zbMATH — the first resource for mathematics

Texture and shape information fusion for facial expression and facial action unit recognition. (English) Zbl 1132.68640
Summary: A novel method based on fusion of texture and shape information is proposed for facial expression and Facial Action Unit (FAU) recognition from video sequences. Regarding facial expression recognition, a subspace method based on Discriminant Non-negative Matrix Factorization (DNMF) is applied to the images, thus extracting the texture information. In order to extract the shape information, the system firstly extracts the deformed Candide facial grid that corresponds to the facial expression depicted in the video sequence. A Support Vector Machine (SVM) system designed on an Euclidean space, defined over a novel metric between grids, is used for the classification of the shape information. Regarding FAU recognition, the texture extraction method is applied on the differences images of the video sequence, calculated taking under consideration the neutral and the expressive frame. An SVM system is used for FAU classification from the shape information. This time, the shape information consists of the grid node coordinate displacements between the neutral and the expressed facial expression frame. The fusion of texture and shape information is performed using various approaches, among which are SVMs and Median Radial Basis Functions, in order to detect the facial expression and the set of present FAUs. The accuracy achieved using the Cohn-Kanade database is 92.3% when recognizing the seven basic facial expressions (anger, disgust, fear, happiness, sadness, surprise and neutral), and 92.1% when recognizing the 17 FAUs that are responsible for facial expression development.
68T10 Pattern recognition, speech recognition
Full Text: DOI
[1] Pentland, A.; Choudhury, T., Face recognition for smart environments, IEEE comput., 33, 2, 50-55, (2000)
[2] Pantic, M.; Rothkrantz, L., Toward an affect-sensitive multimodal human – computer interaction, Proc. IEEE, 91, 9, 1370-1390, (2003)
[3] Ekman, P.; Friesen, W.V., Emotion in the human face, (1975), Prentice-Hall New Jersey
[4] T. Kanade, J. Cohn, Y. Tian, Comprehensive database for facial expression analysis, in: Proceedings of IEEE International Conference on Face and Gesture Recognition, 2000, pp. 46-53.
[5] Pantic, M.; Rothkrantz, L.J.M., Expert system for automatic analysis of facial expressions, Image vision comput., 18, 11, 881-905, (2000)
[6] Pantic, M.; Rothkrantz, L.J.M., Automatic analysis of facial expressions: the state of the art, IEEE trans. pattern anal. Mach. intell., 22, 12, 1424-1445, (2000)
[7] Fasel, B.; Luettin, J., Automatic facial expression analysis: a survey, Pattern recognition, 36, 1, 259-275, (2003) · Zbl 1007.68947
[8] Gu, H.; Zhang, Y.; Ji, Q., Task oriented facial behavior recognition with selective sensing, Cviu, 100, 3, 385-415, (2005)
[9] Bartlett, M.S.; Littlewort, G.; Fasel, I.; Movellan, JR., Real time face detection and facial expression recognition: development and applications to human computer interaction, (), 53-58
[10] G. Littlewort, M. Bartlett, I. Fasel, J. Susskind, J. Movellan, Dynamics of facial expression extracted automatically from video, in: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, Workshop on Face Processing in Video, 2004.
[11] C. Shan, S. Gong, P. McOwan, Appearance manifold of facial expression, in: IEEE International Workshop on Human-Computer Interaction, 2005.
[12] Y. Kosaka, K. Kotani, Facial expression analysis by kernel eigenspace method based on class features (KEMC) using non-linear basis for separation of expression-classes, in: Proceedings of IEEE International Conference on Image Processing (ICIP 2004), Singapore, 2004.
[13] Dubuisson, S.; Davoine, F.; Masson, M., A solution for facial expression representation and recognition, Signal process. image commun., 17, 9, 657-673, (2002)
[14] Thomaz, C.; Feitosa, D.G.A., Using mixture covariance matrices to improve face and facial expression recognitions, Pattern recognition lett., 24, 13, 2159-2165, (2003)
[15] Ma, L.; Khorasani, K., Facial expression recognition using constructive feedforward neural networks, IEEE trans. syst. man cybern. B: cybern., 34, 3, 1588-1595, (2004)
[16] Gizatdinova, Y.; Surakka, V., Feature-based detection of facial landmarks from neutral and expressive facial images, IEEE trans. pattern anal. Mach. intell., 28, 1, 135-139, (2006)
[17] Matsugu, M.; Mori, K.; Mitari, Y.; Kaneda, Y., Subject independent facial expression recognition with robust face detection using a convolutional neural network, Neural networks, 16, 5-6, 555-559, (2003)
[18] Liang, D.; Yang, J.; Zheng, Z.; Chang, Y., A facial expression recognition system based on supervised local linear embedding, Pattern recognition lett., 26, 2374-2389, (2005)
[19] T. Yabui, Y. Kenmochi, K. Kotani, Facial expression analysis from 3d range images; comparison with the analysis from 2d images and their integration, in: Proceedings of IEEE International Conference on Image Processing, 2003, pp. 879-882.
[20] Chen, X.-W.; Huang, T., Facial expression recognition: a clustering-based approach, Pattern recognition lett., 24, 9-10, 1295-1302, (2003) · Zbl 1048.68075
[21] Y.L. Tian, T. Kanade, J. Cohn, Evaluation of Gabor wavelet-based facial action unit recognition in image sequences of increasing complexity, in: Proceedings of the Fifth IEEE International Conference on Automatic Face and Gesture Recognition, 2002, pp. 229-234.
[22] Roweis, S.; Saul, L., Nonlinear dimensionality reduction by locally linear embedding, Science, 290, 2323-2326, (2000)
[23] Gao, Y.; Leung, M.; Hui, S.; Tananda, M., Facial expression recognition from line-based caricatures, IEEE trans. systems man cybern. A: syst. hum., 33, 3, 407-412, (2003)
[24] Zhang, Y.; Ji, Q., Active and dynamic information fusion for facial expression understanding from image sequences, IEEE trans. pattern anal. Mach. intell., 27, 5, 699-714, (2005)
[25] P. Michel, R. Kaliouby, Real time facial expression recognition in video using support vector machines, in: Proceedings of Fifth International Conference on Multimodal Interfaces, Vancouver, British Columbia, Canada, 2003, pp. 258-264.
[26] F. Dornaika, F. Davoine, View- and texture-independent facial expression recognition in videos using dynamic programming, in: Proceedings of IEEE International Conference on Image Processing, Genova, Italy, 2005.
[27] F. Dornaika, F. Davoine, Online appearance-based face and facial feature tracking, in: Proceedings of the 17th International Conference on Pattern Recognition, Cambridge, UK, 2004.
[28] Abboud, B.; Davoine, F.; Dang, M., Facial expression recognition and synthesis based on an appearance model, Signal process. image commun., 19, 8, 723-740, (2004)
[29] M. Malciu, F. Preteux, Tracking facial features in video sequences using a deformable model-based approach, in: Proceedings of the SPIE, vol. 4121, 2000, pp. 51-62.
[30] Zhu, Y.; Silva, L.C.D.; Ko, C.C., Using moment invariants and HMM in facial expression recognition, Pattern recognition lett., 23, 1-3, 83-91, (2002) · Zbl 0996.68167
[31] Chang, Y.; Hu, C.; Feris, R.; Turk, M., Manifold based analysis of facial expression, Image vision comput., 24, 605-614, (2006)
[32] Guo, G.; Dyer, C.R., Learning from examples in the small sample case: face expression recognition, IEEE trans. syst. man cybern. B: cybern., 35, 3, 477-488, (2005)
[33] S.B. Gokturk, C. Tomasi, B. Girod, J.-Y. Bouguet, Model-based face tracking for view-independent facial expression recognition, in: Proceedings of the Fifth IEEE International Conference on Automatic Face and Gesture Recognition, Cambridge, UK, 2002, pp. 287-293.
[34] Kotsia, I.; Pitas, I., Facial expression recognition in image sequences using geometric deformation features and support vector machines, IEEE trans. image process., 16, 1, 172-187, (2007)
[35] D. Lee, H. Seung, Algorithms for non-negative matrix factorization, in: NIPS, 2000, pp. 556-562.
[36] Zafeiriou, S.; Tefas, A.; Buciu, I.; Pitas, I., Exploiting discriminant information in non-negative matrix factorization with application to frontal face verification, IEEE trans. neural networks, 17, 3, 683-695, (2006)
[37] Donato, G.; Bartlett, M.; Hager, J.; Ekman, P.; Sejnowski, T., Classifying facial actions, IEEE trans. pattern anal. Mach. intell., 21, 10, 974-989, (1999)
[38] I. Kotsia, S. Zafeiriou, N. Nikolaidis, I. Pitas, Multiclass support vector machines and metric multidimensional scaling for facial expression recognition, 2007.
[39] Buciu, I.; Pitas, I., DNMF modeling of neural receptive fields involved in human facial expression perception, J. visual commun. image representation, 17, 5, 958-969, (2006)
[40] Li, S.; Hou, X.; Zhang, H., Learning spatially localized, part-based representation, (), 207-212
[41] Cox, M.F.; Cox, M.A.A., Multidimensional scaling, (2001), Chapman & Hall London · Zbl 1004.91067
[42] Borg, I.; Groenen, P., Modern multidimensional scaling: theory and applications, (1997), Springer New York · Zbl 0862.62052
[43] Torgerson, W.S., Theory and methods of scaling, (1958), Wiley New York
[44] S. Krinidis, I. Pitas, Statistical analysis of facial expressions for facial expression synthesis, IEEE Trans. Pattern Anal. Mach. Intell., submitted for publication. · Zbl 1371.94199
[45] M. Collins, R.E. Schapire, Y. Singer, Logistic regression, adaboost and bregman distances, Comput. Learn. Theory (2000) 158-169. · Zbl 0998.68123
[46] J.Y. Bouguet, Pyramidal implementation of the Lucas-Kanade feature tracker, Technical Report, Intel Corporation, Microprocessor Research Labs, 1999.
[47] Wiskott, L.; Fellous, J.M.; Kuiger, N.; von der Malsburg, C., Face recognition by elastic bunch graph matching, IEEE trans. pattern anal. Mach. intell., 19, 7, 775-779, (1997)
[48] Pekalska, E.; Paclik, P.; Duin, R., A generalized kernel approach to dissimilarity-based classification, J. Mach. learn. res., 2, 175-211, (2001) · Zbl 1037.68127
[49] Drucker, H.; Donghui, W.; Vapnik, V., Support vector machines for spam categorization, IEEE trans. neural networks, 10, 5, 1048-1054, (1999)
[50] Ganapathiraju, A.; Hamaker, J.; Picone, J., Applications of support vector machines to speech recognition, IEEE trans. signal process., 52, 8, 2348-2355, (2004)
[51] Pontil, M.; Verri, A., Support vector machines for 3D object recognition, IEEE trans. pattern anal. Mach. intell., 20, 6, 637-646, (1998)
[52] Tefas, A.; Kotropoulos, C.; Pitas, I., Using support vector machines to enhance the performance of elastic graph matching for frontal face authentication, IEEE trans. pattern anal. Mach. intell., 23, 7, 735-746, (2001)
[53] Vapnik, V., Statistical learning theory, (1998), Wiley New York · Zbl 0935.62007
[54] Hsu, C.W.; Lin, C.J., A comparison of methods for multiclass support vector machines, IEEE trans. neural networks, 13, 2, 415-425, (2002)
[55] J. Weston, C. Watkins, Multi-class Support Vector Machines, Technical Report CSD-TR-98-04, 2004.
[56] Weston, J.; Watkins, C., Multi-class support vector machines, ()
[57] C.J.C. Burges, A tutorial on support vector machines for pattern recognition, Data Min. Knowl. Discovery 2(2).
[58] Snelick, R.; Uludag, U.; Mink, A.; Indovina, M.; Jain, A., Large-scale evaluation of multimodal biometric authentication using state-of-the-art systems, IEEE trans. pattern anal. Mach. intell., 27, 3, 450-455, (2005)
[59] Bors, A.G.; Pitas, I., Median radial basis function neural network, IEEE trans. neural networks, 7, 1351-1364, (1996)
[60] Chatzis, V.; Bors, A.; Pitas, I., Multimodal decision-level fusion for person authentication, IEEE trans. syst. man cybern. A, 29, 6, 674-680, (1999)
[61] C.C. Hung, Y. Kim, T. Coleman, A comparative study of radial basis function neural networks and wavelet neural networks in classification of remotely sensed data, in: Fourth Biannual World Automation Congress (WAC’2002), Orlando, FL, USA, 2002.
[62] Schalkof, R.J., Pattern recognition: statistical, structural and neural approaches, (1992), Wiley New York
[63] Scholkopf, B.; Sung, K.-K.; Burges, C.J.C.; Girosi, F.; Niyogi, P.; Poggio, T.; Vapnik, V., Comparing support vector machines with Gaussian kernels to radial basis function classifiers, IEEE trans. signal process., 45, 11, 2758-2765, (1997)
[64] Pitas, I.; Venetsanopoulos, A.N., Nonlinear digital filters: principles and applications, (1990), Kluwer Academic Dordrecht · Zbl 0719.93080
[65] Cohen, I.; Sebe, N.; Garg, S.; Chen, L.S.; Huanga, T.S., Facial expression recognition from video sequences: temporal and static modelling, Comput. vision image understanding, 91, 160-187, (2003)
[66] Aleksic, S.; Katsaggelos, K., Automatic facial expression recognition using facial animation parameters and multi-stream hmms, IEEE trans. inf. forensics secur., 1, 1, 3-11, (2006)
[67] Yeasin, M.; Bullot, B.; Sharma, R., Recognition of facial expressions and measurement of levels of interest from video, IEEE transactions on multimedia, 8, 3, 500-508, (2006)
[68] Zhao, G.; Pietikainen, M., Dynamic texture recognition using local binary patterns with an application to facial expressions, IEEE trans. pattern anal. Mach. intell., 29, 6, 915-928, (2007)
This reference list is based on information provided by the publisher or from digital mathematics libraries. Its items are heuristically matched to zbMATH identifiers and may contain data conversion errors. It attempts to reflect the references listed in the original paper as accurately as possible without claiming the completeness or perfect precision of the matching.