Abstract
We consider two novel representations and feature extraction schemes for automatic recognition of emotion related facial expressions. In one scheme facial landmark points are tracked over successive video frames using an effective detector and tracker to extract landmark trajectories. Features are extracted from landmark trajectories using Independent Component Analysis (ICA) method. In the alternative scheme, the evolution of the emotion expression on the face is captured by stacking normalized and aligned faces into a spatiotemporal face cube. Emotion descriptors are then 3D Discrete Cosine Transform (DCT) features from this prism or DCT & ICA features. Several classifier configurations are used and their performance determined in detecting the 6 basic emotions. Decision fusion applied to classifiers improved the recognition performance of best classifier by 9 percentage points. The proposed method was evaluated user independently on the Cohn-Kanade facial expression database and a state-of-the-art 95.34 % recognition performance is achieved.
Keywords
Download to read the full chapter text
Chapter PDF
Similar content being viewed by others
References
Vinciarelli, A., Pantic, M., Bourlard, H.: Social signal processing: Survey of an emerging domain. Image and Vision Computing 27, 1743–1759 (2009)
Gatica-Perez, D.: Automatic nonverbal analysis of social interaction in small groups: A review. Image and Vision Computing 27, 1775–1787 (2009)
Sebe, N., Lew, M., Sun, Y., Cohen, I., Gevers, T., Huang, T.: Authentic facial expression analysis. Image and Vision Computing 25, 1856–1863 (2007)
Zeng, Z., Pantic, M., Roisman, G., Huang, T.: A survey of affect recognition methods: Audio, visual, and spontaneous expressions. IEEE Transactions on Pattern Analysis and Machine Intelligence 31, 39–58 (2009)
Hupont, I., Cerezo, E., Baldassarri, S.: Facial emotional classifier for natural interaction, vol. 7, pp. 1–12 (2008)
Zhao, G., Pietikainen, M.: Dynamic texture recognition using local binary patterns with an application to facial expressions. IEEE Transactions on Pattern Analysis and Machine Intelligence 29, 915–928 (2007)
Zhang, Y., Ji, Q.: Active and dynamic information fusion for facial expression understanding from image sequences. IEEE Transactions on Pattern Analysis and Machine Intelligence 27, 699–714 (2005)
Wang, T., James Lien, J.J.: Facial expression recognition system based on rigid and non-rigid motion separation and 3D pose estimation. Pattern Recognition 42, 962–977 (2009)
Pantic, M., Patras, I.: Dynamics of facial expression: Recognition of facial actions and their temporal segments from face profile image sequences. IEEE Transactions on Systems, Man, and Cybernetics, Part B: Cybernetics 36, 433–449 (2006)
Yang, P., Liu, Q., Metaxas, D.N.: Boosting encoded dynamic features for facial expression recognition. Pattern Recognition Letters 30, 132–139 (2009)
Bartlett, M.S., Littlewort, G., Frank, M., Lainscsek, C., Fasel, I., Movellan, J.: Recognizing facial expression: Machine learning and application to spontaneous behavior. In: IEEE Computer Society Conference on Computer Vision and Pattern Recognition, vol. 2, pp. 568–573 (2005)
Dornaika, F., Davoine, F.: Simultaneous facial action tracking and expression recognition in the presence of head motion. International Journal of Computer Vision 76, 257–281 (2008)
Tsalakanidou, F., Malassiotis, S.: Real-time 2D+3D facial action and expression recognition. Pattern Recognition 43, 1763–1775 (2010)
Ekman, P., Friesen, W.: Facial Action Coding System: A Technique for the Mea-surement of Facial Movement. Consulting Psychologists Press, Palo Alto (1978)
Tian, Y.L.: Evaluation of face resolution for expression analysis. In: CVPR Workshop on Face and Video, p. 82 (2004)
Shan, C., Gong, S., McOwan, P.W.: Facial expression recognition based on local binary patterns a comprehensive study. Image and Vision Computing 27 (2009)
Zhou, F., De la Torre, F., Cohn, J.: Unsupervised discovery of facial events. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (2010)
Bartlett, M.S., Littlewort, G., Frank, M.G., Lainscsek, C., Fasel, I.R., Movellan, J.R.: Automatic recognition of facial actions in spontaneous expressions. Journal of Multimedia 1, 22–35 (2006)
Kanade, T., Cohn, J., Tian, Y.L.: Comprehensive database for facial expression analysis. In: Proceedings of the 4th IEEE International Conference on Automatic Face and Gesture Recognition, FG 2000, pp. 46–53 (2000)
Zeng, Z., Fu, Y., Roisman, G.I., Wen, Z., Hu, Y., Huang, T.S.: Spontaneous emotional facial expression detection. Journal of Multimedia 1, 1–8 (2006)
Zeng, Z., Hu, Y., Fu, Y., Huang, T.S., Roisman, G.I., Wen, Z.: Audio-visual emotion recognition in adult attachment interview. In: Proceedings of the 8th International Conference on Multimodal Interfaces, pp. 139–145. ACM, New York (2006)
Akakin, H.C., Sankur, B.: Analysis of Head and Facial Gestures Using Facial Landmark Trajectories. In: Fierrez, J., Ortega-Garcia, J., Esposito, A., Drygajlo, A., Faundez-Zanuy, M. (eds.) BioID_MultiComm2009. LNCS, vol. 5707, pp. 105–113. Springer, Heidelberg (2009)
Martinez, A.M.: Matching expression variant faces. Vision Research, 1047–1060 (2003)
Oja, E.: Independent component analysis: algorithms and applications. Neural Networks 13, 411–430 (2000)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2012 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Akakın, H.Ç., Sankur, B. (2012). Spatiotemporal Features for Effective Facial Expression Recognition. In: Kutulakos, K.N. (eds) Trends and Topics in Computer Vision. ECCV 2010. Lecture Notes in Computer Science, vol 6553. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-35749-7_16
Download citation
DOI: https://doi.org/10.1007/978-3-642-35749-7_16
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-642-35748-0
Online ISBN: 978-3-642-35749-7
eBook Packages: Computer ScienceComputer Science (R0)