Abstract
The analysis of facial expression temporal dynamics is of great importance for many real-world applications. Being able to automatically analyse facial muscle actions (Action Units, AUs) in terms of recognising their neutral, onset, apex and offset phases would greatly benefit application areas as diverse as medicine, gaming and security. The base system in this paper uses Support Vector Machines (SVMs) and a set of simple geometrical features derived from automatically detected and tracked facial feature point data to segment a facial action into its temporal phases. We propose here two methods to improve on this base system in terms of classification accuracy. The first technique describes the original time-independent set of features over a period of time using polynomial parametrisation. The second technique replaces the SVM with a hybrid SVM/Hidden Markov Model (HMM) classifier to model time in the classifier. Our results show that both techniques contribute to an improved classification accuracy. Modeling the temporal dynamics by the hybrid SVM-HMM classifier attained a statistically significant increase of recall and precision by 4.5% and 7.0%, respectively.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
Ekman, P., Friesen, W.V., Hager, J.C.: Facial Action Coding System. A Human Face, Salt Lake City (2002)
Cohn, J.F.: Foundations of human computing: Facial expression and emotion. In: Proc. ACM Int’l Conf. Multimodal Interfaces, vol. 1, pp. 610–616 (2006)
Tian, Y., Kanade, T., Cohn, J.: Recognizing action units for facial expression analysis. IEEE Trans. Pattern Analysis and Machine Intelligence 23(2), 97–115 (2001)
Valstar, M.F., Pantic, M.: Fully automatic facial action unit detection and temporal analysis. In: Proc. IEEE Conference on Computer Vision and Pattern Recognition, p. 149 (2006)
Bartlett, M.S., Littlewort, G., Lainscsek, C., Fasel, I., Movellan, J.: Machine learning methods for fully automatic recognition of facial expressions and actions. In: Proc. IEEE Int’l Conf. on Systems, Man and Cybernetics, vol. 1, pp. 592–597 (2004)
Pantic, M., Rothkrantz, L.J.M.: Toward an affect-sensitive multimodal human-computer interaction. Proc. IEEE 91(9), 1370–1390 (2003)
Tian, Y.L., Kanade, T., Cohn, J.F.: Handbook of Face Recognition. Springer, Heidelberg (2005)
Cohn, J.F., Schmidt, K.L.: The timing of facial motion in posed and spontaneous smiles. J. Wavelets, Multi-resolution and Information Processing 2(2), 121–132 (2004)
Bassili, J.N.: Facial motion in the perception of faces and of emotional expression. J. Experimental Psychology 4(3), 373–379 (1978)
Hess, U., Kleck, R.E.: Differentiating emotion elicited and deliberate emotional facial expressions. European J. of Social Psychology 20(5), 369–385 (1990)
de Williams, A.C.: Facial expression of pain: An evolutionary account. Behavioral and Brain Sciences 25(4), 439–488 (2006)
Ekman, P.: Darwin, deception, and facial expression. Annals of New York Ac. of sciences 1000, 105–221 (2003)
Ekman, P., Rosenberg, E.L.: What the face reveals: Basic and applied studies of spontaneous expression using the Facial Action Coding System. Oxford University Press, Oxford (2005)
Valstar, M.F., Pantic, M., Ambadar, Z., Cohn, J.F.: Spontaneous vs. posed facial behavior: automatic analysis of brow actions. In: Proc. ACM Intl. conf. on Multimodal Interfaces, pp. 162–170 (2006)
Fasel, I.R., Fortenberry, B., Movellan, J.R.: A generative framework for real time object detection and classification. Comp. Vision, and Image Understanding 98(1), 181–210 (2005)
Viola, P., Jones, M.: Robust real-time object detection. Technical report CRL 200001/01 (2001)
Friedman, J., Hastie, T., Tibshirani, R.: Additive logistic regression: A statistical view of boosting. The Annals of Statistics 28(2), 337–374 (2000)
Vukdadinovic, D., Pantic, M.: Fully automatic facial feature point detection using gabor feature based boosted features. In: Proc. IEEE Int’l Conf. on Systems, Man and Cybernetics, pp. 1692–1698 (2005)
Patras, I., Pantic, M.: Particle filtering with factorized likelihoods for tracking facial features. In: Proc. Int’l Conf. Automatic Face & Gesture Recognition, pp. 97–102 (2004)
Patras, I., Pantic, M.: Tracking deformable motion. In: Proc. Int’l Conf. Systems, Man and Cybernetics, pp. 1066–1071 (2005)
Bourlard, H., Morgan, N.: Hybrid hmm/ann systemds for speech recognition: Overview and new research directions. In: Giles, C.L., Gori, M. (eds.) Adaptive Processing of Sequences and Data Structures. LNCS (LNAI), vol. 1387, pp. 389–417. Springer, Heidelberg (1998)
Kruger, S.E., Schaffner, M., Katz, M., Andelic, E., Wendemuth, A.: Speech recognition with support vector machines in a hybrid system. In: Interspeech, pp. 993–996 (2005)
Platt, J.: Probabilistic outputs for support vector machines and comparison to regularized likelihood methods, pp. 61–74. Cambridge, MA (2000)
Pantic, M., Valstar, M.F., Rademaker, R., Maat, L.: Web-based database for facial expression analysis. In: Proc. Int’l Conf. Multimedia & Expo, pp. 317–321 (2005)
Kanade, T., Cohn, J., Tian, Y.: Comprehensive database for facial expression analysis. In: IEEE Int’l Conf. on Automatic Face and Gesture Recognition, pp. 46–53 (2000)
Author information
Authors and Affiliations
Editor information
Rights and permissions
Copyright information
© 2007 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Valstar, M.F., Pantic, M. (2007). Combined Support Vector Machines and Hidden Markov Models for Modeling Facial Action Temporal Dynamics. In: Lew, M., Sebe, N., Huang, T.S., Bakker, E.M. (eds) Human–Computer Interaction. HCI 2007. Lecture Notes in Computer Science, vol 4796. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-540-75773-3_13
Download citation
DOI: https://doi.org/10.1007/978-3-540-75773-3_13
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-540-75772-6
Online ISBN: 978-3-540-75773-3
eBook Packages: Computer ScienceComputer Science (R0)