Advertisement

International Journal of Computer Vision

, Volume 76, Issue 3, pp 257–281 | Cite as

Simultaneous Facial Action Tracking and Expression Recognition in the Presence of Head Motion

  • Fadi DornaikaEmail author
  • Franck Davoine
Article

Abstract

The recognition of facial gestures and expressions in image sequences is an important and challenging problem. Most of the existing methods adopt the following paradigm. First, facial actions/features are retrieved from the images, then the facial expression is recognized based on the retrieved temporal parameters. In contrast to this mainstream approach, this paper introduces a new approach allowing the simultaneous retrieval of facial actions and expression using a particle filter adopting multi-class dynamics that are conditioned on the expression. For each frame in the video sequence, our approach is split into two consecutive stages. In the first stage, the 3D head pose is retrieved using a deterministic registration technique based on Online Appearance Models. In the second stage, the facial actions as well as the facial expression are simultaneously retrieved using a stochastic framework based on second-order Markov chains. The proposed fast scheme is either as robust as, or more robust than existing ones in a number of respects. We describe extensive experiments and provide evaluations of performance to show the feasibility and robustness of the proposed approach.

Keywords

Simultaneous tracking and recognition Face and facial feature tracking Facial expression recognition Particle filtering 

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. Ahlberg, J. (2002). An active model for facial feature tracking. EURASIP Journal on Applied Signal Processing, 2002(6), 566–571. CrossRefGoogle Scholar
  2. Bartlett, M., Littlewort, G., Lainscsek, C., Fasel, I., & Movellan, J. (2004). Machine learning methods for fully automatic recognition of facial expressions and facial actions. In IEEE international conference on systems, man and cybernetics. Google Scholar
  3. Bascle, B., & Black, A. (1998). Separability of pose and expression in facial tracking and animation. In Proceedings of the IEEE international conference on computer vision. Google Scholar
  4. Blake, A., & Isard, M. (2000). Active contours. Berlin: Springer. Google Scholar
  5. Blanz, V., & Vetter, T. (2003). Face recognition based on fitting a 3D morphable model. In IEEE transactions on pattern analysis and machine intelligence (pp. 1–12), September 2003. Google Scholar
  6. Chandrasiri, N. P., Naemura, T., & Harashima, H. (2004). Interactive analysis and synthesis of facial expressions based on personal facial expression space. In IEEE international conference on automatic face and gesture recognition. Google Scholar
  7. Cohen, I., Sebe, N., Garg, A., Chen, L., & Huang, T. S. (2003). Facial expression recognition from video sequences: Temporal and static modeling. Computer Vision and Image Understanding, 91(1–2), 160–187. CrossRefGoogle Scholar
  8. Dornaika, F., & Davoine, F. (2005a). Simultaneous facial action tracking and expression recognition using a particle filter. In IEEE international conference on computer vision. Google Scholar
  9. Dornaika, F., & Davoine, F. (2005b). View- and texture-independent facial expression recognition in videos using dynamic programming. In IEEE international conference on image processing. Google Scholar
  10. Dornaika, F., & Davoine, F. (2006). On appearance based face and facial action tracking. IEEE Transactions on Circuits and Systems for Video Technology, 16(9), 2006. Google Scholar
  11. Ekman, P., & Friesen, W. V. (1977). Facial action coding system. Palo Alto: Consulting Psychology Press. Google Scholar
  12. Fasel, B., & Luettin, J. (2003). Automatic facial expression analysis: a survey. Pattern Recognition, 36(1), 259–275. zbMATHCrossRefGoogle Scholar
  13. Gokturk, S. B., Bouguet, J. Y., Tomasi, C., & Girod, B. (2002). Model-based face tracking for view-independent facial expression recognition. In IEEE international conference on automatic face and gesture recognition. Google Scholar
  14. Huang, Y., Huang, T. S., & Niemann, H. (2002). A region-based method for model-free object tracking. In 16th international conference on pattern recognition. Google Scholar
  15. Huber, P. J. (1981). Robust statistics. New York: Wiley. zbMATHGoogle Scholar
  16. Isard, M., & Blake, A. (1998). A mixed-state condensation tracker with automatic model-switching. In Proceedings of the IEEE international conference on computer vision. Google Scholar
  17. Jepson, A. D., Fleet, D. J., & El-Maraghi, T. F. (2003). Robust online appearance models for visual tracking. IEEE Transactions on Pattern Analysis and Machine Intelligence, 25(10), 1296–1311. CrossRefGoogle Scholar
  18. Lee, D. (2005). Effective Gaussian mixture learning for video background subtraction. IEEE Transactions on Pattern Analysis and Machine Intelligence, 27(5), 827–832. CrossRefGoogle Scholar
  19. Liao, W.-K., & Cohen, I. (2005). Classifying facial gestures in presence of head motion. In IEEE workshop on vision for human-computer interaction. Google Scholar
  20. Ljung, L. (1987). System identification: theory for the user. New York: Prentice Hall. zbMATHGoogle Scholar
  21. Lu, L., Zhang, Z., Shum, H. Y., Liu, Z., & Chen, H. (2001). Model- and exemplar-based robust head pose tracking under occlusion and varying expression. In Proceedings of the IEEE workshop on models versus exemplars in computer vision (CVPR’01). Google Scholar
  22. Lu, X., Jain, A. K., & Colbry, D. (2006). Matching 2.5D face scans to 3D models. IEEE Transactions on Pattern Analysis and Machine Intelligence, 28(1), 31–43. CrossRefGoogle Scholar
  23. Lyons, M. J., Budynek, J., & Akamatsu, S. (1999). Automatic classification of single facial images. IEEE Transactions on Pattern Analysis and Machine Intelligence, 21(12), 1357–1362. CrossRefGoogle Scholar
  24. Moreno, F., Tarrida, A., Andrade-Cetto, J., & Sanfeliu, A. (2002). 3D real-time tracking fusing color histograms and stereovision. In IEEE international conference on pattern recognition. Google Scholar
  25. North, B., Blake, A., Isard, M., & Rittscher, J. (2000). Learning and classification of complex dynamics. IEEE Transactions on Pattern Analysis and Machine Intelligence, 22(9), 1016–1034. CrossRefGoogle Scholar
  26. Pantic, M., & Rothkrantz, L. J. M. (2000). Automatic analysis of facial expressions: the state of the art. IEEE Transactions on Pattern Analysis and Machine Intelligence, 22(12), 1424–1445. CrossRefGoogle Scholar
  27. Perez, P., & Vermaak, J. (2005). Bayesian tracking with auxiliary discrete processes. Application to detection and tracking of objects with occlusions. In IEEE ICCV workshop on dynamical vision, Beijing, China. Google Scholar
  28. Tian, Y., Kanade, T., & Cohn, J. F. (2001). Recognizing action units for facial expression analysis. IEEE Transactions on Pattern Analysis and Machine Intelligence, 23, 97–115. CrossRefGoogle Scholar
  29. Wang, Y., Ai, H., Wu, B., & Huang, C. (2004). Real time facial expression recognition with Adaboost. In IEEE international conference on pattern recognition. Google Scholar
  30. Wen, Z., & Huang, T. S. (2003). Capturing subtle facial motions in 3D face tracking. In IEEE international conference on computer vision. Google Scholar
  31. Yacoob, Y., & Davis, L. S. (1996). Recognizing human facial expressions from long image sequences using optical flow. IEEE Transactions on Pattern Analysis and Machine Intelligence, 18(6), 636–642. CrossRefGoogle Scholar
  32. Yilmaz, A., Shafique, K. H., & Shah, M. (2002). Estimation of rigid and non-rigid facial motion using anatomical face model. In IEEE international conference on pattern recognition. Google Scholar
  33. Zhang, Y., & Ji, Q. (2005). Active and dynamic information fusion for facial expression understanding from image sequences. IEEE Transactions on Pattern Analysis and Machine Intelligence, 27(5), 699–714. CrossRefMathSciNetGoogle Scholar
  34. Zhou, S., Krueger, V., & Chellappa, R. (2003). Probabilistic recognition of human faces from video. Computer Vision and Image Understanding, 91(1–2), 214–245. CrossRefGoogle Scholar
  35. Zhou, S., Chellappa, R., & Mogghaddam, B. (2004). Visual tracking and recognition using appearance-adaptive models in particle filters. IEEE Transactions on Image Processing, 13(11), 1473–1490. CrossRefGoogle Scholar

Copyright information

© Springer Science+Business Media, LLC 2007

Authors and Affiliations

  1. 1.Institut Géographique National, Laboratoire MATISSaint-MandéFrance
  2. 2.Heudiasyc Mixed Research Unit, CNRS/UTCCompiegneFrance

Personalised recommendations