Skip to main content
SpringerLink
Log in

Recognising facial expressions in video sequences

  • Theoretical Advances
  • Published:
Pattern Analysis and Applications Aims and scope Submit manuscript

Abstract

We introduce a system that processes a sequence of images of a front-facing human face and recognises a set of facial expressions. We use an efficient appearance-based face tracker to locate the face in the image sequence and estimate the deformation of its non-rigid components. The tracker works in real time. It is robust to strong illumination changes and factors out changes in appearance caused by illumination from changes due to face deformation. We adopt a model-based approach for facial expression recognition. In our model, an image of a face is represented by a point in a deformation space. The variability of the classes of images associated with facial expressions is represented by a set of samples which model a low-dimensional manifold in the space of deformations. We introduce a probabilistic procedure based on a nearest-neighbour approach to combine the information provided by the incoming image sequence with the prior information stored in the expression manifold to compute a posterior probability associated with a facial expression. In the experiments conducted we show that this system is able to work in an unconstrained environment with strong changes in illumination and face location. It achieves an 89% recognition rate in a set of 333 sequences from the Cohn–Kanade database.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13

Similar content being viewed by others

References

  1. Flanagan JL, Huang TS (2003) Special issue on human–computer multimodal interface. Proc IEEE, 91(9):1267–1468

    Google Scholar 

  2. Alkoot FM, Kittler J (2002) Moderating k-nn classifiers. Pattern Anal Appl 5:326–332

    Article  MathSciNet  Google Scholar 

  3. Baker S, Matthews I , Schneider J (2004) Automatic construction of active appearance models as an image coding problem. IEEE Trans Pattern Anal Mach Intell 26(10):1380–1384

    Article  Google Scholar 

  4. Barlett MS, Littlewort G, Braathen B, Sejnowki T, Movellán J (2003) A prototype for automatic recognition of spontaneous facial actions. In: Becker S, Obermayer K (eds) Advances in neural information processing systems, vol 15. MIT Press, Cambridge, pp 1271–1278

  5. Barlett MS, Littlewort G, Frank M, Lainscsek C, Fasel IR, Movellán J (2005) Recognizing facial expression: machine learning and application to spontaneous behaviour. In: Proceedings of CVPR, vol 2, pp 568–573

  6. Bascle B, Blake A (1998) Separability of pose and expression in facial tracing and animation. In: Proceedings of international conference on computer vision. IEEE, Washington, pp 323–328

  7. Basili JN (1979) Emotion recognition: the role of facial movement and the relative importance of upper and lower area of the face. J Pers Soc Psychol 37:2049–2059

    Article  Google Scholar 

  8. Belhumeur PN, Hespanha JP, Kriegman DJ (1997) Eigenfaces vs. fisherfaces: recognition using class specific linear projection. IEEE Trans Pattern Anal Mach Intell 19(7):711–720

    Article  Google Scholar 

  9. Belkin M, Niyogi P (2001) Laplacian eigenmaps and spectral techniques for embedding and clustering. In: Advances in neural information processing systems, pp 585–591

  10. Black MJ, Jepson AD (1998) Eigentracking: robust matching and tracking of articulated objects using a view-based representation. Int J Comput Vis 26(1):63–84

    Article  Google Scholar 

  11. Black MJ, Yacoob Y (1997) Recognizing facial expressions in image sequences using local parameterized models of image motion. Int J Comput Vis 25(1):23–48

    Article  Google Scholar 

  12. Blanz V, Vetter T (1999) A morphable model for the synthesis of 3d faces. In: Proceedings of SIGGRAPH. ACM Press, New York, pp 187–194

  13. Buenaposada JM (2001) Buenaposada and Luis Baumela: variations of grey world for face tracking. Image Process Commun 7(3, 4):51–61

    Google Scholar 

  14. Buenaposada JM (2002) Buenaposada and Luis Baumela: real-time tracking and estimation of plane pose. In: Proceedings of international conference on pattern recognition, vol II, QC, Canada, August 2002. IEEE, Washington, pp 697–700

  15. Buenaposada JM, Muñoz E, Baumela L (2006) Efficiently estimating facial expression and illumination in appearance-based tracking. In: Proceedings British machine vision conference, vol I, pp 57–66

  16. Chang Y, Hu C, Turk M (2004) Probabilistic expression analysis on manifolds. In: Proceedings of CVPR, vol 2, pp 520–527

  17. Cohen I, Sebe N, Garg A, Chen LS, Huang TS (2003) Facial expression recognition from video sequences: temporal and static modeling. Comput Vis Image Underst 91:160–187

    Article  Google Scholar 

  18. Cootes T, Edwards GJ, Taylor C (2001) Active appearance models. IEEE Trans Pattern Anal Mach Intell 23(6):681–685

    Article  Google Scholar 

  19. Cowie R, Douglas-Cowie E, Tsapatsoulis N, Votsis G, Kollias S, Fellenz W, Taylor JG (2001) Emotion recognition in human–computer interaction. Signal Process Mag, 18(1):32–80

    Article  Google Scholar 

  20. DeCarlo D, Metaxas D (2000) Optical flow constraints on deformable models with applications to face tracking. Int J Comput Vis 38(2):99–127

    Article  MATH  Google Scholar 

  21. DeCarlo D, Metaxas D(2000) Optical flow constraints on deformable models with applications to face tracking. Int J Comput Vis 38(2):99–127

    Article  MATH  Google Scholar 

  22. Duda RO, Hart PE, Stork DG (2000) Pattern classification. Wiley, New York

    Google Scholar 

  23. Ekman P (1993) Facial expression and emotion. Am Psychol 44:384–392

    Article  Google Scholar 

  24. Ekman P (1994) Strong evidence for universals in facial expressions: a reply to Russell’s mistaken critique. Psychol Bull 115(2):268–287

    Article  Google Scholar 

  25. Essa I, Pentland A (1997) Coding, analysis, interpretation, recognition of facial expressions. IEEE Trans Pattern Anal Mach Intell 19(7):757–763

    Article  Google Scholar 

  26. Fasel B, Luettin J (2003) Automatic facial expression analysis: a survey. Pattern Recognit 36:259–275

    Article  MATH  Google Scholar 

  27. Gao Y, Leung MKH, Hui SC, Tanada MW (2003) Facial expression recognition from line-based caricatures. Trans SMC A 33(3):407–412

    Google Scholar 

  28. Gee A, Cipolla R (1996) Fast visual tracking by temporal consensus. Image Vis Comput 14(2):105–114

    Article  Google Scholar 

  29. Hager G, Belhumeur P (1998) Efficient region tracking with parametric models of geometry and illumination. IEEE Trans Pattern Anal Mach Intell 20(10):1025–1039

    Article  Google Scholar 

  30. He X, Niyogi P (2003) Locality preserving projections. In: Thrun S, Saul L, Schölkopf B (eds) Advances in neural information processing systems, vol 16. MIT Press, Cambridge

  31. Hu C, Ferris R, Turk M (2003) Active wavelet networks for face alignment. In: Proceedings of British machine vision conference

  32. Jongwoo L, Ross D, Ruei-Sung L, Ming-Hsuan Y (2004) Incremental learning for visual tracking. In: Advances in neural information processing systems, vol 17. MIT Press, Cambridge, 793–800

  33. Kanade T, Cohn J, Tian Y-L (2000) Comprehensive database for facial expression analysis. In: Proceedings of international conference on automatic face and gesture recognition, pp 46–53

  34. Lanitis A, Taylor CJ, Cootes TF (1997) Automatic interpretation and coding of face images using flexible models. IEEE Trans Pattern Anal Mach Intell 19(7):743–756

    Article  Google Scholar 

  35. Lien JJ, Kanade T, Cohn JF, Li C (1997) Detection, tracking and classification of action units in facial expression. J Rob Auton Syst 31:131–146

    Article  Google Scholar 

  36. Lyons MJ, Budynek L, Akamatsu S (1999) Automatic classification of single facial images. IEEE Trans Pattern Anal Mach Intell 21(12):1357–1362

    Article  Google Scholar 

  37. Mase K (1991) Recognition of facial expression from optical flow. IEICE Trans E 74(10):3474–3483

    Google Scholar 

  38. Matthews I, Baker S (2004) Active appearance models revisited. Int J Comput Vis 60(2):135–164

    Article  Google Scholar 

  39. McTear MF (2002) Spoken dialogue technology: enabling the conversational user interface. ACM Comput Surv 34(1):90–169

    Article  Google Scholar 

  40. Michel P, El Kaliouby R (2003) Real time facial expression recognition in video using support vector machines. In: Proceedings of international conference on multimodal interfaces. ACM, New York, pp 258–264

  41. Muñoz E, Buenaposada JM, Baumela L (2005) Efficient model-based 3d tracking of deformable objects. In: Proceedings of international conference on computer vision, Beijing, China, vol I, pp 877–882,

  42. Ohya J, Kitamura Y, Takemura H, Ishi H, Kishino F, Terashima N (1996) Virtual space teleconferencing: real-time reproduction of 3d human images. J Vis Commun Image Represent 6(1):1–25

    Article  Google Scholar 

  43. Ojala T, Pietikainen M, Menpp T (2002) Multirresolution gray-scale and rotation invariant texture classification with local binary patterns. IEEE Trans Pattern Anal Mach Intell 24(7):971–987

    Article  Google Scholar 

  44. Oliver N, Pentland A, Bérard F (2000) Lafter: a real-time face and lips tracker with facial expression recognition. Pattern Recognit 33:1369–1382

    Article  Google Scholar 

  45. Panti M, Rothkrantz LJM (2000) Automatic analysis of facial expressions: the state of the art. IEEE Trans Pattern Anal Mach Intell 22(12):1424–1445

    Article  Google Scholar 

  46. Pantic M, Rothkrantz LJM (2000) Expert system for automatic analysis of facial expressions. Image Vis Comput 18(11):881–905

    Article  Google Scholar 

  47. Picard RW (1997) Affective computing. MIT Press, Cambridge

    Google Scholar 

  48. Raducanu B, Graña M, Albizuri FX, d’Anjou A (2004) A probabilistic hit-and-miss transform for face localization. Pattern Anal Appl 7:117–127

    Article  Google Scholar 

  49. Rani P, Liu C, Sarkar N (2006) An empirical study of machine learning techniques for affect recognition in human-robot interaction. Pattern Anal Appl 9:58–69

    Article  Google Scholar 

  50. Romdhani S, Vetter T (2003) Efficient robust and accurate fitting of a 3rd morphable model. In: Proceedings of international conference on computer vision, vol 1, pp 59–66

  51. Rose N (2006) Facial expression classification using gabor and log-gabor filters. In: Proceedings of international conference on automatic face and gesture recognition

  52. Rosenblum M, Yacoob Y, Davis LS (1996) Human expression recognition from motion using radial basis function network architecture. IEEE Trans Neural Netw 7(5):1121–1138

    Article  Google Scholar 

  53. Roweis S, Saul LK (2000) Nonlinear dimensionality reduction by locally linear embedding. Science 290(5500):2323–2326

    Article  Google Scholar 

  54. Rowley H, Baluja S, Kanade T (1998) Neural network-based face detection. IEEE Trans Pattern Anal Mach Intell 20(1):23–28

    Article  Google Scholar 

  55. Shan C, Gong S, McOwan PW (2005) Appearance manifold of facial expression. In: IEEE international workshop on human–computer interaction

  56. Shan C, Gong S, McOwan PW (2006) Dynamic facial expression recognition using a bayesian temporal manifold model. In: Proceedings of British machine vision conference vol 1, pp, 297–306

  57. Tenenbaum JB, de Silva V, Langford JC (2000) A global geometric framework for nonlinear dimensionality reduction. Science 290(5500):2319–2323

    Article  Google Scholar 

  58. Terzopoulos D, Waters K (1993) Analysis and synthesis of facial image sequences using physical and anatomical models. IEEE Trans Pattern Anal Mach Intell 15(6):569–579

    Article  Google Scholar 

  59. Tian Y, Kanade T, Cohn J (2001) Recognizing action units for facial expression analysis. IEEE Trans Pattern Anal Mach Intell 23(2):97–115

    Article  Google Scholar 

  60. Tong Y, Liao W, Ji Q (2006). Inferring facial action units with causal relations. In: Proceedings of CVPR , vol 2, pp 1623–1630

  61. Turk M, Pentland A (1991) Eigenfaces for recognition. J Cogn Neurosci 3(1):71–86

    Article  Google Scholar 

  62. Viola P, Jones MJ (2004) Robust real-time face detection. Int J Comput Vis 57(2):137–154

    Article  Google Scholar 

  63. Wang J, Yin L, Wei X, Sun Y (2006) 3d facial expression recognition based on primitive surface feature distribution. In: Proceedings of CVPR, vol 2, pp 1399–1406

  64. Xu Y, Roy-Chowdhury AK (2007) Integrating motion, illumination and structure in video sequences with applications in illumination-invariant tracking. IEEE Trans Pattern Anal Mach Intell 29(5):793–806

    Article  Google Scholar 

  65. Yacoob Y, Davis LS (1996) Recognizing human facial expressions from long image sequences using optical flow. IEEE Trans Pattern Anal Mach Intell 18(6):636–642

    Article  Google Scholar 

  66. Yeasin M, Bullot B, Sharma R (2004) From facial expression to levels of interest: a spatio-temporal approach. In: Proceedings of CVPR, vol 2, 922–927

  67. Zhang Y, Ji Q (2003) Facial expression understanding in image sequences using dynamic and active information fusion. In: Proceedings of international conference on computer vision, Nice, France

  68. Zhang Y, Ji Q (2005) Active and dynamic information fusion for facial expression understanding from image sequences. IEEE Trans Pattern Anal Mach Intell 27(5):1–16

    Article  MATH  MathSciNet  Google Scholar 

  69. Zhang Z, Lyons M, Schuster M, Akamatsu S (1998) Comparison between geomtry-based and gabor wavelets-based facial expression recognition using multi-layer perceptron. In: Proceedings of international conference on automatic face and gesture recognition, Nara, Japan, pp 454–459

  70. Zhao G, Piettikäinen M (2007) Dynamic texture recognition using local binary patterns with an application to facial expressions. IEEE Trans Pattern Anal Mach Intell 29(6):915–928

    Article  Google Scholar 

Download references

Acknowledgments

The authors gratefully acknowledge funding from the Spanish Ministerio de Educación y Ciencia under contract TRA2005-08529-C02-02 . They also thank the anonymous reviewers for their comments and Jeffrey Cohn and Takeo Kanade for providing the Cohn–Kanade image database.

Author information

Author notes

  1. Enrique Muñoz

    Present address: Facultad Informática, Universidad Complutense de Madrid, Ciudad Universitaria s/n, 28040, Madrid, Spain

Authors and Affiliations

  1. ESCET, Universidad Rey Juan Carlos, C/Tulipán s/n, 28933, Móstoles, Spain

    José M. Buenaposada

  2. Facultad Informática, Universidad Politécnica de Madrid, Campus Montegancedo s/n, 28660, Boadilla del Monte, Spain

    Enrique Muñoz & Luis Baumela

Authors
  1. José M. Buenaposada
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Enrique Muñoz
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Luis Baumela
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Luis Baumela.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Buenaposada, J.M., Muñoz, E. & Baumela, L. Recognising facial expressions in video sequences. Pattern Anal Applic 11, 101–116 (2008). https://doi.org/10.1007/s10044-007-0084-8

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10044-007-0084-8

Keywords

Search

Navigation