Investigating the Dynamics of Facial Expression

  • Jane Reilly
  • John Ghent
  • John McDonald
Part of the Lecture Notes in Computer Science book series (LNCS, volume 4292)


This paper is concerned with capturing the dynamics of facial expression. The dynamics of facial expression can be described as the intensity and timing of a facial expression and its formation. To achieve this we developed a technique that can accurately classify and differentiate between subtle and similar expressions, involving the lower face. This is achieved by using Local Linear Embedding (LLE) to reduce the dimensionality of the dataset and applying Support Vector Machines (SVMs) to classify expressions. We then extended this technique to estimate the dynamics of facial expression formation in terms of intensity and timing.


Support Vector Machine Facial Expression Facial Expression Recognition Facial Action Linear Embedding 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Darwin, C., Ekman, P.: The expression of the emotions in man and animal. The University of Chicago Press, Chicago (1998); 1st edition in 1872, 2nd edition in 1889, 3rd edition with additional commentry by Ekman, P. (1998)Google Scholar
  2. 2.
    Cohn, J., Kanade, T., Moriyama, T., Ambadar, Z., Xiao, J., Gao, J., Imamura, H.: A comparative study of alternative facs coding algorithms. Technical Report CMU-RI-TR-02-06, Robotics Institute, Carnegie Mellon University, Pittsburgh, PA (2001)Google Scholar
  3. 3.
    Ekman, P., Friesen, W., Hager, J.: Facial action coding system. Consulting Psychologists Press, Palo Alto (1978)Google Scholar
  4. 4.
    Cohn, J.F., Schmidt, K., Gross, R., Ekman, P.: Individual differences in facial expression: Stability over time, relation to self-reported emotion, and ability to inform person identification. In: Proceedings of Intel. Conf. On Multimedia and Expo., 2001 (2002)Google Scholar
  5. 5.
    Ambadar, Z., Schooler, J., Cohn, J.: Deciphering the enigmatic face: The importance of facial dynamics to interpreting subtle facial expressions. Psychological Science (2005)Google Scholar
  6. 6.
    Bartlett, M.S., Movellan, J., Littlewort, G., Braathen, B., Frank, M.G., Sejnowski, T.J.: Towards automatic recognition of spontaneous facial actions. In: Ekman, P. (ed.) What the face reveals. Oxford University Press, Oxford (2003)Google Scholar
  7. 7.
    Cohen, I., Sebe, N., Huang, L.C.A.G.: Facial expression recognition from video sequences: Temporal and static modeling. Computer Vision and Image Understanding, 91(1-2) (2003)Google Scholar
  8. 8.
    Ekman, P.: Facial expression of emotion: New findings, new questions (1992)Google Scholar
  9. 9.
    Hadid, A., Pietikdinen, M.: An experimental investigation about the integration of facial dynamics in video-based face recognition. ELCVIA 5, 1–13 (2005)Google Scholar
  10. 10.
    Littlewort, G., Bartlett, M.S., Fasel, I., Susskind, J., Movellan, J.: Dynamics of facial expression extracted automatically from video. Computer vision and Image understanding (2006)Google Scholar
  11. 11.
    Ekman, P., Friesen, W., Hager, J.: Facial Action Coding System Manual (2002)Google Scholar
  12. 12.
    Ghent, J., McDonald, J., Harper, J.: Statistical model for expression generation using the facial action coding system. Technical Report NUIM-CS-TR-2003-01, NUI Maynooth (2003)Google Scholar
  13. 13.
    Saul, L.K., Roweis, S.T.: An introduction to locally linear embedding (2001),
  14. 14.
    Schmidt, K., Cohn, J.: Dynamics of facial expression: Normative characteristics and individual differences. In: IEEE International Conference on Multimedia and Expo (ICME 2001), pp. 728–731 (2001)Google Scholar
  15. 15.
    Cohn, J.: Automated analysis of the configuration and timing of facial expression. In: Ekman, P., Rosenberg, E. (eds.) Afterword of What the face reveals (2nd edn.): Basic and applied studies of spontaneous expression using the Facial Action Coding System (FACS) (2005)Google Scholar
  16. 16.
    Pantic, M., Patras, I.: Dynamics of facial expression: Recognition of facial actions and their temporal segments from face profile image sequences. SMC-B 36, 433–449 (2006)Google Scholar
  17. 17.
    Zhang, Y., Ji, Q.: Facial expression understanding in image sequences using dynamic and active visual information fusion. In: ICCV, pp. 113–118 (2003)Google Scholar
  18. 18.
    Donato, G., Bartlett, M., Hager, J., Ekman, P., Sejnowski, T.: Classifying facial actions. IEEE Transactions on Pattern Analysis and Machine Intelligence 21(10), 974–989 (1999)CrossRefGoogle Scholar
  19. 19.
    Bartlett, M.S., Littlewort, G., Frank, M., lainscsek, C., Fasel, I., Movellan, J.: Recognising facial expression: Machine learning and application to spontaneous behaviour. In: IEEE International conference on computer vision and pattern recognition (2005)Google Scholar
  20. 20.
    Bartlett, M.S., Littlewort, G., Lainscsek, C., Fasel, I., Movellan, J.: Machine learning methods for fully automatic recognition of facial expressions and facial actions. In: IEEE International conference on systems, man and cybernetics, pp. 592–597 (2004)Google Scholar
  21. 21.
    Bartlett, M.S., Braathen, B., Movellan, J.: Automatic analysis of spontaneous facial behavior: A final report. MPLAB-TR-2001-06, Institute for Neural Computation, University of California, San Diego (2001)Google Scholar
  22. 22.
    Bartlett, M.S., Littlewort, G., Fasel, I., Movellan, J.: Real-time face detection and facial expression recognition: Development and applications to human computer interaction (2003)Google Scholar
  23. 23.
    Bartlett, M.S., Littlewort, G., Braathen, B., Sejnowski, T., Movellan, J.: A prototype for automatic recognition of spontaneous facial actions (2003)Google Scholar
  24. 24.
    Bartlet, M., Littlewort, G., Lainscsek, C., Fasel, I., Frank, M., Movellan, J.: Fully automatic facial action recognition in spontaneous behavior bartlett (2006)Google Scholar
  25. 25.
    Goneid, A., el Kaliouby, R.: Facial feature analysis of spontaneous facial expression. In: Proceedings of the 10th International AI Applications Conference (2002)Google Scholar
  26. 26.
    Fasel, B., Luettin, J.: Automatic facial expression analysis: A survey. Pattern Recognition 36(1), 259–275 (2003)MATHCrossRefGoogle Scholar
  27. 27.
    Saul, L.K., Roweis, S.T.: Think globally, fit locally: unsupervised learning of low dimensional manifolds. Journal of Machine Learning Research 4 (2003)Google Scholar
  28. 28.
    Roweis, S.T., Saul, L.K.: Nonlinear dimensionality reduction by local linear embedding. Science 290, 2323–2326 (2000)CrossRefGoogle Scholar
  29. 29.
    Chang, Y., Yeung, D.Y.: Robust local linear embedding. Technical report, Department of Computer Science, Hong Kong University of Science and Technology, HKUST-CS05-12 (2005)Google Scholar
  30. 30.
    Hadid, A., PietikAanien, M.: Efficient locally linear embeddings of imperfect manifolds. In: Proceedings of the Third International Conference on Machine Learning and Data Mining in Pattern Recognition, Leipzig, Germany, pp. 188–201 (2003)Google Scholar
  31. 31.
    Ridder, D.D., Kouropteva, O., Okun, O., PietikAanien, M., Duin, R.P.W.: Supervised locally linear embedding. In: ICANN 2003, pp. 333–341 (2003)Google Scholar
  32. 32.
    Kouropteva, O., Okun, O., Hadid, A., Soriano, M., Marcos, S., PietikAainen, M.: Beyond locally linear embedding algorithm. Technical report, Department of Electrical and Information Engerring, University of Oulu, Oulu, Finland, MVG-01-2002 (2002)Google Scholar
  33. 33.
    Ridder, D.D., Duin, R.P.W.: Locally linear embedding for classification. Technical report, Pattern Recognition Group, Department of Imaging Science and Technology, Delft University of Technology, Delft, The Netherlands, PH-2002-01 (2002)Google Scholar
  34. 34.
    Campbell, C.: kernel methods: A survey of current techniques. Neurocomputing 48, 63–84 (2002)MATHCrossRefGoogle Scholar
  35. 35.
    Rogers, S., Williams, R.D., Campbell, C.: Class prediction with Microarray Datasets. In: BioInformatics with computational intelligence paradigms. Springer, Heidelberg (2004)Google Scholar
  36. 36.
    Rogers, S.: Machine Learning Techniques for Microarray Analysis. Faculty of engineering mathematics, University of Bristol (2004)Google Scholar
  37. 37.
    ScholKopf, B., Smola, A.J.: Learning with Kernels: Support Vector Machines, Regularization, Optimization, and Beyond. MIT Press, Cambridge (2002)Google Scholar
  38. 38.
    Ghent, J.: A Computational Model of Facial Expression. PhD thesis, National University of Ireland Maynooth, Co., kildare, Ireland (2005)Google Scholar
  39. 39.
    Cohn, J., Kanade: Cohn-kanade au-coded facial expression database. Technical report, Pittsburgh University (1999)Google Scholar
  40. 40.
    Gower, J.C.: Generalised procrustes analysis. Psychometrika 40, 33–50 (1975)MATHCrossRefMathSciNetGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2006

Authors and Affiliations

  • Jane Reilly
    • 1
  • John Ghent
    • 1
  • John McDonald
    • 1
  1. 1.National University of IrelandMaynooth

Personalised recommendations