A Model Based Method for Automatic Facial Expression Recognition

  • Hans van Kuilenburg
  • Marco Wiering
  • Marten den Uyl
Part of the Lecture Notes in Computer Science book series (LNCS, volume 3720)


Automatic facial expression recognition is a research topic with interesting applications in the field of human-computer interaction, psychology and product marketing. The classification accuracy for an automatic system which uses static images as input is however largely limited by the image quality, lighting conditions and the orientation of the depicted face. These problems can be partially overcome by using a holistic model based approach called the Active Appearance Model. A system will be described that can classify expressions from one of the emotional categories joy, anger, sadness, surprise, fear and disgust with remarkable accuracy. It is also able to detect smaller, local facial features based on minimal muscular movements described by the Facial Action Coding System (FACS). Finally, we show how the system can be used for expression analysis and synthesis.


Facial Expression Face Image Emotional Expression Hide Neuron Input Neuron 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


  1. 1.
    Bishop, C.M.: Neural Networks for Pattern Recognition. Clarendon Press, Oxford (1995)Google Scholar
  2. 2.
    Blanz, V., Vetter, T.: A morphable model for the synthesis of 3D faces. In: Proceedings of the 26th annual conference on Computer graphics and interactive techniques, pp. 187–194. ACM Press/Addison-Wesley Publishing Co. (1999)Google Scholar
  3. 3.
    Cohn, J.F., Kanade, T.: Cohn-Kanade AU-Coded Facial Expression Database. Pittsburgh University (1999)Google Scholar
  4. 4.
    Cootes, T., Taylor, C.: Statistical models of appearance for computer vision. Technical report, University of Manchester, Wolfson Image Analysis Unit, Imaging Science and Biomedical Engineering (2000)Google Scholar
  5. 5.
    Ekman, P.: Universal facial expressions of emotion. California Mental Health Research Digest 8, 151–158 (1970)Google Scholar
  6. 6.
    Ekman, P., Davidson, R.J.: The Nature of Emotion - Fundamental Questions. Oxford University Press, New York (1994)Google Scholar
  7. 7.
    Ekman, P., Friesen, W.: Facial Action Coding System: A Technique for the Measurement of Facial Movement. Consulting Psychologists Press, Palo Alto (1978)Google Scholar
  8. 8.
    Ekman, P., Friesen, W.V., Ellsworth, P.: Emotion in the Human Face. Pergamon Press, Oxford (1972)Google Scholar
  9. 9.
    Ekman, P., Friesen, W.V., Hager, J.C.: The Facial Action Coding System. Weidenfeld & Nicolson, London (2002)Google Scholar
  10. 10.
    Fasel, B., Luettin, J.: Automatic facial expression analysis: A survey. Pattern Recognition 36(1), 259–275 (2003)zbMATHCrossRefGoogle Scholar
  11. 11.
    Frijda, N.: The Emotions. Cambridge University Press & Editions de la Maison des Sciences de l’Homme, Cambridge (1986)Google Scholar
  12. 12.
    Hager, J., Ekman, P.: The essential behavioral science of the face and gesture that computer scientists need to know. In: Proceedings of the International Workshop on Automatic Face and Gesture Recognition, pp. 7–11 (1995)Google Scholar
  13. 13.
    Jackson, J.E.: A User’s Guide to Principal Components. John Wiley and Sons, Chichester (1991)zbMATHCrossRefGoogle Scholar
  14. 14.
    Lebert, E.: Facial expression classification. Experiment report, Sentient Machine Research, Amsterdam, the Netherlands (1997)Google Scholar
  15. 15.
    Lundqvist, D., Flykt, A., Öhman, A.: The Karolinska Directed Emotional Faces - KDEF. CD ROM from Department of Clinical Neuroscience, Psychology section, Karolinska Institutet (1998)Google Scholar
  16. 16.
    Nieber, M.: Global structure of the ActiveModelLib. Software architecture description, Vicar Vision BV, Amsterdam, the Netherlands (2003)Google Scholar
  17. 17.
    Pantic, M.: Automatic analysis of facial expressions: The state of the art. IEEE Transactions on Pattern Analysis and Machine Intelligence 22(12), 1424–1445 (2000)CrossRefGoogle Scholar
  18. 18.
    Russell, J.A., Fernandez-Dols, J.M. (eds.): The Psychology of Facial Expression. Cambridge University Press, Cambridge (1997)Google Scholar
  19. 19.
    Shewchuk, J.R.: Triangle: engineering a 2D quality mesh generator and Delaunay triangulator. In: Applied Computational Geometry, FCRC 1996 Workshop, pp. 203–222 (1996)Google Scholar
  20. 20.
    Sung, K.K., Poggio, T.: Example-based learning for view-based human face detection. IEEE Transactions on Pattern Analysis and Machine Intelligence 20(1), 39–51 (1998)CrossRefGoogle Scholar
  21. 21.
    Van Kuilenburg, H.: Expressions exposed: Model based methods for expression analysis. Master’s thesis, Department of Philosophy, Utrecht University, The Netherlands (2005)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2005

Authors and Affiliations

  • Hans van Kuilenburg
    • 1
  • Marco Wiering
    • 2
  • Marten den Uyl
    • 1
  1. 1.VicarVisionAmsterdamThe Netherlands
  2. 2.Utrecht UniversityUtrechtThe Netherlands

Personalised recommendations