A Hidden Markov Model Based Approach for Facial Expression Recognition in Image Sequences

  • Miriam Schmidt
  • Martin Schels
  • Friedhelm Schwenker
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 5998)


One of the important properties of hidden Markov models is the ability to model sequential dependencies. In this study the applicability of hidden Markov models for emotion recognition in image sequences is investigated, i.e. the temporal aspects of facial expressions. The underlying image sequences were taken from the Cohn-Kanade database. Three different features (principal component analysis, orientation histograms and optical flow estimation) from four facial regions of interest (face, mouth, right and left eye) were extracted. The resulting twelve paired combinations of feature and region were used to evaluate hidden Markov models. The best single model with features of principal component analysis in the region face achieved a detection rate of 76.4 %. To improve these results further, two different fusion approaches were evaluated. Thus, the best fusion detection rate in this study was 86.1 %.


Facial Expression Hide Markov Model Gaussian Mixture Model Principle Component Analysis Emotion Recognition 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


  1. 1.
    Bayerl, P., Neumann, H.: Disambiguating visual motion through contextual feedback modulation. Neural Computation 16, 2041–2066 (2004)zbMATHCrossRefGoogle Scholar
  2. 2.
    Bayerl, P., Neumann, H.: A fast biologically inspired algorithm for recurrent motion estimation. IEEE Transactions on Pattern Analysis and Machine Intelligence 29, 246–260 (2007)CrossRefGoogle Scholar
  3. 3.
    Bishop, C.M.: Pattern Recognition and Machine Learning. Springer, New York (2007)Google Scholar
  4. 4.
    Cohen, I., Garg, A., Huang, T.S.: Emotion recognition from facial expressions using multilevel HMM. In: Neural Information Processing Systems (2000)Google Scholar
  5. 5.
    Cohn, J.F., Kanade, T., Tian, Y.: Comprehensive database for facial expression analysis. In: Proceedings of the 4th IEEE International Conference on Automatic Face and Gesture Recognition, pp. 46–53 (2000)Google Scholar
  6. 6.
    Darwin, C.: The Expression of the Emotions in Man and Animals, 1st edn. Oxford University Press Inc., New York (1872)Google Scholar
  7. 7.
    Lin, D.T., Chen, J.: Facial expressions classification with hierarchical radial basis function networks. In: Proceedings of the 6th International Conference on Neural Information Processing, ICONIP, pp. 1202–1207 (1999)Google Scholar
  8. 8.
    Durbin, R., Eddy, S., Krogh, A., Mitchison, G.: Biological Sequence Analysis: Probabilistic Models of Proteins and Nucleic Acids. Cambridge University Press, Cambridge (1998)zbMATHGoogle Scholar
  9. 9.
    Ekman, P., Friesen, W.V.: Facial Action Coding System: A Technique for the Measurement of Facial Movement. Consulting Psychologists Press, Palo Alto (1978)Google Scholar
  10. 10.
    Freeman, W.T., Roth, M.: Orientation histograms for hand gesture recognition. In: International Workshop on Automatic Face and Gesture Recognition, pp. 296–301 (1994)Google Scholar
  11. 11.
    Kuncheva, L.I., Whitaker, C.J.: Measures of diversity in classifier ensembles and their relationship with the ensemble accuracy. Mach. Learn. 51(2), 181–207 (2003)zbMATHCrossRefGoogle Scholar
  12. 12.
    Lang, P.J.: The emotion probe. studies of motivation and attention. The American psychologist 50(5), 372–385 (1995)CrossRefGoogle Scholar
  13. 13.
    Lien, J.J.J., Kanade, T., Cohn, J., Li, C.: A multi-method approach for discriminating between similar facial expressions, including expression intensity estimation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR 1998) (June 1998)Google Scholar
  14. 14.
    Lisetti, C.L., Rumelhart, D.E.: Facial expression recognition using a neural network. In: Proceedings of the Eleventh International Florida Artificial Intelligence Research Society Conference, pp. 328–332. AAAI Press, Menlo Park (1998)Google Scholar
  15. 15.
    Rabiner, L., Juang, B.H.: Fundamentals of Speech Recognition. Prentice Hall PTR, Englewood Cliffs (1993)Google Scholar
  16. 16.
    Rabiner, L.R.: A tutorial on hidden markov models and selected applications in speech recognition. In: Proceedings of the IEEE, pp. 257–286 (1989)Google Scholar
  17. 17.
    Rosenblum, M., Yacoob, Y., Davis, L.: Human expression recognition from motion using a radial basis function network architecture. IEEE Transactions on Neural Networks 7(5), 1121–1138 (1996)CrossRefGoogle Scholar
  18. 18.
    Schwenker, F., Sachs, A., Palm, G., Kestler, H.A.: Orientation histograms for face recognition. In: Schwenker, F., Marinai, S. (eds.) ANNPR 2006. LNCS (LNAI), vol. 4087, pp. 253–259. Springer, Heidelberg (2006)CrossRefGoogle Scholar
  19. 19.
    Turk, M., Pentland, A.: Eigenfaces for recognition. Journal of Cognitive Neuroscience 3(1), 71–86 (1991)CrossRefGoogle Scholar
  20. 20.
    Yeasin, M., Bullot, B., Sharma, R.: From facial expression to level of interest: A spatio-temporal approach. In: IEEE Computer Society Conference on Computer Vision and Pattern Recognition, vol. 2, pp. 922–927 (2004)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2010

Authors and Affiliations

  • Miriam Schmidt
    • 1
  • Martin Schels
    • 1
  • Friedhelm Schwenker
    • 1
  1. 1.Institute of Neural Information ProcessingUniversity of UlmUlmGermany

Personalised recommendations