Evaluation of Expression Recognition Techniques

  • Ira Cohen
  • Nicu Sebe
  • Yafei Sun
  • Michael S. Lew
  • Thomas S. Huang
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 2728)


The most expressive way humans display emotions is through facial expressions. In this work we report on several advances we have made in building a system for classification of facial expressions from continuous video input. We introduce and test different Bayesian network classifiers for classifying expressions from video. In particular we use Naive-Bayes classifiers and to learn the dependencies among different facial motion features we use Tree-Augmented Naive Bayes (TAN) classifiers. We also investigate a neural network approach. Further, we propose an architecture of hidden Markov models (HMMs) for automatically segmenting and recognizing human facial expression from video sequences. We explore both person-dependent and person-independent recognition of expressions and compare the different methods.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. [1]
    C.K. Chow and C.N. Liu. Approximating discrete probability distribution with dependence trees. IEEE Transactions on Information Theory, 14:462–467, 1968.zbMATHCrossRefGoogle Scholar
  2. [2]
    I. Cohen. Automatic facial expression recognition from video sequences using temporal information. In MS Thesis, University of Illinois at Urbana-Champaign, Dept. of Electrical Engineering, 2000.Google Scholar
  3. [3]
    I. Cohen, N. Sebe, L. Chen, and T.S. Huang. Facial expression recognition from video sequences: Temporal and static modeling. to appear in Computer Vision and Image Understanding, 2003.Google Scholar
  4. [4]
    G. Donato, M.S. Bartlett, J.C. Hager, P. Ekman, and T.J. Sejnowski. Classifying facial actions. IEEE Transactions on Pattern Analysis and Machine Intelligence, 21(10):974–989, 1999.CrossRefGoogle Scholar
  5. [5]
    P. Ekman and W.V. Friesen. Facial Action Coding System: Investigator’s Guide. Consulting Psychologists Press, Palo Alto, CA, 1978.Google Scholar
  6. [6]
    I.A. Essa and A.P. Pentland. Coding, analysis, interpretation, and recognition of facial expressions. IEEE Transactions on Pattern Analysis and Machine Intelligence, 19(7):757–763, July 1997.CrossRefGoogle Scholar
  7. [7]
    J.H. Friedman. On bias, variance, 0/1-loss, and the curse-of-dimensionality. Data Mining and Knowledge Discovery, 1(1):55–77, 1997.CrossRefGoogle Scholar
  8. [8]
    N. Friedman, D. Geiger, and M. Goldszmidt. Bayesian network classifiers. Machine Learning, 29(2):131–163, 1997.zbMATHCrossRefGoogle Scholar
  9. [9]
    T. Kanade, J. Cohn, and Y. Tian. Comprehensive database for facial expression analysis, 2000.Google Scholar
  10. [10]
    A. Lanitis, C.J. Taylor, and T.F. Cootes. A unified approach to coding and interpreting face images. In Proc. 5th International Conference on Computer Vision (ICCV’95), pages 368–373, 1995.Google Scholar
  11. [11]
    N. Oliver, A. Pentland, and F. Bérard. LAFTER: A real-time face and lips tracker with facial expression recognition. Pattern Recognition, 33:1369–1382, 2000.CrossRefGoogle Scholar
  12. [12]
    T. Otsuka and J. Ohya. Recognizing multiple persons’ facial expressions using HMM based on automatic extraction of significant frames from image sequences. In Proc. Int. Conf. on Image Processing (ICIP’97), pages 546–549, 1997.Google Scholar
  13. [13]
    C. Padgett and G.W. Cottrell. Representing face images for emotion classification. In Conf. Advances in Neural Information Processing Systems, pages 894–900, 1996.Google Scholar
  14. [14]
    M. Pantic and L.J.M. Rothkrantz. Automatic analysis of facial expressions: The state of the art. IEEE Transactions on Pattern Analysis and Machine Intelligence, 22(12):1424–1445, 2000.CrossRefGoogle Scholar
  15. [15]
    L.R. Rabiner. A tutorial on hidden Markov models and selected applications in speech processing. Proceedings of IEEE, 77(2):257–286, 1989.CrossRefGoogle Scholar
  16. [16]
    M. Rosenblum, Y. Yacoob, and L.S. Davis. Human expression recognition from motion using a radial basis function network architecture. IEEE Transactions on Neural Network, 7(5):1121–1138, September 1996.CrossRefGoogle Scholar
  17. [17]
    H. Tao and T.S. Huang. Connected vibrations: A modal analysis approach to non-rigid motion tracking. In Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR’98), pages 735–740, 1998.Google Scholar
  18. [18]
    N. Ueki, S. Morishima, H. Yamada, and H. Harashima. Expression analysis/synthesis system based on emotion space constructed by multilayered neural network. Systems and Computers in Japan, 25(13):95–103, Nov. 1994.Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2003

Authors and Affiliations

  • Ira Cohen
    • 1
  • Nicu Sebe
    • 2
    • 3
  • Yafei Sun
    • 2
  • Michael S. Lew
    • 3
  • Thomas S. Huang
    • 1
  1. 1.Beckman InstituteUniversity of Illinois at Urbana-ChampaignUSA
  2. 2.Faculty of ScienceUniversity of AmsterdamThe Netherlands
  3. 3.Leiden Institute of Advanced Computer ScienceLeiden UniversityThe Netherlands

Personalised recommendations