Dynamic Visual Time Context Descriptors for Automatic Human Expression Classification

Conference paper
Part of the Advances in Intelligent Systems and Computing book series (AISC, volume 215)


In this paper, we propose two fast dynamic descriptors Vertical-Time-Backward (VTB) and Vertical-Time-Forward (VTF) on spatial–temporal domain to catch the cues of essential facial movements. These dynamic descriptors are used in a two-step system to recognize human facial expression within image sequences. In the first step, the system classifies static images and then it identifies the whole sequence. After combining the visual-time context features with popular LBP, the system can efficiently recognize the expression in a single image, and is especially helpful in highly ambiguous ones. In the second step, we use the evaluation method through the weighted probabilities of all frames to predict the class of the whole sequence. The experiments were performed on 348 sequences from 95 subjects in Cohn–Kanade database and obtained good results as high as 97.6 % in seven-class recognition for frames and 95.7 % in six class for sequences.


Facial expression classification LBP Spatial–temporal descriptor 



This work was supported by National Natural Science Foundation of China (61170124& 61272258), Jiangsu Province Natural Science Foundation (BK2009116) and Basic and Applied Research Program of Suzhou City (SYG201116).


  1. 1.
    Ekman P, Friesen W (1978) Facial action coding system: a technique for the measurement of facial movement. Consulting Psychologists Press, Palo AltoGoogle Scholar
  2. 2.
    Pantic M, Rothkrantz L (2000) Automatic analysis of facial expressions: the state of the art. IEEE Trans Pattern Anal Mach Intell 22:1424–1445CrossRefGoogle Scholar
  3. 3.
    Huang D, Shan C, Ardabilian M, Wang Y, Chen L (2011) Local binary patterns and its application to facial image analysis: a survey. IEEE Trans Syst Man Cybern Part C: Appl Rev 41(4):1–17Google Scholar
  4. 4.
    Chang K, Liu T, Lai S (2009) Learning partially-observed hidden conditional random fields for facial expression recognition. In: CVPR, Miami, pp 533–540Google Scholar
  5. 5.
    Bartlett M, Littlewort G, Lainscsek C, Fasel I, Frank M, Movellan J (2005) Fully automatic facial action recognition in spontaneous behavior. In: Automatic Face and Gesture Recognition, pp 223–228Google Scholar
  6. 6.
    Shan C, Gong S, McOwan P (May 2009) Facial expression recognition based on local binary patterns: a comprehensive study. Image Vis Comput 27(6):803–816Google Scholar
  7. 7.
    Zhao G, Ahonen T, Matas J, Pietikainen M (2012) Rotation-invariant image and video description with local binary pattern features. Image Process, IEEE Trans 21(4):1465–1477Google Scholar
  8. 8.
    Yeasin M, Bullot B, Sharma R (2004) From facial expression to level of interest: a spatio-temporal approach. CVPR 2:922–927Google Scholar
  9. 9.
    Xiang T, Leung M, Cho S (2008) Expression recognition using fuzzy spatio-temporal modeling. Pattern Recognit 41(1):204–216CrossRefMATHGoogle Scholar
  10. 10.
    Zhao G, Pietikäinen M (2009) Boosted multi-resolution spatiotemporal descriptors for facial expression recognition. Pattern Recogn Lett 30(12):1117–1127CrossRefGoogle Scholar
  11. 11.
    Yang P, Liu Q, Metaxas DN (2009) Boosting encoded dynamic features for facial expression recognition. Pattern Recogn Lett 30(2):132–139CrossRefGoogle Scholar
  12. 12.
    Buenaposada J, Munoz E, Baumela L (January 2008) Recognising facial expressions in video sequences. Pattern Anal Appl 1(2):101–116Google Scholar
  13. 13.
    Rudovic O, Pavlovic V, Pantic M (2012) Multi-output laplacian dynamic ordinal regression for facial expression recognition and intensity estimation. CVPR '12 Proceedings of the 2012 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) pp 2634–2641Google Scholar
  14. 14.
    Kienzle W, Bakir G, Franz M, Schölkopf B (2004) Face detection—efficient and rank deficient. NIPS 17(2005):673–680Google Scholar
  15. 15.
    Ji Y, Idrissi K (2012) Automatic facial expression recognition based on spatiotemporal descriptors. Pattern Recognit Lett 33(10):1373–1380CrossRefGoogle Scholar
  16. 16.
    Beveridge J, Bolme D, Draper B, Teixeira M (February 2005) The csu face identification evaluation system: its purpose, features, and structure. Mach Vis Appl 16(2):128–138Google Scholar
  17. 17.
    Ojala T, Pietikäinen M, Harwood D (January 1996) A comparative study of texture measures with classification based on feature distributions. Pattern Recognit 29(1):51–59Google Scholar
  18. 18.
    Bassili JN (1979) Emotion recognition: the role of facial movement and the relative importance of upper and lower areas of the face. J Pers Soc Psychol 37(11):2049–2058CrossRefGoogle Scholar
  19. 19.
    Kanade T, Cohn J, Tian Y-L (2000) Comprehensive database for facial expression analysis. In Proceedings of the 4th IEEE international conference on automatic face and gesture recognition (FG’00), Grenoble, pp 46–53Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2014

Authors and Affiliations

  1. 1.School of Computer Science and TechnologySoochow UniversitySuzhouChina

Personalised recommendations