Towards the Usage of Optical Flow Temporal Features for Facial Expression Classification

  • Raymond Ptucha
  • Andreas Savakis
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 7432)


Psychological evidence suggests that the human ability to recognize facial expression improves with the addition of temporal stimuli. While the facial action coding community has largely migrated towards temporal information, the facial expression recognition community has been slow to utilize facial dynamics. This paper contrasts the contributions of static vs. temporal features, including both dense and sparse facial tracking methodologies in combination with sparse representation classification. The temporal methods of facial feature point tracking, motion history images, free form deformation, and SIFT flow are adapted for facial expression classification. Dense optical flow for facial expression recognition is successfully utilized. We show that when used in isolation, the best temporal methods are just as good as static methods. However, when fusing temporal dynamics with static imagery significant increases in facial expression classification are achieved.


Facial Expression Optical Flow Motion Vector Sparse Representation Local Binary Pattern 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Lew, M., Bakker, E.M., Sebe, N., Huang, T.S.: Human-Computer Intelligent Interaction: A Survey. In: Lew, M., Sebe, N., Huang, T.S., Bakker, E.M. (eds.) HCI 2007. LNCS, vol. 4796, pp. 1–5. Springer, Heidelberg (2007)CrossRefGoogle Scholar
  2. 2.
    Pantic, M., Pentland, A., Nijholt, A., Huang, T.S.: Human computing and machine understanding of human behavior: a survey. In: ICMI 2006 and IJCAI 2007 International Workshops, Artifical Intelligence for Human Computing, Berlin, Germany (2007)Google Scholar
  3. 3.
    Pantic, M., Rothkrantz, L.U.M.: Automatic analysis of facial expressions: The state of the art. IEEE Transactions on Pattern Analysis and Machine Intelligence 22, 1424–1445 (2000)CrossRefGoogle Scholar
  4. 4.
    Zhihong, Z., Pantic, M., Roisman, G.I., Huang, T.S.: A survey of affect recognition methods: audio, visual, and spontaneous expressions. IEEE Transactions on Pattern Analysis and Machine Intelligence 31, 39–58 (2009)CrossRefGoogle Scholar
  5. 5.
    Shuai-Shi, L., Yan-Tao, T., Dong, L.: New research advances of facial expression recognition. In: Eighth International Conference on Machine Learning and Cybernetics (2009)Google Scholar
  6. 6.
    Martin, C., Werner, U., Gross, H.M.: A real-time facial expression recognition system based on active appearance models using gray images and edge images. In: 8th IEEE International Conference on Automatic Face & Gesture Recognition (2008)Google Scholar
  7. 7.
    Buciu, I., Kotropoulos, C., Pitas, I.: ICA and Gabor representation for facial expression recognition. In: Proceedings of International Conference on Image Processing (2003)Google Scholar
  8. 8.
    Shan, C., Gong, S., McOwan, P.W.: Facial expression recognition based on Local Binary Patterns: A comprehensive study. Image and Vision Computing 27, 803–816 (2009)CrossRefGoogle Scholar
  9. 9.
    Zafeiriou, S., Petrou, M.: Sparse representations for facial expressions recognition via l1 optimization. In: IEEE Computer Society Conference on Computer Vision and Pattern Recognition - Workshops, CVPRW 2010, San Francisco, CA (2010)Google Scholar
  10. 10.
    Ekman, P., Friesen, W.V.: The Facial Action Coding System. Consulting Psychologists Press, Inc., San Francisco (1978)Google Scholar
  11. 11.
    Koelstra, S., Pantic, M., Patras, I.: A dynamic texture-based approach to recognition of facial actions and their temporal models. IEEE Transactions on Pattern Analysis and Machine Intelligence 32, 1940–1954 (2010)CrossRefGoogle Scholar
  12. 12.
    Curio, C., Bulthoff, H., Giese, M.: Dynamic Faces: Insights from Experiments and Computation, 1st edn. The MIT Press (2010)Google Scholar
  13. 13.
    Jiang, B., Valstar, M.F., Pantic, M.: Action Unit Detection Using Sparse Appearance Descriptors in Space-Time Video Volumes. In: Face and Gesture Recognition Face and Gesture Recognition, Santa Barbara, CA (2011)Google Scholar
  14. 14.
    Lucey, P., Cohn, J., Matthews, I., Prkachin, K., Solomon, P.: Painful Data: The UNBC-McMaster Shoulder Pain Expression Archive Database. In: Face and Gesture Recognition, Santa Barbara, CA (2011)Google Scholar
  15. 15.
    Bhaskaran, N., Nwogu, I., Frank, M., Govindaraju, V.: Lie to Me: Deceit Detection via Online Behavioral Learning. In: Face and Gesture Recognition, Santa Barbara, CA (2011)Google Scholar
  16. 16.
    Cohn, J.F., et al.: Detecting depression from facial actions and vocal prosody. In: 3rd International Conference on Affective Computing and Intelligent Interaction and Workshops, ACII 2009 (2009)Google Scholar
  17. 17.
    Saragih, J., Lucey, S., Cohn, J.: Real-time Avatar Animation from a Single Image. In: Face and Gesture Recognition, Santa Barbara, CA (2011)Google Scholar
  18. 18.
    Bobick, A.F., Davis, J.W.: The recognition of human movement using temporal templates. IEEE Transactions on Pattern Analysis and Machine Intelligence 23, 257–267 (2001)CrossRefGoogle Scholar
  19. 19.
    Ce, L., Yuen, J., Torralba, A.: SIFT flow: dense correspondence across scenes and its applications. IEEE Transactions on Pattern Analysis and Machine Intelligence 33, 978–994 (2011)CrossRefGoogle Scholar
  20. 20.
    Songfan, Y., Bhanu, B.: Facial expression recognition using emotion avatar image. In: 2011 IEEE International Conference on Automatic Face & Gesture Recognition (2011)Google Scholar
  21. 21.
    Ptucha, R., Tsagkatakis, G., Savakis, A.: Manifold Based Sparse Representation for Robust Expression Recognition without Neutral Subtraction. In: Presented at the BeFIT 2011 Workshop, International Conference on Computer Vision, Barcelona, Spain (2011)Google Scholar
  22. 22.
    Olshausen, B.A., Field, D.J.: Sparse coding with an overcomplete basis set: a strategy employed by V1? Vision Research 37, 3311–3325 (1997)CrossRefGoogle Scholar
  23. 23.
    Olshausen, B.A., Field, D.J.: Emergence of simple-cell receptive field properties by learning a sparse code for natural images. Nature 381, 607–609 (1996)CrossRefGoogle Scholar
  24. 24.
    Wright, J., Yang, A.Y., Ganesh, A., Sastry, S.S., Yi, M.: Robust face recognition via sparse representation. IEEE Transactions on Pattern Analysis and Machine Intelligence 31, 210–227 (2009)CrossRefGoogle Scholar
  25. 25.
    He, X., Niyogi, P.: Locality Preserving Projections. In: Advances in Neural Information Processing Systems, Vancouver, Canada, vol. 16 (2003)Google Scholar
  26. 26.
    Cootes, T.F., Taylor, C.J., Cooper, D.H., Graham, J.: Active shape models - their training and application. Computer Vision and Image Understanding 61, 38–59 (1995)CrossRefGoogle Scholar
  27. 27.
    Kotsia, I., Pitas, I.: Facial expression recognition in image sequences using geometric deformation features and support vector machines. IEEE Transactions on Image Processing 16, 172–187 (2007)MathSciNetCrossRefGoogle Scholar
  28. 28.
    Rueckert, D., Sonoda, L.I., Hayes, C., Hill, D.L.G., Leach, M.O., Hawkes, D.J.: Nonrigid registration using free-form deformations: Application to breast MR images. IEEE Transactions on Medical Imaging 18, 712–721 (1999)CrossRefGoogle Scholar
  29. 29.
    Valstar, M., Jiang, B., Mehu, M., Pantic, M., Scherer, K.R.: The First Facial Expression Recognition and Analysis Challenge. In: Face and Gesture Recognition, Santa Barbara, CA (2011)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2012

Authors and Affiliations

  • Raymond Ptucha
    • 1
  • Andreas Savakis
    • 1
  1. 1.Computing and Information Sciences and Computer EngineeringRochester Institute of TechnologyRochesterUSA

Personalised recommendations