Subtle Expression Recognition Using Optical Strain Weighted Features

  • Sze-Teng LiongEmail author
  • John See
  • Raphael C.-W. Phan
  • Anh Cat Le Ngo
  • Yee-Hui Oh
  • KokSheik Wong
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 9009)


Optical strain characterizes the relative amount of displacement by a moving object within a time interval. Its ability to compute any small muscular movements on faces can be advantageous to subtle expression research. This paper proposes a novel optical strain weighted feature extraction scheme for subtle facial micro-expression recognition. Motion information is derived from optical strain magnitudes, which is then pooled spatio-temporally to obtain block-wise weights for the spatial image plane. By simple product with the weights, the resulting feature histograms are intuitively scaled to accommodate the importance of block regions. Experiments conducted on two recent spontaneous micro-expression databases–CASMEII and SMIC, demonstrated promising improvement over the baseline results.


Support Vector Machine Facial Expression Optical Flow Emotion Recognition Local Binary Pattern 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


  1. 1.
    Ekman, P., Friesen, W.V.: Constants across cultures in the face and emotion. J. Pers. Soc. Psychol. 17(2), 124 (1971)CrossRefGoogle Scholar
  2. 2.
    Ekman, P.: Lie catching and microexpressions. In: Martin, C. (ed.) The Philosophy of Deception, pp. 118–133. Oxford University Press, New York (2009)CrossRefGoogle Scholar
  3. 3.
    Porter, S., ten Brinke, L.: Reading between the lies identifying concealed and falsified emotions in universal facial expressions. Psych. Sci. 19, 508–514 (2008)CrossRefGoogle Scholar
  4. 4.
    Frank, M.G., Herbasz, M., Sinuk, K., Keller, A., Kurylo, A., Nolan, C.: I see how you feel: training laypeople and professionals to recognize fleeting emotions. In: Annual Meeting of the International Communication Association, Sheraton New York, New York City, NY (2009)Google Scholar
  5. 5.
    O’Sullivan, M., Frank, M.G., Hurley, C.M., Tiwana, J.: Police lie detection accuracy: the effect of lie scenario. Law Hum. Behav. 33(6), 530–538 (2009)CrossRefGoogle Scholar
  6. 6.
    Frank, M.G., Maccario, C.J., Govindaraju, V.: Protecting Airline Passengers in the Age of Terrorism. ABC-CLIO, Santa Barbara (2009)Google Scholar
  7. 7.
    D’hooge, J., Heimdal, A., Jamal, F., Kukulski, T., Bijnens, B., Rademakers, F., Hatle, L., Suetens, P., Sutherland, G.R.: Regional strain and strain rate measurements by cardiac ultrasound: principles, implementation and limitations. Eur. J. Echocardiogr. 1(3), 154–170 (2000)CrossRefGoogle Scholar
  8. 8.
    Shreve, M., Manohar, V., Goldgof, D., Sarkar, S.: Face recognition under camouflage and adverse illumination. In: 4th IEEE International Conference on Biometrics: Theory Applications and Systems (BTAS), pp. 1–6 (2010)Google Scholar
  9. 9.
    Manohar, V., Goldgof, D., Sarkar, S.: Facial strain pattern as a soft forensic evidence. In: Applications of Computer Vision (WACV) (2007)Google Scholar
  10. 10.
    Shreve, M., Godavarthy, S., Manohar, V., Goldgof, D., Sarkar, S.: Towards macro-and micro-expression spotting in video using strain patterns. In: Applications of Computer Vision (WACV), pp. 1–6 (2009)Google Scholar
  11. 11.
    Shreve, M., Godavarthy, S., Goldgof, D., Sarkar, S.: Macro-and micro-expression spotting in long videos using spatio- temporal strain. In: Automatic Face, Gesture Recognition and Workshops, pp. 51–56 (2011)Google Scholar
  12. 12.
    Vinciarelli, A., Dielmann, A., Favre, S., Salamin, H.: Canal9: a database of political debates for analysis of social interactions. In: Affective Computing and Intelligent Interaction and Workshops, pp. 1–4 (2009)Google Scholar
  13. 13.
    Ekman, P.: Telling Lies: Clues to Deceit in the Marketplace, Politics, and Marriage. W. W. Norton and Company, New York (2009)Google Scholar
  14. 14.
    Shreve, M., Brizzi, J., Fefilatyev, S., Luguev, T., Goldgof, D., Sarkar, S.: Automatic expression spotting in videos. Image Vis. Comput. 32(8), 476–486 (2014)CrossRefGoogle Scholar
  15. 15.
    Zhao, G., Pietikainen, M.: Dynamic texture recognition using local binary patterns with an application to facial expressions. IEEE Trans. Pattern Anal. Mach. Intell. 20(6), 915–928 (2007)CrossRefGoogle Scholar
  16. 16.
    Yan, W.J., Wang, S.J., Zhao, G., Li, X., Liu, Y.J., Chen, Y.H., Fu, X.: Casme ii: an improved spontaneous micro-expression database and the baseline evaluation. PLoS ONE 9, e86041 (2014)CrossRefGoogle Scholar
  17. 17.
    Pfister, T., Li, X., Zhao, G., Pietikainen, M.: Recognising spontaneous facial micro-expressions. In: Computer Vision (ICCV), pp. 1449–1456 (2011)Google Scholar
  18. 18.
    Li, X., Pfister, T., Huang, X., Zhao, G., Pietikainen, M.: A spontaneous micro-expression database: inducement, collection and baseline. In: Automatic Face and Gesture Recognition, pp. 1–6 (2013)Google Scholar
  19. 19.
    Boureau, Y.L., Ponce, J., LeCun, Y.: A theoretical analysis of feature pooling in visual recognition. In: Machine Learning (ICML 2010), pp. 11–118 (2010)Google Scholar
  20. 20.
    Hamel, P., Lemieux, S., Bengio, Y., Eck, D.: Temporal pooling and multiscale learning for automatic annotation and ranking of music audio. In: International Society for Music Information Retrieval Conference, pp. 729–734 (2011)Google Scholar
  21. 21.
    Forsyth, D.A., Ponce, J.: Computer Vision: A Modern Approach. Prentice Hall, Upper Saddle River (2002)Google Scholar
  22. 22.
    Ekman, P., Friesen, W.V.: Facial Action Coding System. Consulting Psychologists Press, Palo Alto (1978)Google Scholar
  23. 23.
    Lien, J.J.J., Kanade, T., Cohn, J.F., Li, C.C.: Detection, tracking, and classification of action units in facial expression. Rob. Auton. Syst. 31(3), 131–146 (2000)CrossRefGoogle Scholar
  24. 24.
    Liu, Z., Shan, Y., Zhang, Z.: Expressive expression mapping with ratio images. In: Computer Graphics and Interactive Techniques, pp. 271–276 (2001)Google Scholar
  25. 25.
    Anitha, C., Venkatesha, M.K., Adiga, B.S.: A survey on facial expression databases. Int. J. Eng. Sci. Technol. 2(10), 5158–5174 (2010)Google Scholar
  26. 26.
    Yan, W.J., Wang, S.J., Liu, Y.J., Wu, Q., Fu, X.: For micro-expression recognition: database and suggestions. Neurocomputing 136, 82–87 (2014)CrossRefGoogle Scholar
  27. 27.
    Polikovsky, S., Kameda, Y.,O.Y.: Facial micro-expressions recognition using high speed camera and 3D-gradient descriptor. In: Crime Detection and PreventionGoogle Scholar
  28. 28.
    Warren, G., Schertler, E., Bull, P.: Detecting deception from emotional and unemotional cues. J. Nonverbal Behav. 33(1), 59–59 (2009)CrossRefGoogle Scholar
  29. 29.
    Barron, J.L., Thacker, N.A.: Tutorial: Computing 2D and 3D optical flow (2005) Imaging Science and Biomedical Engineering Division, Medical School, University of ManchesterGoogle Scholar
  30. 30.
    Jain, R., Kasturi, R., Schunck, B.G.: Machine Vision, vol. 5. McGraw-Hill Education, New York (1995)Google Scholar
  31. 31.
    Godavarthy, S.: Microexpression spotting in video using optical strain. Masters thesis, University of South Florida (2010)Google Scholar
  32. 32.
    Sun, D., Roth, S., Black, M.J.: Secrets of optical flow estimation and their principles. In: Computer Vision and Pattern Recognition, pp. 2432–2439 (2010)Google Scholar

Copyright information

© Springer International Publishing Switzerland 2015

Authors and Affiliations

  • Sze-Teng Liong
    • 1
    Email author
  • John See
    • 2
  • Raphael C.-W. Phan
    • 3
  • Anh Cat Le Ngo
    • 3
  • Yee-Hui Oh
    • 3
  • KokSheik Wong
    • 1
  1. 1.Faculty of Computer Science and Information TechnologyUniversity of MalayaKuala LumpurMalaysia
  2. 2.Faculty of Computing and InformaticsMultimedia UniversityCyberjayaMalaysia
  3. 3.Faculty of EngineeringMultimedia UniversityCyberjayaMalaysia

Personalised recommendations