Springer Nature is making SARS-CoV-2 and COVID-19 research free. View research | View latest news | Sign up for updates

Hybrid Facial Regions Extraction for Micro-expression Recognition System

  • 636 Accesses

  • 5 Citations

Abstract

Micro-expressions can occur when a person attempts to conceal and suppress his true feelings and emotions, both deliberately or unconsciously.In recent years, facial micro-expression analysis has received tremendous attention in the field of psychology, media and computer vision. However, due to its subtlety and brief duration, development of automated micro-expression detection and recognition system are still great challenges in the field of computer vision. In this paper, we present a novel hybrid facial region extraction framework that combines heuristic and automatic approaches to better recognize spontaneous micro-expressions. Salient facial regions are statistically determined based on the occurrence frequency of facial action units instead of holistic utilization of the entire facial area. The regions were automatically selected according to the facial landmark coordinates. We tested on two recent publicly available datasets that provided sufficient samples while also fulfilling the criteria of being elicited spontaneously. To further confirm the reliability of the proposed method, two distinct feature extractors were employed to describe micro-expression information. Results show consistent and promising performance in all scenarios considered. The best result achieved is an improvement of approximately 10.5% in CASME II and an increment of nearly 10% in SMIC. We also report F-measure, precision and recall performance metrics that are most suited for the imbalanced nature of spontaneous micro-expression datasets.

This is a preview of subscription content, log in to check access.

Figure 1
Figure 2
Figure 3
Figure 4
Figure 5
Figure 6
Figure 7
Figure 8
Figure 9
Figure 10

Notes

  1. 1.

    http://www.cse.oulu.fi/SMICDatabase

  2. 2.

    http://fu.psych.ac.cn/CASME/casme2-en.php

References

  1. 1.

    Asthana, A., Zafeiriou, S., Cheng, S., & Pantic, M. (2013). Robust discriminative response map fitting with constrained local models. In Computer vision and pattern recognition (pp. 3444–3451).

  2. 2.

    Beauchemin, S.S., & Barron, J.L. (1995). The computation of optical flow. ACM Computing Surveys (CSUR), 27(3), 433–466.

  3. 3.

    Cristinacce, D., & Cootes, T.F. (2006). Feature detection and tracking with constrained local models. In British machine vision conference, (Vol. 2 p. 6).

  4. 4.

    Ekman, P. (2009). Lie catching and microexpressions. The Philosophy of Deception, pp. 118–133.

  5. 5.

    Ekman, P. (2009). Telling lies: clues to Deceit in the marketplace, politics, and marriage. New York: W. W. Norton and Company.

  6. 6.

    Ekman, P., & Friesen, W.V. (1969). Nonverbal leakage and clues to deception. Journal for the Study of Interpersonal Processes, 32, 88–106.

  7. 7.

    Ekman, P., & Friesen, W.V. (1971). Constants across cultures in the face and emotion. Journal of Personality and Social Psychology, 17(2), 124.

  8. 8.

    Ekman, P., & Friesen, W.V. (1974). Nonverbal behavior and psychopathology. The Psychology of Depression: Contemporary Theory and Research, pp. 203–232.

  9. 9.

    Fan, X., & Verma, B. (2009). Selection and fusion of facial features for face recognition. Expert Systems with Applications, 36(3), 7157–7169.

  10. 10.

    Fleet, D., & Weiss, Y. (2006). Optical flow estimation. In Handbook of mathematical models in computer vision (pp. 237–257).

  11. 11.

    Friesen, E., & Ekman, P. (1978). Facial action coding system a technique for the measurement of facial movement. Consulting Psychologists Press, 12.

  12. 12.

    Gibson, J.J. (1950). The perception of the visual world.

  13. 13.

    Guo, Z., & Zhang, D. (2010). A completed modeling of local binary pattern operator for texture classification . IEEE Transactions on Image Processing, 19(6), 1657–1663.

  14. 14.

    Haggard, E.A., & Isaacs, K.S. (1966). Micromomentary facial expressions as indicators of ego mechanisms in psychotherapy. In Methods of research in psychotherapy (pp. 154–165).

  15. 15.

    Happy, S., & Routray, A. (2015). Automatic facial expression recognition using features of salient facial patches. IEEE Transactions on Affective Computing, 6(1), 1–12.

  16. 16.

    Horn, B.K., & Schunck, B.G. (1981). Determining optical flow. In International society for optics and photonics (pp. 319–331).

  17. 17.

    Le Ngo, A.C., Phan, R.C.W., & See, J. (2014). Spontaneous subtle expression recognition: Imbalanced databases & solutions. In Asian conference on computer vision.

  18. 18.

    Li, X., Pfister, T., Huang, X., Zhao, G., & Pietikainen, M. (2013). A spontaneous micro-expression database inducement, collection and baseline. In Automatic face and gesture recognition (pp. 1–6).

  19. 19.

    Liong, S.T., Phan, R.C.-W., See, J., Oh, Y.H., & Wong, K. (2014). Optical strain based recognition of subtle emotions. In International symposium on intelligent signal processing and communication systems.

  20. 20.

    Liong, S.T., See, J., Phan, R.C.W., Le Ngo, A.C., Oh, Y.H., & Wong, K. (2014). Subtle expression recognition using optical strain weighted features. In Asian conference on computer vision workshops on computer vision for affective computing.

  21. 21.

    Liong, S.T., See, J., Phan, R.C.W., Oh, Y.H., Ngo, A.C.L., Wong, K., & Tan, S.W. (2016). Spontaneous subtle expression detection and recognition based on facial strain. arXiv.

  22. 22.

    Liong, S.T., See, J., Phan, R.C.W., & Wong, K. (2016), Less is more: Micro-expression recognition from video using apex frame. arXiv.

  23. 23.

    Liu, M., Shan, S., Wang, R., & Chen, X. (2014). Learning expressionlets on spatio-temporal manifold for dynamic facial expression recognition. In Computer vision and pattern recognition (pp. 1749–1756).

  24. 24.

    Meng, H., & Bianchi-Berthouze, N. (2014). Affective state level recognition in naturalistic facial and vocal expressions. IEEE Transactions on Cybernetics, 44(3), 315–328.

  25. 25.

    Oh, Y.H., Le Ngo, A.C., See, J., Liong, S.T., Phan, R.C.W., & Ling, H.C. (2015). Monogenic riesz wavelet representation for micro-expression recognition. In Digital signal processing (pp. 1237–1241). Los Alamitos: IEEE.

  26. 26.

    Ojala, T., Pietikainen, M., & Maenpaa, T. (2002). Multiresolution gray-scale and rotation invariant texture classification with local binary patterns. Pattern Analysis and Machine Intelligence, 24(7), 971–987.

  27. 27.

    Pfister, T., Li, X., Zhao, G., & Pietikainen, M. (2011). Recognising spontaneous facial micro-expressions. In International conference on computer vision (pp. 1449–1456).

  28. 28.

    Polikovsky, S., Kameda, Y., & Ohta, Y. (2009). Facial micro-expressions recognition using high speed camera and 3d-gradient descriptor. In Crime detection and prevention (pp. 16–16).

  29. 29.

    Porter, S., & ten Brinke, L. (2008). Reading between the lies identifying concealed and falsified emotions in universal facial expressions. Psychological Science, 19.5, 508– 514.

  30. 30.

    Saragih, J.M., Lucey, S., & Cohn, J.F. (2011). Deformable model fitting by regularized landmark mean-shift. International Journal of Computer Vision, 91(2), 200–215.

  31. 31.

    Senechal, T., Bailly, K., & Prevost, L. (2014). Impact of action unit detection in automatic emotion recognition. Pattern Analysis and Applications, 17(1), 51–67.

  32. 32.

    Shan, C., Gong, S., & Mcowan, P.W. (2009). Facial expression recognition based on local binary patterns a comprehensive study. Image and Vision Computing, 27(6), 803–816.

  33. 33.

    Shreve, M., Brizzi, J., Fefilatyev, S., Luguev, T., Goldgof, D., & Sarkar, S. (2014). Automatic expression spotting in videos. Image and Vision Computing, 32(8), 476–486.

  34. 34.

    Shreve, M., Godavarthy, S., Manohar, V., Goldgof, D., & Sarkar, S. (2009). Towards macro-and micro-expression spotting in video using strain patterns. In Applications of computer vision (WACV) (pp. 1–6).

  35. 35.

    Wang, L., Li, R.F., Wang, K., & Chen, J. (2014). Feature representation for facial expression recognition based on facs and lbp. Pattern Analysis and Applications, 11(5), 459–468.

  36. 36.

    Wang, S.-J., Chen, H.-L., Yan, W.-J., Chen, Y.-H., & Fu, X. (2014). Face recognition and micro-expression recognition based on discriminant tensor subspace analysis plus extreme learning machine. Neural Processing Letters, 39(1), 25–43.

  37. 37.

    Wang, S.J., Yan, W.J., Zhao, G., Fu, X., & Zhou, C.G. (2014). Micro-expression recognition using robust principal component analysis and local spatiotemporal directional features. In ECCV workshop on spontaneous facial behavior analysis.

  38. 38.

    Wang, Y., See, J., Phan, R.C.W., & Oh, Y.H. (2015). Lbp with six intersection points: Reducing redundant information in lbp-top for micro-expression recognition. In Computer vision–ACCV (pp. 525–537). Los Alamitos: IEEE.

  39. 39.

    Yan, W.J., Wang, S.J., Liu, Y.J., Wu, Q., & Fu, X. (2014). For micro-expression recognition database and suggestions. Neurocomputing, 136, 82–87.

  40. 40.

    Yan, W.-J., Wang, S.-J., Zhao, G., Li, X., Liu, Y.-J., Chen, Y.-H., & Fu, X. (2014). CASME II: An improved spontaneous micro-expression database and the baseline evaluation. PLoS ONE, 9, e86041.

  41. 41.

    Zhang, J., Shan, S., Kan, M., & Chen, X. (2014). Coarse-to-fine auto-encoder networks (cfan) for real-time face alignment. In Computer vision–ECCV (pp. 1–16).

  42. 42.

    Zhao, G., & Pietikainen, M. (2009). Dynamic texture recognition using local binary patterns with an application to facial expressions. IEEE Transactions on Pattern Analysis and Machine Intelligence, 29(6), 915–928.

  43. 43.

    Zhong, L., Liu, Q., Yang, P., Huang, J., & Metaxas, D.N. (2014). Learning multiscale active facial patches for expression analysis. IEEE Transactions on Cybernetics.

  44. 44.

    Zhu, X., & Ramanan, D. (2012). Face detection, pose estimation, and landmark localization in the wild. In Computer vision and pattern recognition (pp. 2879–2886).

Download references

Author information

Correspondence to Sze-Teng Liong.

Additional information

This research was funded in part by TM under the projects UbeAware and 2beAware.

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Liong, S., See, J., Phan, R. et al. Hybrid Facial Regions Extraction for Micro-expression Recognition System. J Sign Process Syst 90, 601–617 (2018). https://doi.org/10.1007/s11265-017-1276-0

Download citation

Keywords

  • Micro-expressions
  • Emotion
  • Region of interest
  • Optical strain
  • Recognition