Detecting Micro-expression Intensity Changes from Videos Based on Hybrid Deep CNN

  • Selvarajah ThuseethanEmail author
  • Sutharshan Rajasegarar
  • John Yearwood
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11441)


Facial micro-expressions, which usually last only for a fraction of a second, are challenging to detect by the human eye or machine. They are useful for understanding the genuine emotional state of a human face, and have various applications in education, medical, surveillance and legal sectors. Existing works on micro-expressions are focused on binary classification of the micro-expressions. However, detecting the micro-expression intensity changes over the spanning time, i.e., the micro-expression profiling, is not addressed in the literature. In this paper, we present a novel deep Convolutional Neural Network (CNN) based hybrid framework for micro-expression intensity change detection together with an image pre-processing technique. The two components of our hybrid framework, namely a micro-expression stage classifier, and an intensity estimator, are designed using a 3D and 2D shallow deep CNNs respectively. Moreover, we propose a fusion mechanism to improve the micro-expression intensity classification accuracy. Evaluation using the recent benchmark micro-expression datasets; CASME, CASME II and SAMM, demonstrates that our hybrid framework can accurately classify the various intensity levels of each micro-expression. Further, comparison with the state-of-the-art methods reveals the superiority of our hybrid approach in classifying the micro-expressions accurately.


Micro-expression intensity Convolutional Neural Networks Hybrid framework Fusion mechanism 


  1. 1.
    Davison, A.K., Lansley, C., Costen, N., Tan, K., Yap, M.H.: SAMM: a spontaneous micro-facial movement dataset. IEEE Trans. Affect. Comput. 9(1), 116–129 (2018)CrossRefGoogle Scholar
  2. 2.
    Ekman, P.: Lie catching and microexpressions. Philos. Decept. 1, 5 (2009)Google Scholar
  3. 3.
    Gottman, J.M., Levenson, R.W.: A two-factor model for predicting when a couple will divorce: exploratory analyses using 14-year longitudinal data. Fam. Process 41(1), 83–96 (2002)CrossRefGoogle Scholar
  4. 4.
    Happy, S.L., Routray, A.: Fuzzy histogram of optical flow orientations for micro-expression recognition. IEEE Trans. Affect. Comput. (2017).
  5. 5.
    Kim, D.H., Baddar, W.J., Ro, Y.M.: Micro-expression recognition with expression-state constrained spatio-temporal feature representations. In: Proceedings of the 2016 ACM on Multimedia Conference, pp. 382–386. ACM (2016)Google Scholar
  6. 6.
    Liong, S.T., See, J., Wong, K., Phan, R.C.W.: Less is more: micro-expression recognition from video using apex frame. Signal Process.: Image Commun. 62, 82–92 (2018)Google Scholar
  7. 7.
    Lucey, P., Cohn, J.F., Prkachin, K.M., Solomon, P.E., Matthews, I.: Painful data: the UNBC-McMaster shoulder pain expression archive database. In: 2011 IEEE International Conference on Automatic Face and Gesture Recognition and Workshops (FG 2011), pp. 57–64. IEEE (2011)Google Scholar
  8. 8.
    Merghani, W., Davison, A.K., Yap, M.H.: A review on facial micro-expressions analysis: datasets, features and metrics. arXiv preprint arXiv:1805.02397 (2018)
  9. 9.
    Peng, M., Wang, C., Chen, T., Liu, G., Fu, X.: Dual temporal scale convolutional neural network for micro-expression recognition. Front. Psychol. 8, 1745 (2017)CrossRefGoogle Scholar
  10. 10.
    Simard, P.Y., Steinkraus, D., Platt, J.C.: Best practices for convolutional neural networks applied to visual document analysis. In: Null, p. 958. IEEE (2003)Google Scholar
  11. 11.
    Yan, W.J., et al.: CASME II: an improved spontaneous micro-expression database and the baseline evaluation. PloS One 9(1), e86041 (2014)CrossRefGoogle Scholar
  12. 12.
    Yan, W.J., Wu, Q., Liu, Y.J., Wang, S.J., Fu, X.: CASME database: a dataset of spontaneous micro-expressions collected from neutralized faces. In: 2013 10th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition (FG), pp. 1–7. IEEE (2013)Google Scholar
  13. 13.
    Zhao, R., Gan, Q., Wang, S., Ji, Q.: Facial expression intensity estimation using ordinal information. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3466–3474 (2016)Google Scholar
  14. 14.
    Zheng, H., Geng, X., Yang, Z.: A relaxed K-SVD algorithm for spontaneous micro-expression recognition. In: Booth, R., Zhang, M.-L. (eds.) PRICAI 2016. LNCS (LNAI), vol. 9810, pp. 692–699. Springer, Cham (2016). Scholar
  15. 15.
    Zhu, X., Ben, X., Liu, S., Yan, R., Meng, W.: Coupled source domain targetized with updating tag vectors for micro-expression recognition. Multimed. Tools Appl. 77(3), 3105–3124 (2018)CrossRefGoogle Scholar
  16. 16.
    Zuiderveld, K.: Contrast limited adaptive histogram equalization. In: Graphics Gems, pp. 474–485 (1994)Google Scholar

Copyright information

© Springer Nature Switzerland AG 2019

Authors and Affiliations

  • Selvarajah Thuseethan
    • 1
    Email author
  • Sutharshan Rajasegarar
    • 1
  • John Yearwood
    • 1
  1. 1.Deakin UniversityGeelongAustralia

Personalised recommendations