Facial micro-expression recognition based on the fusion of deep learning and enhanced optical flow

  • Qiuyu Li
  • Shu Zhan
  • Liangfeng Xu
  • Congzhong Wu


Micro-expression is a kind of split-second subtle expression which could not be controlled by the autonomic nervous system. Micro-expression indicates that a person is hiding his truly emotion consciously. Because the micro-expression is closely interrelated with lie detection, micro-expression recognition has various potential applications in many domains, such as the public security, the clinical medicine, the investigation and the interrogation. Because recognizing the micro-expression through human observation is very difficult, researchers focus on the automatic micro-expression recognition. This research proposed a novel algorithm for automatic micro-expression recognition which combined a deep multi-task convolutional network for detecting the facial landmarks and a fused deep convolutional network for estimating the optical flow features of the micro-expression. Firstly, this research employed the deep multi-task convolutional network to detect facial landmarks with the manifold related tasks and divided the facial region by utilizing these facial landmarks. Furthermore, a fused convolutional network was applied for extracting the optical flow features from the facial regions which contain the muscle changes when the micro-expression presents. Finally the enhanced optical flow was applied for refining the information of the features and these refined optical flow features were classified by Support Vector Machine classifier for recognizing the micro-expression. The result of experiments on two spontaneous micro-expression database demonstrated that the method proposed in this paper achieved good performance in micro-expression recognition.


Micro-expression Recognition Convolutional network Optical flow 



The authors would like to thank the anonymous reviews for their helpful and constructive comments and suggestions regarding this manuscript.


This work was supported in part by National Nature Science Foundation of China Grand No:61371156.


  1. 1.
    Ben X et al (2016) Gait recognition and micro-expression recognition based on maximum margin projection with tensor representation. Neural Comput & Applic 27(8):2629–2646CrossRefGoogle Scholar
  2. 2.
    Burgos-Artizzu XP, Perona P, Dollár P (2013) Robust face landmark estimation under occlusion. Proceedings of the IEEE international conference on computer visionGoogle Scholar
  3. 3.
    Chan CH et al (2012) Local ordinal contrast pattern histograms for spatiotemporal, lip-based speaker authentication. IEEE T Inf Foren Sec 7(2):602–612CrossRefGoogle Scholar
  4. 4.
    Chang C-C, Chih-Jen L (2011) LIBSVM: a library for support vector machines. ACM Trans Intell Syst Technol (TIST) 2(3):27Google Scholar
  5. 5.
    Chaudhry R et al (2009) Histograms of oriented optical flow and binet-cauchy kernels on nonlinear dynamical systems for the recognition of human actions. Computer Vision and Pattern Recognition, 2009. CVPR 2009. IEEE Conference on. IEEEGoogle Scholar
  6. 6.
    Dollár P et al (2005) Behavior recognition via sparse spatio-temporal features. Visual surveillance and performance evaluation of tracking and surveillance, 2005. 2nd Joint IEEE International Workshop on. IEEEGoogle Scholar
  7. 7.
    Dong C et al (2014) Learning a deep convolutional network for image super-resolution. In: European Conference on Computer Vision. Springer, ChamGoogle Scholar
  8. 8.
    Dosovitskiy A et al (2015) Flownet: learning optical flow with convolutional networks. Proceedings of the IEEE international conference on computer visionGoogle Scholar
  9. 9.
    Ekman P (2003) Darwin, deception, and facial expression. Ann N Y Acad Sci 1000(1):205–221CrossRefGoogle Scholar
  10. 10.
    Ekman P (2003) Micro expressions training tool.
  11. 11.
    Ekman P (2009) Telling lies: clues to deceit in the marketplace, politics, and marriage (revised edition). WW Norton & CompanyGoogle Scholar
  12. 12.
    Ekman P (2009) Lie catching and microexpressions. The philosophy of deception, pp 118–133Google Scholar
  13. 13.
    Ekman P, Friesen WV (1969) Nonverbal leakage and clues to deception. Psychiatry 32(1):88–106CrossRefGoogle Scholar
  14. 14.
    Ekman P, Friesen WV (1977) Facial action coding systemGoogle Scholar
  15. 15.
    Haggard EA, Isaacs KS (1966) Micromomentary facial expressions as indicators of ego mechanisms in psychotherapy. In: Methods of research in psychotherapy. Springer, Boston, pp 154–165CrossRefGoogle Scholar
  16. 16.
    He J et al (2017) Multi-task mid-level feature learning for micro-expression recognition. Pattern Recogn 66:44–52CrossRefGoogle Scholar
  17. 17.
    Huang X et al (2012) Spatiotemporal local monogenic binary patterns for facial expression recognition. IEEE Signal Proc Let 19(5):243–246CrossRefGoogle Scholar
  18. 18.
    Huang X et al (2016) Spontaneous facial micro-expression analysis using spatiotemporal completed local quantized patterns. Neurocomputing 175:564–578CrossRefGoogle Scholar
  19. 19.
    Ilg E et al (2016) Flownet 2.0: evolution of optical flow estimation with deep networks. arXiv preprint arXiv:1612.01925Google Scholar
  20. 20.
    Krizhevsky A, Sutskever I, Hinton GE (2012) Imagenet classification with deep convolutional neural networks. Advances in neural information processing systemsGoogle Scholar
  21. 21.
    Le Ngo AC, Phan RC-W, See J (2014) Spontaneous subtle expression recognition: imbalanced databases and solutions. In: Asian conference on computer vision. Springer, ChamGoogle Scholar
  22. 22.
    Le Ngo AC, See J, Phan RC-W (2017) Sparsity in dynamics of spontaneous subtle emotion: analysis & application. IEEE Trans Affect ComputGoogle Scholar
  23. 23.
    Li X, Yu J, Zhan S (2016) Spontaneous facial micro-expression detection based on deep learning. Signal Processing (ICSP), 2016 IEEE 13th International Conference on. IEEEGoogle Scholar
  24. 24.
    Li X et al (2017) Towards reading hidden emotions: a comparative study of spontaneous micro-expression spotting and recognition methods. IEEE transactions on affective computingGoogle Scholar
  25. 25.
    Liu Y-J et al (2016) A main directional mean optical flow feature for spontaneous micro-expression recognition. IEEE Trans Affect Comput 7(4):299–310CrossRefGoogle Scholar
  26. 26.
    Long J, Shelhamer E, Darrell T (2015) Fully convolutional networks for semantic segmentation. Proceedings of the IEEE conference on computer vision and pattern recognitionGoogle Scholar
  27. 27.
    Ma L, Khashayar K (2004) Facial expression recognition using constructive feedforward neural networks. IEEE T Syst Man Cy B 34(3):1588–1595CrossRefGoogle Scholar
  28. 28.
    Mayer N et al (2016) A large dataset to train convolutional networks for disparity, optical flow, and scene flow estimation. Proceedings of the IEEE conference on computer vision and pattern recognitionGoogle Scholar
  29. 29.
    Ni Z et al (2016) Gradient direction for screen content image quality assessment. IEEE Signal Proc Let 23(10):1394–1398CrossRefGoogle Scholar
  30. 30.
    Ni Z et al (2017) ESIM: edge similarity for screen content image quality assessment. IEEE Trans Image Process 26(10):4818–4831MathSciNetCrossRefGoogle Scholar
  31. 31.
    Oh Y-H et al (2015) Monogenic Riesz wavelet representation for micro-expression recognition. Digital Signal Processing (DSP), 2015 IEEE International Conference on. IEEEGoogle Scholar
  32. 32.
    Patel D, Hong X, Zhao G (2016) Selective deep features for micro-expression recognition. Pattern Recognition (ICPR), 2016 23rd International Conference on. IEEEGoogle Scholar
  33. 33.
    Pfister T et al (2011) Recognising spontaneous facial micro-expressions. Computer Vision (ICCV), 2011 IEEE International Conference on. IEEEGoogle Scholar
  34. 34.
    Pfister T et al (2011) Differentiating spontaneous from posed facial expressions within a generic facial expression recognition framework. Computer Vision Workshops (ICCV Workshops), 2011 IEEE International Conference on. IEEEGoogle Scholar
  35. 35.
    Polikovsky S, Kameda Y, Ohta Y (2009) Facial micro-expressions recognition using high speed camera and 3D-gradient descriptor, pp 16–16Google Scholar
  36. 36.
    Ren S et al (2015) Faster R-CNN: towards real-time object detection with region proposal networks. Advances in neural information processing systemsGoogle Scholar
  37. 37.
    Ruder S (2017) An overview of multi-task learning in deep neural networks[J]. arXiv preprint arXiv:1706.05098Google Scholar
  38. 38.
    Sun Y, Wang X, Tang X (2013) Deep convolutional network cascade for facial point detection. Proceedings of the IEEE conference on computer vision and pattern recognitionGoogle Scholar
  39. 39.
    Sun Y, Wang X, Tang X (2014) Deep learning face representation from predicting 10,000 classes. Proceedings of the IEEE conference on computer vision and pattern recognitionGoogle Scholar
  40. 40.
    Tran D et al (2016) Deep end2end voxel2voxel prediction. Proceedings of the IEEE conference on computer vision and pattern recognition workshopsGoogle Scholar
  41. 41.
    Wang Y et al (2011) Large disparity motion layer extraction via topological clustering. IEEE Trans Image Process 20(1):43–52MathSciNetCrossRefGoogle Scholar
  42. 42.
    Wang S-J et al (2014) Micro-expression recognition using dynamic textures on tensor independent color space. Pattern Recognition (ICPR), 2014 22nd International Conference on. IEEEGoogle Scholar
  43. 43.
    Wang Y et al (2014) Lbp with six intersection points: reducing redundant information in lbp-top for micro-expression recognition. In: Asian Conference on Computer Vision. Springer, ChamGoogle Scholar
  44. 44.
    Wang Y et al (2015) Efficient spatio-temporal local binary patterns for spontaneous facial micro-expression recognition. PLoS One 10(5):e0124674CrossRefGoogle Scholar
  45. 45.
    Wang Y et al (2017) Effective recognition of facial micro-expressions with video motion magnification. Multimed Tools Appl 76(20):21665–21690CrossRefGoogle Scholar
  46. 46.
    Xu F, Zhang J, Wang JZ (2017) Microexpression identification and categorization using a facial dynamics map. IEEE Trans Affect Comput 8(2):254–267CrossRefGoogle Scholar
  47. 47.
    Yan W-J et al (2013) CASME database: a dataset of spontaneous micro-expressions collected from neutralized faces. Automatic face and gesture recognition (FG), 2013 10th IEEE International Conference and Workshops on. IEEEGoogle Scholar
  48. 48.
    Yan W-J et al (2014) CASME II: an improved spontaneous micro-expression database and the baseline evaluation. PLoS One 9(1):e86041CrossRefGoogle Scholar
  49. 49.
    Yang A et al (2017) Perceptual feature guided rate distortion optimization for high efficiency video coding. Multidim Syst Sign Process 28(4):1249–1266MathSciNetCrossRefGoogle Scholar
  50. 50.
    Zeng H, Cai C, Ma K-K (2009) Fast mode decision for H. 264/AVC based on macroblock motion activity. IEEE T Circ Syst Vid 19(4):491–499CrossRefGoogle Scholar
  51. 51.
    Zeng H, Ma K-K, Cai C (2010) Hierarchical intra mode decision for H. 264/AVC. IEEE T Circ Syst Vid 20(6):907–912CrossRefGoogle Scholar
  52. 52.
    Zeng H, Ma K-K, Cai C (2011) Fast mode decision for multiview video coding using mode correlation. IEEE T Circ Syst Vid 21(11):1659–1666CrossRefGoogle Scholar
  53. 53.
    Zeng H et al (2014) Fast multiview video coding using adaptive prediction structure and hierarchical mode decision. IEEE T Circ Syst Vid 24(9):1566–1578CrossRefGoogle Scholar
  54. 54.
    Zeng H et al (2015) Sift-flow-based color correction for multi-view video. Signal Process Image Commun 36:53–62CrossRefGoogle Scholar
  55. 55.
    Zeng H et al (2016) Quad binary pattern and its application in mean-shift tracking. Neurocomputing 217:3–10CrossRefGoogle Scholar
  56. 56.
    Zhang Z et al (2014) Facial landmark detection by deep multi-task learning. In: European Conference on Computer Vision. Springer, ChamGoogle Scholar
  57. 57.
    Zhang S et al (2017) Micro-expression recognition by aggregating local spatio-temporal patterns. In: International Conference on Multimedia Modeling. Springer, ChamGoogle Scholar
  58. 58.
    Zhao G, Pietikainen M (2007) Dynamic texture recognition using local binary patterns with an application to facial expressions. IEEE Trans Pattern Anal Mach Intell 29(6):915–928CrossRefGoogle Scholar
  59. 59.
    Zhu Z et al (2013) Deep learning identity-preserving face space. Proceedings of the IEEE international conference on computer visionGoogle Scholar
  60. 60.
    Zhu J et al (2017) Deep hybrid similarity learning for person re-identification. IEEE T Circ Syst VidGoogle Scholar

Copyright information

© Springer Science+Business Media, LLC, part of Springer Nature 2018

Authors and Affiliations

  1. 1.School of Computer and InformationHefei University of TechnologyHefeiChina

Personalised recommendations