Skip to main content
SpringerLink
Log in

Micro-expression recognition based on differential feature fusion

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

Micro-expressions (MEs) are natural facial mechanisms with short duration and subtle changes. It has attracted much attention in the real world due to its accuracy and uncontrollability of mental expression. With the development of computer vision, micro-expression Recognition (MER) methods have been continuously proposed and improved by scholars. However, the existing MER methods still have some deficiencies in processing Spatio-temporal redundant information and feature extraction. This paper proposes an MER network based on Differential Feature Fusion (DFF) method to solve this problem. First, inputs the onset frame and apex frame of the face, divide each image into small blocks, and uses part of the SE-ResNet50 model for feature extraction. Second, the Spatio-Temporal information of the features is extracted by using a DFF module composed of a differential feature module, CapsuleNet, and a Fully Connected (FC) layer. Finally, inputs the feature vector to the FC module for classification. This study is based on the Leave One Subject Out (LOSO) cross-validation protocol and uses the CASMEII dataset. Experiments and comparisons show the effectiveness of the algorithm.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5

Similar content being viewed by others

References

  1. Bai M, Goecke R (2020) Investigating lstm for micro-expression recognition. In: Companion Publication of the 2020 International Conference on Multimodal Interaction. pp 7–11

  2. Cao Q, Shen L, Xie W, Parkhi OM, Zisserman A (2018) Vggface2: A dataset for recognising faces across pose and age. In: 2018 13th IEEE international conference on automatic face & gesture recognition (FG 2018). IEEE, pp 67–74

  3. Cheng G, Si Y, Hong H, Yao X, Guo L (2020) Cross-scale feature fusion for object detection in optical remote sensing images. IEEE Geosci Remote Sens Lett 18(3):431–435

    Article  Google Scholar 

  4. Ekman P, Friesen WV (1969) Nonverbal leakage and clues to deception. Psychiatry 32(1):88–106

    Article  Google Scholar 

  5. Frank MG, Ekman P (1997) The ability to detect deceit generalizes across different types of high-stake lies. J Pers Soc Psychol 72(6):1429

    Article  Google Scholar 

  6. Han J, Yao X, Cheng G, Feng X, Xu D (2019) P-CNN: Part-based convolutional neural networks for fine-grained visual categorization. Transaction Pattern Analysis Machine Intelligence, 44(2):579–590. https://doi.org/10.1109/TPAMI.2019.2933510

  7. He J, Hu JF, Lu X, Zheng WS (2017) Multi-task mid-level feature learning for micro-expression recognition. Pattern Recogn 66:44–52

    Article  Google Scholar 

  8. He K, Zhang X, Ren S, Sun J (2016) Deep residual learning for image recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp 770–778

  9. Hinton GE, Ghahramani Z, Teh YW (2000) Learning to parse images. Adv Neural Inf Process Syst 12:463–469

    Google Scholar 

  10. Hu J, Shen L, Sun G (2017) Squeeze-and-excitation networks. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp 7132–7141

  11. Huang X, Wang SJ, Zhao G, Piteikainen M (2015) Facial micro-expression recognition using spatiotemporal local binary pattern with integral projection. In: Proceedings of the IEEE international conference on computer vision workshops. pp 1–9

  12. Huang X, Zhao G, Hong X, Zheng W, Pietikäinen M (2016) Spontaneous facial micro-expression analysis using spatiotemporal completed local quantized patterns. Neurocomputing 175:564–578

    Article  Google Scholar 

  13. Huang X, Wang SJ, Liu X, Zhao G, Feng X, Pietikäinen M (2017) Discriminative spatiotemporal local binary pattern with revisited integral projection for spontaneous facial micro-expression recognition. IEEE Trans Affect Comput 10(1):32–47

    Article  Google Scholar 

  14. Jovanović MR, Schmid PJ, Nichols JW (2014) Sparsity-promoting dynamic mode decomposition. Phys Fluids 26(2):024103

    Article  Google Scholar 

  15. Khor HQ, See J, Phan RCW, Lin W (2018) Enriched long-term recurrent convolutional network for facial micro-expression recognition. In: 2018 13th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2018). IEEE, pp 667–674

  16. Kosiorek A, Sabour S, Teh YW, Hinton GE, (2019) Stacked capsule autoencoders. Advances in Neural Information Processing Systems, 32

  17. Le Ngo AC, See J, Phan RCW (2016) Sparsity in dynamics of spontaneous subtle emotions: analysis and application. IEEE Trans Affect Comput 8(3):396–411

    Article  Google Scholar 

  18. Lei L, Chen T, Li S, Li J (2021) Micro-expression recognition based on facial graph representation learning and facial action unit fusion. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp 1571–1580

  19. Li J, Wang Y, See J, Liu W (2019) Micro-expression recognition based on 3d flow convolutional neural network. Pattern Anal Applic 22(4):1331–1339

    Article  MathSciNet  Google Scholar 

  20. Li J, Wang SJ, Yap MH, See J, Hong X, Li X (2020) Megc2020-the third facial micro-expression grand challenge. In: 2020 15th IEEE International Conference on Automatic Face and Gesture Recognition (FG 2020)(FG). IEEE Computer Society, pp 234–237

  21. Li k, Zong Y, Song B, Zhu J (2019) Three-stream convolutional neural network for micro-expressionrecognition. In: International Conference on Neural Information Processing

  22. Li X, Pfister T, Huang X, Zhao G, Pietikäinen M (2013) A spontaneous micro-expression database: Inducement, collection and baseline. In: 2013 10th IEEE International Conference and Workshops on Automatic face and gesture recognition (fg). IEEE, pp 1–6

  23. Li X, Hong X, Moilanen A, Huang X, Pfister T, Zhao G, Pietikäinen M (2017) Towards reading hidden emotions: A comparative study of spontaneous micro-expression spotting and recognition methods. IEEE Trans Affect Comput 9(4):563–577

    Article  Google Scholar 

  24. Liong ST, See J, Wong K, Phan RCW (2018) Less is more: Micro-expression recognition from video using apex frame. Signal Process Image Commun 62:82–92

    Article  Google Scholar 

  25. Liu N, Liu X, Zhang Z, Xu X, Chen T (2020) Offset or onset frame: A multi-stream convolutional neural network with capsulenet module for micro-expression recognition. In: 2020 5th International Conference on Intelligent Informatics and Biomedical Sciences (ICIIBMS). IEEE, pp 236–240

  26. Liu YJ, Zhang JK, Yan WJ, Wang SJ, Zhao G, Fu X (2015) A main directional mean optical flow feature for spontaneous micro-expression recognition. IEEE Trans Affect Comput 7(4):299–310

    Article  Google Scholar 

  27. Masi I, Wu Y, Hassner T, Natarajan P (2018) Deep face recognition: A survey. In: 2018 31st SIBGRAPI conference on graphics, patterns and images (SIBGRAPI). IEEE, pp 471–478

  28. Merghani W, Davison A, Yap M (2018) Facial micro-expressions grand challenge 2018: evaluating spatio-temporal features for classification of objective classes. In: 2018 13th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2018). IEEE, pp 662–666

  29. Peng M, Wu Z, Zhang Z, Chen T (2018) From macro to micro expression recognition: Deep learning on small datasets using transfer learning. In: 2018 13th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2018). IEEE, pp 657–661

  30. Peng W, Hong X, Xu Y, Zhao G (2019) A boost in revealing subtle facial expressions: A consolidated eulerian framework. In: 2019 14th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2019). IEEE, pp 1–5

  31. Pfister T, Li X, Zhao G, Pietikäinen M (2011) Recognising spontaneous facial micro-expressions. In: 2011 international conference on computer vision. IEEE, pp 1449–1456

  32. Polikovsky S, Kameda Y, Ohta Y (2009) Facial micro-expressions recognition using high speed camera and 3d-gradient descriptor. In: 3rd International Conference on Imaging for Crime Detection and Prevention (ICDP 2009). pp 1–6

  33. Polikovsky S, Kameda Y, Ohta Y (2013) Facial micro-expression detection in hi-speed video based on facial action coding system (facs). IEICE Trans Inf Syst 96(1):81–92

    Article  Google Scholar 

  34. Russell TA, Chu E, Phillips ML (2006) A pilot study to investigate the effectiveness of emotion recognition remediation in schizophrenia using the micro-expression training tool. British J Clin Psychol 45(4):579–583

    Article  Google Scholar 

  35. Sabour S, Frosst N, Hinton GE (2017) Dynamic routing between capsules. arXiv preprint arXiv:1710.09829

  36. See J, Yap MH, Li J, Hong X, Wang SJ (2019) Megc 2019–the second facial micro-expressions grand challenge. In: 2019 14th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2019). IEEE, pp 1–5

  37. Soh XR, Baskaran VM, Buhari AM, Phan RCW (2017) A real time micro-expression detection system with lbp-top on a many-core processor. In: 2017 Asia-Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA ASC). IEEE, pp 309–315

  38. Su Y, Zhang J, Liu J, Zhai G (2021) Key facial components guided micro-expression recognition based on first & second-order motion. In: 2021 IEEE International Conference on Multimedia and Expo (ICME). IEEE, pp 1–6

  39. Takalkar MA, Xu M (2017) Image based facial micro-expression recognition using deep learning on small datasets. In: 2017 international conference on digital image computing: techniques and applications (DICTA). IEEE, pp 1–7

  40. Takalkar MA, Xu M, Chaczko Z (2020) Manifold feature integration for micro-expression recognition. Multimedia Systems 26(5):535–551

    Article  Google Scholar 

  41. Van Quang N, Chun J, Tokuyama T (2019) Capsulenet for micro-expression recognition. In: 2019 14th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2019). IEEE, pp 1–7

  42. Wang C, Peng M, Bi T, Chen T (2020) Micro-attention for micro-expression recognition. Neurocomputing 410:354–362

    Article  Google Scholar 

  43. Wang SJ, Yan WJ, Zhao G, Fu X, Zhou CG (2014) Micro-expression recognition using robust principal component analysis and local spatiotemporal directional features. In: European Conference on computer vision. Springer, pp 325–338

  44. Wang SJ, Yan WJ, Li X, Zhao G, Zhou CG, Fu X, Yang M, Tao J (2015) Micro-expression recognition using color spaces. IEEE Trans Image Process 24(12):6034–6047

    Article  MathSciNet  Google Scholar 

  45. Wang SJ, He Y, Li J, Fu X (2021) Mesnet: A convolutional neural network for spotting multi-scale micro-expression intervals in long videos. IEEE Trans Image Process 30:3956–3969

    Article  Google Scholar 

  46. Wang Y, See J, Phan RCW, Oh YH (2014) Lbp with six intersection points: Reducing redundant information in lbp-top for micro-expression recognition. In: Asian conference on computer vision. Springer, pp 525–537

  47. Wang Y, See J, Phan RCW, Oh YH (2015) Efficient spatio-temporal local binary patterns for spontaneous facial micro-expression recognition. PLoS ONE 10(5):e0124674

    Article  Google Scholar 

  48. Wang Y, Ma H, Xing X, Pan Z (2020) Eulerian motion based 3dcnn architecture for facial micro-expression recognition. In: International Conference on Multimedia Modeling. Springer, pp 266–277

  49. Wang Y, Huang Y, Liu C, Gu X, Yang D, Wang S, Zhang B (2021) Micro expression recognition via dual-stream spatiotemporal attention network. Journal Healthcare Engineering 2021;2021:7799100. https://doi.org/10.1155/2021/7799100

  50. Wu C, Guo F (2021) Tsnn: Three-stream combining 2d and 3d convolutional neural network for micro-expression recognition. IEEJ Trans Electr Electron Eng 16(1):98–107

    Article  Google Scholar 

  51. Xia Z, Hong X, Gao X, Feng X, Zhao G (2019) Spatiotemporal recurrent convolutional networks for recognizing spontaneous micro-expressions. IEEE Trans Multimedia 22(3):626–640

    Article  Google Scholar 

  52. Xie HX, Lo L, Shuai HH, Cheng WH (2020) Au-assisted graph attention convolutional network for micro-expression recognition. In: Proceedings of the 28th ACM International Conference on Multimedia. pp 2871–2880

  53. Xu F, Zhang J, Wang JZ (2017) Microexpression identification and categorization using a facial dynamics map. IEEE Trans Affect Comput 8(2):254–267

    Article  Google Scholar 

  54. Yan WJ, Li X, Wang SJ, Zhao G, Liu YJ, Chen YH, Fu X (2014) Casme ii: An improved spontaneous micro-expression database and the baseline evaluation. PLoS ONE 9(1):e86041

  55. Yang B, Cheng J, Yang Y, Zhang B, Li J (2021) Merta: micro-expression recognition with ternary attentions. Multimedia Tools Appl 80(11):1–16. https://doi.org/10.1007/s11042-019-07896-4

    Article  Google Scholar 

  56. Zhao G, Pietikainen M (2007) Dynamic texture recognition using local binary patterns with an application to facial expressions. IEEE Trans Pattern Anal Mach Intell 29(6):915–928

    Article  Google Scholar 

  57. Zhou L, Mao Q, Huang X, Zhang F, Zhang Z (2021) Feature refinement: An expression-specific feature learning and fusion method for micro-expression recognition. arXiv preprint arXiv:2101.04838

  58. Zhou L, Shao XY, Mao QR (2021) A survey of micro-expression recognition. Image Vis Comput 105. https://doi.org/10.1016/j.imavis.2020.104043

  59. Zhou Z, Zhao G, Pietikäinen M (2011) Towards a practical lipreading system. In: CVPR 2011. IEEE, pp 137–144

  60. Zong Y, Huang X, Zheng W, Cui Z, Zhao G (2018) Learning from hierarchical spatiotemporal descriptors for micro-expression recognition. IEEE Trans Multimedia 20(11):3160–3172

    Article  Google Scholar 

Download references

Acknowledgements

This work is supported by the Key Project of the Science and Technology Research Program in University of Hebei Province of China (Grant No. ZD2017209), the Natural Science Foundation of Hebei Province of China (Grant No. F2019201329).

Author information

Authors and Affiliations

  1. School of Cyber Security and Computer, Hebei University, Baoding, Hebei, China

    Ziyang Shang, Penghai Wang & Xinfu Li

Authors
  1. Ziyang Shang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Penghai Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Xinfu Li
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Xinfu Li.

Ethics declarations

Conflicts of interest

We declare that we do not have any commercial or associative interest that represents a conflicts of interest in connection with the work submitted.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Shang, Z., Wang, P. & Li, X. Micro-expression recognition based on differential feature fusion. Multimed Tools Appl 83, 11111–11126 (2024). https://doi.org/10.1007/s11042-023-15626-0

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-023-15626-0

Keywords

Search

Navigation