Skip to main content
SpringerLink
Log in

Shallow multi-branch attention convolutional neural network for micro-expression recognition

  • Regular Paper
  • Published:
Multimedia Systems Aims and scope Submit manuscript

Abstract

Micro-expression recognition (MER) has become challenging because it is difficult to extract subtle facial variations of micro-expressions (MEs). Several approaches have been recently exploited to model MER’s global expression features directly. However, these methods do not identify discriminative ME representations, resulting in suboptimal performance. To address the problem of localization and asymmetry in ME movements, we propose a novel shallow multi-branch attention convolutional neural network (SMBANet) to recognize MEs. SMBANet seeks to obtain faintly distinctive ME characteristics and it contains three components: region division, local expression feature learning, and global feature fusion. First, region division partitions facial areas into four regions, i.e., upper left, upper right, lower left, and lower right half-face. Second, four branches with inception module and proposed efficient residual channel spatial (ERCS) attention module are designed to learn local expression features. Last, ME labels are predicted via global fusion with adaptively weighting four branches’ features. Experiments conducted on the composite database published by MEGC 2019 validate the effectiveness of SMBANet under the composite database evaluation protocol. The results show that SMBANet yields salient and discriminative ME representations and achieves more competitive performance than comparable state-of-the-art methods for MER. And ablation experiments’ results exhibit that our proposed ERCS attention outperforms the classical attention module, i.e., ECA and CBAM.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5

Similar content being viewed by others

Data availability

The data that support the findings of this study are available from the corresponding author upon reasonable request.

References

  1. Ekman, P.: Lie catching and microexpressions. Philos. Decept. 1(2), 5 (2009)

    Google Scholar 

  2. Ben, X., Ren, Y., Zhang, J., Wang, S.-J., Kpalma, K., Meng, W., Liu, Y.-J.: Video-based facial micro-expression analysis: a survey of datasets, features and algorithms. IEEE Trans. Pattern Anal. Mach. Intell. 44, 5826–5846 (2021)

    Google Scholar 

  3. Yan, W-Jing., Qi, Wu., Liang, J., Chen, Y.-H., Xiaolan, F.: How fast are the leaked facial expressions: The duration of micro-expressions. J. Nonverbal Behav. 37(4), 217–230 (2013)

    Article  Google Scholar 

  4. Li J, Dong Z, Lu S, Wang S-J, Yan W-J, Ma Y, Liu Y, Huang C, Fu X: Cas (me) 3: A third generation facial spontaneous micro-expression database with depth information and high ecological validity. IEEE Transactions on Pattern Analysis and Machine Intelligence. 45, 2782–2800 (2022)

  5. Frank M, Herbasz M, Sinuk K, Keller A, Nolan C: I see how you feel: Training laypeople and professionals to recognize fleeting emotions. In The Annual Meeting of the International Communication Association. Sheraton New York, New York City, pages 1–35, (2009)

  6. Zhao, G., Pietikainen, M.: Dynamic texture recognition using local binary patterns with an application to facial expressions. IEEE Trans. Pattern Anal. Mach. Intell. 29(6), 915–928 (2007)

    Article  Google Scholar 

  7. Liong, S.-T., See, J., Wong, K., Phan, R.C.-W.: Less is more micro-expression recognition from video using apex frame. Signal Process. 62, 82–92 (2018)

    Google Scholar 

  8. Kumar AJR, Bhanu B: Micro-expression classification based on landmark relations with graph attention convolutional network. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1511–1520, (2021)

  9. Hao-Yu, Wu., Rubinstein, Michael, Shih, Eugene, Guttag, John, Durand, Frédo., Freeman, William: Eulerian video magnification for revealing subtle changes in the world. ACM Trans. Graphics (TOG) 31(4), 1–8 (2012)

    Google Scholar 

  10. Polikovsky, S., Kameda, Y., Ohta, Y.: Facial micro-expression detection in hi-speed video based on facial action coding system (facs). IEICE Trans. Inf. Syst. 96(1), 81–92 (2013)

    Article  Google Scholar 

  11. Rosenberg, E.L., Ekman, P.: What the face reveals: basic and applied studies of spontaneous expression using the facial action coding System (FACS). Oxford University Press, Oxford (2020)

    Google Scholar 

  12. Liu, Y.-J., Zhang, J.-K., Yan, W.-J., Wang, S.-J., Zhao, G., Xiaolan, F.: A main directional mean optical flow feature for spontaneous micro-expression recognition. IEEE Trans. Affective Comput. 7(4), 299–310 (2015)

    Article  Google Scholar 

  13. O’Toole, A.J., Castillo, C.D.: Face recognition by humans and machines: three fundamental advances from deep learning. Ann. Rev Vis. Sci. 7, 543–570 (2021)

    Article  Google Scholar 

  14. Bouguettaya A, Zarzour H, Kechida A, Taberkit AM: Vehicle detection from uav imagery with deep learning: a review. IEEE Transactions on Neural Networks and Learning Systems. 33, 6047–6067 (2021)

  15. Li, J., Wang, Y., See, J., Liu, W.: Micro-expression recognition based on 3d flow convolutional neural network. Pattern Anal. Appl. 22(4), 1331–1339 (2019)

    Article  MathSciNet  Google Scholar 

  16. Li, Y., Huang, X., Zhao, G.: Joint local and global information learning with single apex frame detection for micro-expression recognition. IEEE Trans. Image Process. 30, 249–263 (2020)

    Article  Google Scholar 

  17. Van Quang N, Chun J, Tokuyama T: Capsulenet for micro-expression recognition. In 2019 14th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2019), pages 1–7. IEEE, (2019)

  18. Sun, B., Cao, S., Li, D., He, J., Yu, L.: Dynamic micro-expression recognition using knowledge distillation. IEEE Trans. Affect. Comput. 13, 1037–1043 (2020)

    Article  Google Scholar 

  19. Wang, G., Huang, S., Dong, Z.: Haphazard cuboids feature extraction for micro-expression recognition. IEEE Access 10, 110149–110162 (2022)

    Article  Google Scholar 

  20. Dong Z, Wang G, Lu S, Yan W-J, Wang S-J: A brief guide: Code for spontaneous expressions and micro-expressions in videos. In Proceedings of the 1st Workshop on Facial Micro-Expression: Advanced Techniques for Facial Expressions Generation and Spotting, pages 31–37 (2021)

  21. Dong Z, Wang G, Lu S, Li J, Yan W, Wang S-J: Spontaneous facial expressions and micro-expressions coding: From brain to face. Front. Psychol. 12, 5808 (2022)

  22. Pfister T, Li X, Zhao G, Pietikäinen M: Recognising spontaneous facial micro-expressions. In 2011 international conference on computer vision, pages 1449–1456. IEEE, (2011)

  23. Wang, S.-J., Yan, W.-J., Li, X., Zhao, G., Zhou, C.-G., Xiaolan, F., Yang, M., Tao, J.: Micro-expression recognition using color spaces. IEEE Trans. Image Process. 24(12), 6034–6047 (2015)

    Article  MathSciNet  MATH  Google Scholar 

  24. Huang, X., Zhao, G., Hong, X., Zheng, W., Pietikäinen, M.: Spontaneous facial micro-expression analysis using spatiotemporal completed local quantized patterns. Neurocomputing 175, 564–578 (2016)

    Article  Google Scholar 

  25. Huang, X., Wang, S.-J., Liu, X., Zhao, G., Feng, X., Pietikäinen, M.: Discriminative spatiotemporal local binary pattern with revisited integral projection for spontaneous facial micro-expression recognition. IEEE Trans. Affective Comput. 10(1), 32–47 (2017)

    Article  Google Scholar 

  26. Peng, M., Wang, C., Chen, T., Liu, G., Xiaolan, F.: Dual temporal scale convolutional neural network for micro-expression recognition. Front. Psychol. 8, 1745 (2017)

    Article  Google Scholar 

  27. Khor H-Q, See J, Liong S-T, Phan RCW, Lin W: Dual-stream shallow networks for facial micro-expression recognition. In 2019 IEEE international conference on image processing (ICIP), pages 36–40. IEEE, (2019)

  28. Wang, C., Peng, M., Bi, T., Chen, T.: Micro-attention for micro-expression recognition. Neurocomputing 410, 354–362 (2020)

    Article  Google Scholar 

  29. Zhao, S., Tao, H., Zhang, Y., Tong, X., Zhang, K., Hao, Z., Chen, E.: A two-stage 3d cnn based learning method for spontaneous micro-expression recognition. Neurocomputing 448, 276–289 (2021)

    Article  Google Scholar 

  30. Huang, S., Zhuang, L.: Exponential discriminant locality preserving projection for face recognition. Neurocomputing 208, 373–377 (2016)

    Article  Google Scholar 

  31. Gan, Y.S., Liong, S.-T., Yau, W.-C., Huang, Y.-C., Tan, L.-K.: Off-apexnet on micro-expression recognition system. Signal Processing 74, 129–139 (2019)

    Google Scholar 

  32. See J, Yap MH, Li J, Hong X, Wang S-J: Megc 2019–the second facial micro-expressions grand challenge. In 2019 14th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2019), pages 1–5. IEEE, (2019)

  33. Liu Y, Du H, Zheng L, Gedeon T: A neural micro-expression recognizer. In 2019 14th IEEE international conference on automatic face & gesture recognition (FG 2019), pages 1–4. IEEE, (2019)

  34. Liong S-T, Gan YS, See J, Khor H-Q, Huang Y-C: Shallow triple stream three-dimensional cnn (ststnet) for micro-expression recognition. In 2019 14th IEEE international conference on automatic face & gesture recognition (FG 2019), pages 1–5. IEEE, (2019)

  35. Zhou L, Mao Q, Xue L: Dual-inception network for cross-database micro-expression recognition. In 2019 14th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2019), pages 1–5. IEEE, (2019)

  36. Xia, Z., Peng, W., Khor, H.-Q., Feng, X., Zhao, G.: Revealing the invisible with model and data shrinking for composite-database micro-expression recognition. IEEE Trans. Image Process. 29, 8590–8605 (2020)

    Article  MATH  Google Scholar 

  37. Nie, X., Takalkar, M.A., Duan, M., Zhang, H., Xu, M.: Geme: dual-stream multi-task gender-based micro-expression recognition. Neurocomputing 427, 13–28 (2021)

    Article  Google Scholar 

  38. Chen B, Liu K-H, Xu Y, Wu Qi-Q, Yao J-F: Block division convolutional network with implicit deep features augmentation for micro-expression recognition. IEEE Transactions on Multimedia (2022)

  39. Guo, M.-H., Xu, T.-X., Liu, J.-J., Liu, Z.-N., Jiang, P.-T., Mu, Tai-Jiang., Zhang, Song-Hai., Martin, Ralph R., Cheng, Ming-Ming., Hu, Shi-Min.: Attention mechanisms in computer vision: a survey. Comput. Vis. Media 8, 1–38 (2022)

    Google Scholar 

  40. Hu J, Shen L, Sun G: Squeeze-and-excitation networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 7132–7141. (2018)

  41. Wang Q, Wu B, Zhu P, Li P, Zuo W, Hu Q: Eca-net: Efficient channel attention for deep convolutional neural networks. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 11531–11539, (2020)

  42. Woo S, Park J, Lee J-Y, Kweon IS: Cbam: Convolutional block attention module. In Proceedings of the European conference on computer vision (ECCV), pages 3–19, (2018)

  43. Ekman, P.: Facial expression and emotion. Am. Psychol. 48(4), 384 (1993)

    Article  Google Scholar 

  44. Li X, Pfister T, Huang X, Zhao G, Pietikäinen M: A spontaneous micro-expression database: Inducement, collection and baseline. In 2013 10th IEEE International Conference and Workshops on Automatic face and gesture recognition (fg), pages 1–6. IEEE, (2013)

  45. Liong S-T, See J, Wong K, Le Ngo AC, Oh Y-H, Phan R: Automatic apex frame spotting in micro-expression database. In 2015 3rd IAPR Asian conference on pattern recognition (ACPR), pages 665–669. IEEE, (2015)

  46. Zhou, L., Mao, Q., Huang, X., Zhang, F., Zhang, Z.: Feature refinement: an expression-specific feature learning and fusion method for micro-expression recognition. Pattern Recogn. 122, 108275 (2022)

    Article  Google Scholar 

  47. Yan, W.-J., Li, X., Wang, S.-J., Zhao, G., Liu, Y.-J., Chen, Yu-Hsin., Xiaolan, Fu.: Casme ii: an improved spontaneous micro-expression database and the baseline evaluation. PloS One 9(1), e86041 (2014)

    Article  Google Scholar 

  48. Davison, A.K., Lansley, C., Costen, N., Tan, K., Yap, M.H.: Samm: a spontaneous micro-facial movement dataset. IEEE Trans. Affect. Comput. 9(1), 116–129 (2018)

    Article  Google Scholar 

  49. Zach, C., Pock, T., Bischof, H.: A duality based approach for realtime tv-l 1 optical flow. In: Joint pattern recognition symposium, pp. 214–223. Springer, New York (2007)

    Chapter  Google Scholar 

  50. Sun D, Roth S, Black MJ: Secrets of optical flow estimation and their principles. In 2010 IEEE computer society conference on computer vision and pattern recognition, pages 2432–2439. IEEE, (2010)

  51. Szegedy C, Liu W, Jia Y, Sermanet P, Reed S, Anguelov D, Erhan D, Vanhoucke V, Rabinovich A: Going deeper with convolutions. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1–9. (2015)

  52. Loshchilov I, Hutter F: Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101, (2017)

  53. Krizhevsky, A., Sutskever, I., Hinton, G.E.: Imagenet classification with deep convolutional neural networks. Communications of the ACM 60(6), 84–90 (2017)

    Article  Google Scholar 

  54. Iandola FN, Han S, Moskewicz MW, Ashraf K, Dally WJ, Keutzer K: Squeezenet: Alexnet-level accuracy with 50x fewer parameters and< 0.5 mb model size. arXiv preprint arXiv:1602.07360, (2016)

  55. He K, Zhang X, Ren S, Sun J: Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778, (2016)

  56. Dosovitskiy A, Fischer P, Ilg E, Hausser P, Hazirbas C, Golkov V, Van Der Smagt, P, Cremers D, Brox T: Flownet: Learning optical flow with convolutional networks. In Proceedings of the IEEE international conference on computer vision, pages 2758–2766, (2015)

Download references

Acknowledgements

This work was supported by the National Natural Science Foundation of China under Grants 62276118 and 61772244.

Author information

Authors and Affiliations

  1. School of Computer, Jiangsu University of Science and Technology, Zhenjiang, 212003, Jiangsu, China

    Gang Wang, Shucheng Huang & Zhe Tao

Authors
  1. Gang Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Shucheng Huang
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Zhe Tao
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

GW wrote the main manuscript text, SH and ZT checked the logic and grammar. All authors reviewed the manuscript.

Corresponding author

Correspondence to Shucheng Huang.

Ethics declarations

Conflict of interest

The authors have no competing interests to declare that are relevant to the content of this article.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Wang, G., Huang, S. & Tao, Z. Shallow multi-branch attention convolutional neural network for micro-expression recognition. Multimedia Systems 29, 1967–1980 (2023). https://doi.org/10.1007/s00530-023-01080-3

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00530-023-01080-3

Keywords

Search

Navigation