Skip to main content
SpringerLink
Log in

Dual-stream network with cross-layer attention and similarity constraint for micro-expression recognition

  • Regular Paper
  • Published:
Multimedia Systems Aims and scope Submit manuscript

Abstract

Micro-expression recognition (MER) is a pivotal research area within human emotion analysis. However, the fleeting, subtle, and complex nature of micro-expressions poses challenges for accurate and efficient recognition. To address this, this paper proposes a Dual-Stream Network with Cross-layer Attention and Similarity Constraint (DSN-CASC) for MER. The network is designed with two parallel branches, each dedicated to learning features of stacked optical flow maps and independent micro-expression optical flow components. The network amplifies its focus on representing features at different hierarchical levels via incorporating a cross-layer attention module, thereby enhancing the capture of key features associated with micro-expression variations. Furthermore, a similarity constraint strategy is introduced to ensure that the micro-expression features extracted by the dual branches exhibit similar representations, improving the network’s overall representation capability. Finally, a simple feature fusion approach is employed for micro-expression classification. Extensive experiments on the composite database validate the effectiveness of DSN-CASC under leave-one-subject-out cross-validation and composite database evaluation protocol. The results demonstrate that our proposed approach achieves promising performance improvements, which provide new insights and effective solutions for MER research.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5

Similar content being viewed by others

Data availibility

No datasets were generated or analysed during the current study.

References

  1. Ekman, P.: Lie catching and microexpressions. Philos. Decep. 1(2), 5 (2009)

    Google Scholar 

  2. Yan, W.-J., Wu, Q., Liang, J., Chen, Y.-H., Fu, X.: How fast are the leaked facial expressions: the duration of micro-expressions. J. Nonverbal Behav. 37(4), 217–230 (2013)

    Article  Google Scholar 

  3. Li, Y., Wei, J., Liu, Y., Kauttonen, J., Zhao, G.: Deep learning for micro-expression recognition: a survey. IEEE Trans. Affect. Comput. 13(4), 2028–2046 (2022)

    Article  Google Scholar 

  4. Ben, X., Ren, Y., Zhang, J., Wang, S.-J., Kpalma, K., Meng, W., Liu, Y.-J.: Video-based facial micro-expression analysis: a survey of datasets, features and algorithms. IEEE Trans. Pattern Anal. Mach. Intell. 44(9), 5826–5846 (2021)

    Google Scholar 

  5. Li, J., Dong, Z., Lu, S., Wang, S.-J., Yan, W.-J., Ma, Y., Liu, Y., Huang, C., Fu, X.: Cas (me) 3: a third generation facial spontaneous micro-expression database with depth information and high ecological validity. IEEE Trans. Pattern Anal. Mach. Intell. 45(3), 2782–2800 (2022)

    Google Scholar 

  6. Frank, M., Herbasz, M., Sinuk, K., Keller, A., Nolan, C.: I see how you feel: training laypeople and professionals to recognize fleeting emotions. In: The Annual Meeting of the International Communication Association. Sheraton New York, New York City, pp. 1–35 (2009)

  7. Rosenberg, E.L., Ekman, P.: What the Face Reveals: Basic and Applied Studies of Spontaneous Expression Using the Facial Action Coding System (FACS). Oxford University Press, Oxford (2020)

    Google Scholar 

  8. Liong, S.-T., Gan, Y.S., See, J., Khor, H.-Q., Huang, Y.-C.: Shallow triple stream three-dimensional cnn (ststnet) for micro-expression recognition. In: 2019 14th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2019), pp. 1–5 (2019). IEEE

  9. Zhou, L., Mao, Q., Xue, L.: Dual-inception network for cross-database micro-expression recognition. In: 2019 14th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2019), pp. 1–5 (2019). IEEE

  10. Wang, G., Huang, S., Tao, Z.: Shallow multi-branch attention convolutional neural network for micro-expression recognition. Multimedia Syst. 29(4), 1967–1980 (2023)

    Article  Google Scholar 

  11. Zhou, H., Huang, S., Xu, Y.: Inceptr: micro-expression recognition integrating inception-cbam and vision transformer. Multimedia Syst. 29(6), 3863–3876 (2023)

    Article  Google Scholar 

  12. Zhai, Z., Zhao, J., Long, C., Xu, W., He, S., Zhao, H.: Feature representation learning with adaptive displacement generation and transformer fusion for micro-expression recognition. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 22086–22095 (2023)

  13. Gan, Y., Lien, S.-E., Chiang, Y.-C., Liong, S.-T.: Laenet for micro-expression recognition. Vis. Comput. 40(2), 585–599 (2024)

    Article  Google Scholar 

  14. Wang, C., Peng, M., Bi, T., Chen, T.: Micro-attention for micro-expression recognition. Neurocomputing 410, 354–362 (2020)

    Article  Google Scholar 

  15. Xia, Z., Peng, W., Khor, H.-Q., Feng, X., Zhao, G.: Revealing the invisible with model and data shrinking for composite-database micro-expression recognition. IEEE Trans. Image Process. 29, 8590–8605 (2020)

    Article  Google Scholar 

  16. Liu, D., Liang, Z., Sun, Y.: Micro-expression recognition method based on spatial attention mechanism and optical flow features. J. Comput. Aid. Des. Comput. Gr. 33, 1541–1552 (2021)

    Google Scholar 

  17. Chaudhry, R., Ravichandran, A., Hager, G., Vidal, R.: Histograms of oriented optical flow and binet-cauchy kernels on nonlinear dynamical systems for the recognition of human actions. In: 2009 IEEE Conference on Computer Vision and Pattern Recognition, pp. 1932–1939 (2009)

  18. Pfister, T., Li, X., Zhao, G., Pietikäinen, M.: Recognising spontaneous facial micro-expressions. In: 2011 International Conference on Computer Vision, pp. 1449–1456 (2011)

  19. Happy, S., Routray, A.: Fuzzy histogram of optical flow orientations for micro-expression recognition. IEEE Trans. Affect. Comput. 10(03), 394–406 (2019)

    Article  Google Scholar 

  20. Huang, X., Wang, S.-J., Liu, X., Zhao, G., Feng, X., Pietikainen, M.: Discriminative spatiotemporal local binary pattern with revisited integral projection for spontaneous facial micro-expression recognition. IEEE Trans. Affect. Comput. 10(01), 32–47 (2019)

    Article  Google Scholar 

  21. Wang, G., Huang, S., Dong, Z.: Haphazard cuboids feature extraction for micro-expression recognition. IEEE Access 10, 110149–110162 (2022)

    Article  Google Scholar 

  22. Wei, J., Lu, G., Yan, J., Zong, Y.: Learning two groups of discriminative features for micro-expression recognition. Neurocomputing 479, 22–36 (2022)

    Article  Google Scholar 

  23. Liu, Y.-J., Zhang, J.-K., Yan, W.-J., Wang, S.-J., Zhao, G., Fu, X.: A main directional mean optical flow feature for spontaneous micro-expression recognition. IEEE Trans. Affect. Comput. 7(4), 299–310 (2015)

    Article  Google Scholar 

  24. Dong, Z., Wang, G., Lu, S., Yan, W.-J., Wang, S.-J.: A brief guide: Code for spontaneous expressions and micro-expressions in videos. In: Proceedings of the 1st Workshop on Facial Micro-Expression: Advanced Techniques for Facial Expressions Generation and Spotting, pp. 31–37 (2021)

  25. Dong, Z., Wang, G., Lu, S., Li, J., Yan, W., Wang, S.-J.: Spontaneous facial expressions and micro-expressions coding: from brain to face. Front. Psychol. 5808 (2022)

  26. Liong, S.-T., See, J., Wong, K., Phan, R.C.-W.: Less is more: micro-expression recognition from video using apex frame. Signal Process. Image Commun. 62, 82–92 (2018)

    Article  Google Scholar 

  27. Peng, M., Wang, C., Chen, T., Liu, G., Fu, X.: Dual temporal scale convolutional neural network for micro-expression recognition. Front. Psychol. 8, 1745 (2017)

    Article  Google Scholar 

  28. Khor, H.-Q., See, J., Liong, S.-T., Phan, R.C., Lin, W.: Dual-stream shallow networks for facial micro-expression recognition. In: 2019 IEEE International Conference on Image Processing (ICIP), pp. 36–40 (2019). IEEE

  29. Gao, J., Zhang, T., Xu, C.: Learning to model relationships for zero-shot video classification. IEEE Trans. Pattern Anal. Mach. Intell. 43(10), 3476–3491 (2020)

    Article  Google Scholar 

  30. Gao, J., Xu, C.: Learning video moment retrieval without a single annotated video. IEEE Trans. Circuits Syst. Video Technol. 32(3), 1646–1657 (2021)

    Article  Google Scholar 

  31. Gan, Y.S., Liong, S.-T., Yau, W.-C., Huang, Y.-C., Tan, L.-K.: Off-apexnet on micro-expression recognition system. Signal Process. Image Commun. 74, 129–139 (2019)

    Article  Google Scholar 

  32. Nie, X., Takalkar, M.A., Duan, M., Zhang, H., Xu, M.: Geme: dual-stream multi-task gender-based micro-expression recognition. Neurocomputing 427, 13–28 (2021)

    Article  Google Scholar 

  33. Zhou, H., Huang, S., Li, J., Wang, S.-J.: Dual-atme: dual-branch attention network for micro-expression recognition. Entropy 25(3), 460 (2023)

    Article  Google Scholar 

  34. Hu, Y., Gao, J., Dong, J., Fan, B., Liu, H.: Exploring rich semantics for open-set action recognition. IEEE Trans. Multimedia (2023)

  35. Gao, J., Chen, M., Xu, C.: Vectorized evidential learning for weakly-supervised temporal action localization. IEEE Trans. Pattern Anal. Mach. Intell. (2023)

  36. Van Quang, N., Chun, J., Tokuyama, T.: Capsulenet for micro-expression recognition. In: 2019 14th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2019), pp. 1–7 (2019). IEEE

  37. Yan, W.-J., Li, X., Wang, S.-J., Zhao, G., Liu, Y.-J., Chen, Y.-H., Fu, X.: Casme ii: an improved spontaneous micro-expression database and the baseline evaluation. PLoS One 9(1), 86041 (2014)

    Article  Google Scholar 

  38. Davison, A.K., Lansley, C., Costen, N., Tan, K., Yap, M.H.: Samm: a spontaneous micro-facial movement dataset. IEEE Trans. Affect. Comput. 9(01), 116–129 (2018)

    Article  Google Scholar 

  39. Li, X., Pfister, T., Huang, X., Zhao, G., Pietikäinen, M.: A spontaneous micro-expression database: Inducement, collection and baseline. In: 2013 10th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition (fg), pp. 1–6 (2013). IEEE

  40. Liong, S.-T., See, J., Wong, K., Le Ngo, A.C., Oh, Y.-H., Phan, R.: Automatic apex frame spotting in micro-expression database. In: 2015 3rd IAPR Asian Conference on Pattern Recognition (ACPR), pp. 665–669 (2015). IEEE

  41. Hou, Q., Zhou, D., Feng, J.: Coordinate attention for efficient mobile network design. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 13713–13722 (2021)

  42. See, J., Yap, M.H., Li, J., Hong, X., Wang, S.-J.: Megc 2019—the second facial micro-expressions grand challenge. In: 2019 14th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2019), pp. 1–5 (2019). IEEE

  43. Loshchilov, I., Hutter, F.: Decoupled weight decay regularization. In: International Conference on Learning Representations (2018)

  44. Hong, T., Longjiao, F.S.Z., Hongmei, L.: Micro-expression recognition based on optical flow method and pseudo three-dimensional residual network. J. Signal Process. 38(05), 1075–1087 (2022)

    Google Scholar 

  45. Shu, X., Li, J., Shi, L., Huang, S.: Res-capsnet: an improved capsule network for micro-expression recognition. Multimedia Syst. 29(3), 1593–1601 (2023)

    Article  Google Scholar 

Download references

Acknowledgements

This work was supported by the National Natural Science Foundation of China under Grants 62276118 and 61772244.

Author information

Authors and Affiliations

  1. School of Computer, Jiangsu University of Science and Technology, Zhenjiang, 212003, Jiangsu, China

    Gang Wang & Shucheng Huang

Authors
  1. Gang Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Shucheng Huang
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

GW wrote the main manuscript text. All authors reviewed the manuscript.

Corresponding author

Correspondence to Shucheng Huang.

Ethics declarations

Conflict of interest

The authors have no Conflict of interest to declare that are relevant to the content of this article.

Additional information

Communicated by J. Gao.

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Wang, G., Huang, S. Dual-stream network with cross-layer attention and similarity constraint for micro-expression recognition. Multimedia Systems 30, 147 (2024). https://doi.org/10.1007/s00530-024-01352-6

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s00530-024-01352-6

Keywords

Search

Navigation