Skip to main content
SpringerLink
Log in

Long Short-Term Perception Network for Dynamic Facial Expression Recognition

  • Conference paper
  • First Online:
Pattern Recognition and Computer Vision (PRCV 2023)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 14429))

Included in the following conference series:

  • 373 Accesses

Abstract

Dynamic facial expression recognition (DFER) presents a difficult challenge, and antecedent methodologies leveraging convolutional neural networks (CNNs), recurrent neural networks (RNNs), or Transformers focus on extracting either long-term temporal information or short-term temporal information from facial videos. Unlike prevailing approaches, we design a novel framework named long short-term perception network (LSTPNet). It can easily perceive aforementioned dual temporal cues and bestow notable advantages upon the DFER task. To be specific, a temporal channel excitation (TCE) module is proposed, building upon the previous outstanding efficient channel attention (ECA) module. This extension serves to imbue the backbone network with temporal attention capabilities, thereby facilitating the acquisition of more enriched temporal features. Furthermore, we design a long short-term temporal Transformer (LSTformer) which can capture both short-term and long-term temporal information with efficacy. The empirical findings, as showcased across three benchmark datasets, unequivocally demonstrate the state-of-the-art performance of LSTPNet.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 69.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 89.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Abdat, F., Maaoui, C., Pruski, A.: Human-computer interaction using emotion recognition from facial expression. In: 2011 UKSim 5th European Symposium on Computer Modeling and Simulation. pp. 196–201. IEEE (2011)

    Google Scholar 

  2. Ayral, T., Pedersoli, M., Bacon, S., Granger, E.: Temporal stochastic softmax for 3d cnns: An application in facial expression recognition. In: Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision. pp. 3029–3038 (2021)

    Google Scholar 

  3. Carreira, J., Zisserman, A.: Quo vadis, action recognition? a new model and the kinetics dataset. In: proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp. 6299–6308 (2017)

    Google Scholar 

  4. Chen, W., Zhang, D., Li, M., Lee, D.J.: Stcam: Spatial-temporal and channel attention module for dynamic facial expression recognition. IEEE Transactions on Affective Computing (2020)

    Google Scholar 

  5. Deng, J., Guo, J., Ververas, E., Kotsia, I., Zafeiriou, S.: Retinaface: Single-shot multi-level face localisation in the wild. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 5203–5212 (2020)

    Google Scholar 

  6. Dhall, A., Goecke, R., Lucey, S., Gedeon, T., et al.: Collecting large, richly annotated facial-expression databases from movies. IEEE Multimedia 19(3), 34 (2012)

    Article  Google Scholar 

  7. Fan, Y., Lu, X., Li, D., Liu, Y.: Video-based emotion recognition using cnn-rnn and c3d hybrid networks. In: Proceedings of the 18th ACM international conference on multimodal interaction. pp. 445–450 (2016)

    Google Scholar 

  8. Fei, Z., Yang, E., Li, D.D.U., Butler, S., Ijomah, W., Li, X., Zhou, H.: Deep convolution network based emotion analysis towards mental health care. Neurocomputing 388, 212–227 (2020)

    Article  Google Scholar 

  9. Guo, J., Han, K., Wu, H., Tang, Y., Chen, X., Wang, Y., Xu, C.: Cmt: Convolutional neural networks meet vision transformers. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 12175–12185 (2022)

    Google Scholar 

  10. Hara, K., Kataoka, H., Satoh, Y.: Can spatiotemporal 3d cnns retrace the history of 2d cnns and imagenet? In: Proceedings of the IEEE conference on Computer Vision and Pattern Recognition. pp. 6546–6555 (2018)

    Google Scholar 

  11. He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 770–778 (2016)

    Google Scholar 

  12. Hu, J., Shen, L., Sun, G.: Squeeze-and-excitation networks. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 7132–7141 (2018)

    Google Scholar 

  13. Jiang, X., Zong, Y., Zheng, W., Tang, C., Xia, W., Lu, C., Liu, J.: Dfew: A large-scale database for recognizing dynamic facial expressions in the wild. In: Proceedings of the 28th ACM international conference on multimedia. pp. 2881–2889 (2020)

    Google Scholar 

  14. Jung, H., Lee, S., Yim, J., Park, S., Kim, J.: Joint fine-tuning in deep neural networks for facial expression recognition. In: Proceedings of the IEEE international conference on computer vision. pp. 2983–2991 (2015)

    Google Scholar 

  15. Kim, D.H., Baddar, W.J., Jang, J., Ro, Y.M.: Multi-objective based spatio-temporal feature representation learning robust to expression intensity variations for facial expression recognition. IEEE Trans. Affect. Comput. 10(2), 223–236 (2017)

    Article  Google Scholar 

  16. King, D.E.: Dlib-ml: A machine learning toolkit. The Journal of Machine Learning Research 10, 1755–1758 (2009)

    Google Scholar 

  17. Li, H., Wang, N., Yang, X., Gao, X.: Crs-cont: a well-trained general encoder for facial expression analysis. IEEE Trans. Image Process. 31, 4637–4650 (2022)

    Article  Google Scholar 

  18. Li, H., Wang, N., Yang, X., Wang, X., Gao, X.: Towards semi-supervised deep facial expression recognition with an adaptive confidence margin. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 4166–4175 (2022)

    Google Scholar 

  19. Li, H., Niu, H., Zhu, Z., Zhao, F.: Intensity-aware loss for dynamic facial expression recognition in the wild. arXiv preprint arXiv:2208.10335 (2022)

  20. Li, H., Sui, M., Zhu, Z., et al.: Nr-dfernet: Noise-robust network for dynamic facial expression recognition. arXiv preprint arXiv:2206.04975 (2022)

  21. Li, S., Deng, W., Du, J.: Reliable crowdsourcing and deep locality-preserving learning for expression recognition in the wild. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 2852–2861 (2017)

    Google Scholar 

  22. Liu, M., Li, S., Shan, S., Wang, R., Chen, X.: Deeply learning deformable facial action parts model for dynamic expression analysis. In: Computer Vision-ACCV 2014: 12th Asian Conference on Computer Vision, Singapore, Singapore, November 1–5, 2014, Revised Selected Papers, Part IV 12. pp. 143–157. Springer (2015)

    Google Scholar 

  23. Lowe, D.G.: Object recognition from local scale-invariant features. In: Proceedings of the seventh IEEE international conference on computer vision. vol. 2, pp. 1150–1157. Ieee (1999)

    Google Scholar 

  24. Ma, F., Sun, B., Li, S.: Spatio-temporal transformer for dynamic facial expression recognition in the wild. arXiv preprint arXiv:2205.04749 (2022)

  25. Qiu, Z., Yao, T., Mei, T.: Learning spatio-temporal representation with pseudo-3d residual networks. In: proceedings of the IEEE International Conference on Computer Vision. pp. 5533–5541 (2017)

    Google Scholar 

  26. Tran, D., Bourdev, L., Fergus, R., Torresani, L., Paluri, M.: Learning spatiotemporal features with 3d convolutional networks. In: Proceedings of the IEEE international conference on computer vision. pp. 4489–4497 (2015)

    Google Scholar 

  27. Tran, D., Wang, H., Torresani, L., Ray, J., LeCun, Y., Paluri, M.: A closer look at spatiotemporal convolutions for action recognition. In: Proceedings of the IEEE conference on Computer Vision and Pattern Recognition. pp. 6450–6459 (2018)

    Google Scholar 

  28. Udayakumar, N.: Facial expression recognition system for autistic children in virtual reality environment. Int. J. Sci. Res. Publ. 6(6), 613–622 (2016)

    Google Scholar 

  29. Vielzeuf, V., Pateux, S., Jurie, F.: Temporal multimodal fusion for video emotion classification in the wild. In: Proceedings of the 19th ACM International Conference on Multimodal Interaction. pp. 569–576 (2017)

    Google Scholar 

  30. Wang, Q., Wu, B., Zhu, P., Li, P., Zuo, W., Hu, Q.: Eca-net: Efficient channel attention for deep convolutional neural networks. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 11534–11542 (2020)

    Google Scholar 

  31. Wang, Y., Sun, Y., Huang, Y., Liu, Z., Gao, S., Zhang, W., Ge, W., Zhang, W.: Ferv39k: a large-scale multi-scene dataset for facial expression recognition in videos. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 20922–20931 (2022)

    Google Scholar 

  32. Woo, S., Park, J., Lee, J.Y., Kweon, I.S.: Cbam: Convolutional block attention module. In: Proceedings of the European conference on computer vision (ECCV). pp. 3–19 (2018)

    Google Scholar 

  33. Yu, M., Zheng, H., Peng, Z., Dong, J., Du, H.: Facial expression recognition based on a multi-task global-local network. Pattern Recogn. Lett. 131, 166–171 (2020)

    Article  Google Scholar 

  34. Zhang, Y., Wang, C., Ling, X., Deng, W.: Learn from all: Erasing attention consistency for noisy label facial expression recognition. In: European Conference on Computer Vision. pp. 418–434. Springer (2022)

    Google Scholar 

  35. Zhao, Z., Liu, Q.: Former-dfer: Dynamic facial expression recognition transformer. In: Proceedings of the 29th ACM International Conference on Multimedia. pp. 1553–1561 (2021)

    Google Scholar 

Download references

Acknowledgments

This work was supported in part by the NSFC under No. 62176169, and Sichuan Science and Technology Projects (2023ZHCG0007, 2022YFQ0056).

Author information

Authors and Affiliations

  1. College of Computer Science, Sichuan University, Chengdu, China

    Chengcheng Lu, Yiben Jiang, Keren Fu, Qijun Zhao & Hongyu Yang

  2. National Key Lab of Fundamental Science on Synthetic Vision, Sichuan University, Chengdu, China

    Keren Fu, Qijun Zhao & Hongyu Yang

Authors
  1. Chengcheng Lu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Yiben Jiang
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Keren Fu
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Qijun Zhao
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Hongyu Yang
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Keren Fu .

Editor information

Editors and Affiliations

  1. Nanjing University of Information Science and Technology, Nanjing, China

    Qingshan Liu

  2. Xiamen University, Xiamen, China

    Hanzi Wang

  3. Beijing University of Posts and Telecommunications, Beijing, China

    Zhanyu Ma

  4. Sun Yat-sen University, Guangzhou, China

    Weishi Zheng

  5. Peking University, Beijing, China

    Hongbin Zha

  6. Chinese Academy of Sciences, Beijing, China

    Xilin Chen

  7. Chinese Academy of Sciences, Beijing, China

    Liang Wang

  8. Xiamen University, Xiamen, China

    Rongrong Ji

Rights and permissions

Reprints and permissions

Copyright information

© 2024 The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd.

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Lu, C., Jiang, Y., Fu, K., Zhao, Q., Yang, H. (2024). Long Short-Term Perception Network for Dynamic Facial Expression Recognition. In: Liu, Q., et al. Pattern Recognition and Computer Vision. PRCV 2023. Lecture Notes in Computer Science, vol 14429. Springer, Singapore. https://doi.org/10.1007/978-981-99-8469-5_14

Download citation

  • DOI: https://doi.org/10.1007/978-981-99-8469-5_14

  • Published:

  • Publisher Name: Springer, Singapore

  • Print ISBN: 978-981-99-8468-8

  • Online ISBN: 978-981-99-8469-5

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation