Skip to main content
SpringerLink
Log in

Fusing Multi-scale Binary Convolution with Joint Attention Face Expression Recognition Algorithm

  • Conference paper
  • First Online:
Proceedings of 2023 Chinese Intelligent Systems Conference (CISC 2023)

Part of the book series: Lecture Notes in Electrical Engineering ((LNEE,volume 1091))

Included in the following conference series:

  • 331 Accesses

Abstract

Addressing the challenges of wild facial expression datasets being affected by illumination and pose variations, and the expression features being dispersed across multiple easily overlooked facial regions, this paper proposes a facial expression recognition network called Binary and Joint Attention Network. Firstly, The multi-scale binary convolution module integrates texture features of different granularities. Subsequently, the proposed Multi-head Joint Attention module focusing on multiple distinct facial areas through multiple attention heads. Lastly, the inter-attention map loss is designed to prevent attention overlap while assisting in classification. Experimental results on multiple datasets, including SFEW2.0, RAF-DB, FER2013, demonstrate that the proposed network can effectively recognize various facial expressions.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 259.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Hardcover Book
USD 329.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Lyons, M., Akamatsu, S., Kamachi, M., Gyoba, J.: Coding facial expressions with Gabor wavelets. In: Proceedings Third IEEE International Conference on Automatic Face and Gesture Recognition, pp. 200–205. IEEE computer society Nara, Japan (1998). https://doi.org/10.1109/AFGR.1998.670949

  2. Calder, A.J., Burton, A.M., Miller, P., Young, A.W., Akamatsu, S.: A principal component analysis of facial expressions. Vision Res. 41, 1179–1208 (2001). https://doi.org/10.1016/S0042-6989(01)00002-5, https://doi.org/10.1007/11823285_121

  3. Juefei-Xu, F., Boddeti, V.N., Savvides, M.: Local Binary Convolutional Neural Networks. In: 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 4284–4293. IEEE, Honolulu, HI (2017)

    Google Scholar 

  4. Wang, K., Peng, X., Yang, J., Lu, S., Qiao, Y.: Suppressing uncertainties for large-scale facial expression recognition. In: 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 6896–6905. IEEE, Seattle, WA, USA (2020). https://doi.org/10.1109/CVPR42600.2020.00693

  5. Wang, K., Peng, X., Yang, J., Meng, D., Qiao, Y.: Region attention networks for pose and occlusion robust facial expression recognition. IEEE Trans. Image Process. 29, 4057–4069 (2020). https://doi.org/10.1109/TIP.2019.2956143

  6. Li, Y., Zeng, J., Shan, S., Chen, X.: Patch-gated CNN for occlusion-aware facial expression recognition. In: 2018 24th International Conference on Pattern Recognition (ICPR), pp. 2209–2214 (2018). https://doi.org/10.1109/ICPR.2018.8545853

  7. Zhao, Z., Liu, Q., Wang, S.: Learning deep global multi-scale and local attention features for facial expression recognition in the wild. IEEE Trans. Image Process. 30, 6544-6556 (2021). https://doi.org/10.1109/TIP.2021.3093397

  8. He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 770–778. IEEE, Las Vegas, NV, USA (2016). https://doi.org/10.1109/CVPR.2016.90

  9. Ding, X., Guo, Y., Ding, G., Han, J.: ACNet: strengthening the kernel skeletons for powerful CNN via asymmetric convolution blocks. In: 2019 IEEE/CVF International Conference on Computer Vision (ICCV), pp. 1911–1920. IEEE, Seoul, Korea (South) (2019)

    Chapter  Google Scholar 

  10. Woo, S., Park, J., Lee, J.-Y., Kweon, I.S.: Cbam: convolutional block attention module. In: Proceedings of the European Conference on Computer Vision (ECCV), pp. 3–19 (2018)

    Google Scholar 

  11. Teng, X., Deng, X.: Optimization of a helical flow inducer of endovascular stent based on the principle of swirling flow in arterial system. J. Biomed. Eng. 27(2):429–434 (2010)

    Google Scholar 

  12. Teng, X., Hwang, W.: Chapter 4. Structural and dynamical hierarchy of fibrillar collagen. In: Kaunas, R., Zemel, A., (eds) Cell and matrix mechanics, pp. 101–118. Taylor and Francis (2014)

    Google Scholar 

  13. Dhall, A., Goecke, R., Lucey, S., Gedeon, T.: Static facial expression analysis in tough conditions: Data, evaluation protocol and benchmark. In: 2011 IEEE International Conference on Computer Vision Workshops (ICCV Workshops), pp. 2106–2112. IEEE, Barcelona, Spain (2011). https://doi.org/10.1109/ICCVW.2011.6130508

  14. Li, S., Deng, W., Du, J.: Reliable crowdsourcing and deep locality-preserving learning for expression recognition in the wild. In: 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2584–2593. IEEE, Honolulu, HI (2017). https://doi.org/10.1109/CVPR.2017.277

  15. Teng, X., Hwang, W.: Chain registry and load-dependent conformational dynamics of collagen. Biomacromolecules. 15, 3019–3029 (2014). https://doi.org/10.1021/bm500641f

  16. Li, Y., Zeng, J., Shan, S., Chen, X.: Occlusion aware facial expression recognition using CNN with attention mechanism. IEEE Trans. Image Process. 28(5), 2439–2450 (2018)

    Article  MathSciNet  Google Scholar 

  17. Florea, C., Florea, L., Badea, M.S., Vertan, C., Racoviteanu, A. (Sept 2019) Annealed label transfer for face expression recognition. In: BMVC, p. 104

    Google Scholar 

  18. Li, Y., Zeng, J., Shan, S., Chen, X.: Occlusion aware facial expression recognition using CNN with attention mechanism. IEEE Trans. Image Process. 28, 2439–2450 (2019). https://doi.org/10.1109/TIP.2018.2886767

  19. Li, Y., Lu, Y., Li, J., Lu, G.: Separate loss for basic and compound facial expression recognition in the wild. In: Proceedings of The Eleventh Asian Conference on Machine Learning, pp. 897–911. PMLR (2019)

    Google Scholar 

  20. Ferrari, V., Hebert, M., Sminchisescu, C., Weiss, Y., Zeng, J., Shan, S., Chen, X., (eds).: Facial expression recognition with inconsistently annotated datasets. In: ECCV, pp 222–237. Springer International Publishing, Cham (2018). https://doi.org/10.1007/978-3-030-01261-8

  21. Farzaneh, A.H., Qi, X.: Facial expression recognition in the wild via deep attentive center loss. In: 2021 IEEE Winter Conference on Applications of Computer Vision (WACV), pp. 2401–2410 (2021). https://doi.org/10.1109/WACV48630.2021.00245

  22. Deep disturbance-disentangled learning for facial expression recognition. In: Proceedings of the 28th ACM International Conference on Multimedia, https://dl.acm.org/doi/10.1145/3394171.3413907. Accessed 20 March 2023

  23. Ruan, D., Yan, Y., Lai, S., Chai, Z., Shen, C., Wang, H.: Feature Decomposition and Reconstruction Learning for Effective Facial Expression Recognition (2021) http://arxiv.org/abs/2104.05160

  24. Sadeghi, H., Raie, A.-A.: HistNet: histogram-based convolutional neural network with Chi-squared deep metric learning for facial expression recognition. Inf. Sci. 608, 472–488 (2022). https://doi.org/10.1016/j.ins.2022.06.092

  25. Ruan, D., Mo, R., Yan, Y., Chen, S., Xue, J.-H., Wang, H.: Adaptive deep disturbance-disentangled learning for facial expression recognition. Int. J. Comput. Vis. 130, 455–477 (2022). https://doi.org/10.1007/s11263-021-01556-7

  26. Farzaneh, A.H., Qi, X.: Discriminant distribution-agnostic loss for facial expression recognition in the wild. In: 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pp. 1631–1639 (2020). https://doi.org/10.1109/CVPRW50498.2020.00211

  27. Xie, S., Hu, H., Wu, Y.: Deep multi-path convolutional neural network joint with salient region attention for facial expression recognition. Pattern Recognit. 92, 177–191 (2019). https://doi.org/10.1016/j.patcog.2019.03.019

Download references

Author information

Authors and Affiliations

  1. Automation Division, Department of Control Science and Engineering, University of Shanghai for Science and Technology, Shanghai, 200093, China

    Minbo Qin & Lin Li

Authors
  1. Minbo Qin
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Lin Li
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Lin Li .

Editor information

Editors and Affiliations

  1. School of Automation Science and Electrical Engineering, Beihang University, Beijing, China

    Yingmin Jia

  2. School of Automation and Electrical Engineering, University of Science and Technology Beijing, Beijing, China

    Weicun Zhang

  3. School of Mechanical Engineering and Automation, Beihang University, Beijing, China

    Yongling Fu

  4. Ningbo Institute of Materials Technology&Engineering, Chinese Academy of Sciences, Ningbo, Zhejiang, China

    Jiqiang Wang

Rights and permissions

Reprints and permissions

Copyright information

Âİ 2023 The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd.

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Qin, M., Li, L. (2023). Fusing Multi-scale Binary Convolution with Joint Attention Face Expression Recognition Algorithm. In: Jia, Y., Zhang, W., Fu, Y., Wang, J. (eds) Proceedings of 2023 Chinese Intelligent Systems Conference. CISC 2023. Lecture Notes in Electrical Engineering, vol 1091. Springer, Singapore. https://doi.org/10.1007/978-981-99-6886-2_34

Download citation

Publish with us

Policies and ethics

Search

Navigation