Abstract
Facial expression recognition (FER) has become increasingly important in the field of human-computer interaction. This paper proposes an improved method with attention mechanism to improve FER performance. Our approach is grounded in two crucial observations. Firstly, as multiple categories share similar underlying facial characteristics, distinguishing between them may be subtle. Secondly, recognizing facial expressions demands a comprehensive approach by encoding high-order interactions among localized features from multiple facial regions simultaneously. To address these challenges, we introduce our MAN model consisting of three key components: Multi-branch stack Residual Network (MRN), Transitional Attention Network (TAN), and Appropriate Cascade Structure (ACS). The TAN learns objectives to maximize class separability, while the MRN deploys attention heads to focus on various facial regions and generate attention maps. Additionally, the ACS module provides a more reasonable construction method for the model. Comprehensive experiments on three publicly available datasets (AffectNet, RAF-DB, and CK+) consistently achieves better expression recognition performance. Compared to the ResNet network, our approach yielded an improved accuracy of 3.3%, 3.2%, 4.1% and 1.8% on the Affectnet7 dataset, Affectnet8 dataset, RAF-DB dataset, and CK+ dataset, respectively.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Szajnberg, N.: What the face reveals: basic and applied studies of spontaneous expression using the facial action coding system. J. Am. Psychoanal. Assoc. 70(3), 591–595 (2022)
Wang, X., Zhou, Z.: Facial age estimation by total ordering Preserving Projection. In: Proceedings of PRICAI 2016: Trends in Artificial Intelligence, pp. 603–615 (2016)
Wang, S., Yan, W., Li, X.: Micro-expression recognition using color spaces. IEEE Trans. Image Process. 24(12), 6034–6047 (2015)
Fasel, B.: Automatic facial expression analysis: a survey. Pattern Recogn. 36(1), 259–275 (2003)
Mai, G., Guo, Z., She, Y., Wang, H., Liang, Y.: Video-based emotion recognition in the wild for online education systems. In: Khanna, S., Cao, J., Bai, Q., Guandong, X. (eds.) PRICAI 2022: Trends in Artificial Intelligence: 19th Pacific Rim International Conference on Artificial Intelligence, PRICAI 2022, Shanghai, China, November 10–13, 2022, Proceedings, Part III, pp. 516–529. Springer, Cham (2022). https://doi.org/10.1007/978-3-031-20868-3_38
Fei, Z., Yang, E., Yu, L.: A novel deep neural network-based emotion analysis system for automatic detection of mild cognitive impairment in the elderly. Neurocomputing 468, 306–316 (2022)
Jingyi, W., Qiu, B., Shang, L.: A calibration method for sentiment time series by deep clustering. In: Pham, D.N., Theeramunkong, T., Governatori, G., Liu, F. (eds.) PRICAI 2021: Trends in Artificial Intelligence: 18th Pacific Rim International Conference on Artificial Intelligence, PRICAI 2021, Hanoi, Vietnam, November 8–12, 2021, Proceedings, Part II, pp. 3–16. Springer, Cham (2021). https://doi.org/10.1007/978-3-030-89363-7_1
Cai, J., Meng, Z., Khan, A., et al.: Island loss for learning discriminative features in facial expression recognition. In: Proceedings of 13th IEEE International Conference on Automatic Face & Gesture Recognition (FG), pp. 302–309 (2018)
Farzaneh, A., Qi, X., Facial expression recognition in the wild via deep attentive center loss. In: Proceedings of IEEE Winter Conference on Applications of Computer Vision (WACV), pp. 2401–2410 (2021)
Li, Z., Wu, S., Xiao, G., et al.: Facial expression recognition by multi-scale cnn with regularized center Loss. In: Proceedings of 24th International Conference on Pattern Recognition (ICPR), pp. 3384–3389 (2018)
Wang, K., Peng, X., Yang, J.: Region attention networks for pose and occlusion robust facial expression recognition. IEEE Trans. Image Process. 29, 4057–4069 (2020)
Shao, Z., Liu, Z., Cai, J., et al.: Facial action unit detection using attention and relation learning. IEEE Trans. Affect. Comput. 13(3), 1274–1289 (2022)
Zhang, J., Liu, F., Zhou, A.: Off-tanet: a lightweight neural micro-expression recognizer with optical flow features and integrated attention mechanism. In: Pham, D.N., Theeramunkong, T., Governatori, G., Liu, F. (eds.) PRICAI 2021: Trends in Artificial Intelligence: 18th Pacific Rim International Conference on Artificial Intelligence, PRICAI 2021, Hanoi, Vietnam, November 8–12, 2021, Proceedings, Part I, pp. 266–279. Springer International Publishing, Cham (2021). https://doi.org/10.1007/978-3-030-89188-6_20
Tian, Z., Shen, C., Chen, H., et al.: Fcos: a simple and strong anchor-free object detector. IEEE Trans. Pattern Anal. Mach. Intell. 44(4), 1922–1933 (2022)
He, K., Zhang, X., Ren, S., et al.: Deep residual learning for image recognition. In: Proceedings of 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 770–778 (2016)
Howard, A., Sandler, M., Chu, G., et al.: Searching for mobilenetV3. In: Proceedings of IEEE/CVF International Conference on Computer Vision (ICCV), pp. 1314–24 (2019)
Vaswani, A., Shazeer, N., Parmar, N., et al.: Attention is all you need. In: Proceedings of 31st Annual Conference on Neural Information Processing Systems (NIPS), pp. 2401–2410 (2017)
Guo, M., Xu, T., Liu, J.: Attention mechanisms in computer vision: a survey. Comput. Visual Media 8(3), 331–368 (2022)
Hu, J., Shen, L., Albanie, S.: Squeeze-and-excitation networks. IEEE Trans. Pattern Anal. Mach. Intell. 42(8), 2011–2023 (2020)
Woo, S., Park, J., Lee, J.: Cbam: convolutional block attention module. In: Proceedings of 15th European Conference on Computer Vision (ECCV), pp. 3–19 (2018)
Wang, Q., Wu, B., Zhu, P.: Eca-net: efficient channel attention for deep convolutional neural networks. In: Proceedings of 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2020)
Wen, Z., Lin, W., Wang, T.: Distract your attention: multi-head cross attention network for facial expression recognition. arXiv preprint arXiv:2109.07270 (2021)
Ma, N., Zhang, X., Zheng, H.: ShuffleNet V2: practical guidelines for efficient cnn architecture design. In: Proceedings of 15th European Conference on Computer Vision (ECCV), pp. 122–138 (2018)
Dollar, P., Singh, M., Girshick, R..: Fast and accurate model scaling. In: Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 924–932 (2021)
Wang, C., Bochkovskiy, A., Liao, H.: Scaled-YOLOv4: scaling cross stage partial network. In: Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 13024–13033 (2021)
Lee, Y., Hwang, J., Lee, S.: An energy and gpu-computation efficient backbone network for real-time object detection. In: Proceedings of 32nd IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 752–760 (2019)
Wang, C., Bochkovskiy, A., Liao, H.: YOLOv7: Trainable bag-of-freebies sets new state-of-the-art for real-time object detectors. arXiv preprint arXiv:2207.02696 (2022)
Acknowledgement
This study is supported by the Shanghai Pujiang Program (No. 22PJ1403800) and the Natural Science Foundation of China (NSFC) under Grant 62203290.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2024 The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd.
About this paper
Cite this paper
Feng, W., Fei, Z., Zhou, W., Fei, M. (2024). Multiple Attention Network for Facial Expression Recognition. In: Liu, F., Sadanandan, A.A., Pham, D.N., Mursanto, P., Lukose, D. (eds) PRICAI 2023: Trends in Artificial Intelligence. PRICAI 2023. Lecture Notes in Computer Science(), vol 14327. Springer, Singapore. https://doi.org/10.1007/978-981-99-7025-4_12
Download citation
DOI: https://doi.org/10.1007/978-981-99-7025-4_12
Published:
Publisher Name: Springer, Singapore
Print ISBN: 978-981-99-7024-7
Online ISBN: 978-981-99-7025-4
eBook Packages: Computer ScienceComputer Science (R0)