Abstract
Human facial behavior is an important information for communication. The study of facial behavior is one of the significant research topics in field of psychology, computer vision and artificial intelligence. In order to improve performance of facial expression and action unit recognition, a face recognition method based on deep-block network proposed in this paper. First, to improve the network performance, we preprocess the input of the network facial image, which includes two operations: face detection and face standardization. Second, deep-block network regards facial parts as the core of expression recognition rather than the whole face and key areas are in charge of specific action units to abate the weak correlation bias, which results in better classification and regression effect. Last, with the purpose of reducing impact of image independent factors, relevant feature map is applied to recognize the associated facial action units, which can promote the accuracy of detection to a certain extent. Experimental results on CK+ and MMI show that proposed method can not only capture the correlation of whole face regions globally, but also can increase network speed caused by too few pooling layers.
Similar content being viewed by others
References
Batista JC, Albiero V, Bellon ORP et al (2017) AUMPNet: simultaneous action units detection and intensity estimation on multipose facial images using a single convolutional neural network[C]. IEEE International Conference on Automatic Face & Gesture Recognition
Chen D, Hua G, Wen F et al (2016) Supervised transformer network for efficient face detection[C]. European Conference on Computer Vision:122–138
Cheok AD, Goh KH, Liu W et al (2004) Human Pacman: a Mobile, wide-area entertainment system based on physical, social, and ubiquitous computing[J]. Pers Ubiquit Comput 8(2):71–81
Cootes TF, Taylor CJ (1993) Active shape model search using local Grey-level models: a quantitative evaluation[C]. British Machine Vision Conference
Dalal N, Triggs B (2005) Histograms of oriented gradients for human detection[C]. IEEE Computer Society Conference on Computer Vision & Pattern Recognition
Deng J, Guo J, Ververas E et al (2020) Retinaface: single-shot multi-level face localisation in the wild[C]. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition:5203–5212
Ekman P, Friesen WV (1978) Facial action coding system (FACS): a technique for the measurement of facial actions[J]. Rivista Di Psichiatria 47(2):126–138
Ghosh S, Laksana E, Scherer S et al (2015) A multi-label convolutional neural network approach to cross-domain action unit detection[C]. International Conference on Affective Computing and Intelligent Interaction (ACII), pp 609–615
Gudi A, Tasli HE, Den Uyl TM et al (2015) Deep learning based FACS Action Unit occurrence and intensity estimation[C]. IEEE International Conference & Workshops on Automatic Face & Gesture Recognition, pp 1–5
Li S, Deng W, Du J (2017) Reliable Crowdsourcing and Deep Locality-Preserving Learning for Expression Recognition in the Wild[C]. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp 2584–2593
Liu X, Kumar BVKV, You J et al (2017) Adaptive Deep Metric Learning for Identity-Aware Facial Expression Recognition[C]. IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pp 522–531
Lowe DG (2004) Distinctive image features from scale-invariant Keypoints[J]. Int J Comput Vis 60(2):91–110
Lucey P, Cohn JF, Kanade T et al (2010) The extended Cohn-Kanade dataset (CK+): a complete dataset for action unit and emotion-specified expression[C]. Computer Vision & Pattern Recognition Workshops
Maaten L, Hendriks E (2012) Action unit classification using active appearance models and conditional random fields[J]. Cogn Process 13(S2):507–518
Parr LA, Waller BM, Burrows AM et al (2010) Brief communication: MaqFACS: a muscle-based facial movement coding system for the rhesus macaque[J]. Am J Phys Anthropol 143(4)
Peng G, Wang S (2018) Weakly Supervised Facial Action Unit Recognition Through Adversarial Training[C]. IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp 2188–2196
Ruiz A, Van De Weijer J, Binefa X (2015) From Emotions to Action Units with Hidden and Semi-Hidden-Task Learning[C]. IEEE International Conference on Computer Vision (ICCV), pp 3703–3711
Simonyan K, Zisserman A (2014) Very deep convolutional networks for large-scale image recognition[J]. Computer Science
Tong Y, Ji Q (2008) Learning bayesian networks with qualitative constraints[C]. IEEE Conference on Computer Vision and Pattern Recognition 2008:1–8
Tran DL, Walecki R, Rudovic O et al (2017) DeepCoder: semi-parametric Variational autoencoders for automatic facial action coding[C]. IEEE International Conference on Computer Vision
Tsér Z, Jeni LA, Lrincz A et al (2016) Deep learning for facial action unit detection under large head poses[M]. Springer International Publishing
Valstar M, Pantic M (2010) Induced disgust, happiness and surprise: an addition to the mmi facial expression database[J]. procinternworkshop on emotion corpora for research on emotion & affect
Valstar MF, Pantic M (2012) Fully automatic recognition of the temporal phases of facial actions[J]. IEEE TRANSACTIONS ON SYSTEMS MAN AND CYBERNETICS PART B-CYBERNETICS
Walecki R, Rudovic O, Pavlovic V et al (2017) Deep Structured Learning for Facial Action Unit Intensity Estimation[C]. IEEE Conference on Computer Vision and Pattern Recognition (CVPR)
Wang C, Zeng J, Shan S et al (2019) Multi-task learning of emotion recognition and facial action unit detection with adaptively weights sharing network[C]. IEEE International Conference on Image Processing (ICIP) 2019:56–60
Wang Z, Li Y, Wang S et al (2013) Capturing global semantic relationships for facial action unit recognition[C]. IEEE International Conference on Computer Vision
Wongpatikaseree K, Hnoohom N, Yuenyong S et al (2021) Facial Action Units Recognition using Deep Learning Model with Bottleneck Features[C]. 25th International Computer Science and Engineering Conference (ICSEC), pp 318–323
Yan J, Wang J, Li Q, et al. Weakly supervised regional and temporal learning for facial action unit recognition[J], 2022.
Zeng N, Zhang H, Song B et al (2018) Facial expression recognition via learning deep sparse autoencoders[J]. Neurocomputing 273:643–649
Zeng Z, Pantic M, Roisman GI et al (2009) A survey of affect recognition methods[J]. IEEE Trans Pattern Anal Mach Intell
Zhang KP, Zhang ZP, Li ZF et al (2016) Joint face detection and alignment using multitask cascaded convolutional networks[J]. Ieee Signal Processing Letters 23(10):1499–1503
Zhang S, Zhu X, Lei Z et al (2017) FaceBoxes: a CPU real-time face detector with high accuracy[C]. IEEE International Joint Conference on Biometrics (IJCB) 2017:1–9
Zhang Y, Dong W, Hu BG et al (2018) Weakly-supervised deep convolutional neural network learning for facial action unit intensity estimation[C]. IEEE/CVF Conference on Computer Vision and Pattern Recognition 2018:2314–2323
Zhao Z, Yang Z, Luo L et al (2016) ML-CNN: A novel deep learning based disease named entity recognition architecture[C]. IEEE International Conference on Bioinformatics and Biomedicine (BIBM), pp 794–794
Zhou Y, Pi J, Shi BE (2017) Pose-independent facial action unit intensity regression based on multi-task deep transfer learning[C]. IEEE International Conference on Automatic Face & Gesture Recognition
Acknowledgements
The authors wish to thank the editor-in-chief, associate editor and reviewers for their insightful comments and suggestions. This work was supported by National Key Technology Research and Development Program of China(2017YFB1402103-3), National Natural Science Foundation of China (61901363, 61901362) and Natural Science Foundation of Shaanxi province, China (2020JQ-648,2019JM-381, 2019JQ-729) and Key Laboratory Foundation of Shaanxi Education Department (20JS086).
Data availability statements
The data that support the findings of this study are available from the corresponding author upon reasonable request.
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Conflict of interest
The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.
Additional information
Publisher’s note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
About this article
Cite this article
Zhao, M., Zhi, Y., Yuan, F. et al. Deep-block network for AU recognition and expression migration. Multimed Tools Appl 82, 25733–25746 (2023). https://doi.org/10.1007/s11042-023-14527-6
Received:
Revised:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11042-023-14527-6