Abstract
Based on the depth feature of facial micro-expression recognition, such as by using Convolutional Neural Network, the method of classifying facial micro-expression recognition has been gradually improved. Compared with the traditional feature extraction method, the improved technique can more easily realize real-time application. To perfect the details and extract the fine features of micro-expressions, a new algorithm named MACNN (Multi-scale convolutional neural network model), which combines the kernels of atrous convolutions and automatic face correction, is proposed to improve the feature extraction process of the CNN network. Residual blocks are introduced to solve the gradient disappearance problem and to accelerate convergence during the model training. The model is trained and tested on the micro-expression public data sets CASME and CASME II through real-time detection in real-time application of automatic face correction. The robustness of the model is improved by comparing the loss function schemes. The accuracy of the method is 73%, and the real-time detection occurs at a frame rate of 60 FPS. This method can effectively improve the accuracy of micro-expression detection, meet the real-time requirements, and demonstrate satisfactory robustness and generalizability.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Agrawal A, Mittal N (2019) Using CNN for facial expression recognition: a study of the effects of kernel size and number of filters on accuracy. Vis Comput. https://doi.org/10.1007/s00371-019-01630-9
Ben X, Zhang P, Yan R, Yang M, Ge G (2015) Gait recognition and micro-expression recognition based on maximum margin projection with tensor representation. Neural Comput Appl 27(8):1–18
Ekman P, Rosenberg EL (2005) What the face revealsbasic and applied studies of spontaneous expression using the facial action coding system (FACS). Oxford University Press, New York, pp 21–38. https://doi.org/10.1093/acprof:oso/9780195179644.001.0001
He J, Hu JF, Lu X et al (2017) Multi-task mid-level feature learning for micro-expression recognition. Pattern Recogn 66:44–52
Huang T, Chen L, Feng Y, Ben X, Xiao R, Xue T (2019) A multiview representation framework for micro-expression recognition. IEEE Access 7:120670–120680
Huilan L, Chengtao W, Fansheng K (2016) Salient region detection algorithm via kl divergence and multi-scale merging. J Electron Inform Technol
Krishnappa Babu PR, Lahiri U (2019) Classification approach for understanding implications of emotions using eye-gaze. J Ambient Intell Human Comput. https://doi.org/10.1007/s12652-019-01329-8
Lee CC, Shih CY, Lai WP et al (2012) An improved boosting algorithm and its application to facial emotion recognition. J Ambient Intell Human Comput. https://doi.org/10.1007/s12652-011-0085-8
Li X, Hong X, Moilanen A, Huang X, Pfister T, et al (2015). Reading hidden emotions: spontaneous micro-expression spotting and recognition. Comput Sci
Li Q, Yu J, Kurihara T, et al (2018). [IEEE 2018 5th International Conference on control, decision and information technologies (CoDIT)—Thessaloniki, Greece (2018.4.10–2018.4.13)] 2018 5th International Conference on Control, Decision and Information Technologies (CoDIT)—Micro-expression Analysis by Fusing Deep Convolutional Neural Network and Optical Flow. (pp 265–270)
Lin TY, Goyal P, Girshick R, He K, Dollár, Piotr (2017). Focal loss for dense object detection. IEEE Trans Pattern Anal Mach Intell
Liong ST, See J, Wong KS et al (2018) Less is more: micro-expression recognition from video using apex frame. Signal Process Image Commun 62:82–92
Long J, Shelhamer E, Darrell T (2014) Fully convolutional networks for semantic segmentation. IEEE Trans Pattern Anal Mach Intell 39(4):640–651
Lu Z, Luo Z, Zheng H, et al (2014) A delaunay-based temporal coding model for micro-expression recognition
Lu Z, Jiang X, Kot A (2018) Deep coupled ResNet for low-resolution face recognition. IEEE Signal Process Lett, pp 1–1
Peng M, Wang C, Chen T et al (2017) Dual temporal scale convolutional neural network for micro-expression recognition. Front Psychol 8:1745
Perumal Ramalingam S, Chandra Mouli PVSSR (2018) Modified dimensionality reduced local directional pattern for facial analysis. J Ambient Intell Human Comput. https://doi.org/10.1007/s12652-017-0473-9
Pfister T, Li X, Zhao G, et al (2011). Recognising spontaneous facial micro-expressions. IEEE International Conference on Computer Vision, ICCV 2011, Barcelona, Spain, November 6–13, 2011. IEEE
Qiuyu L, Jun Y, Kurihara T, Shu Z (2018) Micro-expression analysis by fusing deep convolutional neural network and optical flow. 2018 5th International Conference on Control, Decision and Information Technologies (CoDIT), Thessaloniki, pp 265–270
Qu CB, Jiang SY, Wu DY (2019). Multiscale semantic segmentation network based on cavity convolution. 55(24): 91–95
Ragupathy P, Vivekanandan P (2019) A modified fuzzy histogram of optical flow for emotion classification. J Ambient Intell Human Comput. https://doi.org/10.1007/s12652-019-01607-5
Redmon J, Farhadi A (2018). YOLOv3: an incremental improvement
Sharma S, Shanmugasundaram K, Ramasamy SK (2016). FAREC—CNN based efficient face recognition technique using Dlib. 2016 International Conference on Advanced Communication Control and Computing Technologies (ICACCCT). IEEE
Su W, Wang Y, Su F, Zhao Z (2018) Micro-expression recognition based on the spatio-temporal feature. 2018 IEEE International Conference on Multimedia & Expo Workshops (ICMEW). IEEE
SuJing W, BingJun L, YongJin L, WenJing Y, Xinyu O, Xiaohua H, et al (2018). Micro-expression recognition with small sample size by transferring long-term convolutional neural network. Neurocomputing
Takalkar MA, Xu M (2017). Image based facial micro-expression recognition using deep learning on small datasets. 2017 International Conference on Digital Image Computing: Techniques and Applications (DICTA). IEEE
Wang SJ, Yan WJ, Sun TK, Zhao GY, Fu XL (2016). Sparse tensor canonical correlation analysis for micro-expression recognition. Neurocomputing
Xia Z, Feng X, Hong X, Zhao G (2018). Spontaneous facial micro-expression recognition via deep convolutional network. pp 1–6
Xia Z, Hong X, Gao X, Feng X, Zhao G (2019). Spatiotemporal recurrent convolutional networks for recognizing spontaneous micro-expressions. IEEE Trans Multimed
Xiang J, Zhu G (2017) [IEEE 2017 4th International Conference on Information Science and Control Engineering (ICISCE)—Changsha (2017.7.21–2017.7.23)] 2017 4th International Conference on Information Science and Control Engineering (ICISCE)—Joint Face Detection and Facial Expression Recognition with MTCNN. International Conference on Information Science & Control Engineering (pp 424–427). IEEE Computer Society
Xianye B, Mingqiang Y, Peng Z, Juan L (2014) Survey on automatic micro expression recognition methods. J Comput Aided Des Comput Graph 26(9):1385–1395
Xu F, Zhang JP (2017) Facial microexpression recognition: a survey. Zidonghua Xuebao/Acta Automatica Sinica 43(3):333–348
Yan WJ, Wu Q, Liang J, Chen YH, Fu X (2013) How fast are the leaked facial expressions: the duration of micro-expressions. J Nonverbal Behav 37(4):217–230
Yan WJ, Wu Q, Liu YJ, et al (2013) CASME database: a dataset of spontaneous micro-expressions collected from neutralized faces. 10th IEEE conference on automatic face and gesture recognition, Shanghai. IEEE
Yan WJ, Li X, Wang SJ et al (2014) Casme II: an improved spontaneous micro-expression database and the baseline evaluation. PLoS ONE 9(1):1–8
Yolcu G, Oztel I, Kazan S et al (2020) Deep learning-based face analysis system for monitoring customer interest. J Ambient Intell Human Comput. https://doi.org/10.1007/s12652-019-01310-5
Zhang P, Ben XY, Yan R, Wu C, Guo C (2016) Micro-expression recognition system. Optik Int J Light Electron Opt 127(3):1395–1400
Zhang ZL, Zhao JW, Cao FL (2019). Constructing deep neural network with void convolution to reconstruct high resolution image. Pattern recognition and artificial intelligence, pp 259–267
Zhao G, Wang SJ, Yan WJ, Fu X, Zhou CG. (2015). Micro-expression recognition using robust principal component analysis and local spatiotemporal directional features
Zhou L, Mao Q and Xue L (2019) Cross-database micro-expression recognition: a style aggregated and attention transfer approach. 2019 IEEE International Conference on Multimedia & Expo Workshops (ICMEW), Shanghai, China, 2019, pp 102–107
Acknowledgements
First of all, we would like to thank the Editor-in-Chief, Associate Editor, and anonymous Reviewers for their valuable comments that greatly improved the presentation of this paper. We made every effort to revise our manuscript according to the Reviewers’ comments. Finally, I also thank the editorial department and reviewers for their recognition and support of the revised manuscript. By the way, this article is supported by the following fund projects including: 1. National Natural Science Foundation of China (61966035) and (61562086). 2. Innovation Team Project of Education Department of Xinjiang Uygur Autonomous Region (XJEDU2016S035). 3. Xinjiang Uygur Autonomous Region Graduate Innovation Project (XJ2019G071), (XJ2019G069) and (XJ2019G072).
Author information
Authors and Affiliations
Corresponding author
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
About this article
Cite this article
Lai, Z., Chen, R., Jia, J. et al. Real-time micro-expression recognition based on ResNet and atrous convolutions. J Ambient Intell Human Comput 14, 15215–15226 (2023). https://doi.org/10.1007/s12652-020-01779-5
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s12652-020-01779-5