Abstract
Facial Emotion/Expression Recognition (FER) is the key technology which is responsible for analyzing facial emotions from digital images in order to reveal information on the emotional state of a person. It is one of the trending research areas in human computer interaction (HCI). This paper is supposed to provide a recent outlook in this arena by combining saliency map and deep learning approaches. As the usage of deep learning-based approaches in computer vision has revolutionized the way such problems are addressed, deep networks like convolutional neural networks (ConvNet) have become the standard design for image recognition problems. The learning through ConvNet needs a large number of images. Apart from the features of interest, most of the time these image data also carry a bunch of nonessential things, which add to the noise in our dataset. Saliency Maps accomplish this task of fixing critical pixels while ignoring the remaining image background. The present paper divides the whole process of FER into three steps: in the first step, image saliency is detected using a saliency map to focus on important regions of interest from the input dataset. In the second step, data augmentation is applied to balance all the emotion images datasets used in testing. In the third step, a deep convolutional neural network is trained using a modified adaptive moment estimation optimizer (M-Adam) to recognize the facial emotions. After this, the proposed technique is tested on the Japanese female facial expression (JAFFE), the extended Cohn-Kanade (CK+), and the face expression recognition plus dataset (FER+) benchmark datasets. The maximum accuracies achieved with the proposed technique are 97%, 99.8%, and 82.7% for the JAFFE, CK+, and FER + datasets, respectively.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Agrawal A, Mittal N (2019) Using CNN for facial expression recognition: a study of the effects of kernel size and number of filters on accuracy. Visual Comput. https://doi.org/10.1007/s00371-019-01630-9
Al-Sumaidaee SAM, Abdullah MAM, Al-Nima RRO, Dlay SS, Chambers JA (2017) Multi-gradient features and elongated Quinary Pattern encoding for image-based facial expression recognition. Pattern Recogn. https://doi.org/10.1016/j.patcog.2017.06.00
Bhaskar S, Thasleema TM (2023) LSTM model for visual speech recognition through facial expressions. Multimed Tools Appl 82:5455–5472. https://doi.org/10.1007/s11042-022-12796-1
Chang J, Ryoo S (2017) Implementation of an improved facial emotion retrieval method in multimedia system. Multimedia Tools and Applications 77(4):5059–5065. https://doi.org/10.1007/s11042-017-5241-5
Cheng S, Zhou G (2019) Facial expression recognition method based on improved vgg convolutional neural network. Int J Pattern Recognit Artif Intell. https://doi.org/10.1142/s0218001420560030
Choi DY, Song BC (2020) Semi-supervised learning for continuous emotion recognition based on metric learning. IEEE Access. https://doi.org/10.1109/access.2020.3003125
Choi DY, Song BC (2020) Facial micro-expression recognition using two-dimensional landmark feature maps. IEEE Access. https://doi.org/10.1109/access.2020.3006958
Cuiting X, Yan C, Jiang M, Alenezi F, Alhudhaif A, Alnaim N, Polat K, Wanqing W (2022) A novel facial emotion recognition method for stress inference of facial nerve paralysis patients. Expert Syst Appl 197:116705. https://doi.org/10.1016/j.eswa.2022.116705
Deng J, Pang G, Zhang Z, Pang Z, Yang H, Yang G (2019) cGAN based facial expression recognition for human-robot interaction. IEEE Access. https://doi.org/10.1109/access.2019.2891668
Du G, Long S, Yuan H (2020) Non-contact emotion recognition combining heart rate and facial expression for interactive gaming environments. IEEE Access. https://doi.org/10.1109/access.2020.2964794
Duchi J, Hazan E, Singer Y (2011) Adaptive subgradient methods for online learning and stochastic optimization. J Mach Learn Res 12:2121–2159
Ekman P, Friesen W (1978) Facial action coding system: a technique for the measurement of facial movement. Palo Alto: Consulting Psychologists
FER + emotion label. https://github.com/Microsoft/FERPlus (2016) (accessed on 07 August 2020)
Fei Z, Yang E, Li DD-U, Butler S, Ijomah W, Li X, Zhou H (2020) Deep convolution network based emotion analysis towards mental health care. Neurocomputing. https://doi.org/10.1016/j.neucom.2020.01.034
Finzel B, Kollmann R, Rieger I, Pahl J, Schmid (2021) Ute. Deriving temporal prototypes from saliency map clusters for the analysis of deep-learning-based facial action unit classification. In: Proceedings of the LWDA Workshops: FGWM, KDML, FGWI-BIA, and FGIR, 2993 S. 86–97
Gan Y, Chen J, Xu L (2019) Facial expression recognition boosted by soft label with a diverse ensemble. Pattern Recognit Lett 125:105–112. https://doi.org/10.1016/j.patrec.2019.04.002
Gianchandani N, Jaiswal A, Singh D, Kumar V, Kaur M (2020) Rapid COVID-19 diagnosis using ensemble deep transfer learning models from chest radiographic images. J Ambient Intell Humaniz Comput. https://doi.org/10.1007/s12652-020-02669-6
Gupta S, Singh A, Ranjan J (2023) Multimodal, multiview and multitasking depression detection framework endorsed with auxiliary sentiment polarity and emotion detection. Int J Syst Assur Eng Manag. https://doi.org/10.1007/s13198-023-01861-z
Hossain S, Umer S, Rout RK, Tanveer M (2023) Fine-grained image analysis for facial expression recognition using deep convolutional neural networks with bilinear pooling. Appl Soft Comput, p109997
Hua W, Dai F, Huang L, Xiong J, Gui G (2019) HERO: human emotions recognition for realizing intelligent internet of things. IEEE Access 1–1. https://doi.org/10.1109/access.2019.2900231
Itti L, Koch C (2001) Computational modelling of visual attention. Nat Rev Neurosci 2(3):194–203. https://doi.org/10.1038/35058500
Iyer A, Das SS, Teotia R et al (2023) CNN and LSTM based ensemble learning for human emotion recognition using EEG recordings. Multimed Tools Appl 82:4883–4896. https://doi.org/10.1007/s11042-022-12310-7
Jain DK, Shamsolmoali PPS (2019) Extended deep neural network for facial emotion recognition. Pattern Recognit Lett 120:69–74https://doi.org/10.1016/j.patrec.2019.01.008
Jumani SZ, Ali F, Guriro S, Kandhro IA, Khan A, Zaidi A (2019) Facial expression recognition with histogram of oriented gradients using CNN. Indian J Sci Technol 12(24):1–8. https://doi.org/10.17485/ijst/2019/v12i24/145
Khan RA, Meyer A, Konik H et al (2019) Saliency-based framework for facial expression recognition. Front Comput Sci 13:183–198. https://doi.org/10.1007/s11704-017-6114-9
Khattak A, Asghar MZ, Ali M et al (2022) An efficient deep learning technique for facial emotion recognition. Multimed Tools Appl 81:1649–1683. https://doi.org/10.1007/s11042-021-11298-w
Kim J-H, Kima B-G, Roy PP, Jeong D-M (2019) Efficient facial expression recognition lgorithm based on hierarchical deep neural network structure. IEEE Access. https://doi.org/10.1109/access.2019.2907327
Kingma DP, Ba J (2015) Adam: A method for stochastic optimization. In: Published as a conference paper at ICLR 2015
Kumari, Naveen and Bhatia Rekha (2021) Efficient facial emotion recognition model using deep convolutional neural network and modified joint trilateral filter. International Journal of Soft Computing, Vol. 9, No. 1, 2021, https://doi.org/10.1007/s00500-022-06804-7
Lakshmi D, Ponnusamy R (2021) Facial emotion recognition using modified HOG and LBP features with deep stacked autoencoders. Microprocess Microsyst 82:103834. https://doi.org/10.1016/j.micpro.2021.103834
Li D, Wang Z, Wang C, Liu S, Chi W, Dong E, Song Y (2019) The fusion of electroencephalography and facial expression for continuous emotion recognition. IEEE Access 7:155724–155736. https://doi.org/10.1109/access.2019.2949707
Li B, Lima D (2021) Facial expression recognition via ResNet-50. Int J Cognit Comput Eng 2:57–64
Lucey P, Cohn JF, Kanade T, Saragih J, Ambadar Z, Matthews I (2010) The extended Cohn-Kanade dataset (CK+): A complete dataset for action unit and emotion-specified expression. In: 2010 IEEE computer society conference on computer vision and pattern recognition–workshops. https://doi.org/10.1109/cvprw.2010.5543262
Mathur G, Sharma H, Bundele M, Dey N, Paprzycki M (eds) (2020) International Conference on Artificial Intelligence: Advances and Applications 2019. Algorithms for Intelligent Systems https://doi.org/10.1007/978-981-15-1059-5
Minaee S, Minaei M, Abdolrashidi A (2021) Deep-emotion: facial expression recognition using attentional convolutional network. Sensors 21(9):3046. https://doi.org/10.3390/s21093046
Muhammad G, Hossain MS (2021) Emotion recognition for cognitive edge computing using deep learning. IEEE Internet Things J 23(8):16894–16901. https://doi.org/10.1109/JIOT.2021.3058587
Poursaberi A, Noubari HA, Gavrilova M, Yanushkevich SN (2012) Gauss–Laguerre wavelet textural feature fusion with geometrical information for facial expression identification. EURASIP J Image Video Process 2012(1). https://doi.org/10.1186/1687-5281-2012-17
Qayyum H, Majid M, Anwar SM, Khan B (2017) Facial expression recognition using stationary wavelet transform features. Math Problems Eng. doi:https://doi.org/10.1155/2017/9854050
Said Y, Barr M (2021) Human emotion recognition based on facial expressions via deep learning on high-resolution images. Multimed Tools Appl 80:25241–25253. https://doi.org/10.1007/s11042-021-10918-9
Song K (2006) A globally convergent and consistent method for estimating the shape parameter of a generalized gaussian distribution. IEEE Trans Inf Theory 52:510–527. https://doi.org/10.1109/tit.2005.860423
Tieleman T, Hinton G (2012) Lecture 6.5 - RMSProp, COURSERA: Neural Networks for Machine Learning. Technical report
Wang W, Sun Q, Chen T (2019) A fine-grained facial expression database for end-to-end multi-pose facial expression recognition. arXiv: 1907.10838. [Online], Available: https://arxiv.org/abs/1907.10838
Wei Q (2021) “Saliency Maps-Based Convolutional Neural Networks for Facial Expression Recognition,“ in IEEE Access, vol. 9, pp. 76224–76234, doi: https://doi.org/10.1109/ACCESS.2021.3082694
Yadav SP (2021) Emotion recognition model based on facial expressions. Multimed Tools Appl 80:26357–26379. https://doi.org/10.1007/s11042-021-10962-5
Zhang L, Tong MH, Marks TK, Shan H, Cottrell GW (2008) SUN: a bayesian framework for saliency using natural statistics. J Vis 8(7):32. https://doi.org/10.1167/8.7.32
Zhang L, Tong MH, Marks TK, Shan H, Cottrell GW (2008) SUN: a bayesian framework for saliency using natural statistics. J Vis 8:1–20. https://doi.org/10.1167/8.7.32
Zhao K, Chu W-S, Zhang H (2016) Deep region and multi-label learning for facial action unit detection, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3391–3399
Zhe Sun H, Zhang J, Bai M, Liu Zhengping Hu,A discriminatively deep fusion approach with improved conditional GAN (im-cGAN) for facial expression recognition, Pattern Recognition,Volume 135,2023,109157,ISSN 0031-3203,https://doi.org/10.1016/j.patcog.2022.109157
Funding
This research received no specific grant from any funding agency.
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Conflict of interest
The authors declare no competitive interest regarding the publication of this paper.
Human and animal participants
There were no studies in this research that included individuals who were either animals or people.
Informed consent
The article does not have any informed consent.
Additional information
Publisher’s Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
About this article
Cite this article
Kumari, N., Bhatia, R. Deep learning based efficient emotion recognition technique for facial images. Int J Syst Assur Eng Manag 14, 1421–1436 (2023). https://doi.org/10.1007/s13198-023-01945-w
Received:
Revised:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s13198-023-01945-w