Abstract
Human ideas and sentiments are mirrored in facial expressions. Facial expression recognition (FER) is a crucial type of visual data that can be utilized to deduce a person’s emotional state. It gives the spectator a plethora of social cues, such as the viewer’s focus of attention, emotion, motivation, and intention. It’s said to be a powerful instrument for silent communication. AI-based facial recognition systems can be deployed at different areas like bus stations, railway stations, airports, or stadiums to help security forces identify potential threats. There has been a lot of research done in this area. But, it lacks a detailed review of the literature that highlights and analyses the previous work in FER (including work on compound emotion and micro-expressions), and a comparative analysis of different models applied to available datasets, further identifying aligned future directions. So, this paper includes a comprehensive overview of different models that can be used in the field of FER and a comparative study of the traditional methods based on hand-crafted feature extraction and deep learning methods in terms of their advantages and disadvantages which distinguishes our work from existing review studies.This paper also brings you to an eye on the analysis of different FER systems, the performance of different models on available datasets, evaluation of the classification performance of traditional and deep learning algorithms in the context of facial emotion recognition which reveals a good understanding of the classifier’s characteristics. Along with the proposed models, this study describes the commonly used datasets showing the year-wise performance achieved by state-of-the-art methods which lacks in the existing manuscripts. At last, the authors itemize recognized research gaps and challenges encountered by researchers which can be considered in future research work.
Similar content being viewed by others
Data Availability
All the dataset links are provided in the paper.
References
Shiffrar M, Kaiser MD, Chouchourelou A (2011) Seeing human movement as inherently social. Sci Soc Vis 248–264
Mehrabian A (1981) Silent messages: implicit communication of emotions and attitudes. Wadsworth Pub, Co
Wallbott HG (1998) Bodily expression of emotion. Eur J Soc Psychol 28(6):879–896
De Gelder B (2006) Towards the neurobiology of emotional body language. Nature Rev Neurosci 7(3):242–249
Meeren HK, van Heijnsbergen CC, de Gelder B (2005) Rapid perceptual integration of facial expression and emotional body language. Proc Natl Acad Sci 102(45):16518–16523
Aviezer H, Trope Y, Todorov A (2012) Body cues, not facial expressions, discriminate between intense positive and negative emotions. Science 338(6111):1225–1229
Boyle EA, Anderson AH, Newlands A (1994) The effects of visibility on dialogue and performance in a cooperative problem solving task. Lang Speech 37(1):1–20
Nguyen T, Bass I, Li M, Sethi IK (2005) Investigation of combining svm and decision tree for emotion classification. In: Seventh IEEE International Symposium on Multimedia (ISM’05), p 5. IEEE
Luo Y, Ye J, Adams RB, Li J, Newman MG, Wang JZ (2020) Arbee: Towards automated recognition of bodily expression of emotion in the wild. Int j comput vis 128(1):1–25
Zhang T, Zheng W, Cui Z, Zong Y, Yan J, Yan K (2016) A deep neural network-driven feature learning method for multi-view facial expression recognition. IEEE Trans Multimed 18(12):2528–2536
Verma B, Choudhary A (2021) Affective state recognition from hand gestures and facial expressions using grassmann manifolds. Multimed Tools Appl 80(9):14019–14040
Verma B, Choudhary A (2018) Deep learning based real-time driver emotion monitoring. In: 2018 IEEE International Conference on Vehicular Electronics and Safety (ICVES), p 1–6. IEEE
Altameem A, Kumar A, Poonia RC, Kumar S, Saudagar AKJ (2021) Early identification and detection of driver drowsiness by hybrid machine learning. IEEE Access 9:162805–162819
Reece AG, Danforth CM (2017) Instagram photos reveal predictive markers of depression. EPJ Data Sci 6(1):15
Manikonda L, De Choudhury M (2017) Modeling and understanding visual attributes of mental health disclosures in social media. In: Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems, p 170–181
Kindness P, Masthoff J, Mellish C (2017) Designing emotional support messages tailored to stressors. Int J Hum Comp Studies 97:1–22
Kaliouby Re, Picard R, Baron-Cohen S (2006) Affective computing and autism. Ann N Y Acad Sci 1093(1):228–248
Liu C, Conn K, Sarkar N, Stone W (2008) Physiology-based affect recognition for computer-assisted intervention of children with autism spectrum disorder. Int J Hum Comput Studies 66(9):662–677
Muhammad G, Alsulaiman M, Amin SU, Ghoneim A, Alhamid MF (2017) A facial-expression monitoring system for improved healthcare in smart cities. IEEE Access 5:10871–10881
Uddin MZ, Hassan MM, Almogren A, Alamri A, Alrubaian M, Fortino G (2017) Facial expression recognition utilizing local direction-based robust features and deep belief network. IEEE Access 5:4525–4536
Bishay M, Palasek P, Priebe S, Patras I (2019) Schinet: Automatic estimation of symptoms of schizophrenia from facial behaviour analysis. IEEE Trans Affect Comput
Chen Z, Ansari R, Wilkie D (2019) Learning pain from action unit combinations: a weakly supervised approach via multiple instance learning. IEEE Trans Affect Comput
Lamba PS, Virmani D (2019) Information retrieval from facial expression using voting to assert exigency. J Discrete Math Sci Cryptogr 22(2):177–190
Zheng K, Yang D, Liu J, Cui J (2020) Recognition of teachers’ facial expression intensity based on convolutional neural network and attention mechanism. IEEE Access 8:226437–226444
Ashwin T, Guddeti RMR (2019) Unobtrusive behavioral analysis of students in classroom environment using non-verbal cues. IEEE Access 7:150693–150709
Dawood A, Turner S, Perepa P (2018) Affective computational model to extract natural affective states of students with asperger syndrome (as) in computer-based learning environment. IEEE Access 6:67026–67034
Dampage U, Egodagamage D, Waidyaratne A, Dissanayaka D, Senarathne A (2021) Spatial augmented reality based customer satisfaction enhancement and monitoring system. IEEE Access 9:97990–98004
Aghamaleki JA, Ashkani Chenarlogh V (2019) Multi-stream cnn for facial expression recognition in limited training data. Multimed Tools Appl 78(16):22861–22882
Li BY, Mian AS, Liu W, Krishna A (2013) Using kinect for face recognition under varying poses, expressions, illumination and disguise. In: 2013 IEEE Workshop on Applications of Computer Vision (WACV), p 186–192. IEEE
Saeed U, Masood K, Dawood H (2021) Illumination normalization techniques for makeup-invariant face recognition. Comput Electric Eng 89:106921
Štruc V, Pavesic N (2009) Image normalization techniques for robust face recognition, p 155–160
Li S, Deng W (2020) Deep facial expression recognition: A survey. IEEE trans affect comput
Wen G, Chang T, Li H, Jiang L (2020) Dynamic objectives learning for facial expression recognition. IEEE Trans Multimed 22(11):2914–2925
Pisal A, Sor R, Kinage K (2017) Facial feature extraction using hierarchical max (hmax) method. In: 2017 International Conference on Computing, Communication, Control and Automation (ICCUBEA), p 1–5. IEEE
Liliana DY, Widyanto MR, BasaruddinT (2018) Geometric facial components feature extraction for facial expression recognition. In: 2018 International Conference on Advanced Computer Science and Information Systems (ICACSIS), p 391–396. IEEE
Boughrara H, Chtourou M, Ben Amar C, Chen L (2016) Facial expression recognition based on a mlp neural network using constructive training algorithm. Multimed Tools Appl 75(2):709–731
El Zarif N, Montazeri L, Leduc-Primeau F, Sawan M (2021) Mobile-optimized facial expression recognition techniques. IEEE Access 9:101172–101185
Pons G, Masip D (2017) Supervised committee of convolutional neural networks in automated facial expression analysis. IEEE Trans Affect Comput 9(3):343–350
Canal FZ, Müller TR, Matias JC, Scotton GG, de Sa Junior AR, Pozzebon E, Sobieranski AC (2022) A survey on facial emotion recognition techniques: A state-of-the-art literature review. Inf Sci 582:593–617
Harms MB, Martin A, Wallace GL (2010) Facial emotion recognition in autism spectrum disorders: a review of behavioral and neuroimaging studies. Neuropsychol Rev 20:290–322
Nelson CA (2001) The development and neural bases of face recognition. Infant Child Develop Int J Res Pract 10(1–2):3–18
Mostafa A, Khalil MI, Abbas H (2018) Emotion recognition by facial features using recurrent neural networks. In: 2018 13th International Conference on Computer Engineering and Systems (ICCES), p 417–422. IEEE
Calvo MG, Fernández-Martín A, Gutiérrez-García A, Lundqvist D (2018) Selective eye fixations on diagnostic face regions of dynamic emotional expressions: Kdef-dyn database. Sci Rep 8(1):1–10
Pantic M, Rothkrantz LJM (2000) Automatic analysis of facial expressions: The state of the art. IEEE TrAns Pattern Anal Mach Intell 22(12):1424–1445
Fukuda T, Jung M-J, Nakashima M, Arai F, Hasegawa Y (2004) Facial expressive robotic head system for human-robot communication and its application in home environment. Proc IEEE 92(11):1851–1865
Ekman P (1993) Facial expression and emotion. Am Psychol 48(4):384
Benitez-Quiroz CF, Srinivasan R, Martinez AM (2018) Discriminant functional learning of color features for the recognition of facial action units and their intensities. IEEE Trans Pattern Anal Mach Intell 41(12):2835–2845
Cimtay Y, Ekmekcioglu E, Caglar-Ozhan S (2020) Cross-subject multimodal emotion recognition based on hybrid fusion. IEEE Access 8:168865–168878
Kollias D, Zafeiriou S (2020) Exploiting multi-cnn features in cnn-rnn based dimensional emotion recognition on the omg in-the-wild dataset. IEEE Trans Affect Comput 12(3):595–606
Liu J, Wang H, Feng Y (2021) An end-to-end deep model with discriminative facial features for facial expression recognition. IEEE Access 9:12158–12166
Zhou N, Liang R, Shi W (2020) A lightweight convolutional neural network for real-time facial expression detection. IEEE Access 9:5573–5584
Fard AP, Mahoor MH (2022) Ad-corre: Adaptive correlation-based loss for facial expression recognition in the wild. IEEE Access 10:26756–26768
Verma B, Choudhary A (2018) A framework for driver emotion recognition using deep learning and grassmann manifolds. In: 2018 21st International Conference on Intelligent Transportation Systems (ITSC), p 1421–1426. IEEE
Alam M, Vidyaratne LS, Iftekharuddin KM (2018) Sparse simultaneous recurrent deep learning for robust facial expression recognition. IEEE Trans Neural Netw Learn Syst 29(10):4905–4916
Nguyen DH, Kim S, Lee G-S, Yang H-J, Na I-S, Kim SH (2019) Facial expression recognition using a temporal ensemble of multi-level convolutional neural networks. IEEE Trans Affect Comput
Jin X, Lai Z, Jin Z (2021) Learning dynamic relationships for facial expression recognition based on graph convolutional network. IEEE Trans Image Process 30:7143–7155
Perveen N, Roy D, Mohan CK (2018) Spontaneous expression recognition using universal attribute model. IEEE Trans Image Process 27(11):5575–5584
Miao S, Xu H, Han Z, Zhu Y (2019) Recognizing facial expressions using a shallow convolutional neural network. IEEE Access 7:78000–78011
Xia Z, Hong X, Gao X, Feng X, Zhao G (2019) Spatiotemporal recurrent convolutional networks for recognizing spontaneous micro-expressions. IEEE Trans Multimed 22(3):626–640
Liu Y-J, Li B-J, Lai Y-K (2018) Sparse mdmo: Learning a discriminative feature for micro-expression recognition. IEEE Trans Affect Comput 12(1):254–261
Wang S-J, He Y, Li J, Fu X (2021) Mesnet: A convolutional neural network for spotting multi-scale micro-expression intervals in long videos. IEEE Trans Image Process 30:3956–3969
Hoai M, De la Torre F (2014) Max-margin early event detectors. Int J Comput Vis 107(2):191–202
Xie L, Tao D, Wei H (2018) Early expression detection via online multi-instance learning with nonlinear extension. IEEE Trans Neural Netw Learn Syst 30(5):1486–1496
Cortes C, Vapnik V (1995) Support-vector networks. Mach Learn 20(3):273–297
Ruiz-Garcia A, Elshaw M, Altahhan A, Palade V (2018) A hybrid deep learning neural approach for emotion recognition from facial expressions for socially assistive robots. Neural Comput Appl 29(7):359–373
Kulkarni K, Corneanu CA, Ofodile I, Escalera S, Baro X, Hyniewska S, Allik J, Anbarjafari G (2018) Automatic recognition of facial displays of unfelt emotions. IEEE Trans Affect Comput 12(2):377–390
Ab Wahab MN, Nazir A, Ren ATZ, Noor MHM, Akbar MF, Mohamed ASA (2021) Efficientnet-lite and hybrid cnn-knn implementation for facial expression recognition on raspberry pi. IEEE Access 9:134065–134080
Srivastava A (2021) Impact of k-nearest neighbour on classification accuracy in knn algorithm using machine learning. In: Advances in Smart Communication and Imaging Systems, p 363–373. Springer,
Tarnowski P, Kołodziej M, Majkowski A, Rak RJ (2017) Emotion recognition using facial expressions. Proc Comput Sci 108:1175–1184
Salmam FZ, Madani A, Kissi M (2016) Facial expression recognition using decision trees. In: 2016 13th International Conference on Computer Graphics, Imaging and Visualization (CGiV), p 125–130. IEEE
Bailly K, Dubuisson S et al (2017) Dynamic pose-robust facial expression recognition by multi-view pairwise conditional random forests. IEEE Trans Affect Comput 10(2):167–181
Das M, Ghosh SK (2016) Deep-step: A deep learning approach for spatiotemporal prediction of remote sensing data. IEEE Geosci Remote Sens Lett 13(12):1984–1988
Shickel B, Tighe PJ, Bihorac A, Rashidi P (2017) Deep ehr: a survey of recent advances in deep learning techniques for electronic health record (ehr) analysis. IEEE J Biomed Health Inf 22(5):1589–1604
Shao L, Wu D, Li X (2014) Learning deep and wide: A spectral method for learning deep networks. IEEE Trans Neural Netw Learn Syst 25(12):2303–2308
Zhiqiang W, Jun L (2017) A review of object detection based on convolutional neural network. In: 2017 36th Chinese Control Conference (CCC), p 11104–11109. IEEE
Li K, Jin Y, Akram MW, Han R, Chen J (2020) Facial expression recognition with convolutional neural networks via a new face cropping and rotation strategy. Vis Comput 36(2):391–404
Phung VH, Rhee EJ (2019) A high-accuracy model average ensemble of convolutional neural networks for classification of cloud image patches on small datasets. Appl Sci 9(21):4500
Kim J-H, Kim B-G, Roy PP, Jeong D-M (2019) Efficient facial expression recognition algorithm based on hierarchical deep neural network structure. IEEE Access 7:41273–41285
Li M, Xu H, Huang X, Song Z, Liu X, Li X (2018) Facial expression recognition with identity and emotion joint learning. IEEE Trans Affect Comput 12(2):544–550
Fujii K, Sugimura D, Hamamoto T (2020) Hierarchical group-level emotion recognition. IEEE Trans Multimed 23:3892–3906
Zhang H, Jolfaei A, Alazab M (2019) A face emotion recognition method using convolutional neural network and image edge computing. IEEE Access 7:159081–159089
Xie S, Hu H (2017) Facial expression recognition with frr-cnn. Electron Lett 53(4):235–237
Li J, Zhang D, Zhang J, Zhang J, Li T, Xia Y, Yan Q, Xun L (2017) Facial expression recognition with faster r-cnn. Proc Comput Sci 107:135–140
Huang GB, Mattar M, Berg T, Learned-Miller E (2008) Labeled faces in the wild: A database forstudying face recognition in unconstrained environments. In: Workshop on Faces in’Real-Life’Images: Detection, Alignment, and Recognition
Kim J, Kang J-K, Kim Y (2021) A resource efficient integer-arithmetic-only fpga-based cnn accelerator for real-time facial emotion recognition. IEEE Access 9:104367–104381
Zhou Y, Jin L, Liu H, Song E (2020) Color facial expression recognition by quaternion convolutional neural network with gabor attention. IEEE Trans Cogn Develop Syst
Rumelhart G, Hinton R (1986) Williams, learning representations by back-propagating errors. Nature 323:533–536
Jarraya SK, Masmoudi M, Hammami M (2020) Compound emotion recognition of autistic children during meltdown crisis based on deep spatio-temporal analysis of facial geometric features. IEEE Access 8:69311–69326
Wang T, Wen C-K, Wang H, Gao F, Jiang T, Jin S (2017) Deep learning for wireless physical layer: Opportunities and challenges. China Commun 14(11):92–111
Yang B, Cao J, Ni R, Zhang Y (2017) Facial expression recognition using weighted mixture deep neural network based on double-channel facial images. IEEE Access 6:4630–4640
Li T-HS, Kuo P-H, Tsai T-N, Luan P-C (2019) Cnn and lstm based facial expression analysis model for a humanoid robot. IEEE Access 7:93998–94011
Yan S (2016) Understanding lstm and its diagrams. MLReview. com
Dencelin LX, Ramkumar T (2016) Analysis of multilayer perceptron machine learning approach in classifying protein secondary structures. Biomed Res India 27:166–173
Choi DY, Song BC (2020) Facial micro-expression recognition using two-dimensional landmark feature maps. IEEE Access 8:121549–121563
Gong C, Lin F, Zhou X, Lü X (2019) Amygdala-inspired affective computing: To realize personalized intracranial emotions with accurately observed external emotions. China Commun 16(8):115–129
Marques DB, Barradas Filho AO, Romariz AR, Viegas IM, Luz DA, Barros Filho AK, Labidi S, Ferraudo AS (2014) Recent developments on statistical and neural network tools focusing on biodiesel quality. Int J Comput Sci Appl 3(3):97–110
Bhuvaneshwari M, Kanaga EGM, Anitha J, Raimond K, George ST (2021) A comprehensive review on deep learning techniques for a bci-based communication system. Demystifying Big Data Mach Learn Deep Learn Healthcare Anal 131–157
Poux D, Allaert B, Ihaddadene N, Bilasco IM, Djeraba C, Bennamoun M (2021) Dynamic facial expression recognition under partial occlusion with optical flow reconstruction. IEEE Trans Image Process 31:446–457
Fu Y, Wu X, Li X, Pan Z, Luo D (2020) Semantic neighborhood-aware deep facial expression recognition. IEEE Trans Image Process 29:6535–6548
Ng A, Ngiam J, Foo CY, Mai Y, Suen C, Coates A, Maas A, Hannun A, Huval B, Wang T et al (2015) Deep learning tutorial. Univ, Stanford
Zhao Z-j, Gu J-w (2015) Recognition of digital modulation signals based on hybrid three-order restricted boltzmann machine. In: 2015 IEEE 16th International Conference on Communication Technology (ICCT), p 166–169. IEEE
Wang Y, Li Y, Song Y, Rong X (2019) The application of a hybrid transfer algorithm based on a convolutional neural network model and an improved convolution restricted boltzmann machine model in facial expression recognition. IEEE Access 7:184599–184610
Lin R, Yang F, Gao M, Wu B, Zhao Y (2019) Aud-mts: An abnormal user detection approach based on power load multi-step clustering with multiple time scales. Energies 12(16):3144
Goodfellow I, Pouget-Abadie J, Mirza M, Xu B, Warde-Farley D, Ozair S, Courville A, Bengio Y (2014) Generative adversarial nets. Adv neural inf process syst 27
Yan Y, Huang Y, Chen S, Shen C, Wang H (2019) Joint deep learning of facial expression synthesis and recognition. IEEE Trans Multimed 22(11):2792–2807
Li C, Wang Y, Zhang X, Gao H, Yang Y, Wang J (2019) Deep belief network for spectral-spatial classification of hyperspectral remote sensor data. Sensors 19(1):204
Dosovitskiy A, Beyer L, Kolesnikov A, Weissenborn D, Zhai X, Unterthiner T, Dehghani M, Minderer M, Heigold G, Gelly S, Uszkoreit J, Houlsby N (2021) An image is worth 16x16 words: Transformers for image recognition at scale. In: International Conference on Learning Representations. https://openreview.net/forum?id=YicbFdNTTy
Ma F, Sun B, Li S (2021) Facial expression recognition with visual transformers and attentional selective fusion. IEEE Trans Affect Comput
Xue F, Wang Q, Guo G (2021) Transfer: Learning relation-aware facial expression representations with transformers. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, p 3601–3610
Huang Q, Huang C, Wang X, Jiang F (2021) Facial expression recognition with grid-wise attention and visual transformer. Inf Sci 580:35–54
Liu C, Hirota K, Dai Y (2023) Patch attention convolutional vision transformer for facial expression recognition with occlusion. Inf Sci 619:781–794
Vaswani A, Shazeer N, Parmar N, Uszkoreit J, Jones L, Gomez AN, Kaiser Ł, Polosukhin I (2017) Attention is all you need. Advances in neural information processing systems 30
Liu X, Cheng X, Lee K (2020) Ga-svm-based facial emotion recognition using facial geometric features. IEEE Sensors J 21(10):11532–11542
Zhen Q, Huang D, Drira H, Amor BB, Wang Y, Daoudi M (2017) Magnifying subtle facial motions for effective 4d expression recognition. IEEE Trans Affect Comput 10(4):524–536
Xu F, Zhang J, Wang JZ (2017) Microexpression identification and categorization using a facial dynamics map. IEEE Trans Affect Comput 8(2):254–267
Zhao G, Yang H, Yu M (2020) Expression recognition method based on a lightweight convolutional neural network. IEEE Access 8:38528–38537
Li Y, Zeng J, Shan S, Chen X (2018) Occlusion aware facial expression recognition using cnn with attention mechanism. IEEE Trans Image Process 28(5):2439–2450
Liu C, Hirota K, Ma J, Jia Z, Dai Y (2021) Facial expression recognition using hybrid features of pixel and geometry. IEEE Access 9:18876–18889
Zhang H, Su W, Yu J, Wang Z (2020) Identity-expression dual branch network for facial expression recognition. IEEE Trans Cogn Develop Syst 13(4):898–911
Zhang H, Su W, Wang Z (2020) Weakly supervised local-global attention network for facial expression recognition. IEEE Access 8:37976–37987
Kabakus AT (2020) Pyfer: A facial expression recognizer based on convolutional neural networks. IEEE Access 8:142243–142249
Song B, Li K, Zong Y, Zhu J, Zheng W, Shi J, Zhao L (2019) Recognizing spontaneous micro-expression using a three-stream convolutional neural network. IEEE Access 7:184537–184551
Xie S, Hu H (2018) Facial expression recognition using hierarchical features with deep comprehensive multipatches aggregation convolutional neural networks. IEEE Trans Multimed 21(1):211–220
Salih WM, Nadher I, Tariq A (2019) Deep learning for face expressions detection: Enhanced recurrent neural network with long short term memory. In: International Conference on Applied Computing to Support Industry: Innovation and Technology, p 237–247. Springer
Hong Q-B, Wu C-H, Su M-H, Chang C-C (2019) Exploring macroscopic and microscopic fluctuations of elicited facial expressions for mood disorder classification. IEEE Trans Affect Comput 12(4):989–1001
Barman A, Dutta P (2021) Facial expression recognition using distance and shape signature features. Pattern Recognit Lett 145:254–261
Nie S, Wang Z, Ji Q (2015) A generative restricted boltzmann machine based method for high-dimensional motion data modeling. Comput Vis Image Underst 136:14–22
Li D, Li Z, Luo R, Deng J, Sun S (2019) Multi-pose facial expression recognition based on generative adversarial network. IEEE Access 7:143980–143989
Kim S, Nam J, Ko BC (2022) Facial expression recognition based on squeeze vision transformer. Sensors 22(10):3729
Cohn JF, Zlochower AJ, Lien J, Kanade T (1999) Automated face analysis by feature point tracking has high concurrent validity with manual facs coding. Psychophysiology 36(1):35–43
Zeng G, Zhou J, Jia X, Xie W, Shen L (2018) Hand-crafted feature guided deep learning for facial expression recognition. In: 2018 13th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2018), p 423–430. IEEE
Kumar S, Bhuyan MK, Chakraborty BK (2016) Extraction of informative regions of a face for facial expression recognition. IET Comput Vis 10(6):567–576
Kurup AR, Ajith M, Ramón MM (2019) Semi-supervised facial expression recognition using reduced spatial features and deep belief networks. Neurocomputing 367:188–197
Jain DK, Shamsolmoali P, Sehdev P (2019) Extended deep neural network for facial emotion recognition. Pattern Recognit Lett 120:69–74
Sen D, Datta S, Balasubramanian R (2019) Facial emotion classification using concatenated geometric and textural features. Multimed Tools Appl 78(8):10287–10323
Fei Z, Yang E, Li DD-U, Butler S, Ijomah W, Li X, Zhou H (2020) Deep convolution network based emotion analysis towards mental health care. Neurocomputing 388:212–227
Shahid AR, Khan S, Yan H (2020) Contour and region harmonic features for sub-local facial expression recognition. J Vis Commun Image Represent 73:102949
Chowdary MK, Nguyen TN, Hemanth DJ (2021) Deep learning-based facial emotion recognition for human-computer interaction applications. Neural Comput Appl 1–18
Saurav S, Saini R, Singh S (2021) Facial expression recognition using dynamic local ternary patterns with kernel extreme learning machine classifier. IEEE Access 9:120844–120868
Liu J, Feng Y, Wang H (2021) Facial expression recognition using pose-guided face alignment and discriminative features based on deep learning. IEEE Access 9:69267–69277
Niu B, Gao Z, Guo B (2021) Facial expression recognition with lbp and orb features. Comput Intell Neurosci 2021
Kas M, Ruichek Y, Messoussi R et al (2021) New framework for person-independent facial expression recognition combining textural and shape analysis through new feature extraction approach. Inf Sci 549:200–220
Shao J, Cheng Q (2021) E-fcnn for tiny facial expression recognition. Appl Intell 51:549–559
Wan F, Zhi R (2022) Gaussian distribution-based facial expression feature extraction network. Pattern Recognit Lett 164:104–111
Ahadit AB, Jatoth RK (2022) A novel multi-feature fusion deep neural network using hog and vgg-face for facial expression classification. Mach Vis Appl 33(4):1–23
Dar T, Javed A, Bourouis S, Hussein HS, Alshazly H (2022) Efficient-swishnet based system for facial emotion recognition. IEEE Access 10:71311–71328
Umer S, Rout RK, Pero C, Nappi M (2022) Facial expression recognition with trade-offs between data augmentation and deep learning features. J Ambient Intell Humaniz Comput 13(2):721–735
Yang J, Lv Z, Kuang K, Yang S, Xiao L, Tang Q (2022) Rasn: Using attention and sharing affinity features to address sample imbalance in facial expression recognition. IEEE Access 10:103264–103274
Bentoumi M, Daoud M, Benaouali M, Taleb Ahmed A (2022) Improvement of emotion recognition from facial images using deep learning and early stopping cross validation. Multimed Tools appl 1–31
Zou W, Zhang D, Lee D-J (2022) A new multi-feature fusion based convolutional neural network for facial expression recognition. Appl Intell 52(3):2918–2929
Aifanti N, Papachristou C, Delopoulos A (2010) The mug facial expression database. In: 11th International Workshop on Image Analysis for Multimedia Interactive Services WIAMIS 10, p 1–4. IEEE
Gopalan N, Bellamkonda S, Chaitanya VS (2018) Facial expression recognition using geometric landmark points and convolutional neural networks. In: 2018 International Conference on Inventive Research in Computing Applications (ICIRCA), p 1149–1153. IEEE
Barman A, Dutta P (2018) Facial expression recognition using distance signature feature. In: Advanced Computational and Communication Paradigms: Proceedings of International Conference on ICACCP 2017, Volume 2, p 155–163. Springer
Verma M, Vipparthi SK, Singh G (2019) Hinet: Hybrid inherited feature learning network for facial expression recognition. IEEE Lett Comput Soc 2(4):36–39
Barman A, Dutta P (2019) Influence of shape and texture features on facial expression recognition. IET Image Process 13(8):1349–1363
Barman A, Dutta P (2019) Facial expression recognition using distance and texture signature relevant features. Appl Soft Comput 77:88–105
Dirik M (2022) Optimized anfis model with hybrid metaheuristic algorithms for facial emotion recognition. Int J Fuzzy Syst 1–12
Lyons, M., Akamatsu, S., Kamachi, M., Gyoba, J (1998) Coding facial expressions with gabor wavelets. In: Proceedings Third IEEE International Conference on Automatic Face and Gesture Recognition, p 200–205. IEEE
Dubey AK, Jain V (2020) Automatic facial recognition using vgg16 based transfer learning model. J Inf Optim Sci 41(7):1589–1596
Minaee S, Minaei M, Abdolrashidi A (2021) Deep-emotion: Facial expression recognition using attentional convolutional network. Sensors 21(9):3046
Kola DGR, Samayamantula SK (2021) A novel approach for facial expression recognition using local binary pattern with adaptive window. Multimed Tools Appl 80(2):2243–2262
Mahesh VG, Chen C, Rajangam V, Raj ANJ, Krishnan PT (2021) Shape and texture aware facial expression recognition using spatial pyramid zernike moments and law’s textures feature set. IEEE Access 9:52509–52522
Barros P, Sciutti A (2022) Across the universe: Biasing facial representations toward non-universal emotions with the face-stn. IEEE Access 10:103932–103947
Tsalera E, Papadakis A, Samarakou M, Voyiatzis I (2022) Feature extraction with handcrafted methods and convolutional neural networks for facial emotion recognition. Appl Sci 12(17):8455
Su C, Wei J, Lin D, Kong L (2022) Using attention lsgb network for facial expression recognition. Pattern Anal Appl 1–11
Lundqvist D, Flykt A, Öhman A (1998) Karolinska directed emotional faces. Cogn Emot
Pandey RK, Karmakar S, Ramakrishnan A, Saha N (2019) Improving facial emotion recognition systems with crucial feature extractors. In: Image Analysis and Processing–ICIAP 2019: 20th International Conference, Trento, Italy, September 9–13, 2019, Proceedings, Part I 20, p 268–279. Springer
Fei Z, Yang E, Li D, Butler S, Ijomah W, Zhou H (2019) Combining deep neural network with traditional classifier to recognize facial expressions. In: 2019 25th International Conference on Automation and Computing (ICAC), p 1–6. IEEE
Akhand M, Roy S, Siddique N, Kamal MAS, Shimamura T (2021) Facial emotion recognition using transfer learning in the deep cnn. Electronics 10(9):1036
Cho S, Lee J (2022) Learning local attention with guidance map for pose robust facial expression recognition. IEEE Access 10:85929–85940
Goodfellow IJ, Erhan D, Carrier PL, Courville A, Mirza M, Hamner B, Cukierski W, Tang Y, Thaler D, Lee D-H, et al (2013) Challenges in representation learning: A report on three machine learning contests. In: International Conference on Neural Information Processing, p 117–124. Springer
Zhang X, Ma Y (2018) Improving ensemble learning performance with complementary neural networks for facial expression recognition. In: International Conference on Artificial Neural Networks, p 747–759. Springer
Vorontsov A, Averkin A (2018) Comparison of different convolution neural network architectures for the solution of the problem of emotion recognition by facial expression. In: Proceedings of the VIII International Conference “Distributed Computing and Grid-technologies in Science and Education” (GRID 2018), Dubna, Moscow Region, Russia, p 35–40
Ramdhani B, Djamal EC, Ilyas R (2018) Convolutional neural networks models for facial expression recognition. In: 2018 International Symposium on Advanced Intelligent Informatics (SAIN), p 96–101. IEEE
Hua W, Dai F, Huang L, Xiong J, Gui G (2019) Hero: Human emotions recognition for realizing intelligent internet of things. IEEE Access 7:24321–24332
Georgescu M-I, Ionescu RT, Popescu M (2019) Local learning with deep and handcrafted features for facial expression recognition. IEEE Access 7:64827–64836
Talegaonkar I, Joshi K, Valunj S, Kohok R, Kulkarni A (2019) Real time facial expression recognition using deep learning. In: Proceedings of International Conference on Communication and Information Processing (ICCIP)
Jiang P, Wan B, Wang Q, Wu J (2020) Fast and efficient facial expression recognition using a gabor convolutional network. IEEE Signal Process Lett 27:1954–1958
Kim JH, Poulose A, Han DS (2021) The extensive usage of the facial image threshing machine for facial emotion recognition performance. Sensors 21(6):2026
Liang X, Xu L, Zhang W, Zhang Y, Liu J, Liu Z (2022) A convolution-transformer dual branch network for head-pose and occlusion facial expression recognition. Vis Comput 1–14
Yen C-T, Li K-H (2022) Discussions of different deep transfer learning models for emotion recognitions. IEEE Access 10:102860–102875
Liu H, Cai H, Lin Q, Li X, Xiao H (2022) Adaptive multilayer perceptual attention network for facial expression recognition. IEEE Trans Circuits Syst Vid Technol 32(9):6253–6266
Lu Y, Wang S, Zhao W, Zhao Y (2019) Wgan-based robust occluded facial expression recognition. IEEE Access 7:93594–93610
Barsoum E, Zhang C, Ferrer CC, Zhang Z (2016) Training deep networks for facial expression recognition with crowd-sourced label distribution. In: Proceedings of the 18th ACM International Conference on Multimodal Interaction, p 279–283
Albanie S, Nagrani A, Vedaldi A, Zisserman A (2018) Emotion recognition in speech using cross-modal transfer in the wild. In: Proceedings of the 26th ACM International Conference on Multimedia, p 292–301
Wang K, Peng X, Yang J, Meng D, Qiao Y (2020) Region attention networks for pose and occlusion robust facial expression recognition. IEEE Trans Image Process 29:4057–4069
Wang K, Peng X, Yang J, Lu S, Qiao Y (2020) Suppressing uncertainties for large-scale facial expression recognition. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p 6897–6906
Li H, Wang N, Ding X, Yang X, Gao X (2021) Adaptively learning facial expression representation via cf labels and distillation. IEEE Trans Image Process 30:2016–2028
Li S, Deng W, Du J (2017) Reliable crowdsourcing and deep locality-preserving learning for expression recognition in the wild. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, p 2852–2861
Wang Z, Zeng F, Liu S, Zeng B (2021) Oaenet: Oriented attention ensemble for accurate facial expression recognition. Pattern Recognit 112:107694
Cao S, Yao Y, An G (2020) E2-capsule neural networks for facial expression recognition using au-aware attention. IET Image Process 14(11):2417–2424
Zhao Z, Liu Q, Wang S (2021) Learning deep global multi-scale and local attention features for facial expression recognition in the wild. IEEE Trans Image Process 30:6544–6556
Farzaneh AH, Qi X (2021) Facial expression recognition in the wild via deep attentive center loss. In: Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, p 2402–2411
Saurav S, Saini R, Singh S (2021) Emnet: a deep integrated convolutional neural network for facial emotion recognition in the wild. Appl Intell 51:5543–5570
Yan W-J, Li X, Wang S-J, Zhao G, Liu Y-J, Chen Y-H, Fu X (2014) Casme ii: An improved spontaneous micro-expression database and the baseline evaluation. PloS One 9(1):86041
Liong S-T, See J, Wong K, Phan RC-W (2018) Less is more: Micro-expression recognition from video using apex frame. Signal Process Image Commun 62:82–92
Zong Y, Huang X, Zheng W, Cui Z, Zhao G (2018) Learning from hierarchical spatiotemporal descriptors for micro-expression recognition. IEEE Trans Multimed 20(11):3160–3172
Zhi R, Xu H, Wan M, Li T (2019) Combining 3d convolutional neural networks with transfer learning by supervised pre-training for facial micro-expression recognition. IEICE Trans Inf Syst 102(5):1054–1064
Verma M, Vipparthi SK, Singh G, Murala S (2019) Learnet: Dynamic imaging network for micro expression recognition. IEEE Trans Image Process 29:618–1627
Li Y, Huang X, Zhao G (2020) Joint local and global information learning with single apex frame detection for micro-expression recognition. IEEE Trans Image Process 30:249–263
Xie H-X, Lo L, Shuai H-H, Cheng W-H (2020) Au-assisted graph attention convolutional network for micro-expression recognition. In: Proceedings of the 28th ACM International Conference on Multimedia, p 2871–2880
Cen S, Yu Y, Yan G, Yu M, Yang Q (2020) Sparse spatiotemporal descriptor for micro-expression recognition using enhanced local cube binary pattern. Sensors 20(16):4437
Wang C, Peng M, Bi T, Chen T (2020) Micro-attention for micro-expression recognition. Neurocomputing 410:354–362
Kumar AJR, Bhanu B (2021) Micro-expression classification based on landmark relations with graph attention convolutional network. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p 1511–1520
Saeed U (2021) Facial micro-expressions as a soft biometric for person recognition. Pattern Recognit Lett 143:95–103
Wei J, Lu G, Yan J (2021) A comparative study on movement feature in different directions for micro-expression recognition. Neurocomputing 449:159–171
Liu K-H, Jin Q-S, Xu H-C, Gan Y-S, Liong S-T (2021) Micro-expression recognition using advanced genetic algorithm. Signal Process Image Commun 93:116153
Nie X, Takalkar MA, Duan M, Zhang H, Xu M (2021) Geme: Dual-stream multi-task gender-based micro-expression recognition. Neurocomputing 427:13–28
Cai L, Li H, Dong W, Fang H (2022) Micro-expression recognition using 3d densenet fused squeeze-and-excitation networks. Appl Soft Comput 119:108594
Wei J, Lu G, Yan J, Liu H (2022) Micro-expression recognition using local binary pattern from five intersecting planes. Multimed Tools Appl 1–26
Liu S, Ren Y, Li L, Sun X, Song Y, Hung C-C (2022) Micro-expression recognition based on squeezenet and c3d. Multimed Syst 1–10
Langner O, Dotsch R, Bijlstra G, Wigboldus DH, Hawk ST, Van Knippenberg A (2010) Presentation and validation of the radboud faces database. Cogn Emot 24(8):1377–1388
González-Hernández F, Zatarain-Cabada R, Barrón-Estrada ML, Rodríguez-Rangel H (2018) Recognition of learning-centered emotions using a convolutional neural network. J Intell Fuzzy Syst 34(5):3325–3336
Sun N, Li Q, Huan R, Liu J, Han G (2019) Deep spatial-temporal feature fusion for facial expression recognition in static images. Pattern Recognit Lett 119:49–61
Yolcu G, Oztel I, Kazan S, Oz C, Palaniappan K, Lever TE, Bunyak F (2019) Facial expression recognition for monitoring neurological disorders based on convolutional neural network. Multimed Tools Appl 78(22):31581–31603
Sun N, Lu Q, Zheng W, Liu J, Han G (2020) Unsupervised cross-view facial expression image generation and recognition. IEEE Trans Affect Comput
He J, Yu X, Sun B, Yu L (2021) Facial expression and action unit recognition augmented by their dependencies on graph convolutional networks. J Multimodal User Interfaces 15(4):429–440
Fan X, Jiang M, Shahid AR, Yan H (2022) Hierarchical scale convolutional neural network for facial expression recognition. Cogn Neurodynamics 1–12
Mollahosseini A, Hasani B, Mahoor MH (2017) Affectnet: A database for facial expression, valence, and arousal computing in the wild. IEEE Trans Affect Comput 10(1):18–31
Hung SC, Lee J-H, Wan TS, Chen C-H, Chan Y-M, Chen C-S (2019) Increasingly packing multiple facial-informatics modules in a unified deep-learning model via lifelong learning. In: Proceedings of the 2019 on International Conference on Multimedia Retrieval, p 339–343
Vo T-H, Lee G-S, Yang H-J, Kim S-H (2020) Pyramid with super resolution for in-the-wild facial expression recognition. IEEE Access 8:131988–132001
Schoneveld L, Othmani A, Abdelkawy H (2021) Leveraging recent advances in deep learning for audio-visual emotion recognition. Pattern Recognit Lett 146:1–7
Davison AK, Lansley C, Costen N, Tan K, Yap MH (2016) Samm: A spontaneous micro-facial movement dataset. IEEE Trans Affect Comput 9(1):116–129
Khor H-Q, See J, Liong S-T, Phan RC, Lin W (2019) Dual-stream shallow networks for facial micro-expression recognition. In: 2019 IEEE International Conference on Image Processing (ICIP), p 36–40. IEEE
Lei L, Li J, Chen T, Li S (2020) A novel graph-tcn with a graph structured representation for micro-expression recognition. In: Proceedings of the 28th ACM International Conference on Multimedia, p 2237–2245
Pfister T, Li X, Zhao G, Pietikäinen M (2011) Recognising spontaneous facial micro-expressions. In: 2011 International Conference on Computer Vision, p 1449–1456. IEEE
Zhang Z, Yi M, Xu J, Zhang R, Shen J (2020) Two-stage recognition and beyond for compound facial emotion recognition. In: 2020 15th IEEE International Conference on Automatic Face and Gesture Recognition (FG 2020), p 900–904. IEEE
Dai Y, Feng L (2021) Cross-domain few-shot micro-expression recognition incorporating action units. IEEE Access 9:142071–142083
Ali G, Ali A, Ali F, Draz U, Majeed F, Yasin S, Ali T, Haider N (2020) Artificial neural network based ensemble approach for multicultural facial expressions analysis. IEEE Access 8:134950–134963
Ye Y, Pan Y, Liang Y, Pan J (2023) A cascaded spatiotemporal attention network for dynamic facial expression recognition. Appl Intell 53(5):5402–5415
Acknowledgements
The author would like to acknowledge the Department of Information Technology, Delhi Technological University, New Delhi, India for providing me necessary resources to carry out the research.
Funding
We received no funding.
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Conflicts of interest
On behalf of all authors, the corresponding author states that there is no conflict of interest.
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
About this article
Cite this article
Nidhi, Verma, B. From methods to datasets: a detailed study on facial emotion recognition. Appl Intell 53, 30219–30249 (2023). https://doi.org/10.1007/s10489-023-05052-y
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10489-023-05052-y