Skip to main content
SpringerLink
Log in

Facial Expression Recognition Using Machine Learning and Deep Learning Techniques: A Systematic Review

  • Review Article
  • Published:
SN Computer Science Aims and scope Submit manuscript

Abstract

In the contemporary era, Facial Expression Recognition (FER) plays a pivotal role in numerous fields due to its vast application areas, such as e-learning, healthcare, marketing, and psychology, to name a few examples. Several research studies have been conducted on FER, and many reviews are available. The existing FER review paper focused on presenting a standard pipeline for FER to predict basic expressions. However, previous studies have not given an adequate amount of importance to FER datasets and their influence on affecting FER system performance. In this systematic review, 105 papers retrieved papers from IEEE, ACM, Science Direct, Scopus, Web of Science, and Springer from the years 2002 to 2023, following systematic review guidelines. Review protocol and research questions are also developed for the analysis of study results. The review identified that the accuracy of the FER system in controlled and spontaneous facial expression datasets is being affected, along with other challenges such as illumination, pose, and scale variation. Furthermore, this paper comparatively analyzed the FER model in both machine and deep learning techniques, including face detection, pre-processing, handcrafted feature extraction techniques, and emotion classifiers. In addition, we discussed some unresolved issues in FER and suggested solutions to enhance FER system performance further. In the future, multimodal FER systems need to be developed for real-time scenarios, considering the computational efficiency of model performance when integrating more than one model and dataset to achieve promising accuracy and reduce error rates.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13

Similar content being viewed by others

Data Availability

No data availability.

References

  1. Abdul-Hadi MH, Waleed, J. Human speech and facial emotion recognition technique using svm. In: 2020 International Conference on Computer Science and Software Engineering (CSASE). 2020;191–196. IEEE. https://doi.org/10.1186/s42492-019-0034-5

  2. Abdulsalam WH, Alhamdani RS, Abdullah MN. Facial emotion recognition from videos using deep convolutional neural networks. Int J Mach Learn Comput. 2019;9(1):14–9. https://doi.org/10.18178/ijmlc.2019.9.1.759.

    Article  Google Scholar 

  3. Abiyev RH. Facial feature extraction techniques for face recognition. J Comput Sci. 2014;10(12):2360. https://doi.org/10.3844/jcssp.2014.2360.2365.

    Article  Google Scholar 

  4. Adyapady RR, Annappa BA. comprehensive review of facial expression recognition techniques. Multimedia Syst. 2023;29:73–103. https://doi.org/10.1007/s00530-022-00984-w.

    Article  Google Scholar 

  5. Akhand MAH, Roy S, Siddique N, Kamal MAS, Shimamura T. Facial emotion recognition using transfer learning in the deep CNN. Electronics. 2021;10(9):1036. https://doi.org/10.3390/electronics10091036.

    Article  Google Scholar 

  6. Ali G, Ali A, Ali F, Draz U, Majeed F, Yasin S, Haider N. Artificial neural network-based ensemble approach for multicultural facial expressions analysis. IEEE Access. 2023;8:134950–63.

    Article  Google Scholar 

  7. Anwar S, Milanova M. Real-time face expression recognition of children with autism. In: Proc. IAEMR. 2016.

  8. Aouani H, Ayed YB. Speech emotion recognition with deep learning. Procedia Comput Sci. 2020;176:251–60. https://doi.org/10.1016/j.procs.2020.08.027.

    Article  Google Scholar 

  9. Aro T, Abikoye O, Oladipo I, Awotunde B. Enhanced Gabor features based facial recognition using ant colony optimization algorithm. J Sustain Technol. 2019;10(1):1–28

  10. Ashraf A, Gunawan T S, Rahman F D A, Kartiwi M. A Summarization of Image and Video Databases for Emotion Recognition. In: Recent Trends in Mechatronics Towards Industry 4.0, Springer. 2022; 669–680.

  11. Aung H, Bobkov AV, Tun NL. Face detection in real-time live video using Yolo algorithm based on Vgg16 convolutional neural network. In: 2021 International Conference on Industrial Engineering, Applications and Manufacturing (ICIEAM), IEEE. 2021; 697–702.

  12. Ayvaz U, Gürüler H, Devrim MO. Use of facial emotion recognition in e-learning systems. 2017.

  13. Babajee P, Suddul G, Armoogum S, Foogooa R. Identifying human emotions from facial expressions with deep learning. In: 2020 Zooming Innovation in Consumer Technologies Conference (ZINC). IEEE. 2020; 36–9.

  14. Bach S, Binder A, Montavon G, Klauschen F, Müller KR, Samek W. On pixel-wise explanations for non-linear classifier decisions by layer-wise relevance propagation. PLoS One. 2015;10(7): e0130140.

    Article  Google Scholar 

  15. Bagherian E, Rahmat RWO. Facial feature extraction for face recognition: a review. In: 2008 International Symposium on Information Technology, IEEE. 2008;2;1–9.

  16. Bakshi U, Singhal R. A survey on face detection methods and feature extraction techniques of face recognition. Int J Emerg Trends Technol Comput Sci (IJETTCS). 2014;3(3):233–7.

    Google Scholar 

  17. Basak P, De S, Agarwal M, Malhotra A, Vatsa M, Singh R. Multimodal biometric recognition for toddlers and pre-school children. In: 2017 IEEE International Joint Conference on Biometrics (IJCB), IEEE, 2017; 627–33.

  18. Bishop C M, Nasrabadi NM. Pattern recognition and machine learning. New York: springer. 2006;4(4):738.

  19. Boughida A, Kouahla MN, Lafifi Y. A novel approach for facial expression recognition based on Gabor filters and genetic algorithm. Evol Syst. 2021. https://doi.org/10.1007/s12530-021-09393-2.

    Article  Google Scholar 

  20. Bouhabba EM, Shafie AA, Akmeliawati R. Support vector machine for face emotion detection on a real-time basis. In: 2011 4th International Conference on Mechatronics (ICOM). IEEE. 2011; 1–6. IEEE.

  21. Buhari AM, Ooi CP, Baskaran VM, Phan RC, Wong K, Tan WH. FACS-based graph features for real-time micro-expression recognition. J Imaging. 2020;6(12):130.

    Article  Google Scholar 

  22. Cambria E, Das D, Bandyopadhyay S, Feraco A. Affective computing and sentiment analysis. In: A practical guide to sentiment analysis. Springer, Cham. 2017; 1–10.

  23. Canedo D, Neves AJ. Facial expression recognition using computer vision: a systematic review. Appl Sci. 2019;9(21):4678.

    Article  Google Scholar 

  24. Celisse A, Robin S. Nonparametric density estimation by exact leave-p-out cross-validation. Comput Stat Data Anal. 2008;52(5):2350–68.

    Article  MathSciNet  Google Scholar 

  25. Chang W Y, Hsu S H, Chien J H. FATAUVA-Net: an integrated deep learning framework for facial attribute recognition, action unit detection, and valence-arousal estimation. In: Proceedings of the IEEE conference on computer vision and pattern recognition workshops. 2017; 17–25.

  26. Chatterjee S, Das AK, Nayak J, Pelusi D. Improving facial emotion recognition using residual autoencoder coupled affinity-based overlapping reduction. Mathematics. 2022;10(3):406.

    Article  Google Scholar 

  27. Chaudhari S T, Kale A. Face normalization: enhancing face recognition. In: 2010 3rd International Conference on Emerging Trends in Engineering and Technology. IEEE. 2010; 520–5.

  28. Chen T, Pu T, Wu H, Xie Y, Liu L, Lin L. Cross-domain facial expression recognition: A unified evaluation benchmark and adversarial graph learning. IEEE Trans Pattern Anal Mach Intell. 2021;44(12):9887–903.

    Article  Google Scholar 

  29. Chen W, Huang H, Peng S, Zhou C, Zhang C. YOLO-face: a real-time face detector. Vis Comput. 2021;37(4):805–13.

    Article  Google Scholar 

  30. Choi IK, Ahn HE, Yoo J. Facial expression classification using deep convolutional neural network. J Elect Eng Technol. 2018;13(1):485–92.

    Google Scholar 

  31. Cohn J F, Zlochower A J, Lien J J, Kanade T. Feature-point tracking by optical flow discriminates subtle differences in facial expression. In: Proceedings third IEEE international conference on automatic face and gesture recognition. IEEE. 1998; 396–401.

  32. Cubuk E D, Zoph B, Mane D, Vasudevan V, Le Q V. Autoaugment: Learning augmentation strategies from data. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2019; 113–23.

  33. Cuimei L, Zhiliang Q, Nan J, Jianhua W. Human face detection algorithm via Haar cascade classifier combined with three additional classifiers. In: 2017 13th IEEE International Conference on Electronic Measurement & Instruments (ICEMI). IEEE. 2017; 483–7.

  34. Cunningham S, Ridley H, Weinel J, Picking R. Supervised machine learning for audio emotion recognition. Pers Ubiquit Comput. 2021;25(4):637–50.

    Article  Google Scholar 

  35. Dalrymple KA, Gomez J, Duchaine B. The dartmouth database of children’s faces: acquisition and validation of a new face stimulus set. PLoS One. 2013;8(11): e79131.

    Article  Google Scholar 

  36. Dang V T, Do H Q, Vu V V, Yoon B. Facial expression recognition: a survey and its applications. In: 2021 23rd International Conference on Advanced Communication Technology (ICACT), IEEE. 2021; 359–67.

  37. Deng J, Guo J, Zafeiriou S. Single-stage joint face detection and alignment. In: Proceedings of the IEEE/CVF International Conference on Computer Vision Workshops. 2019.

  38. Dino H I, Abdulrazzaq M B. Facial expression classification based on SVM, KNN and MLP classifiers. In: 2019 International Conference on Advanced Science and Engineering (ICOASE). IEEE. 2019; 70–5.

  39. Du S, Tao Y, Martinez AM. Compound facial expression of emotion. Proc Natl Acad Sci. 2014;111(15):E1454–62.

    Article  Google Scholar 

  40. Edwards G J, Cootes TF, Taylor CJ. Face recognition using active appearance models. In: European conference on computer vision. Springer, Berlin, Heidelberg. 1998; 581–95.

  41. Egger HL, Pine DS, Nelson E, Leibenluft E, Ernst M, Towbin KE, Angold A. The NIMH Child Emotional Faces Picture Set (NIMH-ChEFS): a new set of children’s facial emotion stimuli. Int J Methods Psychiatr Res. 2011;20(3):145–56.

    Article  Google Scholar 

  42. Ekman P, Friesen WV. Constants across cultures in the face and emotion. J Pers Soc Psychol. 1971;17(2):124.

    Article  Google Scholar 

  43. Ekman P. An argument for basic emotions. Cogn Emot. 1992;6(3–4):169–200.

    Article  Google Scholar 

  44. Ekman P. Darwin, deception, and facial expression. Ann NY Acad Sci. 2003;1000(1):205–21.

    Article  Google Scholar 

  45. El Hammoumi O, Benmarrakchi F, Ouherrou N, El Kafi J, El Hore A. Emotion recognition in e-learning systems. In: 2018 6th international conference on multimedia computing and systems (ICMCS). IEEE. 2018; 1–6.

  46. Fabian Benitez-Quiroz C, Srinivasan R, Martinez AM. Emotionet: an accurate, real-time algorithm for the automatic annotation of a million facial expressions in the wild. In: Proceedings of the IEEE conference on computer vision and pattern recognition. 2016; 5562–70.

  47. Fasel B, Luettin J. Automatic facial expression analysis: a survey. Pattern Recogn. 2003;36(1):259–75.

    Article  Google Scholar 

  48. Friesen E, Ekman P. Facial action coding system: a technique for the measurement of facial movement. Palo Alto. 1978;3(2):5.

    Google Scholar 

  49. Gavrilescu M, Vizireanu N. Predicting depression, anxiety, and stress levels from videos using the facial action coding system. Sensors. 2019;19(17):3693.

    Article  Google Scholar 

  50. Gehrig T, Ekenel H K. Why is facial expression analysis in the wild challenging? In: Proceedings of 2013 on Emotion recognition in the wild challenge and workshop. 2013; 9–16.

  51. Giuliani NR, Flournoy JC, Ivie EJ, Von Hippel A, Pfeifer JH. Presentation and validation of the DuckEES child and adolescent dynamic facial expressions stimulus set. Int J Methods Psychiatr Res. 2017;26(1): e1553.

    Article  Google Scholar 

  52. Goodfellow I J, Erhan D, Carrier P L, Courville A, Mirza M, Hamner B, Bengio Y. Challenges in representation learning: a report on three machine learning contests. In: International conference on neural information processing. Springer, Berlin, Heidelberg. 2013;117–24.

  53. Gross R, Matthews I, Cohn J, Kanade T, Baker S. Multi-pie. Image Vis Comput. 2010;28(5):807–13.

    Article  Google Scholar 

  54. Gu L, Kanade T. A generative shape regularization model for robust face alignment. In: European conference on computer vision. Springer, Berlin, Heidelberg. 2008; 413–26 (2008).

  55. Gunawan TS, Ashraf A, Riza BS, Haryanto EV, Rosnelly R, Kartiwi M, Janin Z. Development of video-based emotion recognition using deep learning with Google Colab. Telkomnika. 2020;18(5):2463–71.

    Article  Google Scholar 

  56. Gupta S. Facial emotion recognition in real-time and static images. In: 2018 2nd international conference on inventive systems and control (ICISC). IEEE. 2018; 553–60.

  57. Hjelmås E, Low BK. Face detection: a survey. Comput Vis Image Underst. 2001;83(3):236–74.

    Article  Google Scholar 

  58. Ho D, Liang E, Chen X, Stoica I, Abbeel P. Population-based augmentation: Efficient learning of augmentation policy schedules. In: International conference on machine learning. 2019; 2731–41. PMLR.

  59. Hossain S, Umer S, Rout RK, Tanveer M. Fine-grained image analysis for facial expression recognition using deep convolutional neural networks with bilinear pooling. Appl Soft Comput. 2023;134: 109997.

    Article  Google Scholar 

  60. Hossen AMA, Ogla RAA, Ali MM. Face detection by using OpenCV’s Viola-Jones Algorithm based on coding eyes. Iraqi J Sci. 2017;58(2A):735–45.

    Google Scholar 

  61. Huang Y, Chen F, Lv S, Wang X. Facial expression recognition: a survey. Symmetry. 2019;11(10):1189.

    Article  Google Scholar 

  62. Jack RE, Garrod OG, Yu H, Caldara R, Schyns PG. Facial expressions of emotion are not culturally universal. Proc Natl Acad Sci. 2012;109(19):7241–4.

    Article  Google Scholar 

  63. Jaderberg M, Dalibard V, Osindero S, Czarnecki W M, Donahue J, Razavi A, Kavukcuoglu K. Population based training of neural networks. arXiv preprint arXiv:1711.09846. 2017.

  64. Jadhav R, Bhuke J, Patil N. Facial emotion detection using convolutional neural network. Int Res J Eng Technol. e-ISSN, 2395–0056. 2019.

  65. Jain DK, Shamsolmoali P, Sehdev P. Extended deep neural network for facial emotion recognition. Pattern Recogn Lett. 2019;120:69–74.

    Article  Google Scholar 

  66. Jaiswal A, Raju A K, Deb S. Facial emotion detection using deep learning. In: 2020 International Conference for Emerging Technology (INCET). IEEE. 2020; 1–5.

  67. Jiang F, Zhang J, Yan L, Xia Y, Shan S. A three-category face detector with contextual information on finding tiny faces. In: 2018 25th IEEE international conference on image processing (ICIP). IEEE. 2018; 2680–4

  68. Johnston B, de Chazal P. A review of image-based automatic facial landmark identification techniques. EURASIP J Image Video Process. 2018;1:1–23.

    Google Scholar 

  69. Kalinovskii I, Spitsyn, V. Compact convolutional neural network cascade for face detection. arXiv preprint arXiv:1508.01292. 2015.

  70. Kamarol SKA, Jaward MH, Kälviäinen H, Parkkinen J, Parthiban R. Joint facial expression recognition and intensity estimation based on weighted votes of image sequences. Pattern Recogn Lett. 2017;92:25–32.

    Article  Google Scholar 

  71. Kanade T, Cohn J F, Tian Y. Comprehensive database for facial expression analysis. In: Proceedings Fourth IEEE International Conference on Automatic Face and Gesture Recognition (Cat. No. PR00580). IEEE. 2000; 46–53.

  72. Karnati M, Seal A, Yazidi A, Krejcar O. FLEPNet: feature level ensemble parallel network for facial expression recognition. IEEE Trans Affect Comput. 2022;13(4):2058–70.

    Article  Google Scholar 

  73. Kawakura S, Hirafuji M, Ninomiya S, Shibasaki R. Analyses of diverse agricultural worker data with explainable artificial intelligence: Xai based on shap, lime, and lightgbm. Eur J Agric Food Sci. 2022;4(6):11–9.

    Google Scholar 

  74. Khan RA, Crenn A, Meyer A, Bouakaz S. A novel database of children’s spontaneous facial expressions (LIRIS-CSE). Image Vis Comput. 2019;83:61–9.

    Article  Google Scholar 

  75. Kitchenham B. Procedures for performing systematic reviews, vol. 33. Keele: Keele University; 2004. p. 1–26.

    Google Scholar 

  76. Kumar GR, Kumar RK, Sanyal G. Facial emotion analysis using deep convolution neural network. In: 2017 International Conference on Signal Processing and Communication (ICSPC). IEEE. 2017; 369–74.

  77. Langner O, Dotsch R, Bijlstra G, Wigboldus DH, Hawk ST, Van Knippenberg AD. Presentation and validation of the Radboud Faces Database. Cogn Emot. 2010;24(8):1377–88.

    Article  Google Scholar 

  78. Lasri I, Solh A R, El Belkacemi M. Facial emotion recognition of students using a convolutional neural network. In: 2019 third international conference on intelligent computing in data sciences (ICDS), IEEE. 2019;1–6.

  79. LeCun Y, Bengio Y, Hinton G. Deep learning. Nature. 2015;521(7553):436–44.

    Article  Google Scholar 

  80. Li S, Deng W. Deep emotion transfer network for cross-database facial expression recognition. In: 2018 24th International Conference on Pattern Recognition (ICPR). IEEE. 2018;3092–9.

  81. Li S, Deng W. Deep facial expression recognition: a survey. IEEE Trans Affect Comput. 2020. 13(3);1195–1215.

  82. Li X, Lai S, Qian X. DBCFace: towards pure convolutional neural network face detection. IEEE Trans Circ Syst Video Technol. 2021. https://doi.org/10.1109/TCSVT.2021.3082635.

    Article  Google Scholar 

  83. Lim R, MJT Reinders T. Facial landmark detection using a Gabor filter representation and a genetic search algorithm. In: Proceeding, (SITIA’2000), Graha Institut Teknologi Sepuluh November. 2000.

  84. Liu C, Hirota K, Dai Y. Patch attention convolutional vision transformer for facial expression recognition with occlusion. Inf Sci. 2023;619:781–94.

    Article  Google Scholar 

  85. Liu S, Tian Y, Peng C, Li J. Facial expression recognition approaches based on least squares support vector machines with improved particle swarm optimization algorithms. In: 2010 IEEE International Conference on Robotics and Biomimetics. IEEE. 2010; 399–404 (2010).

  86. Liu Y, Wang W, Feng C, Zhang H, Chen Z, Zhan Y. Expression snippet transformer for robust video-based facial expression recognition. Pattern Recogn. 2023;138: 109368.

    Article  Google Scholar 

  87. LoBue V, Thrasher C. The Child Affective Facial Expression (CAFE) set: validity and reliability from untrained adults. Front Psychol. 2015;5:532.

    Article  Google Scholar 

  88. Lopez-Rincon, A. Emotion recognition using facial expressions in children using the NAO Robot. In: 2019 International Conference on Electronics, Communications and Computers (CONIELECOMP), IEEE. 2019;146–53.

  89. Lu H, Yang F. Active shape model and its application to face alignment. In: Subspace methods for pattern recognition in intelligent environment, Springer. 2014; 1–31.

  90. Lucey P, Cohn J F, Kanade T, Saragih J, Ambadar Z, Matthews I. The extended cohn-kanade dataset (ck+): A complete dataset for action unit and emotion-specified expression. In: 2010 ieee computer society conference on computer vision and pattern recognition-workshops. IEEE. 2010; 94–10.

  91. Lundqvist D, Flykt A, Öhman A. The Karolinska directed emotional faces—KDEF [CD ROM]. Karolinska Institutet, Stockholm. 1998.

  92. Luo D, Wen G, Li D, Hu Y, Huan E. Deep-learning-based face detection using iterative bounding-box regression. Multimedia Tools Appl. 2018;77:24663–80.

    Article  Google Scholar 

  93. Michael L, Miyuki K, Jiro G. The Japanese Female Facial Expression (JAFFE) Dataset .1998. Zenodo.

  94. Martinez B, Valstar MF. Advances, challenges, and opportunities in automatic facial expression recognition. Adv Face Detect Facial Image Anal. 2016; 63–100.

  95. Mascaró-Oliver M, Mas-Sansó R, Amengual-Alcover E, Roig-Maimó MF. UIBVFED-mask: a dataset for comparing facial expressions with and without face masks. Data. 2023;8(1):17. https://doi.org/10.3390/data8010017.

    Article  Google Scholar 

  96. Matsumoto D. More evidence for the universality of a contempt expression. Motiv Emot. 1992;16:363–8.

    Article  Google Scholar 

  97. Mehrabian, A. Nonverbal communication. In: Nebraska symposium on motivation. University of Nebraska Press. 1971.

  98. Ming Y, Qian H, Guangyuan L. CNN-LSTM facial expression recognition method fused with two-layer attention mechanism. Comput Intell Neurosci. 2022.

  99. Miolla A, Cardaioli M, Scarpazza C. Padova Emotional Dataset of Facial Expressions (PEDFE): a unique dataset of genuine and posed emotional facial expressions. Behav Res. 2023;55:2559–74. https://doi.org/10.3758/s13428-022-01914-4.

    Article  Google Scholar 

  100. Mohammed OA, Al-Tuwaijari JM. Analysis of challenges and methods for face detection systems: a survey. Int J Nonlinear Anal Appl. 2022;13(1):3997–4015.

    Google Scholar 

  101. Mohana M, Subashini P, Krishnaveni M. Emotion recognition from facial expression using hybrid CNN–LSTM network. Int J Pattern Recognit Artif Intell. 2023;37(08):2356008.

    Article  Google Scholar 

  102. Mollahosseini A, Chan D, Mahoor M H. Going deeper in facial expression recognition using deep neural networks. In: 2016 IEEE Winter conference on applications of computer vision (WACV), IEEE. 2016;1–10.

  103. Mollahosseini A, Hasani B, Mahoor MH. Affectnet: a database for facial expression, valence, and arousal computing in the wild. IEEE Trans Affect Comput. 2017;10(1):18–31.

    Article  Google Scholar 

  104. Nan Y, Ju J, Hua Q, Zhang H, Wang B. A-MobileNet: an approach of facial expression recognition. Alex Eng J. 2022;61(6):4435–44.

    Article  Google Scholar 

  105. Negrão JG, Osorio AAC, Siciliano RF, Lederman VRG, Kozasa EH, D’Antino MEF, Schwartzman JS. The child emotion facial expression set: a database for emotion recognition in children. Front Psychol. 2021;12:1352.

    Article  Google Scholar 

  106. Nojavanasghari B, Baltrušaitis T, Hughes CE, Morency LP. Emoreact: a multimodal approach and dataset for recognizing emotional responses in children. In: Proceedings of the 18th acm international conference on multimodal interaction. 2016; 137–44.

  107. Oliver MM, Amengual AE. UIBVFED: virtual facial expression dataset. PLoS One. 2020;15(4): e0231266.

    Article  Google Scholar 

  108. Page MJ, McKenzie JE, Bossuyt PM, Boutron I, Hoffmann TC, Mulrow CD, Moher D. Updating guidance for reporting systematic reviews: development of the PRISMA 2020 statement. J Clin Epidemiol. 2021;134:103–12.

    Article  Google Scholar 

  109. Palacio S, Lucieri A, Munir M, Ahmed S, Hees J, Dengel A. Xai handbook: towards a unified framework for explainable AI. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. 2021; 3766–75.

  110. Pali V, Goswami S, Bhaiya L P. An extensive survey on feature extraction techniques for facial image processing. In: 2014 International Conference on Computational Intelligence and Communication Networks, IEEE. 2014; 142–148.

  111. Pantic M, Valstar M, Rademaker R, Maat L. Web-based database for facial expression analysis. In: 2005 IEEE international conference on multimedia and Expo, IEEE. 2005; 5.

  112. Park S, Lee K, Lim JA, Ko H, Kim T, Lee JI, Lee EC. Differences in facial expressions between spontaneous and posed smiles: automated method by action units and three-dimensional facial landmarks. Sensors. 2020;20(4):1199.

    Article  Google Scholar 

  113. Patil HY, Bharambe SV, Kothari AG, Bhurchandi KM. Face localization and its implementation on embedded platform. In: 2013 3rd IEEE International Advance Computing Conference (IACC). IEEE. 2013; 741–5

  114. Peng P, Xiang T, Wang Y, Pontil M, Gong S, Huang T, Tian Y. Unsupervised cross-dataset transfer learning for person re-identification. In: Proceedings of the IEEE conference on computer vision and pattern recognition. 2016;1306–15.

  115. Perveen N, Ahmad N, Khan M A Q B, Khalid R, Qadri S. Facial expression recognition through machine learning. Int J Sci Technol Res. 2016;5(03)91–97

  116. Picard RW. Affective computing. MIT press; 2000.

    Book  Google Scholar 

  117. Polikovsky S, Kameda Y, Ohta Y. Facial micro-expression detection in hi-speed video based on facial action coding system (FACS). IEICE Trans Inf Syst. 2013;96(1):81–92.

    Article  Google Scholar 

  118. Pranav E, Kamal S, Chandran C S, Supriya M H. Facial emotion recognition using deep convolutional neural network. In: 2020 6th International conference on advanced computing and communication Systems (ICACCS), IEEE. 2020; 317–20.

  119. Priadana A, Habibi M. Face detection using haar cascades to filter selfie face images on instagram. In: 2019 International Conference of Artificial Intelligence and Information Technology (ICAIIT), IEEE. 2019; 6–9.

  120. Qayyum R, Akre V, Hafeez T, Khattak H A, Nawaz A, Ahmed S, ur Rahman, K. Android based Emotion Detection Using Convolutions Neural Networks. In: 2021 International Conference on Computational Intelligence and Knowledge Economy (ICCIKE), IEEE. 2021; 360–5.

  121. Rathee N, Vaish A, Gupta S. Adaptive system to learn and recognize the emotional state of mind. In: 2016 International Conference on Computing, Communication and Automation (ICCCA). IEEE. 2016; 32–6.

  122. Riyantoko PA, Hindrayani KM. Facial emotion detection using haar-cascade classifier and convolutional neural networks. J Phys Conf Ser. 2021;1844(1): 012004 (IOP Publishing).

    Article  Google Scholar 

  123. Robin M H, Rahman M M U, Taief A M, Eity Q N. Improvement of face and eye detection performance by using multi-task cascaded convolutional networks. In: 2020 IEEE Region 10 Symposium (TENSYMP), IEEE. 2020; 977–980

  124. Rodriguez JD, Perez A, Lozano JA. Sensitivity analysis of k-fold cross validation in prediction error estimation. IEEE Trans Pattern Anal Mach Intell. 2009;32(3):569–75.

    Article  Google Scholar 

  125. Rodriguez Y, Cardinaux F, Bengio S, Mariéthoz J. Measuring the performance of face localization systems. Image Vis Comput. 2006;24(8):882–93.

    Article  Google Scholar 

  126. Romani-Sponchiado A, Sanvicente-Vieira B, Mottin C, Hertzog-Fonini D, Arteche A. Child Emotions Picture Set (CEPS): development of a database of children’s emotional expressions. Psychol Neurosci. 2015;8(4):467.

    Article  Google Scholar 

  127. Roshan K, Zafar A, Haque SBU. Untargeted white-box adversarial attack with heuristic defence methods in real-time deep learning based network intrusion detection system. Comput Commun. 2023. https://doi.org/10.1016/j.comcom.2023.09.030.

    Article  Google Scholar 

  128. Russell JA. A circumplex model of affect. J Pers Soc Psychol. 1980;39(6):1161.

    Article  Google Scholar 

  129. Sajjad M, Ullah FUM, Ullah M, Christodoulou G, Cheikh FA, Hijji M, Rodrigues JJ. A comprehensive survey on deep facial expression recognition: challenges, applications, and future guidelines. Alex Eng J. 2023;68:817–40.

    Article  Google Scholar 

  130. Sebe N, Cohen I, Huang TS. Multimodal emotion recognition. In: Handbook of pattern recognition and computer vision. 2005;387–409.

  131. Selvaraju R R, Cogswell M, Das A, Vedantam R, Parikh D, Batra D. Grad-cam: Visual explanations from deep networks via gradient-based localization. In: Proceedings of the IEEE international conference on computer vision. 2017; 618–626.

  132. Sharafi M, Yazdchi M, Rasti R, Nasimi F. A novel Spatio-temporal convolutional neural framework for multimodal emotion recognition. Biomed Signal Process Control. 2022;78: 103970.

    Article  Google Scholar 

  133. Sharma R, Patterh MS. Face recognition using face alignment and PCA techniques: a literature survey. IOSR J Comput Eng (IOSR-JCE). 2015;17(4):17–30.

    Google Scholar 

  134. Sheikh BUH, Zafar A. White-box inference attack: compromising the security of deep learning-based COVID-19 diagnosis systems. Int J Inf Tecnol. 2023. https://doi.org/10.1007/s41870-023-01538-7.

    Article  Google Scholar 

  135. Sheikh BUH, Zafar A. Unlocking adversarial transferability: a security threat towards deep learning-based surveillance systems via black box inference attack—a case study on face mask surveillance. Multimed Tools Appl. 2023. https://doi.org/10.1007/s11042-023-16439-x.

    Article  Google Scholar 

  136. Sheikh BUH, Zafar A. Untargeted white-box adversarial attack to break into deep learning based COVID-19 monitoring face mask detection system. Multimed Tools Appl. 2023. https://doi.org/10.1007/s11042-023-15405-x.

    Article  Google Scholar 

  137. Sheikh B, Zafar A. Beyond accuracy and precision: a robust deep learning framework to enhance the resilience of face mask detection models against adversarial attacks. Evol Syst. 2023. https://doi.org/10.1007/s12530-023-09522-z.

    Article  Google Scholar 

  138. Sheikh B, Zafar A. RRFMDS: rapid real-time face mask detection system for effective COVID-19 monitoring. SN Comput Sci. 2023;4:288. https://doi.org/10.1007/s42979-023-01738-9.

    Article  Google Scholar 

  139. Shorten C, Khoshgoftaar TM. A survey on image data augmentation for deep learning. J Big Data. 2019;6(1):1–48.

    Article  Google Scholar 

  140. Sikkandar H, Thiyagarajan R. Deep learning based facial expression recognition using improved Cat Swarm Optimization. J Ambient Intell Humaniz Comput. 2021;12(2):3037–53.

    Article  Google Scholar 

  141. Stockman G, Shapiro L G. Computer vision. Prentice Hall PTR.

  142. Su Y, Liu Z, Ban X. Symmetric face normalization. Symmetry. 2019;11(1):96.

    Article  Google Scholar 

  143. Suhaimi NS, Mountstephens J, Teo J. EEG-based emotion recognition: a state-of-the-art review of current trends and opportunities. Comput Intell Neurosci. 2020. https://doi.org/10.1155/2020/8875426.

    Article  Google Scholar 

  144. Talele KT, Kadam S. Face detection and geometric face normalization. In: TENCON 2009–2009 IEEE Region 10 Conference. IEEE. 2009; 1–6.

  145. Tang Y, Zhang X, Hu X, Wang S, Wang H. Facial expression recognition using frequency neural network. IEEE Trans Image Process. 2020;30:444–57.

    Article  Google Scholar 

  146. Tao S, Li Y, Huang Y, Lan X. Face detection algorithm based on deep residual network. J Phys: Conf Ser. 2021;1802(3): 032142 (IOP Publishing).

    Google Scholar 

  147. Tian YI, Kanade T, Cohn JF. Recognizing action units for facial expression analysis. IEEE Trans Pattern Anal Mach Intell. 2001;23(2):97–115.

    Article  Google Scholar 

  148. Tümen V, Söylemez ÖF, Ergen B. Facial emotion recognition on a dataset using convolutional neural network. In: 2017 International Artificial Intelligence and Data Processing Symposium (IDAP). IEEE. 2017; 1–5

  149. Verma A, Singh P, Alex J S R. Modified convolutional neural network architecture analysis for facial emotion recognition. In: 2019 International Conference on Systems, Signals and Image Processing (IWSSIP), IEEE. 2019; 169–173.

  150. Viola P, Jones M. Rapid object detection using a boosted cascade of simple features. In: Proceedings of the 2001 IEEE computer society conference on computer vision and pattern recognition. CVPR 2001. IEEE. 2001; 1; I-I.

  151. Voulodimos A, Doulamis N, Doulamis A, Protopapadakis E. Deep learning for computer vision: a brief review. Comput Intell Neurosci. 2018. https://doi.org/10.1155/2018/7068349.

    Article  Google Scholar 

  152. Wang Y, Ji X, Zhou Z, Wang H, Li Z. Detecting faces using region-based fully convolutional networks. arXiv preprint arXiv:1709.05256. 2017.

  153. Wang Y, Sun Y, Huang Y, Liu Z, Gao S, Zhang W, Zhang W. Ferv39k: a large-scale multi-scene dataset for facial expression recognition in videos. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2022;20922–31.

  154. Wardhani N W S, Rochayani M Y, Iriany A, Sulistyono A D, Lestantyo P. Cross-validation metrics for evaluating classification performance on imbalanced data. In: 2019 international conference on computer, control, informatics and its applications (IC3INA), IEEE. 2019; 14–18.

  155. Wen Z, Lin W, Wang T, Xu G. Distract your attention: multi-head cross attention network for facial expression recognition. Biomimetics. 2023;8(2):199.

    Article  Google Scholar 

  156. Wu Y, Ji Q. Facial landmark detection: a literature survey. Int J Comput Vis. 2019;127(2):115–42.

    Article  Google Scholar 

  157. Wu Y, Zhang L, Gu Z, Lu H, Wan S. Edge-AI-driven framework with efficient mobile network design for facial expression recognition. ACM Trans Embed Comput Syst. 2023;22(3):1–17.

    Article  Google Scholar 

  158. Xie Y, Chen T, Pu T, Wu H, Lin L. Adversarial graph representation adaptation for cross-domain facial expression recognition. In: Proceedings of the 28th ACM International Conference on Multimedia. 2020;1255–64.

  159. Yadav KS, Singha J. Facial expression recognition using modified Viola-John’s algorithm and KNN classifier. Multimed Tools Appl. 2020;79(19):13089–107.

    Article  Google Scholar 

  160. Yan H, Liu Y, Wang X, Li M, Li H. A face detection method based on skin color features and AdaBoost algorithm. J Phys: Conf Ser. 2021;1748(4): 042015 (IOP Publishing).

    Google Scholar 

  161. Yan Z, Yuan C. Ant colony optimization for feature selection in face recognition. In International conference on biometric authentication. Springer, Berlin, Heidelberg. 2004; 221–6.

  162. Yang L, Tian Y, Song Y, Yang N, Ma K, Xie L. A novel feature separation model exchange-GAN for facial expression recognition. Knowl-Based Syst. 2020;204: 106217.

    Article  Google Scholar 

  163. Yang W, Jiachun Z. Real-time face detection based on YOLO. In: 2018 1st IEEE international conference on knowledge innovation and invention (ICKII), IEEE. 2018; 221–4.

  164. Zhang L, Ai H, Xin S, Huang C, Tsukiji S, Lao S. Robust face alignment based on local texture classifiers. In: IEEE International Conference on Image Processing, IEEE. 2005; 2, II-354.

  165. Zhang N, Luo J, Gao W. Research on face detection technology based on MTCNN. In: 2020 International Conference on Computer Network, Electronic and Automation (ICCNEA), IEEE. 2020; 154–8.

  166. Zhang X, Zhang F, Xu C. Joint expression synthesis and representation learning for facial expression recognition. IEEE Trans Circuits Syst Video Technol. 2021;32(3):1681–95.

    Article  Google Scholar 

  167. Zhang Z, Luo P, Loy C C, Tang X. Facial landmark detection by deep multi-task learning. In: European conference on computer vision, Springer, Cham. 2014; 94–108. https://doi.org/10.1007/978-3-319-10599-4_7.

  168. Zhao G, Huang X, Taini M, Li SZ, PietikäInen M. Facial expression recognition from near-infrared videos. Image Vis Comput. 2011;29(9):607–19.

    Article  Google Scholar 

  169. Zhu X, Liu Y, Li J, Wan T, Qin Z. Emotion classification with data augmentation using generative adversarial networks. In: Advances in knowledge discovery and data mining: 22nd Pacific-Asia Conference, PAKDD 2018, Melbourne, VIC, Proceedings, Part III 22. Springer International Publishing. 2018;49–360.https://doi.org/10.1007/978-3-319-93040-4_28

Download references

Acknowledgements

The authors sincerely thank the ISO Certified (ISO/IEC 20000-1:2018) Centre for Machine Learning and Intelligence (CMLI), funded by the Department of Science and Technology (DST-CURIE), India, for providing the facility to carry out this research study.

Funding

This research study received no external funding.

Author information

Authors and Affiliations

  1. Centre for Machine Learning and Intelligence, Department of Computer Science, Avinashilingam Institute, Coimbatore, India

    M. Mohana & P. Subashini

Authors
  1. M. Mohana
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. P. Subashini
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to M. Mohana.

Ethics declarations

Conflict of Interest

The author declared no potential conflict of interest concerning the publishing of this article.

Declaration of AI and AI-assisted Technologies in the Writing Process

During the preparation of this work, the authors utilized Grammarly assistant tools included in Microsoft Word for grammar checking. After using these tools, the authors reviewed and edited the content as necessary and took full responsibility for the publication's content.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Mohana, M., Subashini, P. Facial Expression Recognition Using Machine Learning and Deep Learning Techniques: A Systematic Review. SN COMPUT. SCI. 5, 432 (2024). https://doi.org/10.1007/s42979-024-02792-7

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s42979-024-02792-7

Keywords

Search

Navigation