Skip to main content
SpringerLink
Log in

A comprehensive review of facial expression recognition techniques

  • Regular Paper
  • Published:
Multimedia Systems Aims and scope Submit manuscript

Abstract

Emotion recognition has opened up many challenges, which lead to various advances in computer vision and artificial intelligence. The rapid development in this field has encouraged the development of an automatic system that could accurately analyze and measure the emotions of human beings via facial expressions. This study mainly focuses on facial expression recognition from visual cues, as visual information is the most prominent channel for social communication. The paper provides a comprehensive review of recent advancements in algorithm development, presents the overall findings performed over the past decades, discusses their advantages and constraints. It explores the transition from the laboratory-controlled environment to challenging real-world (in-the-wild) conditions, focusing on essential issues that require further exploration. Finally, relevant opportunities in this field, challenges, and future directions mentioned in this paper assist the researchers and academicians in designing efficient and robust facial expression recognition systems.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4

Similar content being viewed by others

Notes

  1. https://imotions.com/blog/facial-expression-analysis/.

References

  1. Adolphs, R.: Recognizing emotion from facial expressions: psychological and neurological mechanisms. Behav. Cogn. Neurosci. Rev. 1(1), 21–62 (2002)

    Google Scholar 

  2. Agrawal, A., Mittal, N.: Using cnn for facial expression recognition: a study of the effects of kernel size and number of filters on accuracy. Vis. Comput. 36(2), 405–412 (2020)

    Google Scholar 

  3. Azmi, R., Yegane, S.: Facial expression recognition in the presence of occlusion using local gabor binary patterns. In: 20th Iranian Conference on Electrical Engineering (ICEE2012), IEEE, pp 742–747 (2012)

  4. Barros, P., Parisi, G.I., Weber, C., Wermter, S.: Emotion-modulated attention improves expression recognition: a deep learning model. Neurocomputing 253, 104–114 (2017)

    Google Scholar 

  5. Bhushan, B.: Study of facial micro-expressions in psychology. In: Understanding facial expressions in communication, Springer, pp 265–286 (2015)

  6. Boughida, A., Kouahla, M.N., Lafifi, Y.: A novel approach for facial expression recognition based on gabor filters and genetic algorithm. Evol. Syst. 13(2), 331–345 (2022)

    Google Scholar 

  7. Breuer, R., Kimmel, R.: A deep learning perspective on the origin of facial expressions. arXiv preprint arXiv:1705.01842 (2017)

  8. Cai, J., Meng, Z., Khan, A.S., O’Reilly, J., Li, Z., Han, S., Tong, Y.: Identity-free facial expression recognition using conditional generative adversarial network. In: 2021 IEEE International Conference on Image Processing (ICIP), IEEE, pp 1344–1348 (2021)

  9. Chen, B., Guan, W., Li, P., Ikeda, N., Hirasawa, K., Lu, H.: Residual multi-task learning for facial landmark localization and expression recognition. Pattern Recogn. 115, 107893 (2021)

    Google Scholar 

  10. Chen, J., Chen, Z., Chi, Z., Fu, H.: Facial expression recognition in video with multiple feature fusion. IEEE Trans. Affect. Comput. 9(1), 38–50 (2016)

    Google Scholar 

  11. Chen, J., Konrad, J., Ishwar, P.: Vgan-based image representation learning for privacy-preserving facial expression recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pp 1570–1579 (2018)

  12. Cohn, J.F., Ambadar, Z., Ekman, P.: Observer-based measurement of facial expression with the facial action coding system. The handbook of emotion elicitation and assessment pp 203–221 (2007)

  13. Cruz, A.C., Bhanu, B., Thakoor, N.S.: Vision and attention theory based sampling for continuous facial emotion recognition. IEEE Trans. Affect. Comput. 5(4), 418–431 (2014)

    Google Scholar 

  14. Dailey, M.N., Joyce, C., Lyons, M.J., Kamachi, M., Ishi, H., Gyoba, J., Cottrell, G.W.: Evidence and a computational explanation of cultural differences in facial expression recognition. Emotion 10(6), 874–893 (2010)

    Google Scholar 

  15. Darwin, C., Prodger, P.: The expression of the emotions in man and animals. Oxford University Press, USA (1998)

    Google Scholar 

  16. Davison, A.K., Lansley, C., Costen, N., Tan, K., Yap, M.H.: Samm: a spontaneous micro-facial movement dataset. IEEE Trans. Affect. Comput. 9(1), 116–129 (2016)

    Google Scholar 

  17. DeVries, T., Taylor, G.W.: Improved regularization of convolutional neural networks with cutout. arXiv preprint arXiv:1708.04552 (2017)

  18. Dhall, A., Goecke, R., Lucey, S., Gedeon, T.P: Acted facial expressions in the wild database. Australian National University, Canberra, Australia, Technical Report TR-CS-11 2:1 (2011)

  19. Dhall, A., Goecke, R., Joshi, J., Sikka, K., Gedeon, T.: Emotion recognition in the wild challenge 2014: Baseline, data and protocol. In: Proceedings of the 16th international conference on multimodal interaction, pp 461–466 (2014)

  20. Drira, H., Amor, B.B., Srivastava, A., Daoudi, M., Slama, R.: 3d face recognition under expressions, occlusions, and pose variations. IEEE Trans. Pattern Anal. Mach. Intell. 35(9), 2270–2283 (2013)

    Google Scholar 

  21. Du, S., Tao, Y., Martinez, A.M.: Compound facial expressions of emotion. Proc. Natl. Acad. Sci. 111(15), E1454–E1462 (2014)

    Google Scholar 

  22. Edla, D.R., Ansari, M.F., Chaudhary, N., Dodia, S.: Classification of facial expressions from eeg signals using wpt and svm for wheelchair control operations. Proc. Comput. Sci. 132, 1467–1476 (2018)

    Google Scholar 

  23. Ekenel, H.K., Stiefelhagen, R.: Why is facial occlusion a challenging problem? In: International Conference on Biometrics, Springer, pp 299–308 (2009)

  24. Ekman, P.: Lie catching and microexpressions. The philosophy of deception pp 118–133 (2009)

  25. Ekman, P., Friesen, W.V.: Nonverbal leakage and clues to deception. Psychiatry 32(1), 88–106 (1969)

    Google Scholar 

  26. Ekman, P., Friesen, W.V.: Facial action coding system: investigator’s guide. Consulting Psychologists Press, Washington, DC (1978)

    Google Scholar 

  27. Farzaneh, A.H., Qi, X.: Facial expression recognition in the wild via deep attentive center loss. In: Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp 2402–2411 (2021)

  28. Ge, H., Zhu, Z., Dai, Y., Wang, B., Wu, X.: Facial expression recognition based on deep learning. Computer Methods and Programs in Biomedicine p 106621 (2022)

  29. Georgescu, M.I., Ionescu, R.T., Popescu, M.: Local learning with deep and handcrafted features for facial expression recognition. IEEE Access 7, 64827–64836 (2019)

    Google Scholar 

  30. Ghimire, D., Lee, J.: Geometric feature-based facial expression recognition in image sequences using multi-class adaboost and support vector machines. Sensors 13(6), 7714–7734 (2013)

    Google Scholar 

  31. Ghimire, D., Jeong, S., Lee, J., Park, S.H.: Facial expression recognition based on local region specific features and support vector machines. Multimed. Tools Appl. 76(6), 7803–7821 (2017)

    Google Scholar 

  32. Guo, C., Liang, J., Zhan, G., Liu, Z., Pietikäinen, M., Liu, L.: Extended local binary patterns for efficient and robust spontaneous facial micro-expression recognition. IEEE Access 7, 174517–174530 (2019)

    Google Scholar 

  33. Happy, S., Patnaik, P., Routray, A., Guha, R.: The indian spontaneous expression database for emotion recognition. IEEE Trans. Affect. Comput. 8(1), 131–142 (2017)

    Google Scholar 

  34. Hess, U., Thibault, P.: Darwin and emotion expression. Am. Psychol. 64(2), 120–128 (2009)

    Google Scholar 

  35. Hung, J.C., Lin, K.C., Lai, N.X.: Recognizing learning emotion based on convolutional neural networks and transfer learning. Appl. Soft Comput. 84, 105724 (2019)

    Google Scholar 

  36. Jiang, R., Ho, A.T., Cheheb, I., Al-Maadeed, N., Al-Maadeed, S., Bouridane, A.: Emotion recognition from scrambled facial images via many graph embedding. Pattern Recogn. 67, 245–251 (2017)

    Google Scholar 

  37. Jung, H., Lee, S., Yim, J., Park, S., Kim, J.: Joint fine-tuning in deep neural networks for facial expression recognition. In: Proceedings of the IEEE International Conference on Computer Vision, pp 2983–2991 (2015)

  38. Kaya, H., Gürpınar, F., Salah, A.A.: Video-based emotion recognition in the wild using deep transfer learning and score fusion. Image Vis. Comput. 65, 66–75 (2017)

    Google Scholar 

  39. Khorrami, P., Paine, T., Huang, T.: Do deep neural networks learn facial action units when doing expression recognition? In: Proceedings of the IEEE international conference on computer vision workshops, pp 19–27 (2015)

  40. Kim, D.H., Baddar, W.J., Jang, J., Ro, Y.M.: Multi-objective based spatio-temporal feature representation learning robust to expression intensity variations for facial expression recognition. IEEE Trans. Affect. Comput. 10(2), 223–236 (2017)

    Google Scholar 

  41. Ko, B.: A brief review of facial emotion recognition based on visual information. sensors 18(2), 401 (2018)

    Google Scholar 

  42. Koujan, M.R., Alharbawee, L., Giannakakis, G., Pugeault, N., Roussos, A.: Real-time facial expression recognition “in the wild” by disentangling 3d expression from identity. In: 2020 15th IEEE International Conference on Automatic Face and Gesture Recognition (FG 2020), IEEE, pp 24–31 (2020)

  43. Lee, K., Yoon, H., Song, J., Park, K.: Convolutional neural network-based classification of driver’s emotion during aggressive and smooth driving using multi-modal camera sensors. Sensors 18(4), 957 (2018)

    Google Scholar 

  44. Lee, S.H., Ro, Y.M.: Partial matching of facial expression sequence using over-complete transition dictionary for emotion recognition. IEEE Trans. Affect. Comput. 7(4), 389–408 (2016)

    Google Scholar 

  45. Li, S., Deng, W.: Deep facial expression recognition: A survey. arXiv preprint arXiv:1804.08348 (2018a)

  46. Li, S., Deng, W.: Reliable crowdsourcing and deep locality-preserving learning for unconstrained facial expression recognition. IEEE Trans. Image Process. 28(1), 356–370 (2018)

    MathSciNet  MATH  Google Scholar 

  47. Li, S., Deng, W., Du, J.: Reliable crowdsourcing and deep locality-preserving learning for expression recognition in the wild. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 2852–2861 (2017)

  48. Li, Y., Zeng, J., Shan, S., Chen, X.: Occlusion aware facial expression recognition using cnn with attention mechanism. IEEE Trans. Image Process. 28(5), 2439–2450 (2018)

    MathSciNet  Google Scholar 

  49. Li, Y., Lu, Y., Li, J., Lu, G.: Separate loss for basic and compound facial expression recognition in the wild. In: Asian Conference on Machine Learning, pp 897–911 (2019)

  50. Liang, D., Liang, H., Yu, Z., Zhang, Y.: Deep convolutional bilstm fusion network for facial expression recognition. Vis. Comput. 36(3), 499–508 (2020)

    Google Scholar 

  51. Liang, L., Lang, C., Li, Y., Feng, S., Zhao, J.: Fine-grained facial expression recognition in the wild. IEEE Trans. Inf. Forensics Secur. 16, 482–494 (2020)

    Google Scholar 

  52. Liong, S.T., See, J., Wong, K., Phan, R.C.W.: Less is more: micro-expression recognition from video using apex frame. Signal Process. 62, 82–92 (2018)

    Google Scholar 

  53. Liu, P., Lin, Y., Meng, Z., Deng, W., Zhou, J.T., Yang, Y.: Point adversarial self mining: A simple method for facial expression recognition in the wild. arXiv preprint arXiv:2008.11401 (2020)

  54. Liu, Y., Yuan, X., Gong, X., Xie, Z., Fang, F., Luo, Z.: Conditional convolution neural network enhanced random forest for facial expression recognition. Pattern Recogn. 84, 251–261 (2018)

    Google Scholar 

  55. Liu, Y., Dai, W., Fang, F., Chen, Y., Huang, R., Wang, R., Wan, B.: Dynamic multi-channel metric network for joint pose-aware and identity-invariant facial expression recognition. Inf. Sci. 578, 195–213 (2021)

    MathSciNet  Google Scholar 

  56. Liu, Y., Feng, C., Yuan, X., Zhou, L., Wang, W., Qin, J., Luo, Z.: Clip-aware expressive feature learning for video-based facial expression recognition. Inf. Sci. 598, 182–195 (2022)

    Google Scholar 

  57. Lopes, A.T., de Aguiar, E., De Souza, A.F., Oliveira-Santos, T.: Facial expression recognition with convolutional neural networks: coping with few data and the training sample order. Pattern Recogn. 61, 610–628 (2017)

    Google Scholar 

  58. Lucey, P., Cohn, J.F., Kanade, T., Saragih, J., Ambadar, Z., Matthews, I.: The extended cohn-kanade dataset (ck+): A complete dataset for action unit and emotion-specified expression. In: 2010 ieee computer society conference on computer vision and pattern recognition-workshops, IEEE, pp 94–101 (2010)

  59. Ly, T.S., Do, N.T., Kim, S.H., Yang, H.J., Lee, G.S.: A novel 2d and 3d multimodal approach for in-the-wild facial expression recognition. Image and Vision Computing (2019)

  60. Ma, Y., Hao, Y., Chen, M., Chen, J., Lu, P., Košir, A.: Audio-visual emotion fusion (avef): a deep efficient weighted approach. Inform. Fusion 46, 184–192 (2019)

    Google Scholar 

  61. Majumder, A., Behera, L., Subramanian, V.K.: Automatic facial expression recognition system using deep network-based data fusion. IEEE Transact. Cybern. 48(1), 103–114 (2016)

    Google Scholar 

  62. Martinez, B., Valstar, M.F., Jiang, B., Pantic, M.: Automatic analysis of facial actions: A survey. IEEE transactions on affective computing (2017)

  63. Matias, R., Cohn, J.F., Ross, S.: A comparison of two systems that code infant affective expression. Dev. Psychol. 25(4), 483 (1989)

    Google Scholar 

  64. Mavadati, M., Sanger, P., Mahoor, M.H.: Extended disfa dataset: Investigating posed and spontaneous facial expressions. In: proceedings of the IEEE conference on computer vision and pattern recognition workshops, pp 1–8 (2016)

  65. Mavadati, S.M., Mahoor, M.H., Bartlett, K., Trinh, P., Cohn, J.F.: Disfa: a spontaneous facial action intensity database. IEEE Trans. Affect. Comput. 4(2), 151–160 (2013)

    Google Scholar 

  66. Min, R., Hadid, A., Dugelay, J.L.: Improving the recognition of faces occluded by facial accessories. In: Proc. IEEE Int. Conf. Face and Gesture, pp 442–447 (2011)

  67. Min, R., Hadid, A., Dugelay, J.L.: Efficient detection of occlusion prior to robust face recognition. The Scientific World Journal 2014 (2014)

  68. Minaee, S., Abdolrashidi, A., Su, H., Bennamoun, M., Zhang, D.: Biometrics recognition using deep learning: A survey. arXiv preprint arXiv:1912.00271 (2019)

  69. Mollahosseini, A., Chan, D., Mahoor, M.H.: Going deeper in facial expression recognition using deep neural networks. In: 2016 IEEE Winter conference on applications of computer vision (WACV), IEEE, pp 1–10 (2016)

  70. Namba, S., Makihara, S., Kabir, R.S., Miyatani, M., Nakao, T.: Spontaneous facial expressions are different from posed facial expressions: morphological properties and dynamic sequences. Curr. Psychol. 36(3), 593–605 (2017)

    Google Scholar 

  71. Nan, F., Jing, W., Tian, F., Zhang, J., Chao, K.M., Hong, Z., Zheng, Q.: Feature super-resolution based facial expression recognition for multi-scale low-resolution images. Knowl.-Based Syst. 236, 107678 (2022)

    Google Scholar 

  72. Navarathna, R., Carr, P., Lucey, P., Matthews, I.: Estimating audience engagement to predict movie ratings. IEEE Trans. Affect. Comput. 10(1), 48–59 (2017)

    Google Scholar 

  73. Nguyen, D.H., Kim, S., Lee, G.S., Yang, H.J., Na, I.S., Kim, S.H.: Facial expression recognition using a temporal ensemble of multi-level convolutional neural networks. IEEE Transactions on Affective Computing (2019)

  74. Nigam, S., Singh, R., Misra, A.: A review of computational approaches for human behavior detection. Arch. Comput. Method. Eng. 26(4), 831–863 (2019)

    Google Scholar 

  75. Niu, B., Gao, Z., Guo, B.: Facial expression recognition with lbp and orb features. Computational Intelligence and Neuroscience 2021 (2021)

  76. Nonis, F., Dagnes, N., Marcolin, F., Vezzetti, E.: 3d approaches and challenges in facial expression recognition algorithms-a literature review. Appl. Sci. 9(18), 3904 (2019)

    Google Scholar 

  77. Pan, X., Ying, G., Chen, G., Li, H., Li, W.: A deep spatial and temporal aggregation framework for video-based facial expression recognition. IEEE Access 7, 48807–48815 (2019)

    Google Scholar 

  78. Pfister, T., Li, X., Zhao, G., Pietikäinen, M.: Recognising spontaneous facial micro-expressions. In: Proc. IEEE Int. Conf. Com-put. Vision, pp 1449–1456 (2011)

  79. Pham, T.T.D., Kim, S., Lu, Y., Jung, S.W., Won, C.S.: Facial action units-based image retrieval for facial expression recognition. IEEE Access 7, 5200–5207 (2019)

    Google Scholar 

  80. Picard, R.W., Vyzas, E., Healey, J.: Toward machine emotional intelligence: analysis of affective physiological state. IEEE Transact. Pattern Anal. Mach. Intell. 10, 1175–1191 (2001)

    Google Scholar 

  81. Polikovsky, S., Kameda, Y., Ohta, Y.: Facial micro-expressions recognition using high speed camera and 3d-gradient descriptor. In: Proc. IEEE Third Int. Conf. Crime Detection and Prevention (ICDP), pp 1–6 (2009)

  82. Qu, F., Wang, S.J., Yan, W.J., Li, H., Wu, S., Fu, X.: CAS (ME)\(^{2}\): a database for spontaneous macro-expression and micro-expression spotting and recognition. IEEE Trans. Affect. Comput. 9(4), 424–436 (2017)

    Google Scholar 

  83. Rahulamathavan, Y., Phan, R.C.W., Chambers, J.A., Parish, D.J.: Facial expression recognition in the encrypted domain based on local fisher discriminant analysis. IEEE Trans. Affect. Comput. 4(1), 83–92 (2012)

    Google Scholar 

  84. Rashmi, R.A., Annappa, B.: Micro expression recognition using delaunay triangulation and voronoi tessellation. IETE J. Res. 0(0), 1–17 (2022)

    Google Scholar 

  85. Reddy, G.V., Savarni, C.D., Mukherjee, S.: Facial expression recognition in the wild, by fusion of deep learnt and hand-crafted features. Cogn. Syst. Res. 62, 23–34 (2020)

    Google Scholar 

  86. Riaz, M.N., Shen, Y., Sohail, M., Guo, M.: Exnet: an efficient approach for emotion recognition in the wild. Sensors 20(4), 1087 (2020)

    Google Scholar 

  87. Rinn, W.E.: The neuropsychology of facial expression: a review of the neurological and psychological mechanisms for producing facial expressions. Psychol. Bull. 95(1), 52 (1984)

    Google Scholar 

  88. Ruan, D., Yan, Y., Lai, S., Chai, Z., Shen, C., Wang, H.: Feature decomposition and reconstruction learning for effective facial expression recognition. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp 7660–7669 (2021)

  89. Russell, J.A.: Culture and the categorization of emotions. Psychol. Bull. 110(3), 426–450 (1991)

    Google Scholar 

  90. Samadiani, N., Huang, G., Cai, B., Luo, W., Chi, C.H., Xiang, Y., He, J.: A review on automatic facial expression recognition systems assisted by multimodal sensor data. Sensors 19(8), 1863 (2019)

    Google Scholar 

  91. Samal, A., Iyengar, P.A.: Automatic recognition and analysis of human faces and facial expressions: a survey. Pattern Recogn. 25(1), 65–77 (1992)

    Google Scholar 

  92. Saurav, S., Gidde, P., Saini, R., Singh, S.: Dual integrated convolutional neural network for real-time facial expression recognition in the wild. Vis. Comput. 38(3), 1083–1096 (2022)

    Google Scholar 

  93. Sebe, N., Cohen, I., Gevers, T., Huang, T.S.: Multimodal approaches for emotion recognition: a survey. Internet Imaging VI Int. Soc. Optics Photon. 5670, 56–68 (2005)

    Google Scholar 

  94. Sebe, N., Lew, M.S., Sun, Y., Cohen, I., Gevers, T., Huang, T.S.: Authentic facial expression analysis. Image Vis. Comput. 25(12), 1856–1863 (2007)

    Google Scholar 

  95. Sepas-Moghaddam, A., Etemad, A., Pereira, F., Correia, P.L.: Capsfield: Light field-based face and expression recognition in the wild using capsule routing. arXiv preprint arXiv:2101.03503 (2021)

  96. Shao, J., Qian, Y.: Three convolutional neural network models for facial expression recognition in the wild. Neurocomputing 355, 82–92 (2019)

    Google Scholar 

  97. Shreve, M., Godavarthy, S., Goldgof, D., Sarkar, S.: Macro-and micro-expression spotting in long videos using spatio-temporal strain. In: Face and Gesture 2011, IEEE, pp 51–56 (2011)

  98. Sun, B., Cao, S., He, J., Yu, L.: Affect recognition from facial movements and body gestures by hierarchical deep spatio-temporal features and fusion strategy. Neural Netw. 105, 36–51 (2018)

    Google Scholar 

  99. Sun, N., Li, Q., Huan, R., Liu, J., Han, G.: Deep spatial-temporal feature fusion for facial expression recognition in static images. Pattern Recogn. Lett. 119, 49–61 (2019)

    Google Scholar 

  100. Sun, X., Pingping, Xia SL.: A roi-guided deep architecture for robust facial expressions recognition. Inf. Sci. 522, 35–48 (2020)

    Google Scholar 

  101. Tan, L., Zhang, K., Wang, K., Zeng, X., Peng, X., Qiao, Y.: Group emotion recognition with individual facial emotion cnns and global image based cnns. In: Proceedings of the 19th ACM International Conference on Multimodal Interaction, pp 549–552 (2017)

  102. Tang, Y., Zhang, X.M., Wang, H.: Geometric-convolutional feature fusion based on learning propagation for facial expression recognition. IEEE Access 6, 42532–42540 (2018)

    Google Scholar 

  103. Tian, Y.L., Kanade, T., Cohn, J.F.: Facial expression analysis. In: Handbook of face recognition, Springer, pp 247–275 (2005)

  104. Ullah A, Wang J, Anwar MS, Ahmad A, Nazir S, Khan HU, Fei Z.: Fusion of machine learning and privacy preserving for secure facial expression recognition. Secur. Commun. Netw. 2021 (2021) 1–12

    Google Scholar 

  105. Valstar, M., Pantic, M.: Induced disgust, happiness and surprise: an addition to the mmi facial expression database. In: Proc, pp. 65–70. Int’l Conf. Language Resources and Evaluation, Workshop EMOTION (May 2010)

  106. Valstar, M.F., Mehu, M., Jiang, B., Pantic, M., Scherer, K.: Meta-analysis of the first facial expression recognition challenge. IEEE Transact. Syst. Man Cybern Part B (Cybern) 42(4), 966–979 (2012)

    Google Scholar 

  107. Vo, T.H., Lee, G.S., Yang, H.J., Kim, S.H.: Pyramid with super resolution for in-the-wild facial expression recognition. IEEE Access 8, 131988–132001 (2020)

    Google Scholar 

  108. Wang, K., Peng, X., Yang, J., Lu, S., Qiao, Y.: Suppressing uncertainties for large-scale facial expression recognition. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp 6897–6906 (2020a)

  109. Wang, K., Peng, X., Yang, J., Meng, D., Qiao, Y.: Region attention networks for pose and occlusion robust facial expression recognition. IEEE Trans. Image Process. 29, 4057–4069 (2020b)

  110. Wang, P., Wang, Z., Ji, Z., Liu, X., Yang, S., Wu, Z.: Tal emotionet challenge 2020 rethinking the model chosen problem in multi-task learning. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pp 412–413 (2020c)

  111. Wang, S., Wu, C., He, M., Wang, J., Ji, Q.: Posed and spontaneous expression recognition through modeling their spatial patterns. Mach. Vis. Appl. 26(2–3), 219–231 (2015)

    Google Scholar 

  112. Weber, R., Soladié, C., Séguier, R.: A survey on databases for facial expression analysis. In: VISIGRAPP (5: VISAPP), pp 73–84 (2018)

  113. Whitehill, J., Serpell, Y.C., Foster, A., Movellan, J.R.: The faces of engagement: automatic recognition of student engagement from facial expressions. IEEE Trans. Affect. Comput. 5(1), 86–98 (2014)

    Google Scholar 

  114. Xiaohua, W., Muzi, P., Lijuan, P., Min, H., Chunhua, J., Fuji, R.: Two-level attention with two-stage multi-task learning for facial emotion recognition. J. Vis. Commun. Image Represent. 62, 217–225 (2019)

    Google Scholar 

  115. Xie, S., Hu, H.: Facial expression recognition using hierarchical features with deep comprehensive multipatches aggregation convolutional neural networks. IEEE Trans. Multimed. 21(1), 211–220 (2018)

    MathSciNet  Google Scholar 

  116. Xie, S., Hu, H., Wu, Y.: Deep multi-path convolutional neural network joint with salient region attention for facial expression recognition. Pattern Recogn. 92, 177–191 (2019)

    Google Scholar 

  117. Yan, J., Zheng, W., Cui, Z., Tang, C., Zhang, T., Zong, Y.: Multi-cue fusion for emotion recognition in the wild. Neurocomputing 309, 27–35 (2018)

    Google Scholar 

  118. Yan, W.J., Li, X., Wang, S.J., Zhao, G., Liu, Y.J., Chen, Y.H., Fu, X.: Casme ii: an improved spontaneous micro-expression database and the baseline evaluation. PLoS ONE 9(1), e86041 (2014)

    Google Scholar 

  119. Yan, W.J., Li, S., Que, C., Pei, J., Deng, W.: Raf-au database: In-the-wild facial expressions with subjective emotion judgement and objective au annotations. In: Proceedings of the Asian Conference on Computer Vision (2020)

  120. Zhang, F., Yu, Y., Mao, Q., Gou, J., Zhan, Y.: Pose-robust feature learning for facial expression recognition. Front. Comp. Sci. 10(5), 832–844 (2016)

    Google Scholar 

  121. Zhang, K., Huang, Y., Du, Y., Wang, L.: Facial expression recognition based on deep evolutional spatial-temporal networks. IEEE Trans. Image Process. 26(9), 4193–4203 (2017)

    MathSciNet  MATH  Google Scholar 

  122. Zhang, T., Zheng, W., Cui, Z., Zong, Y., Yan, J., Yan, K.: A deep neural network-driven feature learning method for multi-view facial expression recognition. IEEE Trans. Multimed. 18(12), 2528–2536 (2016)

    Google Scholar 

  123. Zhang, Z., Luo, P., Loy, C.C., Tang, X.: From facial expression recognition to interpersonal relation prediction. Int. J. Comput. Vision 126(5), 550–569 (2018)

    MathSciNet  Google Scholar 

  124. Zhao, G., Yang, H., Yu, M.: Expression recognition method based on a lightweight convolutional neural network. IEEE Access 8, 38528–38537 (2020)

    Google Scholar 

  125. Zhao, X., Zou, J., Li, H., Dellandréa, E., Kakadiaris, I.A., Chen, L.: Automatic 2.5-d facial landmarking and emotion annotation for social interaction assistance. IEEE Transact Cybern. 46(9), 2042–2055 (2015)

    Google Scholar 

  126. Zhi, R., Zhou, C., Li, T., Liu, S., Jin, Y.: Action unit analysis enhanced facial expression recognition by deep neural network evolution. Neurocomputing 425, 135–148 (2021)

    Google Scholar 

  127. Zhong, L., Liu, Q., Yang, P., Huang, J., Metaxas, D.N.: Learning multiscale active facial patches for expression analysis. IEEE Transact. Cybern. 45(8), 1499–1510 (2014)

    Google Scholar 

  128. Zhu, J., Luo, B., Zhao, S., Ying, S., Zhao, X., Gao, Y.: Iexpressnet: Facial expression recognition with incremental classes. In: Proceedings of the 28th ACM International Conference on Multimedia, pp 2899–2908 (2020)

  129. Zhu, Q., Mao, Q., Jia, H., Noi, O.E.N., Tu, J.: Convolutional relation network for facial expression recognition in the wild with few-shot learning. Expert Syst. Appl. 189, 116046 (2022)

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Computer Science and Engineering, National Institute of Technology Karnataka, Surathkal, India

    R. Rashmi Adyapady & B. Annappa

Authors
  1. R. Rashmi Adyapady
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. B. Annappa
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

Conceptualization, literature review, original draft preparation: RAR, supervision, guidance and review: AB.

Corresponding author

Correspondence to R. Rashmi Adyapady.

Ethics declarations

Conflict of interest

The authors declare that they have no conflicts of interest.

Additional information

Communicated by R. Huang.

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Adyapady, R.R., Annappa, B. A comprehensive review of facial expression recognition techniques. Multimedia Systems 29, 73–103 (2023). https://doi.org/10.1007/s00530-022-00984-w

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00530-022-00984-w

Keywords

Search

Navigation