Skip to main content
SpringerLink
Log in

FER-net: facial expression recognition using deep neural net

  • Original Article
  • Published:
Neural Computing and Applications Aims and scope Submit manuscript

Abstract

Automatic facial expression recognition (FER) is one of the most challenging tasks in computer vision. FER admits a wide range of applications in human–computer interaction, behavioral psychology, and human expression synthesis. Extensive works have been reported in this field, mainly, based on handcrafted features. However, it is a challenging task to accurately extract all the correlated handcrafted features due to the effect of variations caused by emotional state. Therefore, there is a quest for further research on accurately extracting relevant features that can capture changes in facial expressions (FEs) with high fidelity. In this study, we propose FER-net: a convolution neural network to distinguish FEs efficiently with the help of the softmax classifier. We implement our method FER-net along with twenty-one state-of-the-art methods and test them on five benchmarking datasets, namely FER2013, Japanese Female Facial Expressions, Extended CohnKanade, Karolinska Directed Emotional Faces, and Real-world Affective Faces. Seven FEs, namely neutral, anger, disgust, fear, happiness, sadness, and surprise, are considered in this work. The average accuracies on these datasets are 78.9%, 96.7%, 97.8%, 82.5% and 81.68%, respectively. The obtained results demonstrate that FER-net is preeminent in comparison with twenty-one state-of-the-art methods.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11

Similar content being viewed by others

References

  1. Yeasin M, Bullot B, Sharma R (2006) Recognition of facial expressions and measurement of levels of interest from video. IEEE Trans Multimed 8(3):500–508

    Article  Google Scholar 

  2. Lola C, Philippe G (2005) Emotion understanding: robots as tools and models. University Press England, Oxford

    Google Scholar 

  3. Dornaika F, Raducanu B (2009) Facial expression recognition for hci applications. In: Encyclopedia of artificial intelligence. IGI Global, pp 625–631

  4. Shakya S, Sharma S, Basnet A (2016) Human behavior prediction using facial expression analysis. In: 2016 international conference on computing, communication and automation (ICCCA). IEEE, pp 399–404

  5. Dickey CC, Panych LP, Voglmaier MM, Niznikiewicz MA, Terry DP, Murphy C, Zacks R, Shenton ME, McCarley RW (2011) Facial emotion recognition and facial affect display in schizotypal personality disorder. Schizophr Res 131(1–3):242–249

    Article  Google Scholar 

  6. Jeong M, Ko BC (2018) Driver’s facial expression recognition in real-time for safe driving. Sensors 18(12):4270

    Article  Google Scholar 

  7. Zhou Y, Shi BE (2017) Photorealistic facial expression synthesis by the conditional difference adversarial autoencoder. In: 2017 seventh international conference on affective computing and intelligent interaction (ACII). IEEE, pp 370–376

  8. Viola P, Jones P et al (2001) Rapid object detection using a boosted cascade of simple features. In: CVPR 2001, vol 1, no 511–518, p 3

  9. Siswanto ARS, Nugroho AS, Galinium M (2014) Implementation of face recognition algorithm for biometrics based time attendance system. In: 2014 international conference on ICT For Smart Society (ICISS). IEEE, pp 149–154

  10. Turk MA, Pentland AP (1991) Face recognition using eigenfaces. In: Proceedings. 1991 IEEE computer society conference on computer vision and pattern recognition. IEEE, pp 586–591

  11. Lundqvist D, Flykt A, Öhman A (1998) The karolinska directed emotional faces (kdef), vol 91. CD ROM from Department of Clinical Neuroscience, Psychology section, Karolinska Institutet, Solna, p 30

    Google Scholar 

  12. Mohammadi MR, Fatemizadeh E, Mahoor MH (2014) Pca-based dictionary building for accurate facial expression recognition via sparse representation. J Vis Commun Image Represent 25(5):1082–1092

    Article  Google Scholar 

  13. Liu C, Wechsler H (2002) Gabor feature based classification using the enhanced fisher linear discriminant model for face recognition. IEEE Trans Image Process 11(4):467–476

    Article  Google Scholar 

  14. Lucey P, Cohn JF, Kanade T, Saragih J, Ambadar Z, Matthews I (2010) The extended cohn-kanade dataset (ck+): a complete dataset for action unit and emotion-specified expression. In: 2010 IEEE computer society conference on computer vision and pattern recognition-workshops. IEEE, pp 94–101

  15. Shan C, Gong S, McOwan PW (2009) Facial expression recognition based on local binary patterns: a comprehensive study. Image Vis Comput 27(6):803–816

    Article  Google Scholar 

  16. Kobayashi H, Hara F (1997) Facial interaction between animated 3d face robot and human beings. In: 1997 IEEE international conference on systems, man, and cybernetics. Computational cybernetics and simulation, vol 4. IEEE, pp 3732–3737

  17. Zhong L, Liu Q, Yang P, Huang J, Metaxas DN (2014) Learning multiscale active facial patches for expression analysis. IEEE Trans Cybern 45(8):1499–1510

    Article  Google Scholar 

  18. Mase K (1991) Recognition of facial expression from optical flow. IEICE Trans Inf Syst 74(10):3474–3483

    Google Scholar 

  19. Chen C-R, Wong W-S, Chiu C-T (2010) A 0.64 mm \(^{2}\) real-time cascade face detection design based on reduced two-field extraction. IEEE Trans Very Large Scale Integr (VLSI) Syst 19(11):1937–1948

    Article  Google Scholar 

  20. Kotsia I, Pitas I (2006) Facial expression recognition in image sequences using geometric deformation features and support vector machines. IEEE Trans Image Process 16(1):172–187

    Article  MathSciNet  Google Scholar 

  21. Sohail ASM, Bhattacharya P (2007) Classification of facial expressions using k-nearest neighbor classifier. In: International conference on computer vision/computer graphics collaboration techniques and applications. Springer, pp 555–566

  22. Li X, Ji Q (2004) Active affective state detection and user assistance with dynamic Bayesian networks. IEEE Trans Syst Man Cybern Part A Syst Huma 35(1):93–105

    Article  MathSciNet  Google Scholar 

  23. Fanelli G, Gall J, Van Gool L (2011) Real time head pose estimation with random regression forests. In: CVPR 2011. IEEE, pp 617–624

  24. Salmam FZ, Madani A, Kissi M (2016) Facial expression recognition using decision trees. In: 2016 13th international conference on computer graphics, imaging and visualization (CGiV). IEEE, pp 125–130

  25. Sebe N, Lew MS, Cohen I, Garg A, Huang TS (2002) Emotion recognition using a Cauchy naive Bayes classifier. In: Makihara Y, Takizawa M, Shirai Y, Miura J, Shimada N (eds) Object recognition supported by user interaction for service robots, vol 1. IEEE, New York, pp 17–20

    Chapter  Google Scholar 

  26. Chen J, Chen Z, Chi Z, Hong F (2016) Facial expression recognition in video with multiple feature fusion. IEEE Trans Affect Comput 9(1):38–50

    Article  Google Scholar 

  27. Karlekar A, Seal A (2020) Soynet: Soybean leaf diseases classification. Comput Electron Agric 172:105342

    Article  Google Scholar 

  28. Mohan K, Seal A, Krejcar O, Yazidi A (2020) Facial expression recognition using local gravitational force descriptor based deep convolution neural networks. IEEE Trans Instrum Meas 70:1–12

    Article  Google Scholar 

  29. Zhao K, Chu W-S, Zhang H (2016) Deep region and multi-label learning for facial action unit detection. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp 3391–3399

  30. Sun N, Li Q, Huan R, Liu J, Han G (2017) Deep spatial-temporal feature fusion for facial expression recognition in static images. Pattern Recogn Lett 119:49–61

    Article  Google Scholar 

  31. Orozco D, Lee C, Arabadzhi Y, Gupta D. Transfer learning for facial expression recognition

  32. Alizadeh S, Fazel A (2017) Convolutional neural networks for facial expression recognition arXiv:1704:06756

  33. Zhao G, Pietikainen M (2007) Dynamic texture recognition using local binary patterns with an application to facial expressions. IEEE Trans Pattern Anal Mach Intell 6:915–928

    Article  Google Scholar 

  34. Dhall A, Goecke R, Joshi J, Wagner M, Gedeon T (2013) Emotion recognition in the wild challenge 2013. In: Proceedings of the 15th ACM on international conference on multimodal interaction. ACM, pp 509–516

  35. Yu Z, Zhang C (2015) Image based static facial expression recognition with multiple deep network learning. In Proceedings of the 2015 ACM on international conference on multimodal interaction. ACM, pp 435–442

  36. Jain DK, Shamsolmoali P, Sehdev P (2019) Extended deep neural network for facial emotion recognition. Pattern Recogn Lett 120:69–74

    Article  Google Scholar 

  37. Salmam FZ, Madani A, Kissi M (2019) Fusing multi-stream deep neural networks for facial expression recognition. SIViP 13(3):609–616

    Article  Google Scholar 

  38. Li K, Jin Y, Akram MW, Han R, Chen J (2019) Facial expression recognition with convolutional neural networks via a new face cropping and rotation strategy. Vis Comput 36:391–404

    Article  Google Scholar 

  39. Shao J, Qian Y (2019) Three convolutional neural network models for facial expression recognition in the wild. Neurocomputing 355:82–92

    Article  Google Scholar 

  40. Uddin MZ, Hassan MM, Almogren A, Alamri A, Alrubaian M, Fortino G (2017) Facial expression recognition utilizing local direction-based robust features and deep belief network. IEEE Access 5:4525–4536

    Article  Google Scholar 

  41. Pons G, Masip D (2017) Supervised committee of convolutional neural networks in automated facial expression analysis. IEEE Trans Affect Comput 9(3):343–350

    Article  Google Scholar 

  42. Villanueva MG, Zavala SR, Zavala (2020) Deep neural network architecture: application for facial expression recognition. IEEE Lat Am Trans 18(07):1311–1319

    Article  Google Scholar 

  43. Liu X, Vijaya Kumar BV, You J, Jia P (2017) Adaptive deep metric learning for identity-aware facial expression recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition workshops, pp 20–29

  44. Meng Z, Liu P, Cai J, Han S, Tong Y (2017) Identity-aware convolutional neural network for facial expression recognition. In: 2017 12th IEEE international conference on automatic face & gesture recognition (FG 2017). IEEE, pp 558–565

  45. Minaee S, Abdolrashidi A (2019) Deep-emotion: Facial expression recognition using attentional convolutional network. arXiv:1902.01019

  46. Hua W, Dai F, Huang L, Xiong J, Gui G (2019) Hero: human emotions recognition for realizing intelligent internet of things. IEEE Access 7:24321–24332

    Article  Google Scholar 

  47. Kim J-H, Kim B-G, Roy PP, Jeong D-M (2019) Efficient facial expression recognition algorithm based on hierarchical deep neural network structure. IEEE Access 7:41273–41285

    Article  Google Scholar 

  48. Kim DH, Baddar WJ, Jang J, Ro YM (2017) Multi-objective based spatio-temporal feature representation learning robust to expression intensity variations for facial expression recognition. IEEE Trans Affect Comput 10(2):223–236

    Article  Google Scholar 

  49. Benitez-Quiroz CF, Srinivasan R, Martinez AM (2018) Discriminant functional learning of color features for the recognition of facial action units and their intensities. IEEE Trans Pattern Anal Mach Intell 41(12):2835–2845

    Article  Google Scholar 

  50. Zhang T, Liu Z, Wang X-H, Xing X-F, Chen CP, Chen E (2018) Facial expression recognition via broad learning system. In: 2018 IEEE international conference on systems, man, and cybernetics (SMC). IEEE, pp 1898–1902

  51. Zhao X, Shi X, Zhang S (2015) Facial expression recognition via deep learning. IETE Tech Rev 32(5):347–355

    Article  Google Scholar 

  52. Chellappa R, Wilson CL, Sirohey S et al (1995) Human and machine recognition of faces: a survey. Proc IEEE 83(5):705–740

    Article  Google Scholar 

  53. Samal A, Iyengar PA (1992) Automatic recognition and analysis of human faces and facial expressions: a survey. Pattern Recogn 25(1):65–77

    Article  Google Scholar 

  54. Li H, Lin Z, Shen X, Brandt J, Hua G (2015) A convolutional neural network cascade for face detection. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 5325–5334

  55. Lyons M, Akamatsu S, Kamachi M, Gyoba J (1998) Coding facial expressions with gabor wavelets. In: Proceedings third IEEE international conference on automatic face and gesture recognition. IEEE, pp 200–205

  56. Li S, Deng W (2019) Reliable crowdsourcing and deep locality-preserving learning for unconstrained facial expression recognition. IEEE Trans Image Process 28(1):356–370

    Article  MathSciNet  MATH  Google Scholar 

  57. Li S, Deng W, Du JP (2017) Reliable crowdsourcing and deep locality-preserving learning for expression recognition in the wild. In: 2017 IEEE conference on computer vision and pattern recognition (CVPR). IEEE, pp 2584–2593

  58. LeCun Y, Haffner P, Bottou L, Bengio Y (1999) Object recognition with gradient-based learning. In: Forsyth DA, Mundy JL, di Gesu V, Cipolla R (eds) Shape, contour and grouping in computer vision. Springer, Berlin, pp 319–345

    Chapter  Google Scholar 

  59. Ding H, Zhou SK, Chellappa R (2017) Facenet2expnet: regularizing a deep face recognition net for expression recognition. In: 2017 12th IEEE international conference on automatic face and gesture recognition (FG 2017). IEEE, pp 118–126

  60. Simonyan K, Zisserman A (2014) Very deep convolutional networks for large-scale image recognition. arXiv:1409.1556

  61. He K, Zhang X, Ren S Sun J (2016) Deep residual learning for image recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 770–778

  62. Szegedy C, Liu W, Jia Y, Sermanet P, Reed S, Anguelov D, Erhan D, Vanhoucke V, Rabinovich A (2015) Going deeper with convolutions. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 1–9

  63. Huang G, Liu Z, Van Der Maaten L, Weinberger KQ (2017) Densely connected convolutional networks. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 4700–4708

  64. Chollet F (2017) Xception: deep learning with depthwise separable convolutions. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 1251–1258

  65. Krizhevsky A, Sutskever I, Hinton GE (2012) Imagenet classification with deep convolutional neural networks. In: Advances in neural information processing systems, pp 1097–1105

  66. Karlekar A, Seal A, Krejcar O, Gonzalo-Martin C (2019) Fuzzy k-means using non-linear s-distance. IEEE Access 7:55121–55131

    Article  Google Scholar 

  67. Sharma KK, Seal A (2019) Modeling uncertain data using Monte Carlo integration method for clustering. Expert Syst Appl 137:100–116

    Article  Google Scholar 

  68. Sharma KK, Seal A (2020) Spectral embedded generalized mean based k-nearest neighbors clustering with s-distance. Expert Syst Appl. https://doi.org/10.1016/j.eswa.2020.114326

    Article  Google Scholar 

  69. Sharma KK, Seal A (2020) Outlier-robust multi-view clustering for uncertain data. Knowl Based Syst 211:106567

    Article  Google Scholar 

  70. Jain S, Seal A, Ojha A, Krejcar O, Bureš J, Tachecí I, Yazidi A (2020) Detection of abnormality in wireless capsule endoscopy images using fractal features. Comput Biol Med 127:104094

    Article  Google Scholar 

  71. Sharma KK, Seal A (2020) Clustering analysis using an adaptive fused distance. Eng Appl Artif Intell 96:103928

    Article  Google Scholar 

  72. Sharma KK, Seal A (2020) Multi-view spectral clustering for uncertain objects. Inf Sci 547:723–745

    Article  MathSciNet  Google Scholar 

Download references

Acknowledgements

This work is partially supported by the project “Prediction of diseases through computer-assisted diagnosis system using images captured by minimally invasive and non-invasive modalities”, Computer Science and Engineering, PDPM Indian Institute of Information Technology, Design and Manufacturing, Jabalpur, India (under ID: SPARC-MHRD-231). This work is also partially supported by the project at Universiti Teknologi Malaysia (UTM) under Research University Grant Vot-20H04, Malaysia Research University Network (MRUN) Vot 4L876 and the Fundamental Research Grant Scheme (FRGS) Vot5F073 supported under Ministry of Education Malaysia for the completion of the research.

Author information

Authors and Affiliations

  1. PDPM Indian Institute of Information Technology, Design and Manufacturing, Jabalpur, 482005, India

    Karnati Mohan & Ayan Seal

  2. Faculty of Informatics and Management, Center for Basic and Applied Research, University of Hradec Kralove, Rokitanskeho 62, 50003, Hradec Kralove, Czech Republic

    Ayan Seal & Ondrej Krejcar

  3. Malaysia-Japan International Institute of Technology (MJIIT), Universiti Teknologi Malaysia, Jalan Sultan Yahya Petra, 54100, Kuala Lumpur, Malaysia

    Ondrej Krejcar

  4. Research Group in Applied Artificial Intelligence, Oslo Metropolitan University, 460167, Oslo, Norway

    Anis Yazidi

Authors
  1. Karnati Mohan
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Ayan Seal
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Ondrej Krejcar
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Anis Yazidi
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Ayan Seal.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Mohan, K., Seal, A., Krejcar, O. et al. FER-net: facial expression recognition using deep neural net. Neural Comput & Applic 33, 9125–9136 (2021). https://doi.org/10.1007/s00521-020-05676-y

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00521-020-05676-y

Keywords

Search

Navigation