Skip to main content
SpringerLink
Log in

An efficient facial emotion recognition system using novel deep learning neural network-regression activation classifier

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

In the computer vision field, FER encompasses a significant place. It is being studied for a long period, and in recent decades, it has attained progress, but all is in vain, since, recognizing facial expression with high accuracy is still hard due to disparate facial expressions. To beat such difficulties, an efficient Facial Emotion Recognitions (FER) is proposed by utilizing a novel Deep Learning Neural Network-regression activation (DR) classifier. The proposed method has six phases, namely, pre-processing, facial point extraction, segmentation, feature extraction, feature selection, and classification. Initially the input image has been pre-processed using Gamma-HE technique and then facial points are extracted using Pyramid Histogram of Oriented Gradients (PHOG) based Supervised Descent (SMD) Method. The facial parts are segmented using Viola-Jones Algorithm (VJA) and then Local Tetra Pattern (LTrP), cluster shade, Inverse Divergent Moment (IDM), Local homogeneity, optimum probability, cluster prominence, dissimilarity, autocorrelation, and contrast features have been extracted. Modified Monarch Butterfly Optimization (MMBO) algorithm has been used to select necessary features from the extracted features. From the extracted facial points, the DR classifier classifies the emotions of the particular input image. A ‘2’ datasets were taken for analyzing the proposed system’s performance. Centred on the CK+ database, the proposed work attains 0.9885-accuracy, and centred on the JAFFE database, it has 0.9727-accuracy. Also, the investigational results proved that the proposed work trounces the existing systems centred on statistical metrics.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8

Similar content being viewed by others

References

  1. Arora M, Kumar M (2020) AutoFER: PCA and PSO based automatic facial emotion recognition. Multimedia Tools Appl 1–11. https://doi.org/10.1007/s11042-020-09726-4

  2. Arora M, Kumar M, Garg NK (2018) Facial emotion recognition system based on PCA and gradient features. Natl Acad Sci Lett 41(6):365–368

    Article  Google Scholar 

  3. Bastanfard A, Bastanfard O, Takahashi H, Nakajima M (2004) Toward anthropometrics simulation of face rejuvenation and skin cosmetic. Comput Virtual Worlds 15(34):347–352

    Article  Google Scholar 

  4. Bastanfard A, Takahashi H, Nakajima M (n.d.) 2004Toward E-appearance of human face and hair by age, expression and rejuvenation. International conference on Cyberworlds. https://doi.org/10.1109/cw.2004.65.

  5. Boutorh A, Guessoum A (2016) Complex diseases SNP selection and classification by hybrid association rule mining and artificial neural network—based evolutionary algorithms. Eng Appl Artif Intell 51:58–70. https://doi.org/10.1016/j.engappai.2016.01.004

    Article  Google Scholar 

  6. Chen J, Lv Y, Xu R, Xu C (2019) Automatic social signal analysis: facial expression recognition using difference convolution neural network. Journal of Parallel and Distributed Computing 131:97–102. https://doi.org/10.1016/j.jpdc.2019.04.017

    Article  Google Scholar 

  7. Fan X, Tjahjadi T (2019) Fusing dynamic deep learned features and handcrafted features for facial expression recognition. J Vis Commun Image Represent 65:10265. https://doi.org/10.1016/j.jvcir.2019.102659

    Article  Google Scholar 

  8. Farkhod M, Abdullah-Al-Wadud M, Iqbal MTB, Ryu B, Chae O (2019) Facial expression recognition with local prominent directional pattern. Signal Process Image Commun 74:1–12. https://doi.org/10.1016/j.image.2019.01.002

    Article  Google Scholar 

  9. Ghosh M, Kundu T, Ghosh D, Sarkar R (2019) Feature selection for facial emotion recognition using late hill-climbing based memetic algorithm. Multimed Tools Appl 78(18):25753–25779. https://doi.org/10.1007/s11042-019-07811-x

    Article  Google Scholar 

  10. He H, Tan Y, Ying J, Zhang W (2020) Strengthen EEG-based emotion recognition using firefly integrated optimization algorithm. Appl Soft Comput 94:106426. https://doi.org/10.1016/j.asoc.2020.106426

    Article  Google Scholar 

  11. Hu H, Li Y, Zhu Z, Zhou G (2018) CNNAuth: continuous authentication via two-stream convolutional neural networks. In: IEEE International Conference on Networking, Architecture and Storage (NAS), IEEE, 1–9. https://doi.org/10.1109/NAS.2018.8515693.

  12. Jessica Bone K, Lewis G, Button KS, Duffy L, Harmer CJ, Munafò MR, Penton-Voak IS, Wiles NJ, Lewis G (2019) Variation in recognition of happy and sad facial expressions and self-reported depressive symptom severity: a prospective cohort study. J Affect Disord 257:461–469. https://doi.org/10.1016/j.jad.2019.06.025

    Article  Google Scholar 

  13. Ji Y, Hu Y, Yang Y, Shen F, Shen HT (2019) Cross-domain facial expression recognition via an intra-category common feature and inter-category distinction feature fusion network. Neurocomputing 333:231–239. https://doi.org/10.1016/j.neucom.2018.12.037

    Article  Google Scholar 

  14. Jiana J, Lin J, Xiao-huaa Z, Haoa L (2011) Inversion of neural network rayleigh wave dispersion based on LM algorithm. Adv Control Eng Inf Sci 15:5126–5132. https://doi.org/10.1016/j.proeng.2011.08.951

    Article  Google Scholar 

  15. Kalantarian H, Jedoui K, Washington P, Tariq Q, Dunlap K, Schwartz J, Wall DP (2019) Labeling images with facial emotion and the potential for pediatric healthcare. Artif Intell Med 98:77–86. https://doi.org/10.1016/j.artmed.2019.06.004

    Article  Google Scholar 

  16. Kalsum T, Anwar SM, Majid M, Khan B, Ali SM (2018) Emotion recognition from facial expressions using hybrid feature descriptors. IET Image Process 12(6):1004–1012. https://doi.org/10.1049/iet-ipr.2017.0499

    Article  Google Scholar 

  17. Kim J-H, Kim B-G, Roy PP, Jeong D-M (2019) Efficient facial expression recognition algorithm based on hierarchical deep neural network structure. IEEE Access 7:41273–41285. https://doi.org/10.1109/ACCESS.2019.2907327

    Article  Google Scholar 

  18. Kobayashi M (2017) Gradient descent learning for quaternionic Hopfield neural networks. Neurocomputing 260:174–179. https://doi.org/10.1016/j.neucom.2017.04.025\

    Article  Google Scholar 

  19. Li Y, Hu H, Zhu Z, Zhou G (2020) SCANet: sensor-based continuous authentication with two-stream convolutional neural networks. ACM Trans Sens Netw (TOSN) 16(3):1–27. https://doi.org/10.1145/3397179

    Article  Google Scholar 

  20. Lin C-H (2016) Novel application of continuously variable transmission system using composite recurrent Laguerre orthogonal polynomials modified PSO NN control system. ISA Trans 64:405–417. https://doi.org/10.1016/j.isatra.2016.05.013

    Article  Google Scholar 

  21. Majumder A, Behera L, Subramanian VK (2016) Automatic facial expression recognition system using deep network-based data fusion. IEEE Trans Cybern 48(1):103–114. https://doi.org/10.1109/TCYB.2016.2625419

    Article  Google Scholar 

  22. Maria E, Matthias L, Sten H (2019) Emotion recognition from physiological signal analysis: a review. Electron Notes Theor Comput Sci 343:35–55. https://doi.org/10.1016/j.entcs.2019.04.009

    Article  Google Scholar 

  23. Mlakar U, Fister I, Brest J, Potočnik B (2017) Multi-objective differential evolution for feature selection in facial expression recognition systems. Expert Syst Appl 89:129–137. https://doi.org/10.1016/j.eswa.2017.07.037

    Article  Google Scholar 

  24. Pino M, Montaño S, Agudelo K, Idárraga-Cabrera C, Fernández-Lucas J, Herrera-Mendoza K (2019) Emotion recognition in young male offenders and non-offenders. Physiol Behav 207:73–75. https://doi.org/10.1016/j.physbeh.2019.03.015

    Article  Google Scholar 

  25. Prasada Rao K, Rao MVPCS, Chowdary NH (2019) An integrated approach to emotion recognition and gender classification. J Vis Commun Image Represent 60:339–345. https://doi.org/10.1016/j.jvcir.2019.03.002

    Article  Google Scholar 

  26. Rodriguez P, Cucurull G, Gonzàlez J, Gonfaus JM, Nasrollahi K, Moeslund TB, Xavier Roca F (2017) Deep pain: exploiting long short-term memory networks for facial expression classification. IEEE Trans Cybern. https://doi.org/10.1109/TCYB.2017.2662199

  27. Sadeghi H, Raie AA (2019) Histogram distance metric learning for facial expression recognition. J Vis Commun Image Represent 62:152–165. https://doi.org/10.1016/j.jvcir.2019.05.004

    Article  Google Scholar 

  28. Sánchez D, Melin P, Castillo O (2017) Optimization of modular granular neural networks using a firefly algorithm for human recognition. Eng Appl Artif Intell 64:172–186. https://doi.org/10.1016/j.engappai.2017.06.007

    Article  Google Scholar 

  29. Shao J, Qian Y (2019) Three convolutional neural network models for facial expression recognition in the wild. Neurocomputing 355:82–92. https://doi.org/10.1016/j.neucom.2019.05.005

    Article  Google Scholar 

  30. Sheheryar K, Chen L, Yan H (2017) Co-clustering to reveal salient facial features for expression recognition. IEEE Trans Affect Comput 11:348–360. https://doi.org/10.1109/TAFFC.2017.2780838

    Article  Google Scholar 

  31. Sreedharan NPN, Ganesan B, Raveendran R, Sarala P, Dennis B (2018) Grey wolf optimisation-based feature selection and classification for facial emotion recognition. IET Biom 7(5):490–499

    Article  Google Scholar 

  32. Wang X, Peng M, Pan L, Hu M, Jin C, Ren F (2018) Two-level attention with two-stage multi-task learning for facial emotion recognition. arXiv preprint arXiv 1811.12139

  33. Wang F, Lv J, Ying G, Chen S, Zhang C (2019) Facial expression recognition from image based on hybrid features understanding. J Vis Commun Image Represent 59:84–88. https://doi.org/10.1016/j.jvcir.2018.11.010

    Article  Google Scholar 

  34. Xie S, Hu H (2018) Facial expression recognition using hierarchical features with deep comprehensive multipatches aggregation convolutional neural networks. IEEE Trans Multimedia 21(1):211–220

    Article  Google Scholar 

  35. Yang M, Liu Y, You Z (2017) The Euclidean embedding learning based on convolutional neural network for stereo matching. Neurocomputing 267:195–200. https://doi.org/10.1016/j.neucom.2017.06.007

    Article  Google Scholar 

  36. Ye Y, Zhang X, Lin Y, Wang H (2019) Facial expression recognition via region-based convolutional fusion network. J Vis Commun Image Represent 62:1–11. https://doi.org/10.1016/j.jvcir.2019.04.009

    Article  Google Scholar 

  37. Yin Z, Liu L, Chen J, Zhao B, Wang Y (2020) Locally robust EEG feature selection for individual-independent emotion recognition. Expert Syst Appl 162:113768

    Article  Google Scholar 

  38. Zhuang N, Zhang Q, Pan C, Ni B, Xu Y, Yang X, Zhang W (2019) Recognition oriented facial image quality assessment via deep convolutional neural network. Neurocomputing 358:109–118. https://doi.org/10.1016/j.neucom.2019.04.057

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Computer Science and Engineering, JNTUK, Kakinada, Andhra Pradesh, India

    Anjani Suputri Devi D &  Satyanarayana Ch

Authors
  1. Anjani Suputri Devi D
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Satyanarayana Ch
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Anjani Suputri Devi D.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Anjani Suputri Devi D, Satyanarayana Ch An efficient facial emotion recognition system using novel deep learning neural network-regression activation classifier. Multimed Tools Appl 80, 17543–17568 (2021). https://doi.org/10.1007/s11042-021-10547-2

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-021-10547-2

Keywords

Search

Navigation