Skip to main content

Advertisement

SpringerLink
Log in

Efficient facial emotion recognition model using deep convolutional neural network and modified joint trilateral filter

  • Focus
  • Published:
Soft Computing Aims and scope Submit manuscript

Abstract

Facial emotion recognition extracts the human emotions from the images and videos. As such, it requires an algorithm to understand and model the relationships between faces and facial expressions and to recognize human emotions. Recently, deep learning models are utilized to improve the performance of facial emotion recognition. However, the deep learning models suffer from the overfitting issue. Moreover, deep learning models perform poorly for images which have poor visibility and noise. Therefore, in this paper, an efficient deep learning-based facial emotion recognition model is proposed. Initially, contrast-limited adaptive histogram equalization (CLAHE) is applied to improve the visibility of input images. Thereafter, a modified joint trilateral filter is applied to the obtained enhanced images to remove the impact of impulsive noise. Finally, an efficient deep convolutional neural network is designed. Adam optimizer is also utilized to optimize the cost function of deep convolutional neural networks. Experiments are conducted by using the benchmark dataset and competitive human emotion recognition models. Comparative analysis demonstrates that the proposed facial emotion recognition model performs considerably better compared to the competitive models

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8

Similar content being viewed by others

Data Availability

Datasets is freely available on Kaggle website https://www.kaggle.com/shawon10/ckplus.

References

  • Agrawal A, Mittal N (2020) Using cnn for facial expression recognition: a study of the effects of kernel size and number of filters on accuracy. Vis Comput 36(2):405–412

    Article  Google Scholar 

  • Alam M, Vidyaratne LS, Iftekharuddin KM (2018) Sparse simultaneous recurrent deep learning for robust facial expression recognition. IEEE Trans Neural Net Learn Syst 29(10):4905–4916

    Article  Google Scholar 

  • Baddar WJ, Lee S, Ro YM (2019) On-the-fly facial expression prediction using lstm encoded appearance-suppressed dynamics. IEEE Trans Affect Comput

  • Bargshady G, Zhou X, Deo RC, Soar J, Whittaker F, Wang H (2020) Ensemble neural network approach detecting pain intensity from facial expressions. Artif Intell Med 109:101954

    Article  Google Scholar 

  • Barsoum E, Zhang C, Ferrer CC, Zhang Z (2016) Training deep networks for facial expression recognition with crowd-sourced label distribution. In: Proceedings of the 18th ACM international conference on multimodal interaction, pp 279–283

  • Basavegowda HS, Dagnew G (2020) Deep learning approach for microarray cancer data classification. CAAI Trans Intell Technol 5(1):22–33

    Article  Google Scholar 

  • Chen L, Zhou M, Su W, Wu M, She J, Hirota K (2018) Softmax regression based deep sparse autoencoder network for facial emotion recognition in human-robot interaction. Inf Sci 428:49–61

    Article  MathSciNet  Google Scholar 

  • Cheng S, Zhou G (2020) Facial expression recognition method based on improved vgg convolutional neural network. Int J Pattern Recognit Artif Intell 34(07):2056003

    Article  Google Scholar 

  • Choi DY, Song BC (2020) Semi-supervised learning for continuous emotion recognition based on metric learning. IEEE Access 8:113443–113455

    Article  Google Scholar 

  • Choi DY, Song BC (2020) Facial micro-expression recognition using two-dimensional landmark feature maps. IEEE Access 8:121549–121563

    Article  Google Scholar 

  • Choudhury P, Tumblin J (2003) The trilateral filter for high contrast images and meshes, pp 186–196. https://doi.org/10.1145/1198555.1198565

  • Deng J, Pang G, Zhang Z, Pang Z, Yang H, Yang G (2019) cgan based facial expression recognition for human-robot interaction. IEEE Access 7:9848–9859

    Article  Google Scholar 

  • Du G, Long S, Yuan H (2020) Non-contact emotion recognition combining heart rate and facial expression for interactive gaming environments. IEEE Access 8:11896–11906

    Article  Google Scholar 

  • Duchi HE, Singer JY (2011) Adaptive subgradient methods for online learning and stochastic optimization. J Mach Learn Res 12:2121–2159

    MathSciNet  MATH  Google Scholar 

  • Fei Z, Yang E, Li DD-U, Butler S, Ijomah W, Li X, Zhou H (2020) Deep convolution network based emotion analysis towards mental health care. Neurocomputing 388:212–227

    Article  Google Scholar 

  • Ferreira PM, Marques F, Cardoso JS, Rebelo A (2018) Physiological inspired deep neural networks for emotion recognition. IEEE Access 6:53930–53943

    Article  Google Scholar 

  • Gan Y, Chen J, Xu L (2019) Facial expression recognition boosted by soft label with a diverse ensemble. Pattern Recogn Lett 125:105–112

    Article  Google Scholar 

  • Gao L, Zhang R, Qi L, Chen E, Guan L (2019) The labeled multiple canonical correlation analysis for information fusion. IEEE Trans Multimedia 21(2):375–387

    Article  Google Scholar 

  • Ghosh S, Shivakumara P, Roy P, Pal U, Lu T (2020) Graphology based handwritten character analysis for human behaviour identification. CAAI Trans Intell Technol 5(1):55–65

    Article  Google Scholar 

  • Gupta B, Tiwari M, Lamba SS (2019) Visibility improvement and mass segmentation of mammogram images using quantile separated histogram equalisation with local contrast enhancement. CAAI Trans Intell Technol 4(2):73–79

    Article  Google Scholar 

  • Gupta A, Arunachalam S, Balakrishnan R (2020) Deep self-attention network for facial emotion recognition. Proc Comput Sci 171:1527–1534, third international conference on computing and network communications (CoCoNet’19)

  • He K, Sun J, Tang X (2013) Guided image filtering. IEEE Trans Pattern Anal Mach Intell 35:1397–1409. https://doi.org/10.1109/TPAMI.2012.213

    Article  Google Scholar 

  • Hu G, Chen S-HK, Mazur N (2021) Deep neural network-based speaker-aware information logging for augmentative and alternative communication. J Art Intell Technol 1(2):138–143

    Google Scholar 

  • Hua W, Dai F, Huang L, Xiong J, Gui G (2019) Hero: Human emotions recognition for realizing intelligent internet of things. IEEE Access 7:24321–24332

    Article  Google Scholar 

  • Hung JC, Chang J-W (2021) Multi-level transfer learning for improving the performance of deep neural networks: theory and practice from the tasks of facial emotion recognition and named entity recognition. Appl Soft Comput 109:107491

    Article  Google Scholar 

  • Jain DK, Shamsolmoali P, Sehdev P (2019) Extended deep neural network for facial emotion recognition. Pattern Recogn Lett 120:69–74

    Article  Google Scholar 

  • Ji Y, Hu Y, Yang Y, Shen F, Shen HT (2019) Cross-domain facial expression recognition via an intra-category common feature and inter-category distinction feature fusion network. Neurocomputing 333:231–239

    Article  Google Scholar 

  • Jiang P, Liu G, Wang Q, Wu J (2020) Accurate and reliable facial expression recognition using advanced softmax loss with fixed weights. IEEE Signal Process Lett 27:725–729

    Article  Google Scholar 

  • Jiang D, Hu G, Qi G, Mazur N (2021) A fully convolutional neural network-based regression approach for effective chemical composition analysis using near-infrared spectroscopy in cloud. J Art Intell Technol 1(1):74–82

    Google Scholar 

  • Jung S-W (2012) Enhancement of image and depth map using adaptive joint trilateral filter. IEEE Trans Circuits Syst Video Technol 23(2):258–269

    Article  Google Scholar 

  • Kar NB, Babu KS, Sangaiah AK, Bakshi S (2019) Face expression recognition system based on ripplet transform type ii and least square svm. Multimed Tools Appl 78(4):4789–4812

    Article  Google Scholar 

  • Kim J, Kim B, Roy PP, Jeong D (2019) Efficient facial expression recognition algorithm based on hierarchical deep neural network structure. IEEE Access 7:41273–41285

    Article  Google Scholar 

  • Kim J-H, Kim B-G, Roy PP, Jeong D-M (2019) Efficient facial expression recognition algorithm based on hierarchical deep neural network structure. IEEE Access 7:41273–41285

    Article  Google Scholar 

  • Kingma DP, Ba JL (2015) Adam: a method for stochastic optimization. In: Published as a conference paper at ICLR

  • Kumari N, Bhatia R (2020) Comparative study and analysis of various facial emotion recognition techniques

  • Kumari N, Rekha B (2021) A brief overview of facial emotion recognition system. In: Application of AI and machine learning, pp 97–102

  • Lakshmi D, Ponnusamy R (2021) Facial emotion recognition using modified hog and lbp features with deep stacked autoencoders. Microprocess Microsyst 82:103834

    Article  Google Scholar 

  • Lee C-C, Mower E, Busso C, Lee S, Narayanan S (2011) Emotion recognition using a hierarchical binary decision tree approach. Speech Commun 53(9–10):1162–1171

    Article  Google Scholar 

  • Li S, Deng W (2019) Reliable crowdsourcing and deep locality-preserving learning for unconstrained facial expression recognition. IEEE Trans Image Process 28(1):356–370

    Article  MathSciNet  MATH  Google Scholar 

  • Li B, Lima D (2021) Facial expression recognition via resnet-50. Int J Cogn Comput Eng 2:57–64

    Google Scholar 

  • Li D, Wang Z, Wang C, Liu S, Chi W, Dong E, Song X, Gao Q, Song Y (2019) The fusion of electroencephalography and facial expression for continuous emotion recognition. IEEE Access 7:155724–155736

    Article  Google Scholar 

  • Li TS, Kuo P, Tsai T, Luan P (2019) Cnn and lstm based facial expression analysis model for a humanoid robot. IEEE Access 7:93998–94011

    Article  Google Scholar 

  • Li J, Jin K, Zhou D, Kubota N, Ju Z (2020) Attention mechanism-based cnn for facial expression recognition. Neurocomputing 411:340–350

    Article  Google Scholar 

  • Liu Y, Fu G (2021) Emotion recognition by deeply learned multi-channel textual and eeg features. Futur Gener Comput Syst 119:1–6

    Article  Google Scholar 

  • Lo K-H, Wang Y-CF, Hua K-L (2013) Joint trilateral filtering for depth map super-resolution. Vis Commun Image Process 2013:1–6. https://doi.org/10.1109/VCIP.2013.6706444

    Article  Google Scholar 

  • Lucey P, Cohn JF, Kanade T, Saragih J, Ambadar Z, Matthews I, The extended cohn-kanade dataset (ck+): a complete dataset for action unit and emotion-specified expression. In: IEEE computer society conference on computer vision and pattern recognition-workshops. IEEE 2010:94–101

  • Mehendale N (2020) Facial emotion recognition using convolutional neural networks (ferc). SN Appl Sci 2(3):1–8

    Article  Google Scholar 

  • Muhammad G, Hossain MS. Emotion recognition for cognitive edge computing using deep learning. IEEE Int Things J

  • Ngai WK, Xie H, Zou D, Chou K-L (2022) Emotion recognition based on convolutional neural networks and heterogeneous bio-signal data sources. Inf Fusion 77:107–117

    Article  Google Scholar 

  • Otberdout N, Kacem A, Daoudi M, Ballihi L, Berretti S (2019) Automatic analysis of facial expressions based on deep covariance trajectories. IEEE Trans Neural Netw Learn Syst 31(10):3892–3905

    Article  Google Scholar 

  • Pu X, Fan K, Chen X, Ji L, Zhou Z (2015) Facial expression recognition from image sequences using twofold random forest classifier. Neurocomputing 168:1173–1180

    Article  Google Scholar 

  • Reza AM (2004) Realization of the contrast limited adaptive histogram equalization (clahe) for real-time image enhancement. J VLSI Signal Process Syst Signal Image Video Technol 38(1):35–44

    Article  Google Scholar 

  • Singh D, Kumar V (2017) Dehazing of remote sensing images using improved restoration model based dark channel prior. Imag Sci J 65(5):282–292

    Article  Google Scholar 

  • Singh D, Kumar V (2018) Dehazing of remote sensing images using fourth-order partial differential equations based trilateral filter. IET Comput Vision 12(2):208–219

    Article  Google Scholar 

  • Singh D, Kumar V (2018) Defogging of road images using gain coefficient-based trilateral filter. J Electron Imaging 27(1):013004

    Article  Google Scholar 

  • Sun L, Fu S, Wang F (2019) Decision tree svm model with fisher feature selection for speech emotion recognition. EURASIP J Audio Speech Music Proc 2019 (1) 2

  • Tan Y, Sun Z, Duan F, Solé-Casals J, Caiafa CF (2021) A multimodal emotion recognition method based on facial expressions and electroencephalography. Biomed Signal Process Control 70:103029

    Article  Google Scholar 

  • Tieleman T, Hinton G Adaptive subgradient methods for online learning and stochastic optimization, COURSERA: Neural Networks for Machine Learning. Technical report

  • Tong X, Sun S, Fu M (2019) Data augmentation and second-order pooling for facial expression recognition. IEEE Access 7:86821–86828

    Article  Google Scholar 

  • Valstar M, Gratch J, Schuller B, Ringeval F, Lalanne D, Torres Torres M, Scherer S, Stratou G, Cowie R, Pantic M (2016) Avec 2016: Depression, mood, and emotion recognition workshop and challenge. In: Proceedings of the 6th international workshop on audio/visual emotion challenge, pp 3–10

  • Varma S, Shinde M, Chavan S. S Analysis of pca and lda features for facial expression recognition using svm and hmm classifiers. In: Techno-Societal 2018, Springer, 2020, pp 109–119

  • Vijaya Lakshmi A, Mohanaiah P (2021) Woa-tlbo: Whale optimization algorithm with teaching-learning-based optimization for global optimization and facial emotion recognition. Appl Soft Comput 110:107623

  • Wang X, Chen X, Cao C (2020) Human emotion recognition by optimally fusing facial expression and speech feature. Signal Process Image Commun 84:115831

    Article  Google Scholar 

  • Wang W, Sun Q, Chen T, Cao C, Zheng Z, Xu G, Qiu H, Fu Y A fine-grained facial expression database for end-to-end multi-pose facial expression recognition, arXiv preprint arXiv:1907.10838

  • Xiang ZNCXWZL (2016) Xuezhi. A modified joint trilateral filter based depth map refinement method, Yan, pp 1403–1407

  • Xu Y, Qiu TT (2021) Human activity recognition and embedded application based on convolutional neural network. J Art Intell Technol 1(1):51–60

    Google Scholar 

  • Yin Y, Zheng X, Hu B, Zhang Y, Cui X (2021) Eeg emotion recognition using fusion model of graph convolutional neural networks and lstm. Appl Soft Comput 100:106954

    Article  Google Scholar 

  • Zakraoui J, Elloumi S, Alja’am JM, Ben Yahia S (2019) Improving arabic text to image mapping using a robust machine learning technique. IEEE Access 7:18772–18782

    Article  Google Scholar 

  • Zhang T, Zheng W, Cui Z, Zong Y, Yan J, Yan K (2016) A deep neural network-driven feature learning method for multi-view facial expression recognition. IEEE Trans Multimed 18(12):2528–2536

    Article  Google Scholar 

  • Zhang H, Jolfaei A, Alazab M (2019) A face emotion recognition method using convolutional neural network and image edge computing. IEEE Access 7:159081–159089

  • Zhang T, Zheng W, Cui Z, Zong Y, Li Y (2019) Spatial-temporal recurrent neural network for emotion recognition. IEEE Trans Cybern 49(3):839–847

  • Zhang S, Pan X, Cui Y, Zhao X, Liu L (2019) Learning affective video features for facial expression recognition via hybrid deep learning. IEEE Access 7:32297–32304

    Article  Google Scholar 

  • Zhang Z, Lai C, Liu H, Li Y-F (2020) Infrared facial expression recognition via gaussian-based label distribution learning in the dark illumination environment for human emotion detection. Neurocomputing 409:341–350

    Article  Google Scholar 

Download references

Funding

This research received no specific grant from any funding agency.

Author information

Authors and Affiliations

  1. Department of Computer Science, Punjabi University Regional Centre for IT and Management, Mohali, India

    Naveen Kumari & Rekha Bhatia

Authors
  1. Naveen Kumari
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Rekha Bhatia
    View author publications

    You can also search for this author in PubMed Google Scholar

Ethics declarations

Conflict of interest

The authors declare no competitive interest regarding the publication of this paper.

Additional information

Communicated by Irfan Uddin.

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Kumari, N., Bhatia, R. Efficient facial emotion recognition model using deep convolutional neural network and modified joint trilateral filter. Soft Comput 26, 7817–7830 (2022). https://doi.org/10.1007/s00500-022-06804-7

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00500-022-06804-7

Keywords

Search

Navigation