Skip to main content
SpringerLink
Log in

Facial expression recognition with trade-offs between data augmentation and deep learning features

  • Original Research
  • Published:
Journal of Ambient Intelligence and Humanized Computing Aims and scope Submit manuscript

Abstract

A novel facial expression recognition system has been proposed in this paper. The objective of this paper is to recognize the types of expressions in the human face region. The implementation of the proposed system has been divided into four components. In the first component, a region of interest as face detection has been performed from the captured input image. For extracting more distinctive and discriminant features, in the second component, a deep learning-based convolutional neural network architecture has been proposed to perform feature learning tasks for classification purposes to recognize the types of expressions. To enhance the performance of the proposed system, in the third component, some novel data augmentation techniques have been applied to the facial image to enrich the learning parameters of the proposed CNN model. In the fourth component, a trade-off between data augmentation and deep learning features have been performed for fine-tuning the trained CNN model. Extensive experimental results have been demonstrated using three benchmark databases: KDEF (seven expression classes), GENKI-4k (two expression classes), and CK+ (seven expression classes). The performance of the proposed system respect for each database has been well presented and described and finally, these performances have been compared with the existing state-of-the-art methods. The comparison with competing methods shows the superiority of the proposed system.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16
Fig. 17
Fig. 18
Fig. 19

Similar content being viewed by others

References

  • Abate AF, Barra P, Bisogni C, Nappi M, Ricciardi S (2019) Near real-time three axis head pose estimation without training. IEEE Access 7:64256–64265

    Article  Google Scholar 

  • Alenazy WM, Alqahtani AS (2020) Gravitational search algorithm based optimized deep learning model with diverse set of features for facial expression recognition. J Ambient Intell Human Comput. https://doi.org/10.1007/s12652-020-02235-0

    Article  Google Scholar 

  • An L, Yang S, Bhanu B (2015) Efficient smile detection by extreme learning machine. Neurocomputing 149:354–363

    Article  Google Scholar 

  • Asano T, Bitou S, Motoki M, Usui N (2007) In-place algorithm for image rotation. In: International symposium on algorithms and computation. Springer, pp 704–715

  • Barra P, Barra S, Bisogni C, De Marsico M, Nappi M (2020) Web-shaped model for head pose estimation: An approach for best exemplar selection. IEEE Trans Image Process 29:5457–5468

    Article  Google Scholar 

  • Battiato S, Gallo G, Stanco F (2002) A locally adaptive zooming algorithm for digital images. Image Vis Comput 20(11):805–812

    Article  Google Scholar 

  • Branson S, Wah C, Schroff F, Babenko B, Welinder P, Perona P, Belongie S (2010) Visual recognition with humans in the loop. In: European conference on computer vision. Springer, pp 438–451

  • Castrillón-Santana M, De Marsico M, Nappi M, Riccio D (2017) Meg: texture operators for multi-expert gender classification. Comput Vis Image Underst 156:4–18

    Article  Google Scholar 

  • Chollet F (2015) Keras: Deep learning library for theano and tensorflow. https://keras.io/

  • De Marsico M, Nappi M, Riccio D, Wechsler H (2012) Robust face recognition for uncontrolled pose and illumination changes. IEEE Trans Syst Man Cybern Syst 43(1):149–163

    Article  Google Scholar 

  • De Queiroz RL (2000) On data filling algorithms for MRC layers. In: Proceedings 2000 international conference on image processing (Cat. No. 00CH37101), vol 2. IEEE, pp 586–589

  • Ekman P, Friesen WV (1971) Constants across cultures in the face and emotion. J Personal Soc Psychol 17(2):124

    Article  Google Scholar 

  • Fan Xijian, Tjahjadi Tardi (2019) Fusing dynamic deep learned features and handcrafted features for facial expression recognition. J Vis Commun Image Represent 65:102659

    Article  Google Scholar 

  • Friesen E, Ekman P (1978) Facial action coding system: a technique for the measurement of facial movement. Palo Alto 3

  • Gao Y, Liu H, Pingping W, Wang C (2016) A new descriptor of gradients self-similarity for smile detection in unconstrained scenarios. Neurocomputing 174:1077–1086

    Article  Google Scholar 

  • Hernández-García A, König P (2018) Further advantages of data augmentation on convolutional neural networks. In: International conference on artificial neural networks. Springer, pp 95–103

  • Huang G, Liu Z, Der Maaten LV, Weinberger KQ (2017) Densely connected convolutional networks. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 4700–4708

  • Iliyasu AM, Le PQ, Dong F, Hirota K (2012) Watermarking and authentication of quantum images based on restricted geometric transformations. Inf Sci 186(1):126–149

    Article  MathSciNet  Google Scholar 

  • Ioffe Sergey (2017) Batch renormalization: Towards reducing minibatch dependence in batch-normalized models. In: Advances in neural information processing systems, pp 1945–1953

  • Ioffe S, Szegedy C (2015) Batch normalization: accelerating deep network training by reducing internal covariate shift. arXiv:1502.03167

  • Jaimes Alejandro, Sebe Nicu (2007) Multimodal human–computer interaction: a survey. Comput Vis Image Underst 108(1–2):116–134

    Article  Google Scholar 

  • Jain V, Crowley JL (2013) Smile detection using multi-scale gaussian derivatives

  • Ji Y, Hu Y, Yang Y, Shen F, Shen HT (2019) Cross-domain facial expression recognition via an intra-category common feature and inter-category distinction feature fusion network. Neurocomputing 333:231–239

    Article  Google Scholar 

  • Khan S, Rahmani H, Shah SAA, Bennamoun M (2018) A guide to convolutional neural networks for computer vision. Synth Lect Comput Vis 8(1):1–207

    Article  Google Scholar 

  • Ko BC (2018) A brief review of facial emotion recognition based on visual information. Sensors 18(2):401

    Article  Google Scholar 

  • Lai Z, Chen R, Jia J, Qian Y (2020) Real-time micro-expression recognition based on resnet and atrous convolutions. J Ambient Intell Humaniz Comput 1–12

  • Lee K, Lee EC (2019) Comparison of facial expression recognition performance according to the use of depth information of structured-light type RGB-D camera. J Ambient Intell Humaniz Comput 1–17

  • Lucey P, Cohn JF, Kanade T, Saragih J, Ambadar Z, Matthews I (2010) The extended Cohn–Canade dataset (ck+): a complete dataset for action unit and emotion-specified expression. In: 2010 IEEE computer society conference on computer vision and pattern recognition-workshops. IEEE, pp 94–101

  • Lundqvist D, Flykt A, Öhman A (1998) The karolinska directed emotional faces (kdef). CD ROM from Department of Clinical Neuroscience, Psychology section, Karolinska Institutet, vol 91, no 630, p 2

  • Maheswari VU, Varaprasad G, Viswanadha RS (2020) Local directional maximum edge patterns for facial expression recognition. J Ambient Intell Humaniz Comput 1–9

  • Makhmudkhujaev F, Abdullah-Al-Wadud M, Iqbal MTB, Ryu B, Chae O (2019) Facial expression recognition with local prominent directional pattern. Signal Process Image Commun 74:1–12

    Article  Google Scholar 

  • Meshach WT, Hemajothi S, Anita EAM (2020) Real-time facial expression recognition for affect identification using multi-dimensional SVM. J Ambient Intell Humaniz Comput 1–11

  • Mollahosseini A, Chan D, Mahoor MH (2016) Going deeper in facial expression recognition using deep neural networks. In: 2016 IEEE winter conference on applications of computer vision (WACV), IEEE, pp 1–10

  • Pardo-Igúzquiza E, Chica-Olmo M, Atkinson PM (2006) Downscaling cokriging for image sharpening. Remote Sens Environ 102(1–2):86–98

    Article  Google Scholar 

  • Paris Sylvain, Kornprobst Pierre, Tumblin Jack, Frédo D (2009) Theory and applications. Bilateral filtering. Now Publishers Inc., Norwell

    MATH  Google Scholar 

  • Perez L, Wang J (2017) The effectiveness of data augmentation in image classification using deep learning. arXiv:1712.04621

  • Polesel A, Ramponi G, Mathews VJ (2000) Image enhancement via adaptive unsharp masking. IEEE Trans Image Process 9(3):505–510

    Article  Google Scholar 

  • Proenca H, Neves JC, Barra S, Marques T, Moreno JC (2016) Joint head pose/soft label estimation for human recognition in-the-wild. IEEE Trans Pattern Anal Mach Intell 38(12):2444–2456

    Article  Google Scholar 

  • Rao Q, Qu X, Mao Q, Zhan Y (2015) Multi-pose facial expression recognition based on surf boosting. In: 2015 international conference on affective computing and intelligent interaction (ACII). IEEE, pp 630–635

  • Renda A, Barsacchi M, Bechini A, Marcelloni F (2019) Comparing ensemble strategies for deep learning: an application to facial expression recognition. Expert Syst Appl 136:1–11

    Article  Google Scholar 

  • Sadeghi H, Raie AA (2019) Histogram distance metric learning for facial expression recognition. J Vis Commun Image Represent 62:152–165

    Article  Google Scholar 

  • Sandbach G, Zafeiriou S, Pantic M, Yin L (2012) Static and dynamic 3d facial expression recognition: a comprehensive survey. Image Vis Comput 30(10):683–697

    Article  Google Scholar 

  • Simonyan K, Zisserman A (2014) Very deep convolutional networks for large-scale image recognition. arXiv:1409.1556

  • Srivastava Nitish, Hinton Geoffrey, Krizhevsky Alex, Sutskever Ilya, Salakhutdinov Ruslan (2014) Dropout: a simple way to prevent neural networks from overfitting. J Mach Learn Res 15(1):1929–1958

    MathSciNet  MATH  Google Scholar 

  • Sun Xiao, Xia Pingping, Zhang Luming, Shao Ling (2020) A ROI-guided deep architecture for robust facial expressions recognition. Inf Sci 522:35–48

    Article  Google Scholar 

  • Sun Zhe, Zheng-Ping Hu, Wang Meng, Zhao Shu-Huan (2017) Discriminative feature learning-based pixel difference representation for facial expression recognition. IET Comput Vis 11(8):675–682

    Article  Google Scholar 

  • Szegedy C, Vanhoucke V, Ioffe S, Shlens J, Wojna Z (2016) Rethinking the inception architecture for computer vision. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 2818–2826

  • Szegedy C, Ioffe S, Vanhoucke V, Alemi AA (2017) Inception-v4, inception-resnet and the impact of residual connections on learning. In: Thirty-first AAAI conference on artificial intelligence

  • Tanter Mickaël, Touboul David, Gennisson Jean-Luc, Bercoff Jeremy, Fink Mathias (2009) High-resolution quantitative imaging of cornea elasticity using supersonic shear imaging. IEEE Trans Med Imaging 28(12):1881–1893

    Article  Google Scholar 

  • Tao J, Tan T (2005) Affective computing: a review. In: International conference on affective computing and intelligent interaction. Springer, pp 981–995

  • Targ S, Almeida D, Lyman K (2016) Resnet in resnet: generalizing residual architectures. arXiv:1603.08029

  • Umer S, Dhara BC, Chanda B (2019) Face recognition using fusion of feature learning techniques. Measurement 146:43–54

    Article  Google Scholar 

  • Vedaldi A, Zisserman A (2016) VGG convolutional neural networks practical. Department of Engineering Science, University of Oxford, p 66

  • Wu R, Yan S, Yi S, Dang Q, Sun G (2015) Deep image: scaling up image recognition 7(8). arXiv:1501.02876

  • Xu B, Wang N, Chen T, Li M (2015) Empirical evaluation of rectified activations in convolutional network. arXiv:1505.00853

  • Yan Yan, Zhang Zizhao, Chen Si, Wang Hanzi (2020) Low-resolution facial expression recognition: a filter learning perspective. Signal Process 169:107370

    Article  Google Scholar 

  • Ye Yingsheng, Zhang Xingming, Lin Yubei, Wang Haoxiang (2019) Facial expression recognition via region-based convolutional fusion network. J Vis Commun Image Represent 62:1–11

    Article  Google Scholar 

  • Mingjing Yu, Zheng Huicheng, Peng Zhifeng, Dong Jiayu, Heran Du (2020) Facial expression recognition based on a multi-task global-local network. Pattern Recognit Lett 131:166–171

    Article  Google Scholar 

  • Zavarez MV, Berriel RF, Oliveira-Santos T (2017) Cross-database facial expression recognition based on fine-tuned deep convolutional network. In: 2017 30th SIBGRAPI conference on graphics, patterns and images (SIBGRAPI). IEEE, pp 405–412

  • Zhang Hepeng, Huang Bin, Tian Guohui (2020) Facial expression recognition based on deep convolution long short-term memory networks of double-channel weighted mixture. Pattern Recognit Lett 131:128–134

    Article  Google Scholar 

  • Zhang Kaihao, Huang Yongzhen, Wu Hong, Wang Liang (2015) Facial smile detection based on deep learning features. In: 2015 3rd IAPR Asian conference on pattern recognition (ACPR). IEEE, pp 534–538

  • Zhu X, Ramanan D (2012) Face detection, pose estimation, and landmark localization in the wild. In: 2012 IEEE conference on computer vision and pattern recognition. IEEE, pp 2879–2886

Download references

Author information

Authors and Affiliations

  1. Department of Computer Science and Engineering, Aliah University, Kolkata, India

    Saiyed Umer

  2. Department of Computer Science and Engineering, National Institute of Technology, Srinagar, India

    Ranjeet Kumar Rout

  3. Department of Computer Science, University of Salerno, Fisciano, Italy

    Chiara Pero & Michele Nappi

Authors
  1. Saiyed Umer
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Ranjeet Kumar Rout
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Chiara Pero
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Michele Nappi
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Chiara Pero.

Ethics declarations

Conflict of interest

The authors declare no conflict of interest. The funding agency had no role in the design of the study; in the collection, analyses, or interpretation of data; in the writing of the manuscript, or in the decision to publish the results.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Umer, S., Rout, R.K., Pero, C. et al. Facial expression recognition with trade-offs between data augmentation and deep learning features. J Ambient Intell Human Comput 13, 721–735 (2022). https://doi.org/10.1007/s12652-020-02845-8

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s12652-020-02845-8

Keywords

Search

Navigation