Skip to main content
SpringerLink
Log in

Static facial expression recognition using convolutional neural networks based on transfer learning and hyperparameter optimization

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

Expression recognition (ER), which has been frequently used in human-computer interaction, uses visual data such as video and static images or sensor-based data for recognizing. Facial expression recognition (FER) is a visual data based ER. Since videos have sequential images, it can be easier to recognize emotion in video signals rather than static images which consist of a single plain image. Therefore, FER on static images is a relatively tough task. Recently, deep learning methods have introduced increased success in classification problems. Accordingly, these methods are also used for FER in the literature. Data preparation and hyperparameter optimization can be utilized to increase the success of deep learning methods. With the preparation of data, the features become more pronounced. Increasing the number of training samples directly also generally affects the success rate. Tuning the hyperparameters of deep learning is another factor that increases the performance of the models. In this study, a classification method including data preparation, hyperparameter optimization, and a transfer learning aided convolutional neural network is proposed. Through the study, a new dataset, named ERUFER, was created by using static images. The newly introduced dataset ERUFER and a popular public dataset JAFFE were classified by the proposed method. To the extent of our knowledge, the best result in the literature is achieved by the proposed method for the JAFFE dataset using a 10-fold cross-validation test technique. On the other hand, a success rate with 92.56 % is achieved for the ERUFER dataset.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9

Similar content being viewed by others

References

  1. Aghamaleki JA, Ashkani Chenarlogh V (2019) Multi-stream cnn for facial expression recognition in limited training data. Multimed Tools Appl 78 (16):22,861–22,882. https://doi.org/10.1007/s11042-019-7530-7

    Article  Google Scholar 

  2. Alghamdi A, Hammad M, Ugail H, Abdel-Raheem A, Muhammad K, Khalifa HS, Abd El-Latif AA (2020) Detection of myocardial infarction based on novel deep transfer learning methods for urban healthcare in smart cities. Multimed Tools Appl 1–22

  3. Alghamdi AS, Polat K, Alghoson A, Alshdadi AA, Abd El-Latif AA (2020) A novel blood pressure estimation method based on the classification of oscillometric waveforms using machine-learning methods. Appl Acoust 164 (107):279

    Google Scholar 

  4. Araújo T, Aresta G, Castro E, Rouco J, Aguiar P, Eloy C, Polónia A, Campilho A (2017) Classification of breast cancer histology images using convolutional neural networks. PloS one 12(6):e0177,544

    Article  Google Scholar 

  5. Badem H, Basturk A, Caliskan A, Yuksel ME (2017) A new efficient training strategy for deep neural networks by hybridization of artificial bee colony and limited memory BFGS optimization algorithms. Neurocomputing 266:506–526

    Article  Google Scholar 

  6. Badem H, Basturk A, Caliskan A, Yuksel ME (2018) A new hybrid optimization method combining artificial bee colony and limited-memory BFGS algorithms for efficient numerical optimization. Appl Soft Comput 70:826–844. https://doi.org/10.1016/j.asoc.2018.06.010, http://www.sciencedirect.com/science/article/pii/S1568494618303338

    Article  Google Scholar 

  7. Basturk A, Basturk NS, Qurbanov O (2018) Fingerprint recognition by deep neural networks and fingercodes. In: 2018 26th signal processing and communications applications conference (SIU), pp 1–4. https://doi.org/10.1109/SIU.2018.8404577

  8. Basturk A, Sarikaya Basturk N, Qurbanov O (2018) A comparative performance analysis of various classifiers for fingerprint recognition. Nöhü Müh Bilim Derg 7:504–513

    Google Scholar 

  9. Cai J, Meng Z, Khan A, Li Z, O’Reilly J, Tong Y (2017) Island loss for learning discriminative features in facial expression recognition. arXiv:1710.03144

  10. Caliskan A, Badem H, Basturk A, Yuksel ME (2017) Diagnosis of the parkinson disease by using deep neural network classifier. Istanbul University-J Electr Electron Eng 17(2):3311–3318

    Google Scholar 

  11. Caliskan A, Yuksel ME, Badem H, Basturk A (2018) Performance improvement of deep neural network classifiers by a simple training strategy. Eng Appl Artif Intell 67:14–23

    Article  Google Scholar 

  12. Chang T, Li H, Wen G, Hu Y, Ma J (2019) Facial expression recognition sensing the complexity of testing samples. Appl Intell. https://doi.org/10.1007/s10489-019-01491-8

  13. Devries T, Biswaranjan K, Taylor GW (2014) Multi-task learning of facial landmarks and expression. In: 2014 Canadian conference on computer and robot vision, pp 98–103

  14. Dhall A, Goecke R, Lucey S, Gedeon T (2011) Static facial expression analysis in tough conditions: data, evaluation protocol and benchmark. In: 2011 IEEE international conference on computer vision workshops (ICCV workshops), pp 2106–2112

  15. Ding H, Zhou SK, Chellappa R (2016) Facenet2expnet: regularizing a deep face recognition net for expression recognition. arXiv:1609.06591

  16. Dong Y, Zhang Z, Hong WC (2018) A hybrid seasonal mechanism with a chaotic cuckoo search algorithm with a support vector regression model for electric load forecasting. Energies 11(4):1009

    Article  Google Scholar 

  17. Gogić I, Manhart M, Pandžić IS, Ahlberg J (2020) Fast facial expression recognition using local binary features and shallow neural networks. Vis Comput 36(1):97–112

    Article  Google Scholar 

  18. Goodfellow IJ, Erhan D, Carrier PL, Courville A, Mirza M, Hamner B, Cukierski W, Tang Y, Thaler D, Lee DH, Zhou Y, Ramaiah C, Feng F, Li R, Wang X, Athanasakis D, Shawe-Taylor J, Milakov M, Park J, Ionescu R, Popescu M, Grozea C, Bergstra J, Xie J, Romaszko L, Xu B, Chuang Z, Bengio Y (2015) Challenges in representation learning: a report on three machine learning contests. Neural Netw 64:59–63. Special Issue on “Deep Learning of Representations”

    Article  Google Scholar 

  19. Guo Y, Tao D, Yu J, Xiong H, Li Y, Tao D (2016) Deep neural networks with relativity learning for facial expression recognition. In: 2016 IEEE international conference on multimedia & expo workshops (ICMEW). IEEE, pp 1–6

  20. Hamester D, Barros P, Wermter S (2015) Face expression recognition with a 2-channel convolutional neural network. In: 2015 international joint conference on neural networks (IJCNN), pp 1–8

  21. He K, Zhang X, Ren S, Sun J (2016) Deep residual learning for image recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 770–778

  22. Hong WC, Dong Y, Lai CY, Chen LY, Wei SY (2011) Svr with hybrid chaotic immune algorithm for seasonal load demand forecasting. Energies 4(6):960–977

    Article  Google Scholar 

  23. Hong WC, Li MW, Geng J, Zhang Y (2019) Novel chaotic bat algorithm for forecasting complex motion of floating platforms. Appl Math Model 72:425–443

    Article  MathSciNet  Google Scholar 

  24. Jiang M, Liang Y, Feng X, Fan X, Pei Z, Xue Y, Guan R (2018) Text classification based on deep belief network and softmax regression. Neural Comput & Applic 29(1):61–70

    Article  Google Scholar 

  25. Kennedy J, Eberhart R (1995) Particle swarm optimization. In: International conference on neural networks, pp 1942–1948

  26. Khorrami P, Paine TL, Huang TS (2015) Do deep neural networks learn facial action units when doing expression recognition? arXiv:1510.02969

  27. Kim B, Dong S, Roh J, Kim G, Lee S (2016) Fusing aligned and non-aligned face information for automatic affect recognition in the wild: a deep learning approach. In: 2016 IEEE conference on computer vision and pattern recognition workshops (CVPRW), pp 1499–1508

  28. Kim BK, Lee H, Roh J, Lee SY (2015) Hierarchical committee of deep cnns with exponentially-weighted decision fusion for static facial expression recognition. In: Proceedings of the 2015 ACM on international conference on multimodal interaction, ICMI ’15, pp. 427–434. ACM, New York, NY, USA

  29. Kim T, Yu C, Lee S (2018) Facial expression recognition using feature additive pooling and progressive fine-tuning of cnn. Electron Lett 54(23):1326–1328

    Article  Google Scholar 

  30. King DE (2009) Dlib-ml: a machine learning toolkit. J Mach Learn Res 10:1755–1758

    Google Scholar 

  31. Krizhevsky A, Sutskever I, Hinton GE (2012) Imagenet classification with deep convolutional neural networks. In: Advances in neural information processing systems, pp 1097–1105

  32. Kundra H, Sadawarti H (2015) Hybrid algorithm of cuckoo search and particle swarm optimization for natural terrain feature extraction. Res J Inf Technol 7(1):58–69

    Google Scholar 

  33. Langner O, Dotsch R, Bijlstra G, Wigboldus DHJ, Hawk ST, van Knippenberg A (2010) Presentation and validation of the radboud faces database. Cogn Emot 24(8):1377–1388

    Article  Google Scholar 

  34. Li D, Wen G, Li X, Cai X (2019) Graph-based dynamic ensemble pruning for facial expression recognition. Appl Intell 49(9):3188–3206. https://doi.org/10.1007/s10489-019-01435-2

    Article  Google Scholar 

  35. Li S, Deng W (2018) Deep facial expression recognition: a survey. arXiv:1804.08348

  36. Liu P, Han S, Meng Z, Tong Y (2014) Facial expression recognition via a boosted deep belief network. In: 2014 IEEE conference on computer vision and pattern recognition, pp 1805–1812

  37. Lundqvist D, Flykt A, Ohman A (1998) The karolinska directed emotional faces – KDEF. In: CD ROM from department of clinical neuroscience, psychology section, ISBN 91-630-7164-9

  38. Luo Y, Liu XY, Zhang Y, Chen XF, Chen Z (2019) Facial expression recognition based on improved completed local ternary patterns. Optoelectron Lett 15(3):224–230

    Article  Google Scholar 

  39. Lyons M, Akamatsu S, Kamachi M, Gyoba J (1998) Coding facial expressions with gabor wavelets. In: Proceedings third IEEE International conference on automatic face and gesture recognition, pp 200–205

  40. Mathworks: Documentation. https://www.mathworks.com/help. Online; accessed 18 August 2019

  41. Matsumoto D (1992) More evidence for the universality of a contempt expression. Motiv Emot 16(4):363–368

    Article  Google Scholar 

  42. Meng Z, Liu P, Cai J, Han S, Tong Y (2017) Identity-aware convolutional neural network for facial expression recognition. In: 2017 12th IEEE international conference on automatic face gesture recognition (FG 2017), pp 558–565

  43. Missinglink.ai Convolutional neural networks. https://missinglink.ai/guides/convolutional-neural-networks/ Online; accessed 20 August 2019

  44. Moeini A, Faez K, Moeini H, Safai AM (2017) Facial expression recognition using dual dictionary learning. J Vis Commun Image Represent 45:20–33

    Article  Google Scholar 

  45. Ng A (2018) Machine learning yearning. deeplearning.ai

  46. Ngo TA, Carneiro G (2013) Left ventricle segmentation from cardiac mri combining level set methods with deep belief networks. In: 2013 IEEE international conference on image processing. IEEE, pp 695–699

  47. Ondruska P, Posner I (2016) Deep tracking: seeing beyond seeing using recurrent neural networks. In: Thirtieth AAAI conference on artificial intelligence

  48. Ozcan T, Basturk A (2019) Lip reading using convolutional neural networks with and without pre-trained models. Balkan J Electr Comput Eng 7(2):195–201

    Article  Google Scholar 

  49. Ozcan T, Basturk A (2019) Static image-based emotion recognition using convolutional neural network. In: 2019 27th signal processing and communications applications conference (SIU), pp 1–4. https://doi.org/10.1109/SIU.2019.8806408

  50. Ozcan T, Basturk A (2019) Transfer learning-based convolutional neural networks with heuristic optimization for hand gesture recognition. Neural Comput Applic 31(12):8955–8970

    Article  Google Scholar 

  51. Ozcan T, Basturk A (2020) Human action recognition with deep learning and structural optimization using a hybrid heuristic algorithm. Clust Comput 1–14

  52. Ozcan T, Basturk A (2020) Performance improvement of pretrained convolutional neural networks for action recognition. Comput J 1–12

  53. Poursaberi A, Noubari HA, Gavrilova M, Yanushkevich SN (2012) Gauss–laguerre wavelet textural feature fusion with geometrical information for facial expression identification. EURASIP J Image Vide 2012(1):17

    Article  Google Scholar 

  54. Pramerdorfer C, Kampel M (2016) Facial expression recognition using convolutional neural networks: state of the art. arXiv:1612.02903

  55. Saeed F, Paul A, Karthigaikumar P, Nayyar A (2019) Convolutional neural network based early fire detection. Multimed Tools Appl. https://doi.org/10.1007/s11042-019-07785-w

  56. Schmidt EM, Kim YE (2011) Learning emotion-based acoustic features with deep belief networks. In: 2011 IEEE workshop on applications of signal processing to audio and acoustics (Waspaa). IEEE, pp 65–68

  57. Sekaran K, Chandana P, Krishna NM, Kadry S (2019) Deep learning convolutional neural network (cnn) with gaussian mixture model for predicting pancreatic cancer. Multimed Tools Appl. https://doi.org/10.1007/s11042-019-7419-5

  58. Sun W, Song Y, Jin Z, Zhao H, Chen C (2019) Unsupervised orthogonal facial representation extraction via image reconstruction with correlation minimization. Neurocomputing 337:203–217

    Article  Google Scholar 

  59. Susskind JM, Anderson AK, Hinton GE (2010) The Toronto face database. Tech. rep., University of Toronto, Department of Computer Science, ON, Canada, vol 3

  60. Tang Y (2013) Deep learning using support vector machines. arXiv:1306.0239

  61. Venkatraman S, Balasubramanian S, Gera D (2017) Multiple face-component analysis: a unified approach towards facial recognition tasks. In: 2017 2nd international conference on man and machine interfacing (MAMI). IEEE, pp 1–6

  62. Wang N, Li Q, El-Latif AAA, Peng J, Niu X (2013) A novel multibiometric template security scheme for the fusion of dual iris, visible and thermal face images. J Comput Inf Syst 9(19):1–9

    Google Scholar 

  63. Wang N, Li Q, El-Latif AAA, Peng J, Niu X (2014) An enhanced thermal face recognition method based on multiscale complex fusion for gabor coefficients. Multimed Tools Appl 72(3):2339–2358

    Article  Google Scholar 

  64. Wang N, Li Q, El-Latif AAA, Yan X, Niu X (2013) A novel hybrid multibiometrics based on the fusion of dual iris, visible and thermal face images. In: 2013 international symposium on biometrics and security technologies. IEEE, pp 217–223

  65. Yildirim MT, Basturk A, Yuksel ME (2007) A detail-preserving type-2 fuzzy logic filter for impulse noise removal from digital images. In: 2007 IEEE international fuzzy systems conference, pp 1–6. https://doi.org/10.1109/FUZZY.2007.4295460

  66. Yu Z, Zhang C (2015) Image based static facial expression recognition with multiple deep network learning. In: Proceedings of the 2015 ACM on international conference on multimodal interaction, ICMI ’15, pp. 435–442, ACM, New York, NY, USA

  67. Yuksel ME, Basturk NS, Badem H, Caliskan A, Basturk A (2018) Classification of high resolution hyperspectral remote sensing data using deep neural networks. J Intell Fuzzy Syst 34:2273–2285

    Article  Google Scholar 

  68. Zavaschi TH, Britto AS, Oliveira LE, Koerich AL (2013) Fusion of feature sets and classifiers for facial expression recognition. Expert Syst Appl 40 (2):646–655

    Article  Google Scholar 

  69. Zhang Z, Hong WC (2019) Electric load forecasting by complete ensemble empirical mode decomposition adaptive noise and support vector regression with quantum-based dragonfly algorithm. Nonlinear Dyn 98(2):1107–1136

    Article  Google Scholar 

  70. Zhang Z, Hong WC, Li J (2020) Electric load forecasting by hybrid self-recurrent support vector regression model with variational mode decomposition and improved cuckoo search algorithm. IEEE Access 8:14,642–14,658

    Article  Google Scholar 

  71. Zhang Z, Luo P, Loy CC, Tang X (2015) Learning social relation traits from face images. arXiv:1509.03936

Download references

Acknowledgements

ERUFER includes so many samples from many subjects. Therefore, we had to need help in collecting data. We would like to thank Nimet Tuhtakaya, Hatice Ozturk, and all the participants involved in the project for their help in collecting data.

Author information

Authors and Affiliations

  1. Department of Computer Engineering, Erciyes University, Melikgazi, 38039, Kayseri, Turkey

    Tayyip Ozcan & Alper Basturk

Authors
  1. Tayyip Ozcan
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Alper Basturk
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Tayyip Ozcan.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Ozcan, T., Basturk, A. Static facial expression recognition using convolutional neural networks based on transfer learning and hyperparameter optimization. Multimed Tools Appl 79, 26587–26604 (2020). https://doi.org/10.1007/s11042-020-09268-9

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-020-09268-9

Keywords

Search

Navigation