Skip to main content
SpringerLink
Log in

DTL-I-ResNet18: facial emotion recognition based on deep transfer learning and improved ResNet18

  • Original Paper
  • Published:
Signal, Image and Video Processing Aims and scope Submit manuscript

Abstract

Human facial emotion recognition (FER) has attracted interest from the scientific community for its prospective uses. The fundamental goal of FER is to match distinct facial expressions to different emotional states. Recent state-of-the-art studies have generally adopted more complex methods to achieve this aim, such as large-scale deep learning models or multi-model analysis referring to multiple sub-models. Unfortunately, performance defacement happens in these approaches because to poor layer selection in the convolutional neural networks (CNN) architecture. To resolve this problem and unlike these models, the present work proposes a Deep CNN-based intelligent computer vision system capable of recognizing facial emotions. To do so, we propose, first, a Deep CNN architecture using Transfer Learning (TL) approach for constructing a highly accurate FER system, in which a pre-trained Deep CNN model is adopted by substituting its dense upper layers suitable with FER, and the model is fine-tuned with facial expression data. Second, we propose improving ResNet18 model due to its highest performance in terms of recognition accuracy compared with the state-of-the-art studies. Then, the improved model is trained and tested on two benchmark datasets, FER2013 and CK+. The improved ResNet18 model achieves FER accuracies of 98% and 83% on CK+ and FER2013 test sets, respectively. The obtained results show that the suggested FER system based on the improved model outperforms the Deep TL techniques in terms of both emotion detection accuracy and evaluation metrics.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9

Similar content being viewed by others

Data availability statement

The datasets used and/or analysed during the current study are available from the corresponding author on reasonable request.

References

  1. Abbassi, N., Helaly, R., Hajjaji, M. A., Mtibaa, A.: A deep learning facial emotion classification system: a VGGNet-19 based approach. In 2020 20th International Conference on Sciences and Techniques of Automatic Control and Computer Engineering (STA) (pp. 271-276). (2020) IEEE

  2. Helaly, R., Hajjaji, M. A., M’Sahli, F., Mtibaa, A.: Deep convolution neural network implementation for emotion recognition system. In 2020 20th International Conference on Sciences and Techniques of Automatic Control and Computer Engineering (STA) (pp. 261-265). (2020). IEEE

  3. Akhand, M.A.H., Roy, S., Siddique, N., Kamal, M.A.S., Shimamura, T.: Facial emotion recognition using transfer learning in the deep CNN. Electronics 10(9), 1036 (2021)

    Article  Google Scholar 

  4. Pantic, M., Rothkrantz, L.J.: Facial action recognition for facial expression analysis from static face images. IEEE Trans. Syst. Man Cybern. Part B Cybern. 34(3), 1449–1461 (2004)

    Article  Google Scholar 

  5. Wolf, K.: Measuring facial expression of emotion. Dialogues Clin. Neurosci. (2022)

  6. Kumar, A., Kumar, M., Kaur, A.: Face detection in still images under occlusion and non-uniform illumination. Multimed. Tools Appl. 80(10), 14565–14590 (2021)

    Article  Google Scholar 

  7. Schoneveld, L., Othmani, A., Abdelkawy, H.: Leveraging recent advances in deep learning for audio-visual emotion recognition. Pattern Recogn. Lett. 146, 1–7 (2021)

    Article  Google Scholar 

  8. Song, Z.: Facial expression emotion recognition model integrating philosophy and machine learning theory. Front. Psychol. 12, (2021)

  9. Kumar, A., Kaur, A., Kumar, M.: Face detection techniques: a review. Artif. Intell. Rev. 52(2), 927–948 (2019)

    Article  Google Scholar 

  10. Tian, Y., Kanade, T., Cohn, J. F.: “Facial expression recognition,” In: Handbook Face Recognition. London, U.K.: Springer, pp. 487-519, (2011)

  11. Shan, C., Gong, S., McOwan, P.W.: Facial expression recognition based on local binary patterns: a comprehensive study. Image Vis. Comput. 27(6), 803–816 (2009)

    Article  Google Scholar 

  12. Bansal, M., Kumar, M., Sachdeva, M., Mittal, A.: Transfer learning for image classification using VGG19: caltech-101 image data set. J. Ambient Intel. Hum. Comput. (2021). https://doi.org/10.1007/s12652-021-03488-z

    Article  Google Scholar 

  13. Bansal, M., Kumar, M., Kumar, M., Kumar, K.: An efficient technique for object recognition using Shi-Tomasi corner detection algorithm. Soft. Comput. 25(6), 4423–4432 (2021)

    Article  Google Scholar 

  14. Singh, S., Ahuja, U., Kumar, M., Kumar, K., Sachdeva, M.: Face mask detection using YOLOv3 and faster R-CNN models: COVID-19 environment. Multimed. Tools Appl. 80(13), 19753–19768 (2021)

    Article  Google Scholar 

  15. Kumar, A., Kaur, A., Kumar, M.: Face detection techniques: a review. Artif. Intell. Rev. 52(2), 927–948 (2019)

    Article  Google Scholar 

  16. Zhong, L., Liu, Q., Yang, P., Liu, B., Huang, J., Metaxas, D. N.: “Learning active facial patches for expression analysis,” In: Computer Vision and Pattern Recognition (CVPR), 2012 IEEE Conference on. IEEE, (2012), pp. 2562-2569

  17. Zhao, G., Pietikainen, M.: Dynamic texture recognition using local binary patterns with an application to facial expressions. IEEE Trans. Pattern Anal. Mach. Intell. 29(6), 915–928 (2007)

    Article  Google Scholar 

  18. Zhi, R., Flierl, M., Ruan, Q., Kleijn, W.B.: Graph-preserving sparse nonnegative matrix factorization with application to facial expression recognition. IEEE Trans. Syst. Man Cybern. Part B (Cybern.) 41(1), 38–52 (2011)

    Article  Google Scholar 

  19. Krizhevsky, A., Sutskever, I., Hinton, G. E.: “ImageNet classification with deep convolutional neural networks,” In: Advances in Neural Information Processing systems, (2012), pp. 1097-1105

  20. Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition, arXiv preprint arXiv: 1409.1556, (2014)

  21. Mollahosseini, A. Hasani, B., Salvador, M. J., Abdollahi, H., Chan, D., Mahoor, M. H.: “Facial expression recognition from World Wild Web,” In: Proc. CVPRW, pp. 1509-1516, (2016)

  22. Wen, G., Hou, Z., Li, H., Li, D., Jiang, L., Xun, E.: Ensemble of deep neural networks with probability-based fusion for facial expression recognition. Cogn. Comput. 9(5), 597–610 (2017)

    Article  Google Scholar 

  23. Arora, M., Kumar, M.: AutoFER: PCA and PSO based automatic facial emotion recognition. Multimed. Tools Appl. 80(2), 3039–3049 (2021)

    Article  Google Scholar 

  24. Arora, M., Kumar, M., Garg, N.K.: Facial emotion recognition system based on PCA and gradient features. Natl. Acad. Sci. Lett. 41(6), 365–368 (2018)

    Article  Google Scholar 

  25. Bansal, M., Kumar, M., Kumar, M.: 2D object recognition: a comparative analysis of SIFT, SURF and ORB feature descriptors. Multimed. Tools Appl. 80(12), 18839–18857 (2021)

    Article  Google Scholar 

  26. Reddy, A.H., Kolli, K., Kiran, Y.L.: Deep cross feature adaptive network for facial emotion classification. SIViP 16(2), 369–376 (2022)

    Article  Google Scholar 

  27. Cohn, J.F., Ekman, P.: “Measuring facial action, In: The New Handbook of Methods in Nonverbal Behaviour Research, (2005), pp. 9-64

  28. Szegedy,C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., Rabinovich, A.: Going deeper with convolutions, In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, (2015), pp. 1-9

  29. He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition, In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, (2016), pp. 770-778

  30. Bansal, M., Kumar, M., Sachdeva, M., Mittal, A.: Transfer learning for image classification using VGG19: caltech-101 image data set. J. Ambient Intel. Humaniz. Comput. (2021). https://doi.org/10.1007/s12652-021-03488-z

    Article  Google Scholar 

  31. Gupta, S., Thakur, K., Kumar, M.: 2D-human face recognition using SIFT and SURF descriptors of face’s feature regions. Vis. Comput. 37(3), 447–456 (2021)

    Article  Google Scholar 

  32. Gupta, S., Thakur, K., Kumar, M.: 2D-human face recognition using SIFT and SURF descriptors of face’s feature regions. Vis. Comput. 37(3), 447–456 (2021)

    Article  Google Scholar 

  33. Arora, M., Kumar, M.: AutoFER: PCA and PSO based automatic facial emotion recognition. Multimed. Tools Appl. 80(2), 3039–3049 (2021)

    Article  Google Scholar 

  34. Goodfellow, I. J., Erhan, D., Carrier, P. L., Courville, A., Mirza, M., Hamner, B., Bengio, Y.: Challenges in representation learning: A report on three machine learning contests. In: International Conference on Neural Information Processing (pp. 117-124). Springer, Berlin, Heidelberg.(2013)

  35. Arora, M., Kumar, M., Garg, N.K.: Facial emotion recognition system based on PCA and gradient features. Natl. Acad. Sci. Lett. 41(6), 365–368 (2018)

    Article  Google Scholar 

  36. Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Fei-Fei, L.: Imagenet large scale visual recognition challenge. Int. J. Comput. Vision 115(3), 211–252 (2015)

    Article  MathSciNet  Google Scholar 

  37. Krizhevsky, A., Sutskever, I., Hinton, G. E.: ImageNet classification with deep convolutional neural networks, In: Proc. NIPS, P. Bartlett, F. Pereira, C. Burges, L. Bottou, and K. Weinberger, Eds. Red Hook, NY, USA: Curran, (2012), pp. 1106-1114

  38. Goodfellow, I. J., Erhan, D., Carrier, P. L., Courville, A., Mirza, M. Hamner, B., Cukierski, W., Tang, Y., Thaler, D., Lee, D.-H. et al.: Challenges in representation learning: a report on three machine learning contests, In: International Conference on Neural Information Processing. Springer, (2013), pp. 117-124

  39. Dhall, A., Ramana Murthy, O., Goecke, R., Joshi, J., Gedeon, T.: Video and image based emotion recognition challenges in the wild: Emotiw 2015,” In: Proceedings of the 2015 ACM on International Conference on Multimodal Interaction. ACM, (2015), pp. 423-426

  40. Dhall, A., Goecke, R., Ghosh, S., Joshi, J., Hoey, J., Gedeon, T.: From individual to group-level emotion recognition: Emotiw 5.0, In: Proceedings of the 19th ACM International Conference on Multimodal Interaction. ACM, (2017), pp. 524-528

  41. Tang, Y.: Deep learning using linear support vector machines,” In: Proc. ICML Workshop Challenges Represent. Learn. Workshop, (2013), pp. 1-6

  42. Minaee, S., Minaei, M., Abdolrashidi, A.: Deep-emotion: facial expression recognition using attentional convolutional network. Sensors 21(9), 3046 (2021)

    Article  Google Scholar 

  43. Wen, G., Hou, Z., Li, H., Li, D., Jiang, L., Xun, J.: Ensemble of deep neural networks with probability-based fusion for facial expression recognition. Cognit. Comput. 9(5), 597–610 (2017)

    Article  Google Scholar 

  44. Yu, Z., Zhang, C.: Image based static facial expression recognition with multiple deep network learning, In: Proc. ICMI, (Nov. 2015), pp. 435-442

  45. Li, D., Wen, G.: MRMR-based ensemble pruning for facial expression recognition. Multimed. Tools Appl. 77(12), 15251–15272 (2018)

    Article  Google Scholar 

  46. Hua, W., Dai, F., Huang, L., Xiong, J., Gui, G.: HERO: human emotions recognition for realizing intelligent Internet of Things. IEEE Access 7, 24321–24332 (2019)

    Article  Google Scholar 

  47. Connie, T., Al-Shabi, M., Cheah, W. P., Goh, M.: Facial expression recognition using a hybrid CNN_SIFT aggregator, In: Proc. MIWAI, vol. 10607. Cham, Switzerland: Springer, (2017), pp. 139-149

  48. Kaya, H., Gürpinar, F., Salah, A.A.: Video-based emotion recognition in the wild using deep transfer learning and score fusion. Image Vis. Comput. 65, 66–75 (2017)

    Article  Google Scholar 

  49. Lowe, D.G.: Distinctive image features from scale-invariant keypoints. Int. J. Comput. Vis. 60(2), 91–110 (2004)

    Article  Google Scholar 

  50. Dalal, N., Triggs, B.: Histograms of oriented gradients for human detection. Proc. IEEE Comput. Soc. Conf. Comput. Vis. Pattern Recognit. 1(1), 886–893 (2005)

    Google Scholar 

  51. Kaya, H., Gürpinar, F., Salah, A.A.: Video-based emotion recognition in the wild using deep transfer learning and score fusion. Image Vis. Comput. 65, 66–75 (2017)

    Article  Google Scholar 

  52. Hazourli, A.R., Djeghri, A., Salam, H., Othmani, A.: Multi-facial patches aggregation network for facial expression recognition and facial regions contributions to emotion display. Multimed. Tools Appl. 80(9), 13639–13662 (2021)

    Article  Google Scholar 

  53. Hasani, B., Mahoor, M. H.: Facial expression recognition using enhanced deep 3D convolutional neural networks, In: Proc. CVPRW, (2017), pp. 2278-2288

  54. Liu, X., Kumar, B. V. K. V., You, J., Jia, P.: Adaptive deep metric learning for identity-aware facial expression recognition, In: Proc CVPRW, (2017), pp 522-531

  55. Meng, Z., Liu, P., Cai, J., Han, S., Tong, Y.: Identity-aware convolutional neural network for facial expression recognition, In: Proc. 12th IEEE Int. Conf. Autom. Face Gesture Recognit., (2017), pp. 558-565

  56. Li, S., Deng, W., Du, J.: Reliable crowdsourcing and deep locality preserving learning for expression recognition in the wild, In: Proc. CVPR, (2017), pp. 2584-2593

  57. Liu, X., Kumar, B. V. K. V., You, J., Jia, P.: Adaptive deep metric learning for identity-aware facial expression recognition, In: Proc. CVPRW, (2017), pp. 522-531

  58. Li, Y., Zeng, J., Shan, S., Chen, X.: Patch-Gated CNN for occlusion awarefacial expression recognition, In: Proc. ICPR, (2018), pp. 2209-2214

  59. Hua, W., Dai, F., Huang, L., Xiong, J., Gui, G.: HERO: human emotions recognition for realizing intelligent Internet of Things. IEEE Access 7, 24321–24332 (2019)

    Article  Google Scholar 

  60. Zeng, J., Shan, S., Chen, X.: Facial expression recognition with inconsistentlyannotated datasets, In: Proc. ECCV, pp. 222-237, (2018)

  61. Kaya, H., Gürpinar, F., Salah, A.A.: Video-based emotion recognition in the wild using deep transfer learning and score fusion. Image Vis. Comput. 65, 66–75 (2017)

    Article  Google Scholar 

  62. Ionescu, R. T., Popescu, M., Grozea, C.: Local learning to improve bag of visual words model for facial expression recognition, In: Proc. ICML Workshop Challenges Represent. Learn., pp. 1-6, (2013)

  63. Kaya, H., Gürpinar, F., Salah, A.A.: Video-based emotion recognition in the wild using deep transfer learning and score fusion. Image Vis. Comput. 65, 66–75 (2017)

    Article  Google Scholar 

  64. Kim, B.K., Dong, S.Y., Roh, J., Kim, G., Lee, S.Y.: Fusing Aligned and Non-Aligned Face Information for automatic affect recognition in the wild: A deep learning approach. IEEE Conference on Computer Vision and Pattern Recognition Workshops. (2016)

  65. He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (pp. 770-778), (2016)

  66. Lucey, P., Cohn, J. F., Kanade, T., Saragih, J., Ambadar, Z., Matthews, I.: The extended cohn-kanade dataset (ck+): A complete dataset for action unit and emotion-specified expression, In: Computer Vision and Pattern Recognition Workshops (CVPRW), 2010 IEEE Computer Society Conference on. IEEE, (2010), pp. 94-101

  67. Chaudhari, A., Bhatt, C., Krishna, A., Mazzeo, P.L.: ViTFER: facial emotion recognition with vision transformers. Appl. Syst. Innov. 5(4), 80 (2022)

    Article  Google Scholar 

  68. Kong, Y., Zhang, S., Zhang, K., Ni, Q., Han, J.: Real-time facial expression recognition based on iterative transfer learning and efficient attention network. IET Image Proc. 16(6), 1694–1708 (2022)

    Article  Google Scholar 

  69. Sreevidya, P., Veni, S., Ramana Murthy, O.V.: Elder emotion classification through multimodal fusion of intermediate layers and cross-modal transfer learning. SIViP 16(5), 1281–1288 (2022)

    Article  Google Scholar 

  70. Goodfellow, I. J., Erhan, D., Carrier, P. L., Courville, A., Mirza, M., Hamner, B., Cukierski, W., Tang, Y., Thaler, D., Lee, D.-H. et al.: Challenges in representation learning: a report on three machine learning contests, In: International Conference on Neural Information Processing. Springer, (2013), pp. 117-124

  71. Khattak, A., Asghar, M.Z., Ali, M., Batool, U.: An efficient deep learning technique for facial emotion recognition. Multimed. Tools Appl. 81(2), 1649–1683 (2022)

    Article  Google Scholar 

Download references

Acknowledgements

We thank all the authors who contributed in this study.

Funding

Not Applicable.

Author information

Author notes

  1. Rabie Helaly, Seifeddine Messaoud, Soulef Bouaafia, Mohamed Ali Hajjaji and Abdellatif Mtibaa have contributed equally to this work.

Authors and Affiliations

  1. Electronics and Microelectronics Laboratory, FSM, University of Monastir, Monastir, Tunisia

    Rabie Helaly, Seifeddine Messaoud, Soulef Bouaafia, Mohamed Ali Hajjaji & Abdellatif Mtibaa

  2. National Engineering School of Monastir, University of Monastir, Monastir, Tunisia

    Rabie Helaly

  3. Higher Institute of Applied Science and Technology of Sousse, University of Sousse, Sousse, Tunisia

    Mohamed Ali Hajjaji

  4. National Engineering School of Sfax, University of Sfax, Sfax, Tunisia

    Abdellatif Mtibaa

Authors
  1. Rabie Helaly
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Seifeddine Messaoud
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Soulef Bouaafia
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Mohamed Ali Hajjaji
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Abdellatif Mtibaa
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

All authors contributed to the study conception and design. Material preparation, data collection and analysis were performed by [Rabie Helaly], [Soulef Bouaafia], [Seifeddine Messaoud]. The first draft of the manuscript was written by [Soulef Bouaafia] and [Seifeddine Messaoud] and all authors commented on previous versions of the manuscript. [Mohamed Ali Hajjaji] and [Abdellatif Mtibaa] read and approved the final manuscript.

Corresponding author

Correspondence to Soulef Bouaafia.

Ethics declarations

Conflict of interest

The authors declare that they have no competing interests.

Ethics approval and consent to participate

Not applicable.

Consent for publication

Not applicable.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Helaly, R., Messaoud, S., Bouaafia, S. et al. DTL-I-ResNet18: facial emotion recognition based on deep transfer learning and improved ResNet18. SIViP 17, 2731–2744 (2023). https://doi.org/10.1007/s11760-023-02490-6

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11760-023-02490-6

Keywords

Search

Navigation