Skip to main content
SpringerLink
Log in

Multichannel convolutional neural network for human emotion recognition from in-the-wild facial expressions

  • Original article
  • Published:
The Visual Computer Aims and scope Submit manuscript

Abstract

Facial emotions reflect the person’s moods and show the human affective state that is correlative with non-verbal intentions and behaviors. Despite the advances on computer vision techniques, capturing automatically the facial expressions in-the-wild remains a very difficult task. In this context, we propose a multichannel convolutional neural network based on the quality and the strengths of three well-known pre-trained models, namely VGG19, GoogleNet, and ResNet101. Indeed, the complementarity of the features extracted from the three models is exploited in order to form a more robust feature vector. During the training phase, a freezing weight is applied for each architecture. Then, the layers containing the most relevant information are marked, and the final feature descriptor for emotion prediction is thereafter defined by concatenating the obtained vectors. In fact, the three architectures have showed their efficiency severally in term of emotion recognition, and notably they do not err in the same images. The final vector, obtained by concatenating the features extracted from the different models, is fed to a support vector machine classifier in order to predict the final emotions. Extensive experiments have been conducted on four challenging datasets (JAFFE, CK+, FER2013 and SFEW_2.0) covering in-the-wild as well as in-the-laboratory facial expressions. The obtained results show that the suggested method is not only more accurate compared to each pre-trained CNN model but it also outperforms relevant state-of-the-art methods.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4

Similar content being viewed by others

References

  1. Plutchik, R.: Emotion, a Psychoevolutionary Synthesis. Harper & Row, New York (1980)

    Google Scholar 

  2. Bericat, E.: The sociology of emotions: four decades of progress. Curr. Sociol. 64(3), 491–513 (2016)

    Article  Google Scholar 

  3. Shouse, E.: Feeling, emotion, affect. M/C Journal 8(6) (2005)

  4. Darwin, C., Prodger, P.: The Expression of the Emotions in Man and Animals. Oxford University Press, Oxford (1998)

    Google Scholar 

  5. Damasio, A., Blanc, M.: L’erreur de Descartes: la raison des émotions. Odile Jacob, France (2006)

    Google Scholar 

  6. Karnati, M., Seal, A., Yazidi, A., Krejcar, O.: Lienet: a deep convolution neural networks framework for detecting deception. IEEE Trans. Cognit. Dev. Syst. 14(3), 971–984 (2022)

    Article  Google Scholar 

  7. Hua, W., Dai, F., Huang, L., Xiong, J., Gui, G.: Hero: human emotions recognition for realizing intelligent internet of things. IEEE Access 7, 24321–24332 (2019)

    Article  Google Scholar 

  8. Khorsheed, J.A., Yurtkan, K.: Analysis of local binary patterns for face recognition under varying facial expressions. In: Signal Processing and Communication Application Conference (SIU), pp. 2085–2088 (2016)

  9. Minaee, S., Minaei, M., Abdolrashidi, A.: Deep-emotion: facial expression recognition using attentional convolutional network. Sensors 21(9), 3046 (2021)

    Article  Google Scholar 

  10. Lyons, M., Kamachi, M., Gyoba, J.: The Japanese female facial expression (JAFFE) dataset (1998)

  11. Kanade, T., Cohn, J.F., Tian, Y.: Comprehensive database for facial expression analysis. In: IEEE International Conference on Automatic Face and Gesture Recognition (Cat. No. PR00580), pp. 46–53 (2000)

  12. Lucey, P., Cohn, J.F., Kanade, T., Saragih, J., Ambadar, Z., Matthews, I.: The extended Cohn-Kanade dataset (CK+): a complete dataset for action unit and emotion-specified expression. In: IEEE Computer Society Conference on Computer Vision and Pattern Recognition - Workshops, pp. 94–101 (2010)

  13. Zhao, G., Huang, X., Taini, M., Li, S.Z., Pietikäinen, M.: Facial expression recognition from near-infrared videos. Image Vis. Comput. 29(9), 607–619 (2011)

    Article  Google Scholar 

  14. Mollahosseini, A., Hasani, B., Mahoor, M.H.: Affectnet: a database for facial expression, valence, and arousal computing in the wild. IEEE Trans. Affect. Comput. 10(1), 18–31 (2017)

    Article  Google Scholar 

  15. Dhall, A., Goecke, R., Lucey, S., Gedeon, T.: Collecting large, richly annotated facial-expression databases from movies. IEEE Multimed. 19(3), 34–41 (2012)

    Article  Google Scholar 

  16. Dhall, A., Goecke, R., Joshi, J., Sikka, K., Gedeon, T.: Emotion recognition in the wild challenge 2014: baseline, data and protocol. In: International Conference on Multimodal Interaction, pp. 461–466 (2014)

  17. Goodfellow, I.J., Erhan, D., Carrier, P.L., Courville, A., Mirza, M., Hamner, B., Cukierski, W., Tang, Y., Thaler, D., Lee, D.-H. et al.: Challenges in representation learning: A report on three machine learning contests. In: International Conference on Neural Information Processing (ICONIP), pp. 117–124 (2013)

  18. Zhang, L., Verma, B., Tjondronegoro, D., Chandran, V.: Facial expression analysis under partial occlusion: a survey. ACM Comput. Surv. 51(2), 25 (2018)

    Google Scholar 

  19. Tian, Y.-L., Kanade, T., Cohn, J.F.: Facial expression analysis. In: Handbook of Face Recognition, pp. 247–275. Springer (2005)

  20. Mollahosseini, A., Hasani, B., Mahoor, M.H.: Affectnet: a database for facial expression, valence, and arousal computing in the wild. IEEE Trans. Affect. Comput. 10(1), 18–31 (2019)

    Article  Google Scholar 

  21. Dou, S., Feng, Z., Yang, X., Tian, J.: Real-time multimodal emotion recognition system based on elderly accompanying robot. J. Phys.: Conf. Ser. 1453(1), 012093 (2020)

    Google Scholar 

  22. Khorrami, P., Paine, T., Huang, T.: Do deep neural networks learn facial action units when doing expression recognition? In: IEEE International Conference on Computer Vision Workshop (ICCVW), pp. 19–27 (2015)

  23. Ben Fredj, H., Bouguezzi, S., Souani, C.: Face recognition in unconstrained environment with CNN. Vis. Comput. 37(2), 217–226 (2021)

    Article  Google Scholar 

  24. Kumar, A., Kaur, A., Kumar, M.: Face detection techniques: a review. Artif. Intell. Rev. 52, 927–948 (2019)

    Article  Google Scholar 

  25. Zhang, K., Zhang, Z., Li, Z., Qiao, Y.: Joint face detection and alignment using multitask cascaded convolutional networks. IEEE Signal Process. Lett. 23(10), 1499–1503 (2016)

    Article  Google Scholar 

  26. King, D.E.: Dlib-ml: a machine learning toolkit. J. Mach. Learn. Res. 10, 1755–1758 (2009)

    Google Scholar 

  27. Wirdiani, N.A., Lattifia, T., Supadma, I.K., Mahar, B.K., Taradhita, D.N., Fahmi, A.: Real-time face recognition with eigenface method. Int. J. Image Graph. Signal Process. 11(11), 1–9 (2019)

    Article  Google Scholar 

  28. Viola, P., Jones, M.: Rapid object detection using a boosted cascade of simple features. In: IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR) (2001)

  29. Li, S., Deng, W.: Deep facial expression recognition: a survey. IEEE Trans. Affect. Comput. 13(3), 1195–1215 (2022)

    Article  Google Scholar 

  30. Lopes, A.T., De Aguiar, E., Oliveira-Santos, T.: A facial expression recognition system using convolutional networks. In: SIBGRAPI Conference on Graphics, Patterns and Images, pp. 273–280 (2015)

  31. Li, K., Jin, Y., Akram, M.W., Han, R., Chen, J.: Facial expression recognition with convolutional neural networks via a new face cropping and rotation strategy. Vis. Comput. 36(2), 391–404 (2020)

    Article  Google Scholar 

  32. Ramos, A.L.A., Dadiz, B.G., Santos, A.B.G.: Classifying emotion based on facial expression analysis using Gabor filter: a basis for adaptive effective teaching strategy. In: Computational Science and Technology, vol. 603, pp. 469–479. Springer (2020)

  33. Slimani, K., Kas, M., El Merabet, Y., Messoussi, R., Ruichek, Y.: Facial emotion recognition: A comparative analysis using 22 LBP variants. In: Mediterranean Conference on Pattern Recognition and Artificial Intelligence, pp. 88–94 (2018)

  34. Kumar, P., Happy, S., Routray, A.: A real-time robust facial expression recognition system using hog features. In: International Conference on Computing, Analytics and Security Trends (CAST), pp. 289–293 (2016)

  35. Liu, X., Cheng, X., Lee, K.: GA-SVM-based facial emotion recognition using facial geometric features. IEEE Sens. J. 21(10), 11532–11542 (2020)

    Article  Google Scholar 

  36. Zhang, H., Su, W., Wang, Z.: Weakly supervised local-global attention network for facial expression recognition. IEEE Access 8, 37976–37987 (2020)

    Article  Google Scholar 

  37. Zhang, S., Zhao, X., Lei, B.: Facial expression recognition based on local binary patterns and local fisher discriminant analysis. WSEAS Trans. Signal Process. 8(1), 21–31 (2012)

    Google Scholar 

  38. Abdulrahman, M., Eleyan, A.: Facial expression recognition using support vector machines. In: Signal Processing and Communications Applications Conference (SIU), pp. 276–279 (2015)

  39. Alshamsi, H., Kepuska, V., Meng, H.: Automated facial expression recognition app development on smart phones using cloud computing. In: Annual Ubiquitous Computing, Electronics and Mobile Communication Conference (UEMCON), pp. 577–583 (2017)

  40. Gite, B., Nikhal, K., Palnak, F.: Evaluating facial expressions in real time. In: Intelligent Systems Conference (IntelliSys), pp. 849–855 (2017)

  41. Agrawal, A., Mittal, N.: Using CNN for facial expression recognition: a study of the effects of kernel size and number of filters on accuracy. Vis. Comput. 36(2), 405–412 (2020)

    Article  Google Scholar 

  42. Mohan, K., Seal, A., Krejcar, O., Yazidi, A.: Fer-net: Facial expression recognition using deep neural net. Neural Comput. Appl. 33(15), 9125–9136 (2021)

    Article  Google Scholar 

  43. Siqueira, H., Magg, S., Wermter, S.: Efficient facial feature learning with wide ensemble-based convolutional neural networks. In: AAAI Conference on Artificial Intelligence, pp. 5800–5809 (2020)

  44. Fasel, B.: Robust face analysis using convolutional neural networks. In: International Conference on Pattern Recognition (ICPR), pp. 40–43 (2002)

  45. Mohan, K., Seal, A., Krejcar, O., Yazidi, A.: Facial expression recognition using local gravitational force descriptor-based deep convolution neural networks. IEEE Trans. Instrum. Measur. 70, 1–12 (2021)

    Article  Google Scholar 

  46. Liu, P., Han, S., Meng, Z., Tong, Y.: Facial expression recognition via a boosted deep belief network. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1805–1812 (2014)

  47. Vincent, P., Larochelle, H., Lajoie, I., Bengio, Y., Manzagol, P.-A., Bottou, L.: Stacked denoising autoencoders: learning useful representations in a deep network with a local denoising criterion. J. Mach. Learn. Res. 11(12), 3371–3408 (2010)

    MathSciNet  MATH  Google Scholar 

  48. Fan, Y., Lu, X., Li, D., Liu, Y.: Video-based emotion recognition using CNN-RNN and C3D hybrid networks. In: ACM International Conference on Multimodal Interaction, pp. 445–450 (2016)

  49. Lai, Y.-H., Lai, S.-H.: Emotion-preserving representation learning via generative adversarial network for multi-view facial expression recognition. In: IEEE International Conference on Automatic Face & Gesture Recognition, pp. 263–270 (2018)

  50. Hazarika, D., Poria, S., Zimmermann, R., Mihalcea, R.: Conversational transfer learning for emotion recognition. Inf. Fusion 65, 1–12 (2021)

    Article  Google Scholar 

  51. Shaees, S., Naeem, H., Arslan, M., Naeem, M.R., Ali, S.H., Aldabbas, H.: Facial emotion recognition using transfer learning. In: International Conference on Computing and Information Technology (ICCIT), pp. 1–5 (2020)

  52. Zhang, H., Huang, B., Tian, G.: Facial expression recognition based on deep convolution long short-term memory networks of double-channel weighted mixture. Pattern Recognit. Lett. 131, 128–134 (2020)

    Article  Google Scholar 

  53. Chen, L., Wu, M., Pedrycz, W., Hirota, K.: Deep sparse autoencoder network for facial emotion recognition. In: Emotion Recognition and Understanding for Emotional Human–Robot Interaction Systems, Studies in Computational Intelligence, pp. 25–39. Springer International Publishing (2021)

  54. Liu, Y., Zhang, X., Lin, Y., Wang, H.: Facial expression recognition via deep action units graph network based on psychological mechanism. IEEE Trans. Cognit. Dev. Syst. 12(2), 311–322 (2019)

    Article  Google Scholar 

  55. Fan, X., Tjahjadi, T.: Fusing dynamic deep learned features and handcrafted features for facial expression recognition. J. Vis. Commun. Image Represent. 65, 102659 (2019)

    Article  Google Scholar 

  56. Sun, X., Lv, M.: Facial expression recognition based on a hybrid model combining deep and shallow features. Cognit. Comput. 11(4), 587–597 (2019)

    Article  Google Scholar 

  57. Gogic, I., Manhart, M., Pandžic, I.S., Ahlberg, J.: Fast facial expression recognition using local binary features and shallow neural networks. Vis. Comput. 36, 97–112 (2020)

    Article  Google Scholar 

  58. Alreshidi, A., Ullah, M.: Facial emotion recognition using hybrid features. Informatics 7(1), 6 (2020)

    Article  Google Scholar 

  59. Dhall, A., Ramana Murthy, O., Goecke, R., Joshi, J., Gedeon, T.: Video and image based emotion recognition challenges in the wild: Emotiw 2015. In: ACM International Conference on Multimodal Interaction, pp. 423–426 (2015)

  60. Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. In: International Conference on Learning Representations (ICLR) (2015)

  61. Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., Rabinovich, A.: Going deeper with convolutions. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1–9 (2015)

  62. Li, Y., Lyu, S.: Exposing deepfake videos by detecting face warping artifacts. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 46–52 (2018)

  63. Boughanem, H., Ghazouani, H., Barhoumi, W.: Towards a deep neural method based on freezing layers for in-the-wild facial emotion recognition. In: IEEE/ACS International Conference on Computer Systems and Applications (AICCSA), pp. 1–8 (2021)

  64. Ahmed, T.U., Hossain, S., Hossain, M.S., ul Islam, R., Andersson, K.: Facial expression recognition using convolutional neural network with data augmentation. In: Joint International Conference on Informatics, Electronics & Vision (ICIEV) and International Conference on Imaging, Vision & Pattern Recognition (icIVPR), pp. 336–341 (2019)

  65. Simard, P.Y., Steinkraus, D., Platt, J.C.: Best practices for convolutional neural networks applied to visual document analysis. In: International Conference on Document Analysis and Recognition (ICDAR), pp. 958–963 (2003)

  66. Siam, A.I., Soliman, N.F., Algarni, A.D., El-Samie, A., Fathi, E., Sedik, A.: Deploying machine learning techniques for human emotion detection. Comput. Intell. Neurosci. 2022, 8032673 (2022)

    Article  Google Scholar 

  67. Saurav, S., Gidde, P., Saini, R., Singh, S.: Dual integrated convolutional neural network for real-time facial expression recognition in the wild. Vis. Comput. 38(3), 1083–1096 (2022)

    Article  Google Scholar 

  68. Krizhevsky, A., Sutskever, I., Hinton, G.E.: Imagenet classification with deep convolutional neural networks. Commun. ACM 60(6), 84–90 (2017)

    Article  Google Scholar 

  69. Vo, A., Nguyen, B.T.: Facial expression recognition based on salient regions. In: International Conference on Green Technology and Sustainable Development (GTSD), pp. 739–743 (2018)

  70. Huang, Y., Chen, F., Lv, S., Wang, X.: Facial expression recognition: a survey. Symmetry 11(10), 1189 (2019)

    Article  Google Scholar 

  71. Dhall, A., Goecke, R., Lucey, S., Gedeon, T.: Static facial expression analysis in tough conditions: Data, evaluation protocol and benchmark. In: IEEE International Conference on Computer Vision Workshops (ICCV Workshops), pp. 2106–2112 (2011)

  72. Ravi, R., Yadhukrishna, S., Prithviraj, R.: A face expression recognition using CNN & LBP. In: International Conference on Computing Methodologies and Communication (ICCMC), pp. 684–689 (2020)

  73. Wu, M., Su, W., Chen, L., Liu, Z., Cao, W., Hirota, K.: Weight-adapted convolution neural network for facial expression recognition in human–robot interaction. IEEE Transactions on Systems, Man, and Cybernetics: Systems 51(3), 1473–1484 (2021)

    Article  Google Scholar 

  74. Bodapati, J.D., Srilakshmi, U., Veeranjaneyulu, N.: Fernet: a deep CNN architecture for facial expression recognition in the wild. J. Inst. Eng. (India): Ser. B 103(2), 439–448 (2022)

    Google Scholar 

  75. Salmam, F.Z., Madani, A., Kissi, M.: Fusing multi-stream deep neural networks for facial expression recognition. Signal Image Video Process. 13(3), 609–616 (2019)

    Article  Google Scholar 

  76. Kim, J.-H., Kim, B.-G., Roy, P.P., Jeong, D.-M.: Efficient facial expression recognition algorithm based on hierarchical deep neural network structure. IEEE Access 7, 41273–41285 (2019)

    Article  Google Scholar 

  77. Hung, J.C., Lin, K.-C., Lai, N.-X.: Recognizing learning emotion based on convolutional neural networks and transfer learning. Appl. Soft Comput. 84, 105724 (2019)

    Article  Google Scholar 

  78. Jain, D.K., Shamsolmoali, P., Sehdev, P.: Extended deep neural network for facial emotion recognition. Pattern Recognit. Lett. 120, 69–74 (2019)

    Article  Google Scholar 

  79. Xie, S., Hu, H.: Facial expression recognition using hierarchical features with deep comprehensive multipatches aggregation convolutional neural networks. IEEE Trans. Multimed. 21(1), 211–220 (2018)

    Article  Google Scholar 

  80. Shao, J., Qian, Y.: Three convolutional neural network models for facial expression recognition in the wild. Neurocomputing 355, 82–92 (2019)

    Article  Google Scholar 

  81. Umer, S., Rout, R.K., Pero, C., Nappi, M.: Facial expression recognition with trade-offs between data augmentation and deep learning features. J. Ambient. Intell. Humaniz. Comput. 13(2), 721–735 (2022)

    Article  Google Scholar 

  82. Yan, K., Zheng, W., Zhang, T., Zong, Y., Tang, C., Lu, C., Cui, Z.: Cross-domain facial expression recognition based on transductive deep transfer learning. IEEE Access 7, 108906–108915 (2019)

    Article  Google Scholar 

  83. Saurav, S., Saini, R., Singh, S.: Emnet: a deep integrated convolutional neural network for facial emotion recognition in the wild. Appl. Intell. 51, 5543–5570 (2021)

    Article  Google Scholar 

  84. Zhou, L., Fan, X., Tjahjadi, T., Das Choudhury, S.: Discriminative attention-augmented feature learning for facial expression recognition in the wild. Neural Comput. Appl. 34(2), 925–936 (2022)

    Article  Google Scholar 

  85. Liang, X., Xu, L., Zhang, W., Zhang, Y., Liu, J., Liu, Z.: A convolution-transformer dual branch network for head-pose and occlusion facial expression recognition. Vis. Comput. (2022)

Download references

Author information

Authors and Affiliations

  1. Institut Supérieur d’Informatique d’El Manar, Research Team on Intelligent Systems in Imaging and Artificial Vision (SIIVA), LR16ES06 Laboratoire de recherche en Informatique, Modélisation et Traitement de l’Information et de la Connaissance (LIMTIC), Université de Tunis El Manar, 2 Rue Abou Rayhane Bayrouni, 2080, Ariana, Tunisia

    Hadjer Boughanem, Haythem Ghazouani & Walid Barhoumi

  2. Ecole Nationale d’Ingénieurs de Carthage, Université de Carthage, 45 Rue des Entrepreneurs, 2035, Tunis-Carthage, Tunisia

    Haythem Ghazouani & Walid Barhoumi

Authors
  1. Hadjer Boughanem
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Haythem Ghazouani
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Walid Barhoumi
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Walid Barhoumi.

Ethics declarations

Conflict of interest

The authors have no relevant financial or non-financial interests to disclose.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Boughanem, H., Ghazouani, H. & Barhoumi, W. Multichannel convolutional neural network for human emotion recognition from in-the-wild facial expressions. Vis Comput 39, 5693–5718 (2023). https://doi.org/10.1007/s00371-022-02690-0

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00371-022-02690-0

Keywords

Search

Navigation