Skip to main content
SpringerLink
Log in

Hybrid-Deep Learning Model for Emotion Recognition Using Facial Expressions

  • Article
  • Published:
The Review of Socionetwork Strategies Aims and scope Submit manuscript

Abstract

Humans use facial expressions as a tool to show emotional states. Facial expression recognition remains an interesting and challenging area of research in computer vision. An improved deep learning approach using a convolution neural network (CNN) is proposed in this paper to predict emotions by analysing facial expressions contained in an image. The model developed in this work consists of one CNN to analyse the primary emotion of the image as being happy or sad and a second CNN to predict the secondary emotion of the image. The proposed model was trained on the FER2013 and Japanese female facial expression (JAFFE) datasets with results suggesting its capability of predicting emotions from facial expressions better than existing state-of-the-art approaches.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5

Similar content being viewed by others

References

  1. Arnheim, R. (1965). Art and visual perception: A psychology of the creative eye. Berkeley: University of California Press.

    Google Scholar 

  2. Borth, D., Ji, R., Chen, T., Breuel, T., & Chang, S. F. (2013). Large-scale visual sentiment ontology and detectors using adjective noun pairs. In Proceedings of the 21st ACM International Conference on Multimedia (pp. 223–232). ACM.

  3. Boughrara, H., Chtourou, M., Amar, C. B., & Chen, L. (2016). Facial expression recognition based on a mlp neural network using constructive training algorithm. Multimedia Tools and Applications, 75(2), 709–731.

    Article  Google Scholar 

  4. Busso, C., Deng, Z., Yildirim, S., Bulut, M., Lee, C. M., Kazemzadeh, A., Lee, S., Neumann, U., & Narayanan, S. (2004). Analysis of emotion recognition using facial expressions, speech and multimodal information. In Proceedings of the 6th International Conference on Multimodal Interfaces, (pp. 205–211). ACM.

  5. Chen, T., Borth, D., Darrell, T., & Chang, S.F. (2014). Deepsentibank: Visual sentiment concept classification with deep convolutional neural networks. arXiv preprint arXiv:1410.8586.

  6. Cheng, F., Yu, J., & Xiong, H. (2010). Facial expression recognition in Jaffe dataset based on Gaussian process classification. IEEE Transactions on Neural Networks, 21(10), 1685–1690.

    Article  Google Scholar 

  7. Christou, N., & Kanojiya, N. (2019). Human facial expression recognition with convolution neural networks. In Third International Congress on Information and Communication Technology (pp. 539–545). Springer.

  8. Colombo, C., Del Bimbo, A., & Pala, P. (1999). Semantics in visual information retrieval. IEEE Multimedia, 6(3), 38–53.

    Article  Google Scholar 

  9. Dellandrea, E., Liu, N., & Chen, L. (2010). Classification of affective semantics in images based on discrete and dimensional models of emotions. In 2010 International Workshop on Content Based Multimedia Indexing (CBMI) (pp. 1–6). IEEE.

  10. He, X., & Zhang, W. (2018). Emotion recognition by assisted learning with convolutional neural networks. Neurocomputing, 291, 187–194.

    Article  Google Scholar 

  11. Ionescu, R.T., Popescu, M., & Grozea, C. (2013). Local learning to improve bag of visual words model for facial expression recognition. In Workshop on Challenges in Representation Learning, ICML.

  12. Jain, D. K., Shamsolmoali, P., & Sehdev, P. (2019). Extended deep neural network for facial emotion recognition. Pattern Recognition Letters, 120, 69–74.

    Article  Google Scholar 

  13. Jain, N., Kumar, S., Kumar, A., Shamsolmoali, P., & Zareapoor, M. (2018). Hybrid deep neural networks for face emotion recognition. Pattern Recognition Letters, 115, 101–106.

    Article  Google Scholar 

  14. Joshi, D., Datta, R., Fedorovskaya, E., Luong, Q. T., Wang, J. Z., Li, J., et al. (2011). Aesthetics and emotions in images. IEEE Signal Processing Magazine, 28(5), 94–115.

    Article  Google Scholar 

  15. Kahou, S. E., Bouthillier, X., Lamblin, P., Gulcehre, C., Michalski, V., Konda, K., et al. (2016). Emonets: Multimodal deep learning approaches for emotion recognition in video. Journal on Multimodal User Interfaces, 10(2), 99–111.

    Article  Google Scholar 

  16. Kahou, S.E., Pal, C., Bouthillier, X., Froumenty, P., Gu¨l¸cehre, C¸., Memisevic, R., Vincent, P., Courville, A., Bengio, Y., & Ferrari, R.C., et al. (2013). Combining modality specific deep neural networks for emotion recognition in video. In Proceedings of the 15th ACM on International Conference on Multimodal Interaction (pp. 543–550). ACM.

  17. Khorrami, P., Le Paine, T., Brady, K., Dagli, C., & Huang, T.S. (2016). How deep neural networks can improve emotion recognition on video data. In: 2016 IEEE International Conference on Image Processing (ICIP) (pp. 619–623). IEEE.

  18. Lopes, A. T., de Aguiar, E., De Souza, A. F., & Oliveira-Santos, T. (2017). Facial expression recognition with convolutional neural networks: coping with few data and the training sample order. Pattern Recognition, 61, 610–628.

    Article  Google Scholar 

  19. Lu, X., Suryanarayan, P., Adams Jr, R. B., Li, J., Newman, M. G., & Wang, J. Z. (2012). On shape and the computability of emotions. In Proceedings of the 20th ACM International Conference on Multimedia (pp. 229–238). ACM.

  20. Lyons, M. J., Akamatsu, S., Kamachi, M., Gyoba, J., & Budynek, J. (1998). The Japanese female facial expression (Jaffe) database. In Proceedings of Third International Conference on Automatic Face and Gesture Recognition (pp. 14–16).

  21. Machajdik, J., & Hanbury, A. (2010). Affective image classification using features inspired by psychology and art theory. In Proceedings of the 18th ACM International Conference on Multimedia (pp. 83–92). ACM.

  22. Mavani, V., Raman, S., & Miyapuram, K. P. (2017). Facial expression recognition using visual saliency and deep learning. In: Proceedings of the IEEE International Conference on Computer Vision (pp. 2783–2788).

  23. Minaee, S., & Abdolrashidi, A. (2019). Deep-emotion: Facial expression recognition using attentional convolutional network. arXiv preprint arXiv:1902.01019.

  24. Nicolaou, M. A., Gunes, H., & Pantic, M. (2011). A multi-layer hybrid framework for dimensional emotion classification. In: Proceedings of the 19th ACM international conference on Multimedia (pp. 933–936). ACM.

  25. Nie, D., Wang, X. W., Shi, L. C., & Lu, B. L. (2011). Eeg-based emotion recognition during watching movies. In: 2011 5th International IEEE/EMBS Conference on Neural Engineering (pp. 667–670). IEEE.

  26. Shan, K., Guo, J., You, W., Lu, D., & Bie, R. (2017). Automatic facial expression recognition based on a deep convolutional-neural-network structure. In 2017 IEEE 15th International Conference on Software Engineering Research, Management and Applications (SERA) (pp. 123–128). IEEE.

  27. Srivastava, N., Hinton, G., Krizhevsky, A., Sutskever, I., & Salakhutdinov, R. (2014). Dropout: A simple way to prevent neural networks from overfitting. The Journal of Machine Learning Research, 15(1), 1929–1958.

    Google Scholar 

  28. Tang, Y. (2013). Challenges in representation learning: Facial expression recognition challenge implementation. Toronto: University of Toronto.

    Google Scholar 

  29. Tzirakis, P., Trigeorgis, G., Nicolaou, M. A., Schuller, B. W., & Zafeiriou, S. (2017). End-to-end multimodal emotion recognition using deep neural networks. IEEE Journal of Selected Topics in Signal Processing, 11(8), 1301–1309.

    Article  Google Scholar 

  30. Valdez, P., & Mehrabian, A. (1994). Effects of color on emotions. Journal of experimental psychology: General, 123(4), 394.

    Article  Google Scholar 

  31. Wei-ning, W., Ying-lin, Y., Sheng-ming, J. (2006). Image retrieval by emotional semantics: A study of emotional space and feature extraction. In 2006 IEEE International Conference on Systems, Man and Cybernetics, vol. 4 (pp. 3534–3539). IEEE.

  32. Xu, C., Cetintas, S., Lee, K., & Li, L. (2014). Visual sentiment prediction with deep convolutional neural networks. arXiv preprint arXiv:1411.5731

  33. Zhao, S., Gao, Y., Jiang, X., Yao, H., Chua, T.S., & Sun, X. (2014) Exploring principles-of-art features for image emotion recognition. In Proceedings of the 22nd ACM international conference on Multimedia (pp. 47–56). ACM.

Download references

Author information

Authors and Affiliations

  1. School of Computing, DIT University, Dehradun, India

    Garima Verma

  2. Faculty of Management Studies, DIT University, Dehradun, India

    Hemraj Verma

Authors
  1. Garima Verma
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Hemraj Verma
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Garima Verma.

Ethics declarations

Conflict of interest

We hereby declare that this manuscript is an original work and is not under consideration for publication in any other journal. All authors have approved the submission of the manuscript the journal mentioned above.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Verma, G., Verma, H. Hybrid-Deep Learning Model for Emotion Recognition Using Facial Expressions. Rev Socionetwork Strat 14, 171–180 (2020). https://doi.org/10.1007/s12626-020-00061-6

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s12626-020-00061-6

Keywords

Search

Navigation