Deep Learning for Emotion Recognition in Faces

  • Ariel Ruiz-GarciaEmail author
  • Mark Elshaw
  • Abdulrahman Altahhan
  • Vasile Palade
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 9887)


Deep Learning (DL) has shown real promise for the classification efficiency for emotion recognition problems. In this paper we present experimental results for a deeply-trained model for emotion recognition through the use of facial expression images. We explore two Convolutional Neural Network (CNN) architectures that offer automatic feature extraction and representation, followed by fully connected softmax layers to classify images into seven emotions. The first architecture explores the impact of reducing the number of deep learning layers and the second splits the input images horizontally into two streams based on eye and mouth positions. The first proposed architecture produces state of the art results with an accuracy rate of 96.93 % and the second architecture with split input produces an average accuracy rate of 86.73 %, respectively.


Deep learning Convolution neural networks Emotion recognition Empathic robots 


  1. 1.
    Lewis, M., Haviland-Jones, J., Barrett, L.: Handbook of Emotions. Guilford Press, New York (2008)Google Scholar
  2. 2.
    Chavhan, A., Chavan, S., Dahe, S., Chibhade, S.: A neural network approach for real time emotion recognition. IJARCCE 4(3), 259–263 (2015)CrossRefGoogle Scholar
  3. 3.
    Han, K., Yu, D., Tashev, I.: Speech emotion recognition using deep neural network and extreme learning machine. In: Interspeech, pp. 223–227 (2014)Google Scholar
  4. 4.
    Cohen, I., Garg, A., Huang, T.: Emotion recognition from facial expressions using multi-level HMM. In: Neural Information Processing Systems, vol. 2 (2000)Google Scholar
  5. 5.
    Sarnarawickrame, K., Mindya, S.: Facial expression recognition using active shape models and support vector machines. In: 2013 International Conference on Advances in ICT for Emerging Regions (ICTer), pp. 51–55 (2013)Google Scholar
  6. 6.
    Boughrara, H., Chtourou, M., Ben Amar, C., Chen, L.: Facial expression recognition based on a mlp neural network using constructive training algorithm. Multimed. Tools Appl. 75, 709–731 (2014)CrossRefGoogle Scholar
  7. 7.
    Kahou, S., Michalski, V., Konda, K., Memisevic, R., Pal, C.: Recurrent neural networks for emotion recognition in video. In: Proceedings of the 2015 ACM on International Conference on Multimodal Interaction (ICMI 2015), pp. 467–474 (2015)Google Scholar
  8. 8.
    Levi, G., Hassner, T.: Emotion recognition in the wild via convolutional neural networks and mapped binary patterns. In: Proceedings of the 2015 ACM on International Conference on Multimodal Interaction (ICMI 2015), pp. 503–510 (2015)Google Scholar
  9. 9.
    Ouellet, S.: Realtime emotion recognition for gaming using deep convolutional network features. CoRR. abs/1408.3750 (2014)Google Scholar
  10. 10.
    Szegedy, C., Lui, W., Jia, Y., Sermanet, P., Reed, S., Auguelov, D., Erhan, D., Vanhoucke, V., Rabinovich, A.: Going deeper with convolutions. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1–19 (2014)Google Scholar
  11. 11.
    Krizhevsky, A., Sutskever, I., Hinton, G.: Imagenet classification with deep convolutional neural networks. Adv. Neural Inf. Process. Syst. 25, 1106–1114 (2012)Google Scholar
  12. 12.
    Burkert, P., Trier, F., Afzal, M.Z., Dengel, A., Liwicki, M.: DeXpression: Deep Convolutional Neural Network for Expression Recognition. CoRR. abs/1509.05371 (2015)Google Scholar
  13. 13.
    Lawrence, S., Giles, C., Tsoi, A.C., Back, A.: Face recognition: a convolutional neural network approach. IEEE Trans. Neural Netw. 8, 98–113 (1997)CrossRefGoogle Scholar
  14. 14.
    Brosch, T., Tam, R.: Efficient training of convolutional deep belief networks in the frequency domain for application to high-resolution 2D and 3D images. Neural Computation. 27, 211–227 (2015)CrossRefGoogle Scholar
  15. 15.
    Altahhan, A.: Navigating a robot through big visual sensory data. Procedia Comput. Sci. 53, 478–485 (2015)CrossRefGoogle Scholar
  16. 16.
    Khashman, A.: Application of an emotional neural network to facial recognition. Neural Comput. Appl. 18, 309–320 (2008)CrossRefGoogle Scholar
  17. 17.
    Sohail, A., Bhattacharya, P.: Classifying facial expressions using level set method based lip contour detection and multi-class support vector machines. Int. J. Pattern Recogn. Artif. Intell. 25, 835–862 (2011)MathSciNetCrossRefGoogle Scholar
  18. 18.
    Hewahi, N., Baraka, A.: Impact of ethnic group on human emotion recognition using backpropagation neural network. Broad Res. Artif. Intell. Neurosci. 2, 20–27 (2011)Google Scholar
  19. 19.
    Ahsan, T., Jabid, T., Chong, U.: Facial expression recognition using local transitional pattern on gabor filtered facial images. IETE Tech Rev. 30, 47 (2013)CrossRefGoogle Scholar
  20. 20.
    Chelali, F., Djeradi, A.: Face recognition using MLP and RBF neural network with Gabor and discrete wavelet transform characterization: a comparative study. Math. Prob. Eng. 2015, 116 (2015)CrossRefGoogle Scholar
  21. 21.
    Lundqvist, D., Flykt, A., Ahman, A.: The Karolinska Directed Emotional Faces - KDEF. CD ROM from Department of Clinical Neuroscience, Psychology section, Karolinska Institutet (1998). ISBN 91-630-7164-9Google Scholar

Copyright information

© Springer International Publishing Switzerland 2016

Authors and Affiliations

  • Ariel Ruiz-Garcia
    • 1
    Email author
  • Mark Elshaw
    • 1
  • Abdulrahman Altahhan
    • 1
  • Vasile Palade
    • 1
  1. 1.Faculty of Engineering, Environment and Computing, School of Computing, Electronics and MathematicsCoventry UniversityCoventryUK

Personalised recommendations