Skip to main content
SpringerLink
Log in

A hybrid deep learning neural approach for emotion recognition from facial expressions for socially assistive robots

  • S.I. : EANN 2016
  • Published:
Neural Computing and Applications Aims and scope Submit manuscript

Abstract

We have recently seen significant advancements in the development of robotic machines that are designed to assist people with their daily lives. Socially assistive robots are now able to perform a number of tasks autonomously and without human supervision. However, if these robots are to be accepted by human users, there is a need to focus on the form of human–robot interaction that is seen as acceptable by such users. In this paper, we extend our previous work, originally presented in Ruiz-Garcia et al. (in: Engineering applications of neural networks: 17th international conference, EANN 2016, Aberdeen, UK, September 2–5, 2016, proceedings, pp 79–93, 2016. https://doi.org/10.1007/978-3-319-44188-7_6), to provide emotion recognition from human facial expressions for application on a real-time robot. We expand on previous work by presenting a new hybrid deep learning emotion recognition model and preliminary results using this model on real-time emotion recognition performed by our humanoid robot. The hybrid emotion recognition model combines a Deep Convolutional Neural Network (CNN) for self-learnt feature extraction and a Support Vector Machine (SVM) for emotion classification. Compared to more complex approaches that use more layers in the convolutional model, this hybrid deep learning model produces state-of-the-art classification rate of \(96.26\%\), when tested on the Karolinska Directed Emotional Faces dataset (Lundqvist et al. in The Karolinska Directed Emotional Faces—KDEF, 1998), and offers similar performance on unseen data when tested on the Extended Cohn–Kanade dataset (Lucey et al. in: Proceedings of the third international workshop on CVPR for human communicative behaviour analysis (CVPR4HB 2010), San Francisco, USA, pp 94–101, 2010). This architecture also takes advantage of batch normalisation (Ioffe and Szegedy in Batch normalization: accelerating deep network training by reducing internal covariate shift. http://arxiv.org/abs/1502.03167, 2015) for fast learning from a smaller number of training samples. A comparison between Gabor filters and CNN for feature extraction, and between SVM and multilayer perceptron for classification is also provided.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7

Similar content being viewed by others

References

  1. Scassellati B, Admoni H, Mataric M (2012) Robots for use in autism research. Annu Rev Biomed Eng 14:275–294

    Article  Google Scholar 

  2. Fasola J, Mataric M (2013) A socially assistive robot exercise coach for the elderly. J Hum Robot Interact 2(2):3–32

    Article  Google Scholar 

  3. Chang W-L, Šabanovic S, Huber L (2013) Use of seal-like robot PARO in sensory group therapy for older adults with dementia. In: Proceedings of the 8th ACM/IEEE international conference on Human-robot interaction. IEEE Press

  4. Soler MV, Agüera-Ortiz L, Rodrguez JO, Rebolledo CM, Muoz AP, Prez IR, Ruiz EO, Snchez AB, Cano VH, Chilln LC, Ruiz SF (2015) Social robots in advanced dementia. Front Aging Neurosci 3:133. https://doi.org/10.3389/fnagi.2015.00133

    Google Scholar 

  5. Ruiz-Garcia A, Elshaw M, Altahhan A, Palade V (2016) Emotion recognition using facial expression images for a robotic companion. In: Engineering applications of neural networks: 17th international conference, EANN 2016, Aberdeen, UK, September 2–5, 2016, proceedings, pp 79–93. doi:https://doi.org/10.1007/978-3-319-44188-7_6

  6. Ioffe S, Szegedy C (2015) Batch normalization: accelerating deep network training by reducing internal covariate shift. http://arxiv.org/abs/1502.03167

  7. Lundqvist D, Flykt A, Öhman A (1998) The Karolinska Directed Emotional Faces—KDEF. CD ROM from Department of Clinical Neuroscience, Psychology section, Karolinska Institutet. ISBN 91-630-7164-9

  8. Lucey P, Cohn JF, Kanade T, Saragih J, Ambadar Z, Matthews I (2010) The extended Cohn–Kanade dataset (CK+): a complete expression dataset for action unit and emotion-specified expression. In: Proceedings of the third international workshop on CVPR for human communicative behavior analysis (CVPR4HB 2010), San Francisco, USA, pp 94–101

  9. Robertson J (2014) Human rights vs. robot rights: forecasts from Japan. Crit Asian Stud 46(4):571–598. doi:https://doi.org/10.1080/14672715.2014.960707

    Article  Google Scholar 

  10. Dahl T, Boulos M (2013) Robots in health and social care: a complementary technology to home care and telehealthcare? Robotics 3:1–21

    Article  Google Scholar 

  11. Castellano G, Paiva A, Kappas A, Nabais F, Aylett R, Barendregt W, Hastie H, Bull S (2013) Towards empathic virtual and robotic tutors. In: Lecture notes in computer science, pp 733–736

  12. Toh LPE, Causo A, Tzuo PW, Chen IM, Yeo SH, Link S (2016) A review on the use of robots in education and young children. Sci Direct 19:148–163

    Google Scholar 

  13. Kory Westlund J, Gordon G, Spaulding S, Lee J, Plummer L, Martinez M, Das M, Breazeal C (2015) Learning a second language with a socially assistive robot. In: The 1st international conference on social robots in therapy and education, 2015, Almere, The Netherlands. https://www.media.mit.edu/publications/learning-a-second-language-with-a-socially-assistive-robot/

  14. Affectiva (2016). http://www.affectiva.com/

  15. Kim ES, Berkovits LD, Bernier EP, Leyzberg D, Shic F, Paul R, Scassellati B (2013) Social robots as embedded reinforcers of social behavior in children with autism. J Autism Dev Disord 43(5):1038–1049

    Article  Google Scholar 

  16. Rabbitt SM, Kazdin AE, Hong JH (2015) Acceptability of robot-assisted therapy for disruptive behavior problems in children. Arch Sci Psychol 3(1):101–110. https://doi.org/10.1037/arc0000017

  17. KSERA (2016) Knowledgeable service robots for aging. http://www.aat.tuwien.ac.at/ksera/index_en.html

  18. GeriJoy (2016) Care and companionship for seniors—GeriJoy. http://www.gerijoy.com/

  19. Leite I, Pereira A, Mascarenhas S, Martinho C, Prada R, Paiva A (2013) The influence of empathy in humanrobot relations. Int J Hum Comput Stud 71(3):250–260

    Article  Google Scholar 

  20. Graaf MMA, Ben Allouch S, Dijk JAGM (2016) Long-term acceptance of social robots in domestic environments: insights from a user’s perspective. In: AAAI

  21. Duffy BR (2006) Fundamental issues in social robotics. Int Rev Inf Eth 6:31–36

  22. Boughrara H, Chtourou M, Ben Amar C, Chen L (2014) Facial expression recognition based on a MLP neural network using constructive training algorithm. Multimed Tools Appl 75:709–731

    Article  Google Scholar 

  23. Kahou S, Michalski V, Konda K, Memisevic R, Pal C (2015) Recurrent neural networks for emotion recognition in video. In: Proceedings of the 2015 ACM on international conference on multimodal interaction (ICMI ’15), pp 467–474

  24. Lawrence S, Giles CL, Tsoi AC, Back AD (1997) Face recognition: a convolutional neural-network approach. IEEE Trans Neural Netw 8(1):98–113. http://ieeexplore.ieee.org/lpdocs/epic03/wrapper.htm?arnumber=554195

  25. Gupta A, Garg M (2016) A human emotion recognition system using supervised self-organising maps. In: 2014 International conference on computing for sustainable global development (INDIACom), pp 654–659

  26. Sarnarawickrame K, Mindya S (2013) Facial expression recognition using active shape models and support vector machines. In: 2013 International conference on advances in ICT for emerging regions (ICTer), pp 51–55

  27. Sohail ASM, Bhattacharya P (2011) Classifying facial expressions using level set method based lip contour detection and multi-class support vector machines. Int J Pattern Recognit Artif Intell 25(6):835–862. doi:https://doi.org/10.1142/S0218001411008762

    Article  MathSciNet  Google Scholar 

  28. Ouellet S (2014) Real-time emotion recognition for gaming using deep convolutional network features. CoRR abs/1408.3, 6. http://arxiv.org/abs/1408.3750

  29. Ahsan T, Jabid T, Chong U-P (2013) Facial expression recognition using local transitional pattern on gabor filtered facial images. IETE Tech Rev 30(12):47. http://tr.ietejournals.org/article.asp?issn=0256-4602;year=2013;volume=30;issue=1;spage=47;epage=52;aulast=Ahsan l slightly outperforms Gabor filters as a feature extraction method, at least for this particular dataset

  30. Chelali FZ, Djeradi A (2015) Face recognition using MLP and RBF neural network with Gabor and discrete wavelet transform characterization: a comparative study. Math Probl Eng 2015:1–16. http://www.hindawi.com/journals/mpe/2015/523603/

  31. Mehta N, Jadhav S (2016) Facial emotion recognition using log Gabor filter and PCA. In: Proceedings of international conference on computing communication control and automation (ICCUBEA), pp 1–5

  32. Kanade T, Cohn JF, Tian Y (2000) Comprehensive database for facial expression analysis. In: Proceedings of the fourth IEEE international conference on automatic face and gesture recognition (FG’00), Grenoble, France, pp 46–53

  33. Paul PP, Monwar MM, Gavrilova ML, Wang PSP (2010) Rotation invariant multivew face detection using skin color regressive model and support vector regression. Int J Pattern Recognit Artif Intell 24(8):1261–1280. https://doi.org/10.1142/S0218001410008391

    Article  Google Scholar 

  34. Khan SA, Hussain A, Usman M, Nazir M, Riaz N, Mirza AM (2014) Robust face recognition using computationally efficient features. J Intell Fuzzy Syst 27(6):3131–3143.<GotoISI>://WOS:000345981600037

  35. Hassner T, Harel S, Paz E, Enbar R (2015) Effective face frontalization in unconstrained images. In: 2015 IEEE conference on computer vision and pattern recognition (CVPR), pp 4295–4304. http://ieeexplore.ieee.org/lpdocs/epic03/wrapper.htm?arnumber=7299058

  36. Hewahi NM, Baraka ARM (2012) Impact of ethnic group on human emotion recognition using backpropagation neural network. BRAIN. Broad Res Artif 20–27. http://brain.edusoft.ro/index.php/brain/article/view/284

  37. Khashman A (2009) Application of an emotional neural network to facial recognition. Neural Comput Appl 18(4):309–320

    Article  Google Scholar 

  38. Burkert P, Trier F, Afzal MZ, Dengel A, Liwicki M (2015) DeXpression: Deep Convolutional Neural Network for expression recognition. arXiv preprint 1–8. http://arxiv.org/abs/1509.05371

  39. Beaudry O, Roy-Charland A, Perron M, Cormier I, Tapp R (2014) Featural processing in recognition of emotional facial expressions. Cognit Emot 28(3):416–432. http://www.ncbi.nlm.nih.gov/pubmed/24047413

  40. Kuhn H, Tucker A (1951) Nonlinear programming. In: Proceedings of the second Berkeley symposium on mathematical statistics and probability, pp 481–492

  41. Aldebaran (2016) Who is NAO? https://www.aldebaran.com/en/cool-robots/nao

  42. Tan CC, Eswaran C (2010) Reconstruction and recognition of face and digit images using autoencoders. Neural Comput Appl 19(7):1069–1079

    Article  Google Scholar 

Download references

Acknowledgements

We would like to thank the School of Computing, Electronics and Mathematics, at Coventry University for funding this research. We would like to also acknowledge the contribution of the Barry Gidden fund which partially funded this work. The authors would also like to acknowledge the invaluable contribution of Maria Charalambous, Kenny Ruiz, Ibrahim Alamakky and Danielle Labhardt in creating this paper.

Author information

Authors and Affiliations

  1. School of Computing, Electronics and Mathematics, Faculty of Engineering, Environment and Computing, Coventry University, Coventry, UK

    Ariel Ruiz-Garcia, Mark Elshaw, Abdulrahman Altahhan & Vasile Palade

Authors
  1. Ariel Ruiz-Garcia
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Mark Elshaw
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Abdulrahman Altahhan
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Vasile Palade
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Ariel Ruiz-Garcia.

Ethics declarations

Conflict of interest

The authors declare that they have no conflict of interest.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Ruiz-Garcia, A., Elshaw, M., Altahhan, A. et al. A hybrid deep learning neural approach for emotion recognition from facial expressions for socially assistive robots. Neural Comput & Applic 29, 359–373 (2018). https://doi.org/10.1007/s00521-018-3358-8

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00521-018-3358-8

Keywords

Search

Navigation