Skip to main content
Log in

Deep transfer learning in human–robot interaction for cognitive and physical rehabilitation purposes

  • Original Article
  • Published:
Pattern Analysis and Applications Aims and scope Submit manuscript

Abstract

This paper presents the extraction of the emotional signals from traumatic brain-injured (TBI) patients through the analysis of facial features and implementation of the effective emotion-recognition model through the Pepper robot to assist in the rehabilitation process. The identification of emotional cues from TBI patients is very challenging due to unique and diverse psychological, physiological, and behavioral challenges such as non-cooperation, facial/body paralysis, upper or lower limb impairments, cognitive, motor, and hearing skills inhibition. It is essential to read subtle changes in the emotional cues of TBI patients for effective communication and the development of affect-based systems. To analyze the variations of the emotional signal in TBI patients, a new database is collected in a natural and unconstrained environment from eleven residents of a neurological center in three different modalities, RGB, thermal and depth in three specified scenarios performing physical, cognitive and social communication rehabilitation activities. Due to the lack of labeled data, a deep transfer learning method is applied to efficiently classify emotions. The emotion classification model is tested through closed-field study and installment of a Pepper robot equipped with the trained model. Our deep trained and fine-tuned emotional recognition model composed of CNN-LSTM has improved the performance by 1.47% on MMI, and 4.96% on FER2013 validation data set. In addition, use of temporal information and transfer learning techniques to overcome TBI-data limitations has increased the performance efficacy on challenging dataset of neurologically impaired people. Findings that emerged from the study illustrate the noticeable effectiveness of SoftBank Pepper robot equipped with deep trained emotion recognition model in developing rehabilitation strategies by monitoring the TBI patient’s emotions. This research article presents the technical solution for real therapeutic robot interaction to rehabilitate patients with standard monitoring, assessment, and feedback in the neuro centers.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14

Similar content being viewed by others

Notes

  1. https://sites.google.com/site/emotiwchallenge/

  2. https://www.sundhed.dk/sundhedsfaglig/

    laegehaandbogen/undersoegelser-og-proever/skemaer/geriatri/mms-mini-mental-status/

  3. https://www.mocatest.org/

  4. https://www.ea.com/games/medal-of-honor

  5. https://www.ea.com/games/need-for-speed

References

  1. Adolphs R (2002) Neural systems for recognizing emotion. Curr Opin Neurobiol 12(2):169–177

    Article  Google Scholar 

  2. Albiol A, Monzo D, Martin A, Sastre J, Albiol A (2008) Face recognition using hog-ebgm. Pat Recognit Lett 29(10):1537–1543. https://doi.org/10.1016/j.patrec.2008.03.017

    Article  Google Scholar 

  3. Albuquerque VHC, Damaševičius R, Garcia NM, Pinheiro PR, et al. (2017) Brain computer interface systems for neurorobotics: methods and applications

  4. Balconi M (2012) Neuropsychology of facial expressions. the role of consciousness in processing emotional faces. Neuropsychol Trends 11:19–40

    Google Scholar 

  5. Barman A, Chatterjee A, Bhide R (2016) Cognitive impairment and rehabilitation strategies after traumatic brain injury. Indian J Psychol Med 38(3):172–181. https://doi.org/10.4103/0253-7176.183086

    Article  Google Scholar 

  6. Bellantonio M, Haque MA, Rodriguez P, Nasrollahi K, Telve T, Escalera S, Gonzalez J, Moeslund TB, Rasti P, Anbarjafari G (2017) Spatio-temporal pain recognition in CNN-based super-resolved facial images. Springer International Publishing, Cham, pp 151–162

    Google Scholar 

  7. Bemelmans R, Gelderblom GJ, Jonker P, De Witte L (2012) Socially assistive robots in elderly care: a systematic review into effects and effectiveness. J Am Med Dir Assoc 13(2):114–120

    Article  Google Scholar 

  8. Berretti S, Ben Amor B, Daoudi M, del Bimbo A (2011) 3d facial expression recognition using sift descriptors of automatically detected keypoints. Vis Comput 27(11):1021. https://doi.org/10.1007/s00371-011-0611-x

    Article  Google Scholar 

  9. Breuer R, Kimmel R (2017) A deep learning perspective on the origin of facial expressions. arXiv preprint arXiv:170501842

  10. Burgner-Kahrs J, Rucker DC, Choset H (2015) Continuum robots for medical applications: a survey. IEEE Trans Robot 31(6):1261–1280

    Article  Google Scholar 

  11. Cabibihan JJ, Javed H, Ang M, Aljunied SM (2013) Why robots? a survey on the roles and benefits of social robots in the therapy of children with autism. Int J Soc Robot 5(4):593–618

    Article  Google Scholar 

  12. Calvaresi D, Cesarini D, Sernani P, Marinoni M, Dragoni AF, Sturm A (2017) Exploring the ambient assisted living domain: a systematic review. J Ambient Intell Human Comput 8(2):239–257

    Article  Google Scholar 

  13. Chen L, Zhou M, Su W, Wu M, She J, Hirota K (2018) Softmax regression based deep sparse autoencoder network for facial emotion recognition in human–robot interaction. Inform Sci 428:49–61

    Article  MathSciNet  Google Scholar 

  14. Corneanu CA, Simón MO, Cohn JF, Guerrero SE (2016) Survey on rgb, 3d, thermal, and multimodal approaches for facial expression recognition: history, trends, and affect-related applications. IEEE Trans Pattern Anal Mach Intell 38(8):1548–1568

    Article  Google Scholar 

  15. Dhall A, Goecke R, Lucey S, Gedeon T (2011a) Acted facial expressions in the wild database. Australian National University, Canberra, Australia, Technical Report TR-CS-11 2:1

  16. Dhall A, Goecke R, Lucey S, Gedeon T (2011b) Static facial expression analysis in tough conditions: Data, evaluation protocol and benchmark. In: 2011 IEEE international conference on computer vision workshops (ICCV workshops), IEEE, pp 2106–2112

  17. Dhall A, Goecke R, Ghosh S, Joshi J, Hoey J, Gedeon T (2017) From individual to group-level emotion recognition: Emotiw 5.0. In: Proceedings of the 19th ACM international conference on multimodal interaction, pp 524–528

  18. Du S, Tao Y, Martinez AM (2014) Compound facial expressions of emotion. Proc Natl Acad Sci 111(15):E1454–E1462. https://doi.org/10.1073/pnas.1322355111

    Article  Google Scholar 

  19. Ekman P, Friesen WV, Ellsworth P (2013) Emotion in the human face: guidelines for research and an integration of findings, vol 11. Elsevier, Amsterdam

    Google Scholar 

  20. Elaklouk AM, Zin NAM, Shapii A (2015) Investigating therapists’ intention to use serious games for acquired brain injury cognitive rehabilitation. J King Saud Univ Comput Inf Sci 27(2):160–169

    Google Scholar 

  21. Fan Y, Lu X, Li D, Liu Y (2016) Video-based emotion recognition using cnn-rnn and c3d hybrid networks. In: Proceedings of the 18th ACM international conference on multimodal interaction, pp 445–450

  22. Friesen WV, Ekman P, et al. (1983) Emfacs-7: Emotional facial action coding system. Unpublished manuscript, University of California at San Francisco 2(36):1

  23. Goodfellow IJ, Erhan D, Carrier PL, Courville A, Mirza M, Hamner B, Cukierski W, Tang Y, Thaler D, Lee DH, et al. (2013) Challenges in representation learning: a report on three machine learning contests. In: International conference on neural information processing, Springer, pp 117–124

  24. Guo J, Lei Z, Wan J, Avots E, Hajarolasvadi N, Knyazev B, Kuharenko A, Junior JCSJ, Baró X, Demirel H et al (2018) Dominant and complementary emotion recognition from still images of faces. IEEE Access 6:26391–26403

    Article  Google Scholar 

  25. Hassner T, Harel S, Paz E, Enbar R (2015) Effective face frontalization in unconstrained images. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 4295–4304

  26. Hugentobler JA, Vegh M, Janiszewski B, Quatman-Yates C (2015) Physical therapy intervention strategies for patients with prolonged mild traumatic brain injury symptoms: a case series. Int J Sports Phys Therapy 10(5):676

    Google Scholar 

  27. Ilyas CMA, Haque MA, Rehm M, Nasrollahi K, Moeslund TB (2018a) Effective facial expression recognition through multimodal imaging for traumatic brain injured patient’s rehabilitation. In: International joint conference on computer vision. Springer, Imaging and Computer Graphics, pp 369–389

  28. Ilyas CMA, Nasrollahi K, Rehm M, Moeslund TB (2018b) Rehabilitation of traumatic brain injured patients: Patient mood analysis from multimodal video. In: 2018 25th IEEE international conference on image processing (ICIP), IEEE, pp 2291–2295

  29. Ilyas CMA, Schmuck V, Haque MA, Nasrollahi K, Rehm M, Moeslund TB (2019) Teaching pepper robot to recognize emotions of traumatic brain injured patients using deep neural networks. In: 28th IEEE international conference on robot and human interactive communication (roman)

  30. Jones M, Viola P (2003) Fast multi-view face detection. Mitsubishi Electric Research Lab TR-20003-96 3(14):2

  31. Kahou SE, Pal C, Bouthillier X, Froumenty P, Gülçehre Ç, Memisevic R, Vincent P, Courville A, Bengio Y, Ferrari RC, et al. (2013) Combining modality specific deep neural networks for emotion recognition in video. In: Proceedings of the 15th ACM on International conference on multimodal interaction, ACM, pp 543–550

  32. Karpathy A, Toderici G, Shetty S, Leung T, Sukthankar R, Fei-Fei L (2014) Large-scale video classification with convolutional neural networks. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 1725–1732

  33. Kaya H, Gürpinar F, Afshar S, Salah AA (2015) Contrasting and combining least squares based learners for emotion recognition in the wild. In: Proceedings of the 2015 ACM on international conference on multimodal interaction, pp 459–466

  34. Kim BK, Dong SY, Roh J, Kim G, Lee SY (2016) Fusing aligned and non-aligned face information for automatic affect recognition in the wild: a deep learning approach. In: Proceedings of the IEEE conference on computer vision and pattern recognition workshops, pp 48–57

  35. Kim Y, Yoo B, Kwak Y, Choi C, Kim J (2017) Deep generative-contrastive networks for facial expression recognition. arXiv preprint arXiv:170307140

  36. King DE (2009) Dlib-ml: a machine learning toolkit. J Mach Learn Res 10(Jul):1755–1758

    Google Scholar 

  37. Krishna NM, Sekaran K, Vamsi AVN, Ghantasala GP, Chandana P, Kadry S, Blažauskas T, Damaševičius R (2019) An efficient mixture model approach in brain-machine interface systems for extracting the psychological status of mentally impaired persons using eeg signals. IEEE Access 7:77905–77914

    Article  Google Scholar 

  38. Kulkarni K, Corneanu C, Ofodile I, Escalera S, Baro X, Hyniewska S, Allik J, Anbarjafari G (2018) Automatic recognition of facial displays of unfelt emotions. In: IEEE transactions on affective computing

  39. Kuo CM, Lai SH, Sarkis M (2018) A compact deep learning model for robust facial expression recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition workshops, pp 2121–2129

  40. Langhorne P, Bernhardt J, Kwakkel G (2011) Stroke rehabilitation. Lancet 377(9778):1693–1702

    Article  Google Scholar 

  41. Li S, Deng W (2020) Deep facial expression recognition: A survey. In: IEEE transactions on affective computing

  42. Li S, Deng W, Du J (2017) Reliable crowdsourcing and deep locality-preserving learning for expression recognition in the wild. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 2852–2861

  43. Liu M, Wang R, Li S, Shan S, Huang Z, Chen X (2014a) Combining multiple kernel methods on riemannian manifold for emotion recognition in the wild. In: Proceedings of the 16th international conference on multimodal interaction, ACM, pp 494–501

  44. Liu P, Han S, Meng Z, Tong Y (2014b) Facial expression recognition via a boosted deep belief network. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 1805–1812

  45. Liu X, Vijaya Kumar B, You J, Jia P (2017) Adaptive deep metric learning for identity-aware facial expression recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition workshops, pp 20–29

  46. Lucey P, Cohn JF, Kanade T, Saragih J, Ambadar Z, Matthews I (2010) The extended cohn-kanade dataset (ck+): A complete dataset for action unit and emotion-specified expression. In: 2010 IEEE computer society conference on computer vision and pattern recognition-workshops, IEEE, pp 94–101

  47. Lyons MJ, Akamatsu S, Kamachi M, Gyoba J, Budynek J (1998) The japanese female facial expression (jaffe) database. In: Proceedings of third international conference on automatic face and gesture recognition, pp 14–16

  48. Maskeliūnas R, Damaševičius R, Segal S (2019) A review of internet of things technologies for ambient assisted living environments. Future Internet 11(12):259

    Article  Google Scholar 

  49. Mavadati M, Sanger P, Mahoor MH (2016) Extended disfa dataset: Investigating posed and spontaneous facial expressions. In: Proceedings of the IEEE conference on computer vision and pattern recognition workshops, pp 1–8

  50. Mavadati SM, Mahoor MH, Bartlett K, Trinh P, Cohn JF (2013) Disfa: a spontaneous facial action intensity database. IEEE Trans Affect Comput 4(2):151–160

    Article  Google Scholar 

  51. McKenna K, Cooke DM, Fleming J, Jefferson A, Ogden S (2006) The incidence of visual perceptual impairment in patients with severe traumatic brain injury. Brain Injury 20(5):507–518. https://doi.org/10.1080/02699050600664368

    Article  Google Scholar 

  52. Meng Z, Liu P, Cai J, Han S, Tong Y (2017) Identity-aware convolutional neural network for facial expression recognition. In: 2017 12th IEEE international conference on automatic face and gesture recognition (FG 2017), IEEE, pp 558–565

  53. Mohammadi MR, Fatemizadeh E, Mahoor MH (2014) Pca-based dictionary building for accurate facial expression recognition via sparse representation. J Vis Commun Image Represent 25(5):1082–1092

    Article  Google Scholar 

  54. Müri RM (2016) Cortical control of facial expression. J Comp Neurol 524(8):1578–1585

    Article  Google Scholar 

  55. Ng HW, Nguyen VD, Vonikakis V, Winkler S (2015) Deep learning for emotion recognition on small datasets using transfer learning. In: Proceedings of the 2015 ACM on international conference on multimodal interaction, pp 443–449

  56. Oquab M, Bottou L, Laptev I, Sivic J (2014) Learning and transferring mid-level image representations using convolutional neural networks. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 1717–1724

  57. Otberdout N, Kacem A, Daoudi M, Ballihi L, Berretti S (2019) Automatic analysis of facial expressions based on deep covariance trajectories. In: IEEE transactions on neural networks and learning systems

  58. Pantic M, Valstar M, Rademaker R, Maat L (2005) Web-based database for facial expression analysis. In: 2005 IEEE international conference on multimedia and Expo, IEEE, p 5

  59. Parkhi OM, Vedaldi A, Zisserman A (2015) Deep face recognition

  60. Perry JC, Andureu J, Cavallaro FI, Veneman J, Carmien S, Keller T (2011) Effective game use in neurorehabilitation: user-centered perspectives. In: Handbook of research on improving learning and motivation through educational games: multidisciplinary approaches, IGI Global, pp 683–725

  61. Rapple L (2008) Lotsa helping hands. FOCUS J Respirat Care Sleep Med p 36

  62. Rees L, Marshall S, Hartridge C, Mackie D, Group MWFTE (2007) Cognitive interventions post acquired brain injury. Brain Injury 21(2):161–200. https://doi.org/10.1080/02699050701201813

    Article  Google Scholar 

  63. Robinson H, MacDonald B, Broadbent E (2014) The role of healthcare robots for older people at home: a review. Int J Soc Robot 6(4):575–591

    Article  Google Scholar 

  64. Rodil K, Rehm M, Krummheuer AL (2018) Co-designing social robots with cognitively impaired citizens. In: The 10th Nordic conference on human–computer interaction, association for computing machinery

  65. Rodriguez P, Cucurull G, Gonzàlez J, Gonfaus JM, Nasrollahi K, Moeslund TB, Roca FX (2017) Deep pain: exploiting long short-term memory networks for facial expression classification. IEEE Trans Cybern 99:1–11

    Google Scholar 

  66. Rudovic O, Lee J, Dai M, Schuller B, Picard RW (2018) Personalized machine learning for robot perception of affect and engagement in autism therapy. Sci Robot 3(19):eaao6760

    Article  Google Scholar 

  67. Šalkevicius J, Damaševičius R, Maskeliunas R, Laukien I (2019) Anxiety level recognition for virtual reality therapy system using physiological signals. Electronics 8(9):1039

    Article  Google Scholar 

  68. Sariyanidi E, Gunes H, Cavallaro A (2014) Automatic analysis of facial affect: a survey of registration, representation, and recognition. IEEE Trans Pat Anal Mach Intell 37(6):1113–1133

    Article  Google Scholar 

  69. Shan C, Gong S, McOwan PW (2009) Facial expression recognition based on local binary patterns: a comprehensive study. Image Vis Comput 27(6):803–816. https://doi.org/10.1016/j.imavis.2008.08.005

    Article  Google Scholar 

  70. Shapi’i A, Zin M, Azan N, Elaklouk AM (2015) A game system for cognitive rehabilitation. In: BioMed research international 2015

  71. Stern Y (2009) Cognitive reserve. Neuropsychologia 47(10):2015–2028

    Article  Google Scholar 

  72. Sun N, Li Q, Huan R, Liu J, Han G (2019) Deep spatial-temporal feature fusion for facial expression recognition in static images. Pat Recognit Lett 119:49–61

    Article  Google Scholar 

  73. Sutton M (2012) Apps to aid aphasia. ASHA Leader 17(7):32, https://search.proquest.com/docview/1022993653

  74. Tang Y (2013) Deep learning using support vector machines. CoRR abs/1306.0239, arXiv:1306.0239

  75. Taylor RH, Menciassi A, Fichtinger G, Fiorini P, Dario P (2016) Medical robotics and computer-integrated surgery. In: Springer handbook of robotics, Springer, pp 1657–1684

  76. Tian YI, Kanade T, Cohn JF (2001) Recognizing action units for facial expression analysis. IEEE Trans Pat Anal Mach Intell 23(2):97–115

    Article  Google Scholar 

  77. Tsaousides T, Gordon WA (2009) Cognitive rehabilitation following traumatic brain injury: assessment to treatment. Mount Sinai J Med J Trans Personal Med 76(2):173–181. https://doi.org/10.1002/msj.20099

    Article  Google Scholar 

  78. Uddin MZ, Hassan MM, Almogren A, Alamri A, Alrubaian M, Fortino G (2017) Facial expression recognition utilizing local direction-based robust features and deep belief network. IEEE Access 5:4525–4536

    Article  Google Scholar 

  79. Valstar M, Pantic M (2010) Induced disgust, happiness and surprise: an addition to the mmi facial expression database. In: Proceedings of 3rd international workshop on EMOTION (satellite of LREC): Corpora for Research on Emotion and Affect, Paris, France, p 65

  80. Wan J, Escalera S, Anbarjafari G, Jair Escalante H, Baró X, Guyon I, Madadi M, Allik J, Gorbova J, Lin C, et al. (2017) Results and analysis of chalearn lap multi-modal isolated and continuous gesture recognition, and real versus fake expressed emotions challenges. In: Proceedings of the IEEE international conference on computer vision, pp 3189–3197

  81. Yan J, Zheng W, Cui Z, Tang C, Zhang T, Zong Y (2018) Multi-cue fusion for emotion recognition in the wild. Neurocomputing 309:27–35

    Article  Google Scholar 

  82. Yao A, Shao J, Ma N, Chen Y (2015) Capturing au-aware facial features and their latent relations for emotion recognition in the wild. In: Proceedings of the 2015 ACM on international conference on multimodal interaction, ACM, pp 451–458

  83. Yao A, Cai D, Hu P, Wang S, Sha L, Chen Y (2016) Holonet: towards robust emotion recognition in the wild. In: Proceedings of the 18th ACM international conference on multimodal interaction, pp 472–478

  84. Yin L, Wei X, Sun Y, Wang J, Rosato MJ (2006) A 3d facial expression database for facial behavior research. In: 7th international conference on automatic face and gesture recognition (FGR06), IEEE, pp 211–216

  85. Yu Z, Zhang C (2015) Image based static facial expression recognition with multiple deep network learning. In: Proceedings of the 2015 ACM on international conference on multimodal interaction, pp 435–442

  86. Yu Z, Liu Q, Liu G (2018) Deeper cascaded peak-piloted network for weak expression recognition. Vis Comput 34(12):1691–1699

    Article  Google Scholar 

  87. Zeng Z, Pantic M, Roisman GI, Huang TS (2008) A survey of affect recognition methods: audio, visual, and spontaneous expressions. IEEE Trans Pat Anal Mach Intell 31(1):39–58

    Article  Google Scholar 

  88. Zhang K, Zhang Z, Li Z, Qiao Y (2016) Joint face detection and alignment using multitask cascaded convolutional networks. IEEE Signal Process Lett 23(10):1499–1503

    Article  Google Scholar 

  89. Zhang K, Huang Y, Du Y, Wang L (2017) Facial expression recognition based on deep evolutional spatial-temporal networks. IEEE Trans Image Process 26(9):4193–4203

    Article  MathSciNet  Google Scholar 

  90. Zhang X, Yin L, Cohn JF, Canavan S, Reale M, Horowitz A, Liu P (2013) A high-resolution spontaneous 3d dynamic facial expression database. In: 2013 10th IEEE international conference and workshops on automatic face and gesture recognition (FG), IEEE, pp 1–6

  91. Zhang Z, Luo P, Loy CC, Tang X (2018) From facial expression recognition to interpersonal relation prediction. Int J Comput Vis 126(5):550–569

    Article  MathSciNet  Google Scholar 

  92. Zhao X, Zhang S (2011) Facial expression recognition based on local binary patterns and kernel discriminant isomap. Sensors 11(10):9573–9588

    Article  Google Scholar 

  93. Zhao X, Liang X, Liu L, Li T, Han Y, Vasconcelos N, Yan S (2016) Peak-piloted deep network for facial expression recognition. In: European conference on computer vision, Springer, pp 425–442

Download references

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Chaudhary Muhammad Aqdus Ilyas.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Ilyas, C.M.A., Rehm, M., Nasrollahi, K. et al. Deep transfer learning in human–robot interaction for cognitive and physical rehabilitation purposes. Pattern Anal Applic 25, 653–677 (2022). https://doi.org/10.1007/s10044-021-00988-8

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10044-021-00988-8

Keywords

Navigation