Acceptance and use of a multi-modal avatar-based tool for remediation of social cognition deficits

  • Arturo S. García
  • Patricia Fernández-Sotos
  • Antonio Fernández-CaballeroEmail author
  • Elena Navarro
  • José M. Latorre
  • Roberto Rodriguez-Jimenez
  • Pascual González
Original Research


This paper focuses on the validation of a tool designed to improve affect recognition, a fundamental aspect of social cognition as it greatly affects the functionality and quality of life of patients with mental disorders. The presented tool facilitates the generation of multi-modal avatar-based therapies by mental health professionals in this important clinical domain. Moreover, the tool for remediation of social cognitive deficits may be customised to each patient’s impairment. This paper describes how the tool was assessed by therapists after viewing a video explaining its most relevant aspects. The participants were asked to fill in a questionnaire based on UTAUT2 for the study of the acceptance and use of this technology. In light of the results obtained from 41 therapists about their intention of use, the most important statement is that their interest for this kind of tools is high. Nonetheless, there are some factors that negatively affect their behavioural intention.


Social cognition Affect recognition Virtual reality Avatar 



This work was partially supported by Spanish Ministerio de Ciencia, Innovación y Universidades, Agencia Estatal de Investigación (AEI)/European Regional Development Fund (FEDER, UE) under EmoBioFeedback (DPI2016-80894-R), Vi-SMARt (TIN2016-79100-R) and HA-SYMBIOSIS (TIN2015-72931-EXP) Grants, and by Biomedical Research Networking Centre in Mental Health (CIBERSAM) of the Instituto de Salud Carlos III.

Supplementary material

Supplementary material 1 (mp4 119430 KB)


  1. Abirached B, Zhang Y, Aggarwal JK, Fernandes T, Carlos J, Orvalho V (2011) Improving communication skills of children with ASDs through interaction with virtual characters. In: IEEE 1st international conference on serious games and applications for health, IEEE, pp 3493–3502Google Scholar
  2. Adery LH, Ichinose M, Torregrossa LJ, Wade J, Nichols H, Bekele E, Bian D, Gizdic A, Granholm E, Sarkar N, Park S (2018) The acceptability and feasibility of a novel virtual reality based social skills training game for schizophrenia: preliminary findings. Psychiatry Res 270:496–502. CrossRefGoogle Scholar
  3. Akalin N, Kiselev A, Kristoffersson A, Loutfi A (2017) An evaluation tool of the effect of robots in eldercare on the sense of safety and security. In: Kheddar A, Yoshida E, Ge SS, Suzuki K, Cabibihan J-J, Eyssel F, He H (eds) International conference on social robotics. Springer, Cham, pp 628–637CrossRefGoogle Scholar
  4. Barclay D, Higgins C, Thompson R (1995) The partial least squares (PLS) approach to casual modeling: personal computer adoption ans use as an illustration. Technol Stud 2:285–309Google Scholar
  5. Baron-Cohen S, Wheelwright S, Skinner R, Martin J, Clubley E (2001) The autism-spectrum quotient (AQ): evidence from asperger syndrome/high-functioning autism, malesand females, scientists and mathematicians. J Autism Dev Disord 31(1):5–17. CrossRefGoogle Scholar
  6. Beer JM, Fisk AD, Rogers WA (2009) Emotion recognition of virtual agents facial expressions: the effects of age and emotion intensity. Proc Hum Factors Ergon Soc 53(2):131–135. CrossRefGoogle Scholar
  7. Brothers L (1990) The neural basis of primate social communication. Motiv Emot 14(2):81–91. CrossRefGoogle Scholar
  8. Castillo JC, Castro-González Á, Fernández-Caballero A, Latorre JM, Pastor JM, Fernández-Sotos A, Salichs MA (2016) Software architecture for smart emotion recognition and regulation of the ageing adult. Cogn Comput 8(2):357–367. CrossRefGoogle Scholar
  9. Cerezo E, Hupont I, Baldassarri S, Ballano S (2012) Emotional facial sensing and multimodal fusion in a continuous 2d affective space. J Ambient Intell Hum Comput 3(1):31–46. CrossRefGoogle Scholar
  10. Fernández-Caballero A, Latorre JM, Pastor JM, Fernández-Sotos A (2014) Improvement of the elderly quality of life and care through smart emotion regulation. In: Pecchia L, Chen LL, Nugent C, Bravo J (eds) Ambient assisted living and daily activities. Springer, Cham, pp 348–355. CrossRefGoogle Scholar
  11. Fernández-Caballero A, Martínez-Rodrigo A, Pastor JM, Castillo JC, Lozano-Monasor E, López MT, Zangr’oniz R, Latorre JM, Fernández-Sotos A (2016) Smart environment architecture for emotion detection and regulation. J Biomed Inform 64:55–73. CrossRefGoogle Scholar
  12. Fernández-Caballero A, Navarro E, Fernández-Sotos P, González P, Ricarte JJ, Latorre JM, Rodriguez-Jimenez R (2017) Human-avatar symbiosis for the treatment of auditory verbal hallucinations in schizophrenia through virtual/augmented reality and brain–computer interfaces. Front Neuroinform 11:64. CrossRefGoogle Scholar
  13. Fernández-Sotos P, Navarro E, Torio I, Dompablo M, Fernández-Caballero A, Rodriguez-Jimenez R (2018) Pharmacological interventions in social cognition deficits: a systematic mapping review. Psychiatry Res 270:57–67. CrossRefGoogle Scholar
  14. Fernández-Sotos P, Torio I, Fernández-Caballero A, Navarro E, González P, Dompablo M, Rodriguez-Jimenez R (2019) Social cognition remediation interventions: a systematic mapping review. PLoS One 14:e0218720CrossRefGoogle Scholar
  15. García AS, Molina JP, Martínez D, González P (2008) Enhancing collaborative manipulation through the use of feedback and awareness in CVEs. In: Proceedings of the 7th ACM SIGGRAPH international conference on virtual-reality continuum and its applications in industry. ACM, New York, p 32. Google Scholar
  16. García AS, Navarro E, Fernández-Caballero A, González P (2018) Towards the design of avatar-based therapies for enhancing facial affect recognition. In: Novais P, Jung JJ, Villarrubia G, Fernández-Caballero A, Navarro E, González P, Carneiro D, Pinto A, Campbell AT, Durães D (eds) Ambient intelligence—software and applications, 9th international symposium on ambient intelligence. Springer, Cham, pp 306–313Google Scholar
  17. Garrido MV, Lopes D, Prada M, Rodrigues D, Jerónimo R, Mourão RP (2016) The many faces of a face: comparing stills and videos of facial expressions in eight dimensions (SAVE database). Behav Res Methods 49(4):1343–1360. CrossRefGoogle Scholar
  18. Gottesman II (1991) Schizophrenia genesis: the origins of madness. WH Freeman/Times Books/Henry Holt & Co, New YorkGoogle Scholar
  19. Green MF, Penn DL, Bentall R, Carpenter WT, Gaebel W, Gur RC, Kring AM, Park S, Silverstein SM, Heinssen R (2008) Social cognition in schizophrenia: an NIMH workshop on definitions, assessment, and research opportunities. Schizophr Bull 34(6):1211–1220. CrossRefGoogle Scholar
  20. Hair JF Jr, Hult GTM, Ringle C, Sarstedt M (2014) A primer on partial least squares structural equation modeling (PLS-SEM). Sage Publications, Thousand OakszbMATHGoogle Scholar
  21. Henseler J, Ringle CM, Sarstedt M (2015) A new criterion for assessing discriminant validity in variance-based structural equation modeling. J Acad Mark Sci 43(1):115–135CrossRefGoogle Scholar
  22. Kandalaft MR, Didehbani N, Krawczyk DC, Allen TT, Chapman SB (2013) Virtual reality social cognition training for young adults with high-functioning autism. J Autism Dev Disord 43(1):34–44. CrossRefGoogle Scholar
  23. Khalilzadeh J, Ozturk AB, Bilgihan A (2017) Security-related factors in extended UTAUT model for NFC based mobile payment in the restaurant industry. Comput Human Behav 70:460–474CrossRefGoogle Scholar
  24. Krishnappa Babu PR, Lahiri U (2019) Classification approach for understanding implications of emotions using eye-gaze. J Ambient Intell Hum Comput. Google Scholar
  25. Lahera G, Ruiz-Murugarren S, Iglesias P, Ruiz-Bennasar C, Herrería E, Montes JM, Fernández-Liria A (2012) Social cognition and global functioning in bipolar disorder. J Nerv Menta Dis 200(2):135–141. CrossRefGoogle Scholar
  26. Laumer S, Gubler F, Maier C, Weitzel T (2018) Job seekers’ acceptance of job recommender systems: results of an empirical study. In: Proceedings of the 51st Hawaii international conference on system sciences. Curran Associates, Inc., pp 3172–3181Google Scholar
  27. Lozano-Monasor E, López MT, Vigo-Bustos F, Fernández-Caballero A (2017) Facial expression recognition in ageing adults: from lab to ambient assisted living. J Ambient Intell Hum Comput 8(4):567–578. CrossRefGoogle Scholar
  28. Manis KT, Choi D (2018) The virtual reality hardware acceptance model (vr-ham): Extending and individuating the technology acceptance model (TAM) for virtual reality hardware. J Bus Res. Google Scholar
  29. Marcoulides GA, Saunders C (2006) PLS: a silver bullet? Manag Inf Syst Q 30(2):1CrossRefGoogle Scholar
  30. McDuff D, Mahmoud A, Mavadati M, Amr M, Turcot J, Kaliouby Re (2016) Affdex sdk: a cross-platform real-time multi-face expression recognition toolkit. In: Proceedings of the 2016 CHI conference extended abstracts on human factors in computing systems. ACM, pp 3723–3726Google Scholar
  31. Mendoza-Palechor F, Menezes ML, Sant’Anna A, Ortiz-Barrios M, Samara A, Galway L (2018) Affective recognition from EEG signals: an integrated data-mining approach. J Ambient Intell Hum Comput. Google Scholar
  32. Oechslein O, Fleischmann M, Hess T (2014) An application of utaut2 on social recommender systems: incorporating social information for performance expectancy. In: 2014 47th Hawaii international conference on system sciences. IEEE, pp 3297–3306Google Scholar
  33. Oliver M, Teruel M, Molina J, Romero-Ayuso D, González P (2018) Ambient intelligence environment for home cognitive telerehabilitation. Sensors 18(11):3671CrossRefGoogle Scholar
  34. Pinkham AE, Penn DL, Green MF, Buck B, Healey K, Harvey PD (2014) The social cognition psychometric evaluation study: results of the expert survey and RAND panel. Schizophr Bull 40(4):813–823. CrossRefGoogle Scholar
  35. Pinkham AE, Penn DL, Green MF, Harvey PD (2016) Social cognition psychometric evaluation: results of the initial psychometric study. Schizophr Bull 42(2):494–504. CrossRefGoogle Scholar
  36. Roark DA, Barrett SE, Spence MJ, Abdi H, O’Toole AJ (2003) Psychological and neural perspectives on the role of motion in face recognition. Behav Cogn Neurosci Rev 2(1):15–46. CrossRefGoogle Scholar
  37. Rosenberg H, McDonald S, Dethier M, Kessels RP, Westbrook RF (2014) Facial emotion recognition deficits following moderate-severe traumatic brain injury (TBI): re-examining the valence effect and the role of emotion intensity. J Int Neuropsychol Soc 20(10):994–1003. CrossRefGoogle Scholar
  38. Rus-Calafell M, Gutiérrez-Maldonado J, Ribas-Sabaté J (2014) A virtual reality-integrated program for improving social skills in patients with schizophrenia: a pilot study. J Behav Ther Exp Psychiatry 45(1):81–89. CrossRefGoogle Scholar
  39. Samara A, Galway L, Bond R, Wang H (2019) Affective state detection via facial expression analysis within a human–computer interaction context. J Ambient Intell Hum Comput 10(6):2175–2184. CrossRefGoogle Scholar
  40. Souto YM, Campo MV, Llenderrozas FD, Álvarez MR, Mateos R, Caballero AAG (2018) Randomized clinical trial with e-MotionalTraining® 1.0 for social cognition rehabilitation in schizophrenia. Front Psychiatry 26(9):40. CrossRefGoogle Scholar
  41. Teruel MA, Navarro E, González P (2017) Exploiting awareness for the development of collaborative rehabilitation systems. Mobile Inf Syst. Google Scholar
  42. Tracy JL, Randles D, Steckler CM (2015) The nonverbal communication of emotions. Curr Opin Behav Sci 3:25–30. CrossRefGoogle Scholar
  43. Venkatesh V, Morris MG, Davis GB, Davis FD (2003) User acceptance of information technology: toward a unified view. MIS Q 27:425–478CrossRefGoogle Scholar
  44. Venkatesh V, Thong JY, Xu X (2012) Consumer acceptance and use of information technology: extending the unified theory of acceptance and use of technology. MIS Q 36:157–178CrossRefGoogle Scholar
  45. Woods S, Walters M, Koay KL, Dautenhahn K (2006) Comparing human robot interaction scenarios using live and video based methods: towards a novel methodological approach. In: 9th ieee international workshop on advanced motion control, 2006. IEEE, pp 750–755Google Scholar
  46. Yang YJD, Allen T, Abdullahi SM, Pelphrey K, Volkmar F, Chapman S (2017) Brain responses to biological motion predict treatment outcome in young adults with autism receiving virtual reality social cognition training: preliminary findings. Behav Res Ther 93:55–66. CrossRefGoogle Scholar

Copyright information

© Springer-Verlag GmbH Germany, part of Springer Nature 2019

Authors and Affiliations

  • Arturo S. García
    • 1
  • Patricia Fernández-Sotos
    • 2
    • 6
  • Antonio Fernández-Caballero
    • 1
    • 3
    • 6
    Email author
  • Elena Navarro
    • 1
    • 3
    • 6
  • José M. Latorre
    • 4
  • Roberto Rodriguez-Jimenez
    • 5
    • 6
    • 7
  • Pascual González
    • 1
    • 3
    • 6
  1. 1.Instituto de Investigación en Informática de AlbaceteAlbaceteSpain
  2. 2.Complejo Hospitalario Universitario de Albacete, Servicio de PsiquiatríaAlbaceteSpain
  3. 3.Departamento de Sistemas InformáticosUniversidad de Castilla-La ManchaAlbaceteSpain
  4. 4.Departamento de PsicologíaUniversidad de Castilla-La ManchaAlbaceteSpain
  5. 5.Instituto de Investigación Sanitaria Hospital 12 de Octubre (imas12)MadridSpain
  6. 6.Biomedical Research Networking Centre in Mental Health (CIBERSAM)MadridSpain
  7. 7.CogPsy-GroupUniversidad Complutense de MadridMadridSpain

Personalised recommendations