Skip to main content
SpringerLink
Log in

Towards Enhancing the Multimodal Interaction of a Social Robot to Assist Children with Autism in Emotion Regulation

  • Conference paper
  • First Online:
Pervasive Computing Technologies for Healthcare (PH 2021)

Abstract

Assistive robots are expected to become ubiquitous by transforming everyday life and are expected to be widely used in healthcare therapies. SARs (Socially Assistive Robots) are a class of robots that are at an intersection between the class of assistive robots and that of interactive social robots. SARs are being explored to assist in the diagnosis and treatment of children with ASD (Autism Spectrum Disorder). A SAR called EVA has been used to assist non-pharmacological interventions based on verbal, non-verbal communication and social interaction. The EVA robot can currently speak, listen and express emotions through looking. Towards offering immersive therapies for autistic children, this work enhances EVA’s capabilities to recognize user emotions through facial expression recognition and also to create light sensory effects in order to make the therapy more attractive to users. A therapy session was developed through a serious game where the child should recognize the robot’s emotions. During the game, EVA recognizes the child’s facial expression to check his/her learning progress. We invited a neurotypical 6-year-old child to play the game, with the consent of her parents, and recorded videos of the game session. Those videos were evaluated by 48 expert physicians and psychologists in therapies for ASD using the Technology Acceptance Model (TAM). They considered our work useful and agreed it would help them doing their job more effectively.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 84.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 109.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    https://bit.ly/3kNt4u3.

  2. 2.

    https://bityli.com/0ceIRU.

  3. 3.

    https://bityli.com/IhTkIc.

  4. 4.

    https://youtu.be/PU8BLwTkGaw.

References

  1. Barajas, A.O., Al Osman, H., Shirmohammadi, S.: A serious game for children with autism spectrum disorder as a tool for play therapy. In: 2017 IEEE 5th International Conference on Serious Games and Applications for Health (SeGAH), pp. 1–7. IEEE (2017)

    Google Scholar 

  2. Bevill, R., et al.: Multisensory robotic therapy to promote natural emotional interaction for children with ASD. In: 2016 11th ACM/IEEE International Conference on Human-Robot Interaction (HRI), p. 571 (2016). https://doi.org/10.1109/HRI.2016.7451861

  3. Cibrian, F.L., Peña, O., Ortega, D., Tentori, M.: BendableSound: an elastic multisensory surface using touch-based interactions to assist children with severe autism during music therapy. Int. J. Hum. Comput. Stud. 107, 22–37 (2017)

    Article  Google Scholar 

  4. Cruz-Sandoval, D., Favela, J.: A conversational robot to conduct therapeutic interventions for dementia. IEEE Pervasive Comput. 18(2), 10–19 (2019). https://doi.org/10.1109/MPRV.2019.2907020

    Article  Google Scholar 

  5. Davis, F.D.: Perceived usefulness, perceived ease of use, and user acceptance of information technology. MIS Q. 319–340 (1989)

    Google Scholar 

  6. Fachantidis, N., Syriopoulou-Delli, C.K., Zygopoulou, M.: The effectiveness of socially assistive robotics in children with autism spectrum disorder. Int. J. Dev. Disabil. 66(2), 113–121 (2020)

    Article  Google Scholar 

  7. Feil-Seifer, D., Mataric, M.: Defining socially assistive robotics. In: 9th International Conference on Rehabilitation Robotics, ICORR 2005, pp. 465–468 (2005). https://doi.org/10.1109/ICORR.2005.1501143

  8. Feil-Seifer, D., Mataric, M.: Robot-assisted therapy for children with autism spectrum disorders. In: Proceedings of the 7th International Conference on Interaction Design and Children, pp. 49–52 (2008)

    Google Scholar 

  9. Hirokawa, M., Funahashi, A., Pan, Y., Itoh, Y., Suzuki, K.: Design of a robotic agent that measures smile and facing behavior of children with autism spectrum disorder. In: 2016 25th IEEE International Symposium on Robot and Human Interactive Communication (RO-MAN), pp. 843–848 (2016). https://doi.org/10.1109/ROMAN.2016.7745217

  10. Hu, P.J., Chau, P.Y., Sheng, O.R.L., Tam, K.Y.: Examining the technology acceptance model using physician acceptance of telemedicine technology. J. Manag. Inf. Syst. 16(2), 91–112 (1999)

    Article  Google Scholar 

  11. Javed, H., Park, C.H.: Interactions with an empathetic agent: regulating emotions and improving engagement in autism. IEEE Robot. Autom. Mag. 26(2), 40–48 (2019). https://doi.org/10.1109/MRA.2019.2904638

    Article  Google Scholar 

  12. Kientz, J.A., Goodwin, M.S., Hayes, G.R., Abowd, G.D.: Interactive technologies for autism. Syn. Lect. Assistive Rehabil. Health-Preserving Technol. 2(2), 1–177 (2013)

    Article  Google Scholar 

  13. Lee, J., Takehashi, H., Nagai, C., Obinata, G.: Design of a therapeutic robot for interacting with autistic children through interpersonal touch. In: 2012 IEEE RO-MAN: The 21st IEEE International Symposium on Robot and Human Interactive Communication, pp. 712–717 (2012). https://doi.org/10.1109/ROMAN.2012.6343835

  14. Lee, J., Takehashi, H., Nagai, C., Obinata, G., Stefanov, D.: Which robot features can stimulate better responses from children with autism in robot-assisted therapy? Int. J. Adv. Robot. Syst. 9 (2012). https://doi.org/10.5772/51128

  15. Marangunić, N., Granić, A.: Technology acceptance model: a literature review from 1986 to 2013. Univ. Access Inf. Soc. 14(1), 81–95 (2014). https://doi.org/10.1007/s10209-014-0348-1

    Article  Google Scholar 

  16. Mitjans, A.A.: Affective computation in human-robot interaction. Master thesis, Centro de Investigación CientÍfica y de Educación Superior de Ensenada, Baja California (2020). http://cicese.repositorioinstitucional.mx/jspui/handle/1007/3283

  17. Oei, A.C., Patterson, M.D.: Enhancing cognition with video games: a multiple game training study. PLoS ONE 8(3) (2013). https://doi.org/10.1371/journal.pone.0058546

  18. Olazarán, J., et al.: Nonpharmacological therapies in Alzheimer’s disease: a systematic review of efficacy. Dement. Geriatr. Cogn. Disord. 30(2), 161–178 (2010). https://doi.org/10.1159/000316119

    Article  Google Scholar 

  19. Pares, N., Masri, P., Van Wolferen, G., Creed, C.: Achieving dialogue with children with severe autism in an adaptive multisensory interaction: the “mediate’’ project. IEEE Trans. Visual Comput. Graph. 11(6), 734–743 (2005)

    Article  Google Scholar 

  20. de Paula, G., Valentim, P., Seixas, F., Santana, R., Muchaluat-Saade, D.: Sensory effects in cognitive exercises for elderly users: stroop game. In: 2020 IEEE 33rd International Symposium on Computer-Based Medical Systems (CBMS), pp. 132–137 (2020). https://doi.org/10.1109/CBMS49503.2020.00032

  21. Robins, B., et al.: Human-centred design methods: developing scenarios for robot assisted play informed by user panels and field trials. Int. J. Human Comput. Stud. 68(12), 873–898 (2010)

    Article  Google Scholar 

  22. Salleh, M.H.K., Miskam, M.A., Yussof, H., Omar, A.R.: HRI assessment of ASKNAO intervention framework via typically developed child. Procedia Comput. Sci. 105, 333–339 (2017)

    Article  Google Scholar 

  23. Samonte, M.J.C., Guelos, C.M.C., Madarang, D.K.L., Mercado, M.A.P.: Tap-to-talk: Filipino mobile based learning augmentative and alternative through picture exchange communication intervention for children with autism. In: Proceedings of the 2020 The 6th International Conference on Frontiers of Educational Technologies, pp. 25–29 (2020)

    Google Scholar 

  24. Sandoval Bringas, J.A., Carreño León, M.A., Cota, I.E., Carrillo, A.L.: Development of a videogame to improve communication in children with autism. In: 2016 XI Latin American Conference on Learning Objects and Technology (LACLO), pp. 1–6 (2016). https://doi.org/10.1109/LACLO.2016.7751751

  25. Santatiwongchai, S., Kaewkamnerdpong, B., Jutharee, W., Ounjai, K.: Bliss: using robot in learning intervention to promote social skills for autism therapy. In: Proceedings of the International Convention on Rehabilitation Engineering & Assistive Technology. i-CREATe 2016, Singapore Therapeutic, Assistive & Rehabilitative Technologies (START) Centre, Midview City, SGP (2016)

    Google Scholar 

  26. Santos, L., Geminiani, A., Schydlo, P., Olivieri, I., Santos-Victor, J., Pedrocchi, A.: Design of a robotic coach for motor, social and cognitive skills training toward applications with ASD children. IEEE Trans. Neural Syst. Rehabil. Eng. 29, 1223–1232 (2021)

    Article  Google Scholar 

  27. Shibata, T.: An overview of human interactive robots for psychological enrichment. Proc. IEEE 92(11), 1749–1758 (2004). https://doi.org/10.1109/JPROC.2004.835383

    Article  Google Scholar 

  28. Shibata, T.: Therapeutic seal robot as biofeedback medical device: qualitative and quantitative evaluations of robot therapy in dementia care. Proc. IEEE 100(8), 2527–2538 (2012). https://doi.org/10.1109/JPROC.2012.2200559

    Article  Google Scholar 

  29. Smith, T.: Discrete trial training in the treatment of autism. Focus Autism Other Dev. Disabil. 16(2), 86–92 (2001)

    Article  Google Scholar 

  30. So, W.C., et al.: Who is a better teacher for children with autism? Comparison of learning outcomes between robot-based and human-based interventions in gestural production and recognition. Res. Dev. Disabil. 86, 62–75 (2019)

    Article  Google Scholar 

  31. Tan, C.T., Harrold, N., Rosser, D.: Can you copyme? An expression mimicking serious game. In: SIGGRAPH Asia 2013 Symposium on Mobile Graphics and Interactive Applications, SA ’13. Association for Computing Machinery, New York (2013). https://doi.org/10.1145/2543651.2543657

  32. Tentori, M., Ziviani, A., Muchaluat-Saade, D.C., Favela, J.: Digital healthcare in Latin America: the case of Brazil and Mexico. Commun. ACM 63(11), 72–77 (2020)

    Article  Google Scholar 

  33. Valentim, P.A., Barreto, F., Muchaluat-Saade, D.C.: Possibilitando o reconhecimento de expressões faciais em aplicações Ginga-NCL. In: Anais Estendidos do XXVI Simpósio Brasileiro de Sistemas Multimídia e Web, pp. 53–56. SBC (2020)

    Google Scholar 

  34. Zubrycki, I., Granosik, G.: Designing an interactive device for sensory therapy. In: 2016 11th ACM/IEEE International Conference on Human-Robot Interaction (HRI), pp. 545–546. IEEE (2016)

    Google Scholar 

Download references

Acknowledgment

This work was partially supported by CNPq, FAPERJ, CAPES, CAPES Print and INCT-MACC.

Author information

Authors and Affiliations

  1. MÍdiaCom Lab - UFF, Niterói, Brazil

    Marcelo Rocha, Pedro Valentim, Fábio Barreto & Débora Muchaluat-Saade

  2. CICESE, Ensenada, Mexico

    Adrian Mitjans & Jesus Favela

  3. Computer Science and Engineering, UC San Diego, San Diego, USA

    Dagoberto Cruz-Sandoval

Authors
  1. Marcelo Rocha
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Pedro Valentim
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Fábio Barreto
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Adrian Mitjans
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Dagoberto Cruz-Sandoval
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Jesus Favela
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Débora Muchaluat-Saade
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Marcelo Rocha .

Editor information

Editors and Affiliations

  1. Holon Institute of Technology, Holon, Israel

    Hadas Lewy

  2. Holon Institute of Technology, Holon, Israel

    Refael Barkan

Rights and permissions

Reprints and permissions

Copyright information

© 2022 ICST Institute for Computer Sciences, Social Informatics and Telecommunications Engineering

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Rocha, M. et al. (2022). Towards Enhancing the Multimodal Interaction of a Social Robot to Assist Children with Autism in Emotion Regulation. In: Lewy, H., Barkan, R. (eds) Pervasive Computing Technologies for Healthcare. PH 2021. Lecture Notes of the Institute for Computer Sciences, Social Informatics and Telecommunications Engineering, vol 431. Springer, Cham. https://doi.org/10.1007/978-3-030-99194-4_25

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-99194-4_25

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-99193-7

  • Online ISBN: 978-3-030-99194-4

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation