Play and Learn with an Intelligent Robot: Enhancing the Therapy of Hearing-Impaired Children

  • Andri IoannouEmail author
  • Anna Andreva
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11747)


This study suggests an innovative way of using an intelligent robot to support speech therapy for hearing impaired children through play. Although medical technology (e.g., hearing aid, cochlear implant) for children with hearing impairment has advanced significantly, the amplification itself does not provide optimal development of hearing and speaking; it must be combined with specialized therapy. The present study focuses on the use of the humanoid robot NAO in auditory-verbal therapy, an approach to the development of auditory and verbal skills, which does not allow lipreading or other non-verbal cues to facilitate communication. NAO does not have a human mouth and therefore children with hearing impairment cannot do lipreading; this unique characteristic of the technology has been successfully used in the study to create playful and engaging auditory-verbal therapy sessions for six kindergarten hearing impaired children, allowing them to improve their ability to follow instructions using the hearing aid/cochlear implant rather than visual clues in the environment. Our results, although preliminary, seem to encourage further research in supporting hearing-impaired children via play with intelligent robots.


NAO Intelligent robot Humanoid robot Hearing impairment Deaf Speech therapy Auditory-Verbal therapy Special education Playful learning Technology-enhanced learning 



This work has been partly supported by the project that has received funding from the European Union’s Horizon 2020 research and innovation programme under grant agreement No 739578 (RISE – Call: H2020-WIDESPREAD-01-2016-2017-TeamingPhase2) and the Government of the Republic of Cyprus through the Directorate General for European Programmes, Coordination and Development.


  1. 1.
    Spencer, P.E., Marschark, M.: Evidence-Based Practice in Educating Deaf and Hard-of-Hearing Students. Oxford University Press, Oxford (2010)Google Scholar
  2. 2.
    Lee, C.: Deafness and cochlear implants: a deaf scholar’s perspective. Child Neurol. J. 27(6), 821–823 (2012). Scholar
  3. 3.
    Beal-Alvarez, J., Cannon, J.E.: Technology intervention research with deaf and hard of hearing learners: levels of evidence. Am. Ann. Deaf 158(5), 486–505 (2014)CrossRefGoogle Scholar
  4. 4.
    Kelman, C.A., Branco, A.U.: (Meta) Communication strategies in inclusive classes for deaf students. Am. Ann. Deaf 154(4), 371–381 (2009)CrossRefGoogle Scholar
  5. 5.
    Wilkins, M., Ertmer, D.J.: Introducing young children who are deaf or hard of hearing to spoken LanguageChild’sVoice, an oral school. Lang. Speech Hear. Serv. Schools 33(3), 196–204 (2002)CrossRefGoogle Scholar
  6. 6.
    Ginsburg, K.R.: The importance of play in promoting healthy child development and maintaining strong parent-child bonds. Pediatrics 119(1), 182–191 (2007)CrossRefGoogle Scholar
  7. 7.
    Mills, P.E., Beecher, C.C., Dale, P.S., Cole, K.N., Jenkins, J.R.: Language of children with disabilities to peers at play: impact of ecology. J. Early Interv. 36(2), 111–130 (2014)CrossRefGoogle Scholar
  8. 8.
    Qayyum, A., Khan, A.Z., Rais, R.A.: Exploring play of children with sensory impairments in special schools at Karachi, Pakistan. Qual. Rep. 20(2), 1–17 (2015)Google Scholar
  9. 9.
    Bobzien, J., Richels, C., Raver, S.A., Hester, P., Browning, E., Morin, L.: An observational study of social communication skills in eight preschoolers with and without hearing loss during cooperative play. Early Child. Educ. J. 41(5), 339–346 (2013)CrossRefGoogle Scholar
  10. 10.
    Cejas, I., Barker, D.H., Quittner, A.L., Niparko, J.K.: Development of joint engagement in young deaf and hearing children: effects of chronological age and language skills. J. Speech Lang. Hear. Res. 57(5), 1831–1841 (2014)CrossRefGoogle Scholar
  11. 11.
    Harris, L.G.: Social-emotional development in children with hearing loss. Theses and Dissertations - Communication Sciences and Disorders, vol. 4 (2014). Accessed 01 Aug 2017
  12. 12.
    Sininger, Y.S., Grimes, A., Christensen, E.: Auditory development in early amplified children: factors influencing auditory-based communication outcomes in children with hearing loss. Ear Hear. 31(2), 166–185 (2010)CrossRefGoogle Scholar
  13. 13.
    Yuhan, X.: Peer interaction of children with hearing impairment. Int. J. Psychol. Stud. 5(4), 17–25 (2013)CrossRefGoogle Scholar
  14. 14.
    Arendsen, J., Janssen, J.B., Begeer, S., Stekelenburg, F.C.: The use of robots in social behavior tutoring for children with ASD. In: 28th Proceedings on Annual European Conference on Cognitive Ergonomics, pp. 371–372. ACM (2010)Google Scholar
  15. 15.
    Fridin, M.: Storytelling by a kindergarten social assistive robot: a tool for constructive learning in preschool education. Comput. Educ. 70, 53–64 (2014)CrossRefGoogle Scholar
  16. 16.
    Ioannou, A., Andreou, E., Christofi, M.: Preschoolers’ interest and caring behaviour around a humanoid robot. TechTrends 59(2), 23–26 (2015)CrossRefGoogle Scholar
  17. 17.
    Kartapanis, I., Ioannou, A., Zaphiris, P.: NAO as an assistant in ASD therapy sessions: the case of Joe. In: INTED2015 Proceedings (2015)Google Scholar
  18. 18.
    Lee, H., Hyun, E.: The intelligent robot contents for children with speech-language disorder. J. Educ. Technol. Soc. 18(3), 100–113 (2015)Google Scholar
  19. 19.
    Polycarpou, P., Andreeva, A., Ioannou, A., Zaphiris, P.: Don’t read my lips: assessing listening and speaking skills through play with a humanoid robot. In: Stephanidis, C. (ed.) HCI 2016. CCIS, vol. 618, pp. 255–260. Springer, Cham (2016). Scholar
  20. 20.
    Welling, D.R., Ukstins, C.A.: Fundamentals of Audiology for the Speech-language Pathologist. Jones & Bartlett Learning (2017)Google Scholar
  21. 21.
    Shonkoff, J.P., Phillips, D.A. (ed.): From Neurons to Neighborhoods: The Science of Early Childhood Development. National Academies Press, Washington (DC) (2000).
  22. 22.
  23. 23.
    Marshark, M., Hauser, P.C.: How Deaf Children Learn: What Parents and Teachers Need to Know. Oxford University Press, New York (2012)Google Scholar
  24. 24.
    Decker, K.B.: Influences on Parental Decisions Regarding Communications Options for Children Identified with Hearing Loss. Human Development and Family Studies, p. 1409. Michigan State University, Michigan (2010)Google Scholar
  25. 25.
    Gallaudet Research Institute: Regional and national summary report of data from the 2007–08 annual survey of deaf and hard of hearing children and youth (2008)Google Scholar
  26. 26.
    Crowe, K., McLeod, S., McKinnon, D.H., Ching, T.Y.C.: Speech, sign, or multilingualism for children with hearing loss: Quantitative insights into caregivers’ decision-making. Lang. Speech Hear. Serv. Schools 45, 234–247 (2014)CrossRefGoogle Scholar
  27. 27.
    Crowe, K., Fordham, L.A., McLeod, S., Ching, T.Y.C.: Part of our world: Influences on caregiver decisions about communication choices for children with hearing loss. Deafness Educ. Int. 16(2), 61–85 (2014)CrossRefGoogle Scholar
  28. 28.
    Paul, P.V., Whitelaw, G.M.: Hearing and Deafness an Introduction for Health and Education Professionals. Jones & Bartlett Publishers, Sudbury (2011)Google Scholar
  29. 29.
    Cooper, H., Craddock, L.: Cochlear Implants: A Practical Guide, 2nd edn. Whurr Publishers, London and Philadelphia (2006)Google Scholar
  30. 30.
    Cole, E.B., Flexer, C.: Children with Hearing Loss: Developing Listening and Talking Birth to Six. Plural Publishing, San Diego (2015)Google Scholar
  31. 31.
    Ling, D.: Speech and the Hearing Impaired Child: Theory and Practice, 2nd edn. Alexander Graham Bell Association for the Deaf, Washington, DC (2002)Google Scholar
  32. 32.
    Estabrooks, W., Maclver-Lux, K., Rhoades, E.A.: Auditory-Verbal Therapy: For Young Children with Hearing Loss and their Families, and the Practitioners who Guide Them, p. 286. Plural Publishing, San Diego (2016)Google Scholar
  33. 33.
    MuSiEk, F.E.: Neurobiology, Cognitive Science, and Intervention. Handbook of Central Auditory Processing Disorder, Volume II: Comprehensive Intervention, vol. 2, p. 1. Plural Publishing (2013)Google Scholar
  34. 34.
    Keren, G., Ben-David, A., Fridin, M.: Kindergarten assistive robotics (KAR) as a tool for spatial cognition development in pre-school education. In: International Conference on Intelligent Robots and Systems 7–12 October 2012, Vilamoura, Algarve, Portugal (2012)Google Scholar
  35. 35.
    Malik, N.A., Yussof, H., Hanapiah, F.A.: Development of imitation learning through physical therapy using a humanoid robot. In: International Conference on Robot PRIDE 2013–2014 - Medical and Rehabilitation Robotics and Instrumentation, ConfPRIDE 2013–2014, Procedia Computer Science, vol. 42, pp. 191–197. Elsevier (2014)Google Scholar
  36. 36.
    Hamzah, M.S.J., Shamsuddin, S., Miskam, M.A., Yussof, H., Hashim, K.S.: Development of interaction scenarios based on pre-school curriculum in robotic intervention for children with autism. In: International Conference on Robot PRIDE 2013–2014-Medical and Rehabilitation Robotics and Instrumentation, ConfPRIDE 2013–2014, Procedia Computer Science, vol. 42, pp. 214–221. Elsevier (2014)Google Scholar
  37. 37.
    Robins, B., Dautenhahn, K., Boekhorst, R., Billard, A.: Robotic assistants in therapy and education of children with autism: can a small humanoid robot help encourage social interaction skills? Univ. Access Inf. Soc. 4, 105–120 (2005)CrossRefGoogle Scholar
  38. 38.
    Srinivasan, S.M., Park, I.K., Neelly, L.B., Bhat, A.N.: A comparison of the effects of rhythm and robotic interventions on repetitive behaviors and affective states of children with autism spectrum disorder (ASD). In: Research in Autism Spectrum Disorders, vol. 18, pp. 51–63. Elsevier (2015)Google Scholar
  39. 39.
    Salleh, M.H.K., et al.: Experimental framework for the categorization of special education programs of ASKNAO. Procedia Comput. Sci. 76, 480 (2015)CrossRefGoogle Scholar
  40. 40.
    Sawada, H., Kitani, M., Hayashi, Y.: A robotic voice simulator and the interactive training for hearing-impaired people. In: BioMed Research International 2008 (2008)CrossRefGoogle Scholar
  41. 41.
    Kose-Bagci, H., Yorganci, R.: Tale of a robot: humanoid robot assisted sign language tutoring. In: 11th IEEE-RAS International Conference on Humanoid Robots, pp. 105–111. IEEE, Bled Slovenia (2011)Google Scholar
  42. 42.
    Uluer, P., Akalin, N., Kose, H.: A new robotic platform for sign language tutoring humanoid robots as assistive game companions for teaching sign language. Int. J. Soc. Robot. 7, 571–585 (2015)CrossRefGoogle Scholar
  43. 43.
    Constantinou, V., Ioannou, A., Klironomos, I., Antona, M., Stephanidis, C.: Technology support for the inclusion of deaf students in mainstream schools: a summary of research from 2007 to 2017. Univ. Access Inf. Soc. 1–6 (2018)Google Scholar
  44. 44.
    Besio, S.: The need for play for the sake of play. In: Besio, S., Bulgarelli, D., Stancheva-Popkostadinova, V. (eds.) Play Development in Children With Disabilities (2017)Google Scholar

Copyright information

© IFIP International Federation for Information Processing 2019

Authors and Affiliations

  1. 1.Cyprus Interaction Lab, Cyprus University of TechnologyLimassolCyprus
  2. 2.Research Center on Interactive Media, Smart Systems and Emerging Technologies (RISE)NicosiaCyprus
  3. 3.Department of Logopedics, Faculty of Public Health, Health Care and SportSouth-West University “Neofit Rilski”BlagoevgradBulgaria

Personalised recommendations