Journal on Multimodal User Interfaces

, Volume 12, Issue 3, pp 257–269 | Cite as

Personalisation and automation in a virtual conversation skills tutor for children with autism

  • Marissa MilneEmail author
  • Parimala Raghavendra
  • Richard Leibbrandt
  • David Martin Ward Powers
Original Paper


Existing research involving children with autism suggests that autonomous (self directed) virtual humans can be used successfully to improve language skills, and that authorable (researcher controlled) virtual humans can be used to improve social skills. This research combines these ideas and investigates the use of autonomous virtual humans for teaching and facilitating practice of basic social skills in the areas of greeting, conversation skills, listening and turn taking. The Social Tutor software features three virtual human ‘characters’ who guide the learner through educational tasks and model social scenarios. Participants used the software for 10–15 min per day, 3–5 days per week for 3 weeks, with data collected before software use commenced, immediately after use ended, then again 2 and 4 months after this. The software evaluation revealed that the social tutor was generally well-received by participants and caregivers, with data showing a clear difference in theoretical knowledge of social skills between the experimental (social content) group and the control (placebo content) group. This paper focuses on the automated assessment, dynamic lesson sequencing, feedback, and reinforcement systems of the social tutor software, the impact these systems had on participant performance during the software evaluation, and recommendations for future development. Thus, while data reflecting overall performance of the social tutor system is provided for context, the main data presented is limited to that specifically relating to the performance and user perceptions of the aforementioned systems.


Automated assessment Personalised education Autism Social skills Conversation skills Embodied virtual agents 



The content of this paper represents a subset of the broader software development and evaluation project conducted for the PhD thesis of Marissa Milne. The authors gratefully thank the participants and caregivers involved in the evaluation of the Social Tutor, without which the study would not have been possible. This study was conducted with ethical clearance from the Flinders University Social and Behavioural Research Ethics Committee and Autism SA Professional Practice Committee.


  1. 1.
    Abirached B, Zhang Y, Aggarwal JK, Tamersoy B, Fernandes T, Carlos J (2011) Improving communication skills of children with ASDS through interaction with virtual characters. In: 2011 IEEE 1st international conference on serious games and applications for health (SeGAH), pp 1–4Google Scholar
  2. 2.
    Association AP (2000) Diagnostic and statistical manual of mental disorders: DSM-IV-TR. 4th ed., text revision edn. American Psychiatric Association, Washington, DCGoogle Scholar
  3. 3.
    Bernardini S, Porayska-Pomsta K, Smith TJ (2014) Echoes: an intelligent serious game for fostering social communication in children with autism. Inf Sci 264:41–60CrossRefGoogle Scholar
  4. 4.
    Black P (2015) Assessment: friend or foe of pedagogy and learning. Past as Prologue, p 235Google Scholar
  5. 5.
    Blair K, Schwartz D, Biswas G, Leelawong K (2007) Pedagogical agents for learning by teaching: teachable agents. Educ Technol 47(1):56Google Scholar
  6. 6.
    Bosseler A, Massaro D (2003) Development and evaluation of a computer-animated tutor for vocabulary and language learning in children with autism. J Autism Dev Disord 33(6):653–672CrossRefGoogle Scholar
  7. 7.
    Bowman-Perrott L, Davis H, Vannest K, Williams L, Greenwood C, Parker R (2013) Academic benefits of peer tutoring: a meta-analytic review of single-case research. School Psychol Rev 42(1):39–55Google Scholar
  8. 8.
    Carter AS, Volkmar FR, Sparrow SS, Wang JJ, Lord C, Dawson G, Fombonne E, Loveland K, Mesibov G, Schopler E (1998) The vineland adaptive behavior scales: supplementary norms for individuals with autism. J Autism Dev Disord 28(4):287–302CrossRefGoogle Scholar
  9. 9.
    Cheng Y, Huang CL, Yang CS (2015) Using a 3d immersive virtual environment system to enhance social understanding and social skills for children with autism spectrum disorders. Focus Autism Other Dev Disabil 30(4):222–236CrossRefGoogle Scholar
  10. 10.
    Chi M, Siler S, Jeong H, Yamauchi T, Hausmann R (2001) Learning from human tutoring. Cognit Sci 25(4):471–533CrossRefGoogle Scholar
  11. 11.
    Cooper J, Goodfellow H, Muhlheim E, Paske K, Pearson L (2003) PALS social skills program: playing and learning to socialise: resource book. Inscript Publishing, LauncestonGoogle Scholar
  12. 12.
    Elias MJ, Butler LB (2005) Social decision making/social problem solving for middle school students: skills and activities for academic, social, and emotional success. Research PressPub, OttawaGoogle Scholar
  13. 13.
    Finnegan E, Mazin AL (2016) Strategies for increasing reading comprehension skills in students with autism spectrum disorder: a review of the literature. Educ Treat Child 39(2):187–219CrossRefGoogle Scholar
  14. 14.
    Gray C (2001) How to respond to a bullying attempt: what to think, say and do. The Morning News 13(2)Google Scholar
  15. 15.
    Hattie J, Timperley H (2007) The power of feedback. Rev Educ Res 77(1):81CrossRefGoogle Scholar
  16. 16.
    Hopkins IM, Gower MW, Perez TA, Smith DS, Amthor FR, Casey Wimsatt F, Biasini FJ (2011) Avatar assistant: improving social skills in students with an asd through a computer-based intervention. J Autism Dev Disord 41(11):1543–1555CrossRefGoogle Scholar
  17. 17.
    Hu X, Xia H (2010) Automated assessment system for subjective questions based on LSI. In: IEEE, pp. 250–254Google Scholar
  18. 18.
    Jackson G, Guess R, McNamara D (2010) Assessing cognitively complex strategy use in an untrained domain. Top Cognit Sci 2(1):127–137CrossRefGoogle Scholar
  19. 19.
    Jacovina ME, Jackson GT, Snow EL, McNamara DS (2016) Timing game-based practice in a reading comprehension strategy tutor. In: International conference on intelligent tutoring systems, Springer, Berlin, pp 59–68Google Scholar
  20. 20.
    Ke F, Im T (2013) Virtual-reality-based social interaction training for children with high-functioning autism. J Educ Res 106(6):441–461CrossRefGoogle Scholar
  21. 21.
    de Klerk S, Eggen TJHM, Veldkamp BP (2016) A methodology for applying students’ interactive task performance scores from a multimedia-based performance assessment in a bayesian network. Comput Hum Behav 60:264–279CrossRefGoogle Scholar
  22. 22.
    Kroncke AP, Willard M, Huckabee H (2016) What is Autism? History and foundations. Springer International Publishing, Cham, pp 3–9Google Scholar
  23. 23.
    Luerssen M, Lewis T, Powers D (2011) Head X: customizable audiovisual synthesis for a multi-purpose virtual head, Lecture Notes in Computer Science, vol 6464, book section 49, pp. 486–495. Springer, BerlinGoogle Scholar
  24. 24.
    MacMullin JA, Lunsky Y, Weiss JA (2016) Plugged in: electronics use in youth and young adults with autism spectrum disorder. Autism 20(1):45–54CrossRefGoogle Scholar
  25. 25.
    McGinnis E, Goldstein AP (2012) Skillstreaming the elementary school child: a guide for teaching prosocial skills, 3rd edn. Research Press, ChampaignGoogle Scholar
  26. 26.
    Meyer J, Land R (2010) Threshold concepts and troublesome knowledge (5): dynamics of assessment. Threshold concepts and transformational learning. Routledge, London, pp 61–79Google Scholar
  27. 27.
    Milne M, Leibbrandt R, Raghavendra P, Luerssen M, Lewis T, Powers D (2013) Lesson authoring system for creating interactive activities involving virtual humans the thinking head whiteboard. In: 2013 IEEE symposium on intelligent agent (IA), pp 13–20Google Scholar
  28. 28.
    Mostow J (2005) Project listen: a reading tutor that listens.
  29. 29.
    Neely LC, Ganz JB, Davis JL, Boles MB, Hong ER, Ninci J, Gilliland WD (2016) Generalization and maintenance of functional living skills for individuals with autism spectrum disorder: a review and meta-analysis. Rev J Autism Dev Disord 3(1):37–47CrossRefGoogle Scholar
  30. 30.
    Perry A, Flanagan HE, Dunn Geier J, Freeman NL (2009) Brief report: the vineland adaptive behavior scales in young children with autism spectrum disorders at different cognitive levels. J Autism Dev Disord 39(7):1066–1078CrossRefGoogle Scholar
  31. 31.
    Putnam C, Chong L (2008) Software and technologies designed for people with autism: what do users want? In: Proceedings of the 10th international ACM SIGACCESS conference on computers and accessibility, assets ’08, ACM, New York, NY, pp 3–10Google Scholar
  32. 32.
    Putnam RT (1987) Structuring and adjusting content for students: a study of live and simulated tutoring of addition. Am Educ Res J 24(1):13–48CrossRefGoogle Scholar
  33. 33.
    Rao PA, Beidel DC, Murray MJ (2008) Social skills interventions for children with aspergers syndrome or high-functioning autism: a review and recommendations. J Autism Dev Disord 38:353361CrossRefGoogle Scholar
  34. 34.
    Roberts V, Joiner R (2007) Investigating the efficacy of concept mapping with pupils with autistic spectrum disorder. Br J Spec Educ 34(3):127–135CrossRefGoogle Scholar
  35. 35.
    Saadatzi MN, Pennington RC, Welch KC, Graham JH, Scott RE (2017) The use of an autonomous pedagogical agent and automatic speech recognition for teaching sight words to students with autism spectrum disorder. J Spec Educ Technol 32:173–183CrossRefGoogle Scholar
  36. 36.
    Shute V, Towle B (2003) Adaptive e-learning. Educ Psychol 38(2):105–114CrossRefGoogle Scholar
  37. 37.
    Silver M, Oakes P (2001) Evaluation of a new computer intervention to teach people with autism or asperger syndrome to recognize and predict emotions in others. Autism 5(3):299–316CrossRefGoogle Scholar
  38. 38.
    Sparrow SS (2011) Vineland adaptive behavior scales. Springer, New York, NY, pp 2618–2621Google Scholar
  39. 39.
    Sparrow SS, Cicchetti DV, Balla DA (2005) Vineland-II: vineland adaptive behaviour scales-survey forms manual. NCS Pearson, BloomingtonGoogle Scholar
  40. 40.
    Sturm D, Peppe E, Ploog B (2016) emot-ican: Design of an assessment game for emotion recognition in players with autism. In: 2016 IEEE international conference on serious games and applications for health (SeGAH), IEEE, pp 1–7Google Scholar
  41. 41.
    Tartaro A (2007) Authorable virtual peers for children with autism. In: CHI ’07 extended abstracts on human factors in computing systems, CHI EA ’07, ACM, New York, NY, pp 1677–1680Google Scholar
  42. 42.
    VanLehn K (2011) The relative effectiveness of human tutoring, intelligent tutoring systems, and other tutoring systems. Educ Psychol 46(4):197–221CrossRefGoogle Scholar
  43. 43.
    Woolf B, Arroyo I, Muldner K, Burleson W, Cooper D, Dolan R, Christopherson R (2010) The effect of motivational learning companions on low achieving students and students with disabilities. In: Intelligent Tutoring Systems, Springer, Berlin, pp 327–337Google Scholar

Copyright information

© Springer Nature Switzerland AG 2018

Authors and Affiliations

  • Marissa Milne
    • 1
    Email author
  • Parimala Raghavendra
    • 2
  • Richard Leibbrandt
    • 3
  • David Martin Ward Powers
    • 1
  1. 1.College of Science and Engineering, Flinders Digital Health Research CentreFlinders UniversityAdelaideSouth Australia
  2. 2.College of Nursing and Health Sciences, Disability and Community InclusionFlinders UniversityAdelaideSouth Australia
  3. 3.College of Medicine and Public HealthFlinders UniversityAdelaideSouth Australia

Personalised recommendations