Skip to main content
SpringerLink
Log in

Personalisation and automation in a virtual conversation skills tutor for children with autism

  • Original Paper
  • Published:
Journal on Multimodal User Interfaces Aims and scope Submit manuscript

Abstract

Existing research involving children with autism suggests that autonomous (self directed) virtual humans can be used successfully to improve language skills, and that authorable (researcher controlled) virtual humans can be used to improve social skills. This research combines these ideas and investigates the use of autonomous virtual humans for teaching and facilitating practice of basic social skills in the areas of greeting, conversation skills, listening and turn taking. The Social Tutor software features three virtual human ‘characters’ who guide the learner through educational tasks and model social scenarios. Participants used the software for 10–15 min per day, 3–5 days per week for 3 weeks, with data collected before software use commenced, immediately after use ended, then again 2 and 4 months after this. The software evaluation revealed that the social tutor was generally well-received by participants and caregivers, with data showing a clear difference in theoretical knowledge of social skills between the experimental (social content) group and the control (placebo content) group. This paper focuses on the automated assessment, dynamic lesson sequencing, feedback, and reinforcement systems of the social tutor software, the impact these systems had on participant performance during the software evaluation, and recommendations for future development. Thus, while data reflecting overall performance of the social tutor system is provided for context, the main data presented is limited to that specifically relating to the performance and user perceptions of the aforementioned systems.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3

Similar content being viewed by others

References

  1. Abirached B, Zhang Y, Aggarwal JK, Tamersoy B, Fernandes T, Carlos J (2011) Improving communication skills of children with ASDS through interaction with virtual characters. In: 2011 IEEE 1st international conference on serious games and applications for health (SeGAH), pp 1–4

  2. Association AP (2000) Diagnostic and statistical manual of mental disorders: DSM-IV-TR. 4th ed., text revision edn. American Psychiatric Association, Washington, DC

  3. Bernardini S, Porayska-Pomsta K, Smith TJ (2014) Echoes: an intelligent serious game for fostering social communication in children with autism. Inf Sci 264:41–60

    Article  Google Scholar 

  4. Black P (2015) Assessment: friend or foe of pedagogy and learning. Past as Prologue, p 235

  5. Blair K, Schwartz D, Biswas G, Leelawong K (2007) Pedagogical agents for learning by teaching: teachable agents. Educ Technol 47(1):56

    Google Scholar 

  6. Bosseler A, Massaro D (2003) Development and evaluation of a computer-animated tutor for vocabulary and language learning in children with autism. J Autism Dev Disord 33(6):653–672

    Article  Google Scholar 

  7. Bowman-Perrott L, Davis H, Vannest K, Williams L, Greenwood C, Parker R (2013) Academic benefits of peer tutoring: a meta-analytic review of single-case research. School Psychol Rev 42(1):39–55

    Google Scholar 

  8. Carter AS, Volkmar FR, Sparrow SS, Wang JJ, Lord C, Dawson G, Fombonne E, Loveland K, Mesibov G, Schopler E (1998) The vineland adaptive behavior scales: supplementary norms for individuals with autism. J Autism Dev Disord 28(4):287–302

    Article  Google Scholar 

  9. Cheng Y, Huang CL, Yang CS (2015) Using a 3d immersive virtual environment system to enhance social understanding and social skills for children with autism spectrum disorders. Focus Autism Other Dev Disabil 30(4):222–236

    Article  Google Scholar 

  10. Chi M, Siler S, Jeong H, Yamauchi T, Hausmann R (2001) Learning from human tutoring. Cognit Sci 25(4):471–533

    Article  Google Scholar 

  11. Cooper J, Goodfellow H, Muhlheim E, Paske K, Pearson L (2003) PALS social skills program: playing and learning to socialise: resource book. Inscript Publishing, Launceston

    Google Scholar 

  12. Elias MJ, Butler LB (2005) Social decision making/social problem solving for middle school students: skills and activities for academic, social, and emotional success. Research PressPub, Ottawa

    Google Scholar 

  13. Finnegan E, Mazin AL (2016) Strategies for increasing reading comprehension skills in students with autism spectrum disorder: a review of the literature. Educ Treat Child 39(2):187–219

    Article  Google Scholar 

  14. Gray C (2001) How to respond to a bullying attempt: what to think, say and do. The Morning News 13(2)

  15. Hattie J, Timperley H (2007) The power of feedback. Rev Educ Res 77(1):81

    Article  Google Scholar 

  16. Hopkins IM, Gower MW, Perez TA, Smith DS, Amthor FR, Casey Wimsatt F, Biasini FJ (2011) Avatar assistant: improving social skills in students with an asd through a computer-based intervention. J Autism Dev Disord 41(11):1543–1555

    Article  Google Scholar 

  17. Hu X, Xia H (2010) Automated assessment system for subjective questions based on LSI. In: IEEE, pp. 250–254

  18. Jackson G, Guess R, McNamara D (2010) Assessing cognitively complex strategy use in an untrained domain. Top Cognit Sci 2(1):127–137

    Article  Google Scholar 

  19. Jacovina ME, Jackson GT, Snow EL, McNamara DS (2016) Timing game-based practice in a reading comprehension strategy tutor. In: International conference on intelligent tutoring systems, Springer, Berlin, pp 59–68

  20. Ke F, Im T (2013) Virtual-reality-based social interaction training for children with high-functioning autism. J Educ Res 106(6):441–461

    Article  Google Scholar 

  21. de Klerk S, Eggen TJHM, Veldkamp BP (2016) A methodology for applying students’ interactive task performance scores from a multimedia-based performance assessment in a bayesian network. Comput Hum Behav 60:264–279

    Article  Google Scholar 

  22. Kroncke AP, Willard M, Huckabee H (2016) What is Autism? History and foundations. Springer International Publishing, Cham, pp 3–9

    Google Scholar 

  23. Luerssen M, Lewis T, Powers D (2011) Head X: customizable audiovisual synthesis for a multi-purpose virtual head, Lecture Notes in Computer Science, vol 6464, book section 49, pp. 486–495. Springer, Berlin

  24. MacMullin JA, Lunsky Y, Weiss JA (2016) Plugged in: electronics use in youth and young adults with autism spectrum disorder. Autism 20(1):45–54

    Article  Google Scholar 

  25. McGinnis E, Goldstein AP (2012) Skillstreaming the elementary school child: a guide for teaching prosocial skills, 3rd edn. Research Press, Champaign

    Google Scholar 

  26. Meyer J, Land R (2010) Threshold concepts and troublesome knowledge (5): dynamics of assessment. Threshold concepts and transformational learning. Routledge, London, pp 61–79

    Google Scholar 

  27. Milne M, Leibbrandt R, Raghavendra P, Luerssen M, Lewis T, Powers D (2013) Lesson authoring system for creating interactive activities involving virtual humans the thinking head whiteboard. In: 2013 IEEE symposium on intelligent agent (IA), pp 13–20

  28. Mostow J (2005) Project listen: a reading tutor that listens. http://www.cs.cmu.edu/~listen/research.html

  29. Neely LC, Ganz JB, Davis JL, Boles MB, Hong ER, Ninci J, Gilliland WD (2016) Generalization and maintenance of functional living skills for individuals with autism spectrum disorder: a review and meta-analysis. Rev J Autism Dev Disord 3(1):37–47

    Article  Google Scholar 

  30. Perry A, Flanagan HE, Dunn Geier J, Freeman NL (2009) Brief report: the vineland adaptive behavior scales in young children with autism spectrum disorders at different cognitive levels. J Autism Dev Disord 39(7):1066–1078

    Article  Google Scholar 

  31. Putnam C, Chong L (2008) Software and technologies designed for people with autism: what do users want? In: Proceedings of the 10th international ACM SIGACCESS conference on computers and accessibility, assets ’08, ACM, New York, NY, pp 3–10

  32. Putnam RT (1987) Structuring and adjusting content for students: a study of live and simulated tutoring of addition. Am Educ Res J 24(1):13–48

    Article  Google Scholar 

  33. Rao PA, Beidel DC, Murray MJ (2008) Social skills interventions for children with aspergers syndrome or high-functioning autism: a review and recommendations. J Autism Dev Disord 38:353361

    Article  Google Scholar 

  34. Roberts V, Joiner R (2007) Investigating the efficacy of concept mapping with pupils with autistic spectrum disorder. Br J Spec Educ 34(3):127–135

    Article  Google Scholar 

  35. Saadatzi MN, Pennington RC, Welch KC, Graham JH, Scott RE (2017) The use of an autonomous pedagogical agent and automatic speech recognition for teaching sight words to students with autism spectrum disorder. J Spec Educ Technol 32:173–183

    Article  Google Scholar 

  36. Shute V, Towle B (2003) Adaptive e-learning. Educ Psychol 38(2):105–114

    Article  Google Scholar 

  37. Silver M, Oakes P (2001) Evaluation of a new computer intervention to teach people with autism or asperger syndrome to recognize and predict emotions in others. Autism 5(3):299–316

    Article  Google Scholar 

  38. Sparrow SS (2011) Vineland adaptive behavior scales. Springer, New York, NY, pp 2618–2621

    Google Scholar 

  39. Sparrow SS, Cicchetti DV, Balla DA (2005) Vineland-II: vineland adaptive behaviour scales-survey forms manual. NCS Pearson, Bloomington

    Google Scholar 

  40. Sturm D, Peppe E, Ploog B (2016) emot-ican: Design of an assessment game for emotion recognition in players with autism. In: 2016 IEEE international conference on serious games and applications for health (SeGAH), IEEE, pp 1–7

  41. Tartaro A (2007) Authorable virtual peers for children with autism. In: CHI ’07 extended abstracts on human factors in computing systems, CHI EA ’07, ACM, New York, NY, pp 1677–1680

  42. VanLehn K (2011) The relative effectiveness of human tutoring, intelligent tutoring systems, and other tutoring systems. Educ Psychol 46(4):197–221

    Article  Google Scholar 

  43. Woolf B, Arroyo I, Muldner K, Burleson W, Cooper D, Dolan R, Christopherson R (2010) The effect of motivational learning companions on low achieving students and students with disabilities. In: Intelligent Tutoring Systems, Springer, Berlin, pp 327–337

Download references

Acknowledgements

The content of this paper represents a subset of the broader software development and evaluation project conducted for the PhD thesis of Marissa Milne. The authors gratefully thank the participants and caregivers involved in the evaluation of the Social Tutor, without which the study would not have been possible. This study was conducted with ethical clearance from the Flinders University Social and Behavioural Research Ethics Committee and Autism SA Professional Practice Committee.

Author information

Authors and Affiliations

  1. College of Science and Engineering, Flinders Digital Health Research Centre, Flinders University, GPO Box 2100, Adelaide, 5001, South Australia

    Marissa Milne & David Martin Ward Powers

  2. College of Nursing and Health Sciences, Disability and Community Inclusion, Flinders University, GPO Box 2100, Adelaide, 5001, South Australia

    Parimala Raghavendra

  3. College of Medicine and Public Health, Flinders University, GPO Box 2100, Adelaide, 5001, South Australia

    Richard Leibbrandt

Authors
  1. Marissa Milne
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Parimala Raghavendra
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Richard Leibbrandt
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. David Martin Ward Powers
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Marissa Milne.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Milne, M., Raghavendra, P., Leibbrandt, R. et al. Personalisation and automation in a virtual conversation skills tutor for children with autism. J Multimodal User Interfaces 12, 257–269 (2018). https://doi.org/10.1007/s12193-018-0272-4

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s12193-018-0272-4

Keywords

Search

Navigation