Online Engagement Detection and Task Adaptation in a Virtual Reality Based Driving Simulator for Autism Intervention

  • Dayi BianEmail author
  • Joshua Wade
  • Zachary Warren
  • Nilanjan Sarkar
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 9739)


Individuals with Autism spectrum disorder (ASD) have difficulty functioning independently on essential tasks that require adaptive skills such as driving. Recently, computer-aided technology, such as Virtual Reality (VR), is being widely used in ASD intervention to teach basic skills to children with autism. However, most of these works either do not use feedback or solely use performance feedback from the participant for system adaptation. This paper introduces a physiology-based task adaptation mechanism in a virtual environment for driving skill training. The difficulty of the driving task was autonomously adjusted based on the participant’s performance and engagement level to provide the participant with an optimal level of challenge. The engagement level was detected using an affective model which was developed based on our previous experimental data and a therapist’s ratings. We believe that this physiology-based adaptive mechanism can be useful in teaching driving skills to adolescents with ASD.


Virtual Reality (VR) Driving simulator ASD intervention Dynamic difficulty adjustment (DDA) Physiological signals Machine learning Affective computing Human computer interaction 



We gratefully acknowledge the support provided by the National Institute of Health Grant 1R01MH091102-01A1 to perform the presented research.


  1. 1.
    CDC, Prevalence of Autism Spectrum Disorder Among Children Aged 8 Years—Autism and Developmental Disabilities Monitoring Network, 11 Sites, United States, 2010 (2014)Google Scholar
  2. 2.
    Ganz, M.L.: The Costs of Autism, pp. 475–502. CRC Press, New York (2006)Google Scholar
  3. 3.
    Chasson, G.S., Harris, G.E., Neely, W.J.: Cost comparison of early intensive behavioral intervention and special education for children with autism. J. Child Fam. Stud. 16(3), 401–413 (2007)CrossRefGoogle Scholar
  4. 4.
    Rogers, S.J.: Empirically supported comprehensive treatments for young children with autism. J. Clin. Child Psychol. 27(2), 168–179 (1998)CrossRefGoogle Scholar
  5. 5.
    Palmen, A., Didden, R., Lang, R.: A systematic review of behavioral intervention research on adaptive skill building in high-functioning young adults with autism spectrum disorder. Res. Autism Spectrum Disord. 6(2), 602–617 (2012)CrossRefGoogle Scholar
  6. 6.
    Welch, K.C., Lahiri, U., Liu, C., Weller, R., Sarkar, N., Warren, Z.: An affect-sensitive social interaction paradigm utilizing virtual reality environments for autism intervention. In: Jacko, J.A. (ed.) HCI International 2009, Part III. LNCS, vol. 5612, pp. 703–712. Springer, Heidelberg (2009)Google Scholar
  7. 7.
    Lahiri, U., Warren, Z., Sarkar, N.: Dynamic gaze measurement with adaptive response technology in Virtual Reality based social communication for autism. In: 2011 International Conference on Virtual Rehabilitation (ICVR). IEEE (2011)Google Scholar
  8. 8.
    Lahiri, U., et al.: Design of a virtual reality based adaptive response technology for children with autism. IEEE Trans. Neural Syst. Rehabil. Eng. 21(1), 55–64 (2013)CrossRefGoogle Scholar
  9. 9.
    Lahiri, U., Warren, Z., Sarkar, N.: Design of a gaze-sensitive virtual social interactive system for children with autism. IEEE Trans. Neural Syst. Rehabil. Eng. 19(4), 443–452 (2011)CrossRefGoogle Scholar
  10. 10.
    Liu, C., et al.: Online affect detection and robot behavior adaptation for intervention of children with autism. IEEE Trans. Robotics 24(4), 883–896 (2008)CrossRefGoogle Scholar
  11. 11.
    Tartaro, A., Cassell, J.: Using virtual peer technology as an intervention for children with autism. Towards Universal Usability: Designing Computer Interfaces for Diverse User Populations, vol. 231, p. 62. Wiley, Chichester (2007)Google Scholar
  12. 12.
    Giakoumis, D., et al.: Identifying psychophysiological correlates of boredom and negative mood induced during HCI. In: B-Interface (2010)Google Scholar
  13. 13.
    Pekrun, R., et al.: Boredom in achievement settings: Exploring control–value antecedents and performance outcomes of a neglected emotion. J. Educ. Psychol. 102(3), 531 (2010)CrossRefGoogle Scholar
  14. 14.
    Rani, P., Sarkar, N., Liu, C.: Maintaining optimal challenge in computer games through real-time physiological feedback. In: Proceedings of the 11th international conference on human computer interaction (2005)Google Scholar
  15. 15.
    Dayi, B., et al.: Physiology-based affect recognition during driving in virtual environment for autism intervention. In: 2nd International Conference on Physiological Computing Systems (2015)Google Scholar
  16. 16.
    Bellani, M., et al.: Virtual reality in autism: state of the art. Epidemiol. Psychiatr. Sci. 20(03), 235–238 (2011)CrossRefGoogle Scholar
  17. 17.
    Sarkar, N.: Psychophysiological control architecture for human-robot coordination-concepts and initial experiments. In: IEEE International Conference on Robotics and Automation, Proceedings, ICRA 2002. IEEE (2002)Google Scholar
  18. 18.
    Stemmler, G., et al.: Constraints for emotion specificity in fear and anger: The context counts. Psychophysiology 38(2), 275–291 (2001)CrossRefGoogle Scholar
  19. 19.
    Hall, M., et al.: The WEKA data mining software: an update. ACM SIGKDD explorations newsletter 11(1), 10–18 (2009)CrossRefGoogle Scholar
  20. 20.
    Bian, D., Wade, J.W., Zhang, L., Bekele, E., Swanson, A., Crittendon, J.A., Sarkar, M., Warren, Z., Sarkar, N.: A novel virtual reality driving environment for autism intervention. In: Stephanidis, C., Antona, M. (eds.) UAHCI 2013, Part II. LNCS, vol. 8010, pp. 474–483. Springer, Heidelberg (2013)Google Scholar
  21. 21.
    Wade, J., Bian, D., Fan, J., Zhang, L., Swanson, A., Sarkar, M., Weitlauf, A., Warren, Z., Sarkar, N.: A virtual reality driving environment for training safe gaze patterns: application in individuals with ASD. In: Antona, M., Stephanidis, C. (eds.) UAHCI 2015. LNCS, vol. 9177, pp. 689–697. Springer, Heidelberg (2015)CrossRefGoogle Scholar
  22. 22.
    Lord, C., et al.: The autism diagnostic observation schedule—generic: a standard measure of social and communication deficits associated with the spectrum of autism. J. Autism Dev. Disord. 30(3), 205–223 (2000)CrossRefGoogle Scholar

Copyright information

© Springer International Publishing Switzerland 2016

Authors and Affiliations

  • Dayi Bian
    • 1
    Email author
  • Joshua Wade
    • 3
  • Zachary Warren
    • 4
    • 5
    • 6
  • Nilanjan Sarkar
    • 1
    • 2
  1. 1.Department of Electrical EngineeringVanderbilt UniversityNashvilleUSA
  2. 2.Department of Mechanical EngineeringVanderbilt UniversityNashvilleUSA
  3. 3.Department of Computer ScienceVanderbilt UniversityNashvilleUSA
  4. 4.Department of Pediatrics and PsychiatryVanderbilt UniversityNashvilleUSA
  5. 5.Department of Special EducationVanderbilt UniversityNashvilleUSA
  6. 6.Vanderbilt Kennedy Center, Treatment and Research Institute of Autism Spectrum DisordersVanderbilt UniversityNashvilleUSA

Personalised recommendations