Design of an Assistive Avatar in Improving Eye Gaze Perception in Children with ASD During Virtual Interaction

  • Ashwaq Zaini AmatEmail author
  • Amy Swanson
  • Amy Weitlauf
  • Zachary Warren
  • Nilanjan Sarkar
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 10907)


Children diagnosed with autism spectrum disorder (ASD) usually experience impairment in social interaction and often display reduced gaze sharing when interacting with another person. The lack of gaze sharing or joint attention early in the children’s developmental stage may create a delay in their ability to learn new things and share information with others. The presented study involved the design of a novel virtual reality (VR)-based training game with an avatar and eye tracker aimed to eventually address the joint attention impairment in children with ASD. The assistive avatar provides necessary cues and hints based on both the eye tracking data recorded by the VR system and the task performance of the participant. The system uses the task performance to adaptively change the difficulty level of the game. We believe that the training game will be able to improve participant’s gaze following skill. A usability study was carried out to validate the system design. The result showed that the system was feasible and able to obtain the expected gaze performance from the participants. The details of the system architecture and result of system validation are presented in this paper.


Virtual reality Eye gaze Joint attention Autism spectrum disorder Children with ASD 



We are grateful for the support provided by NIH grants 1R21MH111548-01 for this research. The authors are solely responsible for the contents and opinions expressed in this manuscript.


  1. 1.
    Centers for Disease Control (CDC): Facts About ASD (2014)Google Scholar
  2. 2.
    Van Der Geest, J.N., Kemner, C., Verbaten, M.N., Van Engeland, H.: Gaze behavior of children with pervasive developmental disorder toward human faces: a fixation time study. J. Child Psychol. Psychiatry 43, 669–678 (2002). Scholar
  3. 3.
    Pelphrey, K.A., Sasson, N.J., Reznick, J.S., et al.: J. Autism Dev. Disord. 32, 249 (2002).
  4. 4.
    Dawson, G., et al.: Early social attention impairments in autism: social orienting, joint attention, and attention to distress. Dev. Psychol. 40(2), 271 (2004)CrossRefGoogle Scholar
  5. 5.
    Mundy, P., Newell, L.: Attention, joint attention, and social cognition. Curr. Dir. Psychol. Sci. 16(5), 269–274 (2007). Scholar
  6. 6.
    Charman, T., Swettenham, J., Baron-Cohen, S., Cox, A., Baird, G., Drew, A.: Infants with autism: an investigation of empathy, pretend play, joint attention, and imitation. Dev. Psychol. 33(5), 781–789 (1997). Scholar
  7. 7.
    Mundy, P., Sigman, M., Kasari, C.: A longitudinal study of joint attention and language development in autistic children. J. Autism Dev. Disord. 20(1), 115–128 (1990)CrossRefGoogle Scholar
  8. 8.
    Dawson, G., et al.: Early social attention impairments in autism: social orienting, joint attention, and attention to distress. Dev. Psychol. 40(2), 271 (1990)CrossRefGoogle Scholar
  9. 9.
    Moore, M., Calvert, S.: J. Autism Dev. Disord. 30, 359 (2000).
  10. 10.
    Bian, D., Wade, J.W., Zhang, L., Bekele, E., Swanson, A., Crittendon, J.A., Sarkar, M., Warren, Z., Sarkar, N.: A novel virtual reality driving environment for autism intervention. In: Stephanidis, C., Antona, M. (eds.) UAHCI 2013 Part II. LNCS, vol. 8010, pp. 474–483. Springer, Heidelberg (2013). Scholar
  11. 11.
    Zhang, L., et al.: Design of a mobile collaborative virtual environment for autism intervention. In: Antona, M., Stephanidis, C. (eds.) UAHCI 2016 Part III. LNCS, vol. 9739, pp. 265–275. Springer, Cham (2016). Scholar
  12. 12.
    Bekele, E., et al.: Understanding how adolescents with autism respond to facial expressions in virtual reality environments. IEEE Trans. Vis. Comput. Graph. 19(4), 711–720 (2013)CrossRefGoogle Scholar
  13. 13.
    Zheng, Z., Zhao, H., Swanson, A.R., Weitlauf, A.S., Warren, Z.E., Sarkar, N.: Design, development, and evaluation of a noninvasive autonomous robot-mediated joint attention intervention system for young children with ASD. IEEE Trans. Hum.-Mach. Syst. 48, 125–135 (2017)CrossRefGoogle Scholar
  14. 14.
    Caruana, N., et al.: Joint attention difficulties in autistic adults: an interactive eye-tracking study. Autism (2017).
  15. 15.
    Courgeon, M., et al.: Joint attention simulation using eye-tracking and virtual humans. IEEE Trans. Affect. Comput. 5(3), 238–250 (2014)CrossRefGoogle Scholar
  16. 16.
    Mundy, P., Crowson, M.: Joint attention and early social communication: implications for research on intervention with autism. J. Autism Dev. Dis. 27, 653 (1997)CrossRefGoogle Scholar
  17. 17.
    Kasari, C., Freeman, S., Paparella, T.: Joint attention and symbolic play in young children with autism: a randomized controlled intervention study. J. Child Psychol. Psychiatry 47, 611–620 (2006)CrossRefGoogle Scholar
  18. 18.
    Zheng, Z., Nie, G., Swanson, A., Weitlauf, A., Warren, Z., Sarkar, N.: Longitudinal impact of autonomous robot-mediated joint attention intervention for young children with ASD. In: Agah, A., Cabibihan, J.-J., Howard, A.M., Salichs, M.A., He, H. (eds.) ICSR 2016. LNCS (LNAI), vol. 9979, pp. 581–590. Springer, Cham (2016). Scholar
  19. 19.
  20. 20.
    Gibaldi, A., Vanegas, M., Bex, P.J., Maiello, G.: Evaluation of the Tobii EyeX Eye tracking controller and Matlab toolkit for research. Behav. Res. Methods 49(3), 923–946 (2017)CrossRefGoogle Scholar
  21. 21.
  22. 22.
  23. 23.

Copyright information

© Springer International Publishing AG, part of Springer Nature 2018

Authors and Affiliations

  • Ashwaq Zaini Amat
    • 1
  • Amy Swanson
    • 3
    • 4
  • Amy Weitlauf
    • 3
    • 4
  • Zachary Warren
    • 3
    • 4
  • Nilanjan Sarkar
    • 1
    • 2
  1. 1.Electrical Engineering and Computer ScienceVanderbilt UniversityNashvilleUSA
  2. 2.Mechanical EngineeringVanderbilt UniversityNashvilleUSA
  3. 3.Treatment and Research Institute for Autism Spectrum Disorders (TRIAD)Vanderbilt UniversityNashvilleUSA
  4. 4.Pediatrics, Psychiatry and Special EducationVanderbilt UniversityNashvilleUSA

Personalised recommendations