Multimodal Fusion for Cognitive Load Measurement in an Adaptive Virtual Reality Driving Task for Autism Intervention

  • Lian ZhangEmail author
  • Joshua Wade
  • Dayi Bian
  • Jing Fan
  • Amy Swanson
  • Amy Weitlauf
  • Zachary Warren
  • Nilanjan Sarkar
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 9177)


A virtual reality driving system was designed to improve driving skills in individuals with autism spectrum disorder (ASD). An appropriate level of cognitive load during training can help improve a participant’s long-term performance. This paper studied cognitive load measurement with multimodal information fusion techniques. Features were extracted from peripheral physiological signals, Electroencephalogram (EEG) signals, eye gaze information and participants’ performance data. Multiple classification methods and features from different modalities were used to evaluate participant’s cognitive load. We verified classifications’ result with perceived tasks’ difficulty level, which induced different cognitive load. We fused multimodal information in three levels: feature level, decision level and hybrid level. The best accuracy for cognitive load measurement was 84.66 %, which was achieved with the hybrid level fusion.


Autism Virtual reality Multimodal fusion Cognitive load measurement 



This work was supported in part by the National Institute of Health Grant 1R01MH091102-01A1, National Science Foundation Grant 0967170 and the Hobbs Society Grant from the Vanderbilt Kennedy Center.


  1. 1.
    Wingate, M., Kirby, R.S., Pettygrove, S., et al.: Prevalence of autism spectrum disorder among children aged 8 years-autism and developmental disabilities monitoring network, 11 sites, United States, 2010. MMWR Surveillance Summaries, vol. 63, p. 2 (2014)Google Scholar
  2. 2.
    Rogers, S.J.: Empirically supported comprehensive treatments for young children with autism. J. Clin. Child Psychol. 27(2), 168–179 (1998)CrossRefGoogle Scholar
  3. 3.
    Cohen, H., Amerine-Dickens, M., Smith, T.: Early intensive behavioral treatment: Replication of the UCLA model in a community setting. J. Dev. Behav. Pediatr. 27(2), S145–S155 (2006)CrossRefGoogle Scholar
  4. 4.
    Strickland, D.: Virtual reality for the treatment of autism. In: Studies in Health Technology and Informatics, pp. 81–86 (1997)Google Scholar
  5. 5.
    Tartaro, A., Cassell, J.: Using virtual peer technology as an intervention for children with autism. In: Towards Universal Usability: Designing Computer Interfaces for Diverse User Populations, vol. 231, p. 62. John Wiley, Chichester (2007)Google Scholar
  6. 6.
    Lahiri, U., Bekele, E., Dohrmann, E., et al.: Design of a virtual reality based adaptive response technology for children with autism. IEEE Trans. Neural Syst. Rehabil. Eng. 21(1), 55–64 (2013)CrossRefGoogle Scholar
  7. 7.
    Cox, N.B., Reeve, R.E., Cox, S.M., et al.: Brief Report: Driving and young adults with ASD: Parents’ experiences. J. Autism Dev. Disord. 42(10), 2257–2262 (2012)CrossRefGoogle Scholar
  8. 8.
    Reimer, B., Fried, R., Mehler, B., et al.: Brief report: Examining driving behavior in young adults with high functioning autism spectrum disorders: A pilot study using a driving simulation paradigm. J. Autism Dev. Disord. 43(9), 2211–2217 (2013)CrossRefGoogle Scholar
  9. 9.
    Classen, S., Monahan, M.: Evidence-based review on interventions and determinants of driving performance in teens with attention deficit hyperactivity disorder or autism spectrum disorder. Traff. Inj. Prev. 14(2), 188–193 (2013)CrossRefGoogle Scholar
  10. 10.
    Galy, E., Cariou, M., Mélan, C.: What is the relationship between mental workload factors and cognitive load types? Int. J. Psychophysiol. 83(3), 269–275 (2012)CrossRefGoogle Scholar
  11. 11.
    Hussain, M.S., Calvo, R.A., Chen, F.: Automatic cognitive load detection from face, physiology, task performance and fusion during affective interference. Interacting with computers, p. iwt032 (2013)Google Scholar
  12. 12.
    Koenig, A., Novak, D., Omlin, X., et al.: Real-time closed-loop control of cognitive load in neurological patients during robot-assisted gait training. IEEE Trans. Neural Syst. Rehabil. Eng. 19(4), 453–464 (2011)CrossRefGoogle Scholar
  13. 13.
    Engström, J., Johansson, E., Östlund, J.: Effects of visual and cognitive load in real and simulated motorway driving. Transp. Res. Part F: Traffic Psychol. Behav. 8(2), 97–120 (2005)CrossRefGoogle Scholar
  14. 14.
    Paas, F., Tuovinen, J.E., Tabbers, H., et al.: Cognitive load measurement as a means to advance cognitive load theory. Educat. Psychol. 38(1), 63–71 (2003)CrossRefGoogle Scholar
  15. 15.
    Taelman, J., Vandeput, S., Spaepen, A., et al.: Influence of mental stress on heart rate and heart rate variability, pp. 1366–1369Google Scholar
  16. 16.
    Zhai, J., Barreto, A.: Stress Recognition Using Non-invasive Technology, pp. 395–401Google Scholar
  17. 17.
    Mehler, B., Reimer, B., Coughlin, J.F., et al.: Impact of incremental increases in cognitive workload on physiological arousal and performance in young adult drivers. Transp. Res. Rec.: J. Transp. Res. Board 2138(1), 6–12 (2009)CrossRefGoogle Scholar
  18. 18.
    Palinko, O., Kun, A.L., Shyrokov, A., et al.: Estimating cognitive load using remote eye tracking in a driving simulator, pp. 141–144Google Scholar
  19. 19.
    Pomplun, M., Sunkara, S.: Pupil dilation as an indicator of cognitive workload in human-computer interactionGoogle Scholar
  20. 20.
    Zarjam, P., Epps, J., Lovell, N.H., et al.: Characterization of memory load in an arithmetic task using non-linear analysis of EEG signals, pp. 3519–3522Google Scholar
  21. 21.
    Zarjam, P., Epps, J., Chen, F., et al.: Classification of working memory load using wavelet complexity features of EEG signals, pp. 692–699Google Scholar
  22. 22.
    Novak, D., Mihelj, M., Munih, M.: A survey of methods for data fusion and system adaptation using autonomic nervous system responses in physiological computing. Inter. with Comput. 24(3), 154–172 (2012)CrossRefGoogle Scholar
  23. 23.
    Wilson, G.F., Russell, C.A.: Performance enhancement in an uninhabited air vehicle task using psychophysiologically determined adaptive aiding. Hum. Factors: J. Hum. factors Ergon. Soc. 49(6), 1005–1018 (2007)CrossRefGoogle Scholar
  24. 24.
    Sarkar, N.: Psychophysiological control architecture for human-robot coordination-concepts and initial experiments, pp. 3719–3724Google Scholar
  25. 25.
    Rani, P., Sarkar, N., Smith, C.A., et al.: Affective communication for implicit human-machine interaction, pp. 4896–4903Google Scholar
  26. 26.
    Chen, F.: Robust Multimodal Cognitive Load Measurement, DTIC Document (2014)Google Scholar
  27. 27.
    Son, J., Park, M.: Estimating cognitive load complexity using performance and physiological data in a driving simulatorGoogle Scholar
  28. 28.
    Atrey, P.K., Hossain, M.A., El Saddik, A., et al.: Multimodal fusion for multimedia analysis: a survey. Multimedia Syst. 16(6), 345–379 (2010)CrossRefGoogle Scholar
  29. 29.
    Snoek, C.G., Worring, M., Smeulders, A.W.: Early versus late fusion in semantic video analysis, pp. 399–402Google Scholar
  30. 30.
    Wu, Z., Cai, L., Meng, H.: Multi-level fusion of audio and visual features for speaker identification. In: Zhang, D., Jain, A.K. (eds.) ICB 2005. LNCS, vol. 3832, pp. 493–499. Springer, Heidelberg (2005)CrossRefGoogle Scholar
  31. 31.
    Koelstra, S., Muhl, C., Soleymani, M., et al.: Deap: A database for emotion analysis; using physiological signals. IEEE Trans. Affect. Comput. 3(1), 18–31 (2012)CrossRefGoogle Scholar
  32. 32.
    Liu, E.S., Theodoropoulos, G.K.: Interest management for distributed virtual environments: A survey. ACM Comput. Surv. (CSUR) 46(4), 51 (2014)CrossRefGoogle Scholar
  33. 33.
    Wade, J., Bian, D., Zhang, L., Swanson, A., Sarkar, M., Warren, Z., Sarkar, N.: Design of a virtual reality driving environment to assess performance of teenagers with ASD. In: Stephanidis, C., Antona, M. (eds.) UAHCI 2014, Part II. LNCS, vol. 8514, pp. 466–474. Springer, Heidelberg (2014)Google Scholar
  34. 34.
    Bian, D., Wade, J.W., Zhang, L., Bekele, E., Swanson, A., Crittendon, J.A., Sarkar, M., Warren, Z., Sarkar, N.: A novel virtual reality driving environment for autism intervention. In: Stephanidis, C., Antona, M. (eds.) UAHCI 2013, Part II. LNCS, vol. 8010, pp. 474–483. Springer, Heidelberg (2013)Google Scholar
  35. 35.
    Liu, C., Conn, K., Sarkar, N., et al.: Physiology-based affect recognition for computer-assisted intervention of children with Autism Spectrum Disorder. Int. J. Hum.-Comput. Stud. 66(9), 662–677 (2008)CrossRefGoogle Scholar
  36. 36.
    Liu, C., Rani, P., Sarkar, N.: An empirical study of machine learning techniques for affect recognition in human-robot interaction, pp. 2662–2667Google Scholar
  37. 37.
    Bian, D., Wade, J., Swanson, A., et al.: Physiology-based affect recognition during driving in virtual environment for autism intervention. In: 2nd international conference on physiological computing system (Accepted, 2015)Google Scholar
  38. 38.
    Lord, C., Risi, S., Lambrecht, L., et al.: The Autism Diagnostic Observation Schedule—Generic: A standard measure of social and communication deficits associated with the spectrum of autism. J. Autism Dev. Disord. 30(3), 205–223 (2000)CrossRefGoogle Scholar
  39. 39.
    Elliott, C.D.: Differential Ability Scales-ll. Pearson, San Antonio (2007)Google Scholar
  40. 40.
    Wechsler, D.: Wechsler intelligence scale for children (1949)Google Scholar
  41. 41.
    Komogortsev, O.V., Gobert, D.V., Jayarathna, S., et al.: Standardization of automated analyses of oculomotor fixation and saccadic behaviors. IEEE Trans. Biomed. Eng. 57(11), 2635–2645 (2010)CrossRefGoogle Scholar
  42. 42.
    Benedetto, S., Pedrotti, M., Minin, L., et al.: Driver workload and eye blink duration. Trans. Research Part F: Traffic Psychol. Behav. 14(3), 199–208 (2011)CrossRefGoogle Scholar
  43. 43.
    Klem, G.H., Lüders, H.O., Jasper, H., et al.: The ten-twenty electrode system of the International Federation,” Electroencephalogr. Clin. Neurophysiol. 52 (suppl.), 3 (1999)Google Scholar
  44. 44.
    Delorme, A., Makeig, S.: EEGLAB: an open source toolbox for analysis of single-trial EEG dynamics including independent component analysis. J. Neurosci. Methods 134(1), 9–21 (2004)CrossRefGoogle Scholar

Copyright information

© Springer International Publishing Switzerland 2015

Authors and Affiliations

  • Lian Zhang
    • 1
    Email author
  • Joshua Wade
    • 1
  • Dayi Bian
    • 1
  • Jing Fan
    • 1
  • Amy Swanson
    • 2
  • Amy Weitlauf
    • 2
    • 3
  • Zachary Warren
    • 2
    • 3
  • Nilanjan Sarkar
    • 1
    • 4
  1. 1.Electrical Engineering and Computer Science DepartmentVanderbilt UniversityNashvilleUSA
  2. 2.Treatment and Research in Autism Spectrum Disorder (TRIAD)NashvilleUSA
  3. 3.Pediatrics and Psychiatry DepartmentNashvilleUSA
  4. 4.Mechanical Engineering DepartmentVanderbilt UniversityNashvilleUSA

Personalised recommendations