Skip to main content
SpringerLink
Log in
Book cover

International Conference on Human-Computer Interaction

HCII 2021: Augmented Cognition pp 119–133Cite as

Cognitive Workload Quantified by Physiological Sensors in Realistic Immersive Settings

  • Conference paper
  • First Online:

Part of the book series: Lecture Notes in Computer Science ((LNAI,volume 12776))

Abstract

Cognitive workload changes have been studied and utilized as a means of assessment for engagement and learner’s performance during training. Yet, it is unclear how varying levels of simulator immersion affect learner cognitive workload. Wearable sensors allow us to monitor direct physiological changes associated with cognitive workload in real time. This study seeks to utilize multiple physiological and neurological measures: functional near-infrared spectroscopy (fNIRS), eye-tracking, electrodermal activity (EDA), heart rate, and respiratory rate; in order to assess cognitive workload changes during different training conditions. The National Aeronautics and Space Administration’s (NASA) Task Load Index (TLX) and flow state scale questionnaires were additionally used to record self-reported cognitive workload and subjective experience. Nine law enforcement trainees participated in different immersions conditions in a law enforcement use-of-force (UOF) simulator. Results from a low immersion condition were compared to results from a high immersion condition. Preliminary comparison between these two conditions suggests that the Index of Cognitive Activity (ICA) and respiration rate were greater in the low immersion condition. However, a notable increase in the oxygenated hemoglobin of the right anterior medial prefrontal cortex was detected via fNIRS. Heart rate also increased between the two conditions. Traditional questionnaires used to measure cognitive load showed no significance between conditions. Compared to self-report subjective metrics, biometrics such as fNIRS were operationally more effective at smaller sample sizes. Not only do these results show that features associated with trainees’ workload can viably be collected in realistic simulator settings, but they also suggest that increased immersion in law enforcement simulators may have a measurable effect on biometrics associated with cognitive workload.

This is a preview of subscription content, log in via an institution.

Chapter
USD   29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD   39.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD   54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Learn about institutional subscriptions

References

  1. Hancock, P.A., Chignell, M.H.: Toward a theory of mental workload: stress and adaptability in human-machine systems. In: Proceedings of the International IEEE Conference on Systems, Man and Cybernetics, pp. 378–383 (1986)

    Google Scholar 

  2. Welford, A.T.: Forty years of experimental psychology in relation to age: retrospect and prospect. Exp. Gerontol. 21, 469–481 (1986)

    Article  Google Scholar 

  3. Baldwin, C.L., Coyne, J.T.: Mental Workload as a Function of Traffic Density: Comparison of Physiological, Behavioral, and Subjective Indices (2003)

    Google Scholar 

  4. Hart, S.G., Staveland, L.E.: Development of NASA-TLX (task load index): results of empirical and theoretical research. In: Hancock, P.A., Meshkati, N. (eds.) Advances in Psychology, Human Mental Workload, North-Holland, vol. 52, pp. 139–183 (1988). https://doi.org/10.1016/S0166-4115(08)62386-9

  5. Szulewski, A., Gegenfurtner, A., Howes, D.W., Sivilotti, M.L.A., van Merriënboer, J.J.G.: Measuring physician cognitive load: validity evidence for a physiologic and a psychometric tool. Adv. Health Sci. Educ. 22(4), 951–968 (2016). https://doi.org/10.1007/s10459-016-9725-2

    Article  Google Scholar 

  6. Aksoy, E., Izzetoglu, K., Baysoy, E., Agrali, A., Kitapcioglu, D., Onaral, B.: Performance monitoring via functional near infrared spectroscopy for virtual reality based basic life support training. Front. Neurosci. 13, 1336 (2019). https://doi.org/10.3389/fnins.2019.01336

    Article  Google Scholar 

  7. Shewokis, P.A., Shariff, F.U., Liu, Y., Ayaz, H., Castellanos, A., Lind, D.S.: Acquisition, retention and transfer of simulated laparoscopic tasks using fNIR and a contextual interference paradigm. Am. J. Surg. 213(2), 336–345 (2017). https://doi.org/10.1016/j.amjsurg.2016.11.043

    Article  Google Scholar 

  8. Izzetoglu, K., et al.: The evolution of field deployable fNIR spectroscopy from bench to clinical settings. J. Innov. Opt. Health Sci. 4(3), 239–250 (2011). https://doi.org/10.1142/S1793545811001587

    Article  Google Scholar 

  9. Strangman, G., Boas, D., Sutton, J.: Non-invasive neuroimaging using near-infrared light. Biol. Psychiat. 52(7), 679–693 (2002)

    Article  Google Scholar 

  10. Izzetoglu, K., et al.: UAV operators workload assessment by optical brain imaging technology (fNIR). In: Valavanis, K.P., Vachtsevanos, G.J. (eds.) Handbook of Unmanned Aerial Vehicles, pp. 2475–2500. Springer, Dordrecht (2015). https://doi.org/10.1007/978-90-481-9707-1_22

    Chapter  Google Scholar 

  11. Dawson, M.E., Schell, A.M., Filion, D.L.: The electrodermal system. In: Cacioppo, J.T., Tassinary, L.G., Berntson, G.G. (eds.) Cambridge Handbooks in Psychology, pp. 200–223. Cambridge University Press, Handbook of psychophysiology (2000)

    Google Scholar 

  12. Hess, E.H., Polt, J.M.: Pupil size in relation to mental activity during simple problem-solving. Science 143(3611), 1190–1192 (1964)

    Article  Google Scholar 

  13. Demberg, V., Sayeed, A.: The frequency of rapid pupil dilations as a measure of linguistic processing difficulty. PLoS ONE 11(1), 1–30 (2016)

    Article  Google Scholar 

  14. Vogels, J., Demberg, V., Kray, J.: The index of cognitive activity as a measure of cognitive processing load in dual task settings. Front. Psychol. 9, 1–19 (2018). https://doi.org/10.3389/fpsyg.2018.02276

    Article  Google Scholar 

  15. Thayer, J.F., Åhs, F., Fredrikson, M., Sollers, J.J., Wager, T.D.: A meta-analysis of heart rate variability and neuroimaging studies: implications for heart rate variability as a marker of stress and health. Neurosci. Biobehav. Rev. 36(2), 747–756 (2012). https://doi.org/10.1016/j.neubiorev.2011.11.009

    Article  Google Scholar 

  16. Thayer, J.F., Hansen, A.L., Saus-Rose, E., Johnsen, B.H.: Heart rate variability, prefrontal neural function, and cognitive performance: the neurovisceral integration perspective on self-regulation, adaptation, and health. Ann. Behav. Med. 37(2), 141–153 (2009)

    Article  Google Scholar 

  17. Grassmann, M., Vlemincx, E., Von Leupoldt, A., Mittelstädt, J., Den Bergh, O.: Respiratory Changes in Response to Cognitive Load: A Systematic Review. Hindawi Publishing Corporation (2016)

    Google Scholar 

  18. Marshall, S.P.: The index of cognitive activity: measuring cognitive workload. In: Proceedings of the IEEE 7th Conference on Human Factors and Power Plants, Scottsdale, AZ, USA, p. 7 (2002)

    Google Scholar 

  19. Devos, H., Gustafson, K., Ahmadnezhad, P., Liao, K., Mahnken, J., Brooks, W., Burns, J.: Psychometric properties of NASA-TLX and index of cognitive activity as measures of cognitive workload in older adults. Brain Sci. 10(12):994 (2020). https://doi.org/10.3390/brainsci10120994

  20. Fowles, D., Christie, M., Edelberg, R., Grings, W., Lykken, D., Venables, P.: Publication recommendations for electrodermal measurements. Psychophysiology 18(3), 232–239 (1981)

    Google Scholar 

  21. National Aeronautics and Space Administration. https://humansystems.arc.nasa.gov/groups/tlx/

  22. Jackson, S.A., Marsh, H.W.: Development and validation of a scale to measure optimal experience: the flow state scale. J. Sport Exerc. Psychol. 18(1), 17–35 (1996). https://doi.org/10.1123/jsep.18.1.17

    Article  Google Scholar 

  23. Ledalab. http://www.ledalab.de/

  24. Boucsein, W.: Electrodermal Activity, 2nd edn. Springer, New York (2012)

    Book  Google Scholar 

  25. Benedek, M., Kaernbach, C.: Decomposition of skin conductance data by means of nonnegative deconvolution. Psychophysiology (2010). https://doi.org/10.1111/j.1469-8986.2009.00972.x

    Article  Google Scholar 

  26. Boucsein, W., et al.: Publication recommendations for electrodermal measurements. Psychophysiology 49(8), 1017–1034 (2012). https://doi.org/10.1111/j.1469-8986.2012.01384.x

    Article  Google Scholar 

  27. Reddy, P., Richards, D., Izzetoglu, K.: Cognitive performance assessment of UAS sensor operators via neurophysiological measures. Front. Hum. Neurosci. 12 (2018). https://doi.org/10.3389/conf.fnhum.2018.227.00032

  28. Kerr, J., Reddy, P., Kosti, S., Izzetoglu, K.: UAS operator workload assessment during search and surveillance tasks through simulated fluctuations in environmental visibility. In: Schmorrow, D.D., Fidopiastis, C.M. (eds.) HCII 2019. LNCS (LNAI), vol. 11580, pp. 394–406. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-22419-6_28

    Chapter  Google Scholar 

  29. Izzetoglu, K., Bunce, S., Onaral, B., Pourrezaei, K., Chance, B.: Functional optical brain imaging using near-infrared during cognitive tasks. Int. J. Hum.-Comput. Interact. 17(2), 211–227 (2010). https://doi.org/10.1207/s15327590ijhc1702_6

    Article  Google Scholar 

  30. Razali, N.M., Wah, Y.B.: Power comparisons of shapiro-wilk, kolmogorov-smirnov, lilliefors and anderson-darling tests. J. Statist. Model. Analyt. 2(1), 21–33 (2011)

    Google Scholar 

  31. Visnovcova, Z., Mestanik, M., Gala, M., Mestanikova, A., Tonhajzerova, I.: The complexity of electrodermal activity is altered in mental cognitive stressors. Comput. Biol. Med. 79, 123–129 (2016). https://doi.org/10.1016/j.compbiomed.2016.10.014

    Article  Google Scholar 

  32. Hill, S., Iavecchia, H., Byers, J., Bittner, A., Zaklad, A., Christ, R.: Comparison of four subjective workload rating scales. Hum. Factors 34(4), 429–439 (1992). https://doi.org/10.1177/001872089203400405

    Article  Google Scholar 

  33. Yoshida, K., et al.: The flow state scale for occupational tasks: development, reliability, and validity. Hong Kong J. Occup. Ther. 23(2), 54–61 (2013). https://doi.org/10.1016/j.hkjot.2013.09.002

    Article  Google Scholar 

  34. Schmitz, T., Johnson, S.: Self-appraisal decisions evoke dissociated dorsal—ventral aMPFC networks. NeuroImage Orlando Fla. 30(3), 1050–1058 (2006). https://doi.org/10.1016/j.neuroimage.2005.10.030

    Article  Google Scholar 

  35. Boutcher, Y.N., Boutcher, S.H.: Cardiovascular response to stroop: effect of verbal response and task difficulty. Biol. Psychol. 73(3), 235–241 (2006). https://doi.org/10.1016/j.biopsycho.2006.04.005

    Article  Google Scholar 

  36. Izzetoglu, K., et al.: Applications of functional near infrared imaging: case study on UAV ground controller. In: Schmorrow, D.D., Fidopiastis, C.M. (eds.) FAC 2011. LNCS (LNAI), vol. 6780, pp. 608–617. Springer, Heidelberg (2011). https://doi.org/10.1007/978-3-642-21852-1_70

    Chapter  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. School of Biomedical Engineering, Science and Health Systems, Drexel University, Philadelphia, USA

    Ashley Bishop, Emma MacNeil & Kurtulus Izzetoglu

Authors
  1. Ashley Bishop
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Emma MacNeil
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Kurtulus Izzetoglu
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Kurtulus Izzetoglu .

Editor information

Editors and Affiliations

  1. Soar Technology Inc., Orlando, FL, USA

    Dylan D. Schmorrow

  2. Design Interactive, Inc., Orlando, FL, USA

    Cali M. Fidopiastis

Rights and permissions

Reprints and permissions

Copyright information

© 2021 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Bishop, A., MacNeil, E., Izzetoglu, K. (2021). Cognitive Workload Quantified by Physiological Sensors in Realistic Immersive Settings. In: Schmorrow, D.D., Fidopiastis, C.M. (eds) Augmented Cognition. HCII 2021. Lecture Notes in Computer Science(), vol 12776. Springer, Cham. https://doi.org/10.1007/978-3-030-78114-9_9

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-78114-9_9

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-78113-2

  • Online ISBN: 978-3-030-78114-9

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation