Skip to main content
SpringerLink
Log in

Diagnosing Cognitive Control with Eye-Tracking Metrics in a Multitasking Environment

  • Conference paper
  • First Online:
Engineering Psychology and Cognitive Ergonomics (HCII 2024)

Part of the book series: Lecture Notes in Computer Science ((LNAI,volume 14692))

Included in the following conference series:

  • 276 Accesses

Abstract

In multitasking environments, such as military flight missions, effective task prioritization is crucial to ensure overall flight safety. Cognitive control plays a vital role in this process, balancing stability for goal pursuit and flexibility for reacting to unexpected events. This can be challenging, since cognitive stability is also associated with more difficult task switching and cognitive flexibility is linked to distractedness. This study explores the use of eye-tracking metrics to diagnose the cognitive control state of operators within an adaptive assistance system in a multitasking environment. Three studies, involving 144 participants, manipulated control modes using a task prioritization strategy in a low-fidelity flight simulator. Eye movement data was recorded at 1000 Hz. The study employed eight supervised machine learning algorithms for binary classification, namely random forest, k-nearest neighbors, support vector machines, naïve bayes, decision trees, logistic regression, linear discriminant analysis, and XGBoost. The average accuracy of 0.89 demonstrates that the recognition of the cognitive control mode via eye-tracking is feasible. The findings suggest the potential integration of eye-tracking metrics into real-time user state diagnosis for adaptive assistance systems, especially in safety-critical human-machine systems. However, further research is needed to validate the classifier in more realistic flight environments with an expert sample.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 99.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 74.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Disclosure of Interests

The authors have no competing interests to declare that are relevant to the content of this article.

References

  1. Wickens, C.D.: The structure of attentional resources. In: Nickerson, R. (ed.) Attention and Performance, pp. 239–257. Erlbaum, Hillsdale, NJ (1980)

    Google Scholar 

  2. Oberauer, K., Kliegl R.: A formal model of capacity limits in working memory. J. Memory Lang. 55(4), 601–626 (2006). https://www.sciencedirect.com/science/article/pii/S0749596X06000982

  3. Salvucci, D.D., Taatgen, N.A.: Threaded cognition: an integrated theory of concurrent multitasking. Psychol. Rev. 115(1), 101–130 (2008)

    Article  Google Scholar 

  4. Wickens, C.D., Goh, J., Helleberg, J., Horrey, W.J., Talleur, D.A.: Attentional models of multitask pilot performance using advanced display technology. In: Human Error in Aviation, pp. 155–175 (2017)

    Google Scholar 

  5. Chérif, L., Wood, V., Marois, A., Labonté, K., Vachon, F.: Multitasking in the military: cognitive consequences and potential solutions. Appl. Cogn. Psychol. 32(4), 429–439 (2018)

    Article  Google Scholar 

  6. Loukopoulos, L.D., Dismukes, R.K., Barshi, I.: The Multitasking Myth: Handling Complexity in Real-World Operations. Routledge, London (2016)

    Google Scholar 

  7. Watson, J.M., Strayer, D.L.: Supertaskers: profiles in extraordinary multitasking ability. Psychon. Bull. Rev. 17, 479–485 (2010)

    Article  Google Scholar 

  8. Kelly, D., Efthymiou, M.: An analysis of human factors in fifty controlled flight into terrain aviation accidents from 2007 to 2017. J. Safety Res. 69, 155–165 (2019)

    Article  Google Scholar 

  9. Chou, C.C., Madhavan, D., Funk, K.: Studies of cockpit task management errors. Int. J. Aviat. Psychol. 6(4), 307–320 (1996)

    Article  Google Scholar 

  10. Mackie, M.A., van Dam, N.T., Fan, J.: Cognitive control and attentional functions. Brain Cogn. 82(3), 301–312 (2013)

    Article  Google Scholar 

  11. Liegel, N., Schneider, D., Wascher, E., Arnau, S.: Task prioritization modulates alpha, theta and beta EEG dynamics reflecting proactive cognitive control. Sci. Rep. 12(1), 15072 (2022)

    Article  Google Scholar 

  12. Fischer, R., Gottschalk, C., Dreisbach, G.: Context-sensitive adjustment of cognitive control in dual-task performance. J. Exp. Psychol. Learn. Mem. Cogn. 40(2), 399–416 (2014)

    Article  Google Scholar 

  13. Goschke, T., Bolte, A.: Emotional modulation of control dilemmas: the role of positive affect, reward, and dopamine in cognitive stability and flexibility. Neuropsychologia 62, 403–423 (2014). https://www.sciencedirect.com/science/article/pii/S0028393214002358

  14. Musslick, S., Jang, S.J., Shvartsman, M., Shenhav, A., Cohen, J.D.: Constraints associated with cognitive control and the stability-flexibility dilemma. In: Cognitive Science (2018). https://api.semanticscholar.org/CorpusID:117726732

  15. Dreisbach, G., Fröber, K.: On How to Be Flexible (or Not): modulation of the stability-flexibility balance. Curr. Dir. Psychol. Sci. 28(1), 3–9 (2019)

    Article  Google Scholar 

  16. Nassar, M.R., Troiani, V.: The stability flexibility tradeoff and the dark side of detail. Cogn. Affect. Behav. Neurosci. 21(3), 607–623 (2021)

    Article  Google Scholar 

  17. Stasch, S., Mack, W.: Why the stability-flexibility-dilemma should be taken into consideration when studying pilots multitasking behaviour. In: Praetorius, G., Sellberg, C., Patriarca, R. (eds.) Human Factors in Transportation. AHFE International Conference. AHFE Open Access, vol. 95. AHFE International, USA (2023). https://doi.org/10.54941/ahfe1003846

  18. Feigh, K.M., Dorneich, M.C., Hayes, C.C.: Toward a characterization of adaptive systems: a framework for researchers and system designers. Hum. Factors 54(6), 1008–1024 (2012)

    Article  Google Scholar 

  19. Wickens, C.D.: Engineering Psychology and Human Performance, 2nd edn. HarperCollins Publishers, New York (1992)

    Google Scholar 

  20. Schwarz, J., Fuchs, S., Flemisch, F.: Towards a more holistic view on user state assessment in adaptive human-computer interaction. In: 2014 IEEE International Conference on Systems, Man, and Cybernetics (SMC), pp. 1228–1234. IEEE (2014)

    Google Scholar 

  21. Liu, J., Gardi, A., Ramasamy, S., Lim, Y., Sabatini, R.: Cognitive pilot-aircraft interface for single-pilot operations. Knowl.-Based Syst. 112, 37–53 (2016)

    Article  Google Scholar 

  22. Di Stasi, L.L., Diaz-Piedra, C.: Re-examining the pioneering studies on eye movements in aviation: connecting the past to the present. Int. J. Aerosp. Psychol. 31(2), 122–134 (2021)

    Article  Google Scholar 

  23. Peißl, S., Wickens, C.D., Baruah, R.: Eye-tracking measures in aviation: a selective literature review. Int. J. Aerosp. Psychol. 28(3–4), 98–112 (2018)

    Article  Google Scholar 

  24. Peysakhovich, V., Lefrançois, O., Dehais, F., Causse, M.: The neuroergonomics of aircraft cockpits: the four stages of eye-tracking integration to enhance flight safety. Safety 4(1), 8 (2018)

    Article  Google Scholar 

  25. Lavie, N., Hirst, A., Fockert, J.W., Viding, E.: Load theory of selective attention and cognitive control. J. Exp. Psychol. Gen. 133(3), 339–354 (2004)

    Article  Google Scholar 

  26. Joseph, A.W., Murugesh, R.: Potential eye tracking metrics and indicators to measure cognitive load. Hum.-Comput. Interact. Res. 64(01), 168–175 (2020)

    Google Scholar 

  27. Krejtz, K., Duchowski, A., Krejtz, I., Szarkowska, A., Kopacz, A.: Discerning ambient/focal attention with coefficient K. ACM Trans. Appl. Percept. 13(3), 1–20 (2016)

    Article  Google Scholar 

  28. Shannon, C.E.: A mathematical theory of communication. Bell Syst. Tech. J. 27(3), 379–423 (1948)

    Article  MathSciNet  Google Scholar 

  29. Krejtz, K., et al.: Gaze transition entropy. ACM Trans. Appl. Percept. 13(1), 1–20 (2015)

    Google Scholar 

  30. Yarbus, A.L.: Eye Movements and Vision. Springer, Cham (2013). https://doi.org/10.1007/978-1-4899-5379-7

  31. Young, A.H., Hulleman, J.: Eye movements reveal how task difficulty moulds visual search. J. Exp. Psychol. Hum. Percept. Perform. 39(1), 168 (2013)

    Article  Google Scholar 

  32. Rayner, K.: Eye movements in reading and information processing. Psychol. Bull. 85(3), 618 (1978)

    Article  MathSciNet  Google Scholar 

  33. Salthouse, T.A., Ellis, C.L.: Determinants of eye-fixation duration. Am. J. Psychol. 93, 207–234 (1978)

    Google Scholar 

  34. Shiferaw, B., Downey, L., Crewther, D.: A review of gaze entropy as a measure of visual scanning efficiency. Neurosci. Biobehav. Rev. 96, 353–366 (2019)

    Article  Google Scholar 

  35. Cegarra, J., Valéry, B., Avril, E., Calmettes, C., Navarro, J.: OpenMATB: a multi-attribute task battery promoting task customization, software extensibility and experiment replicability. Behav. Res. Methods 52(5), 1980–1990 (2020)

    Article  Google Scholar 

  36. Stasch, S., Mack, W.: A new experimental method to investigate multitasking strategies in flight environments via the use of gamification. In: Proceedings of the Human Factors and Ergonomics Society Europe Chapter 2023 Annual Conference, pp. 23–33 (2023). http://hfes-europe.org

  37. Stevens, L.M., Mortazavi, B.J., Deo, R.C., Curtis, L., Kao, D.P.: Recommendations for reporting machine learning analyses in clinical research. Circ. Cardiovasc. Qual. Outcomes 13(10), 006556 (2020)

    Google Scholar 

  38. Jeffreys, H.: Theory of Probability. Clarendon Press, Oxford (1939)

    Google Scholar 

  39. Pedregosa, F., et al.: Scikit-learn: machine learning in Python. J. Mach. Learn. Res. 12, 2825–2830 (2011)

    MathSciNet  Google Scholar 

  40. Yang, J.H., Kennedy, Q., Sullivan, J., Fricker, R.D.: Pilot performance: assessing how scan patterns & navigational assessments vary by flight expertise. Aviat. Space Environ. Med. 84(2), 116–124 (2013)

    Article  Google Scholar 

  41. Robinski, M., Stein, M.: Tracking visual scanning techniques in training simulation for helicopter landing. JEMR 6(2) (2013)

    Google Scholar 

  42. Kirby, C.E., Kennedy, Q., Yang, J.H.: Helicopter pilot scan techniques during low-altitude high-speed flight. Aviat. Space Environ. Med. 85(7), 740–744 (2014)

    Article  Google Scholar 

  43. Seong, P.H., Kang, H.G., Na, M.G., Kim, J.H., Heo, G., Jung, Y.: Advanced MMIS toward substantial reduction in human errors in NPPs. Nucl. Eng. Technol. 45(2), 125–140 (2013)

    Article  Google Scholar 

  44. Enjalbert, S., Gandini, L.M., Pereda Baños, A., Ricci, S., Vanderhaegen, F.: Human-machine interface in transport systems: an industrial overview for more extended rail applications. Machines 9(2), 36 (2021)

    Article  Google Scholar 

  45. Liu, C., et al.: Human–machine cooperation research for navigation of maritime autonomous surface ships: a review and consideration. Ocean Eng. 246, 110555 (2022)

    Article  Google Scholar 

  46. Di Flumeri, G., et al.: Brain-computer interface-based adaptive automation to prevent out-of-the-loop phenomenon in air traffic controllers dealing with highly automated systems. Front. Hum. Neurosci. 13, 296 (2019)

    Article  Google Scholar 

Download references

Acknowledgments

This research is funded by dtec.bw – Digitalization and Technology Research Center of the Bundeswehr (project MissionLab). Dtec.bw is funded by the European Union – NextGenerationEU.

Author information

Authors and Affiliations

  1. Universität der Bundeswehr München, Werner-Heisenberg-Weg 39, 85579, Neubiberg, Germany

    Sophie-Marie Stasch & Wolfgang Mack

Authors
  1. Sophie-Marie Stasch
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Wolfgang Mack
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Sophie-Marie Stasch .

Editor information

Editors and Affiliations

  1. Coventry University, Coventry, UK

    Don Harris

  2. Cranfield University, Cranfield, UK

    Wen-Chin Li

Rights and permissions

Reprints and permissions

Copyright information

© 2024 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Stasch, SM., Mack, W. (2024). Diagnosing Cognitive Control with Eye-Tracking Metrics in a Multitasking Environment. In: Harris, D., Li, WC. (eds) Engineering Psychology and Cognitive Ergonomics. HCII 2024. Lecture Notes in Computer Science(), vol 14692. Springer, Cham. https://doi.org/10.1007/978-3-031-60728-8_8

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-60728-8_8

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-60727-1

  • Online ISBN: 978-3-031-60728-8

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation