Abstract
Traditionally, automation was introduced to alleviate workload associated with tedious and repetitive tasks. Recently, automation is being used to augment knowledge work, which includes high-level cognitive activities. As automated systems are being extended to perform skill-based tasks, the work required of humans may be altered, potentially affecting their cognitive workloads. Researchers have investigated the influences of automation on cognitive workload across different domains and tasks by assessing changes in task performance, perceived (subjective) workload, and physiological states. A major challenge in comparing results and drawing inferences across studies is that a profusion of measures is often used to assess cognitive workload. The experimental tasks employed across many domains further complicates synthesizing findings. Thus, the aim of this review is to examine how cognitive workload is assessed when at least two different measures of cognitive workload are used in research focused on human-automated knowledge work. To accomplish this aim, the various approaches employed to measure cognitive workload were first summarized. Then, automated and cognitive experimental tasks were classified, utilizing existing frameworks, to identify associations, dissociations, and insensitivities across task types. Finally, recommendations were provided for aligning task types, study designs, and measurement selections, along with expanding the types of tasks and measures used when studying automation applications supporting knowledge work.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Annett J (2002) Subjective rating scales: science or art? Ergonomics 45(14):966–987. https://doi.org/10.1080/00140130210166951
Beer RD, Rieth CA, Tran R, Cook MB (2017) Framework for multi-human multi-robot interaction: impact of operational context and team configuration on interaction task demands. In: 2017 AAAI spring symposium—technical report, SS-17-01-, pp 287–294
Biondi FN, Balasingam B, Ayare P (2021) On the cost of detection response task performance on cognitive load. Hum Factors 63(5):804–812. https://doi.org/10.1177/0018720820931628
Boehm U, Matzke D, Gretton M, Castro S, Cooper J, Skinner M, Heathcote A et al (2021) Real-time prediction of short-timescale fluctuations in cognitive workload. Cognit Res Princ Implic 6(1):1–29. https://doi.org/10.1186/s41235-021-00289-y
Brookhuis KA, van Driel CJG, Hof T, van Arem B, Hoedemaeker M (2009) Driving with a congestion assistant; mental workload and acceptance. Appl Ergon 40(6):1019–1025. https://doi.org/10.1016/j.apergo.2008.06.010
Brown SD, Heathcote A (2008) The simplest complete model of choice response time: linear ballistic accumulation. Cogn Psychol 57(3):153–178. https://doi.org/10.1016/j.cogpsych.2007.12.002
Brünken R, Plass JL, Leutner D (2003) Direct measurement of cognitive load in multimedia learning. Educ Psychol 38(1):53–61. https://doi.org/10.1207/S15326985EP3801_7
Castro SC, Strayer DL, Matzke D, Heathcote A (2019) Cognitive workload measurement and modeling under divided attention. J Exp Psychol Hum Percept Perform 45(6):826–839. https://doi.org/10.1037/xhp0000638
Charles RL, Nixon J (2019) Measuring mental workload using physiological measures: a systematic review. Appl Ergon 74(May 2018):221–232. https://doi.org/10.1016/j.apergo.2018.08.028
Chen SI, Visser TAW, Huf S, Loft S (2017) Optimizing the balance between task automation and human manual control in simulated submarine track management. J Exp Psychol Appl 23(3):240–262. https://doi.org/10.1037/xap0000126
Colcombe S, Kramer AF (2003) Fitness effects on the cognitive function of older adults: a meta-analytic study. Perspect Psychol Sci 13(2):213–217. https://doi.org/10.1177/1745691617707316
De Visser E, Parasuraman R (2011) Adaptive aiding of human-robot teaming: effects of imperfect automation on performance, trust, and workload. J Cognit Eng Decis Mak 5(2):209–231. https://doi.org/10.1177/1555343411410160
Endsley MR, Kaber DB (1999) Level of automation effects on performance, situation awareness and workload in a dynamic control task. Ergonomics 42(3):462–492. https://doi.org/10.1080/001401399185595
Endsley MR, Kiris EO (1995) The out-of-the-loop performance problem and level of control in automation. Hum Fact 37(2):381–394. https://doi.org/10.1518/001872095779064555
Evans DC, Fendley M (2017) A multi-measure approach for connecting cognitive workload and automation. Int J Hum Comput Stud 97:182–189. https://doi.org/10.1016/j.ijhcs.2016.05.008
Fasola J, Mataric M (2013) A socially assistive robot exercise coach for the elderly. J Hum Robot Interact. https://doi.org/10.5898/JHRI.2.2.Fasola
Galy E, Cariou M, Mélan C (2012) What is the relationship between mental workload factors and cognitive load types? Int J Psychophysiol 83(3):269–275. https://doi.org/10.1016/j.ijpsycho.2011.09.023
Ghani U, Signal N, Niazi IK, Taylor D (2020) ERP based measures of cognitive workload: a review. Neurosci Biobehav Rev 118:18–26. https://doi.org/10.1016/j.neubiorev.2020.07.020
Hameed S, Sarter N (2009) Context-sensitive information presentation: Integrating adaptive and adaptable approaches to display design. In: Proceedings of the 53rd annual meeting of the human factors and ergonomics society. Human Factors and Ergonomics Society, Santa Monica, pp 1694–1698. https://doi.org/10.1518/107118109x12524444081476
Hancock PA (2007a) On the process of automation transition in multitask human–machine systems. IEEE Trans Syst Man Cybern Part A Syst Hum 37(4):586–598. https://doi.org/10.1109/TSMCA.2007.897610
Hancock PA (2007b) Procedure and dynamic display relocation on performance in a multitask environment. IEEE Trans Syst Man Cybern Part A Syst Hum 37(1):47–57. https://doi.org/10.1109/TSMCA.2006.886341
Hancock PA (2017) Whither workload? Mapping a path for its future development. Commun Comput Inf Sci 726:3–17. https://doi.org/10.1007/978-3-319-61061-0_1
Hancock PA, Matthews G (2019) Workload and performance: associations, insensitivities, and dissociations. Hum Factors 61(3):374–392. https://doi.org/10.1177/0018720818809590
Hancock PA, Scallen SF (1997) The performance and workload effects of task re-location during automation. Displays 17(2):61–68. https://doi.org/10.1016/S0141-9382(96)01018-9
Hart SG, Staveland LE (1988) Development of NASA-TLX (Task Load Index): results of empirical and theoretical research. In: Hancock PA, Meshkati N (eds) Human mental workload. North Holland Press, Amsterdam
Hughes AM, Hancock GM, Marlow SL, Stowers K, Salas E (2019) Cardiac measures of cognitive workload: a meta-analysis. Hum Factors 61(3):393–414. https://doi.org/10.1177/0018720819830553
Jiang X, Gramopadhye AK, Melloy BJ, Grimes LW (2003) Evaluation of best system performance: human, automated, and hybrid inspection systems. Hum Factors Ergon Manuf 13(2):137–152. https://doi.org/10.1002/hfm.10031
Jou Y-T, Yenn T-C, Lin CJ, Yang C-W, Chiang C-C (2009) Evaluation of operators’ mental workload of human–system interface automation in the advanced nuclear power plants. Nucl Eng Design 239(11):2537–2542. https://doi.org/10.1016/j.nucengdes.2009.06.023
Jovanović M, De Angeli A, McNeill A, Coventry L (2021) User requirements for inclusive technology for older adults. Int J Hum Comput Interact 37(20):1947–1965. https://doi.org/10.1080/10447318.2021.1921365
Kaber DB, Endsley MR (2004) The effects of level of automation and adaptive automation on human performance, situation awareness and workload in a dynamic control task. Theor Issues Ergon Sci 5(2):113–153. https://doi.org/10.1080/1463922021000054335
Kaber DB, Onal E, Endsley MR (2000) Design of automation for telerobots and the effect on performance, operator situation awareness, and subjective workload. Hum Factors Ergon Manuf 10(4):409–430. https://doi.org/10.1002/1520-6564(200023)10:4%3c409::AID-HFM4%3e3.3.CO
Kaber DB, Wright MC, Sheik-Nainar MA (2006) Investigation of multi-modal interface features for adaptive automation of a human-robot system. Int J Hum Comput Stud 64(6):527–540. https://doi.org/10.1016/j.ijhcs.2005.11.003
Kaber D, Perry C, Segall N, Sheik-Nainar M (2007) Workload state classification with automation during simulated air traffic control. Int J Aviat Psychol 17(4):371–390. https://doi.org/10.1080/10508410701527860
Li H, Wickens CD, Sarter N, Sebok A (2014) Stages and levels of automation in support of space teleoperations. Hum Factors 56(6):1050–1061. https://doi.org/10.1177/0018720814522830
Lohani M, Payne BR, Strayer DL (2019) A review of psychophysiological measures to assess cognitive states in real-world driving. Front Hum Neurosci 13:57. https://doi.org/10.3389/fnhum.2019.00057
Lorenz B, Di Nocera F, Röttger S, Parasuraman R (2002) Automated fault-management in a simulated spaceflight micro-world. Aviat Space Environ Med 73(9):886–897
Manzey D, Reichenbach J, Onnasch L (2012) Human performance consequences of automated decision aids: the impact of degree of automation and system experience. J Cognit Eng Decis Mak 6(1):57–87. https://doi.org/10.1177/1555343411433844
Marmeleira J, Ferreira I, Melo F, Godinho M (2012) Associations of physical activity with driving-related cognitive abilities in older drivers: an exploratory study. Percept Motor Skills 115(2):521–533. https://doi.org/10.2466/10.06.25.PMS.115.5.521-533
Masalonis AJ, Duley JA, Parasuraman R (1999) Effects of manual and autopilot control on mental workload and vigilance during simulated general aviation flight. Transp Hum Factors 1(2):187–200
Matthews G, Reinerman-Jones LE, Barber DJ, Abich J (2015) The psychometrics of mental workload: multiple measures are sensitive but divergent. Hum Factors 57(1):125–143. https://doi.org/10.1177/0018720814539505
Matthews G, De Winter J, Hancock PA (2020) What do subjective workload scales really measure? Operational and representational solutions to divergence of workload measures. Theor Issues Ergon Sci 21(4):369–396. https://doi.org/10.1080/1463922X.2018.1547459
McDonald AD, Alambeigi H, Engström J, Markkula G, Vogelpohl T, Dunne J, Yuma N (2019) Toward computational simulations of behavior during automated driving takeovers: a review of the empirical and modeling literatures. Hum Factors 61(4):642–688. https://doi.org/10.1177/0018720819829572
Metzger U, Parasuraman R (2005) Automation in future air traffic management: effects of decision aid reliability on controller performance and mental workload. Hum Factors 47(1):35–49. https://doi.org/10.1518/0018720053653802
Mitchell GW (1966) Effects of automation on the structure and functioning of banking. Am Econ Assoc 56(1/2):159–166
Moher D, Liberati A, Tetzlaff J, Altman DG, Altman D, Antes G, Tugwell P (2009) Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. PLoS Med. https://doi.org/10.1371/journal.pmed.1000097
Navarro J, Heuveline L, Avril E, Cegarra J (2018) Influence of human-machine interactions and task demand on automation selection and use. Ergonomics 61(12):1601–1612. https://doi.org/10.1080/00140139.2018.1501517
Newlin Lew K, Nowlin S, Chyun D, Melkus GD (2014) State of the science: diabetes self-management interventions. West J Nurs Res 36(9):1111–1157. https://doi.org/10.1177/0193945914532033
Nisser T, Westin C (2006) Human factors challenges in unmanned aerial vehicles: a literature review. School of Aviation of the Lund University, Ljungbyhed
Onnasch L, Wickens CD, Li H, Manzey D (2014) Human performance consequences of stages and levels of automation. Hum Factors 56(3):476–488. https://doi.org/10.1177/0018720813501549
Paas FG, Van Merrienboer JJG (1993) The efficiency of instructional conditions: an approach to combine mental effort and performance measures. Hum Factors 35(4):737–743. https://doi.org/10.1177/001872089303500412
Paas FG, Van Merriënboer JJ, Adam JJ (1994) Measurement of cognitive load in instructional research. Percept Mot Skills 79(1):419–430. https://doi.org/10.2466/pms.1994.79.1.419
Parasuraman R, Sheridan TB, Wickens CD (2000) A model for types and levels of human interaction with automation. IEEE Trans Syst Man Cybern Part A Syst Hum 30(3):286–297. https://doi.org/10.1109/3468.844354
Parasuraman R, Cosenzo KA, De Visser E (2009) Adaptive automation for human supervision of multiple uninhabited vehicles: effects on change detection, situation awareness, and mental workload. Mil Psychol 21(2):270–297. https://doi.org/10.1080/08995600902768800
Prinzel LJ III, Freeman FG, Scerbo MW, Mikulka PJ, Pope AT (2003) Effects of a psychophysiological system for adaptive automation on performance, workload, and the event-related potential P300 component. Hum Factors 45(4):601–614. https://doi.org/10.1518/hfes.45.4.601.27092
Reid GB, Nygren TE (1988) The subjective workload assessment technique: a scaling procedure for measuring mental workload. Advances in psychology, vol 52. North Holland Press, Amsterdam, pp 185–218
Rouse WB (1981) Human–computer interaction in the control of dynamic systems. Comput Surv 13(1):71–99. https://doi.org/10.1145/356835.356839
Rovira E, McGarry K, Parasuraman R (2007) Effects of imperfect automation on decision making in a simulated command and control task. Hum Factors 49(1):76–87. https://doi.org/10.1518/001872007779598082
Sauer J, Nickel P, Wastell D (2013) Designing automation for complex work environments under different levels of stress. Appl Ergon 44(1):119–127. https://doi.org/10.1016/j.apergo.2012.05.008
Scerbo MW (2007) Adaptive automation. In: Parasuraman R, Rizzo M (eds) Neuroergonomics: the brain at work. Oxford University Press, New York, pp 239–252
Sehrawat, Choudhury TA, Raj G (2017) Surveillance drone for disaster management and military security. In: 2017 international conference on computing, communication and automation (ICCCA), pp 470–475. https://doi.org/10.1109/CCAA.2017.8229846
Sheridan TB, Parasuraman R (2006) Human-automation interaction. In: Nickerson RS (ed) Reviews of human factors and ergonomics, vol 1. Human Factors and Ergonomics Society, Santa Monica, pp 89–129
Sheridan TB, Verplank WL (1978) Human and computer control of undersea teleoperators (Technical Report). MIT Man-Machine Systems Laboratory, Cambridge
Squire PN, Parasuraman R (2010) Effects of automation and task load on task switching during human supervision of multiple semi-autonomous robots in a dynamic environment. Ergonomics 53(8):951–961. https://doi.org/10.1080/00140139.2010.489969
Stapel J, Mullakkal-Babu FA, Happee R (2019) Automated driving reduces perceived workload, but monitoring causes higher cognitive load than manual driving. Transp Res Part F 60:590–605. https://doi.org/10.1016/j.trf.2018.11.006
Stojmenova K, Sodnik J (2018) Detection-response task-uses and limitations. Sensors 18(2):594. https://doi.org/10.3390/s18020594
Van Acker BB, Parmentier DD, Vlerick P, Saldien J (2018) Understanding mental workload: from a clarifying concept analysis toward an implementable framework. Cogn Technol Work 20(3):351–365. https://doi.org/10.1007/s10111-018-0481-3
Vanderhaegen F, Carsten O (2017) Can dissonance engineering improve risk analysis of human–machine systems? Cognit Technol Work 19(1):1–12. https://doi.org/10.1007/s10111-017-0405-7
Vanderhaegen F, Crevits I, Debernard S, Millot P (1994) Human–machine cooperation: toward an activity regulation assistance for different air traffic control levels. Int J Hum Comput Interact 6(1):65–104. https://doi.org/10.1080/10447319409526084
Vu K-PL, Strybel TZ, Battiste V, Lachter J, Dao A-QV, Brandt S, Ligda S, Johnson W (2012) Pilot performance in trajectory-based operations under concepts of operation that vary separation responsibility across pilots, air traffic controllers, and automation. Int J Hum Comput Interact 28(2):107–118. https://doi.org/10.1080/10447318.2012.634761
Weinstein R (2005) RFID: a technical overview and its application to the enterprise. IT Prof 7(3):27–33. https://doi.org/10.1109/MITP.2005.69
Wilson GF, Russell CA (2007) Performance enhancement in an uninhabited air vehicle task using psychophysiologically determined adaptive aiding. Hum Factors 49(6):1005–1018. https://doi.org/10.1518/001872007X249875
Woods DD (1996) Decomposing automation: apparent simplicity, real complexity. In: Parasuraman R, Mouloua M (eds) Automation and human performance: theory and applications. Erlbaum, Mahwah, pp 3–17
Zhang W, Feltner D, Shirley J, Swangnetr M, Kaber D (2016) Unmanned aerial vehicle control interface design and cognitive workload: a constrained review and research framework. In: 2016 IEEE international conference on systems, man, and cybernetics (SMC), pp 001821–001826. https://doi.org/10.1109/SMC.2016.7844502
Zijlstra FRH (1993) Efficiency in work behaviour: a design approach for modern tools. Delft University Press, Delft
Funding
None.
Author information
Authors and Affiliations
Contributions
SF was responsible for acquiring, analyzing, and interpreting the data for the manuscript, as well as drafting the document. BJP was responsible for developing content, critically revising the manuscript, and final approval for publication. SM was responsible for was responsible for developing content, critically revising the manuscript, and final approval for publication.
Corresponding author
Ethics declarations
Conflict of interest
The authors declare no competing interests.
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Supplementary Information
Below is the link to the electronic supplementary material.
Rights and permissions
About this article
Cite this article
Frazier, S., Pitts, B.J. & McComb, S. Measuring cognitive workload in automated knowledge work environments: a systematic literature review. Cogn Tech Work 24, 557–587 (2022). https://doi.org/10.1007/s10111-022-00708-0
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10111-022-00708-0