Abstract
Situation awareness and workload are popular constructs in human factors science. It has been hotly debated whether these constructs are scientifically credible, or whether they should merely be seen as folk models. Reflecting on the works of psychophysicist Stanley Smith Stevens and of measurement theorist David Hand, we suggest a resolution to this debate, namely that human factors constructs are situated towards the operational end of a representational–operational continuum. From an operational perspective, human factors constructs do not reflect an empirical reality, but they aim to predict. For operationalism to be successful, however, it is important to have suitable measurement procedures available. To explore how human factors constructs are measured, we focused on (mental) workload and its measurement by questionnaires and applied a culturomic analysis to investigate secular trends in word use. The results reveal an explosive use of the NASA Task Load Index (TLX). Other questionnaires, such as the Cooper Harper rating scale and the Subjective Workload Assessment Technique, show a modest increase, whereas many others appear short lived. We found no indication that the TLX is improved by iterative self-correction towards optimal validity, and we argue that usage of the NASA-TLX has become dominant through a Matthew effect. Recommendations for improving the quality of human factors research are provided.
Similar content being viewed by others
References
Adams S (1998) Practical considerations for measuring Situational Awareness. In: Proceedings for the third annual symposium and exhibition on situational awareness in the tactical air environment, Piney Point, MD, pp 157–164
Boring EG, Stevens SS (1936) The nature of tonal brightness. Proc Natl Acad Sci 22:514–521
Bridgman PW (1927) The logic of modern physics. MacMillan, New York
Byers JC, Bittner AC, Hill SG (1989) Traditional and raw task load index (TLX) correlations: are paired comparisons necessary? In: Mital A (ed) Advances in industrial ergonomics and safety 1. Taylor and Francis, London, pp 481–485
Campbell NR (1920) Physics: the elements. Cambridge University Press, Cambridge
Campbell NR, Jeffreys H (1938) Symposium: measurement and its importance for philosophy. Proc Aristot Soc Suppl Vol 17:121–151
Dawes RM, Smith TL (1985) Attitude and opinion measurement. In: Lindzey G, Aronson E (eds) The handbook of social psychology. Random House, New York, pp 509–566
Dekker SWA (2013) On the epistemology and ethics of communicating a Cartesian consciousness. Saf Sci 56:96–99
Dekker S, Hollnagel E (2004) Human factors and folk models. Cogn Technol Work 6:79–86
Dekker SW, Woods DD (2002) Maba-maba or abracadabra? progress on human-automation co-ordination. Cogn Technol Work 4:240–244
Dekker S, Nyce JM, Van Winsen R, Henriqson E (2010) Epistemological self-confidence in human factors research. J Cogn Eng Decis Mak 4:27–38
Dickinson J, Byblow WD, Ryan LA (1993) Order effects and the weighting process in workload assessment. Appl Ergon 24:357–361
Ferguson A, Myers CS, Bartlett RJ, Banister H, Bartlett FC, Brown W et al (1938) Quantitative estimates of sensory events: interim report. Br Assoc Adv Sci 108:277–334
Ferguson A, Myers CS, Bartlett RJ, Banister H, Bartlett FC, Brown W et al (1940) Quantitative estimates of sensory events. Adv Sci 1:331–349
Flach JM (1995) Situation awareness: proceed with caution. Hum Factors 37:149–157
Hancock PA (1996) Effects of control order, augmented feedback, input device and practice on tracking performance and perceived workload. Ergonomics 39:1146–1162
Hancock PA, Szalma JL (2004) On the relevance of qualitative methods for ergonomics. Theor Issues Ergon Sci 5:499–506
Hand DJ (1996) Statistics and the theory of measurement. J R Stat Soc A 159:445–492
Hand DJ (2004) Measurement: theory and practice. Arnold, London
Hand DJ (2005) Size matters—how measurement defines our world. Significance 2:81–83
Hand DJ, Keynes M (1993) Letter to the Editor: Velleman PF, Wilkinson L (1993) Nominal, ordinal, interval, and ratio typologies are misleading. Am Stat 47, 65–72: Comments by Huberty and Hand; rejoinder by Velleman and Wilkinson. Am Stat 47:314–315
Hart SG, Staveland LE (1988) Development of NASA-TLX (Task Load Index): Results of empirical and theoretical research. In: Hancock PA, Meshkati N (eds) Human mental workload, North Holland Press, Amsterdam, pp 139–183. http://humanfactors.arc.nasa.gov/groups/TLX/downloads/NASA-TLXChapter.pdf
Hendy KC, Hamilton KM, Landry LM (1993) Measuring subjective workload: when is one scale better than many? Hum Factors 35:579–601
Hill SG, Iavecchia HP, Byers JC, Bittner AC Jr, Zaklade AL, Christ RE (1992) Comparison of four subjective workload rating scales. Hum Factors 34:429–439
Ioannidis JP (2012) Why science is not necessarily self-correcting. Perspect Psychol Sci 7:645–654
Kantowitz BH (1992) Selecting measures for human factors research. Hum Factors 34:387–398
Khurshid A, Sahai H (1993) Scales of measurements: an introduction and a selected bibliography. Qual Quant 27:303–324
Liu YL, Wickens CD (1994) Mental workload and cognitive task automaticity: an evaluation of subjective and time-estimation metrics. Ergonomics 37:1843–1854
Lord FM (1953) On the statistical treatment of football numbers. Am Psychol 8:750–751
Luce RD, Suppes P (2002) Representational measurement theory. In: Pashler H, Wixted J (eds) Stevens’ handbook of experimental psychology, vol 4. Methodology in experimental psychology, 3rd edn. Wiley, New York, pp 1–41
Marcus-Roberts HM, Roberts FS (1987) Meaningless statistics. J Educ Behav Stat 12:383–394
Marks LE (2006) S.S. Stevens: a brief scientific biography. In: Fechner Day 2006: Proceedings of the twenty-second annual meeting of the international society for psychophysics, St. Albans, England
Mather M, Cacioppo JT, Kanwisher N (2013) Introduction to the Special Section: 20 Years of fMRI—what has it done for understanding cognition? Persp Psychol Sci 8:41–43
Merton RK (1968) The Matthew effect in science. Science 159:56–63
Michel J-B, Shen YK, Aiden AP et al (2010) Quantitative analysis of culture using millions of digitized books. Science 331:176–182
Miller GA (1974) Stanley Smith Stevens: 1906–1973. Am J Psychol 87:279–288
Moroney WF, Biers DW, Eggemeier FT, Mitchell JA (1992) A comparison of two scoring procedures with the NASA Task Load Index in a simulated flight task. In: Proceedings of the IEEE national aerospace and electronics conference 2, Dayton, OH, pp 734–740
NASA (1986) Task Load Index (NASA-TLX). v. 1.0. Paper and pencil package (instruction manual). NASA Ames Research Center, Moffett Field, CA. http://humanfactors.arc.nasa.gov/groups/TLX/downloads/TLX.pdf
NASA (2014) Task Load Index paper and pencil version. NASA Ames Research Center, Moffett Field, CA. http://humansystems.arc.nasa.gov/groups/TLX/downloads/TLXScale.pdf
Nygren TE (1991) Psychometric properties of subjective workload measurement techniques: implications for their use in the assessment of perceived mental workload. Hum Factors 33:17–33
Parasuraman R, Sheridan TB, Wickens CD (2008) Situation awareness, mental workload, and trust in automation: viable, empirically supported cognitive engineering constructs. J Cogn Eng Decis Mak 2:140–160
Reid GB, Shingledecker CA, Eggemeier FT (1981) Application of conjoint measurement to workload scale development. Proc Hum Factors Ergon Soc Annu Meet 25:522–526
Rubio S, Díaz E, Martín J, Puente JM (2004) Evaluation of subjective mental workload: a comparison of SWAT, NASA-TLX, and Workload Profile methods. Appl Psychol 53:61–86
Siegel S (1956) Nonparametric statistics for the behavioral sciences. McGraw-Hill, New York
Stevens SS (1946) On the theory of scales of measurement. Science 103:677–688
Stevens SS (1959) Measurement, psychophysics and utility. In: Churchman CW, Ratoosh P (eds) Measurement: definitions and theories. Wiley, New York, pp 18–63
Stevens SS (1961) To honor Fechner and repeal his law. Science 133:80–86
Teghtsoonian R (2001) S. S. Stevens. In: Smelser NJ, Baltes PB (eds) International encyclopedia of the social and behavioral sciences 22:15104–15108. Pergamon, Oxford
Townsend JT, Ashby FG (1984) Measurement scales and statistics: the misconception misconceived. Psychol Bull 96:394–401
Underwood BJ (1957) Psychological research. Prentice-Hall, Englewood Cliffs, NJ
Uttal WR (2008) Time, space, and number in physics and psychology. Sloan Publishing. http://j.b5z.net/i/u/2084689/f/Online_Time_Space_Number.pdf
Velleman PF, Wilkinson L (1993) Nominal, ordinal, interval, and ratio typologies are misleading. Am Stat 47:65–72
Acknowledgments
I thank Dr. Dimitra Dodou for useful comments and for helping to create Table 3.
Author information
Authors and Affiliations
Corresponding author
Additional information
This original paper is discussed in the commentaries available at: doi:10.1007/s10111-014-0276-0; doi:10.1007/s10111-014-0277-z; doi:10.1007/s10111-014-0278-y.
Rights and permissions
About this article
Cite this article
de Winter, J.C.F. Controversy in human factors constructs and the explosive use of the NASA-TLX: a measurement perspective. Cogn Tech Work 16, 289–297 (2014). https://doi.org/10.1007/s10111-014-0275-1
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10111-014-0275-1