Skip to main content
SpringerLink
Log in

Assessment of Mental Workload: A Comparison of Machine Learning Methods and Subjective Assessment Techniques

  • Conference paper
  • First Online:
Human Mental Workload: Models and Applications (H-WORKLOAD 2017)

Part of the book series: Communications in Computer and Information Science ((CCIS,volume 726))

Abstract

Mental workload (MWL) measurement is a complex multidisciplinary research field. In the last 50 years of research endeavour, MWL measurement has mainly produced theory-driven models. Some of the reasons for justifying this trend includes the omnipresent uncertainty about how to define the construct of MWL and the limited use of data-driven research methodologies. This work presents novel research focused on the investigation of the capability of a selection of supervised Machine Learning (ML) classification techniques to produce data-driven computational models of MWL for the prediction of objective performance. These are then compared to two state-of-the-art subjective techniques for the assessment of MWL, namely the NASA Task Load Index and the Workload Profile, through an analysis of their concurrent and convergent validity. Findings show that the data-driven models generally tend to outperform the two baseline selected techniques.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 39.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Arlot, S., Celisse, A., et al.: A survey of cross-validation procedures for model selection. Stat. Surv. 4, 40–79 (2010)

    Article  MathSciNet  MATH  Google Scholar 

  2. Ben-David, A.: About the relationship between ROC curves and Cohen’s kappa. Eng. Appl. Artif. Intell. 21(6), 874–882 (2008)

    Article  Google Scholar 

  3. Blankertz, B., Curio, G., Müller, K.R.: Classifying single trial EEG: towards brain computer interfacing. Adv. Neural Inf. Process. Syst. 1(c), 157–164 (2002)

    Google Scholar 

  4. Bunkhumpornpat, C., Sinapiromsaran, K., Lursinsap, C.: DBSMOTE: density-based synthetic minority over-sampling technique. Appl. Intell. 36(3), 664–684 (2012)

    Article  Google Scholar 

  5. Cain, B.: A review of the mental workload literature. In: Defence Research and Development Toronto (Canada), pp. 4-1–4-34 (2007)

    Google Scholar 

  6. Cawley, G.C., Talbot, N.L.: On over-fitting in model selection and subsequent selection bias in performance evaluation. J. Mach. Learn. Res. 11, 2079–2107 (2010)

    MathSciNet  MATH  Google Scholar 

  7. Chapman, P., Clinton, J., Khabaza, T., Reinartz, T., Wirth, R.: The CRISP-DM process model. CRIP-DM Consortium 310 (1999)

    Google Scholar 

  8. Choudhury, S., Bhowal, A.: Comparative analysis of machine learning algorithms along with classifiers for network intrusion detection. In: 2015 International Conference on Smart Technologies and Management for Computing, Communication, Controls, Energy and Materials (ICSTM) (May), pp. 89–95 (2015)

    Google Scholar 

  9. Cinaz, B., Arnrich, B., La Marca, R., Tröster, G.: Monitoring of mental workload levels during an everyday life office-work scenario. Pers. Ubiquit. Comput. 17(2), 229–239 (2013)

    Article  Google Scholar 

  10. Cortes Torres, C.C., Sampei, K., Sato, M., Raskar, R., Miki, N.: Workload assessment with eye movement monitoring aided by non-invasive and unobtrusive micro-fabricated optical sensors. In: Adjunct Proceedings of the 28th Annual ACM Symposium on User Interface Software and Technology, pp. 53–54 (2015)

    Google Scholar 

  11. Di Stasi, L.L., Marchitto, M., Antolí, A., Baccino, T., Cañas, J.J.: Approximation of on-line mental workload index in ATC simulated multitasks. J. Air Transp. Manage. 16(6), 330–333 (2010)

    Article  Google Scholar 

  12. Dornhege, G., Blankertz, B., Curio, G., Múller, K.R.: Boosting bit rates in noninvasive EEG single-trial classifications by feature combination and multiclass paradigms. IEEE Trans. Biomed. Eng. 51(6), 993–1002 (2004)

    Article  Google Scholar 

  13. Elkin-Frankston, S., Bracken, B.K., Irvin, S., Jenkins, M.: Are behavioral measures useful for detecting cognitive workload during human-computer interaction? In: Ahram, T., Karwowski, W. (eds.) Advances in Intelligent Systems and Computing, vol. 494, pp. 127–137. Springer, Cham (2017)

    Google Scholar 

  14. Fatourechi, M., Ward, R.K., Mason, S.G., Huggins, J., Schlógl, A., Birch, G.E.: Comparison of evaluation metrics in classification applications with imbalanced datasets. In: Proceedings - 7th International Conference on Machine Learning and Applications, ICMLA 2008, pp. 777–782 (2008)

    Google Scholar 

  15. Hart, S.G., Staveland, L.E.: Development of NASA-TLX (Task Load Index): results of empirical and theoretical research. Adv. Psychol. 52(C), 139–183 (1988)

    Article  Google Scholar 

  16. Hincks, S.W., Afergan, D., Jacob, R.J.K.: Using fNIRS for real-time cognitive workload assessment. In: Schmorrow, D.D.D., Fidopiastis, C.M.M. (eds.) AC 2016. LNCS, vol. 9743, pp. 198–208. Springer, Cham (2016). doi:10.1007/978-3-319-39955-3_19

    Google Scholar 

  17. Juszczak, P., Tax, D., Duin, R.P.: Feature scaling in support vector data description. In: Proceedings of the ASCI, pp. 95–102. Citeseer (2002)

    Google Scholar 

  18. Kohavi, R., et al.: A study of cross-validation and bootstrap for accuracy estimation and model selection. In: Ijcai, vol. 14, pp. 1137–1145 (1995)

    Google Scholar 

  19. Kumar, M., Arndt, A., Kreuzfeld, S., Thurow, K., Stoll, N., Stoll, R.: Fuzzy techniques for subjective workload-score modeling under uncertainties. IEEE Trans. Syst. Man Cybern. Part B Cybern. 38(6), 1449–1464 (2008)

    Article  Google Scholar 

  20. Lee, J.C., Tan, D.S.: Using a low-cost electroencephalograph for task classification in HCI research. In: Proceedings of the 19th ACM Symposium on User Interface Software and Technology, pp. 81–90 (2006)

    Google Scholar 

  21. Leva, M.C., Kay, A.M., Mattei, F., Kontogiannis, T., Ambroggi, M., Cromie, S.: A dynamic task representation method for a virtual reality application. In: Harris, D. (ed.) EPCE 2009. LNCS, vol. 5639, pp. 32–42. Springer, Heidelberg (2009). doi:10.1007/978-3-642-02728-4_4

    Chapter  Google Scholar 

  22. Longo, L.: Human-computer interaction and human mental workload: assessing cognitive engagement in the world wide web. In: Campos, P., Graham, N., Jorge, J., Nunes, N., Palanque, P., Winckler, M. (eds.) INTERACT 2011. LNCS, vol. 6949, pp. 402–405. Springer, Heidelberg (2011). doi:10.1007/978-3-642-23768-3_43

    Chapter  Google Scholar 

  23. Longo, L.: Formalising human mental workload as non-monotonic concept for adaptive and personalised web-design. In: Masthoff, J., Mobasher, B., Desmarais, M.C., Nkambou, R. (eds.) UMAP 2012. LNCS, vol. 7379, pp. 369–373. Springer, Heidelberg (2012). doi:10.1007/978-3-642-31454-4_38

    Chapter  Google Scholar 

  24. Longo, L.: Formalising human mental workload as a defeasible computational concept. Ph.D. thesis, Trinity College Dublin (2014)

    Google Scholar 

  25. Longo, L.: A defeasible reasoning framework for human mental workload representation and assessment. Behav. Inf. Technol. 34(8), 758–786 (2015)

    Article  Google Scholar 

  26. Longo, L.: Designing medical interactive systems via assessment of human mental workload. In: International Symposium on Computer-Based Medical Systems, pp. 364–365 (2015)

    Google Scholar 

  27. Longo, L.: Subjective usability (system usability scale) and subjective mental workload (NASA-TLX and workload profile) of web-based tasks and interfaces (2015)

    Google Scholar 

  28. Longo, L.: Mental workload in medicine: Foundations, applications, open problems, challenges and future perspectives. In: 2016 IEEE 29th International Symposium on Computer-Based Medical Systems (CBMS), pp. 106–111, June 2016

    Google Scholar 

  29. Longo, L., Barrett, S.: A computational analysis of cognitive effort. In: Nguyen, N.T., Le, M.T., Świątek, J. (eds.) ACIIDS 2010. LNCS, vol. 5991, pp. 65–74. Springer, Heidelberg (2010). doi:10.1007/978-3-642-12101-2_8

    Chapter  Google Scholar 

  30. Longo, L., Barrett, S.: Cognitive effort for multi-agent systems. In: Yao, Y., Sun, R., Poggio, T., Liu, J., Zhong, N., Huang, J. (eds.) BI 2010. LNCS, vol. 6334, pp. 55–66. Springer, Heidelberg (2010). doi:10.1007/978-3-642-15314-3_6

    Chapter  Google Scholar 

  31. Longo, L., Dondio, P.: On the relationship between perception of usability and subjective mental workload of web interfaces. In: IEEE/WIC/ACM International Conference on Web Intelligence and Intelligent Agent Technology, WI-IAT 2015, Singapore, December 6–9, vol. I, pp. 345–352 (2015)

    Google Scholar 

  32. Longo, L., Rusconi, F., Noce, L., Barrett, S.: The importance of human mental workload in web-design. In: 8th International Conference on Web Information Systems and Technologies, pp. 403–409, April 2012

    Google Scholar 

  33. Mannaru, P., Balasingam, B., Pattipati, K., Sibley, C., Coyne, J.: Cognitive context detection in UAS operators using eye-gaze patterns on computer screens. In: SPIE 9851, Next-Generation Analyst IV, vol. 9851, p. 98510F (2016)

    Google Scholar 

  34. Ott, T., Wu, P., Paullada, A., Mayer, D., Gottlieb, J., Wall, P.: ATHENA – a zero-intrusion no contact method for workload detection using linguistics, keyboard dynamics, and computer vision. In: Stephanidis, C. (ed.) HCI 2016. CCIS, vol. 617, pp. 226–231. Springer, Cham (2016). doi:10.1007/978-3-319-40548-3_38

    Chapter  Google Scholar 

  35. O’Donnell, R., Eggemeier, F.: Workload assessment methodology. In: Boff, K.R., Kaufman, L., Thomas, J.P. (eds.) Handbook of Perception and Human Performance, Cognitive Processes and Performance, vol. 2. Wiley, Hoboken (1986)

    Google Scholar 

  36. Pham, T.T., Nguyen, T.D., Vo, T.: Sparse fNIRS feature estimation via unsupervised learning for mental workload classification. In: Bassis, S., Esposito, A., Morabito, F.C., Pasero, E. (eds.) Advances in Neural Networks. SIST, vol. 54, pp. 283–292. Springer, Cham (2016). doi:10.1007/978-3-319-33747-0_28

    Chapter  Google Scholar 

  37. Rao, R.B., Fung, G., Rosales, R.: On the dangers of cross-validation. an experimental evaluation. In: Proceedings of the 2008 SIAM International Conference on Data Mining, pp. 588–596. SIAM (2008)

    Google Scholar 

  38. Reid, G.B., Nygren, T.E.: The Subjective Workload Assessment Technique: A Scaling Procedure for Measuring Mental Workload, vol. 52, North-Holland (1988)

    Google Scholar 

  39. Rizzo, L., Dondio, P., Delany, S.J., Longo, L.: Modeling mental workload via rule-based expert system: a comparison with NASA-TLX and workload profile. In: Iliadis, L., Maglogiannis, I. (eds.) AIAI 2016. IAICT, vol. 475, pp. 215–229. Springer, Cham (2016). doi:10.1007/978-3-319-44944-9_19

    Chapter  Google Scholar 

  40. Rubio, S., Díaz, E., Martín, J., Puente, J.M.: Evaluation of subjective mental workload: a comparison of SWAT, NASA-TLX, and workload profile methods. Appl. Psychol. 53(1), 61–86 (2004)

    Article  Google Scholar 

  41. Solovey, E., Schermerhorn, P., Scheutz, M., Sassaroli, A., Fantini, S., Jacob, R.: Brainput: Enhancing interactive systems with streaming fNIRS Brain Input. In: Proceedings of the 2012 ACM Annual Conference on Human Factors in Computing Systems - CHI 2012. p. 2193. ACM (2012)

    Google Scholar 

  42. Stassen, H.G., Johannsen, G., Moray, N.: Internal representation, internal model, human performance model and mental workload. Automatica 26(4), 811–820 (1990)

    Article  Google Scholar 

  43. Stevens, R., Galloway, T., Berka, C.: Integrating EEG models of cognitive load with machine learning models of scientific problem solving. In: Proceedings of 2nd Annual Augmented Cognition International Conference (September, 2006)

    Google Scholar 

  44. Su, J., Luz, S.: Predicting cognitive load levels from speech data. Smart Innov. Syst. Technol. 48, 255–263 (2016)

    Article  Google Scholar 

  45. Thompson, S.K.: Stratified Sampling, pp. 139–156. Wiley, Hoboken (2012)

    Google Scholar 

  46. Trucco, P., Leva, M.C., Sträter, O.: Human error prediction in ATM via cognitive simulation: preliminary study. In: Proceedings of the 8th International Conference on Probabilistic Safety Assessment and Management (PSAM8), pp. 1–9 (2006)

    Google Scholar 

  47. Tsang, P.S., Velazquez, V.L.: Diagnosticity and multidimensional subjective workload ratings. Ergonomics 39(3), 358–381 (1996)

    Article  Google Scholar 

  48. Wickens, C.D.: Multiple resources and mental workload. Hum. Factors 50(3), 449–455 (2008)

    Article  Google Scholar 

  49. Wiebe, E.N., Roberts, E., Behrend, T.S.: An examination of two mental workload measurement approaches to understanding multimedia learning. Comput. Hum. Behav. 26(3), 474–481 (2010)

    Article  Google Scholar 

  50. Yoshida, Y., Ohwada, H., Mizoguchi, F., Iwasaki, H.: Classifying cognitive load and driving situation with machine learning. Int. J. Mach. Learn. Comput. 4(3), 210–215 (2014)

    Article  Google Scholar 

  51. Zhang, Y.Z.Y., Owechko, Y., Zhang, J.Z.J.: Driver cognitive workload estimation: a data-driven perspective. In: Proceedings of the 7th International IEEE Conference on Intelligent Transportation Systems (IEEE Cat. No.04TH8749), pp. 642–647 (2004)

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. School of Computing, Dublin Institute of Technology, Dublin, Ireland

    Karim Moustafa & Luca Longo

  2. Usher Institute of Population Health Sciences and Informatics, The University of Edinburgh, Edinburgh, Scotland

    Saturnino Luz

  3. The ADAPT Centre, Dublin, Ireland

    Luca Longo

Authors
  1. Karim Moustafa
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Saturnino Luz
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Luca Longo
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Luca Longo .

Editor information

Editors and Affiliations

  1. Dublin Institute of Technology, Dublin, Ireland

    Luca Longo

  2. Dublin Institute of Technology, Dublin, Ireland

    M. Chiara Leva

Rights and permissions

Reprints and permissions

Copyright information

© 2017 Springer International Publishing AG

About this paper

Cite this paper

Moustafa, K., Luz, S., Longo, L. (2017). Assessment of Mental Workload: A Comparison of Machine Learning Methods and Subjective Assessment Techniques. In: Longo, L., Leva, M. (eds) Human Mental Workload: Models and Applications. H-WORKLOAD 2017. Communications in Computer and Information Science, vol 726. Springer, Cham. https://doi.org/10.1007/978-3-319-61061-0_3

Download citation

  • DOI: https://doi.org/10.1007/978-3-319-61061-0_3

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-319-61060-3

  • Online ISBN: 978-3-319-61061-0

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation