Assessment of Mental Workload: A Comparison of Machine Learning Methods and Subjective Assessment Techniques

  • Karim Moustafa
  • Saturnino Luz
  • Luca LongoEmail author
Conference paper
Part of the Communications in Computer and Information Science book series (CCIS, volume 726)


Mental workload (MWL) measurement is a complex multidisciplinary research field. In the last 50 years of research endeavour, MWL measurement has mainly produced theory-driven models. Some of the reasons for justifying this trend includes the omnipresent uncertainty about how to define the construct of MWL and the limited use of data-driven research methodologies. This work presents novel research focused on the investigation of the capability of a selection of supervised Machine Learning (ML) classification techniques to produce data-driven computational models of MWL for the prediction of objective performance. These are then compared to two state-of-the-art subjective techniques for the assessment of MWL, namely the NASA Task Load Index and the Workload Profile, through an analysis of their concurrent and convergent validity. Findings show that the data-driven models generally tend to outperform the two baseline selected techniques.


Support Vector Machine Machine Learn Baseline Model Concurrent Validity Minority Class 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


  1. 1.
    Arlot, S., Celisse, A., et al.: A survey of cross-validation procedures for model selection. Stat. Surv. 4, 40–79 (2010)MathSciNetCrossRefzbMATHGoogle Scholar
  2. 2.
    Ben-David, A.: About the relationship between ROC curves and Cohen’s kappa. Eng. Appl. Artif. Intell. 21(6), 874–882 (2008)CrossRefGoogle Scholar
  3. 3.
    Blankertz, B., Curio, G., Müller, K.R.: Classifying single trial EEG: towards brain computer interfacing. Adv. Neural Inf. Process. Syst. 1(c), 157–164 (2002)Google Scholar
  4. 4.
    Bunkhumpornpat, C., Sinapiromsaran, K., Lursinsap, C.: DBSMOTE: density-based synthetic minority over-sampling technique. Appl. Intell. 36(3), 664–684 (2012)CrossRefGoogle Scholar
  5. 5.
    Cain, B.: A review of the mental workload literature. In: Defence Research and Development Toronto (Canada), pp. 4-1–4-34 (2007)Google Scholar
  6. 6.
    Cawley, G.C., Talbot, N.L.: On over-fitting in model selection and subsequent selection bias in performance evaluation. J. Mach. Learn. Res. 11, 2079–2107 (2010)MathSciNetzbMATHGoogle Scholar
  7. 7.
    Chapman, P., Clinton, J., Khabaza, T., Reinartz, T., Wirth, R.: The CRISP-DM process model. CRIP-DM Consortium 310 (1999)Google Scholar
  8. 8.
    Choudhury, S., Bhowal, A.: Comparative analysis of machine learning algorithms along with classifiers for network intrusion detection. In: 2015 International Conference on Smart Technologies and Management for Computing, Communication, Controls, Energy and Materials (ICSTM) (May), pp. 89–95 (2015)Google Scholar
  9. 9.
    Cinaz, B., Arnrich, B., La Marca, R., Tröster, G.: Monitoring of mental workload levels during an everyday life office-work scenario. Pers. Ubiquit. Comput. 17(2), 229–239 (2013)CrossRefGoogle Scholar
  10. 10.
    Cortes Torres, C.C., Sampei, K., Sato, M., Raskar, R., Miki, N.: Workload assessment with eye movement monitoring aided by non-invasive and unobtrusive micro-fabricated optical sensors. In: Adjunct Proceedings of the 28th Annual ACM Symposium on User Interface Software and Technology, pp. 53–54 (2015)Google Scholar
  11. 11.
    Di Stasi, L.L., Marchitto, M., Antolí, A., Baccino, T., Cañas, J.J.: Approximation of on-line mental workload index in ATC simulated multitasks. J. Air Transp. Manage. 16(6), 330–333 (2010)CrossRefGoogle Scholar
  12. 12.
    Dornhege, G., Blankertz, B., Curio, G., Múller, K.R.: Boosting bit rates in noninvasive EEG single-trial classifications by feature combination and multiclass paradigms. IEEE Trans. Biomed. Eng. 51(6), 993–1002 (2004)CrossRefGoogle Scholar
  13. 13.
    Elkin-Frankston, S., Bracken, B.K., Irvin, S., Jenkins, M.: Are behavioral measures useful for detecting cognitive workload during human-computer interaction? In: Ahram, T., Karwowski, W. (eds.) Advances in Intelligent Systems and Computing, vol. 494, pp. 127–137. Springer, Cham (2017)Google Scholar
  14. 14.
    Fatourechi, M., Ward, R.K., Mason, S.G., Huggins, J., Schlógl, A., Birch, G.E.: Comparison of evaluation metrics in classification applications with imbalanced datasets. In: Proceedings - 7th International Conference on Machine Learning and Applications, ICMLA 2008, pp. 777–782 (2008)Google Scholar
  15. 15.
    Hart, S.G., Staveland, L.E.: Development of NASA-TLX (Task Load Index): results of empirical and theoretical research. Adv. Psychol. 52(C), 139–183 (1988)CrossRefGoogle Scholar
  16. 16.
    Hincks, S.W., Afergan, D., Jacob, R.J.K.: Using fNIRS for real-time cognitive workload assessment. In: Schmorrow, D.D.D., Fidopiastis, C.M.M. (eds.) AC 2016. LNCS, vol. 9743, pp. 198–208. Springer, Cham (2016). doi: 10.1007/978-3-319-39955-3_19 Google Scholar
  17. 17.
    Juszczak, P., Tax, D., Duin, R.P.: Feature scaling in support vector data description. In: Proceedings of the ASCI, pp. 95–102. Citeseer (2002)Google Scholar
  18. 18.
    Kohavi, R., et al.: A study of cross-validation and bootstrap for accuracy estimation and model selection. In: Ijcai, vol. 14, pp. 1137–1145 (1995)Google Scholar
  19. 19.
    Kumar, M., Arndt, A., Kreuzfeld, S., Thurow, K., Stoll, N., Stoll, R.: Fuzzy techniques for subjective workload-score modeling under uncertainties. IEEE Trans. Syst. Man Cybern. Part B Cybern. 38(6), 1449–1464 (2008)CrossRefGoogle Scholar
  20. 20.
    Lee, J.C., Tan, D.S.: Using a low-cost electroencephalograph for task classification in HCI research. In: Proceedings of the 19th ACM Symposium on User Interface Software and Technology, pp. 81–90 (2006)Google Scholar
  21. 21.
    Leva, M.C., Kay, A.M., Mattei, F., Kontogiannis, T., Ambroggi, M., Cromie, S.: A dynamic task representation method for a virtual reality application. In: Harris, D. (ed.) EPCE 2009. LNCS, vol. 5639, pp. 32–42. Springer, Heidelberg (2009). doi: 10.1007/978-3-642-02728-4_4 CrossRefGoogle Scholar
  22. 22.
    Longo, L.: Human-computer interaction and human mental workload: assessing cognitive engagement in the world wide web. In: Campos, P., Graham, N., Jorge, J., Nunes, N., Palanque, P., Winckler, M. (eds.) INTERACT 2011. LNCS, vol. 6949, pp. 402–405. Springer, Heidelberg (2011). doi: 10.1007/978-3-642-23768-3_43 CrossRefGoogle Scholar
  23. 23.
    Longo, L.: Formalising human mental workload as non-monotonic concept for adaptive and personalised web-design. In: Masthoff, J., Mobasher, B., Desmarais, M.C., Nkambou, R. (eds.) UMAP 2012. LNCS, vol. 7379, pp. 369–373. Springer, Heidelberg (2012). doi: 10.1007/978-3-642-31454-4_38 CrossRefGoogle Scholar
  24. 24.
    Longo, L.: Formalising human mental workload as a defeasible computational concept. Ph.D. thesis, Trinity College Dublin (2014)Google Scholar
  25. 25.
    Longo, L.: A defeasible reasoning framework for human mental workload representation and assessment. Behav. Inf. Technol. 34(8), 758–786 (2015)CrossRefGoogle Scholar
  26. 26.
    Longo, L.: Designing medical interactive systems via assessment of human mental workload. In: International Symposium on Computer-Based Medical Systems, pp. 364–365 (2015)Google Scholar
  27. 27.
    Longo, L.: Subjective usability (system usability scale) and subjective mental workload (NASA-TLX and workload profile) of web-based tasks and interfaces (2015)Google Scholar
  28. 28.
    Longo, L.: Mental workload in medicine: Foundations, applications, open problems, challenges and future perspectives. In: 2016 IEEE 29th International Symposium on Computer-Based Medical Systems (CBMS), pp. 106–111, June 2016Google Scholar
  29. 29.
    Longo, L., Barrett, S.: A computational analysis of cognitive effort. In: Nguyen, N.T., Le, M.T., Świątek, J. (eds.) ACIIDS 2010. LNCS, vol. 5991, pp. 65–74. Springer, Heidelberg (2010). doi: 10.1007/978-3-642-12101-2_8 CrossRefGoogle Scholar
  30. 30.
    Longo, L., Barrett, S.: Cognitive effort for multi-agent systems. In: Yao, Y., Sun, R., Poggio, T., Liu, J., Zhong, N., Huang, J. (eds.) BI 2010. LNCS, vol. 6334, pp. 55–66. Springer, Heidelberg (2010). doi: 10.1007/978-3-642-15314-3_6 CrossRefGoogle Scholar
  31. 31.
    Longo, L., Dondio, P.: On the relationship between perception of usability and subjective mental workload of web interfaces. In: IEEE/WIC/ACM International Conference on Web Intelligence and Intelligent Agent Technology, WI-IAT 2015, Singapore, December 6–9, vol. I, pp. 345–352 (2015)Google Scholar
  32. 32.
    Longo, L., Rusconi, F., Noce, L., Barrett, S.: The importance of human mental workload in web-design. In: 8th International Conference on Web Information Systems and Technologies, pp. 403–409, April 2012Google Scholar
  33. 33.
    Mannaru, P., Balasingam, B., Pattipati, K., Sibley, C., Coyne, J.: Cognitive context detection in UAS operators using eye-gaze patterns on computer screens. In: SPIE 9851, Next-Generation Analyst IV, vol. 9851, p. 98510F (2016)Google Scholar
  34. 34.
    Ott, T., Wu, P., Paullada, A., Mayer, D., Gottlieb, J., Wall, P.: ATHENA – a zero-intrusion no contact method for workload detection using linguistics, keyboard dynamics, and computer vision. In: Stephanidis, C. (ed.) HCI 2016. CCIS, vol. 617, pp. 226–231. Springer, Cham (2016). doi: 10.1007/978-3-319-40548-3_38 CrossRefGoogle Scholar
  35. 35.
    O’Donnell, R., Eggemeier, F.: Workload assessment methodology. In: Boff, K.R., Kaufman, L., Thomas, J.P. (eds.) Handbook of Perception and Human Performance, Cognitive Processes and Performance, vol. 2. Wiley, Hoboken (1986)Google Scholar
  36. 36.
    Pham, T.T., Nguyen, T.D., Vo, T.: Sparse fNIRS feature estimation via unsupervised learning for mental workload classification. In: Bassis, S., Esposito, A., Morabito, F.C., Pasero, E. (eds.) Advances in Neural Networks. SIST, vol. 54, pp. 283–292. Springer, Cham (2016). doi: 10.1007/978-3-319-33747-0_28 CrossRefGoogle Scholar
  37. 37.
    Rao, R.B., Fung, G., Rosales, R.: On the dangers of cross-validation. an experimental evaluation. In: Proceedings of the 2008 SIAM International Conference on Data Mining, pp. 588–596. SIAM (2008)Google Scholar
  38. 38.
    Reid, G.B., Nygren, T.E.: The Subjective Workload Assessment Technique: A Scaling Procedure for Measuring Mental Workload, vol. 52, North-Holland (1988)Google Scholar
  39. 39.
    Rizzo, L., Dondio, P., Delany, S.J., Longo, L.: Modeling mental workload via rule-based expert system: a comparison with NASA-TLX and workload profile. In: Iliadis, L., Maglogiannis, I. (eds.) AIAI 2016. IAICT, vol. 475, pp. 215–229. Springer, Cham (2016). doi: 10.1007/978-3-319-44944-9_19 CrossRefGoogle Scholar
  40. 40.
    Rubio, S., Díaz, E., Martín, J., Puente, J.M.: Evaluation of subjective mental workload: a comparison of SWAT, NASA-TLX, and workload profile methods. Appl. Psychol. 53(1), 61–86 (2004)CrossRefGoogle Scholar
  41. 41.
    Solovey, E., Schermerhorn, P., Scheutz, M., Sassaroli, A., Fantini, S., Jacob, R.: Brainput: Enhancing interactive systems with streaming fNIRS Brain Input. In: Proceedings of the 2012 ACM Annual Conference on Human Factors in Computing Systems - CHI 2012. p. 2193. ACM (2012)Google Scholar
  42. 42.
    Stassen, H.G., Johannsen, G., Moray, N.: Internal representation, internal model, human performance model and mental workload. Automatica 26(4), 811–820 (1990)CrossRefGoogle Scholar
  43. 43.
    Stevens, R., Galloway, T., Berka, C.: Integrating EEG models of cognitive load with machine learning models of scientific problem solving. In: Proceedings of 2nd Annual Augmented Cognition International Conference (September, 2006)Google Scholar
  44. 44.
    Su, J., Luz, S.: Predicting cognitive load levels from speech data. Smart Innov. Syst. Technol. 48, 255–263 (2016)CrossRefGoogle Scholar
  45. 45.
    Thompson, S.K.: Stratified Sampling, pp. 139–156. Wiley, Hoboken (2012)Google Scholar
  46. 46.
    Trucco, P., Leva, M.C., Sträter, O.: Human error prediction in ATM via cognitive simulation: preliminary study. In: Proceedings of the 8th International Conference on Probabilistic Safety Assessment and Management (PSAM8), pp. 1–9 (2006)Google Scholar
  47. 47.
    Tsang, P.S., Velazquez, V.L.: Diagnosticity and multidimensional subjective workload ratings. Ergonomics 39(3), 358–381 (1996)CrossRefGoogle Scholar
  48. 48.
    Wickens, C.D.: Multiple resources and mental workload. Hum. Factors 50(3), 449–455 (2008)CrossRefGoogle Scholar
  49. 49.
    Wiebe, E.N., Roberts, E., Behrend, T.S.: An examination of two mental workload measurement approaches to understanding multimedia learning. Comput. Hum. Behav. 26(3), 474–481 (2010)CrossRefGoogle Scholar
  50. 50.
    Yoshida, Y., Ohwada, H., Mizoguchi, F., Iwasaki, H.: Classifying cognitive load and driving situation with machine learning. Int. J. Mach. Learn. Comput. 4(3), 210–215 (2014)CrossRefGoogle Scholar
  51. 51.
    Zhang, Y.Z.Y., Owechko, Y., Zhang, J.Z.J.: Driver cognitive workload estimation: a data-driven perspective. In: Proceedings of the 7th International IEEE Conference on Intelligent Transportation Systems (IEEE Cat. No.04TH8749), pp. 642–647 (2004)Google Scholar

Copyright information

© Springer International Publishing AG 2017

Authors and Affiliations

  1. 1.School of ComputingDublin Institute of TechnologyDublinIreland
  2. 2.Usher Institute of Population Health Sciences and InformaticsThe University of EdinburghEdinburghScotland
  3. 3.The ADAPT CentreDublinIreland

Personalised recommendations