Cognitive Workload of Humans Using Artificial Intelligence Systems: Towards Objective Measurement Applying Eye-Tracking Technology

  • Ricardo Buettner
Part of the Lecture Notes in Computer Science book series (LNCS, volume 8077)


Replying to corresponding research calls I experimentally investigate whether a higher level of artificial intelligence support leads to a lower user cognitive workload. Applying eye-tracking technology I show how the user’s cognitive workload can be measure more objectively by capturing eye movements and pupillary responses. Within a laboratory environment which adequately reflects a realistic working situation, the probands use two distinct systems with similar user interfaces but very different levels of artificial intelligence support. Recording and analyzing objective eye-tracking data (i.e. pupillary diameter mean, pupillary diameter deviation, number of gaze fixations and eye saccade speed of both left and right eyes) – all indicating cognitive workload – I found significant systematic cognitive workload differences between both test systems. My results indicated that a higher AI-support leads to lower user cognitive workload.


artificial intelligence support cognitive workload pupillary diameter eye movements eye saccades eye-tracking argumentation-based negotiation argumentation-generation 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Bernstein, A., Klein, M., Malone, T.W.: Programming the Global Brain. CACM 55(5), 41–43 (2012)CrossRefGoogle Scholar
  2. 2.
    Malone, T.W., Laubacher, R., Dellarocas, C.: The Collective Intelligence Genome. MIT Sloan Management Review 51(3), 21–31 (2010)Google Scholar
  3. 3.
    Davis, J., Lin, H.: Web 3.0 and Crowdservicing. In: AMCIS 2011 Proc. (2011)Google Scholar
  4. 4.
    Davis, R., Smith, R.G.: Negotiation as a Metaphor for Distributed Problem Solving. AI 20(1), 63–109 (1983)Google Scholar
  5. 5.
    Carneiro, D., Novais, P., Andrade, F., Zeleznikow, J., Neves, J.: Online dispute resolution: an artificial intelligence perspective. Artif. Intell. Rev. (2012) (in Press)Google Scholar
  6. 6.
    Park, H.J., Kim, B.K., Lim, K.Y.: Measuring the machine intelligence quotient (MIQ) of human-machine cooperative systems. IEEE TSMC, Part A 31(2), 89–96 (2001)Google Scholar
  7. 7.
    Wilson, M.: Six views of embodied cognition. Psychonomic Bulletin & Review 9(4), 625–636 (2002)CrossRefGoogle Scholar
  8. 8.
    Cain, B.: A Review of the Mental Workload Literature. Report, NATO (2007)Google Scholar
  9. 9.
    Ayyagari, R., Grover, V., Purvis, R.: Technostress: Technological Antecedents and Implications. MISQ 35(4), 831–858 (2011)Google Scholar
  10. 10.
    Tarafdar, M., Tu, Q., Ragu-Nathan, T.S.: Impact of Technostress on End-User Satisfaction and Performance. JMIS 27(3), 303–334 (2010)Google Scholar
  11. 11.
    Gupta, A., Li, H., Sharda, R.: Should I send this message? Understanding the impact of interruptions, social hierarchy and perceived task complexity on user performance and perceived workload. DSS 55(1), 135–145 (2013)Google Scholar
  12. 12.
    Ragu-Nathan, T.S., Tarafdar, M., Ragu-Nathan, B.S., Tu, Q.: The Consequences of Technostress for End Users in Organizations: Conceptual Development and Empirical Validation. ISR 19(4), 417–433 (2008)CrossRefGoogle Scholar
  13. 13.
    Wastell, D.G.: Learning Dysfunctions in Information Systems Development: Overcoming the Social Defenses With Transitional Objects. MISQ 23(4), 581–600 (1999)CrossRefGoogle Scholar
  14. 14.
    Sun, Y., Lim, K.H., Peng, J.Z.: Solving the Distinctiveness - Blindness Debate: A Unified Model for Understanding Banner Processing. JAIS 14(2), 49–71 (2013)Google Scholar
  15. 15.
    Ren, P., Barreto, A., Gao, Y., Adjouadi, M.: Affective Assessment by Digital Processing of the Pupil Diameter. IEEE TAC 4(1), 2–14 (2013)Google Scholar
  16. 16.
    Dimoka, A.: What Does the Brain Tell Us About Trust and Distrust? Evidence from a Functional Neuroimaging Study. MISQ 34(2), 373–396 (2010)Google Scholar
  17. 17.
    Dimoka, A., Pavlou, P.A., Davis, F.D.: NeuroIS: The Potential of Cognitive Neuroscience for Information Systems Research. ISR 22(4), 687–702 (2011)CrossRefGoogle Scholar
  18. 18.
    Stassen, H.G., Johannsen, G., Moray, N.: Internal representation, internal model, human performance model and mental workload. Automatica 26(4), 811–820 (1990)CrossRefGoogle Scholar
  19. 19.
    Steinfeld, A., Fong, T., Kaber, D., Lewis, M., Scholtz, J., Schultz, A., Goodrich, M.: Common Metrics for Human-Robot Interaction. In: HRI 2006 Proc., pp. 33–40 (2006)Google Scholar
  20. 20.
    Johannsen, G., Levis, A.H., Stassen, H.G.: Theoretical Problems in Man-machine Systems and Their Experimental Validation. Automatica 30(2), 217–231 (1992)CrossRefGoogle Scholar
  21. 21.
    Lopes, F., Wooldridge, M., Novais, A.Q.: Negotiation among autonomous computational agents: principles, analysis and challenges. Artif. Intell. Rev. 29(1), 1–44 (2008)CrossRefGoogle Scholar
  22. 22.
    Amgoud, L., Vesic, S.: A formal analysis of the role of argumentation in negotiation dialogues. J. Logic Comput. 22(5), 957–978 (2012)MathSciNetzbMATHCrossRefGoogle Scholar
  23. 23.
    Beatty, J.: Task-evoked pupillary responses, processing load, and the structure of processing resources. Psychol. Bull. 91(2), 276–292 (1982)CrossRefGoogle Scholar
  24. 24.
    Hess, E.H., Polt, J.M.: Pupil Size in Relation to Mental Activity during Simple Problem-Solving. Science 143(3611), 1190–1192 (1964)CrossRefGoogle Scholar
  25. 25.
    Kahneman, D., Beatty, J.: Pupil Diameter and Load on Memory. Science 154(3756), 1583–1585 (1966)CrossRefGoogle Scholar
  26. 26.
    Bradshaw, J.: Pupil Size as a Measure of Arousal during Information Processing. Nature 216(5114), 515–516 (1967)CrossRefGoogle Scholar
  27. 27.
    Simpson, H.M.: Effects of a Task-Relevant Response on Pupil Size. Psychophysiology 6(2), 115–121 (1969)CrossRefGoogle Scholar
  28. 28.
    Kahneman, D., Beatty, J., Pollack, I.: Perceptual Deficit during a Mental Task. Science 157(3785), 218–219 (1967)CrossRefGoogle Scholar
  29. 29.
    Beatty, J., Wagoner, B.L.: Pupillometric signs of brain activation vary with level of cognitive processing. Science 199(4334), 1216–1218 (1978)CrossRefGoogle Scholar
  30. 30.
    Rayner, K.: Eye movements in reading and information processing: 20 years of research. Psychol. Bull. 124(3), 372–422 (1998)CrossRefGoogle Scholar
  31. 31.
    Leigh, R.J., Kennard, C.: Using saccades as a research tool in the clinical neurosciences. Brain 127(3), 460–477 (2004)CrossRefGoogle Scholar
  32. 32.
    Van Orden, K.F., Limbert, W., Makeig, S., Jung, T.P.: Eye Activity Correlates of Workload during a Visuospatial Memory Task. Hum. Factors 43(1), 111–121 (2001)CrossRefGoogle Scholar
  33. 33.
    Just, M., Carpenter, P.: Eye fixations and cognitive processes. Cognit. Psychol. 8(4), 441–480 (1976)CrossRefGoogle Scholar
  34. 34.
    Just, M.A., Carpenter, P.A.: A theory of reading: From eye fixations to comprehension. Psychol. Rev. 87(4), 329–354 (1980)CrossRefGoogle Scholar
  35. 35.
    Wierda, S.M., van Rijn, H., Taatgen, N.A., Martens, S.: Pupil dilation deconvolution reveals the dynamics of attention at high temporal resolution. PNAS 109(22), 8456–8460 (2012)CrossRefGoogle Scholar
  36. 36.
    Loft, S., Sanderson, P., Neal, A., Mooij, M.: Modeling and Predicting Mental Workload in En Route Air Traffic Control: Critical Review and Broader Implications. Hum. Factors 49(3), 376–399 (2007)CrossRefGoogle Scholar
  37. 37.
    Bailey, B.P., Iqbal, S.T.: Understanding Changes in Mental Workload during Execution of Goal-Directed Tasks and Its Application for Interruption Management. ACM TOCHI 14(4), 21:1–21:28 (2008)Google Scholar
  38. 38.
    Baltaci, S., Gokcay, D.: Negative Sentiment in Scenarios Elicit Pupil Dilation Response: An Auditory Study. In: ICMI 2012 Proc., pp. 529–532 (2012)Google Scholar
  39. 39.
    Iqbal, S.T., Adamczyk, P.D., Zheng, X.S., Bailey, B.P.: Towards an Index of Opportunity: Understanding Changes in Mental Workload during Task Execution. In: CHI 2005 Proc., pp. 311–320 (2005)Google Scholar
  40. 40.
    Wang, W., Li, Z., Wang, Y., Chen, F.: Indexing cognitive workload based on pupillary response under luminance and emotional changes. In: IUI 2013 Proc., pp. 247–256 (2013)Google Scholar
  41. 41.
    Bee, N., Prendinger, H., Nakasone, A., André, E., Ishizuka, M.: AutoSelect: What You Want Is What You Get: Real-Time Processing of Visual Attention and Affect. In: André, E., Dybkjær, L., Minker, W., Neumann, H., Weber, M. (eds.) PIT 2006. LNCS (LNAI), vol. 4021, pp. 40–52. Springer, Heidelberg (2006)CrossRefGoogle Scholar
  42. 42.
    Ren, P., Barreto, A., Gao, Y., Adjouadi, M.: Affective Assessment of Computer Users Based on Processing the Pupil Diameter Signal. In: 2011 IEEE Eng. Med. Biol. Soc. Proc., pp. 2594–2597 (2011)Google Scholar
  43. 43.
    Zhai, J., Barreto, A.: Stress Detection in Computer Users Based on Digital Signal Processing of Noninvasive Physiological Variables. In: IEEE EMBS 2006 Proc., pp. 1355–1358 (2006)Google Scholar
  44. 44.
    Pomplun, M., Sunkara, S.: Pupil Dilation as an Indicator of Cognitive Workload in Human-Computer Interaction. In: HCII 2003 Proc., pp. 542–546 (2003)Google Scholar
  45. 45.
    Longo, L.: Human-Computer Interaction and Human Mental Workload: Assessing Cognitive Engagement in the World Wide Web. In: Campos, P., Graham, N., Jorge, J., Nunes, N., Palanque, P., Winckler, M. (eds.) INTERACT 2011, Part IV. LNCS, vol. 6949, pp. 402–405. Springer, Heidelberg (2011)CrossRefGoogle Scholar
  46. 46.
    Cegarra, J., Chevalier, A.: The use of Tholos software for combining measures of mental workload: Toward theoretical and methodological improvements. Behav. Res. Methods 40(4), 988–1000 (2008)CrossRefGoogle Scholar
  47. 47.
    Xu, J., Wang, Y., Chen, F., Choi, E.: Pupillary Response Based Cognitive Workload Measurement under Luminance Changes. In: Campos, P., Graham, N., Jorge, J., Nunes, N., Palanque, P., Winckler, M. (eds.) INTERACT 2011, Part II. LNCS, vol. 6947, pp. 178–185. Springer, Heidelberg (2011)CrossRefGoogle Scholar
  48. 48.
    Steinhauer, S.R., Siegle, G.J., Condray, R., Pless, M.: Sympathetic and parasympathetic innervation of pupillary dilation during sustained processing. Int. J. Psychophysiol. 52(1), 77–86 (2004)CrossRefGoogle Scholar
  49. 49.
    McBurney, P., Eijk, R.M.V., Parsons, S., Amgoud, L.: A Dialogue Game Protocol for Agent Purchase Negotiations. JAAMAS 7(3), 235–273 (2003)Google Scholar
  50. 50.
    Buettner, R.: The State of the Art in Automated Negotiation Models of the Behavior and Information Perspective. ITSSA 1(4), 351–356 (2006)Google Scholar
  51. 51.
    Buettner, R.: A Classification Structure for Automated Negotiations. In: IEEE/WIC/ACM WI-IAT 2006 Proc., pp. 523–530 (2006)Google Scholar
  52. 52.
    Buettner, R., Landes, J.: Web Service-based Applications for Electronic Labor Markets: A Multi-dimensional Price VCG Auction with Individual Utilities. In: ICIW 2012 Proc., pp. 168–177 (2012)Google Scholar
  53. 53.
    Landes, J., Buettner, R.: Argumentation–Based Negotiation? Negotiation–Based Argumentation! In: Huemer, C., Lops, P. (eds.) EC-Web 2012. LNBIP, vol. 123, pp. 149–162. Springer, Heidelberg (2012)CrossRefGoogle Scholar
  54. 54.
    Eckhardt, A., Maier, C., Buettner, R.: The Influence of Pressure to Perform and Experience on Changing Perceptions and User Performance: A Multi-Method Experimental Analysis. In: ICIS 2012 Proc. (2012)Google Scholar
  55. 55.
    Steinhauer, S.R., Condray, R., Kasparek, A.: Cognitive modulation of midbrain function: task-induced reduction of the pupillary light reflex. Int. J. Psychophysiol. 39(1), 21–30 (2000)CrossRefGoogle Scholar
  56. 56.
    Gettinger, J., Koeszegi, S.T., Schoop, M.: Shall we dance? - The effect of information presentations on negotiation processes and outcomes. DSS 53, 161–174 (2012)Google Scholar
  57. 57.
    Schoop, M., Köhne, F., Staskiewicz, D.: An Integrated Decision and Communication Perspective on Electronic Negotiation Support Systems - Challenges and Solutions. Journal of Decision Systems 13(4), 375–398 (2004)CrossRefGoogle Scholar
  58. 58.
    Dimoka, A., Davis, F.D.: Where Does TAM Reside in the Brain? The Neural Mechanisms Underlying Technology Adoption. In: ICIS 2008 Proc., Paper 169 (2008)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2013

Authors and Affiliations

  • Ricardo Buettner
    • 1
  1. 1.Institute of Management & Information SystemsFOM University of Applied SciencesMunichGermany

Personalised recommendations