Skip to main content

Attention Guidance Agents with Eye-Tracking

A Use-Case Based on the MATBII Cockpit Task

  • Conference paper
  • First Online:
Engineering Multi-Agent Systems (EMAS 2021)

Part of the book series: Lecture Notes in Computer Science ((LNAI,volume 13190))

Included in the following conference series:


A first step to keeping the human ‘in the loop’ in the context of developing intelligent multi-task interfaces is to be able to monitor their attention. By combining eye tracking with agent monitoring and decision making, we provide a basis for increasing the user’s attentional bandwidth by offering bottom-up attention guidance. We develop a modified implementation of the MATBII cockpit task simulator embedded in an agent environment in which agents monitor events, including eye tracking, and act to deploy visual cues to guide attention. We explore how such a system may be useful for improving task performance, by also simulating users with agents to demonstrate how the system might work for some examples of user behaviour. We also discuss how our system can act as an experimental platform to benefit future user experience research focusing on attention guidance in complex multi-task interfaces.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
USD 64.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 84.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions


  1. 1.

  2. 2.


  1. PyStarWorlds (2021).

  2. Aha, D.W.: Goal reasoning: foundations, emerging applications, and prospects. AI Mag. 39(2), 3–24 (2018)

    MathSciNet  Google Scholar 

  3. Aschermann, M., Kraus, P., Müller, J.P.: LightJason. In: Criado Pacheco, N., Carrascosa, C., Osman, N., Julián Inglada, V. (eds.) EUMAS/AT -2016. LNCS (LNAI), vol. 10207, pp. 58–66. Springer, Cham (2017).

    Chapter  Google Scholar 

  4. Bagga, P., Paoletti, N., Alrayes, B., Stathis, K.: A deep reinforcement learning approach to concurrent bilateral negotiation. In: Proceedings of the 29th International Joint Conference on Artificial Intelligence, IJCAI 2020, pp. 297–303. IJCAI/AAAI, Yokohama (2020)

    Google Scholar 

  5. Berthelot, B., Mazoyer, P., Egea, S., André, J.M., Grivel, É., Legrand, P.: Self-affinity of an aircraft pilot’s gaze direction as a marker of visual tunneling. Technical report, SAE Technical Paper (2019).

  6. Bolstad, C., Costello, A., Endsley, M.: Bad situation awareness designs: what went wrong and why. In: Proceedings of the 16th World Congress of International Ergonomics Association (2006)

    Google Scholar 

  7. Bromuri, S., Stathis, K.: Situating cognitive agents in GOLEM. In: Weyns, D., Brueckner, S.A., Demazeau, Y. (eds.) EEMMAS 2007. LNCS (LNAI), vol. 5049, pp. 115–134. Springer, Heidelberg (2008).

    Chapter  Google Scholar 

  8. Bromuri, S., Stathis, K.: Distributed agent environments in the ambient event calculus. In: Gokhale, A.S., Schmidt, D.C. (eds.) Proceedings of the Third ACM International Conference on Distributed Event-Based Systems, DEBS 2009, Nashville, Tennessee, USA, 6–9 July 2009. ACM (2009)

    Google Scholar 

  9. Bunch, L., et al.: Software agents for process monitoring and notification. In: Proceedings of the 2004 ACM Symposium on Applied Computing, SAC 2004, pp. 94–100. Association for Computing Machinery, New York (2004).

  10. Calegari, R., Ciatto, G., Mascardi, V., Omicini, A.: Logic-based technologies for multi-agent systems: a systematic literature review. Auton. Agent. Multi-agent Syst. 35(1), 1–67 (2020).

    Article  Google Scholar 

  11. Carrasco, M.: Visual attention: the past 25 years. Vis. Res. 51(13), 1484–1525 (2011).

    Article  Google Scholar 

  12. Cegarra, J., Valéry, B., Avril, E., Calmettes, C., Navarro, J.: OpenMATB: a multi-attribute task battery promoting task customization, software extensibility and experiment replicability. Behav. Res. Methods 52(5), 1980–1990 (2020).

    Article  Google Scholar 

  13. Clark, K., et al.: A framework for integrating symbolic and sub-symbolic representations. In: Proceedings of the Twenty-Fifth International Joint Conference on Artificial Intelligence, IJCAI 2016, pp. 2486–2492. AAAI Press (2016)

    Google Scholar 

  14. Coffey., S., Clark., K.: A hybrid, teleo-reactive architecture for robot control. In: Proceedings of the 2nd International Workshop on Multi-agent Robotic Systems - MARS, (ICINCO 2006), pp. 54–65. INSTICC, SciTePress (2006).

  15. Danieau, F., Guillo, A., Doré, R.: Attention guidance for immersive video content in head-mounted displays. In: 2017 IEEE Virtual Reality (VR), pp. 205–206 (2017).

  16. Davies, A.: The war to remotely control self-driving cars heats up. Wired. Accessed 11 Nov 2019

  17. Debie, E., et al.: Multimodal fusion for objective assessment of cognitive workload: a review. IEEE Trans. Cybern. (2019).

    Article  Google Scholar 

  18. Dongol, B., Hayes, I.J., Robinson, P.J.: Reasoning about goal-directed real-time teleo-reactive programs. Formal Aspects Comput. 26(3), 563–589 (2014)

    Article  MathSciNet  Google Scholar 

  19. Dzindolet, M.T., Peterson, S.A., Pomranky, R.A., Pierce, L.G., Beck, H.P.: The role of trust in automation reliance. Int. J. Hum. Comput. Stud. 58(6), 697–718 (2003).

    Article  Google Scholar 

  20. Endsley, M.R.: Designing for Situation Awareness: An Approach to User-Centered Design. CRC Press, Boca Raton (2016)

    Book  Google Scholar 

  21. Endsley, M.R., Kiris, E.O.: The out-of-the-loop performance problem and level of control in automation. Hum. Factors 37(2), 381–394 (1995).

    Article  Google Scholar 

  22. Fern, A., Natarajan, S., Judah, K., Tadepalli, P.: A decision-theoretic model of assistance. J. Artif. Intell. Res. 50, 71–104 (2014).

    Article  MathSciNet  Google Scholar 

  23. Glass, A., McGuinness, D.L., Wolverton, M.: Toward establishing trust in adaptive agents. In: Proceedings of the 13th International Conference on Intelligent User Interfaces, pp. 227–236 (2008).

  24. Gutzwiller, R.S., Wickens, C.D., Clegg, B.A.: Workload overload modeling: an experiment with MATBII to inform a computational model of task management. In: Proceedings of the Human Factors and Ergonomics Society Annual Meeting, vol. 58, pp. 849–853. SAGE Publications, Los Angeles (2014).

  25. Hancock, P., Scallen, S.: The performance and workload effects of task re-location during automation. Displays 17(2), 61–68 (1997).

    Article  Google Scholar 

  26. Hart, S.G., Staveland, L.E.: Development of NASA-TLX (task load index): results of empirical and theoretical research. In: Advances in Psychology, vol. 52, pp. 139–183. Elsevier (1988).

  27. Hata, H., Koike, H., Sato, Y.: Visual guidance with unnoticed blur effect. In: Proceedings of the International Working Conference on Advanced Visual Interfaces, pp. 28–35 (2016).

  28. Hou, M., Kobierski, R.D., Brown, M.: Intelligent adaptive interfaces for the control of multiple UAVs. J. Cogn. Eng. Decis. Mak. 1(3), 327–362 (2007).

    Article  Google Scholar 

  29. Inagaki, T., et al.: Adaptive automation: sharing and trading of control. In: Handbook of Cognitive Task Design, vol. 8, pp. 147–169 (2003)

    Google Scholar 

  30. Kakas, A., Mancarella, P., Sadri, F., Stathis, K., Toni, F.: Declarative agent control. In: Leite, J., Torroni, P. (eds.) CLIMA 2004. LNCS (LNAI), vol. 3487, pp. 96–110. Springer, Heidelberg (2005).

    Chapter  MATH  Google Scholar 

  31. Kakas, A., Mancarella, P., Sadri, F., Stathis, K., Toni, F.: Computational logic foundations of KGP agents. J. Artif. Intell. Res. 33, 285–348 (2008).

    Article  MathSciNet  MATH  Google Scholar 

  32. Kim, J.H., Yang, X.: Applying fractal analysis to pupil dilation for measuring complexity in a process monitoring task. Appl. Ergon. 65, 61–69 (2017).

    Article  Google Scholar 

  33. Klauck, M., Sugano, Y., Bulling, A.: Noticeable or distractive? A design space for gaze-contingent user interface notifications. In: Proceedings of the 2017 CHI Conference Extended Abstracts on Human Factors in Computing Systems, pp. 1779–1786 (2017).

  34. Kowalski, R.A., Sadri, F.: Teleo-reactive abductive logic programs. In: Artikis, A., Craven, R., Kesim Çiçekli, N., Sadighi, B., Stathis, K. (eds.) Logic Programs, Norms and Action. LNCS (LNAI), vol. 7360, pp. 12–32. Springer, Heidelberg (2012).

    Chapter  MATH  Google Scholar 

  35. Krauth, E.I., van Hillegersberg, J., Van De Velde, S.L.: Agent-based human-computer-interaction for real-time monitoring systems in the trucking industry. In: 2007 40th Annual Hawaii International Conference on System Sciences (HICSS 2007), p. 27 (2007).

  36. Kravari, K., Bassiliades, N.: A survey of agent platforms. J. Artif. Soc. Soc. Simul. 18(1) (2015).

  37. Laird, J.E.: The SOAR Cognitive Architecture. MIT Press, Cambridge (2012)

    Book  Google Scholar 

  38. Maes, P.: Agents that reduce work and information overload. In: Readings in Human-Computer Interaction, pp. 811–821. Elsevier (1995).

  39. Majaranta, P., Bulling, A.: Eye tracking and eye-based human–computer interaction. In: Fairclough, S.H., Gilleade, K. (eds.) Advances in Physiological Computing. HIS, pp. 39–65. Springer, London (2014).

    Chapter  Google Scholar 

  40. Marshall, S.P.: The index of cognitive activity: measuring cognitive workload. In: Proceedings of the IEEE 7th Conference on Human Factors and Power Plants, p. 7. IEEE (2002).

  41. Martens, M., Van Winsum, W.: Measuring Distraction: The Peripheral Detection Task. TNO Human Factors, Soesterberg (2000)

    Google Scholar 

  42. Matthews, G., Davies, D.R., Stammers, R.B., Westerman, S.J.: Human Performance: Cognition, Stress, and Individual Differences. Psychology Press, Hove (2000)

    Google Scholar 

  43. Matthews, T., Dey, A.K., Mankoff, J., Carter, S., Rattenbury, T.: A toolkit for managing user attention in peripheral displays. In: Proceedings of the 17th Annual ACM Symposium on User Interface Software and Technology, pp. 247–256 (2004).

  44. Miller, T.: Explanation in artificial intelligence: insights from the social sciences. Artif. Intell. 267, 1–38 (2019).

    Article  MathSciNet  MATH  Google Scholar 

  45. Navalpakkam, V., Itti, L.: A goal oriented attention guidance model. In: Bülthoff, H.H., Wallraven, C., Lee, S.-W., Poggio, T.A. (eds.) BMCV 2002. LNCS, vol. 2525, pp. 453–461. Springer, Heidelberg (2002).

    Chapter  Google Scholar 

  46. Nelson, J.M., Phillips, C.A., McKinley, R.A., McIntire, L.K., Goodyear, C., Monforton, L.: The effects of transcranial direct current stimulation (tDCS) on multitasking performance and oculometrics. Mil. Psychol. 31(3), 212–226 (2019).

    Article  Google Scholar 

  47. Nilsson, N.J.: Teleo-reactive programs for agent control. J. Artif. Int. Res. 1(1), 139–158 (1994).

    Article  Google Scholar 

  48. Ohneiser, O., Gürlük, H., Jauer, M.L., Szöllősi, Á., Balló, D.: Please have a look here: successful guidance of air traffic controller’s attention (2019)

    Google Scholar 

  49. Olsen, A.: The Tobii IVT Fixation Filter Algorithm description (2012)

    Google Scholar 

  50. Onken, R., Walsdorf, A.: Assistant systems for aircraft guidance: cognitive man-machine cooperation. Aerosp. Sci. Technol. 5(8), 511–520 (2001).

    Article  MATH  Google Scholar 

  51. Peirce, J., MacAskill, M.: Building Experiments in PsychoPy. Sage, Thousand Oaks (2018)

    Google Scholar 

  52. Poitschke, T., Laquai, F., Rigoll, G.: Guiding a driver’s visual attention using graphical and auditory animations. In: Harris, D. (ed.) EPCE 2009. LNCS (LNAI), vol. 5639, pp. 424–433. Springer, Heidelberg (2009).

    Chapter  Google Scholar 

  53. Pomplun, M., Sunkara, S.: Pupil dilation as an indicator of cognitive workload in human-computer interaction. In: Proceedings of the International Conference on HCI, vol. 273 (2003)

    Google Scholar 

  54. Roda, C., Thomas, J.: Attention aware systems: theories, applications, and research agenda. Comput. Hum. Behav. 22(4), 557–587 (2006).

    Article  Google Scholar 

  55. Rodden, T.A., Fischer, J.E., Pantidi, N., Bachour, K., Moran, S.: At home with agents: exploring attitudes towards future smart energy infrastructures. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, CHI 2013, pp. 1173–1182. Association for Computing Machinery, New York (2013).

  56. Rosch, J.L., Vogel-Walcutt, J.J.: A review of eye-tracking applications as tools for training. Cogn. Technol. Work 15(3), 313–327 (2013).

    Article  Google Scholar 

  57. Sadri, F.: Logic-based approaches to intention recognition. In: Handbook of Research on Ambient Intelligence and Smart Environments: Trends and Perspectives, pp. 346–375. IGI Global (2011).

  58. Sadri, F., Stathis, K., Toni, F.: Normative KGP agents. Comput. Math. Organ. Theory 12(2–3), 101–126 (2006)

    Article  Google Scholar 

  59. Santiago-Espada, Y., Myer, R.R., Latorella, K.A., Comstock, J.R., Jr.: The multi-attribute task battery II. A user’s guide (MATB-II) software for human performance and workload research (2011)

    Google Scholar 

  60. de Silva, L., Meneguzzi, F., Logan, B.: BDI agent architectures: a survey. In: Bessiere, C. (ed.) Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence, IJCAI 2020, pp. 4914–4921. (2020).

  61. Stanton, N.A., Chambers, P.R., Piggott, J.: Situational awareness and safety. Saf. Sci. 39(3), 189–204 (2001).

    Article  Google Scholar 

  62. Stathis, K., Kakas, A.C., Lu, W., Demetriou, N., Endriss, U., Bracciali, A.: PROSOCS: a platform for programming software agents in computational logic. In: Müller, J., Petta, P. (eds.) Proceedings of the Fourth International Symposium “From Agent Theory to Agent Implementation” (AT2AI-4 - EMCSR 2004 Session M), Vienna, Austria, pp. 523–528 (2004)

    Google Scholar 

  63. Stratmann, T.C., Kempa, F., Boll, S.: Lame: light-controlled attention guidance for multi-monitor environments. In: Proceedings of the 8th ACM International Symposium on Pervasive Displays, pp. 1–5 (2019).

  64. Tatler, B.W., Hansen, D.W., Pelz, J.B.: Eye movement recordings in natural settings. In: Klein, C., Ettinger, U. (eds.) Eye Movement Research. SNPBE, pp. 549–592. Springer, Cham (2019).

    Chapter  Google Scholar 

  65. Toreini, P., Morana, S.: Designing attention-aware business intelligence and analytics dashboards. In: Designing the Digital Transformation: DESRIST 2017 Research in Progress Proceedings of the 12th International Conference on Design Science Research in Information Systems and Technology, Karlsruhe, Germany. 30 May–1 June, pp. 64–72. Karlsruher Institut für Technologie (KIT) (2017).

  66. Van Orden, K.F., Limbert, W., Makeig, S., Jung, T.P.: Eye activity correlates of workload during a visuospatial memory task. Hum. Factors 43(1), 111–121 (2001).

    Article  Google Scholar 

  67. Wang, H., Chignell, M., Ishizuka, M.: Empathic tutoring software agents using real-time eye tracking. In: Proceedings of the 2006 Symposium on Eye Tracking Research & Applications, ETRA 2006, pp. 73–78. Association for Computing Machinery, New York (2006).

  68. Wickens, C.D., Hollands, J.G., Banbury, S., Parasuraman, R.: Engineering Psychology and Human Performance. Psychology Press, Hove (2015)

    Book  Google Scholar 

Download references


The work for developing this demonstrator was supported by a Human-Like Computing EPSRC Network+ Kickstart grant. Thanks to Suzy Broadbent and Lisa Boyce for discussions and support.

Author information

Authors and Affiliations


Corresponding author

Correspondence to Szonya Durant .

Editor information

Editors and Affiliations

Rights and permissions

Reprints and permissions

Copyright information

© 2022 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Durant, S., Wilkins, B., Woods, C., Uliana, E., Stathis, K. (2022). Attention Guidance Agents with Eye-Tracking. In: Alechina, N., Baldoni, M., Logan, B. (eds) Engineering Multi-Agent Systems. EMAS 2021. Lecture Notes in Computer Science(), vol 13190. Springer, Cham.

Download citation

  • DOI:

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-97456-5

  • Online ISBN: 978-3-030-97457-2

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics