Skip to main content
SpringerLink
Log in

Attention Guidance Agents with Eye-Tracking

A Use-Case Based on the MATBII Cockpit Task

  • Conference paper
  • First Online:
Engineering Multi-Agent Systems (EMAS 2021)

Part of the book series: Lecture Notes in Computer Science ((LNAI,volume 13190))

Included in the following conference series:

Abstract

A first step to keeping the human ‘in the loop’ in the context of developing intelligent multi-task interfaces is to be able to monitor their attention. By combining eye tracking with agent monitoring and decision making, we provide a basis for increasing the user’s attentional bandwidth by offering bottom-up attention guidance. We develop a modified implementation of the MATBII cockpit task simulator embedded in an agent environment in which agents monitor events, including eye tracking, and act to deploy visual cues to guide attention. We explore how such a system may be useful for improving task performance, by also simulating users with agents to demonstrate how the system might work for some examples of user behaviour. We also discuss how our system can act as an experimental platform to benefit future user experience research focusing on attention guidance in complex multi-task interfaces.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 64.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 84.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    https://dicelab-rhul.github.io/ICU/.

  2. 2.

    https://dicelab-rhul.github.io/ICU/documentation/configuration/.

References

  1. PyStarWorlds (2021). https://github.com/dicelab-rhul/pystarworlds

  2. Aha, D.W.: Goal reasoning: foundations, emerging applications, and prospects. AI Mag. 39(2), 3–24 (2018)

    MathSciNet  Google Scholar 

  3. Aschermann, M., Kraus, P., Müller, J.P.: LightJason. In: Criado Pacheco, N., Carrascosa, C., Osman, N., Julián Inglada, V. (eds.) EUMAS/AT -2016. LNCS (LNAI), vol. 10207, pp. 58–66. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-59294-7_6

    Chapter  Google Scholar 

  4. Bagga, P., Paoletti, N., Alrayes, B., Stathis, K.: A deep reinforcement learning approach to concurrent bilateral negotiation. In: Proceedings of the 29th International Joint Conference on Artificial Intelligence, IJCAI 2020, pp. 297–303. IJCAI/AAAI, Yokohama (2020)

    Google Scholar 

  5. Berthelot, B., Mazoyer, P., Egea, S., André, J.M., Grivel, É., Legrand, P.: Self-affinity of an aircraft pilot’s gaze direction as a marker of visual tunneling. Technical report, SAE Technical Paper (2019). https://doi.org/10.4271/2019-01-1852

  6. Bolstad, C., Costello, A., Endsley, M.: Bad situation awareness designs: what went wrong and why. In: Proceedings of the 16th World Congress of International Ergonomics Association (2006)

    Google Scholar 

  7. Bromuri, S., Stathis, K.: Situating cognitive agents in GOLEM. In: Weyns, D., Brueckner, S.A., Demazeau, Y. (eds.) EEMMAS 2007. LNCS (LNAI), vol. 5049, pp. 115–134. Springer, Heidelberg (2008). https://doi.org/10.1007/978-3-540-85029-8_9

    Chapter  Google Scholar 

  8. Bromuri, S., Stathis, K.: Distributed agent environments in the ambient event calculus. In: Gokhale, A.S., Schmidt, D.C. (eds.) Proceedings of the Third ACM International Conference on Distributed Event-Based Systems, DEBS 2009, Nashville, Tennessee, USA, 6–9 July 2009. ACM (2009)

    Google Scholar 

  9. Bunch, L., et al.: Software agents for process monitoring and notification. In: Proceedings of the 2004 ACM Symposium on Applied Computing, SAC 2004, pp. 94–100. Association for Computing Machinery, New York (2004). https://doi.org/10.1145/967900.967921

  10. Calegari, R., Ciatto, G., Mascardi, V., Omicini, A.: Logic-based technologies for multi-agent systems: a systematic literature review. Auton. Agent. Multi-agent Syst. 35(1), 1–67 (2020). https://doi.org/10.1007/s10458-020-09478-3

    Article  Google Scholar 

  11. Carrasco, M.: Visual attention: the past 25 years. Vis. Res. 51(13), 1484–1525 (2011). https://doi.org/10.1016/j.visres.2011.04.012

    Article  Google Scholar 

  12. Cegarra, J., Valéry, B., Avril, E., Calmettes, C., Navarro, J.: OpenMATB: a multi-attribute task battery promoting task customization, software extensibility and experiment replicability. Behav. Res. Methods 52(5), 1980–1990 (2020). https://doi.org/10.3758/s13428-020-01364-w

    Article  Google Scholar 

  13. Clark, K., et al.: A framework for integrating symbolic and sub-symbolic representations. In: Proceedings of the Twenty-Fifth International Joint Conference on Artificial Intelligence, IJCAI 2016, pp. 2486–2492. AAAI Press (2016)

    Google Scholar 

  14. Coffey., S., Clark., K.: A hybrid, teleo-reactive architecture for robot control. In: Proceedings of the 2nd International Workshop on Multi-agent Robotic Systems - MARS, (ICINCO 2006), pp. 54–65. INSTICC, SciTePress (2006). https://doi.org/10.5220/0001225300540065

  15. Danieau, F., Guillo, A., Doré, R.: Attention guidance for immersive video content in head-mounted displays. In: 2017 IEEE Virtual Reality (VR), pp. 205–206 (2017). https://doi.org/10.1109/VR.2017.7892248

  16. Davies, A.: The war to remotely control self-driving cars heats up. Wired. https://www.wired.com/story/designated-driverteleoperations-self-driving-cars/. Accessed 11 Nov 2019

  17. Debie, E., et al.: Multimodal fusion for objective assessment of cognitive workload: a review. IEEE Trans. Cybern. (2019). https://doi.org/10.1109/TCYB.2019.2939399

    Article  Google Scholar 

  18. Dongol, B., Hayes, I.J., Robinson, P.J.: Reasoning about goal-directed real-time teleo-reactive programs. Formal Aspects Comput. 26(3), 563–589 (2014)

    Article  MathSciNet  Google Scholar 

  19. Dzindolet, M.T., Peterson, S.A., Pomranky, R.A., Pierce, L.G., Beck, H.P.: The role of trust in automation reliance. Int. J. Hum. Comput. Stud. 58(6), 697–718 (2003). https://doi.org/10.1016/S1071-5819(03)00038-7

    Article  Google Scholar 

  20. Endsley, M.R.: Designing for Situation Awareness: An Approach to User-Centered Design. CRC Press, Boca Raton (2016)

    Book  Google Scholar 

  21. Endsley, M.R., Kiris, E.O.: The out-of-the-loop performance problem and level of control in automation. Hum. Factors 37(2), 381–394 (1995). https://doi.org/10.1518/001872095779064555

    Article  Google Scholar 

  22. Fern, A., Natarajan, S., Judah, K., Tadepalli, P.: A decision-theoretic model of assistance. J. Artif. Intell. Res. 50, 71–104 (2014). https://doi.org/10.1613/jair.4213

    Article  MathSciNet  Google Scholar 

  23. Glass, A., McGuinness, D.L., Wolverton, M.: Toward establishing trust in adaptive agents. In: Proceedings of the 13th International Conference on Intelligent User Interfaces, pp. 227–236 (2008). https://doi.org/10.1145/1378773.1378804

  24. Gutzwiller, R.S., Wickens, C.D., Clegg, B.A.: Workload overload modeling: an experiment with MATBII to inform a computational model of task management. In: Proceedings of the Human Factors and Ergonomics Society Annual Meeting, vol. 58, pp. 849–853. SAGE Publications, Los Angeles (2014). https://doi.org/10.1177/1541931214581179

  25. Hancock, P., Scallen, S.: The performance and workload effects of task re-location during automation. Displays 17(2), 61–68 (1997). https://doi.org/10.1016/S0141-9382(96)01018-9

    Article  Google Scholar 

  26. Hart, S.G., Staveland, L.E.: Development of NASA-TLX (task load index): results of empirical and theoretical research. In: Advances in Psychology, vol. 52, pp. 139–183. Elsevier (1988). https://doi.org/10.1016/S0166-4115(08)62386-9

  27. Hata, H., Koike, H., Sato, Y.: Visual guidance with unnoticed blur effect. In: Proceedings of the International Working Conference on Advanced Visual Interfaces, pp. 28–35 (2016). https://doi.org/10.1145/2909132.2909254

  28. Hou, M., Kobierski, R.D., Brown, M.: Intelligent adaptive interfaces for the control of multiple UAVs. J. Cogn. Eng. Decis. Mak. 1(3), 327–362 (2007). https://doi.org/10.1518/155534307X255654

    Article  Google Scholar 

  29. Inagaki, T., et al.: Adaptive automation: sharing and trading of control. In: Handbook of Cognitive Task Design, vol. 8, pp. 147–169 (2003)

    Google Scholar 

  30. Kakas, A., Mancarella, P., Sadri, F., Stathis, K., Toni, F.: Declarative agent control. In: Leite, J., Torroni, P. (eds.) CLIMA 2004. LNCS (LNAI), vol. 3487, pp. 96–110. Springer, Heidelberg (2005). https://doi.org/10.1007/11533092_6

    Chapter  MATH  Google Scholar 

  31. Kakas, A., Mancarella, P., Sadri, F., Stathis, K., Toni, F.: Computational logic foundations of KGP agents. J. Artif. Intell. Res. 33, 285–348 (2008). https://doi.org/10.1613/jair.2596

    Article  MathSciNet  MATH  Google Scholar 

  32. Kim, J.H., Yang, X.: Applying fractal analysis to pupil dilation for measuring complexity in a process monitoring task. Appl. Ergon. 65, 61–69 (2017). https://doi.org/10.1016/j.apergo.2017.06.002

    Article  Google Scholar 

  33. Klauck, M., Sugano, Y., Bulling, A.: Noticeable or distractive? A design space for gaze-contingent user interface notifications. In: Proceedings of the 2017 CHI Conference Extended Abstracts on Human Factors in Computing Systems, pp. 1779–1786 (2017). https://doi.org/10.1145/3027063.3053085

  34. Kowalski, R.A., Sadri, F.: Teleo-reactive abductive logic programs. In: Artikis, A., Craven, R., Kesim Çiçekli, N., Sadighi, B., Stathis, K. (eds.) Logic Programs, Norms and Action. LNCS (LNAI), vol. 7360, pp. 12–32. Springer, Heidelberg (2012). https://doi.org/10.1007/978-3-642-29414-3_3

    Chapter  MATH  Google Scholar 

  35. Krauth, E.I., van Hillegersberg, J., Van De Velde, S.L.: Agent-based human-computer-interaction for real-time monitoring systems in the trucking industry. In: 2007 40th Annual Hawaii International Conference on System Sciences (HICSS 2007), p. 27 (2007). https://doi.org/10.1109/HICSS.2007.52

  36. Kravari, K., Bassiliades, N.: A survey of agent platforms. J. Artif. Soc. Soc. Simul. 18(1) (2015). https://doi.org/10.18564/jasss.2661

  37. Laird, J.E.: The SOAR Cognitive Architecture. MIT Press, Cambridge (2012)

    Book  Google Scholar 

  38. Maes, P.: Agents that reduce work and information overload. In: Readings in Human-Computer Interaction, pp. 811–821. Elsevier (1995). https://doi.org/10.1016/B978-0-08-051574-8.50084-4

  39. Majaranta, P., Bulling, A.: Eye tracking and eye-based human–computer interaction. In: Fairclough, S.H., Gilleade, K. (eds.) Advances in Physiological Computing. HIS, pp. 39–65. Springer, London (2014). https://doi.org/10.1007/978-1-4471-6392-3_3

    Chapter  Google Scholar 

  40. Marshall, S.P.: The index of cognitive activity: measuring cognitive workload. In: Proceedings of the IEEE 7th Conference on Human Factors and Power Plants, p. 7. IEEE (2002). https://doi.org/10.1109/HFPP.2002.1042860

  41. Martens, M., Van Winsum, W.: Measuring Distraction: The Peripheral Detection Task. TNO Human Factors, Soesterberg (2000)

    Google Scholar 

  42. Matthews, G., Davies, D.R., Stammers, R.B., Westerman, S.J.: Human Performance: Cognition, Stress, and Individual Differences. Psychology Press, Hove (2000)

    Google Scholar 

  43. Matthews, T., Dey, A.K., Mankoff, J., Carter, S., Rattenbury, T.: A toolkit for managing user attention in peripheral displays. In: Proceedings of the 17th Annual ACM Symposium on User Interface Software and Technology, pp. 247–256 (2004). https://doi.org/10.1145/1029632.1029676

  44. Miller, T.: Explanation in artificial intelligence: insights from the social sciences. Artif. Intell. 267, 1–38 (2019). https://doi.org/10.1016/j.artint.2018.07.007

    Article  MathSciNet  MATH  Google Scholar 

  45. Navalpakkam, V., Itti, L.: A goal oriented attention guidance model. In: Bülthoff, H.H., Wallraven, C., Lee, S.-W., Poggio, T.A. (eds.) BMCV 2002. LNCS, vol. 2525, pp. 453–461. Springer, Heidelberg (2002). https://doi.org/10.1007/3-540-36181-2_45

    Chapter  Google Scholar 

  46. Nelson, J.M., Phillips, C.A., McKinley, R.A., McIntire, L.K., Goodyear, C., Monforton, L.: The effects of transcranial direct current stimulation (tDCS) on multitasking performance and oculometrics. Mil. Psychol. 31(3), 212–226 (2019). https://doi.org/10.3389/fnhum.2016.00589

    Article  Google Scholar 

  47. Nilsson, N.J.: Teleo-reactive programs for agent control. J. Artif. Int. Res. 1(1), 139–158 (1994). https://doi.org/10.1613/jair.30

    Article  Google Scholar 

  48. Ohneiser, O., Gürlük, H., Jauer, M.L., Szöllősi, Á., Balló, D.: Please have a look here: successful guidance of air traffic controller’s attention (2019)

    Google Scholar 

  49. Olsen, A.: The Tobii IVT Fixation Filter Algorithm description (2012)

    Google Scholar 

  50. Onken, R., Walsdorf, A.: Assistant systems for aircraft guidance: cognitive man-machine cooperation. Aerosp. Sci. Technol. 5(8), 511–520 (2001). https://doi.org/10.1016/S1270-9638(01)01137-3

    Article  MATH  Google Scholar 

  51. Peirce, J., MacAskill, M.: Building Experiments in PsychoPy. Sage, Thousand Oaks (2018)

    Google Scholar 

  52. Poitschke, T., Laquai, F., Rigoll, G.: Guiding a driver’s visual attention using graphical and auditory animations. In: Harris, D. (ed.) EPCE 2009. LNCS (LNAI), vol. 5639, pp. 424–433. Springer, Heidelberg (2009). https://doi.org/10.1007/978-3-642-02728-4_45

    Chapter  Google Scholar 

  53. Pomplun, M., Sunkara, S.: Pupil dilation as an indicator of cognitive workload in human-computer interaction. In: Proceedings of the International Conference on HCI, vol. 273 (2003)

    Google Scholar 

  54. Roda, C., Thomas, J.: Attention aware systems: theories, applications, and research agenda. Comput. Hum. Behav. 22(4), 557–587 (2006). https://doi.org/10.1016/j.chb.2005.12.005

    Article  Google Scholar 

  55. Rodden, T.A., Fischer, J.E., Pantidi, N., Bachour, K., Moran, S.: At home with agents: exploring attitudes towards future smart energy infrastructures. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, CHI 2013, pp. 1173–1182. Association for Computing Machinery, New York (2013). https://doi.org/10.1145/2470654.2466152

  56. Rosch, J.L., Vogel-Walcutt, J.J.: A review of eye-tracking applications as tools for training. Cogn. Technol. Work 15(3), 313–327 (2013). https://doi.org/10.1007/s10111-012-0234-7

    Article  Google Scholar 

  57. Sadri, F.: Logic-based approaches to intention recognition. In: Handbook of Research on Ambient Intelligence and Smart Environments: Trends and Perspectives, pp. 346–375. IGI Global (2011). https://doi.org/10.4018/978-1-61692-857-5.ch007

  58. Sadri, F., Stathis, K., Toni, F.: Normative KGP agents. Comput. Math. Organ. Theory 12(2–3), 101–126 (2006)

    Article  Google Scholar 

  59. Santiago-Espada, Y., Myer, R.R., Latorella, K.A., Comstock, J.R., Jr.: The multi-attribute task battery II. A user’s guide (MATB-II) software for human performance and workload research (2011)

    Google Scholar 

  60. de Silva, L., Meneguzzi, F., Logan, B.: BDI agent architectures: a survey. In: Bessiere, C. (ed.) Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence, IJCAI 2020, pp. 4914–4921. ijcai.org (2020). https://doi.org/10.24963/ijcai.2020/684

  61. Stanton, N.A., Chambers, P.R., Piggott, J.: Situational awareness and safety. Saf. Sci. 39(3), 189–204 (2001). https://doi.org/10.1016/S0925-7535(01)00010-8

    Article  Google Scholar 

  62. Stathis, K., Kakas, A.C., Lu, W., Demetriou, N., Endriss, U., Bracciali, A.: PROSOCS: a platform for programming software agents in computational logic. In: Müller, J., Petta, P. (eds.) Proceedings of the Fourth International Symposium “From Agent Theory to Agent Implementation” (AT2AI-4 - EMCSR 2004 Session M), Vienna, Austria, pp. 523–528 (2004)

    Google Scholar 

  63. Stratmann, T.C., Kempa, F., Boll, S.: Lame: light-controlled attention guidance for multi-monitor environments. In: Proceedings of the 8th ACM International Symposium on Pervasive Displays, pp. 1–5 (2019). https://doi.org/10.1145/3321335.3324935

  64. Tatler, B.W., Hansen, D.W., Pelz, J.B.: Eye movement recordings in natural settings. In: Klein, C., Ettinger, U. (eds.) Eye Movement Research. SNPBE, pp. 549–592. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-20085-5_13

    Chapter  Google Scholar 

  65. Toreini, P., Morana, S.: Designing attention-aware business intelligence and analytics dashboards. In: Designing the Digital Transformation: DESRIST 2017 Research in Progress Proceedings of the 12th International Conference on Design Science Research in Information Systems and Technology, Karlsruhe, Germany. 30 May–1 June, pp. 64–72. Karlsruher Institut für Technologie (KIT) (2017). https://doi.org/10.5445/IR/1000069452

  66. Van Orden, K.F., Limbert, W., Makeig, S., Jung, T.P.: Eye activity correlates of workload during a visuospatial memory task. Hum. Factors 43(1), 111–121 (2001). https://doi.org/10.1518/001872001775992570

    Article  Google Scholar 

  67. Wang, H., Chignell, M., Ishizuka, M.: Empathic tutoring software agents using real-time eye tracking. In: Proceedings of the 2006 Symposium on Eye Tracking Research & Applications, ETRA 2006, pp. 73–78. Association for Computing Machinery, New York (2006). https://doi.org/10.1145/1117309.1117346

  68. Wickens, C.D., Hollands, J.G., Banbury, S., Parasuraman, R.: Engineering Psychology and Human Performance. Psychology Press, Hove (2015)

    Book  Google Scholar 

Download references

Acknowledgements

The work for developing this demonstrator was supported by a Human-Like Computing EPSRC Network+ Kickstart grant. Thanks to Suzy Broadbent and Lisa Boyce for discussions and support.

Author information

Authors and Affiliations

  1. Department of Psychology Royal Holloway, University of London, Egham, TW20 0EX, UK

    Szonya Durant & Callum Woods

  2. Department of Computer Science Royal Holloway, University of London, Egham, TW20 0EX, UK

    Benedict Wilkins, Emanuele Uliana & Kostas Stathis

Authors
  1. Szonya Durant
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Benedict Wilkins
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Callum Woods
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Emanuele Uliana
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Kostas Stathis
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Szonya Durant .

Editor information

Editors and Affiliations

  1. Utrecht University, Utrecht, The Netherlands

    Natasha Alechina

  2. University of Turin, Turin, Italy

    Matteo Baldoni

  3. Utrecht University, Utrecht, The Netherlands

    Brian Logan

Rights and permissions

Reprints and permissions

Copyright information

© 2022 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Durant, S., Wilkins, B., Woods, C., Uliana, E., Stathis, K. (2022). Attention Guidance Agents with Eye-Tracking. In: Alechina, N., Baldoni, M., Logan, B. (eds) Engineering Multi-Agent Systems. EMAS 2021. Lecture Notes in Computer Science(), vol 13190. Springer, Cham. https://doi.org/10.1007/978-3-030-97457-2_6

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-97457-2_6

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-97456-5

  • Online ISBN: 978-3-030-97457-2

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation