Advertisement

Designing Interactions with Intention-Aware Gaze-Enabled Artificial Agents

  • Joshua NewnEmail author
  • Ronal Singh
  • Fraser Allison
  • Prashan Madumal
  • Eduardo Velloso
  • Frank Vetere
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11747)

Abstract

As it becomes more common for humans to work alongside artificial agents on everyday tasks, it is increasingly important to design artificial agents that can understand and interact with their human counterparts naturally. We posit that an effective way to do this is to harness nonverbal cues used in human-human interaction. We, therefore, leverage knowledge from existing work on gaze-based intention recognition, where the awareness of gaze can provide insights into the future actions of an observed human subject. In this paper, we design and evaluate the use of a proactive intention-aware gaze-enabled artificial agent that assists a human player engaged in an online strategy game. The agent assists by recognising and communicating the intentions of a human opponent in real-time, potentially improving situation awareness. Our first study identifies the language requirements for the artificial agent to communicate the opponent’s intentions to the assisted player, using an inverted Wizard of Oz method approach. Our second study compares the experience of playing an online strategy game with and without the assistance of the agent. Specifically, we conducted a within-subjects study with 30 participants to compare their experience of playing with (1) detailed AI predictions, (2) abstract AI predictions, and (3) no AI predictions but with a live visualisation of their opponent’s gaze. Our results show that the agent can facilitate awareness of another user’s intentions without adding visual distraction to the interface; however, the cognitive workload was similar across all three conditions, suggesting that the manner in which the agent communicates its predictions requires further exploration. Overall, our work contributes to the understanding of how to support human-agent teams in a dynamic collaboration scenario. We provide a positive account of humans interacting with an intention-aware artificial agent afforded by gaze input, which presents immediate opportunities for improving interactions between the counterparts.

Keywords

Human-AI interaction Intention recognition Explainable interface Human-AI teaming Intention awareness Eye tracking Gaze awareness 

Notes

Acknowledgements

We acknowledge the Australian Commonwealth Government, Microsoft Research Centre for Social Natural User Interfaces and Interaction Design Lab for their support on this project. We extend our gratitude to our colleagues, particularly Vassilis Kostakos, Tilman Dingler and Ryan Kelly for their input on the paper. Dr. Eduardo Velloso is the recipient of an Australian Research Council Discovery Early Career Award (Project Number: DE180100315) funded by the Australian Commonwealth Government.

References

  1. 1.
    Akkil, D., Dey, P., Salian, D., Rajput, N.: Gaze awareness in agent-based early-childhood learning application. In: Bernhaupt, R., Dalvi, G., Joshi, A., Balkrishan, D.K., O’Neill, J., Winckler, M. (eds.) INTERACT 2017. LNCS, vol. 10514, pp. 447–466. Springer, Cham (2017).  https://doi.org/10.1007/978-3-319-67684-5_28CrossRefGoogle Scholar
  2. 2.
    Akkil, D., Thankachan, B., Isokoski, P.: I see what you see: gaze awareness in mobile video collaboration. In: Proceedings of the 2018 ACM Symposium on Eye Tracking Research & Applications, ETRA 2018, pp. 32:1–32:9. ACM, New York (2018).  https://doi.org/10.1145/3204493.3204542
  3. 3.
    Antifakos, S., Schwaninger, A., Schiele, B.: Evaluating the effects of displaying uncertainty in context-aware applications. In: Davies, N., Mynatt, E.D., Siio, I. (eds.) UbiComp 2004. LNCS, vol. 3205, pp. 54–69. Springer, Heidelberg (2004).  https://doi.org/10.1007/978-3-540-30119-6_4CrossRefGoogle Scholar
  4. 4.
    Argyle, M., Cook, M.: Gaze and Mutual Gaze. Cambridge University Press, Cambridge (1976)Google Scholar
  5. 5.
    Bednarik, R., Eivazi, S., Vrzakova, H.: Eye gaze in intelligent user interfaces. In: Nakano, Y., Conati, C., Bader, T. (eds.) A Computational Approach for Prediction of Problem-Solving Behavior Using Support Vector Machines and Eye-Tracking Data, pp. 111–134. Springer, London (2013).  https://doi.org/10.1007/978-1-4471-4784-8_7CrossRefGoogle Scholar
  6. 6.
    Biran, O., Cotton, C.: Explanation and justification in machine learning: a survey. In: IJCAI 2017 Workshop on Explainable AI (XAI), p. 8 (2017)Google Scholar
  7. 7.
    Brewer, J., D’Angelo, S., Gergle, D.: Iris: gaze visualization design made easy. In: Extended Abstracts of the 2018 CHI Conference on Human Factors in Computing Systems, CHI EA 2018, pp. D504:1–D504:4. ACM, New York (2018).  https://doi.org/10.1145/3170427.3186502
  8. 8.
    Brockmyer, J.H., Fox, C.M., Curtiss, K.A., McBroom, E., Burkhart, K.M., Pidruzny, J.N.: The development of the game engagement questionnaire: a measure of engagement in video game-playing. J. Exp. Soc. Psychol. 45(4), 624–634 (2009).  https://doi.org/10.1016/j.jesp.2009.02.016CrossRefGoogle Scholar
  9. 9.
    Buettner, R.: Cognitive workload of humans using artificial intelligence systems: towards objective measurement applying eye-tracking technology. In: Timm, I.J., Thimm, M. (eds.) KI 2013. LNCS (LNAI), vol. 8077, pp. 37–48. Springer, Heidelberg (2013).  https://doi.org/10.1007/978-3-642-40942-4_4CrossRefGoogle Scholar
  10. 10.
    Chen, J., Barnes, M.: Human-agent teaming for multirobot control: a review of human factors issues. IEEE Trans. Hum. Mach. Syst. 44(1), 13–29 (2014).  https://doi.org/10.1109/THMS.2013.2293535CrossRefGoogle Scholar
  11. 11.
    Chen, J.Y.C., Lakhmani, S.G., Stowers, K., Selkowitz, A.R., Wright, J.L., Barnes, M.: Situation awareness-based agent transparency and human-autonomy teaming effectiveness. Theor. Issues Ergon. Sci. 19(3), 259–282 (2018).  https://doi.org/10.1080/1463922X.2017.1315750CrossRefGoogle Scholar
  12. 12.
    D’Angelo, S., Begel, A.: Improving communication between pair programmers using shared gaze awareness. In: Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems, CHI 2017, pp. 6245–6290. ACM, New York (2017).  https://doi.org/10.1145/3025453.3025573
  13. 13.
    D’Angelo, S., Brewer, J., Gergle, D.: Iris: a tool for designing contextually relevant gaze visualizations. In: Proceedings of the 11th ACM Symposium on Eye Tracking Research & Applications, ETRA 2019, pp. 79:1–79:5. ACM, New York (2019).  https://doi.org/10.1145/3317958.3318228
  14. 14.
    D’Angelo, S., Gergle, D.: Gazed and confused: understanding and designing shared gaze for remote collaboration. In: Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems, CHI 2016, pp. 2492–2496. ACM, New York (2016).  https://doi.org/10.1145/2858036.2858499
  15. 15.
    D’Angelo, S., Gergle, D.: An eye for design: gaze visualizations for remote collaborative work. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems, CHI 2018, pp. 349:1–349:12. ACM, New York (2018).  https://doi.org/10.1145/3173574.3173923
  16. 16.
    Dodge, J., Penney, S., Hilderbrand, C., Anderson, A., Burnett, M.: How the experts do it: assessing and explaining agent behaviors in real-time strategy games. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems, CHI 2018, pp. 562:1–562:12. ACM, New York (2018).  https://doi.org/10.1145/3173574.3174136
  17. 17.
    Duchowski, A.T.: Gaze-based interaction: a 30 year retrospective. Comput. Graph. 73, 59–69 (2018).  https://doi.org/10.1016/j.cag.2018.04.002CrossRefGoogle Scholar
  18. 18.
    Duchowski, A.T., et al.: The index of pupillary activity: measuring cognitive load vis-à-vis task difficulty with pupil oscillation. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems, CHI 2018, pp. 282:1–282:13. ACM, New York (2018).  https://doi.org/10.1145/3173574.3173856
  19. 19.
    Eiband, M., Schneider, H., Bilandzic, M., Fazekas-Con, J., Haug, M., Hussmann, H.: Bringing transparency design into practice. In: 23rd International Conference on Intelligent User Interfaces, IUI 2018, pp. 211–223. ACM, New York (2018).  https://doi.org/10.1145/3172944.3172961
  20. 20.
    Endsley, M.R.: Toward a theory of situation awareness in dynamic systems. Hum. Factors J. Hum. Factors Ergon. Soc. 37(1), 32–64 (1995).  https://doi.org/10.1518/001872095779049543CrossRefGoogle Scholar
  21. 21.
    Harbers, M., van den Bosch, K., Meyer, J.-J.C.: A study into preferred explanations of virtual agent behavior. In: Ruttkay, Z., Kipp, M., Nijholt, A., Vilhjálmsson, H.H. (eds.) IVA 2009. LNCS (LNAI), vol. 5773, pp. 132–145. Springer, Heidelberg (2009).  https://doi.org/10.1007/978-3-642-04380-2_17CrossRefGoogle Scholar
  22. 22.
    Hart, S.G., Staveland, L.E.: Development of NASA-TLX (task load index): results of empirical and theoretical research. In: Hancock, P.A., Meshkati, N. (eds.) Human Mental Workload, Advances in Psychology, vol. 52, pp. 139–183. North-Holland (1988).  https://doi.org/10.1016/S0166-4115(08)62386-9Google Scholar
  23. 23.
    Hayes, B., Shah, J.A.: Improving robot controller transparency through autonomous policy explanation. In: Proceedings of the 2017 ACM/IEEE International Conference on Human-Robot Interaction, HRI 2017, pp. 303–312. ACM, New York (2017).  https://doi.org/10.1145/2909824.3020233
  24. 24.
    Higuch, K., Yonetani, R., Sato, Y.: Can eye help you? Effects of visualizing eye fixations on remote collaboration scenarios for physical tasks. In: Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems, CHI 2016, pp. 5180–5190. ACM, New York (2016).  https://doi.org/10.1145/2858036.2858438
  25. 25.
    Hoffman, R.R., Mueller, S.T., Klein, G., Litman, J.: Metrics for explainable AI: challenges and prospects. arXiv preprint arXiv:1812.04608 (2018)
  26. 26.
    Huang, C.M., Andrist, S., Sauppé, A., Mutlu, B.: Using gaze patterns to predict task intent in collaboration. Front. Psychol. 6, 1049 (2015).  https://doi.org/10.3389/fpsyg.2015.01049CrossRefGoogle Scholar
  27. 27.
    Huang, C.M., Mutlu, B.: Anticipatory robot control for efficient human-robot collaboration. In: The Eleventh ACM/IEEE International Conference on Human Robot Interaction, HRI 2016, pp. 83–90. IEEE Press, Piscataway (2016). http://dl.acm.org/citation.cfm?id=2906831.2906846
  28. 28.
    Jacob, R.J.K.: What you look at is what you get: eye movement-based interaction techniques. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, CHI 1990, pp. 11–18. ACM, New York (1990).  https://doi.org/10.1145/97243.97246
  29. 29.
    Kahneman, D., Egan, P.: Thinking, Fast and Slow, vol. 1. Farrar Straus and Giroux, New York (2011)Google Scholar
  30. 30.
    Kass, R., Finin, T.: The need for user models in generating expert system explanation. Int. J. Expert Syst. 1(4), 345–375 (1988). http://dl.acm.org/citation.cfm?id=58447.58452Google Scholar
  31. 31.
    Keil, F.C.: Explanation and understanding. Annu. Rev. Psychol. 57(1), 227–254 (2006).  https://doi.org/10.1146/annurev.psych.57.102904.190100CrossRefGoogle Scholar
  32. 32.
    Kelley, J.F.: An iterative design methodology for user-friendly natural language office information applications. ACM Trans. Inf. Syst. 2(1), 26–41 (1984)CrossRefGoogle Scholar
  33. 33.
    Klien, G., Woods, D.D., Bradshaw, J.M., Hoffman, R.R., Feltovich, P.J.: Ten challenges for making automation a “team player” in joint human-agent activity. IEEE Intell. Syst. 19(6), 91–95 (2004).  https://doi.org/10.1109/MIS.2004.74CrossRefGoogle Scholar
  34. 34.
    Land, M.F.: Vision, eye movements, and natural behavior. Vis. Neurosci. 26(1), 51–62 (2009).  https://doi.org/10.1017/S0952523808080899CrossRefGoogle Scholar
  35. 35.
    Langley, P., Meadows, B., Sridharan, M., Choi, D.: Explainable agency for intelligent autonomous systems. In: Twenty-Ninth IAAI Conference (2017)Google Scholar
  36. 36.
    Lankes, M., Maurer, B., Stiglbauer, B.: An eye for an eye: gaze input in competitive online games and its effects on social presence. In: Proceedings of the 13th International Conference on Advances in Computer Entertainment Technology, ACE 2016, pp. 17:1–17:9. ACM, New York (2016).  https://doi.org/10.1145/3001773.3001774
  37. 37.
    Lesh, N., Marks, J., Rich, C., Sidner, C.L.: Man-computer symbiosis revisited: achieving natural communication and collaboration with computers. IEICE Trans. 87(6), 1290–1298 (2004)Google Scholar
  38. 38.
    Licklider, J.C.R.: Man-computer symbiosis. IRE Trans. Hum. Factors Electron. HFE-1 1, 4–11 (1960).  https://doi.org/10.1109/THFE2.1960.4503259CrossRefGoogle Scholar
  39. 39.
    Lim, B.Y., Dey, A.K.: Investigating intelligibility for uncertain context-aware applications. In: Proceedings of the 13th International Conference on Ubiquitous Computing, UbiComp 2011, pp. 415–424. ACM, New York (2011).  https://doi.org/10.1145/2030112.2030168
  40. 40.
    Madumal, P.: Explainable agency in intelligent agents: doctoral consortium. In: Proceedings of the 18th International Conference on Autonomous Agents and MultiAgent Systems, AAMAS 2019, pp. 2432–2434. International Foundation for Autonomous Agents and Multiagent Systems, Richland (2019). http://dl.acm.org/citation.cfm?id=3306127.3332137
  41. 41.
    Madumal, P., Miller, T., Sonenberg, L., Vetere, F.: Explainable reinforcement learning through a causal lens. arXiv preprint arXiv:1905.10958 (2019)
  42. 42.
    Madumal, P., Miller, T., Sonenberg, L., Vetere, F.: A grounded interaction protocol for explainable artificial intelligence. In: Proceedings of the 18th International Conference on Autonomous Agents and MultiAgent Systems, AAMAS 2019, pp. 1033–1041. International Foundation for Autonomous Agents and Multiagent Systems, Richland (2019). http://dl.acm.org/citation.cfm?id=3306127.3331801
  43. 43.
    Majaranta, P., Bulling, A.: Eye Tracking and eye-based human-computer interaction. In: Fairclough, S.H., Gilleade, K. (eds.) Advances in Physiological Computing. HIS, pp. 39–65. Springer, London (2014).  https://doi.org/10.1007/978-1-4471-6392-3_3CrossRefGoogle Scholar
  44. 44.
    Malle, B.F., Knobe, J.: Which behaviors do people explain? A basic actor-observer asymmetry. J. Pers. Soc. Psychol. 72(2), 288 (1997)CrossRefGoogle Scholar
  45. 45.
    Maurer, B., Lankes, M., Stiglbauer, B., Tscheligi, M.: EyeCo: effects of shared gaze on social presence in an online cooperative game. In: Wallner, G., Kriglstein, S., Hlavacs, H., Malaka, R., Lugmayr, A., Yang, H.-S. (eds.) ICEC 2016. LNCS, vol. 9926, pp. 102–114. Springer, Cham (2016).  https://doi.org/10.1007/978-3-319-46100-7_9CrossRefGoogle Scholar
  46. 46.
    Maurer, B., Lankes, M., Tscheligi, M.: Where the eyes meet: lessons learned from shared gaze-based interactions in cooperative and competitive online games. Entertainment Comput. 27, 47–59 (2018).  https://doi.org/10.1016/j.entcom.2018.02.009, http://www.sciencedirect.com/science/article/pii/S1875952117300629CrossRefGoogle Scholar
  47. 47.
    Miller, T.: Explanation in artificial intelligence: insights from the social sciences. Artif. Intell. 267, 1–38 (2019).  https://doi.org/10.1016/j.artint.2018.07.007MathSciNetCrossRefGoogle Scholar
  48. 48.
    Moore, J.D., Paris, C.L.: Requirements for an expert system explanation facility. Comput. Intell. 7(4), 367–370 (1991)CrossRefGoogle Scholar
  49. 49.
    Narayanan, V., Zhang, Y., Mendoza, N., Kambhampati, S.: Automated planning for peer-to-peer teaming and its evaluation in remote human-robot interaction. In: Proceedings of the Tenth Annual ACM/IEEE International Conference on Human-Robot Interaction Extended Abstracts, HRI 2015 Extended Abstracts, pp. 161–162. ACM, New York (2015).  https://doi.org/10.1145/2701973.2702042
  50. 50.
    Newn, J., Allison, F., Velloso, E., Vetere, F.: Looks can be deceiving: Using gaze visualisation to predict and mislead opponents in strategic gameplay. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems, CHI 2018, pp. 261:1–261:12. ACM, New York (2018).  https://doi.org/10.1145/3173574.3173835
  51. 51.
    Newn, J., Velloso, E., Allison, F., Abdelrahman, Y., Vetere, F.: Evaluating real-time gaze representations to infer intentions in competitive turn-based strategy games. In: Proceedings of the Annual Symposium on Computer-Human Interaction in Play, CHI PLAY 2017, pp. 541–552. ACM, New York (2017).  https://doi.org/10.1145/3116595.3116624
  52. 52.
    Nunes, I., Jannach, D.: A systematic review and taxonomy of explanations in decision support and recommender systems. User Model. User-Adap. Inter. 27(3–5), 393–444 (2017).  https://doi.org/10.1007/s11257-017-9195-0CrossRefGoogle Scholar
  53. 53.
    Qvarfordt, P., Zhai, S.: Conversing with the user based on eye-gaze patterns. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, CHI 2005, pp. 221–230. ACM, New York (2005).  https://doi.org/10.1145/1054972.1055004
  54. 54.
    Riek, L.D.: Wizard of Oz studies in HRI: a systematic review and new reporting guidelines. J. Hum.-Robot Interact. 1(1), 119–136 (2012).  https://doi.org/10.5898/JHRI.1.1.RiekCrossRefGoogle Scholar
  55. 55.
    Rieser, V.: Bootstrapping reinforcement learning-based dialogue strategies from wizard-of-Oz data. DFKI (2008)Google Scholar
  56. 56.
    Singh, R., Miller, T., Newn, J., Sonenberg, L., Velloso, E., Vetere, F.: Combining planning with gaze for online human intention recognition. In: Proceedings of the 17th International Conference on Autonomous Agents and Multiagent System, AAMAS 2018. International Foundation for Autonomous Agents and Multiagent Systems, Richland (2018). http://dl.acm.org/citation.cfm?id=3237383.3237457
  57. 57.
    Stein, R., Brennan, S.E.: Another person’s eye gaze as a cue in solving programming problems. In: Proceedings of the 6th International Conference on Multimodal Interfaces, ICMI 2004, pp. 9–15. ACM, New York (2004).  https://doi.org/10.1145/1027933.1027936
  58. 58.
    Unhelkar, V.V., et al.: Human-aware robotic assistant for collaborative assembly: integrating human motion prediction with planning in time. IEEE Robot. Autom. Lett. 3(3), 2394–2401 (2018).  https://doi.org/10.1109/LRA.2018.2812906CrossRefGoogle Scholar
  59. 59.
    Velloso, E., Carter, M.: The emergence of eyeplay: a survey of eye interaction in games. In: Proceedings of the 2016 Annual Symposium on Computer-Human Interaction in Play, CHI PLAY 2016, pp. 171–185. ACM, New York (2016).  https://doi.org/10.1145/2967934.2968084
  60. 60.
    Vertegaal, R.: The gaze groupware system: mediating joint attention in multiparty communication and collaboration. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, CHI 1999, pp. 294–301. ACM, New York (1999).  https://doi.org/10.1145/302979.303065
  61. 61.
    Wang, D., Yang, Q., Abdul, A., Lim, B.Y.: Designing theory-driven user-centric explainable AI. In: Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems, CHI 2019, pp. 601:1–601:15. ACM, New York (2019)Google Scholar
  62. 62.
    Wang, N., Pynadath, D.V., Hill, S.G.: Trust calibration within a human-robot team: comparing automatically generated explanations. In: The Eleventh ACM/IEEE International Conference on Human Robot Interaction, pp. 109–116. IEEE Press (2016)Google Scholar
  63. 63.
    Zhang, Y., Pfeuffer, K., Chong, M.K., Alexander, J., Bulling, A., Gellersen, H.: Look together: using gaze for assisting co-located collaborative search. Pers. Ubiquit. Comput. 21(1), 173–186 (2017).  https://doi.org/10.1007/s00779-016-0969-xCrossRefGoogle Scholar

Copyright information

© IFIP International Federation for Information Processing 2019

Authors and Affiliations

  1. 1.School of Computing and Information SystemsThe University of MelbourneMelbourneAustralia

Personalised recommendations