Using Human Eye Gaze Patterns as Indicators of Need for Assistance from a Socially Assistive Robot

  • Ulyana KuryloEmail author
  • Jason R. Wilson
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11876)


With current growth in social robotics comes a need for well developed and fine tuned agents which respond to the user in a seamless and intuitive manner. Socially assistive robots in particular have become popular for their uses in care for older adults for medication adherence and socializing. Since eye gaze cues are important mediators in human-human interactions, we hypothesize that gaze patterns can be applied to human-robot interactions to identify when the user may need assistance. We reviewed videos (N = 16) of robot supported collaborative work to explore how recognition of gaze patterns for an assistive robot in the context of a medication management task can help predict when a user needs assistance. We found that mutual gaze is a better predictor than confirmatory request, gaze away, and goal reference. While eye gaze serves as an important indicator for need for assistance, it should be combined with other indicators, such as verbal cues or facial expressions to sufficiently represent assistance needed in the interaction and provide timely assistance.


Gaze detection Gaze patterns Assistive agents 


  1. 1.
    Admoni, H., Scassellati, B.: Social eye gaze in human-robot interaction: a review. J. Hum.-Robot Interact. 6(1), 25–63 (2017)CrossRefGoogle Scholar
  2. 2.
    Andrist, S., Gleicher, M., Mutlu, B.: Looking coordinated: bidirectional gaze mechanisms for collaborative interaction with virtual characters. In: Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems, pp. 2571–2582. ACM (2017)Google Scholar
  3. 3.
    Andrist, S., Tan, X.Z., Gleicher, M., Mutlu, B.: Conversational gaze aversion for humanlike robots. In: Proceedings of the 2014 ACM/IEEE International Conference on Human-Robot Interaction, pp. 25–32. ACM (2014)Google Scholar
  4. 4.
    Bruce, A., Nourbakhsh, I., Simmons, R.: The role of expressiveness and attention in human-robot interaction. In: Proceedings 2002 IEEE International Conference on Robotics and Automation (Cat. No. 02CH37292), vol. 4, pp. 4138–4142. IEEE (2002)Google Scholar
  5. 5.
    Casas, J., et al.: Architecture for a social assistive robot in cardiac rehabilitation. In: 2018 IEEE 2nd Colombian Conference on Robotics and Automation (CCRA), pp. 1–6. IEEE (2018)Google Scholar
  6. 6.
    Fang, R., Doering, M., Chai, J.Y.: Embodied collaborative referring expression generation in situated human-robot interaction. In: 2015 10th ACM/IEEE International Conference on Human-Robot Interaction (HRI), pp. 271–278. IEEE (2015)Google Scholar
  7. 7.
    Feil-Seifer, D., Matarić, M.J.: Defining socially assistive robotics. In: 9th International Conference on Rehabilitation Robotics, ICORR 2005, pp. 465–468 (2005)Google Scholar
  8. 8.
    Grigore, E.C., Eder, K., Pipe, A.G., Melhuish, C., Leonards, U.: Joint action understanding improves robot-to-human object handover. In: 2013 IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 4622–4629. IEEE (2013)Google Scholar
  9. 9.
    Huang, C.M., Andrist, S., Sauppé, A., Mutlu, B.: Using gaze patterns to predict task intent in collaboration. Front. Psychol. 6, 1049 (2015)Google Scholar
  10. 10.
    Jeste, D.V., Depp, C.A.: Positive mental aging. Am. J. Geriatr. Psychiatry 18(1), 1–3 (2010)CrossRefGoogle Scholar
  11. 11.
    Jokinen, K.: Gaze and gesture activity in communication. In: Stephanidis, C. (ed.) UAHCI 2009. LNCS, vol. 5615, pp. 537–546. Springer, Heidelberg (2009). Scholar
  12. 12.
    Kendon, A., Cook, M.: The consistency of gaze patterns in social interaction. Br. J. Psychol. 60(4), 481–494 (1969)CrossRefGoogle Scholar
  13. 13.
    Kleinke, C.L.: Gaze and eye contact: a research review. Psychol. Bull. 100(1), 78 (1986)CrossRefGoogle Scholar
  14. 14.
    Matarić, M.J., Eriksson, J., Feil-Seifer, D.J., Winstein, C.J.: Socially assistive robotics for post-stroke rehabilitation. J. NeuroEng. Rehabil. 4(5), 1–9 (2007)Google Scholar
  15. 15.
    Mehlmann, G., Häring, M., Janowski, K., Baur, T., Gebhard, P., André, E.: Exploring a model of gaze for grounding in multimodal HRI. In: Proceedings of the 16th International Conference on Multimodal Interaction, pp. 247–254. ACM (2014)Google Scholar
  16. 16.
    Pennisi, P., et al.: Autism and social robotics: a systematic review. Autism Res. 9(2), 165–183 (2016)CrossRefGoogle Scholar
  17. 17.
    Satake, S., Kanda, T., Glas, D.F., Imai, M., Ishiguro, H., Hagita, N.: How to approach humans?: strategies for social robots to initiate interaction. In: Proceedings of the 4th ACM/IEEE International Conference on Human Robot Interaction, HRI 2009, pp. 109–116. ACM, New York (2009)Google Scholar
  18. 18.
    Schneider, B., Pea, R.: Real-time mutual gaze perception enhances collaborative learning and collaboration quality. Int. J. Comput.-Supported Collab. Learn. 8(4), 375–397 (2013)CrossRefGoogle Scholar
  19. 19.
    Tapus, A., Maja, M., Scassellatti, B.: The grand challenges in socially assistive robotics. IEEE Robot. Autom. Mag. 14(1), 35–42 (2007)CrossRefGoogle Scholar
  20. 20.
    Wada, K., Shibata, T.: Robot therapy in a care house - change of relationship among the residents and seal robot during a 2-month long study. IEEE Trans. Robot. 23(5), 972–980 (2007)CrossRefGoogle Scholar
  21. 21.
    Wilson, J.R., Lee, N.Y., Saechao, A., Tickle-Degnen, L., Scheutz, M.: Supporting human autonomy in a robot-assisted medication sorting task. Int. J. Soc. Robot. 10(5), 621–641 (2018)CrossRefGoogle Scholar
  22. 22.
    Wilson, J.R., Wransky, M., Tierno, J.: General approach to automatically generating need-based assistance. In: Proceeding of Advances in Cognitive Systems (2018))Google Scholar
  23. 23.
    Yamazaki, K., et al.: Prior-to-request and request behaviors within elderly day care: implications for developing service robots for use in multiparty settings. In: Bannon, L.J., Wagner, I., Gutwin, C., Harper, R.H.R., Schmidt, K. (eds.) ECSCW 2007, pp. 61–78. Springer, London (2007). Scholar

Copyright information

© Springer Nature Switzerland AG 2019

Authors and Affiliations

  1. 1.Northwestern UniversityEvanstonUSA

Personalised recommendations