Skip to main content
SpringerLink
Log in

Towards Social Companions in Augmented Reality: Vision and Challenges

  • Conference paper
  • First Online:
Distributed, Ambient and Pervasive Interactions. Smart Living, Learning, Well-being and Health, Art and Creativity (HCII 2022)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 13326))

Included in the following conference series:

Abstract

In future omnipresent Augmented Reality (AR), virtual creatures (humans and pets) will play an important role, not so much as task-oriented individuals, but as friends and companions with whom we can walk and chat when we want to. A companion can also suggest physical activity, motivate healthy eating behavior, or help in reminding taking medication. We survey existing approaches to introduce virtual humans and pets in AR. We indicate shortcomings and mention research approaches to deal with them. We conclude with a few observations on some aspects of AR-specific situational awareness of (multisensorial) virtual humans and pets that must reflect in their behavior to be believable.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 69.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 89.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

Notes

  1. 1.

    https://www.youtube.com/watch?v=-PzeWxtOGzQ.

  2. 2.

    https://www.magicleap.com/en-us/news/op-ed/i-am-mica.

  3. 3.

    In a Wizard of Oz experiment subjects believe that they interact with a computer system when in reality the interaction is mainly done with an invisible experimenter. In this way it can be examined how people will act in a setting that has not yet been realized with a computer.

References

  1. Azuma, R.T.: The road to ubiquitous consumer augmented reality systems. Hum. Behav. Emerg. Technol. 1, 26–32 (2019)

    Article  Google Scholar 

  2. Abrash, M.: Project ARIA. In: Facebook Connect Keynote: Oculus Quest 2, Project Aria and More! 1.33.20-1.47.50 (2020). https://www.youtube.com/watch?v=woXmJMw2lTM

  3. Grubert, J., Langlotz, T., Zollmann, S., Regenbrecht, H.: Towards pervasive augmented reality: context-awareness in augmented reality. IEEE Trans. Vis. Comput. Graph. 23(6), 1706–1724 (2017)

    Article  Google Scholar 

  4. Schraffenberger, H., van der Heide, E.: Multimodal augmented reality: the norm rather than the exception. In: Proceedings of the 2016 Workshop on Multimodal Virtual and Augmented Reality (MVAR 2016), pp. 1–6. ACM, New York, NY, USA (2016)

    Google Scholar 

  5. Kerruish, E.: Arranging sensations: smell and taste in augmented and virtual reality. Sens. Soc. 14(1), 31–45 (2019)

    Article  Google Scholar 

  6. Nijholt, A.: Social augmented reality interactions. In: Proceedings 10th International IEEE Conference on Informatics, Electronics and Vision (ICIEV 2021), pp. 1–8. IEEE, New York (2021)

    Google Scholar 

  7. Burden, D., Savin-Baden, M.: Virtual Humans: Today and Tomorrow. Taylor & Francis/CRC Press, Boca Raton (2019)

    Book  Google Scholar 

  8. Vardoulakis, L.P., Ring, L., Barry, B., Sidner, C.L., Bickmore, T.: Designing relational agents as long term social companions for older adults. In: Nakano, Y., Neff, M., Paiva, A., Walker, M. (eds.) IVA 2012. LNCS (LNAI), vol. 7502, pp. 289–302. Springer, Heidelberg (2012). https://doi.org/10.1007/978-3-642-33197-8_30

  9. Grolleman, J., van Dijk, B., Nijholt, A., van Emst, A.: Break the habit! Designing an e-therapy intervention using a virtual coach in aid of smoking cessation. In: IJsselsteijn, W.A., de Kort, Y.A.W., Midden, C., Eggen, B., van den Hoven, E. (eds.) PERSUASIVE 2006. LNCS, vol. 3962, pp. 133–141. Springer, Heidelberg (2006). https://doi.org/10.1007/11755494_19

  10. Hospers, M., Kroezen, E., Nijholt, A., op den Akker, R., Heylen, D.: An agent-based intelligent tutoring system for nurse education. In: Nealon, J., Moreno, A. (eds.) Applications of Software Agent Technology in the Health Care Domain, pp. 143–159. Whitestein Series in Software Agent Technologies, Birkhauser Publishing Ltd., Basel, Switzerland (2003)

    Google Scholar 

  11. op den Akker, H.J.A., Klaassen, R., Nijholt, A.: Virtual coaches for healthy lifestyle. In: Esposito, A., Jain, L.C. (eds.) Toward Robotic Socially Believable Behaving Systems - Volume II. ISRL, vol. 106, pp. 121–149. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-31053-4_8

  12. Johnsen, K., et al.: Mixed reality virtual pets to reduce childhood obesity. IEEE Trans. Visual Comput. Graphics 20(4), 523–530 (2014)

    Article  Google Scholar 

  13. Loizides, F., et al.: MyCompanion: a digital social companion for assisted living. In: Lamas, D., Loizides, F., Nacke, L., Petrie, H., Winckler, M., Zaphiris, P. (eds.) INTERACT 2019. LNCS, vol. 11749, pp. 649–653. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-29390-1_55

  14. Heylen, D., van Dijk, B., Nijholt, A.: Robotic rabbit companions: amusing or a nuisance? J. Multimodal User Interf. 5, 53–59 (2012)

    Google Scholar 

  15. Nijholt, A.: Augmented reality humans: towards multisensorial awareness. In: Jallouli, R., Bach Tobji, M.A., Mcheick, H., Piho, G. (eds.) ICDEc 2021. LNBIP, vol. 431, pp. 237–250. Springer, Cham (2021). https://doi.org/10.1007/978-3-030-92909-1_16

  16. Torre, R., Fua, P., Balcisoy, S., Ponder, M., Thalmann, D.: Interaction between real and virtual humans: playing checkers. In: Mulder, J.D., van Liere, R. (eds.) 6th Eurographics Workshop on Virtual Environments, pp. 23–32 (2000)

    Google Scholar 

  17. Azuma, R. T.: A survey of augmented reality. Pres. Teleoper. Virtual Environ. 6(4), 355–385 (1997). https://doi.org/10.1162/pres.1997.6.4.355

    Article  Google Scholar 

  18. Wooldridge, M., Jennings, N.: Intelligent agents: theory and practice. The Knowl. Eng. Rev. 10(2), 115–152 (1995)

    Article  Google Scholar 

  19. Cassell, J., Sullivan, J., Prevost, S., Churchill, E. (eds.): Embodied Conversational Agents. The MIT Press, Cambridge (2000)

    Google Scholar 

  20. Norouzi, N., et al.: A systematic survey of 15 years of user studies published in the intelligent virtual agents conference. In: Proceedings of the 18th International Conference on Intelligent Virtual Agents (IVA 2018), pp. 17–22. ACM, New York, USA (2018)

    Google Scholar 

  21. Swartout, W., et al.: Toward virtual humans. AI Mag. 27(2), 96–108 (2006)

    Google Scholar 

  22. Maes, P., Darrell, T., Blumberg, B., Pentland, A.: The ALIVE system: full-body interaction with autonomous agents. In: Proceedings Computer Animation 1995, pp. 11–18. Geneva, Switzerland (1995)

    Google Scholar 

  23. Blumberg, B.M.: Old Tricks, New Dogs: Ethology and Interactive Creatures. Ph.D. Thesis, Massachusetts Institute of Technology (1997)

    Google Scholar 

  24. Lee, J.Y., Rhee, G.W., Dong Woo Seo, D.W., Kim, N.K.: Ubiquitous home simulation using augmented reality. In: Proceedings of the 2007 annual Conference on International Conference on Computer Engineering and Applications (CEA 2007), pp. 112–116. World Scientific and Engineering Academy and Society (WSEAS), Stevens Point, Wisconsin, USA (2007)

    Google Scholar 

  25. Barakonyi, I., Schmalstieg, D.: Augmented reality agents for user interface adaptation. Comput. Anim. Virtual Worlds 19(1), 23–35 (2008)

    Article  Google Scholar 

  26. Hartholt, A., et al.: All together now. In: Aylett, R., Krenn, Be., Pelachaud, C., Shimodaira, Hi. (eds.) IVA 2013. LNCS (LNAI), vol. 8108, pp. 368–381. Springer, Heidelberg (2013). https://doi.org/10.1007/978-3-642-40415-3_33

    Chapter  Google Scholar 

  27. Hartholt, A., Fast, E., Reilly, A., Whitcup, W., Liewer, M., Mozgai, S.: Ubiquitous virtual humans: a multi-platform framework for embodied AI agents in XR. In: IEEE International Conference on Artificial Intelligence and Virtual Reality (AIVR), pp. 308–3084 (2019)

    Google Scholar 

  28. Dobiesz, W.S., Grajper, A.: Animating the static. case study of the project “Urbanimals”. In: Herneoja, A., et al. (eds.) Proceedings of the 34th International Conference on Education and Research in Computer Aided Architectural Design in Europe (eCAADe 34), vol. 1, pp. 691–697. Oulu School of Architecture, University of Oulu, Finland, Oulu (2016)

    Google Scholar 

  29. Grand Visual. Unbelievable Bus Shelter. https://grandvisual.com/work/pepsi-max-bus-shelter/

  30. Scholz, J., Smith, A.N.: Augmented reality: Designing immersive experiences that maximize consumer engagement. Bus. Horiz. 59(2), 149–161 (2016)

    Article  Google Scholar 

  31. Nijholt, A.: Humorous and playful social interactions in augmented reality. In: Contribuciones a la Lingüística y a la Comunicación Social. Tributo a Vitelio Ruiz Hernández, pp. 167–173. Centro de Lingüística Aplicada, Santiago de Cuba, Cuba (2021)

    Google Scholar 

  32. Dow, S., Mehta, M., Harmon, E., MacIntyre, B., Mateas, M.: Presence and engagement in an interactive drama. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pp. 1475–1484. ACM, New York, USA (2007)

    Google Scholar 

  33. Anabuki, M., Kakuta, H., Yamamoto, H., Tamura, H.: Welbo: an embodied conversational agent living in mixed reality space. In: CHI 2000 Extended Abstracts on Human Factors in Computing Systems (CHI EA 2000), pp. 10–11. ACM, New York, USA (2000)

    Google Scholar 

  34. Schmeil, A., Broll, W.: MARA - a mobile augmented reality-based virtual assistant. In: 2007 IEEE Virtual Reality Conference, pp. 267–270. Charlotte, NC, USA (2007)

    Google Scholar 

  35. Bancroft, J., Bin Zafar, N., Comer, S., Kuribayashi, T., Litt, J., Miller, T.: Mica: a photoreal character for spatial computing. In: ACM SIGGRAPH 2019 Talks (SIGGRAPH 2019), Article 9, pp. 1–2. New York, Association for Computing Machinery, New York, USA (2019)

    Google Scholar 

  36. Hartholt, A., et al.: Virtual humans in augmented reality: a first step towards real-world embedded virtual roleplayers. In: Proceedings of the 7th International Conference on Human-Agent Interaction (HAI 2019), pp. 205–207. ACM, New York, NY, USA (2019)

    Google Scholar 

  37. Wang, I., Smith, J., Ruiz, J.: Exploring virtual agents for augmented reality. In: Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (CHI 2019), pp. 1–12. Association for Computing Machinery, New York, NY, USA (2019)

    Google Scholar 

  38. Mostajeran, F., Steinicke, F., Nunez, O.J.A., Gatsios, D., Fotiadis, D. Augmented reality for older adults: exploring acceptability of virtual coaches for home-based balance training in an aging population. In: Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems, pp. 1–12. Association for Computing Machinery, New York, NY, USA, (2020)

    Google Scholar 

  39. Norouzi, N., et al.: Walking your virtual dog: analysis of awareness and proxemics with simulated support animals in augmented reality. In: Proceedings of the IEEE International Symposium on Mixed and Augmented Reality (ISMAR), pp. 253–264 (2019)

    Google Scholar 

  40. Norouzi, N., Bruder, G., Bailenson, J., Welch, G.: Investigating augmented reality animals as companions. In: IEEE International Symposium on Mixed and Augmented Reality Adjunct (ISMAR-Adjunct), pp. 400–403. Beijing, China (2019)

    Google Scholar 

  41. Wilks, Y.: On being a victorian companion. In: Close Engagements with Artificial Companions. Key Social, Psychological, Ethical and Design Issues, pp. 121–128. John Benjamins Publishing Company, Amsterdam, The Netherlands (2010).

    Google Scholar 

  42. Liang, W., Yu, X., Alghofaili, R., Lang, Y., Yu, L.-F.: Scene-aware behavior synthesis for virtual pets in mixed reality. In: CHI Conference on Human Factors in Computing Systems (CHI 2021), 12 p. ACM, New York, NY, USA (2021)

    Google Scholar 

  43. Gruenefeld, U., Prädel, L., Heuten, W.: Locating nearby physical objects in augmented reality. In: Proceedings of the 18th International Conference on Mobile and Ubiquitous Multimedia (MUM 2019), Article 1, pp. 1–10. ACM, New York, NY, USA (2019)

    Google Scholar 

  44. Kim, K., Welch, G.: Maintaining and enhancing human-surrogate presence in augmented reality. In: IEEE Symposium on Mixed and Augmented Reality Workshops, pp. 15–19 (2015)

    Google Scholar 

  45. Kim, K., Maloney, D., Bruder, G., Bailenson, J.N., Welch, G.F.: The effects of virtual human’s spatial and behavioral coherence with physical objects on social presence in AR. Comput. Anim. Virtual Worlds 28, e1771 (2017)

    Article  Google Scholar 

  46. Lee, M., Bruder, G., Höllerer, T., Welch, G.: Effects of unaugmented periphery and vibrotactile feedback on proxemics with virtual humans in AR. IEEE Trans. Visual Comput. Graphics 24(4), 1525–1534 (2018)

    Article  Google Scholar 

  47. Schraffenberger, H.K.: Arguably Augmented Reality. Relationships Between the Virtual and the Real. Ph.D. Thesis, Leiden University (2018)

    Google Scholar 

  48. Kim, J.C., Laine, T.H., Åhlund, C.: Multimodal interaction systems based on internet of things and augmented reality: a systematic literature review. Appl. Sci. 11, 1738 (2021)

    Article  Google Scholar 

  49. Lee M. et al.: The wobbly table: increased social presence via subtle incidental movement of a real-virtual table. In: Proceedings of the IEEE Virtual Reality, pp. 11–17 (2016)

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Faculty EEMCS, Human Media Interaction, University of Twente, PO Box 217, 7500 AE, Enschede, The Netherlands

    Anton Nijholt

Authors
  1. Anton Nijholt
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Anton Nijholt .

Editor information

Editors and Affiliations

  1. Smart Future Initiative, Frankfurt am Main, Germany

    Norbert A. Streitz

  2. Kyushu University, Fukuoka, Japan

    Shin'ichi Konomi

Rights and permissions

Reprints and permissions

Copyright information

© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Nijholt, A. (2022). Towards Social Companions in Augmented Reality: Vision and Challenges. In: Streitz, N.A., Konomi, S. (eds) Distributed, Ambient and Pervasive Interactions. Smart Living, Learning, Well-being and Health, Art and Creativity. HCII 2022. Lecture Notes in Computer Science, vol 13326. Springer, Cham. https://doi.org/10.1007/978-3-031-05431-0_21

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-05431-0_21

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-05430-3

  • Online ISBN: 978-3-031-05431-0

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation