A Systematic Review of the Convergence of Augmented Reality, Intelligent Virtual Agents, and the Internet of Things

  • Nahal NorouziEmail author
  • Gerd Bruder
  • Brandon Belna
  • Stefanie Mutter
  • Damla Turgut
  • Greg Welch
Part of the Transactions on Computational Science and Computational Intelligence book series (TRACOSCI)


In recent years we are beginning to see the convergence of three distinct research fields: augmented reality (AR), intelligent virtual agents (IVAs), and the Internet of things (IoT). Each of these has been classified as a disruptive technology for our society. Since their emergence, the advancement of knowledge and development of technologies and systems in these fields were traditionally performed with limited input from each other. However, over recent years, we have seen research prototypes and commercial products being developed that cross the boundaries between these distinct fields to leverage their collective strengths. In this paper, we review the body of literature published at the intersections between each two of these fields, and we discuss a vision for the nexus of all three technologies.


Augmented reality Intelligent virtual agents Internet of things Mixed reality Convergence research Literature review 


  1. 1.
    Al-Fuqaha, A., Guizani, M., Mohammadi, M., Aledhari, M., & Ayyash, M. (2015). Internet of things: A survey on enabling technologies, protocols, and applications. IEEE Communications Surveys & Tutorials, 17(4), 2347–2376.CrossRefGoogle Scholar
  2. 2.
    Al-Turjman, F. (2018). Information-centric framework for the Internet of Things (IoT): Traffic modeling & optimization. Future Generation Computer Systems Journal, 80, 63–75.CrossRefGoogle Scholar
  3. 3.
    Al-Turjman, F., & Alturjman, S. (2018). Context-sensitive access in Industrial Internet of Things (IIoT) healthcare applications. IEEE Transactions on Industrial Informatics, 14, 2736–2744.CrossRefGoogle Scholar
  4. 4.
    Anabuki, M., Kakuta, H., Yamamoto, H., & Tamura, H. (2000). Welbo: An embodied conversational agent living in mixed reality space. CHI ‘00 Extended Abstracts on Human Factors in Computing Systems. Proceedings of the ACM SIGCHI Conference on Human Factors 659 in Computing Systems (pp. 10–11).Google Scholar
  5. 5.
    Ashton, K. (2009). That ‘internet of things’ thing. RFID journal, 22(7), 97–114.Google Scholar
  6. 6.
    Austerjost, J., Porr, M., Riedel, N., Geier, D., Becker, T., Scheper, T., et al. (2018). Introducing a virtual assistant to the lab: A voice user Interface for the intuitive control of laboratory instruments. SLAS TECHNOLOGY: Translating Life Sciences Innovation, 23(5), 476–482.Google Scholar
  7. 7.
    Azuma, R. (1997). A survey of augmented reality. Presence: Teleoperators & Virtual Environments, 6(4), 355–385.CrossRefGoogle Scholar
  8. 8.
    Barakonyi, I., & Schmalstieg, D. (2005). Augmented reality agents in the development pipeline of computer entertainment. International Conference on Entertainment Computing (pp. 345–356).Google Scholar
  9. 9.
    Barakonyi, I., & Schmalstieg, D. (2006). Ubiquitous animated agents for augmented reality. 2006 IEEE/ACM International Symposium on Mixed and Augmented Reality (pp. 145–154).Google Scholar
  10. 10.
    Barakonyi, I., Psik, T., & Schmalstieg, D. (2004). Agents that talk and hit back: Animated agents in augmented reality. IEEE and ACM International Symposium on Mixed and Augmented Reality (pp. 141–150).Google Scholar
  11. 11.
    Barakonyi, I., Weilguny, M., Psik, T., & Schmalstieg, D. (2005). Monkey Bridge: Autonomous agents in augmented reality games. Proceedings of the 2005 ACM SIGCHI International Conference on Advances in Computer Entertainment Technology (pp. 172–175).Google Scholar
  12. 12.
    Bimber, O., & Raskar, R. (2005). Spatial augmented reality: Merging real and virtual worlds. Wellesley, MA: A.K. Peters.CrossRefGoogle Scholar
  13. 13.
    Blascovich, J. (2002). Social influence within immersive virtual environments. In The social life of avatars (pp. 127–145). London: Springer.CrossRefGoogle Scholar
  14. 14.
    Blum, L., Wetzel, R., McCall, R., Oppermann, L., & Broll, W. (2012). The final TimeWarp: Using form and content to support player experience and presence when designing location-aware mobile augmented reality games. Proceedings of the Designing Interactive Systems Conference on (pp. 711–720).Google Scholar
  15. 15.
    Bölöni, L., & Turgut, D. (2017). Value of information based scheduling of cloud computing resources. Future Generation Computer Systems Journal, 71, 212–220.CrossRefGoogle Scholar
  16. 16.
    Campagna, G., Ramesh, R., Xu, S., Fischer, M., & Lam, M. (2017). Almond: The architecture of an open, crowdsourced, privacy-preserving, programmable virtual assistant. WWW ‘17 Proceedings of the 26th International Conference on World Wide Web (pp. 341–350).Google Scholar
  17. 17.
    Charles, F., Cavazza, M., Mead, S., Martin, O., Nandi, A., & Marichal, X. (2004). Compelling experiences in mixed reality interactive storytelling. Proceedings of the 2004 ACM SIGCHI International Conference on Advances in Computer Entertainment Technology (pp. 32–40).Google Scholar
  18. 18.
    Chekhlov, D., Gee, A., Calway, A., & Mayol-Cuevas, W. (2007). Ninja on a plane: Automatic discovery of physical planes for augmented reality using visual SLAM. 6th IEEE and ACM International Symposium on Mixed and Augmented reality (pp. 153–156).Google Scholar
  19. 19.
    Chung, H., Iorga, M., Voas, J., & Lee, S. (2017). Alexa, Can I Trust You? IEEE Computer, 50(9), 100–104.CrossRefGoogle Scholar
  20. 20.
    Chung, H., Park, J., & Lee, S. (2017). Digital forensic approaches for Amazon Alexa ecosystem. Digital Investigation, 22, S15–S25.CrossRefGoogle Scholar
  21. 21.
    Daher, S., Kim, K., Lee, M., Bruder, G., Schubert, R., Bailenson, J., et al. (2017). Can social presence be contagious? Effects of social presence priming on interaction with virtual humans. 2017 IEEE Symposium on 3D User Interfaces (3DUI) (pp. 201–202).Google Scholar
  22. 22.
    Dey, A., Billinghurst, M., Lindeman, R., & Swan, J. (2018). A systematic review of 10 years of augmented reality usability studies: 2005 to 2014. Frontiers in Robotics and AI, 5, 37.CrossRefGoogle Scholar
  23. 23.
    Dow, S., Mehta, M., Harmon, E., MacIntyre, B., & Mateas, M. (2007). Presence and engagement in an interactive drama. Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (pp. 1475–1484).Google Scholar
  24. 24.
    Druga, S., Williams, R., Breazeal, C., & Resnick, M. (2017). "Hey Google is it OK if I eat you?": Initial explorations in child-agent interaction. Proceedings of the 2017 Conference on Interaction Design and Children (pp. 595–600).Google Scholar
  25. 25.
    Evans, D. (2011). The internet of things: How the next evolution of the internet is changing everything. Cisco Blog, 1, 1–1.Google Scholar
  26. 26.
    García-Macías, J. A., Alvarez-Lozano, J., Estrada-Martinez, P., & Avilés-López, E. (2011). Browsing the internet of things with sentient visors. Computer, 44(5), 46–52.CrossRefGoogle Scholar
  27. 27.
    Gatebox, Inc. (2018). Retrieved from
  28. 28.
    Gibson, J. (1979). The ecological approach to visual perception. Dallas: Houghtom Mifflin.Google Scholar
  29. 29.
    Gimenez, R., & Pous, M. (2010). Augmented reality as an enabling factor for the internet of things. Proceedings of the W3C Workshop: Augmented Reality on the Web.Google Scholar
  30. 30.
    Growing Convergence Research. (2018). Retrieved from National Science Foundation:
  31. 31.
    Hantono, B., Nugroho, L., & Santosa, P. (2016). Review of augmented reality agent in education. 2016 6th International Annual Engineering Seminar (InAES) (pp. 150–153).Google Scholar
  32. 32.
    Hao, Y., & Helo, P. (2017). The role of wearable devices in meeting the needs of cloud manufacturing: A case study. Robotics and Computer-Integrated Manufacturing, 45, 168–179.CrossRefGoogle Scholar
  33. 33.
    Helal, A., Cho, K., Lee, W., Sung, Y., Lee, J., & Kim, E. (2012). 3D modeling and simulation of human activities in smart spaces. 2012 9th International Conference on Ubiquitous Intelligence and Computing and 9th International Conference on Autonomic and Trusted Computing (pp. 112–119).Google Scholar
  34. 34.
    Heun, V., Hobin, J., & Maes, P. (2013). Reality editor: Programming smarter objects. Proceedings of the 2013 ACM conference on Pervasive and Ubiquitous Computing Adjunct Publication (pp. 307–310).Google Scholar
  35. 35.
    Hoffman, D., & Novak, T. (2018). Consumer and object experience in the internet of things: An assemblage theory approach. Journal of Consumer Research, 44(6), 1178–1204.CrossRefGoogle Scholar
  36. 36.
    Holz, T., Campbell, A., O’Hare, G., Stafford, J., Martin, A., & Dragone, M. (2011). MiRA-Mixed Reality Agents. International Journal of Human-Computer Studies/International Journal of Man-Machine Studies, 69(4), 251–268.Google Scholar
  37. 37.
    Jo, D., & Kim, G. (2016). ARIoT: Scalable augmented reality framework for interacting with internet of things appliances everywhere. IEEE Transactions on Consumer Electronics, 62(3), 334–340.CrossRefGoogle Scholar
  38. 38.
    Kasahara, S., Niiyama, R., Heun, V., & Ishii, H. (2013). exTouch: Spatially-aware embodied manipulation of actuated objects mediated by augmented reality. Proceedings of the 7th International Conference on Tangible, Embedded and Embodied Interaction (pp. 223–228).Google Scholar
  39. 39.
    Kim, K., Bruder, G., Maloney, D., & Welch, G. (2016). The influence of real human personality on social presence with a virtual human in augmented reality. ICAT-EGVE ‘16 proceedings of the 26th International Conference on Artificial Reality and Telexistence and the 21st Eurographics Symposium on Virtual Environments (pp. 115–122).Google Scholar
  40. 40.
    Kim, K., Bruder, G., & Welch, G. (2017). Exploring the effects of observed physicality conflicts on real-virtual human interaction in augmented reality. Proceedings of the 23rd ACM Symposium on Virtual Reality Software and Technology (p. 31).Google Scholar
  41. 41.
    Kim, K., Billinghurst, M., Bruder, G., Duh, H. B.-L., & Welch, G. (2018). Revisiting trends in augmented reality research: A review of the 2nd decade of ISMAR (2008–2017). IEEE Transactions on Visualization and Computer Graphics (TVCG) Special Issue on the International Symposium on Mixed and Augmented Reality (ISMAR).Google Scholar
  42. 42.
    Knote, R., Janson, A., Eigenbrod, L., & Söllner, M. (2018). The what and how of smart personal assistants: Principles and application domains for IS research. In: Multikonferenz Wirtschaftsinformatik (MKWI). Lüneburg: Germany.Google Scholar
  43. 43.
    Kollee, B., Kratz, S., & Dunnigan, A. (2014). Exploring gestural interaction in smart spaces using head mounted devices with ego-centric sensing. Proceedings of the 2nd ACM symposium on spatial user interaction (pp. 40–49).Google Scholar
  44. 44.
    Kotranza, A., & Lok, B. (2008). Virtual human + tangible interface = mixed reality human an initial exploration with a virtual breast exam patient. 2008 IEEE Virtual Reality Conference (pp. 99–106).Google Scholar
  45. 45.
    Kotranza, A., Lok, B., Deladisma, A., Pugh, C., & Lind, D. (2009). Mixed reality humans: Evaluating behavior, usability, and acceptability. IEEE Transactions on Visualization and Computer Graphics, 15(3), 369–382.CrossRefGoogle Scholar
  46. 46.
    Krum, D., Suma, E., & Bolas, M. (2012). Augmented reality using personal projection and retroreflection. Ubiquitous Computing, 16(1), 17–26.CrossRefGoogle Scholar
  47. 47.
    Lee, L.H. & Hui, P. (2018). Interaction Methods for Smart Glasses: A survey. IEEE Access, (pp. 28712–28732).CrossRefGoogle Scholar
  48. 48.
    Lee, M., Kim, K., Daher, S., Raij, A., Schubert, R., Bailenson, J., et al. (2016a). The wobbly table: Increased social presence via subtle incidental movement of a real-virtual table. 2016 IEEE Virtual Reality (VR) (pp. 11–17).Google Scholar
  49. 49.
    Lee, W., Cho, S., Chu, P., Vu, H., Helal, S., Song, W., et al. (2016b). Automatic agent generation for IoT-based smart house simulator. Neurocomputing, 209, 14–24.CrossRefGoogle Scholar
  50. 50.
    Lee, M., Bruder, G., & Welch, G. (2017). Exploring the effect of vibrotactile feedback through the floor on social presence in an immersive virtual environment. 2017 IEEE Virtual Reality (VR) (pp. 105–111).Google Scholar
  51. 51.
    Lee, M., Bruder, G., Hollerer, T., & Welch, G. (2018). Effects of unaugmented periphery and vibrotactile feedback on proxemics with virtual humans in AR. IEEE Transactions on Visualization and Computer Graphics, 24(4), 1525–1534.CrossRefGoogle Scholar
  52. 52.
    Lok, B., Chuah, J., Robb, A., Cordar, A., Lampotang, S., Wendling, A., et al. (2014). Mixed-reality humans for team training. IEEE Computer Graphics and Applications, 34(3), 72–75.CrossRefGoogle Scholar
  53. 53.
    López, G., Quesada, L., & Guerrero, L. A. (2017). Alexa vs. Siri vs. Cortana vs. Google assistant: A comparison of speech-based natural user interfaces. Proceedings of the International Conference on Applied Human Factors and Ergonomics (pp. 241–250).Google Scholar
  54. 54.
    Magic Leap, Inc. (2018). Retrieved from
  55. 55.
    Martin, K., & Laviola, J. (2016). The transreality interaction platform: Enabling interaction across physical and virtual reality. 2016 IEEE International Conference on Internet of Things (iThings) and IEEE Green Computing and Communications (GreenCom) and IEEE Cyber, Physical and Social Computing (CPSCom) and IEEE Smart Data (SmartData) (pp. 177–186).Google Scholar
  56. 56.
    Mayle, A., Bidoki, N. H., Masnadi, S., Bölöni, L., & Turgut, D. (2017). Investigating the value of privacy within the internet of things. Proceedings of IEEE GLOBECOM (pp. 1–6).Google Scholar
  57. 57.
    Milgram, P., Takemura, H., Utsumi, A., & Kishino, F. (1995). Augmented reality: A class of displays on the reality-virtuality continuum. Telemanipulator and Telepresence Technologies, 2351, 282–292.CrossRefGoogle Scholar
  58. 58.
    Newsroom Gartner. (2016). Retrieved from Gartner says worldwide spending on VPA-enabled wireless speakers will top $2 billion by 2020:
  59. 59.
    Norouzi, N., Kim, K., Hochreiter, J., Lee, M., Daher, S., Bruder, G., et al. (2018). A systematic survey of 15 years of user studies Published in the Intelligent virtual agents conference. International Conference on Intelligent Virtual Agents (IVA).Google Scholar
  60. 60.
    Papagiannis, H. (2017). Augmented human: How technology is shaping the new reality. Bejing: O’Reilly Media.Google Scholar
  61. 61.
    Paul, Z., Margarita, P., Vasilis, M., & George, P. (2016). Life-sized group and crowd simulation in Mobile AR. Proceedings of the 29th International Conference on Computer Animation and Social Agents (pp. 79–82).Google Scholar
  62. 62.
    Raskar, R. (2001). Projector-based three dimensional graphics. Chapel Hill: University of North Carolina.Google Scholar
  63. 63.
    Raskar, R., Welch, G., Cutts, M., Lake, A., Stesin, L., & Fuchs, H. (1998). The office of the future: A unified approach to image-based modeling and spatially immersive displays. Proceedings of the 25th Annual Conference on Computer Graphics and Interactive Techniques (pp. 179–188).Google Scholar
  64. 64.
    Raskar, R., Welch, G., & Chen, W.-C. (1999). Table-top spatially-augmented realty: Bringing physical models to life with projected imagery. Proceedings 2nd IEEE and ACM International Workshop on Augmented Reality (IWAR’99) (pp. 64–71).Google Scholar
  65. 65.
    Raskar, R., Welch, G., Low, K.-L., & Bandyopadhyay, D. (2001). Shader lamps: Animating real objects with image-based illumination. Proceedings of the 12th Eurographics Workshop on Rendering Techniques (pp. 89–102).Google Scholar
  66. 66.
    Reis, A., Paulino, D., Paredes, H., & Barroso, J. (2017). Using intelligent personal assistants to strengthen the Elderlies’ social bonds. International Conference on Universal Access in Human-Computer Interaction (pp. 593–602).CrossRefGoogle Scholar
  67. 67.
    Robb, A., Cordar, A., Lampotang, S., White, C., Wendling, A., & Lok, B. (2015). Teaming up with virtual humans: How other people change our perceptions of and behavior with virtual teammates. IEEE Transactions on Visualization and Computer Graphics, 21(4), 511–519.CrossRefGoogle Scholar
  68. 68.
    Salman, T., & Jain, R. (2017). A survey of protocols and standards for internet of things. Advanced Computing and Communications, 1(1), 1–20.Google Scholar
  69. 69.
    Seo, D., Kim, H., Kim, J., & Lee, J. (2016). Hybrid reality-based user experience and evaluation of a context-aware smart home. Computers in Industry, 76, 11–23.CrossRefGoogle Scholar
  70. 70.
    Skarbez, R., Welch, G., Brooks, F., & Whitton, M. (2017). Coherence changes gaze behavior in virtual human interactions. 2017 IEEE Virtual Reality (VR) (pp. 287–288).Google Scholar
  71. 71.
    Soda, S., Nakamura, M., Matsumoto, S., Izumi, S., Kawaguchi, H., & Yoshimoto, M. (2012). Implementing virtual agent as an interface for smart home voice control. 2012 19th Asia-Pacific Software Engineering Conference, 1, pp. 342–345.Google Scholar
  72. 72.
    Sutherland, I. (1968). A head-mounted three dimensional display. Proceedings of the December 9–11, 1968, Fall Joint Computer Conference, Part I on (pp. 757–764).Google Scholar
  73. 73.
    Turgut, D., & Bölöni, L. (2017, September). Value of information and cost of privacy in the internet of things. IEEE Communications Magazine, 55(9), 62–66.CrossRefGoogle Scholar
  74. 74.
    Vugt, H., Bailenson, J., Hoorn, J., & Konijn, E. (2010). Effects of facial similarity on user responses to embodied agents. ACM Transactions on Computer-Human Interaction, 17(2), 7.CrossRefGoogle Scholar
  75. 75.
    Wagner, D., Billinghurst, M., & Schmalstieg, D. (2006). How real should virtual characters be. Proceedings of the 2006 ACM SIGCHI International Conference on Advances in Computer Entertainment Technology (p. 57).Google Scholar
  76. 76.
    Wirtz, H., Rüth, J., Serror, M., Link, J., & Wehrle, K. (2014). Opportunistic interaction in the challenged internet of things. Proceedings of the 9th ACM MobiCom Workshop on Challenged Networks (pp. 7–12).Google Scholar
  77. 77.
    Zehtabian, S., Khodadadeh, S., Pearlman, R., Willenberg, B., Kim, B., Turgut, D., et al. (2018). Supporting rehabilitation prescription compliance with an IoT-augmented four-legged walker. Workshop on AI for Aging, Rehabilitation and Independent Assisted Living (ARIAL’18) in Conjunction with International Joint Conference on Artificial Intelligence (IJCA’18).Google Scholar
  78. 78.
    Zhang, B., Chen, Y.-H., Tuna, C., Dave, A., Li, Y., Lee, E., et al. (2014). HOBS: Head orientation-based selection in physical spaces. Proceedings of the 2nd ACM Symposium on Spatial User Interaction (pp. 17–25).Google Scholar
  79. 79.

Copyright information

© Springer Nature Switzerland AG 2019

Authors and Affiliations

  • Nahal Norouzi
    • 1
    Email author
  • Gerd Bruder
    • 1
  • Brandon Belna
    • 1
  • Stefanie Mutter
    • 2
  • Damla Turgut
    • 1
  • Greg Welch
    • 1
  1. 1.University of Central FloridaOrlandoUSA
  2. 2.Muhlenberg CollegeAllentownUSA

Personalised recommendations