Skip to main content
SpringerLink
Log in

Convergence of IoT and Augmented Reality

  • Chapter
  • First Online:
Springer Handbook of Augmented Reality

Part of the book series: Springer Handbooks ((SHB))

Abstract

This chapter contends that the Internet of Things (IoT), which consists of technologies for networked embedded devices and decentralized software applications, must also have rich interface solutions that both inform and engage its users. This is a new frontier in IoT human-computer interaction (HCI), and meeting these challenges requires interface designs that present information to users in a context-sensitive manner, which can inform users while allowing them to remain engaged in their environmental tasks. Mixed reality, combined with context sensing, also known as X-Reality (XR), allows the display of 3D content in situ, providing novel interface design possibilities for IoT edge devices. These XR-IoT (XRI) hybrid systems can become more personal, immersive, embedded, information-rich, decentralized, multiuser, and agent-driven. This repositions the IoT from the passive background embedded environment into active, engaging, foreground information infrastructures. This chapter highlights the challenges and considerations to meet the convergence of the IoT with mixed reality, toward a transformation of IoT devices into hybrid virtual and physical objects that adapt to their user’s context, and presents information in engaging ways. Such a merger will contribute to the IoT broadly and impact HCI across existing and future IoT deployments.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 309.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Hardcover Book
USD 399.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Ahmed, E., Yaqoob, I., Gani, A., Imran, M., Guizani, M.: Internet-of-things-based smart environments: State of the art, taxonomy, and open research challenges. IEEE Wirel. Commun. 23, 10–16 (2016). https://doi.org/10.1109/MWC.2016.7721736

    Article  Google Scholar 

  2. Bernal, G., Colombo, S., Al Ai Baky, M., Casalegno, F.: Safety++ designing IoT and wearable systems for industrial safety through a user centered design approach. In: Proceedings of the 10th International Conference on Pervasive Technologies Related to Assistive Environments, pp. 163–170 (2017)

    Google Scholar 

  3. Bilda, Z., Edmonds, E., Candy, L.: Designing for creative engagement. Design Stud. 29, 525–540 (2007). https://doi.org/10.1016/j.destud.2008.07.009

    Article  Google Scholar 

  4. Billinghurst, M., Kato, H., Poupyrev, I., et al.: Tangible augmented reality. ACM SIGGRAPH Asia 7(2), 1–10 (2008)

    Google Scholar 

  5. Billinghurst, M., Clark, A., Lee, G., et al.: A survey of augmented reality. Found. Trends® Hum.-Comput. Interact. 8(2-3), 73–272 (2015)

    Google Scholar 

  6. Blanco-Novoa, Ó., Fraga-Lamas, P., Vilar-Montesinos, M.A., Fernández-Caramés, T.M.: Towards the internet of augmented things: an open-source framework to interconnect IoT devices and augmented reality systems. In: Multidisciplinary Digital Publishing Institute Proceedings, vol. 42, p. 50 (2019)

    Google Scholar 

  7. Bogost, I.: Alien Phenomenology, or What It’s Like to Be a Thing. University of Minnesota (2012)

    Google Scholar 

  8. Bridle, J.: New dark age: Technology and the end of the future. Verso Books (2018)

    Google Scholar 

  9. Brudy, F., Holz, C., Rädle, R., Wu, C.J., Houben, S., Klokmose, C.N., Marquardt, N.: Cross-device taxonomy: survey, opportunities and challenges of interactions spanning across multiple devices. In: Conference on Human Factors in Computing Systems - Proceedings pp. 1–28 (2019). https://doi.org/10.1145/3290605.3300792

  10. Dellermann, D., Calma, A., Lipusch, N., Weber, T., Weigel, S., Ebel, P.: The future of human-ai collaboration: A taxonomy of design knowledge for hybrid intelligence systems. In: Proceedings of the 52nd Hawaii International Conference on System Sciences, vol. 6, pp. 274–283 (2019). https://doi.org/10.24251/hicss.2019.034

  11. Dey, A.K., Abowd, G.D.: The what, who, where, when, why and how of context-awareness (2000)

    Google Scholar 

  12. Di Geronimo, L., Braz, L., Fregnan, E., Palomba, F., Bacchelli, A.: Ui dark patterns and where to find them: a study on mobile applications and user perception. In: Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems, pp. 1–14 (2020)

    Google Scholar 

  13. Fisher, J.A.: Utilizing the mixed reality cube taxonomy for interactive documentary research. AltMM 2016 - Proceedings of the 1st International Workshop on Multimedia Alternate Realities, co-located with ACM Multimedia 2016 pp. 9–14 (2016). https://doi.org/10.1145/2983298.2983299

  14. Flavián, C., Ibáñez-Sánchez, S., Orús, C.: The impact of virtual, augmented and mixed reality technologies on the customer experience. J. Bus. Res. 100, 547–560 (2019)

    Article  Google Scholar 

  15. Gubbi, J., Buyya, R., Marusic, S., Palaniswami, M.: Internet of Things (IoT): A vision, architectural elements, and future directions. Fut. Gener. Comput. Syst. 29(7), 1645–1660 (2013)

    Article  Google Scholar 

  16. Gushima, K., Nakajima, T.: A design space for virtuality-introduced Internet of Things. Fut. Internet 9(4), 60 (2017)

    Article  Google Scholar 

  17. Harman, G.: Heidegger Explained, vol. 4. Open Court Publishing (2007)

    Google Scholar 

  18. Iatsyshyn, V.A., Kovach, O.V., Romanenko, O.Y., Deinega, I.I., Iatsyshyn, V.A., Popov, O.O., Kutsan, G.Y., Artemchuck, O.V., Burov, Y.O., Lytvynova, H.S.: Application of augmented reality technologies for preparation of specialists of new technological era. English Article 2018, 181–200 (2020)

    Google Scholar 

  19. Jeschke, S., Brecher, C., Meisen, T., Özdemir, D., Eschert, T.: Industrial internet of things and cyber manufacturing systems. In: Industrial Internet of Things, pp. 3–19. Springer, Berlin (2017)

    Google Scholar 

  20. Jing, Q., Vasilakos, A.V., Wan, J., Lu, J., Qiu, D.: Security of the Internet of Things: perspectives and challenges. Wirel. Netw. 20(8), 2481–2501 (2014)

    Article  Google Scholar 

  21. Jo, D., Kim, G.J.: Ariot: scalable augmented reality framework for interacting with internet of things appliances everywhere. IEEE Trans. Consum. Electron. 62(3), 334–340 (2016)

    Article  Google Scholar 

  22. Jo, D., Kim, G.J.: In-situ ar manuals for IoT appliances. In: 2016 IEEE International Conference on Consumer Electronics (ICCE), pp. 409–410. IEEE, Piscataway (2016)

    Google Scholar 

  23. Jo, D., Kim, G.J.: Ar enabled IoT for a smart and interactive environment: a survey and future directions. Sensors 19(19), 4330 (2019)

    Article  Google Scholar 

  24. Jo, D., Kim, G.J.: IoT+  AR: pervasive and augmented environments for “digi-log” shopping experience. Hum.-Centric Comput. Inform. Sci. 9(1), 1–17 (2019)

    Google Scholar 

  25. Jo, D., Kim, K.H., Kim, G.J.: Effects of avatar and background representation forms to co-presence in mixed reality (MR) tele-conference systems. In: SIGGRAPH ASIA 2016 Virtual Reality Meets Physical Reality: Modelling and Simulating Virtual Humans and Environments, pp. 1–4 (2016)

    Google Scholar 

  26. Karthick, M., Kumar, C.S., Robert, T.P.: Investigation of Human Factors Using HFACS Framework—A Case Study for Unintended Reactor Trip Events in NPP. Springer, Singapore (2020). https://doi.org/10.1007/978-981-13-9008-1

    Book  Google Scholar 

  27. Kayes, A.S., Kalaria, R., Sarker, I.H., Islam, M.S., Watters, P.A., Ng, A., Hammoudeh, M., Badsha, S., Kumara, I.: A survey of context-aware access control mechanisms for cloud and fog networks: Taxonomy and open research issues. Sensors (Switzerland) 20 (2020). https://doi.org/10.3390/s20092464

  28. Hoon Kim, T., Ramos, C., Mohammed, S.: Smart city and IoT. Fut. Gener. Comput. Syst. 76, 159–162 (2017). https://doi.org/10.1016/j.future.2017.03.034

    Article  Google Scholar 

  29. Kim, Y.M., Rhiu, I., Yun, M.H.: A systematic review of a virtual reality system from the perspective of user experience. Int. J. Hum.-Comput. Interact. 36, 893–910 (2020). https://doi.org/10.1080/10447318.2019.1699746

    Article  Google Scholar 

  30. Kranz, M., Holleis, P., Schmidt, A.: Embedded interaction: interacting with the Internet of Things. IEEE Internet Comput. 14(2), 46–53 (2009)

    Article  Google Scholar 

  31. Kress, B., Saeedi, E., Brac-de-la Perriere, V.: The segmentation of the HMD market: optics for smart glasses, smart eyewear, AR and VR headsets. In: Photonics Applications for Aviation, Aerospace, Commercial, and Harsh Environments V, vol. 9202, p. 92020D. International Society for Optics and Photonics (2014)

    Google Scholar 

  32. Kress, B.C., Cummings, W.J.: 11-1: Invited paper: Towards the ultimate mixed reality experience: Hololens display architecture choices. In: SID Symposium Digest of Technical Papers, vol. 48, pp. 127–131. Wiley Online Library (2017)

    Google Scholar 

  33. Lee, L.H., Hui, P.: Interaction methods for smart glasses: a survey. IEEE Access 6, 28712–28732 (2018)

    Article  Google Scholar 

  34. Leotta, F., Mecella, M., Sora, D., Catarci, T.: Surveying human habit modeling and mining techniques in smart spaces. Fut. Internet 11 (2019). https://doi.org/10.3390/fi11010023

  35. Lupiana, D., Driscoll, C.O., Mtenzi, F.: Defining smart space in the context of ubiquitous computing. In: Special Issue onICIT 2009 Conference -Web and Agent Systems, vol. 4, pp. 516–524 (2009)

    Google Scholar 

  36. Mann, S., Furness, T., Yuan, Y., Iorio, J., Wang, Z.: All reality: Values, taxonomy, and continuum, for virtual, augmented, extended/mixed (x), mediated (x,y), and multimediated reality/intelligence. Awe (2018)

    Google Scholar 

  37. Mignonneau, L., Sommerer, C.: Designing emotional, metaphoric, natural and intuitive interfaces for interactive art, edutainment and mobile communications. Comput. Graph. (Pergamon) 29, 837–851 (2005). https://doi.org/10.1016/j.cag.2005.09.001

  38. Milgram, P., Kishino, F.: A taxonomy of mixed reality visual displays. IEICE Trans. Inform. Syst. 77(12), 1321–1329 (1994)

    Google Scholar 

  39. Milgram, P., Takemura, H., Utsumi, A., Kishino, F.: Augmented reality: a class of displays on the reality-virtuality continuum. In: Telemanipulator and Telepresence Technologies, vol. 2351, pp. 282–292. International Society for Optics and Photonics (1995)

    Google Scholar 

  40. Milhorat, P., Schlögl, S., Chollet, G., Boudy, J., Esposito, A., Pelosi, G.: Building the next generation of personal digital assistants. In: 2014 1st International Conference on Advanced Technologies for Signal and Image Processing (ATSIP), pp. 458–463. IEEE, Piscataway (2014)

    Google Scholar 

  41. Morris, A., Lessio, N.: Deriving privacy and security considerations for core: an indoor IoT adaptive context environment. In: Proceedings of the ACM Conference on Computer and Communications Security pp. 2–11 (2018). https://doi.org/10.1145/3267357.3267363

  42. Morris, A., Guan, J., Lessio, N., Shao, Y.: Toward mixed reality hybrid objects with IoT avatar agents. In: 2020 IEEE International Conference on Systems, Man, and Cybernetics (SMC), pp. 766–773. IEEE, Piscataway (2020)

    Google Scholar 

  43. Muhanna, M.A.: Virtual reality and the cave: Taxonomy, interaction challenges and research directions. J. King Saud Univ. Comput. Inform. Sci. 27, 344–361 (2015). https://doi.org/10.1016/j.jksuci.2014.03.023

    Google Scholar 

  44. Nielsen, J.: Iterative user-interface design. Computer 26(11), 32–41 (1993)

    Article  Google Scholar 

  45. Norman, D.: The Design of Everyday Things, doubleday/edn. Basic Books (1990)

    Google Scholar 

  46. Nuamah, J., Seong, Y.: Human machine interface in the internet of things (IoT). In: 2017 12th System of Systems Engineering Conference (SoSE), pp. 1–6. IEEE, Piscataway (2017)

    Google Scholar 

  47. Paradiso, J.A., Landay, J.A.: Guest editors’ introduction: cross-reality environments. IEEE Pervasive Comput. 8(3), 14–15 (2009)

    Article  Google Scholar 

  48. Perera, C., Zaslavsky, A., Christen, P., Georgakopoulos, D.: Context aware computing for the internet of things: a survey. IEEE Commun. Surv. Tutorials 16, 414–454 (2014). https://doi.org/10.1109/SURV.2013.042313.00197

    Article  Google Scholar 

  49. Phupattanasilp, P., Tong, S.R.: Augmented reality in the integrative internet of things (AR-IoT): Application for precision farming. Sustainability 11(9), 2658 (2019)

    Article  Google Scholar 

  50. Rokhsaritalemi, S., Sadeghi-Niaraki, A., Choi, S.M.: A review on mixed reality: current trends, challenges and prospects. Appl. Sci. (Switzerland) 10 (2020). https://doi.org/10.3390/app10020636

  51. Shao, Y., Lessio, N., Morris, A.: IoT avatars: mixed reality hybrid objects for core ambient intelligent environments. Proc. Comput. Sci. 155, 433–440 (2019). https://doi.org/10.1016/j.procs.2019.08.060

    Article  Google Scholar 

  52. Shin, D.: A socio-technical framework for internet-of-things design: a human-centered design for the Internet of Things. Telematics Inform. 31(4), 519–531 (2014)

    Article  Google Scholar 

  53. Shinde, G.R., Dhotre, P.S., Mahalle, P.N., Dey, N.: Internet of Things Integrated Augmented Reality. Springer, Berlin (2020)

    Google Scholar 

  54. Shirehjini, A.A.N., Semsar, A.: Human interaction with IoT-based smart environments. Multimedia Tools Appl. 76(11), 13343–13365 (2017)

    Article  Google Scholar 

  55. Thierer, A., Castillo, A.: Projecting the growth and economic impact of the Internet of Things. George Mason University, Mercatus Center, vol. 15 (2015)

    Google Scholar 

  56. Truelove, A., Chowdhury, F.N., Gnawali, O., Alipour, M.A.: Topics of concern: identifying user issues in reviews of IoT apps and devices. In: 2019 IEEE/ACM 1st International Workshop on Software Engineering Research & Practices for the Internet of Things (SERP4IoT), pp. 33–40. IEEE, Piscataway (2019)

    Google Scholar 

  57. Tuladhar, S., AlSallami, N., Alsadoon, A., Prasad, P.W., Alsadoon, O.H., Haddad, S., Alrubaie, A.: A recent review and a taxonomy for hard and soft tissue visualization-based mixed reality. Int. J. Med. Robot. 16(5), 1–22 (2020). https://doi.org/10.1002/rcs.2120

    Article  Google Scholar 

  58. Ungureanu, D., Bogo, F., Galliani, S., Sama, P., Duan, X., Meekhof, C., Stühmer, J., Cashman, T.J., Tekin, B., Schönberger, J.L., et al.: Hololens 2 research mode as a tool for computer vision research (2020). arXiv preprint arXiv:2008.11239

    Google Scholar 

  59. Vicente, K.: The human factor: revolutionizing the way we live with technology. Vintage Canada (2004)

    Google Scholar 

  60. West, D.M.: How 5g technology enables the health internet of things. Brookings Center for Technology Innovation, vol. 3, pp. 1–20 (2016)

    Google Scholar 

  61. White, G., Cabrera, C., Palade, A., Clarke, S.: Augmented reality in IoT. In: International Conference on Service-Oriented Computing, pp. 149–160. Springer, Berlin (2018)

    Google Scholar 

  62. Zemmoudj, S., Bermad, N., Omar, M.: Context-aware pseudonymization and authorization model for IoT-based smart hospitals. J. Ambient Intell. Hum. Comput. 10, 4473–4490 (2019). https://doi.org/10.1007/s12652-018-1129-0

    Article  Google Scholar 

  63. Zollmann, S., Grasset, R., Langlotz, T., Lo, W.H., Mori, S., Regenbrecht, H.: Visualization techniques in augmented reality: a taxonomy, methods and patterns. IEEE Trans. Vis. Comput. Graph. 2626, 1–20 (2020). https://doi.org/10.1109/TVCG.2020.2986247

    Google Scholar 

Download references

Acknowledgements

The authors gratefully acknowledge support from the Tri-Council of Canada under the Canada Research Chairs program.

Author information

Authors and Affiliations

  1. Adaptive Context Environments Lab, OCAD University, Toronto, ON, Canada

    Alexis Morris & Georgina Yeboah

Authors
  1. Alexis Morris
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Georgina Yeboah
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Alexis Morris .

Editor information

Editors and Affiliations

  1. Department of Mechanical Engineering, National University of Singapore, Singapore, Singapore

    Andrew Yeh Ching Nee

  2. Department of Mechanical Engineering, National University of Singapore, Singapore, Singapore

    Soh Khim Ong

Section Editor information

    Rights and permissions

    Reprints and permissions

    Copyright information

    © 2023 Springer Nature Switzerland AG

    About this chapter

    Check for updates. Verify currency and authenticity via CrossMark

    Cite this chapter

    Morris, A., Yeboah, G. (2023). Convergence of IoT and Augmented Reality. In: Nee, A.Y.C., Ong, S.K. (eds) Springer Handbook of Augmented Reality. Springer Handbooks. Springer, Cham. https://doi.org/10.1007/978-3-030-67822-7_33

    Download citation

    • DOI: https://doi.org/10.1007/978-3-030-67822-7_33

    • Published:

    • Publisher Name: Springer, Cham

    • Print ISBN: 978-3-030-67821-0

    • Online ISBN: 978-3-030-67822-7

    • eBook Packages: Computer ScienceComputer Science (R0)

    Publish with us

    Policies and ethics

    Search

    Navigation