Cross-reality environments in smart buildings to advance STEM cyberlearning


Real time data associated with the Building Information Model plays a critical role in the interpretation of the built environment, which is particularly relevant as an increasing number of education facilities and institutions promote sustainable engineering practices and monitoring data available to the public. However, it is challenging for non-technical audiences to fully comprehend or use information concealed in scientific data related to the performance of structures and materials. It is especially difficult for them to connect these concepts to physical contexts and phenomena. In this paper, we present how cross-reality paradigms in Architecture, Engineering, and Construction, coupled with multimodal representation techniques, enhance data literacy in both professionals and laypeople alike. In particular, we present the design of a learning environment where cutting-edge holographic interfaces and display technologies are combined with sonified and visual data to create a more immersive environment for data analysis and exploration, empowering users with situated data awareness and new ways of understanding real-time data.

This is a preview of subscription content, access via your institution.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8


  1. 1.

    Porter, J.R., Morgan, J.A., Johnson, M.: Building automation and IoT as a platform for introducing STEM education in K-12. In: 2017 ASEE Annual Conference & Exposition (2017)

  2. 2.

    Dickel, S.: Trust in technologies? Science after de-professionalization. J. Sci. Commun. 15, 1–7 (2016).

    Google Scholar 

  3. 3.

    Buckman, A.H., Mayfield, M., Beck, B.M.: What is a smart building? Smart Sustain. Built Environ. 3, 92–109 (2014).

    Article  Google Scholar 

  4. 4.

    Carlson L.E, Sullivan J.F.: (1999) Hands-on engineering: learning by doing in the integrated teaching and learning program. Int. J. Eng. Educ. 15, 20–31

    Google Scholar 

  5. 5.

    Petersen, J., Frantz, C., Shammin, R.: Using sociotechnical feedback to engage, educate, motivate and empower environmental thought and action. Solutions 5, 79–87 (2014)

    Google Scholar 

  6. 6.

    Petersen, J.E., Murray, M.E., Platt, G., Shunturov, V.: Using buildings to teach environmental stewardship: real-time display of environmental performance as a mechanism for educating, motivating, and empowering the student body. In: Proc Green Campus VI Muncie Indiana (2007)

  7. 7.

    Petersen, J.E., Rosenberg Daneri, D., Frantz, C., Shammin, M.R.: Environmental dashboards: fostering pro-environmental and pro-community thought and action through feedback. In: Leal Filho, W., Mifsud, M., Shiel, C., Pretorius, R. (eds.) Handbook of Theory and Practice of Sustainable Development in Higher Education, pp. 149–168. Springer International Publishing, Cham (2017)

    Google Scholar 

  8. 8.

    Squire, K., Klopfer, E.: Augmented reality simulations on handheld computers. J. Learn. Sci. 16, 371–413 (2007).

    Article  Google Scholar 

  9. 9.

    Kamarainen, A.M., Metcalf, S., Grotzer, T., Browne, A., Mazzuca, D., Tutwiler, M.S., Dede, C.: EcoMOBILE: integrating augmented reality and probeware with environmental education field trips. Comput. Educ. 68, 545–556 (2013).

    Article  Google Scholar 

  10. 10.

    Cocciolo, A., Rabina, D.: Does place affect user engagement and understanding? Mobile learner perceptions on the streets of New York. J. Doc. 69, 98–120 (2013).

    Article  Google Scholar 

  11. 11.

    Pimmer, C., Mateescu, M., Gröhbiel, U.: Mobile and ubiquitous learning in higher education settings: a systematic review of empirical studies. Comput. Hum. Behav. 63, 490–501 (2016).

    Article  Google Scholar 

  12. 12.

    Rogers, Y., Price, S., Harris, E., Phelps, T., Underwood, M., Wilde, D., Smith, H., Weal, M.T.M.J., Michaelides, D.T.: Learning through digitally-augmented physical experiences: reflections on the Ambient Wood project (2002)

  13. 13.

    Brown, A., Green, T.: Virtual reality: low-cost tools and resources for the classroom. Tech. Trends. 60, 517–519 (2016).

    Article  Google Scholar 

  14. 14.

    Brown, J.S., Collins, A., Duguid, P.: Situated Cognition and the Culture of Learning. Educ. Res. 18, 32–42 (1989).

    Article  Google Scholar 

  15. 15.

    Smith, S.M., Vela, E.: Environmental context-dependent memory: a review and meta-analysis. Psychon. Bull. Rev. 8, 203–220 (2001).

    Article  Google Scholar 

  16. 16.

    Chun, M.M., Jiang, Y.: Contextual Cueing: implicit Learning and Memory of Visual Context Guides Spatial Attention. Cogn. Psychol. 36, 28–71 (1998).

    Article  Google Scholar 

  17. 17.

    Cook, A.E., Limber, J.E., O’Brien, E.J.: Situation-based context and the availability of predictive inferences. J. Mem. Lang. 44, 220–234 (2001).

    Article  Google Scholar 

  18. 18.

    Smith, S.M., Glenberg, A., Bjork, R.A.: Environmental context and human memory. Mem. Cogn. 6, 342–353 (1978).

    Article  Google Scholar 

  19. 19.

    Chi, M.T.H.: Active-Constructive-Interactive: a conceptual framework for differentiating learning activities. Top. Cogn. Sci. 1, 73–105 (2009).

    Article  Google Scholar 

  20. 20.

    Jang, S., Vitale, J.M., Jyung, R.W., Black, J.B.: Direct manipulation is better than passive viewing for learning anatomy in a three-dimensional virtual reality environment. Comput. Edu. 106, 150–165 (2017).

    Article  Google Scholar 

  21. 21.

    Goldstone, R.L., Son, J.Y.: The transfer of scientific principles using concrete and idealized simulations. J. Learn. Sci. 14, 69–110 (2005).

    Article  Google Scholar 

  22. 22.

    Larkin, J., Simon, H.: Why a diagram is (Sometimes) worth ten thousand words. Cogn. Sci. 11, 65–100 (1987).

    Article  Google Scholar 

  23. 23.

    Schnotz, W., Bannert, M.: Construction and interference in learning from multiple representation. Learn. Instr. 13, 141–156 (2003).

    Article  Google Scholar 

  24. 24.

    Lowe, R.K.: Animation and learning: value for money. In: Beyond the comfort zone: Proceedings of the 21st ASCILITE Conference. pp 558–561 (2004)

  25. 25.

    Sanchez, C.A., Wiley, J.: Sex differences in science learning: closing the gap through animations. Learn. Individ. Differ. 20, 271–275 (2010).

    Article  Google Scholar 

  26. 26.

    Sanchez, C.A., Wiley, J.: The role of dynamic spatial ability in geoscience text comprehension. Learn. Instr. 31, 33–45 (2014).

    Article  Google Scholar 

  27. 27.

    Schiefele, U.: Interest, learning, and motivation. Edu. Psychol. 26, 299–323 (1991).

    Article  Google Scholar 

  28. 28.

    Hsu, Y.-S., Lin, Y.-H., Yang, B.: Impact of augmented reality lessons on students’ STEM interest. Res. Pract. Tech. Enhanced Learn. 12, 2 (2017).

    Article  Google Scholar 

  29. 29.

    Moreno, R., Mayer, R.E.: Learning science in virtual reality multimedia environments: role of methods and media. J. Educ. Psychol. 94, 598–610 (2002).

    Article  Google Scholar 

  30. 30.

    Buckley, P., Doyle, E.: Gamification and student motivation. Interact. Learn. Environ. 24, 1162–1175 (2016).

    Article  Google Scholar 

  31. 31.

    Dicheva, D., Dichev, C., Agre, G., Angelova, G.: Gamification in education: a systematic mapping study. J. Educ. Tech. Soc. 18, 75–88 (2015)

    Google Scholar 

  32. 32.

    McCombs, B.L.: Motivation and lifelong learning. Educ. Psychol. 26, 117–127 (1991).

    Article  Google Scholar 

  33. 33.

    Rashid, T., Asghar, H.M.: Technology use, self-directed learning, student engagement and academic performance: examining the interrelations. Comput. Hum. Behav. 63, 604–612 (2016).

    Article  Google Scholar 

  34. 34.

    Dackermann, U., Crews, K., Kasal, B., Li, J., Riggio, M., Rinn, F., Tannert, T.: In situ assessment of structural timber using stress-wave measurements. Mater. Struct. 47, 787–803 (2014).

    Article  Google Scholar 

  35. 35.

    Dickel, S., Franzen, M.: The “Problem of Extension” revisited: new modes of digital participation in science. J. Sci. Commun. (2016).

    Google Scholar 

  36. 36.

    Posada, J., Toro, C., Barandiaran, I., Oyarzun, D., Stricker, D., de Amicis, R., Pinto, E.B., Eisert, P., Dollner, J., Vallarino, I.: Visual computing as a key enabling technology for industrie 4.0 and industrial internet. IEEE Comput. Graph Appl. 35, 26–40 (2015).

    Article  Google Scholar 

  37. 37.

    Reid J.B., Rhodes D.H.: Digital System Models: an investigation of the non-technical challenges and research needs. In: 2016 Conference on Systems Engineering Research. Huntsville, AL (2016)

  38. 38.

    Schleich, B., Anwer, N., Mathieu, L., Wartzack, S.: Shaping the digital twin for design and production engineering. CIRP Ann. 66, 141–144 (2017).

    Article  Google Scholar 

  39. 39.

    Ciribini, A.L.C., Pasini, D., Tagliabue, L.C., Manfren, M., Daniotti, B., Rinaldi, S., De Angelis, E.: Tracking users’ behaviors through real-time information in BIMs: workflow for interconnection in the brescia smart campus demonstrator. Procedia Eng. 180, 1484–1494 (2017).

    Article  Google Scholar 

  40. 40.

    Donalek, C., Djorgovski, S.G., Cioc, A. et al.: Immersive and collaborative data visualization using virtual reality platforms. In: 2014 IEEE International Conference on Big Data (Big Data), pp. 609–614. IEEE, Washington (2014)

  41. 41.

    Olshannikova, E., Ometov, A., Koucheryavy, Y., Olsson, T.: Visualizing Big Data with augmented and virtual reality: challenges and research agenda. J. Big Data (2015).

    Google Scholar 

  42. 42.

    Napolitano, Rebecca, Blyth, Anna, Glisic, Branko: Virtual environments for visualizing structural health monitoring sensor networks, data, and metadata. Sensors 18, 243 (2018).

    Article  Google Scholar 

  43. 43.

    Alaloul, W.S., Liew, M.S., Zawawi, N.A.W.A., Mohammed, B.S.: Industry revolution IR 4.0: future opportunities and challenges in construction industry. In: MATEC Web Conf, vol. 203, p. 02010 (2018).

  44. 44.

    Dallasega, P., Rauch, E., Linder, C.: Industry 4.0 as an enabler of proximity for construction supply chains: a systematic literature review. Comput. Ind. 99, 205–225 (2018).

    Article  Google Scholar 

  45. 45.

    Wang, P., Wu, P., Wang, J., Chi, H.-L., Wang, X.: A critical review of the use of virtual reality in construction engineering education and training. Int. J. Environ. Res. Public Health 15, 1204 (2018).

    Article  Google Scholar 

  46. 46.

    Ying, H., Lee, S.: Survey of the research of ICT applications in the AEC industry: a view from two mainstream journals. In: Proceedings of the 16th International Conference on Construction Applications of Virtual Reality, pp. 471–483. The Hong Kong University of Science and Technology, Hong Kong (2016)

    Google Scholar 

  47. 47.

    Kramer, G., Walker, B., Bonebright, T., Cook, P., Flowers, J.H., Miner, N., Neuhoff, J.: Sonification report: status of the field and research Agenda. In: Prep Natl Sci Found Memb Int Community Audit Disp (1997)

  48. 48.

    Munzner, T.: Visualization Analysis and Design. CRC Press, Boca Raton (2015)

    Google Scholar 

  49. 49.

    Card, S.K., Mackinlay, J.: The structure of the information visualization design space. In: Proceedings of VIZ’97: Visualization Conference, Information Visualization Symposium and Parallel Rendering Symposium, pp. 92–99. IEEE Comput. Soc, Phoenix (1997)

  50. 50.

    Adcock, M., Barrass, S.: Cultivating Design Patterns for Auditory Displays. In: Proceedings of ICAD 04. Tenth Meeting of the International Conference on Auditory Display. Sydney, Australia (2004)

  51. 51.

    Barrass, S.: Sonification Design Patterns. In: Proceedings of the 9th International Conference on Auditory Display (ICAD2003). pp. 170–175. Boston, MA (2003)

  52. 52.

    Moore, B.C.J.: An Introduction to the Psychology of Hearing, 6th edn. Bingley, Emerald (2012)

    Google Scholar 

  53. 53.

    Ware, C.: Visual Thinking for Design: Active Vision, Attention Visual Queries, Gist, Visual Skills, Color, Narrative, Design. Morgan Kaufmann/Elsevier, Amsterdam (2008)

    Google Scholar 

  54. 54.

    Ferguson, S., Beilharz, K., Calò, C.A.: Navigation of interactive sonifications and visualisations of time-series data using multi-touch computing. J. Multimodal User Interfaces 5, 97–109 (2012).

    Article  Google Scholar 

  55. 55.

    Nesbitt, K.V.: A classification of multi-sensory metaphors for understanding abstract data in a virtual environment. In: 2000 IEEE Conference on Information Visualization. An International Conference on Computer Visualization and Graphics, pp. 493–498. IEEE Comput. Soc, London (2000)

  56. 56.

    Chandler, T., Cordeil, M., Czauderna, T. et al.: Immersive analytics. In: 2015 Big Data Visual Analytics (BDVA), pp. 1–8. IEEE, Hobart (2015)

  57. 57.

    Bell, B., Höllerer, T., Feiner, S.: An annotated situation-awareness aid for augmented reality. In: Proceedings of the 15th Annual ACM Symposium on User Interface Software and Technology—UIST 0’02, p. 213. ACM Press, Paris (2002)

  58. 58.

    Feiner, S., MacIntyre, B., Höllerer, T., Webster, A.: A touring machine: prototyping 3D mobile augmented reality systems for exploring the urban environment. Pers. Technol. 1, 208–217 (1997).

    Article  Google Scholar 

  59. 59.

    Höllerer, T., Feiner, S.: Mobile augmented reality. Telegeoinform. Locat. Comput. Serv. 21, 00533 (2004)

    Google Scholar 

  60. 60.

    Langlotz, T., Nguyen, T., Schmalstieg, D., Grasset, R.: Next-generation augmented reality browsers: rich, seamless, and adaptive. Proc. IEEE 102, 155–169 (2014).

    Article  Google Scholar 

  61. 61.

    Slay, H., Phillips, M., Vernik, R., Thomas, B.: Interaction modes for augmented reality visualization. In: Proceedings of the 2001 Asia-Pacific Symposium on Information Visualisation, vol 9, pp. 71–75. Australian Computer Society, Inc., Darlinghurst (2001)

  62. 62.

    Drascic, D., Milgram, P.: Perceptual issues in augmented reality. In: Bolas, M.T., Fisher, S.S., Merritt, J.O. (eds.) Stereoscopic Displays and Virtual Reality Systems III, pp. 123–134. San Jose, CA (1996)

  63. 63.

    Kruijff, E., Swan, J.E., Feiner, S.: Perceptual issues in augmented reality revisited. In: 2010 IEEE International Symposium on Mixed and Augmented Reality, pp. 3–12. IEEE, Seoul, Korea (South) (2010)

  64. 64.

    Pirolli, P., Card, S.: The sensemaking process and leverage points for analyst technology as identified through cognitive task analysis. In: Proceedings of International Conference on Intelligence Analysis (2005)

  65. 65.

    Yeh, K.-C., Tsai, M.-H., Kang, S.-C.: On-site building information retrieval by using projection-based augmented reality. J. Comput. Civ. Eng. 26, 342–355 (2012).

    Article  Google Scholar 

  66. 66.

    Riexinger, G., Kluth, A., Olbrich, M., Braun, J.-D., Bauernhansl, T.: Mixed reality for on-site self-instruction and self-inspection with building information models. Procedia CIRP 72, 1124–1129 (2018).

    Article  Google Scholar 

  67. 67.

    Kotranza, A., Lind, D.S., Lok, B.: Real-time evaluation and visualization of learner performance in a mixed-reality environment for clinical breast examination. IEEE Trans. Vis. Comput. Graph 18, 1101–1114 (2012).

    Article  Google Scholar 

  68. 68.

    Messadi, T., Newman, W.E., Braham, A., Nutter, D.: Cyber-innovation in the STEM classroom, a mixed reality approach. Creat. Educ. 09, 2385–2393 (2018).

    Article  Google Scholar 

  69. 69.

    Messadi, T., Newman, W.E., Braham, A., Nutter, D.: Immersive learning for sustainable building design and construction practices. J. Civ. Eng. Archit. (2017).

    Google Scholar 

  70. 70.

    Schmidt, E., Riggio, M., Laleicke, P., Barbosa, A., van den Wymelenberg, K.: How monitoring CLT buildings can remove market barriers and support designers in North America: an introduction to preliminary environmental studies. Portuguese J. Struct. Eng. III, 41–48 (2018)

    Google Scholar 

  71. 71.

    Sorin, E., Lanata, F., Boudaud, C.: Behaviour of timber structures under variable environment through long-term monitoring. In: World Conference on Timber Engineering (WCTE 2016). TU Verlag, Vienna (2016)

    Google Scholar 

  72. 72.

    Leyder, C., Chatzi, E., Frangi, A.: Structural health monitoring of an innovative timber building. In: Proceedings of the Second International Conference on Performance-based and Life-cycle Structural Engineering (PLSE 2015), pp. 1383–1392. School of Civil Engineering, The University of Queensland, Brisbane, QLD, Australia (2015)

  73. 73.

    Wang, J., Karsh, E., Finch, G., Cheng, M.: Field measurement of vertical movement and roof moisture performance of the Wood Innovation and Design Centre. In: World Conference on Timber Engineering, pp. 3120–3128 (2016)

  74. 74.

    Fast, P., Gafner, B., Jackson, R., Li, J.: Case study: an 18 storey tall mass timber hybrid student residence at the University of British Columbia, Vancouver. In: Proceedings of the World Conference on Timber Engineering (WCTE2016), Vienna, Austria, pp. 22–25 (2016)

  75. 75.

    Mustapha, G., Khondoker, K., Higgins, J.: Structural Performance Monitoring Technology and Data Visualization Tools and Techniques—Featured Case Study. UBC Tallwood House, Victoria (2018)

    Google Scholar 

  76. 76.

    Oregon State University—College of Forestry: Testing tall wood buildings (2017). Accessed 13 Dec 2018

  77. 77.

    Udomchoksakul, K.: The Peavy Hall OSU in Unreal Engine 4, (2018). Accessed 13 Dec 2018

  78. 78.

    Pike, W.A., Stasko, J., Chang, R., O’Connell, T.A.: The science of interaction. Inf. Vis. 8, 263–274 (2009).

    Article  Google Scholar 

  79. 79.

    Riggio, M., Shahbaz Badr, A., Prather, E.A., de Amicis, R.: Advancing AEC practice and data literacy using Digital Twins in Spatial Augmented Reality Environments. Monterrey, México (2018)

  80. 80.

    Hermann, T., Ritter, H.: Listen to your data: model-based sonification for data analysis. In: Lasker GE, Syed MR (eds.) Advances in intelligent computing and multimedia systems, Int. Inst. for Advanced Studies in System research and cybernetics, Windsor, Ontario, pp. 189–194 (1999)

    Google Scholar 

  81. 81.

    Hunt, A., Hermann, T.: The importance of interaction in sonification. In: Proceedings of the International Conference on Auditory Display (ICAD 2004), Sydney, Australia (2004)

  82. 82.

    Hassenzahl, M., Platz, A., Burmester, M., Lehner, K.: Hedonic and ergonomic quality aspects determine a software’s appeal. In: Proceedings of the SIGCHI Conference on Human factors in Computing Systems—CHI’00, pp. 201–208. ACM Press, The Hague (2000)

  83. 83.

    Sonderegger, A., Sauer, J.: The influence of design aesthetics in usability testing: effects on user performance and perceived usability. Appl. Ergon. 41, 403–410 (2010).

    Article  Google Scholar 

  84. 84.

    Tractinsky, N.: Aesthetics and apparent usability: empirically assessing cultural and methodological issues. In: Proceedings of the SIGCHI Conference on Human factors in Computing Systems—CHI’97, pp. 115–122. ACM Press, Atlanta, Georgia (1997)

  85. 85.

    Djamasbi, S., Siegel, M., Tullis, T., Dai, R.: Efficiency, trust, and visual appeal: usability testing through eye tracking. In: 2010 43rd Hawaii International Conference on System Sciences, pp. 1–10. IEEE, Honolulu (2010)

  86. 86.

    Castro-Alonso, J.C., Ayres, P., Paas, F.: Dynamic visualisations and motor skills. In: Huang, W. (ed.) Handbook of Human Centric Visualization, pp. 551–580. Springer, New York (2014)

    Google Scholar 

  87. 87.

    McAndrew, P., Clough, G.: Affective factors in learning with mobile devices. Big Issues in Mobile Learning: Report of a workshop by the Kaleidoscope Network of Excellence Mobile Learning Initiative, pp. 14–19 (2006)

  88. 88.

    Santos, M.E.C., Chen, A., Taketomi, T., Yamamoto, G., Miyazaki, J., Kato, H.: Augmented reality learning experiences: survey of prototype design and evaluation. IEEE Trans. Learn. Technol. 7, 38–56 (2014).

    Article  Google Scholar 

  89. 89.

    Welsh, E.T., Wanberg, C.R., Brown, K.G., Simmering, M.J.: E-learning: emerging uses, empirical results and future directions. Int. J. Train. Dev. 7, 245–258 (2003).

    Article  Google Scholar 

  90. 90.

    Akçayır, M., Akçayır, G.: Advantages and challenges associated with augmented reality for education: a systematic review of the literature. Educ. Res. Rev. 20, 1–11 (2017).

    Article  Google Scholar 

  91. 91.

    Pfeiffer, V.D.I., Gemballa, S., Bizer, B., Jarodzka, H., Imhof, B., Scheiter, K., Gerjets, P.: Enhancing students’ knowledge of biodiversity in a situated mobile learning scenario: using static and dynamic visualizations in field trips. In: Proceedings of the 8th International Conference on International Conference for the Learning Sciences, vol 2, pp. 204–212. International Society of the Learning Sciences, Utrecht (2008)

  92. 92.

    Gune, A., De Amicis, R., Simoes, B., Sanchez, C.A., Demirel, H.O.: Graphically hearing: enhancing understanding of geospatial data through an integrated auditory and visual experience. IEEE Comput. Graph Appl. 38, 18–26 (2018).

    Article  Google Scholar 

Download references


The Living Lab @ Peavy Hall project is conducted through the TallWood Design Institute with funding by the U.S. Department of Agriculture’s Agricultural Research Service (USDA ARS Agreement No. 58-0202-5-001). The material presented in this contribution is also based upon work that is supported by the National Institute of Food and Agriculture, U.S. Department of Agriculture, McIntire Stennis Project Under 1009740. The author Eric Andrew Prather was supported by AFRI ELI Grant No. 2018-67032-27704, from the USDA National Institute of Food and Agriculture. Findings and conclusions are those of the Authors and do not reflect opinions or views of the supporting agencies.

Author information



Corresponding author

Correspondence to Raffaele De Amicis.

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

De Amicis, R., Riggio, M., Shahbaz Badr, A. et al. Cross-reality environments in smart buildings to advance STEM cyberlearning. Int J Interact Des Manuf 13, 331–348 (2019).

Download citation


  • Augmented reality
  • Cyberlearning
  • Cross-reality environments
  • Data literacy
  • Structural health monitoring
  • Smart building
  • Virtual reality