Cross-reality environments in smart buildings to advance STEM cyberlearning

  • Raffaele De AmicisEmail author
  • Mariapaola Riggio
  • Arash Shahbaz Badr
  • Jason Fick
  • Christopher A. Sanchez
  • Eric Andrew Prather
Original Paper


Real time data associated with the Building Information Model plays a critical role in the interpretation of the built environment, which is particularly relevant as an increasing number of education facilities and institutions promote sustainable engineering practices and monitoring data available to the public. However, it is challenging for non-technical audiences to fully comprehend or use information concealed in scientific data related to the performance of structures and materials. It is especially difficult for them to connect these concepts to physical contexts and phenomena. In this paper, we present how cross-reality paradigms in Architecture, Engineering, and Construction, coupled with multimodal representation techniques, enhance data literacy in both professionals and laypeople alike. In particular, we present the design of a learning environment where cutting-edge holographic interfaces and display technologies are combined with sonified and visual data to create a more immersive environment for data analysis and exploration, empowering users with situated data awareness and new ways of understanding real-time data.


Augmented reality Cyberlearning Cross-reality environments Data literacy Structural health monitoring Smart building Virtual reality 



The Living Lab @ Peavy Hall project is conducted through the TallWood Design Institute with funding by the U.S. Department of Agriculture’s Agricultural Research Service (USDA ARS Agreement No. 58-0202-5-001). The material presented in this contribution is also based upon work that is supported by the National Institute of Food and Agriculture, U.S. Department of Agriculture, McIntire Stennis Project Under 1009740. The author Eric Andrew Prather was supported by AFRI ELI Grant No. 2018-67032-27704, from the USDA National Institute of Food and Agriculture. Findings and conclusions are those of the Authors and do not reflect opinions or views of the supporting agencies.


  1. 1.
    Porter, J.R., Morgan, J.A., Johnson, M.: Building automation and IoT as a platform for introducing STEM education in K-12. In: 2017 ASEE Annual Conference & Exposition (2017)Google Scholar
  2. 2.
    Dickel, S.: Trust in technologies? Science after de-professionalization. J. Sci. Commun. 15, 1–7 (2016). Google Scholar
  3. 3.
    Buckman, A.H., Mayfield, M., Beck, B.M.: What is a smart building? Smart Sustain. Built Environ. 3, 92–109 (2014). CrossRefGoogle Scholar
  4. 4.
    Carlson L.E, Sullivan J.F.: (1999) Hands-on engineering: learning by doing in the integrated teaching and learning program. Int. J. Eng. Educ. 15, 20–31Google Scholar
  5. 5.
    Petersen, J., Frantz, C., Shammin, R.: Using sociotechnical feedback to engage, educate, motivate and empower environmental thought and action. Solutions 5, 79–87 (2014)Google Scholar
  6. 6.
    Petersen, J.E., Murray, M.E., Platt, G., Shunturov, V.: Using buildings to teach environmental stewardship: real-time display of environmental performance as a mechanism for educating, motivating, and empowering the student body. In: Proc Green Campus VI Muncie Indiana (2007)Google Scholar
  7. 7.
    Petersen, J.E., Rosenberg Daneri, D., Frantz, C., Shammin, M.R.: Environmental dashboards: fostering pro-environmental and pro-community thought and action through feedback. In: Leal Filho, W., Mifsud, M., Shiel, C., Pretorius, R. (eds.) Handbook of Theory and Practice of Sustainable Development in Higher Education, pp. 149–168. Springer International Publishing, Cham (2017)CrossRefGoogle Scholar
  8. 8.
    Squire, K., Klopfer, E.: Augmented reality simulations on handheld computers. J. Learn. Sci. 16, 371–413 (2007). CrossRefGoogle Scholar
  9. 9.
    Kamarainen, A.M., Metcalf, S., Grotzer, T., Browne, A., Mazzuca, D., Tutwiler, M.S., Dede, C.: EcoMOBILE: integrating augmented reality and probeware with environmental education field trips. Comput. Educ. 68, 545–556 (2013). CrossRefGoogle Scholar
  10. 10.
    Cocciolo, A., Rabina, D.: Does place affect user engagement and understanding? Mobile learner perceptions on the streets of New York. J. Doc. 69, 98–120 (2013). CrossRefGoogle Scholar
  11. 11.
    Pimmer, C., Mateescu, M., Gröhbiel, U.: Mobile and ubiquitous learning in higher education settings: a systematic review of empirical studies. Comput. Hum. Behav. 63, 490–501 (2016). CrossRefGoogle Scholar
  12. 12.
    Rogers, Y., Price, S., Harris, E., Phelps, T., Underwood, M., Wilde, D., Smith, H., Weal, M.T.M.J., Michaelides, D.T.: Learning through digitally-augmented physical experiences: reflections on the Ambient Wood project (2002)Google Scholar
  13. 13.
    Brown, A., Green, T.: Virtual reality: low-cost tools and resources for the classroom. Tech. Trends. 60, 517–519 (2016). CrossRefGoogle Scholar
  14. 14.
    Brown, J.S., Collins, A., Duguid, P.: Situated Cognition and the Culture of Learning. Educ. Res. 18, 32–42 (1989). CrossRefGoogle Scholar
  15. 15.
    Smith, S.M., Vela, E.: Environmental context-dependent memory: a review and meta-analysis. Psychon. Bull. Rev. 8, 203–220 (2001). CrossRefGoogle Scholar
  16. 16.
    Chun, M.M., Jiang, Y.: Contextual Cueing: implicit Learning and Memory of Visual Context Guides Spatial Attention. Cogn. Psychol. 36, 28–71 (1998). CrossRefGoogle Scholar
  17. 17.
    Cook, A.E., Limber, J.E., O’Brien, E.J.: Situation-based context and the availability of predictive inferences. J. Mem. Lang. 44, 220–234 (2001). CrossRefGoogle Scholar
  18. 18.
    Smith, S.M., Glenberg, A., Bjork, R.A.: Environmental context and human memory. Mem. Cogn. 6, 342–353 (1978). CrossRefGoogle Scholar
  19. 19.
    Chi, M.T.H.: Active-Constructive-Interactive: a conceptual framework for differentiating learning activities. Top. Cogn. Sci. 1, 73–105 (2009). CrossRefGoogle Scholar
  20. 20.
    Jang, S., Vitale, J.M., Jyung, R.W., Black, J.B.: Direct manipulation is better than passive viewing for learning anatomy in a three-dimensional virtual reality environment. Comput. Edu. 106, 150–165 (2017). CrossRefGoogle Scholar
  21. 21.
    Goldstone, R.L., Son, J.Y.: The transfer of scientific principles using concrete and idealized simulations. J. Learn. Sci. 14, 69–110 (2005). CrossRefGoogle Scholar
  22. 22.
    Larkin, J., Simon, H.: Why a diagram is (Sometimes) worth ten thousand words. Cogn. Sci. 11, 65–100 (1987). CrossRefGoogle Scholar
  23. 23.
    Schnotz, W., Bannert, M.: Construction and interference in learning from multiple representation. Learn. Instr. 13, 141–156 (2003). CrossRefGoogle Scholar
  24. 24.
    Lowe, R.K.: Animation and learning: value for money. In: Beyond the comfort zone: Proceedings of the 21st ASCILITE Conference. pp 558–561 (2004)Google Scholar
  25. 25.
    Sanchez, C.A., Wiley, J.: Sex differences in science learning: closing the gap through animations. Learn. Individ. Differ. 20, 271–275 (2010). CrossRefGoogle Scholar
  26. 26.
    Sanchez, C.A., Wiley, J.: The role of dynamic spatial ability in geoscience text comprehension. Learn. Instr. 31, 33–45 (2014). CrossRefGoogle Scholar
  27. 27.
    Schiefele, U.: Interest, learning, and motivation. Edu. Psychol. 26, 299–323 (1991). CrossRefGoogle Scholar
  28. 28.
    Hsu, Y.-S., Lin, Y.-H., Yang, B.: Impact of augmented reality lessons on students’ STEM interest. Res. Pract. Tech. Enhanced Learn. 12, 2 (2017). CrossRefGoogle Scholar
  29. 29.
    Moreno, R., Mayer, R.E.: Learning science in virtual reality multimedia environments: role of methods and media. J. Educ. Psychol. 94, 598–610 (2002). CrossRefGoogle Scholar
  30. 30.
    Buckley, P., Doyle, E.: Gamification and student motivation. Interact. Learn. Environ. 24, 1162–1175 (2016). CrossRefGoogle Scholar
  31. 31.
    Dicheva, D., Dichev, C., Agre, G., Angelova, G.: Gamification in education: a systematic mapping study. J. Educ. Tech. Soc. 18, 75–88 (2015)Google Scholar
  32. 32.
    McCombs, B.L.: Motivation and lifelong learning. Educ. Psychol. 26, 117–127 (1991). CrossRefGoogle Scholar
  33. 33.
    Rashid, T., Asghar, H.M.: Technology use, self-directed learning, student engagement and academic performance: examining the interrelations. Comput. Hum. Behav. 63, 604–612 (2016). CrossRefGoogle Scholar
  34. 34.
    Dackermann, U., Crews, K., Kasal, B., Li, J., Riggio, M., Rinn, F., Tannert, T.: In situ assessment of structural timber using stress-wave measurements. Mater. Struct. 47, 787–803 (2014). CrossRefGoogle Scholar
  35. 35.
    Dickel, S., Franzen, M.: The “Problem of Extension” revisited: new modes of digital participation in science. J. Sci. Commun. (2016). Google Scholar
  36. 36.
    Posada, J., Toro, C., Barandiaran, I., Oyarzun, D., Stricker, D., de Amicis, R., Pinto, E.B., Eisert, P., Dollner, J., Vallarino, I.: Visual computing as a key enabling technology for industrie 4.0 and industrial internet. IEEE Comput. Graph Appl. 35, 26–40 (2015). CrossRefGoogle Scholar
  37. 37.
    Reid J.B., Rhodes D.H.: Digital System Models: an investigation of the non-technical challenges and research needs. In: 2016 Conference on Systems Engineering Research. Huntsville, AL (2016)Google Scholar
  38. 38.
    Schleich, B., Anwer, N., Mathieu, L., Wartzack, S.: Shaping the digital twin for design and production engineering. CIRP Ann. 66, 141–144 (2017). CrossRefGoogle Scholar
  39. 39.
    Ciribini, A.L.C., Pasini, D., Tagliabue, L.C., Manfren, M., Daniotti, B., Rinaldi, S., De Angelis, E.: Tracking users’ behaviors through real-time information in BIMs: workflow for interconnection in the brescia smart campus demonstrator. Procedia Eng. 180, 1484–1494 (2017). CrossRefGoogle Scholar
  40. 40.
    Donalek, C., Djorgovski, S.G., Cioc, A. et al.: Immersive and collaborative data visualization using virtual reality platforms. In: 2014 IEEE International Conference on Big Data (Big Data), pp. 609–614. IEEE, Washington (2014)Google Scholar
  41. 41.
    Olshannikova, E., Ometov, A., Koucheryavy, Y., Olsson, T.: Visualizing Big Data with augmented and virtual reality: challenges and research agenda. J. Big Data (2015). Google Scholar
  42. 42.
    Napolitano, Rebecca, Blyth, Anna, Glisic, Branko: Virtual environments for visualizing structural health monitoring sensor networks, data, and metadata. Sensors 18, 243 (2018). CrossRefGoogle Scholar
  43. 43.
    Alaloul, W.S., Liew, M.S., Zawawi, N.A.W.A., Mohammed, B.S.: Industry revolution IR 4.0: future opportunities and challenges in construction industry. In: MATEC Web Conf, vol. 203, p. 02010 (2018).
  44. 44.
    Dallasega, P., Rauch, E., Linder, C.: Industry 4.0 as an enabler of proximity for construction supply chains: a systematic literature review. Comput. Ind. 99, 205–225 (2018). CrossRefGoogle Scholar
  45. 45.
    Wang, P., Wu, P., Wang, J., Chi, H.-L., Wang, X.: A critical review of the use of virtual reality in construction engineering education and training. Int. J. Environ. Res. Public Health 15, 1204 (2018). CrossRefGoogle Scholar
  46. 46.
    Ying, H., Lee, S.: Survey of the research of ICT applications in the AEC industry: a view from two mainstream journals. In: Proceedings of the 16th International Conference on Construction Applications of Virtual Reality, pp. 471–483. The Hong Kong University of Science and Technology, Hong Kong (2016)Google Scholar
  47. 47.
    Kramer, G., Walker, B., Bonebright, T., Cook, P., Flowers, J.H., Miner, N., Neuhoff, J.: Sonification report: status of the field and research Agenda. In: Prep Natl Sci Found Memb Int Community Audit Disp (1997)Google Scholar
  48. 48.
    Munzner, T.: Visualization Analysis and Design. CRC Press, Boca Raton (2015)Google Scholar
  49. 49.
    Card, S.K., Mackinlay, J.: The structure of the information visualization design space. In: Proceedings of VIZ’97: Visualization Conference, Information Visualization Symposium and Parallel Rendering Symposium, pp. 92–99. IEEE Comput. Soc, Phoenix (1997)Google Scholar
  50. 50.
    Adcock, M., Barrass, S.: Cultivating Design Patterns for Auditory Displays. In: Proceedings of ICAD 04. Tenth Meeting of the International Conference on Auditory Display. Sydney, Australia (2004)Google Scholar
  51. 51.
    Barrass, S.: Sonification Design Patterns. In: Proceedings of the 9th International Conference on Auditory Display (ICAD2003). pp. 170–175. Boston, MA (2003)Google Scholar
  52. 52.
    Moore, B.C.J.: An Introduction to the Psychology of Hearing, 6th edn. Bingley, Emerald (2012)Google Scholar
  53. 53.
    Ware, C.: Visual Thinking for Design: Active Vision, Attention Visual Queries, Gist, Visual Skills, Color, Narrative, Design. Morgan Kaufmann/Elsevier, Amsterdam (2008)Google Scholar
  54. 54.
    Ferguson, S., Beilharz, K., Calò, C.A.: Navigation of interactive sonifications and visualisations of time-series data using multi-touch computing. J. Multimodal User Interfaces 5, 97–109 (2012). CrossRefGoogle Scholar
  55. 55.
    Nesbitt, K.V.: A classification of multi-sensory metaphors for understanding abstract data in a virtual environment. In: 2000 IEEE Conference on Information Visualization. An International Conference on Computer Visualization and Graphics, pp. 493–498. IEEE Comput. Soc, London (2000)Google Scholar
  56. 56.
    Chandler, T., Cordeil, M., Czauderna, T. et al.: Immersive analytics. In: 2015 Big Data Visual Analytics (BDVA), pp. 1–8. IEEE, Hobart (2015)Google Scholar
  57. 57.
    Bell, B., Höllerer, T., Feiner, S.: An annotated situation-awareness aid for augmented reality. In: Proceedings of the 15th Annual ACM Symposium on User Interface Software and Technology—UIST 0’02, p. 213. ACM Press, Paris (2002)Google Scholar
  58. 58.
    Feiner, S., MacIntyre, B., Höllerer, T., Webster, A.: A touring machine: prototyping 3D mobile augmented reality systems for exploring the urban environment. Pers. Technol. 1, 208–217 (1997). CrossRefGoogle Scholar
  59. 59.
    Höllerer, T., Feiner, S.: Mobile augmented reality. Telegeoinform. Locat. Comput. Serv. 21, 00533 (2004)Google Scholar
  60. 60.
    Langlotz, T., Nguyen, T., Schmalstieg, D., Grasset, R.: Next-generation augmented reality browsers: rich, seamless, and adaptive. Proc. IEEE 102, 155–169 (2014). CrossRefGoogle Scholar
  61. 61.
    Slay, H., Phillips, M., Vernik, R., Thomas, B.: Interaction modes for augmented reality visualization. In: Proceedings of the 2001 Asia-Pacific Symposium on Information Visualisation, vol 9, pp. 71–75. Australian Computer Society, Inc., Darlinghurst (2001)Google Scholar
  62. 62.
    Drascic, D., Milgram, P.: Perceptual issues in augmented reality. In: Bolas, M.T., Fisher, S.S., Merritt, J.O. (eds.) Stereoscopic Displays and Virtual Reality Systems III, pp. 123–134. San Jose, CA (1996)Google Scholar
  63. 63.
    Kruijff, E., Swan, J.E., Feiner, S.: Perceptual issues in augmented reality revisited. In: 2010 IEEE International Symposium on Mixed and Augmented Reality, pp. 3–12. IEEE, Seoul, Korea (South) (2010)Google Scholar
  64. 64.
    Pirolli, P., Card, S.: The sensemaking process and leverage points for analyst technology as identified through cognitive task analysis. In: Proceedings of International Conference on Intelligence Analysis (2005)Google Scholar
  65. 65.
    Yeh, K.-C., Tsai, M.-H., Kang, S.-C.: On-site building information retrieval by using projection-based augmented reality. J. Comput. Civ. Eng. 26, 342–355 (2012). CrossRefGoogle Scholar
  66. 66.
    Riexinger, G., Kluth, A., Olbrich, M., Braun, J.-D., Bauernhansl, T.: Mixed reality for on-site self-instruction and self-inspection with building information models. Procedia CIRP 72, 1124–1129 (2018). CrossRefGoogle Scholar
  67. 67.
    Kotranza, A., Lind, D.S., Lok, B.: Real-time evaluation and visualization of learner performance in a mixed-reality environment for clinical breast examination. IEEE Trans. Vis. Comput. Graph 18, 1101–1114 (2012). CrossRefGoogle Scholar
  68. 68.
    Messadi, T., Newman, W.E., Braham, A., Nutter, D.: Cyber-innovation in the STEM classroom, a mixed reality approach. Creat. Educ. 09, 2385–2393 (2018). CrossRefGoogle Scholar
  69. 69.
    Messadi, T., Newman, W.E., Braham, A., Nutter, D.: Immersive learning for sustainable building design and construction practices. J. Civ. Eng. Archit. (2017). Google Scholar
  70. 70.
    Schmidt, E., Riggio, M., Laleicke, P., Barbosa, A., van den Wymelenberg, K.: How monitoring CLT buildings can remove market barriers and support designers in North America: an introduction to preliminary environmental studies. Portuguese J. Struct. Eng. III, 41–48 (2018)Google Scholar
  71. 71.
    Sorin, E., Lanata, F., Boudaud, C.: Behaviour of timber structures under variable environment through long-term monitoring. In: World Conference on Timber Engineering (WCTE 2016). TU Verlag, Vienna (2016)Google Scholar
  72. 72.
    Leyder, C., Chatzi, E., Frangi, A.: Structural health monitoring of an innovative timber building. In: Proceedings of the Second International Conference on Performance-based and Life-cycle Structural Engineering (PLSE 2015), pp. 1383–1392. School of Civil Engineering, The University of Queensland, Brisbane, QLD, Australia (2015)Google Scholar
  73. 73.
    Wang, J., Karsh, E., Finch, G., Cheng, M.: Field measurement of vertical movement and roof moisture performance of the Wood Innovation and Design Centre. In: World Conference on Timber Engineering, pp. 3120–3128 (2016)Google Scholar
  74. 74.
    Fast, P., Gafner, B., Jackson, R., Li, J.: Case study: an 18 storey tall mass timber hybrid student residence at the University of British Columbia, Vancouver. In: Proceedings of the World Conference on Timber Engineering (WCTE2016), Vienna, Austria, pp. 22–25 (2016)Google Scholar
  75. 75.
    Mustapha, G., Khondoker, K., Higgins, J.: Structural Performance Monitoring Technology and Data Visualization Tools and Techniques—Featured Case Study. UBC Tallwood House, Victoria (2018)Google Scholar
  76. 76.
    Oregon State University—College of Forestry: Testing tall wood buildings (2017). Accessed 13 Dec 2018
  77. 77.
    Udomchoksakul, K.: The Peavy Hall OSU in Unreal Engine 4, (2018). Accessed 13 Dec 2018
  78. 78.
    Pike, W.A., Stasko, J., Chang, R., O’Connell, T.A.: The science of interaction. Inf. Vis. 8, 263–274 (2009). CrossRefGoogle Scholar
  79. 79.
    Riggio, M., Shahbaz Badr, A., Prather, E.A., de Amicis, R.: Advancing AEC practice and data literacy using Digital Twins in Spatial Augmented Reality Environments. Monterrey, México (2018)Google Scholar
  80. 80.
    Hermann, T., Ritter, H.: Listen to your data: model-based sonification for data analysis. In: Lasker GE, Syed MR (eds.) Advances in intelligent computing and multimedia systems, Int. Inst. for Advanced Studies in System research and cybernetics, Windsor, Ontario, pp. 189–194 (1999)Google Scholar
  81. 81.
    Hunt, A., Hermann, T.: The importance of interaction in sonification. In: Proceedings of the International Conference on Auditory Display (ICAD 2004), Sydney, Australia (2004)Google Scholar
  82. 82.
    Hassenzahl, M., Platz, A., Burmester, M., Lehner, K.: Hedonic and ergonomic quality aspects determine a software’s appeal. In: Proceedings of the SIGCHI Conference on Human factors in Computing Systems—CHI’00, pp. 201–208. ACM Press, The Hague (2000)Google Scholar
  83. 83.
    Sonderegger, A., Sauer, J.: The influence of design aesthetics in usability testing: effects on user performance and perceived usability. Appl. Ergon. 41, 403–410 (2010). CrossRefGoogle Scholar
  84. 84.
    Tractinsky, N.: Aesthetics and apparent usability: empirically assessing cultural and methodological issues. In: Proceedings of the SIGCHI Conference on Human factors in Computing Systems—CHI’97, pp. 115–122. ACM Press, Atlanta, Georgia (1997)Google Scholar
  85. 85.
    Djamasbi, S., Siegel, M., Tullis, T., Dai, R.: Efficiency, trust, and visual appeal: usability testing through eye tracking. In: 2010 43rd Hawaii International Conference on System Sciences, pp. 1–10. IEEE, Honolulu (2010)Google Scholar
  86. 86.
    Castro-Alonso, J.C., Ayres, P., Paas, F.: Dynamic visualisations and motor skills. In: Huang, W. (ed.) Handbook of Human Centric Visualization, pp. 551–580. Springer, New York (2014)CrossRefGoogle Scholar
  87. 87.
    McAndrew, P., Clough, G.: Affective factors in learning with mobile devices. Big Issues in Mobile Learning: Report of a workshop by the Kaleidoscope Network of Excellence Mobile Learning Initiative, pp. 14–19 (2006)Google Scholar
  88. 88.
    Santos, M.E.C., Chen, A., Taketomi, T., Yamamoto, G., Miyazaki, J., Kato, H.: Augmented reality learning experiences: survey of prototype design and evaluation. IEEE Trans. Learn. Technol. 7, 38–56 (2014). CrossRefGoogle Scholar
  89. 89.
    Welsh, E.T., Wanberg, C.R., Brown, K.G., Simmering, M.J.: E-learning: emerging uses, empirical results and future directions. Int. J. Train. Dev. 7, 245–258 (2003). CrossRefGoogle Scholar
  90. 90.
    Akçayır, M., Akçayır, G.: Advantages and challenges associated with augmented reality for education: a systematic review of the literature. Educ. Res. Rev. 20, 1–11 (2017). CrossRefGoogle Scholar
  91. 91.
    Pfeiffer, V.D.I., Gemballa, S., Bizer, B., Jarodzka, H., Imhof, B., Scheiter, K., Gerjets, P.: Enhancing students’ knowledge of biodiversity in a situated mobile learning scenario: using static and dynamic visualizations in field trips. In: Proceedings of the 8th International Conference on International Conference for the Learning Sciences, vol 2, pp. 204–212. International Society of the Learning Sciences, Utrecht (2008)Google Scholar
  92. 92.
    Gune, A., De Amicis, R., Simoes, B., Sanchez, C.A., Demirel, H.O.: Graphically hearing: enhancing understanding of geospatial data through an integrated auditory and visual experience. IEEE Comput. Graph Appl. 38, 18–26 (2018). CrossRefGoogle Scholar

Copyright information

© Springer-Verlag France SAS, part of Springer Nature 2019

Authors and Affiliations

  1. 1.Electrical Engineering and Computer Science, College of EngineeringOregon State UniversityCorvallisUSA
  2. 2.Wood Science and Engineering, College of ForestryOregon State UniversityCorvallisUSA
  3. 3.Music Technology and Production, College of Liberal ArtsOregon State UniversityCorvallisUSA
  4. 4.School of Psychological Science, College of Liberal ArtsOregon State UniversityCorvallisUSA

Personalised recommendations