Introduction

Science textbooks have traditionally included more information about inventions or products as opposed to science processes,14,15 and often times, the way the information is presented makes it challenging for students to understand, especially if they have low reading skills.8 Over the years, digital technology has found its avenue in science, technology, engineering, and mathematics (STEM) education. More schools have started incorporating digital presentations in classroom settings. While current software such as Adobe PDF complement traditional STEM education, they are comprised of static 2D elements and often mimic non-digital teaching techniques. Basic text and figure layout structure in academic lessons provide students less motivation to dive deeper into scientific content thereby affecting overall academic performance.18 This aspect turns lectures into monologues and affects student engagement levels and comprehension abilities.32

US Department of Education identifies engagement and comprehension as key factors contributing to student success (2026 STEM Vision).30 Academic engagement is multidimensional and examined within three components including behavioral (on-task behavior, persistence, class participation), cognitive (analytical aspects to solve problems), and affective (attitude and interest level).3 Drawing on knowledge about achievement motivation, engaging students in school is greatly influenced by pedagogical approaches7,28). For example, evidence suggests that ICT nurtures learning through a student-centered approach that encourages active learning.22 In such a climate, students report higher levels of engagement compared to students in a teacher-centered approach because they feel more competent in their individual abilities while also feeling a sense of belonging.9 When students have the opportunity to construct their own meanings through different ways of learning, their engagement levels rise10). While there is a dearth of research about the relationship between digital technology and engagement, teachers have reported higher levels of engagement for their students when using technology as learning tools.26,28

Mixed reality (MR) technologies, often combining aspects of virtual reality (VR) or augmented reality (AR), are becoming increasingly popular in classrooms because they behaviorally, cognitively, and affectively engage students. MR blends real and virtual worlds using computer-generated objects. The two worlds are “mixed” together to create a realistic environment, so that users can seamlessly navigate content and “physically” interact with real and virtual objects with the aid of a digital display device (e.g., a smart phone). For instance, middle schoolers who learned gravity and planetary motion using a virtual reality environment revealed significant learning gains, higher levels of engagement, and more positive attitudes toward sciences while using an immerse, whole-body interactive simulation rather than a desktop version of the same simulation.20 In another experiment, eighth graders exhibited significant improvement in their learning attitudes and outcomes for magnetic field instruction using AR technology instead of traditional lab equipment.4 Similarly, AR technologies have shown to improve student learning and retention,16 especially in STEM areas such medical surgery education,17 molecular modeling,27 and astronomy.21 AR technologies also simplify multidimensional concept maps and enable students to stay motivated and understand content better.6 The increase of online instruction requires that digital platforms be developed and validated by looking at engagement levels and academic achievement in comprehension.

In this study, we investigate the effectiveness of MR technology created by Zyndo LLC for Biomedical Engineering and Immunology disciplines. The Zyndo platform creates educational stories, which are referred to as Z-books, in an interactive and immersive MR environment. To improve upon the static 2D elements of PDFs, our MR presentation combines 2D text and images to form a 3D space containing interactive models and objects students can interact with using a smart device. An important innovation of Z-books is that users can freely navigate content in a non-linear way with multiple story path options and advanced triggering of special objects, thereby supporting novice and advanced users alike. Offering a wide selection of language, audio, tactile, and visual capabilities, this MR presentation may provide a flexible digital technology to benefit all types of learners. Comprehension strategies, such as reading and taking margin notes are applied using advanced technology as students are encouraged to explore the MR presentation in a way that challenges their critical and relational thinking in a 3D relatable way. The video-game-adventure-like setting of this MR presentation especially appeals to younger students, who are immersed in a technology-driven world.1

This pilot study was performed on higher education students to first verify the effects of MR on students who are more adept to learning, though are recognized as demographically different from undergraduate or K-12 students. A controlled, randomized group of STEM professionals (N = 20) from an academic, biomedical engineering research center at Harvard Medical School were given two articles to read from the journal Science11,31 as summarized in Z-book or PDF format. While the STEM professionals viewed the article presentations, we recorded their total time spent doing so. Once the STEM professionals finished reading one article in Z-book or PDF format, they were given the other article presented in the other format. After the STEM professionals finished reading either article, they were given a multiple-choice sheet containing comprehension questions that corresponded to the article, followed by survey engagement questions. These questions were developed by a committee of by the senior authors with domain/educational expertise. When the STEM professionals finished reading and experiencing the articles and PDF presentations, a focus group took place and researchers took notes using an observation protocol.

Methods

Before Experimental Portion

Prior to the experiment, two scientific articles, one on Immunology (Anticancer immunotherapy by CTLA-4 blockade relies on the gut microbiota. Science, Nov 27; 2015, https://doi.org/10.1126/science.aad1329) and the other on Bioengineering (First-in-Human Testing of a Wirelessly Controlled Drug Delivery Microchip. Science Translational Medicine, Feb 16, 2012 DOI: https://doi.org/10.1126/scitranslmed.3003276) had been turned into interactive Z-book presentations. Both presentations were recreated, as faithfully as possible, in PDFs. Participants were found in two ways. The first was direct advertising, in which we posted flyers in both Shriner’s Hospitals for Children and Massachusetts General Hospital. An email asking for participants was also sent on an internal hospital listing. The second way was the “snowball method,” meaning we used the participants we initially found to find more participants. All 20 participants were aged 21–40 and received an advanced scientific degree (BS, MS, PhD, MD, MD/PhD). We first met with each participant in a private session, discussing what the study involved. After this, we gave each participant a sheet asking for background information and asked them to fill it out. With all the questions answered, the experimental portion began.

During Experimental Portion

All four versions of the articles (Z1, Z2, PDF1, PDF2) were loaded onto tablet devices. Each participant was given the first tablet and was asked to read the article presentation. While the participant read, we observed and answered basic questions and comments. We also timed the participant on a stopwatch from the start of the article to the end. When the participants indicated they were finished reading, they were given a sheet containing comprehension questions for the article they read. Five multiple choice comprehension questions for each article were written and provided to test whether participants learned more depending on the presentation format. Below the comprehension questions were survey questions, which were meant to provide a quantitative measure of engagement. We used a combination of a pre-existing standard scale, called the System Usability Scale, and our own questions.29 Survey questions were meant to represent very basic criteria that are important for students to know.

When the participants finished answering, we gave them the second tablet loaded to the other article presentation in the other format. For each participant, the order of article and format was rotated. (So, Z1/PDF2 then PDF1/Z2 then Z2/PDF1 then PDF2/Z1 then the same rotation again.) In this way, we ensured that the contents of the article and the order they were read did not impact the experiment. When the participants finished reading, they were given a sheet containing comprehension questions for the second article followed by the same survey questions that followed the first. After participants finished answering these questions, we moved on to an open-ended discussion. This lasted until the end of each participant’s allotted time. We tried to get each participant to talk about their experience, likes, dislikes, etc. While the conversation was free-flowing, we had a list of Usability Questions adapted from standard usability testing surveys. We took notes on their comments, writing down verbatim quotes when possible and later typing up these quotes in combination with the background information.

After Experimental Portion

Following the completion of the experimental aspect, we marked each comprehension question correct/incorrect, totaled correct answers, etc. We then created an Excel file that contained every participant’s correct/incorrect answers, the total correct answers for each file, file type, and the ratings each participant gave in the survey questions.

Results

MR Presentation Provides an Immersive and Interactive Learning Environment

Among other features, MR presentation improves upon standard 2D learning environments through an interactive 3D user interface. Traditional presentation software have users swipe or scroll through pages of static, non-interactive elements, which are suboptimal for maximizing students’ learning (Fig. 1a). However, MR presentation can take these 2D images and position them within an immersive, infinite 3D environment (Fig. 1b). Users are then able to explore the presentation on their own through a virtual camera. Our MR presentation is built using video gaming language and incorporates multiple story paths for users to choose the information they want to hear more about. Moreover, the audiovisual interface is integrative, allowing for an accessible interactive presentation of nearly any information in a 3D environment with the inclusion of complex 2D or 3D elements, a robust synced audio engine, and advanced triggering of new content via nuanced user interactions (Fig. 1c). This multi-sensory engagement of touch, sight, and sound creates a deeply immersive experience for users of varying academic abilities and needs. Furthermore, built-in language translation capabilities allow our MR presentation software to reach international users. The audio engine is especially multifaceted, as it combines detailed narrations, sound effects, and pre-developed soundtracks (Fig. 1d). With its unique 2D-to-3D parallax system and zooming function, our MR presentation effectively brings stories to life by making static 2D images look 3D (Fig. 1e).

Figure 1
figure 1

Z-books’ 3D user interface. (a) A “traditional” published layout of image panels is depicted by pentagons. By and large, the current e-publishing paradigm simply mimics a traditional 2D reading experience: a user swipes through pages or scrolls through static, non-interactive elements. (b) A simple example of the same visual panels displayed by a Z-book-based e-publishing user interface is shown. Each panel is discretely divided as the same 2D image and imported into an infinite 3D environment. A virtual camera “flies” a user to each panel, allowing for a guided view in an immersive 3D world. (c) Our integrated menu system (depicted on a tablet viewer) allows for user-specified language settings to enable learning in many countries. Our multi-language translation functionality is cloud-based and can be updated remotely with new languages added on-the-fly. (d) Our custom “trigger-based” high-fidelity audio engine allows for interaction between complex narration, sound effects, and soundtracks. (e) Z-books’ proprietary 2D to 3D parallax interactivity creates interactive 3D elements from industry-standard 2D assets and places them into a limitless 3D environment with other 3D elements. These features make stories and learning come to life with a unique, interactive viewing experience.

Full size image

MR Presentation Improved Nearly all Immunology and Bioengineering Engagement Individual Question Scores

To first explore the affect of participants, they completed an academic engagement survey immediately following their digestion of both the written content and Zyndo MR created content, which measured their affective, cognitive, and behavioral engagement. The 10-item instrument was developed based on pre-existing measures of engagement,5,13 adapted for STEM graduate students. Each of the items were rated on a 5-point-scale where 1 = strongly disagree and 5 = strongly agree (see Appendix A). We found that generally, users preferred the MR presentation over traditional PDFs (Fig. 2). In almost all metrics related to engagement, MR presentation users reported that they were more engaged with and better enjoyed both the Immunology and Bioengineering MR presentations than did the PDF users. The only engagement question in which PDF users outscored MR presentation users was pertaining to the Bioengineering presentation’s navigation, with an improvement of − 2.70%. Still, this difference had the lowest magnitude of all the engagement questions for both Immunology and Bioengineering articles, showing that MR presentation has a significant positive effect on engagement for STEM learners. As they relate to affective engagement (i.e. I am likely to recommend this presentation software to a friend or colleague), MR presentation users felt more strongly that they would like to read on, create with, and recommend the software than PDF users did for the Immunology article. This finding suggests MR environments are more affectively engaging than PDFs.

Figure 2
figure 2

Metrics of engagement increased. Nearly all questions of engagement, enjoyment, and recommendations of use were answered very positively when using Zyndo.

Full size image

MR Presentation Significantly Improved Overall Immunology and Bioengineering Engagement

After all the individual engagement scores for each MR presentation or PDF group were compiled, we next explored participant affect as measured by total engagement. Total engagement scores were calculated by taking the average and then dividing by the maximum score of 5 to obtain a percentage (Fig. 3). For the Immunology article, MR presentation users had an average engagement score of 82.67% (σ = 0.28), while PDF users had an average engagement score of 70.91% (σ = 0.47). A t test was performed on the raw Immunology engagement and enjoyment data, obtaining a value of 0.00053. This supported our hypothesis that the MR environment would significantly improve users’ engagement while viewing presentations, which has been shown to increase overall learning outcomes related to ease of use, amount of effort exerted, and enjoyment. Similarly, for the Bioengineering article, MR presentation users had an average engagement score of 78.10% (σ = 0.42), while PDF users had an average engagement score of 65.85% (σ = 0.43 A t test was performed on the raw Bioengineering engagement and enjoyment data, obtaining a value of 0.000095. Like the Immunology article results, this supported our hypothesis that the MR presentation would significantly improve users’ learning outcomes. Overall, MR presentation users had ~ 17% overall increases in a composite endpoint of engagement (16.57% between Immunology groups and 18.60% between Bioengineering groups).

Figure 3
figure 3

Mixed reality presentation significantly improved user engagement. Mixed reality presentation users had overall increases in a composite endpoint of engagement and enjoyment that was independent of content viewed (12% or 13% total increase). This equates to ~ 17% improvement in this measure.

Full size image

MR Presentation Improved ovErall Bioengineering Comprehension, Yet Failed to Improve Immunology Comprehension

We then examined cognitive effectiveness, specifically comprehension between MR presentation and PDF users. This was measured based on average scores from multiple choice comprehension questions about content of the articles (see Appendix A). Comprehension trended toward significant improvement from using the MR presentation in a content-dependent manner (Fig. 4). For the Immunology article, comprehension was essentially identical between users who viewed the MR presentation versus PDFs, showing an average score of 0.70 across all 5 questions (σ = 0.24 for MR presentation users and 0.21 for PDF users). A t test was performed on the raw Immunology comprehension data, obtaining a value of 1.00. This suggests that MR presentations do not always improve comprehension of the learning material. For the Bioengineering article, MR presentation users, who had an overall average score of 0.76 (σ = 0.21), exhibited significantly greater comprehension than PDF users, who had an overall average score of 0.58 (σ = 0.26). This revealed an increase in group average test score of +18 points (31% improvement) for the Bioengineering article presented through MR presentation versus PDFs. A t test was performed on the raw Bioengineering comprehension data, obtaining a value of 0.056. Overall, more participants who viewed the MR presentation and not the PDFs for the Bioengineering article answered correctly for 4 of the 5 questions. The fifth question showed equal comprehension, so PDF users never outperformed MR presentation users.

Figure 4
figure 4

Mixed reality presentation sometimes improved comprehension. Comprehension trended toward significant improvement in a content-dependent manner. An increased group average test score (+18 points; 31%) was observed in Bioengineering content using the mixed reality presentation. Immunology content was comparable between the two presentation delivery platforms.

Full size image

MR Presentation Increased Overall Time Spent Viewing Bioengineering Content, Yet Failed to Increase Overall Time Spent Viewing Immunology Content

Finally, behavioral effectiveness was also studies by measuring time-on-task. Using a stopwatch, time spent viewing the material on the two presentation formats was recorded. As a group, there was a strong trend towards more time spent viewing content on the presentations (p < 0.20) in a content-dependent manner (Fig. 5). For the Immunology article, MR presentation users spent slightly less time viewing the presentation with an average of 246.5 min (σ = 66.43) compared to PDF users with 252.1 min (σ = 102.97). A t-test was performed on the raw Immunology time data, obtaining a value of 0.89. These results suggest that MR presentation may detract from users’ interests in the learning material. For the Bioengineering article, MR presentation users spent significantly more time viewing the presentation with an average of 220.7 min (σ = 50.77) compared to PDF users with 189.9 min (σ = 67.81). On an individual level, about 66% of MR presentation users spent a greater amount of time with content. MR presentation users also spent a median of about ½ minute more time with the content than PDF users (range 3–4 min). A t-test was performed on the raw Bioengineering time data, obtaining a value of 0.27. In contrast to the Immunology article results, the time spent viewing the Bioengineering article increased from using the MR presentation than the PDFs, allowing users to better interact with and experience the content. We also examined whether age was related to the differences in times of use to account for age-dependent attention spans, but we found no correlation. These data suggest a potential mechanism for improved comprehension of Bioengineering content that is associated with increased time of viewing. Since this study was performed in a biomedical engineering research center, we speculate that the familiarity with bioengineering concepts helped with immersion into this specific article as a reflection of participant backgrounds.

Figure 5
figure 5

Users sometimes spent more time with the mixed reality presentation. As a group, there was a trend towards more time spent viewing Bioengineering content on the mixed reality presentation as a group, but not so in Immunology. This trend may be associated with improved comprehension of this group. No correlation was seen to age and may be impacted by limited sample size or a function of the specialized topic that was presented for this audience who had advanced education in scientific disciplines at the outset.

Full size image

Discussion

Comprehension can be defined as how people construct meaning from reading and studying written and oral language. Measurements that study active comprehension include patterns of eye movement, reading time, and think aloud protocols. Comprehension is also evaluated using measurements after content is absorbed through recall tests, summaries of information, and answering questions.24 Adequate comprehension of scientific content lays the foundation for advanced STEM learning. A STEM-focused approach to teaching and learning was founded on the belief that students could comprehend and apply the multiple disciplines at once when asked to consider real-world problems.2,23,25 From a larger perspective, when students are presented with real world problems to solve, they must consider solutions that require problem solving and innovation to compete in the international career arena. Innovation is nurtured through STEM programs when they are committed to teaching and learning situations that welcome digital visual technologies that are developmentally appropriate for accurate, in-depth comprehension.12,19

Z-books have analytical tools that can potentially guide curriculum development. Robust analytics features are critical for better understanding students’ responses to scientific content. The exact mechanisms and most helpful features that lead to the success of educational software are unclear. Most educational software is evaluated largely on correlations with test scores. Understanding the way in which users interact with the software may help determine which aspects are most important in retention and engagement. Publishing software analytics typically track only whether the user experience is positive, and not the actual interactions with features that directly lead to user happiness. Z-books contain analytic tools (Supplementary Table 1) that, in addition to user geography, device type, total views, and other analytic data, may also provide metrics aligned with optimizing STEM curricula. Among other things, instructors can see how much of each assignment is actually viewed, time spent within the app, and individual interactions within each section, object, problem, or 3D model. This unique information could potentially allow instructors to determine where students might be stuck, objects with which the most successful students interact, and numerous other data points around which curricula and lesson plans can be refined. New analytical tools, for instance, eye movement tracking that can more precisely quantify student observation, are under development for next generation Z-books.

Z-books have cross-platform capability that can help keep school district costs down and appeal to publishers for further academic incorporation and global distribution. Z-books can be used on computers, TVs, and mobile devices, allowing for cheaper incorporation and a much wider potential base of users. Already, half of all teen smartphone owners are “cell-mostly” internet users. This multi-platform capability also allows publishers to take advantage of the rapid growth of mobile devices to access STEM journals, which is occurring fastest among physicians and healthcare professionals. Furthermore, (though not studied herein) Z-books have integrated language translation that can also service the language gap in education and help smooth the incorporation of new non-native speakers into mixed-language classrooms by allowing instructors to create a single lesson for all students. This study explored this concept, albeit in a limited sample size (N = 20). Z-books are inherently scalable to greater sizes of classrooms and are limited by access to display devices, which may be challenging in resource-limited settings. Higher education students can be an initial demographic to focus on for advanced education and training methods based on these initial results. Our study used a journal article, which is typically narrow in focus, therefore evaluating a diversity of content will also be important moving forward. Results that are equated with comprehension of research article may not translate to other content such as entire introductory course in a subject and is worthy of further investigation by diversifying content materials. For younger demographics, larger studies with more quantitative endpoints and a diverse study body will be critical to perform to evaluate the benefits of Z-books on K-12 students who are developmentally different in regards to learning and brain function.

The results of this exploratory pilot study mainly focused on engagement and comprehension as indicative measures of overall learning outcomes for users of Z-books. Based on the results of this exploratory pilot study, which provided us with a baseline grasp of what metrics are important and how much improvement we should expect using MR platforms over PDFs, we propose using Z-books analytics in future studies while expanding the scale of study, diversifying the learning content and student population, and adding more short- and long-term quantifiable metrics to learning outcomes. Z-books unique back-end analytics platform can extend the scope of our metrics beyond levels of engagement, comprehension, and time of use. Further research involving Z-book’s analytics will not only improve STEM education for people of all ages, but also enhance the capabilities of the Z-book platform by providing more data on which analytical tools are crucial for allowing teachers to effectively gauge student learning.