Introduction

Group-based learning is frequently used in medical and healthcare education. This includes having students practice clinical skills in groups (O’Dunn-Orto et al. 2012; Tolsgaard et al. 2013), where they actively take part in the learning process (Kolb and Kolb 2005). The advantages of group practice are that students can learn from observing each other and help each other when they practice clinical skills, and it can be less resource intensive because it requires less facilitator or teacher time than does one-to-one instruction (Wulf et al. 2010; Räder et al. 2014).

One of the clinical skills that can be practiced in groups is systematic clinical observation, using the airway, breathing, circulation, disability and exposure (ABCDE) approach (Resuscitation Council UK 2015). A range of methods have been used to teach this approach (Thim et al. 2012; Smith and Bowden 2017), but we have not found any studies on the effect of using virtual reality (VR; Bremer et al. 2020). VR is “a computer generated digital environment that can be experienced and interacted with as if that environment was real” (Jerald 2015). With the use of a head-mounted display (HMD) described as an immersive technology, the user becomes completely occluded from reality and can experience a sense of presence in the virtual environment (Martirosov and Kopecek 2017). The user can interact with objects in the virtual environment using hand controllers.

To be able to practice as a group in VR, online, real-time, multiplayer features are necessary so that the group can be present and collaborate in the same environment (Liaw et al. 2018). Simulation-based activity is well suited for group learning (Hughes et al. 2016), and VR can potentially provide effective simulation-based group learning (McGrath et al. 2018; Kyaw et al. 2019). However, it is difficult to draw conclusions about the effect of different types of multiplayer VR applications because of methodological limitations in the studies (Liaw et al. 2018; Kyaw et al. 2019), and more research is, therefore, requested. VR has some disadvantages compared to real life when practicing clinical skills, and it can be expected that it would not give an effect above that of traditional practice with physical equipment, which has been widely used and developed over a long time. This is an argument for choosing a non-inferior design.

Therefore, the aim was to investigate if group self-practice skill training of the ABCDE approach in a multiplayer, immersive, interactive virtual reality application resulted in a non-inferior learning outcome in first-year medical and nursing students compared to using physical equipment.

Method

Study design

This was a non-inferior, parallel-group, open randomized controlled trial (RCT), which was conducted as part of a large trial also recruiting students to another RCT on the effect of individual practice simultaneously (Berg and Steinsbekk 2020).

The study was approved by the Norwegian Centre for Research Data (reference number 535088) and conducted from August to September 2019.

Participants and recruitment

The inclusion criteria were first-year medical and nursing students who had started their study no later than 2 months before this study was conducted at one of the three campus sites of the Faculty of Medicine and Health Sciences, Norwegian University of Science and Technology.

The participants took part in a teaching programme integrated into their curriculum, and as part of this study, they were randomized in groups of three to take part in two different types of practicing the ABCDE approach. The group size of three was chosen based on research showing no difference in learning outcomes between groups of three, four or five (Rezmer et al. 2011), and when self-practicing, smaller groups would not be more resource demanding. The students were informed both verbally and in writing a week before and at the start of the teaching session, that they could consent to participate in the study at the end of the session. Those who attended the session were eligible, and those who consented were included.

Randomization and allocation

Randomization had to consider the practical organization of the teaching, which included that batches of students had to be allocated to participate at separate times. To randomize these students, separate randomization lists were prepared for each batch using the Microsoft Excel RAND function. The lists were printed on identity stickers with identification (ID) numbers and codes for the type of practice in which the students were to participate. These were sealed in identical opaque plastic bags, which were mixed and randomly selected for each batch. The stickers were placed on the desk in ascending order according to the ID numbers. The allocation was done by asking each student entering the classroom to sequentially seat themselves at the desk with the lowest available ID number. They were not informed about what the allocation codes on their stickers meant.

When the introduction part was over, the participants were informed about where to go for their practice according to the allocation codes on their stickers by the person in charge of the session. The instructors for the self-practice part could not influence the allocation and were charged with ensuring that they got students with the right allocation.

Interventions

The teaching session consisted of a 15-min introduction, 20-min group self-practice and approximately 15-min individual testing.

The main learning outcomes were to be able to keep to the order of the ABCDE approach, conduct the eight observations to be made (Table 1), and document the results of the observations. On the basis of the dialogue with those responsible for the curriculum at the study programmes and recommendations in guidelines and studies (Thim et al. 2012; Resuscitation Council UK 2015), it was decided which observations to include and the equipment (a digital blood pressure gauge, a digital oximeter, a digital ear thermometer, a clock, means for documentation and an overview of the ABCDE observations) to be used.

Table1 The information the students got regarding which eight observation to do and the order they should be done in
Full size table

The introduction session, which was the same for all the participants, included a 6-min lecture on the ABCDE approach and a 7-min video made by the authors demonstrating how to do the ABCDE examination on an advanced simulator manikin (ABCDE introduction film; https://www.youtube.com/watch?v=8brQrQPg_2o).

The VR group received brief information on how to wear the VR equipment consisting of an head mounted display and hand controllers (Oculus Rift S or Oculus Quest; https://www.oculus.com/). The participants in the VR group were informed that they were supposed to practice as a group in the virtual patient room, with one student at a time performing the ABCDE procedure.

The authors, with hired help for programming in Unity (https://unity.com/) to make the application usable across platforms, made the ABCDE application (Table 2, video demonstrating the VR application; https://www.youtube.com/watch?v=5MbPHkcavmY). The application has a tutorial part on how to use the VR hand controllers and an ABCDE practicing part. In the tutorial part, the user is alone, and when the part is finished, the user is taken into the patient room where they meet the other group members. The first user entering the room is placed on the left side of the bed where all the equipment is lying and the observations are made (Fig. 1). The second user is placed on the right side of the bed and the third user at the foot-end. All the observations are done on a virtual patient lying on the bed, using virtual versions of the equipment. The two users not in reach of the equipment are activated through knowledge questions. The questions appear on digital boards in front of them each time the user on the left side of the bed starts a new observation. Instructions on how to make clinical observations are provided as a silent subtitled film on a screen on the wall. All the users can interact and communicate verbally and visually in real time and are present as avatars with head and hands only. When the user on the left side of the bed completes the ABCDE observations, feedback on performance is automatically generated and all the users automatically rotate one place to the right around the bed.

Table 2 Features in the practice part of the VR application (the VirSam ABCDE application)
Full size table
Fig. 1
figure 1

To the left, the look of the ‘head and hand’ avatar as seen by another player. To the right, the view for a player standing beside the patient

Full size image

The participants allocated to group self-practice with physical equipment received a printed sheet with pictures of the equipment along with simple instructions on its technical use. They were also informed to practice as a group.

The minimum help from the instructors in both groups was to reflect a self-training situation.

Data collection

The participants completed a baseline questionnaire when they entered the introduction session. The outcome data were collected through a questionnaire and a practical physical test, which was done individually after the group practice part.

After the practice, first, the participants individually answered questions about the correct order of the ABCDE approach, the eight observations, and their experiences with the different parts of the whole teaching session. There was no time limit, but the majority took approximately 6 min. Then, they individually performed a full physical ABCDE examination on an advanced simulator (the 3G or ALS Simulator; Laerdal Inc., Stavanger, Norway), with changing clinical values of a healthy person. On the bedside table, the same physical equipment as described above was available. They used a blank sheet of paper to document the observations, and this sheet was collected as the outcome measure for the practical test. The students were informed that they had 5 min to perform the ABCDE examination. One staff member was present, giving instructions about the time limit and equipment available to the students. They also showed where the values for blood pressure, O2 saturation and temperature were displayed on a monitor if the students did these observations. The staff helped students who struggled with technical issues. Otherwise, they did not interact with the students and were instructed to only answer “do as you think is best” if they asked anything. The staff were blinded to the allocation of students.

All data were scored independently by the first author (H. B.) and a third person hired for the purpose and then checked for accordance. Both were blinded to the allocation in this process.

Implementation of the intervention

To monitor the implementation of the intervention, the technical problems encountered during self-practice were recorded. The participants were also asked how many times they completed the full ABCDE examination during the practice session (0, 1, 2, 3 or more).

Outcome measures

The primary outcome was the number of participants who documented all eight observations in the correct order of the ABCDE approach on the practical test (yes/no).

Additional outcome measures were collected about the participants’ knowledge and performance of the ABCDE approach and their experience with the teaching session (see the “Results” section for details). The ten questions in the System Usability Scale (SUS) were transformed into one single score according to Brooke (1996) and were given a grade using the Curved Grading Scale (CGS; Lewis 2018).

Statistics

Data for the sample size calculation came from previous studies testing clinical learning outcomes, indicating that a non-inferiority limit of 10 to 15% points is fair (Mpotos et al. 2014; Curran et al. 2015), and a limit of 13% points was decided. We conducted a pilot with 18 healthcare worker students in their second year at a vocational high school who had some experience in systematic clinical observation. Twenty percent of these students got everything correct on the primary outcome. We expected a similar outcome, arguing that the university students in our study had less practical experience but more experience in studying to master new tasks. With an expected outcome of 20% correct answers in both groups, with a non-inferiority limit of 13%, power (1-B) of 80% and significance level (alpha) of 0.05, 118 students were required in each arm, using the web calculator for non-inferior trials provided by Sealed Envelope (https://www.sealedenvelope.com/power/binary-noninferior/) (Julious 2004; Sealed Envelope Ltd. 2012).

Baseline variables are presented with descriptive statistics. As there were no deviations from the allocated groups and hardly any missing data, only one outcome analysis was performed using t tests for continuous variables (SPSS, v.26; IBM Corp., Armonk, NY, USA) and tests of proportions for categorical variables (StataMP, v.16; Stata Corp., Texas, USA). Results are presented as an absolute difference. For the primary outcome, a one-sided 95% confidence interval (CI) is reported according to the one-sided non-inferiority limit. The secondary outcomes are reported with a two-sided 95% CI.

Results

Recruitment and baseline characteristics

Of all 689 eligible first-year medical and nursing students in the large study, 289 participated in this study (Fig. 2). A total of 146 students were randomized to group self-practice in the VR application with virtual equipment (the VR group) and 143 to group self-practice with physical equipment (the TP group). The participants in both groups completed interventions and delivered outcome measures for analyses, except for one participant in the VR group who left before the practical physical test.

Fig. 2
figure 2

Flow of participants (VR virtual reality, TP traditional practice)

Full size image

There were some differences between the groups at baseline (Table 3). The participants were younger in the TP group, with 39.7% younger than 20 years, compared to 19.4% in the VR group. More students in the TP group had experience with simulation, clinical observation and cardiopulmonary resuscitation.

Table 3 Baseline characteristics of the participants
Full size table

Implementation of the intervention

There were no major technical or other types of practical problems in the implementation of the intervention. The only problem that was recorded included that the head mounted display had to be restarted two times because of lost tracking of the hand controller. A larger proportion in the TP group reported completing the full ABCDE examination more than two times during the practice session than did the VR group [25.2% in the TP group and 7.5% in the VR group; absolute difference 17.7% points (95% CI 9.3–25.9); data not shown].

Primary outcome

The results of the primary outcome were that 29 (20%) participants in the VR group and 30 (21%) in the TP group got all eight observations documented in the correct order according to the ABCDE approach on the practical physical test. The absolute difference was 1% points, with the one-sided 95% CI upper level being 8.8%, thus demonstrating the non-inferiority of VR, as the confidence interval was within the non-inferior limit of 13% (Fig. 3).

Fig. 3
figure 3

Primary outcome. 30 persons (21%) in the TP group and 29 persons (20%) in the VR group

Full size image

Secondary outcome

The outcomes on the two questions related to the correct order of the observations in the questionnaire were similar in the two groups [absolute difference 9.4% points (95% CI − 2.1 to 20.9) and 1.7% points (95% CI − 10.0 to 6.6)] (Table 4).

Table 4 Secondary outcomes measures concerning the ABCDE approach
Full size table

The other outcomes on the participants’ knowledge and performance of the ABCDE approach were also similar in the two groups, except in the reporting of respiratory frequency where the students in the TP group performed better [97.9% correct in the TP group vs. 92.5% correct in the VR group; absolute difference 5.4% points (95% CI 0.6–10.3)] and in the number of students who did not complete all eight observations but had the correct order of the ABCDE approach on the documented observations, which also was in favour of the TP group [46.9% correct in the TP group vs. 35.2% correct in the VR group; absolute difference 11.7% points (95% CI 0.4–23.0)].

Concerning the participants’ experience with the teaching session, few thought they received enough training on the ABCDE approach before they started practicing, but the students in the VR group were more displeased [absolute difference 10.1% points (95% CI 0.8–19.4)] (Table 5). One in ten in the VR group thought they got enough time to practice compared to one in four in the TP group [absolute difference 15.3% points (95% CI 6.3–24.2)]. The TP group reported being more confident in conducting an ABCDE examination [absolute difference 8.8% points (95% CI − 0.7 to 18.2)]. The rest of the answers were similar between the two groups.

Table 5 Secondary outcomes measures concerning the student’s experiences with the teaching session
Full size table

The outcome on the SUS questionnaire was similar with a grade of C+ (SUS score 72) for the VR group, B− (SUS score 74) for the TP group on the Curved Grading Scale (Lewis 2018).

Discussion

The participants in the TP group reported they had practiced the full ABCDE examination more than the VR group during the 20-min self-practice session. Nevertheless, self-practicing the ABCDE approach in groups in a multiplayer, immersive, interactive virtual reality application was non-inferior to self-practicing in groups with physical equipment. More students in the VR group reported that there was a lack of time to practice. Both groups thought that the way of practice was likable and that it was a good way to learn the ABCDE approach. The system usability test obtained a similar score in both groups, demonstrating the multiplayer, immersive, interactive virtual reality application to be of the same usability as using physical equipment for group practice.

The main strength of this study is its design. The study included the number of students required in the sample size calculation. The high proportion of eligible participants included makes the results generalizable to first-year medical and nursing students at comparable institutions. The groups had some differences at baseline; the TP group had younger students, who reported more experience with the use of simulation and VR which was considered to be a disadvantage for the VR group. There was no blinding of the students due to the nature of the study, but a recent meta-epidemiological study found that blinding did not influence the outcome (Moustgaard et al. 2020). It is also a limitation that the study tested only one type of VR application, with no follow-up for knowledge retention.

The practical test was physical and performed with the same physical equipment used by the TP group, and the students in this group reported having practiced the full ABCDE examination more. Still, it was found that practicing in VR gave a non-inferior learning outcome compared to practicing with physical equipment. Furthermore, emphasis was put on making an intervention that was adaptable and sustainable for use in ordinary teaching settings. VR solutions are scalable as the equipment needed have limited costs and the programmed VR application has no upper limit in numbers of users. This provides initial evidence for the efficiency of self-practicing procedure skills in a multiplayer, immersive, interactive virtual reality application.

We have not found other studies investigating this type of VR application. Studies on group practice in VR that use desktop solutions, have also found desktop VR solutions to give similar outcomes as traditional training (Youngblood et al. 2008; Khanal et al. 2014; Liaw et al. 2018). Together with our study, this lends support to the claim that VR can be used successfully for group training in skills typically trained involving the participants themselves or simulator manikins. This is further supported by recent studies, reviews and meta-analyses, which have demonstrated that single-player VR applications also provide similar or better outcomes than do the control intervention (Bracq et al. 2019; Kyaw et al. 2019; Berg and Steinsbekk 2020).

There are probably several reasons the VR group got a non-inferior learning outcome despite less practice. Studies have shown that students practicing in a group learn from observing each other and helping each other when they practice clinical skills (Wulf et al. 2010), and the VR application provided this by multiplayer functions, allowing the students to see and interact with each other. As becoming passive can make students lose focus, the VR application provided the users in the observer positions with knowledge questions about the ABCDE approach, which is in line with the findings of other studies showing that activity for all users is important in multiplayer VR (Creutzfeldt et al. 2016).

Most importantly, perhaps, the VR application also gave feedback to the users. Feedback is important for the learning outcomes of clinical skills (Khanal et al. 2014; Sahu et al. 2019). Feedback was provided to both the users answering the knowledge questions in passive positions and more extensively the users doing clinical observations. Although the participants could provide feedback to each other in both groups, only the VR group received autonomous feedback. This study does not have data on the effect of feedback in itself, but the importance of feedback in general points to the need for studies on the effect of VR with and without autonomous feedback.

There are studies discussing the importance of avatar realism in multiplayer, immersive, interactive environments, which argue that simple geometric avatars can support successful collaboration on physical tasks between users (Steed and Schroeder 2015; Roth et al. 2016). Other research has found that the looks of avatars affect how users respond to each other (Waltemate et al. 2018). The avatars in our application are gender neutral and have head and hands only, but they have lip synchronization, so the other user could see who was taking.

The strength of the ABCDE VR application developed for this study is the combination of multiplayer possibilities, immersion through the head mounted display and interaction with the environment using hand controllers, all of which contribute to making the experience relevant. This is different for most VR applications used for clinical practice, which is either immersive or interactive (Liaw et al. 2018; Bracq et al. 2019; Kyaw et al. 2019), and the immersive VR applications lacking the multiplayer dimension. Because of being developed in 2019, the application used in this study benefits from the rapid development of technology and the availability of VR solutions for the consumer market. As such, the application is in the forefront, but given the speed of innovations in this field, new and improved possibilities are likely to be available for further improvement of VR solutions to practice clinical skills.

Conclusion

To the best of our knowledge, this is the first study on the effect of multiplayer, immersive, interactive virtual reality application for self-directed group-based procedure practice. With a time limit of 20 min for practice, group practice in VR gives each individual student less opportunity to practice the ABCDE approach compared to using physical equipment probably because of unfamiliarity with doing observations in VR. Still, group self-practice of the ABCDE approach in a multiplayer, immersive, interactive virtual reality application was non-inferior to practice with physical equipment.