Virtual reality reusable e‑resources for clinical skills training: a mixed‑methods evaluation

Virtual reality has long existed, but its wider adoption in education is recent. Studies informed by theoretical underpinned co-creation frameworks and utilization of theoretical informed evaluations are scarce in literature. Thus, this study inter-nationally evaluated the efficacy of three virtual reality reusable e-resources (VRReRs), co-created based on the ASPIRE framework, for teaching clinical skills to university students. The study followed a mixed-methods approach, combining SUS, SUS Presence Questionnaire, TAM, and UTAUT2 with a focus group discussion. Additionally, for one VRReR, a quantitative pre/post evaluation of knowledge and comparison with lecture notes followed. Results demonstrated moderately to highly usability, effectively facilitated a strong sense of presence, confidence while using them, and willingness to continue using VRReRs in the future, while increased knowledge of the learners, highlighted their effectiveness. Although some usability issues were identified, these were considered easy to address. This work evidence, in an international context, that co-created VR resources are highly acceptable and effective, similar to other types of digital or traditional resources developed through participatory inquiry paradigm. By leveraging the benefits of VR technology, VRReRs have the potential to transform and enhance the learning experience in the field of clinical skills, ultimately advancing the digitalization of higher education.


Introduction
Healthcare and medical education is always at the forefront of the adoption of digital innovation to enhance students' and healthcare professionals' competences to ensure a future high skilled workforce and quality of care.A range of different types of resources have been created and used in the last decades to enhance clinical skills of higher education students, such as reusable learning objects (RLOs), virtual patients (VPs), and other on-screen computer or mobile simulation tools, with positive acceptance by the students.While virtual reality exists for a long time, it is only lately that started to have a wider adoption including in education [1][2][3][4][5].
Virtual reality within medical education programs has a positive impact to learning with less immersive VR to be more effective than fully immersive according to a metanalysis of 25 randomized control trial (RCT) studies [4].Similar findings from an earlier review and metanalysis of 31 RCT studies resulted that VR improves postintervention knowledge and skill outcomes when compared with other types of education (digital, traditional, etc.) [6].360° immersive videos, as a cost-effective alternative to VR, have a favorable impact on the user's emotional response to the learning environment, thereby significantly influencing their motivation to engage in learning [7].While positive evidence for the application of VR into healthcare education exists, all reviews agree that the results are based on a small number of studies with weaknesses, and emphasize the need of more high-quality studies.Additionally, there is limited available evidence regarding the utilization of more novel modalities, such as mobile virtual reality (VR) [8], and a noticeable absence of underpinning pedagogical theories [7] and conceptual frameworks or theories in the design of VR experiences [8].Furthermore, the involvement of healthcare professionals on the development process is urged to enhance the effectiveness and specificity of virtual reality (VR) content creation [9].
Thus, this study aims to fill in the gap of high-quality international studies, aiming to explore the efficacy of virtual reality reusable e-resources (VRReRs) in improving the learning outcomes in the healthcare and medical education.This work underpins the design of VRReRs with a sound co-creation theoretical framework and evaluations based on theoretically informed quantitative measures and qualitative focus group discussions, towards more accessible, costeffective, and immersive mobile VR apps.
The remainder of the paper is structured as follows.Initially, we provide a background on the transfer of knowledge and digitalization of content and the co-creation framework utilized for the development of the VRReRs.Next, we describe the protocols used to evaluate the efficacy, feasibility, and acceptability of the VRReR along with the tools used.Results of the findings are presented in the next session.Then, a discussion of our findings in regard with the existing literature follows, alongside the Kirkpatrick model to generalize our evaluation findings.Last but not least, Section 5 summarises the main outputs of this international study together with main limitations and the road ahead.

Transfer of information and digitalization of content
Healthcare professionals have a high demand for high-quality training materials due to the increasing complexity of the healthcare landscape, including more data, information, procedures, equipment, and patient needs [10,11].The transfer of relevant knowledge and skills from experts to learners is a crucial factor in the success or failure of healthcare services.Sub-optimal knowledge and skill transfer can result in learners acquiring improper skills and inaccurate information or not acquiring enough information [12,13].
Additionally, incorporating mobile learning applications into training may not have a positive impact on learners if they are not involved in the development process.Research has shown that the use of mobile applications is low [14], but this is mostly due to factors such as usage in the classroom, biases in studies, and outdated devices [15].Both learners and subject experts must be included in the development process for successful integration of digital pedagogical resources.
Traditional training methods such as lectures, workshops, and hands-on experience through shadowing and supervision are also highly dependent on environmental factors, which are further impacted by events like the COVID-19 pandemic [16].Digitalizing training content has numerous benefits and can overcome these limitations.Reforming the education process in the digital era requires supporting current curricula and promoting open education [17].This is where the European-funded project Co-creation of Virtual Reality reusable e-Resources for European Healthcare Education (COViRR) funded by ERASMUS + can make a positive impact.
COViRR aimed to co-create new pedagogical approaches, specifically virtual reality reusable e-resources, for medical and nursing schools in Europe.The project predicted that learners would adopt this new digital pedagogy and improve their clinical skills and competencies through immersive learning [18].Additionally, the teaching staff would have the opportunity to enhance their competencies in e-learning tool co-creation and make use of best practices for co-creation [19].Reforming the educational process to meet current standards can be achieved through personalized learning opportunities driven by learner focus, as they are the endusers and future independent healthcare providers.

Co-creation and ASPIRE framework for healthcare
The VRReR development process is based on the ASPIRE framework, which stands for Aims, Storyboarding, Population, Implementation, Release, and Evaluation [20][21][22][23].The ASPIRE framework has been refined over time to be flexible and effective for a wide range of applications and has been adapted for use in virtual reality and 360° interactive video resources, as well as virtual learning packages that address multiple learning objectives [24].The combination of the co-creation methods within the ASPIRE framework, the use of virtual reality environments, and the quality control of content by subject experts creates powerful, efficient, and simple learning resources.The development process (Fig. 1) includes three screening checkpoints to ensure quality and necessity: in the storyboarding workshop, learners are facilitated by learning technologists and educators to verbalize their needs, content is checked by experts before development, and another review is performed before the final release [17].This paper presents the design and evaluation of three VRReRs and reflects the efficacy of the entire development process.
CoViRR proposed, and has recently achieved, the integration of immersive technologies in healthcare pedagogy, and this paper presents the design overview and evaluation of three VRReRs.There is focus on feasibility and acceptability of the resources rather than the novelty; however, there are novel elements present through the co-creation elements.The resources are reusable learning objects as they are self-contained packages of highquality information presented to healthcare and medical learners who were involved in the full development process [23,25].Their involvement in the creation of these immersive reality resources is successful from utilizing the ASPIRE framework which has a longstanding and continued evidence-based efficacy for development and implementation of resources using co-creation methodology [17].

Aims and rationale
Therefore, the aim of this paper was to evaluate the VRReRs to understand both the strengths and limitations in acceptability and usability.These have been embedded into the resources either during the development processes or identified in the feedback from the learners who test each resource.The evaluation and understanding of strengths and limitations allow both the ASPIRE framework and the VRReRs to progress towards greater efficacious outcomes in future resources created for healthcare students.Furthermore, it aims to explore the efficacy of VRReR in improving the learning outcomes in the healthcare and medical education.

Virtual reality reusable e-resources for evaluation
The selection of subject areas for the virtual reality reusable e-resources (VRReRs) was guided by healthcare educators and course leaders, who determined the areas that would benefit students the most, as well as have the most significant impact on service users and patients.Collaborative storyboarding was then performed with learners and educators to ensure that the VRReRs were tailored to their needs.An online storyboard canvas named Mural was used to allow multi-user editing of images, text, and videos.Following this, subject matter experts, learning technologists, and a facilitator reviewed the final content in a storyboarding workshop to ensure that the VRReRs were effective for their intended purposes.The VRReRs were designed to be used with mobile phone headsets, which are low-cost and widely available.The proposed model is of little complexity, designed to balance immersive experience with user accessibility.The interaction with the VRReRs involved a gaze dot in the center of the screen, with participants waiting three seconds for an item for it to be triggered.
The first VRReR (UoN VRReR) (Fig. 2) was developed in response to the pandemic's effects on communicationbased scenarios, particularly home visits of adolescents at risk of self-harm.The VRReR aimed to improve communication skills between learners and patients [26,27].
The second VRReR (CYENS VRReR) [28] is a VR mobile resource for a clinical skill course (Fig. 3), with the development process previously presented [24].The resource provides students with the ability to watch 360° videos that highlight several different scenarios, covering the following areas: sterilizing hands/hand hygiene, surgical gloving and ungloving technique, wound sterilization, local anesthetic administration, and excision of skin lesion.Some are techniques that are commonly used during surgical procedures to minimize the risk of infection and to manage pain.
The third VRReR (AUTH VRReR) [29] (Fig. 4) had two sections, one being 12-lead ECG placement application and examination and one being X-ray examination.For the ECG scenario, students perform a non-invasive procedure, the placement of the 12-basic leads and the machine displays single-channel ECG records of various heart diseases like heart arrhythmias, and the medical students must identify the disease which the signals correspond to and choose the correct answer.For the X-ray scenario, alongside the patient, a chest X-ray is displayed.The user must find the problem based solely on the provided X-ray and choose among four options displayed in a banner next to the X-ray.

Experimental protocol
In this study, we employed two different protocols for evaluating the effectiveness, acceptability, and usability of virtual reality reusable e-resources (VRReRs) in improving learning outcomes in healthcare education.A mixed method sequential explanatory methodology was followed [30] for the first protocol in order to triangulate the research findings and gain a better understanding of the relation between qualitative and quantitative data [31].The quantitative part involved collecting data from a convenience sample of 136 participants who completed the Technology Acceptance Model (TAM) [32], System Usability Scale (SUS) [33], Slater-Usoh-Steed Presence Questionnaire (SUS-PQ) [34], and Unified Theory of Acceptance and Use of Technology (UTAUT2) [35] surveys before and after using the VRReRs.There were 52% (71) males and 48% (65) females.Average age was 21.6 (SD = 4.32, mode = 20, Mdn = 21).Therefore, participants were Generation Z, described as "digital natives."There were 60 participants who assessed the AUTH VRReR, 38 for CYENS VRReR, and 38 for UoN VRReR.Participants joined an online session for data collection, and we used descriptive comparisons to analyze the results.For the qualitative part, 21 participants in three groups of seven were The second evaluation protocol involved further investigation of the CYENS VRReR which consisted of several different medical and surgical procedures.The students were recruited (n = 27) from the University of Cyprus and divided into two groups-one group learning the course only through lecture notes and the other group using lecture notes along with 360° videos.
We initiated the study with a pre-TAM evaluation questionnaire to all students, followed by the post-TAM at the end of the evaluation, which took place 1 month later.Students viewed a series of 360 videos (wound suturing, hand sterilization/glove use, skin lesion removal, wound sterilization) each week, taking corresponding pre-and post-SUS evaluations and proficiency tests to gauge improvement.The effects of CYENS VRReR usage over Time and across different Scenarios on knowledge test Scores were evaluated statistically.The "Time" variable in this context represented the pre/post usage of the VRReR, and the "Scenario" variable referred to four distinct medical situations: wound suturing, hand hygiene, wound sterilization/local anesthetic, and excision of skin lesion.The participants were divided into two "Conditions": a control group and a VRReR group.As the knowledge tests varied across the different scenarios, with potential "Scores" ranging from 8 to 32, a normalization process was applied to the scores (converting them between 0 and 1 as Normalized Scores) to ensure comparability across the different scenarios.

Evaluation measures
There are several evaluation metrics for novel technology in healthcare pedagogy and technology.We used the most popular yet relevant and most established systems of measurement for evaluation and briefly described: The System Usability Scale (SUS) is a widely adopted questionnaire consisting of ten questions that provide an unbiased and agnostic measure of usability.The SUS score out of 100 can be compared to a determined average score of 68, with scores of 80 or higher indicating excellent usability and scores of 51 or lower suggesting significant usability problems.
The Slater-Usoh-Steed Presence Questionnaire (SUS-PQ) measures the user experience of being present within a virtual environment through a 6-question, 7-point scale [34].
The Unified Theory of Acceptance and Use of Technology (UTAUT2) combines constructs from previous models and theories to develop a unified form of technology acceptance measure.The Technology Acceptance Model (TAM) identifies determinants involved in computer acceptance, examines information technology usage behaviors, and provides a theoretical explanatory model [36].
A series of short focus group sessions identified the feasibility of CoViRR resources for formal curricular integration.These sessions, spanning no more than 1-1.5 h and consisting of no more than 5-7 persons each explored all axes of curricular integration such as accessibility in the The paper [2] has full details on the measures and expanded results section.This paper expands by further investigating one of the resources and provides a more critical appraisal and practical application of results towards pedagogical changes.

System Usability Scale (SUS) Scores
The SUS score for all data was 68.2, with AUTH VRReR receiving a score of 62.5, CYENS VRReR a score of 72.3, and UoN VRReR a score of 71.6.The collective score is within, and above the median of, 68-which is above the range of average usability.In Fig. 5, the light blue color depicts the collective SUS Score (reversed where needed) for the ten SUS questions (light blue labels around the spider diagram), while the red line depicts the range of average usability.

Slater-Usoh-Steed Presence Questionnaire (SUS-PQ)
The six SUS-PQ questions showed a moderate to high feeling of presence (mean = 4.5, mode = 5) when 0-1 is low, 2-3 low/moderate, 4-5 moderate/high, and 6-7 high.This suggested the mobile VR environments facilitated an immersive environment for learners, which is great from the low fidelity and cost of the equipment.Figure 5 (purple color) depicts the collective SUS Presence score for its different categories (purple labels around the spider diagram).

Technology Acceptance Model
The TAM had three sections (Ease of Use, Perceived Usefulness, and Intention of Use).All had positive mean ratings from approximately 90% of participants, therefore approximately less than 10% neutral or disagreeing for each

Unified Theory of Acceptance and Use of Technology
There were no significant findings for changes in perceived ease of use from before to after VRReR usage.Again, this suggested they felt the VRReRs fell with the range of what the participants are used to using.

CYENS VRReR usage comparison
The effects of CYENS VRReR usage over "Time" (pre/ postintervention) and across different "Scenarios" (wound suturing, hand hygiene, wound sterilization/local anesthetic, and excision of skin lesion) on knowledge test "Scores" were evaluated statistically.After the normalization of Scores as each scenario's knowledge tests asked different topic questions, a two-way ANOVA was conducted on the data.The results revealed a significant main effect of "Time" (F(1, 199) = 8.15, p = 0.0048), demonstrating that the mean normalized "Score" differed significantly between the pre-and post-VRReR usage Times.However, the "Scenario" (wound suturing, hand hygiene, wound sterilization/local anesthetic, excision of skin lesion) did not yield a significant main effect (F(1, 199) = 1.92, p = 0.17), suggesting that the specific scenario did not significantly affect the Normalized Scores.Moreover, the interaction between "Time" (pre/post usage of VRReR as a whole) and "Scenario" did not show a significant effect (F(1, 199) = 0.14, p = 0.71).It is worth noting that in the "Scenario" of wound suturing, the factor "Time" exhibited a significant effect (F = 20.96,p > 0.05), indicating that the progression of time had a significant impact on the outcomes (Score) (Fig. 6a).However, the factor "Condition" did not show a significant effect (F = 0.25, p = 0.61).The interaction between "Time" and "Condition" was also not significant (F = 0.06, p = 1.0).For the "Scenario" of excision of skin lesion, a significant effect was observed for the factor "Condition" (F = 16.6, p = 0.001), suggesting that the specific "Condition" (lecture notes and VRReR) under which the skin lesion was excised significantly influenced the outcomes (Score) (Fig. 6b).The factors "Time" and the interaction between "Time" and "Condition" did not exhibit significant effects.
These results suggest that while the usage of VRReRs significantly affected the participants' knowledge test Scores over Time (pre/post usage), the Condition (lecture notes/ lecture notes and VRReR) on the whole set of Scenarios did not substantially impact these Scores.Furthermore, in one Scenario (wound suturing), there was a significant improvement between the pre-and post-questionnaire for both types of resources (Condition) measuring the knowledge Score interaction, and for the Scenario excision of skin lesion, the participants who used the VRReR showed a significantly Fig. 5 Radar diagram as a concise overview of the 20 main evaluation points across the different dimensions.SUS scores (light blue) were higher than typical literature scores (typical literature scores represented with a red line).Technology Acceptance (orange) score averages were towards strongly agree.SUS Presence (purple) scores were towards strongly agree.Rating in diagram: 1 = "strongly agree," 2 = "agree," 3 = "neutral," 4 = "disagree," 5 = "strongly disagree" increased knowledge Score in comparison with the lecture notes group.

Focus group discussion (FGD)
To supplement these quantitative results and further probe the strengths and limitations found in the data for each VRReR, focus group discussion data from 21 participants (in 3 groups of 7) were analyzed using thematic analysis.
The majority of comments were regarding technical elements of the resources and mostly limitations experienced.Within these, the theme of navigation was most common as users experienced a mismatch between what they expected to be navigated to compared to what occurred.For the effective communication VRReR, one learner stated, it was difficult to go back and choose another example, so I had to refresh (p25).Mention and preference of the VR feature were positive when mentioned, for example, Using the app on touch screen mode was more or less like watching a video (p37), and The app functions well in VR mode (p38).
There were few mentions of the adaptations for use; however, the main issue noted was It took me a while to get used to and understand how the 'mouse' worked to navigate and select things (p2)-this referred to the visual tracking and selection via a gaze UI dot located at the center of the user's vision.Therefore, referring back to the TAM and SUS, there were a portion of participants who gave either neutral or low positive scores.The weaknesses identified in the FGD can lower these occurrences.

Principal results
The aim of this study was to assess the usability and acceptance of the VRReRs through investigating performance expectancy, usage behavior, facilitating conditions, and confidence towards acceptance.The study's findings suggest that VRReRs can be an effective tool for teaching medical and surgical procedures, potentially replacing multiple traditional resources with a single, accessible mobile app.
The results revealed that the VRReRs were well received and exhibited moderate-to-good usability, independence towards self-learning, and ease of use.The VRReRs helped to decrease users' anxieties towards technology and increased their motivation to use the resources following the initial session, which is aligned with the findings of a recent review resulted that 360° videos significantly influencing their motivation to engage in learning [7].
However, the study also identified limitations and weaknesses of the VRReRs, such as technical issues that some learners faced, particularly with navigation and adapting to the visual tracking and selection via gaze UI dot.Furthermore, some participants had neutral or low positive scores in the Technology Acceptance Model and System Usability Scale assessments, indicating that improvements could be made in future updates of the VRReRs.
Despite these limitations, the VRReRs provide a more immersive and interactive learning experience compared to traditional resources, which are often limited in their ability to visualize complex 3D structures and simulate hands-on experiences.
In the second part of the study, no significant differences were found in the usability and acceptance of the VRReRs between the different cohorts of participants.This is a positive outcome as it suggests that this low-fidelity, low-cost, and low-development resource can be just as effective as traditional teaching resources.
Usage of the resources has a significant effect on the "Score" values, while the "Scenario" variable does not have a significant effect.This means that the use of CYENS VRReR improved the scores significantly between the pre-and post-tests, and in one scenario, the immersive scenario had a better effect on participants' knowledge in comparison with the traditional teaching.Thus, similarly to a metanalysis of 31 RCTs [6], the CYENS VRReR was equally or more effective as currently used resources.
Furthermore, the user-orientated approach of the VRReRs caters well to digital learners who are comfortable with using technology in their education.This study confirms for virtual reality technology's previous generic findings that digital literacy influences the effectiveness of learning in blended learning environments [37].These findings highlight the potential of VRReRs as a valuable tool for teaching medical and surgical procedures in a way that is accessible, interactive, and effective for a diverse range of learners.

Mapping to Kirkpatrick model
Kirkpatrick model [38] can be used to evaluate the effectiveness of virtual reality reusable e-resources (VRReRs) in teaching medical and surgical procedures.The four levels-Reaction, Learning, Behavior, and Results-are evaluated using metrics including user feedback, pre-and post-tests, and plotted in Fig. 7 [38].This study focused on evaluating the virtual reality reusable e-resources, and the results obtained can be aligned with the Kirkpatrick model as follows (Fig. 7): Level 1 Reaction: Participants' reactions to the VRReRs were positive indicating good usability and increased motivation to use the resources.Level 2 Learning: Pre-and post-tests showed that the VRReRs were equally effective or more effective from traditional resources in teaching medical and surgical procedures.
Although not directly evaluated, the findings suggest that the VRReRs could simplify the need for multiple resources and streamline trusted information-seeking behavior among medical students.
The fourth level of the model concerns the long-term results of using the VRReRs, which go beyond common measures like patient outcomes and cost savings.Although not assessed in this study, the VRReRs have the potential to be a valuable and accessible tool for teaching medical and surgical procedures, improving their learners' abilities in the long run.
Overall, the study shows good usability and acceptability of VRReRs and improvement of learning outcomes and knowledge over time, and suggests that further investigation may be needed to explore the potential impact of different scenarios on the effectiveness of the resource in the higher levels of Kirkpatrick model.This evaluation endeavor for the CoViRR project was expected to contribute to the digitalization of higher education curricula to enhance learners' knowledge of clinical skills and beyond.Multifaceted measures were used to provide a holistic matrix of the key factors in uptake, efficacy, and distribution.These findings can help modify the ASPIRE process for immersive reality development, emphasizing the need for theoretically underpinned co-creation approaches [39], giving learners more control in the stakeholder workshops to provide feedback on the UI and menus while still maintaining the focus on the primary goal of VRReRs and simplicity.

Conclusions
ASPIRE, a theoretical co-creation framework, was followed to create cost-effective and low-complexity virtual reality reusable e-resources.Experts on clinical skills, learners, academics, and healthcare professionals from three European countries collaborated in participatory workshops to envision and co-create three multi-scenario VRReRs.VRReR evaluation in Reaction and Learning levels of the Kirkpatrick framework was positive, with learners to indicate good usability and increased motivation to use and equal or greater increase in knowledge than traditional resources.Both the creation process and the outcome are considered innovative, addressing current burdens such as the developmental cost of VR resource which is performed mainly by specialist companies without taking into consideration the learners and the equipment needed for VR, as learners do not have easy access (e.g., by their mobile phone and costeffective headsets) to immersive learning experience.While the VRReRs are promising, participants also revealed some practical limitations, reporting issues with navigation and adapting to the gaze UI dot for visual tracking and selection.These were first-time users and may have needed a little more setup time to adjust.Future iterations of the platform will aim to address these issues.
Technology has aided learning since its conception, and VR, in particular, has been seen as a constructivist instrument of instruction.It should come as no surprise then that a qualitative-mixed methods approach, the primary choice for evaluating constructivist pedagogical undertakings [40,41], has been effectively used to provide the holistic framework for its educational evaluation in the CoViRR project.This work was pivotal on the development of VR educational resources based on theoretical co-creation frameworks.It evidences, in an international context, that co-created VR resources are no exception to high acceptability and efficacy, as "traditional" resources had done in the past following the participatory inquiry paradigm for their development.This multifaceted mixed-methods evaluation demonstrated that there is an audience of learners ready to accept VR reusable e-resources as tools for augmenting their learning capacity towards clinical skills training.Sound usability choices, technical ease of use, and adherence to good participatory design choices can make these resources integral supporting parts of contemporary healthcare curricula.

Fig. 1
Fig. 1 VRReR development process based on the ASPIRE framework.Learners are stakeholders from the start when performing collaborative workshops

Fig. 2
Fig. 2 Snapshots of the UoN VRReR to train communication skills with adolescents with home visits.Top left: students can select VR or non-VR view of the application.Bottom left: students are guided to a service user's home to start the scenarios through hotspots inter-

Fig. 3
Fig. 3 Top: the 360° interactive video in which students are introduced to tools.Bottom left: screenshot of the storyboard workshop that stakeholders developed.Bottom right: a synced close-up video showing skills that identified during the co-creation process

Fig. 4
Fig. 4 Bottom: the VR environment showing the 12-leads.Top: the guided scenario of procedural steps for 12-lead EEG placement on the patient

Fig. 6
Fig. 6 Pre/post differences for normalized Score (knowledge Score) for two Scenarios (wound suturing on the left (a) and excision of skin lesion on the right (b) over Time (before and after the intervention)

Fig. 7
Fig. 7 Summary points applied to the Kirkpatrick model of training and evaluation