1 Introduction

Over the last few decades, great strides have been taken in information and communication technologies. The growing use of these technologies is transforming different fields of society, including the health and education sectors (Mohapatra et al. 2015). These new technologies have the potential to improve clinical care and training in the health sciences field (Mohapatra et al. 2015). Specifically, over the next few years, virtual reality (VR) and augmented reality (AR) devices will play an increasingly important role in the health sector (Roessel et al. 2020).

AR allows users to see the real world complemented by computer-generated virtual elements (Buettner et al. 2020). In this way, virtual and real elements coexist in the same physical space (Azuma 1997), augmenting the information that we receive from the real world. AR combines virtual images with real world images. In this way, the real world is complemented or “augmented” with virtual content; but it is not replaced completely or substituted, as occurs in virtual reality (Azuma 1997). AR can be used in various display devices, such as smartphones, tablets or smart glasses (Klinker et al. 2020). The number of AR devices on the market is growing every day, and one of the best known and most technologically advanced are the Microsoft HoloLens® smart glasses that recently launched their second generation (https://www.microsoft.com/en-us/hololens/hardware#document-experiences). These smart glasses allow users to interact with the real world and the digital world, requiring minimum action between the user and the device. Using voice commands or very simple hand gestures, users can view the necessary information while continuing other tasks that require use of their hands. The possibilities of these devices are very broad, from simply watching a video or reading documentation to interacting with complex 3D elements or receiving remote assistance.

In the field of health care, there are numerous examples of the use of AR in surgery, rehabilitation or in training on surgical or medical techniques for health professionals (Chen et al. 2017; Gallos et al. 2018; Gerup et al. 2020). Furthermore, its application in training health science students is also increasingly widespread, either to include information resources or to work on skills such as clinical judgement or communication (Byrne and Senk 2017; Garrett et al. 2018; Frost et al. 2020).

When training these students (or health professionals), they must practice clinical technical skills for the performance invasive techniques (American Association of College of Nursing 2008; Scheckel 2012; Brown et al. 2020). This practical training occurs in specialized skills laboratories or simulation rooms, employing mannequins or task trainers designed to reproduce specific anatomical regions of the human body (Baillie and Curzio 2009; Weller et al. 2012; Kemery and Morrell 2020). The literature also describes other methods for teaching these skills, videos; blended online learning, mobile application use or virtual reality simulation (Bayram and Caliskan 2019; Kemery and Morrell 2020; Rourke 2020). Carrying out certain clinical invasive techniques requires knowledge of the anatomical internal structures involved. To do this, the student needs to imagine these structures inside the body as they practice this type of techniques with the task trainers. AR technology enables the visualization of the internal anatomical structures in this context. Thus far, the application of AR technology in the context of the training of health science students has shown very positive results (Hauze et al. 2019; Wüller et al. 2019), offering many opportunities to reinforce learning and the learning process in regard to the acquisition of competencies (Mendez et al. 2020). Kurt and Öztürk (2021) and Byrne and Senk (2017) have also evaluated the use of AR to provide educational materials to nursing students, concluding that it has a positive effect on their knowledge and skill levels.

The aim of this study was to assess the usability and user expectations of ARSim2care, an augmented reality (AR) application designed for Microsoft HoloLens®. This application allows health science students and health care professionals to visualize internal anatomical structures during the training of the clinical technical skills for the performance of invasive techniques.

2 Methods

2.1 Design

To assess the usability and user expectations of the ARsim2care application as an effective learning tool for training of the clinical technical skills for the performance of invasive techniques, we conducted a descriptive study at three universities located in the northeast of Spain.

2.2 Sample/participants

In the recruitment phase, students of the Bachelor’s Degree in Nursing of the three Spanish universities were invited to participate in a seminar about augmented reality technology, where in addition the Microsoft HoloLens® and the ARSim2care application were presented. At the end of the information session, the students had to decide whether to participate in the study. A convenience sample of 61 undergraduate nursing degree students from different courses participated in the study. The inclusion criteria applied were (1) having already learned the intramuscular injection technique. Based on Alroobaea and Mayhew’s (2014) recommendations, this sample size is considered adequate as for studies aiming at identifying problems with design or function as well as pursuing statistically significant results a sample size higher than 20 is recommended. On the other hand, the U.S. Food and Drug Administration (U.S. Food and Drug Administration 2016) indicates that a sample of 15 people could be enough to detect most of the problems in a user interface constitutes a practical minimum number of participants for human factors validation testing. Both recommendations notes that the size should be adjusted to the complexity of the research objective studied. Therefore, although the Microsoft HoloLens can be considered a device with a certain level of sophistication and complexity, a sample of 61 users could be assessed as high enough to detect the usability problems of the evaluated application, since it far exceeds the minimum established by the aforementioned authors.

2.3 ARSIM2care application

This application has been developed within the context of the European project titled “ARSim2Care: Application of augmented reality in clinical simulation”, with the collaboration of three European higher education institutions, Public University of Navarre (UPNA) (leading institution), Escola Superior de Enfermagem de Coimbra (ESEnfC), Erasmushogeschool Brussel (EhB) and and the Industrial Augmented Reality (iAR) company (https://erasmus-plus.ec.europa.eu/projects/search/details/2017-1-ES01-KA203-038514).

The evaluation presented in this article was developed by the university who was leading the European project, Public University of Navarre, in collaboration with two other Spanish universities, who were associated partners in ARSim2care project.

The application is designed to be used with Microsoft HoloLens® smart glasses. The software for the ARSim2care application allows AR to be applied in training on clinical technical skills for the performance of invasive techniques. Specifically, the application was designed for the procedures of intramuscular injection, nasogastric tube insertion, endotracheal intubation and suctioning via tracheostomy tube. All of these are procedures that require knowledge of the internal structures to perform them as safely and accurately as possible. The application can be used in four languages: English, Spanish, Dutch and Portuguese.

The software was designed for its use with certain anatomical models and manikins. In the intramuscular injection procedure, the task trainer used was the buttock injection simulator distributed by Medical Simulator®. For the other techniques, the Nursing Anne simulator manufactured by Laerdal Medical® was used, that is a real-size, full-body manikin. These models were selected because the universities involved in the project usually worked with them.

Using the HoloLens glasses, AR allows students and professionals to visualize these internal anatomical structures when performing the technique. In addition, the endotracheal intubation, nasogastric tube insertion and suctioning via tracheostomy tube procedures have a simulation option that can display the route of the catheter or tubes that are inserted, using Hall effect sensors and magnets installed in the manikin.

2.3.1 Application development

The design of the application was developed in three phases: digitalization of the images, recognition of the models and software development. First, the different internal anatomical structures to be visualized in each of the procedures were identified. Once the structures had been identified, searches were made for corresponding virtual 3D images among various libraries and the rights to use were purchased. These 3D images were processed and modified to adapt to the project needs.

AR technology uses markers as a point of reference to define the position, scale, and orientation of the virtual content in the physical world. Therefore, the team of project engineers developed the geometric shape, which involved 3D scanning the anatomical model (buttock injection simulator) and the manikin (Nursing Anne simulator). The virtual images of the internal structures were inserted in the scanned models to check their dimension and alignment. With all this information, the application was able to recognize the anatomical model and the manikin and locate the virtual anatomical structures in the correct position.

The application was developed with Unity 3D v2018.4.12f1.

2.3.2 Overview of ARSim2care application

The home screen of the application shows the initial menu (Fig. 1) with the icons for the different procedures that can be performed: intramuscular injection, nasogastric tube insertion, suctioning via tracheostomy tube and endotracheal intubation.

Fig. 1
figure 1

ARsim2care home screen

Full size image

The top part shows two icons, one to access the manual for the procedures and the other for language adjustments, Bluetooth connection and technical support.

Once a procedure has been selected, a screen shows the application’s options for performing the procedure. The top of this screen once again includes the procedure manual icon that gives access to a screen where we find the definition, indications, contraindications, resources, steps for performing it and complications. The information is provided by text and pictures. Voice commands (i.e., next and back) can be used to work through the information. The lower part of the screen offers the different recognition options for the anatomical model or the manikin that the application includes automatic mode or manual mode.

In automatic mode, it works by shape recognition. This option shows a silhouette that must fit in the anatomical model or the manikin so that Microsoft HoloLens can recognize it and allow the internal structures to be placed in the right proportion and position, offering a realistic recreation. In manual mode, the internal structures are visualized, with the possibility of moving them and resizing them to locate them wherever the user wishes. This can be useful to carry out certain procedures, such as endotracheal intubation (Fig. 2) where it might be more comfortable to view the images from the side, or to use the application with another type of anatomical model or manikin on which the recognition mode does not work.

Fig. 2
figure 2

3D images on the right manikin section

Full size image

On each of the recognition mode screens, there is a circular floating menu showing different icons (Fig. 3) for different options: go back to the home screen; disconnect the Bluetooth when you are not going to use the procedure that requires sensorization; eliminate the interaction framework with the internal structures that makes it possible to move and resize them; change from manual to automatic recognition mode and change into QR recognition.

Fig. 3
figure 3

Circular floating menu on the right

Full size image

Two manuals provide support resources for the use of the application. First, the “3D Procedures Manual Included in the App Arsim2care” (San Martín-Rodríguez et al. 2019) explains the protocols for performing the clinical procedures, accompanied by 3D images of the internal anatomical structures to be visualized. Second, the “Arsim2care Application User’s Manual” (Dias Coutinho et al. 2020) includes a protocol for integrating the Arsim2care application in teaching. This manual is particularly intended for teachers, as a guide to develop training for their students through the application.

2.4 Data collection

2.4.1 Procedure

The team leading the project moved to different universities to carry out the data collection process during April and May 2022. For the evaluation of usability and user expectations, each student put on the smart glasses, under the supervision of the research team members, and received instructions to access the ARSim2care application from the initial menu of the Microsoft HoloLens®. Once inside the application, the student visualized the home screen with the different procedures that could be performed and was instructed to select the intramuscular injection procedure. This was the procedure chosen by the research team for this initial evaluation of usability and user expectations of ARSim2care application.

The next screen offered the different options for recognition of the anatomical model or manikin included in the application: automatic mode or manual mode. The student was instructed to select the automatic mode, which works by shape recognition.

Thus, during the use of the application each student virtually visualized the internal anatomical structures (i.e. gluteus maximus muscle, gluteus medius muscle, ischiatic nerve, femoral nerve, superior gluteal artery and vein, inferior gluteal artery and vein, sacrum, pelvic girdle, iliac crest), while performing the procedure as learned. Once the procedure was completed, the student was instructed to exit the application and remove the glasses.

After completing the software experience, the student was asked to respond to an electronic questionnaire using the SurveyMonkey® web platform via a mobile device.

2.4.2 Variables and measurements

A three-part electronic questionnaire was developed. The first part included demographic and academic data: age; gender; university; interest in new technologies (evaluated on a scale of 0 to 10); time since he/she learned the intramuscular injection procedure. In the second part, to evaluate usability, participants were asked to complete the System Usability Scale (SUS) (Brooke 1996). This scale includes 10 items with five response options for respondents (from strongly agree to strongly disagree), with a score range between 0 and 100. It has been translated and validated in Spanish (Sevilla-Gonzalez et al. 2020). It is the most widely used instrument to assess the perceived usability of a wide variety of products and services, including hardware, software, mobile devices, websites and apps and has demonstrated high reliability and validity (Lewis 2018). To interpret their results, Lewis and Sauro (2018) developed a classification model, the curved grading scale (CGS), with 11 grading ranges (from A + to F), setting the average score at 68 points suggesting an average user experience and placing at 78.9 points or higher the excellent level (Grade A-), indicative of an above average user experience. Scores equal to or below 51.7 are indicative of a problematic or poor user experience (Grade F). Subsequently, different qualifying adjectives, related to user experience, were added to these scores (Bangor et al. 2009).

In the third part, the user expectations were assessed by means of an adaptation of the questionnaire developed by Ferrer Torregrosa (Ferrer Torregrosa 2014; Ferrer-Torregrosa et al. 2015) that measures the expectations in relation to learning with the use of AR. This instrument was originally designed, in Spanish, to evaluate the application of the AR in the study of leg and foot anatomy in podiatry students. This scale has demonstrated adequate internal consistency (Cronbach’s Alpha = 0.883) and reliability (Ferrer Torregrosa 2014). The original scale consists of 23 questions that are grouped into 5 domains: (1) “Training, attention and motivation”; (2) “Autonomous work”; (3) “Comparison with cadaveric material”; (4) “Three-dimensional understanding”; (5) “New technology” (AR) (Ferrer Torregrosa 2014). For the present study, the domains “Autonomous work” and “Comparative with cadaveric material” were excluded because they were not relevant to the objectives pursued. Thirteen items of the domains “Training, attention and motivation”, “Three-dimensional understanding” and “New technology” (AR) were included. The questions in this last domain were adapted to specifically evaluate the ARSim2care application. The items are assessed using a 4-point Likert scale, between 1 (total disagreement) and 4 (total agreement) and with a Yes or No response in the case of the questions on the domain “New technology” (AR).

2.5 Data analysis

The data were analyzed using descriptive statistical analysis techniques for quantitative variables (mean and standard deviation) and for qualitative variables (frequencies and percentages). A bivariate analysis, Student´s t test, was used to compare continuous variables. Statistical significance was set at p < 0.05. The data were analyzed by using the with SPSS V25.0 (SPSS Inc., Chicago, IL, USA).

2.6 Ethical considerations

This research project complied with the ethical principles established by the Declaration of Helsinki. It was approved by the Ethics, Animal Experimentation and Biosafety Committee of the Public University of (blinded) (PI-012-21). The nature of the study was explained to each student beforehand. The request for consent to participate appeared on the first screen of the electronic questionnaire, in which the student had to select whether to participate in the study.

The data were collected and stored anonymously. Only the research team had access to the study data, and they were kept under password protection.

3 Results

Of the 61 participants who responded to the questionnaire after using the ARSim2care application, 80.3% were female (n = 49) and 19.7% were male (n = 12). The mean age of the sample was 24.07 years (SD 6.77) with a range between 18 and 56 years. The mean rating of interest in new technologies was 8.521 (SD 1.62).

Among the participants, 41.00% (n = 25) had learned the intramuscular injection procedure more than two years ago, 29.5% (n = 18) between one and two years ago, 18.00% (n = 11) between 6 months and one year ago, and 11.51% (n = 7) less than 6 months ago.

Of the 61 participants who responded to SUS, the mean score was 73.15 (SD 15.40) higher than the described score as a good user experience by Lewis and Sauro (2018). The mean SUS score in women was 72.45 (SD 16.55) and in men was 75.83 (SD 9.37). No statistically significant differences by sex were observed. As observed in Tables 1 and 34.5% (n = 21) of the participants described their experience with the application as best imaginable and excellent and 27.9% (n = 17) as good. Only 11.5% (n = 7) suggested a problematic level of usability.

Table 1 SUS scores according to Lewis-Sauro (2018)
Full size table

Regarding the user expectations, most of the items within the domains “Training, attention and motivation” and "Three-dimensional understanding" obtained scores above 3.5 points and were rated with 4 points by more than 70% of the participants (Table 2). In addition, following the analysis performed by Ferrer Torregrosa (Ferrer Torregrosa 2014), the data were simplified by combining the four response categories into two nominal categories, such as Yes/No. Thus, about 90% of students responded positively to the items related to motivation and stimulation in learning or content retention in the domain “Training, attention and motivation”. And similar percentages were found for the item about anatomical understanding in the domain “Three-dimensional understanding”. In relation to the questions on the domain “New technology” (AR), 88.5% (n = 54) of the participants stated that they already knew about AR technology and more than 90% of students considered it effective for learning clinical procedures and stated that their interest in the subject had increased. Additionally, 85.2% (n = 52) even considered that its use could be reflected in better grades.

Table 2 Responses in relation to user expectations in relation to learning with the use of AR (Ferrer Torregrosa 2014)
Full size table

4 Discussion

The aim of this paper was to assess the usability and user expectations of an augmented reality application that allowed health science students and health professionals to visualize internal anatomical structures during training of the clinical technical skills for the performance of invasive techniques.

ARSim2care application was developed as part of a European project entitled “ARSim2Care: Augmented Applications in Clinical Simulation”, in collaboration with three European higher education institutions. The aim of this project was to offer an innovative method for teaching and training clinical technical skills. Traditionally, training on these skills comprises a theoretical explanation and the corresponding practical demonstration of the procedure by the teacher, combined with the supervised practical training on the procedure by the students (Baillie and Curzio 2009; Weller et al. 2012; Kemery and Morrell 2020). This practical training uses manikins or task trainers that rarely include new technology beyond the materials used to make them or some mechanical or electronic interfaces (Lioce et al. 2020). The use of AR by the designed application offers a new way of learning to perform techniques such as intramuscular injection, nasogastric tube insertion, suctioning via tracheostomy tube or endotracheal intubation. This may enhance the internalization of the technique by facilitating its correct performance, as it is possible to visualize the internal structures. Implementing the ARSim2care application in the Microsoft HoloLens® offers the possibility of using this technology while keeping hands free, interacting with them through voice commands or hand gestures (Klinker et al. 2020). In addition, the use of this type of Head-mounted displays (HMDs) devices allows, if necessary, sterile conditions to be maintained in healthcare environments. This is one of the most advantageous aspects of the application. However, in the systematic review by Rodriguez et al. (2021), the mobile devices (smartphones and tablets) were the most used in augmented reality applications in health sciences. (Rodríguez-Abad et al. 2021). This is due, on the one hand, to the requirements for their implementation, for example in our case, for the performance of procedures that require the use of both hands. On the other hand, the high cost of smart glasses makes widespread use difficult. It is easier to use the mobile electronic devices because the massive use by students (Rodríguez-Abad et al. 2021).

Another economic issue is the high cost of application and software development that also influences the extensive use of augmented reality (Rodríguez-Abad et al. 2021). The availability of some free applications would allow the implementation of this technology in teaching. An example is the ARSIM2care application developed within the context of the European project that facilitates its free availability.

The results for the evaluation of the usability show that the user experience with the ARSim2care application was considered as good or excellent by 62.4% of the participants, with the mean score of 73.15 (SD 15.40). Only 7 students reported a problematic level of usability, difficulties in handling that could be solved by increasing the time to become familiar with the device26. This result is higher than the SUS scores obtained in similar studies. For example, in the usability evaluation of an AR triage scenario visualized in a HMD the mean SUS score was 57 (Anderson et al. 2021). Similar result (58.96) was obtained for the usability assessment of an AR prebriefed activity using a head-mounted display which was designed to offer orientation information before a clinical simulation activity (Anderson et al. 2022).

As suggested by Frost et al., augmented and mixed reality technologies, in their various forms, provide the user with a greater ability to be exposed to ideas that would otherwise be difficult to visualize and contextualize (Frost et al. 2020). ARSim2care application is an example of this, as the use of AR for learning invasive procedures that require a deep understanding of internal anatomical structures has proven to be advantageous. Thus, the results confirm that users perceived that ARSim2care application enhanced their three-dimensional understanding of the anatomy related to the procedure.

Moreover, the positive findings about user expectations in relation to learning with the use of augmented reality demonstrate that the application has the potential of improving motivation, attention, and learning. In line with other studies, the motivation has been widely studied due to its important role in the learning process (Rodríguez-Abad et al. 2021). ARSim2care application scored 3.73/4 on motivation for learning the use of AR was assessed as effective for learning clinical procedures and for strengthening the interest on the subject by a vast majority of participants. These results are in line with the findings of other studies evaluating the use of AR smart glasses for teaching laboratory skills or nursing skill training (Kim et al. 2021; Kapp et al. 2022).

However, it is worth mentioning that the use of the ARSim2care application with Microsoft HoloLens® requires solving certain technical issues to avoid difficulties during teaching activities. For example, the room needs to be well-lit to support object recognition and not have too much background noise to avoid speech recognition problems for some people. The importance of environmental conditions is an issue also identified by other studies (Marschollek et al. 2016; Ingrassia et al. 2020).

Another potential problem is the time it takes to learn how the AR device works, in this case the Microsoft HoloLens®. To prevent this, in this study, the students received a preceding session on augmented reality technology and the smart glasses Microsoft HoloLens® to facilitate its use during the practical teaching session. Additionally, in line with what is suggested in the literature (Mendez et al. 2020), during use, each student was guided by an instructor, who was a member of the research team with experience using the app and Microsoft HoloLens® to resolve any difficulties or problems that arose.

Our results present evidence for a preliminary validation of the ARSim2care application as a learning tool in clinical technical skills training for the performance of invasive techniques. However, to further validate this application, it is envisaged to develop additional research with experimental designs to evaluate the students’ competence to perform the different procedures, comparing the training using the application against traditional training.

Finally, we should mention the possibilities offered by the application to popularize its use. As the ARSim2care design is the result of a project financed by the Erasmus + European project, the software is available for free download (https://erasmus-plus.ec.europa.eu/projects/search/details/2017-1-ES01-KA203-038514).

4.1 Limitations

As a potential limitation of this study, it is worth noting the limited sample size, which should be taken into account when interpreting the results of the study. However, for the usability evaluation, the number of participants involved can be considered adequate (Alroobaea and Mayhew 2014) and the inclusion of students from three different universities could have enhanced the representativeness of the sample. Another issue that could be within the potential limitations of the study is the use of a questionnaire adapted from another study (Ferrer Torregrosa 2014) to measure expectations in relation to learning with the use of AR. However, the adaptations were minimal, and a pilot test was carried out beforehand to ensure adequate understanding and relevance of the items.

5 Conclusion

ARSim2care application, is an application that allows the visualization of internal anatomical structures when performing the procedures of intramuscular injection, nasogastric tube insertion, endotracheal intubation, and suctioning via tracheostomy tube, using Microsoft HoloLens® smart glasses. The positive results about the usability and the excellent user perceptions manifested by participants suggest that it could be an effective resource for training of the clinical technical skills for the performance of invasive techniques. Therefore, further research is worthwhile to demonstrate its effectiveness for the learning of health science students in this field. Therefore, further research is worthwhile to demonstrate the effectiveness of using the ARSim2care application for learning by health science students.