Augmented Reality in Orthopedic Surgery Is Emerging from Proof of Concept Towards Clinical Studies: a Literature Review Explaining the Technology and Current State of the Art

Purpose of Review Augmented reality (AR) is becoming increasingly popular in modern-day medicine. Computer-driven tools are progressively integrated into clinical and surgical procedures. The purpose of this review was to provide a comprehensive overview of the current technology and its challenges based on recent literature mainly focusing on clinical, cadaver, and innovative sawbone studies in the field of orthopedic surgery. The most relevant literature was selected according to clinical and innovational relevance and is summarized. Recent Findings Augmented reality applications in orthopedic surgery are increasingly reported. In this review, we summarize basic principles of AR including data preparation, visualization, and registration/tracking and present recently published clinical applications in the area of spine, osteotomies, arthroplasty, trauma, and orthopedic oncology. Higher accuracy in surgical execution, reduction of radiation exposure, and decreased surgery time are major findings presented in the literature. Summary In light of the tremendous progress of technological developments in modern-day medicine and emerging numbers of research groups working on the implementation of AR in routine clinical procedures, we expect the AR technology soon to be implemented as standard devices in orthopedic surgery.


Introduction
Technologic advances significantly influenced the way orthopedic surgery is practiced today and paved the way for recent innovations in surgical procedures. Complex surgical interventions, as needed for the treatment of musculoskeletal tumors, fracture reduction, corrective osteotomy procedures, or instrumentation adjacent to critical anatomic structures require precise preoperative planning and accurate intra-operative execution. Computer-driven approaches have been established in many areas of orthopedic surgery to support the surgeon in facilitating preoperative planning or improving surgical execution [1,2]. Different technological innovations have been implemented in orthopedic surgery such as robotic surgery [3], 3D-printed patient-specific instrumentation (PSI) [1], navigation tools with tracking visualized on monitors [4], and the emerging technology of augmented reality which allow a human-computer interface [5]. Robotic surgery still has significant limitations that prevent broader clinical use, as devices are very expensive [6], spacious, and their setup and maintenance are time-consuming and cost-intensive [7]. PSI has proven as feasible tool with excellent postoperative results [8,9]. However, the design and manufacturing of patientspecific instruments need up to weeks of lead time. Another drawback is that accurate navigated surgical execution requires extensive bone exposition [10].
AR is defined as "a system", where the real world is enhanced with virtual computer-generated sensory impressions that appear to coexist in the same space as the real world [11]. These virtual impressions can be visual stimuli (e.g., holograms), and any other sensorial information such as sound. The AR technology originated from the military sector [11,12] has been widely used commercially in the area of entertainment and gaming [13] and is now promoted for use in orthopedic surgery [14]. Recently, the FDA (U.S. Food and Drug Administration) approved the first AR applications for elective spinal surgery [15,16].
Orthopedic surgical procedures require a vast amount of numeric and geometric information, such as angles of deformity [17], anatomic relations for instrumentation [18], vital parameters such as blood pressure for blood loss control [19,20] or trajectory orientation for instrumentation and implant placement [21••]. In daily clinical practice, these parameters are analyzed preoperatively by the surgeon and made available via prints and/or digital imaging data on screens in the operating room. Still, information is lost between preoperative planning and the execution of the procedure.
AR represents a valuable solution to improve information transfer and consideration during surgery. Current AR approaches are mainly visual and use monitors [22] or headmounted displays (HMD) [23•].
Recent publications on the topic evaluate the potential benefits of this technology, especially in regard to radiation exposure of the patient and staff, procedure time in the operating room, and improvement of accuracy of surgical execution. This review provides a comprehensive overview of the current state of the technology and recent research of AR in the field of orthopedic surgery ( Table 1). The aim of the article is to elucidate basic principles and current applications of AR to orthopedic surgeons, who lack profound knowledge in computer sciences.

Data Preparation for Visualization of Radiologic Imaging and Navigation
In case of surgical navigation, the relevant information for the surgeon is typically extracted preoperatively from twodimensional (2D) or three-dimensional (3D) radiologic imaging. The virtual model then consists of a computed 3D anatomy model including data about the surgical navigation to execute critical surgical steps. Fürnstahl et al. [45••] described the process of data preparation and surgical planning for the purpose of 3D surgical navigation of long bone deformities as follows.
A CT scan of the pathologic bones is acquired in highresolution with a slice thickness of 1 mm or smaller. The data is then imported in a commercial image processing software such as Mimics (Mimics Medical, Version 19; Materialise, Leuven, Belgium). The anatomical region of interest is segmented from the soft tissue using density-based thresholding and region-growing functions. A 3D triangular surface model is generated from the segmented image using the Marching Cube algorithm [46]. The 3D model is then imported into a surgical planning software in order to elaborate a computerassisted surgical plan by stepwise simulation of the surgery. Machine-learning approaches have shown good results for improving automatization of this process [47][48][49][50][51][52], but the gold standard for clinical-grad surgical planning remains humanbased planning performed by experts [53••].

Registration and Tracking
Registration is the process in which the visualized computergenerated object, which can be a radiologic image or a modality of navigation, is superimposed and oriented into situs in the correct position. After registration, tracking enables the visualized object to stay in the right position when moving and to adapt to the user position as well as to the detect instruments and their orientation and movement in threedimensional space. Tracking requires the AR-system to reference the visualization or instrument from its original registration in a spatial room. For instruments, this is referred to as pose reconstruction. Low accuracy of registration and motion  [22,44,[54][55][56], marker-based registration [25,28,34,38], and surface registration [21,23]. Navab et al. [54••] first engineered the camera-augmented surgical c-arm and discovered its potential for augmented reality in the operating room. This system registers intraoperatively acquired X-rays to the camera image of the operation field in a 3D coordinate system shared with the AR-head-mounted-display (HMD), such that the X-rays can then be visualized in real-time [56].
Marker-based registration is based on the registration of marker positions in a 3D coordinate system in relation to the computed reality augmentation, usually a 3D model of the anatomical region of interest. Its accuracy depends on exact positioning of the markers [25,34,38,57].
Liebmann et al. [21••] first introduced intraoperative radiation-free registration by surface digitalization using only an AR-HMD and a pointing device to navigate pedicle screw instrumentation. Their approach registers a CT-based 3D preoperative plan by superimposing the 3D model to the intraoperative bony surface. A marker senses the surface of the exposed bone surface and samples a point cloud intraoperatively. When sufficient information is collected, the unique surface pattern is recognized, and the 3D model including the surgical plan is superimposed on the surgical field using iterative closest point registration [58].

Visualization
Within the field of human-computer interaction, augmented reality can be understood as a class of displays [59]. The fusion between artificial information and real-world images is performed by either optical or video see-through techniques [11]. There are three main approaches to implementing those displays: head-mounted displays (HMD), monitors, and projectors [11]. Display weight, size, and resolution favor the use of an HMD as a simple method of visualization. HMD with a see-through display visualizes information into the field of view of the user [60].
Regarding visualization, there are two main approaches. First, the visualization can be independent from any spatial relation. This application is mainly useful for displaying additional information such as numeric data [61]. A more sophisticated approach displays a visualization depending on associated spatial positions. Thus, the image changes according to the position of the user. This requires position tracking of the HMD. If the device itself is capable to determine its spatial position by means of integrated sensors, it uses inside-out tracking [62]. On the contrary, outside-in tracking uses an external camera system to detect and track the position of the AR device. In outside-in tracking additional hardware is required, and occlusions may occur, which are two disadvantages of this system [63].
Most commercial AR-devices have originally been developed in the area of entertainment and gaming [13]. Therefore, the commercially available hardware itself is hardly appropriate for clinical use since its original capacity was not intended for high-accuracy visualization. The accuracy is mainly dependent on two factors: registration and exact spatial location for tracking. Liebmann et al. [64•] has pointed out the problem with low-fidelity tracking for navigation purposes resulting in drifting virtual models in orthopedic surgery. In 2019, an HMD device engineered for surgical use, Xvision (Augmedics, Arlington Heights IL, USA), was the first to be cleared by the FDA for spinal surgery navigation [15•] and since early 2020, the Microsoft HoloLens has been approved to be used for spinal surgery navigation in a Swiss first-in-man clinical study by Swissmedic.

Spine Surgery
The spinal cord as well as its emerging spinal nerves and accompanying vessels are prone to iatrogenic injury during instrumentation due to the close proximity to the bony structure of the spine. Mispositioning of pedicle screws during spinal fusion surgery can result in neurological or vascular injury with severe long-term sequelae [22]. Therefore, the majority of AR applications in spine surgery address the surgical navigation of pedicle screw instrumentation [15,16,21,22,25,26,28].
Yoon et al. [26] placed forty pedicle screws using the Google Glass (Foxconn, Google, Mountain View, CA, USA) as a head-mounted display (HMD) in 10 consecutive patients. The study group instrumented cervical, thoracic, and lumbar pedicle screws navigated with the Medtronic Stealth S7 (Medtronic Inc., Littleton, Massachusetts) image-guidance system with radiologic imaging using the O-ARM (Medtronic Inc) and visualized with the HMD. The HMD had a voice control feature to control the information to be displayed. The registration, tracking, and navigation were performed by the Medtronic Stealth S7 system. This feasibility study described the use of HMDs during the procedure of pedicle screw instrumentation as safe. No complications were reported in the results.
Molina et al. [15•] placed Th6 to L5 pedicle screws in five male cadaver torsos using the Xvision (Augmedics, Arlington Heights IL, USA). The group navigated 120 pedicle screws and graded accuracy using the Gertzbein scale (GS) [65], a combination of that scale and the Heary classification [66], referred to in this paper as the Heary-Gertzbein scale (HGS). Overall accuracy when using the AR system was 96.7% based on the HGS and 94.6% based on the GS, which is similar to the accuracy reported for computer-navigated pedicle instrumentation. User experience evaluated with the user experience questionnaire [67] was rated as excellent in terms of usability.
Elmi-Terander et al. [22] navigated 253 pedicle screws from Th1-S1 in a clinical study of twenty patients. They used a modified version of the camera-augmented c-arm [55] and also graded accuracy using the GS and achieved overall accuracy of 94.1%. No screws were assessed as Gertzbein grade 3. The group described a decreased time for instrumentation once experience was gained. From a starting time of 17 min required for screw placement, the average instrumentation time dropped to 1.8 ± 0.9 min with increased experience with the navigation system.
The same research group performed a follow-up study in the form of a case-control-study [27] consisting of 20 ARguided versus 20 free-hand instrumented pedicle screws, which confirmed most of their preliminary findings. The clinical accuracy of AR navigation was 93.9% and thus slightly higher compared to the free-hand group with 89.6%. The percentage of perforation was only half as high with AR compared to free-hand screws. No significant difference in instrumentation time could be shown between the groups.
Liebmann et al. [68] developed a new registration method to superimpose a 3D model of the patient vertebra together with planning information including pedicle screw insertion point and trajectory. Their idea was to register a point cloud of the exposed bone surface using a marker-tracked pointing device to the 3D model of the surgery plan. The group navigated L1-L5 screws using the HoloLens on spine sawbone models and reported an accuracy of 3.38°± 1.73 for the screw trajectory orientation and 2.77 ± 1.46 mm for the entry point localization. The mean time required for surface digitization was 125 ± 27 s.
Müller et al. [25] described an image-based registration approach, which was evaluated in a cadaver study. Three spine cadavers were embedded in opaque agar gel to simulate a lumbar torso. They attached optical markers with radiopaque parts to specified anatomical locations on the cadaver and acquired a CT scan with a cone beam CT device. After segmentation and registration of the bony anatomy, the transformation between markers and anatomy enabled real-time overlay of the surgery plan for pedicle screw instrumentation. The proposed approach could achieve an accuracy in pedicle screw placement comparable to navigation with highprecision optical systems.
Wu et al. [28] superimposed radiologic imaging onto the patient's skin using a commercially available entertainment projector. The used system visualized the patient's anatomy to guide needle instrumentation for vertebroplasty. They evaluated the system on a synthetic phantom and verified precision of the system on an animal cadaver. Later, he assessed accuracy of inserting points during vertebroplasty in three clinical trial participants. The mean transition error in entry point location was 4.4 mm, and the system reduced the time of finding the entry point by 70%. They, however, noted that adipose tissue during surgery influences the accuracy of this system since the overlying skin, where the markers are attached, is mobile.

Osteotomies
Precise surgical execution of osteotomies is crucial in corrective procedures to reconstruct the physiologic anatomy [30]. Particularly, complex corrective osteotomies consisting of multiple oblique or curved osteotomy planes are challenging to perform without support through surgical navigation [29,45,69,70].
Fallavolita et al. [30] presented a method to visualize the mechanical leg axis intraoperatively using AR. The group used the camera-augmented c-arm [55] to create a panorama view of the hip center, knee, and upper ankle joint based on three X-ray images. Twenty-five cadaver legs with random varus or valgus deformities were used to validate their method and confirmed it to ground truth CT data with no statistically significant difference. The group stated that the method allowed reliable tracking of the leg axis intraoperatively requiring only 3 X-rays.
Kosterhorn et al. [29] presented a case report about an AR application integrated into a surgical microscope through an HMD. Surgical planning was visualized in situ and allowed the study group to navigate the osteotomy planes of pedicle subtraction osteotomies. The anatomy was registered to intraoperative accessible landmarks of the vertebral body. The procedure itself is a high-risk intervention consisting of invasive osseous reduction of the vertebral body in proximity to the neuronal structures of the spine. The method was first simulated on a sawbone spine model and later implemented in the operating room. In the presented case, the surgeons resected a 27°posterior wedge of the Th1 vertebral body and reported good match with the virtually overlaid navigation template. The pathologic segmental kyphosis Th11-12 improved from 45 to 5°. No complications or neurologic deficits were observed.

Arthroplasty
In arthroplasty, exact implantation of the prosthetic components with respect to the patient's anatomy is a main contributor to successful outcomes, functional recovery, and longevity [32,34,36]. Three studies performed on sawbones evaluated AR navigation of arthroplasties as more accurate than free-hand procedures in hip [32,36] and knee surgery [33].
Ogawa et al. [34] performed 56 total hip arthroplasties in 54 patients superimposing cup orientation through a smartphone into the surgical filed (group 1) and using a goniometer (group 2) for navigating placement and orientation of the acetabular cup component. Three months after implantation, a CT scan was acquired for assessment of the surgical accuracy. AR navigation was significantly more accurate in terms of radiographic anteversion compared to the goniometer method (2.7°vs. 6.8°). The AR system was evaluated as a safe and effective navigation tool for cup orientation. No information on clinical outcome or complications was provided.
In the next step, Ogawa et al. [31••] conducted a randomized controlled trial, where forty-six patients were randomly assigned to undergo acetabular cup placement during THA using either a marker-based AR navigation system or a conventional mechanical alignment guide. They found no differences in acetabular anteversion accuracy, and no clinically important differences in acetabular inclination.

Trauma
The outcome of trauma surgery is highly dependent on exact anatomic reduction of the fractured bone fragments.
Ortega et al. [44] conducted a multicenter study including 50 patients using an HMD to display in-situ intraoperative fluoroscopic images acquired by a c-arm. With this technique, the surgeons' attention left the operative field only five times compared to 207 times with conventional visualization. Radiation exposure was also significantly reduced.
Shen et al. [43] used an HMD for preoperative bending of osteosynthesis plates in 6 cases of pelvic fractures. The group reduced the fracture in a computer-assisted simulation and evaluated the optimal plate design. After determining the optimal plate shape, they bent the plate preoperatively by visualizing the optimal plate template with an HMD or monitor. After surgical sterilization, the pre-bent plates were used in the surgeries. For all patients, good anatomical reconstruction, good functional recovery, and no complications were reported. The surgery time was reduced by a mean of 10 min.
Von Heide [42] compared osteosynthesis, wiring, and implant removal surgeries using an AR application together with the camera-augmented c-arm [55] in 28 cases, in which registered imaging of the fractures was superimposed in-situ on a monitor and compared to 45 cases performed with conventional c-arm fluoroscopy. The group reduced radiation exposure by 46% (18 X-rays) using the AR visualization, but without observing any reduction in surgery time.
Weidert et al. [41] applied the camera-augmented c-arm for distal intramedullary nail locking in 42 bovine forelimb bones superimposing registered imaging of the X-rays in-situ on a monitor. Three surgeons with different levels of experience (beginner, intermediate, expert) conducted the experiments. The study group analyzed surgical accuracy, radiation dose, and surgery time. The main finding and benefit of their AR application was the significant reduction of radiation use, especially for novice surgeons.

Orthopedic Oncology
Oncologic surgery is faced with a constant compromise between maintaining the safety margin required for sufficient tumor resection and excessive removal of functional tissue. Therefore, high surgical accuracy and exact execution are crucial for patient survival and optimal functional outcome.
Cho et al. [38] published an experimental study focusing on bone tumor resection. The group compared the feasibility of using AR with a tablet in comparison to a conventional tumor resection navigation method in 82 porcine cadaveric femurs. The conventional method consisted of an optical tracking system, a display, and a workstation. The group injected bone cement in the cadaver legs to simulate bone tumors. In the resections, the goal was to maintain a resection margin of 10 mm. The mean error was 1.71 mm in the AR group without any tumor violation and 2.64 mm in the conventional group with three tumor violations. The aimed oncologic margin of 10 mm was achieved in 90.2% of AR-guided resections and in 70.7% in the conventional group.
In addition to the cadaver evaluation, one clinical case of AR-navigated resection of a low-grade osteosarcoma in the diaphysis of the tibia was performed. The preoperative plan aimed for a 1.5-cm safety margin. Histologic workup showed a 1.4-cm margin proximally and a 1.7-cm margin distally.
The same research group described a further AR application for pelvic tumors [37]. As in their previous work, the resections were simulated by injecting bone cement in 18 porcine cadaver pelvises. The resection errors were classified into four grades: ≤ 3 mm, 3 to 6 mm, 6 to 9 mm, and > 9 mm or any tumor violation. After evaluation, average resection margin of the AR group was 1.59 ± 4.13 mm in comparison to 4.55 ± 9.7 mm in the control group. The group described the study as a proof of concept. Still, current results do not yet justify a clinical trial without further in-vivo animal studies.
Choi et al. [39] presented a similar study and resected 60 simulated bone tumors in porcine cadaver pelvises to compare AR-navigation using a tablet with conventional navigation. The conventional navigation method was not further described. As in the study of Cho et al., the aimed oncologic tumor resection margin was 10 mm. After analyzing the resected cadavers, AR showed a mean resection margin of 9.85 mm compared to a 7.11 mm resection margin in the control group.

Conclusion
The technology of augmented reality is on the rise, and its application in orthopedic surgery has gained increasing attention opening up new opportunities in surgical planning and execution. The translation of pre-clinical results from proof of concept and feasibility studies to daily practice has been initiated. However, to this day the way AR and HMDs influence our concentration, perception and cognition is far from understood [71]. The sensory impression delivered through an HMD is a new experience to many surgeons. Avoiding overload of information and providing welldesigned user interfaces will be necessary to smoothly integrate this technology in daily clinic practice. For final deployment and adoption, AR needs to be fully integrated into the surgical workflow [72].
The lack of robust and accurate registration and tracking processes which represent major limitations and refinements of this technology are needed to allow implementation of AR systems in the operating room. Here, inaccuracies can lead to misplaced virtual models making the navigation unreliable. Buggy navigation is widely known from other computer-assisted navigation approaches based on optical markers. Different new innovations are coming up; nevertheless, the technology itself needs further development in this direction. The ideal AR system should work automatically and allow not only surgical navigation but also error detection. Ethical Approval Inapplicable.

Human and Animal Rights and Informed Consent
The review article contains studies with human subjects and cadaver experiments performed by the authors. All these included studies have been approved by the appropriate ethical review board and patients have signed an informed consent as described in the articles.
Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/.