1 Introduction

Orthognathic surgery or corrective jaw surgery involves all surgical operations to correct jaw conditions related to lower face structures and airways, temporomandibular joint (TMJ) disorders, aesthetic issues, and malocclusion problems primarily caused by skeletal disharmonies, asymmetries, and mal proportions (Kanithi et al. 2016). The definition of orthognathic surgery and osteotomy includes genioplasty, mandibular angle reduction, and maxillary osteotomy. For example, the mandibular angle split osteotomy (MASO) is the most used procedure to treat prominent mandibular angles. In contrast, maxillary osteotomy and repositioning are executed to correct upper jaw deformities or open bites. In this last case, the incisions are made below the eye sockets, allowing the upper jaw movement, the roof of the mouth, and the upper teeth as a single unit (Kashani and Rasmusson 2016). Through maxillary repositioning, the cut maxillary fragment is translated, rotated, orientated, and realigned to correct the targeted defect (Cortese 2012).

The consolidated solution to perform these surgical procedures nowadays expects custom-made physical cutting guides and resin dental splints. Although this is an affirmed procedure, it has several drawbacks. First, the time required for the design and production precludes using this solution in emergencies (Pietruski et al. 2019). Also, this technique is costly, and costs are increased when multiple guides are required to treat the clinical case or adjust them to inter-operative changes in the surgical plan (Rassweiler-Seyfried et al. 2020).

These issues can be overcome by using extended reality (XR) systems. XR describes those technologies that improve human vision and perception of the world, intertwining the real environment and digital content. In augmented reality (AR), digital details are overlaid in the real world, enhancing the user experience. In mixed reality (MR), the digital content is not only overlapped with the real world, but the user can also interact and manipulate the digital and real objects.

In this context, the perspective is to completely remove and replace the physical cutting guides by directly projecting the guidance lines on the anatomy. The use of MR applications allows both to visualise the patient-specific cutting lines during the surgery and give interactive feedback regarding the status of the surgical operation. Recent studies provide some applications of MR in craniomaxillofacial surgery, such as for mandible reconstruction (Meng et al. 2021) and mandibular angle split osteotomy (Piramide et al. 2022). The literature is still fragmented, and the studies are insufficient to provide surgeons with mature MR solutions capable of replacing physical cutting guides. Further research in this direction is required.

The application proposed in this paper expects the assistance of maxillary osteotomy and repositioning, guiding the surgeon through specific feedback defined through a surgical plan.

The paper is organised as follows. Section 2 illustrates the state of the art concerning XR in surgery, focusing on craniomaxillofacial. Section 3 presents the methodology underpinning the developed MR app. Here, there is a comparison between the traditional (based on cutting guides) and proposed approach (based on the MR app) preoperatory and surgical procedures. Section 4 gives practical information about software development. At last, Sects. 5 and 6 present the validation protocol and achieved results.

2 Research background

XR in medicine has emerged as a training and surgical tool (Syamlan et al. 2022). XR for learning offers the possibility to create a safe environment where users can experience and simulate surgical procedures and fail in their performances without any dangerous consequences for the patient (Brunzini et al. 2022a, b). In the surgical field, XR can be used in preoperative and operative environments. Most head-worn display applications for manual work guidance are designed to support clinical staff on-field during surgery (Schlosser et al. 2021). Most applications refer to maxillofacial surgery, orthopaedics, spinal, and neurosurgical operations where target bodies or structures are almost stationary throughout the procedure (Andrews et al. 2021; Badiali et al. 2020; Mehrotra and Markus 2021; Park et al. 2021). Indeed, this is a crucial requirement for optimal application accuracy.

AR can be used as an operative tool to render the magnetic resonance or computed tomography datasets of the patient over his/her body during the surgery. AR helps the surgeon with a detailed view of the patient’s internal bones and organs (Billinghurst et al. 2014). Thus, AR permits to perform tailored incisions, follow pre-designed cutting planes, or visualise the localisation of blood vessels, nerves, vital tissues, or any dangerously damageable structure, improving the safety of the whole procedure. It also reduces surgery time because the surgeon does not need to look away from the surgical site to collect patient information or consult acquired data (Vávra et al. 2017).

Although the future of XR in medical applications is broad, several limitations exist; accuracy, occlusion, and technological limitations of head-mounted displays (HMDs) are the most relevant ones (Vávra et al. 2017). Furthermore, even if XR can speed up surgeries, the need to design and develop the application is a non-simple and non-short step. It requires integrating knowledge from software developers and physicians to create an affordable, feasible and consistent solution (Vávra et al. 2017).

The registration is the process through which the real physical world (i.e. the patient’s skull) and the digital contents are aligned to create a coherent augmented environment (Schmalstieg and Höllerer 2016) to display digital objects over the physical ones correctly. The registration can be performed using marker-based or marker-less methods (Arora and Parkar 2017). Registration accuracy is crucial in the medical field to make the whole clinical procedure not fail. It must be stable to lighting changes and rapid camera motions (Schmalstieg and Höllerer 2016). However, even if the marker-based method is considered more accurate, two issues remain: the determination of the user’s pose in the real world and the system delay or lag that may create harmful location shifts of the digital objects (Zlatanova 2002). Indeed, the tracking system (defined as a dynamic process of scanning and recognition of the HMD) is a decisive factor affecting the accuracy of the AR/MR application. The marker-based method reduces computational costs; however, the marker dependency allows tracking only when visible (Afif et al. 2013). The main requirement of a suitable marker is the sizing of enclosed features and the type of pattern adopted. The higher the number of features, the easier the detection and recognition of the marker by the tracking system. After the tracking system recognises the marker, it calculates the marker/camera relation. It permits the visualisation of the digital object at the correct location.

Even if AR/MR is still not widespread in the medical field, several research studies exist in the scientific literature. An overview of the medical applications focused on developing surgical guides during maxillofacial or orthopaedic surgeries is presented.

Kramers et al. (2014), Tsukada et al. (2019), and García-Mato et al. (2021) developed three different marker-based AR applications to be used with a smartphone to assist the surgeon respectively during the external ventricular drain, the tibial bone resection, and the front-orbital advancement procedures. In the first case, two markers are employed: The first one is used to stably visualise the segmented anatomy of the patient’s skull with the ventricles highlighted, and the second one is to improve the depth perception of the user and permit the visualisation of the interaction point between the surgical tool and the projected anatomy (Kramers et al. 2014). In the study of Tsukada et al. (2019), a guide marker is employed to visualise the holograms of the tibial axes stably. Holograms are created with a sampling-point process mediated by a pointing marker equipped with a fiducial marker (i.e. to generate the hologram of the vertical tibial axis, two points of the vertical axis are sampled with the pointing device, whose marker is tracked, and with them, the axis is reproduced). Finally, an oval marker is employed to verify the accuracy of the operation. In the García-Mato et al. (2021) study, the AR application expects the recognition of fiducial markers placed on the temporal region of the skull phantom that permit the visualisation of two different types of holograms, one for the osteotomy (cutting lines) and one for the remodelling (cut bone fragment in the target position).

Jiang et al. (2018) developed a marker-less AR application in dental implant surgery to guide the correct positioning of the drill for the placement of dental implants. Through a vision algorithm, a tracking device that recognises drill and dental cast, the system generates the virtual content as a three-level alarm capable of changing the colour of the virtual content (red-yellow-green) based on the actual pose of the drill concerning the planned position in the preoperatory phase. Also, Ceccariglia et al. (2022) developed a marker-less AR application to guide maxillary osteotomies based on the recognition of the patient’s facial profile. In the proposed case studies, the surgeon, through HoloLens 2, could see the virtual surgical planning and/or the custom-made cutting guides superimposed on the patient’s anatomy. Results showed a discrepancy under 2 mm comparing the cutting lines obtained with the AR app and those drawn with cutting guides.

Lin et al. (2016) successfully applied AR for mandibular angle split osteotomy (MASO) with little error assistance from a robotic arm system. Lin et al. (2016) and Lin et al. (2015) employed AR to superimpose 3D digital models of colourful lines denoting the nerves or the sinuses on the actual environment to prevent collateral damage. They used an image target as a marker to precisely align the real and digital worlds during registration. The image target was merged with the mandible through a splint. The application has been used with the head-mounted displays nVisor ST60 (Lin et al. 2016) and Sony HMZ-T1 (Lin et al. 2015). However, they used physical surgical guidance during MASO and dental implant surgery. Jiang et al. (2019) have created another marker-based guiding AR application to drill holes into the mandibles of beagle dogs. Also, in this case, they deployed and tested the system’s accuracy using nVisor ST60.

Also, Microsoft HoloLens has been used for several AR/MR applications in the craniomaxillofacial field. T. Frantz et al. (2018) and T. Itamiya et al. (2017) employed the Microsoft HoloLens with the SDK Vuforia to superimpose 3D anatomical structures and examine the stability and registration accuracy of the holograms over a skull. Ackermann et al. (2021) proposed an AR application for Microsoft HoloLens to assist pelvic osteotomy and fragment reorientation. Two custom-made 3D-printed mounts are equipped with markers (black-and-white patterns) and placed through small screws respectively on the pelvis and over the fragment to be reoriented. The pelvic marker is employed to visualise the osteotomy cutting planes.

In contrast, the marker placed over the fragment is tracked to manage the feedback given to the user. Also, Gao et al. (2019) developed an AR application for Microsoft HoloLens to assist mandibular angle osteotomy split procedures. The application is a marker-based procedure in which two markers (QR codes) are placed respectively on the splint over the mandible and the employed surgical tool: While the first one is used to visualise the holograms of the alveolar nerve, mandible, and surgical plan, the latter is continuously tracked and employed to manage the feedback given to the user. Benmahdjoub et al. (2022) proposed a multimodal marker approach to align Microsoft HoloLens 2 with an electromagnetic tracking system to project a 3D skull model on its physical counterpart. The multimodal marker consisted of two markers, respectively, trackable by the navigation system (through its tracking device) and the AR device (through its RGB camera).

Therefore, the literature provides several AR/MR solutions for assisting maxillofacial osteotomies and repositioning. However, AR/MR applications commonly help maxillofacial repositioning by displaying anatomical structures at the target position to be reached without any interactive guide for the user. Also, several drawbacks and limitations emerge from studying the state of the art. Beyond the technological issues related to the hardware (e.g. holograms visual discrepancy even after registration), the main drawbacks concerning using augmented or mixed reality applications refer to poor tracking (especially with in-motion parts), occlusion, and accuracy. Indeed, when using marker-based tracking methods, poor lighting conditions, line of sight limitations (e.g. high angles between target and camera), and marker obstruction might reduce or interrupt tracking and even cause inaccuracies in the AR/MR visualisation. On the other side, marker-less tracking methods applied to patient anatomy recognition need further improvements in terms of robustness and stability. For example, Ceccariglia et al. (2022), who used the marker-less Vuforia model target tracking based solely on facial profile recognition, suggested the use of other tracking methods (e.g. trackers with QR codes) to improve registration accuracy since the model target recognition is sensitive to variations in light and/or colour of the object, leading to poor performance.

Other limitations of the previous studies concern the AR/MR app tests, which usually involve in vitro studies (without soft tissues), small samples of participants, and experience-dependent results. In this context, this work proposes a new MR application, useful for osteotomy and repositioning, providing interactive, fast, and intuitive feedback to the surgeon, who is then supported in performing the bone fragment resection and replacement frame by frame. The proposed application can speed up the surgery and reduce under/overshooting errors. Moreover, the idea of integrating osteotomy and repositioning assistance in the same MR application is rarely found in the literature, resulting in the main novelty of the proposed work. It is an entirely novel approach to maxillofacial surgery, based on two 3D cubic markers tracking system that has been adequately designed to improve the app’s stability and accuracy, even when the surgeon changes the point of view and the osteotomised fragment is moved and displaced. Also, the MR app has been tested with several inexperienced subjects to avoid obtaining experience-dependent outcomes.

3 Methodology

This research aims to develop an MR approach to aid surgeons in maxillofacial surgeries (osteotomies and osteotomised fragment repositioning) without using physical cutting guides. The MR app deployment to an HMD will give surgeons interactive, fast, intuitive, and effective feedback during surgery. To explain the characteristics and benefits of the proposed approach, the following sections present the current and the proposed procedures split into preoperatory surgical plan and surgery.

3.1 Preoperatory surgical plan

3.1.1 Current and common procedures

Preoperatory planning is the phase in which the surgeon and related staff define all the requirements and steps of the intervention to be performed on the specific clinical case. Head computed tomography (CT) is the starting point for most maxillofacial surgeries to digitise the patient’s skull. Through a segmentation process, the surgeon gets the 3D model of the skull for the preoperatory surgical plan. In addition to the head CT, a 3D scan of the patient’s teeth is commonly carried out through an intraoral scanner before surgery. This activity allows the surgeon to get a 3D model of the teeth.

During the preoperatory surgical planning, the surgeon, supported by a dedicated software tool, establishes the cutting surfaces/curves for osteotomies (e.g. genioplasty and mandible bevelling) and the new position/orientation of bone fragments (e.g. maxillary and mandible repositioning). Once this step is completed, the following activities differ between the current and proposed approaches (Fig. 1).

Fig. 1
figure 1

Preoperatory surgical plan

Full size image

Following the current approach, the surgeon (or a technician) designs a cutting guide for the osteotomy and, if required, another one for bone repositioning. These guides are typically manufactured employing 3D printing processes. According to the kind of surgery, the cutting guide can be integrated with the lower or upper splints.

3.1.2 Proposed procedure

The MR-based surgery approach uses an HMD to run a proper MR app for maxillofacial surgery. The app includes two use modes that the user can select from the appropriate menu. The first is for osteotomies, where the HMD projects two kinds of holograms. Curved pipes represent the osteotomy lines. Cylindrical holograms indicate where drilling the osteotomised fragment and placing the screws. The second mode is for bone repositioning. The HMD projects the hologram of the bone fragment in the final configuration, plus the distance/angle between the actual (during bone repositioning) and the last position/orientation.

To precisely overlap the holograms over the physical prototype of the patient’s skull, specific (patient-based) reference systems must be defined. HMD recognises them as a reference anchor to project holograms with the proper alignment and positioning to the real environment. Markers must be rich in details or features. They must have the appropriate high contrast and not exhibit repetitive patterns. The most straightforward used markers are image targets. QR codes are images rich in features since they create distinct and non-repetitive patterns of tiny black-and-white squares. Moreover, the marker dimensions must be correctly set. They must suit both practical constraints (e.g. do not hamper the surgical procedure) and technical constraints of the HMD (e.g. to be recognisable and trackable).

For the development of the MR application proposed in this work, a 2-marker system has been employed (Fig. 2a). The first marker, M1, is hypothesised to be fixed on the patient’s anatomy and stable throughout osteotomy and repositioning tasks. M1 permits the visualisation of necessary holograms to aid osteotomised fragment repositioning.

Fig. 2
figure 2

Markers

Full size image

The second marker, named M2, instead, is hypothesised to be solidly placed on the osteotomised fragment. It will be stably fixed during the osteotomy task, becoming a mobile marker when the bony component to which it is attached is osteotomised. It is worth noting that according to the specific surgery (e.g. maxilla or mandibular osteotomy or repositioning), the upper or lower markers (M2u and M2l of Fig. 2b) will be used alternatively. M2 will permit the stable visualisation of holograms uniquely throughout the osteotomy, while the associated holograms will solidly move during repositioning. Indeed, M2 is first used to visualise cutting curves and drilling centres stably. The holes are used for fixing the osteotomised fragment. Then, it is also used to visualise feedback and indications on the final position to reach with the osteotomised component.

The edge length for M1 and M2 is 30 mm, sized to be appropriately recognised by the HMD during the surgery. Five different images for each target allow the HMD to track them, without interruption, moving around the patient.

M1 expects the realisation of metallic squared support precisely dimensioned as the marker box face. The marker is fixed above the patient’s forehead (on the scalp) with a small removable screw (0.75-mm screw). The realisation of this element with metallic materials allows it to reach small thicknesses that do not hamper the screw fastening. The metallic support is equipped with four laminar semi-elastic metallic wings that mediate the assembly of the support component with the marker box. The marker box is a hollowed cube deprived of the bottom face and whose faces are internally dug to host the profile of the four wings. The marker box can be manufactured through a coloured 3D printing technology (e.g. HP® Multi Jet Fusion, Stratasys® PolyJet) to imprint the markers on the box faces directly. The coupling of the elastic wings of the metallic support with these custom-made internal slots of the marker box is a fast and stable assembly mechanism. The marker box is also ensured on the patient’s anatomy after the rapid assembly of support wings on marker box slots. M2, a single body with resin dental splints, is realised with the same technology as the M1 marker box.

Once the markers are placed, the MR app scene must be updated by importing the virtual models of markers and the skull, the cutting lines and repositioned osteotomised fragments. Before proceeding with the surgery, a trial run must verify that the preoperatory surgical plan is correctly defined.

3.2 Surgery

3.2.1 Current and common procedures

This section aims to briefly present how maxillofacial surgery is commonly carried out to highlight differences with the proposed MR-based approach (Fig. 3). Osteotomies are common surgical operations. Nowadays, they are supported by physical and custom-made cutting guides surgeons use to cut bone precisely. These guides are solid (single part) with dental splits, so dental arches act as an accurate reference system. Furthermore, holes are drilled to place the second guide and the prosthesis. Once the bone is cut, the osteotomised fragment is repositioned using another custom-made cutting guide and fixed through fixation plates and screws.

Fig. 3
figure 3

Surgery procedure

Full size image

3.2.2 Proposed procedure

The proposed approach has been thought to avoid using physical and expensive cutting guides and help the surgeon reach more precise results, thus improving the patient’s safety and satisfaction. It begins by fixing the reference markers M1 and M2 (M2u or M2l, depending on the surgery to be carried out). Then, the surgeon wears the HMD with the app installed. Ocular calibration is needed before starting to use the device. The first activity the surgeon has to perform before beginning the osteotomy consists of launching the app and calibrating the markers. Indeed, the HMD has to recognise M1, the global reference system that will not change during the surgery.

Once the calibration task is completed, the surgeon can execute the osteotomy. The HMD recognises M2 and its relative position/orientation to M1. Once M2 is identified, holograms of the cutting curves and drilling centres will be projected over the patient’s skull. The surgeon will see holograms superimposed over the bone to be cut. Osteotomy is carried out in two steps. At first, the surgeon defines (on the bone) the cutting path through the HMD. After that, the surgeon takes off the HDM, and he/she will begin the osteotomy by following the path previously defined.

Once the osteotomy is done, the surgeon will dress the HMD again to start the repositioning. In this phase, the HMD projects the hologram of the osteotomised fragment in its final position and orientation. This hologram represents the first guidance for the surgeon during the repositioning. The HMD informs the surgeon about the distance from the final pose of the osteotomised fragment. This information is to improve the quality of this surgery operation. The gap is measured along the mediolateral, anteroposterior, and longitudinal axes. The hologram colour switches from red to green when the surgeon places the bony piece at a distance lower than 1 mm from the final target position. This visual feedback informs the surgeon that the planned position/orientation is achieved. Now, the fixation plates and screws are used to fix the bone.

4 MR application design and development

4.1 Requirements and MR application overview

The application is conceived to guide the clinician in performing two operations (osteotomy and repositioning). Thus, it must guarantee the following functions:

  • Visualise static holograms (i.e. cutting lines, drilling points, and planned target’s position and orientation to be reached)

  • Track the moving structures (i.e. the osteotomised fragment)

  • Give interactive feedback about the movements to be performed by the clinician to reach the planned final position of the mandible or maxilla.

Therefore, the registration results are highly significant in this specific use case. Cubic QR code markers have been chosen to ensure a stable holograms registration, even when the surgeon’s viewpoint changes and the tracking system could lose markers. In this manner, the tracking system can continuously track them by different orientations.

Another critical requirement concerns the digital content appearance and application usability. The MR application must be practical, easy to use, and intuitive in all digital parts. It should be easily understandable even by inexperienced MR users and allow for choosing one of the two operation modes. Also, a significant amount of displayed digital content may distract the surgeon. Hence, only relevant data should be shown; digital content must be well-organised without confusing the surgeon. A method to accurately choose and display digital content should be enclosed in the application (Vávra et al. 2017).

For this purpose, the MR application was designed with a three-button menu to (1) start the application and choose the operation mode (i.e. (2) osteotomy or (3) repositioning). In this way, the digital content to be shown or hidden is easily selected. The clinician can also easily handle this menu. It was conceived to follow the user’s sight or to be moved, scaled, or fixed by the surgeon in the surrounding space with specific and simple hand gestures. Figure 4 shows all the created digital content in Unity (by Unity Technologies).

Fig. 4
figure 4

Overview of the overall digital content in Unity, alternatively visualised through the Osteotomy and Repositioning modes of the MR application (the cutting lines and drilling cylinders refer to the maxillary osteotomy)

Full size image

4.2 Digital content design

The MR application was designed and developed to be preliminary tested in a laboratory setting. For this reason, a three-dimensional skull model from a free online CAD models library (Sketchfabb) was employed instead of an actual use case. The 3D virtual skull model was first used to define the MR digital content and then 3D printed.

To realise a modular prototype capable of simulating different surgeries, markers and osteotomised bones were conceived as single parts to be added or removed according to the kind of surgery to be emulated. The lower and upper splints, the three markers, the cutting lines, and the drilling points were designed based on the pre-surgical plan established by the surgeon.

The three cubic markers (M1, M2u, and M2l) and the splints integral with M2u and M2l were appropriately modelled through Rhinoceros 3D (by McNeel). The 3D skull model consisted of four mesh components: the upper skull, the upper teeth, the lower teeth, and the mandible. This subdivision allows the design of maxillary/mandibular osteotomy/repositioning markers.

To have a light 3D printable prototype to test and validate the MR application, the overall weight of the skull phantom was reduced. The internal skull’s volume and the exceeding bony structures were removed. Then, it was transformed into a shelled form with a 5-mm thickness, suitable to be 3D printed.

As described in Sect. 3.1.2, it was hypothesised to place the fixed marker M1 on the patient’s forehead. For this case study, it was directly embedded in the 3D printable upper skull as a 30 × 30 × 30 mm cube through a bridge structure, simulating the metallic removable screw. As for M1, two other 30-mm-length cubes, which stand for markers M2u and M2l, were created. They have been respectively solidly attached to the maxilla and the mandible through patient-specific splints for the upper and lower teeth. Bridge support allows the rigid connection between the splints and the two markers.

The QR codes were applied as textures to the cube’s faces (one different QR code for each face) to be 3D printable (e.g. through a Material Jetting 3D printer) with the skull prototype. To make the markers recognisable by the HMD, a database with all the markers accurately dimensioned was created in the Model Target Generator by Vuforia. Vuforia is an AR SDK that employs computer vision technology to recognise and track planar images or three-dimensional objects. This tracking permits the virtual contents to be placed correctly on the scene. In this case, the database comprised the three cuboid markers and related QR codes for each face. Even if model target generator enables MR apps to recognise and track 3D objects in the real world based on the object’s shape, some object-related requirements should be observed. Indeed, to have robust tracking, Vuforia recommends avoiding simple shapes such as cubes or spheres and symmetric objects and, instead, using objects with textures and coloured or patterned surfaces, which typically work better. This is why QR codes rich in features have been added to the cubic markers. The database was then exported to develop the MR application in Unity (by Unity Technologies).

Also, starting from the 3D skull model, the cutting lines and the drilling points were adequately designed in Rhinoceros and imported in Unity (everything correctly aligned with M2u and M2l).

The cutting lines for osteotomies (mandibular and maxillary) were realised as coloured pipes with a diameter of 0.75 mm. Cutting lines are for genioplasty, orthognathic surgery, mandibular angle osteotomy, and maxillary osteotomy (respectively, red, yellow, cyan, and green lines in Fig. 5). Then, for a comprehensive case study, drilling centres were designed to suggest the points on the patient’s skull to fix the repositioned osteotomised fragment through plates and screws at the end of the surgery. Based on the surgeon’s indications, drilling points were created as hollow cylinders with a 1.7-mm diameter. This value corresponds to the diameter of the screws in the Stryker manual (Stryker Craniomaxillofacial 2011), 4-mm length for the upper left/right holes, and 6-mm for the lower left/right holes (Fig. 5).

Fig. 5
figure 5

Cutting lines and drilling points for the fixation plates (digital content)

Full size image

4.3 MR application modes

For this case study, the MR app was developed and built in Unity, compiled and distributed in Visual Studio by Microsoft and deployed on the Microsoft HoloLens V2 HMD.

As a first step, the packages of the markers database, the Vuforia Engine, and the Mixed Reality ToolKit (MRTK) were imported into Unity. The markers database permits the use of markers in the MR scene, and the Vuforia Engine package allows recognising and using them actively. For what concerns the MRTK package, the Mixed Reality Feature Tool—Beta version was used. The Text Mesh Pro (TMP) was installed to manage and use prefabs available in the MRTK foundation package. Then, TMP was used to realise the three-button menu that allows selecting the operation mode.

Since, by default, HoloLens represents environment geometries and the user’s hands as meshes in the scene, they were hidden to avoid annoying and confusing the surgeon. Then, the “Clipping Plane” was set to 0.3 because it is suggested as the optimal value for Microsoft HoloLens V2 to recognise the imported Vuforia markers database.

Afterwards, the Unity’s scene was created: The markers and the digital objects to be visualised as holograms were added to the scene.

The application has been developed to allow the following commands:

  • App Start: calibrate the app for repositioning the osteotomised fragment.

  • Osteotomy: visualise the cutting lines and drilling points on the patient’s skull for the placement of the fixation plate.

  • Repositioning: visualise, close to the osteotomised fragment, the step-by-step real-time feedback to precisely support the surgeon towards the final pre-planned position.

4.3.1 App Start

The “NearMenu3 × 1” prefab was employed to allow the operation mode selection. The menu automatically appears when launching the MR app (Fig. 6).

Fig. 6
figure 6

Three-button menu to switch between different modes

Full size image

This menu can be moved or re-sized according to the user’s preferences. The pin button (on the upper right side) allows the user to arbitrarily move it through the gaze or fix it in the environment during the MR experience. The menu comprises three different buttons, and the “On Click” function is activated for each. This component permits rapidly implementing events based on user interaction by simple gestures (air tapping). In detail, the “On Click” component of the first button “App Start” activates a script that implements the system calibration. It locates and orients the cube hologram at the final pose to be reached. The other two buttons implement the switch between the osteotomy and the repositioning modes.

4.3.2 Osteotomy mode

The “Osteotomy Mode” button activates the osteotomy lines and the drilling points (and turns off the feedback for the repositioning), as in Fig. 7.

Fig. 7
figure 7

Osteotomy mode visualisation for maxillary osteotomy

Full size image

Osteotomy lines and drilling points are visualised over the patient/phantom maxilla through M2u and guide the surgeon during the bone-cutting and drilling phases.

4.3.3 Repositioning mode

The “Repositioning Mode” expects the visualisation of a cube precisely dimensioned as M2 with a superimposed reference system to better understand the target pose in terms of position and orientation. Indeed, in providing a reference system even for M2, the surgeon should match the reference system on M2 with the one on the holographic cube; this way, position and orientation are guaranteed per the preoperatory plan.

The Repositioning command also visualises three kinds of real-time feedback: the cube hologram “colour-alarm system”, the delta visualisation panel, and the 3D arrows (Fig. 8). It turns off the osteotomy lines and the drilling points.

Fig. 8
figure 8

Repositioning mode. Arrows and a red cube appear at the app launch (a). The cube turns green when the target position is reached (and the arrows disappear) (b). Guiding feedback about displacements is displayed on the panel on the right

Full size image

To give interactive feedback to the surgeon and improve the intuitiveness of the feedback, the colour of the M2 target changes based on the deltas value (differences in the tracked position of M2 and the target position of the cube hologram). The colour turns from red, wrong position, to green, correct position. A specific script governs this functioning. Then, 3D arrows and the dialogue panel with real-time delta values give additional feedback to guide the surgeon precisely. While the panel shows the gap between the actual and final locations of the osteotomised bone, the 3D arrows are selectively activated to indicate the direction (positive or negative) that the user must follow to reach the final position with the osteotomised fragment. (They are shown if the distance from the target place is higher than a threshold of 1 mm.)

Two C# scripts developed in Visual Studio implement the main functionalities expected by the repositioning mode. Based on the tracked M2u and surgical plan, the first one implements the positioning and orientation of the cube hologram at the final pose to be reached during the repositioning phase. This script is launched at any frame after the user taps the “App Start” button. The second script allows the management of all the feedback elements for repositioning. This script is launched at any frame after the “Repositioning Mode” the user taps the button. The main functions that the script executes are:

  • Calculate the delta between the cube hologram final pose and the current M2u position.

  • Display delta values in the dedicated dialogue panel.

  • Manage the 3D arrows based on delta values.

  • Manage the cube hologram colour (red-green) based on delta values.

Based on the displacement delta values, the basic concept of this function is to activate the 3D arrows corresponding to the directions to be followed by moving M2u to reach the final pose of the cube hologram. It also allows displaying delta values in the dedicated text box of the dialogue panel and turning the cube hologram colour. Red is set when deltas are out of the acceptance range established by the user; otherwise, green.

5 Trial protocol and evaluation procedure

To quantitatively evaluate the accuracy of the MR app, this section presents the validation protocol and evaluation procedure for the two working modes available within the app (Fig. 9). The test bench consists of Hololens 2 equipped with the MR app and a 3D-printed skull (with maxilla and mandible) with markers.

Fig. 9
figure 9

Trial protocol and evaluation procedure for osteotomy (a) and repositioning (b) modes

Full size image

5.1 Trial protocol

The trial protocol consisted of, firstly, 3D printing the skull phantom. The 3D (rigid) model does not account for soft tissues. ZPrinter 450 (by 3D Systems) was employed for this goal. Even if this 3D printer manages colours, its accuracy in creating a texture like a QR code is not enough to be recognised by an HMD. Thus, to solve this problem, QR codes were printed on paper sheets and then applied to 3D-printed markers using bi-adhesive tape. Single parts are joined together through metallic pins. The modular physical prototype is visible in Fig. 10.

Fig. 10
figure 10

Assembled (a) and exploded (b) views of the physical prototype used for validation

Full size image

The preoperatory surgical plan was used to customise the scene of the MR app (i.e. markers and holograms of cutting lines and drilling centres) for this specific case.

The trial included seven volunteers who had never undergone surgery and used extended reality technologies. Testers signed the informed consent form after receiving detailed information about the procedure. Before starting each tester, the HMD was adjusted according to his/her head; an ocular calibration was then executed.

The first task tests the osteotomy mode (Fig. 11a–b).

Fig. 11
figure 11

Osteotomy (a, b) and repositioning (c, d) trials

Full size image

Before beginning the test, the skull phantom was assembled and covered with gummed paper tape. Then, the user was asked to wear the HoloLens device, look at the dental multi-target (M2), and select the osteotomy mode. Using a pencil guided by the visualised surgical guides (cutting lines and drilling centres), the tester traced the cutting curves over the phantom and the four points. The latter stands for the holes used for fixing the osteotomised fragment.

The second task tests the repositioning mode (Fig. 11c, d). To perform this task, graph paper was used to track the final position and orientation of the osteotomised bone repositioning. Since M2 is rigidly attached to the maxilla, bone repositioning was simulated by considering only marker M2. This solution was to simplify the test bench. Even if it limits translations on the mediolateral and anteroposterior axes and orientation on the vertical axis, this remains a valid alternative to preliminary evaluate the MR application. At first, M2 is positioned over the graph paper, and its position is marked. Then, the user is asked to wear the HMD, look at the multi-targets (M1 and M2), start the app, select the “Repositioning Mode”, and manually position M2 guided by the elements of the repositioning feedback. At the end of the task, the final M2 position is marked, and its centroid is extracted and marked.

5.2 Evaluation procedure

For osteotomy task validation (Fig. 12), the aim was to evaluate the distance between the cutting lines and drilling centres drawn by users over the skull phantom with the reference one established during pre-surgical planning. To achieve this goal, a 3D structured light scanner (Go!Scan 3D by Creaform) was used for its capability of digitising geometries and textures. The test geometry obtained from the scanner was superimposed on the reference one. A 3D CAD system (Rhinoceros 3D) was then used to rebuild the cutting curves (as 3D solid pipes of 0.75 mm in diameter) and the drilling centres (as spheres of 0.5 mm in diameter). CloudCompare (by CloudCompare.org) was used at last to assess the deviation between the reference and test geometries.

Fig. 12
figure 12

Validation test for osteotomy mode. Rebuild test geometries from 3D scans (a). Overlapped test (red)-reference (yellow) geometries (b)

Full size image

For repositioning task validation (Fig. 13), the aim was to compare the M2 centroid position reached with the aid of the MR application with the target one, measured on graph paper. The distance is measured in absolute deviation along the anteroposterior and mediolateral axes. At last, the absolute angular deviation around the vertical axes was also measured. The deviations were calculated directly on a paper graph using rulers and goniometers.

Fig. 13
figure 13

Validation test for repositioning mode

Full size image

6 Results and discussion

6.1 Osteotomies

Results about mandibular osteotomies derive from Brunzini et al. (2022a, b). The trial was extended with the maxillary osteotomies. The overall mean operational time for the first trial of mandibular osteotomies (i.e. drawing five cutting lines) was 2′56″. The mean execution time for the maxillary osteotomy (i.e. drawing two cutting lines and four drilling holes) was 2′15″. Findings in Table 1 demonstrate that for mandibular osteotomies, the overall mean variation between the intended and drawn cutting lines is less than 2 mm for the five surgical guides, except for one trial. For maxillary cutting lines, the mean deviation decreases by around 1 mm. At the same time, it increases a bit for the drilling holes (the average is under 1.5 mm).

Table 1 Mean deviations between the designed and the drawn cutting lines and drilling holes
Full size table

Then, each of the different cutting lines was evaluated to determine whether the MR application in genioplasty (frontal cutting line on the chin), orthognathic surgery (two middle-lateral cutting lines for mandibular advancement), mandibular angle osteotomy (two retro-lateral cutting lines for prominent mandibular angle reduction), and the maxillary osteotomy is feasible and suitable. The Pareto chart was computed for each surgery to assess the deviations at 75% of the total number of occurrences (results in Table 2).

Table 2 Cutting lines Pareto results: 75% of the cumulative distribution function.
Full size table

In oral and maxillofacial surgery, the accuracy is acceptable if it is between 1 and 2 mm (Meulstee et al. 2019). Indeed, 1-mm inaccuracies at the skeleton level are not noticeable at the cutaneous level. For this reason, results are not acceptable only for mandibular angle bevelling (Table 2). On the contrary, genioplasty results are satisfactory since they are lower than the acceptability threshold. Maxillary osteotomy results are comparable to those for the orthognathic surgery due to their similar position and orientation on the mediolateral plane. They both fall within the allowed range for using the MR system during maxillofacial surgical procedures. To notice that the additive manufacturing process used to realise the physical prototype has intrinsic accuracy (± 0.2 mm for ZPrinter 450). Such a value, anyway, does not significantly influence the most relevant conclusions of this study.

Moreover, the obtained results are comparable with the literature on Microsoft HoloLens and nVisor ST60 on such surgical applications, as shown in Table 3. Due to the novelty of the application, a precise comparison with the results obtained in the same kinds of CMF surgeries (genioplasty, orthognathic surgery, mandibular angle and maxillary osteotomies) is impossible. For this reason, Table 3 shows the overall mean error for the proposed application, averaged on the four surgeries, and contrasted towards the overall mean errors of similar applications.

Table 3 Literature results on similar applications
Full size table

6.2 Repositioning

The repositioning task was performed by moving the marker M2 over a graph paper sheet guided by the MR feedback elements. In detail, the repositioning activity was set with a 20-mm translation on the anteroposterior and mediolateral axes. Null rotations on all three axes resemble the type of motion the surgeon performs after the fragment has been osteotomised. Linear deviations on the anteroposterior and mediolateral axes and the angular deviation from the vertical axis were computed.

Each user performed three trials of the same repositioning task. The execution time remained lower than 2 min with a mean value of 52 s (after air tapping the “Repositioning Mode” button). Table 4 reports mean deviation values for each user.

Table 4 Results of repositioning task: anteroposterior, mediolateral, and angular deviations.
Full size table

Technical issues were encountered during a few trials; the main one referred to the cube hologram positioning temporary crash and the need to re-launch the application. Indeed, sometimes, the final pose to be reached did not correspond to the set one, and the app needed to be restarted. Also, the hologram stability appeared unstable once. User 2 positioned M2 uniquely based on cube hologram visualisation. He was unconfident of displayed delta values. User 6 was not able to perform the task due to visual problems.

These results can be compared with literature studies where medical instrumentation positioning or anatomical structure repositioning was treated. The developed MR application shows significantly better results than other studies found in the literature, such as Ackermann et al. (2021) and Gao et al. (2019). The latter found a mean position error of 3.26 ± 1.40 mm, a mean orientation error of 8.11 ± 2.67° and a mean operation time of 17.32 ± 1.86 min (for a mandibular angle split osteotomy performed by engineers through Microsoft HoloLens). Other scientific studies exhibit results in line with the developed application. García-Mato et al. (2021) study reported a 0.70-mm mean translational deviation and a 0.43° angular deviation for supraorbital advancement operations, a 0.67-mm translational deviation, and a 0.39° angular deviation for frontal advancement operations. Koyachi et al. (2021) study reported a 0.38-mm absolute deviation for the maxillary/mandibular repositioning through Microsoft HoloLens. However, it must be noted that Koyachi et al. (2021) performed the repositioning using the MR surgical guides and the custom-made repositioning instrument that is not employed in the present study.

Hence, results are promising, even though the overall application can improve stability to prevent crashes and guarantee the optimal combination of all the feedback elements. Moreover, even though this can be a valid preliminary study to estimate the precision of the developed MR application, a complete preoperatory plan with non-null rotations and translations about all three axes must be tested.

7 Conclusions

The paper presented an MR app deployed in Microsoft HoloLens 2 for craniomaxillofacial surgeries. The adoption of MR overcomes limitations given by physical cutting guides. The MR app provides interactive, fast, and intuitive feedback to the surgeon during maxillofacial osteotomies and bone repositioning. The app manages genioplasty, orthognathic surgery, mandibular angle osteotomy, and maxillary osteotomy. The proposed maxillofacial surgery approach, supported by MR, points towards the direction of augmented surgery. MR can guide surgeons during the entire surgery by following the procedures established during the preoperatory surgical plan.

The MR application was designed with a three-button menu. The “App Start” calibrates the app for repositioning the osteotomised fragment. The “Osteotomy Mode” visualises the holograms of the cutting lines and drilling points. The “Repositioning Mode” visualises the step-by-step real-time feedback to precisely support the surgeon in placing the osteotomised bone fragment towards the final pre-planned position.

A test bench consisting of a 3D-printed skull, an HMD, and a 3D scanner was organised to validate the MR app. Seven testers used the HMD with the app installed for osteotomies and bone repositioning. For genioplasty, orthognathic surgery and maxillary osteotomy, the testers sketched the cutting lines at a distance lower than 2 mm (maximum acceptable threshold) from the target. The deviation was less than 1 mm for genioplasty. The maxilla repositioning task was performed with an accuracy no lower than 0.7 mm. Results obtained for osteotomies and repositioning are often better than the literature. Validation tests were characterised by the app’s instability (some crashes happened). Results achieved in this study suggest adopting MR, with Microsoft Hololens, in surgery. The markers’ dimensions should be no smaller than 30 mm for good tracking. This size could be cumbersome in some applications. The device cannot track markers if the relative distance is too low (around 250 mm). This limitation could be problematic when the surgeon must stand close to the patient.

Future work must first point towards strengthening the app to avoid instability. Furthermore, marker sizes should be reduced to increase the surgeon’s space without losing tracking capabilities. The test bench should be arranged to be more similar to the actual conditions of a surgery room. For example, splints and markers should be 3D printed using material jetting technology to incorporate the texture directly. Specific manikins (i.e. used for maxillofacial surgery) should be used for a more realistic simulation. The sterilisation effects on markers (e.g. shrinkage or deformation) must be assessed to avoid a drop in accuracy. Still, the repositioning task must let the maxilla be free to move and rotate in the space. Concerning the results, a more extensive set of participants should be involved. Moreover, it would be interesting to evaluate the feelings of surgeons and the accuracy of such an app in the case of a different HMD (e.g. Varjo XR3, by Varjo). Varjo XR3 and Hololens 2 are high-end devices with entirely different projection systems.