Keywords

1 Introduction

Surgery requires a high level of knowledge, cognitive decision-making and communication skills as well as a high level of dexterity [1]. Dexterity typically refers to the ability to precisely coordinate the movements of small muscles in one’s wrists, hands and fingers with one’s eyes in order to perform a given task. This psychomotor skill is not necessarily innate or hereditary and can be acquired and reinforced by experience and practice [1,2,3].

The worldwide, long-standing, gold standard for surgical training is the Halstedian apprenticeship model in which the operating room and patient are used as the forum for teaching and learning [4, 5]. Although effective, this model is not optimal because it requires the apprentice to be exposed to a large number of surgeries that can only be taught by a limited number of faculty mentors. This is problematic in modern medical practice where working hour restrictions, institutional financial pressure as well as ethical and medico-legal issues further reduces training opportunities and hands-on experience for apprentices [4,5,6]. In addition, this method only offers a subjective evaluation of skill acquisition. For these reasons, surgical education has been undergoing an important paradigm shift in recent years wherein simulation-based training (SBT) has grown in importance and is used to effectively and efficiently complement traditional patient-based training [3, 5, 7,8,9].

SBT has several advantages. First, is that it is safe for the patient. Usually occurring in dedicated laboratories, it offers the apprentice the possibility to develop their skills in a standardized environment, without the pressure of the operating room, thus permitting consequence-free mistakes to be made. Immediate objective performance measures can also be obtained, permitting measures-based evaluation criteria compared to the traditional subjective evaluation [5, 7]. Today, several important medical organizations and associations recognize the potential benefits of SBT and are advocating its use, such as the Food and Drug Administration (FDA) who has endorsed its role for a decade. In the specific field of orthopaedics, the American Board of Orthoapedic Surgery (ABOS) and American Academy of Orthopaedic Surgeons (AAOS) have also clearly expressed their support [5, 10, 11].

As in other surgical specialties, SBT in orthopaedics is performed with physical (cadaveric, live animal and synthetic) and computer-based models [3, 7,8,9]. Computer-based models are relatively new compared to their physical counterparts but have been quickly evolving over the last decade due to technological advancements and increased clinician involvement [3]. They can potentially circumvent some of the limitations presented by physical models such as ethical and regulatory restrictions, expensive procurement and maintenance, risk for disease transmission as well as reduced realism with respect to anatomy, tactile feedback and physiological dynamics and the need for supervised training and assessment [3, 7,8,9].

Computer-based models typically teach and/or train the key steps of a given surgery using various input and output modalities and levels of immersion. In its simplest form, a training simulator can consist of a smartphone- or tablet-based application, such as the one developed by TouchSurgery [12]. Although this mobile application does not offer an immersive or natural hand-based psychomotor interaction, it permits apprentices to cognitively simulate the key steps of several standard surgical procedures [9, 11]. In its more sophisticated form, a training simulator can consist of a medical cart workstation using 3D glasses or a VR headset for immersion and haptic devices to manipulate a computer-generated surgical scene. While most forms of computer-based simulation offer opportunities for self-paced and self-directed learning, those that incorporate hand-based interactions with haptics can potentially offer a more realistic learning experience and more advanced metrics with regards to manual skills [3, 8, 9]. Finally interactive systems can provide immediate objective performance measures and feedback favoring optimal learning which not only helps to assess trainee competency and progress but also builds confidence and an understanding of the corresponding technique [3, 4, 7, 9]. Despite their potential, there are few commercially-available computer-based orthopaedic training simulators.

This paper describes the development of a new VR orthopaedic simulator by the National Research Council of Canada (NRC). NRC has previously developed interactive simulation platforms for various surgical specialties including endoscopy, otolaryngology, cardiothoracic surgery [13,14,15,16] and most notably for neurosurgery with NeuroTouch (now distributed as NeuroVR by CAE Healthcare, Montreal, Canada) [17,18,19,20,21]. The simulator developed in this work is haptic-based, multi-procedural and allows for task-based bimanual skills training. The prototype targets three different specialties of orthopaedic surgery: transforaminal lumbar interbody fusion (TLIF) in spine surgery, antegrade femoral nailing in traumatology and orbital floor reconstruction in craniomaxillofacial (CMF) surgery. This paper is organized as follows: Sect. 2 consists of a brief overview of existing commercial haptic-based orthopaedic simulators; Sect. 3 describes the development process employed to create the simulator; Sect. 4 presents the resulting design; Sects. 5 and 6 respectively present a discussion and future work.

2 Background

Virtual reality simulation training has been well established in high-risk industries and pioneering works in laparoscopic surgery have shown evidence of shortened learning curves and improved patient outcomes [5]. Recent reviews have gathered, classified and presented the different simulators that have been developed and reported for skills and surgical procedure training in orthopaedics [3, 8, 22]. From these reviews, it appears that, while a number of research initiatives seem to have led to interesting VR training platforms, only a few have become commercially available. To the best of the authors’ knowledge, this short list includes: Sim-Ortho by OSSimTech (Montreal, Canada), TraumaVision and ArthroVision by Sewmac (Linköping, Sweden), ArthroSim by ToLTech (Aurora, USA), ArthroMentor by 3D Systems (formerly Simbionix, Littleton, USA) and ArthroS by VirtaMed (Zurich, Switzerland), summarized in Table 1.

Table 1. Features of commercially available simulators for orthopaedic training
Full size table

All of the commercial simulators that permit the simultaneous use of two instruments (i.e. bimanual) target arthroscopic training. These simulators allow for surgical techniques and skills training by using 3 degree-of-freedom (DOF) haptic systems for force feedback. In addition, they provide 2D graphics rendering for simulated arthroscopic view and, in some cases, offer an additional 2D patient and/or fluoroscopic view. One simulator outside of the arthroscopic domain uses a single 5DOF haptic system. A 5DOF device allows for torque feedback in addition to force. Thus, bimanual skills training is currently not well covered for open orthopaedic procedures involving the anatomical region of the spine, femur, knee and hip; only training exercises involving one instrument are available. Each of the simulator platforms provide to the user either the original stylus of the haptic device at the hand or have modified end-effectors to connect generic or realistic surgical instrument handles.

3 Method

3.1 General Requirements

The key requirements, defined in conjunction with the SMEs, were to (1) develop a multi-procedural prototype accommodating three different anatomical sites on a single platform. This includes permitting 3D, fluoroscopic and endoscopic views as well as surgical instrument interaction with both non-deformable, hard bone and soft tissue (examples: nerve, dura and muscle). Consequently, a haptic system being able to reproduce the appropriate workspace and tactile feedback for a wide range of tissue stiffness was required. As well, this entails the additional requirement to (2) provide multiple instruments with the possibility for bimanual haptic feedback and single-handed dynamic tool exchange as well as to (3) implement the corresponding virtual surgical instrument models. An additional requirement was to (4) create 3D computational models from anatomical segmentations provided by the SMEs. A final requirement was to (5) conceptualize pedagogical exercises targeting the essential skills navigable with a basic user interface.

3.2 Instrument Interaction Requirements

There were requirements specific to the procedure to be simulated. They focused on representing the functionalities and interactions of a given surgical instrument. For each procedure, the key surgical instruments were identified and are described below. Note that the suction and bipolar instruments are available and used for multiple procedures. More specifically, the suction aspirates blood to clear the operative view and retract tissue. The bipolar is used for grasping and to manage bleeding by cauterizing tissue.

Transforaminal Lumbar Interbody Fusion.

Five different surgical instruments were required for this procedure namely the microdrill, Kerrison punch, bone curette, rongeur and implant applicator. The microdrill erodes bone to thin away the planned resection area. The Kerrison punch cuts out soft and hard tissue while protecting underlying delicate structures. The bone curette strips tissues (softer than bone) and is used to scoop away disc nucleus and scrape cartilaginous endplates. The rongeur grasps and bluntly tears away the disc nucleus. The implant applicator is used to manipulate and position the implant. When bound to the instrument, blunt force applied to the applicator allows the implant to be advanced into the disc opening. The implant can interact with surrounding structures and is geometrically constrained by the disc corridor such that if a proper angle is maintained, it will advance into the proper position.

Lateral Femoral Nailing.

Four instruments were to be made available in this simulated procedure namely an awl, guidewire, reamer and drill. The awl, when forced against the bone and simultaneously rotated, pierces through cortical bone to create a canal. The guidewire then enters this canal and penetrates spongy bone to create the pilot path for the reamer. The guidewire is flexible and therefore bends when any part of its length is constrained by the femoral shaft. The reamer erodes the medullary canal in order to accommodate the femoral nail. Finally, the drill erodes through cortical bone to allow for screws to be inserted for distal locking of the nail. Several reamer head and drill bit sizes should be made available.

Orbital Floor Reconstruction.

Five surgical instruments were required for the procedure, namely the malleable retractor, periosteal freer elevator, plate-holding forceps, screwdriver and endoscope. The malleable retractor holds back (retracts) the contents of the orbit to expose the surgical site. The periosteal freer elevator strips away tissue adhesions to expose bone. The plate holding forceps are used to grasp, displace and position the orbital plate (implant used to cover the orbital floor fracture). The screwdriver manipulates (binds to, displaces and inserts) screws into bone in order to fix the orbital plate into place. The endoscope provides a magnified 2D view to inspect the surgical bed.

3.3 Analysis and Mock-Ups

Meetings were held for each of the targeted procedures with the corresponding SME to establish key steps and essential skills to learn. The required surgical instruments functionalities, their interactions with tissues as well as the anatomical structure/tissue representation and behaviour were elaborated. The emerging conclusions from the analysis stage were conceptualized as mock-ups detailing task definition and simulation scene content. The procedures were broken down with start and end points/conditions, including identification of the surgical view. The simulation scenarios were conceptualized with pedagogical content including: training goals, learning activities, surgical landmarks and cues as well as performance measures. A similar process was previously used by the team for neurosurgical simulation [18].

3.4 Iterative Development Process

The overall quality, accuracy and realism of a medical task simulation is contingent upon an optimal balance between several interdependent factors. These factors are related to both surgical task analysis and technical (software and hardware) development and implementation. A strategy is to obtain guidance and expert feedback at the start of the development cycle to assure that what is being developed will be valid and useful for the end user. Their input is not only necessary at the start of development, but also throughout the process because surgical cues are often automated in the expert; as successively more comprehensive and refined simulations are tested, details regarding surgical cues and intervention/simulation features that were not previously discussed can come to surface. In addition, an optimal level of detail in the models that meet the surgical task objectives and real-time simulation constraints must be validated by the SMEs. For example, the level of detail of the mesh and mechanical models should preserve a realistic perception for the user, both on the visual and haptic level. Due to the nature of the task as well as the extensive number of requirements, an iterative design process consisting of several cycles of implementation, internal testing and external demonstration to end users was used.

Implementation.

The implementation consisted of translating the requirements, as well as the results from the analysis phase into their respective hardware and software components: (1) a hardware design to accommodate the various ergonomics constraints imposed by several anatomical sites and procedures was targeted. In order to adequately capture the different workspaces for the three interventions as well as to represent the required hard and soft tactile feedback, it was determined that custom haptics would be required; (2) physical handles with custom connectors to permit dynamic tool exchange were fabricated; (3) virtual models of the instruments were implemented within an in-house simulation software. Visual rendering was based on CAD files provided by the SMEs. Collision detection of instruments was primitive-based: cylindrical, spherical and box primitives were used to represent their geometry. New instrument mechanisms (for example flexible and bending wires) and interaction models (for example tissue punching) were implemented, as well as the functionalities to manipulate rigid bodies (surgical objects). New software features were also developed and integrated, in particular, rigid body dynamics and their interaction with tissues and other manipulated rigid bodies; (4) for the development of 3D models, the basic segmented anatomical models provided by the SMEs were systematically refined via sculpting using Blender [23], capturing the relevant tissues/structures as well as their state (for example: whether the tissue is retracted). The computational tissue models were subsequently generated using in-house software; (5) key skills training in orthopaedics were identified for each of the interventions. Generally, in orthopaedics, a positive outcome depends on the surgeon’s proficiency in using appropriate, albeit high, force to perform specific manipulations while aiming to protect surrounding soft tissue [3]. Accordingly, the exercises were designed with an emphasis on performing force-based tasks while capturing errors involving injury to critical structures.

Internal Testing.

Successive iterations of individual software and hardware developments were tested together. Specifically, the visual and haptic realism was evaluated while verifying that real-time performance was met (i.e. simulation’s clock runs at the same speed or faster as a real clock). This involved iterating on the computational mesh resolution while still capturing key anatomical structures as well as adapting the models representing surgical instrument interaction with tissues. It was also verified that hard bone and soft tissue could be distinguished. Mechanical properties from literature were used as a starting point, and subsequently adjusted to achieve realistic behaviour with optimal simulation performance. The usability of the cart by the end-user, as related to ergonomics, was also tested.

External Demonstration.

Several hands-on and interactive demonstrations were scheduled with the SMEs during the developmental process. These demonstrations served to highlight the advancements and validate the simulations from a clinical perspective. More specifically, the anatomical structures and their representation in terms of visual and tactile feedback, instrument behavior and ergonomics of the recreated surgery were discussed and corroborated. The targeted training goals and performance measures were also validated. Feedback and comments from SMEs were collected and subsequently used to guide the next round of developments.

4 Results

A working VR prototype for interactive training in orthopaedic surgery has been developed. The prototype simulator supports three surgical sites (lumbar spine, right femur and the left eye). The sites correspond to three surgical procedures namely, transforaminal lumbar interbody fusion (TLIF) in spine surgery, antegrade femoral nailing in traumatology and orbital floor reconstruction in craniomaxillofacial (CMF) surgery. The simulated procedures are decomposed into nine scenarios using a total of 16 surgical instruments.

4.1 Cart Hardware

The proposed platform is a customized, transportable cart with adjustable height for user comfort while respecting the various ergonomic requirements (Fig. 1a). The workstation uses a 3.7 GHz Intel Core i7-5930K processor with 6 cores and 16 GB of RAM as well as a GeForce GTX 1080 GPU for computing and graphics. It includes two screens, one for 3D vision and one for 2D display and user-interface (UI) navigation. The lower screen supports NVidia 3D vision with glasses and is used to display the surgical workspace. The upper screen is a touchscreen to facilitate UI navigation and serves as a view for an observer. When required, the upper screen can display 2D fluoroscopic and endoscopic views parallel to the surgical workspace.

Fig. 1.
figure 1

(a) Orthopaedic VR simulation cart with lower display enabling NVidia 3D vision; upper display for 2D patient view and fluoroscopy; Haptic devices; VR ready PC; (b), (c): Customized connector for rapid one-handed instrument exchange and automatic recognition;

Full size image

The haptics are 3DOF devices from Entact Robotics (Toronto, Canada) designed to have a customized level of force (1.5N continuous, 6.0N instantaneous) and negligible friction for high resolution bimanual interactions. In terms of workspace, the two haptic devices are separated by approximately 50 cm while the respective reach of each of their arms is about 35 cm.

The handle of each haptic device was removed to accommodate a customized surgical instrument connector permitting quick-release, automatic instrument recognition and signal transmission for articulating instruments. The developed connector also permits one-handed surgical instrument exchange during the simulation. The dynamic, single-handed tool exchange not only enhances user experience, but also accommodates constraints related to the execution of the selected surgical procedures (Fig. 1b and c). The instrument handles that accompany the customized connectors are shown in Fig. 2. The handles include, when required, an activation mechanism (ex: keyhole at fingertip) or a sensor to detect whether articulating instruments are open or closed.

Fig. 2.
figure 2

Instrument handles replicating the ergonomics of the surgical procedure.

Full size image

The required peripherals for surgical instrument activation or intensity control include foot pedals, control knobs and push buttons. The foot pedals are used to activate certain surgical instruments such as the microdrill and to take fluoroscopic images of the surgical scene. The control knobs are used to adjust the intensity of certain surgical instrument functionalities such as the power of aspiration of the suction tool. Push buttons permit instrument selection without changing the tool handle, used for example, when changing the size of a drill bit. Various accessories can be conveniently attached to the simulation cart between the two haptic devices. For example an armrest can be used for bimanual instrument manipulations or tasks requiring stability (Fig. 3).

Fig. 3.
figure 3

(a) Bimanual interactions; (b) Armrest accessory

Full size image

4.2 Real-Time Software

Architecture and Tissue Modeling.

From a high-level architecture perspective, a multithreaded framework to achieve realistic, real-time, simulation of interactions between surgical instruments and tissue has been developed. The framework divides the computational work into graphic, haptic, physics and simulation (manager for exchange of information) threads, at 60 Hz, 1000 Hz, 1500–3000 Hz and 100–300 Hz respectively (Fig. 4a). Parallel computing on CPU and GPU is used for the resolution of the mechanical system of equations associated with tissue deformation and contact (collision detection and force response) resulting from haptic interactions.

Fig. 4.
figure 4

(a) Schematic of the framework with the simulation thread acting as the manager of information exchanged between threads; (b) Femur head multiresolution mesh representation.

Full size image

From a tissue modeling perspective, a multiresolution mesh strategy was used for achieving realistic visual and haptic feedback while maintaining real-time performances. In this approach, the mechanical behavior of a fine tissue model is computed on a coarse volumetric finite element (FE) mesh while visual and haptic rendering make use of a fine surface attached to the fine model. The surface mesh, which consists of triangular elements, corresponds to the zero isolevel of a level-set function and is efficiently obtained by the Marching Cube method [24]. The level-set function is itself defined on the fine volumetric mesh. Figure 4 offers a view of the different meshes for a femur head. The mechanical behaviour of deformable tissues follows a hyperelastic model and is computed using the Total Lagrangian Explicit Dynamics (TLED) algorithm on a structured mesh [25]. Topology changes associated with soft and hard tissue removal (erosion) are supported through sculpting of the level-set surface. Contact between surgical instruments and deformable tissues uses penalty-based forces and virtual coupling models [26].

Instrument Models.

The simulation framework supports manipulations (contact and/or topology change) of deformable tissue and rigid bodies with surgical instruments. It also permits to control (aspiration and cauterization) bleeding and to render different visualisations of the operating field (stereoscopic, endoscopic and fluoroscopic).

The surgical instruments that were developed for this simulation platform can be classified according to four basic functions: dissection, retraction, grasping and blood management. As such, interaction models corresponding to these functions were implemented. These models were further adapted and/or combined to represent the specific behaviour required for each instrument. For example, tissue dissection is achieved using a cutting primitive wherein the volume of tissue cut is determined by that of the cutting primitive on the instrument. The mechanism by which tissue is removed by a specific instrument can be customized by modifying the number, position, and size of the sphere(s). In addition, combinations of the basic interaction models can be implemented in order to achieve advanced behaviour leading to progressive or blunt tissue removal. The manipulation of rigid objects using surgical instruments allows their interaction with other rigid objects and tissues. More specifically, the current framework supports the handling of implants (grasping, binding with and release from articulating instruments such as forceps, displacement within the surgical scene and their insertion and positioning near soft and hard tissues). The simulation can additionally handle screw insertion and removal from hard tissues using a screwdriver, so as to secure simulated plate placement.

4.3 Simulation Exercises

Three orthopaedic procedures are simulated: (1) TLIF in spine surgery, (2) lateral femoral nailing in traumatology and (3) orbital floor reconstruction in CMF. Tables 2, 3 and 4 provide details for the procedures respectively. Based on the surgical goals and key learning objectives identified in the analysis and mock-up stages with the SMEs, each of the procedures have been divided into simulation exercises with specific start and end points. Each of the exercises recreate the surgical view, simulation scenario concept, essential surgical cues and performance measures with the notation (+) for outcome measures and (−) for errors.

Table 2. Key steps of the TLIF spine surgery. The intervention is performed on lower spine to remove intervertebral disc and join two or more vertebrae. It is generally done to decompress and stabilize the spine. The simulation involves the anatomy of L4 and L5 vertebrae.
Full size table
Table 3. Key steps of lateral femoral nailing. The intervention is performed under fluoroscopy to repair a femoral fracture. The simulation involves the left femur with a reduced shaft fracture.
Full size table
Table 4. Key steps of orbital floor reconstruction. The intervention serves to repair a fracture in the orbital floor and restore ocular movement and orbital integrity. The simulated anatomy involves a fracture in the left eye, extending up to the wall of the nose.
Full size table

5 Discussion

The working prototype has been evaluated by the SMEs. It was concluded that the simulator can adequately reproduce the ergonomics of the three different interventions on a single platform. The anatomical representations of tissues and structures have achieved an appropriate level of detail.

The step requiring the most iterations in the process was the one encompassing the development of the anatomical structures and surgical instrument interaction models as it required the most extensive feedback from the SME. The implementation consisted of successive tuning of tissue mechanical properties with SMEs as the reference to what felt acceptably realistic. At the same time, when the anatomical level of detail was insufficient for realistic haptic and/or visual representation, adjustments of the mesh models were required on the developmental side with possible workarounds to meet real-time constraints. The adjustments were then re-evaluated for realistic perception by the SMEs.

An acceptable level of realism has been achieved for the surgical instrument interaction models. Certain improvements could further enhance realism. Particular to this work, and as already previously noted in similar studies [8], it was determined that the simulation of contact with bone can sometimes give the sensation of a slippery surface. This occurs when an instrument tip contacts a convex hard bone surface. A non-slip contact is often difficult to reproduce because it requires a high resolution mesh capturing bone surface rugosity, not easily amenable to real-time simulation with current computational capabilities. Similarly, the ability to discern between the different types of soft tissue based on their texture could further enhance the learning experience. A potential improvement could lie in the implementation of tissue-specific friction models.

Custom haptics with specialized connectors were designed to permit a more realistic experience. In particular, the inclusion of quick-release connectors allows for dynamic tool exchange to more realistically capture the various configurations of instruments-in-hand that a surgeon may use during the intervention. When hard bone is involved, the surgeon may choose to place both hands on one surgical instrument or one hand on an instrument while the other is stabilized on an arm rest or patient back, for example. With soft tissue, the surgeon may be required to perform bimanual manipulations involving the simultaneous use of two instruments, such as retracting critical structures out of the way with the non-dominant hand while performing the task with the other. In addition, the choice was made to provide realistic surgical instrument handles to more closely replicate the ergonomics of the surgical manipulations. As such, the handles were reproduced 3D printing or other machining techniques based on designs from real instrument CAD files when available or fabricated from repurposed surgical instruments.

3DOF haptic devices were chosen for this platform due to their relative affordability and their capacity to provide force feedback while respecting ergonomic constraints of the three surgeries. This set-up was well received by the SMEs. The realism of certain interactions such as medullary canal opening or implant insertion into a disc could potentially benefit from having an additional torque feedback and higher force feedback. Torque could be made possible by upgrading the platform with 5DOF haptic devices for each hand. A haptic system capable of generating higher continuous force (for example, 10N rather than the current 1.5N) could be used, however concerns for user safety and affordability arise.

6 Conclusions and Future Work

There is little doubt that SBT has an important role to play in the current and future of orthopaedic surgery training. There will be continued advances in technology to improve realism and increased availability of simulators which may help to compensate for the reduced hours of experience in the operating room (OR) of surgeons in training.

A prototype of a VR orthopedic training simulator meeting the requirements of the SMEs has been developed. The efforts have resulted in a platform targeting three orthopaedic procedures at three different anatomical sites with a total of 16 surgical instrument-mimicking handles. Each procedure has been split into a total of three exercises highlighting key tasks, according to SME input and technical feasibility, resulting in a total of nine simulation exercises. The exercises include pedagogical content, complete with training goals, instructions and performance measures within a user interface. The simulator prototype allows bimanual haptic feedback and single-handed dynamic tool exchange. Visual feedback includes a 3D vision-enabled screen also permitting endoscopic view. Additionally, a second screen is available for observer display or fluoroscopic view. The next step includes technical evaluation and initial validation by residents and surgeons for formal skills training. Future work could focus on further increasing the realism of the instrument interaction models as well as increasing haptic forces. The eventual goal for this simulator is to achieve transferability evidence of simulation skills to OR surgical practice, thereby demonstrating a clear benefit of VR for training.