A comprehensive study to learn the impact of augmented reality and haptic interaction in ultrasound-guided percutaneous liver biopsy training and education

Abstract

Simulation based training methods are gaining popularity as they could provide a platform for practitioners to gain hands-on experience without causing ethical issues. By combining augmented reality (AR) and haptics, a training method for percutaneous liver biopsy (PLB) could be developed providing realistic scenarios, and real-time visualization of the human anatomy and needle. Additionally, it could also provide real-time feedback to the practitioner. In this review, we describe the conventional PLB procedure, then discuss AR technology and its application in the field of medicine for image-guided therapies, especially, hepatic biopsy. Next, we summarize the associated devices, models and methods illustrating a few haptic simulators devised for training and gesture assessment. Lastly, we present a few potential approaches to integrate AR and haptic interaction to develop a PLB training simulator by accounting the existing challenges.

1 Introduction

Liver illness can have multiple etiologies, and its diagnosis and treatment can be challenging, even with the utilization of non-invasive imaging procedures or blood testing. The primary purposes of a liver biopsy are to: (1) assist in providing a clear and precise diagnosis, (2) assess the extent of liver damage or classify the stage of the tumor, (3) assist in forecasting the likely outcome or course of a patient’s condition who already has a confirmed diagnosis, (4) facilitate the process of making informed judgments regarding the treatment options, (5) observe the progress of an illness or evaluate the effectiveness of a treatment, (6) acquire liver tissue for non-histological evaluation, such as microbiological, biochemical, or other types of analysis (Neuberger et al. 2020). Liver biopsy is a fundamental procedure for diagnosing whether the liver lesion is benign or malignant, which is conducted through histopathological analysis (Dakua and Nayak 2022; Rai et al. 2021). The present techniques for this purpose include percutaneous liver biopsy (PLB), transvenous liver biopsy, laparoscopic liver biopsy, and endoscopic ultrasound-guided Biopsy (Neuberger et al. 2020). Despite the existence of different techniques, PLB has mostly been preferred. Previously, blind liver biopsies were performed, however, the present guidelines dictate to perform blind liver biopsies subsequent to radiological imaging without exceeding three months after the acquisition of imaging data. In order to achieve greater accuracy and reduce the procedural and post biopsy complications, it is recommended to guide the biopsy procedure. Visualization is important for guidance, thus, the selection of imaging modality is crucial.

1.1 Imaging modalities

1.1.1 Computed tomography (CT)

Besides bones or hard objects, the computed tomographic (CT) scans provide better visualization of deeper lesions. Additionally, it allows better placement of biopsy needles to obtain viable tumor tissue (Adnan and Sheth 2021).

1.1.2 Positron emission tomography (PET)

PET scan with the aid of 2-fluoro-deoxy-d-glucose (FDG) produces functional images giving information on metabolic areas. It is helpful in reviewing and locating viable and active lesions (Fei and Schuster 2017). PET could be used with CT guidance in case of multiple lesions to target the most threatening lesion(s) to be biopsied.

1.1.3 Magnetic resonance imaging (MRI)

Magnetic resonance or interventional MRIs provide better visualization of those lesions that are difficult to perceive on US/CT. They have also the advantage of multi-planar accessibility (Adnan and Sheth 2021).

Although CT scans provide better visualization, it is unable to provide real-time guidance due to ionizing radiation. Furthermore, MRI has good visualization with real-time guidance but requires specialized equipment that is compatible with MRI.

1.1.4 Ultrasound (US)

In context with above challenges and limitations of above imaging modalities, the most common imaging modality for PLB is ultrasound (Tselikas et al. 2019). It is helpful in both focal and non-focal liver biopsies providing decent visualization, and safe accessibility via subcostal and intercostal routes with real-time guidance. US is highly popular as it is widely available and cost effective with no radiation exposure, especially for the pregnant and pediatric population. Additionally, US could provide access to visualization of both axial and non-axial planes.

1.2 Training and education

Like other clinical procedures, the risks and challenges are associated with the skill level of the clinician; the third most predominant cause of death in the United States is due to the lack of appropriate skills, and therapeutic errors (Corrêa et al. 2019). Thus, for effective skills transfer, it is necessary to ensure adequate training is provided to the practitioners. It is reported that, on an average, 90 epidural insertions are required to achieve an 80% success rate (Delbos et al. 2022). The undesired biopsy outcome is resulted either due to the improper placement of the needle, needle deviation, tissue puncturing, or unclear visualization of the radiological images (Ravali and Manivannan 2017). Thus, it is vital to ensure practitioners have adequate training in both needle insertion and radiology. Training is further considered to enhance safety and accuracy minimizing the complications (Neuberger et al. 2020). Most importantly, adequate training could offer better tissue samples. Current training methods are restricted to manikins, cadavers, and phantoms. However, cadavers are subject to decay and deform once it is used, whereas manikins are limited to use during certain procedures (Alamilla et al. 2022). Furthermore, the cadaver tissue properties are certainly different from the real patients. Phantoms are subject to wear-and-tear after their repeated usage. With the advancements in technology, it is possible to devise realistic alternative training solutions for medical practitioners. This would offer a platform to practice the procedures and acquire effective skills before practicing on real patients (Favier et al. 2021). Training simulators would allow practitioners to perform multiple attempts without the need of damaging cadavers, harming animals or wearing out materials (Corrêa et al. 2019). Since visualization and feedback to exert the required force is the key in the biopsy procedure, Augmented Reality (AR) and Haptics could be considered in these procedures, particularly those that are guided using imaging modalities (Zhao et al. 2021).

AR has gained decent popularity in clinical domain due to its ability to allow visualization of the human anatomy in three dimensional (3D) view. The patient’s 2D images by US, CT or MRI are rendered, and the corresponding 3D models are visualized through AR decreasing the cognitive load on the practitioner. Additionally, AR has improved the stages of diagnosis and prognosis by contributing to better planning prior to the surgery, reducing the procedure time, radiation exposure, and resources during the procedure. AR decreases the shifts in focus, thereby improving the situational awareness (Park et al. 2020; Tang et al. 2018; Lin et al. 2018). AR has been the point of consideration to perform a percutaneous biopsy procedure and provide real-time guidance with US to view the needle trajectory. The problem in real-life application is the needle deflection upon insertion, producing error in the needle placement. This could be overcome by sensing and updating the needle shape via rendering to achieve precision.

As mentioned earlier, haptic feedback, or the pressures experienced by the practitioners in their hand(s) as they enter the needle into the patient’s body (Corrêa et al. 2019) is another key aspect. Using computer-based simulations (CBS) or AR alone would deprive trainees of haptic feedback during their training period (Delbos et al. 2022). Thus, incorporating haptic simulation along with CBS or AR could provide a more realistic scenario for training (Lelevé et al. 2020). For the field of medicine, the authors Overtoom et al. (2019) propose a haptic simulator for training in laparoscopy. The haptic simulators use sensors to provide critical assessment on medical gestures. Furthermore, active haptic interfaces could replace the phantoms and other passive materials as they provide adaptive simulators, thereby mitigating material damage.

Over the years, the medical sector has experienced rapid developments as traditional manual systems of operation have been substituted by digital healthcare technologies, including electronic health records (EHRs), smart health products, e-prescriptions, wearable devices, telemedicine, and patient interaction management powered by artificial intelligence (AI) (Alolayyan et al. 2020; Chenet al. 2020; Kumar et al. 2023; Amann et al. 2020). Incorporating technologically advanced tools in medicine provides a multitude of advantages for both patients and practitioners, including rapid and precise retrieval of patients’ information, enhanced accessibility to healthcare, dependable diagnoses, and the ability to deliver remote care to the patients remotely located, who may otherwise face difficulties in obtaining adequate medical treatment (Edo et al. 2023). Furthermore, the medical training process has witnessed a paradigm shift from traditional training to technologically aided training (Corrêa et al. 2019; Lelevé et al. 2020; Overtoom et al. 2019; Alolayyan et al. 2020). Numerous studies have emerged that employ the use of various technologies for medical support and training (Corrêa et al. 2019; Ravali and Manivannan 2017; Alamilla et al. 2022; Zhao et al. 2021; Park et al. 2020; Tang et al. 2018; Neves et al. 2020). Thus, it is vital to possess an awareness of innovations in technology and their practicality gaining acceptability within the realm of medicine.

This paper provides a detailed understanding of PLB, concepts and heterogeneous applications of AR, and haptic interfaces in medicine. The detailed review studies the use of AR for visualization and depth perception during image-guided procedures to determine whether it could be considered for efficiently tracking the needle trajectory. Furthermore, we study the role of haptic simulations in training to determine whether haptic feedback could be useful for novice practitioners. The paper is organized as follows: Sect. 2 discusses the procedure followed in PLB (Sect. 2.1) along with its associated risks and complications (Sect. 2.2). In Sect. 3, various AR uses are discussed such as, the display technology (Sect. 3.1), rendering methods (Sect. 3.2), accuracy, tracking, and registration (Sect. 3.3), and a few applications of AR in medicine (Sect. 3.4). In Sect. 4, the mathematical models and method of haptic feedback (Sect. 4.1), the mechanical properties, types and examples of haptic devices in medicine (Sect. 4.2), and gesture assessment (Sect. 4.3) is discussed. In Sect. 5, we discuss the integration of AR (Sect. 5.1) and haptics (Sect. 5.2) to develop a training simulator for PLB procedure, along with the challenges and solutions. We finally conclude the paper in Sect. 6.

2 Percutaneous liver biopsy

Liver biopsy is mainly a diagnostic test to obtain the liver tissue sample and analyze it histopathologically. The liver biopsy is essential to: (1) diagnose the suspected lesion, (2) help assess the extent of liver damage due to the lesion, (3) help examine the tumor staging, and (4) help monitor the disease progression and effectiveness of treatment. Liver biopsies could be taken for non-histopathological assessments, such as biochemical or microbiological assessments (Neuberger et al. 2020).

PLB is primarily performed by inserting a needle into the liver and extracting hepatic lesion tissues. Prior to the procedure, the patient’s clinical evaluation is necessary to evaluate the level of suitability and safety. The patient must lie in a supine position or on their left side. The skin at the site of insertion is prepared by cleaning and injecting local anesthesia. In addition to this, the patient is required to pause their respiration while the needle is being inserted.

Fig. 1

Steps followed in the PLB procedure

2.1 Procedure followed in percutaneous liver biopsy

The steps followed in PLB procedure is depicted in Fig. 1. Once the site of the biopsy is cleaned, local anesthesia is given. Following this, the patient is asked to hold his/her breath commonly after a deep expiration. This is to avoid movement of the liver during respiration as it may cause laceration or intraperitoneal bleeding with possibility of risk of pneumothorax. It is essential to insert the biopsy needle and complete the specimen collection in the same part of the respiratory cycle to avoid complications (Neuberger et al. 2020).

Since the needle insertion and retrieval during liver biopsy should be performed while the patient holds his/her breath, the duration to complete this procedure is a few seconds. However, the entire sample collection procedure takes about 15 min to complete. Previously, a gastroenterologist or hepatologist would perform a blind biopsy. However, with increasing advances in technology, a variety of imaging modalities are now used to perform image guided biopsies. Additionally, various studies have been performed to investigate the benefits of image guided biopsies and the associated risks and complications. It is found that image guided biopsy certainly reduces the risks and complications (Neuberger et al. 2020). This could potentially reduce the need for repeated attempts to obtain a sample of the lesion for laboratory evaluation.

Biopsy needles vary based on the gauge, tip configuration and the desired method to acquire the sample. The two main categories of the needle are aspirating needles and coring needles. However, the needle is selected based on the insertion site, preference of the practitioner, and the availability. Image guidance during the procedure could be used to visualize the size and location of the lesion, the surrounding anatomy and the biopsy needle. The most common imaging modality is US (Tselikas et al. 2019) as it is cost effective, ubiquitous, does not have ionizing radiation and offers real-time guidance (Adnan and Sheth 2021). Other imaging modalities are used when the lesion is not visible using US-guidance.

The minimum recommended diameter of the sample obtained should be 20–25 mm containing at least $\ge {11}$ complete portal triads in it for reliable analysis (Boyd et al. 200). This could be achieved either by aspiration (Menghini) or cutting needle (tru-cut needle) (Neuberger et al. 2020). The tru-cut needles of 16G or 18G in caliber are commonly used; one pass for the biopsy is recommended unless there is insufficient material collected for adequate diagnosis. There is a link established with increasing number of passes giving an improved diagnostic yield. However, studies have shown increased risk of complications and morbidity, when needle passes have been increased to three or more (Chi et al. 2017). The procedure from sterilization to obtaining the biopsy may take a few minutes. Post biopsy, there is a chance of hypotension, tachycardia or any bleeding at the site due to improper biopsy. While performing the procedure, it is important to note that the best needle path is unobstructed from intervening structures and is the shortest straight path to the lesion. Additionally, using imaging modalities, the shaft and tip of the needle must be visualized in real-time during the procedure (Adnan and Sheth 2021).

2.2 Associated risks and complications

PLB is a procedure with high bleeding risk due to breech in the liver capsule causing intra-peritoneal bleeding (Dohan et al. 2015). It is commonly done in patients with prior liver diseases or coagulation disorders with deranged hematological values (Adnan and Sheth 2021). The patient may be on anticoagulant or antiplatelet medications, thereby increasing the risk of hemorrhage due to deranged clotting mechanisms. Hence, necessitating the use of PLB in such cases. According to Society of Interventional Radiology (SIR) guidelines for periprocedural management of thrombotic and risk of hemorrhage, it is recommended to have routine monitoring of prothrombin time (PT), international normalised ratio (INR) and platelet count hemoglobin (Patel et al. 2019). In case of no chronic liver disease, INR correction is made to be maintained $\le 1.8$ and platelet transfusion is recommended, if platelet count goes down below the threshold. Other complications of PLB are those that are common with invasive procedures such as risk of infection (Neuberger et al. 2020). Occasionally, there could be injury to bile duct or bile leak within the liver, which could be reduced with good image guidance (Takyar et al. 2017; Neuberger et al. 2020).

2.2.1 Challenges of using CT in PLB

CT scans provide better placement of biopsy needles to obtain viable tumor tissue information. However, CT scans require the patient to lie in the supine position giving access solely to the axial plane. There is a need for frequent repeated imaging as real-time guidance is not available, thereby exposing patient and clinical technicians to increased ionizing radiation. Additionally, as the biopsy needle obstructs the visualization of the lesion during the procedure, there could be a challenge that the beam of the metal artifact would harden (Adnan and Sheth 2021). To overcome this challenge, the CT scans could be used in conjunction with the US. CT could be used preprocedurally to plan the best route in order to minimize procedural time and could depict anatomical variations to avoid complications later. Whereas, the US would provide real-time guidance in accordance to pre-planned route during the procedure, hence avoiding any complications or delays in obtaining the specimen with reduced radiation exposure. Another alternative could be the use of CT fluoroscopy (CTF), which could provide near real-time guidance and transient contrast enhancement of the lesions. However, it is not typically used for PLB due to narrow access routes and damage to critical nearby structures.

2.2.2 Challenges of using PET in PLB

Both PET and CT could be used intraprocedurally; additionally, PET could be used prior to the procedure and PET-CT during the procedure. However, PET is found to provide no real-time guidance (Adnan and Sheth 2021), as a result, limiting its use during image-guided procedures.

2.2.3 Challenges of using MRI in PLB

MRI requires special biopsy equipment to be compatible with it. Although the titanium needles have been useful in obtaining MRI guided biopsies, they are of limited availability.

2.2.4 Challenges of using US in PLB

Using US, the needle guidance system could help predict the trajectory of the needle prior to the procedure enabling a better planned approach. Additionally, the Doppler flow imaging prevents hemorrhage by accidental puncturing of the surrounding vessels or tumor vasculature. Sometimes in case of chronic liver disease with fibrosis and necrosis present, the use of contrast helps better differentiation of lesions (Sparchez et al. 2019). Furthermore, it is highly operator dependent, and requires adequate training. However, sole dependence on the operator could pose a risk to the outcomes. Thus, US could be integrated with other techniques to aid better visualization of challenging lesions or in case of chronic liver disease with intense fibrosis (Adnan and Sheth 2021).

Fig. 2

AR visualization of a 3D constructed model of the patient liver

3 Augmented reality

Using AR technology, the real world is augmented by either superimposing or placing virtual content in it. AR creates an interactive environment allowing the interaction between the virtual and real world objects while maintaining direct line of sight. This feature makes AR suitable for its use in medicine, particularly during image-guided procedures. Additionally, AR has the ability to provide depth cues and thereby reduce cyber-sickness that may arise. However, the placement of the virtual content is crucial to minimize the shifts in focus of the user and ensuring that the visualization of the desired objects in the real world is not occluded (Park et al. 2020). In image-guided procedures, the images acquired from the different imaging modalities could be converted to 3D virtual objects and could be superimposed on the human anatomical structures as shown in Fig. 2. This aids the practitioner while performing various image-guided procedures.

3.1 Display technology

AR devices could be categorized as head-mounted display (HMD), handheld display (HHD) and stationary display. The HMD displays either make use of optical see-through (OST) or video see-through (VST) displays to augment the real world. As the name suggests, OST displays consist of transparent lenses that provide direct view of the surrounding environment, whereas, VST displays provide an indirect view of the surrounding environment by utilizing a video feed. In contrast, the augmented virtual objects can be projected directly onto physical surfaces in the real world utilizing stationary setups in projection-based AR, such as permanently mounted projectors, or handheld devices like smartphones and tablets. Based on the number of viewing channels in these displays, the devices are categorized as either monocular (having a single viewing channel) or binocular (having two separate viewing channels) (Park et al. 2020). For clinical procedural planning, AR systems superimpose the virtual 3D medical images on the procedural field.

3.1.1 Video-based display

Video-based displays utilize either an external device or an HMD device allowing the practitioner to concurrently view the operating field and the 3D virtual medical images. This display allows multiple practitioners to view the same synced contents simultaneously. Additionally, it manipulates and sets the specifications for the AR system and the practitioners. Therefore, these displays are mostly used in robotic, laparoscopic, and endoscopic procedures (Tang et al. 2018). However, care must be taken to ensure that the 3D camera used to capture the procedural field of view has adequate resolution.

3.1.2 See-through display

See-through displays make use of half-silvered mirrors to reflect the superimposed 3D virtual images. These mirrors are semi-transparent and could either be set up in a HMD device or eyeglasses. The practitioners would be able to visualize the anatomical structures and the operating field simultaneously. AR systems, using see-through displays, do not reduce the tactile feedback during the procedures, and offer a wider field of view in comparison to video-based displays (Tang et al. 2018). However, since HMD devices could add an overhead due to its weight, and suffer from resolution and registration constraints, they are often not preferable in clinical routine.

3.1.3 Projection-based display

Projection-based displays augment the operating field by directly projecting the virtual images onto the field. The projected images are 2D in nature, thereby lacking 3D depth information. Medical AR systems that utilize projection-based displays must take into consideration the curvature of the patient’s body and account for the difference in the visualization of the projected images by additional calibration, tracking, and registration of different anatomical structures. Consequently, the principal tracking and registration of the overall system may be impeded making it unsuitable for use in the procedures that involve small anatomical regions. However, the drawbacks in the conventional projection-based displays could be mitigated by expanding to 3D holographic technology in projection-based displays (Tang et al. 2018).

Fig. 3

Meshes of a CT image of the abdomen that is generated using volume and surface rendering

3.2 Rendering

AR system for image-guided procedures requires rendering the images from the imaging-modalities to 3D virtual images that could be augmented onto the real world. Conventionally, in image-guided procedures, imaging data is acquired before and during the procedure. Therefore, this pre-procedural and intra-procedural medical imaging data could be recorded to reconstruct 3D virtual images. Recording data prior to the procedure and during the procedure helps overcome the challenges posed by organ deformation and movement (Tang et al. 2018). The methods for rendering 3D data are volume rendering and surface rendering. Figure 3 illustrates meshes created from a CT image of the abdomen using volume and surface rendering methods.

3.2.1 Volume rendering

Volume rendering (or direct volume rendering) is a computationally intense process that continuously processes the entire volume of the raw data (Park et al. 2020) by using different algorithms such as volume element projection, footprint table, shear deformation, etc. This process replaces the density values of all the voxels in the imaging data by corresponding opacity values and RGB shades or gray scale based on the RGB transfer function. The by-products of volume rendering are moderately translucent virtual images that will allow the practitioners to visualize the anatomical structures with varying densities. However, in order to visualize the clear-cut volume of soft tissues, the system must first delineate the images and then apply volume rendering algorithms (Tang et al. 2018).

3.2.2 Surface rendering

Surface rendering (or indirect volume rendering) or shaded surface display is a two-fold process in which a small portion of the raw data is segmented (Dakua 2013a, b, 2014, 2015, 2017; Dakua et al. 2016, 2018; Dakua and Abi-Nahed 2013; Akhtar et al. 2021; Dakua and Sahambi 2011) and visualized as surface meshes (Park et al. 2020). This process requires fewer data points and creates triangular meshes using mesh generation methods. In contrast to volume rendering, surface rendering could generate meshes by using the contour outline of the imaging data and eliminate the need to utilize the inner voxels of the data, thereby requiring less computational power (Tang et al. 2018). The ability to render volumetric data using partial data is an advantage of the surface rendering. However, it might compromise on the accuracy of the rendered images.

3.3 Accuracy, tracking, and registration

In order to augment the real world with virtual elements, accurate calibration, tracking, and registration (Mohanty and Dakua year) are of prime importance. In order to project the augmented objects to the user’s interpupillary distance (IPD), the see-through HMD devices must be calibrated to avoid eye fatigue, misalignment, and image distortion (Grubert et al. 2018). Furthermore, prolonged use of these see-through HMD devices may cause discomfort due to distorted depth perception that is caused as a result of these devices having fixed focal planes (Park et al. 2020). Projection-based displays suffer from the need to undergo recalibration in the event of environmental changes to avoid disruption; they must have high resolution and brightness to ensure visibility of the projected AR content. Calibrating projectors presents a substantial problem while achieving the necessary coverage and engagement; it also determines the ideal location and orientation while avoiding distortion or occlusion. Thus, the dimensional properties of the surrounding environment must be accurately tracked. In addition to this, the dimensional properties of the virtual elements must be calibrated and aligned to the tracked properties of the spatial surrounding. In practice, there are several methods that are deployed to accurately calibrate the AR system by achieving precise tracking and registration. Therefore, the AR devices must have the desired specifications to achieve accurate calibration and be useful in image-guided procedures. Presently, the Microsoft HoloLens is considered to be a a suitable tool by providing close to perfect accuracy (Incekara et al. 2018).

3.3.1 Accuracy

AR systems for image-guided procedures must have foolproof accuracy to avoid fatal errors. Since the z-plane in 3D objects suffer from vergence-accommodation conflict (Singh et al. 2020), the additional depth factor in the third axis, or the z-plane, is more challenging to measure. As a result, it is challenging to calibrate the accuracy in 3D planes than in 2D planes. The calibration of the AR system must be done using known markers to account for the zero point, the focal distance and length, and the positioning of the virtual elements, which include the virtual camera and the elements (Tang et al. 2018).

3.3.2 Tracking

The process of recognizing and computing the dimensional properties is known as tracking. For accurate tracking, AR devices must have a minimum of three degrees of freedom (DoF) for position, and three DoF for orientation in the 3D plane. Obtaining high accuracy and registration could be complex using inside-out tracking; it could be enhanced by deploying additional computer vision algorithms that use built-in sensors (Solbiati et al. 2018). Outside-in tracking provides greater accuracy in contrast to inside-out tracking, in addition to offering the advantage of accounting for additional hardware (Lin et al. 2018). The tracking methods used in AR systems could either be infrared, optical, or electromagnetic (Tang et al. 2018). The tracking methods that are used in AR systems can either be marker-based or markerless (Kim et al. 2024). The marker-based technique can either incorporate the fiducial markers, or QR codes in the surrounding. These markers are images or objects that the application can readily identify and use as a point of reference. When the system detects a marker, it is able to accurately ascertain the position and orientation of the camera in relation to that marker (Craig 2013). In contrast, marker-less tracking avoids placing arbitrary markers in the surrounding. It employs methods such as feature and depth tracking to track the relative surrounding (Kim et al. 2024). Another marker-less approach is employing Simultaneous Localization And Mapping, SLAM, to regularly survey and update the position of the device; it augments the objects using multiple sensors embedded in the device (Cadena et al. 2016).

3.3.3 Registration

In the registration process, the calibrated 3D data points and virtual dimensional properties must be aligned with the dimensional properties of the surrounding physical environment. Registration could be a challenging process; it is usually achieved using either built-in sensors or external sensors to track the spatial properties. This process could be either manual, automatic or semi-automatic (Andreß et al. 2018). Similar to the tracking technique, the registration process can either be marker-based or marker-less depending on the employed tracking technique. Many AR systems make use of fiducial markers to facilitate the registration process. The AR system can accurately position augmented objects in the user’s range of view by identifying the markers. By doing so, the accuracy and consistency of the registration process can be ascertained as it is not affected by the change in the user’s perspective. Furthermore, AR systems must consider the dynamic changes that may occur during the procedures, and must therefore, ensure that the tracking and registration processes are continuous and dynamic. Dynamic tracking and registration are computationally intense, yet of prime importance in image-guided procedures. Patient respiration, organ deformation, and patient motion are crucial factors that must be considered in the AR systems used for image-guided procedures (Park et al. 2020). Many systems deploy rigid to elastic switch (Si et al. 2018) or respiratory gating to account for patient respiration and patient motion while superimposing structures onto the patient.

3.4 AR in medicine

The anatomy of the vascular structure is complex to understand from 2D scans. The surgeon/operator has to cognitively use the 2D image and mentally reconstruct a 3D image (Park et al. 2020). AR systems could help proper visualization of the human organ structures in real-time. AR helps surgeons integrate 2D images into 3D images, which reduces the cognitive load on the surgeon. Enhanced imaging reduces procedural time and complications. AR could be used prior to the procedure for viewing the vasculature and planning the most suited path. During the procedure, the 3D model could be referred to anytime to visualize the selected augmented vessel and the catheter position. With the use of AR, the radiation dose exposure could be reduced. For instance, patient specific phantom and AR jointly provide the feel of real patient organ visualization, where AR and electromagnetic (EM) tracking help create a real-time 3D model. This could be considered during the treatment planning phase. This characteristic of AR in regards to percutaneous liver biopsies could help in planning better needle approach, thereby increasing the confidence of the radiologist. Any error in interpreting images or placing the needle could significantly increase the complications. Hence, the practitioners largely insert the needle multiple times to accurately place the needle tip on the lesion and cognitively use the 2D images in 3D setting. AR facilitates this process to make this task easier by reducing the stress level of the practitioners. A study has presented two techniques for interactive registration that strive to have minimum impact on laparoscopic surgical procedure while replicating the effortless control of 3D objects (Joeres et al. 2021).The technique employs laparoscopic instruments that are tracked in place to edit the virtual 3D information.

The HoloLens in computer assisted orthopedic surgery uses data from CT preoperatively and US imaging during the procedure to visualize and generate a 3D AR model. Fiducial markers play an important role in AR; they are used occasionally to assess the system in comparison to the real location (El-Hariri et al. 2018). The fiducials attached to the bone could be used to track the location of the same.

Leger et al. have developed MARIN, a complete system based on a mobile AR system that uses the Intraoperative Brain Imaging System platform (Leger et al. 2020). MARIN is used in image guided neurosurgery, where the mobile device executes in real-time. The AR system was evaluated on a phantom to localize already defined lesions. The evaluation showed the time taken to achieve the task was significantly reduced.

The authors in Neves et al. (2020) used Magic Leap AR goggles to access the frontal sinus and perform external osteoplastic flaps on six cadavers with the help of head and frontal sinus CT images. A 3D hologram is created and visualized on the Magic Leap AR goggles guiding the surgical procedure (Neves et al. 2020).

Using AR in laparoscopic surgery could increase the field of view, motility, tactile feedback and provide good depth perception (Zhao et al. 2021). Lung et al. have developed an AR system that uses electromagnetic (EM) tracking system overlaying the laparoscopic videos and real-time laparoscopic ultrasound (LUS) images for laparoscopic liver resection (Lau et al. 2018). Here, the ablation control over the lesion created is limited.

Another work discusses a tabletop EM-based tracking system for laparoscopic surgery that is overlaid with LUS to view the abdominal structures. The depth of perception is reported to have improved after using a stereoscopic laparoscopic video (Liu et al. 2016).

During a percutaneous bone intervention, AR has shown to reduce the number of attempts for needle positioning and placement, minimizing the radiation exposure and procedure time. AR in percutaneous procedure has additionally minimized the resources being utilized. In microwave liver ablation simulation, the ablation rates are reported to be higher with use of AR as it provides an increased visualization of the tumor margin (Park et al. 2020) improving the treatment coverage.

Liver tumors of large size may not always be shaped regularly. As a result, the needle tip might not be able to reach the location. Percutaneous thermal ablation, being the best approach for large tumors, does face this challenge. The traditional CT or angiography with digital subtraction could provide better guidance to assess the depth and location of the tumor but at a cost of repeated radiation dose (Tang et al. 2018). Although the US imaging modality could help, it provides a 2D image with the ribs obstructing the view of the liver. This could be overcome with the use of AR in fusion with CT scans. AR guided percutaneous liver biopsy with CT scan was proposed and tests were performed on a phantom where respiratory gating methods were used to predict liver movements, the accuracy was $\le 4.5$ mm during expiration and error margin that was $\le 2$ mm.

A needle guidance system (Li et al. 2018) is developed that displays the planned angle of the needle in real time using smart see-through glasses; the accuracy is within 1.33 ± 0.73 degrees.

The usefulness of AR in percutaneous interventions is exemplified by calculating automated path planning while taking the factors such as length of path, proximity to risk structures, and insertion orientation into account (Schwenderling et al. 2022). The authors exhibit the outcomes on a phantom using projection-based AR with a method of choosing an access point through the use of an insertion needle. The user study assesses two versions of the insertion visualization and three different target displays. A visual depiction of insertion points, along with an indication of the quality of the path, leads to the selection of safer routes.

The irreversible electroporation (IRE) helps identify the needle trajectory using HoloLens providing a guide for needle insertion into the pancreas (Kuzhagaliyev et al. 2018).

The HoloLens as a head-mounted display (HMD) is used to visualize the US and holographic MRI images in prostatic intervention by providing image based planning guidance (Morales Mojica 2018).

An AR based app on the mobile/smart glass is used to plan the needle trajectory and give real-time guidance with prefixed trackable targets (Li et al. 2019) for a transperineal prostate procedure. However, the virtual features of the system displayed on the smart glasses are unstable, thereby making it difficult to use.

The AR system specially designed for biopsy and ablation techniques is Endosight (Endosight, Milan, Italy) (Zhao et al. 2021). The endosight forms 3D images from preprocedural CT scans of the patient and the 3D model could be seen on AR goggles or tablets. Endosight with CT images and trackable targets such as fiducials are used to locate tumors and help the needle in targeting (Solbiati et al. 2018).

As stated previously, AR in various studies has shown to enhance pre-procedural and intra-procedural planning while providing real-time guidance. AR is increasingly gaining popularity since it could provide the opportunity to train the novice surgeons or residents using real-life like scenarios (Huang et al. 2018). By comparing VST, OST, projection-based AR, and a traditional monitor-based approach for the needle insertion task, the advantages of offering AR guidance in contrast to a traditionally employed monitor-based approach is determined. The results showing less angular divergence, reduced task completion time, and positive subjective feedback from participants signify the benefits of utilizing projection-based AR (Heinrich et al. 2020). Furthermore, real-time collaborative guidance by experts has been made possible by AR, where experts could view and consult another operator’s AR display (Wang et al. 2017). Distant learning has been made possible through AR on a large scale for those in developing countries to view the live or the recorded procedures that are performed by an expert interventional radiologist (Park et al. 2020). Table 1 summarizes a the discussed applications of AR systems in medicines and highlights the method used, display type, device, advantages and disadvantages of the AR systems.

Table 1 Examples of AR in medicine

4 Haptic interaction

Human–machine interfaces (HMIs) act as a communication channel between the two parties of interface to achieve effective and natural manipulation (Flesher et al. 2021). These interfaces deploy actuators (Mishra et al. 2020), sensors (Yang et al. 2019), and skin-integrated systems (Jung et al. 2021) that take up signals from both the human body and the outside world. Its success depends on the creation of a closed-loop HMI that synchronizes the detection of sensing the inputs with execution directives necessitating the integration of a feedback interface (Yin et al. 2021). When employing HMIs, an ergonomic haptic feedback system uses immersive HMIs to entertain, and assists in coherently performing the tasks (Zhu et al. 2020). With advancement in technologies such as AR and Virtual Reality (VR), there is an increase in the demand for haptic interaction in order to provide more realistic scenarios (Huang et al. 2022). This is due to the fact that AR/VR systems lack haptic feedback and interaction (Sainsbury et al. 2020). In the field of medicine, haptic interaction is categorized into three main groups (Corrêa et al. 2019): (1) haptic systems for medical tests and procedures, (2) haptic systems for training and assessing medical practitioners, and (3) haptic systems for medical intervention procedures.

Currently, many haptic systems are in use for various procedures such as needle insertion, palpation, endovascular procedures, arthroscopy, laparoscopy, endoscopy, etc (Corrêa et al. 2019). Integrating multimodal haptic interfaces could provide compound haptic interaction (Abiri et al. 2019). Therefore, combining haptic interaction with AR could create a realistic environment that could help the practitioners by offering increased immersion, enhancing visualization, and allowing force feedback in training simulators (Herzig et al. 2018). The haptic interaction could be integrated by a haptic device with a simulation model to provide force feedback and gesture assessment. The haptic simulators could thus be used in training for percutaneous needle insertion (Ferrier-Barbut et al. 2021).

4.1 Haptic feedback

Haptic feedback could broadly be categorized into two: tactile feedback and kinesthetic feedback (Huang et al. 2022). Tactile feedback creates sensations on the skin allowing us to experience good tactile perception, such as the texture of the object surface, the vibration, the pressure, etc (Germani et al. 2013). Whereas, kinesthetic feedback is concerned with the sensations of human muscles, joints and tendons, providing us with insight about the weight, tension, gestures, and movements (Mintchev et al. 2019). Haptic interfaces can be force-, thermal-, or nerve stimulation-based. Force-based haptic interfaces are mechanical systems that use sensors to exert force on the skin and muscles, creating skin deformations that mimic how the human body touches or interacts with objects in real life. By utilizing the thermal characteristics of simulated substances, thermal-based haptic interfaces alter the perceived fluctuation in temperature on the skin. Haptic interfaces that use nerve stimulation effectively utilize the electrical impulses to activate human neurons, transmitting information to the brain and producing electrotactile feedback. These technologies greatly enhance users’ sense of immersion across multiple fields, such as robotics and clinical instrumentations (Huang et al. 2022). Force-based haptic interfaces are specifically designed to deliver kinesthetic feedback by means of mechanical forces. Yet, these systems may encounter difficulties when attempting to provide tactile feedback. As a result, they are considered atypical although they can provide great physical feedback, acquiring an elevated degree of realism (Pacchierotti et al. 2017). This typically requires the integration of mechanical actuators (Wang et al. 2019). Haptic devices built for needle insertion procedures must provide haptic feedback, when the needle and tissue come in contact with each other. Haptic feedback is generated by incorporating mathematical models and methods that establish the feedback generated by the haptic interface when it comes in contact with the region of interest.

4.1.1 Mathematical models

Mathematical models are used to determine the behavior of haptic devices and the haptic feedback. In haptic interfaces designed for needle insertion, the most commonly used mathematical models are: Coulomb’s friction, viscous friction, Karnopp, static friction, stribeck effect for static friction and presliding displacement, rising static friction, dwell time, frictional memory, seven parameter model, Dahl Model, Lugre Model for dynamic friction, Hooke’s Law, and Kelvin–Voigt model (Corrêa et al. 2019; Ravali and Manivannan 2017). Figure 4 illustrates the friction modeled by the Coulumb’s, viscous, static friction mathematical model and the stribeck curve.

Fig. 4

Friction curve depicting Coulumb’s fiction, viscous friction, static friction and the stribeck curve on a force vs velocity graph

In the Coulomb’s friction model, the contact area and sliding velocity have less or no effect on the degree of friction. In viscous friction model, the sliding velocity is proportional to the viscous frictional force. The Stribeck curve depicts the Stribeck effect that is caused by the friction produced at steady-state velocity. However, the Coulomb, Viscous and Stribeck models are little effective at zero velocity (Ravali and Manivannan 2017). Static friction is the frictional force, when the interface is static. Having been derived from the Coulomb’s model, the Karnopp model considers a meantime pseudovelocity at zero velocity and changes at the rate of variance between the force acting on the system and the modeled frictional force. However, since the force acting on the system is indefinite, this model is not suitable for real-time interfaces (Ravali and Manivannan 2017). The Presliding displacement model depicts the frictional force that arises due to the displacement of the haptic interface contacts before the actual sliding. The rising static friction and dwell time take stiction force into account and is mathematically depicted by the Armstrong model. The lag between the changes in frictional force and velocity (or normal load) is counted in the frictional memory model. In the seven parameter model, the Coulomb, viscous, static, stribeck, dwell time, presliding friction, and frictional memory are included. However, this model does not specify a mechanism to switch from the stiction phase to the sliding phase (Ravali and Manivannan 2017). The Dahl’s model could be used in dynamic systems as it has the capability to perform simulations for control systems. This model depicts the relationship between friction and the stress–strain property, however, it does not account for the stribeck effect, stiction and stick–slip phenomenon (Ravali and Manivannan 2017). The Lugre Model includes Coulomb’s friction, static friction, viscous friction, stribeck friction, frictional memory, and presliding displacement, thereby overcoming the shortcomings of the Dahl model, and is well grounded for use in linear and rotational coordinates (Ravali and Manivannan 2017). Hooke’s Law in physics is a theory of elasticity which illustrates that the deformation of any material is proportional to the stress applied. Fukushima and Naemura (2014) design a single layer model to gauge frictional force during needle insertion. The authors took the variance between the frictional force and the total insertion force to estimate and analyze the needle tip force. Multiple models could be used in haptic device simulators, and each model requires different mechanical properties to perform the computations providing haptic feedback (Corrêa et al. 2019).

4.1.2 Methods

The employed methods ascertain about the mathematical models being applied and adopted to provide haptic feedback. In order to accurately depict the needle insertion forces, needle insertion simulation models are required to compute the forces using the above discussed mathematical models. Two broad categories of needle insertion simulation models are the deformation-based and the fracture-based models (Delbos et al. 2022).

Fig. 5

Phases of needle insertion

Fig. 6

Forces acting on the shaft of the needle; the red arrows indicate the direction of the force during needle insertion

Deformation-based models take into account the forces that occur when the needle penetrates the tissue layers without considering the underlying physics (Delbos et al. 2022). Okamura et al. (2004) categorize the needle insertion into four phases: prepuncture, puncture, relaxation, and extraction as depicted in Fig. 5. They then devise a method with three parts to measure the forces, which are (1) cutting force, at the tip of the needle, (2) friction forces, along the shaft of the needle, and (3) stiffness forces, which are caused due to visco-elastic deformations as a result of the elasticity of tissues; as depicted in Fig. 6. These models are used in several existing systems (Delbos et al. 2022). With the intention of rendering multiple layers of tissue having its own stiffness and cutting force, recent works have reported the emergence of piece-wise exponential models (Pepley et al. 2017) and nested boxes (Delbos et al. 2022). Furthermore, these models develop and account for different patient morphologies by considering the size of the tissue layers (Pepley et al. 2017) and the box depths (Delbos et al. 2022). The tracking wall algorithm is applied to enhance the cutting force (Delbos et al. 2022). Another approach is to use lateral forces and clamping forces on the sides of the needle to mimic the elasticity of the tissues (Di Vece et al. 2021). The needle divergence from the path is increased as the depth of insertion increases (Delbos et al. 2022). However, none of the approaches have yet considered the needle deflection. Additionally, these models do not consider the effects of velocity on the needle insertion as it penetrates viscoelastic tissues (Barnett et al. 2015). To overcome this issue, a 3-phase piece-wise model (Sadeghnejad et al. 2019) is proposed that considers the tissue fracture and cutting force as a function of needle penetration and velocity. The three phases of the model are: tissue loading deformation, fracture, and cutting. However, it did not account for the extraction phase of the needle insertion. In another approach, contact forces are rendered using non-holonomic restrictions as opposed to stiffness-like functions (Castro-Díaz et al. 2020). In order to render simulations specific to patients, Meshless Total Lagrangian Explicit Dynamics is used to simulate the deformation of the tissues (Wittek et al. 2020).

Fracture-based models regard the needle insertions as cracks that disseminate with applied energy (Delbos et al. 2022). The fracture-based models are categorized into two: energy-based model (EBM) and finite-element model (FEM) that consider the tissue fracture to be caused by the penetration of needles into the tissues, thereby increasing the precision. These models take into account the needle velocity during the insertion. The total insertion force for porcine skin is the sum of tissue fracture (61%), friction (21%), and tissue deformation (18%) (Barnett et al. 2015). The principle of EBM is based on the proposition that the crack propagation cutting the tissue is an outcome of the energy exchange between the tissues and the needle. Realistic simulations (Mohammadi et al. 2021) develop the needle trajectory that use FEM to illustrate the tissue fracture and evaluate the needle tissue interaction forces. Multiple layers of tissue cannot be used for real-time systems since they take a long duration to run (Jushiddi et al. 2021). To overcome this issue, the complexity of mesh generation could be diminished and coupled with the cut finite element method (CutFEM) algorithm (Bui et al. 2019). The CutFEM algorithm has double the computational speed in comparison to classical FEM models allowing for 2D and 3D rendering. However, it is still slow in rendering force feedback. In addition to the force of insertion (Barnett et al. 2015), the diameter and shape of the needle determine the needle deflection during insertion. The needles are reviewed as rigid and deformable (Bui et al. 2019) in FEM models.

Both methods have varying complexities and render forces differently during needle insertion simulations. Accurate guidelines for setting mechanical properties and selecting mathematical models must be in place to simulate needle-tissue interaction forces that occur during needle insertion in order to provide realistic haptic feedback.

4.2 Haptic devices

Haptic devices use the mechanical actuators to provide feedback; the actuators have different mechanical properties and are used to specify the haptics (Pacchierotti et al. 2017). Additionally, haptic devices could be classified based on whether they are specifically built for a purpose, available for commercial purposes, or built artifacts (Corrêa et al. 2019). Haptic devices when integrated with other simulation environments permit the amalgamation of real and virtual elements (Sutherland et al. 2013). However, in most cases the ergonomic facet of haptic devices limit its usability. In order to overcome this issue, commercially available haptic devices could be customized for a specific objective using the simulation models and actuators.

4.2.1 Mechanical properties

The mechanical properties of the haptic device help distinguish different haptic devices and assess its capability and usability while developing haptic systems for specific use cases. The common mechanical properties of haptic devices are: Degrees of Freedom (DoF), Degrees of Force feedback (DoFF) workspace, peak force, inertia, friction, precision, resolution, and bandwidth (Pacchierotti et al. 2017; Delbos et al. 2022).

Haptic devices need to have at least three DoF to render forces (Pacchierotti et al. 2017). In needle insertion simulation, five DoF are necessary to be able to orient the needle at the time of insertion (Delbos et al. 2022). To feel the resistance offered by the axial tissues and sense the forces produced by lateral tissues while orienting and inserting the needle, five DoFs are required. Workspace of the haptic device depends on its usability; in case of needle insertion, the motion sweep and the angle of the needle must be taken into account. The peak force of the haptic device could either be long, short transient, or persistent transient. In haptic devices, inertia and friction must be reduced around the end-effector of contact and workspace. This could be achieved by either mechanical design or control (Gurari and Baud-Bovy 2014). The precision of a haptic device is determined by how accurately the desired feedback and action are fulfilled, whereas the resolution is a more important factor as it determines the magnitude by which the nanoscopic deviations from the desired equilibrium are detected. Lastly, the bandwidth of the haptic device is the measure of how well the system is able to track at any given instance.

4.2.2 Types of devices

As discussed earlier, the types of haptic devices are classified as definite, commercial devices, artifacts, and force capture devices (Corrêa et al. 2019). The definite devices are built for research purposes; the commercial devices are available in the market; the artifact haptic devices are built to record the information other than haptic feedback in forms of either boxes or silicone materials; whereas, the force capture devices are specifically designed to record and evaluate the specifications and characteristics of the anatomical structures. Commercial devices are easy to integrate as they are readily compatible with existing software frameworks. Some of the commercially available haptic devices are: Touch by 3D Systems, North Carolina, Virtuose 6D by Haption GmbH, Falcon by Novint Tech. Inc., Omega 6 by Force Dimension, Switzerland, and Phantom Premium 1.5. While building a customized haptic interface or artifacts, the mechanical properties should be taken into account and the minimum values of these mechanical properties must be attained. The parameters required for haptic models and methods are recorded by the capture devices. Deploying a mock needle on the haptic interface terminal tool could make the simulator more immersive (Delbos et al. 2022).

4.2.3 Haptics in medicine

Many haptic simulators have been developed over the years and are in use, especially in the field of medicine. The simulators are used to train and assess the trainees/novice clinicians (Pozner and Eyre 2021). For instance, a simulator has been developed to train on perilous epidural insertions (Moo-Young et al. 2021). In this system, the realism of the simulator is enhanced by replacing the end effector of the Novint Falcon with a custom-end effector. This simulator has been reported to be both innovative and cost efficient. In order to replicate the force occurring during ophthalmic surgeries, an end effector is developed by using 2 Touch C interfaces (Heimann et al. 2021) and then a virtual world is added to increase the immersion. The developed end effector mimicked the tools used in real-life ophthalmic surgical procedures. A central venous catheterization (CVC) procedure is simulated (Pepley et al. 2017), where the needle is basically simulated featuring a virtual ultrasound probe consisting of a 3D tracker and a touch interface. This simulator is reported to acquire real-time ultrasound images depicting the virtual needle. This work is further enhanced by replacing the virtual ultrasound probe with another Touch interface in order to render the interaction forces between the patient and the probe (Li et al. 2019); the investigators take respiration of the virtual patient into consideration while reproducing real-time haptic feedback. A VR haptic simulator is developed for ultrasound guided needle insertions (Alamilla et al. 2022), where the researchers use a Virtuose 6D desktop in the mechanical part of the simulator to replicate the needle and the stylus of the Geomagic Touch representing the ultrasound probe. Using VR, they depict a real-time 3D and 2D view in addition to depicting an ultrasound view of the 3D scene. A custom haptic interface is developed to provide training for epidural procedures (Sénac et al. 2019) by mounting an artificial needle integrated with a pneumatic cylinder on the Virtuose 6D. As a result, the trainees receive haptic feedback from the needle which is replicated by the haptic device and the syringe. Sometimes electric actuators are used to develop custom haptic interfaces along with a hexapod design (Aygün et al. 2020). The MURAB project has introduced a revolutionary method for conducting cancer biopsies in collaboration with KUKA Robotics and SIEMENS. This methodology establishes a new workflow for combining MRI with US. Users can detect and measure the distortion of specific regions by utilizing relative force feedback and volumetric data. These processes guarantee the accurate identification and removal of small abnormalities with meticulous control, guided by an AR navigation system (Welleweerd et al. 2020). Li et al. create an endovascular catheterization robotic system that utilizes 2DoF force feedback teleoperation system; this technology allows for the effective control of the insertion of catheters and guide wires. The investigators employ hydrogel, heological fluid, and solid magnet to provide haptic feedback on the catheter insertion and guidewire handles. This enables the catheterisation procedure with a smaller amount of effort and time (Li et al. 2022). For robotic MRI-guided needle biopsy of the prostate, Mendoza and Whitney (2019) design a teleoperated system and instrument testbed. The proposed device has 3DoF and a swivel with 2DoF, which securely retains a biopsy needle at the end-effector. The device could effectively identify a sheath puncture event that may be undetected by clinicians. In the field of robot-assisted tele-interventional surgery (RATIS), the primary obstacle is the provision of accurate and precise force feedback. To address this, Shi et al. (2021) develop a catheter operating system that utilizes a robot and a spring-based haptic force interface. The haptic force interface is equipped with a closed-circuit force modulation system, enabling it to deliver realistic force feedback. Additionally, a collision protection feature is employed, which utilizes a collision detection algorithm relying on proximal force to enhance surgical safety. In the event that the algorithm identifies a collision, the haptic force feedback generates a forceful warning to alert the operator, who then modifies their actions to minimize harm to the vessel. Olsson et al. (2013) present an innovative approach for determining the reconstruction of skeletal architecture in patients with face damage, utilizing a virtual model created from patient specific CT scans. The proposed device integrates AR stereo visualization with 6DoF, high-fidelity haptic input to facilitate the study, planning, and preoperative testing of several approaches for repositioning bone pieces accurately. The stereo display offers exact visual spatial perception, while the haptics system provides intuitive haptic feedback when bone fragments come into contact. Table 2 summarizes the aforementioned examples of haptics in the field of medicine.

Table 2 Examples of haptics in medicine

4.3 Gesture assessment

The skills acquired by medical practitioners determine how successfully a medical procedure could be carried out. Haptic interaction simulators could be used to assess and provide feedback to the trainees as they practice the medical procedure in a simulation environment (Marvel et al. 2020). Furthermore, this helps in assessing the results of classical training methods with training simulators. The sensors and actuators are crucial in a haptic device to qualify and quantify the gesture and progress of a trainee. In order to evaluate the gesture and progress, standardized evaluation metrics are necessary; they could be categorized as: (1) subjective tests and (2) objective tests (Close et al. 2020). Objective tests assess the skills and performance of the trainee using some selected metrics (Sharon et al. 2021), whereas the subjective tests assess the trainees based on interviews and questionnaires (Charles and Nixon 2019).

4.3.1 Objective tests

The skills acquired by the trainee are qualified based on the performed gestures. There are many methods to evaluate the gestures, such as the OSATS (Asif et al. 2022) for classical surgical procedures and the GOALS (Higuchi et al. 2020) for minimally invasive surgical procedures. These methods use predefined metrics to qualify the gesture of a trainee and are reviewed in various studies (Cotin et al. 2002). These metrics are: (1) task completion time, (2) needle deviation from the optimal path, (3) regularity of needle movement, (4) economy of needle movement, (5) length of needle movement, (6) trajectory velocity of the needle, and (7) affine velocity (Delbos et al. 2022). These metrics are mostly used to qualify gestures in 3D space and are specific to surgical gestures. However, it is difficult to generalize these metrics as they must be predefined and require references. Furthermore, there is a lack of data that is needed to collocate gestures and define the metrics needed to evaluate each. However, this problem could be overcome by incorporating gesture simulators that could be capable of replicating clinical use cases and extracting data from the gestures performed by trainees. These simulators could be developed in VR (Alaker et al. 2016), AR or haptic devices (Sénac et al. 2021).

4.3.2 Subjective tests

In subjective tests, trainees are assessed based on the interviews and questionnaires presented to them post the training period. There are 3 conventional questionnaires: (1) National Aeronautics and Space Administration Task Load Index (NASA TLX) (Hart and Staveland 1988), (2) After-Scenario Questionnaire-Inter-national Business Machine (ASQ-IBM) (Lewis 1995) and (3) Bibliographic Collection and Usability Scale System (Klug 2017). However, since these questionnaires are specific to the trainees understanding and perception, they cannot be used as a standard metric to quantify the learning of a trainee. Furthermore, these questionnaires are specific to certain applications and cannot be used as a mainstream metric that would be applicable for all applications. Despite its drawbacks, subjective tests are usually preferred as they provide an insight into the confidence gained by a trainee in performing the desired tasks. Another important realization is that, currently, there are no subjective tests that could be applied to quantify the learning of trainees in needle insertion.

5 Discussion

Since liver disease is reported to be the second most cause of dealth world-wide, it is crucial to assess whether the tumor is either benign or malignant at an early stage to prevent it from further growing. PLB is the most common procedure to acquire hepatic cells for histopathological testing. PLB is usually conducted by well trained practitioners to avoid any complications and risks due to improper placement of the needle, needle deviation, or tissue puncturing. Practitioners must master the procedure in terms of the required force and velocity during needle insertion and tissue penetration. Ideally, the practitioners should be well trained and efficient to accurately acquire the lesion samples for diagnosis. During training, the practitioner must be able to visualize the lesion in real-time, the trajectory and progression of the needle, tissues deformations, etc. thereby requiring real-time imaging. Thus, ultrasound would be the ideal choice as it offers a cost effective and multiplanar approach. Although CT and PET scans provide the necessary anatomical image, it fails to be used in real time during the biopsy procedure, making it a less favorable. Furthermore, MRI has its drawbacks in terms of usage as it introduces additional radiation exposure and requires the use of MRI compatible equipment to be used during the biopsy procedure. The ethical issues are bound to arise if cadavers and animals are used as in traditional training. Interests in simulation-based training has grown recently with the development of AR, VR, and Mixed Reality (MR); these visualizations could give the shape of mannikins, phantoms, or any other tangible items without requiring the actual or tangible ones. Furthermore, combining haptics to these systems could make the overall system cutthroat. Mannikins, phantoms, and tangible items are all subject to wear and tear; additionally, they offer limited usage and have to be replaced. The present computer-based simulations lack the haptic feedback aspect, and thus, are less preferred. By combining AR and haptics, hybrid systems could be ideal for training purposes and gesture evaluation. While AR could be used to enhance the visualization of radiological images, haptic interaction could be used to create a realistic training scenario involving needle-tissue interaction forces. Additionally, haptics could be used to evaluate the gesture of trainees and provide feedback to help improve their gestures.

5.1 Integration of AR

AR enables the overlay of digital content over real-world visual information. As a result, AR has found many applications in the medical domain enhancing the visualization of radiological images. Selim et al. (2024) justify that the incorporation of haptic elements into AR inside a teleoperation system would enhance the effectiveness of physicians and the overall security of needle insertion procedures in the liver. As a result, the patient would experience a reduced number of complications following surgery, and the clinicians’ cognitive burden would be minimized. While both AR and VR have their own advantages, distinct features of AR make it especially suitable for complex and interactive medical procedures such as liver biopsies. For liver biopsy training, the trainees have the option to practice on physical models or real patients, where the 3D visual instruction is overlaid on the procedure region. The direct engagement with the physical environment boosts the authenticity and situational relevance, hence optimizing the efficacy of the learning experience. In contrast to VR, which completely immerses users in a purely digital environment, AR allows the users to remain linked to their real-life surroundings. It is vital to keep awareness of the procedural environment and the patient during training. AR allows for the superimposition of digital data onto the real world, providing practical training on real physical objects or patients. The practical experience is extremely helpful for the cultivation of precise manual dexterity and spatial perception necessary for intricate operations such as liver biopsies. AR devices have the capability to promptly deliver feedback on the trainee’s actions, specifically regarding the angle and depth of needle insertion, which is a crucial element of liver biopsy. This feedback has the ability to immediately rectify faults in technique, hence strengthening learning through repeated practice. A similar approach was employed by Ma et al. (2022), where they use a HoloLens for Stereo visual servo-based tracking of surgical tool for the change of the perspective and depth of the robotic stereo flexible endoscope (RSFE). While VR offers advantages for training, such as the ability to collaborate and communicate in real-time, it also enhances teamwork between the anesthesiologist and surgeons (Chheang et al. 2020), AR enables instructors and students to engage in collaborative learning experiences by sharing the same augmented world. Instructors have the ability to provide real-time guidance, demonstration, and correction of techniques, which improve the educational effectiveness of the instruction (Wang et al. 2017). AR apps can easily be used on commonly accessible devices like tablets and smartphones (Leger et al. 2020), allowing for seamless integration into current educational environments without requiring specialized VR equipment. This enhanced accessibility expands the availability of sophisticated training tools to a wider range of places, including distant or resource-constrained settings (Park et al. 2020). Despite the high initial development expenses, AR training modules can prove to be more cost-effective in the long run when compared to VR installations, which sometimes necessitate the use of pricier and more specialized technology (Chheang et al. 2020). AR has the ability to utilize pre-existing technology such as smartphones and smart glasses as shown in Sect. 3.4. Through the utilization of an augmented environment, trainees have the opportunity to make errors without facing any real-life repercussions, hence minimizing the potential harm to real patients during the learning phase. AR provides a secure and regulated setting that closely replicates real-life situations. In conclusion, VR provides immersive experiences that are advantageous for comprehending intricate anatomy and simulating treatments in a virtual environment. On the other hand, AR offers a more sophisticated and practical method for medical training by seamlessly integrating digital aspects with the actual world. AR has numerous advantages for liver biopsy training, including increased realism, hands-on experience, collaborative learning, and improved safety and cost-effectiveness. This makes it the preferred alternative for both educators and trainees. With the recent advancements in machine learning (ML) and deep learning (DL), many studies have emerged that enhance AR and its effectiveness in the field of medicine. von Atzigen et al. (2022) combine the benefits of DL and AR in spinal fusion surgery. The authors present a stereo neural network model for navigating the rod bending process without the need for markers. The AR navigation system converts the identified positions of pedicle screw heads into bending parameters that are utilized to enhance a surgical bending bench, providing guidance to the surgeon. The pipeline not only has the capacity to decrease the time required for bending rods, but also substantially reduces the need for additional bending actions compared to the freehand benchmark technique. DL can be used to remove noise in medical images using techniques like image deformation methods, segmentation, definition of safe areas inside anatomical regions, and delineation. Registration algorithms use neural network architectures to extract and change specific 2D or 3D surfaces on deformable heterogeneous images (Seetohul et al. 2023).

Although many AR systems have been developed for medical domain, it has many challenges. Regardless of AR abundance in the market, projection-based AR could be the ideal choice for training in PLB (Heinrich et al. 2020). For radiological image data rendering, AR is sufficient; it is crucial to ensure that the calibration, accuracy of the tracked objects, and the registrations are accomplished precisely to avoid any misalignment of the overlapped structures. Interactive registration techniques have been introduced which strive to have minimum impact on surgical procedures (Joeres et al. 2021). Zhang et al. (2019) introduce a markerless automated framework for deformable registration in laparoscopic partial nephrectomy video see through AR navigation. Virtual reconstruction of pre-acquired radiological images paves way for possible errors that could occur due to organ deformation and patient breathing. As a result, real-time rendering and registration of virtual images are necessary. Real-time intraprocedural US and its rendering to virtual images need to be overlapped on the liver model, which is quite challenging. Tanzi et al. (2021) propose a two-step automated approach to align a 3D virtual model of a patient’s organ with a 2D endoscopic image. The method is designed to aid surgeons throughout the surgery; the method is based on Convolutional Neural Network (CNN) architecture for semantic segmentation. Brunet et al. (2019) make use of UNet architecture to simulate preoperative organs anticipating mesh deformation in human anatomical boundaries. Furthermore, the AR device battery life, its field of view might limit the intent for the use in real-time. Additionally, the integrated sensors within the AR device must be calibrated to provide accurate tracking of the spatial surrounding and must have the computational capacity to render in real-time. Another important factor is the streaming of data from standalone or portable AR devices to computers that are connected to haptic devices. Many solutions are present to ensure low latency data streaming for these configurations, such as WebSockets, real-time cloud server, distributed cloud server, 5G networks, and Web Real Time Communication (WebRTC). Black et al. (2024) present a method for remotely streaming and controlling point-of-care ultrasound systems, regardless of their manufacturer. The authors propose a communication system for HoloLens 2 that can be used with the ResearchMode API sensor stream access for efficient streaming of sensor data.

5.2 Integration of haptics

Haptic technology encompasses the use of various tactile components, including motors, actuators, pneumatics, hydraulics, and other devices, to simulate touch sensations. It has a vital function in improving user experiences in diverse applications. Numerous haptic feedback models have been created and utilized, particularly; these systems enable precise navigation of medical instruments along a path, ensuring flawless precision without harming crucial regions. Additionally, it aids in differentiating between various layers of liver tissue (Selim et al. 2024). Haptic interaction could be used to create a realistic training scenario involving needle-tissue interaction forces and evaluate the gesture of trainees providing assessment feedback. The triboelectric nanogenerator (TENG) and electro-tactile (ET) system represents the beginning of the development of self-sustaining virtual tactile stimulation system, offering a solution to the limitations of traditional VR/AR techniques in terms of sustainability and wiring constraints. Therefore, it can facilitate the implementation of tactile VR/AR in different domains, such as tactile prosthetics, Braille education, and others (Shi et al. 2021). Furthermore, virtual models of the liver can be created to demonstrate its dynamic deformations and mechanical characteristics. These models have the potential to educate the physicians to use a teleoperation system and its integration with AR, resulting in a more immersive experience for them during remote operations. Various techniques are available for creating virtual liver models such as FEM, ML, and 3D modelling algorithms like Haptics3D (H3D) (Selim et al. 2024). The utilization of haptic feedback with force sensors enables surgeons to precisely regulate the force exerted at the interface between the surgical tool and the tissue. Additionally, impedance sensing of tissue electrical properties can be employed for tumor diagnosis (Qiu et al. 2024). The transformative evolution of metaverse wearables, using AR/VR and haptics, in assisted healthcare, particularly in the fields of rehabilitation, medical/nursing education, and remote patient treatment offers numerous advantages, including better patient prognosis, increased access to high-quality care, and superior standards of practitioner teaching (Kim et al. 2023). Furthermore, DL can be applied in amalgamation with haptics to enhance the effect of force-feedback.

The active haptic interfaces offer the capability to incorporate sensors to capture and record data for gesture evaluation and assessment. A mock needle and mock ultrasound probe could be incorporated in the haptic interface. Many commercially available haptic devices offer the advantage of being compatible with existing frameworks and libraries, and would be easy for development by appropriate customization. One such customization could be a hand rest for the practitioners along with a knob to hold the needle and prevent it from falling. The selected haptic device must be able to produce the required forces to create a realistic training scenario for the trainees. To produce realistic forces, it is vital to ensure that the mechanical properties of the devices are within the required constraints. Additionally, the respective mathematical models must take the elasticity of the tissues into account, and must not necessarily consider them as rigid structures. Furthermore, the parameters necessary to render the tissues and needle must be determined. For liver, the parameters could be obtained by conducting insertion tests on porcine and bovine samples. For simulations, although the FEM method is proved effective, it does not allow real-time computation; a solution to this could be a hybrid approach. For instance, the CuTFEM approach is faster than the traditional FEM approach, yet it is slow in terms of rendering force feedback. In the hybrid solution, the CuTFEM approach is reported promising. Pellicer-Valero et al. (2020) apply a ML technique to a large dataset of liver models that are created using the FEM approach; the algorithm achieves good outcomes by enabling real-time force simulation on the liver with a Euclidean error of less than 1 mm. Another additional variable using AR and haptics for training in PLB could visually illustrate the insertion point along with the angle of insertion.

6 Conclusion

Haptics could be integrated to provide feedback if the angle of insertion is different from the illustrated angle. Additionally, if the needle’s path deviates from the ideal trajectory path, haptics could be of help. Embedding sensors in training simulators to record data could be a solution for future developments. Furthermore, evaluating different AR and haptic devices, mathematical models/methods and classifying them based on their utility and performance could help the developers select the optimal approach when developing an AR haptic simulator for PLB training. To conclude, AR and haptic interaction, when integrated could provide a realistic, efficient, and immersive training simulator.