1 Introduction

The incidence of renal cell carcinoma (RCC) ranks within the top 20 among all solid tumors [1]. With the increasing number of early-stage RCC cases and the improvement of surgical techniques and instruments, the proportion of partial nephrectomy (PN) is increasing [2]. The European Association of Urology guidelines have suggested that the oncological outcomes achieved by PN are comparable to those achieved by radical nephrectomy (RN) for T1 RCC. PN also preserves kidney function better and potentially limits the incidence of cardiovascular disorders [3].

PN is technically demanding and has various challenges. (1) During the operation, the key targets that need to be handled are blocked by perirenal fat or adjacent organs, making them invisible to the naked eye, and they often cannot be found quickly and accurately due to the lack of guidance of clear anatomical landmarks. Delays or mistakes in this process may lead to vascular injury, opening of the collecting system, tumor rupture, or other risks. (2) The information provided by two-dimensional (2D) CT/MR imaging is insufficient. Thus, it is necessary for surgeons to “translate” cross-sectional planar imaging into stereoscopic imaging, which is a process of cognitive reconstruction. This building-in-mind process is particularly difficult for complex lesions or for inexperienced surgeons. (3) Due to the insufficient grasp of the local anatomical details of renal tumors, surgeons sometimes avoid PN for some complex cases and instead use a safer method, such as radical nephrectomy. (4) Similarly, due to the insufficient information obtained before the operation, the PN operation often turns into an “encounter”, such as encountering unknown vascular variants or failing to find endogenous tumors after kidney incision, which introduces risks to the operation.

In recent years, new technological tools have been developed for the reconstruction of three-dimensional (3D) virtual models from standard 2D imaging. Holographic imaging (also known as 3D imaging, augmented reality (AR) imaging, 3D visualization models, and holograms) is reconstructed based on surface rendering techniques from contrast CT or MRI DICOM data using 3D virtual reconstruction technology [4]. Holographic imaging provides more intuitive three-dimensional images, enhances the spatial understanding of the operator, and has a powerful interactive function, guiding the precise implementation of the operation. Previous studies have shown that the application of the holographic imaging technique in laparoscopic partial nephrectomy (LPN) and robotic-assisted partial nephrectomy (RAPN) results in reduced operative time, estimated blood loss, complications, and length of hospital stay [5]. The aim of this comprehensive literature review was to update the current status regarding the application of holographic imaging techniques in PN.

2 Methods

According to the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) specification, a systematic search was conducted for original articles or reviews published prior to and including November 11, 2022. The databases searched included Medline, Embase, and Cochrane Central. The inclusion criteria for the studies, including original articles, reviews, and case reports, were as follows: (1) patients with renal tumors; (2) laparoscopic or robot-assisted minimally invasive partial nephrectomy; and (3) 3D reconstruction model/holographic imaging or 3D printed model. The exclusion criteria for the studies were as follows: (1) only the title of the paper could be searched; and (2) non-English abstract/article. The main search terms included renal cell carcinoma, partial nephrectomy, nephron-sparing surgery, three dimensional, holographic, virtual, imaging, model, visualization, augmented reality, and 3D print. Different combinations and strategies were applied during different database retrievals (see Appendix).

3 Results

A total of 606 papers were retrieved from the above databases. After deleting duplicate papers and reviewing the abstracts, 304 papers were in line with the objective and were read in full. We summarized and presented the results in the following four sections: history and development; clinical applications; impact on PN outcomes; and new perspectives.

3.1 History and development

Holographic images are reconstructed based on the surface rendering technique, which is different from traditional CT angiography 3D models based on volume rendering. Surface rendering is a voxel-based surface reconstruction method that processes a series of 2D images by boundary recognition and segmentation to restore the 3D shape of the target organs. Surface rendering creates higher accuracy and better interactive functions than traditional volume rendering models [6].

Although the application of surface rendering and volume rendering technology in medical holographic image reconstruction can all be traced back to the late 1980s [7, 8], most of the 3D models reported before the 2010s were built through volume rendering. In 2008, Ukimura and Gill reported the first use of AR in urology [9]. A surgical 3D virtual model was reconstructed from conventional CT data and superimposed onto the surgical view to help the surgeon understand the 3D anatomy beyond the surgical view during LPN. In 2009, Teber et al. developed an AR-based soft tissue navigation system to enhance the surgeon’s perception and to provide decision-making guidance before the initiation of LPN [10]. However, due to the lack of technological sophistication and insufficient fineness of the 3D models, this intraoperative guidance has not been popularized.

The number of papers based on 3D images for preoperative assessment and surgical planning has been increased significantly since the 2000s, mainly because surgeons have gradually accepted 3D models “reconstructed” with software, such as Mimics. Some studies have also reported the use of manual image fusion to superimpose 3D images on two-dimensional (2D) laparoscopic images [11]. During the same period, 3D printing models attracted clinical interest and were used for preoperative planning, surgical simulation, doctor‒patient communication, and intraoperative reference [12,13,14]. The 3D printing boom did not last long because the inconvenience of 3D printing instruments resulted in limited acceptance by surgeons and patients.

Later, with the advent of head-mounted AR devices, urological surgeons used this spatial projection and interaction for PN. Surgeons wear Microsoft’s HoloLens to holographically project 3D models for interactive discussion before surgery or enter the operating room space to observe anatomical details and guide surgery [15, 16]. However, the application value of this interaction method in LPN/RAPN is limited due to the cost, accuracy, and inconvenience.

In RAPN, there is a transitional mode for image-guided surgery. The TilePro ™ (Intuitive Surgical Inc., Sunnyvale, CA, USA) multi-input display on the da Vinci robotic surgical platform allows the surgeon to view a 3D virtual kidney model on a subscreen below the main screen showing the intraoperative view, and the surgeon can rotate and orient the 3D model to align it with the intraoperative view [6]. In this process, the software used for 3D rendering is constantly iterating and upgrading. As an increasing number of “3D imaging engineers” specialize in medical 3D image reconstruction, the quality of 3D images has greatly improved and can show the anatomy of renal tumors and the renal hilum in very fine detail to help surgeons formulate more personalized surgical strategies before surgery.

In recent years, a fusion software, named Navigator, has been developed to better guide LPN/RAPN [17, 18]. The fusion software fuses and superimposes the 3D reconstruction models on the 2D laparoscopic images, synchronously guiding the implementation of LPN surgery. The fusion of the 3D models is manual and is usually performed by a surgical assistant. This image fusion “navigation” mode is also used in the da Vinci robotic surgical system. 3D models are displayed on the Tile-Pro display through the AR Viewer application and fused with the real-time endoscopic view [19]. After more than 10 years of development, image-guided surgery in AR mode is becoming a trend [20].

Currently, CT data in DICOM format are processed using a software platform, such as the Renxin platform (Renxin MedTech, Beijing, China) [4, 18]. The CT images are first evaluated by a DICOM viewer, and segmentation processing is then performed semiautomatically. The holographic images of the kidney and adjacent organs are reconstructed with layer-by-layer image segmentation. Using this reconstruction and virtual image technology, the abdominal organs, kidneys, vessels, tumors, renal pelvis, ureter, and other targets of interest can be accurately mapped, and holographic images can be reconstructed for use [21]. Holographic imaging has the functions of splitting, rotation, and transparency, and a high-quality holographic image can display a patient’s anatomical details precisely with high definition. To achieve such a result, original data in DICOM format should be of high quality; for example, the 2D CT images should be ≤ 1.5 mm slices.

At present, the segmentation process mainly relies on manual work. Even experienced imaging engineers require approximately 1 h to complete detailed segmentation and modeling. This process is usually performed with the assistance of experienced urologists and radiologists as professional medical knowledge is needed. Currently, holographic imaging reconstruction is a time- and labor-consuming process with low efficiency.

3.2 Clinical application

3.2.1 Patient consultation

Traditionally, surgeons provide their patients and their family members with extensive verbal, written, and drawn information to provide a detailed explanation of their CT scan and preoperative findings. The reality is that nonmedical professionals cannot be expected to understand CT images and anatomy. However, with the help of advanced AR techniques, patients can obtain a much deeper understanding of the surgical procedures and the related risks of complications by watching the holographic images and surgical simulation [22]. Holographic imaging is a useful tool to deliver comprehensible and personalized information regarding a specific procedure and to understand the potential causes of surgical complications, thus enhancing the communication between patients and surgeons, which will result in positive patient satisfaction and may also improve shared decision-making [23].

3.2.2 Nephrometry Score

Assessment of the location, anatomical details, and interrelationships of renal tumors with other structures, such as vessels and the collecting system, plays an important role in preoperative surgical planning because it can predict surgical complexity [24]. There are several CT image-based nephrometry score systems; for example, the PADUA and R.E.N.A.L. scores have been developed to objectively determine how complex PN will be and the likelihood of postoperative complications [25, 26]. Despite a good correlation between nephrometry score and PN outcomes, as supported by extensive literature, the full understanding of the morphological and anatomical renal tumor and kidney characteristics is suboptimal as the assessment is made on 2D images; surgeons need to envision a 3D image by reading the 2D images in the three spatial axes (axial, coronal, and sagittal), which can be easily achieved on holographic imaging [27].

Holographic images provide much richer information than conventional CT or MRI and, therefore, can assess PN surgical complexity more objectively and avoid the abstraction process needed when using 2D images [28]. Porpiglia et al. showed that 3D virtual imaging is more precise than 2D standard imaging in evaluating the surgical complexity of kidney tumors, according to nephrometry score, as it allows a better perception of tumor depth and its relationships with intrarenal structures [29]. A higher accuracy of 3D imaging in predicting postoperative complications has also been confirmed. Liu et al. reported a PN surgical complexity scoring system based on holographic imaging [30]. The maximum diameter of the tumor in the kidney (D), the compression degree of renal segmental vessels by the tumor (C), the area of the renal sinus occupied by the tumor (O), and the mass of the exophytic rate (M) constitute the complexity scoring system named the DCOM score for PN. Receiver operator characteristic (ROC) curve analysis demonstrated that in predicting a negative surgical margin, warm ischemia time < 20 min, and no major complications in the perioperative period, the area under the curve (AUC) values of the R.E.N.A.L., PADUA, and DCOM scoring systems were 0.657, 0.655, and 0.746, respectively (P = 0.025). Thus, the DCOM scoring system can predict PN operative outcomes more accurately than 2D-based nephrometry systems.

3.2.3 Surgical planning for PN

Holographic imaging can be reformed to meet the needs of facilitating surgical planning. For example, a simulating incision line or clamping of renal artery branches for PN can be achieved on a kidney holographic image. These factors all contribute to a safer surgery and better outcomes [31]. Bertolo et al. compared the ability of 3D images to 2D images in expanding the PN indication for complex renal tumors [32], and they reported that more than 20% of surgeons change their decision for these complex renal tumors from RN to PN after reviewing the 3D images. These results support the potential of holographic imaging in surgical planning. Shirk et al. compared 3D VR models with conventional CT/MR imaging for surgical planning and surgical outcomes in RAPN; their results showed a reduction in operative time, estimated blood loss, clamp time, and hospital stay in the 3D group [33].

3.2.4 Navigation in surgery

Holographic imaging superimposes holographic images on the endoscopic view of the anatomy, allowing intraoperative navigation. The holographic imaging navigation technique achieves holographic images fused with the real-time intraoperative endoscopic view, allowing the surgeon to access the targets directly and minimizing the damage to surrounding vessels and other structures. Currently, although it is not yet an automatic navigation system, holographic imaging still helps the console surgeon in perceiving the three-dimensionality of the kidney and correctly localizing the tumor, resulting in precise and safe tumor resection. Holographic imaging is particularly useful in complex renal tumor cases, such as hilar tumors, in which surgeons must perform tumor resection close to the renal vein/artery.

During holographic imaging navigation surgery, it is currently necessary to expose some anatomical landmarks, such as renal hilar vessels or renal contours, to achieve registration and tracking of holographic images with an intraoperative endoscopic view [17]. In holographic imaging navigation PN, when the virtual renal pedicle is precisely fused with the real renal vascular pedicle, the surgeon is guided to perform safe vascular dissection and to identify the renal artery or vein branches and clamping as well as to implement personalized vascular management strategies, such as high selective renal artery clamping. When the virtual kidney is completely fused with the real kidney, by adjusting the transparency of the model, the location of the endophytic tumor and the tumor relationship with the adjacent blood vessels, calyces, and other intraparenchymal structures can be visualized, which is conducive to more accurate tumor excision [34].

However, there are some concerns regarding holographic imaging navigation. One is the accuracy of registration of holographic images on static anatomical structures because it is not easy to precisely align virtual holographic images and their physical counterparts in spatial and rotational coordinates [35]. In LPN or RAPN, the establishment of pneumoperitoneum deforms the abdominal cavity and changes the spatial relationship of the kidney compared to that before the operation. In addition, the kidney shifts, deforms, rotates, and changes its relative position to neighboring organs due to gravity and the jostling of surgical tools. Based on these factors, the previous rigid matching technology produces a large deviation after organ deformation. Deformable models have been introduced by some researchers, and this problem can be alleviated to some extent by modifying the preoperative model during surgery [36]. There is also a method called nonlinear parametric deformation to simulate the deformation of an organ during surgery [34].

Currently, to maximize the accuracy of superposition, it is often necessary to use manual registration. The assistant will manipulate the fusion system during the entire procedure to create proper orientation and deformation of the model. Manual registration and tracking are simple methods to “anchor” a 3D model to its counterpart in real time. However, these methods require an additional assistant surgeon to control the AR workstation [18]. This work is labor-consuming, and the accuracy of image fusion depends on the experience of the assistant.

Intraoperative tracking is another major challenge. It is a great challenge to maintain satisfactory real-time accuracy in laparoscopic AR surgery because the endoscopic view of a surgical scene is highly dynamic. It is difficult for the assistant to adjust the model and match it with the endoscopic image in time. Similar to manual registration, manual tracking is also a common method under current conditions, but it is labor intensive. In addition, the efficacy of manual tracking is affected by the experience of the assistants.

3.2.5 Surgical training

Holographic imaging can also be used to enhance the education of medical students and fellows, thus aiding their professional development. Rai et al. reported that medical students who use the interactive 3D VR simulator based on PN cases significantly improve their subjective ability to localize the tumor position [37]. Knoedler et al. evaluated the effect of 3D printed physical renal models on enhancing medical trainees’ understanding of kidney tumor characterization and localization [38]; they reported that the overall trainee nephrometry score accuracy is significantly improved with the 3D model vs. CT scan, and there is also more consistent agreement among trainees when using the 3D models compared to CT scans to assess the nephrometry score. Thus, 3D models improve trainees’ understanding and characterization of kidney tumors in patients.

3.3 Impact on PN outcomes

Zhu et al. reported their experience of holographic image navigation in urological laparoscopic and robotic surgery, including 27 partial nephrectomy cases; they reported that this technology reduces tissue injury, decreases complications, and improves the surgical success rate [5]. Zeng et al. reported the use of 3D navigation imaging and real-time navigation in 41 RAPN cases; they found that the use of 3D navigation provides precise intraoperative guidance, subsequently reducing the risk of major blood vessel injury [21]. Schiavina et al. reported 15 cases of RAPNs, of which 8 (53.3%) were selectively clamped and 4 (26.7%) were superselectively clamped after 3D image guidance, which was higher than with traditional planning with 2D images (P = 0.03); they also reported that intraoperative management of the renal hilum was consistent with preoperative planning in 86.7% of patients [39].

The application of 3D imaging in PN for complex renal tumors, such as renal hilar tumors, has attracted extensive attention. Wang et al. included 26 cases of renal hilar tumors and found that 3D imaging reconstruction and navigation technology have the advantages of accurate localization, a high complete resection rate, and fewer perioperative complications [40]. Porpiglia et al. reported their results of using 3D imaging during RAPN for complex renal tumors (PADUA ≥ 10) [34]. Compared to 2D ultrasound guidance, the 3D imaging and AR guidance group had a lower rate of global ischemia, a higher rate of enucleation, a lower rate of collecting system violation, a low risk of surgery-related complications, and lower renal blood flow decrease 3 months after the operation. The combination of holographic imaging with da Vinci robotic surgical systems allowed accurate recognition, increased flexibility, and real-time navigation, which made the RAPN easier and safer for renal hilar tumors. Zhang et al. reported their series of combining holographic imaging with RAPN for renal hilar tumor treatment [18]; they reported that this technique reduces the risk of conversion to open surgery or RN for renal hilar tumors, increases the success rate, and decreases complications. Zhang et al. also reported a new technique of combining holographic imaging and clipping tumor bed artery branches outside the kidney to reduce PN-related secondary bleeding, to reduce the need for postoperative interventional embolization, and to shorten the length of hospital stay.

Endophytic kidney tumors present a great challenge as they are not visible on the kidney surface. Porpiglia et al. presented their use of AR images to visualize endophytic tumors [34]. AR technology potentially increases the 3D perception of the lesion’s features and the surgeon’s confidence in tumor excision and guide precise resection. Compared to the ultrasound-guided group, Porpiglia et al. observed that the enucleation rate of the 3D AR group was higher (p = 0.02), the percentage of preserved healthy renal parenchyma was higher, and the opening rate of the collecting system was lower (p = 0.0003).

A systematic review has examined the effectiveness of AR-assisted technology in LPN compared to conventional techniques [41]. Compared to conventional LPN, a shorter procedure time and lower intraoperative blood loss are found for AR-assisted LPN, while the positive surgical margin rate, warm ischemia time, complications, eGFR decline, and length of stay are equivalent between the two technologies.

3.4 New perspectives

3.4.1 Attempt for automation

Automatic registration and tracking have always presented an interesting direction. As early as 2014, a new AR system was reported, which used a new weighted sliding window registration method. This AR system automatically and accurately restored the overlay by predicting the image location of a large number of anchor points lost after a sudden image change, and it maintained long-term (more than 2 min) and accurate (less than 1 mm) augmentation in a set of real PN laparoscopic videos [42]. Infrared trackers are alternative tracking modalities [43], but they are still far from mature applications. Kobayashi et al. attached infrared reflective markers to the da Vinci endoscope and automatically measured the position of the endoscope using an optical tracking system. 3D images can be moved automatically and synchronously with the movement of the endoscope. However, the method still requires manual registration at the beginning and reregistration when gaps occur [44]. Amparore et al. used near-infrared fluorescence (NIRF) firefly fluorescence imaging technology to superenhance the kidney to overcome the limitation of color similarity between the kidney and its neighboring structures [45]. In addition, a specially developed software, called Indocyanine Green Auto Augmented Reality (IGNITE), allows 3D models to be automatically anchored to real organs and takes advantage of the enhanced views provided by NIRF technology. There are also some other reports of surgical tracking technology [46], and the subsequent development of these technologies deserves our attention.

3.4.2 Artificial intelligence

Artificial intelligence (AI) should be an important direction for future holographic imaging-guided surgery. Recently, it has been reported that kidneys, renal tumors, arteries, and veins can be automatically segmented and 3D-modeled by deep learning [47, 48]. He Y et al. proposed the first deep learning framework, called the Meta Grayscale Adaptive Network (MGANet), which simultaneously segments the kidney, renal tumors, arteries, and veins on CTA images in one inference, resulting in a better 3D integrated renal structure segmentation quality [49]. Houshyar et al. developed and evaluated a convolutional neural network (CNN) to act as a surgical planning aid by determining renal tumor and kidney volumes through segmentation on single-phase CT. The results showed that the end-to-end trained CNNs perform renal parenchyma and tumor segmentation on test cases in an average of 5.6 s. Houshyar et al. concluded that the deep learning model rapidly and accurately segments kidneys and renal tumors on single-phase contrast-enhanced CT scans as well as calculates tumor and renal volumes [50]. Zhang et al. developed a 3D kidney perfusion model based on deep learning techniques to automatically demonstrate the segmentation of renal arteries, and they verified its accuracy and reliability in LPN [51]. It is technically feasible for AI to realize automatic 3D modeling, but the process relies on a large amount of calibrated data for training.

Marker-based and deformation-based registration techniques have been preliminarily reported to achieve more accurate registration [42]. The use of deep learning to automatically recognize image and video information is expected to achieve automatic registration and tracking. Recently, Padovan et al. introduced a deep learning framework through convolutional neural networks and motion analysis, which determines the position and rotation information of target organs in endoscopic video in real time [52]. This work has taken an important step for the application of deep learning to generalize the automatic registration process.

4 Conclusion

PN is a challenging surgical procedure. Holographic imaging helps surgeons to thoroughly understand the individualized anatomy of the kidney and tumor as well as to set up a more optimized surgical plan and to facilitate patient counseling. The implementation of holographic imaging navigation helps the surgeon to accurately identify and locate the target tumor, renal artery/vein and branches, and collecting system, thus reducing complications and conversion to open surgery or RN.

The application of holographic imaging in PN has significant benefits in reducing the warm ischemia time, collecting system opening, blood loss, and incidence of serious complications, but it is similar to traditional technology in conversion to RN, complication rate, changes in glomerular infiltration rate, and surgical margins. Holographic imaging in PN is particularly valuable in cases of endophytic renal tumors, complex renal hilar tumors, and super-selective clamping.

At present, the main deficiency in holographic imaging is that automatic 3D modeling and intraoperative automatic registration have not yet been fully realized, and the accuracy of registration still needs to be improved. Deep learning is expected to solve these challenges in the future.