Stereoscopic augmented reality for laparoscopic surgery
- 1.9k Downloads
Conventional laparoscopes provide a flat representation of the three-dimensional (3D) operating field and are incapable of visualizing internal structures located beneath visible organ surfaces. Computed tomography (CT) and magnetic resonance (MR) images are difficult to fuse in real time with laparoscopic views due to the deformable nature of soft-tissue organs. Utilizing emerging camera technology, we have developed a real-time stereoscopic augmented-reality (AR) system for laparoscopic surgery by merging live laparoscopic ultrasound (LUS) with stereoscopic video. The system creates two new visual cues: (1) perception of true depth with improved understanding of 3D spatial relationships among anatomical structures, and (2) visualization of critical internal structures along with a more comprehensive visualization of the operating field.
The stereoscopic AR system has been designed for near-term clinical translation with seamless integration into the existing surgical workflow. It is composed of a stereoscopic vision system, a LUS system, and an optical tracker. Specialized software processes streams of imaging data from the tracked devices and registers those in real time. The resulting two ultrasound-augmented video streams (one for the left and one for the right eye) give a live stereoscopic AR view of the operating field. The team conducted a series of stereoscopic AR interrogations of the liver, gallbladder, biliary tree, and kidneys in two swine.
The preclinical studies demonstrated the feasibility of the stereoscopic AR system during in vivo procedures. Major internal structures could be easily identified. The system exhibited unobservable latency with acceptable image-to-video registration accuracy.
We presented the first in vivo use of a complete system with stereoscopic AR visualization capability. This new capability introduces new visual cues and enhances visualization of the surgical anatomy. The system shows promise to improve the precision and expand the capacity of minimally invasive laparoscopic surgeries.
KeywordsAugmented reality Surgical visualization Stereoscopic visualization Multimodality image fusion Laparoscopic surgery
The funding for this project came from internal institutional sources.
X. Kang, M. Azizian, E. Wilson, K. Wu, A. D. Martin, T. D. Kane, C. A. Peters, K. Cleary, and R. Shekhar have no conflicts of interest or financial ties to disclose.
Supplementary material 1 (MOV 18773 kb)
Supplementary material 2 (MOV 64925 kb)
Supplementary material 3 (MOV 6371 kb)
- 7.Leven J, Burschka D, Kumar R, Zhang G, Blumenkranz S, Dai XD, Awad M, Hager GD, Marohn M, Choti M, Hasser C, Taylor RH (2005) DaVinci canvas: a telerobotic surgical system with integrated, robot-assisted, laparoscopic ultrasound capability. Med Image Comput Comput Assist Interv 8:811–818PubMedGoogle Scholar
- 11.Ieiri S, Uemura M, Konishi K, Souzaki R, Nagao Y, Tsutsumi N, Akahoshi T, Ohuchida K, Ohdaira T, Tomikawa M, Tanoue K, Hashizume M, Taguchi T (2012) Augmented reality navigation system for laparoscopic splenectomy in children based on preoperative CT image using optical tracking device. Pediatr Surg Int 28(4):341–346PubMedCrossRefGoogle Scholar
- 14.Yaron A, Bar-Zohar M, Horesh N (2007) Miniature stereoscopic video system provides real-time 3D registration and image fusion for minimally invasive surgery. Proc SPIE 6490, Stereoscopic Displays and Virtual Reality Systems XIV 649009Google Scholar
- 20.Lu F, Hartley R (2007) A fast optimal algorithm for L2 triangulation. ACCV 2:279–288Google Scholar