Skip to main content
SpringerLink
Log in

Stereoscopic augmented reality for laparoscopic surgery

  • Dynamic Manuscript
  • Published:
Surgical Endoscopy Aims and scope Submit manuscript

Abstract

Background

Conventional laparoscopes provide a flat representation of the three-dimensional (3D) operating field and are incapable of visualizing internal structures located beneath visible organ surfaces. Computed tomography (CT) and magnetic resonance (MR) images are difficult to fuse in real time with laparoscopic views due to the deformable nature of soft-tissue organs. Utilizing emerging camera technology, we have developed a real-time stereoscopic augmented-reality (AR) system for laparoscopic surgery by merging live laparoscopic ultrasound (LUS) with stereoscopic video. The system creates two new visual cues: (1) perception of true depth with improved understanding of 3D spatial relationships among anatomical structures, and (2) visualization of critical internal structures along with a more comprehensive visualization of the operating field.

Methods

The stereoscopic AR system has been designed for near-term clinical translation with seamless integration into the existing surgical workflow. It is composed of a stereoscopic vision system, a LUS system, and an optical tracker. Specialized software processes streams of imaging data from the tracked devices and registers those in real time. The resulting two ultrasound-augmented video streams (one for the left and one for the right eye) give a live stereoscopic AR view of the operating field. The team conducted a series of stereoscopic AR interrogations of the liver, gallbladder, biliary tree, and kidneys in two swine.

Results

The preclinical studies demonstrated the feasibility of the stereoscopic AR system during in vivo procedures. Major internal structures could be easily identified. The system exhibited unobservable latency with acceptable image-to-video registration accuracy.

Conclusions

We presented the first in vivo use of a complete system with stereoscopic AR visualization capability. This new capability introduces new visual cues and enhances visualization of the surgical anatomy. The system shows promise to improve the precision and expand the capacity of minimally invasive laparoscopic surgeries.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5

Similar content being viewed by others

References

  1. Himal HS (2002) Minimally invasive (laparoscopic) surgery. Surg Endosc 16(12):1647–1652

    Article  CAS  PubMed  Google Scholar 

  2. Rosen M, Ponsky J (2001) Minimally invasive surgery. Endoscopy 33(4):358–366

    Article  CAS  PubMed  Google Scholar 

  3. Storz P, Buess GF, Kunert W, Kirschniak A (2012) 3D HD versus 2D HD: surgical task efficiency in standardised phantom tasks. Surg Endosc 26(5):1454–1460

    Article  PubMed  Google Scholar 

  4. Smith R, Day A, Rockall T, Ballard K, Bailey M, Jourdan I (2012) Advanced stereoscopic projection technology significantly improves novice performance of minimally invasive surgical skills. Surg Endosc 26(6):1522–1527

    Article  CAS  PubMed  Google Scholar 

  5. Nakamoto M, Ukimura O, Faber K, Gill IS (2012) Current progress on augmented reality visualization in endoscopic surgery. Curr Opin Urol 22(2):121–126

    Article  PubMed  Google Scholar 

  6. Linte CA, Wiles A, Moore J, Wedlake C, Peters TM (2008) Virtual reality-enhanced ultrasound guidance for atrial ablation: in vitro epicardial study. Med Image Comput Comput Assist Interv 11(Pt 2):644–651

    PubMed  Google Scholar 

  7. Leven J, Burschka D, Kumar R, Zhang G, Blumenkranz S, Dai XD, Awad M, Hager GD, Marohn M, Choti M, Hasser C, Taylor RH (2005) DaVinci canvas: a telerobotic surgical system with integrated, robot-assisted, laparoscopic ultrasound capability. Med Image Comput Comput Assist Interv 8:811–818

    PubMed  Google Scholar 

  8. Su LM, Vagvolgyi BP, Agarwal R, Reiley CE, Taylor RH, Hager GD (2009) Augmented reality during robot-assisted laparoscopic partial nephrectomy: toward real-time 3D-CT to stereoscopic video registration. Urology 73(4):896–900

    Article  PubMed  Google Scholar 

  9. Cheung CL, Wedlake C, Moore J, Pautler SE, Ahmad A, Peters TM (2009) Fusion of stereoscopic video and laparoscopic ultrasound for minimally invasive partial nephrectomy. Proc SPIE 7261:726109–726110

    Article  Google Scholar 

  10. Cheung CL, Wedlake C, Moore J, Pautler SE, Peters TM (2010) Fused video and ultrasound images for minimally invasive partial nephrectomy: a phantom study. Med Image Comput Comput Assist Interv 13(Pt 3):408–415

    PubMed  Google Scholar 

  11. Ieiri S, Uemura M, Konishi K, Souzaki R, Nagao Y, Tsutsumi N, Akahoshi T, Ohuchida K, Ohdaira T, Tomikawa M, Tanoue K, Hashizume M, Taguchi T (2012) Augmented reality navigation system for laparoscopic splenectomy in children based on preoperative CT image using optical tracking device. Pediatr Surg Int 28(4):341–346

    Article  PubMed  Google Scholar 

  12. Teber D, Guven S, Simpfendörfer T, Baumhauer M, Güven EO, Yencilek F, Gözen AS, Rassweiler J (2009) Augmented reality: a new tool to improve surgical accuracy during laparoscopic partial nephrectomy? Preliminary in vitro and in vivo results. Eur Urol 56(2):332–338

    Article  PubMed  Google Scholar 

  13. Simpfendörfer T, Baumhauer M, Müller M, Gutt CN, Meinzer HP, Rassweiler JJ, Guven S, Teber D (2011) Augmented reality visualization during laparoscopic radical prostatectomy. J Endourol 25(12):1841–1845

    Article  PubMed  Google Scholar 

  14. Yaron A, Bar-Zohar M, Horesh N (2007) Miniature stereoscopic video system provides real-time 3D registration and image fusion for minimally invasive surgery. Proc SPIE 6490, Stereoscopic Displays and Virtual Reality Systems XIV 649009

  15. Hostettler A, George D, Rémond Y, Nicolau SA, Soler L, Marescaux J (2010) Bulk modulus and volume variation measurement of the liver and the kidneys in vivo using abdominal kinetics during free breathing. Comput Methods Progr Biomed 100(2):149–157

    Article  Google Scholar 

  16. Hostettler A, Nicolau SA, Rémond Y, Marescaux J, Soler L (2010) A real-time predictive simulation of abdominal viscera positions during quiet free breathing. Prog Biophys Mol Biol 103(2–3):169–184

    Article  CAS  PubMed  Google Scholar 

  17. Shekhar R, Dandekar O, Bhat V, Philip M, Lei P, Godinez C, Sutton E, George I, Kavic S, Mezrich R, Park A (2010) Live augmented reality: a new visualization method for laparoscopic surgery using continuous volumetric computed tomography. Surg Endosc 24(8):1976–1985

    Article  PubMed  Google Scholar 

  18. Zhang Z (2000) A flexible new technique for camera calibration. IEEE Trans Pattern Anal Mach Intell 22(11):1330–1334

    Article  Google Scholar 

  19. Yaniv Z, Foroughi P, Kang HJ, Boctor E (2011) Ultrasound calibration framework for the image-guided surgery toolkit. Medical Imaging 2011: visualization, image-guided procedures, and modeling. Proc SPIE 7964:79641N–1–79641N–11

    Article  Google Scholar 

  20. Lu F, Hartley R (2007) A fast optimal algorithm for L2 triangulation. ACCV 2:279–288

    Google Scholar 

  21. Chen TK, Heffter T, Lasso A, Pinter C (2011) Automated intraoperative calibration for prostate cancer brachytherapy. Med Phys 38(11):6285–6299

    Article  Google Scholar 

Download references

Acknowledgments

The funding for this project came from internal institutional sources.

Disclosures

X. Kang, M. Azizian, E. Wilson, K. Wu, A. D. Martin, T. D. Kane, C. A. Peters, K. Cleary, and R. Shekhar have no conflicts of interest or financial ties to disclose.

Author information

Authors and Affiliations

  1. Sheikh Zayed Institute for Pediatric Surgical Innovation, Children’s National Medical Center, 111 Michigan Avenue NW, Washington, DC, 20010, USA

    Xin Kang, Mahdi Azizian, Emmanuel Wilson, Kyle Wu, Aaron D. Martin, Timothy D. Kane, Craig A. Peters, Kevin Cleary & Raj Shekhar

Authors
  1. Xin Kang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Mahdi Azizian
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Emmanuel Wilson
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Kyle Wu
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Aaron D. Martin
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Timothy D. Kane
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Craig A. Peters
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. Kevin Cleary
    View author publications

    You can also search for this author in PubMed Google Scholar

  9. Raj Shekhar
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Xin Kang.

Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary material 1 (MOV 18773 kb)

Supplementary material 2 (MOV 64925 kb)

Supplementary material 3 (MOV 6371 kb)

Rights and permissions

Reprints and permissions

About this article

Cite this article

Kang, X., Azizian, M., Wilson, E. et al. Stereoscopic augmented reality for laparoscopic surgery. Surg Endosc 28, 2227–2235 (2014). https://doi.org/10.1007/s00464-014-3433-x

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00464-014-3433-x

Keywords

Search

Navigation