Augmented Reality in Minimally Invasive Digestive Surgery

  • Luc SolerEmail author
  • Stéphane Nicolau
  • Patrick Pessaux
  • Didier Mutter
  • Jacques Marescaux


Minimally invasive surgery represents one of the main evolutions of surgical techniques. However, minimally invasive surgery adds difficulty that can be reduced by computer technology. Indeed, from a patient’s medical image (US, CT, or MRI), augmented reality (AR) can increase the surgeon’s intraoperative vision by providing virtual transparency of the patient. AR is based on two main processes: 3D visualization of the anatomical or pathological structures appearing in the medical image and registration of this visualization onto the real patient. 3D visualization can be performed directly from the medical image by direct volume rendering or after image processing by 3D surface rendering. Registration can be performed interactively or automatically. Several interactive systems have been developed and applied to humans, demonstrating the benefit of AR in surgical oncology, but also their limits due to tool tracking and organ deformation. Automatic AR (AAR) efficiently overcomes the surgical instrument tracking limitations, but is still currently imperfect due to the high complexity of real-time organ deformation and human movements. However, the first results of AAR show that it will become a system requirement for future computer-assisted surgical oncology. In this article, we will illustrate several applications and results of AR in digestive surgical oncology.


Augmented Reality External View Nonrigid Registration Augmented Reality System Laparoscopic Camera 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


  1. 1.
    Sielhorst T, Feuerstein M, Navab N (2008) Advanced medical displays: a literature review of augmented reality. J Display Technol 4(4):451–467CrossRefGoogle Scholar
  2. 2.
    OsiriX Imaging Software.
  3. 3.
    Lima M, Reinberg O, Ruggeri G, De Buys Roessingh AS, Gardano T, Soler L, Mogiatti M, Cantone N (2013) 3D virtual rendering before laparoscopic partial splenectomy in children. Journal of Pediatric Surgery 48(8):1784–1788CrossRefPubMedGoogle Scholar
  4. 4.
    Destro F, Maffi M, Gargano T, Ruggeri G, Soler L, Lima M (2013) Thoracoscopic treatment of congenital malformation of the lung: preliminary experience with preoperative 3D virtual rendering. JEMIS 1:1–4Google Scholar
  5. 5.
    Bégin A, Martel G, Lapointe R, Belblidia A, Lepanto L, Soler L, Mutter D, Marescaux J, Vandenbroucke-Menu F (2014) Accuracy of preoperative automatic measurement of the liver volume by CT-scan combined to a 3D virtual surgical planning software. Surg Endosc 28(12):3408–3412CrossRefPubMedGoogle Scholar
  6. 6.
    Soler L, Nicolau S, Pessaux P, Mutter D, Marescaux J (2014) Real-time 3D image reconstruction guidance in liver resection surgery. Hepatobiliary Surg Nutr 3(2):73–81PubMedPubMedCentralGoogle Scholar
  7. 7.
    Mutter D, Soler L, Marescaux J (2010) Recent advances in liver imaging. Informa Healthc 4(5):613–621Google Scholar
  8. 8.
    Osorio A, Galan JA, Nauroy J, Lobato JJ, Dahdouh S, Navarro I, Donars P, Caballero JP, Frenoux E (2008) Planning and validating percutaneous nephrolithotomies in a non-classical patient’s position using a new 3D augmented reality system. Int J CARS 3(1):S130–S132Google Scholar
  9. 9.
    Sugimoto M, Yasuda H, Koda K, Suzuki M, Yamazaki M, Tezuka T, Kosugi C, Higuchi R, Watayo Y, Yagawa Y, Uemura S, Tsuchiya H, Azuma T (2010) Image overlay navigation by markerless surface registration in gastrointestinal, hepatobiliary and pancreatic surgery. J Hepatobiliary Pancreat Sci 17(5):629–636CrossRefPubMedGoogle Scholar
  10. 10.
    Masamune K, Fichtinger G, Deguet A, Matsuka D, Taylor R (2002) An image overlay system with enhanced reality for percutaneous therapy performed inside CT scanner. MICCAI 2002, LNCS 2489:77–84Google Scholar
  11. 11.
    Marescaux J, Rubino F, Arena M, Soler L (2004) Augmented reality assisted laparoscopic adrenalectomy. JAMA 292(18):2214–2215PubMedGoogle Scholar
  12. 12.
    Pessaux P, Soler L, Marzano E, Diana M, Piardi T, Mutter D, Marescaux J (2015) Towards cybernetic surgery: robotic and augmented reality-assisted liver segmentectomy. Langenbecks Arch Surg 400(3):381–385CrossRefPubMedGoogle Scholar
  13. 13.
    Hallet J, Soler L, Diana M, Mutter D, Baumert TF, Habersetzer F, Marescaux J, Pessaux P (2015) Trans-thoracic minimally invasive liver resection guided by augmented reality. J Am Coll Surg 220(5):e55–e60CrossRefPubMedGoogle Scholar
  14. 14.
    Pessaux P, Diana M, Soler L, Piardi T, Mutter D, Marescaux J (2014) Robotic duodenopancreatectomy assisted with augmented reality and real-time fluorescence guidance. Surg Endosc 28(8):2493–2498CrossRefPubMedGoogle Scholar
  15. 15.
    Bernhardt S, Nicolau SA, Agnus V, Soler L, Doignon C, Marescaux J (2014) Automatic detection of endoscope in intraoperative CT image: application to AR guidance in laparoscopic surgery. IEEE international symposium on biomedical imaging (ISBI 2014), pp 563–567Google Scholar
  16. 16.
    Bano J, Hostettler A, Nicolau S, Cotin S, Doignon C, Wu HS, Huang MH, Soler L, Marescaux J (2012) Simulation of pneumoperitoneum for laparoscopic surgery planning. Proceedings of MICCAI, 2012, pp 91–98Google Scholar
  17. 17.
    Bano J, Nicolau SA, Hostettler A, Doignon C, Marescaux J, Soler L (2014) Registration of preoperative liver model for laparoscopic surgery from intraoperative 3d acquisition. In Augmented reality environments for medical imaging and computer-assisted interventions. Springer, Heidelberg New York Dordrecht London, LNCS 8198:201–210Google Scholar
  18. 18.
    Hostettler A, Nicolau SA, Marescaux J, Soler L (2010) A real-time predictive simulation of abdominal viscera position during quiet free breathing. Prog Biophys Mol Biol 103(2-3):169–184CrossRefPubMedGoogle Scholar
  19. 19.
    Mountney P, Yang GZ (2010) Motion compensated slam for image guided surgery. In: Jiang T, Navab N, Pluim JPW, Viergever MA (eds) MICCAI 2010, Part II. LNCS, vol 6362. Springer, Heidelberg, pp 496–504Google Scholar
  20. 20.
    Mountney P, Fallert J, Nicolau S, Soler L, Mewes PW (2014) An augmented reality framework for soft tissue surgery. Med Image Comput Comput Assist Interv 17(1):423–431PubMedGoogle Scholar
  21. 21.
    Haouchine N, Dequidt J, Peterlik I, Kerrien E, Berger MO, Cotin S (2013) Image-guided simulation of heterogeneous tissue deformation for augmented reality during hepatic surgery. ISMAR 2013, pp 199–208Google Scholar
  22. 22.
    Plantefeve R, Peterlik I, Haouchine N, Cotin S (2016) Patient-specific biomechanical modeling for guidance during minimally-invasive hepatic surgery. Ann Biomed Eng 44(1):139–153CrossRefPubMedGoogle Scholar
  23. 23.
    Vappou J, Hou G, Marquet F, Shahmirzadi D, Grondin J, Konofagou E (2015) Non-contact, ultrasound-based indentation method for measuring elastic properties of biological tissues using Harmonic Motion Imaging (HMI). Phys Med Biol 60(7):2853–2868CrossRefPubMedPubMedCentralGoogle Scholar

Copyright information

© Springer International Publishing Switzerland 2017

Authors and Affiliations

  • Luc Soler
    • 1
    Email author
  • Stéphane Nicolau
    • 2
  • Patrick Pessaux
    • 3
  • Didier Mutter
    • 3
  • Jacques Marescaux
    • 3
  1. 1.Luc Soler, IRCADStrasbourgFrance
  2. 2.IRCADStrasbourgFrance
  3. 3.IHU Strasbourg, IRCADStrasbourgFrance

Personalised recommendations