Surgical Endoscopy

, Volume 31, Issue 7, pp 2863–2871 | Cite as

Robust augmented reality registration method for localization of solid organs’ tumors using CT-derived virtual biomechanical model and fluorescent fiducials

  • Seong-Ho Kong
  • Nazim Haouchine
  • Renato Soares
  • Andrey Klymchenko
  • Bohdan Andreiuk
  • Bruno Marques
  • Galyna Shabat
  • Thierry Piechaud
  • Michele Diana
  • Stéphane Cotin
  • Jacques Marescaux



Augmented reality (AR) is the fusion of computer-generated and real-time images. AR can be used in surgery as a navigation tool, by creating a patient-specific virtual model through 3D software manipulation of DICOM imaging (e.g., CT scan). The virtual model can be superimposed to real-time images enabling transparency visualization of internal anatomy and accurate localization of tumors. However, the 3D model is rigid and does not take into account inner structures’ deformations. We present a concept of automated AR registration, while the organs undergo deformation during surgical manipulation, based on finite element modeling (FEM) coupled with optical imaging of fluorescent surface fiducials.


Two 10 × 1 mm wires (pseudo-tumors) and six 10 × 0.9 mm fluorescent fiducials were placed in ex vivo porcine kidneys (n = 10). Biomechanical FEM-based models were generated from CT scan. Kidneys were deformed and the shape changes were identified by tracking the fiducials, using a near-infrared optical system. The changes were registered automatically with the virtual model, which was deformed accordingly. Accuracy of prediction of pseudo-tumors’ location was evaluated with a CT scan in the deformed status (ground truth). In vivo: fluorescent fiducials were inserted under ultrasound guidance in the kidney of one pig, followed by a CT scan. The FEM-based virtual model was superimposed on laparoscopic images by automatic registration of the fiducials.


Biomechanical models were successfully generated and accurately superimposed on optical images. The mean measured distance between the estimated tumor by biomechanical propagation and the scanned tumor (ground truth) was 0.84 ± 0.42 mm. All fiducials were successfully placed in in vivo kidney and well visualized in near-infrared mode enabling accurate automatic registration of the virtual model on the laparoscopic images.


Our preliminary experiments showed the potential of a biomechanical model with fluorescent fiducials to propagate the deformation of solid organs’ surface to their inner structures including tumors with good accuracy and automatized robust tracking.


Augmented reality Automatic registration Optical imaging Finite element modeling Solid organ tumor Fluorescence-guided surgery Fiducials 



Authors are grateful to Christopher Burel, professional in medical English proofreading, for their valuable help in revising the manuscript.

Compliance with ethical standards


SH Kong, N Haouchine, R Soares, A Klymchenko, B Andreiuk, B Marques, G Shabat, T Piechaud, M Diana, S Cotin, and J Marescaux have no conflicts of interest or financial ties to disclose.


  1. 1.
    Kim SP, Thompson RH (2013) Kidney function after partial nephrectomy: current thinking. Curr Opin Urol 23:105–111CrossRefPubMedGoogle Scholar
  2. 2.
    Tang YH, Wen TF, Chen X (2013) Anatomic versus non-anatomic liver resection for hepatocellular carcinoma: a systematic review. Hepato-gastroenterology 60:2019–2025PubMedGoogle Scholar
  3. 3.
    Hou W, Yan W, Ji Z (2015) Anatomic features involved in technical complexity of partial nephrectomy. Urology 85:1–7CrossRefPubMedGoogle Scholar
  4. 4.
    Santambrogio R, Opocher E, Ceretti AP, Barabino M, Costa M, Leone S, Montorsi M (2007) Impact of intraoperative ultrasonography in laparoscopic liver surgery. Surg Endosc 21:181–188CrossRefPubMedGoogle Scholar
  5. 5.
    Marescaux J, Diana M (2015) Next step in minimally invasive surgery: hybrid image-guided surgery. J Pediatr Surg 50:30–36CrossRefPubMedGoogle Scholar
  6. 6.
    D’Agostino J, Diana M, Vix M, Soler L, Marescaux J (2012) Three-dimensional virtual neck exploration before parathyroidectomy. N Engl J Med 367:1072–1073CrossRefPubMedGoogle Scholar
  7. 7.
    Marescaux J, Diana M (2015) Inventing the future of surgery. World J Surg 39:615–622CrossRefPubMedGoogle Scholar
  8. 8.
    Marzano E, Piardi T, Soler L, Diana M, Mutter D, Marescaux J, Pessaux P (2013) Augmented reality-guided artery-first pancreatico-duodenectomy. J Gastrointest Surg 17:1980–1983CrossRefPubMedGoogle Scholar
  9. 9.
    Pessaux P, Diana M, Soler L, Piardi T, Mutter D, Marescaux J (2014) Robotic duodenopancreatectomy assisted with augmented reality and real-time fluorescence guidance. Surg Endosc 28:2493–2498CrossRefPubMedGoogle Scholar
  10. 10.
    Pessaux P, Diana M, Soler L, Piardi T, Mutter D, Marescaux J (2015) Towards cybernetic surgery: robotic and augmented reality-assisted liver segmentectomy. Langenbecks Arch Surg 400:381–385CrossRefPubMedGoogle Scholar
  11. 11.
    Haouchine N, Cotin S, Peterlik I, Dequidt J, Lopez MS, Kerrien E, Berger MO (2015) Impact of soft tissue heterogeneity on augmented reality for liver surgery. IEEE Trans Vis Comput Graph 21:584–597CrossRefPubMedGoogle Scholar
  12. 12.
    Collins BT, Erickson K, Reichner CA, Collins SP, Gagnon GJ, Dieterich S, McRae DA, Zhang Y, Yousefi S, Levy E, Chang T, Jamis-Dow C, Banovac F, Anderson ED (2007) Radical stereotactic radiosurgery with real-time tumor motion tracking in the treatment of small peripheral lung tumors. Radiat Oncol 2:39CrossRefPubMedPubMedCentralGoogle Scholar
  13. 13.
    Dieterich S, Gibbs IC (2011) The CyberKnife in clinical use: current roles, future expectations. Front Radiat Ther Oncol 43:181–194CrossRefPubMedGoogle Scholar
  14. 14.
    Chavalitdhamrong D, DiMaio CJ, Siersema PD, Wagh MS (2015) Technical advances in endoscopic ultrasound-guided fiducial placement for the treatment of pancreatic cancer. Endosc Int Open 3:E373–E377CrossRefPubMedPubMedCentralGoogle Scholar
  15. 15.
    Kong SH, Noh YW, Suh YS, Park HS, Lee HJ, Kang KW, Kim HC, Lim YT, Yang HK (2015) Evaluation of the novel near-infrared fluorescence tracers pullulan polymer nanogel and indocyanine green/gamma-glutamic acid complex for sentinel lymph node navigation surgery in large animal models. Gastric Cancer 18:55–64CrossRefPubMedGoogle Scholar
  16. 16.
    Crum WR, Camara O, Hill DL (2006) Generalized overlap measures for evaluation and validation in medical image analysis. IEEE Trans Med Imaging 25:1451–1461CrossRefPubMedGoogle Scholar
  17. 17.
    Faure F, Duriez C, Delingette H, Allard J, Gilles B, Marchesseau S, Talbot H, Courtecuisse H, Bousquet G, Peterlik I (2012) Sofa: a multi-model framework for interactive physical simulation. Soft tissue biomechanical modeling for computer assisted surgery. Springer, New York, pp 283–321CrossRefGoogle Scholar
  18. 18.
    Yamada H, Evans FG (1970) Strength of biological materials. Williams & Wilkins, PhiladelphiaGoogle Scholar
  19. 19.
    Marescaux J, Diana M, Soler L (2013) Augmented reality and minimally invasive surgery. J Gastroenterol Hepatol Res 2:555–560Google Scholar
  20. 20.
    Pilet J, Lepetit V, Fua P (2008) Fast non-rigid surface detection, registration and realistic augmentation. Int J Comput Vis 76:109–122CrossRefGoogle Scholar
  21. 21.
    Gay-Bellile V, Bartoli A, Sayd P (2010) Direct estimation of nonrigid registrations with image-based self-occlusion reasoning. IEEE Trans Pattern Anal 32:87–104CrossRefGoogle Scholar
  22. 22.
    Haouchine N, Dequidt J, Berger MO, Cotin S (2015) Monocular 3D reconstruction and augmentation of elastic surfaces with self-occlusion handling. IEEE Trans Vis Comput Graph 21:1363–1376CrossRefPubMedGoogle Scholar
  23. 23.
    Marescaux J, Rubino F, Arenas M, Mutter D, Soler L (2004) Augmented-reality-assisted laparoscopic adrenalectomy. JAMA J Am Med Assoc 292:2214–2215Google Scholar
  24. 24.
    Teber D, Guven S, Simpfendorfer T, Baumhauer M, Guven EO, Yencilek F, Gozen AS, Rassweiler J (2009) Augmented reality: a new tool to improve surgical accuracy during laparoscopic partial nephrectomy? Preliminary in vitro and in vivo results. Eur Urol 56:332–338CrossRefPubMedGoogle Scholar
  25. 25.
    Chen Y, Li H, Wu D, Bi K, Liu C (2014) Surgical planning and manual image fusion based on 3D model facilitate laparoscopic partial nephrectomy for intrarenal tumors. World J Urol 32:1493–1499CrossRefPubMedGoogle Scholar
  26. 26.
    Wang D, Zhang B, Yuan X, Zhang X, Liu C (2015) Preoperative planning and real-time assisted navigation by three-dimensional individual digital model in partial nephrectomy with three-dimensional laparoscopic system. Int J Comput Assist Radiol Surg 10:1461–1468CrossRefPubMedGoogle Scholar
  27. 27.
    Marvik R, Lango T, Tangen GA, Andersen JO, Kaspersen JH, Ystgaard B, Sjolie E, Fougner R, Fjosne HE, Nagelhus Hernes TA (2004) Laparoscopic navigation pointer for three-dimensional image-guided surgery. Surg Endosc 18:1242–1248CrossRefPubMedGoogle Scholar
  28. 28.
    Ieiri S, Uemura M, Konishi K, Souzaki R, Nagao Y, Tsutsumi N, Akahoshi T, Ohuchida K, Ohdaira T, Tomikawa M, Tanoue K, Hashizume M, Taguchi T (2012) Augmented reality navigation system for laparoscopic splenectomy in children based on preoperative CT image using optical tracking device. Pediatr Surg Int 28:341–346CrossRefPubMedGoogle Scholar
  29. 29.
    Nam WH, Kang DG, Lee D, Lee JY, Ra JB (2012) Automatic registration between 3D intra-operative ultrasound and pre-operative CT images of the liver based on robust edge matching. Phys Med Biol 57:69–91CrossRefPubMedGoogle Scholar
  30. 30.
    Shekhar R, Dandekar O, Bhat V, Philip M, Lei P, Godinez C, Sutton E, George I, Kavic S, Mezrich R, Park A (2010) Live augmented reality: a new visualization method for laparoscopic surgery using continuous volumetric computed tomography. Surg Endosc 24:1976–1985CrossRefPubMedGoogle Scholar
  31. 31.
    Noh YW, Kong SH, Choi DY, Park HS, Yang HK, Lee HJ, Kim HC, Kang KW, Sung MH, Lim YT (2012) Near-infrared emitting polymer nanogels for efficient sentinel lymph node mapping. ACS Nano 6:7820–7831CrossRefPubMedGoogle Scholar
  32. 32.
    Sutherland SE, Resnick MI, Maclennan GT, Goldman HB (2002) Does the size of the surgical margin in partial nephrectomy for renal cell cancer really matter? J Urol 167:61–64CrossRefPubMedGoogle Scholar
  33. 33.
    Kim SP, Thompson RH, Boorjian SA, Weight CJ, Han LC, Murad MH, Shippee ND, Erwin PJ, Costello BA, Chow GK, Leibovich BC (2012) Comparative effectiveness for survival and renal function of partial and radical nephrectomy for localized renal tumors: a systematic review and meta-analysis. J Urol 188:51–57CrossRefPubMedGoogle Scholar
  34. 34.
    Sun M, Trinh QD, Bianchi M, Hansen J, Hanna N, Abdollah F, Shariat SF, Briganti A, Montorsi F, Perrotte P, Karakiewicz PI (2012) A non-cancer-related survival benefit is associated with partial nephrectomy. Eur Urol 61:725–731CrossRefPubMedGoogle Scholar
  35. 35.
    Lam JS, Bergman J, Breda A, Schulam PG (2008) Importance of surgical margins in the management of renal cell carcinoma. Nat Clin Pract Urol 5:308–317PubMedGoogle Scholar
  36. 36.
    Marszalek M, Carini M, Chlosta P, Jeschke K, Kirkali Z, Knuchel R, Madersbacher S, Patard JJ, Van Poppel H (2012) Positive surgical margins after nephron-sparing surgery. Eur Urol 61:757–763CrossRefPubMedGoogle Scholar
  37. 37.
    Di Pierro GB, Tartaglia N, Aresu L, Polara A, Cielo A, Cristini C, Grande P, Gentile V, Grosso G (2014) Laparoscopic partial nephrectomy for endophytic hilar tumors: feasibility and outcomes. Eur J Surg Oncol 40:769–774CrossRefPubMedGoogle Scholar
  38. 38.
    Autorino R, Khalifeh A, Laydner H, Samarasekera D, Rizkala E, Eyraud R, Stein RJ, Haber GP, Kaouk JH (2014) Robot-assisted partial nephrectomy (RAPN) for completely endophytic renal masses: a single institution experience. BJU Int 113:762–768CrossRefPubMedGoogle Scholar
  39. 39.
    Moreno-Noguer F, Lepetit V, Fua P (2007) Accurate non-iterative o (n) solution to the pnp problem. In: IEEE 11th international conference on computer vision. IEEE, pp 1–8Google Scholar
  40. 40.
    Kim JH, Hong SS, Kim JH, Park HJ, Chang YW, Chang AR, Kwon SB (2012) Safety and efficacy of ultrasound-guided fiducial marker implantation for CyberKnife radiation therapy. Korean J Radiol 13:307–313CrossRefPubMedPubMedCentralGoogle Scholar
  41. 41.
    Kothary N, Heit JJ, Louie JD, Kuo WT, Loo BW Jr, Koong A, Chang DT, Hovsepian D, Sze DY, Hofmann LV (2009) Safety and efficacy of percutaneous fiducial marker implantation for image-guided radiation therapy. J Vasc Interv Radiol JVIR 20:235–239CrossRefPubMedGoogle Scholar
  42. 42.
    Paulus CJ, Haouchine N, Cazier D, Cotin S (2015) Surgical augmented reality with topological changes. International conference on medical image computing and computer-assisted intervention. Springer, New York, pp 413–420Google Scholar

Copyright information

© Springer Science+Business Media New York 2016

Authors and Affiliations

  • Seong-Ho Kong
    • 1
    • 2
  • Nazim Haouchine
    • 3
  • Renato Soares
    • 1
  • Andrey Klymchenko
    • 4
  • Bohdan Andreiuk
    • 4
  • Bruno Marques
    • 3
  • Galyna Shabat
    • 5
  • Thierry Piechaud
    • 6
  • Michele Diana
    • 1
    • 5
  • Stéphane Cotin
    • 3
  • Jacques Marescaux
    • 1
    • 5
  1. 1.IHU-Strasbourg, Institute of Image-Guided SurgeryStrasbourgFrance
  2. 2.Department of SurgerySeoul National University HospitalSeoulKorea
  3. 3.Institut national de recherche en informatique et en automatique (INRIA) MimesisStrasbourgFrance
  4. 4.Biophotonic and Pharmacology Lab, UMR 7213 CNRS, Pharmacological FacultyUniversity of StrasbourgStrasbourgFrance
  5. 5.IRCAD, Research Institute against Cancer of the Digestive SystemStrasbourgFrance
  6. 6.Division of UrologyClinique Saint-AugustinBordeauxFrance

Personalised recommendations