Skip to main content

Advertisement

SpringerLink
Log in

Robust augmented reality registration method for localization of solid organs’ tumors using CT-derived virtual biomechanical model and fluorescent fiducials

  • Published:
Surgical Endoscopy Aims and scope Submit manuscript

Abstract

Background

Augmented reality (AR) is the fusion of computer-generated and real-time images. AR can be used in surgery as a navigation tool, by creating a patient-specific virtual model through 3D software manipulation of DICOM imaging (e.g., CT scan). The virtual model can be superimposed to real-time images enabling transparency visualization of internal anatomy and accurate localization of tumors. However, the 3D model is rigid and does not take into account inner structures’ deformations. We present a concept of automated AR registration, while the organs undergo deformation during surgical manipulation, based on finite element modeling (FEM) coupled with optical imaging of fluorescent surface fiducials.

Methods

Two 10 × 1 mm wires (pseudo-tumors) and six 10 × 0.9 mm fluorescent fiducials were placed in ex vivo porcine kidneys (n = 10). Biomechanical FEM-based models were generated from CT scan. Kidneys were deformed and the shape changes were identified by tracking the fiducials, using a near-infrared optical system. The changes were registered automatically with the virtual model, which was deformed accordingly. Accuracy of prediction of pseudo-tumors’ location was evaluated with a CT scan in the deformed status (ground truth). In vivo: fluorescent fiducials were inserted under ultrasound guidance in the kidney of one pig, followed by a CT scan. The FEM-based virtual model was superimposed on laparoscopic images by automatic registration of the fiducials.

Results

Biomechanical models were successfully generated and accurately superimposed on optical images. The mean measured distance between the estimated tumor by biomechanical propagation and the scanned tumor (ground truth) was 0.84 ± 0.42 mm. All fiducials were successfully placed in in vivo kidney and well visualized in near-infrared mode enabling accurate automatic registration of the virtual model on the laparoscopic images.

Conclusions

Our preliminary experiments showed the potential of a biomechanical model with fluorescent fiducials to propagate the deformation of solid organs’ surface to their inner structures including tumors with good accuracy and automatized robust tracking.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6

Similar content being viewed by others

References

  1. Kim SP, Thompson RH (2013) Kidney function after partial nephrectomy: current thinking. Curr Opin Urol 23:105–111

    Article  PubMed  Google Scholar 

  2. Tang YH, Wen TF, Chen X (2013) Anatomic versus non-anatomic liver resection for hepatocellular carcinoma: a systematic review. Hepato-gastroenterology 60:2019–2025

    PubMed  Google Scholar 

  3. Hou W, Yan W, Ji Z (2015) Anatomic features involved in technical complexity of partial nephrectomy. Urology 85:1–7

    Article  PubMed  Google Scholar 

  4. Santambrogio R, Opocher E, Ceretti AP, Barabino M, Costa M, Leone S, Montorsi M (2007) Impact of intraoperative ultrasonography in laparoscopic liver surgery. Surg Endosc 21:181–188

    Article  CAS  PubMed  Google Scholar 

  5. Marescaux J, Diana M (2015) Next step in minimally invasive surgery: hybrid image-guided surgery. J Pediatr Surg 50:30–36

    Article  PubMed  Google Scholar 

  6. D’Agostino J, Diana M, Vix M, Soler L, Marescaux J (2012) Three-dimensional virtual neck exploration before parathyroidectomy. N Engl J Med 367:1072–1073

    Article  PubMed  Google Scholar 

  7. Marescaux J, Diana M (2015) Inventing the future of surgery. World J Surg 39:615–622

    Article  PubMed  Google Scholar 

  8. Marzano E, Piardi T, Soler L, Diana M, Mutter D, Marescaux J, Pessaux P (2013) Augmented reality-guided artery-first pancreatico-duodenectomy. J Gastrointest Surg 17:1980–1983

    Article  PubMed  Google Scholar 

  9. Pessaux P, Diana M, Soler L, Piardi T, Mutter D, Marescaux J (2014) Robotic duodenopancreatectomy assisted with augmented reality and real-time fluorescence guidance. Surg Endosc 28:2493–2498

    Article  PubMed  Google Scholar 

  10. Pessaux P, Diana M, Soler L, Piardi T, Mutter D, Marescaux J (2015) Towards cybernetic surgery: robotic and augmented reality-assisted liver segmentectomy. Langenbecks Arch Surg 400:381–385

    Article  PubMed  Google Scholar 

  11. Haouchine N, Cotin S, Peterlik I, Dequidt J, Lopez MS, Kerrien E, Berger MO (2015) Impact of soft tissue heterogeneity on augmented reality for liver surgery. IEEE Trans Vis Comput Graph 21:584–597

    Article  PubMed  Google Scholar 

  12. Collins BT, Erickson K, Reichner CA, Collins SP, Gagnon GJ, Dieterich S, McRae DA, Zhang Y, Yousefi S, Levy E, Chang T, Jamis-Dow C, Banovac F, Anderson ED (2007) Radical stereotactic radiosurgery with real-time tumor motion tracking in the treatment of small peripheral lung tumors. Radiat Oncol 2:39

    Article  PubMed  PubMed Central  Google Scholar 

  13. Dieterich S, Gibbs IC (2011) The CyberKnife in clinical use: current roles, future expectations. Front Radiat Ther Oncol 43:181–194

    Article  PubMed  Google Scholar 

  14. Chavalitdhamrong D, DiMaio CJ, Siersema PD, Wagh MS (2015) Technical advances in endoscopic ultrasound-guided fiducial placement for the treatment of pancreatic cancer. Endosc Int Open 3:E373–E377

    Article  PubMed  PubMed Central  Google Scholar 

  15. Kong SH, Noh YW, Suh YS, Park HS, Lee HJ, Kang KW, Kim HC, Lim YT, Yang HK (2015) Evaluation of the novel near-infrared fluorescence tracers pullulan polymer nanogel and indocyanine green/gamma-glutamic acid complex for sentinel lymph node navigation surgery in large animal models. Gastric Cancer 18:55–64

    Article  CAS  PubMed  Google Scholar 

  16. Crum WR, Camara O, Hill DL (2006) Generalized overlap measures for evaluation and validation in medical image analysis. IEEE Trans Med Imaging 25:1451–1461

    Article  PubMed  Google Scholar 

  17. Faure F, Duriez C, Delingette H, Allard J, Gilles B, Marchesseau S, Talbot H, Courtecuisse H, Bousquet G, Peterlik I (2012) Sofa: a multi-model framework for interactive physical simulation. Soft tissue biomechanical modeling for computer assisted surgery. Springer, New York, pp 283–321

    Chapter  Google Scholar 

  18. Yamada H, Evans FG (1970) Strength of biological materials. Williams & Wilkins, Philadelphia

    Google Scholar 

  19. Marescaux J, Diana M, Soler L (2013) Augmented reality and minimally invasive surgery. J Gastroenterol Hepatol Res 2:555–560

    Google Scholar 

  20. Pilet J, Lepetit V, Fua P (2008) Fast non-rigid surface detection, registration and realistic augmentation. Int J Comput Vis 76:109–122

    Article  Google Scholar 

  21. Gay-Bellile V, Bartoli A, Sayd P (2010) Direct estimation of nonrigid registrations with image-based self-occlusion reasoning. IEEE Trans Pattern Anal 32:87–104

    Article  Google Scholar 

  22. Haouchine N, Dequidt J, Berger MO, Cotin S (2015) Monocular 3D reconstruction and augmentation of elastic surfaces with self-occlusion handling. IEEE Trans Vis Comput Graph 21:1363–1376

    Article  PubMed  Google Scholar 

  23. Marescaux J, Rubino F, Arenas M, Mutter D, Soler L (2004) Augmented-reality-assisted laparoscopic adrenalectomy. JAMA J Am Med Assoc 292:2214–2215

    CAS  Google Scholar 

  24. Teber D, Guven S, Simpfendorfer T, Baumhauer M, Guven EO, Yencilek F, Gozen AS, Rassweiler J (2009) Augmented reality: a new tool to improve surgical accuracy during laparoscopic partial nephrectomy? Preliminary in vitro and in vivo results. Eur Urol 56:332–338

    Article  PubMed  Google Scholar 

  25. Chen Y, Li H, Wu D, Bi K, Liu C (2014) Surgical planning and manual image fusion based on 3D model facilitate laparoscopic partial nephrectomy for intrarenal tumors. World J Urol 32:1493–1499

    Article  PubMed  Google Scholar 

  26. Wang D, Zhang B, Yuan X, Zhang X, Liu C (2015) Preoperative planning and real-time assisted navigation by three-dimensional individual digital model in partial nephrectomy with three-dimensional laparoscopic system. Int J Comput Assist Radiol Surg 10:1461–1468

    Article  PubMed  Google Scholar 

  27. Marvik R, Lango T, Tangen GA, Andersen JO, Kaspersen JH, Ystgaard B, Sjolie E, Fougner R, Fjosne HE, Nagelhus Hernes TA (2004) Laparoscopic navigation pointer for three-dimensional image-guided surgery. Surg Endosc 18:1242–1248

    Article  CAS  PubMed  Google Scholar 

  28. Ieiri S, Uemura M, Konishi K, Souzaki R, Nagao Y, Tsutsumi N, Akahoshi T, Ohuchida K, Ohdaira T, Tomikawa M, Tanoue K, Hashizume M, Taguchi T (2012) Augmented reality navigation system for laparoscopic splenectomy in children based on preoperative CT image using optical tracking device. Pediatr Surg Int 28:341–346

    Article  PubMed  Google Scholar 

  29. Nam WH, Kang DG, Lee D, Lee JY, Ra JB (2012) Automatic registration between 3D intra-operative ultrasound and pre-operative CT images of the liver based on robust edge matching. Phys Med Biol 57:69–91

    Article  PubMed  Google Scholar 

  30. Shekhar R, Dandekar O, Bhat V, Philip M, Lei P, Godinez C, Sutton E, George I, Kavic S, Mezrich R, Park A (2010) Live augmented reality: a new visualization method for laparoscopic surgery using continuous volumetric computed tomography. Surg Endosc 24:1976–1985

    Article  PubMed  Google Scholar 

  31. Noh YW, Kong SH, Choi DY, Park HS, Yang HK, Lee HJ, Kim HC, Kang KW, Sung MH, Lim YT (2012) Near-infrared emitting polymer nanogels for efficient sentinel lymph node mapping. ACS Nano 6:7820–7831

    Article  CAS  PubMed  Google Scholar 

  32. Sutherland SE, Resnick MI, Maclennan GT, Goldman HB (2002) Does the size of the surgical margin in partial nephrectomy for renal cell cancer really matter? J Urol 167:61–64

    Article  PubMed  Google Scholar 

  33. Kim SP, Thompson RH, Boorjian SA, Weight CJ, Han LC, Murad MH, Shippee ND, Erwin PJ, Costello BA, Chow GK, Leibovich BC (2012) Comparative effectiveness for survival and renal function of partial and radical nephrectomy for localized renal tumors: a systematic review and meta-analysis. J Urol 188:51–57

    Article  PubMed  Google Scholar 

  34. Sun M, Trinh QD, Bianchi M, Hansen J, Hanna N, Abdollah F, Shariat SF, Briganti A, Montorsi F, Perrotte P, Karakiewicz PI (2012) A non-cancer-related survival benefit is associated with partial nephrectomy. Eur Urol 61:725–731

    Article  PubMed  Google Scholar 

  35. Lam JS, Bergman J, Breda A, Schulam PG (2008) Importance of surgical margins in the management of renal cell carcinoma. Nat Clin Pract Urol 5:308–317

    PubMed  Google Scholar 

  36. Marszalek M, Carini M, Chlosta P, Jeschke K, Kirkali Z, Knuchel R, Madersbacher S, Patard JJ, Van Poppel H (2012) Positive surgical margins after nephron-sparing surgery. Eur Urol 61:757–763

    Article  PubMed  Google Scholar 

  37. Di Pierro GB, Tartaglia N, Aresu L, Polara A, Cielo A, Cristini C, Grande P, Gentile V, Grosso G (2014) Laparoscopic partial nephrectomy for endophytic hilar tumors: feasibility and outcomes. Eur J Surg Oncol 40:769–774

    Article  PubMed  Google Scholar 

  38. Autorino R, Khalifeh A, Laydner H, Samarasekera D, Rizkala E, Eyraud R, Stein RJ, Haber GP, Kaouk JH (2014) Robot-assisted partial nephrectomy (RAPN) for completely endophytic renal masses: a single institution experience. BJU Int 113:762–768

    Article  PubMed  Google Scholar 

  39. Moreno-Noguer F, Lepetit V, Fua P (2007) Accurate non-iterative o (n) solution to the pnp problem. In: IEEE 11th international conference on computer vision. IEEE, pp 1–8

  40. Kim JH, Hong SS, Kim JH, Park HJ, Chang YW, Chang AR, Kwon SB (2012) Safety and efficacy of ultrasound-guided fiducial marker implantation for CyberKnife radiation therapy. Korean J Radiol 13:307–313

    Article  PubMed  PubMed Central  Google Scholar 

  41. Kothary N, Heit JJ, Louie JD, Kuo WT, Loo BW Jr, Koong A, Chang DT, Hovsepian D, Sze DY, Hofmann LV (2009) Safety and efficacy of percutaneous fiducial marker implantation for image-guided radiation therapy. J Vasc Interv Radiol JVIR 20:235–239

    Article  PubMed  Google Scholar 

  42. Paulus CJ, Haouchine N, Cazier D, Cotin S (2015) Surgical augmented reality with topological changes. International conference on medical image computing and computer-assisted intervention. Springer, New York, pp 413–420

    Google Scholar 

Download references

Acknowledgments

Authors are grateful to Christopher Burel, professional in medical English proofreading, for their valuable help in revising the manuscript.

Author information

Authors and Affiliations

  1. IHU-Strasbourg, Institute of Image-Guided Surgery, Strasbourg, France

    Seong-Ho Kong, Renato Soares, Michele Diana & Jacques Marescaux

  2. Department of Surgery, Seoul National University Hospital, Seoul, Korea

    Seong-Ho Kong

  3. Institut national de recherche en informatique et en automatique (INRIA) Mimesis, Strasbourg, France

    Nazim Haouchine, Bruno Marques & Stéphane Cotin

  4. Biophotonic and Pharmacology Lab, UMR 7213 CNRS, Pharmacological Faculty, University of Strasbourg, Strasbourg, France

    Andrey Klymchenko & Bohdan Andreiuk

  5. IRCAD, Research Institute against Cancer of the Digestive System, 1, Place de l’Hôpital, 67091, Strasbourg, France

    Galyna Shabat, Michele Diana & Jacques Marescaux

  6. Division of Urology, Clinique Saint-Augustin, Bordeaux, France

    Thierry Piechaud

Authors
  1. Seong-Ho Kong
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Nazim Haouchine
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Renato Soares
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Andrey Klymchenko
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Bohdan Andreiuk
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Bruno Marques
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Galyna Shabat
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. Thierry Piechaud
    View author publications

    You can also search for this author in PubMed Google Scholar

  9. Michele Diana
    View author publications

    You can also search for this author in PubMed Google Scholar

  10. Stéphane Cotin
    View author publications

    You can also search for this author in PubMed Google Scholar

  11. Jacques Marescaux
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Michele Diana.

Ethics declarations

Disclosures

SH Kong, N Haouchine, R Soares, A Klymchenko, B Andreiuk, B Marques, G Shabat, T Piechaud, M Diana, S Cotin, and J Marescaux have no conflicts of interest or financial ties to disclose.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Kong, SH., Haouchine, N., Soares, R. et al. Robust augmented reality registration method for localization of solid organs’ tumors using CT-derived virtual biomechanical model and fluorescent fiducials. Surg Endosc 31, 2863–2871 (2017). https://doi.org/10.1007/s00464-016-5297-8

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00464-016-5297-8

Keywords

Search

Navigation