Skip to main content

Advertisement

SpringerLink
Log in

Augmented Reality during Open Liver Surgery Using a Markerless Non-rigid Registration System

  • Original Article
  • Published:
Journal of Gastrointestinal Surgery

Abstract

Introduction

Intraoperative navigation during liver resection remains difficult and requires high radiologic skills because liver anatomy is complex and patient-specific. Augmented reality (AR) during open liver surgery could be helpful to guide hepatectomies and optimize resection margins but faces many challenges when large parenchymal deformations take place. We aimed to experiment a new vision-based AR to assess its clinical feasibility and anatomical accuracy.

Patients and Methods

Based on preoperative CT scan 3-D segmentations, we applied a non-rigid registration method, integrating a physics-based elastic model of the liver, computed in real time using an efficient finite element method. To fit the actual deformations, the model was driven by data provided by a single RGB-D camera. Five livers were considered in this experiment. In vivo AR was performed during hepatectomy (n = 4), with manual handling of the livers resulting in large realistic deformations. Ex vivo experiment (n = 1) consisted in repeated CT scans of explanted whole organ carrying internal metallic landmarks, in fixed deformations, and allowed us to analyze our estimated deformations and quantify spatial errors.

Results

In vivo AR tests were successfully achieved in all patients with a fast and agile setup installation (< 10 min) and real-time overlay of the virtual anatomy onto the surgical field displayed on an external screen. In addition, an ex vivo quantification demonstrated a 7.9 mm root mean square error for the registration of internal landmarks.

Conclusion

These first experiments of a markerless AR provided promising results, requiring very little equipment and setup time, yet providing real-time AR with satisfactory 3D accuracy. These results must be confirmed in a larger prospective study to definitively assess the impact of such minimally invasive technology on pathological margins and oncological outcomes.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6

Similar content being viewed by others

Abbreviations

AR:

Augmented reality

FEM:

Finite element method

FPS:

Frame per second

P:

Patient

RGB-D camera:

Red Green Blue-depth camera

References

  1. Torzilli G, Adam R, Viganò L, Imai K, Goransky J, Fontana A, et al. Surgery of Colorectal Liver Metastases: Pushing the Limits. Liver Cancer. 2016;6(1):80–9.

    Article  Google Scholar 

  2. Majno P, Mentha G, Toso C, Morel P, Peitgen HO, Fasel JHD. Anatomy of the liver: an outline with three levels of complexity--a further step towards tailored territorial liver resections. J Hepatol. Elsevier; 2014;60(3):654–62.

    Article  Google Scholar 

  3. Truant S, Oberlin O, Sergent G, Lebuffe G, Gambiez L, Ernst O, et al. Remnant Liver Volume to Body Weight Ratio ≥0.5%: A New Cut-Off to Estimate Postoperative Risks after Extended Resection in Noncirrhotic Liver. J Am Coll Surg 2007;204(1):22–33.

    Article  Google Scholar 

  4. Traynor O, Castaing D, Bismuth H. Peroperative ultrasonography in the surgery of hepatic tumours. Br J Surg 1988;75(3):197–202.

    Article  CAS  Google Scholar 

  5. Viganò L, Torzilli G, Cimino M, Imai K, Vibert E, Donadon M, et al. Drop-out between the two liver resections of two-stage hepatectomy. Patient selection or loss of chance? Eur J Surg Oncol. 2016 Apr 26.

  6. Owen JW, Fowler KJ, Doyle MB, Saad NE, Linehan DC, Chapman WC. Colorectal liver metastases: disappearing lesions in the era of Eovist hepatobiliary magnetic resonance imaging. HPB (Oxford). 2016;18(3):296–303.

    Article  Google Scholar 

  7. Ntourakis D, Memeo R, Soler L, Marescaux J, Mutter D, Pessaux P. Augmented Reality Guidance for the Resection of Missing Colorectal Liver Metastases: An Initial Experience. World J Surg. 2015 Aug 28.

  8. Kenngott HG, Wagner M, Gondan M, Nickel F, Nolden M, Fetzer A, et al. Real-time image guidance in laparoscopic liver surgery: first clinical experience with a guidance system based on intraoperative CT imaging. Surg Endosc. 2014;28(3):933–40.

    Article  Google Scholar 

  9. Ahn SJ, Lee JM, Lee DH, Lee SM, Yoon J-H, Kim YJ, et al. Real-time US-CT/MR fusion imaging for percutaneous radiofrequency ablation of hepatocellular carcinoma. J Hepatol. 2017;66(2):347–54.

    Article  Google Scholar 

  10. Terasawa M, Ishizawa T, Mise Y, Inoue Y, Ito H, Takahashi Y, et al. Applications of fusion-fluorescence imaging using indocyanine green in laparoscopic hepatectomy. Surg Endosc. 2017;31(12):5111–8.

    Article  Google Scholar 

  11. Faure F, Duriez C, Delingette H, Allard J, Gilles B, Marchesseau S, et al. SOFA: A Multi-Model Framework for Interactive Physical Simulation. In: Soft Tissue Biomechanical Modeling for Computer Assisted Surgery. Berlin, Heidelberg: Springer, Berlin, Heidelberg; 2012. pp. 283–321. (Studies in Mechanobiology, Tissue Engineering and Biomaterials; vol. 11).

  12. Petit A, Lippiello V, Fontanelli GA, Siciliano B. Tracking elastic deformable objects with an RGB-D sensor for a pizza chef robot. Robotics and Autonomous Systems. Elsevier B.V; 2016 Nov 16;:1–15.

  13. Felippa CA, Haugen B. A unified formulation of small-strain corotational finite elements: I. Theory. Comput Methods Appl Mech Engrg. North-Holland; 2005;194(21–24):2285–335.

    Google Scholar 

  14. Cook RD. Finite element modeling for stress analysis. Wiley; 1994 (5).

  15. Petit, A., Cotin, S., Lippiello, V., Siciliano, B.: Capturing deformations of interacting non- rigid objects using rgb-d data. Intelligent Robots and Systems (IROS), 2018 IEEE/RSJ In- ternational Conference on (2018) 3, 5

  16. Petit, A., Lippiello, V., Siciliano, B.: Real-time tracking of 3d elastic objects with an rgb-d sensor. Intelligent Robots and Systems (IROS), 2015 IEEE/RSJ International Conference on (2015) 6

  17. Haouchine, N., Dequidt, J., Peterlik, I., Kerrien, E., Berger, M.O., Cotin, S.: Image-guided simulation of heterogeneous tissue deformation for augmented reality during hepatic surgery. In: Mixed and Augmented Reality (ISMAR), 2013 IEEE International Symposium on, IEEE (2013) 199–208 3, 6

  18. Nishino H, Hatano E, Seo S, Nitta T, Saito T, Nakamura M, et al. Real-time Navigation for Liver Surgery Using Projection Mapping With Indocyanine Green Fluorescence: Development of the Novel Medical Imaging Projection System. Ann Surg. 2017 Feb 8.

  19. Brunet JN, Mendizabal A, Petit A, Golse N, Vibert E, Cotin S. Physics-based Deep Neural Network for Augmented Reality during Liver Surgery. Medical image computing and computer-assisted intervention MICCAI. 2019;16:1–9.

  20. Hirai R, Sakata Y, Tanizawa A, Mori S. Real-time tumor tracking using fluoroscopic imaging with deep neural network analysis. Phys Med. 2019;59:22–9.

    Article  Google Scholar 

  21. Sauer IM, Queisner M, Tang P, Moosburner S, Hoepfner O, Horner R, et al. Mixed Reality in Visceral Surgery: Development of a Suitable Workflow and Evaluation of Intraoperative Use-cases. Ann Surg. Annals of Surgery; 2017;266(5):706–12.

    Article  Google Scholar 

  22. Nicolau S, Soler L, Mutter D, Marescaux J. Augmented reality in laparoscopic surgical oncology. Surg Oncol. Elsevier; 2011;20(3):189–201.

    Article  Google Scholar 

  23. Dilley JWR, Hughes-Hallett A, Pratt PJ, Pucher PH, Camara M, Darzi AW, et al. Perfect Registration Leads to Imperfect Performance: A Randomized Trial of Multimodal Intraoperative Image Guidance. Ann Surg. 2019;269(2):236–42.

    Article  Google Scholar 

  24. Pessaux P, Diana M, Soler L, Piardi T, Mutter D, Marescaux J. Towards cybernetic surgery: robotic and augmented reality-assisted liver segmentectomy. Langenbecks Arch Surg. 2015;400(3):381–5.

    Article  Google Scholar 

  25. Buchs NC, Volonté F, Pugin F, Toso C, Fusaglia M, Gavaghan K, et al. Augmented environments for the targeting of hepatic lesions during image-guided robotic liver surgery. J Surg Res. 2013;184(2):825–31.

    Article  Google Scholar 

  26. Okamoto T, Onda S, Matsumoto M, Gocho T, Futagawa Y, Fujioka S, et al. Utility of augmented reality system in hepatobiliary surgery. J Hepatobiliary Pancreat Sci. 2013;20(2):249–53.

    Article  Google Scholar 

  27. Onda S, Okamoto T, Kanehira M, Fujioka S, Suzuki N, Hattori A, et al. Short rigid scope and stereo-scope designed specifically for open abdominal navigation surgery: clinical application for hepatobiliary and pancreatic surgery. J Hepatobiliary Pancreat Sci. 2013;20(4):448–53.

    Article  Google Scholar 

  28. Tang R, Ma L, Xiang C, Wang X, Li A, Liao H, et al. Augmented reality navigation in open surgery for hilar cholangiocarcinoma resection with hemihepatectomy using video-based in situ three-dimensional anatomical modeling: A case report. Medicine (Baltimore). 2017;96(37):e8083.

    Article  Google Scholar 

  29. Clements LW, Chapman WC, Dawant BM, Galloway RL, Miga MI. Robust surface registration using salient anatomical features for image-guided liver surgery: algorithm and validation. Med Phys. 2008;35(6):2528–40.

    Article  Google Scholar 

  30. Clements LW, Collins JA, Weis JA, Simpson AL, Kingham TP, Jarnagin WR, et al. Deformation correction for image guided liver surgery: An intraoperative fidelity assessment. Surgery. 2017;162(3):537–47.

    Article  Google Scholar 

  31. Rucker DC, Wu Y, Clements LW, Ondrake JE, Pheiffer TS, Simpson AL, et al. A Mechanics-Based Nonrigid Registration Method for Liver Surgery Using Sparse Intraoperative Data. IEEE Trans Med Imaging. 2014;33(1):147–58.

    Article  Google Scholar 

  32. Adagolodjo Y, Golse N, Vibert E, De Mathelin M, Cotin S, Courtecuisse H. Marker-Based Registration for Large Deformations - Application to Open Liver Surgery. IEEE; 2018. pp. 1–6.

Download references

Author information

Author notes

  1. Nicolas Golse and Antoine Petit contributed equally to this work.

Authors and Affiliations

  1. Department of Surgery, Paul-Brousse Hospital, Assistance Publique Hôpitaux de Paris, Centre Hépato-Biliaire, 12 Avenue Paul Vaillant Couturier, 94804, Villejuif Cedex, France

    Nicolas Golse & Eric Vibert

  2. DHU Hepatinov, 94800, Villejuif, France

    Nicolas Golse & Eric Vibert

  3. INSERM, Unit 1193, 94800, Villejuif, France

    Nicolas Golse & Eric Vibert

  4. Univ Paris-Sud, UMR-S 1193, 94800, Villejuif, France

    Nicolas Golse & Eric Vibert

  5. Inria, Strasbourg, France

    Nicolas Golse, Antoine Petit & Stéphane Cotin

  6. Department of Radiology, Paul-Brousse Hospital, Assistance Publique Hôpitaux de Paris, Centre Hépato-Biliaire, 94800, Villejuif, France

    Maïté Lewin

Authors
  1. Nicolas Golse
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Antoine Petit
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Maïté Lewin
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Eric Vibert
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Stéphane Cotin
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

N Golse and A Petit: wrote manuscript, performed experiments. M Lewin: CT scan acquisition, images analysis. S Cotin and E Vibert: research supervision, manuscript rewriting, scientific advices.

Corresponding author

Correspondence to Nicolas Golse.

Ethics declarations

Conflict of Interest

The authors declare that they have no conflict of interest.

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Golse, N., Petit, A., Lewin, M. et al. Augmented Reality during Open Liver Surgery Using a Markerless Non-rigid Registration System. J Gastrointest Surg 25, 662–671 (2021). https://doi.org/10.1007/s11605-020-04519-4

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11605-020-04519-4

Keywords

Search

Navigation