Skip to main content

Advertisement

SpringerLink
Log in

Learning soft tissue behavior of organs for surgical navigation with convolutional neural networks

  • Original Article
  • Published:
International Journal of Computer Assisted Radiology and Surgery Aims and scope Submit manuscript

Abstract

Purpose

In surgical navigation, pre-operative organ models are presented to surgeons during the intervention to help them in efficiently finding their target. In the case of soft tissue, these models need to be deformed and adapted to the current situation by using intra-operative sensor data. A promising method to realize this are real-time capable biomechanical models.

Methods

We train a fully convolutional neural network to estimate a displacement field of all points inside an organ when given only the displacement of a part of the organ’s surface. The network trains on entirely synthetic data of random organ-like meshes, which allows us to use much more data than is otherwise available. The input and output data are discretized into a regular grid, allowing us to fully utilize the capabilities of convolutional operators and to train and infer in a highly parallelized manner.

Results

The system is evaluated on in-silico liver models, phantom liver data and human in-vivo breathing data. We test the performance with varying material parameters, organ shapes and amount of visible surface. Even though the network is only trained on synthetic data, it adapts well to the various cases and gives a good estimation of the internal organ displacement. The inference runs at over 50 frames per second.

Conclusion

We present a novel method for training a data-driven, real-time capable deformation model. The accuracy is comparable to other registration methods, it adapts very well to previously unseen organs and does not need to be re-trained for every patient. The high inferring speed makes this method useful for many applications such as surgical navigation and real-time simulation.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7

Similar content being viewed by others

Notes

  1. Code publically available at: https://gitlab.com/nct_tso_public/cnn-deformation-estimation.

References

  1. Adagolodjo Y, Trivisonne R, Haouchine N, Cotin S, Courtecuisse H (2017) Silhouette-based pose estimation for deformable organs application to surgical augmented reality. In: IROS 2017—IEEE/RSJ international conference on intelligent robots and systems, Vancouver, Canada

  2. Allard J, Courtecuisse H, Faure F (2011) Implicit FEM solver on GPU for interactive deformation simulation. In: Hwu WW (ed) GPU computing gems jade. Elsevier, New York

    Google Scholar 

  3. Bui HP, Tomar S, Chouly F, Lozinski A, Bordas S (2018) Real-time patient specific surgical simulation using corotational cut finite element method: application to needle insertion simulation. In: 13th world congress in computational mechanics, New York, United States

  4. Chen Y, Medioni G (1992) Object modelling by registration of multiple range images. Image Vis Comput 10(3):144–155

    Article  Google Scholar 

  5. Geuzaine C, Remacle JF (2009) Gmsh: a 3-D finite element mesh generator with built-in pre- and post-processing facilities. Int J Numer Methods Eng 79(11):1309–1331

    Article  Google Scholar 

  6. Giannarou S, Visentini-Scarzanella M, Yang G (2013) Probabilistic tracking of affine-invariant anisotropic regions. IEEE Trans Pattern Anal Mach Intell 35(1):130–143

    Article  PubMed  Google Scholar 

  7. Goodfellow I, Bengio Y, Courville A (2016) Deep learning. MIT Press, Cambridge

    Google Scholar 

  8. Guo X, Li W, Iorio F (2016) Convolutional neural networks for steady flow approximation. In: Proceedings of the 22Nd ACM SIGKDD international conference on knowledge discovery and data mining, KDD ’16. ACM

  9. Hambli R, Chamekh A, Salah HBH (2006) Real-time deformation of structure using finite element and Neural networks in virtual reality applications. Finite Elem Anal Des 42:985–991

    Article  Google Scholar 

  10. Kingma DP, Ba JL (2015) Adam: a method for stochastic optimization. In: International conference on learning representations (ICLR), San Diego, Ithaca

  11. Kenngott HG, Wünscher JJ, Wagner M, Preukschas A, Wekerle AL, Neher P, Suwelack S, Speidel S, Nickel F, Oladokun D, Maier-Hein L, Dillmann R, Meinzer HP, Müller-Stich BP (2015) OpenHELP (Heidelberg laparoscopy phantom): development of an open-source surgical evaluation and training tool. Surg Endosc 29:3338–3347

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  12. Koo B, Özgür E, Le Roy B, Buc E, Bartoli A (2017) Deformable registration of a preoperative 3D liver volume to a laparoscopy image using contour and shading cues. In: Medical image computing and computer assisted intervention—MICCAI 2017. Springer International Publishing, Berlin

  13. Lorente D, Martínez-Martínez F, Rupérez MJ, Lago MA, Martínez-Sober M, Escandell-Montero P, Martínez-Martínez JM, Martínez-Sanchis S, Serrano-López AJ, Monserrat C, Martín-Guerrero JD (2017) A framework for modelling the biomechanical behaviour of the human liver during breathing in real time using machine learning. Expert Syst Appl 71:342–357

    Article  Google Scholar 

  14. Malinen M, Råback P (2013) Elmer finite element solver for multiphysics and multiscale problems. Multiscale modelling methods for applications in materials science. Forschungszentrum Jülich, Jülich, pp 101–113

    Google Scholar 

  15. Martínez-Martínez F, Rupérez-Moreno MJ, Martínez-Sober M, Solves-Llorens JA, Lorente D, Serrano-López AJ, Martínez-Sanchis S, Monserrat C, Martín-Guerrero JD (2017) A finite element-based machine learning approach for modeling the mechanical behavior of the breast tissues under compression in real-time. Comput Biol Med 90:116–124

    Article  PubMed  Google Scholar 

  16. Mendizabal A, Duparc RB, Bui HP, Paulus CJ, Peterlik I, Cotin S (2017) Face-based smoothed finite element method for real-time simulation of soft tissue. In: Proceedings of SPIE, vol 10135

  17. Misra S, Macura KJ, Ramesh KT, Okamura AM (2009) The importance of organ geometry and boundary constraints for planning of medical interventions. Med Eng Phys 31(2):195–206

    Article  CAS  PubMed  Google Scholar 

  18. Morooka K, Chen X, Kurazume R, Uchida S, Hara K, Iwashita Y, Hashizume M (2008) Real-time nonlinear fem with neural network for simulating soft organ model deformation. In: Medical image computing and computer-assisted intervention—MICCAI 2008. Springer, Berlin

  19. Myronenko, A., Song, X., Á. Carreira-Perpiñán, M.: Non-rigid point set registration: coherent point drift. In: Advances in neural information processing systems, vol 19 (2006)

  20. Peterlik I, Courtecuisse H, Rohling R, Abolmaesumi P, Nguan C, Cotin S, Salcudean S (2017) Fast elastic registration of soft tissues under large deformations. Med Image Anal 45:24–40

    Article  PubMed  Google Scholar 

  21. Peterlik I, Haouchine N, Ručka L, Cotin S (2017) Image-driven stochastic identification of boundary conditions for predictive simulation. In: 20th international conference on medical image computing and computer assisted intervention, Québec, Canada

  22. Plantefève R, Peterlik I, Haouchine N, Cotin S (2016) Patient-specific biomechanical modeling for guidance during minimally-invasive hepatic surgery. Ann Biomed Eng 44(1):139–153

    Article  PubMed  Google Scholar 

  23. Rechowicz KJ, McKenzie FD (2013) Development and validation methodology of the Nuss procedure surgical planner. Simulation 89(12):1474–1488

    Article  Google Scholar 

  24. Reichard D, Häntsch D, Bodenstedt S, Suwelack S, Wagner M, Kenngott H, Müller-Stich B, Maier-Hein L, Dillmann R, Speidel S (2017) Projective biomechanical depth matching for soft-tissue registration in laparoscopic surgery. Int J Comput Assist Radiology and Surgery (IJCARS) 12(7):1101–1110

    Article  Google Scholar 

  25. Ronneberger O, Fischer P, Brox T (2015) U-Net: convolutional networks for biomedical image segmentation. In: Medical image computing and computer-assisted intervention (MICCAI), LNCS, vol 9351. Springer

  26. Simpson AL, Dumpuri P, Jarnagin WR, Miga MI (2012) Model-assisted image-guided liver surgery using sparse intraoperative data. Springer, Berlin

    Book  Google Scholar 

  27. Suwelack S, Röhl S, Bodenstedt S, Reichard D, Dillmann R, dos Santos T, Maier-Hein L, Wagner M, Wünscher J, Kenngott H, Müller BP, Speidel S (2014) Physics based shape matching for intraoperative image guidance. Med Phys 41:111901

    Article  PubMed  Google Scholar 

  28. Suwelack S, Talbot H, Röhl S, Dillmann R, Speidel S (2011) A biomechanical liver model for intraoperative soft tissue registration. In: Progress in biomedical optics and imaging—proceedings of SPIE, vol 7964

  29. Tonutti M, Gras G, Yang GZ (2017) A machine learning approach for real-time modelling of tissue deformation in image-guided neurosurgery. Artif Intell Med 80:39–47

    Article  PubMed  Google Scholar 

  30. Wu J, Westermann R, Dick C (2014) Real-time haptic cutting of high-resolution soft tissues. Stud Health Technol Inform 196:469–475

    PubMed  Google Scholar 

  31. Yamamoto U, Nakao M, Ohzeki M, Matsuda T (2017) Deformation estimation of an elastic object by partial observation using a neural network. CoRR arXiv:abs/1711.10157

Download references

Author information

Authors and Affiliations

  1. National Center for Tumor Diseases (NCT), Partner Site Dresden, Dresden, Germany

    Micha Pfeiffer & Stefanie Speidel

  2. Department for Visceral, Thoracic and Vascular Surgery, University Hospital, Technical University Dresden, Dresden, Germany

    Carina Riediger & Jürgen Weitz

Authors
  1. Micha Pfeiffer
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Carina Riediger
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Jürgen Weitz
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Stefanie Speidel
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Micha Pfeiffer.

Ethics declarations

Conflict of interest

The authors declare that they have no conflict of interest.

Human and animal rights

This article does not contain any studies with human participants or animals performed by any of the authors.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Pfeiffer, M., Riediger, C., Weitz, J. et al. Learning soft tissue behavior of organs for surgical navigation with convolutional neural networks. Int J CARS 14, 1147–1155 (2019). https://doi.org/10.1007/s11548-019-01965-7

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11548-019-01965-7

Keywords

Search

Navigation