Learning soft tissue behavior of organs for surgical navigation with convolutional neural networks

  • Micha PfeifferEmail author
  • Carina Riediger
  • Jürgen Weitz
  • Stefanie Speidel
Original Article



In surgical navigation, pre-operative organ models are presented to surgeons during the intervention to help them in efficiently finding their target. In the case of soft tissue, these models need to be deformed and adapted to the current situation by using intra-operative sensor data. A promising method to realize this are real-time capable biomechanical models.


We train a fully convolutional neural network to estimate a displacement field of all points inside an organ when given only the displacement of a part of the organ’s surface. The network trains on entirely synthetic data of random organ-like meshes, which allows us to use much more data than is otherwise available. The input and output data are discretized into a regular grid, allowing us to fully utilize the capabilities of convolutional operators and to train and infer in a highly parallelized manner.


The system is evaluated on in-silico liver models, phantom liver data and human in-vivo breathing data. We test the performance with varying material parameters, organ shapes and amount of visible surface. Even though the network is only trained on synthetic data, it adapts well to the various cases and gives a good estimation of the internal organ displacement. The inference runs at over 50 frames per second.


We present a novel method for training a data-driven, real-time capable deformation model. The accuracy is comparable to other registration methods, it adapts very well to previously unseen organs and does not need to be re-trained for every patient. The high inferring speed makes this method useful for many applications such as surgical navigation and real-time simulation.


Surgical navigation Soft tissue Biomechanical model Organ deformation Convolutional neural network 


Compliance with ethical standards

Conflict of interest

The authors declare that they have no conflict of interest.

Human and animal rights

This article does not contain any studies with human participants or animals performed by any of the authors.


  1. 1.
    Adagolodjo Y, Trivisonne R, Haouchine N, Cotin S, Courtecuisse H (2017) Silhouette-based pose estimation for deformable organs application to surgical augmented reality. In: IROS 2017—IEEE/RSJ international conference on intelligent robots and systems, Vancouver, CanadaGoogle Scholar
  2. 2.
    Allard J, Courtecuisse H, Faure F (2011) Implicit FEM solver on GPU for interactive deformation simulation. In: Hwu WW (ed) GPU computing gems jade. Elsevier, New YorkGoogle Scholar
  3. 3.
    Bui HP, Tomar S, Chouly F, Lozinski A, Bordas S (2018) Real-time patient specific surgical simulation using corotational cut finite element method: application to needle insertion simulation. In: 13th world congress in computational mechanics, New York, United StatesGoogle Scholar
  4. 4.
    Chen Y, Medioni G (1992) Object modelling by registration of multiple range images. Image Vis Comput 10(3):144–155CrossRefGoogle Scholar
  5. 5.
    Geuzaine C, Remacle JF (2009) Gmsh: a 3-D finite element mesh generator with built-in pre- and post-processing facilities. Int J Numer Methods Eng 79(11):1309–1331CrossRefGoogle Scholar
  6. 6.
    Giannarou S, Visentini-Scarzanella M, Yang G (2013) Probabilistic tracking of affine-invariant anisotropic regions. IEEE Trans Pattern Anal Mach Intell 35(1):130–143CrossRefGoogle Scholar
  7. 7.
    Goodfellow I, Bengio Y, Courville A (2016) Deep learning. MIT Press, CambridgeGoogle Scholar
  8. 8.
    Guo X, Li W, Iorio F (2016) Convolutional neural networks for steady flow approximation. In: Proceedings of the 22Nd ACM SIGKDD international conference on knowledge discovery and data mining, KDD ’16. ACMGoogle Scholar
  9. 9.
    Hambli R, Chamekh A, Salah HBH (2006) Real-time deformation of structure using finite element and Neural networks in virtual reality applications. Finite Elem Anal Des 42:985–991CrossRefGoogle Scholar
  10. 10.
    Kingma DP, Ba JL (2015) Adam: a method for stochastic optimization. In: International conference on learning representations (ICLR), San Diego, IthacaGoogle Scholar
  11. 11.
    Kenngott HG, Wünscher JJ, Wagner M, Preukschas A, Wekerle AL, Neher P, Suwelack S, Speidel S, Nickel F, Oladokun D, Maier-Hein L, Dillmann R, Meinzer HP, Müller-Stich BP (2015) OpenHELP (Heidelberg laparoscopy phantom): development of an open-source surgical evaluation and training tool. Surg Endosc 29:3338–3347CrossRefGoogle Scholar
  12. 12.
    Koo B, Özgür E, Le Roy B, Buc E, Bartoli A (2017) Deformable registration of a preoperative 3D liver volume to a laparoscopy image using contour and shading cues. In: Medical image computing and computer assisted intervention—MICCAI 2017. Springer International Publishing, BerlinGoogle Scholar
  13. 13.
    Lorente D, Martínez-Martínez F, Rupérez MJ, Lago MA, Martínez-Sober M, Escandell-Montero P, Martínez-Martínez JM, Martínez-Sanchis S, Serrano-López AJ, Monserrat C, Martín-Guerrero JD (2017) A framework for modelling the biomechanical behaviour of the human liver during breathing in real time using machine learning. Expert Syst Appl 71:342–357CrossRefGoogle Scholar
  14. 14.
    Malinen M, Råback P (2013) Elmer finite element solver for multiphysics and multiscale problems. Multiscale modelling methods for applications in materials science. Forschungszentrum Jülich, Jülich, pp 101–113Google Scholar
  15. 15.
    Martínez-Martínez F, Rupérez-Moreno MJ, Martínez-Sober M, Solves-Llorens JA, Lorente D, Serrano-López AJ, Martínez-Sanchis S, Monserrat C, Martín-Guerrero JD (2017) A finite element-based machine learning approach for modeling the mechanical behavior of the breast tissues under compression in real-time. Comput Biol Med 90:116–124CrossRefGoogle Scholar
  16. 16.
    Mendizabal A, Duparc RB, Bui HP, Paulus CJ, Peterlik I, Cotin S (2017) Face-based smoothed finite element method for real-time simulation of soft tissue. In: Proceedings of SPIE, vol 10135Google Scholar
  17. 17.
    Misra S, Macura KJ, Ramesh KT, Okamura AM (2009) The importance of organ geometry and boundary constraints for planning of medical interventions. Med Eng Phys 31(2):195–206CrossRefGoogle Scholar
  18. 18.
    Morooka K, Chen X, Kurazume R, Uchida S, Hara K, Iwashita Y, Hashizume M (2008) Real-time nonlinear fem with neural network for simulating soft organ model deformation. In: Medical image computing and computer-assisted intervention—MICCAI 2008. Springer, BerlinGoogle Scholar
  19. 19.
    Myronenko, A., Song, X., Á. Carreira-Perpiñán, M.: Non-rigid point set registration: coherent point drift. In: Advances in neural information processing systems, vol 19 (2006)Google Scholar
  20. 20.
    Peterlik I, Courtecuisse H, Rohling R, Abolmaesumi P, Nguan C, Cotin S, Salcudean S (2017) Fast elastic registration of soft tissues under large deformations. Med Image Anal 45:24–40CrossRefGoogle Scholar
  21. 21.
    Peterlik I, Haouchine N, Ručka L, Cotin S (2017) Image-driven stochastic identification of boundary conditions for predictive simulation. In: 20th international conference on medical image computing and computer assisted intervention, Québec, CanadaGoogle Scholar
  22. 22.
    Plantefève R, Peterlik I, Haouchine N, Cotin S (2016) Patient-specific biomechanical modeling for guidance during minimally-invasive hepatic surgery. Ann Biomed Eng 44(1):139–153CrossRefGoogle Scholar
  23. 23.
    Rechowicz KJ, McKenzie FD (2013) Development and validation methodology of the Nuss procedure surgical planner. Simulation 89(12):1474–1488CrossRefGoogle Scholar
  24. 24.
    Reichard D, Häntsch D, Bodenstedt S, Suwelack S, Wagner M, Kenngott H, Müller-Stich B, Maier-Hein L, Dillmann R, Speidel S (2017) Projective biomechanical depth matching for soft-tissue registration in laparoscopic surgery. Int J Comput Assist Radiology and Surgery (IJCARS) 12(7):1101–1110CrossRefGoogle Scholar
  25. 25.
    Ronneberger O, Fischer P, Brox T (2015) U-Net: convolutional networks for biomedical image segmentation. In: Medical image computing and computer-assisted intervention (MICCAI), LNCS, vol 9351. SpringerGoogle Scholar
  26. 26.
    Simpson AL, Dumpuri P, Jarnagin WR, Miga MI (2012) Model-assisted image-guided liver surgery using sparse intraoperative data. Springer, BerlinCrossRefGoogle Scholar
  27. 27.
    Suwelack S, Röhl S, Bodenstedt S, Reichard D, Dillmann R, dos Santos T, Maier-Hein L, Wagner M, Wünscher J, Kenngott H, Müller BP, Speidel S (2014) Physics based shape matching for intraoperative image guidance. Med Phys 41:111901CrossRefGoogle Scholar
  28. 28.
    Suwelack S, Talbot H, Röhl S, Dillmann R, Speidel S (2011) A biomechanical liver model for intraoperative soft tissue registration. In: Progress in biomedical optics and imaging—proceedings of SPIE, vol 7964Google Scholar
  29. 29.
    Tonutti M, Gras G, Yang GZ (2017) A machine learning approach for real-time modelling of tissue deformation in image-guided neurosurgery. Artif Intell Med 80:39–47CrossRefGoogle Scholar
  30. 30.
    Wu J, Westermann R, Dick C (2014) Real-time haptic cutting of high-resolution soft tissues. Stud Health Technol Inform 196:469–475Google Scholar
  31. 31.
    Yamamoto U, Nakao M, Ohzeki M, Matsuda T (2017) Deformation estimation of an elastic object by partial observation using a neural network. CoRR arXiv:abs/1711.10157

Copyright information

© CARS 2019

Authors and Affiliations

  1. 1.National Center for Tumor Diseases (NCT), Partner Site DresdenDresdenGermany
  2. 2.Department for Visceral, Thoracic and Vascular Surgery, University HospitalTechnical University DresdenDresdenGermany

Personalised recommendations