Learning soft tissue behavior of organs for surgical navigation with convolutional neural networks
- 7 Downloads
In surgical navigation, pre-operative organ models are presented to surgeons during the intervention to help them in efficiently finding their target. In the case of soft tissue, these models need to be deformed and adapted to the current situation by using intra-operative sensor data. A promising method to realize this are real-time capable biomechanical models.
We train a fully convolutional neural network to estimate a displacement field of all points inside an organ when given only the displacement of a part of the organ’s surface. The network trains on entirely synthetic data of random organ-like meshes, which allows us to use much more data than is otherwise available. The input and output data are discretized into a regular grid, allowing us to fully utilize the capabilities of convolutional operators and to train and infer in a highly parallelized manner.
The system is evaluated on in-silico liver models, phantom liver data and human in-vivo breathing data. We test the performance with varying material parameters, organ shapes and amount of visible surface. Even though the network is only trained on synthetic data, it adapts well to the various cases and gives a good estimation of the internal organ displacement. The inference runs at over 50 frames per second.
We present a novel method for training a data-driven, real-time capable deformation model. The accuracy is comparable to other registration methods, it adapts very well to previously unseen organs and does not need to be re-trained for every patient. The high inferring speed makes this method useful for many applications such as surgical navigation and real-time simulation.
KeywordsSurgical navigation Soft tissue Biomechanical model Organ deformation Convolutional neural network
Compliance with ethical standards
Conflict of interest
The authors declare that they have no conflict of interest.
Human and animal rights
This article does not contain any studies with human participants or animals performed by any of the authors.
- 1.Adagolodjo Y, Trivisonne R, Haouchine N, Cotin S, Courtecuisse H (2017) Silhouette-based pose estimation for deformable organs application to surgical augmented reality. In: IROS 2017—IEEE/RSJ international conference on intelligent robots and systems, Vancouver, CanadaGoogle Scholar
- 2.Allard J, Courtecuisse H, Faure F (2011) Implicit FEM solver on GPU for interactive deformation simulation. In: Hwu WW (ed) GPU computing gems jade. Elsevier, New YorkGoogle Scholar
- 3.Bui HP, Tomar S, Chouly F, Lozinski A, Bordas S (2018) Real-time patient specific surgical simulation using corotational cut finite element method: application to needle insertion simulation. In: 13th world congress in computational mechanics, New York, United StatesGoogle Scholar
- 7.Goodfellow I, Bengio Y, Courville A (2016) Deep learning. MIT Press, CambridgeGoogle Scholar
- 8.Guo X, Li W, Iorio F (2016) Convolutional neural networks for steady flow approximation. In: Proceedings of the 22Nd ACM SIGKDD international conference on knowledge discovery and data mining, KDD ’16. ACMGoogle Scholar
- 10.Kingma DP, Ba JL (2015) Adam: a method for stochastic optimization. In: International conference on learning representations (ICLR), San Diego, IthacaGoogle Scholar
- 11.Kenngott HG, Wünscher JJ, Wagner M, Preukschas A, Wekerle AL, Neher P, Suwelack S, Speidel S, Nickel F, Oladokun D, Maier-Hein L, Dillmann R, Meinzer HP, Müller-Stich BP (2015) OpenHELP (Heidelberg laparoscopy phantom): development of an open-source surgical evaluation and training tool. Surg Endosc 29:3338–3347CrossRefGoogle Scholar
- 12.Koo B, Özgür E, Le Roy B, Buc E, Bartoli A (2017) Deformable registration of a preoperative 3D liver volume to a laparoscopy image using contour and shading cues. In: Medical image computing and computer assisted intervention—MICCAI 2017. Springer International Publishing, BerlinGoogle Scholar
- 13.Lorente D, Martínez-Martínez F, Rupérez MJ, Lago MA, Martínez-Sober M, Escandell-Montero P, Martínez-Martínez JM, Martínez-Sanchis S, Serrano-López AJ, Monserrat C, Martín-Guerrero JD (2017) A framework for modelling the biomechanical behaviour of the human liver during breathing in real time using machine learning. Expert Syst Appl 71:342–357CrossRefGoogle Scholar
- 14.Malinen M, Råback P (2013) Elmer finite element solver for multiphysics and multiscale problems. Multiscale modelling methods for applications in materials science. Forschungszentrum Jülich, Jülich, pp 101–113Google Scholar
- 15.Martínez-Martínez F, Rupérez-Moreno MJ, Martínez-Sober M, Solves-Llorens JA, Lorente D, Serrano-López AJ, Martínez-Sanchis S, Monserrat C, Martín-Guerrero JD (2017) A finite element-based machine learning approach for modeling the mechanical behavior of the breast tissues under compression in real-time. Comput Biol Med 90:116–124CrossRefGoogle Scholar
- 16.Mendizabal A, Duparc RB, Bui HP, Paulus CJ, Peterlik I, Cotin S (2017) Face-based smoothed finite element method for real-time simulation of soft tissue. In: Proceedings of SPIE, vol 10135Google Scholar
- 18.Morooka K, Chen X, Kurazume R, Uchida S, Hara K, Iwashita Y, Hashizume M (2008) Real-time nonlinear fem with neural network for simulating soft organ model deformation. In: Medical image computing and computer-assisted intervention—MICCAI 2008. Springer, BerlinGoogle Scholar
- 19.Myronenko, A., Song, X., Á. Carreira-Perpiñán, M.: Non-rigid point set registration: coherent point drift. In: Advances in neural information processing systems, vol 19 (2006)Google Scholar
- 21.Peterlik I, Haouchine N, Ručka L, Cotin S (2017) Image-driven stochastic identification of boundary conditions for predictive simulation. In: 20th international conference on medical image computing and computer assisted intervention, Québec, CanadaGoogle Scholar
- 24.Reichard D, Häntsch D, Bodenstedt S, Suwelack S, Wagner M, Kenngott H, Müller-Stich B, Maier-Hein L, Dillmann R, Speidel S (2017) Projective biomechanical depth matching for soft-tissue registration in laparoscopic surgery. Int J Comput Assist Radiology and Surgery (IJCARS) 12(7):1101–1110CrossRefGoogle Scholar
- 25.Ronneberger O, Fischer P, Brox T (2015) U-Net: convolutional networks for biomedical image segmentation. In: Medical image computing and computer-assisted intervention (MICCAI), LNCS, vol 9351. SpringerGoogle Scholar
- 28.Suwelack S, Talbot H, Röhl S, Dillmann R, Speidel S (2011) A biomechanical liver model for intraoperative soft tissue registration. In: Progress in biomedical optics and imaging—proceedings of SPIE, vol 7964Google Scholar
- 30.Wu J, Westermann R, Dick C (2014) Real-time haptic cutting of high-resolution soft tissues. Stud Health Technol Inform 196:469–475Google Scholar
- 31.Yamamoto U, Nakao M, Ohzeki M, Matsuda T (2017) Deformation estimation of an elastic object by partial observation using a neural network. CoRR arXiv:abs/1711.10157