Physics-Based Deep Neural Network for Augmented Reality During Liver Surgery

  • Jean-Nicolas Brunet
  • Andrea Mendizabal
  • Antoine Petit
  • Nicolas Golse
  • Eric Vibert
  • Stéphane CotinEmail author
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11768)


In this paper we present an approach combining a finite element method and a deep neural network to learn complex elastic deformations with the objective of providing augmented reality during hepatic surgery. Derived from the U-Net architecture, our network is built entirely from physically-based simulations of a preoperative segmentation of the organ. These simulations are performed using an immersed-boundary method, which offers several numerical and practical benefits, such as not requiring boundary-conforming volume elements. We perform a quantitative assessment of the method using synthetic and ex vivo patient data. Results show that the network is capable of solving the deformed state of the organ using only a sparse partial surface displacement data and achieve similar accuracy as a FEM solution, while being about 100\(\times \) faster. When applied to an ex vivo liver example, we achieve the registration in only 3 ms with a mean target registration error (TRE) of 2.9 mm.


Deep learning Real-time simulation Augmented reality 



This study was supported by H2020-MSCA-ITN Marie Skłodowska-Curie Actions, Innovative Training Networks (ITN) - H2020 MSCA ITN 2016 GA EU project number 722068 High Performance Soft Tissue Navigation (HiPerNav).

Supplementary material

Supplementary material 1 (mp4 3952 KB)


  1. 1.
    Ronneberger, O., Fischer, P., Brox, T.: U-Net: convolutional networks for biomedical image segmentation. In: Navab, N., Hornegger, J., Wells, W.M., Frangi, A.F. (eds.) MICCAI 2015. LNCS, vol. 9351, pp. 234–241. Springer, Cham (2015). Scholar
  2. 2.
    Plantefeve, R., et al.: Patient-specific biomechanical modeling for guidance during minimally-invasive hepatic surgery. Ann. Biomed. Eng. 44(1), 139–153 (2016)CrossRefGoogle Scholar
  3. 3.
    Clements, L.W., Chapman, W.C., Dawant, B.M., Galloway, R.L., Miga, M.I.: Robust surface registration using salient anatomical features for image-guided liver surgery: algorithm and validation. Med. Phys. 35(6Part1), 2528–2540 (2008)CrossRefGoogle Scholar
  4. 4.
    Haouchine, N., Dequidt, J., et al.: Image-guided simulation of heterogeneous tissue deformation for augmented reality during hepatic surgery. In: ISMAR, pp. 199–208 (2013)Google Scholar
  5. 5.
    Suwelack, S., Röhl, S., Bodenstedt, S., Reichard, D., et al.: Physics-based shape matching for intraoperative image guidance. Med. Phys. 41(11), 111901 (2014)CrossRefGoogle Scholar
  6. 6.
    Alvarez, P., et al.: Lung deformation between preoperative CT and intraoperative CBCT for thoracoscopic surgery: a case study. In: Medical Imaging, vol. 10576D (2018)Google Scholar
  7. 7.
    Modrzejewski, R., Collins, T., Bartoli, A., Hostettler, A., Marescaux, J.: Soft-body registration of pre-operative 3D models to intra-operative RGBD partial body scans. In: Frangi, A.F., Schnabel, J.A., Davatzikos, C., Alberola-López, C., Fichtinger, G. (eds.) MICCAI 2018. LNCS, vol. 11073, pp. 39–46. Springer, Cham (2018). Scholar
  8. 8.
    Peterlík, I., Duriez, C., Cotin, S.: Modeling and real-time simulation of a vascularized liver tissue. In: Ayache, N., Delingette, H., Golland, P., Mori, K. (eds.) MICCAI 2012. LNCS, vol. 7510, pp. 50–57. Springer, Heidelberg (2012). Scholar
  9. 9.
    Petit, A., Cotin, S.: Environment-aware non-rigid registration in surgery using physics-based simulation. In: ACCV - 14th Asian Conference on Computer Vision (2018)Google Scholar
  10. 10.
    Meier, U., López, O., Monserrat, C., et al.: Real-time deformable models for surgery simulation: a survey. Comput. Methods Programs Biomed. 77(3), 183–197 (2005)CrossRefGoogle Scholar
  11. 11.
    Marchesseau, S., Heimann, T., Chatelin, S., Willinger, R., Delingette, H.: Multiplicative Jacobian energy decomposition method for fast porous visco-hyperelastic soft tissue model. In: Jiang, T., Navab, N., Pluim, J.P.W., Viergever, M.A. (eds.) MICCAI 2010. LNCS, vol. 6361, pp. 235–242. Springer, Heidelberg (2010). Scholar
  12. 12.
    Miller, K., Joldes, G., Lance, D., Wittek, A.: Total Lagrangian explicit dynamics finite element algorithm for computing soft tissue deformation. Commun. Numer. Methods Eng. 23(2), 121–134 (2007)MathSciNetCrossRefGoogle Scholar
  13. 13.
    Collins, T., Pizarro, D., Bartoli, A., Canis, M., Bourdel, N.: Real-time wide-baseline registration of the uterus in monocular laparoscopic videos. In: MICCAI (2013)Google Scholar
  14. 14.
    Heiselman, J.S., et al.: Characterization and correction of intraoperative soft tissue deformation in image-guided laparoscopic liver surgery. J. Med. Imag. 5(2), 021203 (2017)CrossRefGoogle Scholar
  15. 15.
    Petit, A., Lippiello, V., Siciliano, B.: Real-time tracking of 3D elastic objects with an RGB-D sensor. In: IROS, pp. 3914–3921 (2015)Google Scholar
  16. 16.
    Düster, A., et al.: The finite cell method for three-dimensional problems of solid mechanics. Comput. Methods Appl. Mech. Eng. 197(45–48), 3768–3782 (2008)MathSciNetCrossRefGoogle Scholar
  17. 17.
    Niroomandi, S., et al.: Real-time deformable models of non-linear tissues by model reduction techniques. Comput. Methods Programs Biomed. 91(3), 223–231 (2008)CrossRefGoogle Scholar
  18. 18.
    Cifuentes, A., et al.: A performance study of tetrahedral and hexahedral elements in 3-D finite element structural analysis. Finite Elem. Anal. Des. 12, 313–318 (1992)CrossRefGoogle Scholar
  19. 19.
    Benzley, S.E., et al.: A comparison of all hexagonal and all tetrahedral finite element meshes for elastic and elasto-plastic analysis. In: 4th IMR, vol. 17, pp. 179–191 (1995)Google Scholar
  20. 20.
    Wang, E., Nelson, T., Rauch, R.: Back to elements-tetrahedra vs. hexahedra. In: Proceedings of the 2004 International ANSYS Conference (2004)Google Scholar
  21. 21.
    Miller, K., Lu, J.: On the prospect of patient-specific biomechanics without patient-specific properties of tissues. J. Mech. Behav. Biomed. Mater. 27, 154–166 (2013)CrossRefGoogle Scholar

Copyright information

© Springer Nature Switzerland AG 2019

Authors and Affiliations

  • Jean-Nicolas Brunet
    • 1
  • Andrea Mendizabal
    • 1
    • 2
  • Antoine Petit
    • 1
  • Nicolas Golse
    • 3
  • Eric Vibert
    • 3
  • Stéphane Cotin
    • 1
    Email author
  1. 1.INRIAStrasbourgFrance
  2. 2.University of Strasbourg, ICubeStrasbourgFrance
  3. 3.Hôpital Paul-BrousseParisFrance

Personalised recommendations