Projective biomechanical depth matching for soft tissue registration in laparoscopic surgery

  • Daniel Reichard
  • Dominik Häntsch
  • Sebastian Bodenstedt
  • Stefan Suwelack
  • Martin Wagner
  • Hannes Kenngott
  • Beat Müller-Stich
  • Lena Maier-Hein
  • Rüdiger Dillmann
  • Stefanie Speidel
Original Article



A key component of computer- assisted surgery systems is the accurate and robust registration of preoperative planning data with intraoperative sensor data. In laparoscopic surgery, this image-based registration remains challenging due to soft tissue deformations. This paper presents a novel approach for biomechanical soft tissue registration of preoperative CT data with stereo endoscopic image data.


The proposed method consists of two registrations steps. First, we use a 3D surface mosaic from partial surfaces reconstructed from stereo endoscopic images to initially align the biomechanical model with the intraoperative position and shape of the organ. After this initialization, the biomechanical model is projected onto newly captured surfaces, resulting in displacement boundary conditions, which in turn are used to update the biomechanical model.


The method is evaluated in silico, using a human liver model, and in vivo, using porcine data. The quantitative in silico data shows a stable behaviour of the biomechanical model and root-mean-square deviation of volume vertices of under 3 mm with adjusted biomechanical parameters.


This work contributes a fully automatic featureless non-rigid registration approach. The results of the in silico and in vivo experiments suggest that our method is able to handle dynamic deformations during surgery. Additional experiments, especially regarding human tissue behaviour, are an important next step towards clinical applications.


Endoscopic vision Biomechanical registration Minimally invasive procedures Intraoperative registration 



The present research was supported by the Klaus Tschira Foundation.

Compliance with ethical standards

Conflict of interest

The authors declare that they have no conflict of interest.

Ethical approval

All applicable international, national and/or institutional guidelines for the care and use of animals were followed. All procedures performed in studies involving animals were in accordance with the ethical standards of the institution or practice at which the studies were conducted.

Informed consent

This article does not contain patient data.

Supplementary material

11548_2017_1613_MOESM1_ESM.mp4 (28.7 mb)
Supplementary material 1 (mp4 29376 KB)


  1. 1.
    Nicolau S, Soler L, Mutter D, Marescaux J (2011) Augmented reality in laparoscopic surgical oncology. Surg Oncol 20(3):189–201CrossRefPubMedGoogle Scholar
  2. 2.
    Maier-Hein L, Mountney P, Bartoli A, Elhawary H, Elson D, Groch A, Kolb A, Rodrigues M, Sorger J, Speidel S, Stoyanov D (2013) Optical techniques for 3D surface reconstruction in computer-assisted laparoscopic surgery. Med Image Anal 17(8):974–996CrossRefPubMedGoogle Scholar
  3. 3.
    Maier-Hein L, Groch A, Bartoli A, Bodenstedt S, Boissonnat G, Chang PL, Clancy NT, Elson DS, Haase S, Heim E, Hornegger J (2014) Comparative validation of single-shot optical techniques for laparoscopic 3-D surface reconstruction. IEEE Trans Med Imaging 33(10):1913–1930CrossRefPubMedGoogle Scholar
  4. 4.
    Allan M, Kapoor A, Mewes P, Mountney P (2015) Non rigid registration of 3D images to laparoscopic video for image guided surgery. International workshop on computer-assisted and robotic endoscopy. Springer International Publishing, BerlinGoogle Scholar
  5. 5.
    Stefansic JD, Herline AJ, Shyr Y, Chapman WC, Fitzpatrick JM, Dawant BM, Galloway RL (2002) Registration of physical space to laparoscopic image space for use in minimally invasive hepatic surgery. In: 5th IEEE EMBS international summer school on biomedical imaging. IEEE, p 12Google Scholar
  6. 6.
    Su LM, Vagvolgyi BP, Agarwal R, Reiley CE, Taylor RH, Hager GD (2009) Augmented reality during robot-assisted laparoscopic partial nephrectomy: toward real-time 3D-CT to stereoscopic video registration. Urology 73(4):896–900CrossRefPubMedGoogle Scholar
  7. 7.
    Herline AJ, Stefansic JD, Debelak JP, Hartmann SL, Pinson CW, Galloway RL, Chapman WC (1999) Image-guided surgery: preliminary feasibility studies of frameless stereotactic liver surgery. Arch Surg 134(6):644–650CrossRefPubMedGoogle Scholar
  8. 8.
    Gupta T, Shin D, Sivagnanadasan N, Hoiem D (2016) 3DFS: deformable dense depth fusion and segmentation for object reconstruction from a handheld camera. arXiv:1606.05002
  9. 9.
    Newcombe RA, Fox D, Seitz SM (2015) Dynamicfusion: reconstruction and tracking of non-rigid scenes in real-time. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 343–352Google Scholar
  10. 10.
    Bregler C, Hertzmann A, Biermann H (2000) Recovering non-rigid 3D shape from image streams. In: Proceedings of IEEE conference on computer vision and pattern recognition, 2000, vol 2. IEEE, pp 690–696Google Scholar
  11. 11.
    Rucker DC, Wu Y, Clements L, Ondrake J, Pheiffer T, Simpson A, Jarnagin W, Miga M (2014) A mechanics-based nonrigid registration method for liver surgery using sparse intraoperative data. IEEE Trans Med Imaging 33(1):147–158CrossRefPubMedGoogle Scholar
  12. 12.
    Plantefve R, Peterlik I, Haouchine N, Cotin S (2015) Patient-specific biomechanical modeling for guidance during minimally-invasive hepatic surgery. Ann. Biomed. Eng. 44(1):139–153CrossRefGoogle Scholar
  13. 13.
    Suwelack S, Röhl S, Bodenstedt S, Reichard D, Dillmann R, Thiago S, Maier-Hein L, Wagner M, Wnscher J, Kenngott H, Müller BP, Speidel S (2014) Physics-based shape matching for intraoperative image guidance. Med Phys 41(11):111901CrossRefPubMedGoogle Scholar
  14. 14.
    Pratt P, Stoyanov D, Visentini-Scarzanella M, Yang G (2010) Dynamic guidance for robotic surgery using image-constrained biomechanical models. In: Medical image computing and computer-assisted intervention-MICCAI, pp 77–85Google Scholar
  15. 15.
    Oktay O, Zhang L, Mansi T, Mountney P, Mewes P, Nicolau S, Soler L, Chefd’hotel C (2013) Biomechanically driven registration of pre-to intra-operative 3D images for laparoscopic surgery. In: Medical image computing and computer-assisted intervention-MICCAI, pp 1–9Google Scholar
  16. 16.
    Nosrati MS, Abugharbieh R, Peyrat JM, Abinahed J, Al-Alao O, Al-Ansari A, Hamarneh G (2016) Simultaneous multi-structure segmentation and 3D nonrigid pose estimation in image-guided robotic surgery. IEEE Trans Med Imaging 35(1):1–9CrossRefPubMedGoogle Scholar
  17. 17.
    Collins T, Bartoli A, Bourdel N, Canis M (2016) Robust, real-time, dense and deformable 3D organ tracking in laparoscopic videos. In: International conference on medical image computing and computer-assisted intervention. Springer International Publishing (2016)Google Scholar
  18. 18.
    dos Santos TR, Seitel A, Kilgus T, Suwelack S, Wekerle AL, Kenngott H, Speidel S, Schlemmer H, Meinzer H, Heimann T, Maier-Hein L (2014) Pose-independent surface matching for intra-operative soft-tissue marker-less registration. Med Image Anal 18(7):1101–1114CrossRefPubMedGoogle Scholar
  19. 19.
    Rusu RB, Blodow N, Beetz M (2009) Fast point feature histograms (FPFH) for 3D registration. In: IEEE international conference on robotics and automation, 2009. ICRA’09. IEEEGoogle Scholar
  20. 20.
    Bodenstedt S, Goertler J,Wagner M, Kenngott H, Mïller-Stich B, Dillmann R, Speidel S (2016) Superpixel-based structure classification for laparoscopic surgery. In: Webster RJ, Yaniv ZR (eds) Medical Imaging 2016: Image-Guided Procedures, Robotic Interventions, and Modeling, vol 9786. Bellingham. doi: 10.1117/12.2216750
  21. 21.
    Roehl S, Bodenstedt S, Suwelack S, Kenngott H, Müller-Stich BP, Dillmann R, Speidel S (2012) Dense GPU-enhanced surface reconstruction from stereo endoscopic images for intraoperative registration. Med Phys 39(3):1632–1645CrossRefGoogle Scholar
  22. 22.
    Nolden M, Zelzer S, Seitel A, Wald D, Mller M, Franz AM, Maleike D, Fangerau M, Baumhauer M, Maier-Hein L, Maier-Hein KH, Meinzer HP, Wolf I (2013) The medical imaging interaction toolkit: challenges and advances. Int J Comput Assist Radiol Surg 8:607–620CrossRefPubMedGoogle Scholar
  23. 23.
    Yeh WC, Li PC, Jeng YM, Hsu HC, Kuo PL, Li ML, Lee PH (2002) Elastic modulus measurements of human liver and correlation with pathology. Ultrasound Med Biol 28(4):467–474CrossRefPubMedGoogle Scholar
  24. 24.
    Fung YC (1981) Biomechanics: mechanical properties of living tissues. Springer, New YorkCrossRefGoogle Scholar
  25. 25.
    Reichard D, Bodenstedt S, Suwelack S, Mayer B, Preukschas A, Wagner M, Kenngott H, Müller-Stich B, Dillmann R, Speidel S (2015) Intraoperative on-the-fly organ-mosaicking for laparoscopic surgery. J Med Imaging 2(4):045001CrossRefGoogle Scholar
  26. 26.
    Shi H (2007) Finite element modeling of soft tissue deformation. ProQuest Dissertations Publishing, University of LouisvilleGoogle Scholar

Copyright information

© CARS 2017

Authors and Affiliations

  • Daniel Reichard
    • 1
  • Dominik Häntsch
    • 1
  • Sebastian Bodenstedt
    • 1
  • Stefan Suwelack
    • 1
  • Martin Wagner
    • 2
  • Hannes Kenngott
    • 2
  • Beat Müller-Stich
    • 2
  • Lena Maier-Hein
    • 3
  • Rüdiger Dillmann
    • 1
  • Stefanie Speidel
    • 1
  1. 1.Karlsruhe Institute of TechnologyKarlsruheGermany
  2. 2.Department of General, Abdominal and Transplantation SurgeryUniversity of HeidelbergHeidelbergGermany
  3. 3.Junior Group Computer-Assisted Interventions, Division of Medical and Biological InformaticsGerman Cancer Research Center (DKFZ)HeidelbergGermany

Personalised recommendations