Skip to main content
SpringerLink
Log in

Non-rigid scene reconstruction of deformable soft tissue with monocular endoscopy in minimally invasive surgery

  • Original Article
  • Published:
International Journal of Computer Assisted Radiology and Surgery Aims and scope Submit manuscript

Abstract

Purpose

The utilization of image-guided surgery has demonstrated its ability to improve the precision and safety of minimally invasive surgery (MIS). Non-rigid scene reconstruction is a challenge in image-guided system duo to uniform texture, smoke, and instrument occlusion, etc.

Methods

In this paper, we introduced an algorithm for 3D reconstruction aimed at non-rigid surgery scenes. The proposed method comprises two main components: firstly, the front-end process involves the initial reconstruction of 3D information for deformable soft tissues using embedded deformation graph (EDG) on the basis of dual quaternions, enabling the reconstruction without the need for prior knowledge of the target. Secondly, the EDG is integrated with isometric nonrigid structure from motion (Iso-NRSFM) to facilitate centralized optimization of the observed map points and camera motion across different time instances in deformable scenes.

Results

For the quantitative evaluation of the proposed method, we conducted comparative experiments with both synthetic datasets and publicly available datasets against the state-of-the-art 3D reconstruction method, DefSLAM. The test results show that our proposed method achieved a maximum reduction of 1.6 mm in average reconstruction error compared to method DefSLAM across all datasets. Additionally, qualitative experiments were performed on video scene datasets involving surgical instrument occlusions.

Conclusion

Our method proved to outperform DefSLAM on both synthetic datasets and public datasets through experiments, demonstrating its robustness and accuracy in the reconstruction of soft tissues in dynamic surgical scenes. This success highlights the potential clinical application of our method in delivering surgeons with critical shape and depth information for MIS.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6

Similar content being viewed by others

References

  1. Liu Z, Gao W, Zhu J, Yu Z, Fu Y (2023) Surface deformation tracking in monocular laparoscopic video. Med Image Anal 86:102775. https://doi.org/10.1016/j.media.2023.102775

    Article  PubMed  Google Scholar 

  2. Dou M, Khamis S, Degtyarev Y, Davidson P, Fanello SR, Kowdle A, Escolano SO, Rhemann C, Kim D, Taylor J, Pushmeet K, Vladimir T, Shahram I (2016) Fusion4d: Real-time performance capture of challenging scenes. ACM Trans Graph (ToG) 35(4):1–13

    Article  Google Scholar 

  3. Fletcher J (2022) Methods and applications of 3d patient-specific virtual reconstructions in surgery. Adv Exp Med Biol 1356:53–71. https://doi.org/10.1007/978-3-030-87779-8_3

  4. Wang Y, Long Y, Fan S, Dou Q (2022) Neural rendering for stereo 3d reconstruction of deformable tissues in robotic surgery. In: International conference on medical image computing and computer-assisted intervention. Springer, pp 431–441

  5. Gao W, Tedrake R (2018) Surfelwarp: efficient non-volumetric single view dynamic reconstruction. In: Proceedings of robotics: science and systems, Pittsburgh, Pennsylvania. https://doi.org/10.15607/RSS.2018.XIV.029

  6. Whelan T, Salas-Moreno RF, Glocker B, Davison AJ, Leutenegger S (2016) Elasticfusion: real-time dense slam and light source estimation. Int J Robot Res 35(14):1697–1716

    Article  Google Scholar 

  7. Long Y, Li Z, Yee CH, Ng CF, Taylor RH, Unberath M, Dou Q (2021) E-dssr: efficient dynamic surgical scene reconstruction with transformer-based stereoscopic depth perception. In: Medical image computing and computer assisted intervention–MICCAI 2021: 24th international conference, Strasbourg, France, September 27–October 1, 2021, Proceedings, Part IV 24. Springer, pp 415–425

  8. Song J, Bai F, Zhao L, Huang S, Xiong R (2020) Efficient two step optimization for large embedded deformation graph based slam. In: 2020 IEEE international conference on robotics and automation (ICRA). IEEE, pp 9419–9425

  9. Zhou H, Jayender J (2021) Emdq-slam: real-time high-resolution reconstruction of soft tissue surface from stereo laparoscopy videos. Springer, Berlin, Heidelberg, pp 331–340. https://doi.org/10.1007/978-3-030-87202-1_32

  10. Lamarca J, Parashar S, Bartoli A, Montiel J (2020) Defslam: tracking and mapping of deforming scenes from monocular sequences. IEEE Trans Rob 37(1):291–303

    Article  Google Scholar 

  11. You Y, Wei P, Cai J, Weibo H, Risheng K, Hong L (2022) Misd-slam: multimodal semantic slam for dynamic environments. Wirel Commun Mob Comput 2022:1–13. https://doi.org/10.1155/2022/7600669

    Article  Google Scholar 

  12. Ranftl R, Lasinger K, Hafner D, Schindler K, Koltun V (2022) Towards robust monocular depth estimation: Mixing datasets for zero-shot cross-dataset transfer. IEEE Trans Pattern Anal Mach Intell 44(3):1623–1637. https://doi.org/10.1109/TPAMI.2020.3019967.

  13. Hirohata Y, Sogabe M, Miyazaki T, Toshihiro K, Kenji K (2023) Confidence-aware self-supervised learning for dense monocular depth estimation in dynamic laparoscopic scene. Sci Rep 13:15380. https://doi.org/10.1038/s41598-023-42713-x

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  14. Godard C, Mac Aodha O, Firman M, Brostow GJ (2019) Digging into self-supervised monocular depth estimation. In: Proceedings of the IEEE/CVF international conference on computer vision, pp 3828–3838

  15. Mur-Artal R, Montiel JMM, Tardós JD (2015) Orb-slam: a versatile and accurate monocular slam system. IEEE Trans Rob 31(5):1147–1163

    Article  Google Scholar 

  16. Kavan L, Collins S, Žára J, O’Sullivan C (2007) Skinning with dual quaternions. In: Proceedings of the 2007 symposium on interactive 3D graphics and games. I3D ’07. Association for Computing Machinery, New York, pp 39–46. https://doi.org/10.1145/1230100.1230107

  17. Jia T, Taylor ZA, Chen X (2021) Long term and robust 6dof motion tracking for highly dynamic stereo endoscopy videos. Comput Med Imaging Graph 94:101995

    Article  PubMed  Google Scholar 

  18. Parashar S, Pizarro D, Bartoli A (2017) Isometric non-rigid shape-from-motion with Riemannian geometry solved in linear time. IEEE Trans Pattern Anal Mach Intell 40(10):2442–2454

    Article  PubMed  Google Scholar 

  19. Cartucho J, Tukra S, Li Y, Elson SD, Giannarou S (2021) Visionblender: a tool to efficiently generate computer vision datasets for robotic surgery. Comput Methods Biomech Biomed Eng Imaging Vis 9(4):331–338

  20. Chen Z, Marzullo A, Alberti D, Lievore E, Fontana M, De Cobelli O, Musi G, Ferrigno G, De Momi E (2023) Frsr: framework for real-time scene reconstruction in robot-assisted minimally invasive surgery. Comput Biol Med 163:107121

    Article  PubMed  Google Scholar 

  21. Chen Z, Cruciani L, Lievore E, Fontana M, De Cobelli O, Musi G, Ferrigno G, De Momi E (2024) Spatio-temporal layers based intra-operative stereo depth estimation network via hierarchical prediction and progressive training. Comput Methods Programs Biomed 244:107937

    Article  PubMed  Google Scholar 

  22. Heiselman JS, Jarnagin WR, Miga MI (2020) Intraoperative correction of liver deformation using sparse surface and vascular features via linearized iterative boundary reconstruction. IEEE Trans Med Imaging 39(6):2223–2234

Download references

Acknowledgements

This work was supported by grants from the National Natural Science Foundation of China (82330063; M-0019), the Foundation of Science and Technology Commission of Shanghai Municipality (21S31905200; 22Y11911700), Shanghai Jiao Tong University Foundation on Medical and Technological Joint Science Research (YG2021ZD21; YG2021QN72; YG2022QN056; YG2023ZD19; YG2023ZD15), and the Funding of Xiamen Science and Technology Bureau (No. 3502Z20221012).

Author information

Authors and Affiliations

  1. School of Mechanical Engineering, Shanghai Jiao Tong University, No. 800, Road Dongchuan, Shanghai, 200240, China

    Enpeng Wang, Yueang Liu, Jiangchang Xu & Xiaojun Chen

Authors
  1. Enpeng Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Yueang Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Jiangchang Xu
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Xiaojun Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Xiaojun Chen.

Ethics declarations

Conflict of interest

The authors declare that they have no conflict of interest.

Ethical approval

All procedures performed in studies involving human participants were in accordance with the ethical standards of the institutional and/or national research committee and with the 1964 Helsinki Declaration and its later amendments or comparable ethical standards.

Informed consent

There was no informed consent required for the work.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Wang, E., Liu, Y., Xu, J. et al. Non-rigid scene reconstruction of deformable soft tissue with monocular endoscopy in minimally invasive surgery. Int J CARS (2024). https://doi.org/10.1007/s11548-024-03149-4

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s11548-024-03149-4

Keywords

Search

Navigation