Colonoscopic 3D reconstruction by tubular non-rigid structure-from-motion

Abstract

Purpose

The visual examination of colonoscopic images fails to extract precise geometric information of the colonic surface. Reconstructing the 3D surface of the colon from colonoscopic image sequences may thus add valuable clinical information. We address this problem of extracting precise spatio-temporal 3D structure information from colonoscopic images.

Methods

Using just the intrinsically calibrated monocular image stream, we develop a technique to compute the depth of certain feature points that have been tracked across images. Our method uses the prior knowledge of an approximate geometry of the colon, called the (TTP). It works by fitting a deformable cylindrical model to points reconstructed independently by non-rigid structure-from-motion (NRSfM), compromising between the data term and a novel tubular smoothing prior. Our method represents the first method ever to exploit a very weak topological prior to improve NRSfM. As such, it lies in-between standard NRSfM, which does not use a topological prior beyond the mere plane, and shape-from-template (SfT), which uses a very strong prior as a full deformable 3D object model.

Results

We validate our method on both synthetic images of tubular structures and real colonoscopic data. Our method improves the results obtained by existing NRSfM methods by 71.74% on average on synthetic data and succeeds in obtaining 3D reconstruction from a real colonoscopic sequence defeating the existing methods.

Conclusion

Colonoscopic 3D reconstruction is a difficult problem, which is yet unresolved by the existing methods from computer vision. Our proposed dedicated NRSfM method and experiments show that the visual motion might be the right visual cue to use in colonoscopy.

This is a preview of subscription content, access via your institution.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5

References

  1. 1.

    Barron JT, Malik J (2014) Shape, illumination, and reflectance from shading. IEEE Trans Pattern Anal Mach Intel 37(8):1670–1687

    Article  Google Scholar 

  2. 2.

    Besl PJ, McKay ND (1992) Method for registration of 3-d shapes. In: Sensor fusion IV: control paradigms and data structures, vol. 1611, pp. 586–606. International Society for Optics and Photonics

  3. 3.

    Boyd S, Boyd SP, Vandenberghe L (2004) Convex optimization. Cambridge University Press, Cambridge

    Book  Google Scholar 

  4. 4.

    Bregler C, Hertzmann A, Biermann H (2000) Recovering non-rigid 3d shape from image streams. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, vol. 2, pp. 690–696. IEEE

  5. 5.

    Chhatkuli A, Pizarro D, Bartoli A (2014) Non-rigid shape-from-motion for isometric surfaces using infinitesimal planarity. In: British Machine Vision Conference

  6. 6.

    Chhatkuli A, Pizarro D, Collins T, Bartoli A (2017) Inextensible non-rigid structure-from-motion by second-order cone programming. IEEE Trans Pattern Anal Mach Intel 40(10):2428–2441

    Article  Google Scholar 

  7. 7.

    Collins T, Bartoli A (2012) Towards live monocular 3d laparoscopy using shading and specularity information. In: International Conference on Information Processing in Computer-Assisted Interventions, pp. 11–21. Springer

  8. 8.

    Kumar S, Dai Y, Li H (2017) Spatio-temporal union of subspaces for multi-body non-rigid structure-from-motion. Pattern Recognit 71:428–443

    Article  Google Scholar 

  9. 9.

    Liu X, Sinha A, Ishii M, Hager GD, Reiter A, Taylor RH, Unberath M (2019) Dense depth estimation in monocular endoscopy with self-supervised learning methods. IEEE Trans Med Imaging 39(5):1438–1447

    Article  Google Scholar 

  10. 10.

    Mahmood F, Durr NJ (2018) Deep learning and conditional random fields-based depth estimation and topographical reconstruction from conventional endoscopy. Med Image Anal 48:230–243

    Article  Google Scholar 

  11. 11.

    Parashar S, Pizarro D, Bartoli A (2017) Isometric non-rigid shape-from-motion with riemannian geometry solved in linear time. IEEE Trans Pattern Anal Mach Intel 40(10):2442–2454

    Article  Google Scholar 

  12. 12.

    Salzmann M, Fua P (2010) Linear local models for monocular reconstruction of deformable surfaces. IEEE Trans Pattern Anal Mach Intel 33(5):931–944

    Article  Google Scholar 

  13. 13.

    Salzmann M, Hartley R, Fua P (2007) Convex optimization for deformable surface 3-d tracking. In: 2007 IEEE 11th International Conference on Computer Vision, pp. 1–8. IEEE

  14. 14.

    Saxena A, Chung SH, Ng AY (2006) Learning depth from single monocular images. In: Advances in Neural Information Processing Systems, pp. 1161–1168

  15. 15.

    Xiong Y, Chakrabarti A, Basri R, Gortler SJ, Jacobs DW, Zickler T (2014) From shading to local shape. IEEE Trans Pattern Anal Mach Intell 37(1):67–79

    Article  Google Scholar 

Download references

Author information

Affiliations

Authors

Corresponding author

Correspondence to Agniva Sengupta.

Ethics declarations

Conflict of interest

The authors declare that they have no conflict of interest. Informed consent was obtained from all individual participants included in the study. This article does not contain any studies with animals performed by any of the authors.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

This work was funded by the FET-Open grant 863146 Endomapper.

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Sengupta, A., Bartoli, A. Colonoscopic 3D reconstruction by tubular non-rigid structure-from-motion. Int J CARS 16, 1237–1241 (2021). https://doi.org/10.1007/s11548-021-02409-x

Download citation

Keywords

  • Non-rigid structure-from-motion
  • 3D reconstruction
  • Colonoscopy