The visual examination of colonoscopic images fails to extract precise geometric information of the colonic surface. Reconstructing the 3D surface of the colon from colonoscopic image sequences may thus add valuable clinical information. We address this problem of extracting precise spatio-temporal 3D structure information from colonoscopic images.
Using just the intrinsically calibrated monocular image stream, we develop a technique to compute the depth of certain feature points that have been tracked across images. Our method uses the prior knowledge of an approximate geometry of the colon, called the (TTP). It works by fitting a deformable cylindrical model to points reconstructed independently by non-rigid structure-from-motion (NRSfM), compromising between the data term and a novel tubular smoothing prior. Our method represents the first method ever to exploit a very weak topological prior to improve NRSfM. As such, it lies in-between standard NRSfM, which does not use a topological prior beyond the mere plane, and shape-from-template (SfT), which uses a very strong prior as a full deformable 3D object model.
We validate our method on both synthetic images of tubular structures and real colonoscopic data. Our method improves the results obtained by existing NRSfM methods by 71.74% on average on synthetic data and succeeds in obtaining 3D reconstruction from a real colonoscopic sequence defeating the existing methods.
Colonoscopic 3D reconstruction is a difficult problem, which is yet unresolved by the existing methods from computer vision. Our proposed dedicated NRSfM method and experiments show that the visual motion might be the right visual cue to use in colonoscopy.
This is a preview of subscription content, access via your institution.
Buy single article
Instant access to the full article PDF.
Tax calculation will be finalised during checkout.
Subscribe to journal
Immediate online access to all issues from 2019. Subscription will auto renew annually.
Tax calculation will be finalised during checkout.
Barron JT, Malik J (2014) Shape, illumination, and reflectance from shading. IEEE Trans Pattern Anal Mach Intel 37(8):1670–1687
Besl PJ, McKay ND (1992) Method for registration of 3-d shapes. In: Sensor fusion IV: control paradigms and data structures, vol. 1611, pp. 586–606. International Society for Optics and Photonics
Boyd S, Boyd SP, Vandenberghe L (2004) Convex optimization. Cambridge University Press, Cambridge
Bregler C, Hertzmann A, Biermann H (2000) Recovering non-rigid 3d shape from image streams. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, vol. 2, pp. 690–696. IEEE
Chhatkuli A, Pizarro D, Bartoli A (2014) Non-rigid shape-from-motion for isometric surfaces using infinitesimal planarity. In: British Machine Vision Conference
Chhatkuli A, Pizarro D, Collins T, Bartoli A (2017) Inextensible non-rigid structure-from-motion by second-order cone programming. IEEE Trans Pattern Anal Mach Intel 40(10):2428–2441
Collins T, Bartoli A (2012) Towards live monocular 3d laparoscopy using shading and specularity information. In: International Conference on Information Processing in Computer-Assisted Interventions, pp. 11–21. Springer
Kumar S, Dai Y, Li H (2017) Spatio-temporal union of subspaces for multi-body non-rigid structure-from-motion. Pattern Recognit 71:428–443
Liu X, Sinha A, Ishii M, Hager GD, Reiter A, Taylor RH, Unberath M (2019) Dense depth estimation in monocular endoscopy with self-supervised learning methods. IEEE Trans Med Imaging 39(5):1438–1447
Mahmood F, Durr NJ (2018) Deep learning and conditional random fields-based depth estimation and topographical reconstruction from conventional endoscopy. Med Image Anal 48:230–243
Parashar S, Pizarro D, Bartoli A (2017) Isometric non-rigid shape-from-motion with riemannian geometry solved in linear time. IEEE Trans Pattern Anal Mach Intel 40(10):2442–2454
Salzmann M, Fua P (2010) Linear local models for monocular reconstruction of deformable surfaces. IEEE Trans Pattern Anal Mach Intel 33(5):931–944
Salzmann M, Hartley R, Fua P (2007) Convex optimization for deformable surface 3-d tracking. In: 2007 IEEE 11th International Conference on Computer Vision, pp. 1–8. IEEE
Saxena A, Chung SH, Ng AY (2006) Learning depth from single monocular images. In: Advances in Neural Information Processing Systems, pp. 1161–1168
Xiong Y, Chakrabarti A, Basri R, Gortler SJ, Jacobs DW, Zickler T (2014) From shading to local shape. IEEE Trans Pattern Anal Mach Intell 37(1):67–79
Conflict of interest
The authors declare that they have no conflict of interest. Informed consent was obtained from all individual participants included in the study. This article does not contain any studies with animals performed by any of the authors.
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
This work was funded by the FET-Open grant 863146 Endomapper.
About this article
Cite this article
Sengupta, A., Bartoli, A. Colonoscopic 3D reconstruction by tubular non-rigid structure-from-motion. Int J CARS 16, 1237–1241 (2021). https://doi.org/10.1007/s11548-021-02409-x
- Non-rigid structure-from-motion
- 3D reconstruction