3D-Reconstruction and Semantic Segmentation of Cystoscopic Images
- 172 Downloads
Bladder cancer (BCa) is the fourth most common cancer and the eighth most common cause of cancer-related mortality in men. Although roughly 75% of patients are diagnosed with non-muscle invasive bladder cancer (NMIBC), recurrence rates are high even at a low stage and grade. Transurethral resection (TURB) is required for establishing the pathological diagnosis and clinical staging of patients. In daily clinical practice, however, conventional tumor documentation after TURB lacks accuracy, posing a major limitation for patient follow-up and surveillance. Novel technologies to facilitate data documentation and interpretation processes are imperative to maximize patients’ outcomes. As part of the RaVeNNA-4pi initiative, our contribution is twofold: first, we propose a bladder 3D-reconstruction method using Structure-from-Motion (SfM). Second, we propose deep convolutional neural networks (DCNN) for cystoscopic image segmentation to improve the interpretation of cystoscopic findings and localization of tumors. 3D reconstruction of endoscopic images assists physicians in navigating the bladder and monitoring successive resections. Nevertheless, this process is challenging due to an endoscope’s narrow field of view (FoV), illumination conditions and the bladder’s highly dynamic structure. So far in our project, the SfM approach has been tested on a bladder phantom, demonstrating that the processing sequence permits a 3D reconstruction. Subsequently, we will test our approach on bladder images from patients generated in real-time with a rigid cystoscope. In recent years, deep learning (DL) has enabled significant progress in medical image analysis. Accurate localization of structures such as tumors is of particular interest in processing medical images. In this work, we apply a DCNN for multi-class semantic segmentation of cystoscopic images. Moreover, we introduce a new training dataset for evaluating state-of-the-art DL models on cystoscopic images. Our results show that on average a 0.67 Dice score coefficient (DSC) can be achieved.
KeywordsCystoscopy Bladder cancer 3D-reconstruction Structure from motion (SfM) Deep learning Neural networks Semantic segmentation
This study was funded by the German Federal Ministry of Education and Research (grant number 13GW0203A). We also thank Grigor Andreev for annotation of cystoscopic images that were used to build dataset for training of neural networks. Furthermore, we thank Karl Storz GmbH Tuettlingen for their support in building the experimental setup for the phantom bladder recording.
Conflict of Interest
The authors declare that they have no conflict of interest.
- 1.Fradet, Y., Grossman, H.B., Gomella, L., et al.: A comparison of hexaminolevulinate fluorescence cystoscopy and white light cystoscopy for the detection of carcinoma in situ in patients with bladder cancer: a phase III, multicenter study. J. Urol. 178(1), 68–73 (2007). https://doi.org/10.1016/j.juro.2007.03.028. discussion 73CrossRefGoogle Scholar
- 4.Babjuk, M., Burger, M., Compérat, E., Gontero, P., Mostafid, A.H., Palou, J., van Rhijn, B.W.G., Rouprêt, M., Shariat, S.F., Sylvester, R., Zigeuner, R.: EAU Guidelines edn. presented at the EAU Annual Congress Barcelona (2019)Google Scholar
- 7.Widya, A.R., Monno, Y., Imahori, K. et al.: 3D Reconstruction of whole stomach from endoscope video using structure-from-motion. CoRR abs/1905.12988 (2019)Google Scholar
- 9.Péntek, Q.: 3D-Rekonstruktion menschlicher Harnblasen anhand Videoaufnahmen eines monokularen Endoskops (2016)Google Scholar
- 10.Hanna, S.: Structure from Motion (SfM) (2019). http://gsp.humboldt.edu/OLM/Courses/GSP_216_Online/lesson8-2/SfM.html. Accessed 12 Nov 2019
- 11.Nyimbili, P., Demirel, H., Seker, D., Erden, T.: Structure from motion (SfM) - approaches and applications (2016)Google Scholar
- 12.Lowe, D.G.: Distinctive image features from scale-invariant keypoints. Int. J. Comput. Vis. 60(2), 91–110 (2004). https://doi.org/10.1023/B:VISI.0000029664.99615.94CrossRefGoogle Scholar
- 14.Triggs, B., Mclauchlan, P., Hartley, R. et al.: Bundle adjustment – a modern synthesis. In: International Workshop on Vision Algorithms, pp. 298–372, September 2000Google Scholar
- 15.Ronneberger, O., Fischer, P., Brox, T.: U-Net: Convolutional Networks For Biomedical Image Segmentation. Springer, Cham (2015)Google Scholar
- 16.Long, J., Shelhamer, E., Darrell, T.: Fully convolutional networks for semantic segmentation (2015). https://arxiv.org/pdf/1411.4038
- 17.Ma, X., Hadjiiski, L., Wei, J. et al.: 2D and 3D bladder segmentation using U-Net-based deep-learning. In: International Society for Optics and Photonics, p. 109500Y (2019)Google Scholar
- 18.Dolz, J., Xu, X., Rony, J. et al.: Multi-region segmentation of bladder cancer structures in MRI with progressive dilated convolutional networks (2018). https://arxiv.org/pdf/1805.10720