Endoscopic scene labelling and augmentation using intraoperative pulsatile motion and colour appearance cues with preoperative anatomical priors
- 313 Downloads
Despite great advances in medical image segmentation, the accurate and automatic segmentation of endoscopic scenes remains a challenging problem. Two important aspects have to be considered in segmenting an endoscopic scene: (1) noise and clutter due to light reflection and smoke from cutting tissue, and (2) structure occlusion (e.g. vessels occluded by fat, or endophytic tumours occluded by healthy kidney tissue).
In this paper, we propose a variational technique to augment a surgeon’s endoscopic view by segmenting visible as well as occluded structures in the intraoperative endoscopic view. Our method estimates the 3D pose and deformation of anatomical structures segmented from 3D preoperative data in order to align to and segment corresponding structures in 2D intraoperative endoscopic views. Our preoperative to intraoperative alignment is driven by, first, spatio-temporal, signal processing based vessel pulsation cues and, second, machine learning based analysis of colour and textural visual cues. To our knowledge, this is the first work that utilizes vascular pulsation cues for guiding preoperative to intraoperative registration. In addition, we incorporate a tissue-specific (i.e. heterogeneous) physically based deformation model into our framework to cope with the non-rigid deformation of structures that occurs during the intervention.
We validated the utility of our technique on fifteen challenging clinical cases with 45 % improvements in accuracy compared to the state-of-the-art method.
A new technique for localizing both visible and occluded structures in an endoscopic view was proposed and tested. This method leverages both preoperative data, as a source of patient-specific prior knowledge, as well as vasculature pulsation and endoscopic visual cues in order to accurately segment the highly noisy and cluttered environment of an endoscopic video. Our results on in vivo clinical cases of partial nephrectomy illustrate the potential of the proposed framework for augmented reality applications in minimally invasive surgeries.
KeywordsRobotic surgery Partial nephrectomy Image-guided surgery Segmentation 3D pose estimation Endoscopy Patient-specific model Occluded vessels Kidney
This publication was made possible by NPRP Grant #4-161-2-056 from the Qatar National Research Fund (a member of the Qatar Foundation). The statements made herein are solely the responsibility of the authors.
Compliance with ethical standards
Conflicts of interest
The authors declare that they have no conflict of interest.
All procedures performed in studies involving human participants were in accordance with the ethical standards of the institutional and/or national research committee and with the 1964 Helsinki declaration and its later amendments or comparable ethical standards.
This articles does not contain patient information.
- 1.Agudo A, Agapito L, Calvo B, Montiel J (2014) Good vibrations: a modal analysis approach for sequential non-rigid structure from motion. In: IEEE conference on computer vision and pattern recognition (CVPR), pp 1558–1565Google Scholar
- 2.Amir-Khalili A, Hamarneh G, Peyrat JM, Abinahed J, Al-Alao O, Al-Ansari A, Abugharbieh R (2015) Automatic segmentation of occluded vasculature via pulsatile motion analysis in endoscopic robot-assisted partial nephrectomy video. Med Image Anal 25(1):103–110Google Scholar
- 3.Amir-Khalili A, Peyrat JM, Abinahed J, Al-Alao O, Al-Ansari A, Hamarneh G, Abugharbieh R (2014) Auto localization and segmentation of occluded vessels in robot-assisted partial nephrectomy. In: Medical image computing and computer-assisted intervention (MICCAI), pp 407–414Google Scholar
- 4.Andrews S, Hamarneh G (2015) The generalized log-ratio transformation: learning shape and adjacency priors for simultaneous thigh muscle segmentation. IEEE Trans Med Imaging 34(9):1773–1787Google Scholar
- 5.Andrews S, McIntosh C, Hamarneh G (2011) Convex multi-region probabilistic segmentation with shape prior in the isometric log-ratio transformation space. In: IEEE international conference on computer vision (ICCV). IEEE, pp 2096–2103Google Scholar
- 6.Brown E, Chan T, Bresson X (2009) Convex formulation and exact global solutions for multi-phase piecewise constant Mumford–Shah image segmentation. UCLA CAM report, pp 09–66Google Scholar
- 9.Delong A, Boykov Y (2009) Globally optimal segmentation of multi-region objects. In: ieee international conference on computer vision (IEEE ICCV), pp 285–292Google Scholar
- 13.Figueiredo IN, Moreno JC, Prasath VBS, Figueiredo PN (2012) A segmentation model and application to endoscopic images. In: Image analysis and recognition. Springer, pp 164–171Google Scholar
- 16.Gill S, Abolmaesumi P, Vikal S, Mousavi P, Fichtinger G (2008) Intraoperative prostate tracking with slice-to-volume registration in MR. In: International conference of the society for medical innovation and technology, pp 154–158Google Scholar
- 17.Hernes N, Toril A, Lindseth F, Selbekk T, Wollf A, Solberg OV, Harg E, Rygh OM, Tangen GA, Rasmussen I et al (2006) Computer-assisted 3D ultrasound-guided neurosurgery: technological contributions, including multimodal registration and advanced display, demonstrating future perspectives. Int J Med Robot Comput Assist Surg 2(1):45–59CrossRefGoogle Scholar
- 19.McIntosh C, Hamarneh G (2009) Optimal weights for convex functionals in medical image segmentation. In: Advances in visual computing. Springer, pp 1079–1088Google Scholar
- 20.Merritt SA, Rai L, Higgins WE (2006) Real-time CT-video registration for continuous endoscopic guidance. In: Medical imaging. International Society for Optics and Photonics, pp 614313–614313Google Scholar
- 21.Mewes PW, Neumann D, Licegevic O, Simon J, Juloski AL, Angelopoulou E (2011) Automatic region-of-interest segmentation and pathology detection in magnetically guided capsule endoscopy. In: Medical image computing and computer-assisted intervention (MICCAI 2011). Springer, pp 141–148Google Scholar
- 22.Nosrati MS, Andrews S, Hamarneh G (2013) Bounded labeling function for global segmentation of multi-part objects with geometric constraints. In: IEEE international conference on computer vision (ICCV), pp 2032–2039Google Scholar
- 23.Nosrati MS, Peyrat JM, Abinahed J, Al-Alao O, Al-Ansari A, Abugharbieh R, Hamarneh G (2014) Efficient multi-organ segmentation in multi-view endoscopic videos using pre-operative priors. In: Medical image computing and computer-assisted intervention (MICCAI), pp 324–331Google Scholar
- 25.Pickering MR, Muhit AA, Scarvell JM, Smith PN (2009) A new multi-modal similarity measure for fast gradient-based 2D–3D image registration. In: Engineering in medicine and biology society. EMBC 2009. Annual international conference of the IEEE. IEEE, pp 5821–5824Google Scholar
- 27.Puerto-Souza GA, Mariottini GL (2013) Toward long-term and accurate augmented-reality display for minimally-invasive surgery. In: IEEE international conference on robotics and automation (ICRA). IEEE, pp 5384–5389Google Scholar
- 28.Singh I (2009) Robot-assisted laparoscopic partial nephrectomy: current review of the technique and literature. J Min Access Surg 5(4):87Google Scholar
- 33.Zikic D, Glocker B, Kutter O, Groher M, Komodakis N, Khamene A, Paragios N, Navab N (2010) Markov random field optimization for intensity-based 2D–3D registration. In: SPIE medical imaging. International Society for Optics and Photonics, pp 762334–762334Google Scholar