Recovering tissue deformation during robotic-assisted minimally invasive surgery procedures is important for providing intra-operative guidance, enabling in vivo imaging modalities and enhanced robotic control. The tissue motion can also be used to apply motion stabilization and to prescribe dynamic constraints for avoiding critical anatomical structures.
Image-based methods based independently on salient features or on image intensity have limitations when dealing with homogeneous soft tissues or complex reflectance. In this paper, we use a triangular geometric mesh model in order to combine the advantages of both feature and intensity information and track the tissue surface reliably and robustly.
Synthetic and in vivo experiments are performed to provide quantitative analysis of the tracking accuracy of our method, and we also show exemplar results for registering multispectral images where there is only a weak image signal.
Compared to traditional methods, our hybrid tracking method is more robust and has improved convergence in the presence of larger displacements, tissue dynamics and illumination changes.
This is a preview of subscription content, log in to check access.
Buy single article
Instant unlimited access to the full article PDF.
Price includes VAT for USA
Subscribe to journal
Immediate online access to all issues from 2019. Subscription will auto renew annually.
This is the net price. Taxes to be calculated in checkout.
Baker S, Matthews I (2004) Lucas–Kanade 20 years on: a unifying framework. Int J Comput Vis 56(3):221–255
Bay H, Tuytelaars T, Van Gool L (2006) Surf: speeded up robust features. In: Computer Vision-ECCV 2006, Springer, Berlin, Heidelberg, pp 404–417
Bradski G (2000) The opencv library. Doctor Dobbs Journal 25(11):120–126
Braux-Zin J, Dupont R, Bartoli A (2013) Combining features and intensity for wide-baseline non-rigid surface registration. In: British machine vision conference (BMVC), BMVA
Clancy NT, Stoyanov D, James DR, Di Marco A, Sauvage V, Clark J, Yang GZ, Elson DS (2012) Multispectral image alignment using a three channel endoscope in vivo during minimally invasive surgery. Biomed Opt Exp 3(10):2567–2578
Delabarre B, Marchand E (2012) Visual servoing using the sum of conditional variance. In: Intelligent robots and systems (IROS), 2012 IEEE/RSJ international conference on, IEEE, pp 1689–1694
Fua P, Brechbühler C (1996) Imposing hard constraints on soft snakes. In: Computer vision-ECCV 1996, Springer, pp 495– 506
Giannarou S, Visentini-Scarzanella M, Yang GZ (2009) Affine-invariant anisotropic detector for soft tissue tracking in minimally invasive surgery. In: Biomedical imaging: from Nano to Macro, 2009. ISBI’09. IEEE international symposium on, IEEE, pp 1059–1062
Giannarou S, Visentini-Scarzanella M, Yang GZ (2013) Probabilistic tracking of affine-invariant anisotropic regions. Pattern Anal Mach Intell IEEE Trans 35(1):130–143
Gröger M, Sepp W, Ortmaier T, Hirzinger G (2001) Reconstruction of image structure in presence of specular reflections. In: Pattern recognition, Springer, pp 53–60
Kalal Z, Mikolajczyk K, Matas J (2010) Forward–backward error: automatic detection of tracking failures. In: Pattern recognition (ICPR), 2010 20th international conference on, IEEE, pp 2756–2759
Keerthi SS, DeCoste D (2005) A modified finite newton method for fast solution of large scale linear svms. J Mach Learn Res 6:341–361
Maier-Hein L, Mountney P, Bartoli A, Elhawary H, Elson D, Groch A, Kolb A, Rodrigues M, Sorger J, Speidel S, Stoyanov D (2013) Optical techniques for 3d surface reconstruction in computer-assisted laparoscopic surgery. Med Image Anal 17(8):974–996
Mangasarian OL (2002) A finite newton method for classification. Opt Method Softw 17(5):913–929
Matthews I, Ishikawa T, Baker S (2004) The template update problem. Pattern Anal Mach Intell IEEE Trans 26(6):810–815
Mountney P, Yang GZ (2008) Soft tissue tracking for minimally invasive surgery: learning local deformation online. In: Medical image computing and computer-assisted intervention-MICCAI 2008, Springer, pp 364–372
Ortmaier TJ (2003) Motion compensation in minimally invasive robotic surgery. PhD thesis, Universität München
Pickering MR, Muhit AA, Scarvell JM, Smith PN (2009) A new multi-modal similarity measure for fast gradient-based 2d–3d image registration. In: Engineering in medicine and biology society, 2009. EMBC 2009, annual international conference of the IEEE, IEEE, pp 5821–5824
Pilet J, Lepetit V, Fua P (2008) Fast non-rigid surface detection, registration and realistic augmentation. Int J Comput Vis 76(2):109–122
Puerto-Souza GA, Mariottini GL (2012) Hierarchical multi-affine (hma) algorithm for fast and accurate feature matching in minimally-invasive surgical images. In: Intelligent robots and systems (IROS), 2012 IEEE/RSJ international conference on, IEEE, pp 2007–2012
Richa R, Sznitman R, Taylor R, Hager G (2011) Visual tracking using the sum of conditional variance. In: Intelligent robots and systems (IROS), 2011 IEEE/RSJ international conference on, IEEE, pp 2953–2958
Selka F, Nicolau SA, Agnus V, Bessaid A, Marescaux J, Soler L (2013) Evaluation of endoscopic image enhancement for feature tracking: a new validation framework. In: Augmented reality environments for medical imaging and computer-assisted interventions, Springer, pp 75–85
Stoyanov D (2012) Stereoscopic scene flow for robotic assisted minimally invasive surgery. In: Medical Image Computing and Computer-Assisted Intervention-MICCAI 2012, Springer, pp 479–486
Stoyanov D (2012) Surgical vision. Ann Biomed Eng 40(2):332–345
Stoyanov D, Yang GZ (2005) Removing specular reflection components for robotic assisted laparoscopic surgery. In: Image processing, 2005. ICIP 2005. IEEE international conference on, IEEE, vol 3, pp III-632
Stoyanov D, Darzi A, Yang GZ (2004) Dense 3d depth recovery for soft tissue deformation during robotically assisted laparoscopic surgery. In: Medical image computing and computer-assisted intervention-MICCAI 2004, Springer, pp 41–48
Stoyanov D, Darzi A, Yang GZ (2005) A practical approach towards accurate dense 3d depth recovery for robotic laparoscopic surgery. Comput Aided Surg 10(4):199–208
Stoyanov D, Rayshubskiy A, Hillman E (2012) Robust registration of multispectral images of the cortical surface in neurosurgery. In: Biomedical imaging (ISBI), 2012 9th IEEE international symposium on, IEEE, pp 1643–1646
Yip MC, Lowe DG, Salcudean SE, Rohling RN, Nguan CY (2012) Real-time methods for long-term tissue feature tracking in endoscopic scenes. In: Information processing in computer-assisted interventions, Springer, pp 33–43
Zhu J, Lyu MR, Huang TS (2009) A fast 2d shape recovery approach by fusing features and appearance. Pattern Anal Mach Intell IEEE Trans 31(7):1210–1224
Xiaofei Du is supported by the China Scholarship Council scholarship. Neil Clancy is supported by an Imperial College Junior Research Fellowship. Shobhit Arya is supported by an NIHR-HTD 240 Grant. The authors would like to thank the Northwick Park Institute for Medical Research (NPIMR) for their assistance with surgical arrangements.
Conflicts of interest
The authors declare no conflict of interest.
All procedures performed in studies involving human participants were in accordance with the ethical standards of the institutional and/or national research committee and with the 1964 Helsinki declaration and its later amendments or comparable ethical standards.
Informed consent was obtained from all patients who were included in the study.
About this article
Cite this article
Du, X., Clancy, N., Arya, S. et al. Robust surface tracking combining features, intensity and illumination compensation. Int J CARS 10, 1915–1926 (2015). https://doi.org/10.1007/s11548-015-1243-9
- Non-rigid surface tracking
- Multispectral imaging
- Minimally invasive surgery
- Illumination compensation