Robust surface tracking combining features, intensity and illumination compensation



Recovering tissue deformation during robotic-assisted minimally invasive surgery procedures is important for providing intra-operative guidance, enabling in vivo imaging modalities and enhanced robotic control. The tissue motion can also be used to apply motion stabilization and to prescribe dynamic constraints for avoiding critical anatomical structures.


Image-based methods based independently on salient features or on image intensity have limitations when dealing with homogeneous soft tissues or complex reflectance. In this paper, we use a triangular geometric mesh model in order to combine the advantages of both feature and intensity information and track the tissue surface reliably and robustly.


Synthetic and in vivo experiments are performed to provide quantitative analysis of the tracking accuracy of our method, and we also show exemplar results for registering multispectral images where there is only a weak image signal.


Compared to traditional methods, our hybrid tracking method is more robust and has improved convergence in the presence of larger displacements, tissue dynamics and illumination changes.

This is a preview of subscription content, log in to check access.

Access options

Buy single article

Instant unlimited access to the full article PDF.

US$ 39.95

Price includes VAT for USA

Subscribe to journal

Immediate online access to all issues from 2019. Subscription will auto renew annually.

US$ 99

This is the net price. Taxes to be calculated in checkout.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13


  1. 1.


  1. 1.

    Baker S, Matthews I (2004) Lucas–Kanade 20 years on: a unifying framework. Int J Comput Vis 56(3):221–255

  2. 2.

    Bay H, Tuytelaars T, Van Gool L (2006) Surf: speeded up robust features. In: Computer Vision-ECCV 2006, Springer, Berlin, Heidelberg, pp 404–417

  3. 3.

    Bradski G (2000) The opencv library. Doctor Dobbs Journal 25(11):120–126

  4. 4.

    Braux-Zin J, Dupont R, Bartoli A (2013) Combining features and intensity for wide-baseline non-rigid surface registration. In: British machine vision conference (BMVC), BMVA

  5. 5.

    Clancy NT, Stoyanov D, James DR, Di Marco A, Sauvage V, Clark J, Yang GZ, Elson DS (2012) Multispectral image alignment using a three channel endoscope in vivo during minimally invasive surgery. Biomed Opt Exp 3(10):2567–2578

  6. 6.

    Delabarre B, Marchand E (2012) Visual servoing using the sum of conditional variance. In: Intelligent robots and systems (IROS), 2012 IEEE/RSJ international conference on, IEEE, pp 1689–1694

  7. 7.

    Fua P, Brechbühler C (1996) Imposing hard constraints on soft snakes. In: Computer vision-ECCV 1996, Springer, pp 495– 506

  8. 8.

    Giannarou S, Visentini-Scarzanella M, Yang GZ (2009) Affine-invariant anisotropic detector for soft tissue tracking in minimally invasive surgery. In: Biomedical imaging: from Nano to Macro, 2009. ISBI’09. IEEE international symposium on, IEEE, pp 1059–1062

  9. 9.

    Giannarou S, Visentini-Scarzanella M, Yang GZ (2013) Probabilistic tracking of affine-invariant anisotropic regions. Pattern Anal Mach Intell IEEE Trans 35(1):130–143

  10. 10.

    Gröger M, Sepp W, Ortmaier T, Hirzinger G (2001) Reconstruction of image structure in presence of specular reflections. In: Pattern recognition, Springer, pp 53–60

  11. 11.

    Kalal Z, Mikolajczyk K, Matas J (2010) Forward–backward error: automatic detection of tracking failures. In: Pattern recognition (ICPR), 2010 20th international conference on, IEEE, pp 2756–2759

  12. 12.

    Keerthi SS, DeCoste D (2005) A modified finite newton method for fast solution of large scale linear svms. J Mach Learn Res 6:341–361

  13. 13.

    Maier-Hein L, Mountney P, Bartoli A, Elhawary H, Elson D, Groch A, Kolb A, Rodrigues M, Sorger J, Speidel S, Stoyanov D (2013) Optical techniques for 3d surface reconstruction in computer-assisted laparoscopic surgery. Med Image Anal 17(8):974–996

  14. 14.

    Mangasarian OL (2002) A finite newton method for classification. Opt Method Softw 17(5):913–929

  15. 15.

    Matthews I, Ishikawa T, Baker S (2004) The template update problem. Pattern Anal Mach Intell IEEE Trans 26(6):810–815

  16. 16.

    Mountney P, Yang GZ (2008) Soft tissue tracking for minimally invasive surgery: learning local deformation online. In: Medical image computing and computer-assisted intervention-MICCAI 2008, Springer, pp 364–372

  17. 17.

    Ortmaier TJ (2003) Motion compensation in minimally invasive robotic surgery. PhD thesis, Universität München

  18. 18.

    Pickering MR, Muhit AA, Scarvell JM, Smith PN (2009) A new multi-modal similarity measure for fast gradient-based 2d–3d image registration. In: Engineering in medicine and biology society, 2009. EMBC 2009, annual international conference of the IEEE, IEEE, pp 5821–5824

  19. 19.

    Pilet J, Lepetit V, Fua P (2008) Fast non-rigid surface detection, registration and realistic augmentation. Int J Comput Vis 76(2):109–122

  20. 20.

    Puerto-Souza GA, Mariottini GL (2012) Hierarchical multi-affine (hma) algorithm for fast and accurate feature matching in minimally-invasive surgical images. In: Intelligent robots and systems (IROS), 2012 IEEE/RSJ international conference on, IEEE, pp 2007–2012

  21. 21.

    Richa R, Sznitman R, Taylor R, Hager G (2011) Visual tracking using the sum of conditional variance. In: Intelligent robots and systems (IROS), 2011 IEEE/RSJ international conference on, IEEE, pp 2953–2958

  22. 22.

    Selka F, Nicolau SA, Agnus V, Bessaid A, Marescaux J, Soler L (2013) Evaluation of endoscopic image enhancement for feature tracking: a new validation framework. In: Augmented reality environments for medical imaging and computer-assisted interventions, Springer, pp 75–85

  23. 23.

    Stoyanov D (2012) Stereoscopic scene flow for robotic assisted minimally invasive surgery. In: Medical Image Computing and Computer-Assisted Intervention-MICCAI 2012, Springer, pp 479–486

  24. 24.

    Stoyanov D (2012) Surgical vision. Ann Biomed Eng 40(2):332–345

  25. 25.

    Stoyanov D, Yang GZ (2005) Removing specular reflection components for robotic assisted laparoscopic surgery. In: Image processing, 2005. ICIP 2005. IEEE international conference on, IEEE, vol 3, pp III-632

  26. 26.

    Stoyanov D, Darzi A, Yang GZ (2004) Dense 3d depth recovery for soft tissue deformation during robotically assisted laparoscopic surgery. In: Medical image computing and computer-assisted intervention-MICCAI 2004, Springer, pp 41–48

  27. 27.

    Stoyanov D, Darzi A, Yang GZ (2005) A practical approach towards accurate dense 3d depth recovery for robotic laparoscopic surgery. Comput Aided Surg 10(4):199–208

  28. 28.

    Stoyanov D, Rayshubskiy A, Hillman E (2012) Robust registration of multispectral images of the cortical surface in neurosurgery. In: Biomedical imaging (ISBI), 2012 9th IEEE international symposium on, IEEE, pp 1643–1646

  29. 29.

    Yip MC, Lowe DG, Salcudean SE, Rohling RN, Nguan CY (2012) Real-time methods for long-term tissue feature tracking in endoscopic scenes. In: Information processing in computer-assisted interventions, Springer, pp 33–43

  30. 30.

    Zhu J, Lyu MR, Huang TS (2009) A fast 2d shape recovery approach by fusing features and appearance. Pattern Anal Mach Intell IEEE Trans 31(7):1210–1224

Download references


Xiaofei Du is supported by the China Scholarship Council scholarship. Neil Clancy is supported by an Imperial College Junior Research Fellowship. Shobhit Arya is supported by an NIHR-HTD 240 Grant. The authors would like to thank the Northwick Park Institute for Medical Research (NPIMR) for their assistance with surgical arrangements.

Author information

Correspondence to Xiaofei Du.

Ethics declarations

Conflicts of interest

The authors declare no conflict of interest.

Ethical standard

All procedures performed in studies involving human participants were in accordance with the ethical standards of the institutional and/or national research committee and with the 1964 Helsinki declaration and its later amendments or comparable ethical standards.

Informed consent

Informed consent was obtained from all patients who were included in the study.

Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary material 1 (mpg 5780 KB)

Supplementary material 2 (mpg 5789 KB)

Supplementary material 1 (mpg 5780 KB)

Supplementary material 2 (mpg 5789 KB)

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Du, X., Clancy, N., Arya, S. et al. Robust surface tracking combining features, intensity and illumination compensation. Int J CARS 10, 1915–1926 (2015).

Download citation


  • Non-rigid surface tracking
  • Multispectral imaging
  • Minimally invasive surgery
  • Illumination compensation