Robust ultrasound probe tracking: initial clinical experiences during robot-assisted partial nephrectomy



In order to assist in the identification of renal vasculature and tumour boundaries in robot-assisted partial nephrectomy, robust ultrasound probe calibration and tracking methods are introduced. Contemporaneous image guidance during these crucial stages of the procedure should ultimately lead to improved safety and quality of outcome for the patient, through reduced positive margin rates, segmental clamping, shorter ischaemic times and nephron-sparing resection.


Small KeyDot markers with circular dot patterns are attached to a miniature pickup ultrasound probe. Generic probe calibration is superseded by a more robust scheme based on a sequence of physical transducer measurements. Motion prediction combined with a reduced region-of-interest in the endoscopic video feed facilitates real-time tracking and registration performance at full HD resolutions.


Quantitative analysis confirms that circular dot patterns result in an improved translational and rotational working envelope, in comparison with the previous chessboard pattern implementation. Furthermore, increased robustness is observed with respect to prevailing illumination levels and out-of-focus images due to relatively small endoscopic depths of field.


Circular dot patterns should be employed in this context as they result in improved performance and robustness. This facilitates clinical usage and interpretation of the combined video and ultrasound overlay. The efficacy of the overall system is demonstrated in the first human clinical case.

This is a preview of subscription content, log in to check access.

Access options

Buy single article

Instant unlimited access to the full article PDF.

US$ 39.95

Price includes VAT for USA

Subscribe to journal

Immediate online access to all issues from 2019. Subscription will auto renew annually.

US$ 99

This is the net price. Taxes to be calculated in checkout.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13


  1. 1.

    Hughes-Hallett A, Mayer E, Marcus H, Cundy T, Pratt P, Darzi A, Vale J (2013) Augmented reality partial nephrectomy: examining the current status and future perspectives. Urology 83(2). Elsevier, 266–273

  2. 2.

    Hughes-Hallett A, Pratt P, Mayer E, Martin S, Darzi A, Vale J (2014) Image guidance for all—TilePro display of 3-dimensionally reconstructed images in robotic partial nephrectomy. Urology 84(1). Elsevier, 237–243

  3. 3.

    Sun M, Wagner A, San Francisco I, Brook A, Kavoussi L, Russo P, Steele G, Viterbo R, Pedrosa I (2012) Need for intraoperative ultrasound and surgical recommendation for partial nephrectomy: correlation with tumor imaging features and urologist practice patterns. Ultrasound Q 28(1):21–27

  4. 4.

    Leven J, Burschka D, Kumar R, Zhang G, Blumenkranz S, Xiangtian D, Awad M, Hager G, Marohn M, Choti M, Hasser C, Taylor R (2005) DaVinci canvas: a telerobotic surgical system with integrated, robot-assisted, laparoscopic ultrasound capability. In: MICCAI, LNCS 3749. Springer, Berlin, pp 811–818

  5. 5.

    Schneider C, Dachs G, Hasser C, Choti M, DiMaio S, Taylor R (2010) Robot-assisted laparoscopic ultrasound. In: Information Processing and in Computer-Assisted Interventions, LNCS 6135. Springer, Berlin, pp 67–80

  6. 6.

    Schneider C, Guerrero J, Nguan CY, Rohling R, Salcudean SE (2011) Intra-operative “Pick-Up” ultrasound for robot assisted surgery with vessel extraction and registration: a feasibility study. In: International conference on information processing in computer assisted interventions. Springer, Berlin, pp 122–132

  7. 7.

    Cheung CL, Wedlake C, Moore J, Pautler SE, Peters TM (2010) Fused video and ultrasound images for minimally invasive partial nephrectomy: a phantom study. In: MICCAI, Part III, LNCS 6363. Springer, Berlin, pp 408–415

  8. 8.

    Pratt P, Di Marco A, Payne C, Darzi A, Yang G-Z (2012) Intraoperative ultrasound guidance for transanal endoscopic microsurgery. In: MICCAI, Part I, LNCS 7510. Springer, Berlin, pp 463–470

  9. 9.

    Jayarathne U, McLeod AJ, Peters T, Chen E (2013) Robust intraoperative US probe tracking using a monocular endoscopic camera. In: MICCAI, Part III, LNCS 8151. Springer, Berlin, pp 363–370

  10. 10.

    Hughes-Hallett A, Pratt P, Mayer E, Di Marco A, Yang G-Z, Vale J, Darzi A (2013) Intraoperative ultrasound overlay in robot-assisted partial nephrectomy: first clinical experience. Eur Urol 65(3). Elsevier, 671–972

  11. 11.

    Bradski G, Kaehler A (2008) Learning OpenCV: computer vision with the OpenCV library. O’Reilly Media, Inc., Sebastopol

  12. 12.

    Suzuki S, Abe K (1985) Topological structural analysis of digitized binary images by border following. Comput Vis Graph Image Process 30(1):32–46

  13. 13.

    Zhang Z (2000) A flexible new technique for camera calibration. IEEE Trans Pattern Anal Mach Intell 22(11):1330–1334

  14. 14.

    Fuchs H, State A, Pisano E, Garrett W, Hirota G, Livingston M, Whitton M, Pizer S (1996) Towards performing ultrasound-guided needle biopsies from within a head-mounted display. In: Visualization in Biomedical Computing, LNCS 1131. Springer, pp 591–600

  15. 15.

    Pratt P, Bergeles C, Darzi A, Yang G-Z (2014) Practical intraoperative stereo camera calibration. In: Proceedings of the international conference on medical image computing and computer-assisted intervention. Part II, LNCS 8674. Springer, Berlin, pp 667–675

  16. 16.

    Yushkevich P, Piven J, Cody-Hazlett H, Gimpel-Smith R, Ho S, Gee J, Gerig G (2006) User-guided 3D active contour segmentation of anatomical structures. Neuroimage 31(3):1116–1128

  17. 17.

    Yip M, Adebar T, Rohling R, Salcudean S, Nguan C (2010) 3D Ultrasound to stereoscopic camera registration through an air-tissue boundary. In: MICCAI, LNCS 6362. Springer, Berlin, pp 626–634

  18. 18.

    Cheng A, Kang J, Taylor R, Boctor E (2012) Direct 3D Ultrasound to video registration using photoacoustic effect. In: MICCAI, LNCS 7511. Springer, Berlin, pp 552–559

Download references


The authors are grateful for support from The Hamlyn Centre and the NIHR Biomedical Research Centre funding scheme and would like to thank members of the operating theatre team at the Surgical Innovation Centre, St Mary’s Hospital, Paddington, London.

Author information

Correspondence to Philip Pratt.

Ethics declarations

Conflict of interest

The authors declare that they have no conflict of interest.

Ethical approval

All procedures performed in studies involving human participants were in accordance with the ethical standards of the institutional and/or national research committee and with the 1964 Helsinki Declaration and its later amendments or comparable ethical standards. This article does not contain any studies with animals performed by any of the authors.

Informed consent

Informed consent was obtained from all individual participants included in the study.

Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary material 1 (avi 33606 KB)

Supplementary material 1 (avi 33606 KB)

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Pratt, P., Jaeger, A., Hughes-Hallett, A. et al. Robust ultrasound probe tracking: initial clinical experiences during robot-assisted partial nephrectomy. Int J CARS 10, 1905–1913 (2015) doi:10.1007/s11548-015-1279-x

Download citation


  • Ultrasound
  • Tracking
  • Image guidance
  • Partial nephrectomy
  • Robot-assisted