Uncertainty-Encoded Augmented Reality for Robot-Assisted Partial Nephrectomy: A Phantom Study

  • Alborz Amir-Khalili
  • Masoud S. Nosrati
  • Jean-Marc Peyrat
  • Ghassan Hamarneh
  • Rafeef Abugharbieh
Part of the Lecture Notes in Computer Science book series (LNCS, volume 8090)


In most robot-assisted surgical interventions, multimodal fusion of pre- and intra-operative data is highly valuable, affording the surgeon a more comprehensive understanding of the surgical scene observed through the stereo endoscopic camera. More specifically, in the case of partial nephrectomy, fusing pre-operative segmentations of kidney and tumor with the stereo endoscopic view can guide tumor localization and the identification of resection margins. However, the surgeons are often unable to reliably assess the levels of trust they can bestow on what is overlaid on the screen. In this paper, we present the proof-of-concept of an uncertainty-encoded augmented reality framework and novel visualizations of the uncertainties derived from the pre-operative CT segmentation onto the surgeon’s stereo endoscopic view. To verify its clinical potential, the proposed method is applied to an ex vivo lamb kidney. The results are contrasted to different visualization solutions based on crisp segmentation demonstrating that our method provides valuable additional information that can help the surgeon during the resection planning.


Augmented Reality Partial Nephrectomy Stereo Match Stereo Camera Normalize Cross Correlation 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Pratt, P., Mayer, E., Vale, J., Cohen, D., Edwards, E., Darzi, A., Yang, G.-Z.: An effective visualisation and registration system for image-guided robotic partial nephrectomy. Journal of Robotic Surgery, 1–9 (2012)Google Scholar
  2. 2.
    Udupa, J., Grevera, G.: Go digital, go fuzzy. Pattern Recognition Letters 23(6), 743–754 (2002)CrossRefzbMATHGoogle Scholar
  3. 3.
    Zhang, Y., Brady, M., Smith, S.: Segmentation of brain MR images through a hidden markov random field model and the expectation-maximization algorithm. IEEE Transactions on Medical Imaging 20(1), 45–57 (2001)CrossRefGoogle Scholar
  4. 4.
    Grady, L.: Random walks for image segmentation. IEEE Transactions on Pattern Analysis and Machine Intelligence 28(11), 1768–1783 (2006)CrossRefGoogle Scholar
  5. 5.
    Warfield, S., Zou, K., Wells, W.: Simultaneous truth and performance level estimation (STAPLE): An algorithm for the validation of image segmentation. IEEE Transactions on Medical Imaging 23(7), 903–921 (2004)CrossRefGoogle Scholar
  6. 6.
    Konukoglu, E., Clatz, O., Bondiau, P., Delingette, H., Ayache, N.: Extrapolating glioma invasion margin in brain magnetic resonance images: suggesting new irradiation margins. Medical Image Analysis 14(2), 111–125 (2010)CrossRefGoogle Scholar
  7. 7.
    Sinop, A.K., Grady, L.: A seeded image segmentation framework unifying graph cuts and random walker which yields a new algorithm. In: ICCV, pp. 1–8 (2007)Google Scholar
  8. 8.
    Hartley, R., Zisserman, A.: Multiple view geometry in computer vision, 2nd edn. Cambridge Univ. Press (2004)Google Scholar
  9. 9.
    Pollefeys, M., Koch, R., Van Gool, L.: A simple and efficient rectification method for general motion. In: ICCV, pp. 496–501 (1999)Google Scholar
  10. 10.
    Stoyanov, D., Scarzanella, M.V., Pratt, P., Yang, G.-Z.: Real-time stereo reconstruction in robotically assisted minimally invasive surgery. In: Jiang, T., Navab, N., Pluim, J.P.W., Viergever, M.A. (eds.) MICCAI 2010, Part I. LNCS, vol. 6361, pp. 275–282. Springer, Heidelberg (2010)CrossRefGoogle Scholar
  11. 11.
    Bernhardt, S., Abi-Nahed, J., Abugharbieh, R.: Robust dense endoscopic stereo reconstruction for minimally invasive surgery. In: Menze, B.H., Langs, G., Lu, L., Montillo, A., Tu, Z., Criminisi, A. (eds.) MCV 2012. LNCS, vol. 7766, pp. 254–262. Springer, Heidelberg (2013)CrossRefGoogle Scholar
  12. 12.
    Röhl, S., Bodenstedt, S., Suwelack, S., Kenngott, H., Müller-Stich, B., Dillmann, R., Speidel, S.: Real-time surface reconstruction from stereo endoscopic images for intraoperative registration. In: Proc. SPIE, vol. 7964, p. 796414 (2011)Google Scholar
  13. 13.
    Klein, S., Staring, M., Murphy, K., Viergever, M., Pluim, J.: Elastix: A toolbox for intensity based medical image registration. IEEE Transactions on Medical Imaging 29, 196–205 (2010)CrossRefGoogle Scholar
  14. 14.
    Huber, J.S., Peng, Q., Moses, W.W.: Multi-modality phantom development. IEEE Transactions on Nuclear Science 56(5), 2722–2727 (2009)CrossRefGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2013

Authors and Affiliations

  • Alborz Amir-Khalili
    • 1
  • Masoud S. Nosrati
    • 2
  • Jean-Marc Peyrat
    • 3
  • Ghassan Hamarneh
    • 2
  • Rafeef Abugharbieh
    • 1
  1. 1.University of British ColumbiaVancouverCanada
  2. 2.Simon Fraser UniversityBurnabyCanada
  3. 3.Qatar Robotic Surgery CentreQatar Science & Technology ParkDohaQatar

Personalised recommendations