Mobile augmented reality for computer-assisted percutaneous nephrolithotomy

  • Michael Müller
  • Marie-Claire Rassweiler
  • Jan Klein
  • Alexander Seitel
  • Matthias Gondan
  • Matthias Baumhauer
  • Dogu Teber
  • Jens J. Rassweiler
  • Hans-Peter Meinzer
  • Lena Maier-Hein
Original Article



Percutaneous nephrolithotomy (PCNL) plays an integral role in treatment of renal stones. Creating percutaneous renal access is the most important and challenging step in the procedure. To facilitate this step, we evaluated our novel mobile augmented reality (AR) system for its feasibility of use for PCNL.


A tablet computer, such as an iPad\(^{\circledR }\), is positioned above the patient with its camera pointing toward the field of intervention. The images of the tablet camera are registered with the CT image by means of fiducial markers. Structures of interest can be superimposed semi-transparently on the video images. We present a systematic evaluation by means of a phantom study. An urological trainee and two experts conducted 53 punctures on kidney phantoms.


The trainee performed best with the proposed AR system in terms of puncturing time (mean: 99 s), whereas the experts performed best with fluoroscopy (mean: 59 s). iPad assistance lowered radiation exposure by a factor of 3 for the inexperienced physician and by a factor of 1.8 for the experts in comparison with fluoroscopy usage. We achieve a mean visualization accuracy of 2.5 mm.


The proposed tablet computer-based AR system has proven helpful in assisting percutaneous interventions such as PCNL and shows benefits compared to other state-of-the-art assistance systems. A drawback of the system in its current state is the lack of depth information. Despite that, the simple integration into the clinical workflow highlights the potential impact of this approach to such interventions.


Augmented reality Mobile Image-guided surgery  Computer vision CT 



The presented work was conducted within the setting of the “Research group 1126: Intelligent Surgery-Development of new computer-based methods for the future workplace in surgery” funded by the German Research Foundation (DFG). Furthermore, we want to thank the staff in the urological departments of our partner hospitals for the support during our experiments. The presented software was developed as part of the Medical Imaging Interaction Toolkit (MITK,

Conflict of Interest


Supplementary material

11548_2013_828_MOESM1_ESM.mpeg (17.7 mb)
Supplementary material 1 (mpeg 18092 KB)


  1. 1.
    Romero V, Akpinar H, Assimos DG (2010) Kidney stones: a global picture of prevalence, incidence, and associated risk factors. Rev Urol 12(2–3):86–96Google Scholar
  2. 2.
    Skolarikos A, Alivizatos G, de la Rosette JJ (2005) Percutaneous nephrolithotomy and its legacy. Eur Urol 47(1):22–28PubMedCrossRefGoogle Scholar
  3. 3.
    Michel MS, Trojan L, Rassweiler JJ (2007) Complications in percutaneous nephrolithotomy. Eur Urol 51(4):899–906Google Scholar
  4. 4.
    Andonian S, Scoffone C, Louie MK, Gross AJ, Grabe M, Daels F, Shah HN, De La Rosette J (2012) Does imaging modality used for percutaneous renal access make a difference? a matched case analysis. J Endourol 27(1):24–28Google Scholar
  5. 5.
    Nicolau S, Soler L, Mutter D, Marescaux J (2011) Augmented reality in laparoscopic surgical oncology. Surg Oncol 20(3): 189–201Google Scholar
  6. 6.
    Maier-Hein L, Tekbas A, Seitel A, Pianka F, Müller SA, Satzl S, Schawo S, Radeleff B, Tetzlaff R, Franz AM, Müller-Stich BP, Wolf I, Kauczor H-U, Schmied BM, Meinzer H-P (2008) In vivo accuracy assessment of a needle-based navigation system for CT-guided radiofrequency ablation of the liver. Med Phys 35(12):5385–5396PubMedCrossRefGoogle Scholar
  7. 7.
    Fichtinger G, Deguet A, Fischer G, Iordachita I, Balogh E, Masamune K, Taylor RH, Fayad LM, de Oliveira M, Zinreich SJ (2005) Image overlay for CT-guided needle insertions. Comput Aided Surg 10(4):241–255PubMedGoogle Scholar
  8. 8.
    Song DY, Burdette EC, Fiene J, Armour E, Kronreif G, Deguet A, Zhang Z, Iordachita I, Fichtinger G, Kazanzides P (2011) Robotic needle guide for prostate brachytherapy: clinical testing of feasibility and performance. Brachytherapy 10(1):57–63PubMedCrossRefGoogle Scholar
  9. 9.
    Wein W, Brunke S, Khamene A, Callstrom MR, Navab N (2008) Automatic CT-ultrasound registration for diagnostic imaging and image-guided intervention. Med Image Anal 12(5):577–585PubMedCrossRefGoogle Scholar
  10. 10.
    Wood BJ, Kruecker J, Abi-Jaoudeh N, Locklin JK, Levy E, Xu S, Solbiati L, Kapoor A, Amalou H, Venkatesan AM (2010) Navigation systems for ablation. J Vasc Interv Radiol 21(8 Suppl):257–263CrossRefGoogle Scholar
  11. 11.
    Lazarus J, Williams J (2011) The locator: novel percutaneous nephrolithotomy apparatus to aid collecting system puncture-a preliminary report. J Endourol 25(5):747–750PubMedCrossRefGoogle Scholar
  12. 12.
    Huber J, Wegner I, Meinzer HP, Hallscheidt P, Hadaschik B, Pahernik S, Hohenfellner M (2011) Navigated renal access using electromagnetic tracking: an initial experience. Surg Endosc 25(4):1307–1312PubMedCrossRefGoogle Scholar
  13. 13.
    Ritter M, Rassweiler MC, Hacker A, Michel MS (2012) Laser-guided percutaneous kidney access with the Uro Dyna-CT: first experience of three-dimensional puncture planning with an ex vivo model. World J Urol. 30:1–5Google Scholar
  14. 14.
    Baumhauer M, Simpfendörfer T, Stich BM, Teber D, Gutt C, Rassweiler J, Meinzer H-P, Wolf I (2008) Soft tissue navigation for laparoscopic partial nephrectomy. Int J Comput Assist Radiol Surg 3:307–314CrossRefGoogle Scholar
  15. 15.
    Maier-Hein L, Franz AM, Fangerau M, Schmidt M, Seitel A, Mersmann S, Kilgus T, Groch A, Yung K, dos Santos TR, Meinzer H-P (2011) Towards mobile augmented reality for on-patient visualization of medical images. In: Handels H, Ehrhardt J, Deserno TM, Meinzer H-P, Tolxdorff T (eds) Bildverarbeitung für die Medizin. Springer, Berlin, pp 389–393Google Scholar
  16. 16.
    Rassweiler JJ, Müller M, Fangerau M, Klein J, Goezen AS, Pereira P, Meinzer HP, Teber D (2012) iPad-assisted percutaneous access to the kidney using marker-based navigation: initial clinical experience. Eur Urol 61(3):628–631PubMedCrossRefGoogle Scholar
  17. 17.
    Wolf I, Vetter M, Wegner I, Böttger T, Nolden M, Schöbinger M, Hastenteufel M, Kunert T, Meinzer H-P (2005) The medical imaging interaction toolkit. Med Image Anal 9(6):594–604Google Scholar
  18. 18.
    Fangerau M (2011) Medical imaging interaction toolkit for mobile devices. Master’s thesis, Hochschule Mannheim, University of Applied SciencesGoogle Scholar
  19. 19.
    Müller M, Groch A, Baumhauer M, Maier-Hein L, Teber D, Rassweiler J, Meinzer H-P, Wegner I (2012) Robust and efficient fiducial tracking for augmented reality in HD-laparoscopic video streams. In: DRH III, Wong KH, (eds) SPIE medical imaging 2012: visualization, image-guided procedures, and modeling, vol 8316, No 1. SPIE, p 83161MGoogle Scholar
  20. 20.
    Zhang Z (2000) A flexible new technique for camera calibration. IEEE T Pattern Anal 22:1330–1334CrossRefGoogle Scholar
  21. 21.
    Lowe DG (1991) Fitting parameterized three-dimensional models to images. IEEE Trans Pattern Anal Mach Intell 13:441–450CrossRefGoogle Scholar
  22. 22.
    DeMenthon DF, Davis LS (1995) Model-based object pose in 25 lines of code. Int J Comput Vis 15:123–141CrossRefGoogle Scholar
  23. 23.
    Lu C-P, Hager GD, Mjolsness E (2000) Fast and globally convergent pose estimation from video images. IEEE Trans Pattern Anal Mach Intell 22:610–622CrossRefGoogle Scholar
  24. 24.
    Lepetit V, Moreno-Noguer F, Fua P (2008) EP\(n\)P: an accurate o(\(n\)) solution to the P\(n\)P problem. Int J Comput Vis 81(2):155–166CrossRefGoogle Scholar
  25. 25.
    Sarkis M, Diepold K (2012) Camera-pose estimation via projective Newton optimization on the manifold. IEEE Trans Image Process 21(4):1729–1741PubMedCrossRefGoogle Scholar
  26. 26.
    Li S, Xu C, Xie M (2012) A robust o(n) solution to the perspective-n-point problem. IEEE Trans Pattern Anal Mach Intell. 34(7):1444–1450CrossRefGoogle Scholar
  27. 27.
    Fischler MA, Bolles RC (1981) Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography. Commun ACM 24(6):381–395CrossRefGoogle Scholar
  28. 28.
    Nocedal J, Wright SJ (2006) Numerical optimization, 2nd edn. ser. Springer series in operations research and financial engineering. Springer, New YorkGoogle Scholar
  29. 29.
    Holloway RL (1997) Registration error analysis for augmented reality. Presence Teleoper Virtual Environ 6(4):413–432Google Scholar
  30. 30.
    Baumhauer M (2008) Real-time compensation of organ motion for augmented reality in laparoscopic surgery. Ph.D. dissertation, Ruprecht-Karls University HeidelbergGoogle Scholar
  31. 31.
    Simpfendörfer T, Baumhauer M, Müller M, Gutt CN, Meinzer HP, Rassweiler JJ, Guven S, Teber D (2011) Augmented reality visualization during laparoscopic radical prostatectomy. J Endourol 25:1841–1845 Google Scholar
  32. 32.
    Laganière R (2011) OpenCV 2 computer vision application programming cookbook. Packt Publishing, BirminghamGoogle Scholar
  33. 33.
    Miller NL, Lingeman JE (2007) Management of kidney stones. BMJ 334(7591):468–472PubMedCrossRefGoogle Scholar
  34. 34.
    Ritter M, Siegel F, Krombach P, Martinschek A, Weiss C, Hacker A, Pelzer AE (2012) Influence of surgeon’s experience on fluoroscopy time during endourological interventions. World J Urol 31(1): 183–187Google Scholar
  35. 35.
    Seitel A, Maier-Hein L, Schawo S, Radeleff B, Müller SA, Pianka F, Schmied BM, Wolf I, Meinzer H-P (2007) In-vitro evaluation of different visualization approaches for computer assisted targeting in soft tissue. In: Lemke H, Inamura K, Doi K, Vannier M, Farman A (eds) International journal of Computer Assisted Radiology and Surgery. Berlin (Germany), pp 188–190, June 2007Google Scholar
  36. 36.
    Karami H, Rezaei A, Mohammadhosseini M, Javanmard B, Mazloomfard M, Lotfi B (2010) Ultrasonography-guided percutaneous nephrolithotomy in the flank position versus fluoroscopy-guided percutaneous nephrolithotomy in the prone position: a comparative study. J Endourol 24(8):1357–1361Google Scholar
  37. 37.
    Agarwal M, Agrawal MS, Jaiswal A, Kumar D, Yadav H, Lavania P (2011) Safety and efficacy of ultrasonography as an adjunct to fluoroscopy for renal access in percutaneous nephrolithotomy (PCNL). BJU Int 108(8):1346–1349PubMedCrossRefGoogle Scholar
  38. 38.
    Maier-Hein L, Walsh CJ, Seitel A, Hanumara NC, Shepard J-A, Franz AM, Pianka F, Müller SA, Schmied B, Slocum AH, Gupta R, Meinzer H-P (2009) Human vs. robot operator error in a needle-based navigation system for percutaneous liver interventions. In: SPIE medical imaging 2009: visualization, image-guided procedures, and modelling, vol 7261, p 72610Y (12 p)Google Scholar
  39. 39.
    Franz A, März K, Hummel J, Birkfellner W, Bendl R, Delorme S, Schlemmer H-P, Meinzer H-P, Maier-Hein L (2012) Electromagnetic tracking for us-guided interventions: standardized assessment of a new compact field generator. Int J Comp Assist Radiol Surg 7:1–6Google Scholar
  40. 40.
    Yaniva Z, Wilson E, Lindisch D, Cleary K (2009) Electromagnetic tracking in the clinical environment. Med Phys 36(3):876–892CrossRefGoogle Scholar
  41. 41.
    Lee T, Hollerer T (2009) Multithreaded hybrid feature tracking for markerless augmented reality. IEEE Trans Vis Comput Graph 15(3):355–368PubMedCrossRefGoogle Scholar
  42. 42.
    Grimm R, Bauer S, Sukkau J, Hornegger J, Greiner G (2012) Markerless estimation of patient orientation, posture and pose using range and pressure imaging. Int J Comput Assist Radiol Surg 1:1–9Google Scholar
  43. 43.
    Newcombe RA, Davison AJ, Izadi S, Kohli P, Hilliges O, Shotton J, Molyneaux D, Hodges S, Kim D, Fitzgibbon A (2011) Kinectfusion: real-time dense surface mapping and tracking. In: Mixed and augmented reality (ISMAR), 2011 10th IEEE international symposium on, pp. 127–136, Oct 2011Google Scholar
  44. 44.
    Mersmann S, Gergel I, Seitel A, Gaa J, Wegner I, Meinzer H-P, Maier-Hein L (2011) Microsoft kinect controller as intra-operative imaging modality. Int J Comput Assist Radiol Surg 6(Suppl 1): 251–252Google Scholar
  45. 45.
    Seitel A, Engel M, Sommer CM, Radeleff BA, Essert-Villard C, Baegert C, Fangerau M, Fritzsche KH, Yung K, Meinzer H-P, Maier-Hein L (2011) Computer-assisted trajectory planning for percutaneous needle insertions. Med Phys 38(6):3246–3259PubMedCrossRefGoogle Scholar

Copyright information

© CARS 2013

Authors and Affiliations

  • Michael Müller
    • 1
  • Marie-Claire Rassweiler
    • 2
  • Jan Klein
    • 3
  • Alexander Seitel
    • 1
  • Matthias Gondan
    • 4
  • Matthias Baumhauer
    • 1
  • Dogu Teber
    • 5
  • Jens J. Rassweiler
    • 3
  • Hans-Peter Meinzer
    • 1
  • Lena Maier-Hein
    • 1
  1. 1.Department of Medical and Biological InformaticsGerman Cancer Research Center (DKFZ)HeidelbergGermany
  2. 2.Department of UrologyMannheim University HospitalMannheimGermany
  3. 3.Department of UrologySLK-Kliniken HeilbronnHeilbronnGermany
  4. 4.Institute of Medical Biometry and InformaticsUniversity of HeidelbergHeidelbergGermany
  5. 5.Department of UrologyHeidelberg University HospitalHeidelbergGermany

Personalised recommendations