Dual-robot ultrasound-guided needle placement: closing the planning-imaging-action loop

  • Risto KojcevEmail author
  • Bernhard Fuerst
  • Oliver Zettinig
  • Javad Fotouhi
  • Sing Chun Lee
  • Benjamin Frisch
  • Russell Taylor
  • Edoardo Sinibaldi
  • Nassir Navab
Original Article



Precise needle placement is an important task during several medical procedures. Ultrasound imaging is often used to guide the needle toward the target region in soft tissue. This task remains challenging due to the user’s dependence on image quality, limited field of view, moving target, and moving needle. In this paper, we present a novel dual-robot framework for robotic needle insertions under robotic ultrasound guidance.


We integrated force-controlled ultrasound image acquisition, registration of preoperative and intraoperative images, vision-based robot control, and target localization, in combination with a novel needle tracking algorithm. The framework allows robotic needle insertion to target a preoperatively defined region of interest while enabling real-time visualization and adaptive trajectory planning to provide safe and quick interactions. We assessed the framework by considering both static and moving targets embedded in water and tissue-mimicking gelatin.


The presented dual-robot tracking algorithms allow for accurate needle placement, namely to target the region of interest with an error around 1 mm.


To the best of our knowledge, we show the first use of two independent robots, one for imaging, the other for needle insertion, that are simultaneously controlled using image processing algorithms. Experimental results show the feasibility and demonstrate the accuracy and robustness of the process.


Robotic system and software  Software architecture Robotics Robotic architecture and devices Ultrasound Instrument and patient localization and tracking 



The authors wish to thank Wolfgang Wein and his team (ImFusion GmbH, Munich, Germany) for the great support and opportunity to use the ImFusion framework. This work was partially funded by the Bayerische Forschungsstiftung award number AZ-1072-13 (project RoBildOR).

Compliance with ethical standards

Conflict of interest

The authors declare that they have no conflict of interest.

Human and animal rights

This article does not contain any studies with human participants or animals performed by any of the authors. This article does not contain patient data.

Supplementary material

Supplementary material 1 (mp4 12286 KB)


  1. 1.
    Abolhassani N, Patel R, Moallem M (2007) Needle insertion into soft tissue: a survey. Med Eng Phys 29(4):413–431CrossRefPubMedGoogle Scholar
  2. 2.
    Curley SA, Izzo F, Delrio P, Ellis LM, Granchi J, Vallone P, Fiore F, Pignata S, Daniele B, Cremona F (1999) Radiofrequency ablation of unresectable primary and metastatic hepatic malignancies: results in 123 patients. Ann Surg 230(1):1CrossRefPubMedPubMedCentralGoogle Scholar
  3. 3.
    Yılmaz S, Özdoğan M, Cevener M, Ozluk A, Kargi A, Kendiroglu F, Ogretmen I, Yildiz A (2016) Use of cryoablation beyond the prostate, Insights into imaging, pp. 1–10Google Scholar
  4. 4.
    Taylor RH (2006) A perspective on medical robotics. Proc IEEE 94(9):1652–1664CrossRefGoogle Scholar
  5. 5.
    Esposito M, Busam B, Hennersperger C, Rackerseder J, Lu A, Navab N, Frisch B (2015) Cooperative robotic gamma imaging: enhancing us-guided needle biopsy, in Medical Image Computing and Computer-Assisted Intervention-MICCAI. Springer, BerlinGoogle Scholar
  6. 6.
    Graumann C, Fuerst B, Hennersperger C, Bork F, Navab N (2016) Robotic ultrasound trajectory planning for volume of interest coverage. In: IEEE international conference on robotics and automation (ICRA)Google Scholar
  7. 7.
    Zettinig O, Fuerst B, Kojcev R, Esposito M, Salehi M, Wein W, Rackerseder J, Sinibaldi E, Frisch B, Navab N (2016) Toward real-time 3d ultrasound registration-based visual servoing for interventional navigation. In: IEEE international conference on robotics and automation (ICRA)Google Scholar
  8. 8.
    Hong J, Dohi T, Hashizume M, Konishi K, Hata N (2004) An ultrasound-driven needle-insertion robot for percutaneous cholecystostomy. Phys Med Biol 49(3):441CrossRefPubMedGoogle Scholar
  9. 9.
    Wei Z, Ding M, Downey D, Fenster A (2005) 3d trus guided robot assisted prostate brachytherapy. In Medical image computing and computer-assisted intervention-MICCAI. Springer, Berlin, pp 17–24Google Scholar
  10. 10.
    DiMaio SP, Salcudean S (2005) Needle steering and motion planning in soft tissues. IEEE Trans Biomed Eng 52(6):965–974CrossRefPubMedGoogle Scholar
  11. 11.
    Alterovitz R, Goldberg K, Okamura A (2005) Planning for steerable bevel-tip needle insertion through 2d soft tissue with obstacles, in Robotics and Automation, 2005. Proceedings of the IEEE international conference on IEEE, pp. 1640–1645Google Scholar
  12. 12.
    Abayazid M, Vrooijink GJ, Patil S, Alterovitz R, Misra S (2014) Experimental evaluation of ultrasound-guided 3D needle steering in biological tissue. Int J Comput Assist Radiol Surg 9(6):931–939CrossRefPubMedPubMedCentralGoogle Scholar
  13. 13.
    Niccolini M, Castelli V, Diversi C, Kang B, Mussa F, Sinibaldi E (2016) Development and preliminary assessment of a robotic platform for neuroendoscopy based on a lightweight robot. Int J Med Robot Comput Assist Surg 12(1):4–17CrossRefGoogle Scholar
  14. 14.
    Shiu YC, Ahmad S (1989) Calibration of wrist-mounted robotic sensors by solving homogeneous transform equations of the form AX=XB. IEEE Trans Robot Autom 5(1):16–29CrossRefGoogle Scholar
  15. 15.
    Zhang Z (2000) A flexible new technique for camera calibration. IEEE Trans Pattern Anal Mach Intell 22(11):1330–1334CrossRefGoogle Scholar
  16. 16.
    Mercier L, Langø T, Lindseth F, Collins LD (2005) A review of calibration techniques for freehand 3-d ultrasound systems. Ultrasound Med Biol 31(2):143–165CrossRefPubMedGoogle Scholar
  17. 17.
    Rusu RB, Marton ZC, Blodow N, Dolha M, Beetz M (2008) Towards 3D point cloud based object maps for household environments. Rob Auton Syst 56(11):927–941CrossRefGoogle Scholar
  18. 18.
    Karamalis A, Wein W, Kutter O, Navab N (2009) Fast hybrid freehand ultrasound volume reconstruction. In: SPIE medical imaging. International society for optics and photonics, pp 726 114–726 114Google Scholar
  19. 19.
    Albu-Schäffer A, Ott C, Frese U, Hirzinger G (2003) Cartesian impedance control of redundant robots: Recent results with the DLR-Light-Weight-arms. In Proceedings on international conference on robotics and automation, vol. 3. IEEE, 2003, pp. 3704–3709Google Scholar
  20. 20.
    Wein W, Ladikos A, Fuerst B, Shah A, Sharma K, Navab N (2013) Global registration of ultrasound to mri using the lc2 metric for enabling neurosurgical guidance. In: Medical image computing and computer-assisted intervention–MICCAI. Springer, Berlin, pp. 34–41Google Scholar
  21. 21.
    Fuerst B, Wein W, Müller M, Navab N (2014) Automatic ultrasound-MRI registration for neurosurgery using the 2D and 3D LC2 metric. Med Image Anal 18(8):1312–1319CrossRefPubMedGoogle Scholar
  22. 22.
    Powell MJ (2009) The BOBYQA algorithm for bound constrained optimization without derivativesGoogle Scholar
  23. 23.
    Chatelain P, Krupa A, Marchal M (2013) Real-time needle detection and tracking using a visually servoed 3d ultrasound probe. In Robotics and Automation (ICRA), 2013 IEEE International Conference on IEEE, pp 1676–1681Google Scholar
  24. 24.
    Otsu N (1975) A threshold selection method from gray-level histograms. Automatica 11(285–296):23–27Google Scholar
  25. 25.
    Uherčík M, Liebgott H, Kybic J, Cachard C (2009) Needle localization methods in 3D ultrasound data. In International congress on ultrasonics, pp. 11–17Google Scholar
  26. 26.
    Julier s, Uhlmann J (2004) Unscented filtering and nonlinear estimation. In Proceedings of the IEEE, vol 92, no 3, pp. 401–422Google Scholar
  27. 27.
    Bischoff R, Kurth J, Schreiber G, Koeppe R, Albu-Schaeffer A, Beyer A, Eiberger O, Haddadin S, Stemmer A, Grunwald G, Hirzinger G (2010) The kuka-dlr lightweight robot arm—a new reference platform for robotics research and manufacturing, In Robotics (ISR), 2010 41st International Symposium on and 2010 6th German Conference on Robotics (ROBOTIK), June 2010, pp. 1–8Google Scholar
  28. 28.
    Schreiber G, Stemmer A, Bischoff R (2010) The fast research interface for the KUKA lightweight robot. In: IEEE workshop on innovative robot control architectures for demanding applications how to modify and enhance commercial controllersGoogle Scholar
  29. 29.
    Lasso A, Heffter T, Rankin A, Pinter C, Ungi T, Fichtinger G (2014) Plus: Open-source toolkit for ultrasound-guided intervention systems. IEEE Trans Biomed Eng 10:2527–2537CrossRefGoogle Scholar
  30. 30.
    Cook JR, Bouchard RR, Emelianov SY (2011) Tissue-mimicking phantoms for photoacoustic and ultrasonic imaging. Biomed Opt Express 2(11):3193–3206CrossRefPubMedPubMedCentralGoogle Scholar
  31. 31.
    Hall TJ, Bilgen M, Insana MF, Krouskop TA (1997) Phantom materials for elastography. IEEE Trans Ultrason Ferroelectr Freq Control 44(6):1355–1365CrossRefGoogle Scholar
  32. 32.
    Künzli BM, Abitabile P, Maurer CA (2011) Radiofrequency ablation of liver tumors: actual limitations and potential solutions in the future. World J Hepatol 3(1):8–14PubMedPubMedCentralGoogle Scholar
  33. 33.
    Bhan SN, Pautler SE, Shayegan B, Voss MD, Goeree RA, You JJ (2013) Active surveillance, radiofrequency ablation, or cryoablation for the nonsurgical management of a small renal mass: a cost-utility analysis. Ann Surg Oncol 20(11):3675–3684CrossRefPubMedGoogle Scholar

Copyright information

© CARS 2016

Authors and Affiliations

  • Risto Kojcev
    • 1
    • 2
    Email author
  • Bernhard Fuerst
    • 1
    • 3
  • Oliver Zettinig
    • 3
  • Javad Fotouhi
    • 1
  • Sing Chun Lee
    • 1
  • Benjamin Frisch
    • 3
  • Russell Taylor
    • 4
  • Edoardo Sinibaldi
    • 2
  • Nassir Navab
    • 1
    • 3
  1. 1.Computer Aided Medical ProceduresJohns Hopkins UniversityBaltimoreUSA
  2. 2.Center for Micro-BioRoboticsIstituto Italiano di TecnologiaPontederaItaly
  3. 3.Computer Aided Medical ProceduresTechnische Universität MünchenMunichGermany
  4. 4.Laboratory of Computational Sensing and RoboticsJohns Hopkins UniversityBaltimoreUSA

Personalised recommendations