Multimodal US–gamma imaging using collaborative robotics for cancer staging biopsies

  • Marco EspositoEmail author
  • Benjamin Busam
  • Christoph Hennersperger
  • Julia Rackerseder
  • Nassir Navab
  • Benjamin Frisch
Original Article



The staging of female breast cancer requires detailed information about the level of cancer spread through the lymphatic system. Common practice to obtain this information for patients with early-stage cancer is sentinel lymph node (SLN) biopsy, where LNs are radioactively identified for surgical removal and subsequent histological analysis. Punch needle biopsy is a less invasive approach but suffers from the lack of combined anatomical and nuclear information. We present and evaluate a system that introduces live collaborative robotic 2D gamma imaging in addition to live 2D ultrasound to identify SLNs in the surrounding anatomy.


The system consists of a robotic arm equipped with both a gamma camera and a stereoscopic tracking system that monitors the position of an ultrasound probe operated by the physician. The arm cooperatively places the gamma camera parallel to the ultrasound imaging plane to provide live multimodal visualization and guidance. We validate the system by evaluating the target registration errors between fused nuclear and US image data in a phantom consisting of two spheres, one of which is filled with radioactivity. Medical experts perform punch biopsies on agar–gelatine phantoms with complex configurations of hot and cold lesions to provide a qualitative and quantitative evaluation of the system.


The average point registration error for the overlay is \(1.12 \pm 0.57\) mm. The time of the entire procedure was reduced by 36 %, with 80v of the biopsies being successful. The users’ feedback was very positive, and the system was deemed to be very intuitive, with handling similar to classic US-guided needle biopsy.


We present and evaluate the first medical collaborative robotic imaging system. Feedback from potential users for SLN punch needle biopsy is encouraging. Ongoing work investigates the clinical feasibility with more complex and realistic phantoms.


Medical robotics Interventional imaging Multimodal imaging Breast cancer Inside-out tracking 


Compliance with ethical standards

Ethical disclosure

This article does not contain any studies with human participants or animals performed by any of the authors. This article does not contain patient data.


This work was partially funded by the Bayerische Forschungsstiftung award number AZ-1072-13 (project RoBildOR).

Conflict of interest

Benjamin Busam is an employee of FRAMOS GmbH, Taufkirchen, Germany. The other authors declare no conflict of interest.

Supplementary material

Supplementary material 1 (mp4 38708 KB)


  1. 1.
    Bradski G (2000) Opencv. Dr. Dobb’s journal of software toolsGoogle Scholar
  2. 2.
    Bricou A, Duval MA, Charon Y, Barranger E (2013) Mobile gamma cameras in breast cancer care—a review. Eur J Surg Oncol 39(5):409–416CrossRefPubMedGoogle Scholar
  3. 3.
    Busam B, Esposito M, Che’Rose S, Navab N, Frisch B (2015) A stereo vision approach for cooperative robotic movement therapy. In: IEEE international conference on computer vision workshop (ICCVW)Google Scholar
  4. 4.
    Dang J, Frisch B, Lasaygues P, Zhang D, Tavernier S, Felix N, Lecoq P, Auffray E, Varela J, Mensah S, Wan M (2011) Development of an anthropomorphic breast phantom for combined PET, B-mode ultrasound and elastographic imaging. IEEE Trans Nucl Sci 58(3):660–667CrossRefGoogle Scholar
  5. 5.
    Esposito M, Busam B, Hennersperger C, Rackerseder J, Lu A, Navab N, Frisch B (2015) Cooperative robotic gamma imaging: enhancing US-guided needle biopsy. In: Navab N, Hornegger J, Wells WM, Frangi AF (eds) Medical image computing and computer-assisted intervention–MICCAI 2015, Lecture Notes in Computer Science, vol 9350. Springer, Berlin, pp 611–618Google Scholar
  6. 6.
    Foote T, Marder-Eppstein E, Meeussen W (2015) Ros tf package. Accessed 06 June 2016
  7. 7.
    Gardiazabal J, Reichl T, Okur A, Lasser T, Navab N (2013) First flexible robotic intra-operative nuclear imaging for image-guided surgery. In: Information processing in computer-assisted interventionsGoogle Scholar
  8. 8.
    Giammarile F, Alazraki N, Aarsvold JN, Audisio RA, Glass E, Grant SF, Kunikowska J, Leidenius M, Moncayo VM, Uren RF, Oyen WJG, Valdés Olmos RA, Vidal Sicart S (2013) The EANM and SNMMI practice guideline for lymphoscintigraphy and sentinel node localization in breast cancer. Europ J Nucl Med Mol Imaging 40(12):1932–1947CrossRefGoogle Scholar
  9. 9.
    Hermann A, Mauch F, Fischnaller K, Klemm S, Ronnau A, Dillmann R (2015) Anticipate your surroundings: predictive collision detection between dynamic obstacles and planned robot trajectories on the GPU. In: IEEE European conference on mobile robots (ECMR), pp 1–8Google Scholar
  10. 10.
    Kohler D, Conley K (2011) rosjava–an implementation of ROS in pure Java with Android support. Accessed 28 Jan 2016
  11. 11.
    Krag DN, Anderson SJ, Julian TB, Brown AM, Harlow SP, Costantino JP, Ashikaga T, Weaver DL, Mamounas EP, Jalovec LM, Frazier TG, Noyes RD, Robidoux A, Scarth HMC, Wolmark N (2010) Sentinel-lymph-node resection compared with conventional axillary-lymph-node dissection in clinically node-negative patients with breast cancer: overall survival findings from the NSABP B-32 randomised phase 3 trial. Lancet Oncol 11(10):927–933CrossRefPubMedPubMedCentralGoogle Scholar
  12. 12.
    KUKA GmbH (2016) Kuka lwr iiwa. Accessed 28 Jan 2016
  13. 13.
    KUKA GmbH (2016) KUKA sunrise. Accessed 28 Jan 2016
  14. 14.
    Lasso A, Heffter T, Rankin A, Pinter C, Ungi T, Fichtinger G (2014) Plus: open-source toolkit for ultrasound-guided intervention systems. IEEE Trans Biomed Eng 61(10):2527–2537CrossRefPubMedPubMedCentralGoogle Scholar
  15. 15.
    Lyman G, Giuliano A, Somerfield M, Benson A, Bodurka D, Burstein H, Cochran A, Cody H, Edge S, Galper S, Hayman J, Kim T, Perkins C, Podoloff D, Sivasubramaniam V, Turner R, Wahl R, Weaver D, Wolff A, Winer E (2005) American Society of Clinical Oncology guideline recommendations for sentinel lymph node biopsy in early-stage breast cancer. J Clin Oncol 23(30):7703–7720CrossRefPubMedGoogle Scholar
  16. 16.
    Lyman GH, Temin S, Edge SB, Newman LA, Turner RR, Weaver DL, Benson AB, Bosserman LD, Burstein HJ, Cody H, Hayman J, Perkins CL, Podoloff DA, Giuliano AE (2014) Sentinel lymph node biopsy for patients with early-stage breast cancer: American Society of Clinical Oncology clinical practice guideline update. J Clin Oncol 32(13):1365–1383CrossRefPubMedGoogle Scholar
  17. 17.
    Marchand É, Spindler F, Chaumette F (2005) ViSP for visual servoing: a generic software platform with a wide class of robot control skills. IEEE Robot Autom Mag 12(4):40–52CrossRefGoogle Scholar
  18. 18.
    Matthies P, Gardiazabal J, Okur A, Vogel J, Lasser T, Navab N (2014) Mini gamma cameras for intra-operative nuclear tomographic reconstruction. Med Image Anal 18(8):1329–1336CrossRefPubMedGoogle Scholar
  19. 19.
    Okur A, Hennersperger C, Runyan B, Gardiazabal J, Keicher M, Paepke S, Wendler T, Navab N (2014) fhSPECT-US guided needle biopsy of sentinel lymph nodes in the axilla: is it feasible? In: Golland P, Hata N, Barillot C, Hornegger J, Howe R (eds) Medical image computing and computer-assisted intervention–MICCAI 2014. Lecture notes in computer science, vol 8673. Springer, Heidelberg, pp 577–584Google Scholar
  20. 20.
    Padoy N, Hager G (2011) Human-machine collaborative surgery using learned models. In: 2011 IEEE international conference on robotics and automation (ICRA), pp 5285–5292Google Scholar
  21. 21.
    Quigley M, Conley K, Gerkey B, Faust J, Foote T, Leibs J, Wheeler R, Ng AY (2009) ROS: an open-source robot operating system. In: ICRA workshop on open source software, vol 3, pp 5Google Scholar
  22. 22.
    Rivas-Blanco I, Estebanez B, Cuevas-Rodriguez M, Bauzano E, Munoz V (2014) Towards a cognitive camera robotic assistant. In: 5th IEEE international conference on biomedical robotics and biomechatronics, pp 739–744Google Scholar
  23. 23.
    Senkus E, Kyriakides S, Ohno S, Penault-Llorca F, Poortmans P, Rutgers E, Zackrisson S, Cardoso F (2015) Primary breast cancer: ESMO Clinical Practice Guidelines for diagnosis, treatment and follow-up. Ann Oncol 26(suppl 5):v8–v30CrossRefPubMedGoogle Scholar
  24. 24.
    Siegel RL, Miller KD, Jemal A (2015) Cancer statistics, 2015. CA Cancer J Clin 65(1):5–29CrossRefPubMedGoogle Scholar
  25. 25.
    Slagelse C, Petersen KL, Dahl JB, Finnerup K, Greene K, Leong SP, Levine J, Rowbotham M, Werner MU, Finnerup NB (2014) Persistent postoperative pain and sensory changes following lymph node excision in melanoma patients: a topical review. Melanoma Res 24(2):93–98CrossRefPubMedGoogle Scholar
  26. 26.
    Swenson KK, Nissen MJ, Ceronsky C, Swenson L, Lee MW, Tuttle TM (2002) Comparison of side effects between sentinel lymph node and axillary lymph node dissection for breast cancer. Ann Surg Oncol 9(8):745–753CrossRefPubMedGoogle Scholar
  27. 27.
    Taylor RH (2006) A perspective on medical robotics. Proc IEEE 94(9):1652–1664CrossRefGoogle Scholar
  28. 28.
    Tokuda J, Fischer GS, Papademetris X, Yaniv Z, Ibanez L, Cheng P, Liu H, Blevins J, Arata J, Golby AJ, Kapur T, Pieper S, Burdette EC, Fichtinger G, Tempany CM, Hata N (2009) OpenIGTLink: an open network protocol for image-guided therapy environment. Int J Med Robot 5(4):423–434Google Scholar
  29. 29.
    Tsai R, Lenz R (1989) A new technique for fully autonomous and efficient 3D robotics hand/eye calibration. IEEE Trans Robot Autom 5(3):345–358CrossRefGoogle Scholar
  30. 30.
    Vidal-Sicart S, Valdés Olmos R (2012) Sentinel node mapping for breast cancer: current situation. J Oncol 2012:361341. doi: 10.1155/2012/361341 CrossRefPubMedPubMedCentralGoogle Scholar
  31. 31.
    Virga S (2016) ROS iiwa stack. Accessed 28 Jan 2016
  32. 32.
    Wendler T, Feuerstein M, Traub J, Lasser T, Vogel J, Daghighian F, Ziegler S, Navab N (2007) Real-time fusion of ultrasound and gamma probe for navigated localization of liver metastases. In: Ayache N, Ourselin S, Maeder A (eds) Medical image computing and computer-assisted intervention MICCAI 2007, Lecture Notes in Computer Science, vol 4792. Springer, Berlin, pp 252–260Google Scholar
  33. 33.
    Wendler T, Herrmann K, Schnelzer A, Lasser T, Traub J, Kutter O, Ehlerding A, Scheidhauer K, Schuster T, Kiechle M, Schwaiger M, Navab N, Ziegler SI, Buck AK (2010) First demonstration of 3-D lymphatic mapping in breast cancer using freehand SPECT. Eur J Nucl Med 37(8):1452–1461CrossRefGoogle Scholar
  34. 34.
    Wilke LG, McCall LM, Posther KE, Whitworth PW, Reintgen DS, Leitch AM, Gabram SGA, Lucci A, Cox CE, Hunt KK, Herndon JE, Giuliano AE (2006) Surgical complications associated with sentinel lymph node biopsy: results from a prospective international cooperative group trial. Ann Surg Oncol 13(4):491–500. doi: 10.1245/ASO.2006.05.013 CrossRefPubMedGoogle Scholar
  35. 35.
    Zhang Z (2000) A flexible new technique for camera calibration. IEEE Trans Pattern Anal Mach Intell 22(11):1330–1334CrossRefGoogle Scholar
  36. 36.
    Zu Berge CS, Grunau A, Mahmud H, Navab N (2014) CAMPVis—a game engine-inspired research framework for medical imaging and visualization. Technical report, Technische Universität München.

Copyright information

© CARS 2016

Authors and Affiliations

  • Marco Esposito
    • 1
    Email author
  • Benjamin Busam
    • 1
    • 2
  • Christoph Hennersperger
    • 1
  • Julia Rackerseder
    • 1
  • Nassir Navab
    • 1
    • 3
  • Benjamin Frisch
    • 1
  1. 1.Computer Aided Medical Procedures (CAMP)Technische Universität MünchenMunichGermany
  2. 2.FRAMOS GmbHTaufkirchenGermany
  3. 3.Computer Aided Medical ProceduresJohns Hopkins UniversityBaltimoreUS

Personalised recommendations