Transferring from ex-vivo to in-vivo: Instrument Localization in 3D Cardiac Ultrasound Using Pyramid-UNet with Hybrid Loss

  • Hongxu YangEmail author
  • Caifeng Shan
  • Tao Tan
  • Alexander F. Kolen
  • Peter H. N. de With
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11768)


Automated instrument localization during cardiac interventions is essential to accurately and efficiently interpret a 3D ultrasound (US) image. In this paper, we propose a method to automatically localize the cardiac intervention instrument (RF-ablation catheter or guidewire) in a 3D US volume. We propose a Pyramid-UNet, which exploits the multi-scale information for better segmentation performance. Furthermore, a hybrid loss function is introduced, which consists of contextual loss and class-balanced focal loss, to enhance the performance of the network in cardiac US images. We have collected a challenging ex-vivo dataset to validate our method, which achieves a Dice score of 69.6% being 18.8% higher than the state-of-the-art methods. Moreover, with the pre-trained model on the ex-vivo dataset, our method can be easily adapted to the in-vivo dataset with several iterations and then achieves a Dice score of 65.8% for a different instrument. With segmentation, instruments can be localized with an average error less than 3 voxels in both datasets. To the best of our knowledge, this is the first work to validate the image-based method on in-vivo cardiac datasets.


Instrument localization 3D US Pyramid-UNet Hybrid loss 


  1. 1.
    Arif, M., Moelker, A., van Walsum, T.: Automatic needle detection and real-time bi-planar needle visualization during 3D ultrasound scanning of the liver. Med. Image Anal. 53, 104–110 (2019)CrossRefGoogle Scholar
  2. 2.
    Çiçek, Ö., Abdulkadir, A., Lienkamp, S.S., Brox, T., Ronneberger, O.: 3D U-Net: learning dense volumetric segmentation from sparse annotation. In: Ourselin, S., Joskowicz, L., Sabuncu, M.R., Unal, G., Wells, W. (eds.) MICCAI 2016. LNCS, vol. 9901, pp. 424–432. Springer, Cham (2016). Scholar
  3. 3.
    Dou, Q., Chen, H., Jin, Y., Yu, L., Qin, J., Heng, P.-A.: 3D deeply supervised network for automatic liver segmentation from CT volumes. In: Ourselin, S., Joskowicz, L., Sabuncu, M.R., Unal, G., Wells, W. (eds.) MICCAI 2016. LNCS, vol. 9901, pp. 149–157. Springer, Cham (2016). Scholar
  4. 4.
    Lin, T.Y., Dollár, P., Girshick, R., He, K., Hariharan, B., Belongie, S.: Feature pyramid networks for object detection. In: IEEE CVPR, pp. 2117–2125 (2017)Google Scholar
  5. 5.
    Pourtaherian, A., et al.: Medical instrument detection in 3-dimensional ultrasound data volumes. IEEE Trans. Med. Imaging 36(8), 1664–1675 (2017)CrossRefGoogle Scholar
  6. 6.
    Pourtaherian, A., Zanjani, F.G., Zinger, S., Mihajlovic, N., Ng, G.C., Korsten, H.H., et al.: Robust and semantic needle detection in 3D ultrasound using orthogonal-plane convolutional neural networks. IJCARS 13(9), 1321–1333 (2018)Google Scholar
  7. 7.
    Wong, K.C.L., Moradi, M., Tang, H., Syeda-Mahmood, T.: 3D segmentation with exponential logarithmic loss for highly unbalanced object sizes. In: Frangi, A.F., Schnabel, J.A., Davatzikos, C., Alberola-López, C., Fichtinger, G. (eds.) MICCAI 2018. LNCS, vol. 11072, pp. 612–619. Springer, Cham (2018). Scholar
  8. 8.
    Yang, H., Shan, C., Kolen, A.F., de With, P.H.: Catheter detection in 3D ultrasound using triplanar-based convolutional neural networks. In: IEEE ICIP, pp. 371–375. IEEE (2018)Google Scholar
  9. 9.
    Yang, H., Shan, C., Pourtaherian, A., Kolen, A.F., et al.: Catheter segmentation in three-dimensional ultrasound images by feature fusion and model fitting. J. Med. Imaging 6(1), 015001 (2019)CrossRefGoogle Scholar
  10. 10.
    Yang, X., et al.: Towards automatic semantic segmentation in volumetric ultrasound. In: Descoteaux, M., Maier-Hein, L., Franz, A., Jannin, P., Collins, D.L., Duchesne, S. (eds.) MICCAI 2017. LNCS, vol. 10433, pp. 711–719. Springer, Cham (2017). Scholar

Copyright information

© Springer Nature Switzerland AG 2019

Authors and Affiliations

  • Hongxu Yang
    • 1
    Email author
  • Caifeng Shan
    • 2
  • Tao Tan
    • 1
  • Alexander F. Kolen
    • 2
  • Peter H. N. de With
    • 1
  1. 1.Eindhoven University of TechnologyEindhovenThe Netherlands
  2. 2.Philips ResearchEindhovenThe Netherlands

Personalised recommendations