Advertisement

Linking Convolutional Neural Networks with Graph Convolutional Networks: Application in Pulmonary Artery-Vein Separation

  • Zhiwei ZhaiEmail author
  • Marius Staring
  • Xuhui Zhou
  • Qiuxia Xie
  • Xiaojuan Xiao
  • M. Els Bakker
  • Lucia J. Kroft
  • Boudewijn P. F. Lelieveldt
  • Gudula J. A. M. Boon
  • Frederikus A. Klok
  • Berend C. Stoel
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11849)

Abstract

Graph Convolutional Networks (GCNs) are a novel and powerful method for dealing with non-Euclidean data, while Convolutional Neural Networks (CNNs) can learn features from Euclidean data such as images. In this work, we propose a novel method to combine CNNs with GCNs (CNN-GCN), that can consider both Euclidean and non-Euclidean features and can be trained end-to-end. We applied this method to separate the pulmonary vascular trees into arteries and veins (A/V). Chest CT scans were pre-processed by vessel segmentation and skeletonization, from which a graph was constructed: voxels on the skeletons resulting in a vertex set and their connections in an adjacency matrix. 3D patches centered around each vertex were extracted from the CT scans, oriented perpendicularly to the vessel. The proposed CNN-GCN classifier was trained and applied on the constructed vessel graphs, where each node is then labeled as artery or vein. The proposed method was trained and validated on data from one hospital (11 patient, 22 lungs), and tested on independent data from a different hospital (10 patients, 10 lungs). A baseline CNN method and human observer performance were used for comparison. The CNN-GCN method obtained a median accuracy of 0.773 (0.738) in the validation (test) set, compared to a median accuracy of 0.817 by the observers, and 0.727 (0.693) by the CNN. In conclusion, the proposed CNN-GCN method combines local image information with graph connectivity information, improving pulmonary A/V separation over a baseline CNN method, approaching the performance of human observers.

Notes

Acknowledgements

This research was supported by the China Scholarship Council (No. 201406120046) and supported in part by a research grant from the Investigator-Initiated Studies Program of Merck Sharp & Dohme Limited and Bayer AG. The opinions expressed in this paper are those of the authors and do not necessarily represent those of Merck Sharp & Dohme Limited or Bayer AG.

References

  1. 1.
    Kipf, T.N., Welling, M.: Semi-supervised classification with graph convolutional networks. arXiv preprint arXiv:1609.02907 (2016)
  2. 2.
    Wu, F., Zhang, T., et al.: Simplifying Graph Convolutional Networks. arXiv preprint arXiv:1902.07153 (2019)
  3. 3.
    Monti, F., Boscaini, D., et al.: Geometric deep learning on graphs and manifolds using mixture model CNNs. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 5115–5124 (2017)Google Scholar
  4. 4.
    Parisot, S., Ktena, S.I., et al.: Disease prediction using graph convolutional networks: application to Autism Spectrum Disorder and Alzheimer’s disease. Med. Image Anal. 48, 117–130 (2018)CrossRefGoogle Scholar
  5. 5.
    Shin, S.Y., Lee, S., et al.: Deep vessel segmentation by learning graphical connectivity. arXiv preprint arXiv:1806.02279 (2018)
  6. 6.
    Charbonnier, J.P., Brink, M., et al.: Automatic pulmonary artery-vein separation and classification in computed tomography using tree partitioning and peripheral vessel matching. IEEE Trans. Med. Imaging 35(3), 882–892 (2015)CrossRefGoogle Scholar
  7. 7.
    Payer, C., Pienn, M., et al.: Automated integer programming based separation of arteries and veins from thoracic CT images. Med. Image Anal. 34, 109–122 (2016)CrossRefGoogle Scholar
  8. 8.
    Nardelli, P., Jimenez-Carretero, D., et al.: Pulmonary artery-vein classification in CT images using deep learning. IEEE Trans. Med. Imaging 37(11), 2428–2440 (2018)CrossRefGoogle Scholar
  9. 9.
    Zhai, Z., Staring, M., et al.: Automatic quantitative analysis of pulmonary vascular morphology in CT images [published online ahead of print June 18, 2019; Medical Physics (2019).  https://doi.org/10.1002/mp.13659]CrossRefGoogle Scholar
  10. 10.
    Selle, D., Preim, B., et al.: Analysis of vasculature for liver surgical planning. IEEE Trans. Med. Imaging 21(11), 1344–1357 (2002)CrossRefGoogle Scholar

Copyright information

© Springer Nature Switzerland AG 2019

Authors and Affiliations

  • Zhiwei Zhai
    • 1
    Email author
  • Marius Staring
    • 1
  • Xuhui Zhou
    • 3
  • Qiuxia Xie
    • 3
  • Xiaojuan Xiao
    • 3
  • M. Els Bakker
    • 1
  • Lucia J. Kroft
    • 1
  • Boudewijn P. F. Lelieveldt
    • 1
  • Gudula J. A. M. Boon
    • 2
  • Frederikus A. Klok
    • 2
  • Berend C. Stoel
    • 1
  1. 1.Department of RadiologyLeiden University Medical CenterLeidenThe Netherlands
  2. 2.Department of Thrombosis and HemostasisLeiden University Medical CenterLeidenThe Netherlands
  3. 3.Department of RadiologySun Yat-sen UniversityShenzhenChina

Personalised recommendations