Skip to main content
SpringerLink
Log in

Transformer-based 3D U-Net for pulmonary vessel segmentation and artery-vein separation from CT images

  • Original Article
  • Published:
Medical & Biological Engineering & Computing Aims and scope Submit manuscript

Abstract

Transformer-based methods have led to the revolutionizing of multiple computer vision tasks. Inspired by this, we propose a transformer-based network with a channel-enhanced attention module to explore contextual and spatial information in non-contrast (NC) and contrast-enhanced (CE) computed tomography (CT) images for pulmonary vessel segmentation and artery-vein separation. Our proposed network employs a 3D contextual transformer module in the encoder and decoder part and a double attention module in skip connection to effectively finish high-quality vessel and artery-vein segmentation. Extensive experiments are conducted on the in-house dataset and the ISICDM2021 challenge dataset. The in-house dataset includes 56 NC CT scans with vessel annotations and the challenge dataset consists of 14 NC and 14 CE CT scans with vessel and artery-vein annotations. For vessel segmentation, Dice is 0.840 for CE CT and 0.867 for NC CT. For artery-vein separation, the proposed method achieves a Dice of 0.758 of CE images and 0.602 of NC images. Quantitative and qualitative results demonstrated that the proposed method achieved high accuracy for pulmonary vessel segmentation and artery-vein separation. It provides useful support for further research associated with the vascular system in CT images. The code is available at https://github.com/wuyanan513/Pulmonary-Vessel-Segmentation-and-Artery-vein-Separation.

Graphical Abstract

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9

Similar content being viewed by others

References

  1. Resten A, Maitre S, Musset D (2005) CT imaging of peripheral pulmonary vessel disease. Eur Radiol 15:2045–2056

    Article  PubMed  Google Scholar 

  2. Lahousse L, Tiemeier H, Ikram MA, Brusselle GG (2015) Chronic obstructive pulmonary disease and cerebrovascular disease: a comprehensive review. Respir Med 109(11):1371–1380

    Article  PubMed  Google Scholar 

  3. Wang M et al (2023) CE-NC-VesselSegNet: supervised by contrast-enhanced CT images but utilized to segment pulmonary vessels from non-contrast-enhanced CT images. Biomed Signal Process Control 82:104565

    Article  Google Scholar 

  4. Hida K, Ohga N, Kurosu T, Totsuka Y, Shindoh M (2009) Crosstalk between blood vessels and tumor microenvironment. Oral Sci Int 7(1):1–10

    Article  Google Scholar 

  5. Coste F et al (2016) CT evaluation of small pulmonary vessels area in patients with COPD with severe pulmonary hypertension. Thorax 71(9):830–837

    Article  PubMed  Google Scholar 

  6. Hoeper MM, Ghofrani H-A, Grünig E, Klose H, Olschewski H, Rosenkranz S (2017) Pulmonary hypertension, Deutsches Ärzteblatt. International 114(5):73

    Google Scholar 

  7. Washko GR, Coxson HO, O’Donnell DE, Aaron SD (2017) CT imaging of chronic obstructive pulmonary disease: insights, disappointments, and promise. The Lancet Respir Med 5:11

    Article  Google Scholar 

  8. Sajja T, Devarapalli R, Kalluri H (2019) Lung cancer detection based on CT scan images by using deep transfer learning. Traitement du Signal 36(4):339–344

    Article  Google Scholar 

  9. Wu R, Xin Y, Qian J, Dong Y (2023) A multi-scale interactive U-Net for pulmonary vessel segmentation method based on transfer learning. Biomed Signal Process Control 80:104407

    Article  Google Scholar 

  10. Poletti J et al (2022) Automated lung vessel segmentation reveals blood vessel volume redistribution in viral pneumonia. Eur J Radiol 150:110259

    Article  PubMed  Google Scholar 

  11. Li X, Bala R, Monga V (2022) Robust deep 3D blood vessel segmentation using structural priors. IEEE Trans Image Process 31:1271–1284

    Article  PubMed  Google Scholar 

  12. Pan L et al (2023) Learning multi-view and centerline topology connectivity information for pulmonary artery-vein separation. Comput Biol Med 155:106669

  13. Gao Y-R, Drew PJ (2014) Determination of vessel cross-sectional area by thresholding in Radon space. J Cereb Blood Flow Metab 34(7):1180–1187

    Article  PubMed  PubMed Central  Google Scholar 

  14. Hojjatoleslami S, Kittler J (1998) Region growing: a new approach. IEEE Trans Image Process 7(7):1079–1084

    Article  CAS  PubMed  Google Scholar 

  15. Duan H-H, Su G-Q, Huang Y-C, Song L-T, Nie S-D (2019) Segmentation of pulmonary vascular tree by incorporating vessel enhancement filter and variational region-growing. J Xray Sci Technol 27(2):343–360

    PubMed  Google Scholar 

  16. Ziou D, Tabbone S (1998) Edge detection techniques-an overview. Pattern Recognition and Image Analysis C/C of Raspoznavaniye Obrazov I Analiz Izobrazhenii 8:537–559

    Google Scholar 

  17. Ozkava U, Ozturk S, Akdemir B, Sevfi L (2018) An efficient retinal blood vessel segmentation using morphological operations. In: 2018 2nd International Symposium on Multidisciplinary Studies and Innovative Technologies (ISMSIT). IEEE, pp 1–7. https://doi.org/10.1109/ISMSIT.2018.8567239

  18. Kaftan JN, Kiraly AP, Bakai A, Das M, Novak CL, Aach T (2008) Fuzzy pulmonary vessel segmentation in contrast enhanced CT data. Medical Imaging 2008: Image Processing 6914:585–596 (SPIE)

    Google Scholar 

  19. Wu C, Agam G, Roy AS, Armato SG III (2004) Regulated morphology approach to fuzzy shape analysis with application to blood vessel extraction in thoracic CT scans. Med Imaging 2004 Image Process 5370:1262–1270

    Article  Google Scholar 

  20. Meye D, Wien F (2015) Support vector machines. The Interface to libsvm in package e1071, 28(20):597

  21. Wu Y et al (2023) Deep CNN for COPD identification by multi-view snapshot integration of 3D airway tree and lung field. Biomed Signal Process Control 79:104162

    Article  Google Scholar 

  22. Zou Y, Chen S, Sun Q, Liu B, Zhang J (2021) DCET-Net: dual-stream convolution expanded transformer for breast cancer histopathological image classification. In: 2021 IEEE International Conference on Bioinformatics and Biomedicine (BIBM). IEEE, pp 1235–1240

  23. Sünnetci KM, Alkan A (2022) Lung cancer detection by using probabilistic majority voting and optimization techniques. Int J Imaging Syst Technol 32(6):2049–2065

    Article  Google Scholar 

  24. Zhang B et al (2022) Multi-scale segmentation squeeze-and-excitation UNet with conditional random field for segmenting lung tumor from CT images. Comput Methods Programs Biomed 222:106946

    Article  PubMed  Google Scholar 

  25. Pang H et al (2022) A fully automatic segmentation pipeline of pulmonary lobes before and after lobectomy from computed tomography images. Comput Biol Med 147:105792

    Article  PubMed  Google Scholar 

  26. Krizhevsky A, Sutskever I, Hinton G (2012) ImageNet classification with deep convolutional neural networks. Adv Neural Inf Process Syst 25:2

    Google Scholar 

  27. He K, Zhang X, Ren S, Sun J (2016) Deep residual learning for image recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 770–778

  28. Isensee F, Jaeger PF, Kohl SA, Petersen J, Maier-Hein KH (2021) nnU-Net: a self-configuring method for deep learning-based biomedical image segmentation. Nat Methods 18(2):203–211

    Article  CAS  PubMed  Google Scholar 

  29. Wang A, Islam M, Xu M, Ren H (2022) Rethinking surgical instrument segmentation: a background image can be all you need. Medical Image Computing and Computer Assisted Intervention–MICCAI 2022: 25th International Conference, Singapore, September 18–22, 2022, Proceedings, Part VII. Springer, pp 355–364

    Chapter  Google Scholar 

  30. Ronneberger O, Fischer P, Brox T (2015) U-net: convolutional networks for biomedical image segmentation. Medical Image Computing and Computer-Assisted Intervention–MICCAI 2015: 18th International Conference, Munich, Germany, October 5–9, 2015, Proceedings, Part III 18. Springer, pp 234–241

    Google Scholar 

  31. Milletari F, Navab N, Ahmadi S-A (2016) V-net: fully convolutional neural networks for volumetric medical image segmentation. In: 2016 fourth international conference on 3D vision (3DV). IEEE, pp 565–571

  32. Çiçek Ö, Abdulkadir A, Lienkamp SS, Brox T, Ronneberger O (2016) 3D U-Net: learning dense volumetric segmentation from sparse annotation. Medical Image Computing and Computer-Assisted Intervention–MICCAI 2016: 19th International Conference, Athens, Greece, October 17–21, 2016, Proceedings, Part II 19. Springer, pp 424–432

    Google Scholar 

  33. Nardelli P et al (2018) Pulmonary artery–vein classification in CT images using deep learning. IEEE Trans Med Imaging 37(11):2428–2440

    Article  PubMed  PubMed Central  Google Scholar 

  34. Zhai Z et al (2019) Linking convolutional neural networks with graph convolutional networks: application in pulmonary artery-vein separation. Graph Learning in Medical Imaging: First International Workshop, GLMI 2019, Held in Conjunction with MICCAI 2019, Shenzhen, China, October 17, 2019, Proceedings 1. Springer, pp 36–43

    Chapter  Google Scholar 

  35. Qin Y et al (2021) Learning tubule-sensitive CNNs for pulmonary airway and artery-vein segmentation in CT. IEEE Trans Med Imaging 40(6):1603–1617

    Article  PubMed  Google Scholar 

  36. Gu X, Wang J, Zhao J, Li Q (2019) Segmentation and suppression of pulmonary vessels in low-dose chest CT scans. Med Phys 46(8):3603–3614

    Article  PubMed  Google Scholar 

  37. Cui H, Liu X, Huang N (2019) Pulmonary vessel segmentation based on orthogonal fused u-net++ of chest CT images. Medical Image Computing and Computer Assisted Intervention–MICCAI 2019: 22nd International Conference, Shenzhen, China, October 13–17, 2019, Proceedings, Part VI 22. Springer, pp 293–300

    Chapter  Google Scholar 

  38. Xu S, Zhang Z, Zhou Q, Shao W, Tan W (2021) A pulmonary vascular extraction algorithm from chest CT/CTA images. J Healthcare Eng 2021:1–7

    Google Scholar 

  39. Vaswani A et al (2017) Attention is all you need. Adv Neural Inf Process Syst 30:5998–6008

    Google Scholar 

  40. Duong LT, Le NH, Tran TB, Ngo VM, Nguyen PT (2021) Detection of tuberculosis from chest X-ray images: boosting the performance with vision transformer and transfer learning. Expert Syst Appl 184:115519

    Article  Google Scholar 

  41. Niu C, Wang G (2022) Unsupervised contrastive learning based transformer for lung nodule detection. Phys Med Biol 67(20):204001

    Article  Google Scholar 

  42. Wu H, Chen S, Chen G, Wang W, Lei B, Wen Z (2022) FAT-Net: feature adaptive transformers for automated skin lesion segmentation. Med Image Anal 76:102327

    Article  PubMed  Google Scholar 

  43. Shen Z, Yang H, Zhang Z, Zheng S (2022) Automated kidney tumor segmentation with convolution and transformer network. In: Kidney and Kidney Tumor Segmentation: MICCAI 2021 Challenge, KiTS 2021, Held in Conjunction with MICCAI 2021, Strasbourg, France, September 27, 2021, Proceedings: Springer, pp 1–12

  44. Li Y et al (2022) Global transformer and dual local attention network via deep-shallow hierarchical feature fusion for retinal vessel segmentation. IEEE Transactions on Cybernetics, pp 1–14

  45. Tan W, Huang P, Li X, Ren G, Chen Y, Yang J (2021) Analysis of segmentation of lung parenchyma based on deep learning methods. J Xray Sci Technol 29(6):945–959

    PubMed  Google Scholar 

  46. Tan W, Zhou L, Li X, Yang X, Chen Y, Yang J (2021) Automated vessel segmentation in lung CT and CTA images via deep neural networks. J Xray Sci Technol 29(6):1123–1137

    PubMed  Google Scholar 

  47. Tan W, Liu P, Li X, Xu S, Chen Y, Yang J (2022) Segmentation of lung airways based on deep learning methods. IET Image Proc 16(5):1444–1456

    Article  Google Scholar 

  48. Chen Y, Kalantidis Y, Li J, Yan S, Feng J (2018) A^ 2-nets: double attention networks. Adv Neural Inf Process Syst 31:350–359

    Google Scholar 

  49. Li Y, Yao T, Pan Y, Mei T (2022) Contextual transformer networks for visual recognition. IEEE Trans Pattern Anal Mach Intell 4(2):1489–1500

    Article  Google Scholar 

  50. Zhao R et al (2020) Rethinking dice loss for medical image segmentation. In: 2020 IEEE International Conference on Data Mining (ICDM). IEEE, pp 851–860

  51. Lin T-Y, Goyal P, Girshick R, He K, Dollár P (2017) Focal loss for dense object detection. In: Proceedings of the IEEE international conference on computer vision, pp 2980–2988

  52. Sunnetci KM, Kaba E, Beyazal Çeliker F, Alkan A (2023) Comparative parotid gland segmentation by using ResNet-18 and MobileNetV2 based DeepLab v3+ architectures from magnetic resonance images. Concurrency and Computation: Practice and Experience 35:e7405

  53. Tuncer SA, Alkan A (2019) Spinal cord based kidney segmentation using connected component labeling and K-means clustering algorithm. Traitement du Signal 36:6

    Article  Google Scholar 

  54. Wu Y, Zhao S, Qi S et al (2022) Two-stage contextual transformer-based convolutional neural network for airway extraction from CT images. arXiv preprint arXiv:2212.07651

Download references

Acknowledgements

This work was partly supported by the National Natural Science Foundation of China (82072008), Natural Science Foundation of Liaoning Province (2021-YGJC-21), Key R&D Program Guidance Projects in Liaoning Province (2019JH8/10300051), Hong Kong Research Grants Council (RGC) Collaborative Research Fund (C4026-21G), Hong Kong General Research Fund (RGC-GRF 14211420), Hong Kong RGC Research Impact Fund (RIF) R4020-22, and the Fundamental Research Funds for the Central Universities (N2124006-3, N2224001-10).

Author information

Authors and Affiliations

  1. College of Medicine and Biological Information Engineering, Northeastern University, Shenyang, China

    Yanan Wu, Shouliang Qi, Meihuan Wang, Shuiqing Zhao & Haowen Pang

  2. Key Laboratory of Intelligent Computing in Medical Image, Ministry of Education, Northeastern University, Shenyang, China

    Yanan Wu & Shouliang Qi

  3. Department of Electronic Engineering, Faculty of Engineering, The Chinese University of Hong Kong, Hong Kong, China

    Yanan Wu, Long Bai & Hongliang Ren

  4. State Key Laboratory of Respiratory Disease, National Clinical Research Center for Respiratory Disease, The National Center for Respiratory Medicine, Guangzhou Institute of Respiratory Health, The First Affiliated Hospital of Guangzhou Medical University, Guangzhou, China

    Jiaxuan Xu

  5. Department of Biomedical Engineering (BME), National University of Singapore, Singapore, Singapore

    Hongliang Ren

  6. Research Institute, National University of Suzhou, Suzhou, Jiangsu, China

    Hongliang Ren

  7. Shun Hing Institute of Advanced Engineering, The Chinese University of Hong Kong (CUHK), Hong Kong, China

    Hongliang Ren

Authors
  1. Yanan Wu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Shouliang Qi
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Meihuan Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Shuiqing Zhao
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Haowen Pang
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Jiaxuan Xu
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Long Bai
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. Hongliang Ren
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding authors

Correspondence to Shouliang Qi or Hongliang Ren.

Ethics declarations

Ethics approval

All procedures performed in studies involving human participants were in accordance with the ethical standards of the institutional and/or national research committee and with the 1964 Helsinki declaration and its later amendments or comparable ethical standards.

Consent to participate

Informed consent was obtained from all individual participants included in the study.

Competing interests

The authors declare no competing interests.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Wu, Y., Qi, S., Wang, M. et al. Transformer-based 3D U-Net for pulmonary vessel segmentation and artery-vein separation from CT images. Med Biol Eng Comput 61, 2649–2663 (2023). https://doi.org/10.1007/s11517-023-02872-5

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11517-023-02872-5

Keywords

Search

Navigation