Skip to main content
SpringerLink
Log in

DS-MSFF-Net: Dual-path self-attention multi-scale feature fusion network for CT image segmentation

  • Published:
Applied Intelligence Aims and scope Submit manuscript

Abstract

Computed tomography (CT) is an important technique that is widely used in disease screening and diagnosis. In order to assist doctors in diagnosis and treatment plans, an efficient and accurate automatic image segmentation technology is urgently needed. CT images of different lesions always have problems such as different resolutions, different numbers of lesions, and inconspicuous contrast between lesions and background areas, which brings considerable challenges to the automated segmentation process. To this end, we propose a dual-path self-attention multi-scale feature fusion network (DS-MSFF-Net) that fuses self-attention mechanism and dilated convolution. It is worth noting that this network includes two parallel branch paths, which enables it to extract long-range semantic feature information effectively while extracting detailed feature information of CT images. Additionally, a novel feature extraction module is designed to focus limited learning resources on low-resolution high-order semantic feature maps, which can improve the segmentation accuracy without significant additional computational overhead. We extensively evaluate our method on the LIDC-IDRI lung nodule segmentation dataset and the LiTS2017 liver segmentation dataset, which outperforms other recent state-of-the-art methods on various CT image segmentation tasks.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9

Similar content being viewed by others

Data availability and access

The data that support the findings of this study are available from the corresponding author upon reasonable request.

References

  1. Jun Y, Zhu C, Zhang J, Huang Q, Tao D (2019) Spatial pyramid-enhanced netvlad with weighted triplet loss for place recognition. IEEE Trans Neural Netw Learn Syst 31(2):661–674

    Google Scholar 

  2. Liu F, Song Q, Jin G (2020) The classification and denoising of image noise based on deep neural networks. Appl Intell 50(7):2194–2207

    Article  Google Scholar 

  3. Ronneberger O, Fischer P, Brox T (2015) U-net: Convolutional networks for biomedical image segmentation. In: International conference on medical image computing and computer-assisted intervention, pp 234–241. Springer

  4. Kirillov A, Wu Y, He K, Girshick R (2020) Pointrend: Image segmentation as rendering. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 9799–9808

  5. Zhang Y, Bai Y, Ding M, Shibiao X, Ghanem B (2020) Kgsnet: key-point-guided super-resolution network for pedestrian detection in the wild. IEEE Trans Neural Netw Learn Syst 32(5):2251–2265

    Article  Google Scholar 

  6. Pal SK, Pramanik A, Maiti J, Mitra P (2021) Deep learning in multi-object detection and tracking: state of the art. Appl Intell 51(9):6400–6429

    Article  Google Scholar 

  7. Shin Y-G, Sagong M-C, Yeo Y-J, Kim S-W, Ko S-J (2020) Pepsi++: Fast and lightweight network for image inpainting. IEEE Trans Neural Netw Learn Syst 32(1):252–265

    Article  Google Scholar 

  8. Chen Y, Zhang H, Liu L, Chen X, Zhang Q, Yang K, Xia R, Xie J (2021) Research on image inpainting algorithm of improved gan based on two-discriminations networks. Appl Intell 51(6):3460–3474

    Article  Google Scholar 

  9. Hong C, Jun Y, Zhang J, Jin X, Lee K-H (2018) Multimodal face-pose estimation with multitask manifold deep learning. IEEE Trans Industr Inform 15(7):3952–3961

    Article  Google Scholar 

  10. Zhongxu H, Youmin H, Bo W, Liu J, Han D, Kurfess T (2018) Hand pose estimation with multi-scale network. Appl Intell 48(8):2501–2515

    Article  Google Scholar 

  11. Li X, Zhou Y, Peng D, Lang G, Min X, Wei W (2021) A deep learning system that generates quantitative ct reports for diagnosing pulmonary tuberculosis. Appl Intell 51(6):4082–4093

    Article  Google Scholar 

  12. Xia H, Ma M, Li H, Song S (2021) Mc-net: multi-scale context-attention network for medical ct image segmentation. Appl Intell 1–12

  13. Liao F, Liang M, Li Z, Xiaolin H, Song S (2019) Evaluate the malignancy of pulmonary nodules using the 3-d deep leaky noisy-or network. IEEE Trans Neural Netw Learn Syst 30(11):3484–3495

    Article  Google Scholar 

  14. Roy R, Chakraborti T, Chowdhury AS (2019) A deep learning-shape driven level set synergism for pulmonary nodule segmentation. Pattern Recognit Lett 123:31–38

    Article  Google Scholar 

  15. Shakibapour E, Cunha A, Aresta G, Mendonça AM, Campilho A (2019) An unsupervised metaheuristic search approach for segmentation and volume measurement of pulmonary nodules in lung ct scans. Exp Syst Appl 119:415–428

    Article  Google Scholar 

  16. Zhu W, Liu C, Fan W, Xie X (2018) Deeplung: Deep 3d dual path nets for automated pulmonary nodule detection and classification. In: 2018 IEEE Winter conference on applications of computer vision (WACV), pp 673–681. IEEE

  17. Wei Y, Liang X, Chen Y, Shen X, Cheng M-M, Feng J, Zhao Y, Yan S (2016) Stc: A simple to complex framework for weakly-supervised semantic segmentation. IEEE Trans Pattern Anal Mach Intell 39(11):2314–2320

    Article  Google Scholar 

  18. Li M, Hsu W, Xie X, Cong J, Gao W (2020) Sacnn: Self-attention convolutional neural network for low-dose ct denoising with self-supervised perceptual loss network. IEEE Trans Med Imaging 39(7):2289–2301

    Article  Google Scholar 

  19. Long J, Shelhamer E, Darrell T (2015) Fully convolutional networks for semantic segmentation. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 3431–3440

  20. Christ PF, Ettlinger F, Grün F, Elshaera MEA, Lipkova J, Schlecht S, Ahmaddy F, Tatavarty S, Bickel M, Bilic P et al (2017) Automatic liver and tumor segmentation of ct and mri volumes using cascaded fully convolutional neural networks. arXiv:1702.05970

  21. Bi L, Feng D, Kim J (2018) Dual-path adversarial learning for fully convolutional network (fcn)-based medical image segmentation. Vis Comput 34(6):1043–1052

    Article  Google Scholar 

  22. Wu Y, Lin L (2020) Automatic lung segmentation in ct images using dilated convolution based weighted fully convolutional network. In: Journal of Physics: Conference Series, vol 1646, pp 012032. IOP Publishing

  23. Chen L-C, Papandreou G, Kokkinos I, Murphy K, Yuille AL (2017) Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs. IEEE Trans Pattern Anal Mach Intell 40(4):834–848

    Article  Google Scholar 

  24. Men K, Dai J, Li Y (2017) Automatic segmentation of the clinical target volume and organs at risk in the planning ct for rectal cancer using deep dilated convolutional neural networks. Med Phys 44(12):6377–6389

    Article  Google Scholar 

  25. Xia H, Sun W, Song S, Mou X (2020) Md-net: multi-scale dilated convolution network for ct images segmentation. Neural Process Lett 51(3):2915–2927

    Article  Google Scholar 

  26. Liu Z, Liu X, Xiao B, Wang S, Miao Z, Sun Y, Zhang F (2020) Segmentation of organs-at-risk in cervical cancer ct images with a convolutional neural network. Phys Med 69:184–191

    Article  Google Scholar 

  27. Zhao X, Zhang P, Song F, Fan G, Sun Y, Wang Y, Tian Z, Zhang L, Zhang G (2021) D2a u-net: Automatic segmentation of covid-19 ct slices based on dual attention and hybrid dilated convolution. Comput Biol Med 104526

  28. Zhou Z, Siddiquee Md MR, Tajbakhsh N, Liang J (2018) Unet++: A nested u-net architecture for medical image segmentation. In: Deep learning in medical image analysis and multimodal learning for clinical decision support, pp 3–11. Springer

  29. Li X, Chen H, Qi X, Dou Q, Chi-Wing F, Heng P-A (2018) H-denseunet: hybrid densely connected unet for liver and tumor segmentation from ct volumes. IEEE Trans Med Imaging 37(12):2663–2674

    Article  Google Scholar 

  30. He K, Lian C, Zhang B, Zhang X, Cao X, Nie D, Gao Y, Zhang J, Shen D (2021) Hf-unet: learning hierarchically inter-task relevance in multi-task u-net for accurate prostate segmentation in ct images. IEEE Trans Med Imaging 40(8):2118–2128

    Article  Google Scholar 

  31. Khan RA, Luo Y, Wu F-X (2022) Rms-unet: Residual multi-scale unet for liver and lesion segmentation. Artif Intell Med 124:102231

    Article  Google Scholar 

  32. Yin S, Deng H, Zelin X, Zhu Q, Cheng J (2022) Sd-unet: A novel segmentation framework for ct images of lung infections. Electronics 11(1):130

    Article  Google Scholar 

  33. Mehta S, Rastegari M, Caspi A, Shapiro L, Hajishirzi H (2018) Espnet: Efficient spatial pyramid of dilated convolutions for semantic segmentation. In: Proceedings of the european conference on computer vision (ECCV), pp 552–568

  34. Chen Y, Wang K, Liao X, Qian Y, Wang Q, Yuan Z, Heng P-A (2019) Channel-unet: a spatial channel-wise convolutional neural network for liver and tumors segmentation. Front Genet 10:1110

    Article  Google Scholar 

  35. Kushnure DT, Talbar SN (2021) Ms-unet: A multi-scale unet with feature recalibration approach for automatic liver and tumor segmentation in ct images. Comput Med Imaging Graph 89:101885

    Article  Google Scholar 

  36. Xie X, Pan X, Shao F, Zhang W, An J (2022) Mci-net: Multi-scale context integrated network for liver ct image segmentation. Comput Electric Eng 101:108085

    Article  Google Scholar 

  37. Xia H, Ma M, Li H, Song S (2022) Mc-net: multi-scale context-attention network for medical ct image segmentation. Appl Intell 52(2):1508–1519

    Article  Google Scholar 

  38. Woo S, Park J, Lee J-Y, Kweon IS (2018) Cbam: Convolutional block attention module. In: Proceedings of the European conference on computer vision (ECCV), pp 3–19

  39. Srinivas A, Lin T-Y, Parmar N, Shlens J, Abbeel P, Vaswani A (2021) Bottleneck transformers for visual recognition. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 16519–16529

  40. Sinha A, Dolz J (2020) Multi-scale self-guided attention for medical image segmentation. IEEE J Biomed Health Inform 25(1):121–130

    Article  Google Scholar 

  41. Zhang C, Jingben L, Hua Q, Li C, Wang P (2022) Saa-net: U-shaped network with scale-axis-attention for liver tumor segmentation. Biomed Signal Process Control 73:103460

    Article  Google Scholar 

  42. Tang H, Liu X, Han K, Xie X, Chen X, Qian H, Liu Y, Sun S, Bai N (2021) Spatial context-aware self-attention model for multi-organ segmentation. In: Proceedings of the IEEE/CVF winter conference on applications of computer vision, pp 939–949

  43. Chen L-C, Papandreou G, Schroff F, Adam H (2017) Rethinking atrous convolution for semantic image segmentation. arXiv:1706.05587

  44. Chen L-C, Zhu Y, Papandreou G, Schroff F, Adam H (2018) Encoder-decoder with atrous separable convolution for semantic image segmentation. In: Proceedings of the European conference on computer vision (ECCV), pp 801–818

  45. He K, Zhang X, Ren S, Sun J (2016) Deep residual learning for image recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 770–778

  46. Wang C-Y, Liao H-YM, Wu Y-H, Chen P-Y, Hsieh J-W, Yeh I-H (2020) Cspnet: A new backbone that can enhance learning capability of cnn. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition workshops, pp 390–391

  47. Cao Y, Xu J, Lin S, Wei F, Hu H (2019) Gcnet: Non-local networks meet squeeze-excitation networks and beyond. In: Proceedings of the IEEE/CVF international conference on computer vision workshops

  48. Wang Q, Wu B, Zhu P, Li P, Zuo W, Hu Q (2020) Eca-net: efficient channel attention for deep convolutional neural networks, 2020 IEEE. In: CVF conference on computer vision and pattern recognition (CVPR). IEEE

  49. Maqsood M, Yasmin S, Mehmood I, Bukhari M, Kim M (2021) An efficient da-net architecture for lung nodule segmentation. Mathematics 9(13):1457

    Article  Google Scholar 

  50. Shuo Wang M, Zhou ZL, Liu Z, Dongsheng G, Zang Y, Dong D, Gevaert O, Tian J (2017) Central focused convolutional neural networks: Developing a data-driven model for lung nodule segmentation. Med Image Anal 40:172–183

    Article  Google Scholar 

  51. Jiang J, Hu Y-C, Liu C-J, Halpenny D, Hellmann MD, Deasy JO, Mageras G, Veeraraghavan H (2018) Multiple resolution residually connected feature streams for automatic lung tumor segmentation from ct images. IEEE Tran Med Imaging 38(1):134–144

    Article  Google Scholar 

  52. Huang X, Shan J, Vaidya V (2017) Lung nodule detection in ct using 3d convolutional neural networks. In: 2017 IEEE 14th international symposium on biomedical imaging (ISBI 2017), pp 379–383. IEEE

  53. Zhao X, Sun W, Qian W, Qi S, Sun J, Zhang B, Yang Z (2019) Fine-grained lung nodule segmentation with pyramid deconvolutional neural network. In Medical Imaging 2019: Computer-Aided Diagnosis, vol 10950, pp 956–961. SPIE

  54. Hancock MC, Magnan JF (2019) Lung nodule segmentation via level set machine learning. arXiv:1910.03191

  55. Huang X, Sun W, Tseng T-LB, Li C, Qian W (2019) Fast and fully-automated detection and segmentation of pulmonary nodules in thoracic ct scans using deep convolutional neural networks. Comput Med Imaging Graph 74:25–36

    Article  Google Scholar 

  56. Wang J, Lv P, Wang H, Shi C (2021) Sar-u-net: squeeze-and-excitation block and atrous spatial pyramid pooling based residual u-net for automatic liver segmentation in computed tomography. Comput Methods Programs Biomed 208:106268

    Article  Google Scholar 

  57. Oktay O, Schlemper J, Folgoc LL, Lee M, Heinrich M, Misawa K, Mori K, McDonagh S, Hammerla NY, Kainz B et al (2018) Attention u-net: Learning where to look for the pancreas. arXiv:1804.03999

  58. Changling L, Xiangfen S, Hang Z, Li F, Tao H, Yuchen Z, Jun J, Jianan W, Jianping X, Yong S (2021) An 8-layer residual u-net with deep supervision for segmentation of the left ventricle in cardiac ct angiography. Comput Methods Programs Biomed 200:105876

Download references

Acknowledgements

This work has been supported in part by the National Natural Science Foundation of China under Grant 62102331, 62176125 and 61772272, which also has been supported in part by Natural Science Foundation of Sichuan Province 2022NSFSC0839 and Southwest University of Science and Technology Doctoral Research Fund Project 22zx7110.

Author information

Authors and Affiliations

  1. School of Information Engineering, Southwest University of Science and Technology, Mianyang, 621010, China

    Xiaoqian Zhang, Lei Pu & Xiao Wang

  2. School of Computer Science and Technology, Southwest University of Science and Technology, Mianyang, 621010, China

    Liming Wan

  3. Radiology Department, Mianyang Central Hospital, Mianyang, 621010, China

    Ying Zhou

Authors
  1. Xiaoqian Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Lei Pu
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Liming Wan
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Xiao Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Ying Zhou
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

Xiaoqian Zhang: Software, Writing-Reviewing and Editing. Lei Pu: Conceptualization, Methodology, Software. Liming Wan: Writing-Original draft preparation, Data curation, Software. Xiao Wang: Software, Editing. Ying Zhou: Data curation.

Corresponding author

Correspondence to Xiaoqian Zhang.

Ethics declarations

Competing interests

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Conflict of interest

The authors declare that they have no conflict of interest.

Ethical and informed consent for data used

This article does not contain any studies with human participants or animals performed by the author. We obtain ethical and informed consent from data subjects before collecting, using, or disclosing their personal data.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Zhang, X., Pu, L., Wan, L. et al. DS-MSFF-Net: Dual-path self-attention multi-scale feature fusion network for CT image segmentation. Appl Intell (2024). https://doi.org/10.1007/s10489-024-05372-7

Download citation

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s10489-024-05372-7

Keywords

Search

Navigation