Skip to main content
SpringerLink
Log in

Global relationship memory network for retinal capillary segmentation on optical coherence tomography angiography images

  • Published:
Applied Intelligence Aims and scope Submit manuscript

Abstract

Automatic retinal capillary segmentation is a necessary prerequisite for quantitatively analyzing retinal vessels. In recent years, active research has been using deep learning-based methods in this field. However, deep learning methods inevitably lose spatial information of vessels when downsampling, thereby limiting the segmentation performance for fine vessels. Additionally, existing methods must pay more attention to the dynamic correlations between feature mappings in deep learning frameworks, resulting in inefficient acquisition of multi-scale decoder features. To address these limitations, we propose a Global Relationship Memory Network (GRM-Net) that considers the relationship between frequency domain and decoder hierarchy. Specifically, we first design a frequency relation learning module to preserve fine details of vessels during downsampling. This module decouples encoder features into frequency domain features of different dimensions and employs globally learnable filters to better guide the network’s attention towards vessels of different sizes and shapes. Secondly, we investigate a hierarchical relation selection module that leverages gate mechanisms to dynamically adjust the collaboration between two adjacent decoder blocks, thereby adaptively aggregating multi-scale decoder features to address the issue of underutilized decoding information. Comparative experimental results on two retinal vessel datasets validate the effectiveness of the proposed GRM-Net segmentation method. Compared to other state-of-the-art methods (Unet, CS-Net, DeeplabV3, MiniSeg, and OCTA-Net), this method achieves more remarkable segmentation results, preserving more details in the tiniest retinal capillaries. Code is available at https://github.com/WeiliJiang/Global-Relationship-Memory-Network.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6

Similar content being viewed by others

References

  1. Orlando JI, Barbosa Breda J, Keer Kv, Blaschko MB, Blanco PJ, Bulant CA (2018) Towards a glaucoma risk index based on simulated hemodynamics from fundus images. In: International conference on medical image computing and computer-assisted intervention, Springer, pp 65–73

  2. Kobat SG, Baygin N, Yusufoglu E, Baygin M, Barua PD, Dogan S, Yaman O, Celiker U, Yildirim H, Tan R-S et al (2022) Automated diabetic retinopathy detection using horizontal and vertical patch division-based pre-trained densenet with digital fundus images, Diagnostics 12(8):1975

  3. Fenner BJ, Tan GS, Tan AC, Yeo IY, Wong TY, Cheung GC (2018) Identification of imaging features that determine quality and repeatability of retinal capillary plexus density measurements in oct angiography. British journal of ophthalmology 102(4):509–514

    Article  Google Scholar 

  4. Murphy OC, Kwakyi O, Iftikhar M, Zafar S, Lambe J, Pellegrini N, Sotirchos ES, Gonzalez-Caldito N, Ogbuokiri E, Filippatou A et al (2020) Alterations in the retinal vasculature occur in multiple sclerosis and exhibit novel correlations with disability and visual function measures. Multiple Sclerosis Journal 26(7):815–828

    Article  Google Scholar 

  5. Staal J, Abràmoff MD, Niemeijer M, Viergever MA, Van Ginneken B (2004) Ridge-based vessel segmentation in color images of the retina. IEEE Trans Med Imaging 23(4):501–509

    Article  Google Scholar 

  6. Soares JV, Leandro JJ, Cesar RM, Jelinek HF, Cree MJ (2006) Retinal vessel segmentation using the 2-d gabor wavelet and supervised classification. IEEE Transactions on medical Imaging 25(9):1214–1222

    Article  Google Scholar 

  7. Ilayarajaa K, Logashanmugam E (2020) Retinal blood vessel segmentation using morphological and canny edge detection technique. In: 2020 International conference on system, computation, automation and networking (ICSCAN), IEEE, pp 1–5

  8. Chaki J, Woźniak M (2023) Deep learning for neurodegenerative disorder (2016 to 2022): A systematic review. Biomed Signal Process Control 80:104223

    Article  Google Scholar 

  9. Nath MK, Dandapat S, Barna C (2020) Automatic detection of blood vessels and evaluation of retinal disorder from color fundus images. J Intell & Fuzzy Syst 38(5):6019–6030

    Article  Google Scholar 

  10. Dash S, Verma S, Kavita Bevinakoppa S, Wozniak M, Shafi J, Ijaz MF (2022) Guidance image-based enhanced matched filter with modified thresholding for blood vessel extraction. Symmetry 14(2):194

    Article  Google Scholar 

  11. Mou L, Zhao Y, Chen L, Cheng J, Gu Z, Hao H, Qi H, Zheng Y, Frangi A, Liu J (2019) Cs-net: channel and spatial attention network for curvilinear structure segmentation, In: International conference on medical image computing and computer-assisted intervention, Springer, pp 721–730

  12. Ma Y, Hao H, Xie J, Fu H, Zhang J, Yang J, Wang Z, Liu J, Zheng Y, Zhao Y (2020) Rose: a retinal oct-angiography vessel segmentation dataset and new model. IEEE Trans Med Imaging 40(3):928–939

    Article  Google Scholar 

  13. Ronneberger O, Fischer P, Brox T (2015) U-net: Convolutional networks for biomedical image segmentation In: International conference on medical image computing and computer-assisted intervention, Springer, pp 234–241

  14. Khouloud S, Ahlem M, Fadel T, Amel S (2022) W-net and inception residual network for skin lesion segmentation and classification. Appl Intell pp 1–19

  15. Priyanka SN, Lal S, Nalini J, Reddy CS, Dell’Acqua F (2022) Diresunet: Architecture for multiclass semantic segmentation of high resolution remote sensing imagery data. Appl Intell 52(13):15462–15482

    Article  Google Scholar 

  16. Gu Z, Cheng J, Fu H, Zhou K, Hao H, Zhao Y, Zhang T, Gao S, Liu J (2019) Ce-net: Context encoder network for 2d medical image segmentation. IEEE Trans Med Imaging 38(10):2281–2292

    Article  Google Scholar 

  17. Chen L-C, Papandreou G, Kokkinos I, Murphy K, Yuille AL (2017) Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs. IEEE Trans Pattern Anal Mach Intell 40(4):834–848

    Article  Google Scholar 

  18. Rahaman N, Baratin A, Arpit D, Draxler F, Lin M, Hamprecht F, Bengio Y, Courville A (2019) On the spectral bias of neural networks. In: International conference on machine learning, PMLR, pp 5301–5310

  19. Zunair H, Hamza AB (2021) Sharp u-net: Depthwise convolutional network for biomedical image segmentation. Comput Biol Med 136:104699

    Article  Google Scholar 

  20. Jiang Y, Xu S, Fan H, Qian J, Luo W, Zhen S, Tao Y, Sun J, Lin H (2021) Ala-net: Adaptive lesion-aware attention network for 3d colorectal tumor segmentation. IEEE Trans Med Imaging 40(12):3627–3640

    Article  Google Scholar 

  21. Qiu Y, Liu Y, Li S, Xu J (2021) Miniseg: An extremely minimum network for efficient covid-19 segmentation. Proceedings of the AAAI conference on artificial intelligence 35:4846–4854

    Article  Google Scholar 

  22. Kaul C, Manandhar S, Pears N (2019) Focusnet: An attention-based fully convolutional network for medical image segmentation. In: 2019 IEEE 16th International symposium on biomedical imaging (ISBI 2019), IEEE, pp 455–458

  23. Gao C, Ye H, Cao F, Wen C, Zhang Q, Zhang F (2021) Multiscale fused network with additive channel-spatial attention for image segmentation. Knowl-Based Syst 214:106754

    Article  Google Scholar 

  24. Barua PD, Chan WY, Dogan S, Baygin M, Tuncer T, Ciaccio EJ, Islam N, Cheong KH, Shahid ZS, Acharya UR (2021) Multilevel deep feature generation framework for automated detection of retinal abnormalities using oct images. Entropy 23(12):1651

    Article  Google Scholar 

  25. Huang J, Meng Y, Guo F, Ji H, Han J (2020) Weakly-supervised aspect-based sentiment analysis via joint aspect-sentiment topic embedding. arXiv:2010.06705

  26. Dauphin YN, Fan A, Auli M, Grangier D (2017) Language modeling with gated convolutional networks. In: International conference on machine learning. PMLR, pp 933–941

  27. Wang H, Wu X, Huang Z, Xing EP (2020) High-frequency component helps explain the generalization of convolutional neural networks. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 8684–8694

  28. Yin D, Gontijo Lopes R, Shlens J, Cubuk ED, Gilmer J (2019) A fourier perspective on model robustness in computer vision. Adv Neural Inform Process Syst 32

  29. Hao J, Fu H, Xu Y, Hu Y, Li F, Zhang X, Liu J, Zhao Y (2020) Reconstruction and quantification of 3d iris surface for angle-closure glaucoma detection in anterior segment oct. In: International conference on medical image computing and computer-assisted intervention, Springer, pp 704–714

  30. Siłka W, Wieczorek M, Siłka J, Woźniak M (2023) Malaria detection using advanced deep learning architecture. Sensors 23(3):1501

    Article  Google Scholar 

  31. Tan Y, Shen W-D, Wu M-Y, Liu G-N, Zhao S-X, Chen Y, Yang K-F, Li Y-J (2023) Retinal layer segmentation in oct images with boundary regression and feature polarization, IEEE Trans Med Imaging, 1–1. https://doi.org/10.1109/TMI.2023.3317072

  32. Jiang W, Li Y, Jia Y, Feng Y, Yi Z, Chen M, Wang J (2023) Ori-net: Orientation-guided neural network for automated coronary arteries segmentation. Expert Syst Appl pp 121905

  33. Zhao Y, Que D, Tan J, Xiao Y, Yu Y (2019) Automated breast lesion segmentation from ultrasound images based on ppu-net. In: 2019 International conference on medical imaging physics and engineering (ICMIPE), IEEE, pp 1–4

  34. He K, Zhang X, Ren S, Sun J (2015) Spatial pyramid pooling in deep convolutional networks for visual recognition. IEEE Trans Pattern Anal Mach Intell 37(9):904–1916

    Article  Google Scholar 

  35. Zhang Z, Sabuncu M (2018) Generalized cross entropy loss for training deep neural networks with noisy labels, Advances in neural information processing systems 31

  36. Soomro TA, Afifi AJ, Gao J, Hellwich O, Paul M, Zheng L (2018) Strided u-net model: Retinal vessels segmentation using dice loss. In: 2018 Digital image computing: techniques and applications (DICTA), IEEE, pp 1–8

  37. Taha A, Lo P, Li J, Zhao T (2018) Kid-net: convolution networks for kidney vessels segmentation from ct-volumes. In: International conference on medical image computing and computer-assisted intervention, Springer, pp 463–471

  38. Rao Y, Zhao W, Zhu Z, Lu J, Zhou J (2021) Global filter networks for image classification. Adv Neural Inform Process Syst 34:980–993

    Google Scholar 

  39. Giarratano Y, Bianchi E, Gray C, Morris A, MacGillivray T, Dhillon B, Bernabeu MO (2020) Automated segmentation of optical coherence tomography angiography images: benchmark data and clinically relevant metrics. Trans Vis Sci & Technol 9(13):5–5

    Article  Google Scholar 

  40. Sudre CH, Li W, Vercauteren T, Ourselin S, Jorge Cardoso M (2017) Generalised dice overlap as a deep learning loss function for highly unbalanced segmentations. In: Deep learning in medical image analysis and multimodal learning for clinical decision support, Springer, pp 240–248

  41. Carballal A, Novoa FJ, Fernandez-Lozano C, García-Guimaraes M, Aldama-López G, Calviño-Santos R, Vazquez-Rodriguez JM, Pazos A (2018) Automatic multiscale vascular image segmentation algorithm for coronary angiography. Biomed Signal Process Control 46:1–9

    Article  Google Scholar 

  42. Zhao L, Li D, Chen J, Wan T (2018) Automated coronary tree segmentation for x-ray angiography sequences using fully-convolutional neural networks. In: 2018 IEEE Visual Communications And Image Processing (VCIP), IEEE, pp 1–4

  43. Wang J, Ju R, Chen Y, Zhang L, Hu J, Wu Y, Dong W, Zhong J, Yi Z (2018) Automated retinopathy of prematurity screening using deep neural networks. EBioMedicine 35:361–368

    Article  Google Scholar 

  44. Gegúndez-Arias ME, Aquino A, Bravo JM, Marín D (2011) A function for quality evaluation of retinal vessel segmentations. IEEE Trans Med Imaging 31(2):231–239

    Article  Google Scholar 

  45. Kofler F, Shit S, Ezhov I, Fidon L, Horvath I, Al-Maskari R, Li HB, Bhatia H, Loehr T, Piraud M et al (2023) blob loss: instance imbalance aware loss functions for semantic segmentation. In: International conference on information processing in medical imaging, Springer, pp 755–767

Download references

Acknowledgements

This work was supported by the National Natural Science Foundation of China (Grant No. 82302300); Innovation and Entrepreneurship Teams Project of Guangdong Pearl River Talents Program (2019ZT08Y105); Foshan HKUST Projects (FSUST21-HKUST10E); Guangdong Eye Intelligent Medical Imaging Equipment Engineering Technology Research Center (2022E076)

Author information

Authors and Affiliations

  1. College of Computer Science, Sichuan University, Chengdu, Sichuan, People’s Republic of China

    Weili Jiang

  2. Guangdong Weiren Meditech Co., Ltd, Foshan, 528000, Guangdong Province, China

    Lin An & Jia Qin

  3. Capital Institute of Pediatrics, Beijing, China

    Weijing Jiang

  4. Health Management Center, Foshan First People’s Hospital, Foshan, China

    Lushi Chen

  5. Department of Radiology, Guangdong Provincial People’s Hospital (Guangdong Academy of Medical Sciences), Southern Medical University, Guangzhou, China

    Chubin Ou

Authors
  1. Weili Jiang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Weijing Jiang
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Lin An
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Jia Qin
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Lushi Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Chubin Ou
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Chubin Ou.

Ethics declarations

Conflict of Interest

The authors declare that they have no conflict of interest.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Jiang, W., Jiang, W., An, L. et al. Global relationship memory network for retinal capillary segmentation on optical coherence tomography angiography images. Appl Intell 53, 30027–30040 (2023). https://doi.org/10.1007/s10489-023-05107-0

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10489-023-05107-0

Keywords

Search

Navigation