Skip to main content
SpringerLink
Log in

TSNet: Task-specific network for joint diabetic retinopathy grading and lesion segmentation of ultra-wide optical coherence tomography angiography images

  • Research
  • Published:
The Visual Computer Aims and scope Submit manuscript

Abstract

Diabetic retinopathy (DR) is a common complication of diabetes which may lead to blindness. Early diagnosis can effectively prevent the deterioration of the disease and enable timely treatment. Ophthalmologists diagnose DR by observing ultra-wide optical coherence tomography angiography (UW-OCTA) images, which visualize unprecedented detail of DR lesions. In this paper, we propose an end-to-end task-specific network (TSNet) for joint DR grading and lesion segmentation of UW-OCTA images. Specifically, we design task-specific attention block to generate task-specific feature maps for respective segmentation and classification tasks. Furthermore, we devise task-specific fusion block to fuse the original task-specific feature map and augmented task-specific feature map for the following segmentation and classification decoders to generate DR lesion predictive mask and DR grading predictive result. Experiments on a public-available UW-OCTA dataset demonstrate that our model outperforms state-of-the-art (SOTA) multi-task models and achieves promising results on both DR lesion segmentation and DR grading classification tasks

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6

Similar content being viewed by others

References

  1. Antonetti, D.A., Klein, R., Gardner, T.W.: Mechanisms of disease diabetic retinopathy[J]. New England J. Med. 366(13), 1227–1239 (2012)

    Article  Google Scholar 

  2. Kobrin Klein, B.E.: Overview of epidemiologic studies of diabetic retinopathy[J]. Ophthal. Epidemiol. 14(4), 179–183 (2007)

    Article  Google Scholar 

  3. Yang, Q.H., Zhang, Y., Zhang, X.M., et al.: Prevalence of diabetic retinopathy, proliferative diabetic retinopathy and non-proliferative diabetic retinopathy in Asian T2DM patients: a systematic review and meta-analysis[J]. Int. J. Ophthalmol. 12(2), 302 (2019)

    Google Scholar 

  4. Vieira-Potter, V. J., Karamichos, D., Lee, D. J.: Ocular complications of diabetes and therapeutic approaches[J]. BioMed Res. Int. (2016)

  5. Massin, P., Bandello, F., Garweg, J.G., et al.: Safety and Efficacy of Ranibizumab in Diabetic Macular Edema (RESOLVE Study) A 12-month, randomized, controlled, double-masked, multicenter phase II study[J]. Diabetes care 33(11), 2399–2405 (2010)

    Article  Google Scholar 

  6. Elman, M.J., Aiello, L.P., Beck, R.W., et al.: Randomized trial evaluating ranibizumab plus prompt or deferred laser or triamcinolone plus prompt laser for diabetic macular edema[J]. Ophthalmology 117(6), 1064–1077 (2010)

    Article  Google Scholar 

  7. Michaelides, M., Kaines, A., Hamilton, R.D., et al.: A prospective randomized trial of intravitreal bevacizumab or laser therapy in the management of diabetic macular edema (BOLT study): 12-month data: report 2[J]. Ophthalmology 117(6), 1078–1086 (2010)

    Article  Google Scholar 

  8. Chen, G., Li, W., Tzekov, R., et al.: Ranibizumab monotherapy or combined with laser versus laser monotherapy for diabetic macular edema: a meta-analysis of randomized controlled trials[J]. PLoS One 9(12), e115797 (2014)

    Article  Google Scholar 

  9. Imran, A., Li, J., Pei, Y., et al.: Fundus image-based cataract classification using a hybrid convolutional and recurrent neural network[J]. Vis. Comput. 37, 2407–2417 (2021)

    Article  Google Scholar 

  10. Chandrasekaran, R., Loganathan, B.: Retinopathy grading with deep learning and wavelet hyper-analytic activations[J]. Vis. Comput. 39(7), 2741–2756 (2023)

    Article  Google Scholar 

  11. Dai, L., Wu, L., Li, H., et al.: A deep learning system for detecting diabetic retinopathy across the disease spectrum[J]. Nat. Commun. 12(1), 3242 (2021)

    Article  Google Scholar 

  12. Islam, S. M. S., Hasan, M. M., Abdullah, S.: Deep learning based early detection and grading of diabetic retinopathy using retinal fundus images[J] (2018) arXiv preprint arXiv:1812.10595

  13. Lee, R., Wong, T.Y., Sabanayagam, C.: Epidemiology of diabetic retinopathy, diabetic macular edema and related vision loss[J]. Eye Vision 2(1), 1–25 (2015)

    Article  Google Scholar 

  14. Prescott, G., Sharp, P., Goatman, K., et al.: Improving the cost-effectiveness of photographic screening for diabetic macular oedema: a prospective, multi-centre, UK study[J]. Br. J. Ophthalmol. 98(8), 1042–1049 (2014)

    Article  Google Scholar 

  15. De Carlo, T.E., Romano, A., Waheed, N.K., et al.: A review of optical coherence tomography angiography (OCTA)[J]. Int. J. Retina Vitreous 1, 1–15 (2015)

    Article  Google Scholar 

  16. Spaide, R.F., Fujimoto, J.G., Waheed, N.K.: Optical coherence tomography angiography[J]. Retina (Philadelphia, Pa) 35(11), 2161 (2015)

    Article  Google Scholar 

  17. Hwang, T.S., Jia, Y., Gao, S.S., et al.: Optical coherence tomography angiography features of diabetic retinopathy[J]. Retina (Philadelphia, Pa) 35(11), 2371 (2015)

    Article  Google Scholar 

  18. Tan, A.C.S., Tan, G.S., Denniston, A.K., et al.: An overview of the clinical applications of optical coherence tomography angiography[J]. Eye 32(2), 262–286 (2018)

    Article  Google Scholar 

  19. Liu, X., Huang, Z., Wang, Z., et al.: A deep learning based pipeline for optical coherence tomography angiography[J]. J. Biophoton. 12(10), e201900008 (2019)

    Article  Google Scholar 

  20. Jiang, Z., Huang, Z., Qiu, B., et al.: Comparative study of deep learning models for optical coherence tomography angiography[J]. Biomed. Opt. Exp. 11(3), 1580–1597 (2020)

    Article  Google Scholar 

  21. Kim, G., Kim, J., Choi, W.J., et al.: Integrated deep learning framework for accelerated optical coherence tomography angiography[J]. Sci. Rep. 12(1), 1289 (2022)

    Article  Google Scholar 

  22. Le, D., Son, T., Yao, X.: Machine learning in optical coherence tomography angiography[J]. Exp. Biol. Medi. 246(20), 2170–2183 (2021)

    Article  Google Scholar 

  23. Safi, H., Safi, S., Hafezi-Moghadam, A., et al.: Early detection of diabetic retinopathy[J]. Surv. Ophthalmol. 63(5), 601–608 (2018)

    Article  Google Scholar 

  24. Singh, N., Tripathi, R.C.: Automated early detection of diabetic retinopathy using image analysis techniques[J]. Int. J. Comput. Appl. 8(2), 18–23 (2010)

    Google Scholar 

  25. Lin, S., Masood, A., Li, T., et al.: Deep learning-enabled automatic screening of SLE diseases and LR using OCT images[J]. The Visual Computer. 1-11 (2023)

  26. Xiao, H., Ran, Z., Mabu, S., et al.: SAUNet++: an automatic segmentation model of COVID-19 lesion from CT slices[J]. Vis. Comput. 39(6), 2291–2304 (2023)

    Article  Google Scholar 

  27. Li, X., Pi, J., Lou, M., et al.: Multi-level feature fusion network for nuclei segmentation in digital histopathological images[J]. Vis. Comput. 39(4), 1307–1322 (2023)

    Google Scholar 

  28. Zang, P., Hormel, T.T., Wang, X., et al.: A diabetic retinopathy classification framework based on deep-learning analysis of OCT angiography[J]. Translat. Vision Sci. Technol. 11(7), 10–10 (2022)

    Article  Google Scholar 

  29. Ryu, G., Lee, K., Park, D., et al.: A deep learning model for identifying diabetic retinopathy using optical coherence tomography angiography[J]. Sci. Rep. 11(1), 23024 (2021)

    Article  Google Scholar 

  30. Le, D., Alam, M., Yao, C.K., et al.: Transfer learning for automated OCTA detection of diabetic retinopathy[J]. Translat. Vis. Sci. Technol. 9(2), 35–35 (2020)

  31. Sultana, F., Sufian, A., Dutta, P.: Automatic Diabetic Retinopathy Lesion Segmentation in UW-OCTA Images Using Transfer Learning[M]//MICCAI Challenge on Mitosis Domain Generalization, pp. 186–194. Springer Nature Switzerland, Cham (2022)

    Google Scholar 

  32. Gao, Z., Guo, J.: Diabetic Retinal Overlap Lesion Segmentation Network[M]//MICCAI Challenge on Mitosis Domain Generalization, pp. 38–45. Springer Nature Switzerland, Cham (2022)

    Google Scholar 

  33. Hou, J., Xiao, F., Xu, J., et al.: Deep-OCTA: Ensemble Deep Learning Approaches for Diabetic Retinopathy Analysis on OCTA Images[M]//MICCAI Challenge on Mitosis Domain Generalization, pp. 74–87. Springer Nature Switzerland, Cham (2022)

    Google Scholar 

  34. Xie, E., Wang, W., Yu, Z., et al.: Segformer: Simple and efficient design for semantic segmentation with transformers[J]. Adv. Neural Inf. Process. Syst. 34, 12077–12090 (2021)

    Google Scholar 

  35. Kendall, A., Gal, Y., Cipolla, R.: Multi-task learning using uncertainty to weigh losses for scene geometry and semantics[C]//Proceedings of the IEEE conference on computer vision and pattern recognition. 7482-7491 (2018)

  36. Wang, P., Patel, V.M., Hacihaliloglu, I., Simultaneous segmentation and classification of bone surfaces from ultrasound using a multi-feature guided CNN[C], , Medical Image Computing and Computer Assisted Intervention-MICCAI,: 21st International Conference, Granada, Spain, September 16–20, 2018, Proceedings, Part IV 11. Springer International Publishing 2018, 134–142 (2018)

  37. Zhou, Y., Chen, H., Li, Y., et al.: Multi-task learning for segmentation and classification of tumors in 3D automated breast ultrasound images[J]. Med. Imag. Anal. 70, 101918 (2021)

    Article  Google Scholar 

  38. Kang, Q., Lao, Q., Li, Y., et al.: Thyroid nodule segmentation and classification in ultrasound images through intra-and inter-task consistent learning[J]. Med. Imag. Anal. 79, 102443 (2022)

    Article  Google Scholar 

  39. Vaswani, A., Shazeer, N., Parmar, N., et al.: Attention is all you need[J]. Adv. Neural Inf. Process. Syst. 30 (2017)

  40. Ma, J., Chen, J., Ng, M., et al.: Loss odyssey in medical image segmentation[J]. Med. Imag. Anal. 71, 102035 (2021)

    Article  Google Scholar 

  41. Lin, T. Y., Goyal, P., Girshick, R., et al.: Focal loss for dense object detection[C]//Proceedings of the IEEE international conference on computer vision. 2980-2988 (2017)

  42. Qian, B., Chen, H., Wang, X., et al.: (2023) DRAC: Diabetic Retinopathy Analysis Challenge with Ultra-Wide Optical Coherence Tomography Angiography Images[J]. arXiv preprint arXiv:2304.02389

  43. Long, J., Shelhamer, E., Darrell, T.: Fully convolutional networks for semantic segmentation[C]//Proceedings of the IEEE conference on computer vision and pattern recognition. 3431-3440 (2015)

  44. Ronneberger, O., Fischer, P., Brox, T., U-net: Convolutional networks for biomedical image segmentation[C], , Medical Image Computing and Computer-Assisted Intervention-MICCAI,: 18th International Conference, Munich, Germany, October 5–9, 2015, Proceedings, Part III 18. Springer International Publishing 2015, 234–241 (2015)

  45. Jha, D., Smedsrud, P. H., Riegler, M. A., et al.: Resunet++: An advanced architecture for medical image segmentation[C]//2019 IEEE international symposium on multimedia (ISM). IEEE. 225-2255 (2019)

  46. Wang, J., Huang, Q., Tang, F., et al.: Stepwise feature fusion: Local guides global[C]//International Conference on Medical Image Computing and Computer-Assisted Intervention. Cham: Springer Nature Switzerland, 110-120 (2022)

  47. Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition[J] (2014) arXiv preprint arXiv:1409.1556

  48. He, K., Zhang, X., Ren, S., et al.: Deep residual learning for image recognition[C]//Proceedings of the IEEE conference on computer vision and pattern recognition. 770-778 (2016)

  49. Dosovitskiy, A., Beyer, L., Kolesnikov, A., et al.: An image is worth 16x16 words: Transformers for image recognition at scale[J] (2020) arXiv preprint arXiv:2010.11929

  50. Liu, Z., Lin, Y., Cao, Y., et al.: Swin transformer: Hierarchical vision transformer using shifted windows[C]//Proceedings of the IEEE/CVF international conference on computer vision. 10012-10022 (2021)

Download references

Funding

This work was supported in part by the National Key Research and Development Program of China under grant number 2022YFC2407000, in part by the Interdisciplinary Program of Shanghai Jiao Tong University under grant number YG2023LC11 and YG2022ZD007, in part by National Natural Science Foundation of China under grant number 62272298 and 62077037, in part by the College-level Project Fund of Shanghai Jiao Tong University Affiliated Sixth People’s Hospital under grant number ynlc201909, and in part by the Medical industrial Cross-fund of Shanghai Jiao Tong University under the grant number YG2022QN089. This work was supported in part by the National Science Foundation of China under Grants 62101346 and 62301330, in part by the Guangdong Basic and Applied Basic Research Foundation under Grants 2021A1515011702 and 2022A1515110101, in part by the Shaanxi Provincial Department of Education Special Scientific Research Project under Grants 20JK0613, in part by the Clinical Special Program of Shanghai Municipal Health Commission under Grants 20224044, and in part by the 3-year action plan to strengthen the construction of public health system in Shanghai 2023-2025 GWVI\(-\)11.1-28.

Author information

Author notes

  1. Jixue Tang, Xiang-ning Wang and Xiaolong Yang have contributed equally to this work. Yang Wen and Bin Sheng are both corresponding authors.

Authors and Affiliations

  1. Department of Computer Science and Engineering, Shanghai Jiao Tong University, Shanghai, China

    Jixue Tang, Bo Qian & Bin Sheng

  2. Huadong Sanatorium, Wuxi, Jiangsu, China

    Xiaolong Yang & Tingli Chen

  3. Department of Ophthalmology, Shanghai Sixth People’s Hospital Affiliated to Shanghai Jiao Tong University School of Medicine, Shanghai, China

    Xiang-ning Wang

  4. School of Electronic and Information Engineering, Shenzhen University, Shenzhen, China

    Yang Wen

Authors
  1. Jixue Tang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Xiang-ning Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Xiaolong Yang
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Yang Wen
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Bo Qian
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Tingli Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Bin Sheng
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

JT contributed to method design and paper writing. XW and XY were involved in experiments and participated in method discussions. BQ and TC participated in method discussions and assisted in experiments. YW and BS contributed to method design and paper writing, supervision and project administration.

Corresponding authors

Correspondence to Yang Wen or Bin Sheng.

Ethics declarations

Conflict of interest

The authors declare no competing interests.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Tang, J., Wang, Xn., Yang, X. et al. TSNet: Task-specific network for joint diabetic retinopathy grading and lesion segmentation of ultra-wide optical coherence tomography angiography images. Vis Comput (2023). https://doi.org/10.1007/s00371-023-03145-w

Download citation

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s00371-023-03145-w

Keywords

Search

Navigation