Skip to main content
SpringerLink
Log in

GFF-Net: Graph-based feature fusion network for diagnosing plus disease in retinopathy of prematurity

  • Published:
Applied Intelligence Aims and scope Submit manuscript

Abstract

Retinopathy of prematurity (ROP) is a retinal proliferative disorder, and it is the primary cause of childhood blindness. Accurate and convenient automatic diagnostic tools are required to assist ophthalmologists in diagnosing ROP. Existing methods only extract information from fundus image captured from posterior angle, while images captured from other angles are ignored, which limits the performance of the algorithm. In this paper, we propose a graph-based feature fusion network (GFF-Net) that can jointly analyze multiple images and make full use of the relevant information between these images to diagnose the plus disease in ROP. The convolutional features of different fundus images are connected into a graph, where the edges of the graph model the correlation between these images. A graph-based feature fusion module is proposed to aggregate features from the constructed feature graph and produce the final prediction. We compared the proposed GFF-Net with state-of-the-art methods on a clinical dataset and a low-quality “attack dataset". The GFF-Net achieved superior performance compared to other methods on both datasets. The results show that the proposed GFF-Net could be more effective than existing methods in clinical practice.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6

Similar content being viewed by others

References

  1. Heidary G, Löfqvist C, Mantagos IS, Vanderveen DK, Hellström A, Smith LE (2009) Retinopathy of prematurity: clinical insights from molecular studies. NeoReviews 10(11):e550–e557

    Article  Google Scholar 

  2. Steinkuller PG, Du L, Gilbert C, Foster A, Collins ML, Coats DK (1999) Childhood blindness. J AAPOS 3(1):26–32

    Article  Google Scholar 

  3. (1984) T.C. for the Classification of Retinopathy of Prematurity, An International Classification of Retinopathy of Prematurity. Pediatrics 74(1):127-133

  4. Aaberg T, Ben-Sira et al. (1987) An international classification of retinopathy of prematurity: II. The classification of retinal detachment. Arch Ophthalmol 105(7) 906-912

  5. (2005) C. of Retinopathy of Prematurity*, The International Classification of Retinopathy of Prematurity Revisited. Archives of Ophthalmology 123(7):991-999

  6. Higgins RD (2019) Oxygen saturation and retinopathy of prematurity. Clin Perinatol 46(3) 593-599. Oxygen Therapy

  7. Wu T, Zhang L, Tong Y, Qu Y, Xia B, Mu D (2018) Retinopathy of prematurity among very low-birth-weight infants in China: incidence and perinatal risk factors. Investig Ophthalmol Vis Sci 59(2):757–763

    Article  Google Scholar 

  8. Senjam S, Chandra P (2020) Retinopathy of prematurity: Addressing the emerging burden in developing countries. J Fam Med Prim Care 9(6):2600

    Article  Google Scholar 

  9. Cabrera MT, Sharon FF, Amanda EK, Michael FC, David KW (2011) Combining ROPtool measurements of vascular tortuosity and width to quantify plus disease in retinopathy of prematurity. J Am Assoc Pediatr Ophthalmol Strabismus 15(1):40–44

    Article  Google Scholar 

  10. Ataer-Cansizoglu E, Bolon-Canedo V, Campbell JP, Bozkurt A, Erdogmus D, Kalpathy-Cramer J, Patel S, Jonas K, Chan RV, Ostmo S, Chiang MF (2015) Computer-based image analysis for plus disease diagnosis in retinopathy of prematurity: performance of the i-ROP system and image features associated with expert diagnosis. Transl Vis Sci Technol 4(6):5

    Article  Google Scholar 

  11. Krizhevsky A, Sutskever I, Hinton GE (2017) Imagenet classification with deep convolutional neural networks. Communications of the ACM 60(6):84–90

    Article  Google Scholar 

  12. Szegedy, Christian and Vanhoucke, Vincent and Ioffe, Sergey and Shlens, Jon and Wojna, Zbigniew (2016) Rethinking the inception architecture for computer vision. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 2818–2826

  13. He K, Zhang X, Ren S, Sun J (2016) Deep residual learning for image recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 770–778

  14. Worall DE, Wilson CM, Brostow GJ (2016) Automated retinopathy of prematurity case detection with convolutional neural networks. In:Carneiro G, Mateus D, Peter L, Bradley L, Tavares JMRS, Belagiannis V, Papa JP, Nascimento JC, Loog M, Lu Z, Cardoso J, Jaime SC (eds) Deep learning and data labeling for medical applications (Springer International Publishing, Cham, 2016), pp 68-76

  15. Szegedy C, Liu W, Jia Y, Sermanet P, Reed S Anguelov D, Erhan D, Vanhoucke V, Rabinovich A (2015) Going deeper with convolutions. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 1–9

  16. Brown JM, Campbell JP, Beers A, Chang K, Ostmo S, Chan RVP, Dy J, Erdogmus D, Ioannidis S, Kalpathy-Cramer J, Chiang MF (2018) for the Imaging and Informatics in Retinopathy of Prematurity (i-ROP) Research Consortium Automated Diagnosis of Plus Disease in Retinopathy of Prematurity Using Deep Convolutional Neural Networks. JAMA Ophthalmol 136(7):803–810

    Article  Google Scholar 

  17. Brown JM, Campbell, JP, Beers A, Chang K, Donohue K, Ostmo S, Chan RP, Dy J, Erdogmus D, Ioannidis S, et al. (2018) Fully automated disease severity assessment and treatment monitoring in retinopathy of prematurity using deep learning. In:Medical Imaging 2018: Imaging Informatics for Healthcare, Research, and Applications vol 10579. SPIE, pp 149–155

  18. Morrison SL, Dukhovny D, Chan RP, Chiang MF, Campbell JP (2022) Cost-effectiveness of artificial intelligence–based retinopathy of prematurity screening. JAMA ophthalmol 140(4) 401–409. American Medical Association

  19. Chiang MF, Quinn GE, Fielder AR, Ostmo SR, Paul Chan R, Berrocal A, Binenbaum G, Blair M, Peter Campbell J, Capone A, Chen Y, Dai S, Ells A, Fleck BW, Good WV, Hartnett ME, Holmstrom G, Kusaka S, Kychenthal A, Lepore D, Lorenz B, Martinez-Castellanos MA, Özdek Ş Ademola-Popoola D, Reynolds JD, Shah PK, Shapiro M, Stahl A, Toth C, Vinekar A, Visser L, Wallace DK, Wu W-C, Zhao P, Zin A (2021) International Classification of Retinopathy of Prematurity, 3rd edition. Ophthalmology 128(10) e51-e68. https://doi.org/10.1016/j.ophtha.2021.05.031, https://www.sciencedirect.com/science/article/pii/S0161642021004164

  20. Nisha KL, Sreelekha G, Sathidevi PS, Mohanachandran P, Vinekar A (2019) A computer-aided diagnosis system for plus disease in retinopathy of prematurity with structure adaptive segmentation and vessel based features. Computerized Medical Imaging and Graphics 74:72–94. https://doi.org/10.1016/j.compmedimag.2019.04.003, https://www.sciencedirect.com/science/article/pii/S0895611118303495

  21. Yildiz VM, Tian P, Yildiz I, Brown JM, Kalpathy-Cramer J, Dy J, Ioannidis S, Erdogmus D, Ostmo S, Kim SJ et al (2020) Plus disease in retinopathy of prematurity: convolutional neural network performance using a combined neural network and feature extraction approach. Transl Vision Sci Technol 9(2):10–10

    Article  Google Scholar 

  22. Mao J, Luo Y, Liu L, Lao J, Shao Y, Zhang M, Zhang C, Sun M, Shen L (2020) Automated diagnosis and quantitative analysis of plus disease in retinopathy of prematurity based on deep convolutional neural networks. Acta Ophthalmol 98(3):e339–e345

    Article  Google Scholar 

  23. Ronneberger O, Fischer P (2015) Brox T (2015) U-Net: Convolutional Networks for Biomedical Image Segmentation. In: Navab N, Hornegger J, Wells WM, Frangi AF (eds) Medical Image Computing and Computer-Assisted Intervention - MICCAI 2015. Springer International Publishing, Cham, pp 234–241

    Chapter  Google Scholar 

  24. Huang G, Liu Z, Van Der Maaten L, Weinberger KQ (2017) Densely connected convolutional networks. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 4700–4708

  25. Tan Z, Simkin S, Lai C, Dai S (2019) Deep learning algorithm for automated diagnosis of retinopathy of prematurity plus disease. Transl Vision Sci Technol 8(6):23–23

    Article  Google Scholar 

  26. Wang J, Ji J, Zhang M, Lin J-W, Zhang G, Gong W, Cen L-P, Lu Y, Huang X, Huang D et al (2021) Automated explainable multidimensional deep learning platform of retinal images for retinopathy of prematurity screening. JAMA Netw Open 4(5):e218758–e218758

    Article  Google Scholar 

  27. Diakogiannis FI, Waldner F, Caccetta P, Wu C (2020) ResUNet-a: A deep learning framework for semantic segmentation of remotely sensed data. ISPRS J Photogramm Remote Sens 162:94–114

    Article  Google Scholar 

  28. He K, Gkioxari G, Dollár, P, Girshick R (2017) Mask r-cnn. In: Proceedings of the IEEE international conference on computer vision, pp 2961–2969

  29. Redd TK, Campbell JP, Brown JM, Kim SJ, Ostmo S, Chan RVP, Dy J, Erdogmus D, Ioannidis S, Kalpathy-Cramer J et al (2019) Evaluation of a deep learning image assessment system for detecting severe retinopathy of prematurity. Br J Ophthalmol 103(5):580–584

    Article  Google Scholar 

  30. Ramanathan A, Athikarisamy SE, Lam GC (2022) Artificial intelligence for the diagnosis of retinopathy of prematurity: a systematic review of current algorithms. Eye, pp 1–9

  31. Ramachandran S, Niyas P, Vinekar A, John R (2021) A deep learning framework for the detection of Plus disease in retinal fundus images of preterm infants. Biocybernetics Biomed Eng 41(2):362–375

    Article  Google Scholar 

  32. Redmon J, Divvala S, Girshick R, Farhadi A (2016) You only look once: Unified, real-time object detection. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 779–788

  33. Tong Y, Lu W, Deng Q-q, Chen C, Shen Y (2020) Automated identification of retinopathy of prematurity by image-based deep learning. Eye and Vision 7:1–12

    Article  Google Scholar 

  34. Girshick R (2015) Fast r-cnn. In: Proceedings of the IEEE international conference on computer vision, pp 1440–1448

  35. Lin T-Y, Dollár P, Girshick R, He K, Hariharan B, Belongie S (2017) Feature pyramid networks for object detection. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 2117–2125

  36. Liu S, Qi L, Qin H, Shi J, Jia J (2018) Path aggregation network for instance segmentation. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 8759–8768

  37. Ganaie MA, Hu M, Malik A, Tanveer M, Suganthan P (2022) Ensemble deep learning: a review. Eng Appl Artif Intell 115:105151

    Article  Google Scholar 

  38. Hu J, Chen Y, Zhong J, Ju R, Yi Z (2019) Automated Analysis for Retinopathy of Prematurity by Deep Neural Networks. IEEE Trans Med Imaging 38(1):269–279

    Article  Google Scholar 

  39. Peng Y, Zhu W, Chen Z, Wang M, Geng L, Yu K, Zhou Y, Wang T, Xiang D, Chen F, Chen X (2021) Automatic staging for retinopathy of prematurity with deep feature fusion and ordinal classification strategy. IEEE Trans Med Imaging 40(7):1750–1762

    Article  Google Scholar 

  40. Tan M, Le Q (2019) Efficientnet: rethinking model scaling for convolutional neural networks. In: International conference on machine learning, pp 6105–6114

  41. Vaswani A, Shazeer N, Parmar N, Uszkoreit J, Jones L, Gomez AN, Kaiser, Ł, Polosukhin I (2017) Attention is all you need. Adv Neural Inf Process Syst 30

  42. Peng Y, Zhu W, Chen Z, Shi F, Wang M, Zhou Y, Wang L, Shen Y, Xiang D, Chen F, Chen X (2022) AFENet: attention fusion enhancement network for optic disc segmentation of premature infants. Frontiers in Neuroscience 16. https://doi.org/10.3389/fnins.2022.836327

  43. Hoshen Y (2017) Vain: attentional multi-agent predictive modeling. Adv Neural Inf Process Syst 30

  44. Kipf T, Fetaya E, Wang K-C, Welling M, Zemel R (2018) Neural relational inference for interacting systems. In: International conference on machine learning (PMLR 2018), pp 2688–2697

  45. Do K, Tran T, Venkatesh S (2019) Graph transformation policy network for chemical reaction prediction. In: Proceedings of the 25th ACM SIGKDD international conference on knowledge discovery & data mining, pp 750–760

  46. Cui Z, Henrickson K, Ke R, Wang Y (2019) Traffic graph convolutional recurrent neural network: a deep learning framework for network-scale traffic learning and forecasting. IEEE Trans Intell Transp Syst 21(11):4883–4894

    Article  Google Scholar 

  47. Zheng C, Fan X, Wang C, Qi J (2020) Gman: a graph multi-attention network for traffic prediction. Proc AAAI Conf Artif Intell 34(01):1234–1241

    Google Scholar 

  48. Guo Shengnan, Lin Youfang, Feng Ning, Song Chao, Wan H (2019) Attention based spatial-temporal graph convolutional networks for traffic flow forecasting. Proc AAAI Conf Artif Intell 33(01):922–929

    Google Scholar 

  49. Kipf TN, Welling M (2016) Semi-supervised classification with graph convolutional networks. arXiv:1609.02907

  50. Velickovic P, Cucurull G, Casanova A, Romero A, Lio P, Bengio Y et al (2017) Graph attention networks. Stat 1050(20):10–48550

    Google Scholar 

  51. Hamilton W, Ying Z, Leskovec J (2017) Inductive representation learning on large graphs. Adv Neural Inf Process Syst 30

  52. Liu S, Gong L, Ma K, Zheng Y (2020) GREEN: a graph residual re-ranking network for grading diabetic retinopathy. In: Medical image computing and computer assisted intervention–MICCAI 2020: 23rd international conference, Lima, Peru, October 4–8, 2020, Proceedings, Part V 23 (Springer, 2020), pp 585–594

  53. Yang H, Zhen X, Chi Y, Zhang L, Hua X-S (2020) Cpr-gcn: conditional partial-residual graph convolutional network in automated anatomical labeling of coronary arteries. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 3803–3811

  54. Simonyan K, Zisserman A (2014) Very deep convolutional networks for large-scale image recognition. arXiv:1409.1556

  55. Srivastava, RK, Greff K, Schmidhuber J (2015) Highway networks. arXiv:1505.00387

  56. Wang L, Lee C-Y, Tu Z, Lazebnik S (2015) Training deeper convolutional networks with deep supervision. arXiv:1505.02496

  57. Lee C-Y, Xie S, Gallagher P, Zhang Z, Tu Z (2015) Deeply-supervised nets. In: Artificial intelligence and statistics, pp 562–570

  58. Zhao H, Shi J, Qi X, Wang X, Jia J (2017) Pyramid scene parsing network. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 2881–2890

  59. Bradski G (2000) The openCV library. Dr. Dobb’s Journal of Software Tools

  60. Stark JA (2000) Adaptive image contrast enhancement using generalizations of histogram equalization. IEEE Trans Image Process 9(5):889–896

    Article  Google Scholar 

  61. Deng J, Dong W, Socher R, Li L-J, Kai L, Li F-F (2009) ImageNet: A large-scale hierarchical image database. In: 2009 IEEE conference on computer vision and pattern recognition, pp 248-255 https://doi.org/10.1109/CVPR.2009.5206848

  62. Herbert ER (2007) A stochastic approximation method. Ann Math Stat 22:400–407

    MathSciNet  MATH  Google Scholar 

  63. Loshchilov I, Hutter F (2017) Decoupled weight decay regularization. arXiv:1711.05101

  64. Paszke A, Gross S, et al. (2019) PyTorch: an imperative style, high-performance deep learning library. In: Wallach H, Larochelle H, Beygelzimer A, d’Alché-Buc F, Fox E, Garnett R (eds) Advances in Neural Information Processing Systems (Curran Associates, Inc., 2019) pp 8024–8035

  65. Xie S, Girshick R, Dollár P, Tu Z, He K (2017) Aggregated residual transformations for deep neural networks. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 1492–1500

  66. Tan M, Chen B, Pang R, Vasudevan V, Sandler M, Howard A, Le QV (2019) Mnasnet: Platform-aware neural architecture search for mobile. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 2820–2828

  67. Koonce B, Koonce B (2021) MobileNetV3. Image Recognition and Dataset Categorization, Convolutional Neural Networks with Swift for Tensorflow, pp 125–144

    Google Scholar 

  68. Ma N, Zhang X, Zheng H-T, Sun J (2018) Shufflenet v2: Practical guidelines for efficient cnn architecture design. In: Proceedings of the European conference on computer vision (ECCV), pp 116–131

Download references

Acknowledgements

This work was supported by the National Major Science and Technology Projects of China under Grant 2018AAA0100201 and the Sichuan Cadre Health Care Project under Grant ZH2019-201.

Author information

Authors and Affiliations

  1. Machine Intelligence Laboratory, College of Computer Science, Sichuan University, Chengdu, Sichuan Province, 610065, China

    Kaide Huang, Yuanyuan Chen & Zhang Yi

  2. Department of Ophthalmology, Sichuan Academy of Medical Sciences and the Sichuan Provincial Peoples Hospital, Chengdu, Sichuan Province, 610072, China

    Wentao Dong, Jie Li & Jie Zhong

  3. School of Medicine, University of Electronic Science and Technology of China, Chengdu, Sichuan Province, 610072, China

    Wentao Dong, Jie Li & Jie Zhong

Authors
  1. Kaide Huang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Wentao Dong
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Jie Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Yuanyuan Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Jie Zhong
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Zhang Yi
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Yuanyuan Chen.

Ethics declarations

Conflicts of interest

The authors declare that they have no conflict of interest.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Kaide Huang and Wentao Dong These authors contributed equally to this work.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Huang, K., Dong, W., Li, J. et al. GFF-Net: Graph-based feature fusion network for diagnosing plus disease in retinopathy of prematurity. Appl Intell 53, 25259–25281 (2023). https://doi.org/10.1007/s10489-023-04766-3

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10489-023-04766-3

Keywords

Search

Navigation