Skip to main content

Visual Explanations for the Detection of Diabetic Retinopathy from Retinal Fundus Images

  • Conference paper
  • First Online:
Medical Image Computing and Computer Assisted Intervention – MICCAI 2022 (MICCAI 2022)


In medical image classification tasks like the detection of diabetic retinopathy from retinal fundus images, it is highly desirable to get visual explanations for the decisions of black-box deep neural networks (DNNs). However, gradient-based saliency methods often fail to highlight the diseased image regions reliably. On the other hand, adversarially robust models have more interpretable gradients than plain models but suffer typically from a significant drop in accuracy, which is unacceptable for clinical practice. Here, we show that one can get the best of both worlds by ensembling a plain and an adversarially robust model: maintaining high accuracy but having improved visual explanations. Also, our ensemble produces meaningful visual counterfactuals which are complementary to existing saliency-based techniques. Code is available under

V. Boreiko and I. Ilanchezian–Equal Contribution.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
USD 39.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions


  1. 1.


  1. Kaggle competition on diabetic retinopathy detection (2015). Accessed 02 Feb 2022

  2. Arun, N., et al.: Assessing the trustworthiness of saliency maps for localizing abnormalities in medical imaging. Radiol. Artif. Intell. 3(6), e200267 (2021)

    Google Scholar 

  3. Augustin, M., Meinke, A., Hein, M.: Adversarial robustness on in- and out-distribution improves explainability. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, J.-M. (eds.) ECCV 2020. LNCS, vol. 12371, pp. 228–245. Springer, Cham (2020).

    Chapter  Google Scholar 

  4. Ayhan, M.S., Kühlewein, L., Aliyeva, G., Inhoffen, W., Ziemssen, F., Berens, P.: Expert-validated estimation of diagnostic uncertainty for deep neural networks in diabetic retinopathy detection. Med. Image Anal. 64, 101724 (2020)

    Google Scholar 

  5. Ayhan, M.S., et al.: Clinical validation of saliency maps for understanding deep neural networks in ophthalmology. Med. Image Anal. 77, 102364 (2022)

    Google Scholar 

  6. Boreiko, V., Augustin, M., Croce, F., Berens, P., Hein, M.: Sparse visual counterfactual explanations in image space. arXiv preprint arXiv:2205.07972 (2022)

  7. Croce, F., Hein, M.: Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attacks. In: ICML (2020)

    Google Scholar 

  8. Croce, F., Hein, M.: Mind the box: \(l_1\)-APGD for sparse adversarial attacks on image classifiers. In: ICML (2021)

    Google Scholar 

  9. Decencière, E., et al.: Feedback on a publicly distributed database: the Messidor database. Image Anal. Stereol. 33(3), 231–234 (2014).

    Article  MATH  Google Scholar 

  10. Etmann, C., Lunz, S., Maass, P., Schönlieb, C.B.: On the connection between adversarial robustness and saliency map interpretability. In: ICML (2019)

    Google Scholar 

  11. Geirhos, R., Rubisch, P., Michaelis, C., Bethge, M., Wichmann, F.A., Brendel, W.: ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness. In: ICLR (2019)

    Google Scholar 

  12. Geirhos, R., et al.: Shortcut learning in deep neural networks. Nat. Mach. Intell. 2(11), 665–673 (2020)

    Article  Google Scholar 

  13. González-Gonzalo, C., Liefers, B., van Ginneken, B., Sánchez, C.I.: Iterative augmentation of visual evidence for weakly-supervised lesion localization in deep interpretability frameworks. IEEE Trans. Med. Imaging (2019)

    Google Scholar 

  14. González-Gonzalo, C., et al.: Trustworthy AI: closing the gap between development and integration of AI systems in ophthalmic practice. Prog. Retinal Eye Res., 101034 (2021)

    Google Scholar 

  15. Grote, T., Berens, P.: On the ethics of algorithmic decision-making in healthcare. J. Med. Ethics 46(3), 205–211 (2020)

    Article  Google Scholar 

  16. Guo, C., Pleiss, G., Sun, Y., Weinberger, K.Q.: On calibration of modern neural networks. In: ICML (2017)

    Google Scholar 

  17. He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: CVPR. pp. 770–778 (2016)

    Google Scholar 

  18. Ilanchezian, I., Kobak, D., Faber, H., Ziemssen, F., Berens, P., Ayhan, M.S.: Interpretable gender classification from retinal fundus images using BagNets. In: de Bruijne, M., et al. (eds.) MICCAI 2021. LNCS, vol. 12903, pp. 477–487. Springer, Cham (2021).

    Chapter  Google Scholar 

  19. Jaggi, M.: Revisiting Frank-Wolfe: projection-free sparse convex optimization. In: ICML (2013)

    Google Scholar 

  20. Lang, O., et al.: Explaining in style: training a GAN to explain a classifier in stylespace. arXiv preprint arXiv:2104.13369 (2021)

  21. Liu, X., et al.: A comparison of deep learning performance against health-care professionals in detecting diseases from medical imaging: a systematic review and meta-analysis. Lancet Digit. Health 1(6), e271–e297 (2019)

    Article  Google Scholar 

  22. Madry, A., Makelov, A., Schmidt, L., Tsipras, D., Vladu, A.: Towards deep learning models resistant to adversarial attacks. In: ICLR (2018)

    Google Scholar 

  23. Margeloiu, A., Simidjievski, N., Jamnik, M., Weller, A.: Improving interpretability in medical imaging diagnosis using adversarial training. arXiv preprint arXiv:2012.01166 (2020)

  24. Moraru, V.: An algorithm for solving quadratic programming problems. Comput. Sci. J. Moldova (1997)

    Google Scholar 

  25. Ozbulak, U.: PyTorch CNN visualizations. (2019)

  26. Porwal, P., et al.: Indian diabetic retinopathy image dataset (IDRiD): a database for diabetic retinopathy screening research. Data 3(3), 25 (2018)

    Article  Google Scholar 

  27. Rudin, C.: Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead. Nat. Mach. Intell. 1(5), 206–215 (2019)

    Article  Google Scholar 

  28. Saporta, A., et al.: Deep learning saliency maps do not accurately highlight diagnostically relevant regions for medical image interpretation. medRxiv (2021)

    Google Scholar 

  29. Springenberg, J.T., Dosovitskiy, A., Brox, T., Riedmiller, M.: Striving for simplicity: the all convolutional net. In: ICLR (Workshop Track) (2014)

    Google Scholar 

  30. Sundararajan, M., Taly, A., Yan, Q.: Axiomatic attribution for deep networks. In: ICML (2017)

    Google Scholar 

  31. Tan, M., Le, Q.: EfficientNet: rethinking model scaling for convolutional neural networks. In: ICML (2019)

    Google Scholar 

  32. Tsipras, D., Santurkar, S., Engstrom, L., Turner, A., Madry, A.: Robustness may be at odds with accuracy. In: ICLR (2019)

    Google Scholar 

  33. Van Craenendonck, T., Elen, B., Gerrits, N., De Boever, P.: Systematic comparison of heatmapping techniques in deep learning in the context of diabetic retinopathy lesion detection. Transl. Vis. Sci. Technol. 9(2), 64–64 (2020).

  34. Younis, N., Broadbent, D.M., Vora, J.P., Harding, S.P.: Incidence of sight-threatening retinopathy in patients with type 2 diabetes in the liverpool diabetic eye study: a cohort study. Lancet 361(9353), 195–200 (2003)

    Article  Google Scholar 

  35. Zhang, H., Yu, Y., Jiao, J., Xing, E.P., Ghaoui, L.E., Jordan, M.I.: Theoretically principled trade-off between robustness and accuracy. In: ICML (2019)

    Google Scholar 

  36. Zuiderveld, K.: Contrast limited adaptive histogram equalization. Graph. Gems, 474–485 (1994)

    Google Scholar 

Download references


We acknowledge support by the German Ministry of Science and Education (BMBF, 01GQ1601 and 01IS18039A) and the German Science Foundation (BE5601/8-1 and EXC 2064, project number 390727645). The authors thank the International Max Planck Research School for Intelligent Systems (IMPRS-IS) for supporting I.I.

Author information

Authors and Affiliations


Corresponding author

Correspondence to Valentyn Boreiko .

Editor information

Editors and Affiliations

1 Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary material 1 (pdf 3117 KB)

Supplementary material 2 (mp4 2423 KB)

Supplementary material 3 (mp4 2239 KB)

Rights and permissions

Reprints and permissions

Copyright information

© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Boreiko, V. et al. (2022). Visual Explanations for the Detection of Diabetic Retinopathy from Retinal Fundus Images. In: Wang, L., Dou, Q., Fletcher, P.T., Speidel, S., Li, S. (eds) Medical Image Computing and Computer Assisted Intervention – MICCAI 2022. MICCAI 2022. Lecture Notes in Computer Science, vol 13432. Springer, Cham.

Download citation

  • DOI:

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-16433-0

  • Online ISBN: 978-3-031-16434-7

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics