Skip to main content

Agents that Argue and Explain Classifications of Retinal Conditions



Expertise for auditing AI systems in medical domain is only now being accumulated. Conformity assessment procedures will require AI systems: (1) to be transparent, (2) not to rely decisions solely on algorithms, or (3) to include safety assurance cases in the documentation to facilitate technical audit. We are interested here in obtaining transparency in the case of machine learning (ML) applied to classification of retina conditions. High performance metrics achieved using ML has become common practice. However, in the medical domain, algorithmic decisions need to be sustained by explanations. We aim at building a support tool for ophthalmologists able to: (i) explain algorithmic decision to the human agent by automatically extracting rules from the ML learned models; (ii) include the ophthalmologist in the loop by formalising expert rules and including the expert knowledge in the argumentation machinery; (iii) build safety cases by creating assurance argument patterns for each diagnosis.


For the learning task, we used a dataset consisting of 699 OCT images: 126 Normal class, 210 with Diabetic Retinopathy (DR) and 363 with Age Related Macular Degeneration (AMD). The dataset contains patients from the Ophthalmology Department of the County Emergency Hospital of Cluj-Napoca. All ethical norms and procedures, including anonymisation, have been performed. We applied three machine learning algorithms: decision tree (DT), support vector machine (SVM) and artificial neural network (ANN). For each algorithm we automatically extract diagnosis rules. For formalising expert knowledge, we relied on the normative dataset (Invernizzi et al. in Ophthalmol Retina 2(8):808–815, 2018). For arguing between agents, we used the Jason multi-agent platform. We assume different knowledge base and reasoning capabilities for each agent. The agents have their own optical coherence tomography (OCT) images on which they apply a distinct machine learning algorithm. The learned model is used to extract diagnosis rules. With distinct learned rules, the agents engage in an argumentative process. The resolution of the debate outputs a diagnosis that is then explained to the ophthalmologist, by means of assurance cases.


For diagnosing the retina condition, our AI solution deals with the following three issues: first, the learned models are automatically translated into rules. These rules are then used to build an explanation by tracing the reasoning chain supporting the diagnosis. Hence, the proposed AI solution complies with the requirement that “algorithmic decision should be explained to the human agent”. Second, the decision is not solely based on ML-algorithms. The proposed architecture includes expert knowledge. The diagnosis is taken based on exchanging arguments between ML-based algorithms and expert knowledge. The conflict resolution among arguments is verbalised, so that the ophthalmologist can supervise the diagnosis. Third, the assurance cases are generated to facilitate technical audit. The assurance cases structure the evidence among various safety goals such as: machine learning methodology, transparency, or data quality. For each dimension, the auditor can check the provided evidence against the current best practices or safety standards.


We developed a multi-agent system for retina conditions in which algorithmic decisions are sustained by explanations. The proposed tool goes behind most software in medical domain that focuses only on performance metrics. Our approach helps the technical auditor to approve software in the medical domain. Interleaving knowledge extracted from ML-models with expert knowledge is a step towards balancing the benefits of ML with explainability, aiming at engineering reliable medical applications.

This is a preview of subscription content, access via your institution.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5

Data Availability

Please contact Simona Delia Nicoara for the OCT images dataset.

Code Availability


  1. 1.

    De Graaf, M. M., & Malle, B. F. (2017). How people explain action (and autonomous intelligent systems should too). In: Proceedings of the 2017 AAAI Fall Symposium Series.

  2. 2.

    Adadi, A., & Berrada, M. (2018). Peeking inside the black-box: A survey on explainable artificial intelligence (XAI). IEEE Access, 6, 52138–52160.

    Article  Google Scholar 

  3. 3.

    Čyras, K., Letsios, D., Misener, R., & Toni, F. (2019). Argumentation for explainable scheduling. Proceedings of the AAAI Conference on Artificial Intelligence, 33, 2752–2759.

    Article  Google Scholar 

  4. 4.

    Čyras, K., Birch, D., Guo, Y., Toni, F., Dulay, R., Turvey, S., et al. (2019). Explanations by arbitrated argumentative dispute. Expert Systems with Applications, 127, 141–156.

    Article  Google Scholar 

  5. 5.

    Sassoon, I., Kökciyan, N., Sklar, E., Parsons, S.: Explainable argumentation for wellness consultation. In: Proceedings of the Int. Workshop on Explainable, Transparent Autonomous Agents and Multi-Agent Systems, pp. 186–202. Springer (2019)

  6. 6.

    Memari, N., Saripan, M. I. B., Mashohor, S., Moghbel, M., et al. (2019). Retinal blood vessel segmentation by using matched filtering and fuzzy c-means clustering with integrated level set method for diabetic retinopathy assessment. Journal of Medical and Biological Engineering, 39(5), 713–731.

    Article  Google Scholar 

  7. 7.

    Sendi, N., Abchiche-Mimouni, N., & Zehraoui, F. (2019). A new transparent ensemble method based on deep learning. Procedia Computer Science, 159, 271–280.

    Article  Google Scholar 

  8. 8.

    Marginean, A., Groza, A., Nicoara, S. D., Muntean, G., Slavescu, R. R., & Letia, I. A. (2019). Towards balancing the complexity of convolutional neural network with the role of optical coherence tomography in retinal conditions. In: Proceedings of the 2019 IEEE 15th International Conference on Intelligent Computer Communication and Processing (ICCP), pp. 475–482. IEEE.

  9. 9.

    Liu, X., Faes, L., Kale, A. U., Wagner, S. K., Fu, D. J., Bruynseels, A., et al. (2019). A comparison of deep learning performance against health-care professionals in detecting diseases from medical imaging: A systematic review and meta-analysis. The Lancet Digital Health, 1(6), e271–e297.

    Article  Google Scholar 

  10. 10.

    Thulkar, D., Daruwala, R., & Sardar, N. (2020). An integrated system for detection exudates and severity quantification for diabetic macular edema. Journal of Medical and Biological Engineering, 40(6), 798–820.

    Article  Google Scholar 

  11. 11.

    Zeng, Z., Fan, X., Miao, C., Leung, C., Jih, C. J., & Soon, O. Y. (2018). In: Proceedings of the 17th International Conference on Autonomous Agents and MultiAgent Systems, pp. 1114–1122. International Foundation for Autonomous Agents and Multiagent Systems.

  12. 12.

    Invernizzi, A., Pellegrini, M., Acquistapace, A., Benatti, E., Erba, S., Cozzi, M., et al. (2018). Normative data for retinal-layer thickness maps generated by spectral-domain oct in a white population. Ophthalmology Retina, 2(8), 808–815.

    Article  Google Scholar 

  13. 13.

    Coscas, G., Lupidi, M., & Coscas, F. (2016). Heidelberg spectralis optical coherence tomography angiography: Technical aspects. OCT Angiography in Retinal and Macular Diseases, 56, 1–5.

    Article  Google Scholar 

  14. 14.

    Wang, Z., Lin, J., Wang, R., & Zheng, W. (2019). Data augmentation is more important than model architectures for retinal vessel segmentation. In: Proceedings of the 2019 International Conference on Intelligent Medicine and Health, pp. 48–52.

  15. 15.

    Zhang, C., Tavanapong, W., Wong, J., de Groen, P.C., Oh, J (2017) Real data augmentation for medical image classification. In: Proceedings of the Intravascular imaging and computer assisted stenting, and large-scale annotation of biomedical data and expert label synthesis, pp. 67–76. Springer.

  16. 16.

    Russell, S., & Norvig, P. (2020). Artificial intelligence—A modern approach. 4th ed.. Pearson Education, London.

  17. 17.

    Domingos, P. (2015). The master algorithm: How the quest for the ultimate learning machine will remake our world. New York: Basic Books.

    Google Scholar 

  18. 18.

    Chang, C. C., & Lin, C. J. (2011). Libsvm: A library for support vector machines. ACM Transactions on Intelligent Systems and Technology (TIST), 2(3), 1–27.

    Article  Google Scholar 

  19. 19.

    Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V., Thirion, B., Grisel, O., et al. (2011). Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 12, 2825–2830.

    Google Scholar 

  20. 20.

    Ming, Y., Qu, H., & Bertini, E. (2018). Rulematrix: Visualizing and understanding classifiers with rules. IEEE Transactions on Visualization and Computer Graphics, 25(1), 342–352.

    Article  Google Scholar 

  21. 21.

    Bordini, R. H., Hübner, J. F., & Wooldridge, M. (2007). Programming multi-agent systems in AgentSpeak using Jason (Vol. 8). Hoboken: Wiley.

    Book  Google Scholar 

  22. 22.

    Nieves-Moreno, M., Martínez-de-la Casa, J. M., Cifuentes-Canorea, P., Sastre-Ibáñez, M., Santos-Bueso, E., Sáenz-Francés, F., et al. (2017). Normative database for separate inner retinal layers thickness using spectral domain optical coherence tomography in caucasian population. PLoS ONE.

    Article  PubMed  PubMed Central  Google Scholar 

  23. 23.

    Chalam, K. V., Bressler, S. B., Edwards, A. R., Berger, B. B., Bressler, N. M., Glassman, A. R., et al. (2012). Retinal thickness in people with diabetes and minimal or no diabetic retinopathy: Heidelberg spectralis optical coherence tomography. Investigative Ophthalmology & Visual Science, 53(13), 8154–8161.

    Article  Google Scholar 

  24. 24.

    Chan, A., Duker, J. S., Ko, T. H., Fujimoto, J. G., & Schuman, J. S. (2006). Normal macular thickness measurements in healthy eyes using stratus optical coherence tomography. Archives of Ophthalmology, 124(2), 193–198.

    Article  Google Scholar 

  25. 25.

    Picardi, C., Hawkins, R., Paterson, C., & Habli, I. (2019). A pattern for arguing the assurance of machine learning in medical diagnosis systems. In: Proceedings of the International Conference on Computer Safety, Reliability, and Security, pp. 165–179. Springer.

  26. 26.

    Kelly, T., & Weaver, R. (2004). The goal structuring notation–a safety argument notation. In: Proceedings of the dependable systems and networks 2004 workshop on assurance cases, p. 6. Citeseer.

  27. 27.

    Laying down harmonised rules on artificial intelligence (artificial intelligence act) and amending certain union legislative acts. (2021). Tech. rep., European Commission

  28. 28.

    Guerra-Hernández, A., González-Alarcón, C. A., & El FallahSeghrouchni, A. (2010) Jason induction of logical decision trees: A learning library and its application to commitment. In: Proceedings of the Mexican International Conference on Artificial Intelligence, pp. 374–385. Springer.

  29. 29.

    van Voorst Vader, P.J.: Multi-agent based military health system for the future battlefield. Master Thesis (2018)

Download references


Authors would like to thank Anca Marginean and the anonymous reviewers for their valuable comments and suggestions.


No funding was received for conducting this study.

Author information




AG: conceptualization, methodology, writing—original draft, writing—review & editing; LT: software, investigation, writing—original draft; GAM: investigation, writing—original draft; SDN: supervision, resources, writing—original draft.

Corresponding author

Correspondence to Adrian Groza.

Ethics declarations

Conflict of interest

The authors declare that they have no conflict of interest.

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Groza, A., Toderean, L., Muntean, G.A. et al. Agents that Argue and Explain Classifications of Retinal Conditions. J. Med. Biol. Eng. (2021).

Download citation


  • Explainable artificial intelligence
  • Argumentative agents
  • Machine learning
  • AgentSpeak
  • Retina conditions

Mathematics Subject Classification

  • 68T42
  • 68T05
  • 68T30