Expertise for auditing AI systems in medical domain is only now being accumulated. Conformity assessment procedures will require AI systems: (1) to be transparent, (2) not to rely decisions solely on algorithms, or (3) to include safety assurance cases in the documentation to facilitate technical audit. We are interested here in obtaining transparency in the case of machine learning (ML) applied to classification of retina conditions. High performance metrics achieved using ML has become common practice. However, in the medical domain, algorithmic decisions need to be sustained by explanations. We aim at building a support tool for ophthalmologists able to: (i) explain algorithmic decision to the human agent by automatically extracting rules from the ML learned models; (ii) include the ophthalmologist in the loop by formalising expert rules and including the expert knowledge in the argumentation machinery; (iii) build safety cases by creating assurance argument patterns for each diagnosis.
For the learning task, we used a dataset consisting of 699 OCT images: 126 Normal class, 210 with Diabetic Retinopathy (DR) and 363 with Age Related Macular Degeneration (AMD). The dataset contains patients from the Ophthalmology Department of the County Emergency Hospital of Cluj-Napoca. All ethical norms and procedures, including anonymisation, have been performed. We applied three machine learning algorithms: decision tree (DT), support vector machine (SVM) and artificial neural network (ANN). For each algorithm we automatically extract diagnosis rules. For formalising expert knowledge, we relied on the normative dataset (Invernizzi et al. in Ophthalmol Retina 2(8):808–815, 2018). For arguing between agents, we used the Jason multi-agent platform. We assume different knowledge base and reasoning capabilities for each agent. The agents have their own optical coherence tomography (OCT) images on which they apply a distinct machine learning algorithm. The learned model is used to extract diagnosis rules. With distinct learned rules, the agents engage in an argumentative process. The resolution of the debate outputs a diagnosis that is then explained to the ophthalmologist, by means of assurance cases.
For diagnosing the retina condition, our AI solution deals with the following three issues: first, the learned models are automatically translated into rules. These rules are then used to build an explanation by tracing the reasoning chain supporting the diagnosis. Hence, the proposed AI solution complies with the requirement that “algorithmic decision should be explained to the human agent”. Second, the decision is not solely based on ML-algorithms. The proposed architecture includes expert knowledge. The diagnosis is taken based on exchanging arguments between ML-based algorithms and expert knowledge. The conflict resolution among arguments is verbalised, so that the ophthalmologist can supervise the diagnosis. Third, the assurance cases are generated to facilitate technical audit. The assurance cases structure the evidence among various safety goals such as: machine learning methodology, transparency, or data quality. For each dimension, the auditor can check the provided evidence against the current best practices or safety standards.
We developed a multi-agent system for retina conditions in which algorithmic decisions are sustained by explanations. The proposed tool goes behind most software in medical domain that focuses only on performance metrics. Our approach helps the technical auditor to approve software in the medical domain. Interleaving knowledge extracted from ML-models with expert knowledge is a step towards balancing the benefits of ML with explainability, aiming at engineering reliable medical applications.
This is a preview of subscription content, access via your institution.
Buy single article
Instant access to the full article PDF.
Tax calculation will be finalised during checkout.
Subscribe to journal
Immediate online access to all issues from 2019. Subscription will auto renew annually.
Tax calculation will be finalised during checkout.
Please contact Simona Delia Nicoara for the OCT images dataset.
De Graaf, M. M., & Malle, B. F. (2017). How people explain action (and autonomous intelligent systems should too). In: Proceedings of the 2017 AAAI Fall Symposium Series.
Adadi, A., & Berrada, M. (2018). Peeking inside the black-box: A survey on explainable artificial intelligence (XAI). IEEE Access, 6, 52138–52160.
Čyras, K., Letsios, D., Misener, R., & Toni, F. (2019). Argumentation for explainable scheduling. Proceedings of the AAAI Conference on Artificial Intelligence, 33, 2752–2759.
Čyras, K., Birch, D., Guo, Y., Toni, F., Dulay, R., Turvey, S., et al. (2019). Explanations by arbitrated argumentative dispute. Expert Systems with Applications, 127, 141–156.
Sassoon, I., Kökciyan, N., Sklar, E., Parsons, S.: Explainable argumentation for wellness consultation. In: Proceedings of the Int. Workshop on Explainable, Transparent Autonomous Agents and Multi-Agent Systems, pp. 186–202. Springer (2019)
Memari, N., Saripan, M. I. B., Mashohor, S., Moghbel, M., et al. (2019). Retinal blood vessel segmentation by using matched filtering and fuzzy c-means clustering with integrated level set method for diabetic retinopathy assessment. Journal of Medical and Biological Engineering, 39(5), 713–731.
Sendi, N., Abchiche-Mimouni, N., & Zehraoui, F. (2019). A new transparent ensemble method based on deep learning. Procedia Computer Science, 159, 271–280.
Marginean, A., Groza, A., Nicoara, S. D., Muntean, G., Slavescu, R. R., & Letia, I. A. (2019). Towards balancing the complexity of convolutional neural network with the role of optical coherence tomography in retinal conditions. In: Proceedings of the 2019 IEEE 15th International Conference on Intelligent Computer Communication and Processing (ICCP), pp. 475–482. IEEE.
Liu, X., Faes, L., Kale, A. U., Wagner, S. K., Fu, D. J., Bruynseels, A., et al. (2019). A comparison of deep learning performance against health-care professionals in detecting diseases from medical imaging: A systematic review and meta-analysis. The Lancet Digital Health, 1(6), e271–e297.
Thulkar, D., Daruwala, R., & Sardar, N. (2020). An integrated system for detection exudates and severity quantification for diabetic macular edema. Journal of Medical and Biological Engineering, 40(6), 798–820.
Zeng, Z., Fan, X., Miao, C., Leung, C., Jih, C. J., & Soon, O. Y. (2018). In: Proceedings of the 17th International Conference on Autonomous Agents and MultiAgent Systems, pp. 1114–1122. International Foundation for Autonomous Agents and Multiagent Systems.
Invernizzi, A., Pellegrini, M., Acquistapace, A., Benatti, E., Erba, S., Cozzi, M., et al. (2018). Normative data for retinal-layer thickness maps generated by spectral-domain oct in a white population. Ophthalmology Retina, 2(8), 808–815.
Coscas, G., Lupidi, M., & Coscas, F. (2016). Heidelberg spectralis optical coherence tomography angiography: Technical aspects. OCT Angiography in Retinal and Macular Diseases, 56, 1–5.
Wang, Z., Lin, J., Wang, R., & Zheng, W. (2019). Data augmentation is more important than model architectures for retinal vessel segmentation. In: Proceedings of the 2019 International Conference on Intelligent Medicine and Health, pp. 48–52.
Zhang, C., Tavanapong, W., Wong, J., de Groen, P.C., Oh, J (2017) Real data augmentation for medical image classification. In: Proceedings of the Intravascular imaging and computer assisted stenting, and large-scale annotation of biomedical data and expert label synthesis, pp. 67–76. Springer.
Russell, S., & Norvig, P. (2020). Artificial intelligence—A modern approach. 4th ed.. Pearson Education, London.
Domingos, P. (2015). The master algorithm: How the quest for the ultimate learning machine will remake our world. New York: Basic Books.
Chang, C. C., & Lin, C. J. (2011). Libsvm: A library for support vector machines. ACM Transactions on Intelligent Systems and Technology (TIST), 2(3), 1–27.
Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V., Thirion, B., Grisel, O., et al. (2011). Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 12, 2825–2830.
Ming, Y., Qu, H., & Bertini, E. (2018). Rulematrix: Visualizing and understanding classifiers with rules. IEEE Transactions on Visualization and Computer Graphics, 25(1), 342–352.
Bordini, R. H., Hübner, J. F., & Wooldridge, M. (2007). Programming multi-agent systems in AgentSpeak using Jason (Vol. 8). Hoboken: Wiley.
Nieves-Moreno, M., Martínez-de-la Casa, J. M., Cifuentes-Canorea, P., Sastre-Ibáñez, M., Santos-Bueso, E., Sáenz-Francés, F., et al. (2017). Normative database for separate inner retinal layers thickness using spectral domain optical coherence tomography in caucasian population. PLoS ONE. https://doi.org/10.1371/journal.pone.0180450.
Chalam, K. V., Bressler, S. B., Edwards, A. R., Berger, B. B., Bressler, N. M., Glassman, A. R., et al. (2012). Retinal thickness in people with diabetes and minimal or no diabetic retinopathy: Heidelberg spectralis optical coherence tomography. Investigative Ophthalmology & Visual Science, 53(13), 8154–8161.
Chan, A., Duker, J. S., Ko, T. H., Fujimoto, J. G., & Schuman, J. S. (2006). Normal macular thickness measurements in healthy eyes using stratus optical coherence tomography. Archives of Ophthalmology, 124(2), 193–198.
Picardi, C., Hawkins, R., Paterson, C., & Habli, I. (2019). A pattern for arguing the assurance of machine learning in medical diagnosis systems. In: Proceedings of the International Conference on Computer Safety, Reliability, and Security, pp. 165–179. Springer.
Kelly, T., & Weaver, R. (2004). The goal structuring notation–a safety argument notation. In: Proceedings of the dependable systems and networks 2004 workshop on assurance cases, p. 6. Citeseer.
Laying down harmonised rules on artificial intelligence (artificial intelligence act) and amending certain union legislative acts. (2021). Tech. rep., European Commission
Guerra-Hernández, A., González-Alarcón, C. A., & El FallahSeghrouchni, A. (2010) Jason induction of logical decision trees: A learning library and its application to commitment. In: Proceedings of the Mexican International Conference on Artificial Intelligence, pp. 374–385. Springer.
van Voorst Vader, P.J.: Multi-agent based military health system for the future battlefield. Master Thesis (2018)
Authors would like to thank Anca Marginean and the anonymous reviewers for their valuable comments and suggestions.
No funding was received for conducting this study.
Conflict of interest
The authors declare that they have no conflict of interest.
About this article
Cite this article
Groza, A., Toderean, L., Muntean, G.A. et al. Agents that Argue and Explain Classifications of Retinal Conditions. J. Med. Biol. Eng. (2021). https://doi.org/10.1007/s40846-021-00647-7
- Explainable artificial intelligence
- Argumentative agents
- Machine learning
- Retina conditions
Mathematics Subject Classification