Explaining Multi-label Black-Box Classifiers for Health Applications

  • Cecilia PaniguttiEmail author
  • Riccardo Guidotti
  • Anna Monreale
  • Dino Pedreschi
Part of the Studies in Computational Intelligence book series (SCI, volume 843)


Today the state-of-the-art performance in classification is achieved by the so-called “black boxes”, i.e. decision-making systems whose internal logic is obscure. Such models could revolutionize the health-care system, however their deployment in real-world diagnosis decision support systems is subject to several risks and limitations due to the lack of transparency. The typical classification problem in health-care requires a multi-label approach since the possible labels are not mutually exclusive, e.g. diagnoses. We propose MARLENA, a model-agnostic method which explains multi-label black box decisions. MARLENA explains an individual decision in three steps. First, it generates a synthetic neighborhood around the instance to be explained using a strategy suitable for multi-label decisions. It then learns a decision tree on such neighborhood and finally derives from it a decision rule that explains the black box decision. Our experiments show that MARLENA performs well in terms of mimicking the black box behavior while gaining at the same time a notable amount of interpretability through compact decision rules, i.e. rules with limited length.



This work is partially supported by the European H2020 Program under the funding scheme “INFRAIA-1-2014-2015: Research Infrastructures” g.a. 654024 “SoBigData”,


  1. 1.
    Abe, S.: Fuzzy support vector machines for multilabel classification. PR 48(6), 2110 (2015)CrossRefGoogle Scholar
  2. 2.
    Bai, T., et al.: Interpretable representation learning for healthcare via capturing disease progression through time. In: KDD, pp. 43–51. ACM (2018)Google Scholar
  3. 3.
    Blockeel, H., Schietgat, L., Struyf, J., Clare, A., Dzeroski, S.: Hierarchical multilabel classification trees for gene function prediction. In: MLSB, pp. 9–14 (2006)Google Scholar
  4. 4.
    Che, Z., Kale, D., Li, W., Bahadori, M.T., Liu, Y.: Deep computational phenotyping. In: KDD, pp. 507–516. ACM (2015)Google Scholar
  5. 5.
    Che, Z., Purushotham, S., Khemani, R., Liu, Y.: Interpretable deep models for ICU outcome prediction. In: AMIA Annual Symposium Proceedings, vol. 2016, p. 371. American Medical Informatics Association (2017)Google Scholar
  6. 6.
    Choi, E., et al.: Doctor AI: predicting clinical events via recurrent neural networks. In: Machine Learning for Healthcare Conference, pp. 301–318 (2016)Google Scholar
  7. 7.
    Chui, M.: Artificial intelligence the next digital frontier? McKinsey and CGI, p. 47 (2017)Google Scholar
  8. 8.
    Elisseeff, A., Weston, J.: A kernel method for multi-labelled classification. In: Advances in Neural Information Processing Systems, pp. 681–687 (2002)Google Scholar
  9. 9.
    Feng, Y., et al.: Patient outcome prediction via convolutional neural networks based on multi-granularity medical concept embedding. In: BIBM, pp. 770–777. IEEE (2017)Google Scholar
  10. 10.
    Guidotti, R., Monreale, A., Ruggieri, S., Pedreschi, D., Turini, F., Giannotti, F.: Local rule-based explanations of black box decision systems. CoRR (2018). arXiv:abs/1805.10820
  11. 11.
    Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Pedreschi, D., Giannotti, F.: A survey of methods for explaining black box models. ACM CSUR 51(5), 93:1–93:42 (2018)CrossRefGoogle Scholar
  12. 12.
    Guidotti, R., Soldani, J., Neri, D., Brogi, A., Pedreschi, D.: Helping your Docker images to spread based on explainable models. In: ECML-PKDD. Springer, Berlin (2018)Google Scholar
  13. 13.
    Lasko, T.A., et al.: Computational phenotype discovery using unsupervised feature learning over noisy, sparse, and irregular clinical data. PloS One 8(6), e66341 (2013)CrossRefGoogle Scholar
  14. 14.
    LeCun, Y., Bengio, Y., Hinton, G.: Deep learning. Nature 521(7553), 436 (2015)CrossRefGoogle Scholar
  15. 15.
    Malgieri, G., Comandé, G.: Why a right to legibility of automated decision-making exists in the general data protection regulation. Int. Data Priv. Law 7(4), 243–265 (2017)CrossRefGoogle Scholar
  16. 16.
    Miotto, R., et al.: Deep patient: an unsupervised representation to predict the future of patients from the electronic health records. Sci. Rep. 6, 26094 (2016)CrossRefGoogle Scholar
  17. 17.
    Pestian, J.P., et al.: A shared task involving multi-label classification of clinical free text. In: BioNLP, pp. 97–104. Association for Computational Linguistics (2007)Google Scholar
  18. 18.
    Rajkomar, A., et al.: Scalable and accurate deep learning with EHR. DM 1(1), 18 (2018)Google Scholar
  19. 19.
    Ribeiro, M.T., Singh, S., Guestrin, C.: Why should i trust you?: explaining the predictions of any classifier. In: KDD, pp. 1135–1144. ACM (2016)Google Scholar
  20. 20.
    Sapozhnikova, E.P.: Art-based neural networks for multi-label classification. In: International Symposium on Intelligent Data Analysis, pp. 167–177. Springer, Berlin (2009)CrossRefGoogle Scholar
  21. 21.
    Shickel, B., et al.: Deep EHR: a survey of recent advances in deep learning techniques for EHR analysis. J. Biomed. Health Inform. 22(5), 1589–1604 (2018)CrossRefGoogle Scholar
  22. 22.
    Tan, P.-N. et al.: Introduction to data mining. Pearson Education India (2007)Google Scholar
  23. 23.
    Tsoumakas, G., Katakis, I.: Multi-label classification: an overview. Int. J. Data Warehous. Min. (IJDWM) 3(3), 1–13 (2007)CrossRefGoogle Scholar
  24. 24.
    Vens, C., Struyf, J., Schietgat, L., Džeroski, S., Blockeel, H.: Decision trees for hierarchical multi-label classification. Mach. Learn. 73(2), 185 (2008)CrossRefGoogle Scholar
  25. 25.
    Wachter, S., et al.: Why a right to explanation of automated decision-making does not exist in the general data protection regulation. Int. Data Priv. Law 7(2), 76–99 (2017)CrossRefGoogle Scholar
  26. 26.
    Yadav, P., Steinbach, M., Kumar, V., Simon, G.: Mining electronic health records (EHRS): a survey. ACM Comput. Surv. (CSUR) 50(6), 85 (2018)CrossRefGoogle Scholar

Copyright information

© Springer Nature Switzerland AG 2020

Authors and Affiliations

  • Cecilia Panigutti
    • 1
    Email author
  • Riccardo Guidotti
    • 2
    • 3
  • Anna Monreale
    • 3
  • Dino Pedreschi
    • 3
  1. 1.Scuola Normale SuperiorePisaItaly
  2. 2.ISTI-CNRPisaItaly
  3. 3.University of PisaPisaItaly

Personalised recommendations