Knowledge and Information Systems

, Volume 41, Issue 3, pp 647–665 | Cite as

Explaining prediction models and individual predictions with feature contributions

  • Erik ŠtrumbeljEmail author
  • Igor Kononenko
Regular Paper


We present a sensitivity analysis-based method for explaining prediction models that can be applied to any type of classification or regression model. Its advantage over existing general methods is that all subsets of input features are perturbed, so interactions and redundancies between features are taken into account. Furthermore, when explaining an additive model, the method is equivalent to commonly used additive model-specific methods. We illustrate the method’s usefulness with examples from artificial and real-world data sets and an empirical analysis of running times. Results from a controlled experiment with 122 participants suggest that the method’s explanations improved the participants’ understanding of the model.


Knowledge discovery Data mining Visualization  Interpretability  Decision support 


  1. 1.
    Achen CH (1982) Intepreting and Using Regression. Sage Publications, Thousand OaksGoogle Scholar
  2. 2.
    Allahyari H, Lavesson N (2011) User-oriented assessment of classification model understandability. In: Proceedings of the 11th Scandinavian conference on artificial intelligence, SCAI 2011, pp 11–19Google Scholar
  3. 3.
    Becker B, Kohavi R, Sommerfield D (1997) Visualizing the simple Bayesian classier. KDD workshop on issues in the integration of data mining and data visualizationGoogle Scholar
  4. 4.
    Bhattacharya S, Xu D, Kumar K (2011) An ANN-based auditor decision support system using Benford’s law. Decis Support Syst 50(3):576–584CrossRefGoogle Scholar
  5. 5.
    Bhattacharyya S, Jha S, Tharakunnel K, Westland JC (2011) Data mining for credit card fraud: a comparative study. Decis Support Syst 50(3):602–613CrossRefGoogle Scholar
  6. 6.
    Blanchard J, Guillet F, Briand H (2007) Interactive visual exploration of association rules with rule-focusing methodology. Knowl Inf Syst 13:43–75CrossRefGoogle Scholar
  7. 7.
    Castro J, Gómez D, Tejada J (2009) Polynomial calculation of the shapley value based on sampling. Comput Oper Res 36(5):1726–1730CrossRefzbMATHMathSciNetGoogle Scholar
  8. 8.
    De Falco I, Della Cioppa A (2005) An evolutionary approach for automatically extracting intelligible classification rules. Knowl Inf Syst 7:179–201CrossRefGoogle Scholar
  9. 9.
    Frank A, Asuncion A (2011) Uci machine learning repositoryGoogle Scholar
  10. 10.
    Hall M, Frank E, Holmes G, Pfahringer B, Reutemann P, Witten IH (2009) The weka data mining software: an update. SIGKDD Explor Newsl 11(1):10–18CrossRefGoogle Scholar
  11. 11.
    Huang Z, Chen H, Hsu CJ, Chen WH, Wu S (2004) Credit rating analysis with support vector machines and neural networks: a market comparative study. Decis Support Syst 37(4):543–558CrossRefGoogle Scholar
  12. 12.
    Huysmans J, Dejaeger K, Mues C, Vanthienen J, Baesens B (2011) An empirical evaluation of the comprehensibility of decision table, tree and rule based predictive models. Decis Support Syst 51(1):141–154CrossRefGoogle Scholar
  13. 13.
    Jaeckel P (2002) Monte Carlo methods in finance. Wiley, New YorkGoogle Scholar
  14. 14.
    Jakulin A, Možina M, Demšar J, Bratko I, Zupan B (2005) Nomograms for visualizing support vector machines. KDD ’05: 11th ACM SIGKDD, ACM, pp 108–117Google Scholar
  15. 15.
    Kattan MW, Eastham JA, Stapleton AM, Wheeler TM, Scardino PT (1998) A preoperative nomogram for disease recurrence following radical prostatectomy for prostate cancer. J Natl Cancer Inst 90:766–771CrossRefGoogle Scholar
  16. 16.
    Knuth DE (1998) The art of computer programming, volume 2: seminumerical algorithms. Addison-Wesley, BostonGoogle Scholar
  17. 17.
    Kononenko I (1993) Inductive and bayesian learning in medical diagnosis. Appl Artif Intell 7:317–337CrossRefGoogle Scholar
  18. 18.
    Lee S (2010) Using data envelopment analysis and decision trees for efficiency analysis and recommendation of B2C controls. Decis Support Syst 49(4):486–497CrossRefGoogle Scholar
  19. 19.
    Lemaire V, Feraud R, Voisine N (2008) Contact personalization using a score understanding method. In: International joint conference on neural networks (IJCNN)Google Scholar
  20. 20.
    Lim BY, Dey AK, Avrahami D (2009) Why and why not explanations improve the intelligibility of context-aware intelligent systems. In: Proceedings of the 27th international conference on Human factors in computing systems, CHI ’09, ACM, New York, NY, USA, pp 2119–2128Google Scholar
  21. 21.
    Lubsen J, Pool J, van der Does E (1978) A practical device for the application of a diagnostic or prognostic function. Methods Inf Med 17:127–129Google Scholar
  22. 22.
    Melli G (n.d.) The datgen dataset generator.
  23. 23.
    Možina M, Demšar J, Kattan M, Zupan B (2004) Nomograms for visualization of naive Bayesian classifier. PKDD 2004, Springer, pp 337–348Google Scholar
  24. 24.
    Robnik-Šikonja M, Kononenko I (2008) Explaining classifications for individual instances. IEEE TKDE 20:589–600Google Scholar
  25. 25.
    Shapley LS (1953) A value for n-person games, vol II of Contributions to the theory of games. Princeton University Press, PrincetonGoogle Scholar
  26. 26.
    Szafron D, Poulin B, Eisner R, Lu P, Greiner R, Wishart D, Fyshe A, Pearcy B, Macdonell C, Anvik J (2006) Visual explanation of evidence in additive classifiers. In: Proceedings of innovative applications of artificial intelligenceGoogle Scholar
  27. 27.
    Štrumbelj E, Bosnić Z, Zakotnik B, Grašič-Kuhar C, Kononenko I (2010) Explanation and reliability of breast cancer recurrence predictions. Knowl Inf Syst 24(2):305–324CrossRefGoogle Scholar
  28. 28.
    Štrumbelj E, Kononenko I (2010) An efficient explanation of individual classifications using game theory. J Mach Learn Res 11:1–18zbMATHMathSciNetGoogle Scholar
  29. 29.
    Štrumbelj E, Kononenko I (2011) A general method for visualizing and explaining black-box regression models. In: Dobnikar A, Lotric U, Ster B (eds) ICANNGA (2), vol 6594 of Lecture notes in computer science. Springer, Berlin, pp 21–30Google Scholar
  30. 30.
    Welford BP (1962) Note on a method for calculating corrected sums of squares and products. Technometrics 4(3):419–420CrossRefMathSciNetGoogle Scholar

Copyright information

© Springer-Verlag London 2013

Authors and Affiliations

  1. 1.Faculty of Computer and Information ScienceUniversity of LjubljanaLjubljanaSlovenia

Personalised recommendations