Explaining prediction models and individual predictions with feature contributions

Knowledge and Information Systems volume 41, pages647–665(2014)Cite this article

Abstract

We present a sensitivity analysis-based method for explaining prediction models that can be applied to any type of classification or regression model. Its advantage over existing general methods is that all subsets of input features are perturbed, so interactions and redundancies between features are taken into account. Furthermore, when explaining an additive model, the method is equivalent to commonly used additive model-specific methods. We illustrate the method’s usefulness with examples from artificial and real-world data sets and an empirical analysis of running times. Results from a controlled experiment with 122 participants suggest that the method’s explanations improved the participants’ understanding of the model.

This is a preview of subscription content, log in to check access.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5

References

  1. 1.

    Achen CH (1982) Intepreting and Using Regression. Sage Publications, Thousand Oaks

    Google Scholar 

  2. 2.

    Allahyari H, Lavesson N (2011) User-oriented assessment of classification model understandability. In: Proceedings of the 11th Scandinavian conference on artificial intelligence, SCAI 2011, pp 11–19

  3. 3.

    Becker B, Kohavi R, Sommerfield D (1997) Visualizing the simple Bayesian classier. KDD workshop on issues in the integration of data mining and data visualization

  4. 4.

    Bhattacharya S, Xu D, Kumar K (2011) An ANN-based auditor decision support system using Benford’s law. Decis Support Syst 50(3):576–584

    Article  Google Scholar 

  5. 5.

    Bhattacharyya S, Jha S, Tharakunnel K, Westland JC (2011) Data mining for credit card fraud: a comparative study. Decis Support Syst 50(3):602–613

    Article  Google Scholar 

  6. 6.

    Blanchard J, Guillet F, Briand H (2007) Interactive visual exploration of association rules with rule-focusing methodology. Knowl Inf Syst 13:43–75

    Article  Google Scholar 

  7. 7.

    Castro J, Gómez D, Tejada J (2009) Polynomial calculation of the shapley value based on sampling. Comput Oper Res 36(5):1726–1730

    Article  MATH  MathSciNet  Google Scholar 

  8. 8.

    De Falco I, Della Cioppa A (2005) An evolutionary approach for automatically extracting intelligible classification rules. Knowl Inf Syst 7:179–201

    Article  Google Scholar 

  9. 9.

    Frank A, Asuncion A (2011) Uci machine learning repository

  10. 10.

    Hall M, Frank E, Holmes G, Pfahringer B, Reutemann P, Witten IH (2009) The weka data mining software: an update. SIGKDD Explor Newsl 11(1):10–18

    Article  Google Scholar 

  11. 11.

    Huang Z, Chen H, Hsu CJ, Chen WH, Wu S (2004) Credit rating analysis with support vector machines and neural networks: a market comparative study. Decis Support Syst 37(4):543–558

    Article  Google Scholar 

  12. 12.

    Huysmans J, Dejaeger K, Mues C, Vanthienen J, Baesens B (2011) An empirical evaluation of the comprehensibility of decision table, tree and rule based predictive models. Decis Support Syst 51(1):141–154

    Article  Google Scholar 

  13. 13.

    Jaeckel P (2002) Monte Carlo methods in finance. Wiley, New York

    Google Scholar 

  14. 14.

    Jakulin A, Možina M, Demšar J, Bratko I, Zupan B (2005) Nomograms for visualizing support vector machines. KDD ’05: 11th ACM SIGKDD, ACM, pp 108–117

  15. 15.

    Kattan MW, Eastham JA, Stapleton AM, Wheeler TM, Scardino PT (1998) A preoperative nomogram for disease recurrence following radical prostatectomy for prostate cancer. J Natl Cancer Inst 90:766–771

    Article  Google Scholar 

  16. 16.

    Knuth DE (1998) The art of computer programming, volume 2: seminumerical algorithms. Addison-Wesley, Boston

    Google Scholar 

  17. 17.

    Kononenko I (1993) Inductive and bayesian learning in medical diagnosis. Appl Artif Intell 7:317–337

    Article  Google Scholar 

  18. 18.

    Lee S (2010) Using data envelopment analysis and decision trees for efficiency analysis and recommendation of B2C controls. Decis Support Syst 49(4):486–497

    Article  Google Scholar 

  19. 19.

    Lemaire V, Feraud R, Voisine N (2008) Contact personalization using a score understanding method. In: International joint conference on neural networks (IJCNN)

  20. 20.

    Lim BY, Dey AK, Avrahami D (2009) Why and why not explanations improve the intelligibility of context-aware intelligent systems. In: Proceedings of the 27th international conference on Human factors in computing systems, CHI ’09, ACM, New York, NY, USA, pp 2119–2128

  21. 21.

    Lubsen J, Pool J, van der Does E (1978) A practical device for the application of a diagnostic or prognostic function. Methods Inf Med 17:127–129

    Google Scholar 

  22. 22.

    Melli G (n.d.) The datgen dataset generator. http://www.datasetgenerator.com

  23. 23.

    Možina M, Demšar J, Kattan M, Zupan B (2004) Nomograms for visualization of naive Bayesian classifier. PKDD 2004, Springer, pp 337–348

  24. 24.

    Robnik-Šikonja M, Kononenko I (2008) Explaining classifications for individual instances. IEEE TKDE 20:589–600

    Google Scholar 

  25. 25.

    Shapley LS (1953) A value for n-person games, vol II of Contributions to the theory of games. Princeton University Press, Princeton

  26. 26.

    Szafron D, Poulin B, Eisner R, Lu P, Greiner R, Wishart D, Fyshe A, Pearcy B, Macdonell C, Anvik J (2006) Visual explanation of evidence in additive classifiers. In: Proceedings of innovative applications of artificial intelligence

  27. 27.

    Štrumbelj E, Bosnić Z, Zakotnik B, Grašič-Kuhar C, Kononenko I (2010) Explanation and reliability of breast cancer recurrence predictions. Knowl Inf Syst 24(2):305–324

    Article  Google Scholar 

  28. 28.

    Štrumbelj E, Kononenko I (2010) An efficient explanation of individual classifications using game theory. J Mach Learn Res 11:1–18

    MATH  MathSciNet  Google Scholar 

  29. 29.

    Štrumbelj E, Kononenko I (2011) A general method for visualizing and explaining black-box regression models. In: Dobnikar A, Lotric U, Ster B (eds) ICANNGA (2), vol 6594 of Lecture notes in computer science. Springer, Berlin, pp 21–30

  30. 30.

    Welford BP (1962) Note on a method for calculating corrected sums of squares and products. Technometrics 4(3):419–420

    Article  MathSciNet  Google Scholar 

Download references

Author information

Affiliations

  1. Faculty of Computer and Information Science, University of Ljubljana, Tržaška 25, 1000, Ljubljana, Slovenia

    Erik Štrumbelj & Igor Kononenko

Authors
  1. Erik Štrumbelj
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Igor Kononenko
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Erik Štrumbelj.

Rights and permissions

Reprints and Permissions

About this article

Cite this article

Štrumbelj, E., Kononenko, I. Explaining prediction models and individual predictions with feature contributions. Knowl Inf Syst 41, 647–665 (2014). https://doi.org/10.1007/s10115-013-0679-x

Download citation

Keywords

  • Knowledge discovery
  • Data mining
  • Visualization
  • Interpretability
  • Decision support