Visualizing the Feature Importance for Black Box Models
In recent years, a large amount of model-agnostic methods to improve the transparency, trustability, and interpretability of machine learning models have been developed. Based on a recent method for model-agnostic global feature importance, we introduce a local feature importance measure for individual observations and propose two visual tools: partial importance (PI) and individual conditional importance (ICI) plots which visualize how changes in a feature affect the model performance on average, as well as for individual observations. Our proposed methods are related to partial dependence (PD) and individual conditional expectation (ICE) plots, but visualize the expected (conditional) feature importance instead of the expected (conditional) prediction. Furthermore, we show that averaging ICI curves across observations yields a PI curve, and integrating the PI curve with respect to the distribution of the considered feature results in the global feature importance. Another contribution of our paper is the Shapley feature importance, which fairly distributes the overall performance of a model among the features according to the marginal contributions and which can be used to compare the feature importance across different models. Code related to this paper is available at: https://github.com/giuseppec/featureImportance.
KeywordsInterpretable machine learning Explainable AI Feature importance Variable importance Feature effect Partial dependence
This work is funded by the Bavarian State Ministry of Education, Science and the Arts in the framework of the Centre Digitisation. Bavaria (ZD.B).
- 5.Casalicchio, G., et al.: OpenML: an R package to connect to the machine learning platform OpenML. Comput. Stat. 1–15 (2017). https://doi.org/10.1007/s00180-017-0742-2
- 8.Datta, A., Sen, S., Zick, Y.: Algorithmic transparency via quantitative input influence: theory and experiments with learning systems. In: Proceedings of IEEE Symposium on Security and Privacy, SP, pp. 598–617 (2016)Google Scholar
- 9.Fisher, A., Rudin, C., Dominici, F.: Model class reliance: variable importance measures for any machine learning model class, from the "Rashomon" perspective (2018). arXiv preprint arXiv:1801.01489
- 10.Friedman, J.H.: Greedy function approximation: a gradient boosting machine. Annals of Statistics, pp. 1189–1232 (2001)Google Scholar
- 14.Lipton, Z.C.: The mythos of model interpretability. In: ICML WHI 2016 (2016)Google Scholar
- 15.Lundberg, S.M., Erion, G.G., Lee, S.I.: Consistent individualized feature attribution for tree ensembles (2018). arXiv preprint arXiv:1802.03888
- 16.Lundberg, S.M., Lee, S.I.: A unified approach to interpreting model predictions. In: NIPS, vol. 30, pp. 4765–4774. Curran Associates, Inc., Red Hook (2017)Google Scholar