Advertisement

Explaining Image Classifiers Using Statistical Fault Localization

  • Youcheng SunEmail author
  • Hana Chockler
  • Xiaowei Huang
  • Daniel Kroening
Conference paper
First Online:
  • 236 Downloads
Part of the Lecture Notes in Computer Science book series (LNCS, volume 12373)

Abstract

The black-box nature of deep neural networks (DNNs) makes it impossible to understand why a particular output is produced, creating demand for “Explainable AI”. In this paper, we show that statistical fault localization (SFL) techniques from software engineering deliver high quality explanations of the outputs of DNNs, where we define an explanation as a minimal subset of features sufficient for making the same decision as for the original input. We present an algorithm and a tool called DeepCover, which synthesizes a ranking of the features of the inputs using SFL and constructs explanations for the decisions of the DNN based on this ranking. We compare explanations produced by DeepCover with those of the state-of-the-art tools gradcam, lime, shap, rise and extremal and show that explanations generated by DeepCover are consistently better across a broad set of experiments. On a benchmark set with known ground truth, DeepCover achieves \(76.7\%\) accuracy, which is \(6\%\) better than the second best extremal.

Keywords

Deep learning Explainability Statistical fault localization Software testing 
This is a preview of subscription content, log in to check access.

References

  1. 1.
    Adebayo, J., Gilmer, J., Muelly, M., Goodfellow, I., Hardt, M., Kim, B.: Sanity checks for saliency maps. In: Advances in Neural Information Processing Systems, pp. 9505–9515 (2018)Google Scholar
  2. 2.
    Chajewska, U., Halpern, J.Y.: Defining explanation in probabilistic systems. In: Uncertainty in Artificial Intelligence (UAI), pp. 62–71. Morgan Kaufmann (1997)Google Scholar
  3. 3.
    Chen, J., Song, L., Wainwright, M., Jordan, M.: Learning to explain: an information-theoretic perspective on model interpretation. In: International Conference on Machine Learning (ICML), vol. 80, pp. 882–891. PMLR (2018)Google Scholar
  4. 4.
    Datta, A., Sen, S., Zick, Y.: Algorithmic transparency via quantitative input influence: theory and experiments with learning systems. In: Security and Privacy (S&P), pp. 598–617. IEEE (2016)Google Scholar
  5. 5.
    Eiter, T., Lukasiewicz, T.: Complexity results for explanations in the structural-model approach. Artif. Intell. 154(1–2), 145–198 (2004)MathSciNetCrossRefGoogle Scholar
  6. 6.
    Fong, R., Patrick, M., Vedaldi, A.: Understanding deep networks via extremal perturbations and smooth masks. In: International Conference on Computer Vision (ICCV), pp. 2950–2958. IEEE (2019)Google Scholar
  7. 7.
    Gärdenfors, P.: Knowledge in Flux. MIT Press, Cambridge (1988)zbMATHGoogle Scholar
  8. 8.
    Gonzalez-Sanchez, A.: Automatic error detection techniques based on dynamic invariants. M.S. thesis, Delft University of Technology, The Netherlands (2007)Google Scholar
  9. 9.
    Gunning, D.: Explainable artificial intelligence (XAI) - program information. Defense Advanced Research Projects Agency (2017). https://www.darpa.mil/program/explainable-artificial-intelligence
  10. 10.
    Halpern, J.Y., Pearl, J.: Causes and explanations: a structural-model approach. Part I: causes. Br. J. Philos. Sci. 56(4), 843–887 (2005)CrossRefGoogle Scholar
  11. 11.
    Halpern, J.Y., Pearl, J.: Causes and explanations: a structural-model approach. Part II: explanations. Br. J. Philos. Sci. 56(4), 889–911 (2005)CrossRefGoogle Scholar
  12. 12.
    Hempel, C.G.: Aspects of Scientific Explanation. Free Press, New York (1965)Google Scholar
  13. 13.
    Huang, X., et al.: A survey of safety and trustworthiness of deep neural networks: verification, testing, adversarial attack and defence, and interpretability. Comput. Sci. Rev. 37, 100270 (2020)MathSciNetCrossRefGoogle Scholar
  14. 14.
    Jones, J.A., Harrold, M.J.: Empirical evaluation of the Tarantula automatic fault-localization technique. In: Proceedings of ASE, pp. 273–282. ACM (2005)Google Scholar
  15. 15.
    Landsberg, D., Chockler, H., Kroening, D., Lewis, M.: Evaluation of measures for statistical fault localisation and an optimising scheme. In: Egyed, A., Schaefer, I. (eds.) FASE 2015. LNCS, vol. 9033, pp. 115–129. Springer, Heidelberg (2015).  https://doi.org/10.1007/978-3-662-46675-9_8CrossRefGoogle Scholar
  16. 16.
    Landsberg, D., Sun, Y., Kroening, D.: Optimising spectrum based fault localisation for single fault programs using specifications. In: Russo, A., Schürr, A. (eds.) FASE 2018. LNCS, vol. 10802, pp. 246–263. Springer, Cham (2018).  https://doi.org/10.1007/978-3-319-89363-1_14CrossRefGoogle Scholar
  17. 17.
    Liu, Y., et al.: Trojaning attack on neural networks. In: Network and Distributed System Security Symposium (NDSS). The Internet Society (2018)Google Scholar
  18. 18.
    Lundberg, S.M., Lee, S.I.: A unified approach to interpreting model predictions. In: Advances in Neural Information Processing Systems, pp. 4765–4774 (2017)Google Scholar
  19. 19.
    Naish, L., Lee, H.J., Ramamohanarao, K.: A model for spectra-based software diagnosis. ACM TOSEM 20(3), 11 (2011)CrossRefGoogle Scholar
  20. 20.
    Noller, Y., Păsăreanu, C.S., Böhme, M., Sun, Y., Nguyen, H.L., Grunske, L.: HyDiff: hybrid differential software analysis. In: Proceedings of the International Conference on Software Engineering (ICSE) (2020)Google Scholar
  21. 21.
    Ochiai, A.: Zoogeographic studies on the soleoid fishes found in Japan and its neighbouring regions. Bull. Jpn. Soc. Sci. Fish. 22, 526–530 (1957)CrossRefGoogle Scholar
  22. 22.
    Odena, A., Olsson, C., Andersen, D., Goodfellow, I.: TensorFuzz: debugging neural networks with coverage-guided fuzzing. In: International Conference on Machine Learning, pp. 4901–4911 (2019)Google Scholar
  23. 23.
    Olah, C., et al.: The building blocks of interpretability. Distill 3, e10 (2018)CrossRefGoogle Scholar
  24. 24.
    Pearl, J.: Probabilistic Reasoning in Intelligent Systems. Morgan Kaufmann, Burlington (1988)zbMATHGoogle Scholar
  25. 25.
    Petsiuk, V., Das, A., Saenko, K.: RISE: randomized input sampling for explanation of black-box models. In: British Machine Vision Conference (BMVC). BMVA Press (2018)Google Scholar
  26. 26.
    Rahwan, I., et al.: Machine behaviour. Nature 568(7753), 477 (2019)CrossRefGoogle Scholar
  27. 27.
    Ribeiro, M.T., Singh, S., Guestrin, C.: Why should I trust you? Explaining the predictions of any classifier. In: Knowledge Discovery and Data Mining (KDD), pp. 1135–1144. ACM (2016)Google Scholar
  28. 28.
    Salmon, W.C.: Four Decades of Scientific Explanation. University of Minnesota Press, Minneapolis (1989)Google Scholar
  29. 29.
    Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-CAM: visual explanations from deep networks via gradient-based localization. In: International Conference on Computer Vision (ICCV), pp. 618–626. IEEE (2017)Google Scholar
  30. 30.
    Shrikumar, A., Greenside, P., Kundaje, A.: Learning important features through propagating activation differences. In: International Conference on Machine Learning (ICML), vol. 70, pp. 3145–3153. PMLR (2017)Google Scholar
  31. 31.
    Sun, Y., Wu, M., Ruan, W., Huang, X., Kwiatkowska, M., Kroening, D.: Concolic testing for deep neural networks. In: Proceedings of the 33rd ACM/IEEE International Conference on Automated Software Engineering, ASE, pp. 109–119 (2018)Google Scholar
  32. 32.
    Sun, Y., Zhou, Y., Maskell, S., Sharp, J., Huang, X.: Reliability validation of learning enabled vehicle tracking. In: International Conference on Robotics and Automation (ICRA). IEEE (2020)Google Scholar
  33. 33.
    Wong, W.E., Gao, R., Li, Y., Abreu, R., Wotawa, F.: A survey on software fault localization. IEEE TSE 42(8), 707–740 (2016)Google Scholar
  34. 34.
    Wong, W.E., Qi, Y., Zhao, L., Cai, K.: Effective fault localization using code coverage. In: Computer Software and Applications Conference (COMPSAC), pp. 449–456 (2007)Google Scholar
  35. 35.
    Zemla, J.C., Sloman, S., Bechlivanidis, C., Lagnado, D.A.: Evaluating everyday explanations. Psychon. Bull. Rev. 24(5), 1488–1500 (2017).  https://doi.org/10.3758/s13423-017-1258-zCrossRefGoogle Scholar
  36. 36.
    Ziegler, C.: A Google self-driving car caused a crash for the first time. The Verge (2016). https://www.theverge.com/2016/2/29/11134344/google-self-driving-car-crash-report

Copyright information

© Springer Nature Switzerland AG 2020

Authors and Affiliations

  1. 1.Queen’s University BelfastBelfastNorthern Ireland
  2. 2.King’s College LondonLondonEngland
  3. 3.University of LiverpoolLiverpoolEngland
  4. 4.University of OxfordOxfordEngland

Personalised recommendations

Cite paper

Buy options