A Study on Trust in Black Box Models and Post-hoc Explanations

  • Nadia El BekriEmail author
  • Jasmin Kling
  • Marco F. Huber
Conference paper
Part of the Advances in Intelligent Systems and Computing book series (AISC, volume 950)


Machine learning algorithms that construct complex prediction models are increasingly used for decision-making due to their high accuracy, e.g., to decide whether a bank customer should receive a loan or not. Due to the complexity, the models are perceived as black boxes. One approach is to augment the models with post-hoc explainability. In this work, we evaluate three different explanation approaches based on the users’ initial trust, the users’ trust in the provided explanation, and the established trust in the black box by a within-subject design study.


Machine learning Black box Explainability Interpretability Trust 


  1. 1.
    Alexandrov, N.: Explainable AI decisions for human-autonomy interactions. In: 17th AIAA Aviation Technology, Integration, and Operations Conference, p. 3991 (2017)Google Scholar
  2. 2.
    Davenport, T.H., Harris, J.G.: Automated decision making comes of age. MIT Sloan Manag. Rev. 46(4), 83 (2005)Google Scholar
  3. 3.
    Dheeru, D., Karra Taniskidou, E.: UCI Machine Learning Repository (2017).
  4. 4.
    Doshi-Velez, F., Kim, B.: Towards a rigorous science of interpretable machine learning. arXiv preprint arXiv:1702.08608 (2017)
  5. 5.
    Goldstein, A., Kapelner, A., Bleich, J., Pitkin, E.: Peeking inside the black box: visualizing statistical learning with plots of individual conditional expectation. J. Comput. Graph. Stat. 24(1), 44–65 (2015)MathSciNetCrossRefGoogle Scholar
  6. 6.
    Lakkaraju, H., Kamar, E., Caruana, R., Leskovec, J.: Interpretable & explorable approximations of black box models. arXiv preprint arXiv:1707.01154 (2017)
  7. 7.
    Lipton, Z.C.: The mythos of model interpretability. arXiv preprint arXiv:1606.03490 (2016)
  8. 8.
    Mohseni, S., Ragan, E.D.: A human-grounded evaluation benchmark for local explanations of machine learning. arXiv preprint arXiv:1801.05075 (2018)
  9. 9.
    Poursabzi-Sangdeh, F., Goldstein, D.G., Hofman, J.M., Vaughan, J.W., Wallach, H.: Manipulating and measuring model interpretability. arXiv preprint arXiv:1802.07810 (2018)
  10. 10.
    Ribeiro, M.T., Singh, S., Guestrin, C.: Why should I trust you?: explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 1135–1144. ACM (2016)Google Scholar
  11. 11.
    Saabas, A.: Treeinterpreter (2015).
  12. 12.
    SoSciSurvey: SoSci survey tool (2018).
  13. 13.
    Tolomei, G., Silvestri, F., Haines, A., Lalmas, M.: Interpretable predictions of tree-based ensembles via actionable feature tweaking. In: Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 465–474. ACM (2017)Google Scholar
  14. 14.
    Turner, R.: A model explanation system. In: 2016 IEEE 26th International Workshop on Machine Learning for Signal Processing (MLSP), pp. 1–6. Vietri sul Mare (2016)Google Scholar
  15. 15.
    Tzeng, F.Y., Ma, K.L.: Opening the black box-data driven visualization of neural networks. In: IEEE Visualization, VIS 2005, pp. 383–390. IEEE (2005)Google Scholar
  16. 16.
    Wang, T., Rudin, C., Doshi-Velez, F., Liu, Y., Klampfl, E., MacNeille, P.: OR’s of AND’s for interpretable classification, with application to context-aware recommender systems. arXiv preprint arXiv:1504.07614 (2015)

Copyright information

© Springer Nature Switzerland AG 2020

Authors and Affiliations

  • Nadia El Bekri
    • 1
    Email author
  • Jasmin Kling
    • 1
  • Marco F. Huber
    • 2
    • 3
  1. 1.Fraunhofer IOSBKarlsruheGermany
  2. 2.Institute of Industrial Manufacturing and Management IFFUniversity of StuttgartStuttgartGermany
  3. 3.Center for Cyber Cognitive Intelligence (CCI)Fraunhofer IPAStuttgartGermany

Personalised recommendations