Skip to main content
SpringerLink
Log in

Interpretability of Deep Neural Models

  • Chapter
  • First Online:
Ethics in Artificial Intelligence: Bias, Fairness and Beyond

Part of the book series: Studies in Computational Intelligence ((SCI,volume 1123))

  • 234 Accesses

Abstract

The rise of deep neural networks in machine learning has been remarkable, leading to their deployment in algorithmic decision-making. However, this has raised questions about the explainability and interpretability of these models, given their growing importance in society. To address this, the field of interpretability in machine learning has been developed, with the goal of creating frameworks that can explain the decisions of a machine learning model in a way that is comprehensible to humans. This could be essential in building trust in the system, as well as debugging models for potential errors and meeting legal requirements (e.g., GDPR). Even though the success of deep neural network is attributed to its ability to capture higher level feature interactions, most of existing frameworks still focus on highlighting important individual features (e.g., words in text or pixels in images). Hence, to further improve interpretability, we propose to quantify the importance of feature interactions in addition to individual features. In this work, we introduce integrated directional gradients (IDG), a game-theory inspired method for assigning importance scores to higher level feature interactions. Our experiments with DNN-based text classifiers on the task of sentiment classification demonstrate that IDG is able to effectively capture the importance of feature interactions.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 139.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Hardcover Book
USD 179.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

Notes

  1. 1.

    For detailed proofs refer to the original paper [36].

References

  1. Bach S, Binder A, Montavon G, Klauschen F, Müller K-R, Samek W (2015) On pixel-wise explanations for non-linear classifier decisions by layer-wise relevance propagation. PLoS ONE 10(7):e0130140

    Article  Google Scholar 

  2. Camburu O-M (2020) Explaining deep neural networks. arXiv:2010.01496

  3. Chen H, Zheng G, Ji Y (2020) Generating hierarchical explanations on text classification via feature interaction detection. In: Annual meeting of the association for computational linguistics, pp 5578–5593

    Google Scholar 

  4. Chen J, Jordan M (2020) Ls-tree: model interpretation when the data are linguistic. In: AAAI conference on artificial intelligence, vol 34, pp 3454–3461

    Google Scholar 

  5. Cui T, Marttinen P, Kaski S et al (2020) Learning global pairwise interactions with bayesian neural networks. In: European conference on artificial intelligence. IOS Press

    Google Scholar 

  6. Devlin J, Chang M-W, Lee K, Toutanova K (2018) Bert: pre-training of deep bidirectional transformers for language understanding. arXiv:1810.04805

  7. Engler J, Sikdar S, Lutz M, Strohmaier M (2022) SensePOLAR: word sense aware interpretability for pre-trained contextual word embeddings. In: Findings of the association for computational linguistics: EMNLP, pp 4607–4619

    Google Scholar 

  8. Frye C, Rowat C, Feige I (2020) Asymmetric Shapley values: incorporating causal knowledge into model-agnostic explainability. In: Advances in neural information processing systems, vol 33

    Google Scholar 

  9. Ghorbani A, Zou J (2020) Neuron Shapley: discovering the responsible neurons. arXiv:2002.09815

  10. Harsanyi JC (1963) A simplified bargaining model for the n-person cooperative game. Int Econ Rev 4(2):194–220

    Article  Google Scholar 

  11. Hendricks LA, Akata Z, Rohrbach M, Donahue J, Schiele B, Darrell T (2016) Generating visual explanations. In: European conference on computer vision. Springer, Berlin, pp 3–19

    Google Scholar 

  12. Hooker S, Erhan D, Kindermans P-J, Kim B (2019) A benchmark for interpretability methods in deep neural networks. In: Advances in neural information processing systems

    Google Scholar 

  13. Ibrahim M, Louie M, Modarres C, Paisley J (2019) Global explanations of neural networks: Mapping the landscape of predictions. In: Proceedings of the 2019 AAAI/ACM conference on AI, ethics, and society, pp 279–287

    Google Scholar 

  14. Janizek JD, Sturmfels P, Lee S-I (2020) Explaining explanations: axiomatic feature interactions for deep networks. arXiv:2002.04138

  15. Jin X, Wei Z, Du J, Xue X, Ren X (2019) Towards hierarchical importance attribution: explaining compositional semantics for neural sequence models. In: International conference on learning representations

    Google Scholar 

  16. Kim B, Wattenberg M, Gilmer J, Cai C, Wexler J, Viegas F et al (2018) Interpretability beyond feature attribution: quantitative testing with concept activation vectors (TCAV). In: International conference on machine learning. PMLR, pp 2668–2677

    Google Scholar 

  17. Kim J, Rohrbach A, Darrell T, Canny J, Akata Z (2018) Textual explanations for self-driving vehicles. In: Proceedings of the European conference on computer vision (ECCV), pp 563–578

    Google Scholar 

  18. Koh PW, Liang P (2017) Understanding black-box predictions via influence functions. In: International conference on machine learning. PMLR, pp 1885–1894

    Google Scholar 

  19. LeCun Y, Bengio Y, Hinton G (2015) Deep learning. Nature 521(7553):436–444

    Article  Google Scholar 

  20. Lei T, Barzilay R, Jaakkola T (2016) Rationalizing neural predictions. In: Proceedings of the 2016 conference on empirical methods in natural language processing, pp 107–117

    Google Scholar 

  21. Lewis M, Liu Y, Goyal N, Ghazvininejad M, Mohamed A, Levy O, Stoyanov V, Zettlemoyer L (2019) Bart: denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. arXiv:1910.13461

  22. Liu Z, Song Q, Zhou K, Wang T-H, Shan Y, Hu X (2020) Detecting interactions from neural networks via topological analysis. In: Advances in neural information processing systems, vol 33

    Google Scholar 

  23. Lundberg SM, Erion G, Chen H, DeGrave A, Prutkin JM, Nair B, Katz R, Himmelfarb J, Bansal N, Lee S-I (2020) From local explanations to global understanding with explainable AI for trees. Nat Mach Intell 2(1):2522–5839

    Article  Google Scholar 

  24. Lundberg SM, Lee S-I (2017) A unified approach to interpreting model predictions. In: Advances in neural information processing systems, pp 4765–4774

    Google Scholar 

  25. Maas A, Daly RE, Pham PT, Huang D, Ng AY, Potts C (2011) Learning word vectors for sentiment analysis. In: Annual meeting of the association for computational linguistics: human language technologies, pp 142–150

    Google Scholar 

  26. Mathew B, Sikdar S, Lemmerich F, Strohmaier M (2020) The polar framework: polar opposites enable interpretability of pre-trained word embeddings. In: Proceedings of the web conference, pp 1548–1558

    Google Scholar 

  27. Murdoch WJ, Liu PJ, Yu B (2018) Beyond word importance: contextual decomposition to extract interactions from LSTMS. In: International conference on learning representations

    Google Scholar 

  28. Park DH, Hendricks LA, Akata Z, Rohrbach A, Schiele B, Darrell T, Rohrbach M (2018) Multimodal explanations: justifying decisions and pointing to the evidence. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 8779–8788

    Google Scholar 

  29. Rajani NF, McCann B, Xiong C, Socher R (2019) Explain yourself! leveraging language models for commonsense reasoning. arXiv:1906.02361

  30. Ribeiro MT, Singh S, Guestrin C (2016) why should i trust you? explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining, pp 1135–1144

    Google Scholar 

  31. Ribeiro MT, Wu T, Guestrin C, Singh S (2020) Beyond accuracy: behavioral testing of nlp models with checklist. arXiv:2005.04118

  32. Rudin C (2019) Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead. Nat Mach Intell 1(5):206–215

    Article  Google Scholar 

  33. Selvaraju RR, Cogswell M, Das A, Vedantam R, Parikh D, Batra D (2017) Grad-cam: visual explanations from deep networks via gradient-based localization. In: IEEE international conference on computer vision, pp 618–626

    Google Scholar 

  34. Şenel LK, Şahinuç F, Yücesoy V, Schütze H, Çukur T, Koç A (2022) Learning interpretable word embeddings via bidirectional alignment of dimensions with semantic concepts. Inf Process Manag 59(3):102925

    Article  Google Scholar 

  35. Shrikumar A, Greenside P, Kundaje A (2017) Learning important features through propagating activation differences. In: International conference on machine learning. PMLR, pp 3145–3153

    Google Scholar 

  36. Sikdar S, Bhattacharya P, Heese K (2021) Integrated directional gradients: feature interaction attribution for neural NLP models. In: Proceedings of the 59th annual meeting of the association for computational linguistics and the 11th international joint conference on natural language processing, vol 1: Long Papers, pp 865–878

    Google Scholar 

  37. Singh C, Murdoch WJ, Yu B (2018) Hierarchical interpretations for neural network predictions. In: International conference on learning representations

    Google Scholar 

  38. Socher R, Perelygin A, Wu J, Chuang J, Manning CD, Ng AY, Potts C (2013) Recursive deep models for semantic compositionality over a sentiment treebank. In: Conference on empirical methods in natural language processing, pp 1631–1642

    Google Scholar 

  39. Sun C, Qiu X, Xu Y, Huang X (2019) How to fine-tune bert for text classification? In: China National conference on Chinese computational linguistics. Springer, Berlin, pp 194–206

    Google Scholar 

  40. Sundararajan M, Dhamdhere K, Agarwal A (2020) The Shapley Taylor interaction index. In: International conference on machine learning. PMLR, pp 9259–9268

    Google Scholar 

  41. Sundararajan M, Najmi A (2020) The many Shapley values for model explanation. In: International conference on machine learning. PMLR, pp 9269–9278

    Google Scholar 

  42. Sundararajan M, Taly A, Yan Q (2017) Axiomatic attribution for deep networks. In: International conference on machine learning. PMLR, pp 3319–3328

    Google Scholar 

  43. Tsang M, Rambhatla S, Liu Y (2020) How does this interaction affect me? interpretable attribution for feature interactions. In: Advances in neural information processing systems, vol 33

    Google Scholar 

  44. Yang Z, Dai Z, Yang Y, Carbonell J, Salakhutdinov RR, Le QV (2019) Xlnet: generalized autoregressive pretraining for language understanding. In: Advances in neural information processing systems, pp 5753–5763

    Google Scholar 

  45. Yoon J, Jordon J, van der Schaar M (2018) INVASE: instance-wise variable selection using neural networks. In: International conference on learning representations

    Google Scholar 

  46. Zhang X, Zhao J, LeCun Y (2015) Character-level convolutional networks for text classification. In: Advances in neural information processing systems, pp 649–657

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. L3S Research Center, Leibniz University Hannover, Hanover, Germany

    Sandipan Sikdar

  2. Biocomplexity Institute, University of Virginia, Charlottesville, USA

    Parantapa Bhattacharya

Authors
  1. Sandipan Sikdar
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Parantapa Bhattacharya
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Sandipan Sikdar .

Editor information

Editors and Affiliations

  1. Department of Computer Science and Engineering, IIT Kharagpur, West Bengal, India

    Animesh Mukherjee

  2. Department of Computer Science, Aalto University, Espoo, Finland

    Juhi Kulshrestha

  3. Department of Computer Science and Engineering, IIT Delhi, New Delhi, Delhi, India

    Abhijnan Chakraborty

  4. School of Computational Science and Engineering, College of Computing, Georgia Institute of Technology, Atlanta, GA, USA

    Srijan Kumar

Rights and permissions

Reprints and permissions

Copyright information

© 2023 The Institution of Engineers (India)

About this chapter

Check for updates. Verify currency and authenticity via CrossMark

Cite this chapter

Sikdar, S., Bhattacharya, P. (2023). Interpretability of Deep Neural Models. In: Mukherjee, A., Kulshrestha, J., Chakraborty, A., Kumar, S. (eds) Ethics in Artificial Intelligence: Bias, Fairness and Beyond. Studies in Computational Intelligence, vol 1123. Springer, Singapore. https://doi.org/10.1007/978-981-99-7184-8_8

Download citation

Publish with us

Policies and ethics

Search

Navigation