Layer-Wise Relevance Propagation for Deep Neural Network Architectures

  • Alexander BinderEmail author
  • Sebastian Bach
  • Gregoire Montavon
  • Klaus-Robert Müller
  • Wojciech Samek
Conference paper
Part of the Lecture Notes in Electrical Engineering book series (LNEE, volume 376)


We present the application of layer-wise relevance propagation to several deep neural networks such as the BVLC reference neural net and googlenet trained on ImageNet and MIT Places datasets. Layer-wise relevance propagation is a method to compute scores for image pixels and image regions denoting the impact of the particular image region on the prediction of the classifier for one particular test image. We demonstrate the impact of different parameter settings on the resulting explanation.


Deep neural networks Non-linear explanations 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Bach, S., Binder, A., Montavon, G., Klauschen, F., Müller, K.R., Samek, W.: On pixel-wise explanations for non-linear classifier decisions by layer-wise relevance propagation. PLOS ONE 10(7), e0130140 (2015)CrossRefGoogle Scholar
  2. 2.
    Bach, S., Binder, A., Montavon, G., Müller, K., Samek, W.: Analyzing classifiers: Fisher vectors and deep neural networks. CoRR abs/1512.00172 (2015)Google Scholar
  3. 3.
    Baehrens, D., Schroeter, T., Harmeling, S., Kawanabe, M., Hansen, K., Müller, K.R.: How to explain individual classification decisions. Journal of Machine Learning Research 11, 1803–1831 (2010)MathSciNetzbMATHGoogle Scholar
  4. 4.
    Braun, M.L., Buhmann, J.M., Müller, K.: On relevant dimensions in kernel feature spaces. Journal of Machine Learning Research 9, 1875–1908 (2008)MathSciNetzbMATHGoogle Scholar
  5. 5.
    Chatfield, K., Simonyan, K., Vedaldi, A., Zisserman, A.: Return of the devil in the details: delving deep into convolutional nets. In: British Machine Vision Conference (BMVC) (2014)Google Scholar
  6. 6.
    Ciresan, D.C., Giusti, A., Gambardella, L.M., Schmidhuber, J.: Deep neural networks segment neuronal membranes in electron microscopy images. In: Adv. in NIPS, pp. 2852–2860 (2012)Google Scholar
  7. 7.
    Collobert, R., Weston, J., Bottou, L., Karlen, M., Kavukcuoglu, K., Kuksa, P.P.: Natural language processing (almost) from scratch. Journal of Machine Learning Research 12, 2493–2537 (2011)zbMATHGoogle Scholar
  8. 8.
    Dosovitskiy, A., Brox, T.: Inverting convolutional networks with convolutional networks. CoRR abs/1506.02753 (2015)Google Scholar
  9. 9.
    Erhan, D., Bengio, Y., Courville, A., Vincent, P.: Visualizing higher-layer features of a deep network. Tech. Rep. 1341, University of Montreal, June 2009Google Scholar
  10. 10.
    Everingham, M., Van Gool, L., Williams, C.K.I., Winn, J., Zisserman, A.: The PASCAL Visual Object Classes Challenge 2012 (VOC 2012) ResultsGoogle Scholar
  11. 11.
    Goodfellow, I.J., Shlens, J., Szegedy, C.: Explaining and harnessing adversarial examples. CoRR abs/1412.6572 (2014)Google Scholar
  12. 12.
    Hansen, K., Baehrens, D., Schroeter, T., Rupp, M., Müller, K.R.: Visual interpretation of kernel-based prediction models. Molecular Informatics 30(9), 817–826 (2011)CrossRefGoogle Scholar
  13. 13.
    Jia, Y., Shelhamer, E., Donahue, J., Karayev, S., Long, J., Girshick, R., Guadarrama, S., Darrell, T.: Caffe: Convolutional architecture for fast feature embedding. CoRR abs/1408.5093 (2014)Google Scholar
  14. 14.
    Krizhevsky, A., Sutskever, I., Hinton, G.E.: Imagenet classification with deep convolutional neural networks. In: Adv. in NIPS 25, pp. 1106–1114 (2012)Google Scholar
  15. 15.
    Le, Q.V.: Building high-level features using large scale unsupervised learning. In: ICASSP, pp. 8595–8598 (2013)Google Scholar
  16. 16.
    Mahendran, A., Vedaldi, A.: Understanding deep image representations by inverting them. In: Proceedings of the IEEE Conf. on Computer Vision and Pattern Recognition (CVPR) (2015)Google Scholar
  17. 17.
    Montavon, G., Bach, S., Binder, A., Samek, W., Müller, K.R.: Explaining nonlinear classification decisions with deep taylor decomposition. CoRR abs/1512.02479 (2015)Google Scholar
  18. 18.
    Montavon, G., Braun, M., Müller, K.R.: Kernel analysis of deep networks. Journal of Machine Learning Research 12, 2563–2581 (2011)MathSciNetzbMATHGoogle Scholar
  19. 19.
    Montavon, G., Braun, M.L., Krueger, T., Müller, K.R.: Analyzing local structure in kernel-based learning: Explanation, complexity and reliability assessment. Signal Processing Magazine, IEEE 30(4), 62–74 (2013)CrossRefGoogle Scholar
  20. 20.
    Nguyen, A.M., Yosinski, J., Clune, J.: Deep neural networks are easily fooled: High confidence predictions for unrecognizable images. CoRR abs/1412.1897 (2014)Google Scholar
  21. 21.
    Samek, W., Binder, A., Montavon, G., Bach, S., Müller, K.R.: Evaluating the visualization of what a deep neural network has learned. CoRR abs/1509.06321 (2015)Google Scholar
  22. 22.
    Simonyan, K., Vedaldi, A., Zisserman, A.: Deep inside convolutional networks: Visualising image classification models and saliency maps. CoRR abs/1312.6034 (2013)Google Scholar
  23. 23.
    Socher, R., Perelygin, A., Wu, J., Chuang, J., Manning, C.D., Ng, A.Y., Potts, C.: Recursive deep models for semantic compositionality over a sentiment treebank. In: Proc. of EMNLP, pp. 1631–1642 (2013)Google Scholar
  24. 24.
    Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Van- houcke, V., Rabinovich, A.: Going deeper with convolutions. CoRR abs/1409.4842 (2014)Google Scholar
  25. 25.
    Szegedy, C., Zaremba, W., Sutskever, I., Bruna, J., Erhan, D., Goodfellow, I.J., Fergus, R.: Intriguing properties of neural networks. CoRR abs/1312.6199 (2013)Google Scholar
  26. 26.
    Yosinski, J., Clune, J., Nguyen, A.M., Fuchs, T., Lipson, H.: Understanding neural networks through deep visualization. CoRR abs/1506.06579 (2015)Google Scholar
  27. 27.
    Yu, D., Deng, L.: Automatic Speech Recognition - A Deep Learning Approach. Springer (2014)Google Scholar
  28. 28.
    Zeiler, M.D., Fergus, R.: Visualizing and understanding convolutional networks. In: ECCV, pp. 818–833 (2014)Google Scholar
  29. 29.
    Zhou, B., Lapedriza, A., Xiao, J., Torralba, A., Oliva, A.: Learning deep features for scene recognition using places database. In: Adv. in NIPS, pp. 487–495 (2014)Google Scholar

Copyright information

© Springer Science+Business Media Singapore 2016

Authors and Affiliations

  • Alexander Binder
    • 1
    Email author
  • Sebastian Bach
    • 2
  • Gregoire Montavon
    • 3
  • Klaus-Robert Müller
    • 3
  • Wojciech Samek
    • 2
  1. 1.ISTD PillarSingapore University of Technology and DesignSingaporeSingapore
  2. 2.Machine Learning GroupFraunhofer HHIBerlinGermany
  3. 3.Machine Learning GroupTU BerlinBerlinGermany

Personalised recommendations