Skip to main content

Shap-CAM: Visual Explanations for Convolutional Neural Networks Based on Shapley Value

  • Conference paper
  • First Online:
Computer Vision – ECCV 2022 (ECCV 2022)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 13672))

Included in the following conference series:

Abstract

Explaining deep convolutional neural networks has been recently drawing increasing attention since it helps to understand the networks’ internal operations and why they make certain decisions. Saliency maps, which emphasize salient regions largely connected to the network’s decision-making, are one of the most common ways for visualizing and analyzing deep networks in the computer vision community. However, saliency maps generated by existing methods cannot represent authentic information in images due to the unproven proposals about the weights of activation maps which lack solid theoretical foundation and fail to consider the relations between each pixels. In this paper, we develop a novel post-hoc visual explanation method called Shap-CAM based on class activation mapping. Unlike previous gradient-based approaches, Shap-CAM gets rid of the dependence on gradients by obtaining the importance of each pixels through Shapley value. We demonstrate that Shap-CAM achieves better visual performance and fairness for interpreting the decision making process. Our approach outperforms previous methods on both recognition and localization tasks.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 89.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 119.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Adebayo, J., Gilmer, J., Goodfellow, I., Kim, B.: Local explanation methods for deep neural networks lack sensitivity to parameter values. arXiv preprint arXiv:1810.03307 (2018)

  2. Adebayo, J., Gilmer, J., Muelly, M., Goodfellow, I., Hardt, M., Kim, B.: Sanity checks for saliency maps. In: Advances in Neural Information Processing Systems (2018)

    Google Scholar 

  3. Al-Shedivat, M., Dubey, A., Xing, E.P.: Contextual explanation networks. arXiv preprint arXiv:1705.10301 (2017)

  4. Castro, J., Gomez, D., Tejada, J.: A polynomial rule for the problem of sharing delay costs in pert networks. In: Computers and Operation Research (2008)

    Google Scholar 

  5. Castro, J., Gomez, D., Tejada, J.: Polynomial calculation of the shapley value based on sampling. Comput. Oper. Res. 36, 1726–1730 (2009)

    Google Scholar 

  6. Chang, C.H., Creager, E., Goldenberg, A., Duvenaud, D.: Explaining image classifiers by counterfactual generation. arXiv preprint arXiv:1807.08024 (2018)

  7. Chattopadhay, A., Sarkar, A., Howlader, P., Balasubramanian, V.N.: Grad-cam++: generalized gradient-based visual explanations for deep convolutional networks. In: IEEE Winter Conference on Applications of Computer Vision, pp. 839–847 (2018)

    Google Scholar 

  8. Dabkowski, P., Gal, Y.: Real time image saliency for black box classifiers. In: Advances in Neural Information Processing Systems (2017)

    Google Scholar 

  9. Deng, X., Papadimitriou, C.: On the complexity of cooperative solution concepts. Math. Oper. Res. 19 (1994)

    Google Scholar 

  10. Fatima, S.S., Wooldridge, M., Jennings, N.R.: An analysis of the shapley value and its uncertainty for the voting game. In: La Poutré, H., Sadeh, N.M., Janson, S. (eds.) AMEC/TADA -2005. LNCS (LNAI), vol. 3937, pp. 85–98. Springer, Heidelberg (2006). https://doi.org/10.1007/11888727_7

    Chapter  Google Scholar 

  11. Fernández, J., et al.: Generating functions for computing the myerson value. Ann. Oper. Res. 109(1), 143–158 (2002)

    Google Scholar 

  12. Fong, R.C., Vedaldi, A.: Interpretable explanations of black boxes by meaningful perturbation. In: ICCV (2017)

    Google Scholar 

  13. Granot, D., Kuipers, J., Chopra, S.: Cost allocation for a tree network with heterogeneous customers. Math. Oper. Res. 27, 647–661 (2002)

    Google Scholar 

  14. He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: CVPR (2016)

    Google Scholar 

  15. Heo, J., Joo, S., Moon, T.: Fooling neural network interpretations via adversarial model manipulation. In: NeurIPS (2019)

    Google Scholar 

  16. Hinton, G., Vinyals, O., Dean, J.: Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531 (2015)

  17. Lin, M., Chen, Q., Yan, S.: Network in network. arXiv preprint arXiv:1312.4400 (2013)

  18. Omeiza, D., Speakman, S., Cintas, C., Weldermariam, K.: Smooth Grad-CAM++: an enhanced inference level visualization technique for deep convolutional neural network models. arXiv preprint arXiv:1908.01224 (2019)

  19. Oquab, M., Bottou, L., Laptev, I., Sivic, J.: Is object localization for free? weakly-supervised learning with convolutional neural networks. In: CVPR, pp. 685–694 (2015)

    Google Scholar 

  20. Owen, G.: Multilinear extensions of games. In: Management Science Series B-Application, vol. 18, No. 5, Theory Series, Part 2, Game Theory and Gaming, January 1972, pp. P64–P79. INFORMS (1972)

    Google Scholar 

  21. Petsiuk, V., Das, A., Saenko, K.: Rise: Randomized input sampling for explanation of black-box models. arXiv preprint arXiv:1806.07421 (2018)

  22. Pinheiro, P.O., Collobert, R.: From image-level to pixel-level labeling with convolutional networks. In: CVPR, pp. 1713–1721 (2015)

    Google Scholar 

  23. Ribeiro, M.T., Singh, S., Guestrin, C.: Why should I trust you?: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 1135–1144 (2016)

    Google Scholar 

  24. Selvaraju, R.R., Das, A., Vedantam, R., Cogswell, M., Parikh, D., Batrat, D.: Grad-CAM: why did you say that? Visual explanations from deep networks via gradient-based localization. arXiv preprint arXiv:1610.02391 (2016)

  25. Shapley, L.S.: A value for n-person games. Contribut. Theory Games 2(28), 307–317 (1953)

    MathSciNet  MATH  Google Scholar 

  26. Shrikumar, A., Greenside, P., Kundaje, A.: Learning important features through propagating activation differences. arXiv preprint arXiv:1704.02685 (2017)

  27. Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556 (2014)

  28. Smilkov, D., Thorat, N., Kim, B., Viegas, F., Wattenberg, M.: SmoothGrad: removing noise by adding noise. arXiv preprint arXiv:1706.03825 (2017)

  29. Springenberg, J.T., Dosovitskiy, A., Brox, T., Riedmiller, M.: Striving for simplicity: the all convolutional net. arXiv preprint arXiv:1412.6806 (2014)

  30. Sundararajan, M., Taly, A., Yan, Q.: Axiomatic attribution for deep networks. In: Proceedings of the 34th International Conference on Machine Learning (2017)

    Google Scholar 

  31. Wang, H., et al.: Score-cam: Score-weighted visual explanations for convolutional neural networks. In: CVPR (2020)

    Google Scholar 

  32. Zagoruyko, S., Komodakis, N.: Wide residual networks. arXiv preprint arXiv:1605.07146 (2016)

  33. Zeiler, M.D., Fergus, R.: Visualizing and understanding convolutional networks. In: ECCV,. pp. 818–833 (2014)

    Google Scholar 

  34. Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: CVPR, pp. 2921–2929 (2016)

    Google Scholar 

Download references

Acknowledgement

This work was supported in part by the National Key Research and Development Program of China under Grant 2017YFA0700802, in part by the National Natural Science Foundation of China under Grant 62125603 and Grant U1813218, in part by a grant from the Beijing Academy of Artificial Intelligence (BAAI).

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Jiwen Lu .

Editor information

Editors and Affiliations

Rights and permissions

Reprints and permissions

Copyright information

© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Zheng, Q., Wang, Z., Zhou, J., Lu, J. (2022). Shap-CAM: Visual Explanations for Convolutional Neural Networks Based on Shapley Value. In: Avidan, S., Brostow, G., Cissé, M., Farinella, G.M., Hassner, T. (eds) Computer Vision – ECCV 2022. ECCV 2022. Lecture Notes in Computer Science, vol 13672. Springer, Cham. https://doi.org/10.1007/978-3-031-19775-8_27

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-19775-8_27

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-19774-1

  • Online ISBN: 978-3-031-19775-8

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics