Skip to main content
SpringerLink
Log in

ERIC: Extracting Relations Inferred from Convolutions

  • Conference paper
  • First Online:
Computer Vision – ACCV 2020 (ACCV 2020)

Part of the book series: Lecture Notes in Computer Science ((LNIP,volume 12624))

Included in the following conference series:

Abstract

Our main contribution is to show that the behaviour of kernels across multiple layers of a convolutional neural network can be approximated using a logic program. The extracted logic programs yield accuracies that correlate with those of the original model, though with some information loss in particular as approximations of multiple layers are chained together or as lower layers are quantised. We also show that an extracted program can be used as a framework for further understanding the behaviour of CNNs. Specifically, it can be used to identify key kernels worthy of deeper inspection and also identify relationships with other kernels in the form of the logical rules. Finally, we make a preliminary, qualitative assessment of rules we extract from the last convolutional layer and show that kernels identified are symbolic in that they react strongly to sets of similar images that effectively divide output classes into sub-classes with distinct characteristics.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 84.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 109.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    Preliminary experiments found that l1 norm yielded higher fidelity than l2 norm.

  2. 2.

    Limited space made it difficult to show all 10 without compromising clarity.

References

  1. Ribeiro, M.T., Singh, S., Guestrin, C.: Why should I trust you?: explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 1135–1144. ACM (2016)

    Google Scholar 

  2. Gilpin, L.H., Bau, D., Yuan, B.Z., Bajwa, A., Specter, M., Kagal, L.: Explaining explanations: an overview of interpretability of machine learning. In: 2018 IEEE 5th International Conference on Data Science and Advanced Analytics (DSAA), pp. 80–89. IEEE (2018)

    Google Scholar 

  3. Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. (CSUR) 51, 93 (2018)

    Google Scholar 

  4. Andrews, R., Diederich, J., Tickle, A.B.: Survey and critique of techniques for extracting rules from trained artificial neural networks. Knowl.-Based Syst. 8, 373–389 (1995)

    Article  Google Scholar 

  5. Jacobsson, H.: Rule extraction from recurrent neural networks: a taxonomy and review. Neural Comput. 17, 1223–1263 (2005)

    Article  MathSciNet  Google Scholar 

  6. Townsend, J., Chaton, T., Monteiro, J.M.: Extracting relational explanations from deep neural networks: a survey from a neural-symbolic perspective. IEEE Trans. Neural Networks Learn. Syst. 31, 3456 (2019)

    Article  MathSciNet  Google Scholar 

  7. Zhang, Q., Zhu, S.: Visual interpretability for deep learning: a survey. Front. Inf. Technol. Electron. Eng. 19, 27–39 (2018)

    Article  Google Scholar 

  8. Lamb, L., Garcez, A., Gori, M., Prates, M., Avelar, P., Vardi, M.: Graph neural networks meet neural-symbolic computing: a survey and perspective. arXiv preprint arXiv:2003.00330 (2020)

  9. Simonyan, K., Vedaldi, A., Zisserman, A.: Deep inside convolutional networks: visualising image classification models and saliency maps. arXiv preprint arXiv:1312.6034 (2013)

  10. Zeiler, M.D., Fergus, R.: Visualizing and understanding convolutional networks. In: Fleet, D., Pajdla, T., Schiele, B., Tuytelaars, T. (eds.) ECCV 2014. LNCS, vol. 8689, pp. 818–833. Springer, Cham (2014). https://doi.org/10.1007/978-3-319-10590-1_53

    Chapter  Google Scholar 

  11. Springenberg, J.T., Dosovitskiy, A., Brox, T., Riedmiller, M.: Striving for simplicity: the all convolutional net. arXiv preprint arXiv:1412.6806 (2014)

  12. Bojarski, M., et al.: Visualbackprop: efficient visualization of CNNs. arXiv preprint arXiv:1611.05418 (2016)

  13. Bach, S., Binder, A., Montavon, G., Klauschen, F., Müller, K., Samek, W.: On pixel-wise explanations for non-linear classifier decisions by layer-wise relevance propagation. PloS One 10, e0130140 (2015)

    Article  Google Scholar 

  14. Samek, W., Binder, A., Montavon, G., Lapuschkin, S., Müller, K.: Evaluating the visualization of what a deep neural network has learned. IEEE Trans. Neural Networks Learn. Syst. 28, 2660–2673 (2017)

    Article  MathSciNet  Google Scholar 

  15. Shrikumar, A., Greenside, P., Kundaje, A.: Learning important features through propagating activation differences. arXiv preprint arXiv:1704.02685 (2017)

  16. Frosst, N., Hinton, G.: Distilling a neural network into a soft decision tree. arXiv preprint arXiv:1711.09784 (2017)

  17. Chen, C., Li, O., Tao, D., Barnett, A., Rudin, C., Su, J.K.: This looks like that: deep learning for interpretable image recognition. In: Advances in Neural Information Processing Systems, pp. 8930–8941 (2019)

    Google Scholar 

  18. Bologna, G., Fossati, S.: A two-step rule-extraction technique for a CNN. Electronics 9, 990 (2020)

    Article  Google Scholar 

  19. Zhang, Q., Cao, R., Wu, Y.N., Zhu, S.: Growing interpretable part graphs on convnets via multi-shot learning. In: Thirty-First AAAI Conference on Artificial Intelligence (2017)

    Google Scholar 

  20. Zhang, Q., Cao, R., Shi, F., Wu, Y.N., Zhu, S.: Interpreting CNN knowledge via an explanatory graph. In: Thirty-Second AAAI Conference on Artificial Intelligence (2018)

    Google Scholar 

  21. Zhang, Q., Yang, Y., Ma, H., Wu, Y.N.: Interpreting CNNs via decision trees. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 6261–6270 (2019)

    Google Scholar 

  22. Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Object detectors emerge in deep scene CNNs. arXiv preprint arXiv:1412.6856 (2014)

  23. Zhang, Q., Nian Wu, Y., Zhu, S.: Interpretable convolutional neural networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 8827–8836 (2018)

    Google Scholar 

  24. Percy, C., d’Avila Garcez, A.S., Dragicevic, S., França, M.V., Slabaugh, G.G., Weyde, T.: The need for knowledge extraction: understanding harmful gambling behavior with neural networks. Front. Artif. Intell. Appl. 285, 974–981 (2016)

    Google Scholar 

  25. Ribeiro, M.T., Singh, S., Guestrin, C.: Anchors: High-precision model-agnostic explanations. In: AAAI Conference on Artificial Intelligence (2018)

    Google Scholar 

  26. Zilke, J.R., Loza Mencía, E., Janssen, F.: DeepRED – rule extraction from deep neural networks. In: Calders, T., Ceci, M., Malerba, D. (eds.) DS 2016. LNCS (LNAI), vol. 9956, pp. 457–473. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-46307-0_29

    Chapter  Google Scholar 

  27. Schaaf, N., Huber, M.F.: Enhancing decision tree based interpretation of deep neural networks through l1-orthogonal regularization. arXiv preprint arXiv:1904.05394 (2019)

  28. Nguyen, T.D., Kasmarik, K.E., Abbass, H.A.: Towards interpretable deep neural networks: an exact transformation to multi-class multivariate decision trees. arXiv preprint arXiv:2003.04675 (2020)

  29. Murdoch, W.J., Szlam, A.: Automatic rule extraction from long short term memory networks. arXiv preprint arXiv:1702.02540 (2017)

  30. Tran, S.N., d’Avila Garcez, A.S.: Deep logic networks: inserting and extracting knowledge from deep belief networks. IEEE Trans. Neural Networks Learn. Syst. 29, 246 (2016)

    Article  MathSciNet  Google Scholar 

  31. Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921–2929. IEEE (2016)

    Google Scholar 

  32. Denil, M., Demiraj, A., De Freitas, N.: Extraction of salient sentences from labelled documents. arXiv preprint arXiv:1412.6815 (2014)

  33. Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: visual explanations from deep networks via gradient-based localization. In: Proceedings of the IEEE international conference on computer vision, pp. 618–626 (2017)

    Google Scholar 

  34. Simon, M., Rodner, E., Denzler, J.: Part detector discovery in deep convolutional neural networks. In: Cremers, D., Reid, I., Saito, H., Yang, M.-H. (eds.) ACCV 2014. LNCS, vol. 9004, pp. 162–177. Springer, Cham (2015). https://doi.org/10.1007/978-3-319-16808-1_12

    Chapter  Google Scholar 

  35. Xie, N., Sarker, M.K., Doran, D., Hitzler, P., Raymer, M.: Relating input concepts to convolutional neural network decisions. arXiv preprint arXiv:1711.08006 (2017)

  36. Zhou, B., Lapedriza, A., Khosla, A., Oliva, A., Torralba, A.: Places: a 10 million image database for scene recognition. IEEE Trans. Pattern Anal. Mach. Intell. 40, 1452 (2017)

    Article  Google Scholar 

  37. Quinlan, J.R.: C4.5: programming for machine learning. The Morgan Kaufmann Series in Machine Learning, pp. 38–48. Morgan Kaufmann, San Mateo, CA (1993)

    Google Scholar 

  38. Chollet, F., et al.: Keras (2015). https://github.com/fchollet/keras

Download references

Author information

Authors and Affiliations

  1. Fujitsu Laboratories of Europe LTD., 4th Floor, Building 3, Hyde Park Hayes, 11 Millington Road, Hayes, Middlesex, UB3 4AZ, UK

    Joe Townsend, Theodoros Kasioumis & Hiroya Inakoshi

Authors
  1. Joe Townsend
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Theodoros Kasioumis
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Hiroya Inakoshi
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Joe Townsend .

Editor information

Editors and Affiliations

  1. Waseda University, Tokyo, Japan

    Hiroshi Ishikawa

  2. Institute of Automation of Chinese Academy of Sciences, Beijing, China

    Cheng-Lin Liu

  3. Czech Technical University in Prague, Prague, Czech Republic

    Tomas Pajdla

  4. University of Pennsylvania, Philadelphia, PA, USA

    Jianbo Shi

Rights and permissions

Reprints and permissions

Copyright information

© 2021 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Townsend, J., Kasioumis, T., Inakoshi, H. (2021). ERIC: Extracting Relations Inferred from Convolutions. In: Ishikawa, H., Liu, CL., Pajdla, T., Shi, J. (eds) Computer Vision – ACCV 2020. ACCV 2020. Lecture Notes in Computer Science(), vol 12624. Springer, Cham. https://doi.org/10.1007/978-3-030-69535-4_13

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-69535-4_13

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-69534-7

  • Online ISBN: 978-3-030-69535-4

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation