Skip to main content
SpringerLink
Log in

Explainable Methods for Image-Based Deep Learning: A Review

  • Review article
  • Published:
Archives of Computational Methods in Engineering Aims and scope Submit manuscript

Abstract

With Artificial Intelligence advancement, Deep neural networks (DNN) are extensively used for decision-making in intelligent systems. However, improved performance and accuracy have been achieved through the increasing use of complex models, which makes it challenging for users to understand and trust. This ambiguous nature of these Deep machine learning models of high accuracy and low interpretability is problematic for their adoption in critical domains where it is vital to be able to explain the decisions made by the system. Explainable Artificial Intelligence has become an exciting field for explaining and interpreting machine learning models. Among the different data types used in machine learning, image data is considered hard to train because of the factors such as class, scale, viewpoint, and background variations. This paper aims to provide a rounded view of emerging methods to explain DNN models as a way to boost transparency in image-based deep learning with the analysis of the current and upcoming trends.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13

Similar content being viewed by others

References

  1. Sze V, Chen YH, Yang TJ, Emer JS (2017) Efficient processing of deep neural networks: a tutorial and survey. Proc IEEE 105:2295–2329

    Article  Google Scholar 

  2. Adadi A, Berrada M (2018) Peeking inside the black-box: a survey on explainable artificial intelligence (XAI). IEEE Access 6:52160

    Article  Google Scholar 

  3. Preece AD, Harborne D, Braines D, Tomsett R, Chakraborty S (2018) Stakeholders in Explainable AIa. CoRR, vol. bs/1810.00184

  4. Schlegel U, Arnout H, El-Assady M, Oelke D, Keim DA (2019) Towards a rigorous evaluation of xai methods on time series. 2019 IEEE/CVF International conference on computer vision workshop (ICCVW), pp. 4197–4201

  5. Kaminski ME (2019) The right to explanation, explained. Berkeley Tech LJ 34:189–189

    Google Scholar 

  6. Yang, M., Wang, H., Hu, K., Yin, G., & Wei, Z. (2022). IA-Net: An Inception–Attention-Module-Based Network for Classifying Underwater Images From Others. IEEE Journal of Oceanic Engineering, 47(3), 704–717. https://doi.org/10.1109/JOE.2021.3126090

  7. Arrieta AB (2020) Explainable artificial intelligence (XAI): concepts, taxonomies, opportunities and challenges toward responsible AI. Inf Fusion 58:82–115

    Article  Google Scholar 

  8. Guidotti R, Monreale A, Ruggieri S, Turini F, Giannotti F, Pedreschi D (2018) A survey of methods for explaining black box models. ACM Comput Surv 51(5):1–42

    Article  Google Scholar 

  9. Montavon G, Samek W, Müller KR (2018) Methods for interpreting and understanding deep neural networks. Digit Signal Process 73:1–15

    Article  MathSciNet  Google Scholar 

  10. Fernandez A, Herrera F, Cordon O, Jesus MJ, Marcelloni F (2019) Evolutionary fuzzy systems for explainable artificial intelligence: why, when, what for, and where to? IEEE Comput Intell Mag 14(1):69–81

    Article  Google Scholar 

  11. Lipton ZC (2018) The mythos of model interpretability: in machine learning, the concept of interpretability is both important and slippery. pp. 31–57

  12. Mohseni S, Zarei N, Ragan ED (2018) A multidisciplinary survey and framework for design and evaluation of explainable AI systems

  13. Zhou, W., Wang, H., & Wan, Z. (2022). Ore Image Classification Based on Improved CNN. Computers & electrical engineering, 99(1). https://doi.org/10.1016/j.compeleceng.2022.107819

  14. Bhatt U (2020) Explainable machine learning in deployment. Proceedings of the 2020 conference on fairness, accountability, and transparency. pp. 648–657

  15. Choo J, Liu S (2018) Visual analytics for explainable deep learning. IEEE Comput Graph Appl 38(4):84–92

    Article  Google Scholar 

  16. Liu, H., Liu, M., Li, D., Zheng, W., Yin, L.,... Wang, R. (2022). Recent Advances in Pulse-Coupled Neural Networks with Applications in Image Processing. Electronics, 11(20). https://doi.org/10.3390/electronics11203264

  17. Das A, Rad P (2020) Opportunities and challenges in explainable artificial intelligence (XAI): a survey. CoRR. vol. abs/2006.11371

  18. Ivanovs M, Kadikis R, Ozols K (2021) Perturbation-based methods for explaining deep neural networks: a survey. Pattern Recognit Lett

  19. Yang L, Kenny EM, Ng TLJ, Yang Y, Smyth B, Dong R (2020) generating plausible counterfactual explanations for deep transformers in financial text classification. CoRR, vol. abs/2010.12512

  20. Zhao, L., & Wang, L. (2022). A new lightweight network based on MobileNetV3. KSII TRANSACTIONS ON INTERNET AND INFORMATION SYSTEMS. https://doi.org/10.3837/tiis.2022.01.001

  21. Ploug T, Holm S (2020) The four dimensions of contestable AI diagnosticsA patient-centric approach to explainable AI. Artif Intell Med 107:101–901

    Article  Google Scholar 

  22. Huang, C., Jiang, F., Huang, Q., Wang, X., Han, Z.,... Huang, W. (2022). Dual-Graph Attention Convolution Network for 3-D Point Cloud Classification. IEEE Transactions on Neural Networks and Learning Systems, 1–13. https://doi.org/10.1109/TNNLS.2022.3162301

  23. Jiawei Xu, Sicheng Pan, Poly Z.H. Sun, Seop Hyeong Park, and Kun Guo, “Human-Factors-in-Driving-Loop: Driver Identification and Verification via a Deep Learning Approach using Psychological Behavioral Data,” IEEE Transactions on Intelligent Transportation Systems (IEEE-TITS), 2022. https://doi.org/10.1109/TITS.2022.3225782

  24. Pak M, Kim S, (2017) A review of deep learning in image recognition. 2017 4th international conference on computer applications and information processing technology, pp. 1–3

  25. Dong Y, Su H, Zhu J, Zhang B (2017) Improving interpretability of deep neural networks with semantic information. Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 4306–4314

  26. Zhuang, Y., Chen, S., Jiang, N., & Hu, H. (2022). An Effective WSSENet-Based Similarity Retrieval Method of Large Lung CT Image Databases. KSII Transactions on Internet & Information Systems, 16(7). https://doi.org/10.3837/tiis.2022.07.013

  27. Cortez P, Embrechts MJ (2011) Opening black box data mining models using sensitivity analysis. 2011 IEEE Symposium on computational intelligence and data mining (CIDM), pp. 341–348

  28. Cortex P (2013) Using sensitivity analysis and visualization techniques to open black box data mining models. Inf Sci 225:1–17

    Article  Google Scholar 

  29. Caruana R (2000) Case-based explanation for artificial neural nets. In Artificial neural networks in medicine and biology. pp. 303–308

  30. Keane MT, Kenny EM (2019) The twin-system approach as one generic solution for XAI: an overview of ANN-CBR twins for explaining deep learning. CoRR, vol. abs/1905.08069

  31. Guidotti R, Monreale A, Ruggieri S, Pedreschi D, Turini F, Giannotti F (2018) Local rule-based explanations of black box decision systems,” CoRR, vol. abs/1805.10820

  32. Krishnan S, Wu E (2017) Palm: machine learning explanations for iterative debugging. Proceedings of the 2nd workshop on human-in-the-loop data analytics. pp. 1–6

  33. Ribeiro MT, Singh S, Guestrin C (2016) ‘Why should i trust you?’ Explaining the predictions of any classifier. Proceedings of the ACM SIGKDD international conference on knowledge discovery and data mining, vol. 13

  34. Simonyan K, Vedaldi A, Zisserman A (2014) Deep inside convolutional networks: visualising image classification models and saliency maps. CoRR, vol. abs/1312.6034

  35. Shrikumar A, Greenside P, Shcherbina A, Kundaje A, (2016) Not just a black box: learning important features through propagating activation differences. CoRR, vol. abs/1605.01713

  36. Xu, S., He, Q., Tao, S., Chen, H., Chai, Y.,... Zheng, W. (2022). Pig Face Recognition Based on Trapezoid Normalized Pixel Difference Feature and Trimmed Mean Attention Mechanism. IEEE Transactions on Instrumentation and Measurement, 1. https://doi.org/10.1109/TIM.2022.3232093

  37. Springenberg JT, Dosovitskiy A, Brox T, Riedmiller MA (2015) Striving for simplicity: the all convolutional net. CoRR, vol. abs/1412.6806

  38. Bach S, Binder A, Montavon G, Klauschen F, Müller KR, Samek W (2015) On pixel-wise explanations for non-linear classifier decisions by layer-wise relevance propagation. PLoS One 10(7):140–130

    Article  Google Scholar 

  39. Kindermans P-J, Schütt KT, Alber M, Müller K-R, Erhan D, Kim B, Dähne S (2018) Learning how to explain neural networks: PatternNet and PatternAttribution,” in ICLR

  40. Sundararajan M, Taly A, Yan Q (2017) Axiomatic attribution for deep networks. 34th International conference on machine learning, vol. 7

  41. Shrikumar A, Greenside P, Kundaje A (2017) Learning important features through propagating activation differences. International conference on machine learning. pp. 3145–3153

  42. Ancona M, Ceolini E, Öztireli AC, Gross MH, (2017) A unified view of gradient-based attribution methods for deep neural networks. CoRR, vol. abs/1711.06104

  43. Zhou B, Khosla A, Lapedriza A, Oliva A, Torralba A (2016) Learning deep features for discriminative localization. Proceedings of the IEEE computer society conference on computer vision and pattern recognition

  44. Lin M, Chen Q, Yan S (2014) Network in network. CoRR, vol. abs/1312.4400

  45. Selvaraju RR, Cogswell M, Das A, Vedantam R, Parikh D, Batra D (2017) Grad-cam: visual explanations from deep networks via gradientbased localization. Proceedings of the IEEE international conference on computer vision. pp. 618–626

  46. Chattopadhay A, Sarkar A, Howlader P, Balasubramanian VN (2018) Grad-CAM++: generalized gradient-based visual explanations for deep convolutional networks. Proceedings 2018 IEEE winter conference on applications of computer vision

  47. Smilkov D, Thorat N, Kim B, Viégas FB, Wattenberg M (2017) SmoothGrad: removing noise by adding noise. CoRR, vol. abs/1706.03825

  48. Aumann RJ, Shapley LS (2016) Values of non-atomic games. Princeton University Press, Princeton

    MATH  Google Scholar 

  49. Mitchell R, Cooper J, Frank E, Holmes G, (2021) Sampling permutations for shapley value estimation

  50. Lundberg SM, Lee SI (2017) A unified approach to interpreting model predictions. Advances in neural information processing systems. pp. 2017–2017

  51. Chen H, Lundberg S, Lee SI (2021) Explaining models by propagating shapley values of local components. Studies in computational intelligence. vol. 914

  52. Ancona M, Oztireli C, Gross M (2019) Explaining deep neural networks with a polynomial time algorithm for shapley value approximation. International conference on machine learning. pp. 272–281

  53. Petsiuk V, Das A, Saenko K (2018) Rise: randomized input sampling for explanation of black-box models

  54. Zintgraf LM, Cohen TS, Adel T, Welling M, (2017) Visualizing deep neural network decisions: prediction difference analysis. CoRR, vol. abs/1702.04595

  55. Burns C, Thomason J, Tansey W (2020) Interpreting black box models via hypothesis testing. Proceedings of the 2020 ACM-IMS on foundations of data science conference. pp. 47–57

  56. Fong RC, Vedaldi A (2017) Interpretable explanations of black boxes by meaningful perturbation. Proceedings of the IEEE international conference on computer vision. pp. 3429–3437

  57. Dabkowski P, Gal Y (2017) Real time image saliency for black box classifiers. In Proceedings of the 31st international conference on neural information processing systems. Red Hook, NY, USA: Curran Associates Inc., pp. 6970–6979

  58. Fong R, Patrick M, Vedaldi A (2019) Understanding deep networks via extremal perturbations and smooth masks. Proceedings of the IEEE/CVF international conference on computer vision. pp. 2950–2958

  59. Yang Q, Zhu X, Fwu JK, Ye Y, You G, Zhu Y (2021) MFPP: morphological fragmental perturbation pyramid for black-box model explanations. 2020 25th International conference on pattern recognition (ICPR). pp. 1376–1383

  60. Kim B (2018) Interpretability beyond feature attribution: quantitative testing with concept activation vectors (tcav). In International conference on machine learning. pp. 2668–2677

  61. Ghorbani A, Wexler J, Zou J, Kim B (2019) Towards Automatic ConceptBased Explanations. Curran Associates Inc., Red Hook, NY, USA

    Google Scholar 

  62. Goyal Y, Shalit U, Kim B (2019) Explaining classifiers with causal concept effect (CaCE). CoRR, vol. abs/1907.07165

  63. Agarwal R, Frosst N, Zhang X, Caruana R, Hinton GE (2020) Neural additive models: interpretable machine learning with neural nets. CoRR, vol. abs/2004.13912

  64. Lapuschkin S, Wäldchen S, Binder A, Montavon G, Samek W, Müller KR (2019) Unmasking clever hans predictors and assessing what machines really learn. Nat Commun 10(1):1–8

    Article  Google Scholar 

  65. Ibrahim M, Louie M, Modarres C, Paisley J (2019) Global explanations of neural networks: Mapping the landscape of predictions. Proceedings of the 2019 AAAI/ACM conference on AI, pp. 279–287

  66. Deng J, Dong W, Socher R, Li LJ, Li K, Fei-Fei L (2009) Imagenet: a large-scale hierarchical image database. 2009 IEEE conference on computer vision and pattern recognition, pp. 248–255

  67. Krizhevsky A, Hinton G (2009) Learning multiple layers of features from tiny images

  68. Russakovsky O (2015) Imagenet large scale visual recognition challenge. Int J Comput Vis 115(3):211–252

    Article  MathSciNet  Google Scholar 

  69. Deng L (2012) The mnist database of handwritten digit images for machine learning research. IEEE Signal Process Mag 29(6):141–142

    Article  Google Scholar 

  70. Huang, G. B., Mattar, M., Berg, T., & Learned-Miller, E. (2008, October). Labeled faces in the wild: A database forstudying face recognition in unconstrained environments. In Workshop on faces in‘Real-Life’Images: detection, alignment, and recognition.

  71. Everingham M, Gool LV, Williams CKI, Winn J, Zisserman A (2010) The pascal visual object classes (voc) challenge. Int J Comput Vis 88(2):303–338

    Article  Google Scholar 

  72. Lin TY (2014) Microsoft coco: common objects in context. pp. 740–755

  73. Kim B, Kim H, Kim K, Kim S, Kim J, (2019) Learning not to learn: training deep neural networks with biased data. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (CVPR), June 2019

  74. Yang M, Kim B (2019) BIM: towards quantitative evaluation of interpretability methods with ground truth. ArXiv, vol. abs/1907.09701

  75. Eykholt K (2018) Robust physical-world attacks on deep learning visual classification. Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 1625–1634

Download references

Author information

Authors and Affiliations

  1. School of Computer Science, University of Petroleum and Energy Studies, Dehradun, India

    Lav Kumar Gupta, Deepika Koundal & Shweta Mongia

  2. Department of Computer Science and Engineering, Faculty of Engineering and Technology, Manav Rachna International Institute of Research and Studies, Faridabad, India

    Shweta Mongia

Authors
  1. Lav Kumar Gupta
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Deepika Koundal
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Shweta Mongia
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Deepika Koundal.

Ethics declarations

Conflict of interest

There is no conflict of interest.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Gupta, L.K., Koundal, D. & Mongia, S. Explainable Methods for Image-Based Deep Learning: A Review. Arch Computat Methods Eng 30, 2651–2666 (2023). https://doi.org/10.1007/s11831-023-09881-5

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11831-023-09881-5

Search

Navigation