Skip to main content
SpringerLink
Log in

Overview of Visualization Methods for Artificial Neural Networks

  • INFORMATION SCIENCE
  • Published:
Computational Mathematics and Mathematical Physics Aims and scope Submit manuscript

Abstract

Modern algorithms based on artificial neural networks (ANNs) are extremely useful in solving a variety of complicated problems in computer vision, robust control, and natural language analysis of sound and texts as applied to data processing, robotics, etc. However, for the ANN approach to be successfully incorporated into critically important systems, for example, in medicine or jurisprudence, a clear interpretation is required for the internal architecture of ANN and for ANN-based decision-making processes. In recent years, analysis methods based on various visualization techniques applied to computation graphs, loss function profiles, parameters of single network layers, and even single neurons have become especially popular as tools for creating explainable deep learning models. This survey systematizes existing mathematical analysis methods and explanations of the behavior of underlying algorithms and presents formulations of corresponding problems in computational mathematics. The study and visualization of deep neural networks are new poorly studied yet rapidly developing areas. The considered methods give a deeper insight into the operation of neural network algorithms.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1.
Fig. 2.
Fig. 3.
Fig. 4.
Fig. 5.

Similar content being viewed by others

REFERENCES

  1. Y. LeCun, Y. Bengio, and G. Hinton, “Deep learning,” Nature 521, 436–444 (2015).

    Article  Google Scholar 

  2. N. Shahid, T. Rappon, and W. Berta, “Applications of artificial neural networks in health care organizational decision-making: A scoping review,” PLoS ONE 14 (2), e0212356 (2019).

    Article  Google Scholar 

  3. A. B. Nassif, I. Shahin, I. Attili, M. Azzeh, and K. Shaalan, “Speech recognition using deep neural networks: A systematic review,” IEEE Access 7, 19143–19165 (2019).

    Article  Google Scholar 

  4. H. H. Alkinani, A. T. T. Al-Hameedi, S. Dunn-Norman, R. E. Flori, M. T. Alsaba, and A. S. Amer, “Applications of artificial neural networks in the petroleum industry: A review,” SPE Middle East Oil and Gas Show and Conference (Society of Petroleum Engineers, 2019).

  5. E. Tjoa and C. Guan, “A survey on explainable artificial intelligence (XAI): Toward medical XAI,” Proceedings of the IEEE Transactions on Neural Networks and Learning Systems (2020).

  6. F. Xu, H. Uszkoreit, Y. Du, W. Fan, D. Zhao, and J. Zhu, “Explainable AI: A brief survey on history, research areas, approaches and challenges,” CCF International Conference on Natural Language Processing and Chinese Computing (2019), pp. 563–574.

  7. Explainable AI: Interpreting, Explaining, and Visualizing Deep Learning, Ed. by W. Samek, G. Montavon, A. Vedaldi, L. K. Hansen, and K.-R. Müller (Springer, Berlin, 2019).

    Google Scholar 

  8. Z. C. Lipton, “The mythos of model interpretability,” Queue 16 (3), 31–57 (2018).

    Article  Google Scholar 

  9. N. Cowan, “The many faces of working memory and short-term storage,” Psychon. Bull. Rev. 24 (4), 1158–1170 (2017).

    Article  Google Scholar 

  10. K. Anokhin, O. Ivashkina, K. Toropova, A. Gruzdeva, O. B. Rogozhnikova, V. Plushnin, and I. Fedotov, “Neuronal encoding of object-type and object-place memories in hippocampus and neocortex of young and old mice,” FASEB J. 34 (S1), 1 (2020).

    Article  Google Scholar 

  11. P. Zhigulina, V. Ushakov, S. Kartashov, D. Malakhov, V. Orlov, K. Novikov, A. Korotkova, K. Anokhin, and V. Nourkova, “The architecture of neural networks for enhanced autobiographical memory access: A functional MRI study,” Proc. Comput. Sci. 169, 787–794 (2020).

    Article  Google Scholar 

  12. A. A. Tiunova, N. V. Komissarova, and K. V. Anokhin, “Mapping the neural substrates of recent and remote visual imprinting memory in the chick brain,” Frontiers Physiol. 10, 351–351 (2019).

    Article  Google Scholar 

  13. J. H. Marshel, Y. S. Kim, T. A. Machado, S. Quirin, B. Benson, J. Kadmon, C. Raja, A. Chibukhchyan, C. Ramakrishnan, and M. Inoue, “Cortical layer-specific critical dynamics triggering perception,” Science 365 (6453) (2019).

  14. D. Erhan, Y. Bengio, A. Courville, and P. Vincent, “Visualizing higher-layer features of a deep network,” Technical Report, ICML 2009 Workshop on Learning Feature Hierarchies (Montréal, Canada, 2009).

  15. D. Hubel and T. Wiesel, “Receptive fields of single neurons in the cat’s striate cortex,” J. Physiol. 148, 574–591 (1959).

    Article  Google Scholar 

  16. A. Nguyen, J. Yosinski, and J. Clune, “Understanding neural networks via feature visualization: A survey,” in Explainable AI: Interpreting, Explaining, and Visualizing Deep Learning (2019), pp. 55–76.

    Google Scholar 

  17. A. Nguyen, A. Dosovitskiy, J. Yosinski, T. Brox, and J. Clune, “Synthesizing the preferred inputs for neurons in neural networks via deep generator networks,” Advances in Neural Information Processing Systems 30 (2016), pp. 3395–3403.

    Google Scholar 

  18. K. Simonyan, A. Vedaldi, and A. Zisserman, “Deep inside convolutional networks: Visualizing image classification models and saliency maps,” Workshop at International Conference on Learning Representations (2014).

  19. J. Yosinski, J. Clune, A. Nguyen, T. Fuchs, and H. Lipson, “Understanding neural networks through deep visualization,” Deep Learning Workshop at ICML 2015 (2015).

  20. D. Wei, B. Zhou, A. Torrabla, and W. Freeman, “Understanding intra-class knowledge inside CNN” (2015). https://arxiv.org/abs/1507.02379.

  21. A. Mahendran and A. Vedaldi, “Visualizing deep convolutional neural networks using natural pre-images,” Int. J. Comput. Vision 120 (3), 233–255 (2016).

    Article  MathSciNet  Google Scholar 

  22. A. Nguyen, J. Yosinski, and J. Clune, “Multifaceted feature visualization: Uncovering the different types of features learned by each neuron in deep neural networks” (2016). https://arxiv.org/abs/1602.03616.

  23. I. J. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio, “Generative adversarial nets,” Proceedings of the 27th International Conference on Neural Information Processing Systems (2014), Vol. 2, pp. 2672–2680.

  24. S. Lapuschkin, S. Wӓldchen, A. Binder, G. Montavon, W. Samek, and K.-R. Müller, “Unmasking Clever Hans predictors and assessing what machines really learn,” Nat. Commun. 10 (1) (2019). https://doi.org/10.1038/s41467-019-08987-4

  25. M. Everingham, S. M. A. Eslami, L. Van Gool, C. K. I. Williams, J. Winn, and A. Zisserman, “The Pascal visual object classes challenge: A retrospective,” Int. J. Comput. Vision 111 (1), 98–136 (2015).

    Article  Google Scholar 

  26. S. Lapuschkin, A. Binder, G. Montavon, F. Klauschen, K.-R. Müller, and W. Samek, “On pixel-wise explanations for non-linear classifier decisions by layer-wise relevance propagation,” PLoS ONE 10 (2015).

  27. W. Samek, T. Wiegand, and K.-R. Müller, “Explainable artificial intelligence: Understanding, visualizing, and interpreting deep learning models,” ITU J. ICT Discoveries 1, 39–48 (2019).

    Google Scholar 

  28. W. Samek, A. Binder, G. Montavon, S. Lapushckin, and K.-R. Müller, “Evaluating the visualization of what a deep neural network has learned,” IEEE Trans. Neural Networks Learn. Syst. 28 (11), 2660–2673 (2017).

    Article  MathSciNet  Google Scholar 

  29. A. Shrikumar, P. Greenside, and A. Kundaje, “Learning important features through propagating activation differences,” Proceedings of the 34th International Conference on Machine Learning, PLMR (2017), pp. 3145–3153.

  30. J. T. Springenberg, A. Dosovitskiy, T. Brox, and R. Riedmiller, “Striving for simplicity: The all convolutional net” (2014). https://arxiv.org/abs/1412.6806.

  31. M. Sundararajan, A. Taly, and Q. Yan, “Axiomatic attribution for deep networks,” Proceedings of the International Conference on Machine Learning, ICML (2017), pp. 3319–3328.

  32. D. Smilkov, N. Thorat, B. Kim, F. Viégas, and M. Wattenberg, “Smoothgrad: Removing noise by adding noise,” Workshop on Visualization for Deep Learning, ICML (2017).

  33. B. Zhou, A. Khosla, A. Lapedriza, A. Oliva, and A. Torralba, “Learning deep features for discriminative localization,” Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (2016), pp. 2921–2929.

  34. R. R. Selvaraju, M. Cogswell, A. Das, R. Vedantam, D. Parikh, and D. Batra, “Grad-CAM: Visual explanations from deep networks via gradient-based localization,” Proceedings of the IEEE International Conference on Computer Vision (2017), pp. 618–626.

  35. H. Wang, Z. Wang, M. Du, F. Yang, Z. Zhang, S. Ding, P. Mardziel, and X. Hu, “Score-CAM: Score-weighted visual explanations for convolutional neural networks,” Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (2020), pp. 24–25.

  36. S. M. Lundberg and S.-I. Lee, “A unified approach to interpreting model predictions,” Advances in Neural Information Processing Systems 30 (2017), pp. 4765–4774.

    Google Scholar 

  37. M. D. Zeiler and R. Fergus, “Visualizing and understanding convolutional networks,” European Conference on Computer Vision (Springer, 2014), pp. 818–833.

  38. A. Choromanska, M. Henaff, M. Mathieu, G. B. Arous, and Y. LeCun, “The loss surfaces of multilayer networks,” J. Mach. Learn. Res. 38, 192–204 (2015).

    Google Scholar 

  39. H. Li, Z. Xu, G. Taylor, C. Studer, and T. Goldstein, “Visualizing the loss landscape of neural nets,” Advances in Neural Information Processing Systems 31 (2018), pp. 6389–6399.

    Google Scholar 

  40. L. Dinh, R. Pascanu, S. Bengio, and Y. Bengio, “Sharp minima can generalize for deep nets,” Proceedings of the 34th International Conference on Machine Learning (PMLR) (2017), pp. 1019–1028.

  41. N. S. Keskar, D. Mudigere, J. Nocedal, M. Smelyanskiy, and P. T. P. Tang, “On large-batch training for deep learning: Generalization gap and sharp minima,” 5th International Conference on Learning Representations (ICLR) (2017).

  42. I. J. Goodfellow, O. Vinyals, and A. M. Saxe, “Qualitatively characterizing neural network optimization problems,” International Conference on Learning Representations (2015).

  43. D. J. Im, M. Tao, and K. Branson, “An empirical analysis of deep network loss surfaces” (2016). https://arxiv.org/abs/1612.04010.

  44. M. Abadi, A. Agarwal, P. Barham, E. Brevdo, Z. Chen, C. Citro, G. S. Corrado, A. Davis, J. Dean, M. Devin, S. Ghemawat, I. Goodfellow, A. Harp, G. Irving, M. Isard, Y. Jia, R. Jozefowicz, L. Kaiser, M. Kudlur, J. Levenberg, D. Mané, R. Monga, S. Moore, D. Murray, C. Olah, M. Schuster, J. Shlens, B. Steiner, I. Sutskever, K. Talwar, P. Tucker, V. Vanhoucke, V. Vasudevan, F. Viégas, O. Vinyals, P. Warden, M. Wattenberg, M. Wicke, Y. Yu, and X. Zheng, “TensorFlow: Large-scale machine learning on heterogeneous systems” (2016). https://arxiv.org/abs/1603.04467.

  45. A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, A. Desmaison, A. Kopf, E. Yang, Z. DeVito, M. Raison, A. Tejani, S. Chilamkurthy, B. Steiner, L. Fang, J. Bai, and S. Chintala, “Pytorch: An imperative style, high-performance deep learning library,” Advances in Neural Information Processing Systems 32 (2019), pp. 8024–8035.

    Google Scholar 

  46. A. Kapishnikov, T. Bolukbasi, F. Viégas, and M. Terry, “Xrai: Better attributions through regions,” Proceedings of the IEEE International Conference on Computer Vision (2019), pp. 4948–4957.

  47. B. Kim, M. Wattenberg, J. Gilmer, C. Cai, J. Wexler, F. Viegas, et al., “Interpretability beyond feature attribution: Quantitative testing with concept activation vectors (TCAV),” International Conference on Machine Learning (PMLR) (2018), pp. 2668–2677.

  48. S. Liu, X. Wang, M. Liu, and J. Zhu, “Towards better analysis of machine learning models: A visual analytics perspective,” Visual Inf. 1 (1), 48–56 (2017).

    Article  Google Scholar 

  49. C. Seifert, A. Aamir, A. Balagopalan, D. Jain, A. Sharma, S. Grottel, and S. Gumhold, “Visualizations of deep neural networks in computer vision: A survey,” Transparent Data Mining for Big and Small Data (2017), pp. 123–144.

    Book  Google Scholar 

  50. R. Yu and L. Shi, “A user-based taxonomy for deep learning visualization,” Visual Inf. 2 (3), 147–154 (2018).

    Article  Google Scholar 

  51. Q.-S. Zhang and S.-C. Zhu, “Visual interpretability for deep learning: A survey,” Front. Inf. Technol. Electron. Eng. 19 (1), 27–39 (2018).

    Article  Google Scholar 

  52. Z. Qin, F. Yu, C. Liu, and X. Chen, “How convolutional neural network see the world: A survey of convolutional neural network visualization methods,” Math. Found. Comput. 1 (2), 149–180 (2018).

    Article  Google Scholar 

  53. J. Choo and S. Liu, “Visual analytics for explainable deep learning,” IEEE Comput. Graphics Appl. 38 (4), 84–92 (2018).

    Article  Google Scholar 

  54. F. Hohman, M. Kahng, R. Pienta, and D. H. Chau, “Visual analytics in deep learning: An interrogative survey for the next frontiers,” IEEE Trans. Visualization Comput. Graphics 25 (8), 2674–2693 (2018).

    Article  Google Scholar 

  55. R. Garcia, A. C. Telea, B. C. da Silva, J. Tørresen, and J. L. D. Comba, “A task-and-technique centered survey on visual analytics for deep learning model engineering,” Comput. Graphics 77, 30–49 (2018).

    Article  Google Scholar 

  56. J. Yuan, C. Chen, W. Yang, M. Liu, J. Xia, and S. Liu, “A survey of visual analytics techniques for machine learning,” Comput. Visual Media 7, 3–36 (2021).

    Article  Google Scholar 

  57. A. Chatzimparmpas, R. M. Martins, I. Jusufi, and A. Kerren, “A survey of surveys on the use of visualization for interpreting machine learning models,” Inf. Visualization 19 (3), 207–233 (2020).

    Article  Google Scholar 

Download references

Funding

This work was supported by the Ministry of Science and Higher Education of the Russian Federation, grant no. 075-15-2020-801.

Author information

Authors and Affiliations

  1. Skolkovo Institute of Science and Technology (Skoltech), 121205, Moscow, Russia

    S. A. Matveev, I. V. Oseledets, E. S. Ponomarev & A. V. Chertkov

  2. Marchuk Institute of Numerical Mathematics, Russian Academy of Sciences, 119333, Moscow, Russia

    S. A. Matveev & I. V. Oseledets

Authors
  1. S. A. Matveev
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. I. V. Oseledets
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. E. S. Ponomarev
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. A. V. Chertkov
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding authors

Correspondence to S. A. Matveev, I. V. Oseledets or E. S. Ponomarev.

Additional information

Translated by I. Ruzanova

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Matveev, S.A., Oseledets, I.V., Ponomarev, E.S. et al. Overview of Visualization Methods for Artificial Neural Networks. Comput. Math. and Math. Phys. 61, 887–899 (2021). https://doi.org/10.1134/S0965542521050134

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1134/S0965542521050134

Keywords:

Search

Navigation