Hierarchical Modeling with Neurodynamical Agglomerative Analysis

Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 12396)


We propose a new analysis technique for neural networks, Neurodynamical Agglomerative Analysis (NAA), an analysis pipeline designed to compare class representations within a given neural network model. The proposed pipeline results in a hierarchy of class relationships implied by the network representation, i.e. a semantic hierarchy analogous to a human-made ontological view of the relevant classes. We use networks pretrained on the ImageNet benchmark dataset to infer semantic hierarchies and show the similarity to human-made semantic hierarchies by comparing them with the WordNet ontology. Further, we show using MNIST training experiments that class relationships extracted using NAA appear to be invariant to random weight initializations, tending toward equivalent class relationships across network initializations in sufficiently parameterized networks.


Neural network theory Deep learning Cognitive models 


  1. 1.
    Bach, S., et al.: On pixel-wise explanations for non-linear classifier decisions by layer-wise relevance propagation. PLoS ONE 10(7), e0130140 (2015)CrossRefGoogle Scholar
  2. 2.
    Raghu, M., et al.: SVCCA: singular vector canonical correlation analysis for deep learning dynamics and interpretability. In: Advances in Neural Information Processing Systems (2017)Google Scholar
  3. 3.
    Deng, L.: The MNIST database of handwritten digit images for machine learning research [best of the web]. IEEE Signal Process. Mag. 29(6), 141–142 (2012)CrossRefGoogle Scholar
  4. 4.
    Deng, J., et al.: ImageNet: a large-scale hierarchical image database. In: 2009 IEEE Conference on Computer Vision and Pattern Recognition. IEEE (2009)Google Scholar
  5. 5.
    Fellbaum, C.: WordNet. In: Poli, R., Healy, M., Kameas, A. (eds.) Theory and Applications of Ontology: Computer Applications, pp. 231–243. Springer, Dordrecht (2010). Scholar
  6. 6.
    Li, Y., et al.: Convergent learning: do different neural networks learn the same representations?. In: FE@ NIPS (2015)Google Scholar
  7. 7.
    Hardoon, D.R., Szedmak, S., Shawe-Taylor, J.: Canonical correlation analysis: an overview with application to learning methods. Neural Comput. 16(12), 2639–2664 (2004)CrossRefGoogle Scholar
  8. 8.
    Jaeger, H.: Controlling recurrent neural networks by conceptors. arXiv preprint arXiv:1403.3369 (2014)
  9. 9.
    Müllner, D.: Modern hierarchical, agglomerative clustering algorithms. arXiv preprint arXiv:1109.2378 (2011)
  10. 10.
    Pedregosa, F., et al.: Scikit-learn: machine learning in Python. J. Mach. Learn. Res. 12(Oct), 2825–2830 (2011)MathSciNetzbMATHGoogle Scholar
  11. 11.
    Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556 (2014)
  12. 12.
    He, K., et al.: Deep residual learning for image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (2016)Google Scholar
  13. 13.
    Szegedy, C., et al.: Going deeper with convolutions. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (2015)Google Scholar

Copyright information

© Springer Nature Switzerland AG 2020

Authors and Affiliations

  1. 1.Universität OsnabrückOsnabrückGermany
  2. 2.DFKI Laboratory Niedersachsen, Plan-Based Robot Control DepartmentOsnabrückGermany

Personalised recommendations