Advertisement

Self-organising Neural Network Hierarchy

  • Satya BorgohainEmail author
  • Gideon KowadloEmail author
  • David RawlinsonEmail author
  • Christoph BergmeirEmail author
  • Kok LooEmail author
  • Harivallabha RangarajanEmail author
  • Levin KuhlmannEmail author
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 12576)

Abstract

Mammalian brains exhibit functional self-organisation between different neocortical regions to form virtual hierarchies from a physical 2D sheet. We propose a biologically-inspired self-organizing neural network architecture emulating the same. The network is composed of autoencoder units and driven by a meta-learning rule based on maximizing the Shannon entropy of latent representations of the input, which optimizes the receptive field placement of each unit within a feature map. Unlike Neural Architecture Search, here both the network parameters and the architecture are learned simultaneously. In a case study on image datasets, we observe that the meta-learning rule causes a functional hierarchy to form, and leads to learning progressively better topological configurations and higher classification performance overall, starting from randomly initialized architectures. In particular, our approach yields competitive performance in terms of classification accuracy compared to optimal handcrafted architecture(s) with desirable topological features for this network type, on both MNIST and CIFAR-10 datasets, even though it is not as significant for the latter.

Keywords

Biologically plausible networks Self-supervised learning Greedy training 

References

  1. 1.
    Chen, L.C., et al.: Searching for efficient multi-scale architectures for dense image prediction. In: Advances in Neural Information Processing Systems, pp. 8699–8710 (2018)Google Scholar
  2. 2.
    Corominas-Murtra, B., Goñi, J., Solé, R.V., Rodríguez-Caso, C.: On the origins of hierarchy in complex networks. Proc. Natl. Acad. Sci. 110(33), 13316–13321 (2013)MathSciNetCrossRefGoogle Scholar
  3. 3.
    Elsken, T., Metzen, J.H., Hutter, F.: Neural architecture search: a survey. arXiv preprint arXiv:1808.05377 (2018)
  4. 4.
    Friston, K.: A free energy principle for biological systems. Entropy 14(11), 2100–2121 (2012)MathSciNetCrossRefGoogle Scholar
  5. 5.
    Goldberg, E.: Gradiental approach to neocortical functional organization. J. Clin. Exp. Neuropsychol. 11(4), 489–517 (1989)CrossRefGoogle Scholar
  6. 6.
    Huang, G., Sun, Yu., Liu, Z., Sedra, D., Weinberger, K.Q.: Deep networks with stochastic depth. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9908, pp. 646–661. Springer, Cham (2016).  https://doi.org/10.1007/978-3-319-46493-0_39CrossRefGoogle Scholar
  7. 7.
    Kandel, E.R., et al.: Principles of Neural Science, vol. 4. McGraw-Hill, New York (2000)Google Scholar
  8. 8.
    Krizhevsky, A., Hinton, G., et al.: Learning multiple layers of features from tiny images (2009) Google Scholar
  9. 9.
    Larsson, G., Maire, M., Shakhnarovich, G.: FractalNet: ultra-deep neural networks without residuals. arXiv preprint arXiv:1605.07648 (2016)
  10. 10.
    Lee, C.Y., Xie, S., Gallagher, P., Zhang, Z., Tu, Z.: Deeply-supervised nets. In: Artificial Intelligence and Statistics, pp. 562–570 (2015)Google Scholar
  11. 11.
    Lin, M., Chen, Q., Yan, S.: Network in network. arXiv preprint arXiv:1312.4400 (2013)
  12. 12.
    Lin, Z., Memisevic, R., Konda, K.: How far can we go without convolution: improving fully-connected networks. arXiv preprint arXiv:1511.02580 (2015)
  13. 13.
    Purwani, S., Nahar, J., Twining, C.: Analyzing bin-width effect on the computed entropy. In: AIP Conference Proceedings, vol. 1868, p. 040008. AIP Publishing LLC (2017)Google Scholar
  14. 14.
    Romero, A., Ballas, N., Kahou, S.E., Chassang, A., Gatta, C., Bengio, Y.: FitNets: hints for thin deep nets. arXiv preprint arXiv:1412.6550 (2014)
  15. 15.
    Russell, S.J., Norvig, P.: Artificial Intelligence: A Modern Approach. Pearson Education Limited, Kuala Lumpur (2016)zbMATHGoogle Scholar
  16. 16.
    Saxe, A.M., et al.: On the information bottleneck theory of deep learning. J. Stat. Mech. Theory Exp. 2019(12), 124020 (2019)CrossRefGoogle Scholar
  17. 17.
    Shwartz-Ziv, R., Tishby, N.: Opening the black box of deep neural networks via information. arXiv preprint arXiv:1703.00810 (2017)
  18. 18.
    Springenberg, J.T., Dosovitskiy, A., Brox, T., Riedmiller, M.: Striving for simplicity: the all convolutional net. arXiv preprint arXiv:1412.6806 (2014)
  19. 19.
    Srivastava, R.K., Greff, K., Schmidhuber, J.: Highway networks. arXiv preprint arXiv:1505.00387 (2015)
  20. 20.
    Stanley, K.O., Miikkulainen, R.: Evolving neural networks through augmenting topologies. Evol. Comput. 10(2), 99–127 (2002)CrossRefGoogle Scholar
  21. 21.
    Tishby, N., Zaslavsky, N.: Deep learning and the information bottleneck principle. In: 2015 IEEE Information Theory Workshop (ITW), pp. 1–5. IEEE (2015)Google Scholar
  22. 22.
    Zoph, B., Le, Q.V.: Neural architecture search with reinforcement learning. arXiv preprint arXiv:1611.01578 (2016)
  23. 23.
    Zoph, B., Vasudevan, V., Shlens, J., Le, Q.V.: Learning transferable architectures for scalable image recognition. In: Proceedings of the IEEE Conference on Computer Vision And Pattern Recognition, pp. 8697–8710 (2018)Google Scholar

Copyright information

© Springer Nature Switzerland AG 2020

Authors and Affiliations

  1. 1.Monash UniversityClaytonAustralia
  2. 2.CerenautRichmondAustralia

Personalised recommendations