Self-organising Neural Network Hierarchy
- 1 Mentions
- 115 Downloads
Abstract
Mammalian brains exhibit functional self-organisation between different neocortical regions to form virtual hierarchies from a physical 2D sheet. We propose a biologically-inspired self-organizing neural network architecture emulating the same. The network is composed of autoencoder units and driven by a meta-learning rule based on maximizing the Shannon entropy of latent representations of the input, which optimizes the receptive field placement of each unit within a feature map. Unlike Neural Architecture Search, here both the network parameters and the architecture are learned simultaneously. In a case study on image datasets, we observe that the meta-learning rule causes a functional hierarchy to form, and leads to learning progressively better topological configurations and higher classification performance overall, starting from randomly initialized architectures. In particular, our approach yields competitive performance in terms of classification accuracy compared to optimal handcrafted architecture(s) with desirable topological features for this network type, on both MNIST and CIFAR-10 datasets, even though it is not as significant for the latter.
Keywords
Biologically plausible networks Self-supervised learning Greedy trainingReferences
- 1.Chen, L.C., et al.: Searching for efficient multi-scale architectures for dense image prediction. In: Advances in Neural Information Processing Systems, pp. 8699–8710 (2018)Google Scholar
- 2.Corominas-Murtra, B., Goñi, J., Solé, R.V., Rodríguez-Caso, C.: On the origins of hierarchy in complex networks. Proc. Natl. Acad. Sci. 110(33), 13316–13321 (2013)MathSciNetCrossRefGoogle Scholar
- 3.Elsken, T., Metzen, J.H., Hutter, F.: Neural architecture search: a survey. arXiv preprint arXiv:1808.05377 (2018)
- 4.Friston, K.: A free energy principle for biological systems. Entropy 14(11), 2100–2121 (2012)MathSciNetCrossRefGoogle Scholar
- 5.Goldberg, E.: Gradiental approach to neocortical functional organization. J. Clin. Exp. Neuropsychol. 11(4), 489–517 (1989)CrossRefGoogle Scholar
- 6.Huang, G., Sun, Yu., Liu, Z., Sedra, D., Weinberger, K.Q.: Deep networks with stochastic depth. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9908, pp. 646–661. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-46493-0_39CrossRefGoogle Scholar
- 7.Kandel, E.R., et al.: Principles of Neural Science, vol. 4. McGraw-Hill, New York (2000)Google Scholar
- 8.Krizhevsky, A., Hinton, G., et al.: Learning multiple layers of features from tiny images (2009) Google Scholar
- 9.Larsson, G., Maire, M., Shakhnarovich, G.: FractalNet: ultra-deep neural networks without residuals. arXiv preprint arXiv:1605.07648 (2016)
- 10.Lee, C.Y., Xie, S., Gallagher, P., Zhang, Z., Tu, Z.: Deeply-supervised nets. In: Artificial Intelligence and Statistics, pp. 562–570 (2015)Google Scholar
- 11.Lin, M., Chen, Q., Yan, S.: Network in network. arXiv preprint arXiv:1312.4400 (2013)
- 12.Lin, Z., Memisevic, R., Konda, K.: How far can we go without convolution: improving fully-connected networks. arXiv preprint arXiv:1511.02580 (2015)
- 13.Purwani, S., Nahar, J., Twining, C.: Analyzing bin-width effect on the computed entropy. In: AIP Conference Proceedings, vol. 1868, p. 040008. AIP Publishing LLC (2017)Google Scholar
- 14.Romero, A., Ballas, N., Kahou, S.E., Chassang, A., Gatta, C., Bengio, Y.: FitNets: hints for thin deep nets. arXiv preprint arXiv:1412.6550 (2014)
- 15.Russell, S.J., Norvig, P.: Artificial Intelligence: A Modern Approach. Pearson Education Limited, Kuala Lumpur (2016)zbMATHGoogle Scholar
- 16.Saxe, A.M., et al.: On the information bottleneck theory of deep learning. J. Stat. Mech. Theory Exp. 2019(12), 124020 (2019)CrossRefGoogle Scholar
- 17.Shwartz-Ziv, R., Tishby, N.: Opening the black box of deep neural networks via information. arXiv preprint arXiv:1703.00810 (2017)
- 18.Springenberg, J.T., Dosovitskiy, A., Brox, T., Riedmiller, M.: Striving for simplicity: the all convolutional net. arXiv preprint arXiv:1412.6806 (2014)
- 19.Srivastava, R.K., Greff, K., Schmidhuber, J.: Highway networks. arXiv preprint arXiv:1505.00387 (2015)
- 20.Stanley, K.O., Miikkulainen, R.: Evolving neural networks through augmenting topologies. Evol. Comput. 10(2), 99–127 (2002)CrossRefGoogle Scholar
- 21.Tishby, N., Zaslavsky, N.: Deep learning and the information bottleneck principle. In: 2015 IEEE Information Theory Workshop (ITW), pp. 1–5. IEEE (2015)Google Scholar
- 22.Zoph, B., Le, Q.V.: Neural architecture search with reinforcement learning. arXiv preprint arXiv:1611.01578 (2016)
- 23.Zoph, B., Vasudevan, V., Shlens, J., Le, Q.V.: Learning transferable architectures for scalable image recognition. In: Proceedings of the IEEE Conference on Computer Vision And Pattern Recognition, pp. 8697–8710 (2018)Google Scholar