Abstract
Plastic neural networks provide a new type of biological heuristic method for “meta-learning” or “learning to learn”, opening a new door for research on lifelong learning and fast memory of artificial intelligence. They can combine typical neural networks with the famous Hebb’s rule in biological neurology, and uses Hebbian trace to express the connection strength between neurons. However, redundancy exists in previously designed plastic neural networks since not all neurons have strong connections. In this paper, we first time propose a model compression strategy for an RNN-based plastic neural network in a maze exploration scenario. With a reinforcement learning process, the network is trained and the hidden neurons have different variation trends. As convergent hidden neurons are conceptually able to memory some invariant feature of the maze, connections of nonconvergent hidden neurons are skillfully pruned. We successfully realize our approach and achieve plastic neural network compression in experiments. While ensuring the performance of the algorithm, the compression rate reaches 3.8 and the speedup rate is about 4.1 in theory and up to 16.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Miconi, T., Stanley, K., Clune, J.: Differentiable plasticity: training plastic neural networks with backpropagation. In: International Conference on Machine Learning, pp. 3559–3568. PMLR (2018)
Miconi, T., Rawal, A., Clune, J., Stanley, K.O.: Backpropamine: training self-modifying neural networks with differentiable neuromodulated plasticity. arXiv preprint arXiv:2002.10585 (2020)
Najarro, E., Risi, S.: Meta-learning through Hebbian plasticity in random networks. arXiv preprint arXiv:2007.02686 (2020)
Hebb, D.O.: The Organization of Behavior; A Neuropsycholocigal Theory, vol. 62, p. 78 (1949). A Wiley Book in Clinical Psychology
Denil, M., Shakibi, B., Dinh, L., Ranzato, M., De Freitas, N.: Predicting parameters in deep learning. arXiv preprint arXiv:1306.0543 (2013)
Cheng, Y., Wang, D., Zhou, P., Zhang, T.: A survey of model compression and acceleration for deep neural networks. arXiv preprint arXiv:1710.09282 (2017)
Mao, H., et al.: Exploring the regularity of sparse structure in convolutional neural networks. arXiv preprint arXiv:1705.08922 (2017)
Liu, Z., Li, J., Shen, Z., Huang, G., Yan, S., Zhang, C.: Learning efficient convolutional networks through network slimming. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 2736–2744 (2017)
Denton, E., Zaremba, W., Bruna, J., LeCun, Y., Fergus, R.: Exploiting linear structure within convolutional networks for efficient evaluation. arXiv preprint arXiv:1404.0736 (2014)
Tulloch, A., Jia, Y.: High performance ultra-low-precision convolutions on mobile devices. arXiv preprint arXiv:1712.02427 (2017)
Howard, A.G., et al.: MobileNets: efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv:1704.04861 (2017)
Zhang, X., Zhou, X., Lin, M., Sun, J.: ShuffleNet: an extremely efficient convolutional neural network for mobile devices. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 6848–6856 (2018)
Polino, A., Pascanu, R., Alistarh, D.: Model compression via distillation and quantization. arXiv preprint arXiv:1802.05668 (2018)
Anil, R., Pereyra, G., Passos, A., Ormandi, R., Dahl, G.E., Hinton, G.E.: Large scale distributed neural network training through online distillation. arXiv preprint arXiv:1804.03235 (2018)
Moczulski, M., Denil, M., Appleyard, J., de Freitas, N.: ACDC: a structured efficient linear layer. arXiv preprint arXiv:1511.05946 (2015)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2021 Springer Nature Switzerland AG
About this paper
Cite this paper
Yu, B., Huang, W., Lan, L., Tang, Y. (2021). Model Compression for a Plasticity Neural Network in a Maze Exploration Scenario. In: Mantoro, T., Lee, M., Ayu, M.A., Wong, K.W., Hidayanto, A.N. (eds) Neural Information Processing. ICONIP 2021. Communications in Computer and Information Science, vol 1516. Springer, Cham. https://doi.org/10.1007/978-3-030-92307-5_57
Download citation
DOI: https://doi.org/10.1007/978-3-030-92307-5_57
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-92306-8
Online ISBN: 978-3-030-92307-5
eBook Packages: Computer ScienceComputer Science (R0)