Abstract
Neural network pruning has attracted enormous attention since it offers a promising prospect to facilitate the deployment of deep neural networks on resource-limited devices. However, the core of most existing methods lies in the criteria of selection of filters which were pre-defined by researchers. With the advancement of network pruning research, the criteria are becoming increasingly complex. In this paper, we propose a brain-inspired filter pruning algorithm for deep neural networks, which requires no selection criteria. Inspired by the reorganization of brain function in humans when irreversible damage occurs, we treat the weight to be pruning as damaged neurons, and complete the reorganization of the network function in the novel training process proposed in this paper. After pruning, the kept parameters can take over the function of those that have been pruned. The pruning method is widely applicable to common architectures and does not require any artificially designed filter importance measurement functions. As the first attempt on weight-importance irrelevant pruning, BFRIFP provides novel insight into the network pruning problem. Experiments on CIFAR-10 and ImageNet demonstrate the effectiveness of our new perspective of network pruning compared to traditional network pruning algorithms.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Change history
07 September 2021
The originally published article identified the incorrect corresponding author for the paper. The correct corresponding author has been updated now.
References
Molchanov, P., Mallya, A., Tyree, S., Frosio, I., Kautz, J.: Importance estimation for neural network pruning. In: 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 11264–11272 (2019)
Frankle, J., Carbin, M.: The lottery ticket hypothesis: finding sparse, trainable neural networks. In: International Conference on Learning Representations (2018)
Liu, Z., Sun, M., Zhou, T., Huang, G., Darrell, T.: Rethinking the value of network pruning. arXiv preprint arXiv:1810.05270 (2018)
Wang, W., Wang, A., Limin, Yu., Han, X., Jiang, G., Weng, C., Zhang, H., Zhou, Z.: Constraint-induced movement therapy promotes brain functional reorganization in stroke patients with hemiplegia. Neural Regen. Res. 7(32), 2548 (2012)
Leclerc, C., Saint-Amour, D., Lavoie, M.E., Lassonde, M., Lepore, F.: Brain functional reorganization in early blind humans revealed by auditory event-related potentials. NeuroReport 11(3), 545–550 (2000)
Liou, S.: (2010)
Levin, H.S., Grafman, J.: Cerebral reorganization of function after brain damage. Oxford University Press (2000)
Marsh, E.B., Hillis, A.E., the role of reorganization: Recovery from aphasia following brain injury. Prog. Brain Res. 157, 143–156 (2006)
Sze, V., Chen, Y.-H., Yang, T.-J., Emer, J.S.: Efficient processing of deep neural networks: a tutorial and survey. Proc. IEEE 105(12), 2295–2329 (2017)
Hassibi, B., Stork, D.G.: Second order derivatives for network pruning: optimal brain surgeon. Adv. Neural. Inf. Process. Syst. 5, 164–171 (1992)
Han, S., Pool, J., Tran, J., Dally, W.J.: Learning both weights and connections for efficient neural networks. In NIPS’15 Proceedings of the 28th International Conference on Neural Information Processing Systems - Volume 1, pp. 1135–1143 (2015)
Liu, Z., Xu, J., Peng, X., Xiong, R.: Frequency-domain dynamic pruning for convolutional neural networks. In: Advances in Neural Information Processing Systems, pp. 1043–1053 (2018)
Wen, W., Wu, C., Wang, Y., Chen, Y., Li, H.: Learning structured sparsity in deep neural networks. In: Proceedings of the 30th International Conference on Neural Information Processing Systems, pp. 2074–2082 (2016)
Huang, Z., Wang, N.: Data-driven sparse structure selection for deep neural networks. In: Proceedings of the European Conference on Computer Vision (ECCV), pp. 317–334 (2018)
Lin, S., et al.: Towards optimal structured CNN pruning via generative adversarial learning. In: 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2790–2799 (2019)
Panaretos, V.M., Zemel, Y.: Statistical aspects of wasserstein distances. arXiv preprint arXiv:1806.05500, 6(1), 405–431 (2019)
Krizhevsky, A., Hinton, G., et al.: Learning multiple layers of features from tiny images (2009)
Russakovsky, O., Deng, J., Hao, S., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., et al.: Imagenet large scale visual recognition challenge. Int. J. Comput. Vision 115(3), 211–252 (2015)
Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556 (2014)
He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 770–778 (2016)
Li, H., Kadav, A., Durdanovic, I., Samet, H., Graf, H.P.: Pruning filters for efficient convnets. arXiv preprint arXiv:1608.08710 (2016)
Zhao, C., Ni, B., Zhang, J., Zhao, Q., Zhang, W., Tian, Q.: Variational convolutional neural network pruning. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2780–2789 (2019)
Lin, M.: Hrank: filter pruning using high-rank feature map. In: 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1529–1538 (2020)
Yu, R., et al.: Nisp: pruning networks using neuron importance score propagation. In: 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9194–9203 (2018)
Lin, M., et al.: Filter sketch for network pruning. arXiv preprint arXiv:2001.08514 (2020)
Luo, J.-H., Wu, J., Lin, W.: Thinet: a filter level pruning method for deep neural network compression. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 5058–5066 (2017)
Acknowledgments
The research presented in this paper was partially supported by the National Science and Technology Major Project from Minister of Science and Technology, China (Grant No. 2018AAA0103100), the Key Research Program of Frontier Sciences of the Chinese Academy of Sciences under Grant QYZDY-SSW-JSC034, and Shanghai Municipal Science and Technology Major Project (ZHANGJIANG LAB) under Grant 2018SHZDZX01.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2021 Springer Nature Switzerland AG
About this paper
Cite this paper
Qiu, S., Gu, Y., Zhang, X. (2021). BFRIFP: Brain Functional Reorganization Inspired Filter Pruning. In: Farkaš, I., Masulli, P., Otte, S., Wermter, S. (eds) Artificial Neural Networks and Machine Learning – ICANN 2021. ICANN 2021. Lecture Notes in Computer Science(), vol 12894. Springer, Cham. https://doi.org/10.1007/978-3-030-86380-7_2
Download citation
DOI: https://doi.org/10.1007/978-3-030-86380-7_2
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-86379-1
Online ISBN: 978-3-030-86380-7
eBook Packages: Computer ScienceComputer Science (R0)