Abstract
We present a new method for structured pruning of neural networks, based on the recently proposed neuron merging trick in which following a pruning operation, the weights of the next layer are suitably modified. By a rigorous mathematical analysis of the neuron merging technique we prove an upper bound on the reconstruction error. This bound defines a new objective function for pruning-and-merging. Our new optimal algorithm provably achieves the lowest objective cost among all possible prune-and-merge strategies. We also show empirically that nuclear norm regularization can be used to obtain even better pruning-and-merging accuracy; this finding is supported by our theoretical analysis.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
Notes
- 1.
The bias also needs to be taken into account - see [9, Section 6.1] for details how to do that.
References
Church, R.L.: Beamr: an exact and approximate model for the p-median problem. Comput. Operat. Res. 35(2), 417–426 (2008)
Enderich, L., Timm, F., Burgard, W.: Holistic filter pruning for efficient deep neural networks. In: Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 2596–2605 (2021)
Hakimi, S.L.: Optimum distribution of switching centers in a communication network and some related graph theoretic problems. Oper. Res. 13(3), 462–475 (1965)
He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 770–778 (2016)
He, Y., Kang, G., Dong, X., Fu, Y., Yang, Y.: Soft filter pruning for accelerating deep convolutional neural networks. arXiv preprint arXiv:1808.06866 (2018)
He, Y., Liu, P., Wang, Z., Hu, Z., Yang, Y.: Filter pruning via geometric median for deep convolutional neural networks acceleration. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4340–4349 (2019)
He, Y., Zhang, X., Sun, J.: Channel pruning for accelerating very deep neural networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1389–1397 (2017)
Hoefler, T., Alistarh, D., Ben-Nun, T., Dryden, N., Peste, A.: Sparsity in deep learning: pruning and growth for efficient inference and training in neural networks. arXiv preprint arXiv:2102.00554 (2021)
Kim, W., Kim, S., Park, M., Jeon, G.: Neuron merging: compensating for pruned neurons. In: Advances in Neural Information Processing Systems, vol. 33 (2020)
Krizhevsky, A., Hinton, G., et al.: Learning multiple layers of features from tiny images (2009)
Le, D.H., Hua, B.S.: Network pruning that matters: a case study on retraining variants. In: International Conference on Learning Representations (2021). https://openreview.net/forum?id=Cb54AMqHQFP
Li, T., Li, J., Liu, Z., Zhang, C.: Few sample knowledge distillation for efficient network compression. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14639–14647 (2020)
Liu, Z., Li, J., Shen, Z., Huang, G., Yan, S., Zhang, C.: Learning efficient convolutional networks through network slimming. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 2736–2744 (2017)
Luo, J.H., Wu, J., Lin, W.: Thinet: a filter level pruning method for deep neural network compression. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 5058–5066 (2017)
Mao, H., et al.: Exploring the granularity of sparsity in convolutional neural networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pp. 13–20 (2017)
Nie, F., Huang, H., Ding, C.: Low-rank matrix recovery via efficient Schatten p-norm minimization. In: Twenty-sixth AAAI Conference on Artificial Intelligence (2012)
Recht, B., Xu, W., Hassibi, B.: Necessary and sufficient conditions for success of the nuclear norm heuristic for rank minimization. In: 2008 47th IEEE Conference on Decision and Control, pp. 3065–3070. IEEE (2008)
Reese, J.: Solution methods for the p-median problem: an annotated bibliography. Netw. Int. J. 48(3), 125–142 (2006)
Renda, A., Frankle, J., Carbin, M.: Comparing rewinding and fine-tuning in neural network pruning. In: International Conference on Learning Representations (2020). https://openreview.net/forum?id=S1gSj0NKvB
Russakovsky, O., et al.: Imagenet large scale visual recognition challenge. Int. J. Comput. Vis. 115(3), 211–252 (2015)
Vadera, S., Ameen, S.: Methods for pruning deep neural networks. arXiv preprint arXiv:2011.00241 (2020)
You, Z., Yan, K., Ye, J., Ma, M., Wang, P.: Gate decorator: Global filter pruning method for accelerating deep convolutional neural networks. arXiv preprint arXiv:1909.08174 (2019)
Zagoruyko, S., Komodakis, N.: Wide residual networks. arXiv preprint arXiv:1605.07146 (2016)
Zhou, D., et al.: Go wide, then narrow: efficient training of deep thin networks. In: International Conference on Machine Learning, pp. 11546–11555. PMLR (2020)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2022 Springer Nature Switzerland AG
About this paper
Cite this paper
Goldberg, F., Lubarsky, Y., Gaissinski, A., Botchan, D., Kisilev, P. (2022). Pruning Neural Nets by Optimal Neuron Merging. In: El Yacoubi, M., Granger, E., Yuen, P.C., Pal, U., Vincent, N. (eds) Pattern Recognition and Artificial Intelligence. ICPRAI 2022. Lecture Notes in Computer Science, vol 13363. Springer, Cham. https://doi.org/10.1007/978-3-031-09037-0_56
Download citation
DOI: https://doi.org/10.1007/978-3-031-09037-0_56
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-09036-3
Online ISBN: 978-3-031-09037-0
eBook Packages: Computer ScienceComputer Science (R0)