Abstract
In recent years, channel pruning is one of the important methods for deep model compression. But the resulting model still has tremendous redundant feature maps. In this paper, we propose a novel method, namely federated pruning algorithm, to achieve narrower model with negligible performance degradation. Different from many existing approaches, the federated pruning algorithm removes all filters in the pre-trained model together with their connecting feature map by combining the weights with the importance of the channels, rather than pruning the network in terms of a single criterion. Finally, we fine-tune the resulting model to restore network performance. Extensive experiments demonstrate the effectiveness of federated pruning algorithm. VGG-19 network pruned by federated pruning algorithm on CIFAR-10 achieves 92.5% reduction in total parameters and \(13.58\times \) compression ratio with only 0.23% decrease in accuracy. Meanwhile, tested on SVHN, VGG-19 achieves 94.5% reduction in total parameters and \(18.01\times \) compression ratio with only 0.43% decrease in accuracy.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Srivastava, R.K., Greff, K., Schmidhuber, J.: Training very deep networks. In: Advances in Neural Information Processing Systems (2015)
Schroff, F., Kalenichenko, D., Philbin, J.: Facenet: a unified embedding for face recognition and clustering. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (2015)
Pei, S., Tang, F., Ji, Y., Fan, J., Ning, Z.: Localized traffic sign detection with multi-scale deconvolution networks (2018)
Pei, S., et al.: 3DACN: 3D augmented convolutional network for time series data. Inf. Sci. 513, 17–29 (2020)
Ma, N., Zhang, X., Zheng, H.-T., Sun, J.: ShuffleNet V2: practical guidelines for efficient CNN architecture design. In: Ferrari, V., Hebert, M., Sminchisescu, C., Weiss, Y. (eds.) Computer Vision – ECCV 2018. LNCS, vol. 11218, pp. 122–138. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-01264-9_8
Sandler, M., Howard, A., Zhu, M., Zhmoginov, A., Chen, L.-C.: Mobilenetv 2: Inverted residuals and linear bottlenecks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (2018)
Rastegari, M., Ordonez, V., Redmon, J., Farhadi, A.: XNOR-Net: ImageNet classification using binary convolutional neural networks. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9908, pp. 525–542. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-46493-0_32
Zhou, A., Yao, A., Guo, Y., Xu, L., Chen, Y.: Incremental network quantization: Towards lossless cnns with low-precision weights. arXiv preprint arXiv:1702.03044 (2017)
Han, S., Mao, H., Dally, W.J.: Compressing deep neural networks with pruning trained quantization and huffman coding. arXiv preprint (2015)
Liu, Z., Li, J., Shen, Z., Huang, G., Yan, S., Zhang, C.: Learning efficient convolutional networks through network slimming. In: Proceedings of the IEEE International Conference on Computer Vision (2017)
Ye, J., Lu, X., Lin, Z., Wang, J.Z.: Rethinking the smaller-norm-less-informative assumption in channel pruning of convolution layers. arXiv preprint arXiv:1802.00124 (2018)
Han, S., Pool, J., Tran, J., Dally, W.: Learning both weights and connections for efficient neural network. In: Advances in Neural Information Processing Systems (2015)
Hu, H., Peng, R., Tai, Y.-W., Tang, C.-K.: Network trimming: a data-driven neuron pruning approach towards efficient deep architectures. arXiv preprint arXiv:1607.03250 (2016)
Li, H., Kadav, A., Durdanovic, I., Samet, H., Graf, H.P.: Pruning filters for efficient convnets. arXiv preprint arXiv:1608.08710 (2016)
He, Y., Zhang, X., Sun, J.: Channel pruning for accelerating very deep neural networks. In: Proceedings of the IEEE International Conference on Computer Vision (2017)
LeCun, Y., Denker, J.S., Solla, S.A.: Optimal brain damage. In: Advances in Neural Information Processing Systems (1990)
Guo, Y., Yao, A., Chen, Y.: Dynamic network surgery for efficient DNNs. In: Advances in Neural Information Processing Systems (2016)
Louizos, C.M.W., Kingma, D.P.: Learning sparse neural networks through \( l\_0 \) regularization. arXiv preprint arXiv:1712.01312 (2017)
Qiu, M., Sha, E.H.-M.: Cost minimization while satisfying hard/soft timing constraints for heterogeneous embedded systems. In: ACM Transactions on Design Automation of Electronic Systems (TODAES) (2009)
Qiu, M., et al.: Data allocation for hybrid memory with genetic algorithm. IEEE Trans. Emerg. Top. Comput. 3(4), 544–555 (2015)
Shao, Z., Xue, C., Zhuge, Q., Qiu, M., Xiao, B., Sha, E.H.-M.: Security protection and checking for embedded system integration against buffer overflow attacks via hardware/software. IEEE Trans. Comput. 55, 443–453 (2006)
Qiu, H., Qiu, M., Zhihui, L., Memmi, G.: An efficient key distribution system for data fusion in v2x heterogeneous networks. Inf. Fusion 50, 212–220 (2019)
Song, Y., Gao, S., Li, Y., Jia, L., Li, Q., Pang, F.: Distributed attention-based temporal convolutional network for remaining useful life prediction. IEEE Int. Things J. https://doi.org/10.1109/JIOT.2020.3004452
Acknowledgement
We would like to thank the anonymous reviewers for their invaluable comments. This work was partially funded by the National Natural Science Foundation of China under Grant (61975124, 61332009, and 61775139), the Shanghai Natural Science Foundation(20ZR1438500), the Open Project Program of Shanghai Key Laboratory of Data Science (2020090600003), and the Open Project Funding from the State Key Lab of Computer Architecture, ICT, CAS under Grant CARCH201807. Any opinions, findings and conclusions expressed in this paper are those of the authors and do not necessarily reflect the views of the sponsors.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2020 Springer Nature Switzerland AG
About this paper
Cite this paper
Pei, S., Wu, Y., Qiu, M. (2020). Neural Network Compression and Acceleration by Federated Pruning. In: Qiu, M. (eds) Algorithms and Architectures for Parallel Processing. ICA3PP 2020. Lecture Notes in Computer Science(), vol 12453. Springer, Cham. https://doi.org/10.1007/978-3-030-60239-0_12
Download citation
DOI: https://doi.org/10.1007/978-3-030-60239-0_12
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-60238-3
Online ISBN: 978-3-030-60239-0
eBook Packages: Mathematics and StatisticsMathematics and Statistics (R0)