Skip to main content
SpringerLink
Log in

Progressive kernel pruning CNN compression method with an adjustable input channel

  • Published:
Applied Intelligence Aims and scope Submit manuscript

Abstract

Deep neural network pruning is an effective model compression and acceleration method. In the initial pruning stage, maintaining the integrity of the input channel of the convolution layer is very important to improve the performance of the pruning model. This paper proposes a two-stage multi-strategy progressive kernel pruning method with adjustable input channels. First, a Hybrid Norm Sparse Index (HNSI) is defined as the basis for selecting the number of retained kernels, and then a two-stage progressive pruning technique is adopted. In the first stage, HNSI is used in the groups for moderate kernel pruning. HNSI in the group can reserve at least one kernel in each group, allowing the input channel information of each layer to be mapped to the next layer and ensuring better network optimization through moderate pruning. In the second stage, HNSI is used in the layers for adjustable full kernel pruning. At this stage, the HNSI of each layer is the basis of preserving the kernel number, and the pruning process is divided into two strategies. The first strategy is kernel pruning in the low-level layer. On the basis of the overall kernel pruning in the layer, each group is forced to retain at least one kernel, thus ensuring that the primary feature of each input channel can be transmitted to the next layer. The second strategy is kernel pruning in the high-level layer. Because of the stronger information abstraction ability in the high-level layer, only the valid input channel information can be passed to the next layer, no longer forcing each group to retain the kernel, which can greatly improve the efficiency of network pruning. Model analysis and experiments show that the two-stage kernel pruning can not only obtain better network optimization direction under moderate pruning but also obtain better network performance under a higher pruning rate.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11

Similar content being viewed by others

References

  1. Simonyan K, Zisserman A (2014) Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556

  2. He K, Zhang X, Ren S, Sun J (2016) Deep residual learning for image recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 770–778

  3. Liu W, Anguelov D, Erhan D, Szegedy C, Reed S, Fu C-Y, Berg AC (2016) SSD: Single shot multibox detector. In: European conference on computer vision, pp 21–37. Springer

  4. Redmon J, Divvala S, Girshick R, Farhadi A (2016) You only look once: Unified, real-time object detection. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 779–788

  5. Long J, Shelhamer E, Darrell T (2015) Fully convolutional networks for semantic segmentation. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 3431–3440

  6. Li Y, Qi H, Dai J, Ji X, Wei Y (2017) Fully convolutional instance-aware semantic segmentation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp 2359–2367

  7. Yan S, Teng Y, Smith JS, Zhang B (2016) Driver behavior recognition based on deep convolutional neural networks. In: 2016 12th International Conference on Natural Computation, Fuzzy Systems and Knowledge Discovery (ICNC-FSKD), IEEE, pp 636–641

  8. Reiß S, Roitberg A, Haurilet M, Stiefelhagen R (2020) Deep classification-driven domain adaptation for cross-modal driver behavior recognition. In: 2020 IEEE Intelligent Vehicles Symposium (IV), IEEE, pp 1042–1047

  9. Takahashi N, Gygli M, Van Gool L (2017) Aenet: Learning deep audio features for video analysis. IEEE Transactions on Multimedia 20(3):513–524

    Article  Google Scholar 

  10. Chen J, Li K, Deng Q, Li K, Philip SY (2019) Distributed deep learning model for intelligent video surveillance systems with edge computing. IEEE Transactions on Industrial Informatics

  11. Mozaffari S, Al-Jarrah OY, Dianati M, Jennings P, Mouzakitis A (2019) Deep learning-based vehicle behaviour prediction for autonomous driving applications: A review. arXiv preprint arXiv:1912.11676

  12. Fayjie AR, Hossain S, Oualid D, Lee D-J (2018) Driverless car: Autonomous driving using deep reinforcement learning in urban environment. In: 2018 15th International Conference on Ubiquitous Robots (UR), IEEE, pp 896–901

  13. LeCun Y, Denker JS, Solla A (1990) Optimal brain damage. In: Advances in neural information processing systems, pp 598–605

  14. Hassibi B, Stork DG (1993) Second order derivatives for network pruning: Optimal brain surgeon. In: Advances in neural information processing systems, pp 164–171

  15. Han S, Pool J, Tran J, Dally W (2015) Learning both weights and connections for efficient neural network. In: Advances in neural information processing systems, pp 1135–1143

  16. Han S, Mao H, Dally WJ (2015) Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding. arXiv preprint arXiv:1510.00149

  17. Ullrich K, Meeds E, Welling M (2017) Soft weight-sharing for neural network compression. arXiv preprint arXiv:1702.04008

  18. Xiao X, Wang Z, Rajasekaran S (2019) Autoprune: Automatic network pruning by regularizing auxiliary parameters. In: Advances in Neural Information Processing Systems, pp 13681–13691

  19. Wang Y, Xu C, You S, Tao D, Xu C (2016) Cnnpack: Packing convolutional neural networks in the frequency domain. In: Advances in neural information processing systems, pp 253–261

  20. Liu Z, Xu J, Peng X, Xiong R (2018) Frequency-domain dynamic pruning for convolutional neural networks. In: Advances in Neural Information Processing Systems, pp 1043–1053

  21. Li H, Kadav A, Durdanovic I, Samet H, Graf HP (2016) Pruning filters for efficient convnets. arXiv preprint arXiv:1608.08710

  22. He Y, Kang G, Dong X, Fu Y, Yang Y (2018) Soft filter pruning for accelerating deep convolutional neural networks. arXiv preprint arXiv:1808.06866

  23. He Y, Liu P, Wang Z, Hu Z, Yang Y (2019) Filter pruning via geometric median for deep convolutional neural networks acceleration. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp 4340–4349

  24. Luo J-H, Wu J, Lin W (2017) ThiNet: A filter level pruning method for deep neural network compression. In: Proceedings of the IEEE international conference on computer vision, pp 5058–5066

  25. Yu R, Li A, Chen C-F, Lai J-H, Morariu VI, Han X, Gao M, Lin C-Y, Davis LS (2018) Nisp: Pruning networks using neuron importance score propagation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp 9194–9203

  26. Zhuang Z, Tan M, Zhuang B, Liu J, Guo Y, Wu Q, Huang J, Zhu J (2018) Discrimination-aware channel pruning for deep neural networks. In: Advances in Neural Information Processing Systems, pp 875–886

  27. He Y, Zhang X, Sun J (2017) Channel pruning for accelerating very deep neural networks. In: Proceedings of the IEEE International Conference on Computer Vision, pp 1389–1397

  28. Liu Z, Li J, Shen Z, Huang G, Yan S, Zhang C (2017) Learning efficient convolutional networks through network slimming. In: Proceedings of the IEEE International Conference on Computer Vision, pp 2736–2744

  29. Lin S, Ji R, Li Y, Wu Y, Huang F, Zhang B (2018) Accelerating convolutional networks via global & dynamic filter pruning.. In: IJCAI, pp 2425–2432

  30. Lin S, Ji R, Li Y, Deng C, Li X (2019) Toward compact convnets via structure-sparsity regularized filter pruning. IEEE transactions on neural networks and learning systems 31(2):574–588

    Article  MathSciNet  Google Scholar 

  31. Zhu X, Zhou W, Li H (2018) Improving deep neural network sparsity through decorrelation regularization.. In: IJCAI, pp 3264–3270

  32. Liu C, Wang Y, Han K, Xu C, Xu C (2019) Learning instance-wise sparsity for accelerating deep models. arXiv preprint arXiv:1907.11840

  33. Huang Z, Wang N (2018) Data-driven sparse structure selection for deep neural networks. In: Proceedings of the European conference on computer vision (ECCV), pp 304–320

  34. Louizos C, Welling M, Kingma DP (2017) Learning sparse neural networks through l_0 regularization. arXiv preprint arXiv:1712.01312

  35. Molchanov D, Ashukha A, Vetrov D (2017) Variational dropout sparsifies deep neural networks. In: Proceedings of the 34th International Conference on Machine Learning-Volume 70, JMLR. org, pp 2498–2507

  36. Wang Z, Lin S, Xie J, Lin Y (2019) Pruning blocks for CNN compression and acceleration via online ensemble distillation. IEEE Access 7:175703–175716

    Article  Google Scholar 

  37. Lin S, Ji R, Yan C, Zhang B, Cao L, Ye Q, Huang F, Doermann D (2019) Towards optimal structured CNN pruning via generative adversarial learning. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp 2790–2799

  38. Liu N, Ma X, Xu Z, Wang Y, Tang J, Ye J (2020) Autocompress: An automatic DNN structured pruning framework for ultra-high compression rates.. In: AAAI, pp 4876–4883

  39. He Y, Lin J, Liu Z, Wang H, Li L-J, Han S (2018) Amc: Automl for model compression and acceleration on mobile devices. In: Proceedings of the European Conference on Computer Vision (ECCV), pp 784–800

  40. Lin C, Zhong Z, Wei W, Yan J (2018) Synaptic strength for convolutional neural network. In: Advances in Neural Information Processing Systems, pp 10149–10158

  41. Li Y, Lin S, Zhang B, Liu J, Doermann D, Wu Y, Huang F, Ji R (2019) Exploiting kernel sparsity and entropy for interpretable CNN compression. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp 2800–2809

  42. Mao H, Han S, Pool J, Li W, Liu X, Wang Y, Dally WJ (2017) Exploring the regularity of sparse structure in convolutional neural networks. arXiv preprint arXiv:1705.08922

  43. Wen W, Wu C, Wang Y, Chen Y, Li H (2016) Learning structured sparsity in deep neural networks. In: Advances in neural information processing systems, pp 2074–2082

  44. Wang H, Zhang Q, Wang Y, Hu R (2018) Structured deep neural network pruning by varying regularization parameters. ArXiv preprint:1804.09461 3

  45. Aubry M, Russell BC (2015) Understanding deep features with computer-generated imagery. In: Proceedings of the IEEE International Conference on Computer Vision, pp 2875–2883

  46. Zhang Q, Nian Wu Y, Zhu S-C (2018) Interpretable convolutional neural networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp 8827–8836

  47. Wagner J, Kohler JM, Gindele T, Hetzel L, Wiedemer JT, Behnke S (2019) Interpretable and fine-grained visual explanations for convolutional neural networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp 9097–9107

  48. Sandler M, Howard A, Zhu M, Zhmoginov A, Chen L-C (2018) Mobilenetv2: Inverted residuals and linear bottlenecks. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 4510–4520

  49. Krizhevsky A (2009) Learning multiple layers of features from tiny images. Master’s thesis, University of Tront

  50. Deng J, Dong W, Socher R, Li L-J, Li K, Fei-Fei L (2009) Imagenet: A large-scale hierarchical image database. In: 2009 IEEE conference on computer vision and pattern recognition, Ieee, pp 248–255

  51. Adam P, Sam G, Soumith C, Gregory C, Edward Y, Zachary D, Zeming L, Alban D, Luca A, Adam L (2017) Automatic differentiation in pytorch. In: Proceedings of Neural Information Processing Systems

  52. Li Z, Gong Y, Ma X, Liu S, Sun M, Zhan Z, Kong Z, Yuan G, Wang Y (2020) SS-Auto: A single-shot, automatic structured weight pruning framework of DNNs with ultra-high efficiency. arXiv preprint arXiv:2001.08839

  53. Zuo Y, Chen B, Shi T, Sun M (2020) Filter pruning without damaging networks capacity. IEEE Access 8:90924–90930

    Article  Google Scholar 

  54. Ding G, Zhang S, Jia Z, Zhong J, Han J (2020) Where to prune: Using lstm to guide data-dependent soft pruning. IEEE Transactions on Image Processing 30:293–304

    Article  Google Scholar 

  55. Ding X, Ding G, Guo Y, Han J (2019) Centripetal sgd for pruning very deep convolutional networks with complicated structure. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp 4943–4953

Download references

Acknowledgements

This work was supported in part by the National Natural Science Foundation of China under Grant(62071303, 61871269), Guangdong Basic and Applied Basic Research Foundation(2019A1515011861), Shenzhen Science and Technology Projection (JCYJ20190808151615540).

Author information

Authors and Affiliations

  1. College of Electronics and Information Engineering, Shenzhen University, Shenzhen, 518060, China

    Jihong Zhu & Jihong Pei

Authors
  1. Jihong Zhu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Jihong Pei
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Jihong Pei.

Ethics declarations

Conflict of Interests

The authors declare that they have no conflict of interest.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Zhu, J., Pei, J. Progressive kernel pruning CNN compression method with an adjustable input channel. Appl Intell 52, 10519–10540 (2022). https://doi.org/10.1007/s10489-021-02932-z

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10489-021-02932-z

Keywords

Search

Navigation