Advertisement

Compression of Deep Convolutional Neural Networks Using Effective Channel Pruning

  • Qingbei Guo
  • Xiao-Jun WuEmail author
  • Xiuyang Zhao
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11901)

Abstract

Pruning is a promising technology for convolutional neural networks (CNNs) to address the problems of high computational complexity and high memory requirement. However, there is a principal challenge in channel pruning. Although the least important feature-map is removed each time based on one pruning criterion, these pruning may produce considerable fluctuation in classification performance, which easily results in failing to restore its capacity. We propose an effective channel pruning criterion to reduce redundant parameters, while significantly reducing such fluctuations. This criterion adopts the loss-approximating Taylor expansion based on not the pruned parameters but the parameters in the subsequent convolutional layer, which differentiates our method from existing methods, to evaluate the importance of each channel. To improve the learning effectivity and efficiency, the importance of these channels is ranked using a small proportion of training dataset. Furthermore, after each least important channel is pruned, a small fraction of training dataset is used to fine-tune the pruned network to partially recover its accuracy. Periodically, more proportion of training dataset is used for the intensive recovery in accuracy. The proposed criterion significantly addresses the aforementioned problems and shows outstanding performance compared to other criteria, such as Random, APoZ and Taylor pruning criteria. The experimental results demonstrate the excellent compactness performances of our approach, using several public image classification datasets, on some popular deep network architectures. Our code is available at: https://github.com/QingbeiGuo/Based-Taylor-Pruning.git.

Keywords

Deep neural network Classification Pruning 

Notes

Acknowledgment

This work is supported by the National Key R&D Program of China (Grant No. 2018YFB1004901), by the National Natural Science Foundation of China (Grant No.61672265, U1836218), by the 111 Project of Ministry of Education of China (Grant No. B12018), by UK EPSRC GRANT EP/N007743/1, MURI/EPSRC/DSTL GRANT EP/R018456/1.

References

  1. 1.
    Girshick, R., Donahue, J., Darrell, T., Malik, J.: Rich feature hierarchies for accurate object detection and semantic segmentation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Columbus, OH, USA, pp. 580–587 (2014)Google Scholar
  2. 2.
    Girshick, R., Donahue, J., Darrell, T., Malik, J.: Region-based convolutional networks for accurate object detection and segmentation. IEEE Trans. Pattern Anal. Mach. Intell. 38(1), 142–158 (2016)CrossRefGoogle Scholar
  3. 3.
    Glorot, X., Bordes, A., Bengio, Y.: Deep sparse rectifier neural networks. In: 14th International Conference on Artificial Intelligence and Statistics, Fort Lauderdale, United States, vol. 15, pp. 315–323 (2011)Google Scholar
  4. 4.
    Han, S., Mao, H., Dally, W.J.: Deep compression: compressing deep neural network with pruning, trained quantization and Huffman coding. CoRR abs/1510.00149 (2015). http://arxiv.org/abs/1510.00149
  5. 5.
    Han, S., Pool, J., Tran, J., Dally, W.J.: Learning both weights and connections for efficient neural networks. CoRR abs/1506.02626 (2015). http://arxiv.org/abs/1506.02626
  6. 6.
    He, Y., Zhang, X., Sun, J.: Channel pruning for accelerating very deep neural networks. CoRR abs/1707.06168 (2017). http://arxiv.org/abs/1707.06168
  7. 7.
    Hu, H., Peng, R., Tai, Y., Tang, C.: Network trimming: a data-driven neuron pruning approach towards efficient deep architectures. CoRR abs/1607.03250 (2016). http://arxiv.org/abs/1607.03250
  8. 8.
    Krizhevsky, A., Hinton, G.: Learning multiple layers of features from tiny images. Technical report, Science Department, University of Toronto, Tech (2009)Google Scholar
  9. 9.
    Krizhevsky, A., Sutskever, I., Hinton, G.E.: ImageNet classification with deep convolutional neural networks. In: Proceedings of the 25th International Conference on Neural Information Processing Systems, NIPS 2012, USA, vol. 1, pp. 1097–1105 (2012)Google Scholar
  10. 10.
    LeCun, Y., Bottou, L., Bengio, Y.: Gradient-based learning applied to document recognition. Proc. IEEE 86(11), 2278–2324 (1998)CrossRefGoogle Scholar
  11. 11.
    LeCun, Y., et al.: Handwritten digit recognition with a back-propagation network. In: Advances in Neural Information Processing Systems, vol. 2, pp. 396–404 (1990)Google Scholar
  12. 12.
    LeCun, Y., Denker, J.S., Solla, S.A.: Optimal brain damage. In: Proceedings of the 2nd International Conference on Neural Information Processing Systems, NIPS 1989, Cambridge, MA, USA, pp. 598–605 (1989)Google Scholar
  13. 13.
    Li, H., Kadav, A., Durdanovic, I., Samet, H., Graf, H.P.: Pruning filters for efficient convnets. CoRR abs/1608.08710 (2016). http://arxiv.org/abs/1608.08710
  14. 14.
    Mao, H., et al.: Exploring the regularity of sparse structure in convolutional neural networks. CoRR abs/1705.08922 (2017). http://arxiv.org/abs/1705.08922
  15. 15.
    Mishkin, D., Matas, J.: All you need is a good init. CoRR abs/1511.06422 (2015). http://arxiv.org/abs/1511.06422
  16. 16.
    Molchanov, P., Tyree, S., Karras, T., Aila, T., Kautz, J.: Pruning convolutional neural networks for resource efficient transfer learning. CoRR abs/1611.06440 (2016). http://arxiv.org/abs/1611.06440
  17. 17.
    Ren, S., He, K., Girshick, R., Sun, J.: Faster R-CNN: towards real-time object detection with region proposal networks. In: Proceedings of the 28th International Conference on Neural Information Processing Systems, NIPS 2015, Cambridge, MA, USA, vol. 1, pp. 91–99 (2015)Google Scholar
  18. 18.
    Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. CoRR abs/1409.1556 (2014). http://arxiv.org/abs/1409.1556
  19. 19.
    Wen, W., Wu, C., Wang, Y., Chen, Y., Li, H.: Learning structured sparsity in deep neural networks. CoRR abs/1608.03665 (2016). http://arxiv.org/abs/1608.03665
  20. 20.
    Wu, C., Wen, W., Afzal, T., Zhang, Y., Chen, Y., Li, H.: A compact DNN: approaching GoogLeNet-level accuracy of classification and domain adaptation. CoRR abs/1703.04071 (2017). http://arxiv.org/abs/1703.04071
  21. 21.
    Xu, T., Feng, Z.H., Wu, X.J., Kittler, J.: Learning adaptive discriminative correlation filters via temporal consistency preserving spatial feature selection for robust visual tracking. arXiv preprint arXiv:1807.11348 (2018)
  22. 22.
    Xu, T., Wu, X.J., Kittler, J.: Non-negative subspace representation learning scheme for correlation filter based tracking. In: 2018 24th International Conference on Pattern Recognition (ICPR), pp. 1888–1893. IEEE (2018)Google Scholar
  23. 23.
    Zhang, X., Zou, J., He, K., Sun, J.: Accelerating very deep convolutional networks for classification and detection. IEEE Trans. Pattern Anal. Mach. Intell. 38(10), 1943–1955 (2016)CrossRefGoogle Scholar

Copyright information

© Springer Nature Switzerland AG 2019

Authors and Affiliations

  1. 1.Jiangsu Provincial Engineering Laboratory of Pattern Recognition and Computational IntelligenceJiangnan UniversityWuxiChina
  2. 2.Shandong Provincial Key Laboratory of Network Based Intelligent ComputingUniversity of JinanJinanChina

Personalised recommendations