Advertisement

EagleEye: Fast Sub-net Evaluation for Efficient Neural Network Pruning

Conference paper
  • 892 Downloads
Part of the Lecture Notes in Computer Science book series (LNCS, volume 12347)

Abstract

Finding out the computational redundant part of a trained Deep Neural Network (DNN) is the key question that pruning algorithms target on. Many algorithms try to predict model performance of the pruned sub-nets by introducing various evaluation methods. But they are either inaccurate or very complicated for general application. In this work, we present a pruning method called EagleEye, in which a simple yet efficient evaluation component based on adaptive batch normalization is applied to unveil a strong correlation between different pruned DNN structures and their final settled accuracy. This strong correlation allows us to fast spot the pruned candidates with highest potential accuracy without actually fine-tuning them. This module is also general to plug-in and improve some existing pruning algorithms. EagleEye achieves better pruning performance than all of the studied pruning algorithms in our experiments. Concretely, to prune MobileNet V1 and ResNet-50, EagleEye outperforms all compared methods by up to 3.8%. Even in the more challenging experiments of pruning the compact model of MobileNet V1, EagleEye achieves the highest accuracy of 70.9% with an overall 50% operations (FLOPs) pruned. All accuracy results are Top-1 ImageNet classification accuracy. Source code and models are accessible to open-source community (https://github.com/anonymous47823493/EagleEye).

Keywords

Model compression Neural network pruning 

Notes

Acknowledgements

Jiang Su is the corresponding author of this work. This work was supported in part by the National Natural Science Foundation of China (NSFC) under Grant No.U1811463.

Supplementary material

504434_1_En_38_MOESM1_ESM.pdf (528 kb)
Supplementary material 1 (pdf 528 KB)

References

  1. 1.
    Ayinde, B.O., Zurada, J.M.: Building efficient convnets using redundant feature pruning. ArXiv abs/1802.07653 (2018)Google Scholar
  2. 2.
    Cohen, T.S., Geiger, M., Köhler, J., Welling, M.: Spherical CNNs. In: ICLR (2018)Google Scholar
  3. 3.
    Deng, J., Dong, W., Socher, R., Li, L.J., Li, K., Fei-Fei, L.: Imagenet: a large-scale hierarchical image database. In: 2009 IEEE Conference on Computer Vision and Pattern Recognition, pp. 248–255. IEEE (2009)Google Scholar
  4. 4.
    He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 770–778 (2016)Google Scholar
  5. 5.
    He, Y., Kang, G., Dong, X., Fu, Y., Yang, Y.: Soft filter pruning for accelerating deep convolutional neural networks. arXiv preprint arXiv:1808.06866 (2018)
  6. 6.
    He, Y., Liu, P., Wang, Z., Yang, Y.: Pruning filter via geometric median for deep convolutional neural networks acceleration. arXiv preprint arXiv:1811.00250 (2018)
  7. 7.
    He, Y., Lin, J., Liu, Z., Wang, H., Li, L.-J., Han, S.: AMC: AutoML for model compression and acceleration on mobile devices. In: Ferrari, V., Hebert, M., Sminchisescu, C., Weiss, Y. (eds.) ECCV 2018. LNCS, vol. 11211, pp. 815–832. Springer, Cham (2018).  https://doi.org/10.1007/978-3-030-01234-2_48CrossRefGoogle Scholar
  8. 8.
    He, Y., Zhang, X., Sun, J.: Channel pruning for accelerating very deep neural networks. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 1389–1397 (2017)Google Scholar
  9. 9.
    Howard, A.G., et al.: Mobilenets: efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv:1704.04861 (2017)
  10. 10.
    Ioffe, S., Szegedy, C.: Batch normalization: accelerating deep network training by reducing internal covariate shift. arXiv preprint arXiv:1502.03167 (2015)
  11. 11.
    Kendall, M.G.: A new measure of rank correlation. Biometrika 30, 81–93 (1938)Google Scholar
  12. 12.
    Krizhevsky, A.: Learning multiple layers of features from tiny images (2009)Google Scholar
  13. 13.
    Li, H., Kadav, A., Durdanovic, I., Samet, H., Graf, H.P.: Pruning filters for efficient convnets. arXiv preprint arXiv:1608.08710 (2016)
  14. 14.
    Li, Y., Wang, N., Shi, J., Liu, J., Hou, X.: Revisiting batch normalization for practical domain adaptation. arXiv preprint arXiv:1603.04779 (2016)
  15. 15.
    Lin, M., et al.: Hrank: filter pruning using high-rank feature map. ArXiv abs/2002.10179 (2020)Google Scholar
  16. 16.
    Lin, S., Ji, R., Li, Y., Wu, Y., Huang, F., Zhang, B.: Accelerating convolutional networks via global & dynamic filter pruning. In: IJCAI, pp. 2425–2432 (2018)Google Scholar
  17. 17.
    Lin, S., et al.: Towards optimal structured CNN pruning via generative adversarial learning. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2790–2799 (2019)Google Scholar
  18. 18.
    Lin, S., et al.: Towards optimal structured CNN pruning via generative adversarial learning. In: 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2785–2794 (2019)Google Scholar
  19. 19.
    Liu, N., Ma, X., Xu, Z., Wang, Y., Tang, J., Ye, J.: AutoCompress: an automatic DNN structured pruning framework for ultra-high compression rates (2020)Google Scholar
  20. 20.
    Liu, Z., et al.: Metapruning: meta learning for automatic neural network channel pruning. In: 2019 IEEE/CVF International Conference on Computer Vision (ICCV), pp. 3295–3304 (2019)Google Scholar
  21. 21.
    Liu, Z., Li, J., Shen, Z., Huang, G., Yan, S., Zhang, C.: Learning efficient convolutional networks through network slimming. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 2736–2744 (2017)Google Scholar
  22. 22.
    Luo, J.H., Wu, J., Lin, W.: Thinet: a filter level pruning method for deep neural network compression. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 5058–5066 (2017)Google Scholar
  23. 23.
    Molchanov, P., Mallya, A., Tyree, S., Frosio, I., Kautz, J.: Importance estimation for neural network pruning. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (2019)Google Scholar
  24. 24.
    Soper, H., Young, A., Cave, B., Lee, A., Pearson, K.: On the distribution of the correlation coefficient in small samples. Appendix II to the papers of “student” and RA fisher. Biometrika 11(4), 328–413 (1917)Google Scholar
  25. 25.
    Wen, W., Wu, C., Wang, Y., Chen, Y., Li, H.: Learning structured sparsity in deep neural networks. In: Advances in Neural Information Processing Systems, pp. 2074–2082 (2016)Google Scholar
  26. 26.
    Yang, T.-J., et al.: NetAdapt: platform-aware neural network adaptation for mobile applications. In: Ferrari, V., Hebert, M., Sminchisescu, C., Weiss, Y. (eds.) ECCV 2018. LNCS, vol. 11214, pp. 289–304. Springer, Cham (2018).  https://doi.org/10.1007/978-3-030-01249-6_18CrossRefGoogle Scholar
  27. 27.
    You, Z., Yan, K., Ye, J., Ma, M., Wang, P.: Gate decorator: global filter pruning method for accelerating deep convolutional neural networks. In: Advances in Neural Information Processing Systems (NeurIPS) (2019)Google Scholar
  28. 28.
    Yu, J., Huang, T.: Network slimming by slimmable networks: towards one-shot architecture search for channel numbers. arXiv preprint arXiv:1903.11728 (2019)
  29. 29.
    Yu, R., et al.: Nisp: pruning networks using neuron importance score propagation. In: 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9194–9203 (2017)Google Scholar
  30. 30.
    Zhuang, Z., et al.: Discrimination-aware channel pruning for deep neural networks. In: Advances in Neural Information Processing Systems, pp. 875–886 (2018)Google Scholar

Copyright information

© Springer Nature Switzerland AG 2020

Authors and Affiliations

  1. 1.Dark Matter AI Inc.GuangzhouChina
  2. 2.Sun Yat-sen UniversityGuangzhouChina

Personalised recommendations