Skip to main content
SpringerLink
Log in

Bayesian Optimization with Clustering and Rollback for CNN Auto Pruning

  • Conference paper
  • First Online:
Computer Vision – ECCV 2022 (ECCV 2022)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 13683))

Included in the following conference series:

Abstract

Pruning is an effective technique for convolutional neural networks (CNNs) model compression, but it is difficult to find the optimal pruning policy due to the large design space. To improve the usability of pruning, many auto pruning methods have been developed. Recently, Bayesian optimization (BO) has been considered to be a competitive algorithm for auto pruning due to its solid theoretical foundation and high sampling efficiency. However, BO suffers from the curse of dimensionality. The performance of BO deteriorates when pruning deep CNNs, since the dimension of the design spaces increase. We propose a novel clustering algorithm that reduces the dimension of the design space to speed up the searching process. Subsequently, a rollback algorithm is proposed to recover the high-dimensional design space so that higher pruning accuracy can be obtained. We validate our proposed method on ResNet, MobileNetV1, and MobileNetV2 models. Experiments show that the proposed method significantly improves the convergence rate of BO when pruning deep CNNs with no increase in running time. The source code is available at https://github.com/fanhanwei/BOCR.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 89.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 119.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

References

  1. Brochu, E., Cora, V.M., De Freitas, N.: A tutorial on Bayesian optimization of expensive cost functions, with application to active user modeling and hierarchical reinforcement learning. arXiv preprint arXiv:1012.2599 (2010)

  2. Bull, A.D.: Convergence rates of efficient global optimization algorithms. J. Mach. Learn. Res. 12(10), 2879–2904 (2011)

    MathSciNet  MATH  Google Scholar 

  3. Chen, C., Tung, F., Vedula, N., Mori, G.: Constraint-aware deep neural network compression. In: Ferrari, V., Hebert, M., Sminchisescu, C., Weiss, Y. (eds.) ECCV 2018. LNCS, vol. 11212, pp. 409–424. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-01237-3_25

    Chapter  Google Scholar 

  4. Guo, Y., Yao, A., Chen, Y.: Dynamic network surgery for efficient DNNs. Advances in Neural Information Processing Systems, vol. 29 (2016)

    Google Scholar 

  5. Han, S., et al.: Learning both weights and connections for efficient neural network. In: Advances in Neural Information Processing Systems (2015)

    Google Scholar 

  6. Hayou, S., Ton, J.F., Doucet, A., Teh, Y.W.: Robust pruning at initialization. arXiv preprint arXiv:2002.08797 (2020)

  7. He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 770–778 (2016)

    Google Scholar 

  8. He, Y., et al.: Channel pruning for accelerating very deep neural networks. In: Proceedings of the IEEE ICCV (2017)

    Google Scholar 

  9. He, Y., Lin, J., Liu, Z., Wang, H., Li, L.-J., Han, S.: AMC: autoML for model compression and acceleration on mobile devices. In: Ferrari, V., Hebert, M., Sminchisescu, C., Weiss, Y. (eds.) ECCV 2018. LNCS, vol. 11211, pp. 815–832. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-01234-2_48

    Chapter  Google Scholar 

  10. Howard, A.G., et al.: MobileNets: efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv:1704.04861 (2017)

  11. Hu, H., Peng, R., Tai, Y.W., Tang, C.K.: Network trimming: a data-driven neuron pruning approach towards efficient deep architectures. arXiv preprint arXiv:1607.03250 (2016)

  12. Joe, S., Kuo, F.Y.: Remark on algorithm 659: implementing Sobol’s quasirandom sequence generator. ACM Trans. Math. Soft. (TOMS) 29(1), 49–57 (2003)

    Article  MathSciNet  MATH  Google Scholar 

  13. Joe, S., Kuo, F.Y.: Constructing Sobol sequences with better two-dimensional projections. SIAM J. Sci. Comput. 30(5), 2635–2654 (2008)

    Article  MathSciNet  MATH  Google Scholar 

  14. Kandasamy, K., Schneider, J., Póczos, B.: High dimensional bayesian optimisation and bandits via additive models. In: International Conference on Machine Learning, pp. 295–304. PMLR (2015)

    Google Scholar 

  15. Krizhevsky, A., Sutskever, I., Hinton, G.E.: Imagenet classification with deep convolutional neural networks. Adv. Neural. Inf. Process. Syst. 25, 1097–1105 (2012)

    Google Scholar 

  16. Letham, B., Calandra, R., Rai, A., Bakshy, E.: Re-examining linear embeddings for high-dimensional bayesian optimization. Adv. Neural. Inf. Process. Syst. 33, 1546–1558 (2020)

    Google Scholar 

  17. Lillicrap, T.P., et al.: Continuous control with deep reinforcement learning. arXiv preprint arXiv:1509.02971 (2015)

  18. Liu, N., Ma, X., Xu, Z., Wang, Y., Tang, J., Ye, J.: Autocompress: an automatic DNN structured pruning framework for ultra-high compression rates. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 34, pp. 4876–4883 (2020)

    Google Scholar 

  19. Liu, Z., Mu, H., Zhang, X., Guo, Z., Yang, X., Cheng, K.T., Sun, J.: Metapruning: meta learning for automatic neural network channel pruning. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 3296–3305 (2019)

    Google Scholar 

  20. Louizos, C., et al.: Learning sparse neural networks through \( l_0 \) regularization. arXiv preprint arXiv:1712.01312 (2017)

  21. Ma, X., Blaschko, M.B.: Additive tree-structured conditional parameter spaces in Bayesian optimization: a novel covariance function and a fast implementation. IEEE Trans. Pattern Anal. Mach. Intell. 43, 3024–3036 (2020)

    Article  Google Scholar 

  22. Ma, X., Triki, A.R., Berman, M., Sagonas, C., Cali, J., Blaschko, M.B.: A bayesian optimization framework for neural network compression. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 10274–10283 (2019)

    Google Scholar 

  23. Mockus, J., Mockus, L.: Bayesian approach to global optimization and application to multiobjective and constrained problems. J. Optim. Theory Appl. 70(1), 157–172 (1991)

    Article  MathSciNet  MATH  Google Scholar 

  24. Mockus, J.: Application of bayesian approach to numerical methods of global and stochastic optimization. J. Global Optim. 4(4), 347–365 (1994)

    Article  MathSciNet  MATH  Google Scholar 

  25. Mockus, J., Tiesis, V., Zilinskas, A.: The application of Bayesian methods for seeking the extremum. Towards Global Opt. 2(117–129), 2 (1978)

    MATH  Google Scholar 

  26. Molchanov, P., Tyree, S., Karras, T., Aila, T., Kautz, J.: Pruning convolutional neural networks for resource efficient inference. arXiv preprint arXiv:1611.06440 (2016)

  27. Müllner, D.: Modern hierarchical, agglomerative clustering algorithms. arXiv preprint arXiv:1109.2378 (2011)

  28. Qian, H., Hu, Y.Q., Yu, Y.: Derivative-free optimization of high-dimensional non-convex functions by sequential random embeddings. In: IJCAI, pp. 1946–1952 (2016)

    Google Scholar 

  29. Rana, S., Li, C., Gupta, S., Nguyen, V., Venkatesh, S.: High dimensional Bayesian optimization with elastic gaussian process. In: International Conference on Machine Learning, pp. 2883–2891. PMLR (2017)

    Google Scholar 

  30. Rasmussen, C.E.: Gaussian processes in machine learning. In: Bousquet, O., von Luxburg, U., Rätsch, G. (eds.) ML -2003. LNCS (LNAI), vol. 3176, pp. 63–71. Springer, Heidelberg (2004). https://doi.org/10.1007/978-3-540-28650-9_4

    Chapter  Google Scholar 

  31. Rolland, P., Scarlett, J., Bogunovic, I., Cevher, V.: High-dimensional bayesian optimization via additive models with overlapping groups. In: International Conference on Artificial Intelligence and Statistics, pp. 298–307. PMLR (2018)

    Google Scholar 

  32. Sandler, M., Howard, A., Zhu, M., Zhmoginov, A., Chen, L.C.: Mobilenetv 2: inverted residuals and linear bottlenecks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4510–4520 (2018)

    Google Scholar 

  33. SheffieldML: Gpyopt: a Bayesian optimization framework in python. GuitHub (2016). https://github.com/SheffieldML/GPyOpt

  34. Srinivas, S., Babu, R.V.: Data-free parameter pruning for deep neural networks. arXiv preprint arXiv:1507.06149 (2015)

  35. Tung, F., Muralidharan, S., Mori, G.: Fine-pruning: joint fine-tuning and compression of a convolutional network with bayesian optimization. arXiv preprint arXiv:1707.09102 (2017)

  36. Wang, K., et al.: HAQ: hardware-aware automated quantization with mixed precision. In: Proceedings of the IEEE Conference on CVPR, pp. 8612–8620 (2019)

    Google Scholar 

  37. Wang, Z., Li, C., Wang, X.: Convolutional neural network pruning with structural redundancy reduction. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 14913–14922 (2021)

    Google Scholar 

  38. Wang, Z., Li, C., Jegelka, S., Kohli, P.: Batched high-dimensional bayesian optimization via structural kernel learning. In: International Conference on Machine Learning, pp. 3656–3664. PMLR (2017)

    Google Scholar 

  39. Wang, Z., Hutter, F., Zoghi, M., Matheson, D., de Feitas, N.: Bayesian optimization in a billion dimensions via random embeddings. J. Artif. Intell. Res. 55, 361–387 (2016)

    Article  MathSciNet  MATH  Google Scholar 

  40. Wang, Z., Zoghi, M., Hutter, F., Matheson, D., De Freitas, N., et al.: Bayesian optimization in high dimensions via random embeddings. In: IJCAI, pp. 1778–1784 (2013)

    Google Scholar 

  41. Ward, J.H., Jr.: Hierarchical grouping to optimize an objective function. J. Am. Stat. Assoc. 58(301), 236–244 (1963)

    Article  MathSciNet  Google Scholar 

  42. Wen, W., Wu, C., Wang, Y., Chen, Y., Li, H.: Learning structured sparsity in deep neural networks. Advances in Neural Information Processing Systems, vol. 29 (2016)

    Google Scholar 

  43. Xu, K., et al.: TACC: a full-stack cloud computing infrastructure for machine learning tasks (2021)

    Google Scholar 

  44. Yu, R., et al.: NISP: pruning networks using neuron importance score propagation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 9194–9203 (2018)

    Google Scholar 

  45. Yu, S., Mazaheri, A., Jannesari, A.: Auto graph encoder-decoder for neural network pruning. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 6362–6372 (2021)

    Google Scholar 

  46. Zhang, Y., Gao, S., Huang, H.: Exploration and estimation for model compression. In: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 487–496 (2021)

    Google Scholar 

  47. Zhuang, Z., et al.: Discrimination-aware channel pruning for deep neural networks. In: Advances in Neural Information Processing Systems (2018)

    Google Scholar 

Download references

Acknowledgment

We would like to thank the anonymous reviewers for their valuable comments. We also thank the Turing AI Computing Cloud (TACC) [43] and HKUST iSING Lab for providing us computation resources on their platform. This research was supported in part by Hong Kong Research Grants Council General Research Fund (Grant No. 16215319).

Author information

Authors and Affiliations

  1. The Hong Kong University of Science and Technology (Guangzhou), Guangzhou, China

    Hanwei Fan

  2. The Hong Kong University of Science and Technology, Hong Kong, China

    Hanwei Fan, Jiandong Mu & Wei Zhang

Authors
  1. Hanwei Fan
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Jiandong Mu
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Wei Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Wei Zhang .

Editor information

Editors and Affiliations

  1. Tel Aviv University, Tel Aviv, Israel

    Shai Avidan

  2. University College London, London, UK

    Gabriel Brostow

  3. Google AI, Accra, Ghana

    Moustapha Cissé

  4. University of Catania, Catania, Italy

    Giovanni Maria Farinella

  5. Facebook (United States), Menlo Park, CA, USA

    Tal Hassner

1 Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary material 1 (pdf 140 KB)

Rights and permissions

Reprints and permissions

Copyright information

© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Fan, H., Mu, J., Zhang, W. (2022). Bayesian Optimization with Clustering and Rollback for CNN Auto Pruning. In: Avidan, S., Brostow, G., Cissé, M., Farinella, G.M., Hassner, T. (eds) Computer Vision – ECCV 2022. ECCV 2022. Lecture Notes in Computer Science, vol 13683. Springer, Cham. https://doi.org/10.1007/978-3-031-20050-2_29

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-20050-2_29

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-20049-6

  • Online ISBN: 978-3-031-20050-2

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation