Skip to main content
SpringerLink
Log in

Joint filter and channel pruning of convolutional neural networks as a bi-level optimization problem

  • Regular Research Paper
  • Published:
Memetic Computing Aims and scope Submit manuscript

Abstract

Deep neural networks, specifically deep convolutional neural networks (DCNNs), have been highly successful in machine learning and computer vision, but a significant challenge when using these networks is choosing the right hyperparameters. As the number of layers in the network increases, the search space also becomes larger. To overcome this issue, researchers in deep learning have suggested using deep compression techniques to decrease memory usage and computational complexity. In this paper, we present a new approach for compressing deep CNNs by combining filter and channel pruning methods based on Evolutionary Algorithms (EA). This method involves eliminating filters and channels in order to decrease the number of parameters and computational complexity of the model. Additionally, we propose a bi-level optimization problem that interacts between the hyperparameters of the convolution layer. Bi-level optimization problems are known to be difficult as they involve two levels of optimization tasks, where only the optimal solutions to the lower-level problem are considered as feasible candidates for the upper-level problem. In this work, the upper-level problem is represented by a set of filters to be pruned in order to minimize the number of selected filters, while the lower-level problem is represented by a set of channels to be pruned in order to minimize the number of selected channels per filter. Our research has focused on developing a new method for solving bi-level problems, which we have named Bi-CNN-Pruning. To achieve this, we have adopted the Co-Evolutionary Migration-Based Algorithm (CEMBA) as our search engine. The Bi-CNN-Pruning method is then evaluated using image classification benchmarks on well-known datasets such as CIFAR-10 and CIFAR-100. The results of our evaluation demonstrate that our bi-level proposal outperforms state-of-the-art architectures, and we provide a detailed analysis of the results using commonly employed performance metrics.

Graphical abstract

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8

Similar content being viewed by others

Data availability

CIFAR-10, CIFAR-100 Any additional data could be requested from the corresponding author, Hassen Louati.

References

  1. Ankur S, Pekka M, Anton F, Kalyanmoy D (2015) Multi-objective stackelberg game between a regulating authority and a mining company: a case study in environmental economics. Evolut Comput 23(2):217–248

    Google Scholar 

  2. Anwar S, Hwang K, Sung W (2017) Structured pruning of deep convolutional neural networks. ACM J Emerg Technol Comput Syst 13(3):1–18

    Article  Google Scholar 

  3. Bai X, Wang X, Liu X, Liu Q, Song J, Sebe N, Kim B (2021) Explainable deep learning for efficient and robust pattern recognition: a survey of recent developments. Pattern Recognit 120(1):108102

    Article  Google Scholar 

  4. Bhattacharya S, Lane ND (2016) Sparsification and separation of deep learning layers for constrained resource inference on wearables. In: SenSys pp 176–189

  5. Chakraborty UK, Janikow CZ (2003) An analysis of gray versus binary encoding in genetic search. Inform Sci 156(3–4):253–269

    Article  MathSciNet  Google Scholar 

  6. Chauhan J, Rajasegaran J, Seneviratne S, Misra A, Seneviratne A, Lee Y (2018) Performance characterization of deep learning models for breathing-based authentication on resource-constrained devices. In IMWUT pp 1–24

  7. Chen S, Lin L, Zhang Z, Gen M (2019) Evolutionary netarchitecture search for deep neural networks pruning. In: ICCV pp 189–196

  8. Chollet F (2017) Xception: Deep learning with depthwise separable convolutions. In: IEEE conference on computer vision and pattern recognition pp 1251–1258

  9. Denton EL, Zaremba W, Bruna J, Y. L., Fergus R (2014) Exploiting linear structure within convolutional networks for efficient evaluation. In: NIPS pp 1269–1277

  10. Ding X, Ding G, Han J, Tang S (2018) Auto-balanced filter pruning for efficient convolutional neural networks. In: AAAI conference on artificial intelligence Vol. 32, No. 1

  11. Dwork C, Feldman V, Hardt M, Pitassi T, Reingold O, Roth A (2015) The reusable holdout: preserving validity in adaptive data analysis. Science 349(6248):636–638

    Article  ADS  MathSciNet  CAS  PubMed  Google Scholar 

  12. Francisco E, Fernandes J, Yen GG (2021) Pruning deep convolutional neural networks architectures with evolution strategy. Inform Sci 552:29–47

    Article  MathSciNet  Google Scholar 

  13. Han S, Liu X, Mao H, J. P., Pedram A, Horowitz MA, Dally WJ (2016) Eie: efficient inference engine on compressed deep neural network. In: Eie: efficient inference engine on compressed deep neural network, vol. 44, no. 3, pp 243–254

  14. Han S, Mao H, Dally WJ (2015) Deep compression: compressing deep neural networks with pruning, trained quantization and huffman coding. arXiv:1510.00149

  15. Hao L, Kadav A, Durdanovic I, Samet H, Graf HP (2016) Pruning filters for efficient convnets. arXiv:1608.08710

  16. He K, Zhang X, Ren S, Sun J (2016) Deep residual learning for image recognition. In: IEEE conference on computer vision and pattern recognition pp 770–778

  17. He X, Zhou Z, Thiele L (2018) Multi-task zipping via layer-wise neuron sharing. In: NIPS pp 6016–6026

  18. He Y, Zhang X, Sun J (2017) Channel pruning for accelerating very deep neural networks. In: ICCV pp 1389–1397

  19. Howard AG, Zhu M, Kalenichenko BC, D., Wang W, Andreetto TW, M., Adam H (2017) Mobilenets: efficient convolutional neural networks for mobile vision applications. arXiv:1704.04861

  20. Hu, Sun, Y, Li S, Wang J, Gu X (2018) A novel channel pruning method for deep neural network compression. arXiv:1805.11394

  21. Hu H, Peng R, Tai Y-W, Tang C-K (2016) Network trimming: a datadriven neuron pruning approach towards efficient deep architectures. arXiv:1607.03250, vol. 13, no 3, pp 1–18

  22. Huynh LN, Lee Y, Balan RK (2017) Deepmon: mobile GPU-based deep learning framework for continuous vision applications. In: MobiSys pp 82–95

  23. Jian-Hao L, Jianxin W (2020) Autopruner: an end-to-end trainable filter pruning method for efficient deep model inference. Pattern Recognit 107:107461

    Article  Google Scholar 

  24. Kaixuan Y, Feilong C, Yee L, Jiye L (2021) Deep neural network compression through interpretability-based filter pruning. Pattern Recognit 119:108056

    Article  Google Scholar 

  25. Kollár J (1989) Flops. Nagoya Math J 113(1):15–36

    Article  MathSciNet  Google Scholar 

  26. Krizhevsky A, Sutskever I, Hinton GE (2012) Imagenet classification with deep convolutional neural networks. In: Advances in neural information processing systems pp 1097–1105

  27. LeCun Y, Denker J, Solla S, Howard R, Jackel L (1989) Optimal brain damage. In: NIPS vol. 2, pp 598605

  28. Liu C, Liu Q (2018) Improvement of pruning method for convolution neural network compression. In: ICDLT pp 57–60

  29. Liu Z, Li J, Shen Z, Huang G, Yan S, Zhang C (2017) Learning efficient convolutional networks through network slimming. In: ICCV pp 2736–2744

  30. Louati A, Louati H, Li Z (2021) Deep learning and case-based reasoning for predictive and adaptive traffic emergency management. J Supercomput 77:4389–4418

    Article  Google Scholar 

  31. Louati A, Louati H, Nusir M, Hardjono B (2020) Multi-agent deep neural networks coupled with LQF-MWM algorithm for traffic control and emergency vehicles guidance. J Ambi Intell Humanized Comput 11:5611–5627

    Article  Google Scholar 

  32. Louati H, Bechikh S, Louati A, Aldaej A, Said LB (2021b) Evolutionary optimization of convolutional neural network architecture design for thoracic x-ray image classification. In: IEA/AIE Vol. 32, No. 1

  33. Louati H, Bechikh S, Louati A, Hung C-C, Said LB (2021) Deep convolutional neural network architecture design as a bi-level optimization problem. Neurocomputing 439:44–62

    Article  Google Scholar 

  34. Luo J, Wu J, Lin W (2017) Thinet: A filter level pruning method for deep neural network compression. In: ICCV pp 5058–5066

  35. Mart’ın A, Lara-Cabrera R, Fuentes-Hurtado F, Naranjo V, Camacho D (2018) Evodeep: a new evolutionary approach for automatic deep neural networks parametrisation. J Parallel Distribut Comput 117:180–191

  36. Peng H, Wu J, Chen S, Huang J (2019) Collaborative channel pruning for deep networks. In: ICML pp 5113-5122

  37. Poli R, Langdon WB (1998) A new schema theorem for genetic programming with one-point crossover and point mutation. Evolut Comput 6(3):231–252

    Article  CAS  Google Scholar 

  38. Qin Q, Ren J, Yu J, Wang H, Gao L, Zheng J, Feng Y, Fang J, Wang Z (2018) To compress, or not to compress: Characterizing deep learning model compression for embedded inference. In: ISPA/IUCC/BDCloud/SocialCom/SustainCom pp 729–736

  39. Rahul M, Gupta HP, Dutta T (2020) A survey on deep neural network compression: Challenges, overview, and solutions. arXiv:2010.03954

  40. Said R, Bechikh S, Louati A, Aldaej A, Said LB (2020) Solving combinatorial multi-objective bi-level optimization problems using multiple populations and migration schemes. EEE Access 8:141674–141695

    Google Scholar 

  41. Simonyan K, Zisserman A (2014) Very deep convolutional networks for large-scale image recognition. arXiv:1409.1556

  42. Singh P, Kadi VSR, Verma N, Namboodiri VP (2019) Stability based filter pruning for accelerating deep cnns. In: WACV pp 1166–1174

  43. Spears VM, Jong KAD (1991) On the virtues of parameterized uniform crossover. In Fourth international conference on genetic algorithms pp 230–236

  44. Sun Y, Xue B, Zhang M, Yen GG (2019) Evolving deep convolutional neural networks for image classification. IEEE Trans Evolut Comput. https://doi.org/10.1109/TEVC.2019.2916183

    Article  Google Scholar 

  45. Tan M, Chen B, Pang R, Vasudevan V, Sandler M, Howard A, Le QV (2019) Mnasnet: platform-aware neural architecture search for mobile. In: CVPR pp 2820–2828

  46. Wu H, Gu X (2015) Towards dropout training for convolutional neural networks. Neural Netw 71(1):1–10

    Article  MathSciNet  PubMed  Google Scholar 

  47. Yan M, Zhao M, Xu Z, Zhang Q, Wang G, Su Z (2019) Vargfacenet: An efficient variable group convolutional neural network for lightweight face recognition. In: ICCV pp 1–8

  48. Yao S, Zhao Y, Zhang A, Su L, Abdelzaher T (2017) Deepiot: compressing deep neural network structures for sensing systems with a compressor-critic framework. In: SenSys pp 1–14

  49. Zhou Y, Yen GG, Yi Z (2019) A knee-guided evolutionary algorithm for compressing deep neural networks. IEEE Trans Cybern 51(3):1–13

    CAS  Google Scholar 

Download references

Acknowledgements

This study is supported via funding from Prince Sattam bin Abdulaziz University project number (PSAU/2024/R/1445).

Funding

Funding from Prince Sattam bin Abdulaziz University project number (PSAU/2024/R/1445).

Author information

Authors and Affiliations

  1. College of Information Technology, Kingdom University, Riffa, 40434, Bahrain

    Hassen Louati

  2. SMART Lab, ISG, University of Tunis, Tunis, Tunisia

    Hassen Louati & Slim Bechikh

  3. Department of Information Systems, College of Computer Engineering and Sciences, Prince Sattam bin Abdulaziz University, Al-Kharj, 11942, Saudi Arabia

    Ali Louati & Elham Kariri

Authors
  1. Hassen Louati
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Ali Louati
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Slim Bechikh
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Elham Kariri
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

Conceptualization, methodology, and experimentation: Hassen Louati; - Writing–review and editing: Hassen Louati, Ali Louati, Slim Bechikh and Elham Kariri.

Corresponding author

Correspondence to Ali Louati.

Ethics declarations

Conflict of interest

The authors declare no conflict of interest.

Ethical approval

This material is the authors’ own original work, which has not been previously published elsewhere. The paper is not currently being considered for publication elsewhere. The paper reflects the authors’ own research and analysis in a truthful and complete manner.

Consent for publication

Not applicable.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Louati, H., Louati, A., Bechikh, S. et al. Joint filter and channel pruning of convolutional neural networks as a bi-level optimization problem. Memetic Comp. 16, 71–90 (2024). https://doi.org/10.1007/s12293-024-00406-6

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s12293-024-00406-6

Keywords

Search

Navigation