Skip to main content
SpringerLink
Log in

DyPrune: Dynamic Pruning Rates for Neural Networks

  • Conference paper
  • First Online:
Progress in Artificial Intelligence (EPIA 2023)

Part of the book series: Lecture Notes in Computer Science ((LNAI,volume 14115))

Included in the following conference series:

  • 375 Accesses

Abstract

Neural networks have achieved remarkable success in various applications such as image classification, speech recognition, and natural language processing. However, the growing size of neural networks poses significant challenges in terms of memory usage, computational cost, and deployment on resource-constrained devices. Pruning is a popular technique to reduce the complexity of neural networks by removing unnecessary connections, neurons, or filters. In this paper, we present novel pruning algorithms that can reduce the number of parameters in neural networks by up to 98% without sacrificing accuracy. This is done by scaling the pruning rate of the models to the size of the model and scheduling the pruning to execute throughout the training of the model. Code related to this work is openly available.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 59.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 79.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    https://github.com/richardjonker2000/DyPrune.

  2. 2.

    Due to the fact that the server is a shared resource for the research group, only one GPU core was used.

References

  1. Blalock, D., Gonzalez Ortiz, J.J., Frankle, J., Guttag, J.: What is the state of neural network pruning? Proc. Mach. Learn. Syst. 2, 129–146 (2020)

    Google Scholar 

  2. Finnoff, W., Hergert, F., Zimmermann, H.G.: Improving model selection by nonconvergent methods. Neural Netw. 6(6), 771–783 (1993). https://doi.org/10.1016/S0893-6080(05)80122-4

  3. Gale, T., Elsen, E., Hooker, S.: The State of Sparsity in Deep Neural Networks (2019). https://doi.org/10.48550/arXiv.1902.09574, arXiv:1902.09574 [cs, stat]

  4. Hagiwara, M.: Removal of hidden units and weights for back propagation networks. In: Proceedings of 1993 International Conference on Neural Networks (IJCNN-93-Nagoya, Japan). vol. 1, pp. 351–354 vol 1 (1993). https://doi.org/10.1109/IJCNN.1993.713929

  5. Han, S., Pool, J., Tran, J., Dally, W.: Learning both Weights and Connections for Efficient Neural Network. In: Advances in Neural Information Processing Systems. vol. 28. Curran Associates, Inc. (2015)

    Google Scholar 

  6. Hoefler, T., Alistarh, D., Ben-Nun, T., Dryden, N., Peste, A.: Sparsity in deep learning: pruning and growth for efficient inference and training in neural networks. J. Mach. Learn. Res. 22(1), 241:10882–241:11005 (2021)

    Google Scholar 

  7. Janowsky, S.A.: Pruning versus clipping in neural networks. Phys. Rev. A 39(12), 6600–6603 (1989). https://doi.org/10.1103/PhysRevA.39.6600,

    Article  Google Scholar 

  8. Kruschke, J.K., Movellan, J.R.: Benefits of gain: speeded learning and minimal hidden layers in back-propagation networks. IEEE Trans. Syst. Man Cybern. 21(1), 273–280 (1991)

    Article  MathSciNet  Google Scholar 

  9. Lecun, Y., Bottou, L., Bengio, Y., Haffner, P.: Gradient-based learning applied to document recognition. Proc. IEEE 86(11), 2278–2324 (1998). https://doi.org/10.1109/5.726791

    Article  Google Scholar 

  10. LeCun, Y., Denker, J., Solla, S.: Optimal brain damage. Advances in Neural Information Processing Systems, vol. 2 (1989)

    Google Scholar 

  11. Li, H., Kadav, A., Durdanovic, I., Samet, H., Graf, H.P.: Pruning filters for efficient convnets (2016). arXiv:1608.08710

  12. Molchanov, P., Tyree, S., Karras, T., Aila, T., Kautz, J.: Pruning convolutional neural networks for resource efficient inference (2016). arXiv:1611.06440

  13. Narasimha, P.L., Delashmit, W.H., Manry, M.T., Li, J., Maldonado, F.: An integrated growing-pruning method for feedforward network training. Neurocomputing 71(13), 2831–2847 (2008). https://doi.org/10.1016/j.neucom.2007.08.026,

    Article  Google Scholar 

  14. Thimm, G., Fiesler, E.: Evaluating pruning methods. In: Proceedings of the International Symposium on Artificial Neural Networks, pp. 20–25 (1995)

    Google Scholar 

  15. Xiao, H., Rasul, K., Vollgraf, R.: Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms (2017). https://doi.org/10.48550/arXiv.1708.07747, arXiv:1708.07747 [cs, stat]

  16. Zhang, Q., Zhang, R., Sun, J., Liu, Y.: How Sparse Can We Prune A Deep Network: A Geometric Viewpoint (2023). https://doi.org/10.48550/arXiv.2306.05857, arXiv:2306.05857 [cs, stat]

  17. Zhu, M., Gupta, S.: To prune, or not to prune: exploring the efficacy of pruning for model compression (2017). https://doi.org/10.48550/arXiv.1710.01878,, arXiv:1710.01878 [cs, stat]

Download references

Acknowledgments

This work was supported by national funds through the Foundation for Science and Technology (FCT) in the context of the project DSAIPA/AI/0088/2020 and project UIDB/00127/2020.

Author information

Authors and Affiliations

  1. IEETA- Institute of Electronics and Informatics Engineering of Aveiro, University of Aveiro, Aveiro, Portugal

    Richard Adolph Aires Jonker, Olga Fajarda, José Luís Oliveira & Sérgio Matos

  2. Research Centre in Digitalization and Intelligent Robotics, Polytechnic Institute of Bragança, Bragança, Portugal

    Roshan Poudel & Rui Pedro Lopes

  3. LASI - Intelligent Systems Associate Laboratory, Guimaråes, Portugal

    Olga Fajarda, José Luís Oliveira, Rui Pedro Lopes & Sérgio Matos

Authors
  1. Richard Adolph Aires Jonker
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Roshan Poudel
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Olga Fajarda
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. José Luís Oliveira
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Rui Pedro Lopes
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Sérgio Matos
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Richard Adolph Aires Jonker .

Editor information

Editors and Affiliations

  1. Lucy Family Institute for Data & Society, Notre Dame, IN, USA

    Nuno Moniz

  2. GECAD, Polytechnic of Porto, Porto, Portugal

    Zita Vale

  3. GRIA - LIACC, University of Azores, Ponta-Delgada, Portugal

    José Cascalho

  4. CISUC, University of Coimbra, Coimbra, Portugal

    Catarina Silva

  5. IEETA, University of Aveiro, Aveiro, Portugal

    Raquel Sebastião

Rights and permissions

Reprints and permissions

Copyright information

© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Jonker, R.A.A., Poudel, R., Fajarda, O., Oliveira, J.L., Lopes, R.P., Matos, S. (2023). DyPrune: Dynamic Pruning Rates for Neural Networks. In: Moniz, N., Vale, Z., Cascalho, J., Silva, C., Sebastião, R. (eds) Progress in Artificial Intelligence. EPIA 2023. Lecture Notes in Computer Science(), vol 14115. Springer, Cham. https://doi.org/10.1007/978-3-031-49008-8_12

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-49008-8_12

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-49007-1

  • Online ISBN: 978-3-031-49008-8

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation