Skip to main content
SpringerLink
Log in

Reconstruction Error Aware Pruning for Accelerating Neural Networks

  • Conference paper
  • First Online:
Advances in Visual Computing (ISVC 2019)

Part of the book series: Lecture Notes in Computer Science ((LNIP,volume 11844))

Included in the following conference series:

Abstract

This paper presents a pruning method, Reconstruction Error Aware Pruning (REAP), to reduce the redundancy of convolutional neural network models for accelerating the inference. REAP is an extension of one of the state-of-the-art channel pruning methods. Our method takes 3 steps, (1) evaluating the importance of each channel based on the reconstruction error of the outputs in each convolutional layer, (2) pruning less important channels, (3) updating the remaining weights by the least squares method so as to reconstruct the outputs. By pruning with REAP, one can produce a fast and accurate model out of a large pretrained model. Besides, REAP saves us lots of time and efforts required for retraining the pruned model. As our method requires a large computational cost, we have developed an algorithm based on biorthogonal system to conduct the computation efficiently. In the experiments, we show that REAP can conduct pruning with smaller sacrifice of the model performances than several existing state-of-the-art methods such as CP [9], ThiNet [17], DCP [25], and so on.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 39.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Aghasi, A., Abdi, A., Nguyen, N., Romberg, J.: Net-Trim: convex pruning of deep neural networks with performance guarantee. In: Advances in Neural Information Processing Systems, vol. 30, pp. 3177–3186. Curran Associates Inc. (2017)

    Google Scholar 

  2. akamaster. Proper implementation of resnet-s for cifar10/100 in pytorch that matches description of the original paper (2019)

    Google Scholar 

  3. Courbariaux, M., Bengio, Y., David, J.-P.: BinaryConnect: training deep neural networks with binary weights during propagations. In: Advances in Neural Information Processing Systems, vol. 28, pp. 3123–3131. Curran Associates Inc. (2015)

    Google Scholar 

  4. Deng, J., Dong, W., Socher, R., Li, J., Li, K., Fei-Fei, L.: ImageNet: a large-scale hierarchical image database. In: CVPR (2009)

    Google Scholar 

  5. Dong, X., Chen, S., Pan, S.: Learning to prune deep neural networks via layer-wise optimal brain surgeon. In: Advances in Neural Information Processing Systems, vol. 30, pp. 4857–4867. Curran Associates Inc. (2017)

    Google Scholar 

  6. Han, S., Mao, H., Dally, W.J.: Deep compression: compressing deep neural networks with pruning, trained quantization and Huffman coding. In: Proceedings of International Conference on Learning Representations, pp. 1–14 (2016)

    Google Scholar 

  7. He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition, pp. 770–778 (2016)

    Google Scholar 

  8. He, T., Fan, Y., Qian, Y., Tan, T., Yu, K.: Reshaping deep neural network for fast decoding by node-pruning, pp. 245–249 (2014)

    Google Scholar 

  9. He, Y., Zhang, X., Sun, J.: Channel pruning for accelerating very deep neural networks. In: Proceedings of International Conference on Computer Vision (2017)

    Google Scholar 

  10. He, Y., Lin, J., Liu, Z., Wang, H., Li, L.-J., Han, S.: AMC: AutoML for model compression and acceleration on mobile devices. In: Ferrari, V., Hebert, M., Sminchisescu, C., Weiss, Y. (eds.) ECCV 2018. LNCS, vol. 11211, pp. 815–832. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-01234-2_48

    Chapter  Google Scholar 

  11. Huang, G., Liu, Z., van der Maaten, L., Weinberger, K.Q.: Densely connected convolutional networks, pp. 2261–2269 (2017)

    Google Scholar 

  12. Khosla, A., Jayadevaprakash, N., Yao, B., Fei-Fei, L.: Novel dataset for fine-grained image categorization. In: First Workshop on Fine-Grained Visual Categorization, IEEE Conference on Computer Vision and Pattern Recognition, Colorado Springs, CO, June 2011

    Google Scholar 

  13. Krizhevsky, A., Nair, V., Hinton, G.: Cifar-10 (Canadian institute for advanced research)

    Google Scholar 

  14. LeCun, Y., Denker, J.S., Solla, S.A.: Optimal brain damage. In: Advances in Neural Information Processing Systems, vol. 2, pp. 598–605. Morgan-Kaufmann (1990)

    Google Scholar 

  15. Liu, B., Wang, M., Foroosh, H., Tappen, M.F., Pensky, M.: Sparse convolutional neural networks, pp. 806–814 (2015)

    Google Scholar 

  16. Liu, Z., Li, J., Shen, Z., Huang, G., Yan, S., Zhang, C.: Learning efficient convolutional networks through network slimming. In: Proceedings of International Conference on Computer Vision (2017)

    Google Scholar 

  17. Luo, J.-H., Wu, J., Lin, W.: ThiNet: a filter level pruning method for deep neural network compression. In: Proceedings of International Conference on Computer Vision (2017)

    Google Scholar 

  18. Simonyan, K., Zisserman, A.: Very deep convoolutional networks for large-scale image recognition. In: Proceedings of International Conference on Learning Representations, pp. 1–14 (2015)

    Google Scholar 

  19. Wang, H., Zhang, Q., Wang, Y., Hu, H.: Structured probabilistic pruning for convolutional neural network acceleration. In: Proceedings of British Machine Vision Conference (2018)

    Google Scholar 

  20. Xie, G., Wang, J., Zhang, T., Lai, J., Hong, R., Qi, G.-J.: Interleaved structured sparse convolutional neural networks. In: Proceedings of Computer Vision and Pattern Recognition (2018)

    Google Scholar 

  21. Xue, J., Li, J., Gong, Y.: Restructuring of deep neural network acoustic models with singular value decomposition. In: INTERSPEECH (2013)

    Google Scholar 

  22. Ye, J., et al.: Learning compact recurrent neural networks with block-term tensor decomposition. In: Proceedings of Computer Vision and Pattern Recognition (2018)

    Google Scholar 

  23. Yu, X., Liu, T., Wang, X., Tao, D.: On compressing deep models by low rank and sparse decomposition. In: The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), July 2017

    Google Scholar 

  24. Zhou, A., Yao, A., Wang, K., Chen, Y.: Explicit loss-error-aware quantization for low-bit deep neural networks. In: Proceedings of Computer Vision and Pattern Recognition (2018)

    Google Scholar 

  25. Zhuang, Z., et al.: Discrimination-aware channel pruning for deep neural networks. In: Proceedings of Advances in Neural Information Processing Systems (2018)

    Google Scholar 

  26. Chetlur, S., Woolley, C., Vandermersch, P., Cohen, J., Tran, J., Catanzaro, B., Shelhamer, E.: cuDNN: efficient Primitives for Deep Learning. Technical report (2011)

    Google Scholar 

  27. Zhao, Q., et al.: M2Det: a single-shot object detector based on multi-level feature pyramid network. In: Proceedings of AAAI Conference on Artificial Intelligence (AAAI) (2019)

    Google Scholar 

Download references

Acknowledgment

This work was supported by JSPS KAKENHI Grant Number 19K12020 and the Environment Research and Technology Development Fund (3-1905) of the Environmental Restoration and Conservation Agency of Japan.

Author information

Authors and Affiliations

  1. Wakayama University, 930 Sakaedani, Wakayama-shi, Wakayama, 640-8510, Japan

    Koji Kamma & Toshikazu Wada

Authors
  1. Koji Kamma
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Toshikazu Wada
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Koji Kamma .

Editor information

Editors and Affiliations

  1. University of Nevada, Reno, NV, USA

    George Bebis

  2. NASA Ames Research Center, Moffett Field, CA, USA

    Richard Boyle

  3. University of Nevada, Reno, NV, USA

    Bahram Parvin

  4. Desert Research Institute, Reno, NV, USA

    Darko Koracin

  5. Lawrence Berkeley National Laboratory, Berkeley, CA, USA

    Daniela Ushizima

  6. Latent AI, Palo Alto, CA, USA

    Sek Chai

  7. Texas A&M University, College Station, TX, USA

    Shinjiro Sueda

  8. Louisiana State University, Baton Rouge, LA, USA

    Xin Lin

  9. University of North Carolina at Charlotte, Charlotte, NC, USA

    Aidong Lu

  10. École Polytechnique Fédérale de Lausanne, Lausanne, Switzerland

    Daniel Thalmann

  11. Notre Dame University, Notre Dame, IN, USA

    Chaoli Wang

  12. Bosch Research North America, Palo Alto, CA, USA

    Panpan Xu

Rights and permissions

Reprints and permissions

Copyright information

© 2019 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Kamma, K., Wada, T. (2019). Reconstruction Error Aware Pruning for Accelerating Neural Networks. In: Bebis, G., et al. Advances in Visual Computing. ISVC 2019. Lecture Notes in Computer Science(), vol 11844. Springer, Cham. https://doi.org/10.1007/978-3-030-33720-9_5

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-33720-9_5

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-33719-3

  • Online ISBN: 978-3-030-33720-9

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation