Abstract
Convolutional neural networks have achieved great success in many fields. However, the practical application of convolutional neural networks is hindered due to the high consumption of memory and computational. In this paper, we propose a two-stage method for model compression and acceleration. More specifically, the training process mainly includes the search stage and train stage, the approach is abbreviated as ST. In the search stage, we first search and remove the unnecessary parts of a large pre-trained network (named supernet) by certain evaluation criteria to get a pruned network. Then the weights in the pruned network are initialized to get a small network (called a subnet). During the training stage, the supernet is untrainable, and the subnet will be trained under the supervision of the supernet. The knowledge extracted from the supernet will be transmitted to the subnet, then the subnet will be able to learn from the dataset and the knowledge at the same time. We have proved the effectiveness of our method through implement extensive experiments on several state-of-the-art CNN models (including VGGNet, ResNet, and DenseNet). The subnet only with 1/10 parameters and 1/2 calculations achieves more significant performance than the supernet.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Dong, X., Yang, Y.: Network pruning via transformable architecture search. In: Wallach, H.M., Larochelle, H., Beygelzimer, A., d’Alché-Buc, F., Fox, E.B., Garnett, R. (eds.) Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, 8–14 December 2019, Vancouver, BC, Canada, pp. 759–770 (2019)
Frankle, J., Carbin, M.: The lottery ticket hypothesis: Finding sparse, trainable neural networks. arXiv preprint arXiv:1803.03635 (2018)
He, K., Zhang, X., Ren, S., Jian, S.: Deep residual learning for image recognition. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2016)
Hinton, G., Vinyals, O., Dean, J.: Distilling the knowledge in a neural network. Comput. Sci. 14(7), 38–39 (2015)
Li, H., Kadav, A., Durdanovic, I., Samet, H., Graf, H.P.: Pruning filters for efficient convnets (2016)
Liu, Z., Li, J., Shen, Z., Huang, G., Yan, S., Zhang, C.: Learning efficient convolutional networks through network slimming. In: IEEE International Conference on Computer Vision, ICCV 2017, Venice, Italy, 22–29 October 2017, pp. 2755–2763. IEEE Computer Society (2017). https://doi.org/10.1109/ICCV.2017.298
Liu, Z., Li, J., Shen, Z., Huang, G., Yan, S., Zhang, C.: Learning efficient convolutional networks through network slimming. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 2736–2744 (2017)
Liu, Z., Sun, M., Zhou, T., Huang, G., Darrell, T.: Rethinking the value of network pruning. In: 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, 6–9 May 2019. OpenReview.net (2019). https://openreview.net/forum?id=rJlnB3C5Ym
Peng, B., et al.: Correlation congruence for knowledge distillation (2019)
Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. Computer Science (2014)
Acknowledgements
This research is supported by Sichuan Provincial Science and Technology Program (No.2019YFS0146).
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2020 Springer Nature Switzerland AG
About this paper
Cite this paper
Jiang, N., Tang, J., Zhang, Z., Yu, W., Deng, X., Zhou, J. (2020). Search-and-Train: Two-Stage Model Compression and Acceleration. In: Yang, H., Pasupa, K., Leung, A.CS., Kwok, J.T., Chan, J.H., King, I. (eds) Neural Information Processing. ICONIP 2020. Communications in Computer and Information Science, vol 1333. Springer, Cham. https://doi.org/10.1007/978-3-030-63823-8_73
Download citation
DOI: https://doi.org/10.1007/978-3-030-63823-8_73
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-63822-1
Online ISBN: 978-3-030-63823-8
eBook Packages: Computer ScienceComputer Science (R0)