Abstract
Deep neural networks have present state-of-the-art results in many different machine learning tasks. In the traditional machine learning task, we train models on the formerly prepared dataset. However, in the real-world scenarios, training data are always collected in an incremental manner, in which new samples and new classes will be added to the training data gradually. Since the traditional training method with stochastic gradient descent will suffer from catastrophic forgetting problem when training on the new data set, in this paper, we proposed a new scheme to train deep neural networks incrementally. We first train the deep model on the original dataset with a weight-pruning manner, then on the newly added training data, we train the former pruned weights while remaining the former trained core-part weights unchanged. Experiments on MNIST demonstrated that our method is efficient and can even get better performance than training from scratch on the whole dataset in the traditional manner.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
He, K., Zhang, X., Ren, S., et al.: Delving deep into rectifiers: surpassing human-level performance on imagenet classification 1026–1034 (2015)
Sermanet, P., Eigen, D., Zhang, X., et al.: OverFeat: integrated recognition, localization and detection using convolutional networks. Eprint Arxiv (2013)
Hinton, G., Deng, L., Yu, D., et al.: Deep neural networks for acoustic modeling in speech recognition: the shared views of four research groups. IEEE Signal Process. Mag. 29(6), 82–97 (2012)
Mikolov, T., Deoras, A., Povey, D., et al.: Strategies for training large scale neural network language models. In: Automatic Speech Recognition and Understanding. IEEE, pp. 196–201 (2011)
Hubel, D.H., Wiesel, T.N.: Receptive fields and functional architecture of monkey striate cortex. J. Physiol. 195(1), 215–243 (1968)
Lin, M., Chen, Q., Yan, S.: Network in network (2013). arXiv preprint arXiv:1312.4400
Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. Comput. Sci. (2014)
Szegedy, C., Liu, W., Jia, Y., et al.: Going deeper with convolutions. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1–9 (2015)
Moe-Helgesen, O.M., Stranden, H.: Catastophic forgetting in neural networks. Dept. Comput. Inf. Sci. Nor. Univ. Sci. Technol. (NTNU) Trondheim Nor. Tech. Rep. 1, 22 (2005)
Xiao, T., Zhang, J., Yang, K., et al.: Error-driven incremental learning in deep convolutional neural network for large-scale image classification. In: Proceedings of the 22nd ACM International Conference on Multimedia. ACM, pp. 177–186 (2014)
Han, S., Pool, J., Tran, J., et al.: Learning both weights and connections for efficient neural network. In: Advances in Neural Information Processing Systems, pp. 1135–1143 (2015)
Srinivas, S., Babu, R.V.: Data-free parameter pruning for deep neural networks (2015). arXiv preprint arXiv:1507.06149
Han, S., Mao, H., Dally, W.J.: Deep compression: compressing deep neural networks with pruning, trained quantization and Huffman coding (2015). arXiv preprint arXiv:1510.00149
Acknowledgements
This research work is funded by the National Key Research and Development Project of China (2016YFB0801003)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2020 Springer Nature Singapore Pte Ltd.
About this paper
Cite this paper
Guo, H., Yan, Z., Yang, J., Li, S. (2020). An Incremental Scheme with Weight Pruning to Train Deep Neural Network. In: Liang, Q., Liu, X., Na, Z., Wang, W., Mu, J., Zhang, B. (eds) Communications, Signal Processing, and Systems. CSPS 2018. Lecture Notes in Electrical Engineering, vol 517. Springer, Singapore. https://doi.org/10.1007/978-981-13-6508-9_37
Download citation
DOI: https://doi.org/10.1007/978-981-13-6508-9_37
Published:
Publisher Name: Springer, Singapore
Print ISBN: 978-981-13-6507-2
Online ISBN: 978-981-13-6508-9
eBook Packages: EngineeringEngineering (R0)