Abstract
Deep learning is the most useful tool for may applications, such as image recognize, nature language processing. But huge computation power and millions of parameters are needed in large models which may can’t be supported and stored. For this problem, some works tried to compress the dense weight matrices with sparse representations technologies, such as matrix decomposition and tensor decomposition. But it is still unknown which is the largest compress ratio. Therefore, in this paper, we analyse the relationship between the shape of tensor and the number of parameters, formulate the problem of minimizing the number of parameters, and solve it to find the best compress ratio. We compare the compressed ration on three data sets.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Denil, B., Shakibi, L., Dinh, N., de Freitas et al., Predicting parameters in deep learning. In: Advances in Neural Information Processing Systems, pp. 2148–2156. IEEE Press, New York (2013)
Chien, J.T., Bao, Y.T.: Tensor-factorized neural networks. IEEE Trans. Neural Netw. Learn. Syst. 29(5), 1998–2011 (2018)
Tjandra, A., Sakti, S., Nakamura, S., Compressing recurrent neural network with tensor train. In: 2017 International Joint Conference on in Neural Networks (IJCNN), pp. 4451–4458. IEEE Press, New York (2017)
Lathauwer, L.D., Moor, B.D., Vandewalle, J.: A multilinear singular value decomposition. SIAM J. Matrix Anal. Appl 21(4), 1253–1278 (2000)
LeCun, Y., Bottou, L., Bengio, Y., Haffner, P.: Gradient-based learning applied to document recognition. Proc. IEEE 86(11), 2278–2324 (1998)
Netzer, Y., Wang, T., Coates, A., Bissacco, A., Wu, B., Ng, A.Y.: Reading digits in natural images with unsupervised feature learning. In: Proceedings of NIPS Workshop Deep Learning and Unsupervised Feature Learning, p. 5. IEEE Press. New York (2011)
Liao, C.P., Chien J.T.: Graphical modeling of conditional random fields for human motion recognition. In: Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 1969–1972. IEEE Press, New York (2008)
Acknowledgments
This work was supported by National Natural Science Foundation of China (Nos. 61802030, 61572184, 61502054), the Science and Technology Projects of Hunan Province (No. 2016JC2075), the Research Foundation of Education Bureau of Hunan Province, China (Nos. 16C0047, 16B085).
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2020 Springer Nature Singapore Pte Ltd.
About this paper
Cite this paper
He, S., Li, Z., Wang, J., Xie, K., Zhang, D. (2020). Compressing Deep Neural Network. In: Park, J., Park, DS., Jeong, YS., Pan, Y. (eds) Advances in Computer Science and Ubiquitous Computing. CUTE CSA 2018 2018. Lecture Notes in Electrical Engineering, vol 536. Springer, Singapore. https://doi.org/10.1007/978-981-13-9341-9_107
Download citation
DOI: https://doi.org/10.1007/978-981-13-9341-9_107
Published:
Publisher Name: Springer, Singapore
Print ISBN: 978-981-13-9340-2
Online ISBN: 978-981-13-9341-9
eBook Packages: EngineeringEngineering (R0)