Compressing Deep Neural Network
Deep learning is the most useful tool for may applications, such as image recognize, nature language processing. But huge computation power and millions of parameters are needed in large models which may can’t be supported and stored. For this problem, some works tried to compress the dense weight matrices with sparse representations technologies, such as matrix decomposition and tensor decomposition. But it is still unknown which is the largest compress ratio. Therefore, in this paper, we analyse the relationship between the shape of tensor and the number of parameters, formulate the problem of minimizing the number of parameters, and solve it to find the best compress ratio. We compare the compressed ration on three data sets.
KeywordsDeep neural network Parameters compressing Matrix decomposition Tensor decomposition
This work was supported by National Natural Science Foundation of China (Nos. 61802030, 61572184, 61502054), the Science and Technology Projects of Hunan Province (No. 2016JC2075), the Research Foundation of Education Bureau of Hunan Province, China (Nos. 16C0047, 16B085).
- 1.Denil, B., Shakibi, L., Dinh, N., de Freitas et al., Predicting parameters in deep learning. In: Advances in Neural Information Processing Systems, pp. 2148–2156. IEEE Press, New York (2013)Google Scholar
- 3.Tjandra, A., Sakti, S., Nakamura, S., Compressing recurrent neural network with tensor train. In: 2017 International Joint Conference on in Neural Networks (IJCNN), pp. 4451–4458. IEEE Press, New York (2017)Google Scholar
- 6.Netzer, Y., Wang, T., Coates, A., Bissacco, A., Wu, B., Ng, A.Y.: Reading digits in natural images with unsupervised feature learning. In: Proceedings of NIPS Workshop Deep Learning and Unsupervised Feature Learning, p. 5. IEEE Press. New York (2011)Google Scholar
- 7.Liao, C.P., Chien J.T.: Graphical modeling of conditional random fields for human motion recognition. In: Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 1969–1972. IEEE Press, New York (2008)Google Scholar