Abstract
With the help of available big data and powerful computing units, deep learning has achieved remarkable performance in a series of data processing applications, such as speech recognition, computer vision, and natural language processing. Most of the existing neural networks take a lot of matrix computation operators in their architecture, which requires that the multidimensional input or output must be transformed into matrix form and the multilinear structure information may inevitably be lost. In order to avoid the performance degeneration from the structure loss, tensor computation operators have been introduced to extend the matrix-based neural networks. In this chapter, we first give a brief introduction of classical deep neural networks. Tensor computation is used to extend deep neural networks mainly from three ways in existing works. The first one extends the matrix product in each neural layer to tensor counterparts, inspired by different tensor decompositions, including t-product connected deep neural network, mode-n product-based tensorized neural network, etc. In the second way, different tensor decompositions have been used to compress the whole network parameters by low-rank tensor approximation. The third way builds up the connection between tensor decompositions and neural networks, which helps to theoretically analyze deep learning. To demonstrate the effectiveness of tensor analysis for deep neural networks, experimental results on low-rank tensor approximation for network compression show that the computational complexity and storage requirement can be largely reduced, while the image classification accuracy can be maintained.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Calvi, G.G., Moniri, A., Mahfouz, M., Zhao, Q., Mandic, D.P.: Compression and interpretability of deep neural networks via Tucker tensor layer: from first principles to tensor valued back-propagation (2019, eprints). arXiv–1903
Cao, X., Rabusseau, G.: Tensor regression networks with various low-rank tensor approximations (2017, preprint). arXiv:1712.09520
Chien, J.T., Bao, Y.T.: Tensor-factorized neural networks. IEEE Trans. Neural Netw. Learn. Syst. 29, 1998–2011 (2017). https://doi.org/10.1109/TNNLS.2017.2690379
Cohen, N., Sharir, O., Levine, Y., Tamari, R., Yakira, D., Shashua, A.: Analysis and design of convolutional networks via hierarchical tensor decompositions (2017, eprints). arXiv–1705
Garipov, T., Podoprikhin, D., Novikov, A., Vetrov, D.: Ultimate tensorization: compressing convolutional and FC layers alike (2016, eprints). arXiv–1611
Goldberg, Y.: A primer on neural network models for natural language processing. J. Artif. Intell. Res. 57, 345–420 (2016)
Guo, J., Li, Y., Lin, W., Chen, Y., Li, J.: Network decoupling: from regular to depthwise separable convolutions (2018, eprints). arXiv–1808
Hawkins, C., Zhang, Z.: End-to-end variational bayesian training of tensorized neural networks with automatic rank determination (2020, eprints). arXiv–2010
Hochreiter, S., Schmidhuber, J.: Long short-term memory. Neural Comput. 9(8), 1735–1780 (1997)
Jaderberg, M., Vedaldi, A., Zisserman, A.: Speeding up convolutional neural networks with low rank expansions. In: Proceedings of the British Machine Vision Conference. BMVA Press, Saint-Ouen-l’Aumône (2014)
Kim, Y.: Convolutional neural networks for sentence classification. In: Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 1746–1751. Association for Computational Linguistics, Doha, Qatar (2014). https://doi.org/10.3115/v1/D14-1181. https://www.aclweb.org/anthology/D14-1181
Kossaifi, J., Khanna, A., Lipton, Z., Furlanello, T., Anandkumar, A.: Tensor contraction layers for parsimonious deep nets. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pp. 26–32 (2017)
Kossaifi, J., Bulat, A., Tzimiropoulos, G., Pantic, M.: T-net: Parametrizing fully convolutional nets with a single high-order tensor. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 7822–7831 (2019)
Lebedev, V., Ganin, Y., Rakhuba, M., Oseledets, I.V., Lempitsky, V.S.: Speeding-up convolutional neural networks using fine-tuned CP-Decomposition. In: International Conference on Learning Representations ICLR (Poster) (2015)
Lecun, Y., Bottou, L., Bengio, Y., Haffner, P.: Gradient-based learning applied to document recognition. Proc. IEEE 86, 2278–2324 (1998). https://doi.org/10.1109/5.726791
Long, J., Shelhamer, E., Darrell, T.: Fully convolutional networks for semantic segmentation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3431–3440 (2015)
Makantasis, K., Voulodimos, A., Doulamis, A., Bakalos, N., Doulamis, N.: Space-time domain tensor neural networks: an application on human pose recognition (2020, e-prints). arXiv–2004
Mhaskar, H., Liao, Q., Poggio, T.: Learning functions: when is deep better than shallow. preprint arXiv:1603.00988
Newman, E., Horesh, L., Avron, H., Kilmer, M.: Stable tensor neural networks for rapid deep learning (2018, e-prints). arXiv–1811
Novikov, A., Podoprikhin, D., Osokin, A., Vetrov, D.P.: Tensorizing neural networks. In: Cortes, C., Lawrence, N., Lee, D., Sugiyama, M., Garnett, R. (eds.) Advances in Neural Information Processing Systems, vol. 28. Curran Associates, Red Hook (2015). https://proceedings.neurips.cc/paper/2015/file/6855456e2fe46a9d49d3d3af4f57443d-Paper.pdf
Qi, J., Hu, H., Wang, Y., Yang, C.H.H., Siniscalchi, S.M., Lee, C.H.: Tensor-to-vector regression for multi-channel speech enhancement based on tensor-train network. In: IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 7504–7508. IEEE, Piscataway (2020)
Selvan, R., Dam, E.B.: Tensor networks for medical image classification. In: Medical Imaging with Deep Learning, pp. 721–732. PMLR, Westminster (2020)
Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. In: International Conference on Learning Representations (2015)
Tachioka, Y., Ishii, J.: Long short-term memory recurrent-neural-network-based bandwidth extension for automatic speech recognition. Acoust. Sci. Technol. 37(6), 319–321 (2016)
Tai, K.S., Socher, R., Manning, C.D.: Improved semantic representations from tree-structured long short-term memory networks. In: Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pp. 1556–1566 (2015)
Tran, D., Bourdev, L., Fergus, R., Torresani, L., Paluri, M.: Learning spatiotemporal features with 3d convolutional networks. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 4489–4497 (2015)
Wang, W., Sun, Y., Eriksson, B., Wang, W., Aggarwal, V.: Wide compression: tensor ring nets. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 9329–9338 (2018)
Xu, Y., Li, Y., Zhang, S., Wen, W., Wang, B., Dai, W., Qi, Y., Chen, Y., Lin, W., Xiong, H.: Trained rank pruning for efficient deep neural networks (2019, e-prints). arXiv–1910
Yang, Y., Krompass, D., Tresp, V.: Tensor-train recurrent neural networks for video classification. In: International Conference on Machine Learning, pp. 3891–3900. PMLR, Westminster (2017)
Ye, J., Li, G., Chen, D., Yang, H., Zhe, S., Xu, Z.: Block-term tensor neural networks. Neural Netw. 130, 11–21 (2020)
Yin, M., Liao, S., Liu, X.Y., Wang, X., Yuan, B.: Compressing recurrent neural networks using hierarchical tucker tensor decomposition (2020, e-prints). arXiv–2005
Yu, D., Deng, L., Seide, F.: The deep tensor neural network with applications to large vocabulary speech recognition. IEEE Trans. Audio Speech Lang. Process. 21(2), 388–396 (2012)
Zhang, X., Zou, J., He, K., Sun, J.: Accelerating ery deep convolutional networks for classification and detection. IEEE Trans. Pattern Anal. Mach. Intell. 38(10), 1943–1955 (2015)
Author information
Authors and Affiliations
Rights and permissions
Copyright information
© 2022 Springer Nature Switzerland AG
About this chapter
Cite this chapter
Liu, Y., Liu, J., Long, Z., Zhu, C. (2022). Tensor Decomposition in Deep Networks. In: Tensor Computation for Data Analysis. Springer, Cham. https://doi.org/10.1007/978-3-030-74386-4_10
Download citation
DOI: https://doi.org/10.1007/978-3-030-74386-4_10
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-74385-7
Online ISBN: 978-3-030-74386-4
eBook Packages: EngineeringEngineering (R0)