Skip to main content
Log in

Practical Sketching Algorithms for Low-Rank Tucker Approximation of Large Tensors

  • Published:
Journal of Scientific Computing Aims and scope Submit manuscript

Abstract

Low-rank approximation of tensors has been widely used in high-dimensional data analysis. It usually involves singular value decomposition (SVD) of large-scale matrices with high computational complexity. Sketching is an effective data compression and dimensionality reduction technique applied to the low-rank approximation of large matrices. This paper presents two practical randomized algorithms for low-rank Tucker approximation of large tensors based on sketching and power scheme, with a rigorous error-bound analysis. Numerical experiments on synthetic and real-world tensor data demonstrate the competitive performance of the proposed algorithms.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8

Similar content being viewed by others

Data Availability

Enquiries about data availability should be directed to the authors.

Notes

  1. https://www.hdu.edu.cn/landscape.

  2. http://trace.eas.asu.edu/yuv/index.html.

References

  1. Comon, P.: Tensors: a brief introduction. IEEE Signal Process. Mag. 31(3), 44–53 (2014)

    Article  Google Scholar 

  2. Hitchcock, F.L.: Multiple invariants and generalized rank of a P-Way matrix or tensor. J. Math. Phys. 7(1–4), 39–79 (1928)

    Article  MATH  Google Scholar 

  3. Kiers, H.A.L.: Towards a standardized notation and terminology in multiway analysis. J. Chemom Soc. 14(3), 105–122 (2000)

    Article  Google Scholar 

  4. Tucker, L.R.: Implications of factor analysis of three-way matrices for measurement of change. Probl. Meas. Change 15, 122–137 (1963)

    Google Scholar 

  5. Tucker, L.R.: Some mathematical notes on three-mode factor analysis. Psychometrika 31(3), 279–311 (1966)

    Article  MathSciNet  Google Scholar 

  6. De Lathauwer, L., De Moor, B., Vandewalle, J.: A multilinear singular value decomposition. SIAM J. Matrix Anal. Appl. 21(4), 1253–1278 (2000)

    Article  MathSciNet  MATH  Google Scholar 

  7. Hackbusch, W., Kühn, S.: A new scheme for the tensor representation. J. Fourier Anal. Appl. 15(5), 706–722 (2009)

    Article  MathSciNet  MATH  Google Scholar 

  8. Grasedyck, L.: Hierarchical singular value decomposition of tensors. SIAM J. Matrix Anal. Appl. 31(4), 2029–2054 (2010)

    Article  MathSciNet  MATH  Google Scholar 

  9. Oseledets, I.V.: Tensor-train decomposition. SIAM J. Sci. Comput. 33(5), 2295–2317 (2011)

    Article  MathSciNet  MATH  Google Scholar 

  10. De Lathauwer, L., De Moor, B., Vandewalle, J.: On the best rank-1 and rank-(r1, r2,...,rn) approximation of higher-order tensors. SIAM J. Matrix Anal. Appl. 21(4), 1324–1342 (2000)

    Article  MathSciNet  MATH  Google Scholar 

  11. Vannieuwenhoven, N., Vandebril, R., Meerbergen, K.: A new truncation strategy for the higher-order singular value decomposition. SIAM J. Sci. Comput. 34(2), A1027–A1052 (2012)

    Article  MathSciNet  MATH  Google Scholar 

  12. Zhou, G., Cichocki, A., Xie, S.: Decomposition of big tensors with low multilinear rank. arXiv preprint, arXiv:1412.1885 (2014)

  13. Che, M., Wei, Y.: Randomized algorithms for the approximations of Tucker and the tensor train decompositions. Adv. Comput. Math. 45(1), 395–428 (2019)

    Article  MathSciNet  MATH  Google Scholar 

  14. Minster, R., Saibaba, A.K., Kilmer, M.E.: Randomized algorithms for low-rank tensor decompositions in the Tucker format. SIAM J. Math. Data Sci. 2(1), 189–215 (2020)

    Article  MathSciNet  MATH  Google Scholar 

  15. Che, M., Wei, Y., Yan, H.: The computation of low multilinear rank approximations of tensors via power scheme and random projection. SIAM J. Matrix Anal. Appl. 41(2), 605–636 (2020)

    Article  MathSciNet  MATH  Google Scholar 

  16. Che, M., Wei, Y., Yan, H.: Randomized algorithms for the low multilinear rank approximations of tensors. J. Comput. Appl. Math. 390(2), 113380 (2021)

    Article  MathSciNet  MATH  Google Scholar 

  17. Sun, Y., Guo, Y., Luo, C., Tropp, J., Udell, M.: Low-rank tucker approximation of a tensor from streaming data. SIAM J. Math. Data Sci. 2(4), 1123–1150 (2020)

    Article  MathSciNet  MATH  Google Scholar 

  18. Tropp, J.A., Yurtsever, A., Udell, M., Cevher, V.: Streaming low-rank matrix approximation with an application to scientific simulation. SIAM J. Sci. Comput. 41(4), A2430–A2463 (2019)

    Article  MathSciNet  MATH  Google Scholar 

  19. Malik, O.A., Becker, S.: Low-rank tucker decomposition of large tensors using Tensorsketch. Adv. Neural. Inf. Process. Syst. 31, 10116–10126 (2018)

    Google Scholar 

  20. Ahmadi-Asl, S., Abukhovich, S., Asante-Mensah, M.G., Cichocki, A., Phan, A.H., Tanaka, T.: Randomized algorithms for computation of Tucker decomposition and higher order SVD (HOSVD). IEEE Access. 9, 28684–28706 (2021)

    Article  Google Scholar 

  21. Halko, N., Martinsson, P.-G., Tropp, J.A.: Finding structure with randomness: probabilistic algorithms for constructing approximate matrix decompositions. SIAM Rev. 53(2), 217–288 (2011)

    Article  MathSciNet  MATH  Google Scholar 

  22. Tropp, J.A., Yurtsever, A., Udell, M., Cevher, V.: Practical sketching algorithms for low-rank matrix approximation. SIAM J. Matrix Anal. Appl. 38(4), 1454–1485 (2017)

    Article  MathSciNet  MATH  Google Scholar 

  23. Rokhlin, V., Szlam, A., Tygert, M.: A randomized algorithm for principal component analysis. SIAM J. Matrix Anal. Appl. 31(3), 1100–1124 (2009)

    Article  MathSciNet  MATH  Google Scholar 

  24. Xiao, C., Yang, C., Li, M.: Efficient alternating least squares algorithms for low multilinear rank approximation of tensors. J. Sci. Comput. 87(3), 1–25 (2021)

    Article  MathSciNet  MATH  Google Scholar 

  25. Zhang, J., Saibaba, A.K., Kilmer, M.E., Aeron, S.: A randomized tensor singular value decomposition based on the t-product. Numer. Linear Algebra Appl. 25(5), e2179 (2018)

    Article  MathSciNet  MATH  Google Scholar 

Download references

Acknowledgements

We would like to thank the anonymous referees for their comments and suggestions on our paper, which lead to great improvements of the presentation.

Funding

G. Yu’s work was supported in part by the National Natural Science Foundation of China (No. 12071104) and the Natural Science Foundation of Zhejiang Province (No. LD19A010002).

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Gaohang Yu.

Ethics declarations

Conflict of interest

The authors have not disclosed any competing interests.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Appendix

Appendix

Lemma 1

([25], Theorem 2) Let \(\varrho < k-1\) be a positive natural number and \({\varvec{\Omega }}\in \mathbb {R}^{k\times n}\) be a Gaussian random matrix. Suppose \(\textbf{Q}\) is obtained from Algorithm 7. Then \(\forall \textbf{A}\in \mathbb {R}^{m\times n}\), we have

$$\begin{aligned} \mathbb {E}_{{\varvec{\Omega }}}\Vert \textbf{A}-\textbf{QQ}^\top \textbf{A}\Vert _F^2\le (1+f(\varrho ,k)\varpi _k^{4q})\cdot \tau _{\varrho +1}^2(\textbf{A}) \end{aligned}$$
(23)

Lemma 2

([22], Lemma A.3) Let \(\textbf{A}\in \mathbb {R}^{m\times n}\) be an input matrix and \(\hat{\textbf{A}}=\textbf{QX}\) be the approximation obtained from Algorithm 7. The approximation error can be decomposed as

$$\begin{aligned} \Vert \textbf{A}-\hat{\textbf{A}}\Vert _F^2=\Vert \textbf{A}-\textbf{QQ}^\top \textbf{A}\Vert _F^2+\Vert \textbf{X}-\textbf{Q}^\top \textbf{A}\Vert _F^2 \end{aligned}$$
(24)

Lemma 3

([22], Lemma A.5) Assume \({\varvec{\Psi }}\in \mathbb {R}^{l\times n}\) is a standard normal matrix independent from \({\varvec{\Omega }}\). Then

$$\begin{aligned} \mathbb {E}_{\varvec{\Psi }}\Vert \textbf{X}-\textbf{Q}^\top \textbf{A}\Vert _F^2=f(k,l)\cdot \Vert \textbf{A}-\textbf{QQ}^\top \textbf{A}\Vert _F^2 \end{aligned}$$
(25)

The error-bound for Algorithm 7 can be shown in Lemma 4 below.

Lemma 4

Assume the sketch size parameter satisfies \(l>k+1\). Draw random test matrices \(\varvec{\Omega }\in \mathbb {R}^{n\times k}\) and \(\varvec{\Psi }{\in \mathbb {R}}^{l\times m}\) independently from the standard normal distribution. Then the rank-k approximation \(\hat{\textbf{A}}\) obtained from Algorithm 7 satisfies

$$\begin{aligned} \begin{aligned} \mathbb {E}\parallel \textbf{A}-\hat{\textbf{A}}\parallel _F^2&\le (1+f(k,l))\cdot \min _{\varrho <k-1}(1+f(\varrho ,k){\varpi _k}^{4q})\cdot \tau _{\varrho +1}^2(\textbf{A}) \end{aligned} \end{aligned}$$
(26)

Proof

Using Eqs. (23), (24) and (25), we have

$$\begin{aligned} \begin{aligned} \mathbb {E}\parallel \textbf{A}-\hat{\textbf{A}}\parallel _F^2&=\mathbb {E}_{\varvec{\Omega }}\Vert \textbf{A}-\textbf{QQ}^\top \textbf{A}\Vert _F^2+\mathbb {E}_{\varvec{\Omega }}\mathbb {E}_{\varvec{\Psi }}\Vert \textbf{X}-\textbf{Q}^\top \textbf{A}\Vert _F^2\\&=(1+f(k,l))\cdot \mathbb {E}_{\varvec{\Omega }}\Vert \textbf{A}-\textbf{QQ}^\top \textbf{A}\Vert _F^2\\&\le (1+f(k,l))\cdot (1+f(\varrho ,k){\varpi _k}^{4q})\cdot \tau _{\varrho +1}^2(\textbf{A}). \end{aligned} \end{aligned}$$

After minimizing over eligible index \(\varrho <k-1\), the proof is completed. \(\square \)

We are now in the position to prove Theorem 5. Combining Theorem 2 and Lemma 4, we have

$$\begin{aligned} \begin{aligned}&\mathbb {E}_{\{{\varvec{\Omega }} _j\}_{j = 1}^N}\Vert {\varvec{\mathcal {X}}} - \hat{{\varvec{\mathcal {X}}}}\Vert _F^2 \\&\quad = \sum \limits _{n = 1}^N\mathbb {E}_{\{{\varvec{\Omega }} _j\}_{j = 1}^N} \Vert \hat{{\varvec{\mathcal {X}}}}^{(n - 1)} - \hat{{\varvec{\mathcal {X}}}}^{(n )}\Vert _F^2\\&\quad = \sum \limits _{n = 1}^N \mathbb {E}_{\{{\varvec{\Omega }} _j\}_{j = 1}^{n-1}}\left\{ \mathbb {E}_{\varvec{\Omega }_n}\Vert \hat{{\varvec{\mathcal {X}}}}^{(n - 1)} - \hat{{\varvec{\mathcal {X}}}}^{(n)}\Vert _F^2 \right\} \\&\quad = \sum \limits _{n = 1}^N \mathbb {E}_{\{{\varvec{\Omega }} _j\}_{j = 1}^{n-1}}\left\{ \mathbb {E}_{\varvec{\Omega }_n}\Vert {\varvec{\mathcal {G}}}^{(n - 1)} \times _{i = 1}^{n - 1} {\textbf{U}^{(i)}}{ \times _n}(\textbf{I}_{I_n} - {\textbf{U}^{(n)}}{} \textbf{U}^{(n)\top }) \Vert _F^2 \right\} \\&\quad \le \sum \limits _{n = 1}^N \mathbb {E}_{\{{\varvec{\Omega }} _j\}_{j = 1}^{n-1}}\left\{ \mathbb {E}_{\varvec{\Omega }_n}\Vert (\textbf{I}_{I_n} - {\textbf{U}^{(n)}}{} \textbf{U}^{(n)\top })\textbf{G}_{(n)}^{(n-1)}) \Vert _F^2 \right\} \\&\quad \le \sum \limits _{n = 1}^N \mathbb {E}_{\{{\varvec{\Omega }} _j\}_{j = 1}^{n-1}}(1+f(r_n,l_n))\cdot \min _{\varrho _n<r_n-1}(1+f(\varrho _n,r_n){\varpi _r}^{4q})\sum \limits _{i = r_n+1}^{I_n}\sigma _{i}^{2}(\textbf{G}_{(n)}^{(n-1)})\\&\quad \le \sum \limits _{n = 1}^N \mathbb {E}_{\{{\varvec{\Omega }} _j\}_{j = 1}^{n-1}} (1+f(r_n,l_n))\cdot \min _{\varrho _n<r_n-1}(1+f(\varrho _n,r_n){\varpi _r}^{4q}) \Delta _n^2({\varvec{\mathcal {X}}})\\&\quad = \sum \limits _{n = 1}^N (1+f(r_n,l_n))\cdot \min _{\varrho _n<r_n-1}(1+f(\varrho _n,r_n){\varpi _r}^{4q})\Delta _n^2({\varvec{\mathcal {X}}})\\&\quad \le \sum \limits _{n = 1}^N (1+f(r_n,l_n))\cdot \min _{\varrho _n<r_n-1}(1+f(\varrho _n,r_n){\varpi _r}^{4q}) \Vert {\varvec{\mathcal {X}}}-\hat{{\varvec{\mathcal {X}}}}_\textrm{opt}\Vert _F^2 \, \end{aligned} \end{aligned}$$

which completes the proof of Theorem 5.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Dong, W., Yu, G., Qi, L. et al. Practical Sketching Algorithms for Low-Rank Tucker Approximation of Large Tensors. J Sci Comput 95, 52 (2023). https://doi.org/10.1007/s10915-023-02172-y

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s10915-023-02172-y

Keywords

Mathematics Subject Classification

Navigation