Skip to main content
SpringerLink
Log in

Higher-order principal component analysis for the approximation of tensors in tree-based low-rank formats

  • Published:
Numerische Mathematik Aims and scope Submit manuscript

Abstract

This paper is concerned with the approximation of tensors using tree-based tensor formats, which are tensor networks whose graphs are dimension partition trees. We consider Hilbert tensor spaces of multivariate functions defined on a product set equipped with a probability measure. This includes the case of multidimensional arrays corresponding to finite product sets. We propose and analyse an algorithm for the construction of an approximation using only point evaluations of a multivariate function, or evaluations of some entries of a multidimensional array. The algorithm is a variant of higher-order singular value decomposition which constructs a hierarchy of subspaces associated with the different nodes of the tree and a corresponding hierarchy of interpolation operators. Optimal subspaces are estimated using empirical principal component analysis of interpolations of partial random evaluations of the function. The algorithm is able to provide an approximation in any tree-based format with either a prescribed rank or a prescribed relative error, with a number of evaluations of the order of the storage complexity of the approximation format. Under some assumptions on the estimation of principal components, we prove that the algorithm provides either a quasi-optimal approximation with a given rank, or an approximation satisfying the prescribed relative error, up to constants depending on the tree and the properties of interpolation operators. The analysis takes into account the discretization errors for the approximation of infinite-dimensional tensors. For a tensor with finite and known rank in a tree-based format, the algorithm is able to recover the tensor in a stable way using a number of evaluations equal to the storage complexity of the representation of the tensor in this format. Several numerical examples illustrate the main results and the behavior of the algorithm for the approximation of high-dimensional functions using hierarchical Tucker or tensor train tensor formats, and the approximation of univariate functions using tensorization.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6

Similar content being viewed by others

Notes

  1. Uniqueness comes from (1b) while existence comes from (1a) and (1b).

  2. Note that \(V'\subset \tilde{U}'\) and we may have \(W \not \subset V'.\)

  3. Note that since \({\mathrm {rank}}_{\{d\}}(v) = {\mathrm {rank}}_{\{1,\ldots ,d-1\}}(v)\), adding the node \(\{d\}\) in the set of active nodes A would yield an equivalent tensor format.

  4. For all \(m \ge r_\alpha \), we have \(\mathcal {P}_{U_\alpha ^\star } u_{m} = \sum _{k=1}^{m} \sigma _k^\alpha (P_{U_\alpha ^\star } u_k^\alpha ) \otimes u_k^{\alpha ^c} = \sum _{k=1}^{r_\alpha } \sigma _k^\alpha u_k^\alpha \otimes u_k^{\alpha ^c} = u_{r_\alpha }\). Then using the continuity of \(\mathcal {P}_{U_\alpha ^\star }\) and taking the limit with m, we obtain \(\mathcal {P}_{U_\alpha ^\star } u = u_{r_\alpha }\).

  5. Note that when \(\mathcal {H}\) is equipped with a norm stronger than the norm in \(L^2_{\mu }(\mathcal {X})\), then \(\Vert \cdot \Vert _\alpha \) does not coincides with the norm \(\Vert \cdot \Vert \) on \(\mathcal {H}\), so that the subspaces solutions of (7) and (11) will be different in general.

  6. For the last example, \(\mathcal {X}\) is a finite product set equipped with the uniform measure and \(L^2_\mu (\mathcal {X})\) then corresponds to the space of multidimensional arrays equipped with the canonical norm.

References

  1. Bachmayr, M., Schneider, R., Uschmajew, A.: Tensor networks and hierarchical tensors for the solution of high-dimensional partial differential equations. Found. Comput. Math. 16(6), 1423–1472 (2016)

    Article  MathSciNet  MATH  Google Scholar 

  2. Ballani, J., Grasedyck, L., Kluge, M.: Black box approximation of tensors in hierarchical Tucker format. Linear Algebra Appl. 438(2), 639–657 (2013). Tensors and Multilinear Algebra

    Article  MathSciNet  MATH  Google Scholar 

  3. Blanchard, G., Bousquet, O., Zwald, L.: Statistical properties of kernel principal component analysis. Mach. Learn. 66(2–3), 259–294 (2007)

    Article  MATH  Google Scholar 

  4. Bungartz, H.-J., Griebel, M.: Sparse grids. Acta Numer. 13, 147–269 (2004)

    Article  MathSciNet  MATH  Google Scholar 

  5. Chevreuil, M., Lebrun, R., Nouy, A., Rai, P.: A least-squares method for sparse low rank approximation of multivariate functions. SIAM/ASA J. Uncertain. Quantif. 3(1), 897–921 (2015)

    Article  MathSciNet  MATH  Google Scholar 

  6. Cohen, A., DeVore, R.: Approximation of high-dimensional parametric pdes. Acta Numer. 24, 1–159 (2015)

    Article  MathSciNet  MATH  Google Scholar 

  7. Cohen, N., Sharir, O., Shashua, A.: On the expressive power of deep learning: a tensor analysis. In: Conference on Learning Theory, pp. 698–728 (2016)

  8. De Lathauwer, L., De Moor, B., Vandewalle, J.: A multilinear singular value decomposition. SIAM J. Matrix Anal. Appl. 21(4), 1253–1278 (2000)

    Article  MathSciNet  MATH  Google Scholar 

  9. de Silva, V., Lim, L.-H.: Tensor rank and ill-posedness of the best low-rank approximation problem. SIAM J. Matrix Anal. Appl. 30(3), 1084–1127 (2008)

    Article  MathSciNet  MATH  Google Scholar 

  10. DeVore, R.A.: Nonlinear approximation. Acta Numer. 7, 51–150 (1998)

    Article  MATH  Google Scholar 

  11. Doostan, A., Validi, A., Iaccarino, G.: Non-intrusive low-rank separated approximation of high-dimensional stochastic models. Comput. Methods Appl. Mech. Eng. 263, 42–55 (2013)

    Article  MathSciNet  MATH  Google Scholar 

  12. Espig, M., Grasedyck, L., Hackbusch, W.: Black box low tensor-rank approximation using fiber-crosses. Constr. Approx. 30, 557–597 (2009)

    Article  MathSciNet  MATH  Google Scholar 

  13. Falcó, A., Hackbusch, W.: On minimal subspaces in tensor representations. Found. Comput. Math. 12, 765–803 (2012)

    Article  MathSciNet  MATH  Google Scholar 

  14. Falco, A., Hackbusch, W., Nouy, A.: Geometric Structures in Tensor Representations (Final Release). ArXiv e-prints (2015)

  15. Falcó, A., Hackbusch, W., Nouy, A.: On the Dirac–Frenkel variational principle on tensor Banach spaces. Found. Comput. Math. (2018). https://doi.org/10.1007/s10208-018-9381-4

  16. Falcó, A., Hackbusch, W., Nouy, A.: Tree-based tensor formats. SeMA J. (2018). https://doi.org/10.1007/s40324-018-0177-x

  17. Grasedyck, L.: Hierarchical singular value decomposition of tensors. SIAM J. Matrix Anal. Appl. 31, 2029–2054 (2010)

    Article  MathSciNet  MATH  Google Scholar 

  18. Grasedyck, L., Kressner, D., Tobler, C.: A literature survey of low-rank tensor approximation techniques. GAMM-Mitteilungen 36(1), 53–78 (2013)

    Article  MathSciNet  MATH  Google Scholar 

  19. Grelier, E., Nouy, A., Chevreuil, M.: Learning with tree-based tensor formats (2018). arXiv e-prints arXiv:1811.04455

  20. Hackbusch, W.: Tensor Spaces and Numerical Tensor Calculus, Volume 42 of Springer Series in Computational Mathematics. Springer, Heidelberg (2012)

    Book  Google Scholar 

  21. Hackbusch, W., Kuhn, S.: A new scheme for the tensor representation. J. Fourier Anal. Appl. 15(5), 706–722 (2009)

    Article  MathSciNet  MATH  Google Scholar 

  22. Hillar, C., Lim, L.-H.: Most tensor problems are np-hard. J. ACM (JACM) 60(6), 45 (2013)

    Article  MathSciNet  MATH  Google Scholar 

  23. Holtz, S., Rohwedder, T., Schneider, R.: On manifolds of tensors of fixed tt-rank. Numer. Math. 120(4), 701–731 (2012)

    Article  MathSciNet  MATH  Google Scholar 

  24. Jirak, M., Wahl, M.: A tight \(\sin \varTheta \) theorem for empirical covariance operators. ArXiv e-prints (2018)

  25. Jirak, M., Wahl, M.: Relative perturbation bounds with applications to empirical covariance operators. ArXiv e-prints (2018)

  26. Khoromskij, B.: O (dlog n)-quantics approximation of nd tensors in high-dimensional numerical modeling. Constr. Approx. 34(2), 257–280 (2011)

    Article  MathSciNet  MATH  Google Scholar 

  27. Khoromskij, B.: Tensors-structured numerical methods in scientific computing: survey on recent advances. Chemometr. Intell. Lab. Syst. 110(1), 1–19 (2012)

    Article  MathSciNet  Google Scholar 

  28. Kolda, T.G., Bader, B.W.: Tensor decompositions and applications. SIAM Rev. 51(3), 455–500 (2009)

    Article  MathSciNet  MATH  Google Scholar 

  29. Kressner, D., Steinlechner, M., Uschmajew, A.: Low-rank tensor methods with subspace correction for symmetric eigenvalue problems. SIAM J. Sci. Comput. 36(5), A2346–A2368 (2014)

    Article  MathSciNet  MATH  Google Scholar 

  30. Lubich, C., Rohwedder, T., Schneider, R., Vandereycken, B.: Dynamical approximation by hierarchical tucker and tensor-train tensors. SIAM J. Matrix Anal. Appl. 34(2), 470–494 (2013)

    Article  MathSciNet  MATH  Google Scholar 

  31. Luu, T.H., Maday, Y., Guillo, M., Guérin, P.: A new method for reconstruction of cross-sections using Tucker decomposition. J. Comput. Phys. 345, 189–206 (2017)

    Article  MathSciNet  MATH  Google Scholar 

  32. Maday, Y., Nguyen, N.C., Patera, A.T., Pau, G.S.H.: A general multipurpose interpolation procedure: the magic points. Commun. Pure Appl. Anal. 8(1), 383–404 (2009)

    Article  MathSciNet  MATH  Google Scholar 

  33. Megginson, R.E: An Introduction to Banach Space Theory, Vol. 183. Springer, Berlin (2012)

  34. Nouy, A.: Low-rank methods for high-dimensional approximation and model order reduction. In: Benner, P., Cohen, A., Ohlberger, M., Willcox, K. (eds.) Model Reduction and Approximation: Theory and Algorithms. SIAM, Philadelphia (2017)

    Google Scholar 

  35. Nouy, A.: Low-Rank Tensor Methods for Model Order Reduction, pp. 857–882. Springer, Cham (2017)

    Google Scholar 

  36. Orus, R.: A practical introduction to tensor networks: matrix product states and projected entangled pair states. Ann. Phys. 349, 117–158 (2014)

    Article  MathSciNet  MATH  Google Scholar 

  37. Oseledets, I.: Tensor-train decomposition. SIAM J. Sci. Comput. 33(5), 2295–2317 (2011)

    Article  MathSciNet  MATH  Google Scholar 

  38. Oseledets, I., Tyrtyshnikov, E.: Breaking the curse of dimensionality, or how to use svd in many dimensions. SIAM J. Sci. Comput. 31(5), 3744–3759 (2009)

    Article  MathSciNet  MATH  Google Scholar 

  39. Oseledets, I., Tyrtyshnikov, E.: TT-cross approximation for multidimensional arrays. Linear Algebra Appl. 432(1), 70–88 (2010)

    Article  MathSciNet  MATH  Google Scholar 

  40. Oseledets, I., Tyrtyshnikov, E.: Algebraic wavelet transform via quantics tensor train decomposition. SIAM J. Sci. Comput. 33(3), 1315–1328 (2011)

    Article  MathSciNet  MATH  Google Scholar 

  41. Reiß, M., Wahl, M.: Non-asymptotic upper bounds for the reconstruction error of PCA (2016). arXiv preprint arXiv:1609.03779

  42. Schneider, R., Uschmajew, A.: Approximation rates for the hierarchical tensor format in periodic Sobolev spaces. J. Complex. 30(2), 56–71 (2014). Dagstuhl 2012

    Article  MathSciNet  MATH  Google Scholar 

  43. Temlyakov, V.: Nonlinear methods of approximation. Found. Comput. Math. 3(1), 33–107 (2003)

    Article  MathSciNet  MATH  Google Scholar 

  44. Temlyakov, V.: Greedy Approximation. Cambridge Monographs on Applied and Computational Mathematics. Cambridge University Press, Cambridge (2011)

    Book  Google Scholar 

  45. Uschmajew, A., Vandereycken, B.: The geometry of algorithms using hierarchical tensors. Linear Algebra Appl. 439(1), 133–166 (2013)

    Article  MathSciNet  MATH  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Centrale Nantes, LMJL, UMR CNRS 6629, 1 rue de la Noë, 44321, Nantes, France

    Anthony Nouy

Authors
  1. Anthony Nouy
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Anthony Nouy.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Nouy, A. Higher-order principal component analysis for the approximation of tensors in tree-based low-rank formats. Numer. Math. 141, 743–789 (2019). https://doi.org/10.1007/s00211-018-1017-8

Download citation

  • Received:

  • Revised:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00211-018-1017-8

Mathematics Subject Classification

Search

Navigation