Numerische Mathematik

, Volume 141, Issue 3, pp 743–789 | Cite as

Higher-order principal component analysis for the approximation of tensors in tree-based low-rank formats

  • Anthony NouyEmail author


This paper is concerned with the approximation of tensors using tree-based tensor formats, which are tensor networks whose graphs are dimension partition trees. We consider Hilbert tensor spaces of multivariate functions defined on a product set equipped with a probability measure. This includes the case of multidimensional arrays corresponding to finite product sets. We propose and analyse an algorithm for the construction of an approximation using only point evaluations of a multivariate function, or evaluations of some entries of a multidimensional array. The algorithm is a variant of higher-order singular value decomposition which constructs a hierarchy of subspaces associated with the different nodes of the tree and a corresponding hierarchy of interpolation operators. Optimal subspaces are estimated using empirical principal component analysis of interpolations of partial random evaluations of the function. The algorithm is able to provide an approximation in any tree-based format with either a prescribed rank or a prescribed relative error, with a number of evaluations of the order of the storage complexity of the approximation format. Under some assumptions on the estimation of principal components, we prove that the algorithm provides either a quasi-optimal approximation with a given rank, or an approximation satisfying the prescribed relative error, up to constants depending on the tree and the properties of interpolation operators. The analysis takes into account the discretization errors for the approximation of infinite-dimensional tensors. For a tensor with finite and known rank in a tree-based format, the algorithm is able to recover the tensor in a stable way using a number of evaluations equal to the storage complexity of the representation of the tensor in this format. Several numerical examples illustrate the main results and the behavior of the algorithm for the approximation of high-dimensional functions using hierarchical Tucker or tensor train tensor formats, and the approximation of univariate functions using tensorization.

Mathematics Subject Classification

15A69 41A05 41A63 65D15 65J99 



  1. 1.
    Bachmayr, M., Schneider, R., Uschmajew, A.: Tensor networks and hierarchical tensors for the solution of high-dimensional partial differential equations. Found. Comput. Math. 16(6), 1423–1472 (2016)MathSciNetzbMATHCrossRefGoogle Scholar
  2. 2.
    Ballani, J., Grasedyck, L., Kluge, M.: Black box approximation of tensors in hierarchical Tucker format. Linear Algebra Appl. 438(2), 639–657 (2013). Tensors and Multilinear AlgebraMathSciNetzbMATHCrossRefGoogle Scholar
  3. 3.
    Blanchard, G., Bousquet, O., Zwald, L.: Statistical properties of kernel principal component analysis. Mach. Learn. 66(2–3), 259–294 (2007)zbMATHCrossRefGoogle Scholar
  4. 4.
    Bungartz, H.-J., Griebel, M.: Sparse grids. Acta Numer. 13, 147–269 (2004)MathSciNetzbMATHCrossRefGoogle Scholar
  5. 5.
    Chevreuil, M., Lebrun, R., Nouy, A., Rai, P.: A least-squares method for sparse low rank approximation of multivariate functions. SIAM/ASA J. Uncertain. Quantif. 3(1), 897–921 (2015)MathSciNetzbMATHCrossRefGoogle Scholar
  6. 6.
    Cohen, A., DeVore, R.: Approximation of high-dimensional parametric pdes. Acta Numer. 24, 1–159 (2015)MathSciNetzbMATHCrossRefGoogle Scholar
  7. 7.
    Cohen, N., Sharir, O., Shashua, A.: On the expressive power of deep learning: a tensor analysis. In: Conference on Learning Theory, pp. 698–728 (2016)Google Scholar
  8. 8.
    De Lathauwer, L., De Moor, B., Vandewalle, J.: A multilinear singular value decomposition. SIAM J. Matrix Anal. Appl. 21(4), 1253–1278 (2000)MathSciNetzbMATHCrossRefGoogle Scholar
  9. 9.
    de Silva, V., Lim, L.-H.: Tensor rank and ill-posedness of the best low-rank approximation problem. SIAM J. Matrix Anal. Appl. 30(3), 1084–1127 (2008)MathSciNetzbMATHCrossRefGoogle Scholar
  10. 10.
    DeVore, R.A.: Nonlinear approximation. Acta Numer. 7, 51–150 (1998)zbMATHCrossRefGoogle Scholar
  11. 11.
    Doostan, A., Validi, A., Iaccarino, G.: Non-intrusive low-rank separated approximation of high-dimensional stochastic models. Comput. Methods Appl. Mech. Eng. 263, 42–55 (2013)MathSciNetzbMATHCrossRefGoogle Scholar
  12. 12.
    Espig, M., Grasedyck, L., Hackbusch, W.: Black box low tensor-rank approximation using fiber-crosses. Constr. Approx. 30, 557–597 (2009)MathSciNetzbMATHCrossRefGoogle Scholar
  13. 13.
    Falcó, A., Hackbusch, W.: On minimal subspaces in tensor representations. Found. Comput. Math. 12, 765–803 (2012)MathSciNetzbMATHCrossRefGoogle Scholar
  14. 14.
    Falco, A., Hackbusch, W., Nouy, A.: Geometric Structures in Tensor Representations (Final Release). ArXiv e-prints (2015)Google Scholar
  15. 15.
    Falcó, A., Hackbusch, W., Nouy, A.: On the Dirac–Frenkel variational principle on tensor Banach spaces. Found. Comput. Math. (2018).
  16. 16.
    Falcó, A., Hackbusch, W., Nouy, A.: Tree-based tensor formats. SeMA J. (2018).
  17. 17.
    Grasedyck, L.: Hierarchical singular value decomposition of tensors. SIAM J. Matrix Anal. Appl. 31, 2029–2054 (2010)MathSciNetzbMATHCrossRefGoogle Scholar
  18. 18.
    Grasedyck, L., Kressner, D., Tobler, C.: A literature survey of low-rank tensor approximation techniques. GAMM-Mitteilungen 36(1), 53–78 (2013)MathSciNetzbMATHCrossRefGoogle Scholar
  19. 19.
    Grelier, E., Nouy, A., Chevreuil, M.: Learning with tree-based tensor formats (2018). arXiv e-prints arXiv:1811.04455
  20. 20.
    Hackbusch, W.: Tensor Spaces and Numerical Tensor Calculus, Volume 42 of Springer Series in Computational Mathematics. Springer, Heidelberg (2012)CrossRefGoogle Scholar
  21. 21.
    Hackbusch, W., Kuhn, S.: A new scheme for the tensor representation. J. Fourier Anal. Appl. 15(5), 706–722 (2009)MathSciNetzbMATHCrossRefGoogle Scholar
  22. 22.
    Hillar, C., Lim, L.-H.: Most tensor problems are np-hard. J. ACM (JACM) 60(6), 45 (2013)MathSciNetzbMATHCrossRefGoogle Scholar
  23. 23.
    Holtz, S., Rohwedder, T., Schneider, R.: On manifolds of tensors of fixed tt-rank. Numer. Math. 120(4), 701–731 (2012)MathSciNetzbMATHCrossRefGoogle Scholar
  24. 24.
    Jirak, M., Wahl, M.: A tight \(\sin \varTheta \) theorem for empirical covariance operators. ArXiv e-prints (2018)Google Scholar
  25. 25.
    Jirak, M., Wahl, M.: Relative perturbation bounds with applications to empirical covariance operators. ArXiv e-prints (2018)Google Scholar
  26. 26.
    Khoromskij, B.: O (dlog n)-quantics approximation of nd tensors in high-dimensional numerical modeling. Constr. Approx. 34(2), 257–280 (2011)MathSciNetzbMATHCrossRefGoogle Scholar
  27. 27.
    Khoromskij, B.: Tensors-structured numerical methods in scientific computing: survey on recent advances. Chemometr. Intell. Lab. Syst. 110(1), 1–19 (2012)MathSciNetCrossRefGoogle Scholar
  28. 28.
    Kolda, T.G., Bader, B.W.: Tensor decompositions and applications. SIAM Rev. 51(3), 455–500 (2009)MathSciNetzbMATHCrossRefGoogle Scholar
  29. 29.
    Kressner, D., Steinlechner, M., Uschmajew, A.: Low-rank tensor methods with subspace correction for symmetric eigenvalue problems. SIAM J. Sci. Comput. 36(5), A2346–A2368 (2014)MathSciNetzbMATHCrossRefGoogle Scholar
  30. 30.
    Lubich, C., Rohwedder, T., Schneider, R., Vandereycken, B.: Dynamical approximation by hierarchical tucker and tensor-train tensors. SIAM J. Matrix Anal. Appl. 34(2), 470–494 (2013)MathSciNetzbMATHCrossRefGoogle Scholar
  31. 31.
    Luu, T.H., Maday, Y., Guillo, M., Guérin, P.: A new method for reconstruction of cross-sections using Tucker decomposition. J. Comput. Phys. 345, 189–206 (2017)MathSciNetzbMATHCrossRefGoogle Scholar
  32. 32.
    Maday, Y., Nguyen, N.C., Patera, A.T., Pau, G.S.H.: A general multipurpose interpolation procedure: the magic points. Commun. Pure Appl. Anal. 8(1), 383–404 (2009)MathSciNetzbMATHCrossRefGoogle Scholar
  33. 33.
    Megginson, R.E: An Introduction to Banach Space Theory, Vol. 183. Springer, Berlin (2012)Google Scholar
  34. 34.
    Nouy, A.: Low-rank methods for high-dimensional approximation and model order reduction. In: Benner, P., Cohen, A., Ohlberger, M., Willcox, K. (eds.) Model Reduction and Approximation: Theory and Algorithms. SIAM, Philadelphia (2017)Google Scholar
  35. 35.
    Nouy, A.: Low-Rank Tensor Methods for Model Order Reduction, pp. 857–882. Springer, Cham (2017)Google Scholar
  36. 36.
    Orus, R.: A practical introduction to tensor networks: matrix product states and projected entangled pair states. Ann. Phys. 349, 117–158 (2014)MathSciNetzbMATHCrossRefGoogle Scholar
  37. 37.
    Oseledets, I.: Tensor-train decomposition. SIAM J. Sci. Comput. 33(5), 2295–2317 (2011)MathSciNetzbMATHCrossRefGoogle Scholar
  38. 38.
    Oseledets, I., Tyrtyshnikov, E.: Breaking the curse of dimensionality, or how to use svd in many dimensions. SIAM J. Sci. Comput. 31(5), 3744–3759 (2009)MathSciNetzbMATHCrossRefGoogle Scholar
  39. 39.
    Oseledets, I., Tyrtyshnikov, E.: TT-cross approximation for multidimensional arrays. Linear Algebra Appl. 432(1), 70–88 (2010)MathSciNetzbMATHCrossRefGoogle Scholar
  40. 40.
    Oseledets, I., Tyrtyshnikov, E.: Algebraic wavelet transform via quantics tensor train decomposition. SIAM J. Sci. Comput. 33(3), 1315–1328 (2011)MathSciNetzbMATHCrossRefGoogle Scholar
  41. 41.
    Reiß, M., Wahl, M.: Non-asymptotic upper bounds for the reconstruction error of PCA (2016). arXiv preprint arXiv:1609.03779
  42. 42.
    Schneider, R., Uschmajew, A.: Approximation rates for the hierarchical tensor format in periodic Sobolev spaces. J. Complex. 30(2), 56–71 (2014). Dagstuhl 2012MathSciNetzbMATHCrossRefGoogle Scholar
  43. 43.
    Temlyakov, V.: Nonlinear methods of approximation. Found. Comput. Math. 3(1), 33–107 (2003)MathSciNetzbMATHCrossRefGoogle Scholar
  44. 44.
    Temlyakov, V.: Greedy Approximation. Cambridge Monographs on Applied and Computational Mathematics. Cambridge University Press, Cambridge (2011)CrossRefGoogle Scholar
  45. 45.
    Uschmajew, A., Vandereycken, B.: The geometry of algorithms using hierarchical tensors. Linear Algebra Appl. 439(1), 133–166 (2013)MathSciNetzbMATHCrossRefGoogle Scholar

Copyright information

© Springer-Verlag GmbH Germany, part of Springer Nature 2019

Authors and Affiliations

  1. 1.Centrale Nantes, LMJL, UMR CNRS 6629NantesFrance

Personalised recommendations