Abstract
In this paper, we introduce a method for multivariate function approximation using function evaluations, Chebyshev polynomials, and tensor-based compression techniques via the Tucker format. We develop novel randomized techniques to accomplish the tensor compression, provide a detailed analysis of the computational costs, provide insight into the error of the resulting approximations, and discuss the benefits of the proposed approaches. We also apply the tensor-based function approximation to develop low-rank matrix approximations to kernel matrices that describe pairwise interactions between two sets of points; the resulting low-rank approximations are efficient to compute and store (the complexity is linear in the number of points). We present an adaptive version of the function and kernel approximation that determines an approximation that satisfies a user-specified relative error over a set of random points. We extend our approach to the case where the kernel requires repeated evaluations for many values of (hyper)parameters that govern the kernel. We give detailed numerical experiments on example problems involving multivariate function approximation, low-rank matrix approximations of kernel matrices involving well-separated clusters of sources and target points, and a global low-rank approximation of kernel matrices with an application to Gaussian processes. We observe speedups up to 18X over standard matrix-based approaches.
Similar content being viewed by others
Data availability
No new data was generated as part of this study.
References
U.S. hourly precipitation data, https://catalog.data.gov/dataset/u-s-hourly-precipitation-data
Ahmadi-Asl, S., Abukhovich, S., Asante-Mensah, M.G., Cichocki, A., Phan, A.H., Tanaka, T., Oseledets, I.: Randomized algorithms for computation of Tucker decomposition and higher order SVD (HOSVD). IEEE Access 9, 28684–28706 (2021)
Ahmadi-Asl, S., Caiafa, C.F., Cichocki, A., Phan, A.H., Tanaka, T., Oseledets, I., Wang, J.: Cross tensor approximation methods for compression and dimensionality reduction. IEEE Access 9, 150809–150838 (2021)
Barnes, J., Hut, P.: A hierarchical O(n log n) force-calculation algorithm. Nature 324(6096), 446–449 (1986)
Bebendorf, M., Kunis, S.: Recompression techniques for adaptive cross approximation. The Journal of Integral Equations and Applications, pp. 331–357 (2009)
Benner, P., Khoromskaia, V., Khoromskij, B.N.: Range-separated tensor format for many-particle modeling. SIAM J. Sci. Comput. 40(2), A1034–A1062 (2018)
Bigoni, D., Engsig-Karup, A.P., Marzouk, Y.M.: Spectral tensor-train decomposition. SIAM J. Sci. Comput. 38(4), A2405–A2439 (2016)
Bingham, D.: Virtual library of simulation experiments: test functions and datasets, http://www.sfu.ca/ssurjano/index.html
Börm, S., Grasedyck, L., Hackbusch, W.: Hierarchical matrices. Lecture notes 21, 2003 (2003)
Bungartz, H.-J., Griebel, M.: Sparse grids. Acta Numerica 13, 147–269 (2004)
Cambier, L., Darve, E.: Fast low-rank kernel matrix factorization using skeletonized interpolation. SIAM J. Sci. Comput. 41(3), A1652–A1680 (2019)
Chen, C., Aubry, S., Oppelstrup, T., Arsenlis, A., Darve, E.: Fast algorithms for evaluating the stress field of dislocation lines in anisotropic elastic media. Modelling and Simulation in Materials Science and Engineering 26(4), 045007 (2018)
Constantine, P.G., Diaz, P.: Global sensitivity metrics from active subspaces. Reliability Engineering & System Safety 162, 1–13 (2017)
Corona, E., Rahimian, A., Zorin, D.: A tensor-train accelerated solver for integral equations in complex geometries. J. Comput. Phys. 334, 145–169 (2017)
De Lathauwer, L., De Moor, B., Vandewalle, J.: A multilinear singular value decomposition. SIAM Journal on Matrix Analysis and Applications 21 (4), 1253–1278 (2000)
Dolgov, S., Kressner, D., Strössner, C.: Functional Tucker approximation using Chebyshev interpolation. SIAM J. Sci. Comput. 43(3), A2190–A2210 (2021)
Dong, K., Eriksson, D., Nickisch, H., Bindel, D., Wilson, A.G.: Scalable log determinants for Gaussian process kernel learning. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems, vol. 30. Curran Associates, Inc. (2017)
Drmač, Z., Saibaba, A.K.: The discrete empirical interpolation method: canonical structure and formulation in weighted inner product spaces. SIAM Journal on Matrix Analysis and Applications 39(3), 1152–1180 (2018)
Fong, W., Darve, E.: The black-box fast multipole method. J. Comput. Phys. 228(23), 8712–8725 (2009)
Gaß, M., Glau, K., Mahlstedt, M., Mair, M.: Chebyshev interpolation for parametric option pricing. Finance Stochast. 22(3), 701–731 (2018)
Golub, G.H., Van Loan, C.F.: Matrix Computations, 4th edn. Johns Hopkins Studies in the Mathematical Sciences, Johns Hopkins University Press, Baltimore, MD (2013)
Gorodetsky, A., Karaman, S., Marzouk, Y.: A continuous analogue of the tensor-train decomposition. Comput. Methods Appl. Mech. Eng. 347, 59–84 (2019)
Grasedyck, L., Hackbusch, W.: Construction and arithmetics of H-matrices. Computing 70(4), 295–334 (2003)
Greengard, L., Rokhlin, V.: A fast algorithm for particle simulations. Journal of computational physics 73(2), 325–348 (1987)
Greengard, L., Strain, J.: The fast Gauss transform. SIAM J. Sci. Stat. Comput. 12(1), 79–94 (1991)
Griebel, M., Harbrecht, H.: Analysis of tensor approximation schemes for continuous functions. arXiv:1903.04234 (2019)
Gu, M., Eisenstat, S.C.: Efficient algorithms for computing a strong rank-revealing QR factorization. SIAM J. Sci. Comput. 17(4), 848–869 (1996)
Hackbusch, W.: A sparse matrix arithmetic based on H-matrices. Part I: introduction to H-matrices. Computing 62(2), 89–108 (1999)
Hackbusch, W.: Hierarchical matrices: algorithms and analysis, vol. 49 Springer (2015)
Hackbusch, W., Börm, S.: Data-sparse approximation by adaptive H2-matrices. Computing 69(1), 1–35 (2002)
Halko, N., Martinsson, P.-G., Tropp, J.A.: Finding structure with randomness: probabilistic algorithms for constructing approximate matrix decompositions. SIAM Rev. 53(2), 217–288 (2011)
Hashemi, B., Trefethen, L.N.: Chebfun in three dimensions. SIAM J. Sci. Comput. 39(5), C341–C363 (2017)
Ho, K.L., Greengard, L.: A fast semidirect least squares algorithm for hierarchically block separable matrices. SIAM Journal on Matrix Analysis and Applications 35(2), 725–748 (2014)
Khoromskaia, V., Khoromskij, B.N.: Grid-based lattice summation of electrostatic potentials by assembled rank-structured tensor approximation. Comput. Phys. Commun. 185(12), 3162–3174 (2014)
Khoromskaia, V., Khoromskij, B.N.: Tensor numerical methods in quantum chemistry. De Gruyter (2018)
Khoromskaia, V., Khoromskij, B.N.: Prospects of tensor-based numerical modeling of the collective electrostatics in many-particle systems. Comput. Math. Math. Phys. 61(5), 864–886 (2021)
Khoromskaia, V., Khoromskij, B.N.: Reduced higher order SVD: ubiquitous rank-reduction method in tensor-based scientific computing. arXiv:2201.12663 (2022)
Khoromskij, B.N.: Tensors-structured numerical methods in scientific computing: survey on recent advances. Chemom. Intell. Lab. Syst. 110(1), 1–19 (2012)
Khoromskij, B.N.: Tensor numerical methods in scientific computing. De Gruyter (2018)
Kolda, T.G., Bader, B.W.: Tensor decompositions and applications. SIAM Rev. 51(3), 455–500 (2009)
Li, M., Bi, W., Kwok, J.T., Lu, B.-L.: Large-scale Nyström kernel matrix approximation using randomized SVD. IEEE Transactions on Neural Networks and Learning Systems 26(1), 152–164 (2014)
Litvinenko, A., Keyes, D., Khoromskaia, V., Khoromskij, B.N., Matthies, H.G.: Tucker tensor analysis of Matérn functions in spatial statistics. Computational Methods in Applied Mathematics 19(1), 101–122 (2019)
Liu, H., Ong, Y.-S., Shen, X., Cai, J.: When Gaussian process meets big data: a review of scalable GPs. IEEE Transactions on Neural Networks and Learning Systems 31(11), 4405–4423 (2020)
Mason, J.C., Handscomb, D.C.: Chebyshev Polynomials. CRC press (2002)
Minster, R., Saibaba, A.K., Kilmer, M.E.: Randomized algorithms for low-rank tensor decompositions in the Tucker format. SIAM Journal on Mathematics of Data Science 2(1), 189–215 (2020)
Rai, P., Kolla, H., Cannada, L., Gorodetsky, A.: Randomized functional sparse Tucker tensor for compression and fast visualization of scientific data. arXiv:1907.05884 (2019)
Rakhuba, M., Oseledets, I.V.: Grid-based electronic structure calculations: the tensor decomposition approach. J. Comput. Phys. 312, 19–30 (2016)
Sauter, S.A., Schwab, C.: Boundary element methods. In: Boundary Element Methods, Springer, pp. 183–287 (2010)
Si, S., Hsieh, C.-J., Dhillon, I.: Memory efficient kernel approximation. In: International Conference on Machine Learning, PMLR, pp. 701–709 (2014)
Sun, Y., Guo, Y., Luo, C., Tropp, J., Udell, M.: Low-rank tucker approximation of a tensor from streaming data. SIAM Journal on Mathematics of Data Science 2(4), 1123–1150 (2020)
Szyld, D.B.: The many proofs of an identity on the norm of oblique projections. Numerical Algorithms 42(3-4), 309–323 (2006)
Takahashi, T., Coulier, P., Darve, E.: Application of the inverse fast multipole method as a preconditioner in a 3D Helmholtz boundary element method. J. Comput. Phys. 341, 406–428 (2017)
Trefethen, L.: Multivariate polynomial approximation in the hypercube. Proceedings of the American Mathematical Society 145(11), 4837–4844 (2017)
Trefethen, L.N.: Approximation Theory and Approximation Practice. Society for Industrial and Applied Mathematics. SIAM, Philadelphia, PA (2013)
Wilson, A., Nickisch, H.: Kernel interpolation for scalable structured Gaussian processes (KISS-GP). In: International Conference on Machine Learning, PMLR, pp. 1775–1784 (2015)
Xia, J., Chandrasekaran, S., Gu, M., Li, X.S.: Fast algorithms for hierarchically semiseparable matrices. Numerical Linear Algebra with Applications 17(6), 953–976 (2010)
Xu, K.: The Chebyshev points of the first kind. Appl. Numer. Math. 102, 17–30 (2016)
Xu, Z., Cambier, L., Rouet, F.-H., L’Eplatennier, P., Huang, Y., Ashcraft, C., Darve, E.: Low-rank kernel matrix approximation using skeletonized interpolation with endo- or exo-vertices. arXiv:1807.04787 (2018)
Ye, X., Xia, J., Ying, L.: Analytical low-rank compression via proxy point selection. SIAM Journal on Matrix Analysis and Applications 41(3), 1059–1085 (2020)
Ying, L., Biros, G., Zorin, D.: A kernel-independent adaptive fast multipole algorithm in two and three dimensions. J. Comput. Phys. 196 (2), 591–626 (2004)
Yu, W., Gu, Y., Li, Y.: Efficient randomized algorithms for the fixed-precision low-rank matrix approximation. SIAM Journal on Matrix Analysis and Applications 39(3), 1339–1359 (2018)
Zhang, J., Saibaba, A.K., Kilmer, M.E., Aeron, S.: A randomized tensor singular value decomposition based on the t-product. Numerical Linear Algebra with Applications 25(5), e2179 (2018)
Funding
A.K.S. was supported, in part, by the National Science Foundation (NSF) through the grants DMS-1821149, DMS-1845406, and DMS-1745654. M.E.K. was supported, in part, by the National Science Foundation under Grants NSF DMS 1821148 and Tufts T-Tripods Institute NSF HDR grant CCF-1934553.
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Conflict of interest
The authors declare no competing interests.
Additional information
Communicated by: Gunnar J Martinsson
Publisher’s note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Appendix.: Proof of Theorem 1
Appendix.: Proof of Theorem 1
For each \(j = 1,\dots ,N\), let \(\mathbf {\Pi }_{j} = \mathbf {A}_{j}\mathbf {P}_{j}^{\top }\). Recall that \(\widehat {\boldsymbol {{\mathcal{M}}}} = \boldsymbol {{\mathcal{M}}} \times _{j=1}^{N} \mathbf {\Pi }_{j}\). Considering the term \(\boldsymbol {{\mathcal{M}}}-\widehat {\boldsymbol {{\mathcal{M}}}}\), we can add and subtract the terms \(\boldsymbol {{\mathcal{M}}} \times _{i=1}^{j} \mathbf {\Pi }_{i}\) for \(j = 1,\dots ,N-1\) as follows.
Then, taking the Frobenius norm and applying the triangle inequality, we have
Using the linearity of expectations, we get
Now, consider the term \(\|\boldsymbol{\mathcal{M}} \times _{i=1}^{j-1} \mathbf {\Pi }_{i} \times _{j} (\mathbf {I}-\mathbf {\Pi }_{j}) \|_{F}\). We can use the submultiplicativity property ∥AB∥F ≤∥A∥2∥B∥F to obtain
Note that \(\mathbf {\Pi }_{j}\mathbf {Q}_{j} \mathbf {Q}_{j}^{\top } = \mathbf {Q}_{j} \mathbf {Q}_{j}^{\top }\), meaning that \(\mathbf {I}-\mathbf {\Pi }_{j} = (\mathbf {I}-\mathbf {\Pi }_{j})(\mathbf {I}-\mathbf {Q}_{j}\mathbf {Q}_{j}^{\top })\). We can then rewrite each term \(\|\boldsymbol{\mathcal{M}} \times _{j} (\mathbf {I}-\mathbf {\Pi }_{j})\|_{F}\) as
We note that \(\boldsymbol\Pi_{j} \neq \mathbf{I}\) since rank \((\boldsymbol\Pi_{j})\) ≤ rank (Qj) = rt + p < n, and \(\boldsymbol\Pi_{j} \neq \mathbf{0}\) since Qj has orthonormal columns and \(\mathbf {P}_{j}^{\top } \mathbf {Q}_{j}\) is invertible. Therefore, using the main result of [51], \(\| \mathbf{I} - \boldsymbol\Pi_{j} \|_{2} = \| \boldsymbol\Pi_{j} \|_{2}\). Then using [18, Lemma 2.1], \(\|\mathbf {I}-\mathbf {\Pi }_{j}\|_{2} = \|\mathbf {\Pi }_{j}\|_{2} = \|(\mathbf {P}_{j}^{\top }\mathbf {Q}_{j})^{-1}\|_{2} \leq g(n,\ell ).\) Combining this result with (19) and (20), we obtain
The last equality comes by unfolding the term \(\boldsymbol {{\mathcal{M}}} \times _{j} (\mathbf {I}-\mathbf {Q}_{j}\mathbf {Q}_{j}^{\top })\) along mode-j. By taking expectations and applying [62, Theorem 3], we have
Combining this result with (18), we obtain the desired bound.
Rights and permissions
Springer Nature or its licensor holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
About this article
Cite this article
Saibaba, A.K., Minster, R. & Kilmer, M.E. Efficient randomized tensor-based algorithms for function approximation and low-rank kernel interactions. Adv Comput Math 48, 66 (2022). https://doi.org/10.1007/s10444-022-09979-7
Received:
Accepted:
Published:
DOI: https://doi.org/10.1007/s10444-022-09979-7
Keywords
- Multivariate function approximation
- Tucker format
- Randomized algorithms
- Low-rank approximations
- Kernel methods