Efficient randomized tensor-based algorithms for function approximation and low-rank kernel interactions

Abstract

In this paper, we introduce a method for multivariate function approximation using function evaluations, Chebyshev polynomials, and tensor-based compression techniques via the Tucker format. We develop novel randomized techniques to accomplish the tensor compression, provide a detailed analysis of the computational costs, provide insight into the error of the resulting approximations, and discuss the benefits of the proposed approaches. We also apply the tensor-based function approximation to develop low-rank matrix approximations to kernel matrices that describe pairwise interactions between two sets of points; the resulting low-rank approximations are efficient to compute and store (the complexity is linear in the number of points). We present an adaptive version of the function and kernel approximation that determines an approximation that satisfies a user-specified relative error over a set of random points. We extend our approach to the case where the kernel requires repeated evaluations for many values of (hyper)parameters that govern the kernel. We give detailed numerical experiments on example problems involving multivariate function approximation, low-rank matrix approximations of kernel matrices involving well-separated clusters of sources and target points, and a global low-rank approximation of kernel matrices with an application to Gaussian processes. We observe speedups up to 18X over standard matrix-based approaches.

Appendix.: Proof of Theorem 1

For each $j = 1,\dots ,N$, let $\mathbf {\Pi }_{j} = \mathbf {A}_{j}\mathbf {P}_{j}^{\top }$. Recall that $\widehat {\boldsymbol {{\mathcal{M}}}} = \boldsymbol {{\mathcal{M}}} \times _{j=1}^{N} \mathbf {\Pi }_{j}$. Considering the term $\boldsymbol {{\mathcal{M}}}-\widehat {\boldsymbol {{\mathcal{M}}}}$, we can add and subtract the terms $\boldsymbol {{\mathcal{M}}} \times _{i=1}^{j} \mathbf {\Pi }_{i}$ for $j = 1,\dots ,N-1$ as follows.

$$\begin{aligned} \boldsymbol{\mathcal{M}}-\widehat{\boldsymbol{\mathcal{M}}} &= \boldsymbol{\mathcal{M}}-\boldsymbol{\mathcal{M}} \times_{j=1}^{N} \mathbf{\Pi}_{j} \\ &= \boldsymbol{\mathcal{M}} \times_{1} (\mathbf{I}-\mathbf{\Pi}_{1}) + \boldsymbol{\mathcal{M}} \times_{1} \mathbf{\Pi}_{1} \times_{2} (\mathbf{I}-\mathbf{\Pi}_{2}) + {\dots} + \boldsymbol{\mathcal{M}} \times_{j=1}^{N-1} \mathbf{\Pi}_{j} \times_{N} (\mathbf{I}-\mathbf{\Pi}_{N}). \end{aligned}$$

Then, taking the Frobenius norm and applying the triangle inequality, we have

$$\|\boldsymbol{\mathcal{M}}-\widehat{\boldsymbol{\mathcal{M}}}\|_{F} \leq \sum\limits_{j=1}^{N} \| \boldsymbol{\mathcal{M}} \times_{i=1}^{j-1} \mathbf{\Pi}_{i} \times_{j} (\mathbf{I}-\mathbf{\Pi}_{j}) \|_{F}.$$

Using the linearity of expectations, we get

$$\mathbb{E}_{\{\mathbf{\Omega}\}_{k=1}^{N}} \|\boldsymbol{\mathcal{M}}-\widehat{\boldsymbol{\mathcal{M}}}\|_{F} \leq \sum\limits_{j=1}^{N} \mathbb{E}_{\mathbf{\Omega}_{j}} \| \boldsymbol{\mathcal{M}} \times_{i=1}^{j-1} \mathbf{\Pi}_{i} \times_{j} (\mathbf{I}-\mathbf{\Pi}_{j}) \|_{F}.$$

(18)

Now, consider the term $\|\boldsymbol{\mathcal{M}} \times _{i=1}^{j-1} \mathbf {\Pi }_{i} \times _{j} (\mathbf {I}-\mathbf {\Pi }_{j}) \|_{F}$. We can use the submultiplicativity property ∥AB∥_F ≤∥A∥₂∥B∥_F to obtain

$$\|\boldsymbol{\mathcal{M}} \times_{i=1}^{j-1} \mathbf{\Pi}_{i} \times_{j} (\mathbf{I}-\mathbf{\Pi}_{j}) \|_{F} \leq {\prod}_{i=1}^{j-1} \|\mathbf{\Pi}_{i}\|_{2} \| \boldsymbol{\mathcal{M}} \times_{j} (\mathbf{I}-\mathbf{\Pi}_{j})\|_{F}.$$

(19)

Note that $\mathbf {\Pi }_{j}\mathbf {Q}_{j} \mathbf {Q}_{j}^{\top } = \mathbf {Q}_{j} \mathbf {Q}_{j}^{\top }$, meaning that $\mathbf {I}-\mathbf {\Pi }_{j} = (\mathbf {I}-\mathbf {\Pi }_{j})(\mathbf {I}-\mathbf {Q}_{j}\mathbf {Q}_{j}^{\top })$. We can then rewrite each term $\|\boldsymbol{\mathcal{M}} \times _{j} (\mathbf {I}-\mathbf {\Pi }_{j})\|_{F}$ as

$$\begin{aligned} \| \boldsymbol{\mathcal{M}} \times_{j} (\mathbf{I}-\mathbf{\Pi}_{j})\|_{F} &= \| \boldsymbol{\mathcal{M}} \times_{j} (\mathbf{I}-\mathbf{\Pi}_{j})(\mathbf{I}-\mathbf{Q}_{j} \mathbf{Q}_{j}^{\top})\|_{F} \\ &\leq \|\boldsymbol{\mathcal{M}} \times_{j} (\mathbf{I}-\mathbf{Q}_{j}\mathbf{Q}_{j}^{\top})\|_{F} \|\mathbf{I}-\mathbf{\Pi}_{j}\|_{2}. \end{aligned}$$

(20)

We note that $\boldsymbol\Pi_{j} \neq \mathbf{I}$ since rank $(\boldsymbol\Pi_{j})$ ≤ rank (Q_j) = r_t + p < n, and $\boldsymbol\Pi_{j} \neq \mathbf{0}$ since Q_j has orthonormal columns and $\mathbf {P}_{j}^{\top } \mathbf {Q}_{j}$ is invertible. Therefore, using the main result of [51], $\| \mathbf{I} - \boldsymbol\Pi_{j} \|_{2} = \| \boldsymbol\Pi_{j} \|_{2}$. Then using [18, Lemma 2.1], $\|\mathbf {I}-\mathbf {\Pi }_{j}\|_{2} = \|\mathbf {\Pi }_{j}\|_{2} = \|(\mathbf {P}_{j}^{\top }\mathbf {Q}_{j})^{-1}\|_{2} \leq g(n,\ell ).$ Combining this result with (19) and (20), we obtain

$$\begin{aligned} \|\boldsymbol{\mathcal{M}} \times_{i=1}^{j-1} \mathbf{\Pi}_{j} \times (\mathbf{I}-\mathbf{\Pi}_{j})\|_{F} &\leq \left(g(n,\ell)\right)^{j} \|\boldsymbol{\mathcal{M}} \times_{j} (\mathbf{I}-\mathbf{Q}_{j}\mathbf{Q}_{j}^{\top})\|_{F} \\ &= \left(g(n,\ell)\right)^{j} \|(\mathbf{I}-\mathbf{Q}_{j}\mathbf{Q}_{j}^{\top})\mathbf{M}_{(j)}\|_{F}. \end{aligned}$$

The last equality comes by unfolding the term $\boldsymbol {{\mathcal{M}}} \times _{j} (\mathbf {I}-\mathbf {Q}_{j}\mathbf {Q}_{j}^{\top })$ along mode-j. By taking expectations and applying [62, Theorem 3], we have

$$\begin{aligned} \mathbb{E}_{\mathbf{\Omega}_{j}}\|\boldsymbol{\mathcal{M}} \times_{i=1}^{j-1} \mathbf{\Pi}_{j} \times_{j} (\mathbf{I}-\mathbf{\Pi}_{j})\|_{F} &\leq (g(n,\ell))^{j} \mathbb{E}_{\mathbf{\Omega}_{j}} \|(\mathbf{I}-\mathbf{Q}_{j}\mathbf{Q}_{j}^{\top})\mathbf{M}_{(j)}\|_{F} \\ &\leq (g(n,\ell))^{j} \left(\left(1+\frac{r_{t}}{p-1} \right) {\sum}_{i=r_{t}+1}^{n} {\sigma_{i}^{2}}\left(\mathbf{M}_{(j)}\right)\right)^{1/2}. \end{aligned}$$

Combining this result with (18), we obtain the desired bound.