Efficient reduced-rank methods for Gaussian processes with eigenfunction expansions

Abstract

In this work, we introduce a reduced-rank algorithm for Gaussian process regression. Our numerical scheme converts a Gaussian process on a user-specified interval to its Karhunen–Loève expansion, the $L^2$-optimal reduced-rank representation. Numerical evaluation of the Karhunen–Loève expansion is performed once during precomputation and involves computing a numerical eigendecomposition of an integral operator whose kernel is the covariance function of the Gaussian process. The Karhunen–Loève expansion is independent of observed data and depends only on the covariance kernel and the size of the interval on which the Gaussian process is defined. The scheme of this paper does not require translation invariance of the covariance kernel. We also introduce a class of fast algorithms for Bayesian fitting of hyperparameters and demonstrate the performance of our algorithms with numerical experiments in one and two dimensions. Extensions to higher dimensions are mathematically straightforward but suffer from the standard curses of high dimensions.

A Legendre polynomials

We now provide a brief overview of Legendre polynomials and Gaussian quadrature (Abramowitz and Stegun 1964). For a more in-depth analysis of these tools and their role in (numerical) approximation theory see, for example, (Trefethen 2020).

In accordance with standard practice, we denote by $P_i: [-1, 1] \rightarrow \mathbb {R}$ the Legendre polynomial of degree i defined by the three-term recursion

$$\begin{aligned} P_{i+1}(x) = \frac{2i + 1}{i + 1} \, x \, P_{i}(x) - \frac{i}{i + 1} P_{i-1}(x) \end{aligned}$$

(A.1)

with initial conditions

$$\begin{aligned} P_0(x) = 1 \qquad \text {and} \qquad P_1(x) = x. \end{aligned}$$

(A.2)

Legendre polynomials are orthogonal on $[-1, 1]$ and satisfy

$$\begin{aligned} \int _{-1}^{1} P_i(x) \, P_j(x) \, \mathrm{d}x = {\left\{ \begin{array}{ll} 0 &{} i \ne j, \\ \frac{2}{2i+1} &{} i = j. \end{array}\right. } \end{aligned}$$

We denote the $L^2$ normalized Legendre polynomials, $\overline{P}_i$, which are defined by

$$\begin{aligned} \overline{P}_i(x) = \sqrt{\frac{2i + 1}{2}} P_i(x). \end{aligned}$$

(A.3)

For each n, the Legendre polynomial $P_n$ has n distinct roots which we denote in what follows by $x_1,...,x_n$. Furthermore, for all n, there exist n positive real numbers $w_1,...,w_n$ such that for any polynomial p of degree $\le 2n - 1$,

$$\begin{aligned} \int _{-1}^1 p(x) \, \mathrm{d}x = \sum _{i=1}^n w_i \, p(x_i). \end{aligned}$$

(A.4)

The roots $x_1,\ldots ,x_n$ are usually referred to as order-n Gaussian nodes and $w_1,...,w_n$ the associated Gaussian quadrature weights. Classical Gaussian quadratures such as this are associated with many families of orthogonal polynomials: Chebyshev, Hermite, Laguerre, etc. The quadratures we mention above, associated with Legendre polynomials, provide a high-order method for discretizing (i.e., interpolating) and integrating square-integrable functions on a finite interval. Legendre polynomials are the natural orthogonal polynomial basis for square-integrable functions on the interval $[-1,1]$, and the associated interpolation and quadrature formulae provide nearly optimal approximation tools for these functions, even if they are not, in fact, polynomials.

The following well-known lemma regarding interpolation using Legendre polynomials will be used in the numerical schemes discussed in this paper. A proof can be found in Stoer and Bulirsch (1992), for example.

Theorem 4

Let $x_1,...,x_n$ be the order-n Gaussian nodes and $w_1,...,w_n$ the associated order-n Gaussian weights. Then, there exists an $n \times n$ matrix $\varvec{\mathsf {M}}$ that maps a function tabulated at these Gaussian nodes to the corresponding Legendre expansion, i.e., the interpolating polynomial expressed in terms of Legendre polynomials. That is to say, defining $\varvec{\mathsf {f}}$ by

$$\begin{aligned} \varvec{\mathsf {f}} = \left( f(x_1) \cdots f(x_n) \right) ^{\mathsf {T}}, \end{aligned}$$

(A.5)

the vector

$$\begin{aligned} {{\varvec{\alpha }}} = \varvec{\mathsf {M}}\varvec{\mathsf {f}} \end{aligned}$$

(A.6)

are the coefficients of the order-n Legendre expansion p such that

$$\begin{aligned} \begin{aligned} p(x_j)&= \sum _{i=1}^n \alpha _i \, P_{i-1}(x_j)\\&= f(x_j), \end{aligned} \end{aligned}$$

(A.7)

where $\alpha _i$ denotes the ith entry of the vector ${{\varvec{\alpha }}}$.

From a computational standpoint, algorithms for efficient evaluation of Legendre polynomials and Gaussian nodes and weights are available in standard software packages (e.g., Driscoll et al. 2014). Furthermore, the entries of the matrix $\varvec{\mathsf {M}}$ can be computed directly via $M_{i, j} = w_j P_{i-1}(x_j)$.