Hilbert space methods for reduced-rank Gaussian process regression

2.1 Spectral densities of homogeneous and isotropic Gaussian processes

2.2 The covariance operator as a pseudo-differential operator

Consider an isotropic covariance function \(k(\mathbf {x},\mathbf {x}') \triangleq k(||\mathbf {r}||)\) (recall that \(||\cdot ||\) denotes the Euclidean norm). The spectral density of the Gaussian process and thus the transfer function corresponding to the covariance operator will now have the form \(S(||{\omega }||)\). We can formally write it as a function of \(||{\omega }||^2\) such that

$$\begin{aligned} S(||{\omega }||) = \psi (||{\omega }||^2). \end{aligned}$$

(5)

Assume that the spectral density \(S(\cdot )\) and hence \(\psi (\cdot )\) have the following polynomial expansion:

$$\begin{aligned} \psi (||{\omega }||^2) = a_0 + a_1 ||{\omega }||^2 + a_2 (||{\omega }||^2)^2 + a_3 (||{\omega }||^2)^3 + \cdots . \end{aligned}$$

(6)

This can be ensured, for example, by requiring that \(\psi (\cdot )\) is an analytic function. Thus we also have

$$\begin{aligned} S(||{\omega }||) = a_0 + a_1 ||{\omega }||^2 + a_2 (||{\omega }||^2)^2 + a_3 (||{\omega }||^2)^3 + \cdots . \end{aligned}$$

(7)

Recall that the transfer function corresponding to the Laplace operator \(\nabla ^2\) is \(-\,||{\omega }||^2\) in the sense that for a regular enough function f we have

$$\begin{aligned} \mathscr {F}[\nabla ^2 f]({\omega }) = -\,||{\omega }||^2 \mathscr {F}[f]({\omega }), \end{aligned}$$

(8)

where \(\mathscr {F}[\cdot ]\) denotes the Fourier transform of its argument. If we take the inverse Fourier transform of (7), we get the following representation for the covariance operator \(\mathcal {K}\), which defines a pseudo-differential operator (Shubin 1987) as a formal series of Laplace operators:

$$\begin{aligned} \mathcal {K} = a_0 + a_1 (-\,\nabla ^2) + a_2 (-\,\nabla ^2)^2 + a_3 (-\,\nabla ^2)^3 + \cdots . \end{aligned}$$

(9)

In the next section, we will use this representation to form a series expansion approximation for the covariance function.

Open image in new window

2.3 Hilbert space approximation of the covariance operator

The right-hand side of (20) is very easy to evaluate, because it corresponds to evaluating the spectral density at the square roots of the eigenvalues and multiplying them with the eigenfunctions of the Laplace operator. Because the eigenvalues of the Laplace operator are monotonically increasing with j and for bounded covariance functions the spectral density goes to zero fast with higher frequencies, we can expect to obtain a good approximation of the right-hand side by retaining only a finite number of terms in the series. However, even with an infinite number of terms this is only an approximation, because we assumed a compact domain with boundary conditions. The approximation can be, though, expected to be good at the input values which are not near the boundary of \(\varOmega \), where the Laplacian was taken to be zero.

As an example, Fig. 1 shows Matérn covariance functions of various degrees of smoothness \(\nu \) (see, e.g., Rasmussen and Williams 2006, Ch. 4) and approximations for different numbers of basis functions in the approximation. The basis consists of the eigenfunctions of the Laplacian in (10) with \(\varOmega = [-\,L,L]\) which gives the eigenfunctions \(\phi _j(x) = L^{-1/2} \sin (\pi j (x + L)/ (2L))\) and the eigenvalues \(\lambda _j = (\pi \, j / (2L))^2\). In the figure, we have set \(L = 1\) and \(\ell = 0.1\). For the squared exponential, the approximation is indistinguishable from the exact curve already at \(m=12\), whereas the less smooth functions require more terms.

2.4 Inner product point of view

For the negative Laplacian, the corresponding definition is

$$\begin{aligned} \mathcal {D} f = -\,\nabla ^2 [f \, w], \end{aligned}$$

(26)

which implies

$$\begin{aligned} \langle \mathcal {D} f,g\rangle = -\int f(\mathbf {x}) \, w(\mathbf {x}) \nabla ^2[g(\mathbf {x})\, w(\mathbf {x})] \,{\mathrm {d}}\mathbf {x}, \end{aligned}$$

(27)

and the operator defined by (26) can be seen to be self-adjoint. Again, there exists an orthonormal basis \(\{ \phi _j(\mathbf {x}) {{\mid }} j=1,2,\ldots \}\) and positive eigenvalues \(\lambda _j\) which satisfy the eigenvalue equation

$$\begin{aligned} (\mathcal {D} \, \phi _j)(\mathbf {x}) = \lambda _j \, \phi _j(\mathbf {x}). \end{aligned}$$

(28)

Thus the kernel of \(\mathcal {D}\) has a series expansion similar to Eq. (13) and thus an approximation can be given in the same form as in Equation (20). In this case, the approximation error comes from approximating the Laplace operator with the smoother operator,

$$\begin{aligned} \nabla ^2 f \approx \nabla ^2 [f \, w], \end{aligned}$$

(29)

which is closely related to assumption of an input density \(w(\mathbf {x})\) for the Gaussian process. However, when the weight function \(w(\cdot )\) is close to constant in the area where the inputs points are located, the approximation is accurate.

Open image in new window

2.5 Connection to Sturm–Liouville theory

3.1 Gaussian process regression

3.2 Learning the hyperparameters

Once the marginal likelihood and its derivatives are available, it is also possible to use other methods for parameter inference such as Markov chain Monte Carlo methods (Liu 2001; Brooks et al. 2011) including Hamiltonian Monte Carlo (HMC, Duane et al. 1987; Neal 2011) as well as numerous others.

3.3 Discussion on the computational complexity

As can be noted from Eq. (20), the basis functions in the reduced-rank approximation do not depend on the hyperparameters of the covariance function. Thus it is enough to calculate the product \(\varvec{\Phi }^\mathsf {T}\varvec{\Phi }\) only once, which means that the method has a overall asymptotic computational complexity of \(\mathcal {O}(nm^2)\). After this initial cost, evaluating the marginal likelihood and the marginal likelihood gradient is an \(\mathcal {O}(m^3)\) operation—which in practice comes from the Cholesky factorization of \(\mathbf {Z}\) on each step.

If the number of observations n is so large that storing the \(n \times m\) matrix \(\varvec{\Phi }\) is not feasible, the computations of \(\varvec{\Phi }^\mathsf {T}\varvec{\Phi }\) can be carried out in blocks. Storing the evaluated eigenfunctions in \(\varvec{\Phi }\) is not necessary, because the \(\phi _j(\mathbf {x})\) are closed-form expressions that can be evaluated when necessary. In practice, it might be preferable to cache the result of \(\varvec{\Phi }^\mathsf {T}\varvec{\Phi }\) (causing a memory requirement scaling as \(\mathcal {O}(m^2)\)), but this is not required.

The computational complexity of conventional sparse GP approximations typically scale as \(\mathcal {O}(nm^2)\) in time for each step of evaluating the marginal likelihood. The scaling in demand of storage is \(\mathcal {O}(nm)\). This comes from the inevitable cost of re-evaluating all results involving the basis functions on each step and storing the matrices required for doing this. This applies to all the methods that will be discussed in Sect. 5, with the exception of SSGP, where the storage demand can be relaxed by re-evaluating the basis functions on demand.

We can also consider the rather restricting, but in certain applications often encountered case, where the measurements are constrained to a regular grid. This causes the product of the orthonormal eigenfunction matrices \(\varvec{\Phi }^\mathsf {T}\varvec{\Phi }\) to be diagonal, avoiding the calculation of the matrix inverse altogether. This relates to the FFT-based methods for GP regression (Paciorek 2007; Fritz et al. 2009), and the projections to the basis functions can be evaluated by fast Fourier transform in \(\mathcal {O}(n \log n)\) time complexity.

3.4 Inverse problems and latent force models

4.1 Univariate Dirichlet case

The univariate convergence result can be summarized as the following theorem which is proved in “Appendix A.2.”

As such, the results above only ensure the convergence of the prior covariance functions. However, it turns out that this also ensures the convergence of the posterior as is summarized in the following corollary.

4.2 Multivariate Cartesian Dirichlet case

The following result for this d-dimensional case is proved in “Appendix A.3.”

4.3 Scaling of error with increasing \({\hat{m}}\)

It is worth noting that due to Remarks 17 and 20, we could actually tighten the bound by introducing \({\hat{m}}\)-dependence to E, but it does not affect the order of scaling, because the dependence on the dimensionality in that term is linear. Furthermore, the latter term actually depends on the ratio \({\hat{m}} / L\) and hence there is a coupling between the number of terms and the size of the domain L. However, we can still get idea of the convergence speed by fixing L.

Let us start by considering the case when \(S(\Vert {{{\omega }}}\Vert )\) is bounded by a reciprocal of a polynomial which is the case, for example, for the Matérn covariance function. We get the following theorem.

The result in the above theorem tells that by selecting an appropriate differentiation order for the covariance function, we can make the convergence speed arbitrarily large. In particular, if we select \(a = d/2\), we get the Monte Carlo rate, and with \(a = d\), we get a convergence rate of \(\sim 1/m\).

The above theorem tells that the convergence in the squared exponential case is faster than \(\sim 1 / m\), independently of the dimensionality d. It is worth noting though that the bound is not independent of the dimensionality in the sense that the constants do depend on it. Strictly speaking, the convergence rate is h(d) / m, for some function h which depends on d. However, as function of m, this rate is independent of the dimensionality.

4.4 Other domains

It would also be possible carry out similar convergence analysis, for example, in a spherical domain. In that case the technical details become slightly more complicated, because instead of sinusoidals we will have Bessel functions and the eigenvalues no longer form a uniform grid. This means that instead of Riemann integrals we need to consider weighted integrals where the distribution of the zeros of Bessel functions is explicitly accounted for. It might also be possible to use some more general theoretical results from mathematical analysis to obtain the convergence results. However, due to these technical challenges more general convergence proof will be developed elsewhere.

There is also a similar technical challenge in the analysis when the basis functions are formed by assuming an input density (see Sect. 2.4) instead of a bounded domain. Because explicit expressions for eigenfunctions and eigenvalues cannot be obtained in general, the elementary proof methods which we used here cannot be applied. Therefore the convergence analysis of this case is also left as a topic for future research.

5.1 Methods from the Nyström family

Figure 3 illustrates the effect of the approximations compared to the exact correlation structure in the GP. The dashed contours show the exact correlation contours computed for three locations with the squared exponential covariance function. Figure 3a shows the results for the FIC approximation with 16 inducing points (locations shown in the figure). It is clear that the number of inducing points or their locations is not sufficient to capture the correlation structure. For similar figures and discussion on the effects of the inducing points, see Vanhatalo et al. (2010). This behavior is not unique to SOR or FIC, but applies to all the methods from the Nyström family.

Open image in new window

5.2 Direct spectral methods

In the case that \(\varOmega \) is not compact, but covers the whole \(\mathbb {R}^d\), and when the covariance function is homogeneous, the eigenvalues defined by (77) are no longer discrete, but they can only be expressed as the spectral density \(S({\omega })\) which can be seen as a continuum of eigenvalues. The eigenfunctions become complex exponentials, that is, sines and cosines—which in turn are a subset of eigenfunctions of Laplace operator. In this background, what (20) essentially says is that we can approximate the Mercer expansion (76) by using the basis consisting of the Laplacian eigenfunctions \(\varphi _j(\mathbf {x}) \approx \phi _j(\mathbf {x})\) and point-wise evaluations of the spectral density at the Laplacian eigenvalues \(\gamma _j \approx S(\sqrt{-\lambda _j})\).

Another related classical connection is to the works in the relationship of spline interpolation and Gaussian process priors (Wahba 1978; Kimeldorf and Wahba 1970; Wahba 1990). In particular, it is well known (see, e.g., Wahba 1990) that spline smoothing can be seen as Gaussian process regression with a specific choice of covariance function. The relationship of the spline regularization with Laplace operators then leads to series expansion representations that are closely related to the approximations considered here.

Contours for the sparse spectrum SSGP method are visualized in Fig. 3c. Here the spectral points were chosen at random following Lázaro-Gredilla (2010). Because the basis functions are spanned using both sines and cosines, the number of spectral points was \(h=8\) in order to match the rank \(m=16\). These results agree well with those presented in the Lázaro-Gredilla et al. (2010) for a one-dimensional example. For this particular set of spectral points, some directions of the contours happen to match the true values very well, while other directions are completely off. Increasing the rank from 16 to 100 would give comparable results to the other methods.

Recently Hensman et al. (2018) presented a variational Fourier feature approximation for Gaussian processes that was derived for the Matérn class of kernels, where the approximation structure is set up by a low-rank plus diagonal structure. The key differences here are the fully diagonal (independent) structure in the \(\mathbf {K}_{u,u}\) matrix (giving rise to additional speedup) and the generality of only requiring the spectral density function to be known.

While SSGP is based on a sparse spectrum, the reduced-rank method proposed in this paper aims to make the spectrum as “full” as possible at a given rank. While SSGP can be interpreted as a Monte Carlo integral approximation, the corresponding interpretation to the proposed method would be a numerical quadrature-based integral approximation (cf. the convergence proof in “Appendix A.2”). Figure 3d shows the same contours obtained by the proposed reduced-rank method. Here the eigendecomposition of the Laplace operator has been obtained for the square \(\varOmega = [-L,L] \times [-L,L]\) with Dirichlet boundary conditions. The contours match well with the full solution toward the middle of the domain. The boundary effects drive the process to zero, which is seen as distortion near the edges.

Figure 3e shows how extending the boundaries just by 25% and keeping the number of basis functions fixed at 16, gives good results. The last Fig. 3f corresponds to using a disk-shaped domain instead of the rectangular. The eigendecomposition of the Laplace operator is done in polar coordinates, and the Dirichlet boundary is visualized by a circle in the figure.

5.3 Structure exploiting and decomposition methods

Other methods for scalable Gaussian processes include many structure exploiting techniques that, similarly to general inducing input methods, aim to be agnostic to the choice of covariance function. They rather exploit the structure of the inputs (see Saatçi 2012, for discussion on Kronecker and Toeplitz algebra) and not the GP prior per se. Most notably, scalable kernel interpolation (SKI, Wilson and Nickisch 2015) is an inducing point method that achieves \(O(n + m \log m)\) time complexity and \(\mathcal {O}(n + m)\) space complexity. Through local cubic kernel interpolation, the SKI framework is used in KISS-GP (see Wilson and Nickisch 2015, for details) which uses Kronecker and Toeplitz algebra on grids of inducing inputs to speed up inference.

The computational complexity of the SKI approach scales cubically in the input dimenionality d. Other recent methods (e.g., Gardner et al. 2018; Izmailov et al. 2018) have reduced the time complexity to linear in d as well (e.g., \(\mathcal {O}(d n + d m \log m)\)). These methods typically leverage parallelization (well suited for GPU calculations) or iterative methods.

Furthermore, general methods form numerical linear algebra for approximately solving eigenvalue and singular value problems allow for fast low-rank decompositions. These methods ignore the kernel learning perspective, but can provide useful tools in practice. For example, the pivoted Cholesky decomposition (Harbrecht et al. 2012; Bach 2013) allows constructing a low-rank approximation to an \(n \times n\) positive definite matrix in \(O(n m^2)\) time. There are also methods for fast randomized singular value decompositions based on subsampled Hadamard transformations (e.g., Boutsidis and Gittens 2013), with some further details in Le et al. (2013). These methods provide speedup to the general linear algebraic problem, but ignore the well-structured nature of the specific application to Gaussian process regression with stationary prior covariance functions.

6.1 Variation of domain size

In addition to the theoretical analysis of approximation error, we provide a study of the effect of choosing the domain size. We set up an experiment where we simulate data (\(n=100\) and all results averaged over 10 independent draws) from GP priors with a squared exponential covariance function with unit hyperparameters and corrupting additive Gaussian noise with variance \(\sigma _{n}^2 = 0.1^2\). The inputs are chosen uniformly randomly in \([-\tilde{L},\tilde{L}]\) with \(\tilde{L}=1\). We study the effect of varying the boundary location \(L \in (1,10]\).

Figure 4 shows the Kullback–Leibler (KL) divergence (see, e.g., Rasmussen and Williams 2006, Appendix A for the identities for the KL between two multivariate Gaussians) between the approximative GP posterior and the exact GP posterior evaluated over ten uniformly spaced points. The same curve is recalculated for \(m=5, 10, 15,\) and 20. The figure shows that the KL has a single minimum that describes the trade-off of being far enough from the data but close enough not to start losing representative power with the given number of basis functions m. Even though the KL suggests there would be a single best choice for L, the practical sensitivity to the choice of L is low. Already for \(m=5\), the MSE in the posterior mean is \(10^{-5}\) (note that the data has unit magnitude scale) when L is chosen one to two length-scales from the data boundary \(\tilde{L}\).

Open image in new window

Open image in new window

6.2 Comparison study

For assessing the performance of different methods, we use 10-fold cross-validation and evaluate the following measures based on the validation set: the standardized mean squared error (SMSE) and the mean standardized log loss (MSLL), respectively defined as:

$$\begin{aligned} {\mathrm {SMSE}} = \sum _{i=1}^{n_*} {(y_{*i} - \mu _{*i})^2 \over {\mathrm {Var}}[y]}, \end{aligned}$$

(81)

and

$$\begin{aligned} {\mathrm {MSLL}} = \frac{1}{2 n_*} \sum _{i=1}^{n_*} \bigg ( { (y_{*i} - \mu _{*i})^2 \over \sigma _{*i}^2} + \log 2\pi \sigma _{*i}^2 \bigg ), \end{aligned}$$

(82)

where \(\mu _{*i} = \mathbb {E}[f(\mathbf {x}_{*i})]\) and \(\sigma _{*i}^2 = \mathbb {V}[f(\mathbf {x}_{*i})] + \sigma _{n}^2\) are the predictive mean and variance for test sample \(i=1,2,\ldots ,n_*\), and \(y_{*i}\) is the actual test value. The training data variance is denoted by \({\mathrm {Var}}[y]\). For all experiments, the values reported are averages over ten repetitions.

Open image in new window

We compare our solution to SOR, DTC, VAR and FIC using the implementations provided in the GPstuff software package (version 4.3.1, see Vanhatalo et al. 2013) for Mathworks Matlab. The sparse spectrum SSGP method (Lázaro-Gredilla et al. 2010) was implemented into the GPstuff toolbox for the comparisons.¹ The reference implementation was modified such that also non-ARD covariances could be accounted for.

The m inducing inputs for SOR, DTC, VAR, and FIC were chosen at random as a subset from the training data and kept fixed between the methods. For low-dimensional inputs, this tends to lead to good results and avoid overfitting to the training data, while optimizing the input locations alongside hyperparameters becomes the preferred approach in high input dimensions (Quiñonero-Candela and Rasmussen 2005b). The results are averaged over ten repetitions in order to present the average performance of the methods. In Sects. 6.2 and 6.3, we used a Cartesian domain with Dirichlet boundary conditions for the new reduced-rank method. To avoid boundary effects, the domain was extended by 10% outside the inputs in each direction.

In the comparisons, we followed the guidelines given by Chalupka et al. (2013) for making comparisons between the actual performance of different methods. For hyperparameter optimization, we used the fminunc routine in Matlab with a Quasi-Newton optimizer. We also tested several other algorithms, but the results were not sensitive to the choice of optimizer. The optimizer was run with a termination tolerance of \(10^{-5}\) on the target function value and on the optimizer inputs. The number of required target function evaluations stayed fairly constant for all the comparisons, making the comparisons for the hyperparameter learning bespoke.

Figure 5 shows a simulated example, where 256 data points are drawn from a Gaussian process prior with a squared exponential covariance function. We use the same parametrization as Rasmussen and Williams (2006) and denote the signal variance \(\sigma ^2\), length scale \(\ell \), and noise variance \(\sigma _{n}^2\). Figure 5b shows the negative marginal log likelihood curves both for the full GP and the approximation with \(m=32\) basis functions. The likelihood curve approximations are almost exact and only differs from the full GP likelihood for small length scales (roughly for values smaller than 2L / m). Figure 5a shows the approximate GP solution. The mean estimate follows the exact GP mean, and the shaded region showing the 95% confidence area differs from the exact solution (dashed) only near the boundaries.

Open image in new window

Figure 6a, b shows the SMSE and MSLL values for \(m=8,10,\ldots ,32\) inducing inputs and basis functions for the toy dataset from Fig. 5. The convergence of the proposed reduced rank method is fast and as soon as the number of eigenfunctions is large enough (\(m=20\)) to account for the short length scales, the approximation converges to the exact full GP solution (shown by the dashed line).

In this case, the SOR method that uses the Nyström approximation to directly approximate the spectrum of the full GP (see Sect. 5) seems to give good results. However, as the resulting approximation in SOR corresponds to a singular Gaussian distribution, the predictive variance is underestimated. This can be seen in Fig. 6b, where SOR seems to give better results than the full GP. These results are however due to the smaller predictive variance on the test set. DTC tries to fix this shortcoming of SOR—they are identical in other respects except predictive variance evaluation—and while SOR and DTC give identical results in terms of SMSE, they differ in MSLL. We also note that additional trace term in the marginal likelihood in VAR makes the likelihood surface flat, which explains the differences in the results in comparison to DTC.

The sparse spectrum SSGP method did not perform well on average. Still, it can be seen that it converges toward the performance of the full GP. The dependence on the number of spectral points differs from the rest of the methods, and a rank of \(m=32\) is not enough to meet the other methods. However, in terms of best case performance over the ten repetitions with different inducing inputs and spectral points, both FIC and SSGP outperformed SOR, DTC, and VAR. Because of its “dense spectrum” approach, the proposed reduced-rank method is not sensitive to the choice of spectral points, and thus, the performance remained the same between repetitions. In terms of variance over the 10-fold cross-validation folds, the methods in order of growing variance in the figure legend (the variance approximately doubling between FULL and SSGP).

6.3 Precipitation data

As a real-data example, we consider a precipitation data set that contain US annual precipitation summaries for year 1995 (\(d=2\) and \(n=5776\), available online, see Vanhatalo et al. 2013). The observation locations are shown on a map in Fig. 7a.

We limit the number of inducing inputs and spectral points to \(m=128,192,\ldots ,512\). For the our Hilbert-GP method, we additionally consider ranks \(m = 1024,1536, \ldots , 4096\), and show that this causes a computational burden of the same order as the conventional sparse GP methods with smaller ms. To avoid boundary effects, the domain was extended by 10% outside the inputs in each direction.

In order to demonstrate the computational benefits of the proposed model, we also present the running time of the GP inference (including hyperparameter optimization). All methods were implemented under a similar framework in the GPstuff package, and they all employ similar reformulations for numerical stability. The key difference in the evaluation times comes from hyperparameter optimization, where SOR, DTC, VAR, FIC, and SSGP scale as \(\mathcal {O}(nm^2)\) for each evaluation of the marginal likelihood. The proposed reduced-rank method scales as \(\mathcal {O}(m^3)\) for each evaluation (after an initial cost of \(\mathcal {O}(nm^2)\)).

Figure 6c, d shows the SMSE and MSLL results for this data against evaluation time. On this scale, we note that the evaluation time and accuracy, both in terms of SMSE and MSLL, are alike for SOR, DTC, VAR, and FIC. SSGP is faster to evaluate in comparison with the Nyström family of methods, which comes from the simpler structure of the approximation. Still, the number of required spectral points to meet a certain average error level is larger for SSGP.

The results for the proposed reduced-rank method(Hilbert-GP) show that with two input dimensions, the required number of basis functions is larger. For the first seven points, we notice that even though the evaluation is two orders of magnitude faster, the method performs only slightly worse in comparison with conventional sparse methods. By considering higher ranks (the next seven points), our method converges to the performance of the full GP (both in SMSE and MSLL), while retaining a computational time comparable to the conventional methods. This type of spatial medium-size GP regression problems can thus be solved in seconds.

Figure 7b, c show interpolation of the precipitation levels using a full GP model and the reduced-rank method (\(m = 1728\)), respectively. The results are practically identical, as is easy to confirm from the color surfaces. Obtaining the reduced-rank result (including initialization and hyperparameter learning) took slightly less than 30 s on a laptop computer (MacBook Air, Late 2010 model, 2.13 GHz, 4 GB RAM), while the full GP inference took approximately 18 minutes.

Open image in new window

6.4 Temperature data on the surface of the globe

We also demonstrate the use of the method in non-Cartesian coordinates. We consider modeling of the spatial mean temperature over a number of \(n = 11{,}028\) locations around the globe.²

As earlier demonstrated in Fig. 2, we use the Laplace operator in spherical coordinates as defined in (31). The eigenfunctions for the angular part are the Laplace’s spherical harmonics. The evaluation of the approximation does not depend on the coordinate system, and thus, all the equations presented in the earlier sections remain valid. We use the squared exponential covariance function and \(m = 1089\) basis functions.

Figure 8 visualizes the modeling outcome. The results are visualized using an interrupted projection (an adaption of the Goode homolosine projection) in order to preserve the length-scale structure across the map. Fig. 8a shows the posterior mean temperature. The uncertainty is visualized in Fig. 8b, which corresponds to the \(n = 11{,}028\) observation locations that are mostly spread over the continents and Western countries (the white areas in Fig. 8b contain no observations). Obtaining the reduced-rank result (including initialization and hyperparameter learning) took approximately 50 s on a laptop computer (MacBook Air, Late 2010 model, 2.13 GHz, 4 GB RAM), which scales with n in comparison to the evaluation time in the previous section.

6.5 Additive modeling of airline delays

In order to fully use the computational benefits and also underline a way of applying the method to high-dimensional inputs, we consider a large dataset for predicting airline delays. The US flight delay prediction example (originally considered by Hensman et al. 2013) is a standard test data set in Gaussian process regression. This is due to the clearly non-stationary behavior and its massive size, with nearly 6 million records.

We aim to replicate and extend to the results previously presented in the work by Hensman et al. (2018) for the Variational Fourier Features (VFF) method. This example has also been used by Deisenroth and Ng (2015), where it was solved using distributed Gaussian processes, and by Samo and Roberts (2016) who use this example for demonstrating the computational efficiency of string Gaussian processes. Adam et al. (2016) used this dataset as an example where the model can be formed by the addition of multiple underlying components.

The data consists of flight arrival and departure times for every commercial flight in the USA for the year 2008. We use the standard eight covariates \(\mathbf {x}\) (see Hensman et al. 2013) which are the age of the aircraft (number of years since deployment), route distance, airtime, departure time, arrival time, day of the week, day of the month, and month. The target is to predict the delay of the aircraft at landing (in minutes), y.

We consider several subset sizes of the data, each selected uniformly at random: \(n = 10{,}000\), 100,000, 1,000,000, and 5,929,413 (all data). In each case, two-thirds of the data is used for training and one third for testing. For each subset size, the training is repeated ten times. The random splits are exactly the same as in Hensman et al. (2018).

Table 1 shows the (normalized) predictive mean squared errors (MSEs) and the negative log predictive densities (NLPDs) with one standard deviation on the airline arrival delays experiment. The table shows that the Hilbert-GP method is directly on par with the variational Fourier features (VFF) method. For the smaller subsets, some variability in the results is visible, even though the MSEs and NLPDs are within one standard deviation of one another for VFF and Hilbert-GP. For the datasets in the millions, VFF and Hilbert-GP perform practically equally well. Further analysis and interpretation of the data and model can be found in Hensman et al. (2018). We have omitted reporting results for the String GP method (Samo and Roberts 2016), the Bayesian committee machine (BCM, Tresp 2000), and the robust Bayesian committee machine (rBCM, Deisenroth and Ng 2015). Each of these performed worse than any of the included methods, and the resulting numbers can be found listed in Hensman et al. (2018) and Samo and Roberts (2016).

6.6 Gaussian process driven Poisson equation

As discussed in Sect. 3.4, our framework also directly extends to inverse problems and latent force models. As this final experiment, we demonstrate the use of the approximation in the latent force model (LFM)

$$\begin{aligned} \begin{aligned} -\,\nabla ^2 g(\mathbf {x})&= f(\mathbf {x}), \\ y_i&= g(\mathbf {x}_i) + \varepsilon _i, \end{aligned} \end{aligned}$$

(84)

where \(\mathbf {x} \in \mathbb {R}^2\) and \(f(\mathbf {x}) \sim \mathcal {GP}(0, k(\mathbf {x}, \mathbf {x}'))\) is the input with a squared exponential covariance function prior. This problem can also be interpreted as a inverse problem where the measurement operator is the Green’s operator \(\mathcal {H} = (-\,\nabla ^2)^{-1}\):

$$\begin{aligned} \begin{aligned} y_i&= (\mathcal {H} f)(\mathbf {x}_i) + \varepsilon _i. \end{aligned} \end{aligned}$$

(85)

If we assume that the boundary conditions of the problem are the same as we used for forming the basis functions in (10), then if we put \(g(\mathbf {x}) \approx \sum _{j=1}^m g_j \, \phi _j(\mathbf {x})\), we get

$$\begin{aligned} \begin{aligned} -\,\nabla ^2 g(\mathbf {x})&\approx -\sum _{j=1}^m g_j \, \nabla ^2 \phi _j(\mathbf {x}) = \sum _{j=1}^m g_j \, \lambda _j \, \phi _j(\mathbf {x}) \end{aligned} \end{aligned}$$

(86)

and thus by further putting \(f(\mathbf {x}) \approx \sum _{j=1}^m f_j \, \phi _j(\mathbf {x})\), the approximation to the equation \(-\,\nabla ^2 g(\mathbf {x}) = f(\mathbf {x})\) becomes

$$\begin{aligned} \begin{aligned} \sum _{j=1}^m g_j \, \lambda _j \, \phi _j(\mathbf {x}) = \sum _{j=1}^m f_j \, \phi _j(\mathbf {x}) \end{aligned} \end{aligned}$$

(87)

which allows us to solve \(f_j = g_j / \lambda _j\). This implies that we approximately have \((-\,\nabla ^2)^{-1} \phi _j = \phi _j / \lambda _j\) which reduces Eq. (52) to

$$\begin{aligned} \begin{aligned} (\mathcal {H} \, \mathcal {H}' \, k)(\mathbf {x},\mathbf {x}')&\approx \sum _j \lambda _j^{-2} \, S(\sqrt{\lambda _j}) \, \phi _j(\mathbf {x}) \, \phi _j(\mathbf {x}'), \\ (\mathcal {H}' k(\mathbf {x}_*,\cdot ))(\mathbf {x}')&\approx \sum _j \lambda _j^{-1} \, S(\sqrt{\lambda _j}) \, \phi _j(\mathbf {x}_*) \, \phi _j(\mathbf {x}'), \end{aligned} \end{aligned}$$

(88)

after which we can proceed with (51). Alternatively we can directly use (53) with \(\tilde{\varvec{\Phi }}_{ij} = \phi _j(\mathbf {x}_i) / \lambda _j\).

Open image in new window

Figure 9 shows the result of applying the proposed method to this model with the input function shown in Fig. 9b. The true solution and the simulated measurements (with standard deviation of 1 / 10) are shown in Fig. 9a. The scale \(\sigma ^2\) and length scale \(\ell \) of the SE covariance function were estimated by maximum likelihood method, and the number of basis functions used for solving the GP regression problem was 100 (for simulation we used 255 basis functions). The estimates of the input and the solution function are shown in Fig. 9a, b, respectively. As can be seen in the figures, the estimate of the solution is very good, as can be expected from the fact that we obtain direct (although noisy) measurements from it. The estimate of the input is less accurate, but still approximates the true input well.