Multi-Scale Vecchia Approximations of Gaussian Processes

Abstract

Gaussian processes (GPs) are popular models for functions, time series, and spatial fields, but direct application of GPs is computationally infeasible for large datasets. We propose a multi-scale Vecchia (MSV) approximation of GPs for modeling and analysis of multi-scale phenomena, which are ubiquitous in geophysical and other applications. In the MSV approach, increasingly large sets of variables capture increasingly small scales of spatial variation, to obtain an accurate approximation of the spatial dependence from very large to very fine scales. For a given set of observations, the MSV approach decomposes the data into different scales, which can be visualized to obtain insights into the underlying processes. We explore properties of the MSV approximation and propose an algorithm for automatic choice of the tuning parameters. We provide comparisons to existing approaches based on simulated data and using satellite measurements of land-surface temperature.

Appendices

Computing $\mathbf {U}$

Extending the derivations in Section 2.4.1, the sparse upper triangular matrix $\mathbf {U}$ can be specified by the following rules:

(1) For each $\ell = 1,2,..., L-1$, denote $\mathbf {U}^{(\ell )}$ as the block of $\mathbf {U}$ corresponding to level $\ell$ with size $n_\ell \times n_\ell$. For each $i=1,2,...,n_\ell$,

$$\begin{aligned} \mathbf {U}^{(\ell )}_{ii} = (D_i^{(\ell )} )^{-1/2}. \end{aligned}$$

For the conditioning set of $y^{(\ell )}_i$, suppose the s-th element in its conditioning set is $y^{(\ell )}_{i'}$, then

$$\begin{aligned} \mathbf {U}^{(\ell )}_{i'i} = -\{B_i^{(\ell )}\}^s (D_i^{(\ell )} )^{-1/2}, \end{aligned}$$

where $\{B_i^{(\ell )}\}^s$ is the s-th element of $B_i^{(\ell )}$.

(2) For the data level L, first denote an $n \times n$ diagonal matrix by

$$\begin{aligned} \mathbf {U}^{(L)(L)}=diag\left( (D_1^{(L)} )^{-1/2},(D_2^{(L)} )^{-1/2},..., (D_n^{(L)} )^{-1/2}\right) . \end{aligned}$$

Next, for $\ell =1,2,..., L-1$, denote an $n_\ell \times n$ matrix $\mathbf {U}^{(L)(\ell )}$ as the block of $\mathbf {U}$ corresponding to $\{ N_{z_i}^{(\ell )}, i=1,2,...,n\}$. Then, for each i, suppose the s-th element in $N_{z_i}^{(\ell )}$ is $y_{i'}^{(\ell )}$, then

$$\begin{aligned} \mathbf {U}^{(L)(\ell )}_{i'i} = -\{B_i^{(L)}\}^s (D_i^{(L)} )^{-1/2}. \end{aligned}$$

(3) Finally, the matrix $\mathbf {U}$ is

All unmentioned entries in $\mathbf {U}$ are 0.

Prediction at unobserved locations

For simplicity in the proof, denote $\mathcal {S}$ as the locations corresponding to the knot set at level $\ell$. First, we show $\left( C^{(\ell )}(\mathcal {S}, \mathcal {S}) \right) ^{-1} =\mathbf {U}^{(\ell )}\mathbf {U}^{(\ell )\top }$. When $\ell =1$, denote $\mathbf {U}= \left( \begin{array}{cc} \mathbf {U}_1 &{} \mathbf {U}_2 \\ \mathbf{0}&{} \mathbf {U}_3 \end{array} \right)$, where $\mathbf {U}_1= \mathbf {U}^{(1)}$ is the block of $\mathbf {U}$ corresponding to knot variables at level 1. Then,

$$\begin{aligned} \hat{\mathbf {C}}^{-1}=\mathbf {U} \mathbf {U}^ \top =\left( \begin{array}{cc} \mathbf {U}_1 &{} \mathbf {U}_2 \\ \mathbf{0}&{} \mathbf {U}_3 \end{array} \right) \left( \begin{array}{cc} \mathbf {U}_1^\top &{} \mathbf{0}\\ \mathbf {U}_2^\top &{} \mathbf {U}_3^\top \end{array} \right) = \left( \begin{array}{cc} \mathbf {U}_1 \mathbf {U}_1^\top +\mathbf {U}_2 \mathbf {U}_2^\top &{} \mathbf {U}_2\mathbf {U}_3^\top \\ \mathbf {U}_3\mathbf {U}_2^\top &{} \mathbf {U}_3\mathbf {U}_3^\top \end{array} \right) . \end{aligned}$$

Since we can also write $\hat{\mathbf {C}}^{-1}$ as $\hat{\mathbf {C}}^{-1}=\left( \begin{array}{cc} C^{(1)}(\mathcal {S}, \mathcal {S}) &{} A \\ A^\top &{} B \end{array} \right) ^{-1}=\left( \begin{array}{cc} E &{} F \\ F^\top &{} G \end{array} \right)$, by the property of matrix inverse in block form, $C^{(1)}(\mathcal {S}, \mathcal {S})^{-1}=E-FG^{-1}F^\top$. Thus we have

$$\begin{aligned} C^{(1)}(\mathcal {S}, \mathcal {S})^{-1}=\mathbf {U}_1 \mathbf {U}_1^\top + \mathbf {U}_2 \mathbf {U}_2^\top -\mathbf {U}_2\mathbf {U}_3^\top (\mathbf {U}_3\mathbf {U}_3^\top )^{-1}\mathbf {U}_3\mathbf {U}_2^\top =\mathbf {U}_1 \mathbf {U}_1^\top = \mathbf {U}^{(1)}\mathbf {U}^{(1)\top }. \end{aligned}$$

For any $\ell >1$, similar results hold: $C^{(\ell )}(\mathcal {S}, \mathcal {S})^{-1}= \mathbf {U}^{(\ell )}\mathbf {U}^{(\ell )\top }.$

Using Algorithm 2 and achieving a KL divergence of (almost) zero, the MSV approximation based on the knot set $\mathbf {y}_\ell$ at level $\ell$ is (almost) exact, and so we assume that all information about the process at level $\ell$ is captured by knot set $\mathbf {y}_\ell$. Then, the posterior mean in (5) at an unobserved location $\mathbf {s}_0$ can be computed as

$$\begin{aligned} E(y^{(\ell )}(\mathbf {s}_0) |\mathbf {z}) =&E\left( E(y^{(\ell )}(\mathbf {s}_0) |\mathbf {y}_\ell ,\mathbf {z}) |\mathbf {z}\right) = E\left( E(y^{(\ell )}(\mathbf {s}_0) |\mathbf {y}_\ell ) |\mathbf {z}\right) \\ =&C^{(\ell )}(\mathbf {s}_0, \mathcal {S}) \left( C^{(\ell )}(\mathcal {S}, \mathcal {S}) \right) ^{-1} E(\mathbf {y}_\ell |\mathbf {z})\\ =&C^{(\ell )}(\mathbf {s}_0, \mathcal {S}) \mathbf {U}^{(\ell )}\mathbf {U}^{(\ell )\top } \mathbf {H}_{\ell }\varvec{\mu }. \end{aligned}$$

The posterior variance in (6) can be computed as

$$\begin{aligned}&var(y^{(\ell )}(\mathbf {s}_0) |\mathbf {z}) \nonumber \\ =&E\left( var(y^{(\ell )}(\mathbf {s}_0) |\mathbf {y}_\ell ,\mathbf {z}) |\mathbf {z}\right) + var\left( E(y^{(\ell )}(\mathbf {s}_0) |\mathbf {y}_\ell ,\mathbf {z}) |\mathbf {z}\right) \nonumber \\ =&E\left( var(y^{(\ell )}(\mathbf {s}_0) |\mathbf {y}_\ell ) |\mathbf {z}\right) + var\left( E(y^{(\ell )}(\mathbf {s}_0) |\mathbf {y}_\ell ) |\mathbf {z}\right) \nonumber \\ =&E \left( C^{(\ell )}(\mathbf {s}_0, \mathbf {s}_0)- C^{(\ell )}(\mathbf {s}_0, \mathcal {S}) \left( C^{(\ell )}(\mathcal {S}, \mathcal {S}) \right) ^{-1} C^{(\ell )}(\mathcal {S},\mathbf {s}_0) |\mathbf {z}\right) \nonumber \\&+ var\left( C^{(\ell )}(\mathbf {s}_0, \mathcal {S}) \left( C^{(\ell )}(\mathcal {S}, \mathcal {S}) \right) ^{-1} \mathbf {y}_\ell |\mathbf {z}\right) \nonumber \\ =&\, C^{(\ell )}(\mathbf {s}_0, \mathbf {s}_0)- C^{(\ell )}(\mathbf {s}_0, \mathcal {S}) \left( C^{(\ell )}(\mathcal {S}, \mathcal {S}) \right) ^{-1} C^{(\ell )}(\mathcal {S},\mathbf {s}_0)\nonumber \\&+ C^{(\ell )}(\mathbf {s}_0, \mathcal {S}) \left( C^{(\ell )}(\mathcal {S}, \mathcal {S}) \right) ^{-1} var(\mathbf {y}_\ell |\mathbf {z}) \left( C^{(\ell )}(\mathcal {S}, \mathcal {S}) \right) ^{-1} C^{(\ell )}(\mathbf {s}_0, \mathcal {S}) ^\top \nonumber \\ =&\,C^{(\ell )}(\mathbf {s}_0, \mathbf {s}_0)- C^{(\ell )}(\mathbf {s}_0, \mathcal {S}) \mathbf {U}^{(\ell )}\mathbf {U}^{(\ell )\top } C^{(\ell )}(\mathcal {S},\mathbf {s}_0)\nonumber \\&+ C^{(\ell )}(\mathbf {s}_0, \mathcal {S}) \mathbf {U}^{(\ell )}\mathbf {U}^{(\ell )\top } \varvec{\Sigma }_{\mathbf {H}_{\ell }} \mathbf {U}^{(\ell )}\mathbf {U}^{(\ell )\top } C^{(\ell )}(\mathbf {s}_0, \mathcal {S}) ^\top . \end{aligned}$$

(7)

The posterior predictive variance for the entire latent process is

$$\begin{aligned}&var(y^{(1)}(\mathbf {s}_0)+y^{(2)}(\mathbf {s}_0) |\mathbf {z}) \\&\quad = E\left( var \left( y^{(1)}(\mathbf {s}_0)+y^{(2)}(\mathbf {s}_0) |\mathbf {y}_1,\mathbf {y}_2,\mathbf {z}\right) |\mathbf {z}\right) + var\left( E \left( y^{(1)}(\mathbf {s}_0)+y^{(2)}(\mathbf {s}_0) |\mathbf {y}_1,\mathbf {y}_2,\mathbf {z}\right) |\mathbf {z}\right) \\&\quad = E\left( var \left( y^{(1)}(\mathbf {s}_0)+y^{(2)}(\mathbf {s}_0) |\mathbf {y}_1,\mathbf {y}_2 \right) |\mathbf {z}\right) + var\left( E \left( y^{(1)}(\mathbf {s}_0)+y^{(2)}(\mathbf {s}_0) |\mathbf {y}_1,\mathbf {y}_2 \right) |\mathbf {z}\right) \\&\quad = E\left( \left( var(y^{(1)}(\mathbf {s}_0)|\mathbf {y}_1) + var(y^{(2)}(\mathbf {s}_0) |\mathbf {y}_2)\right) |\mathbf {z}\right) + var\left( \left( E(y^{(1)}(\mathbf {s}_0)|\mathbf {y}_1)+ E(y^{(2)}(\mathbf {s}_0) |\mathbf {y}_2) \right) |\mathbf {z}\right) \\&\quad = E\left( var(y^{(1)}(\mathbf {s}_0)|\mathbf {y}_1) |\mathbf {z}\right) + E \left( var(y^{(2)}(\mathbf {s}_0) |\mathbf {y}_2) |\mathbf {z}\right) \\&\qquad + var\left( C^{(1)}(\mathbf {s}_0, \mathcal {S}_1) \left( C^{(1)}(\mathcal {S}_1, \mathcal {S}_1) \right) ^{-1} \mathbf {y}_1 + C^{(2)}(\mathbf {s}_0, \mathcal {S}_2) \left( C^{(2)}(\mathcal {S}_2, \mathcal {S}_2) \right) ^{-1} \mathbf {y}_2|\mathbf {z}\right) \end{aligned}$$

The first two terms can be computed similar to (7). The last term is a linear combination of $\mathbf {y}_{1:L-1}$, and so it can be calculated by $var (\mathbf {H}\mathbf {y}_{1:L-1}|z) = (\mathbf {V}^{-1} \mathbf {H}^\top )^\top (\mathbf {V}^{-1} \mathbf {H}^\top )$.

Proofs

Proposition 1

For a polynomial $y^{(\ell )}(\mathbf {s}) = \mathbf {p}(\mathbf {s})^\top \varvec{\beta }$ as a function of spatial location $\mathbf {s}$ with p coefficients $\varvec{\beta } \sim \mathcal {N}_p(\mathbf {0},\varvec{\Sigma }_\beta )$, the corresponding covariance function $C^{(\ell )}(\mathbf {s}_i, \mathbf {s}_j) = \mathbf {p}(\mathbf {s}_i)^\top \varvec{\Sigma }_\beta \mathbf {p}(\mathbf {s}_j)$ can be captured exactly by setting the knot and conditioning set to be any distinct p locations.

Proof of Proposition 1

Denote any p distinct locations as $\{\mathbf {s}_1, \mathbf {s}_2,..., \mathbf {s}_p\}$. For polynomial $y(\mathbf {s}) = \mathbf {p}(\mathbf {s})^\top \varvec{\beta }$ with $\varvec{\beta } \sim \mathcal {N}_p(\mathbf {0},\varvec{\Sigma }_\beta )$, the system of equations $\{\mathbf {p}(\mathbf {s}_1)^\top \varvec{\beta } = y(\mathbf {s}_1) , \mathbf {p}(\mathbf {s}_2)^\top \varvec{\beta } = y(\mathbf {s}_2) ,..., \mathbf {p}(\mathbf {s}_p)^\top \varvec{\beta } = y(\mathbf {s}_p) , \mathbf {p}(\mathbf {s})^\top \varvec{\beta } = y(\mathbf {s}) \}$ is equivalent to the system of equations $\{\mathbf {p}(\mathbf {s}_1)^\top \varvec{\beta } = y(\mathbf {s}_1) , \mathbf {p}(\mathbf {s}_2)^\top \varvec{\beta } = y(\mathbf {s}_2) ,..., \mathbf {p}(\mathbf {s}_p)^\top \varvec{\beta } = y(\mathbf {s}_p) \}$, thus $P \left( y(\mathbf {s}) | y(\mathbf {s}_1), y(\mathbf {s}_2),...,y(\mathbf {s}_p) \right) = 1$. Then, the exact distribution for $\mathbf {y}=(y(\mathbf {s}_1), y(\mathbf {s}_2), ..., y(\mathbf {s}_n))$ can be written as

$$\begin{aligned} f(\mathbf {y})=&\prod _{i=1}^{n} f(y(\mathbf {s}_i)| y(\mathbf {s}_{h_i}))\\ =&f(y(\mathbf {s}_1)) f(y(\mathbf {s}_2)|y(\mathbf {s}_1)) f(y(\mathbf {s}_3)|y(\mathbf {s}_1),y(\mathbf {s}_2)) \cdots f(y(\mathbf {s}_p)|y(\mathbf {s}_1),y(\mathbf {s}_2), ...,y(\mathbf {s}_{p-1}) ) \cdot \\&\prod _{i=p+1}^{n} f(y(\mathbf {s}_i)| y(\mathbf {s}_1), y(\mathbf {s}_2),..., y(\mathbf {s}_p)), \end{aligned}$$

which equals $\hat{f}(\mathbf {y})$ in Vecchia by setting the knot and conditioning set to be $\{\mathbf {s}_1, \mathbf {s}_2, ...,\mathbf {s}_p\}$. Thus, the covariance can be captured exactly. $\square$

Proof of Theorem 1

The following proof is related to Guinness (2018, Thm. 1). Suppose the true covariance is $\Sigma _0$ and the approximated covariance is $\hat{\Sigma }$. At each level $\ell$, the KL divergence between the two normal distributions can be written as $KL \left( f(\mathbf {y}^{(\ell )}) || \hat{f}(\mathbf {y}^{(\ell )}) \right) =\frac{1}{2} E \left( -(\mathbf {y}^{(\ell )})^\top \Sigma _0^{-1}\mathbf {y}^{(\ell )} \right) +\frac{1}{2} E \left( (\mathbf {y}^{(\ell )})^\top \hat{\Sigma }^{-1}\mathbf {y}^{(\ell )}\right) +\frac{1}{2} \log \frac{ |\hat{\Sigma } |}{|\Sigma _0 |}$. Since $\Sigma _0$ is the true covariance, the first term $E \left( -(\mathbf {y}^{(\ell )})^\top \Sigma _0^{-1}\mathbf {y}^{(\ell )} \right) = -n$. Based on MSV, we have $\log |\hat{\Sigma } |=\sum _{i=1}^{n} \log D_i^{(\ell )}$. Suppose $L_0$ is the Cholesky factor of $\Sigma _0$, then $E \left( (\mathbf {y}^{(\ell )})^\top \hat{\Sigma }^{-1}\mathbf {y}^{(\ell )}\right) =tr(UU^T \Sigma _0)= \sum _{i,j} (L_0^T U)_{ij}^2 =n$. Thus, the KL divergence can be written as $KL \left( f(\mathbf {y}^{(\ell )}) || \hat{f}(\mathbf {y}^{(\ell )}) \right) =\frac{1}{2} \left( -n+n+ \sum _{i=1}^{n} \log D_i^{(\ell )}-\log |\Sigma _0 |\right) =\frac{1}{2} \sum _{i=1}^{n} \log D_i^{(\ell )}-constant$. $\square$