Parallel inference for massive distributed spatial data using low-rank models

Abstract

Due to rapid data growth, statistical analysis of massive datasets often has to be carried out in a distributed fashion, either because several datasets stored in separate physical locations are all relevant to a given problem, or simply to achieve faster (parallel) computation through a divide-and-conquer scheme. In both cases, the challenge is to obtain valid inference that does not require processing all data at a single central computing node. We show that for a very widely used class of spatial low-rank models, which can be written as a linear combination of spatial basis functions plus a fine-scale-variation component, parallel spatial inference and prediction for massive distributed data can be carried out exactly, meaning that the results are the same as for a traditional, non-distributed analysis. The communication cost of our distributed algorithms does not depend on the number of data points. After extending our results to the spatio-temporal case, we illustrate our methodology by carrying out distributed spatio-temporal particle filtering inference on total precipitable water measured by three different satellite sensor systems.

Appendices

Appendix 1: Derivation of the likelihood

We derive here the expression of the likelihood in (5). First, note that $\mathbf {z}_{1:J} | {\varvec{\theta }}\sim N_n(\mathbf {B}_{1:J}{\varvec{\nu }}_0,{\varvec{\Sigma }}_{1:J})$, where ${\varvec{\Sigma }}_{1:J} = \mathbf {B}_{1:J} \mathbf {K}_0 \mathbf {B}_{1:J}' + \mathbf {V}_{1:J}$. Hence, the likelihood is given by,

$$\begin{aligned} -2 \log [\mathbf {z}_{1:J} | {\varvec{\theta }}]&= \log |{\varvec{\Sigma }}_{1:J}| \\&\quad + (\mathbf {z}_{1:J} - \mathbf {B}_{1:J}{\varvec{\nu }}_0)'{\varvec{\Sigma }}_{1:J}^{-1}(\mathbf {z}_{1:J} - \mathbf {B}_{1:J}{\varvec{\nu }}_0)\\&\quad -(n/2)\log (2\pi ). \end{aligned}$$

Applying a matrix determinant lemma (e.g., Harville 1997 Thm. 18.1.1), we can write the log determinant as,

$$\begin{aligned} \log |{\varvec{\Sigma }}_{1:J}|&= \log |\mathbf {V}_{1:J}| + \log |\mathbf {K}_0| \\&\quad + \log | \mathbf {B}_{1:J}'\mathbf {V}_{1:J}^{-1} \mathbf {B}_{1:J} + \mathbf {K}_0^{-1} | \\&= \textstyle \sum _{j=1}^J \log |\mathbf {V}_j| - \log |\mathbf {K}_0^{-1}| + \log |\mathbf {K}_z^{-1}|. \end{aligned}$$

Further, using the Sherman-Morrison-Woodbury formula (Sherman and Morrison 1950; Woodbury 1950; Henderson and Searle 1981), we can show that ${\varvec{\Sigma }}_{1:J}^{-1} = \mathbf {V}_{1:J}^{-1} - \mathbf {V}_{1:J}^{-1} \mathbf {B}_{1:J} \mathbf {K}_z \mathbf {B}_{1:J}' \mathbf {V}_{1:J}^{-1}$, and so

$$\begin{aligned}&(\mathbf {z}_{1:J} - \mathbf {B}_{1:J}{\varvec{\nu }}_0)'{\varvec{\Sigma }}_{1:J}^{-1}(\mathbf {z}_{1:J} - \mathbf {B}_{1:J}{\varvec{\nu }}_0) \\&\quad = \textstyle \sum _{j=1}^J (\mathbf {z}_j - \mathbf {B}_j{\varvec{\nu }}_0)'\mathbf {V}_j^{-1}(\mathbf {z}_j - \mathbf {B}_j{\varvec{\nu }}_0) \\&\qquad - \big (\sum _{j=1}^J \mathbf {B}_j'\mathbf {V}_j^{-1}(\mathbf {z}_j-\mathbf {B}_j{\varvec{\nu }}_0)\big )'\mathbf {K}_z\\&\qquad \times \textstyle \big (\sum _{j=1}^J \mathbf {B}_j'\mathbf {V}_j^{-1}(\mathbf {z}_j-\mathbf {B}_j{\varvec{\nu }}_0)\big )\\&\quad = \sum _j \mathbf {z}_j'\mathbf {V}_j^{-1} \mathbf {z}_j -2 {\varvec{\nu }}_0'(\mathbf {K}_z^{-1}{\varvec{\nu }}_z-\mathbf {K}_0^{-1}{\varvec{\nu }}_0) \\&\qquad + {\varvec{\nu }}_0'(\mathbf {K}_z^{-1}- \mathbf {K}_0^{-1}) {\varvec{\nu }}_0 \\&\qquad - \big ( (\mathbf {K}_z^{-1}{\varvec{\nu }}_z-\mathbf {K}_0^{-1}{\varvec{\nu }}_0)-(\mathbf {K}_z^{-1}-\mathbf {K}_0^{-1}){\varvec{\nu }}_0\big )'\mathbf {K}_z\\&\qquad \times \big ( (\mathbf {K}_z^{-1}{\varvec{\nu }}_z-\mathbf {K}_0^{-1}{\varvec{\nu }}_0)-(\mathbf {K}_z^{-1}-\mathbf {K}_0^{-1}){\varvec{\nu }}_0\big )\\&= \sum _j \mathbf {z}_j'\mathbf {V}_j^{-1} \mathbf {z}_j - 2{\varvec{\nu }}_0'\mathbf {K}_z^{-1}{\varvec{\nu }}_z + {\varvec{\nu }}_0'\mathbf {K}_0^{-1}{\varvec{\nu }}_0 + {\varvec{\nu }}_0'\mathbf {K}_z^{-1}{\varvec{\nu }}_0 \\&\qquad - (\mathbf {K}_z^{-1}{\varvec{\nu }}_z)'\mathbf {K}_z(\mathbf {K}_z^{-1}{\varvec{\nu }}_z)-{\varvec{\nu }}_0'\mathbf {K}_z^{-1}\mathbf {K}_z\mathbf {K}_z^{-1}{\varvec{\nu }}_0 \\&\qquad + 2(\mathbf {K}_z^{-1}{\varvec{\nu }}_z)'\mathbf {K}_z \mathbf {K}_z^{-1} {\varvec{\nu }}_0\\&\quad = \textstyle \sum _j \mathbf {z}_j'\mathbf {V}_j^{-1} \mathbf {z}_j + {\varvec{\nu }}_0'\mathbf {K}_0^{-1}{\varvec{\nu }}_0 - {\varvec{\nu }}_z'\mathbf {K}_z^{-1}{\varvec{\nu }}_z, \end{aligned}$$

where $\sum _{j=1}^J \mathbf {B}_j'\mathbf {V}_j^{-1}\mathbf {B}_j = \mathbf {K}_z^{-1}-\mathbf {K}_0^{-1}$ and $\sum _{j=1}^J \mathbf {B}_j'\mathbf {V}_j^{-1}\mathbf {z}_j = \mathbf {K}_z^{-1}{\varvec{\nu }}_z-\mathbf {K}_0^{-1}{\varvec{\nu }}_0$ both follow from (3).

Appendix 2: Spatial prediction when observed and prediction locations coincide

Here we describe how to do spatial prediction when a small number, q say, of the observed locations are also in the set of desired prediction locations. Define ${\varvec{\delta }}_{P,O}$ to be the vector of the first q elements of ${\varvec{\delta }}^P$, which we assume to correspond to the q observed prediction locations, and let $\mathbf {P}_j$ be a sparse $n_j \times q$ matrix with $(\mathbf {P}_j)_{k,l} = I(\mathbf {s}_{j,k} = \mathbf {s}_l^P)$. We write our model in state-space form with identity evolution equation, $\mathbf {z}_j = \tilde{\mathbf {B}}_j \tilde{{\varvec{\eta }}} + \tilde{{\varvec{\xi }}}_j$, where $\tilde{\mathbf {B}}_j {:}{=}(\mathbf {B}_j, \mathbf {P}_j)$, $\tilde{{\varvec{\eta }}} {:}{=}({\varvec{\eta }}', {\varvec{\delta }}_{P,O}')' \sim N(\tilde{{\varvec{\nu }}}_0,\tilde{\mathbf {K}}_0)$, $\tilde{{\varvec{\nu }}}_0 {:}{=}({\varvec{\nu }}_0', \mathbf {0}_q')'$, $\tilde{\mathbf {K}}_0$ is blockdiagonal with first block $\mathbf {K}_0$ and second block $diag\{ v_\delta (\mathbf {s}^P_1),\ldots ,v_\delta (\mathbf {s}^P_q)\}$, $\tilde{{\varvec{\xi }}}_j \sim N_{n_j}(\mathbf {0},\tilde{\mathbf {V}}_j)$, and $\tilde{\mathbf {V}}_j$ is the same as $\mathbf {V}_j$ except that the ith diagonal element is now $v_\epsilon (\mathbf {s}_{j,i})$ if $\mathbf {s}_{j,i}$ is one of the prediction locations.

The decentralized Kalman filter (Rao et al. 1993) gives $\tilde{\mathbf {K}}_z^{-1} = \tilde{\mathbf {K}}_{0}^{-1} + \textstyle \sum _{j=1}^J \tilde{\mathbf {R}}_j$ and $\tilde{{\varvec{\nu }}}_z = \tilde{\mathbf {K}}_z(\tilde{\mathbf {K}}_0^{-1} \tilde{{\varvec{\nu }}}_0 + \textstyle \sum _{j=1}^J \tilde{{\varvec{\gamma }}}_j)$, where $\tilde{\mathbf {R}}_j {:}{=}\tilde{\mathbf {B}}_j'\tilde{\mathbf {V}}_j^{-1} \tilde{\mathbf {B}}_j$ and $\tilde{{\varvec{\gamma }}}_j {:}{=}\tilde{\mathbf {B}}_j'\tilde{\mathbf {V}}_j^{-1}\mathbf {z}_j$ are the only quantities that need to be calculated at and transfered from server j, which is feasible due to sparsity if q is not too large. The predictive distribution is then given by $\mathbf {y}^P | \mathbf {z}_{1:J} \sim N( \tilde{\mathbf {B}}^P \tilde{{\varvec{\nu }}}_J, \tilde{\mathbf {B}}^P \tilde{\mathbf {K}}_z \tilde{\mathbf {B}}^P{}' + \tilde{\mathbf {V}}_\delta ^P )$, where $\tilde{\mathbf {B}}^P {:}{=}(\mathbf {B}^P, (\mathbf {I}_q, \mathbf {0})')$ and $\tilde{\mathbf {V}}_\delta ^P {:}{=}diag\{\mathbf {0}_q', v_\delta (\mathbf {s}^P_{q+1}),\ldots ,v_\delta (\mathbf {s}^P_{n_P})\}$.