Bayesian non-parametric modeling for integro-difference equations

Abstract

Integro-difference equations (IDEs) provide a flexible framework for dynamic modeling of spatio-temporal data. The choice of kernel in an IDE model relates directly to the underlying physical process modeled, and it can affect model fit and predictive accuracy. We introduce Bayesian non-parametric methods to the IDE literature as a means to allow flexibility in modeling the kernel. We propose a mixture of normal distributions for the IDE kernel, built from a spatial Dirichlet process for the mixing distribution, which can model kernels with shapes that change with location. This allows the IDE model to capture non-stationarity with respect to location and to reflect a changing physical process across the domain. We address computational concerns for inference that leverage the use of Hermite polynomials as a basis for the representation of the process and the IDE kernel, and incorporate Hamiltonian Markov chain Monte Carlo steps in the posterior simulation method. An example with synthetic data demonstrates that the model can successfully capture location-dependent dynamics. Moreover, using a data set of ozone pressure, we show that the spatial Dirichlet process mixture model outperforms several alternative models for the IDE kernel, including the state of the art in the IDE literature, that is, a Gaussian kernel with location-dependent parameters.

Appendices

Appendix 1: Stationarity of IDE processes

Brown et al. (2000) show that IDE models are stationary in space when the IDE kernel has parameters which are not spatially varying. The following lemma handles the more general case with spatially varying IDE kernels.

Lemma 1

Consider an IDE process, $X_t(s)$, with a stationary initial process $X_0(s)$, and with a kernel that belongs to a location family of distributions. The covariance function of the process is non-stationary with respect to location for all $t>0$, when the kernel parameters depend on the location s.

Proof

Assume $\hbox {Cov}[X_{t-1}(s),X_{t-1}(r)]=\rho (|s-r|)$, a stationary covariance function, and that the error process $\omega _t(s)$ is also stationary with covariance function $\gamma (|s-r|)$. Then,

$$\begin{aligned}&\hbox {Cov}[X_t(s),X_t(s+r)] \\&\quad = \hbox {Cov}\left[ \int k(u \mid s,\varvec{\theta }_s)X_{t-1}(u) \hbox {d}u \right. \\&\qquad \left. + \,\,\omega _t(s), \int k(v \mid s+r,\varvec{\theta }_{s+r})X_{t-1}(v) \hbox {d}v+ \omega _t(s+r)\right] \\&\quad = \int \int k(u \mid s,\varvec{\theta }_s) k(v \mid s{+}\,r,\varvec{\theta }_{s+r})\rho (|u{-}v|) \hbox {d}u\hbox {d}v {+} \gamma (r). \end{aligned}$$

Using the transformations $\eta = u-v$ and $w=v-s$, and the assumption of a kernel that belongs to a location family, i.e., $k(u \mid s,\varvec{\theta }_s)=$ $k(u-s \mid \varvec{\theta }_s)$, the covariance function can be written as

$$\begin{aligned}&\hbox {Cov}[X_t(s),X_t(s+r)] \\&\quad = \int \int k(\eta {+}\,w \mid \varvec{\theta }_{s}) k(w-r \mid \varvec{\theta }_{s+r})\rho (|\eta |) \hbox {d}\eta \hbox {d}w + \gamma (r). \end{aligned}$$

The covariance is a function of s and r, which implies non-stationarity in space. $\square$

When the parameter vector does not depend on the location, i.e., $\varvec{\theta }_s=\varvec{\theta }$, the location s disappears from the covariance. This is consistent with the previous work which shows that Gaussian kernel IDE models are stationary when the parameters do not change with location.

Appendix 2: Hamiltonian Markov chain Monte Carlo

Hamiltonian Markov chain Monte Carlo (HMCMC) involves taking the derivative with respect to unknown parameters of the negative log of the target function, which in this case is the posterior (Neal 2011). Letting $\varvec{W} = \tau ^2 \varvec{G} \varvec{V} \varvec{G}$, the negative log of the relevant parts of the posterior is

$$\begin{aligned} -l(\varvec{\theta })= & {} -\log (p(\varvec{\theta })) \\&{+} \frac{1}{2}\sum _{t=1}^T \left( \varvec{a}_t{-}\varvec{G}' \varvec{B}_\theta \varvec{a}_{t-1}\right) '\varvec{W}^{-1}\left( \varvec{a}_t{-}\varvec{G}' \varvec{B}_\theta \varvec{a}_{t-1}\right) , \end{aligned}$$

which can be expanded out as

$$\begin{aligned} -l(\varvec{\theta })= & {} -\log (p(\varvec{\theta })) + \frac{1}{2}\sum _{t=1}^T \left( \varvec{a}_t'\varvec{W}^{-1}\varvec{a}_t \right. \\&\left. {-}2\varvec{a}_t\varvec{W}^{-1}\varvec{G}'\varvec{B}_\theta \varvec{a}_{t-1}{+}\,\varvec{a}_{t-1}'\varvec{B}_\theta '\varvec{G} \varvec{W}^{-1} \varvec{G}' \varvec{B}_\theta \varvec{a}_{t-1}\right) . \end{aligned}$$

Using matrix calculus, the derivative is

$$\begin{aligned}&\frac{\partial (-l(\varvec{\theta }))}{\partial \theta _i} = -\frac{\partial (-\log (p(\varvec{\theta })))}{\partial \theta _i}\\&\quad +\frac{1}{2}\sum _{t=1}^T -2\varvec{a}_t\varvec{W}^{-1}\varvec{G}'\frac{\partial \varvec{B}_\theta }{\partial \theta _i}\varvec{a}_{t-1}\\&\quad +\,2\ tr\left( \varvec{a}_{t-1}\varvec{a}_{t-1}'\varvec{B}_\theta \varvec{G} \varvec{W}^{-1} \varvec{G}'\frac{\partial \varvec{B}_\theta }{\partial \theta _i}\right) , \end{aligned}$$

where $\frac{\partial \varvec{B}_\theta }{\partial \theta _i}$ is a element-wise derivative of $\varvec{B}_\theta$ with respect to $\theta _i$, and $\theta _i$ is the i-th element of the parameter vector, $\varvec{\theta }$.

Let $E(\varvec{\theta })$ refer to the negative log posterior. A step size $\epsilon$ and number of iterations, L, must be defined prior to the algorithm. Latent variables, $p_i$, are introduced for each parameter as independent normal variables with zero mean and variance $M_i$. Then, one “leapfrog” step given current iteration $(\varvec{\theta }^b, {{\mathbf {p}}}^b)$ is

$$\begin{aligned}&{{\mathbf {p}}}^{\left( b+\epsilon /2\right) } = {{\mathbf {p}}}^b - \frac{\epsilon }{2} \frac{\partial E}{\partial \varvec{\theta }}\left( \varvec{\theta }^{\left( b\right) }\right) \\&\varvec{\theta }^{\left( b+\epsilon \right) } = \theta ^b+\epsilon \frac{{{\mathbf {p}}}^{\left( b+\epsilon /2\right) }}{{{\mathbf {m}}}} \\&{{\mathbf {p}}}^{\left( b+\epsilon \right) } = {{\mathbf {p}}}^{\left( b+\epsilon /2\right) }-\frac{\epsilon }{2}\frac{\partial E}{\partial \varvec{\theta }}\left( \varvec{\theta }^{\left( b+\epsilon \right) }\right) \end{aligned}$$

The parameters leapfrog L times ending at new proposals for the posterior. The function $H(\varvec{\theta },p)$ is defined to be the sum of $E(\varvec{\theta })$ and $K(p) = \frac{1}{2}\sum \frac{p_i^2}{M_i}$. The new values $(\varvec{\theta }^{(b+1)}, {{\mathbf {p}}}^{(b+1)})$ are accepted with probability $\min (1,\exp \left( H(\varvec{\theta }^{(b+1)}, {{\mathbf {p}}}^{(b+1)})-H(\varvec{\theta }^b, {{\mathbf {p}}}^b)\right)$. If the new value is rejected, it is set to the previous values. The method must be tuned to accept and reject at reasonable rates, perhaps accepting between 40 and 60% of proposed samples. Both L and $\epsilon$ can be tuned, where $L \times \epsilon$ is closely associated with acceptance rates.

For a normal distribution, the derivative $\frac{\partial B_\theta }{\partial \theta _i}$ is found by taking element-wise derivatives of the coefficients found in Eq. (6). The required derivative with respect to the mean parameter using Hermite basis functions is

$$\begin{aligned} \frac{\partial b_n}{\partial \mu }= & {} \frac{1}{\sigma ^2}\frac{1}{\sqrt{\left( \sqrt{\pi }2^n n!\right) (1+\sigma ^2)}}\\&\exp \left( -\frac{\mu ^2}{2(1+\sigma ^2)}\right) \sum _{k=0}^n H_{n,k} \left( \mu m_k-m_{k+1}\right) . \end{aligned}$$

Again, $H_{n,k}$ is the k-th coefficient in the n-th Hermite polynomial and $m_k$ is the k-th raw moment of a normal distribution with mean $\mu /(\sigma ^2+1)$ and variance $\sigma ^2/(\sigma ^2+1)$. In terms of the basis coefficients, the derivative can be written as

$$\begin{aligned} \frac{\partial b_n}{\partial \mu }= & {} \frac{1}{\sigma ^2}\left( \mu b_n-b_{n+1}+\frac{1}{\sqrt{\left( \sqrt{\pi }2^n n!\right) \left( 1+\sigma ^2\right) }}\right. \nonumber \\&\left. \exp \left( -\frac{\mu ^2}{2\left( 1+\sigma ^2\right) }\right) \right) . \end{aligned}$$

(13)

Mixtures of normals result in more complex calculations than the normal, but the basis coefficients of the mixture as a whole are given through the sum of the individual basis coefficients. The derivative of the basis coefficients needed for the HMCMC is the weighted sum of the derivatives of the basis coefficients of the normal components.