Combining functional data with hierarchical Gaussian process models

Abstract

Gaussian process models have been used in applications ranging from machine learning to fisheries management. In the Bayesian framework, the Gaussian process is used as a prior for unknown functions, allowing the data to drive the relationship between inputs and outputs. In our research, we consider a scenario in which response and input data are available from several similar, but not necessarily identical, sources. When little information is known about one or more of the populations it may be advantageous to model all populations together. We present a hierarchical Gaussian process model with a structure that allows distinct features for each source as well as shared underlying characteristics. Key features and properties of the model are discussed and demonstrated in a number of simulation examples. The model is then applied to a data set consisting of three populations of Rotifer Brachionus calyciflorus Pallas. Specifically, we model the log growth rate of the populations using a combination of lagged population sizes. The various lag combinations are formally compared to obtain the best model inputs. We then formally compare the leading hierarchical Gaussian process model with the inferential results obtained under the independent Gaussian process model.

Appendices

Appendix 1: Rprop algorithm for HGP

We utilized the Rprop algorithm to establish the initial values of the parameters ${{\varvec{\theta }}} = (\tau ^2, \sigma ^2, \phi , \rho )'$ for the MCMC. We apply the algorithm described by Blum and Riedmiller (2013) for the standard Gaussian Process Regression model for The HGP. In general, the Rprop algorithm is a gradient-based optimization method. We use the algorithm to find the parameter set that maximizes the log likelihood of the HGP model with $R_C = R_\varSigma$ being the squared exponential function. Note that we also assume the same variance parameter for the corresponding covariance functions. As described in Sect. 2.3, we marginalize the HGP of the unknown functions $f_i(\cdot )$ to obtain the model stated in (4) and (5). The log likelihood of the model is then given by:

$$\begin{aligned} l({{\varvec{y}}})\propto & {} -0.5{{\varvec{y}}}'\varPhi ^{-1}{{\varvec{y}}}- 0.5log|\varPhi | \end{aligned}$$

The derivative of the log likelihood with respect to the kth parameter, $\theta _k$, is given by:

$$\begin{aligned} \frac{\partial }{\partial \theta _k} l({{\varvec{y}}})\propto & {} 0.5tr\left( \varPhi ^{-1}{{\varvec{y}}}{{\varvec{y}}}'\varPhi ^{-1} - \varPhi ^{-1}\right) \frac{\partial \varPhi }{\partial \theta _k} \end{aligned}$$

Specifically,

$$\begin{aligned} \frac{\partial \varPhi }{\partial \tau ^2}(x_{ij},x_{ i'j'})= & {} \rho R_C(x_{ij},x_{i'j'}) + \delta _{ii'}(1 -\rho )R_\varSigma (x_{ij},x_{i'j'}) + \delta _{ij}\delta _{i'j'}\sigma ^2\\ \frac{\partial \varPhi }{\partial \sigma ^2} (x_{ij},x_{ i'j'})= & {} \delta _{ii'}\delta _{jj'}\sigma ^2\\ \frac{\partial \varPhi }{\partial \phi }(x_{ij},x_{ i'j'})= & {} -\rho \tau ^2R_C(x_{ij},x_{i'j'}) (x_{ij}-x_{i'j'})^2\\&- \delta _{ii'}(1-\rho )\tau ^2R_C(x_{ij},x_{i'j'}) (x_{ij}-x_{i'j'})^2 + \delta _{ii'}\delta _{jj'}\sigma ^2\\ \frac{\partial \varPhi }{\partial \rho }(x_{ij},x_{ i'j'})= & {} \tau ^2R_C(x_{ij},x_{i'j'}) - \delta _{ii'}\tau ^2R_C(x_{ij},x_{i'j'}) + \delta _{ii'}\delta _{jj'}\sigma ^2 \end{aligned}$$

The sequential algorithm according to Rpop for time step, $t = 1,\ldots ,T$ is as followings:

$$\begin{aligned} \theta _k^{(t+1)}= & {} \theta _k^{(t)} - sign \left( \frac{ -\partial l({{\varvec{y}}})}{\partial \theta _k}^{(t)} \right) \varDelta ^{(t)}_k\\ \text{ where } \varDelta _k^{(t)}= & {} \left\{ \begin{array}{lll} \eta ^+ \varDelta ^{(t-1)}_k, &{}\quad \text{ if } \ \frac{-\partial l({{\varvec{y}}})}{\partial \theta _k}^{(t-1)}\cdot \frac{-\partial l ({{\varvec{y}}})}{\partial \theta _k}^{(t)} > 0\\ \eta ^-\varDelta ^{(t-1)}_k, &{}\quad \text{ if } \ \frac{-\partial l({{\varvec{y}}})}{\partial \theta _k}^{(t-1)}\cdot \frac{-\partial l({{\varvec{y}}})}{\partial \theta _k}^{(t)} <0\\ \varDelta ^{(t-1)}_k, &{}\quad \text{ else } \end{array} \right. \end{aligned}$$

such that $\eta ^+ > 1$ and $0<\eta ^-<1$ are specified. If the sign of the partial derivative of the negative log likelihood with respect to parameter $\theta _k$ remains unchanged, then $\theta _k$ continues in the same direction if already heading towards the minimum or otherwise reverses direction by a larger change of $\eta ^+\varDelta _k^{(t-1)}$. If the partial derivative changes, then $\theta _k$ reverses direction if the minimum was overshot or otherwise continues in the same direction by a smaller change of $\eta ^-\varDelta$. If a local minimum has been reached no change is performed. An initial $\varDelta _0$ value is specified along with a $\varDelta _{ min }$ and $\varDelta _{ max }$ such that the value of the change is bounded.

Appendix 2: Posterior sampling and predictive algorithms for HGP

This appendix describes the MCMC algorithm used to obtain posterior samples of the model parameters, $(\tau ^2, \sigma ^2, \phi , \rho )$, as well as predictive inference for new responses, ${{\varvec{y}}}^*$, over the vector of new input values, ${{\varvec{x}}}^*_i$. Recall, that the model is given by,

$$\begin{aligned} {{\varvec{y}}}|{{\varvec{X}}}, \rho , \phi , \tau ^2, \sigma ^2\sim & {} N_N({{\varvec{0}}}, \varPhi ) \nonumber \\ \rho , \phi , \tau ^2, \sigma ^2\sim & {} p(\rho )p( \phi ) p(\tau ^2)p( \sigma ^2) \end{aligned}$$

The elements of $\varPhi$ are given by:

$$\begin{aligned} Cov\left[ y_{ij},y_{i'j'}\right]= & {} C(x_{ij}, x_{i'j'}) + \delta _{ii'}\varSigma (x_{ij}, x_{i'j'})+ \delta _{ii'}\delta _{jj'} \sigma ^2 \end{aligned}$$

where $\delta _{ii'}$ is Kronecker delta function which is 1 if $i=i'$ and 0 otherwise. We assume the same squared exponential covariance functions for $C(\cdot )$ and $\varSigma (\cdot )$. We use a uniform prior for the correlation parameter $\rho$, and inverse gamma functions, $\varGamma ^{-1}(a,b)$, with mean a / b for the other three: $p(\tau ^2) \equiv \varGamma ^{-1}(2, b_\tau ), \ \ p(\sigma ^2) \equiv \varGamma ^{-1}(2, b_\sigma ), \ \ p(\phi ) \equiv \varGamma ^{-1}(2, b_\phi )$.

For the parameter updates, we use the Metropolis–Hastings algorithm (Hastings 1970) as conjugacy is not available. Initial values for $\tau ^2_0$, $\sigma ^2_0$, $\phi _0$, and $\rho _0$ are obtained using Rprop (see “Appendix 1”), and posterior samples $t = 1,\ldots ,T$ are obtained via the Metropolis–Hastings algorithm below.

Values of the parameters are proposed by a multivariate normal distribution:

The proposed values are accepted with probability,

$$\begin{aligned} min\left\{ 1, \frac{N({{\varvec{y}}}| {{\varvec{X}}}, \rho ^{(*)}, \phi ^{(*)}, \tau ^{2(*)}, \sigma ^{2(*)})\varGamma ^{-1}(\tau ^{2(*)})\varGamma ^{-1}(\sigma ^{2(*)}) \varGamma ^{-1}(\phi ^{(*)}) \tau ^{2(*)}\sigma ^{2(*)}\phi ^{(*)}\rho ^{(*)}(1-\rho ^{(*)})}{N({{\varvec{y}}}| { {\varvec{X}}}, \rho ^{(t)}, \phi ^{(t)}, \tau ^{2(t)}, \sigma ^{2(t)})\varGamma ^{-1}(\tau ^{2(t)})\varGamma ^{-1}(\sigma ^{2(t)}) \varGamma ^{-1}(\phi ^{(t)}) \tau ^{2(t)}\sigma ^{2(t)}\phi ^{(t)}\rho ^{(t)}(1-\rho ^{(t)})} \right\} . \end{aligned}$$

Preliminary runs are made to establish the form of the covariance matrix of the proposal distribution, and $c >0$ is specified to facilitate stability and good mixing of the posterior samples. A burn-in period is performed and thinning is performed to obtain independent posterior samples.

Predictive values of new responses for the ith population, ${{\varvec{y}}}_i^*$, over a grid of new input values, ${{\varvec{X}}}^*$ can easily be obtained by integrating over the posterior samples of the model parameters:

$$\begin{aligned}&p\left( {{\varvec{y}}}_i^*| {{\varvec{X}}}^*, \mathscr {D}\right) = \int _{{\varvec{\theta }}} N\left( {{\varvec{m}}}^* ({{\varvec{\theta }}}), {{\varvec{S}}}^* ({{\varvec{\theta }}})\right) d{{\varvec{\theta }}}\\&\text{ where } \\&\quad {{\varvec{m}}}_i^*({{\varvec{\theta }}}) = B\varPhi ^{-1}{{\varvec{y}}}\\&\quad {{\varvec{S}}}_i^*({{\varvec{\theta }}}) = A - B\varPhi ^{-1}B' \end{aligned}$$

where $A(x^*_{ij},x^*_{ij'}) = C(x^*_{ij},x^*_{ij'}) + \varSigma (x^*_{ij},x^*_{ij'}) + \delta _{jj'}\sigma ^2$ and $B(x^*_{ij}, x_{i'j'}) = C(x^*_{ij}, x_{i'j'}) + \delta _{ii'}\varSigma (x^*_{ij}, x_{i'j'})$. Therefore, for each posterior sample of the model parameters, ${{\varvec{\theta }}}^{(t)}$, we can obtain a sample of the predicted response for the ith population, ${{\varvec{y}}}^*_i$, over the vector of new input values, ${{\varvec{x}}}^*_i$:

$$\begin{aligned} {{\varvec{y}}}^{*(t)}_i\sim & {} N_N\left( {{\varvec{m}}}_i^* ({{\varvec{\theta }}}^{(t)}), {{\varvec{S}}}_i^* ({{\varvec{\theta }}}^{(t)})\right) \nonumber \\ \end{aligned}$$