Water flow probabilistic predictions based on a rainfall–runoff simulator: a two-regime model with variable selection

Abstract

Probabilistic forecasting aims at producing a predictive distribution of the quantity of interest instead of a single best guess point-wise estimate. With regard to water flow forecasts, the two main sources of uncertainty stem from unknown future rainfall and temperature (input error, i.e., meteorological uncertainty) and from the inadequacy of the deterministic simulator mimicking the rainfall–runoff (RR) transformation (hydrological uncertainty or RR error). These two sources of uncertainty can be dealt with separately and only the latter will be considered here. Only hydrological uncertainty is at stake when recorded meteorological data (instead of meteorological forecasts) are used as inputs to feed the RR simulator (RRS) for probabilistic predictions. The predictive performance of the RRS may strongly depend on the hydrological regimes: rapid flood variations induce large errors of anticipation but a series of dry events will translate into a much more smoother sequence of river levels due to the easily predictable behavior of the soil reservoir emptying. Consequently, a model with several regimes adapted to different error structures appears as a solution to cope with the issue of unstationary predictive variance. The river regime is modeled as a latent variable, the distribution of which is based on additional outputs of the RRS to be selected. Inference is performed by the EM algorithm with both steps leading to explicit analytic expressions. Asymptotic confidence regions for the estimates are provided within the same EM framework. Model selection is also performed, including the length of the model memory as well as the choice of explanatory variables for the latent regimes. The model is applied to a series of water flow forecasts routinely issued by two hydroelectricity producers in France and in Québec and compared with their present operational forecasting methods.

Appendices

Appendix 1: Operational predictive method

EDF’s operational predictive method consists of 3 independent modules: a deterministic model, an error model and an empirical copula.

Deterministic model (Gailhard 2014) The deterministic model in use at EDF is an autoregressive model combined with exponential smoothing. The strength of the autocorrelation is supposed to increase with the rate of water flow coming from the deep reservoirs of the watershed.

Error model (Mathevet 2010) The error model is an heteroscedastic conditional normal model derived for each forecasting lead time h (after normalization):

$$\begin{aligned} \left( Y_{t+h}|X_{t+h}=x\right) =b_{h}\left( x\right) +x+\sigma _{h}\left( x\right) \varepsilon ,\;\;\varepsilon \sim \mathcal {N}\left( 0,1\right) , \end{aligned}$$

where $b_{h}$ and $\sigma _{h}$ are tabulated functions of x.

Empirical mopula One finally resorts to an empirical copula to get samples of a space and time multivariate distribution from samples from the marginal (lead time by lead time) distributions.

Appendix 2: Fisher information matrix

$$\begin{aligned} G_{\gamma _k}(\mathbf {Y},\mathbf {Z})&=\sum _{t>t_{\min }} \mathbb {I}_{\{S_{t}=k\}}\left( \mathbf {U}_t/\sigma _k^2 \cdot Y_t +0 \cdot Y_t^2-\mathbf {U}_t\mathbf {U}_t^T\varvec{\gamma }_k/\sigma ^2_k \right) \\&=\sum _{t>t_{\min }} \mathbb {I}_{\{S_{t}=k\}}\left( \mathbf {U}_t/\sigma _k^2\cdot Y_t-\mathbf {U}_t\mathbf {U}_t^T\varvec{\gamma }_k/\sigma ^2_k \right) ,\\ G_{\sigma ^2_k}(\mathbf {Y},\mathbf {Z})&=\sum _{t>t_{\min }} \mathbb {I}_{\{S_{t}=k\}}\left( \frac{\left( Y_t-\varvec{\gamma }_k^T\mathbf {U}_t \right) ^2}{2\sigma ^4_k}-\frac{1}{2\sigma _k^2}\right) ,\\ G_{\mathbf {B}}(\mathbf {Y},\mathbf {Z})&= \sum _{t>t_{\min }} (Z_t-\mathbf {B}^T\mathbf {V}_t)\mathbf {V}_t. \end{aligned}$$

For any k, $k'\not =k$,

$$\begin{aligned} \frac{\partial }{\partial \varvec{\gamma }_k}G_{\varvec{\gamma }_k}(\mathbf {Y},\mathbf {Z})= & {} -\sum _{t>t_{\min }} \mathbb {I}_{\{S_{t}=k\}} \mathbf {U}_t\mathbf {U}_t^T/\sigma ^2_k,\\ \frac{\partial }{\partial \sigma _k^2}G_{\sigma ^2_k}(\mathbf {Y},\mathbf {Z})= & {} \sum _{t>t_{\min }} \mathbb {I}_{\{S_{t}=k\}} \left( \frac{-\left( Y_t-\varvec{\gamma }_k^T\mathbf {U}_t \right) ^2}{\sigma ^6_k}+\frac{1}{2\sigma _k^4}\right) ,\\ \frac{\partial }{\partial \varvec{\gamma }_k}G_{\sigma ^2_k}(\mathbf {Y},\mathbf {Z})= & {} \sum _{t>t_{\min }} \mathbb {I}_{\{S_{t}=k\}}\left( \frac{-\mathbf {U}_t \left( Y_t-\varvec{\gamma }_k^T\mathbf {U}_t \right) }{\sigma ^4_k}\right) ,\\ \frac{\partial }{\partial \theta _{k'}}G_{\theta _k}(\mathbf {Y},\mathbf {Z})= & {} 0,\\ \frac{\partial }{\partial \mathbf {B}}G_{\mathbf {B}}(\mathbf {Y},\mathbf {Z})= & {} -\sum _{t>t_{\min }}\mathbf {V}_t\mathbf {V}_t^T,\\ \frac{\partial }{\partial \theta _k}G_{\mathbf {B}}(\mathbf {Y},\mathbf {Z})= & {} 0. \end{aligned}$$

Then, the first term in the Louis decomposition is computed since $\mathbb {E}(\mathbb {I}_{\{S_{t}=k\}}|\mathbf {Y};(\varvec{\theta },\mathbf {B}))=\tau _{kt}$ is computed above.

For the second term in the Louis decomposition, we notice that:

$$\begin{aligned} \mathbb {E}(\mathbb {I}_{\{S_{t}=k\}}^2|\mathbf {Y};\varvec{\theta },\mathbf {B})= & {} \mathbb {E}(\mathbb {I}_{\{S_{t}=k\}}|\mathbf {Y};\varvec{\theta },\mathbf {B})=\tau _{kt},\\ \mathbb {E}(\mathbb {I}_{\{S_{t}=k\}}\mathbb {I}_{\{S_{t}=k'\}}|\mathbf {Y};\varvec{\theta },\mathbf {B})= & {} 0 \text { for } k\not =k',\\ \mathbb {E}(\mathbb {I}_{\{S_{t}=k\}}\mathbb {I}_{\{S_{t'}=k\}}|\mathbf {Y};\varvec{\theta },\mathbf {B})= & {} \mathbb {E}(\mathbb {I}_{\{S_{t}=k\}}|\mathbf {Y};\varvec{\theta },\mathbf {B})\mathbb {E}(\mathbb {I}_{\{S_{t'}=k\}}|\mathbf {Y};\varvec{\theta },\mathbf {B})\text { by independence}. \end{aligned}$$

We also need to compute:

$$\begin{aligned} \mathbb {E}(Z_t|\mathbf {Y};\varvec{\theta },\mathbf {B})= & {} \sum _k \tau _{kt}E_{kt}\quad \text {and}\\ \mathbb {E}\left( (Z_t-\mathbf {B}^T\mathbf {V}_t)^2|\mathbf {Y};\varvec{\theta },\mathbf {B}\right)= & {} \text {var}(Z_t|\mathbf {Y};\varvec{\theta },\mathbf {B})=\sum _k \tau _{kt}\varsigma _{kt}, \end{aligned}$$

where

$$\begin{aligned} \varsigma _{1t}= & {} 1-\frac{\phi (-\mathbf {B}^T \mathbf {V}_t)}{\varPhi (-\mathbf {B}^T\mathbf {V}_t)}\left( \frac{\phi (-\mathbf {B}^T\mathbf {V}_t)}{\varPhi (-\mathbf {B}^T\mathbf {V}_t)}-\mathbf {B}^T\mathbf {V}_t\right) ,\\ \varsigma _{0t}= & {} 1-\frac{\phi (-\mathbf {B}^T \mathbf {V}_t)}{1-\varPhi (-\mathbf {B}^T\mathbf {V}_t)}\left( \frac{\phi (-\mathbf {B}^T\mathbf {V}_t)}{1-\varPhi (-\mathbf {B}^T\mathbf {V}_t)}+\mathbf {B}^T\mathbf {V}_t\right) . \end{aligned}$$

Again, we rely on the independence between the $Z_t$s.

The remaining terms are easily evaluated:

$$\begin{aligned} \mathbb {E}(\mathbb {I}_{\{S_{t'}=k\}}Z_t|\mathbf {Y};\varvec{\theta },\mathbf {B})= & {} \tau _{kt'}\mathbb {E}(Z_t|\mathbf {Y};\varvec{\theta },\mathbf {B})\text { by independence},\\ \mathbb {E}(\mathbb {I}_{\{S_{t}=k\}}Z_t|\mathbf {Y};\varvec{\theta },\mathbf {B})= & {} \tau _{kt}E_{kt}. \end{aligned}$$