Post-processing Multiensemble Temperature and Precipitation Forecasts Through an Exchangeable Normal-Gamma Model and Its Tobit Extension

Abstract

Meteorological ensemble members are a collection of scenarios for future weather issued by a meteorological center. Such ensembles nowadays form the main source of valuable information for probabilistic forecasting which aims at producing a predictive probability distribution of the quantity of interest instead of a single best guess point-wise estimate. Unfortunately, ensemble members cannot generally be considered as a sample from such a predictive probability distribution without a preliminary post-processing treatment to re-calibrate the ensemble. Two main families of post-processing methods, either competing such as the BMA or collaborative such as the EMOS, can be found in the literature. This paper proposes a mixed-effect model belonging to the collaborative family. The structure of the model is formally justified by Bruno de Finetti’s representation theorem which shows how to construct operational statistical models of ensemble based on judgments of invariance under the relabeling of the members. Its interesting specificities are as follows: (1) exchangeability contributes to parsimony, with an interpretation of the latent pivot of the ensemble in terms of a statistical synthesis of the essential meteorological features of the ensemble members, (2) a multiensemble implementation is straightforward, allowing to take advantage of various information so as to increase the sharpness of the forecasting procedure. Focus is cast onto normal statistical structures, first with a direct application for temperatures, then with its very convenient Tobit extension for precipitation. Inference is performed by expectation maximization (EM) algorithms with both steps leading to explicit analytic expressions in the Gaussian temperature case, and recourse is made to stochastic conditional simulations in the zero-inflated precipitation case. After checking its good behavior on artificial data, the proposed post-processing technique is applied to temperature and precipitation ensemble forecasts produced for lead times from 1 to 9 days over five river basins managed by Hydro-Québec, which ranks among the world’s largest electric companies. These ensemble forecasts, provided by three meteorological global forecast centers (Canadian, USA and European), were extracted from the THORPEX Interactive Grand Global Ensemble (TIGGE) database. The results indicate that post-processed ensembles are calibrated and generally sharper than the raw ensembles for the five watersheds under study.

Supplementary materials accompanying this paper appear on-line.

Details on Inference in the Gaussian Case

E-step  

We need to compute the conditional distributions $\left[ Z_t|\omega _t^{-2},{\mathbf {X}}_t,Y_t\right]$ and $\Big [\omega _t^{-2}|{\mathbf {X}}_t,Y_t\Big ]$ for each time t of the training set. We interpret the joint distribution at time t

$$\begin{aligned} \left[ Z_t,\omega _t^{2},{\mathbf {X}}_t,Y_t\right]&=[{\mathbf {X}}_t,Y_t|Z_t,\omega _t^{-2}][\omega _t^{-2},Z_t]\\&=[{\mathbf {X}}_t|Z_t,\omega _t^{-2}][Y_t|Z_t,\omega _t^{-2}][\omega _t^{-2},Z_t] \end{aligned}$$

as a function of $\left( \omega _t^{-2},Z_t\right) ,$ and try to recognize the probability distribution function (pdf) of $\left( Z_t,\omega _t^{-2}|{\mathbf {X}}_t,Y_t\right)$ up to a multiplicative constant, since $\left[ Z_t,\omega _t^{-2}|{\mathbf {X}}_t,Y_t\right] =[\omega _t^{-2},Z_t,{\mathbf {X}}_t,Y_t]\times \left( \frac{1}{[{\mathbf {X}}_t,Y_t]}\right)$.

The complete deviance (minus twice the complete log-likelihood) at time t can be written as a quadratic form in ${\mathbf {X}}_t$ and $Y_t$, up to known normalizing constants:

$$\begin{aligned}&\sum _{e=0}^{E}\left\{ \sum _{k=1}^{K_{e}}(X_{e,k,t}-b_{e}Z_t-a_{e})^{2}\omega _t^{-2}c_{e}^{-2}-K_{e}\log \left( \omega _t^{-2}\right) -K_{e}\log \left( c_{e}^{-2}\right) \right\} \nonumber \\&\qquad +\,Z_t^{2}\lambda ^{-1}\omega _t^{-2}-\log (\lambda ^{-1})-\log \left( \omega _t^{-2}\right) \nonumber \\&\quad \quad +\,2\beta \omega _t^{-2}-2(\alpha -1)\log \left( \omega _t^{-2}\right) -2\log \left( \frac{\beta ^{\alpha }}{\Gamma (\alpha )}\right) . \end{aligned}$$

(8)

The pdf we are looking for can be further decomposed as:

$$\begin{aligned} \left[ Z_t,\omega _t^{-2}|{\mathbf {X}}_t,Y_t\right] =[Z_t|\omega _t^{-2},{\mathbf {X}}_t,Y_t][\omega _t^{-2}|{\mathbf {X}}_t,Y_t]\,. \end{aligned}$$

We now check whether we can still benefit from a conjugate situation, i.e., given $\left( {\mathbf {X}}_t,Y_t\right)$ we would still get a normal pdf for $\left( Z_t|\omega _t^{-2},{\mathbf {X}}_t,Y_t\right)$ under the form ${\mathcal {N}}(m^{\prime },\lambda ^{\prime }\omega _t^{2})$ and a gamma pdf for $\left( \omega _t^{-2}|{\mathbf {X}}_t,Y_t\right)$, $\Gamma \left( \alpha ^{\prime },\beta ^{\prime }\right)$. Expressed as a function of $[Z_t,\omega _t^{-2}|{\mathbf {X}}_t,Y_t]$, the deviance exhibits the following shape:

$$\begin{aligned} (Z_t-m_t^{\prime })^{2}\lambda ^{\prime -1}\omega _t^{-2}-\log (\omega _t^{-2})-\log (\lambda ^{\prime -1})-2(\alpha ^{\prime }-1)\log (\omega _t^{-2})+2\beta _t^{\prime }\,(\omega _t^{-2})\,. \end{aligned}$$

We proceed by trying to identify parameters $\alpha ^{\prime },\beta _t^{\prime },\lambda ^{\prime },m_t^{\prime }$ in the above equation to match the deviance for their joint distribution given by Eq. (8).

By matching both expressions, we obtain:

$$\begin{aligned} \begin{array}{lcl} \lambda ^{\prime -1}&{}=&{}\sum _{e=0}^{E}K_{e}b_{e}^{2}c_{e}^{-2}+\lambda ^{-1},\\ m_t^{\prime }&{}=&{}\lambda ^{\prime }\cdot \sum _{e=0}^{E}c_{e}^{-2}b_{e}K_{e}\left( {\bar{X}}_{e,t}-a_{e}\right) \\ \alpha ^{\prime }&{}=&{}\alpha +\frac{\sum _{e=0}^{E}K_{e}}{2},\\ \beta _t^{\prime }&{}=&{}\beta +\frac{1}{2}\left\{ \sum _{e=0}^{E}\sum _{k=1}^{K_{e}}c_{e}^{-2}(X_{e,k,t}-a_{e})^{2}-m_t^{\prime 2}\lambda ^{\prime -1}\right\} , \end{array} \end{aligned}$$

where ${\bar{X}}_{e,t}=\frac{1}{K_{e}}\sum _{k=1}^{K_{e}}X_{e,k,t}$. Therefore, we are in a conjugate situation since the conditional pdf $\left[ Z_t,\omega _t^{-2}|{\mathbf {X}}_t,Y_t\right]$ is in the normal-gamma model as is the marginal pdf $\left[ Z_t,\omega _t^{-2}\right]$.

Denoting $\phi (\cdot )$ the first derivative of function $\log \left\{ \Gamma (\cdot )\right\}$, the moments necessary for performing the E-step are:

$$\begin{aligned} \begin{array}{lcl} {\mathbb {E}}\left( \log (\omega _t^{-2})|{\mathbf {X}}_t,Y_t\right) &{} =&{}-\log (\beta _t^{\prime })+\phi (\alpha ^{\prime })\,,\\ {\mathbb {E}}(\omega _t^{-2}|{\mathbf {X}}_t,Y_t) &{}=&{}\frac{\alpha ^{\prime }}{\beta _t^{\prime }}\,,\\ {\mathbb {E}}(Z_t^{2}\omega _t^{-2}|{\mathbf {X}}_t,Y_t) &{}=&{}\lambda ^{\prime }+m_t^{\prime 2}\frac{\alpha ^{\prime }}{\beta _t^{\prime }}\,,\\ {\mathbb {E}}(Z_t \omega _t^{-2}|{\mathbf {X}}_t,Y_t) &{} =&{}m_t^{\prime }\frac{\alpha ^{\prime }}{\beta _t^{\prime }}\,. \end{array} \end{aligned}$$

M-step  

We write the complete deviance $D(\varvec{\theta })=D(\alpha ,\beta ,\lambda ,{\mathbf {a}},{\mathbf {b}},{\mathbf {c}})$, denoting n the number of records in the dataset (each of them indexed by t):

$$\begin{aligned} \begin{aligned}D(\varvec{\theta })&= \sum _{t=1}^{n} \Bigg \{ Z_{t}^{2}\lambda ^{-1}\omega _{t}^{-2}-\log (\lambda ^{-1})-2\alpha \log (\omega _{t}^{-2})+2\beta \omega _{t}^{-2}-2\alpha \log (\beta )\\&\qquad +2\log \left\{ \Gamma (\alpha )\right\} \\&\qquad + \sum _{e=0}^{E}\left\{ \sum _{k=1}^{K_{e}}\left( X_{e,k,t}-a_{e}-b_{e}Z_{t}\right) ^{2}c_{e}^{-2}\omega _{t}^{-2}-\log \left( c_{e}^{-2}\right) \right\} \Bigg \}+ \text {Cst} \,, \end{aligned} \end{aligned}$$

where $\text {Cst}$ is a constant term with respect to the parameters to estimate.

First, the expectation of $D(\varvec{\theta })$ is computed by using the moments computed in the E-step. Then, this expectation is differentiated with respect to the parameters to be updated. This leads to the following explicit update formulas, and the subscript new indicates the new value of the parameter:

$$\begin{aligned} \begin{array}{lcl} b_{e,new} &{} =&{}\frac{\frac{D_{e}}{B}-\frac{C_{e}}{G}}{\frac{G}{B}-\frac{H}{G}-\frac{n\lambda ^{\prime }}{G\alpha ^{\prime }}}\quad \text { for } e\in \{1,\ldots ,E\},\\ a_{e,new} &{} =&{}\frac{D_{e}}{B}-b_{e,new}\frac{G}{B} \quad \text { for } e\in \{0,\ldots ,E\},\\ c_{e,new}^{2}&{}=&{}K_{e}b_{e,new}^{2}\lambda ^{\prime }+\frac{1}{n}\sum _{t=1}^{n}\frac{\alpha ^{\prime }}{\beta _{t}^{\prime }}\sum _{k=1}^{K_{e}}\left( X_{e,k,t}-a_{e,new}-b_{e,new}m_{t}^{\prime }\right) ^{2}\quad \text { for } e\in \{1,\ldots ,E\},\\ \lambda _{new}&{}=&{}\lambda ^{\prime }+\frac{\alpha ^{\prime }}{n}\sum _{t=1}^{n}\frac{m_{t}^{\prime 2}}{\beta _{t}^{\prime }},\\ \beta _{new} &{} =&{}\frac{n\alpha _{new}}{\alpha ^{\prime }\sum _{t=1}^{n}\frac{1}{\beta _{t}^{\prime }}}, \end{array} \end{aligned}$$

where $G=\sum _{t=1}^{n}\frac{m_{t}^{\prime }}{\beta _{t}^{\prime }}$, $B=\sum _{t=1}^{n}\frac{1}{\beta _{t}^{\prime }}$, $C_{e}=\sum _{t=1}^{n}\frac{m_{t}^{\prime }{\bar{X}}_{e,t}}{\beta _{t}^{\prime }}$, $D_{e}=\sum _{t=1}^{n}\frac{{\bar{X}}_{e,t}}{\beta _{t}^{\prime }}$ et $H=\sum _{t=1}^{n}\frac{m_{t}^{\prime 2}}{\beta _{t}^{\prime }}$. For updating $\alpha$, we use a numeric solver of the following equation:

$$\begin{aligned} \log \left( \frac{n\alpha _{new}}{\alpha ^{\prime }\sum _{t=1}^{n}\frac{1}{\beta _{t}^{\prime }}}\right) -\phi (\alpha _{new}) =\frac{1}{n}\sum _{t=1}^{n}\log (\beta _{t}^{\prime })-\phi (\alpha ^{\prime }). \end{aligned}$$