Abstract
We address the calibration constraint of probability forecasting. We propose a generic method for recalibration, which allows us to enforce this constraint. It remains to be known the impact on forecast quality, measured by predictive distributions sharpness, or specific scores. We show that the impact on the Continuous Ranked Probability Score (CRPS) is weak under some hypotheses and that it is positive under more restrictive ones. We used this method on temperature ensemble forecasts and compared the quality of the recalibrated forecasts with that of the raw ensemble and of a more specific method, that is Ensemble Model Output Statistics (EMOS). Better results are shown with our recalibration rather than with EMOS in this case study.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
S. Ben Taieb, R. Huser, R.J. Hyndman, M.G. Genton, Forecasting uncertainty in electricity smart meter data by boosting additive quantile regression. IEEE Trans. Smart Grid 7, 2448–2455 (2016)
J. Bröcker, Reliability, sufficiency, and the decomposition of proper scores. Q. J. R. Meteorol. Soc. (2009). https://doi.org/10.1002/qj.456
V. Corradi, N.R. Swanson, Predictive density evaluation, in Handbook of Economic Forecasting ed. by G. Elliott, C.W.J. Granger, A. Timmermann (Elsevier, Amsterdam, 2006), pp. 197–284
V. Fortin, A.-C. Favre, M. Said, Probabilistic forecasting from ensemble prediction systems: improving upon the best-member method by using a different weight and dressing kernel for each member. Q. J. R. Meteorol. Soc. (2006). https://doi.org/10.1256/qj.05.167
T. Gneiting, Calibration of medium-range weather forecasts, Technical Memorandum, European Centre for Medium-Range Weather Forecasts. https://www.ecmwf.int/en/elibrary/9607-calibration-medium-range-weather-forecasts (2014)
T. Gneiting, M. Katzfuss, Probabilistic forecasting. Annu. Rev. Stat. Appl. (2014). https://doi.org/10.1146/annurev-statistics-062713-085831
T. Gneiting, A.E. Raftery, A.H. Westveld III, T. Goldman, Calibrated probabilistic forecasting using ensemble model output statistics and minimum CRPS estimation. Mon. Weather Rev. (2005). https://doi.org/10.1175/MWR2904.1
T. Gneiting, F. Balabdaoui, A.E. Raftery, Probabilistic forecasts, calibration and sharpness. J. R. Stat. Soc. Ser. B (Stat. Methodol.) (2007). https://doi.org/10.1111/j.1467-9868.2007.00587.x
A. Gogonel, J. Collet, A. Bar-Hen, Improving the calibration of the best member method using quantile regression to forecast extreme temperatures. Nat. Hazards Earth Syst. Sci. (2013). https://doi.org/10.5194/nhess-13-1161-2013
H. Hersbach, Decomposition of the continuous ranked probability score for ensemble prediction systems. Weather Forecast. (2000). https://doi.org/10.1175/1520-0434(2000)015<0559:DOTCRP>2.0.CO;2
R. Krzysztofowicz, Bayesian processor of output: a new technique for probabilistic weather forecasting, in 17th Conference on Probability and Statistics in the Atmospheric Sciences (American Meteorological Society, 2004). https://ams.confex.com/ams/pdfpapers/69608.pdf
P.-A. Michelangeli, M. Vrac, H. Loukos, Probabilistic downscaling approaches: application to wind cumulative distribution functions. Geophys. Res. Lett. (2009). https://doi.org/10.1029/2009GL038401
D.S. Wilks, Statistical Methods in the Atmospheric Sciences. International Geophysics Series ed. by D.S. Wilks, vol. 100 (Academic Press, Cambridge, 2011)
D.S. Wilks, Enforcing calibration in ensemble postprocessing. Q. J. R. Meteorol. Soc. (2017). https://doi.org/10.1002/qj.3185
Acknowledgements
This research was supported by the ANR project FOREWER (ANR-14-CE05-0028).
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
8.5 Appendix
8.5 Appendix
Here are gathered all the proofs concerning the results presented in the chapter. The first section is concerned by proofs of results in an infinite sample and the second by result in a finite sample.
Lemma 8.1
with \(\tau \),\(\tau _c \in [0,1]\) and \(p_e\) the frequency of appearance of the state e. Under the Assumption A.2.1, we prove Lemma 8.1.
Proof
We have:
First, we only focus on a particular e. Thus, we are interested in:
For ease of notation and comprehension, we suppress e in the notation since there is no confusion. Moreover, we suppose, for ease of notation again (and since we obtain the same result if we inverse the inequality) that \(G^{-1}(\tau ) \le \,G^{-1}(\tau _c)\). So, we have:
Using integral by parts, we have:
Replacing it in (8.8) finishes the demonstration. \(\square \)
1.1 8.5.1 Impact on Score: Conditions for Improvement
In this section, the reader can find the proofs of results mentioned in Sect. 8.3.1 of the chapter. We first demonstrate how to approximate the difference of \(L_\tau \) expectation before showing that under some hypotheses, our correction improves systematically the quality of the forecasts.
1.1.1 8.5.1.1 Rewriting the Difference of \(L_\tau \) Expectation
Under the Assumptions A.2.1, A.3.1.1–A.3.1.4 and using functional derivatives and the implicit function theorem, we prove (8.2).
Proof
Remember: Let H be a functional, h a function, \(\alpha \) a scalar and \(\delta \) an arbitrary function.
We can write the expression of the functional evaluated at \(f+\delta \alpha \) as follow:
with Rem(\(\alpha \)) the remainder. Denote:
For ease of notation, denote \(\varDelta PL_{e} [F_{e} + \delta _{e} \alpha ] \equiv \varDelta PL_{F,\delta ,e}\). Choosing \(H= \varDelta PL_{e}\), \(h=F_{e}\) and \(\eta _e = \alpha \delta _e\) (even if we use \(\alpha \delta _e\) in the development in order to use functional derivatives, directional derivatives and the implicit function theorem), we have:
To calculate \(\frac{\text {d}\tau _{c}}{\text {d}\alpha }\), we will use the equation which link \(\tau _{c}\) and \(\alpha \):
Using the implicit function theorem, we find:
Now, we need to calculate partial derivatives:
Thus, we have:
and hence:
Now, let’s focus on the remainders. Following the Taylor–Lagrange inequality, if M such that \(\left| \frac{\text {d}^{3} \varDelta PL_{F,\delta ,e}}{\text {d}\alpha ^{3}} \right| \le M\) exists, we have \( \vert \,\text {Rem}_e(\alpha )\,\vert \le \frac{M |\alpha ^{3}|}{3!}\). Let’ s find conditions for the existence of M. The third derivative is:
Let’s calculate the partial derivatives of order 3:
Moreover, we have:
Since \(\eta _e\), its first, second and third derivatives are finite in \(F_e^{-1}(\tau )\), it is also the case for \(\delta _e\) and the partial derivatives are finite. Furthermore, \(f_e\), \(\delta _e\) and their derivatives are bounded (since \(\eta _e\) and their derivatives are bounded), which implies that the second derivatives of \(\varDelta PL_{e} [F_{e} + \delta _{e} \alpha ]\) are also bounded. Thus, under these conditions, M exists. Then, we can write \(\frac{\text {d}^{3} \varDelta PL_{F,\delta ,e}}{\text {d}\alpha ^{3}} = M_1 \delta _e^3\) and hence \(\left| \text {Rem}_e(\alpha ) \right| \le \frac{|M_1| |\alpha \delta _e|^3}{3!}\) which implies that \(\lim \,\,\frac{\text {Rem}_e(\alpha )}{(\alpha \delta _e)^2} = 0\), \(\alpha \delta _e \rightarrow 0\), which shows that \(\text {Rem}_e(\alpha )\) is negligible compared to \(\frac{\text {d}^{2} \varDelta PL_{F,\delta ,e}}{\text {d}\alpha ^{2}}\).
Moreover, since \(\forall \,\,e \in E\) the functions \(F_e\) are \(C^{3}\) and the functions \(f_e\) and their derivatives are bounded by a constant which doesn’t depend on e, \(\forall \,\, e \in E\), the development is valid for all directions and thus, since \(\eta _e = G_e - F_e\), we have:
To finish the demonstration, remark that Lemma 8.1 proves that:
\(\square \)
1.1.2 8.5.1.2 Systematic Improvement of the Quality
Under the Assumption A.3.1.5 or A.3.1.6, if \( \exists \,\,\nu \ge 0\) (sufficiently small)\(\, \, \forall \,\, e \in E\) \(\,\, \forall \,\, y \in \mathbf {R};|\eta _e(y)| \le \nu \), we show (8.3) and (8.4):
Proof
Prove (8.3) is equivalent to show that \(\varDelta PL [G]\) is positive, and if we rewrite:
it is clear that the Assumption A.3.1.6 ensures the positivity of \(\varDelta PL [G]\).
However, we need more argumentation to understand the complete utility of the Assumption A.3.1.5. Let’s look at one of the two worst cases: only two states of the world, the correlation coefficient \(\rho =-1\), \(\eta > 0\) (the other case is when \(\rho =1\) and \(\eta < 0\)) and at each bound of the support of \(\delta \) and \(f^{-1}\), there is half of the probability mass. We also consider that the ratios between max and min of the supports are equal. If we define \( max_e=M\) and \(min_e= \frac{M}{r}\), one has the following equation:
Solving this equation in r produces the expected result concerning the ratio between max and min values of \(\eta \) and \(f^{-1}\).
Now, let’s prove (8.4). According to (8.1), we have:
We can rewrite:
and using the Fubini–Tonelli theorem, one obtains:
\(\square \)
1.2 8.5.2 Impact on Score: Bounds on Degradation
Under the Assumptions A.2.1, A.3.2.1–A.3.2.6 we prove (8.6) and (8.7).
Proof
adding and substracting \(\textsc {E}_Y[\,L_{\tau }(Y,G^{-1}(\tau _c))\,]\) to \(\textsc {E}_Y[L_{\,\widehat{\tau }_{c}}-L_{\tau }\,]\), we obtain:
and finally:
To begin with, we treat the third term on the right side. We have:
Using the change of variable \(y=G_{e}^{-1}(z)\) and taking the absolute value, we find:
Now, one needs to distinguish two cases.
If \(\tau \,>\,\tau _c\), one has:
Since \(|\,F_{e}(z)- G_{e}(z)\,| \le \varepsilon ,\) \(\forall z \in \mathbf {R},\,\forall e \in E\), one obtains \(|\,F_e \circ G_{e}^{-1}(z)-z\,| \le \, \varepsilon ,\) \(\forall z \in [0,1],\,\forall e \in E\) and then:
-
if \(z=\,\tau \), one has \(\left| \,F_e \circ G_{e}^{-1}(\tau )-\tau \,\right| \le \varepsilon \),
-
if \(z=\,\tau _c\), \(\left| \,F_e \circ G_{e}^{-1}(\tau _c)-\tau )\,\right| = \left| \,F_e \circ G_{e}^{-1}(\tau _c)-\tau _{c} + \tau _{c}-\tau \,\right| \).
Moreover, one has:
and finally:
One deduces, when \(\tau \,>\,\tau _c\):
When \(\tau \,<\,\tau _c\), one obtains:
and using the same arguments as previously:
Hence, one concludes that:
To finish, replacing \(\textsc {E}_{Y,e}[\,L_{\tau ,\tau _{c}}\,]\) in (8.8), we have:
Now let’s focus on the remainder on the right side. First, we only focus on a particular e. Thus, we are interested in:
For ease of notation and comprehension, we suppress e in the notation since there is no confusion. So, we have:
We find:
Let’s focus on the second term on the right side. Using a Taylor series approximation around \(\tau _c \in [0,1]\) and the Taylor–Lagrange formula for the remainder, one has:
with \(\gamma = \tau _c +(Q_{\tau }-\tau _c)\,\theta \), and \(0< \theta < 1\).
And so
Now, one can study the first term on the right side. Some useful remarks before the next: one can easily see that the study of such a function can be restricted to a study on the interval \(I_{y}:= ]-\infty ,G^{-1}(\tau _c)]\), since we can find results on the interval \([G^{-1}(\tau _c),\infty [\) using the same arguments.
Let’s define \(G^{-1}(Q_\tau ) \equiv Z_\tau \), \(G^{-1}_{\tau _c} \equiv G^{-1}({\tau _c})\) and \(f_{Y}^{\,\,G^{-1}_{\tau _c}} \equiv f_{Y}(G^{-1}(\tau _c))\), for ease of notation.
Thus, we are interested in calculating:
However, the function studied in the integral is complicated to work with. So, one will prefer to use its integral version, that is,
For the bounds of the integral, the upper one is obvious. To justify the lower one, it is important to note that \(\lim \,\,\textsc {E}_{Z_\tau } [\,|Z_\tau -y| - Z_\tau \,]+y=0\), \(y\rightarrow -\infty \).
Indeed, one has:
with h and H the p.d.f and the c.d.f of the variable \(Z_t\). If the variable \(Z_\tau \) has a finite mean, \(\lim ,h(y)=0\), \(y\rightarrow -\infty \), and thus it is clear that the choice of \(-\infty \) for the lower bound of the integral is the good one.
At this stage, it is not easy to see the usefulness of the transformation, but it will be after the following calculus:
Finally, we have:
Now, it is clear that this transformation could help us for the calculus of (8.10) since it is equivalent to study:
A difficulty remains, though. Indeed, \(f_{Y}\) in unknown, and in consequence, not easy to work with. That’s why, at first, one will use \(f_{Y}^{\,\,G^{-1}_{\tau _c}}\) for our calculus, and then we will study the impact of such a manipulation.
Let’s start with the first task. Using an integral by part on Half Int:
One obtains:
Since \(\left( \,u\,G^{-1}_{\tau _c}-\frac{u^{2}}{2}\,\right) = \left( \,\frac{(u-G^{-1}_{\tau _c})^{2}}{2}-\frac{(G^{-1}_{\tau _c})^{2}}{2}\right) \), we have:
Now, using the change of variable \(G(u)= z\), a Taylor series approximation around \(\tau _c\) and the Taylor–Lagrange formula, one has the following approximation for Half Int:
with \(\phi \) the p.d.f of the random variable \(Q_{\tau }\). Using the Jensen inequality and since \(0\le z \le \tau _c\), we find:
Since \(\frac{\lambda }{n}\), which is the variance of the random variable \(Q_{\tau }\), is decreasing with n, let’s study:
Since one supports the hypothesis that \(f_{Y}^{'}\) is bounded, using the mean value theorem, one has:
and thus,
Finally, we obtain with the same change of variable and Taylor approximation as previously:
Thus, one has \(\left| \,\textsc {E}_{Y,e}[L_{\,\widehat{\tau }_{c}}-L_{\tau _c}\,]\,\right| \le \,\frac{(C_{int} + C_s)\lambda }{n}\). Since \(C_{int}\) and \(C_s\) do not depend on e, this result remains meaningful when we are interested in the conditional expectation with respect to the random variable E and so \(\left| \,\textsc {E}_Y[L_{\,\widehat{\tau }_{c}}-L_{\tau }\,]\,\right| \le \, 2\,\varepsilon ^{2}\,\xi + \frac{C\,\lambda }{n}\).
Rights and permissions
Copyright information
© 2018 Springer Nature Switzerland AG
About this paper
Cite this paper
Collet, J., Richard, M. (2018). A Generic Method for Density Forecasts Recalibration. In: Drobinski, P., Mougeot, M., Picard, D., Plougonven, R., Tankov, P. (eds) Renewable Energy: Forecasting and Risk Management. FRM 2017. Springer Proceedings in Mathematics & Statistics, vol 254. Springer, Cham. https://doi.org/10.1007/978-3-319-99052-1_8
Download citation
DOI: https://doi.org/10.1007/978-3-319-99052-1_8
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-99051-4
Online ISBN: 978-3-319-99052-1
eBook Packages: Mathematics and StatisticsMathematics and Statistics (R0)