# Attributes of Forecast Quality

## Abstract

Forecast verification is a process used to assess the quality of hydrometeorological ensemble forecasts. This chapter describes the many aspects of forecast quality using a distributions-oriented approach. Using the joint distribution of forecasts and observations, or one of its factorizations into a conditional and marginal distribution, the aspects of forecast quality are defined. Hypothetical ensemble forecasts are then used to illustrate aspects of forecast quality. The hypothetical ensemble forecasts are used to construct single-valued forecasts, probability forecasts for an event, and ensemble probability distribution forecasts. Their forecast quality is then diagnosed using visual comparisons and numerical comparisons of forecast quality measures. The examples illustrate that a single aspect of forecast quality is insufficient and that many aspects are needed to understand the nature of the forecasts. Some practical considerations in the application of the framework to ensemble forecast verification are discussed.

## Keywords

Forecast verification Ensemble forecasts Deterministic forecasts Probabilistic forecasts Distributions-oriented approach## References

- C. Accadia, S. Mariani, M. Casaioli, A. Lavagnini, A. Speranza, Sensitivity of precipitation forecast skill scores to bilinear interpolation and a simple nearest-neighbor average method on high-resolution verification grids. Weather Forecast.
**18**, 918–932 (2003)CrossRefGoogle Scholar - C. Accadia, S. Mariani, M. Casaioli, A. Lavagnini, A. Speranza, Verification of precipitation forecasts from two limited-area models over Italy and comparison with ECMWF forecasts using a resampling technique. Weather Forecast.
**20**, 276–300 (2005)CrossRefGoogle Scholar - F. Atger, Spatial and interannual variability of the reliability of ensemble-based probabilistic forecasts: Consequences for calibration. Monthly Weather Review.
**131**(8), 1509–1523 (2003)CrossRefGoogle Scholar - A.A. Bradley, S.S. Schwartz, Summary verification measures and their interpretation for ensemble forecasts. Mon. Weather Rev.
**139**, 3075–3089 (2011)CrossRefGoogle Scholar - A.A. Bradley, T. Hashino, S.S. Schwartz, Distributions-oriented verification of probability forecasts for small data samples. Weather Forecast.
**18**, 903–917 (2003)CrossRefGoogle Scholar - A.A. Bradley, S.S. Schwartz, T. Hashino, Distributions-oriented verification of ensemble streamflow predictions. J. Hydrometeorol.
**5**, 532–545 (2004)CrossRefGoogle Scholar - A.A. Bradley, S.S. Schwartz, T. Hashino, Sampling uncertainty and confidence intervals for the Brier score and Brier skill score. Weather Forecast.
**23**, 992–1006 (2008)CrossRefGoogle Scholar - G.W. Brier, Verification of forecasts expressed in terms of probability. Mon. Weather Rev.
**78**, 1–3 (1950)CrossRefGoogle Scholar - G.W. Brier, R.A. Allen, Verification of weather forecasts, in
*Compendium of Meteorology*, ed. by T.F. Malone (American Meteorological Society, Boston, 1951), pp. 841–848CrossRefGoogle Scholar - J.D. Brown, D.J. Seo, A nonparametric postprocessor for bias correction of hydrometeorological and hydrologic ensemble forecasts. J. Hydrometeorol.
**11**, 642–665 (2010)CrossRefGoogle Scholar - J.D. Brown, L. Wu, M. He, S. Regonda, H. Lee, D.-J. Seo, Verification of temperature, precipitation, and streamflow forecasts from the NOAA/NWS Hydrologic Ensemble Forecast Service (HEFS): 1. Experimental design and forcing verification. J. Hydrol.
**519**, 2869–2889 (2014)CrossRefGoogle Scholar - J. Demargne, J. Brown, Y.Q. Liu, D.J. Seo, L.M. Wu, Z. Toth, Y.J. Zhu, Diagnostic verification of hydrometeorological and hydrologic ensembles. Atmos. Sci. Lett.
**11**, 114–122 (2010)CrossRefGoogle Scholar - F.J. Doblas-Reyes, V. Pavan, D.B. Stephenson, The skill of multi-model seasonal forecasts of the wintertime North Atlantic Oscillation. Climate Dynam.
**21**, 501–514 (2003)CrossRefGoogle Scholar - E.E. Ebert, L.J. Wilson, B.G. Brown, P. Nurmi, H.E. Brooks, J. Bally, M. Jaeneke, Verification of nowcasts from the WWRP Sydney 2000 forecast demonstration project. Weather Forecast.
**19**, 73–96 (2004)CrossRefGoogle Scholar - B. Efron, Nonparametric estimates of standard error: the jacknife, the bootstrap and other methods. Biometrika
**68**, 589–599 (1981)CrossRefGoogle Scholar - C.A. Ferro, Comparing probabilistic forecasting systems with the Brier score. Weather Forecast.
**22**, 1076–1088 (2007)CrossRefGoogle Scholar - K.J. Franz, H.C. Hartmann, S. Sorooshian, R. Bales, Verification of national weather service ensemble streamflow predictions for water supply forecasting in the Colorado River basin. J. Hydrometeorol.
**4**, 1105–1118 (2003)CrossRefGoogle Scholar - T. Gneiting, A.E. Raftery, A.H. Westveld, T. Goldman, Calibrated probabilistic forecasting using ensemble model output statistics and minimum CRPS estimation. Mon. Weather Rev.
**133**, 1098–1118 (2005)CrossRefGoogle Scholar - T. Gneiting, F. Balabdaoui, A.E. Raftery, Probabilistic forecasts, calibration and sharpness. J. R. Stat. Soc. Ser. B Stat Methodol.
**69**, 243–268 (2007)CrossRefGoogle Scholar - T. Gorgas and M. Dorninger, Quantifying verification uncertainty by reference data variation. Meteorologishe Zeitschrift.
**21**(3), 259–277 (2012)CrossRefGoogle Scholar - T.M. Hamill, Hypothesis tests for evaluating numerical precipitation forecasts. Weather Forecast.
**14**, 155–167 (1999)CrossRefGoogle Scholar - T.M. Hamill, J. Juras, Measuring forecast skill: is it real skill or is it the varying climatology? Q. J. Roy. Meteorol. Soc.
**132**, 2905–2923 (2006)CrossRefGoogle Scholar - T. Hashino, A.A. Bradley, S.S. Schwartz, Evaluation of bias-correction methods for ensemble streamflow volume forecasts. Hydrol. Earth Syst. Sci.
**11**, 939–950 (2007)CrossRefGoogle Scholar - H. Hersbach, Decomposition of the continuous ranked probability score for ensemble prediction systems. Weather Forecast.
**15**, 559–570 (2000)CrossRefGoogle Scholar - S. Jaun, B. Ahrens, Evaluation of a probabilistic hydrometeorological forecast system. Hydrol. Earth Syst. Sci.
**13**, 1031–1043 (2009)CrossRefGoogle Scholar - I. Jolliffe, Uncertainty and inference for verification measures. Weather Forecast.
**22**, 637–650 (2007)CrossRefGoogle Scholar - I.T. Jolliffe, D.B. Stephenson, Introduction, in
*Forecast Verification*, ed. by I.T. Jolliffe, D.B. Stephenson (Wiley, Chichester, 2011), pp. 1–9CrossRefGoogle Scholar - T.L. Kane, B.G. Brown, Confidence intervals for some verification measures – a survey of several methods, in
*Preprints, 15th Conference on Probability and Statistics in the Atmospheric Sciences*(American Meteorologic Society, Asheville, 2000)Google Scholar - R. Krzysztofowicz, The case for probabilistic forecasting in hydrology. J. Hydrol.
**249**, 2–9 (2001)CrossRefGoogle Scholar - S.J. Mason, N.E. Graham, Areas beneath the relative operating characteristics (ROC) and relative operating levels (ROL) curves: statistical significance and interpretation. Q. J. Roy. Meteorol. Soc.
**128**, 2145–2166 (2002)CrossRefGoogle Scholar - S.J. Mason, G.M. Mimmack, The use of bootstrap confidence intervals for the correlation coefficient in climatology. Theor. Appl. Climatol.
**45**, 229–233 (1992)CrossRefGoogle Scholar - J.E. Matheson, R.E. Winkler, Scoring rules for continuous probability distributions. Manag. Sci.
**22**, 1087–1095 (1976)CrossRefGoogle Scholar - A.H. Murphy, Skill scores based on the mean-square error and their relationships to the correlation-coefficient. Mon. Weather Rev.
**116**, 2417–2425 (1988)CrossRefGoogle Scholar - A.H. Murphy, What is a good forecast – an essay on the nature of goodness in weather forecasting. Weather Forecast.
**8**, 281–293 (1993)CrossRefGoogle Scholar - A.H. Murphy, Forecast verification, in
*Economic Value of Weather and Climate Forecasts*, ed. by R.W. Katz, A.H. Murphy (Cambridge University Press, Cambridge, 1997), pp. 19–74CrossRefGoogle Scholar - A.H. Murphy, R.L. Winkler, A general framework for forecast verification. Mon. Weather Rev.
**115**, 1330–1338 (1987)CrossRefGoogle Scholar - A.H. Murphy, R.L. Winkler, Diagnostic verification of probability forecasts. Int. J. Forecast.
**7**, 435–455 (1992)CrossRefGoogle Scholar - D.N. Politis, J.P. Romano, The stationary bootstrap. J. Am. Stat. Assoc.
**89**, 1303–1313 (1994)CrossRefGoogle Scholar - J.M. Potts, Basic concepts, in
*Forecast Verification*, ed. by I.T. Jolliffe, D.B. Stephenson (Wiley, Chichester, 2011), pp. 11–29Google Scholar - D.J. Seo, H.D. Herr, J.C. Schaake, A statistical post-processor for accounting of hydrologic uncertainty in short-range ensemble streamflow prediction. Hydrol. Earth Syst. Sci. Discuss.
**3**, 1987–2035 (2006)CrossRefGoogle Scholar - D.B. Stephenson, Use of the “odds ratio” for diagnosing forecast skill. Weather Forecast.
**15**, 221–232 (2000)CrossRefGoogle Scholar - J.E. Thornes, D.B. Stephenson, How to judge the quality and value of weather forecast products. Meteorol. Appl.
**8**, 307–314 (2001)CrossRefGoogle Scholar - D.A. Unger, A method to estimate the continuous ranked probability score, in
*Nineth Conference on Probability and Statistics in Atmospheric Sciences*(American Meteorological Society, Virginia Beach, 1985), pp. 206–213Google Scholar - E. Welles, S. Sorooshian, G. Carter, B. Olsen, Hydrologic verification – a call for action and collaboration. Bull. Am. Meteorol. Soc.
**88**, 503 (2007)CrossRefGoogle Scholar - D.S. Wilks, Statistical significance of long-range “optimal climate normal” temperature and precipitation forecasts. J. Climate
**9**, 827–839 (1996)CrossRefGoogle Scholar - D.S. Wilks, Sampling distributions of the Brier score and Brier skill score under serial dependence. Q. J. Roy. Meteorol. Soc.
**136**, 2109–2118 (2010)CrossRefGoogle Scholar - D.S. Wilks,
*Statistical Methods in the Atmospheric Sciences*, 3rd edn. (Academic, Amsterdam, 2011), p. 704Google Scholar - H. Zhang, T. Casey, Verification of categorical probability forecasts. Weather Forecast.
**15**, 80–89 (2000)CrossRefGoogle Scholar