Attributes of Forecast Quality

  • A. Allen BradleyEmail author
  • Julie DemargneEmail author
  • Kristie J. FranzEmail author
Reference work entry


Forecast verification is a process used to assess the quality of hydrometeorological ensemble forecasts. This chapter describes the many aspects of forecast quality using a distributions-oriented approach. Using the joint distribution of forecasts and observations, or one of its factorizations into a conditional and marginal distribution, the aspects of forecast quality are defined. Hypothetical ensemble forecasts are then used to illustrate aspects of forecast quality. The hypothetical ensemble forecasts are used to construct single-valued forecasts, probability forecasts for an event, and ensemble probability distribution forecasts. Their forecast quality is then diagnosed using visual comparisons and numerical comparisons of forecast quality measures. The examples illustrate that a single aspect of forecast quality is insufficient and that many aspects are needed to understand the nature of the forecasts. Some practical considerations in the application of the framework to ensemble forecast verification are discussed.


Forecast verification Ensemble forecasts Deterministic forecasts Probabilistic forecasts Distributions-oriented approach 


  1. C. Accadia, S. Mariani, M. Casaioli, A. Lavagnini, A. Speranza, Sensitivity of precipitation forecast skill scores to bilinear interpolation and a simple nearest-neighbor average method on high-resolution verification grids. Weather Forecast. 18, 918–932 (2003)CrossRefGoogle Scholar
  2. C. Accadia, S. Mariani, M. Casaioli, A. Lavagnini, A. Speranza, Verification of precipitation forecasts from two limited-area models over Italy and comparison with ECMWF forecasts using a resampling technique. Weather Forecast. 20, 276–300 (2005)CrossRefGoogle Scholar
  3. F. Atger, Spatial and interannual variability of the reliability of ensemble-based probabilistic forecasts: Consequences for calibration. Monthly Weather Review. 131(8), 1509–1523 (2003)CrossRefGoogle Scholar
  4. A.A. Bradley, S.S. Schwartz, Summary verification measures and their interpretation for ensemble forecasts. Mon. Weather Rev. 139, 3075–3089 (2011)CrossRefGoogle Scholar
  5. A.A. Bradley, T. Hashino, S.S. Schwartz, Distributions-oriented verification of probability forecasts for small data samples. Weather Forecast. 18, 903–917 (2003)CrossRefGoogle Scholar
  6. A.A. Bradley, S.S. Schwartz, T. Hashino, Distributions-oriented verification of ensemble streamflow predictions. J. Hydrometeorol. 5, 532–545 (2004)CrossRefGoogle Scholar
  7. A.A. Bradley, S.S. Schwartz, T. Hashino, Sampling uncertainty and confidence intervals for the Brier score and Brier skill score. Weather Forecast. 23, 992–1006 (2008)CrossRefGoogle Scholar
  8. G.W. Brier, Verification of forecasts expressed in terms of probability. Mon. Weather Rev. 78, 1–3 (1950)CrossRefGoogle Scholar
  9. G.W. Brier, R.A. Allen, Verification of weather forecasts, in Compendium of Meteorology, ed. by T.F. Malone (American Meteorological Society, Boston, 1951), pp. 841–848CrossRefGoogle Scholar
  10. J.D. Brown, D.J. Seo, A nonparametric postprocessor for bias correction of hydrometeorological and hydrologic ensemble forecasts. J. Hydrometeorol. 11, 642–665 (2010)CrossRefGoogle Scholar
  11. J.D. Brown, L. Wu, M. He, S. Regonda, H. Lee, D.-J. Seo, Verification of temperature, precipitation, and streamflow forecasts from the NOAA/NWS Hydrologic Ensemble Forecast Service (HEFS): 1. Experimental design and forcing verification. J. Hydrol. 519, 2869–2889 (2014)CrossRefGoogle Scholar
  12. J. Demargne, J. Brown, Y.Q. Liu, D.J. Seo, L.M. Wu, Z. Toth, Y.J. Zhu, Diagnostic verification of hydrometeorological and hydrologic ensembles. Atmos. Sci. Lett. 11, 114–122 (2010)CrossRefGoogle Scholar
  13. F.J. Doblas-Reyes, V. Pavan, D.B. Stephenson, The skill of multi-model seasonal forecasts of the wintertime North Atlantic Oscillation. Climate Dynam. 21, 501–514 (2003)CrossRefGoogle Scholar
  14. E.E. Ebert, L.J. Wilson, B.G. Brown, P. Nurmi, H.E. Brooks, J. Bally, M. Jaeneke, Verification of nowcasts from the WWRP Sydney 2000 forecast demonstration project. Weather Forecast. 19, 73–96 (2004)CrossRefGoogle Scholar
  15. B. Efron, Nonparametric estimates of standard error: the jacknife, the bootstrap and other methods. Biometrika 68, 589–599 (1981)CrossRefGoogle Scholar
  16. C.A. Ferro, Comparing probabilistic forecasting systems with the Brier score. Weather Forecast. 22, 1076–1088 (2007)CrossRefGoogle Scholar
  17. K.J. Franz, H.C. Hartmann, S. Sorooshian, R. Bales, Verification of national weather service ensemble streamflow predictions for water supply forecasting in the Colorado River basin. J. Hydrometeorol. 4, 1105–1118 (2003)CrossRefGoogle Scholar
  18. T. Gneiting, A.E. Raftery, A.H. Westveld, T. Goldman, Calibrated probabilistic forecasting using ensemble model output statistics and minimum CRPS estimation. Mon. Weather Rev. 133, 1098–1118 (2005)CrossRefGoogle Scholar
  19. T. Gneiting, F. Balabdaoui, A.E. Raftery, Probabilistic forecasts, calibration and sharpness. J. R. Stat. Soc. Ser. B Stat Methodol. 69, 243–268 (2007)CrossRefGoogle Scholar
  20. T. Gorgas and M. Dorninger, Quantifying verification uncertainty by reference data variation. Meteorologishe Zeitschrift. 21(3), 259–277 (2012)CrossRefGoogle Scholar
  21. T.M. Hamill, Hypothesis tests for evaluating numerical precipitation forecasts. Weather Forecast. 14, 155–167 (1999)CrossRefGoogle Scholar
  22. T.M. Hamill, J. Juras, Measuring forecast skill: is it real skill or is it the varying climatology? Q. J. Roy. Meteorol. Soc. 132, 2905–2923 (2006)CrossRefGoogle Scholar
  23. T. Hashino, A.A. Bradley, S.S. Schwartz, Evaluation of bias-correction methods for ensemble streamflow volume forecasts. Hydrol. Earth Syst. Sci. 11, 939–950 (2007)CrossRefGoogle Scholar
  24. H. Hersbach, Decomposition of the continuous ranked probability score for ensemble prediction systems. Weather Forecast. 15, 559–570 (2000)CrossRefGoogle Scholar
  25. S. Jaun, B. Ahrens, Evaluation of a probabilistic hydrometeorological forecast system. Hydrol. Earth Syst. Sci. 13, 1031–1043 (2009)CrossRefGoogle Scholar
  26. I. Jolliffe, Uncertainty and inference for verification measures. Weather Forecast. 22, 637–650 (2007)CrossRefGoogle Scholar
  27. I.T. Jolliffe, D.B. Stephenson, Introduction, in Forecast Verification, ed. by I.T. Jolliffe, D.B. Stephenson (Wiley, Chichester, 2011), pp. 1–9CrossRefGoogle Scholar
  28. T.L. Kane, B.G. Brown, Confidence intervals for some verification measures – a survey of several methods, in Preprints, 15th Conference on Probability and Statistics in the Atmospheric Sciences (American Meteorologic Society, Asheville, 2000)Google Scholar
  29. R. Krzysztofowicz, The case for probabilistic forecasting in hydrology. J. Hydrol. 249, 2–9 (2001)CrossRefGoogle Scholar
  30. S.J. Mason, N.E. Graham, Areas beneath the relative operating characteristics (ROC) and relative operating levels (ROL) curves: statistical significance and interpretation. Q. J. Roy. Meteorol. Soc. 128, 2145–2166 (2002)CrossRefGoogle Scholar
  31. S.J. Mason, G.M. Mimmack, The use of bootstrap confidence intervals for the correlation coefficient in climatology. Theor. Appl. Climatol. 45, 229–233 (1992)CrossRefGoogle Scholar
  32. J.E. Matheson, R.E. Winkler, Scoring rules for continuous probability distributions. Manag. Sci. 22, 1087–1095 (1976)CrossRefGoogle Scholar
  33. A.H. Murphy, Skill scores based on the mean-square error and their relationships to the correlation-coefficient. Mon. Weather Rev. 116, 2417–2425 (1988)CrossRefGoogle Scholar
  34. A.H. Murphy, What is a good forecast – an essay on the nature of goodness in weather forecasting. Weather Forecast. 8, 281–293 (1993)CrossRefGoogle Scholar
  35. A.H. Murphy, Forecast verification, in Economic Value of Weather and Climate Forecasts, ed. by R.W. Katz, A.H. Murphy (Cambridge University Press, Cambridge, 1997), pp. 19–74CrossRefGoogle Scholar
  36. A.H. Murphy, R.L. Winkler, A general framework for forecast verification. Mon. Weather Rev. 115, 1330–1338 (1987)CrossRefGoogle Scholar
  37. A.H. Murphy, R.L. Winkler, Diagnostic verification of probability forecasts. Int. J. Forecast. 7, 435–455 (1992)CrossRefGoogle Scholar
  38. D.N. Politis, J.P. Romano, The stationary bootstrap. J. Am. Stat. Assoc. 89, 1303–1313 (1994)CrossRefGoogle Scholar
  39. J.M. Potts, Basic concepts, in Forecast Verification, ed. by I.T. Jolliffe, D.B. Stephenson (Wiley, Chichester, 2011), pp. 11–29Google Scholar
  40. D.J. Seo, H.D. Herr, J.C. Schaake, A statistical post-processor for accounting of hydrologic uncertainty in short-range ensemble streamflow prediction. Hydrol. Earth Syst. Sci. Discuss. 3, 1987–2035 (2006)CrossRefGoogle Scholar
  41. D.B. Stephenson, Use of the “odds ratio” for diagnosing forecast skill. Weather Forecast. 15, 221–232 (2000)CrossRefGoogle Scholar
  42. J.E. Thornes, D.B. Stephenson, How to judge the quality and value of weather forecast products. Meteorol. Appl. 8, 307–314 (2001)CrossRefGoogle Scholar
  43. D.A. Unger, A method to estimate the continuous ranked probability score, in Nineth Conference on Probability and Statistics in Atmospheric Sciences (American Meteorological Society, Virginia Beach, 1985), pp. 206–213Google Scholar
  44. E. Welles, S. Sorooshian, G. Carter, B. Olsen, Hydrologic verification – a call for action and collaboration. Bull. Am. Meteorol. Soc. 88, 503 (2007)CrossRefGoogle Scholar
  45. D.S. Wilks, Statistical significance of long-range “optimal climate normal” temperature and precipitation forecasts. J. Climate 9, 827–839 (1996)CrossRefGoogle Scholar
  46. D.S. Wilks, Sampling distributions of the Brier score and Brier skill score under serial dependence. Q. J. Roy. Meteorol. Soc. 136, 2109–2118 (2010)CrossRefGoogle Scholar
  47. D.S. Wilks, Statistical Methods in the Atmospheric Sciences, 3rd edn. (Academic, Amsterdam, 2011), p. 704Google Scholar
  48. H. Zhang, T. Casey, Verification of categorical probability forecasts. Weather Forecast. 15, 80–89 (2000)CrossRefGoogle Scholar

Copyright information

© Springer-Verlag GmbH Germany, part of Springer Nature 2019

Authors and Affiliations

  1. 1.IIHR–Hydroscience and EngineeringThe University of IowaIowa CityUSA
  2. 2.HYDRIS HydrologieSaint Mathieu de TréviersFrance
  3. 3.Department of Geological and Atmospheric SciencesIowa State UniversityAmesUSA

Personalised recommendations