Verification Metrics for Hydrological Ensemble Forecasts

  • François AnctilEmail author
  • Maria-Helena Ramos
Living reference work entry


This chapter reviews the most commonly used verification metrics for measuring the performance of hydrological ensemble forecasts. It links metrics to the different attributes of forecast quality and discusses the links between verification variables, metrics, and applications in a broad perspective. It provides an overview of the use of these metrics in forecast evaluation studies and general insights into what forecasters, practitioners, and end-users should consider when applying verification measures in the practice of hydrological ensemble forecasting.


Forecast evaluation Verification metrics Skill Scores Hydrological ensemble forecasts 


  1. M. Abaza, F. Anctil, V. Fortin, R. Turcotte, A comparison of the Canadian global and regional meteorological ensemble prediction systems for short-term hydrological forecasting. Mon. Weather Rev. 141, 3462–3472 (2013). Corrigendum. Mon. Weather Rev. 142, 2561–2562Google Scholar
  2. M. Abaza, F. Anctil, V. Fortin, R. Turcotte, Sequential streamflow assimilation for short-term hydrological ensemble forecasting. J. Hydrol. 519, 2692–2706 (2014)CrossRefGoogle Scholar
  3. L. Alfieri, F. Pappenberger, F. Wetterhall, T. Haiden, D. Richardson, P. Salamon, Evaluation of ensemble streamflow predictions in Europe. J. Hydrol. 517, 913–922 (2014)CrossRefGoogle Scholar
  4. C. Alvarez-Garreton, D. Ryu, A.W. Western, C.-H. Su, W.T. Crow, D.E. Robertson, C. Leahy, Improving operational flood ensemble prediction by the assimilation of satellite soil moisture: comparison between lumped and semi-distributed schemes. Hydrol. Earth Syst. Sci. 19, 1659–1676 (2015)CrossRefGoogle Scholar
  5. F. Anctil, C. Michel, C. Perrin, V. Andréassian, A soil moisture index as an auxiliary ANN input for stream flow forecasting. J. Hydrol. 286, 155–167 (2004)CrossRefGoogle Scholar
  6. J.L. Anderson, A method for producing and evaluating probabilistic forecasts from ensemble model integrations. J. Clim. 9, 1518–1530 (1996)CrossRefGoogle Scholar
  7. D. Anghileri, N. Voisin, A.F. Castelletti, F. Pianosi, B. Nijssen, D.P. Lettenmaier, Value of long-term streamflow forecast to reservoir operations for water supply in snow-dominated catchments. Water Resour. Res. 52(6), 4209–4225 (2016). CrossRefGoogle Scholar
  8. F. Atger, Verification of intense precipitation forecasts from single models and ensemble prediction systems. Nonlinear Process. Geophys. 8, 401–417 (2001)CrossRefGoogle Scholar
  9. L. Baringhaus, C. Franz, On a new multivariate two-sample test. J. Multivar. Anal. 88(1), 190–206 (2004)CrossRefGoogle Scholar
  10. J.C. Bartholmes, J. Thielen, M.H. Ramos, S. Gentilini, The European Flood Alert System EFAS – part 2: statistical skill assessment of probabilistic and deterministic operational forecasts. Hydrol. Earth Syst. Sci. 13(2), 141–153 (2009)CrossRefGoogle Scholar
  11. M.A. Boucher, J.P. Laliberté, F. Anctil, An experiment on the evolution of an ensemble of neural networks for streamflow forecasting. Hydrol. Earth Syst. Sci. 14, 603–612 (2010)CrossRefGoogle Scholar
  12. M.-A. Boucher, D. Tremblay, L. Delorme, L. Perreault, F. Anctil, Hydro-economic assessment of hydrological forecasting systems. J. Hydrol. 416, 133–144 (2012). CrossRefGoogle Scholar
  13. F. Bourgin, M.-H. Ramos, G. Thirel, V. Andréassian, Investigating the interactions between data assimilation and post-processing in hydrological ensemble forecasting. J. Hydrol. 519, 2775–2784 (2014)CrossRefGoogle Scholar
  14. A.A. Bradley, T. Hashino, S.S. Schwartz, Distributions-oriented verification of probability forecasts for small data samples. Weather Forecast. 18, 903–917 (2003)CrossRefGoogle Scholar
  15. A.A. Bradley, J. Demargne, J.J. Franz, Attributes of forecast quality, in Handbook of Hydrometeorological Ensemble Forecasting, ed. by Q. Duan, F. Pappenberger, J. Thielen, A. Wood, H. Cloke, J. Schaake (Springer, Berlin/Heidelberg, 2016), 46p.
  16. D. Brochero, F. Anctil, C. Gagné, Simplifying a hydrological ensemble prediction system with a backward greedy selection of members, part I: optimization criteria. Hydrol. Earth Syst. Sci. 15, 3307–3325 (2011)CrossRefGoogle Scholar
  17. J.D. Brown, J. Demargne, D.J. Seo, Y. Liu, The Ensemble Verification System (EVS): a software tool for verifying ensemble forecasts of hydrometeorological and hydrologic variables at discrete locations. Environ. Model. Softw. 25(7), 854–872 (2010). CrossRefGoogle Scholar
  18. J.D. Brown, M. He, S. Regonda, L. Wu, H. Lee, D.-J. Seo, Verification of temperature, precipitation, and streamflow forecasts from the NOAA/NWS Hydrologic Ensemble Forecast Service (HEFS): 2. Streamflow verification. J. Hydrol. 519, 2847–2868 (2014)CrossRefGoogle Scholar
  19. T.M. Carpenter, K.P. Georgakakos, Impacts of parametric and radar rainfall uncertainty on the ensemble streamflow simulations of a distributed hydrologic model. J. Hydrol. 298, 202–221 (2004)CrossRefGoogle Scholar
  20. B. Casati, L.J. Wilson, D.B. Stephenson, Forecast verification: current status and future directions. Meteorol. Appl. 15(1), 3–18 (2008)CrossRefGoogle Scholar
  21. M.P. Clark, A.G. Slater, Probabilistic quantitative precipitation estimation in complex terrain. J. Hydrometeorol. 7, 3–22 (2006)CrossRefGoogle Scholar
  22. H. Cloke, F. Pappenberger, Evaluating forecasts of extreme events for hydrological applications: an approach for screening unfamiliar performance measures. Meteorol. Appl. 15, 181–197 (2008)CrossRefGoogle Scholar
  23. C. Corradini, F. Melone, L. Ubertini, A semi-distributed adaptive model for real-time flood forecasting. Water Resour. Bull. 22, 1031–1038 (1986)CrossRefGoogle Scholar
  24. L. Crochemore, M.-H. Ramos, F. Pappenberger, Bias correcting precipitation forecasts to improve the skill of seasonal streamflow forecasts. Hydrol. Earth Syst. Sci. 20, 3601–3618 (2016). CrossRefGoogle Scholar
  25. J. Demargne, J.D. Brown, Y. Liu, D.-J. Seo, L. Wu, Z. Toth, Y. Zhu, Diagnostic verification of hydrometeorological and hydrologic ensembles. Atmos. Sci. Lett. 11(2), 114–122 (2010)CrossRefGoogle Scholar
  26. D. Demeritt, S. Nobert, H.L. Cloke, F. Pappenberger, The European Flood Alert System and the communication, perception, and use of ensemble predictions for operational flood risk management. Hydrol. Process. 27, 147–157 (2013). CrossRefGoogle Scholar
  27. K. Engeland, I. Steinsland, Probabilistic postprocessing models for flow forecasts for a system of catchments and several lead times. Water Resour. Res. 50, 182–197 (2014). CrossRefGoogle Scholar
  28. F.M. Fan, W. Collischonn, A. Meller, L.C.M. Botelho, Ensemble streamflow forecasting experiments in a tropical basin: the São Francisco River case study. J. Hydrol. 519, 2906–2919 (2014)CrossRefGoogle Scholar
  29. V. Fortin, M. Abaza, A. Anctil, R. Turcotte, Why should ensemble spread match the RMSE of the ensemble mean? J. Hydrometeorol. 15, 1708–1713 (2014)CrossRefGoogle Scholar
  30. K.J. Franz, T.S. Hogue, M. Barik, Assessment of SWE data assimilation for ensemble streamflow predictions. J. Hydrol. 519(Part D), 2737–2746 (2014)CrossRefGoogle Scholar
  31. T. Gneiting, A.E. Raftery, A.H. Westveld, T. Goldman, Calibrated probabilistic forecasting using ensemble model output statistics and minimum CRPS estimation. Mon. Weather Rev. 133, 1098–1118 (2005). CrossRefGoogle Scholar
  32. T. Gneiting, A.E. Raftery, Strictly proper scoring rules, prediction, and estimation. Journal of the American Statistical Association 102, 359–378 (2007)CrossRefGoogle Scholar
  33. T. Gneiting, F. Balabdaoui, A.E. Raftery, Probabilistic forecasts, calibration and sharpness. J. R. Stat. Soc. Ser. B Stat. Methodol. 69, 243–268 (2007)CrossRefGoogle Scholar
  34. I.J. Good, Rational decisions. J. R. Stat. Soc. 14, 107–114 (1952)Google Scholar
  35. T. Hamill, Interpretation of rank histograms for verifying ensemble forecasts. Mon. Weather Rev. 129, 550–560 (2001)CrossRefGoogle Scholar
  36. T.M. Hamill, S.J. Colucci, Verification of Eta RSM short-range ensemble forecasts. Mon. Weather Rev. 125, 1312–1327 (1997)CrossRefGoogle Scholar
  37. T. Hashino, A.A. Bradley, S.S. Schwartz, Evaluation of bias-correction methods for ensemble streamflow volume forecasts. Hydrol. Earth Syst. Sci. 11, 939–950 (2007)CrossRefGoogle Scholar
  38. H. Hersbach, Decomposition of the continuous ranked probability score for ensemble prediction systems. Weather Forecast. 15, 559–570 (2000)CrossRefGoogle Scholar
  39. H. Holt, J. Pullen, C. Bishop, Urban and ocean ensembles for improved meteorological and dispersion modeling of the coastal zone. Tellus 61A, 232–249 (2009)CrossRefGoogle Scholar
  40. I.T. Jolliffe, D.B. Stephenson, Forecast Verification: A practitioner’s Guide in Atmospheric Science, 2nd edn. (Wiley, New York, 2012). Google Scholar
  41. Y.-O. Kim, H. Eum, E.G. Lee, I.H. Ko, Optimizing operational policies of a Korean multireservoir system using sampling stochastic dynamic programming with ensemble streamflow prediction. J. Water Resour. Plan. Manag. 133, 4–14 (2007). CrossRefGoogle Scholar
  42. P.K. Kitanidis, R.L. Bras, Real-time forecasting with a conceptual hydrologic model. 2. Applications and results. Water Resour. Res. 16(6), 1034–1044 (1980)CrossRefGoogle Scholar
  43. F. Laio, S. Tamea, Verification tools for probabilistic forecasts of continuous hydrological variables. Hydrol. Earth Syst. Sci. 11(4), 1267–1277 (2007)CrossRefGoogle Scholar
  44. K. Liechti, M. Zappa, F. Fundel, U. Germann, The potential of radar-based ensemble forecasts for flash-flood early warning in the southern Swiss Alps. Hydrol. Earth Syst. Sci. 17, 3853–3869 (2013)CrossRefGoogle Scholar
  45. Y. Liu, J.D. Brown, J. Demargne, D.-J. Seo, A wavelet-based approach to assessing timing errors in hydrologic predictions. J. Hydrol. 397(3–4), 210–224 (2011)CrossRefGoogle Scholar
  46. S.J. Mason, A model for assessment of weather forecast. Aust. Meteorol. Mag. 30, 291–303 (1982)Google Scholar
  47. S. Matte, M.-A. Boucher, V. Boucher, T.-C. Fortier Filion, Moving beyond the cost–loss ratio: economic assessment of streamflow forecasts for a risk-averse decision maker. Hydrol. Earth Syst. Sci. 21, 2967–2986 (2017)CrossRefGoogle Scholar
  48. D.N. Moriasi, J.G. Arnold, M.W. Van Liew, R.L. Bingner, R.D. Harmel, T.L. Veith, Model evaluation guidelines for systematic quantification of accuracy in watershed simulations. Trans. ASABE 50, 885–900 (2007)CrossRefGoogle Scholar
  49. A.H. Murphy, A new vector partition of the probability score. J. Appl. Meteorol. 12(4), 595–600 (1973)CrossRefGoogle Scholar
  50. A. Murphy, What is a good forecast? An essay on the nature of goodness in weather forecasting. Weather Forecast. 8(2), 281–293 (1993)CrossRefGoogle Scholar
  51. T. Palmer, R. Buizza, R. Hagedorn, A. Lawrence, M. Leutbecher, L. Smith, Ensemble prediction: a pedagogical perspective. ECMWF Newsl. 106, 10–17 (2005). ECMWF, ReadingGoogle Scholar
  52. F. Pappenberger, K. Scipal, R. Buizza, Hydrological aspects of meteorological verification. Atmos. Sci. Lett. 9, 43–52 (2008)CrossRefGoogle Scholar
  53. F. Pappenberger, M.H. Ramos, H.L. Cloke, F. Wetterhall, L. Alfieri, K. Bogner, A. Mueller, P. Salamon, How do I know if my forecasts are better? Using benchmarks in hydrological ensemble prediction. J. Hydrol. 522, 697–713 (2015)CrossRefGoogle Scholar
  54. W.W. Peterson, T.G. Birdsall, W.C. Fox, The theory of signal detectability. Trans. IRE Prof. Group Inf. Theory 2–4, 171–212 (1954)CrossRefGoogle Scholar
  55. M.H. Ramos, J. Bartholmes, J. Thielen-del Pozo, Development of decision support products based on ensemble forecasts in the European Flood Alert System. Atmos. Sci. Lett. 8, 113–119 (2007). CrossRefGoogle Scholar
  56. M.H. Ramos, T. Mathevet, J. Thielen, F. Pappenberger, Communicating uncertainty in hydro-meteorological forecasts: mission impossible? Meteorol. Appl. 17, 223–235 (2010)CrossRefGoogle Scholar
  57. A. Randrianasolo, M.H. Ramos, G. Thirel, V. Andreassian, E. Martin, Comparing the scores of hydrological ensemble forecasts issued by two different hydrological models. Atmos. Sci. Lett. 11(2), 100–107 (2010)CrossRefGoogle Scholar
  58. B. Renard, D. Kavetski, G. Kuczera, M. Thyer, S.W. Franks, Understanding predictive uncertainty in hydrologic modeling: the challenge of identifying input and structural errors. Water Resour. Res. 46, W05521 (2010)CrossRefGoogle Scholar
  59. D.S. Richardson, Skill and relative economic value of ECMWF ensemble prediction system. Q. J. R. Meteorol. Soc. 126, 649–667 (2000)CrossRefGoogle Scholar
  60. E. Roulin, Skill and relative economic value of medium-range hydrological ensemble predictions. Hydrol. Earth Syst. Sci. 11, 725–737 (2007)CrossRefGoogle Scholar
  61. E. Roulin, S. Vannitsem, Post-processing of medium-range probabilistic hydrological forecasting: impact of forcing, initial conditions and model errors. Hydrol. Process. 29(6), 1434–1449 (2015)CrossRefGoogle Scholar
  62. M.S. Roulston, L.A. Smith, Evaluating probabilistic forecasts using information theory. Mon. Weather Rev. 130, 1653–1660 (2002)CrossRefGoogle Scholar
  63. M.S. Roulston, L.A. Smith, Combining dynamical and statistical ensembles. Tellus A 55, 16–30 (2003). CrossRefGoogle Scholar
  64. D.L. Shrestha, D.E. Robertson, J.C. Bennett, Q.J. Wang, Improving precipitation forecasts by generating ensembles through postprocessing. Mon. Weather Rev. 143, 3642–3663 (2015)CrossRefGoogle Scholar
  65. G. Székely, M. Rizzo, A new test for multivariate normality. J. Multivar. Anal. 1(93), 58–80 (2005)CrossRefGoogle Scholar
  66. O. Talagrand, R. Vautard, B. Strauss, Evaluation of probabilistic prediction systems, in Workshop on Predictability, ed. by for Medium-Range Weather Forecasts, E. C., Shinfield Park, Reading (1997), pp. 1–25Google Scholar
  67. A. Thiboult, F. Anctil, M.-A. Boucher, Accounting for three sources of uncertainty in ensemble hydrological forecasting. Hydrol. Earth Syst. Sci. 20, 1809–1825 (2016)CrossRefGoogle Scholar
  68. A. Thiboult, F. Anctil, M.H. Ramos, How does the quantification of uncertainties affect the quality and value of flood early warning systems? J. Hydrol. 551, 365–373 (2017). CrossRefGoogle Scholar
  69. P. Trambauer, M. Werner, H.C. Winsemius, S. Maskey, E. Dutra, S. Uhlenbrook, Hydrological drought forecasting and skill assessment for the Limpopo River basin, southern Africa. Hydrol. Earth Syst. Sci. 19, 1695–1711 (2015). CrossRefGoogle Scholar
  70. J. Van den Bergh, E. Roulin, Hydrological ensemble prediction and verification for the Meuse and Scheldt basins. Atmos. Sci. Lett. 11, 64–71 (2010). CrossRefGoogle Scholar
  71. J.-A. Velázquez, F. Anctil, C. Perrin, Performance and reliability of multimodel hydrological ensemble simulations based on seventeen lumped models and a thousand catchments. Hydrol. Earth Syst. Sci. 14, 2303–2317 (2010)CrossRefGoogle Scholar
  72. J.S. Verkade, M.G.F. Werner, Estimating the benefits of single value and probability forecasting for flood warning. Hydrol. Earth Syst. Sci. 15, 3751–3765 (2011)CrossRefGoogle Scholar
  73. J.S. Verkade, J.D. Brown, P. Reggiani, A.H. Weerts, Post-processing ECMWF precipitation and temperature ensemble reforecasts for operational hydrologic forecasting at various spatial scales. J. Hydrol. 501, 73–91 (2013)CrossRefGoogle Scholar
  74. N. Voisin, F. Pappenberger, D.P. Lettenmaier, R. Buizza, J.C. Schaake, Application of a medium-range global hydrologic probabilistic forecast scheme to the Ohio River Basin. Weather Forecast. 26, 425–446 (2011)CrossRefGoogle Scholar
  75. A.S. Weigend, S. Shi, Predicting daily probability distributions of S&P500 returns. J. Forecast. 19, 375–392 (2000)CrossRefGoogle Scholar
  76. S.V. Weijs, R. van Nooijen, N. van de Giesen, Kullback–Leibler divergence as a forecast skill score with classic reliability–resolution–uncertainty decomposition. Mon. Weather Rev. 138, 3387–3399 (2010)CrossRefGoogle Scholar
  77. K. Werner, J.S. Verkade, T.C. Pagano, Application of hydrological forecast verification information, in Handbook of Hydrometeorological Ensemble Forecasting, ed. by Q. Duan, F. Pappenberger, J. Thielen, A. Wood, H. Cloke, J. Schaake (Springer, Berlin/Heidelberg, 2016), 22p.
  78. D.S. Wilks, Statistical Methods in the Atmospheric Sciences: An Introduction (Academic, 2011), Amsterdam, 676pGoogle Scholar
  79. A.W. Wood, A. Kumar, D.P. Lettenmaier, A retrospective assessment of National Centers for Environmental Prediction climate model–based ensemble hydrologic forecasting in the western United States. J. Geophys. Res. Atmos. 110, D04105 (2005). CrossRefGoogle Scholar
  80. X. Yuan, J. Roundy, E. Wood, J. Sheffield, Seasonal forecasting of global hydrologic extremes: system development and evaluation over GEWEX basins. Bull. Am. Meteorol. Soc., 1895–1912 (2015).
  81. I. Zalachori, M.H. Ramos, R. Garçon, T. Mathevet, J. Gailhard, Statistical processing of forecasts for hydrological ensemble prediction: a comparative study of different bias correction strategies. Adv. Sci. Res. 8, 135–141 (2012)CrossRefGoogle Scholar
  82. M. Zappa, F. Fundel, S. Jaun, A ‘Peak-Box’ approach for supporting interpretation and verification of operational ensemble peak-flow forecasts. Hydrol. Process. 27(1), 117–131 (2013). CrossRefGoogle Scholar

Copyright information

© Springer-Verlag GmbH Germany 2018

Authors and Affiliations

  1. 1.Département de génie civil et de génie des eauxUniversité LavalQuébecCanada
  2. 2.IRSTEA, UR HBANAntonyFrance

Personalised recommendations