Abstract
Given a collection of science-based computational models that all estimate states of the same environmental system, we compare the forecast skill of the average of the collection to the skills of the individual members. We illustrate our results through an analysis of regional climate model data and give general criteria for the average to perform more or less skillfully than the most skillful individual model, the “best” model. The average will only be more skillful than the best model if the individual models in the collection produce very different forecasts; if the individual forecasts generally agree, the average will not be as skillful as the best model.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Beven K (2006) A manifesto for the equifinality thesis. J Hydrol 320:18–36
Boer GJ (2004) Long time-scale potential predictability in an ensemble of coupled climate models. Clim Dyn 23:29–44
Christakos G (2003) Another look at the conceptual fundamentals of porous media upscaling. Stoch Env Res Risk Assess 17(5):276–290
Epstein ES (1969) A scoring system for probability forecasts of ranked categories. J Appl Meteorol 8:985–987
Epstein ES, Murphy AH (1989) Skill scores and correlation coefficients in model verification. Mon Weather Rev 117:572–581
Giorgi F, Mearns LO (2002) Calculation of average, uncertainty range, and reliability of regional climate changes from AOGCM simulations via the “reliability ensemble averaging” (REA) method. J Clim 15:1141–1158
Gomez-Hernandez JJ, Wen X-H (1998) To be or not to be multi-Gaussian? A reflection on stochastic hydrogeology. Adv Water Resour 21(1):47–61
Guadagnini A, Guadagnini L, Tartakovsky DM, Winter CL (2003) Random domain decomposition for flow in heterogeneous stratified aquifer. Stoch Env Res Risk Assess 17(6):394–407
Jones PD, New M, Parker DE, Martin S, Rigor IG (1999) Surface air temperature and its variations over the last 150 years. Rev Geophys 37(173):173–199
Jun M, Knutti R, Nychka DW (2008) Spatial analysis to quantify numerical model bias and dependence: how many climate models are there? J Am Stat Assoc 103(483):934–947
Knutti R, Jun M, Nychka DW (2008) Local eigenvalue analysis of CMIP3 climate model errors. Tellus A 60(5):992–1000
Latif M, Collins M, Pohlmann H, Keenlyside N (2006) A review of predictability studies of Atlantic sector climate on decadal time scales. J Clim 19:5971–5987
Meehl GA, Stocker TF, Collins WD, Friedlingstein P, Gaye AT, Gregory JM, Kitoh A, Knutti R, Murphy JM, Noda A, Raper SCB, Watterson IG, Weaver AJ, Zhao ZC (2007) Global climate projections. In: Climate change 2007: the physical science basis. Contribution of working group I to the fourth assessment report of the intergovernmental panel on climate change. Cambridge University Press, Cambridge
Murphy AH (1971) A note on the ranked probability score. J Appl Meteorol 10:155–156
Murphy AH (1989) Skill scores based on the mean-square error and their relationships to the correlation coefficient. Mon Weather Rev 116:2417–2424
Murphy AH (1996) General decompositions of MSE based skill scores: measures of some basic aspects of forecast quality. Mon Weather Rev 124:2353–2369
Neuman SP (2003) Maximum likelihood Bayesian averaging of uncertain model predictions. Stoch Env Res Risk Assess 17(5):291–305
Neuman SP, Wierenga PJ (2003) A comprehensive strategy of hydrogeologic modeling and uncertainty analysis for nuclear facilities and sites (NUREG/CR-6805). Office of Nuclear Regulatory Research, U.S. Nuclear Regulatory Commission, Washington, DC, July
Palmer TN, Alessandri A, Andersen U, Cantelaube P, Davey M, Deleclue P, Deque M, Diez E, Doblas-Reyes J, Feddersen H, Graham R, Gualdi S, Gueremy J-F, Hagedorn R, Hoshen M, Keenlyside H, Latif M, Lazar A, Maisonnave E, Marletto V, Morse AP, Orfila B, Rogel P, Terres J-M, Thomson MC (2004) Development of a European multimodel ensemble system for seasonal-to-interannual prediction (DEMETER). Bull Am Meteorol Soc 6:853–872
Peirce CS (1884) The numerical measure of the success of predictions. Science 4:453–454
Poeter E, Anderson D (2005) Multimodel ranking and inference in ground water modeling. Ground Water 43(4):597–605
Rubin Y (1995) Flow and transport in bimodal heterogeneous formations. Water Resour Res 31(10):2461–2468
Swets JA (1973) The relative operating characteristic in psychology. Science 182:990–1000
Tebaldi C, Knutti R (2007) The use of the multi-model ensemble in probabilistic climate projections. Philos Trans R Soc A 365(1857):2053–2075
Tebaldi C, Smith RL, Nychka D, Mearns LO (2005) Quantifying uncertainty in projections of regional climate change: a Bayesian approach to the analysis of multimodel ensembles. J Clim 18(10):1524–1540
Thornes JE, Stephenson DB (2001) How to judge the quality and value of weather forecast products. Meteorol Appl 8:307–314
Wilks DS (2005) Statistical methods in the atmospheric sciences, 2nd edn. Academic Press, New York
Winter CL, Tartakovsky DM (2000) Mean flow in composite porous media. Geophys Res Lett 27(12):1759–1762
Winter CL, Tartakovsky DM (2002) Groundwater flow in heterogeneous composite aquifers. Water Resour Res 38(8):1148
Ye M, Neuman SP, Meyer PD (2004) Maximum likelihood Bayesian averaging of spatial variability models in unsaturated fractured tuff. Water Resour Res 40(5):W05113
Ye M, Neuman SP, Meyer PD, Pohlmann KF (2005) Sensitivity analysis and assessment of prior model probabilities in MLBMA with application to unsaturated fractured tuff. Water Resour Res 41(12):W12429
Ye M, Meyer PD, Neuman SP (2008) On model selection criteria in multimodel analysis. Water Resour Res 44(3):W03428
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Winter, C.L., Nychka, D. Forecasting skill of model averages. Stoch Environ Res Risk Assess 24, 633–638 (2010). https://doi.org/10.1007/s00477-009-0350-y
Published:
Issue Date:
DOI: https://doi.org/10.1007/s00477-009-0350-y