Relationships between potential, attainable, and actual skill in a decadal prediction experiment

  • G. J. Boer
  • W. J. Merryfield
  • V. V. Kharin


A sequence of ensemble forecasts permits the estimation of the “predictable” and “unpredictable” components of the forecasts and hence the predictability of the system. Predictability is a feature of a physical and/or mathematical system which characterizes its “ability to be predicted” as measured, for instance, by the rate at which initially close states separate. Modern climate models may be used to predict climate evolution on seasonal to decadal timescales and they may also be used to estimate predictability. The predictability in this case measures the model’s ability to predict its own evolution and, for a well behaved climate model, this may be a useful measure of the predictability of the climate system itself. Forecast skill, where forecasts are compared to observations, indicates a forecast system’s “ability to predict” the evolution of the climate system. Common measures of skill include correlation, mean square error and mean square skill score. When the model is used to predict its own evolution, rather than the evolution of the actual climate system, the same measures of skill may be used but are then predictability measures and termed measures of “potential skill”. The expectation is that the model’s skill in predicting its own evolution will be greater than its ability to predict the actual evolution of the system in the face of observational uncertainty and model deficiencies. The further expectation is that the differences between potential and actual skill can give an indication of where and to what extent there is the potential for improvement in forecast skill. Predictability studies may be undertaken with available models but it may not be immediately obvious which statistical conditions are necessary if potential skill is to be a reasonable estimate of attainable skill. The purpose of this paper is to investigate and illustrate the spatial and temporal behaviour of the predictable and unpredictable components of surface air temperature in a decadal prediction experiment and to discuss some of the implications for predictability studies and for analyzing model behaviour. Formal relationships between potential, actual, and attainable skill are developed which, however, provide only modest constraints on model results if they are to be used to infer predictability. The results of various scalings of the forecasts on potential and actual skill and their relationship are also considered. Differences in the actual and potential forecast skill between models and/or model versions can be partitioned into components associated with known statistics as a guide to forecast improvement.


Decadal prediction Predictability Skill Potential skill 



We are pleased to acknowledge the important contributions of many members of the CCCma team in the development of the model and the forecasting system that has led to this investigation and to Woo-Sung Lee for her contribution in producing the forecasts.


  1. Boer GJ, Kharin VV, Merryfield WJ (2018) Differences in potential and actual skill in a decadal prediction experiment. Clim Dyn (accepted for publication) Google Scholar
  2. Boer GJ, Lambert SJ (2008) Multi-model decadal potential predictability of precipitation and temperature. Geophys Res Lett.
  3. Boer GJ (2004) Long time-scale potential predictability in an ensemble of coupled climate models. Clim Dyn 23:29–44CrossRefGoogle Scholar
  4. Boer GJ (2009) Climate trends in a seasonal forecasting system. Atmos Ocean 47:123–138. CrossRefGoogle Scholar
  5. Boer GJ et al (2016) The decadal climate prediction project (DCPP) contribution to CMIP6. Geosci Model Dev 9:3751–3777. CrossRefGoogle Scholar
  6. Boer GJ, Kharin VV, Merryfield WJ (2013) Decadal predictability and forecast skill. Clim Dyn 41:1817–1833. CrossRefGoogle Scholar
  7. Deque M (2003) Continuous variables. In: Jolliffe I, Stephenson D (eds) Forecast verification: a practitioner’s guide in atmospheric science. Wiley, Chichester, pp 97–119Google Scholar
  8. Doblas-Reyes FJ, Hagedorn R, Palmer TN, Morcrette JJ (2005) Impact of increasing greenhouse gas concentrations in seasonal ensemble forecasts. Geophys Res Lett 33:L07708. Google Scholar
  9. Eade R, Smith D, Scaife A, Wallace E, Dunstone N, Hermanson L, Robinson N (2014) Do seasonal-to- decadal climate predictions under- estimate the predictability of the real world? Geophys Res Lett.
  10. Eyring V, Bony S, Meehl G, Senior C, Stevens B, Stouffer R, Taylor K (2016) Overview of the Coupled Model Intercomparison Project Phase 6 (CMIP6) experimental design and organization. Geosci Model Dev 9:1937–1958. CrossRefGoogle Scholar
  11. Feng X, DelSole T, Houser P (2013) Comparison of statistical estimates of potential seasonal predictability. J Geophys Res Atmos. 118:6002–6016. CrossRefGoogle Scholar
  12. Flato G et al (2013) Evaluation of climate models. In: Stocker TF, Qin D, Plattner G-K, Tignor M, Allen SK, Boschung J, Nauels A, Xia Y, Bex V, Midgley PM (eds) Climate change 2013: the physical science basis. Contribution of working group I to the fifth assessment report of the intergovernmental panel on climatechange. Cambridge University Press, Cambridge, New YorkGoogle Scholar
  13. Hartmann DL et al (2013) Observations: Atmosphere and Surface. In: Stocker TF, Qin D, Plattner G-K, Tignor M, Allen SK, Boschung J, Nauels A, Xia Y, Bex V, Midgley PM (eds) Climate change 2013: the physical science basis. Contribution of Working Group I to the fifth assessment report of the intergovernmental panel on climate change. Cambridge University Press, Cambridge, New YorkGoogle Scholar
  14. Jolliffe I, Stephenson D (eds) (2003) Forecast verification: a practitioner’s guide in atmospheric science. Wiley, ChichesterGoogle Scholar
  15. Kharin VV, Teng Q, Zwiers FW, Boer GJ, Derome J, Fontecilla JF (2009) Skill assessment of seasonal hindcasts from the Canadian historical forecast project. Atmos Ocean 47:204–223. CrossRefGoogle Scholar
  16. Kharin VV, Merryfield WJ, Boer GJ, Lee WS (2017) A post-processing method for seasonal forecasts using temporally and spatially smoothed statistics. Mon Weather Rev 145:3545–3561. CrossRefGoogle Scholar
  17. Kumar A, Peng P, Chen M (2014) Is there a relationship between potential and actual skill? Mon Weather Rev 142:2220–2227. CrossRefGoogle Scholar
  18. Mailier PJ, Jolliffe IT, Stephenson DB (2006) Quality of weather forecasts: review and recommendations. Royal Meteorological Society Project Report, p 89Google Scholar
  19. Merryfield WJ et al (2013) The Canadian seasonal to interannual prediction system. Part I: models and initialization. Mon Weather Rev 141:2910–2945. CrossRefGoogle Scholar
  20. National Research Council (U.S.) (2010) Assessment of intraseasonal to interannual climate prediction and predictability. National Academies Press, Washington, D.CGoogle Scholar
  21. Palmer TN, Buizza R, Hagedorn R, Lawrence A, Leutbecher M, Smith L (2006) Ensemble prediction: a pedagogical perspective. ECMWF Newslett 106:10–17Google Scholar
  22. Pohlmann H, Botzet M, Latif M, Rioesch A, Wild M, Schuck P (2004) Estimating the decadal predictability of a coupled AOGCM. J Clim 17:4463–4472. CrossRefGoogle Scholar
  23. Scaife AA et al (2014) Skillful long-range prediction of European and North American winters. Geophys Res Lett 41:2514–2519. CrossRefGoogle Scholar
  24. Stockdale TN, Molteni F, Ferranti L (2015) Atmospheric initial conditions and the predictability of the Arctic Oscillation. Geophys Res Lett 42:1173–1179. CrossRefGoogle Scholar
  25. Storch vH, Zwiers FW (1999) Statistical analysis in climate research. Cambridge University Press, CambridgeCrossRefGoogle Scholar
  26. Toth Z, Talagrand O, Candille G, Zhu Y, (2003) Chapter 7, Probability and ensemble forecasts. In: Forecast verification: a practitioner’s guide in atmospheric science. Wiley, Chichester, p 240Google Scholar
  27. Weisheimer A, Doblas-Reyes FJ, Palmer TN, Alessandri A, Arribas A, Deque M, Keenlyside N, MacVean M, Navarra A, Rogel P (2009) ENSEMBLES—a new multi-model ensemble for seasonal-to-annual predictions: skill and progress beyond DEMETER in forecasting tropical Pacific SST. Geophys Res Lett 36:L21711. CrossRefGoogle Scholar

Copyright information

© Crown 2018

Authors and Affiliations

  1. 1.Canadian Centre for Climate Modelling and AnalysisVictoriaCanada

Personalised recommendations