Differences in potential and actual skill in a decadal prediction experiment

  • G. J. Boer
  • V. V. Kharin
  • W. J. Merryfield


Decadal prediction results are analyzed for the predictability and skill of annual mean temperature. Forecast skill is assessed in terms of correlation, mean square error (MSE) and mean square skill score. The predictability of the forecast system is assessed by calculating the corresponding “potential” skill measures based on the behaviour of the forecast ensemble. The expectation is that potential skill, where the model predicts its own evolution, will be greater than the actual skill, where the model predicts the evolution of the real system, and that the difference is an indication of the potential for forecast improvement. This will depend, however, on the agreement of the second order climate statistics of the forecasts with those of the climate system. In this study the forecast variance differs from the variance of the verifying observations over non-trivial parts of the globe. Observation-based values of variance from different sources also differ non-trivially. This is an area of difficulty independent of the forecasting system and also affects the comparison of actual and potential mean square error. It is possible to scale the forecast variance estimate to match that of the verifying data so as to avoid this consequence but a variance mismatch, whatever its source, remains a difficulty when considering forecast system improvements. Maps of actual and potential correlation indicate that over most of the globe potential correlation is greater than actual correlation, as expected, with the difference suggesting, but not demonstrating, that it might be possible to improve skill. There are exceptions, mainly over some land areas in the Northern Hemisphere and at later forecast ranges, where actual correlation can exceed potential correlation, and this behaviour is ascribed to excessive noise variance in the forecasts, at least as compared to the verifying data. Sampling error can also play a role, but significance testing suggests it is not sufficient to explain the results. Similar results are obtained for MSE but only after scaling the forecasts to match the variance of the verifying observations. It is immediately clear that the forecast system is deficient, independent of other considerations, if the actual correlation is greater than the potential correlation and/or the actual MSE is less than the potential MSE and this gives some indication of the nature of the deficiency in the forecasts in these regions. The predictable and noise components of an ensemble of forecasts can be estimated but this is not the case for the actual system. The degree to which the difference between actual and potential skill indicates the potential for improvement of the forecasting can only be judged indirectly. At a minimum the variances of the forecasts and of the verifying data should be in reasonable accord. If the potential skill is greater than the actual skill for a forecasting system based on a well behaved model it suggests, as a working hypothesis, that forecast skill can be improved so as to more closely approach potential skill.


Decadal prediction Predictability Skill Potential skill 



We acknowledge the important contributions of many members of the CCCma team in developing the model and the forecasting system Woo-Sung Lee for her contribution in producing the forecasts.


  1. Baker LH, Shaffrey LC, Sutton RT, Weisheimer A, Scaife AA (2018) An intercomparison of skill and over/underconfidence of the wintertime North Atlantic Oscillation in multi-model seasonal forecasts. Res Letts Geophys. CrossRefGoogle Scholar
  2. Boer GJ, Merryfield WJ, Kharin VV (2018) Relationships between potential, attainable, and actual skill in a decadal prediction experiment. Clim Dyn. CrossRefGoogle Scholar
  3. Boer GJ, Kharin VV, Merryfield WJ (2013) Decadal predictability and forecast skill. Clim Dyn 41:1817–1833. CrossRefGoogle Scholar
  4. Boer GJ (2009) Climate trends in a seasonal forecasting system. Atmos-Ocean 47:123–138. CrossRefGoogle Scholar
  5. Boer GJ, Lambert SJ (2008) Multi-model decadal potential predictability of precipitation and temperature. Geophys Res Lett.
  6. Boer GJ (2004) Long time-scale potential predictability in an ensemble of coupled climate models. Clim Dyn 23:29–44CrossRefGoogle Scholar
  7. Boer GJ, Lambert SJ (2001) Second order space-time climate difference statistics. Clim Dyn 17:213–218CrossRefGoogle Scholar
  8. Dee DP (2011) The ERA-interim reanalysis: configuration and performance of the data assimilation system. Q J R Meteorol Soc 137:553–597CrossRefGoogle Scholar
  9. Deque M (2003) Continuous Variables. In: Jolliffe IT, Stephenson DB (eds) 2003: Forecast verification: a practitioner’s guide in atmospheric science. Wiley, Chichester, p 240Google Scholar
  10. Dunstone NJ, Smith DM, Scaife AA, Hermanson L, Eade R, Robinson N, Andrews M, Knight J (2016) Skilful predictions of the winter North Atlantic Oscillation one year ahead. Nat Geosci. CrossRefGoogle Scholar
  11. Eade R, Smith D, Scaife A, Wallace E, Dunstone N, Hermanson L, Robinson N (2014) Do seasonal-to-decadal climate predictions under-estimate the predictability of the real world? Geophys Res Lett. CrossRefGoogle Scholar
  12. Hansen J, Ruedy R, Sato M, Lo K (2010) Global surface temperature change. Rev Geophys 48:RG4004. CrossRefGoogle Scholar
  13. Jin Y, Rong X, Liu Z (2017) Potential predictability and forecast skill in ensemble climate forecast: a skill-persistence rule. Clim Dyn. CrossRefGoogle Scholar
  14. Kumar A, Peng P, Chen M (2014) Is there a relationship between potential and actual skill? Mon Weather Rev 142:2220–2227. CrossRefGoogle Scholar
  15. Pitman JG (1939) A note on normal correlation. Biometrika 31:9–12CrossRefGoogle Scholar
  16. Pohlmann H, Botzet M, Latif M, Roesch A, Wild M, Tschuck P (2004) Estimating the Decadal Predictability of a Coupled AOGCM. J Clim 17:4463–4472. CrossRefGoogle Scholar
  17. Scaife AA (2014) Skillful long- range prediction of European and North American winters. Geophys Res Lett 41:2514–2519. CrossRefGoogle Scholar
  18. Scaife AA, Smith DM (2018) A signal-to-noise paradox in climate science. Nat Clim Atmos Sci.
  19. Taylor KE (2001) Summarizing multiple aspects of model performance in a single diagram. J Geophys Res 106:7183–7192CrossRefGoogle Scholar
  20. Merryfield WJ et al (2013) The Canadian seasonal to interannual prediction system. Part I: Models and initialization. Mon Weather Rev 141:2910–2945. CrossRefGoogle Scholar
  21. Stockdale TN, Molteni F, Ferranti L (2015) Atmospheric initial conditions and the predictability of the Arctic Oscillation. Geophys Res Lett 42:1173–1179. CrossRefGoogle Scholar
  22. Uppala SM (2005) The ERA-40 re-analysis. Q J R Meteorol Soc 131:2961–3012CrossRefGoogle Scholar

Copyright information

© © Crown 2018

Authors and Affiliations

  1. 1.Canadian Centre for Climate Modelling and AnalysisVictoriaCanada

Personalised recommendations