# Differences in potential and actual skill in a decadal prediction experiment

## Abstract

Decadal prediction results are analyzed for the predictability and skill of annual mean temperature. Forecast skill is assessed in terms of correlation, mean square error (MSE) and mean square skill score. The predictability of the forecast system is assessed by calculating the corresponding “potential” skill measures based on the behaviour of the forecast ensemble. The expectation is that potential skill, where the model predicts its own evolution, will be greater than the actual skill, where the model predicts the evolution of the real system, and that the difference is an indication of the potential for forecast improvement. This will depend, however, on the agreement of the second order climate statistics of the forecasts with those of the climate system. In this study the forecast variance differs from the variance of the verifying observations over non-trivial parts of the globe. Observation-based values of variance from different sources also differ non-trivially. This is an area of difficulty independent of the forecasting system and also affects the comparison of actual and potential mean square error. It is possible to scale the forecast variance estimate to match that of the verifying data so as to avoid this consequence but a variance mismatch, whatever its source, remains a difficulty when considering forecast system improvements. Maps of actual and potential correlation indicate that over most of the globe potential correlation is greater than actual correlation, as expected, with the difference suggesting, but not demonstrating, that it might be possible to improve skill. There are exceptions, mainly over some land areas in the Northern Hemisphere and at later forecast ranges, where actual correlation can exceed potential correlation, and this behaviour is ascribed to excessive noise variance in the forecasts, at least as compared to the verifying data. Sampling error can also play a role, but significance testing suggests it is not sufficient to explain the results. Similar results are obtained for MSE but only after scaling the forecasts to match the variance of the verifying observations. It is immediately clear that the forecast system is deficient, independent of other considerations, if the actual correlation is greater than the potential correlation and/or the actual MSE is less than the potential MSE and this gives some indication of the nature of the deficiency in the forecasts in these regions. The predictable and noise components of an ensemble of forecasts can be estimated but this is not the case for the actual system. The degree to which the difference between actual and potential skill indicates the potential for improvement of the forecasting can only be judged indirectly. At a minimum the variances of the forecasts and of the verifying data should be in reasonable accord. If the potential skill is greater than the actual skill for a forecasting system based on a well behaved model it suggests, as a working hypothesis, that forecast skill can be improved so as to more closely approach potential skill.

## Keywords

Decadal prediction Predictability Skill Potential skill## Notes

### Acknowledgements

We acknowledge the important contributions of many members of the CCCma team in developing the model and the forecasting system Woo-Sung Lee for her contribution in producing the forecasts.

## References

- Baker LH, Shaffrey LC, Sutton RT, Weisheimer A, Scaife AA (2018) An intercomparison of skill and over/underconfidence of the wintertime North Atlantic Oscillation in multi-model seasonal forecasts. Res Letts Geophys. https://doi.org/10.1029/2018GL078838 CrossRefGoogle Scholar
- Boer GJ, Merryfield WJ, Kharin VV (2018) Relationships between potential, attainable, and actual skill in a decadal prediction experiment. Clim Dyn. https://doi.org/10.1007/s00382-018-4417-7 CrossRefGoogle Scholar
- Boer GJ, Kharin VV, Merryfield WJ (2013) Decadal predictability and forecast skill. Clim Dyn 41:1817–1833. https://doi.org/10.1007/s00382-013-1705-0 CrossRefGoogle Scholar
- Boer GJ (2009) Climate trends in a seasonal forecasting system. Atmos-Ocean 47:123–138. https://doi.org/10.3137/AO1002.2009 CrossRefGoogle Scholar
- Boer GJ, Lambert SJ (2008) Multi-model decadal potential predictability of precipitation and temperature. Geophys Res Lett. https://doi.org/10.1029/2008GL033234
- Boer GJ (2004) Long time-scale potential predictability in an ensemble of coupled climate models. Clim Dyn 23:29–44CrossRefGoogle Scholar
- Boer GJ, Lambert SJ (2001) Second order space-time climate difference statistics. Clim Dyn 17:213–218CrossRefGoogle Scholar
- Dee DP (2011) The ERA-interim reanalysis: configuration and performance of the data assimilation system. Q J R Meteorol Soc 137:553–597CrossRefGoogle Scholar
- Deque M (2003) Continuous Variables. In: Jolliffe IT, Stephenson DB (eds) 2003: Forecast verification: a practitioner’s guide in atmospheric science. Wiley, Chichester, p 240Google Scholar
- Dunstone NJ, Smith DM, Scaife AA, Hermanson L, Eade R, Robinson N, Andrews M, Knight J (2016) Skilful predictions of the winter North Atlantic Oscillation one year ahead. Nat Geosci. https://doi.org/10.1038/NGEO2824 CrossRefGoogle Scholar
- Eade R, Smith D, Scaife A, Wallace E, Dunstone N, Hermanson L, Robinson N (2014) Do seasonal-to-decadal climate predictions under-estimate the predictability of the real world? Geophys Res Lett. https://doi.org/10.1002/2014GL061146 CrossRefGoogle Scholar
- Hansen J, Ruedy R, Sato M, Lo K (2010) Global surface temperature change. Rev Geophys 48:RG4004. https://doi.org/10.1029/2010RG000345 CrossRefGoogle Scholar
- Jin Y, Rong X, Liu Z (2017) Potential predictability and forecast skill in ensemble climate forecast: a skill-persistence rule. Clim Dyn. https://doi.org/10.1007/s00382-017-4040-z CrossRefGoogle Scholar
- Kumar A, Peng P, Chen M (2014) Is there a relationship between potential and actual skill? Mon Weather Rev 142:2220–2227. https://doi.org/10.1175/MWR-D-13-00287.1 CrossRefGoogle Scholar
- Pitman JG (1939) A note on normal correlation. Biometrika 31:9–12CrossRefGoogle Scholar
- Pohlmann H, Botzet M, Latif M, Roesch A, Wild M, Tschuck P (2004) Estimating the Decadal Predictability of a Coupled AOGCM. J Clim 17:4463–4472. https://doi.org/10.1175/3209.1 CrossRefGoogle Scholar
- Scaife AA (2014) Skillful long- range prediction of European and North American winters. Geophys Res Lett 41:2514–2519. https://doi.org/10.1002/2014GL059637 CrossRefGoogle Scholar
- Scaife AA, Smith DM (2018) A signal-to-noise paradox in climate science. Nat Clim Atmos Sci. https://doi.org/10.1038/s41612-018-0038-4
- Taylor KE (2001) Summarizing multiple aspects of model performance in a single diagram. J Geophys Res 106:7183–7192CrossRefGoogle Scholar
- Merryfield WJ et al (2013) The Canadian seasonal to interannual prediction system. Part I: Models and initialization. Mon Weather Rev 141:2910–2945. https://doi.org/10.1175/MWR-D-12-00216.1 CrossRefGoogle Scholar
- Stockdale TN, Molteni F, Ferranti L (2015) Atmospheric initial conditions and the predictability of the Arctic Oscillation. Geophys Res Lett 42:1173–1179. https://doi.org/10.1002/2014GL062681 CrossRefGoogle Scholar
- Uppala SM (2005) The ERA-40 re-analysis. Q J R Meteorol Soc 131:2961–3012CrossRefGoogle Scholar