Climate Dynamics

, Volume 53, Issue 12, pp 7251–7265 | Cite as

Assessing the fidelity of predictability estimates

  • Kathy PegionEmail author
  • Timothy DelSole
  • Emily Becker
  • Teresa Cicerone


Predictability is an intrinsic limit of the climate system due to uncertainty in initial conditions and the chaotic nature of the atmosphere. Estimates of predictability together with calculations of current prediction skill are used to define the gaps in our prediction capabilities, inform future model developments, and indicate to stakeholders the potential for making forecasts that can inform their decisions. The true predictability of the climate system is not known and must be estimated, typically using a perfect model estimate from an ensemble prediction system. However, different prediction systems can give different estimates of predictability. Can we determine which estimate of predictability is most representative of the true predictability of the climate system? We test three metrics as potential indicators of the fidelity of predictability estimates in an idealized framework—the spread-error relationship, autocorrelation and skill. Using the North American multi-model ensemble re-forecast database, we quantify whether these metrics accurately indicate a model’s ability to properly estimate predictability. It is found that none of these metrics is a robust measure for determining whether a predictability estimate is realistic for El Nino-Southern oscillation events. For temperature and precipitation over land, errors in the spread-error ratio are related to errors in estimating predictability at the shortest lead-times, while skill is not related to predictability errors. The relationship between errors in the autocorrelation and errors in estimating predictability varies by lead-time and region.


Predictability NMME 



Constructive comments from three anonymous reviewers helped to improve a previous version of this manuscript. We are also grateful to Dr. Laurie Ternary for her assistance with LaTeX. This study was supported by NOAA’s Climate Program Office’s Modeling, Analysis, Predictions, and Projections Program, Grant #NA15OAR4310072. We acknowledge the agencies that support the NMME system, and we thank the climate modeling groups (Environment Canada, NASA, NCAR, NOAA/GFDL, NOAA/NCEP, and University of Miami) for producing and making available their model output. NOAA/NCEP, NOAA/CTB, and NOAA/CPO jointly provided coordinating support and led development of the NMME system. Additional support was provided by the National Science Foundation (AGS-1338427), National Aeronautics and Space Administration (NNX14AM19G), the National Oceanic and Atmospheric Administration (NA14OAR4310160). The views expressed herein are those of the authors and do not necessarily reflect the views of these agencies.


  1. Barker TW (1991) The relationship between spread and forecast error in extended-range forecasts. J Clim 4(7):733–742CrossRefGoogle Scholar
  2. Barnston AG (2017) Deterministic skill of ENSO predictions from the North American multimodel ensemble. Clim Dyn. doi: 10.1007/s00382-017-3603-3 CrossRefGoogle Scholar
  3. Barnston AG, Tippett MK, L’Heureux ML, Li S, DeWitt DG (2012) Skill of real-time seasonal ENSO model predictions during 2002–11: is our capability increasing? Bull Am Meteorol Soc 93(5):631–651CrossRefGoogle Scholar
  4. Becker E, den Dool Hv, Zhang Q, (2014) Predictability and forecast skill in NMME. J Clim 27(15):5891–5906CrossRefGoogle Scholar
  5. Berner J, Jung T, Palmer TN (2012) Systematic model error: the impact of increased horizontal resolution versus improved stochastic and deterministic parameterizations. J Clim 25:4946–4962CrossRefGoogle Scholar
  6. Boer GJ (1984) A spectral analysis of predictability and error in an operational forecast system. Mon Weather Rev 112(6):1183–1197CrossRefGoogle Scholar
  7. Buizza R (1997) Potential forecast skill of ensemble prediction and spread and skill distributions of the ECMWF ensemble prediction system. Mon Weather Rev 125(1):99–119CrossRefGoogle Scholar
  8. Chen M, Kumar A (2014) Influence of ENSO SSTs on the spread of the probability density function for precipitation and land surface temperature. Clim Dyn 45:965–974CrossRefGoogle Scholar
  9. Chen M, Shi W, Xie P, Silva VBS, Kousky VE, Wayne Higgins R, Janowiak JE (2008) Assessing objective techniques for gauge-based analyses of global daily precipitation. J Geophys Res 113(D4):D04 (110–13) Google Scholar
  10. Christensen HM, Moroz IM, Palmer TN (2014) Evaluation of ensemble forecast uncertainty using a new proper score: application to medium-range and seasonal forecasts. Q J R Meteorol Soc 141(687):538–549CrossRefGoogle Scholar
  11. Compo GP, Sardeshmukh PD (2004) Storm track predictability on seasonal and decadal scales. J Clim 17(19):3701–3720CrossRefGoogle Scholar
  12. Dalcher A, Kalnay E (1987) Error growth and predictability in operational ECMWF forecasts. Tellus A 39A(5):474–491CrossRefGoogle Scholar
  13. Delworth TL, Broccoli AJ, Rosati A (2006) GFDL’s CM2 global coupled climate models. Part I: formulation and simulation characteristics. J Clim 19(5):643–674CrossRefGoogle Scholar
  14. DeWitt DG (2005) Diagnosis of the tropical Atlantic near-equatorial SST bias in a directly coupled atmosphere–ocean general circulation model. Geophys Res Lett 32(L01):703Google Scholar
  15. Doblas Reyes FJ, Déqué M, Piedelievre JP (2000) Multi-model spread and probabilistic seasonal forecasts in PROVOST. Q J R Meteorol Soc 126(567):2069–2087CrossRefGoogle Scholar
  16. Eade R, Smith D, Scaife A, Wallace E (2014) Do seasonal-to-decadal climate predictions underestimate the predictability of the real world? Geophys Res Lett 41:5620–5628CrossRefGoogle Scholar
  17. Fan Y, Van Den Dool H (2008) A global monthly land surface air temperature analysis for 1948–present. J Geophys Res 113(D1):D01 (103–18) Google Scholar
  18. Fu X, Yang B, Bao Q, Wang B (2008) Sea surface temperature feedback extends the predictability of tropical intraseasonal oscillation. Mon Weather Rev 136(2):577–597CrossRefGoogle Scholar
  19. Giorgi F, Francisco R (2000) Uncertainties in regional climate change prediction: a regional analysis of ensemble simulations with the HADCM2 coupled AOGCM. Clim Dyn 16(2–3):169–182CrossRefGoogle Scholar
  20. Goswami BN, Shukla J (1991) Predictability of a coupled ocean–atmosphere model. J Clim 4(1):3–22CrossRefGoogle Scholar
  21. Infanti JM, Kirtman BP (2013) Southeast US rainfall prediction in the North American multi-model ensemble. J Hydrometeorol p 131031134244006Google Scholar
  22. Jin EK, Kinter JL III (2009) Characteristics of tropical Pacific SST predictability in coupled GCM forecasts using the NCEP CFS. Clim Dyn 32:675–691CrossRefGoogle Scholar
  23. Jin EK, Kinter JL III, Wang B, Park CK, Kang IS, Kirtman BP, Kug JS, Kumar A, Luo JJ, Schemm J, Shukla J, Yamagata T (2008) Current status of ENSO prediction skill in coupled ocean–atmosphere models. Clim Dyn 31(6):647–664CrossRefGoogle Scholar
  24. Jolliffe IT, Stephenson DB (2008) Proper scores for probability forecasts can never be equitable. Mon Weather Rev 136:1505–1510CrossRefGoogle Scholar
  25. Kalnay E, Dalcher A (1987) Forecasting forecast skill. Mon Weather Rev 115(2):349–356CrossRefGoogle Scholar
  26. Kirtman BP, Min D (2009) Multimodel ensemble ENSO prediction with CCSM and CFS. Mon Weather Rev 137:2908–2930CrossRefGoogle Scholar
  27. Kirtman BP, Min D, Infanti JM, Kinter JL III, Paolino DA, Zhang Q, Van Den Dool H, Saha S, Mendez MP, Becker E, Peng P, Tripp P, Huang J, DeWitt DG, Tippett MK, Barnston AG, Li S, Rosati A, Schubert SD, Rienecker M, Suarez M, Li ZE, Marshak J, Lim YK, Tribbia J, Pegion K, Merryfield WJ, Denis B, Wood EF (2014) The North American multimodel ensemble: phase-1 seasonal-to-interannual prediction; phase-2 toward developing intraseasonal prediction. Bull Am Meteorol Soc 95(4):585–601CrossRefGoogle Scholar
  28. Kumar A, Peng P, Chen M (2014) Is there a relationship between potential and actual skill? Mon Weather Rev 142(6):2220–2227CrossRefGoogle Scholar
  29. Kumar A, Hu ZZ, Jha B, Peng P (2017) Estimating ENSO predictability based on multi-model hindcasts. Clim Dyn 48(1–2):39–51CrossRefGoogle Scholar
  30. Lorenz EN (1965) A study of the predictability of a 28-variable atmospheric model. Tellus 17(3):321–333CrossRefGoogle Scholar
  31. Lorenz EN (1969) The predictability of a flow which possesses many scales of motion. Tellus 21(3):289–307CrossRefGoogle Scholar
  32. Lorenz EN (1982) Atmospheric predictability experiments with a large numerical model. Tellus 34(6):505–513CrossRefGoogle Scholar
  33. Merryfield WJ, Lee WS, Boer GJ (2013) The Canadian seasonal to interannual prediction system. Part I: models and initialization. Mon Weather Rev 141:2910–2945CrossRefGoogle Scholar
  34. Min Q, Su J, Zhang R, Rong X (2015) What hindered the El Niño pattern in 2014? Geophys Res Lett 42:6762–6770. doi: 10.1002/2015GL064899 CrossRefGoogle Scholar
  35. Murphy AH (1969) On the “ranked probability score”. J Appl Meteorol 8:988–989CrossRefGoogle Scholar
  36. Neena JM, Lee JY, Waliser D, Wang B (2014) Predictability of the Madden–Julian oscillation in the intraseasonal variability hindcast experiment (ISVHE). J Clim 27:4531–4543CrossRefGoogle Scholar
  37. Palmer TN, Tibaldi S (1988) On the prediction of forecast skill. Mon Weather Rev 116(12):2453–2480CrossRefGoogle Scholar
  38. Palmer TN, Williams PD (2008) Introduction. Stochastic physics and climate modelling. Philos Trans R Soc A Math Phys Eng Sci 366:2421–2427Google Scholar
  39. Pegion K, Kirtman BP (2008) The impact of air-sea interactions on the predictability of the tropical intraseasonal oscillation. J Clim 21:5870–5886CrossRefGoogle Scholar
  40. Pegion K, Sardeshmukh PD (2011) Prospects for improving subseasonal predictions. Mon Weather Rev 139(11):3648–3666CrossRefGoogle Scholar
  41. Reynolds RW, Rayner NA, Smith TM (2002) An improved in situ and satellite SST analysis for climate. J Clim 15(13):1609–1625CrossRefGoogle Scholar
  42. Rienecker MM, Suarez MJ, Gelaro R (2011) MERRA: NASA’s modern-era retrospective analysis for research and applications. J Clim 24:3624–3648CrossRefGoogle Scholar
  43. Saha S, Nadiga S, Thiaw C, Wang J, Wang W, Zhang Q, Van den Dool HM, Pan HL, Moorthi S, Behringer D, Stokes D, Peña M, Lord S, White G, Ebisuzaki W, Peng P, Xie P (2006) The NCEP climate forecast system. J Clim 19(15):3483–3517CrossRefGoogle Scholar
  44. Saha S, Moorthi S, Pan HL, Wu X, Wang J, Nadiga S, Tripp P, Kistler R, Woollen J, Behringer D, Liu H, Stokes D, Grumbine R, Gayno G, Wang J, Hou YT, Chuang HY, Juang HMH, Sela J, Iredell M, Treadon R, Kleist D, Van Delst P, Keyser D, Derber J, Ek M, Meng J, Wei H, Yang R, Lord S, Van Den Dool H, Kumar A, Wang W, Long C, Chelliah M, Xue Y, Huang B, Schemm JK, Ebisuzaki W, Lin R, Xie P, Chen M, Zhou S, Higgins W, Zou CZ, Liu Q, Chen Y, Han Y, Cucurull L, Reynolds RW, Rutledge G, Goldberg M (2010) The NCEP climate forecast system reanalysis. Bull Am Meteorol Soc 91(8):1015–1057CrossRefGoogle Scholar
  45. Saha S, Moorthi S, Wu X, Wang J, Nadiga S, Tripp P, Behringer D, Hou YT, Chuang HY, Iredell M, Ek M, Meng J, Yang R, Mendez MP, Van Den Dool H, Zhang Q, Wang W, Chen M, Becker E (2014) The NCEP climate forecast system version 2. J Clim 27(6):2185–2208CrossRefGoogle Scholar
  46. Sanchez C, Williams KD (2016) Improved stochastic physics schemes for global weather and climate models. Q J R Meteorol Soc 142:147–159CrossRefGoogle Scholar
  47. Shi W, Schaller N, MacLeod D, Palmer TN, Weisheimer A (2015) Impact of hindcast length on estimates of seasonal climate predictability. Geophys Res Lett 42:1554–1559. doi: 10.1002/2014GL062829 CrossRefGoogle Scholar
  48. Shukla J (1981) Dynamical predictability of monthly means. J Atmos Sci 38:2547–2572CrossRefGoogle Scholar
  49. Slingo J, Palmer T (2011) Uncertainty in weather and climate prediction. Philos Trans R Soc A Math Phys Eng Sci 369(1956):4751–4767CrossRefGoogle Scholar
  50. Su J, Xiang B, Wang B, Li T (2014) Abrupt termination of the 2012 Pacific warming and its implication on ENSO prediction. Geophys Res Lett 41:9058–9064. doi: 10.1002/2014GL062380 CrossRefGoogle Scholar
  51. Tang Y, Lin H, Moore AM (2008) Measuring the potential predictability of ensemble climate predictions. J Geophys Res 113(D04):108Google Scholar
  52. Trenberth KE (1997) The definition of el nino. Bull Am Meteorol Soc 78(12):2771–2777CrossRefGoogle Scholar
  53. Vecchi GA, Delworth T, Gudgel R, Kapnick S (2014) On the seasonal forecasting of regional tropical cyclone activity. J Clim 27:7994–8016CrossRefGoogle Scholar
  54. Vernieres G, Rienecker M, Kovach CL, Robin ad Keppenne, (2013) Technical report series on global modeling and data assimilation, vol 30. Tech. rep, NASAGoogle Scholar
  55. Waliser DE, Lau KM, Stern W (2003) Potential predictability of the Madden–Julian oscillation. Bull Am Meteorol Soc 84(1):33–50CrossRefGoogle Scholar
  56. Weisheimer A, Doblas-Reyes FJ, Palmer TN, Alessandri A, Arribas A, Déqué M, Keenlyside N, MacVean M, Navarra A, Rogel P (2009) ENSEMBLES: a new multi-model ensemble for seasonal-to-annual predictions–skill and progress beyond DEMETER in forecasting tropical Pacific SSTs. Geophys Res Lett 36(21):L21 (711) Google Scholar
  57. Whitaker JS, Loughe AF (1998) The relationship between ensemble spread and ensemble mean skill. Mon Weather Rev 126:3292–3302. doi: 10.1175/1520-0493(1998)126<3292:TRBESA>2.0.CO;2 CrossRefGoogle Scholar
  58. Winkler RL, Murphy AH (1968) “Good” probability assessors. J Appl Meteorol 7(5):751–758CrossRefGoogle Scholar
  59. Wobus RL, Kalnay E (1995) Three years of operational prediction of forecast skill at NMC. Mon Weather Rev 123(7):2132–2148CrossRefGoogle Scholar
  60. Xue Y, Chen M, Kumar A, Hu ZZ, Wang W (2013) Prediction skill and bias of tropical Pacific sea surface temperatures in the NCEP climate forecast system version 2. J Clim 26:5358–5378CrossRefGoogle Scholar

Copyright information

© Springer-Verlag GmbH Germany 2017

Authors and Affiliations

  1. 1.Department of Atmospheric, Oceanic, and Earth Sciences and Center for Ocean-Land-Atmosphere StudiesGeorge Mason UniversityFairfaxUSA
  2. 2.NOAA/Climate Prediction Center and InnovimLLCCollege ParkUSA

Personalised recommendations