Skip to main content

Advertisement

SpringerLink
Log in

Assessing the fidelity of predictability estimates

  • Published:
Climate Dynamics Aims and scope Submit manuscript

Abstract

Predictability is an intrinsic limit of the climate system due to uncertainty in initial conditions and the chaotic nature of the atmosphere. Estimates of predictability together with calculations of current prediction skill are used to define the gaps in our prediction capabilities, inform future model developments, and indicate to stakeholders the potential for making forecasts that can inform their decisions. The true predictability of the climate system is not known and must be estimated, typically using a perfect model estimate from an ensemble prediction system. However, different prediction systems can give different estimates of predictability. Can we determine which estimate of predictability is most representative of the true predictability of the climate system? We test three metrics as potential indicators of the fidelity of predictability estimates in an idealized framework—the spread-error relationship, autocorrelation and skill. Using the North American multi-model ensemble re-forecast database, we quantify whether these metrics accurately indicate a model’s ability to properly estimate predictability. It is found that none of these metrics is a robust measure for determining whether a predictability estimate is realistic for El Nino-Southern oscillation events. For temperature and precipitation over land, errors in the spread-error ratio are related to errors in estimating predictability at the shortest lead-times, while skill is not related to predictability errors. The relationship between errors in the autocorrelation and errors in estimating predictability varies by lead-time and region.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9

Similar content being viewed by others

Notes

  1. http://iridl.ldeo.columbia.edu/SOURCES/.Models/.NMME/index.html.

References

  • Barker TW (1991) The relationship between spread and forecast error in extended-range forecasts. J Clim 4(7):733–742

    Article  Google Scholar 

  • Barnston AG (2017) Deterministic skill of ENSO predictions from the North American multimodel ensemble. Clim Dyn. doi:10.1007/s00382-017-3603-3

    Article  Google Scholar 

  • Barnston AG, Tippett MK, L’Heureux ML, Li S, DeWitt DG (2012) Skill of real-time seasonal ENSO model predictions during 2002–11: is our capability increasing? Bull Am Meteorol Soc 93(5):631–651

    Article  Google Scholar 

  • Becker E, den Dool Hv, Zhang Q, (2014) Predictability and forecast skill in NMME. J Clim 27(15):5891–5906

    Article  Google Scholar 

  • Berner J, Jung T, Palmer TN (2012) Systematic model error: the impact of increased horizontal resolution versus improved stochastic and deterministic parameterizations. J Clim 25:4946–4962

    Article  Google Scholar 

  • Boer GJ (1984) A spectral analysis of predictability and error in an operational forecast system. Mon Weather Rev 112(6):1183–1197

    Article  Google Scholar 

  • Buizza R (1997) Potential forecast skill of ensemble prediction and spread and skill distributions of the ECMWF ensemble prediction system. Mon Weather Rev 125(1):99–119

    Article  Google Scholar 

  • Chen M, Kumar A (2014) Influence of ENSO SSTs on the spread of the probability density function for precipitation and land surface temperature. Clim Dyn 45:965–974

    Article  Google Scholar 

  • Chen M, Shi W, Xie P, Silva VBS, Kousky VE, Wayne Higgins R, Janowiak JE (2008) Assessing objective techniques for gauge-based analyses of global daily precipitation. J Geophys Res 113(D4):D04 (110–13)

  • Christensen HM, Moroz IM, Palmer TN (2014) Evaluation of ensemble forecast uncertainty using a new proper score: application to medium-range and seasonal forecasts. Q J R Meteorol Soc 141(687):538–549

    Article  Google Scholar 

  • Compo GP, Sardeshmukh PD (2004) Storm track predictability on seasonal and decadal scales. J Clim 17(19):3701–3720

    Article  Google Scholar 

  • Dalcher A, Kalnay E (1987) Error growth and predictability in operational ECMWF forecasts. Tellus A 39A(5):474–491

    Article  Google Scholar 

  • Delworth TL, Broccoli AJ, Rosati A (2006) GFDL’s CM2 global coupled climate models. Part I: formulation and simulation characteristics. J Clim 19(5):643–674

    Article  Google Scholar 

  • DeWitt DG (2005) Diagnosis of the tropical Atlantic near-equatorial SST bias in a directly coupled atmosphere–ocean general circulation model. Geophys Res Lett 32(L01):703

    Google Scholar 

  • Doblas Reyes FJ, Déqué M, Piedelievre JP (2000) Multi-model spread and probabilistic seasonal forecasts in PROVOST. Q J R Meteorol Soc 126(567):2069–2087

    Article  Google Scholar 

  • Eade R, Smith D, Scaife A, Wallace E (2014) Do seasonal-to-decadal climate predictions underestimate the predictability of the real world? Geophys Res Lett 41:5620–5628

    Article  Google Scholar 

  • Fan Y, Van Den Dool H (2008) A global monthly land surface air temperature analysis for 1948–present. J Geophys Res 113(D1):D01 (103–18)

  • Fu X, Yang B, Bao Q, Wang B (2008) Sea surface temperature feedback extends the predictability of tropical intraseasonal oscillation. Mon Weather Rev 136(2):577–597

    Article  Google Scholar 

  • Giorgi F, Francisco R (2000) Uncertainties in regional climate change prediction: a regional analysis of ensemble simulations with the HADCM2 coupled AOGCM. Clim Dyn 16(2–3):169–182

    Article  Google Scholar 

  • Goswami BN, Shukla J (1991) Predictability of a coupled ocean–atmosphere model. J Clim 4(1):3–22

    Article  Google Scholar 

  • Infanti JM, Kirtman BP (2013) Southeast US rainfall prediction in the North American multi-model ensemble. J Hydrometeorol p 131031134244006

  • Jin EK, Kinter JL III (2009) Characteristics of tropical Pacific SST predictability in coupled GCM forecasts using the NCEP CFS. Clim Dyn 32:675–691

    Article  Google Scholar 

  • Jin EK, Kinter JL III, Wang B, Park CK, Kang IS, Kirtman BP, Kug JS, Kumar A, Luo JJ, Schemm J, Shukla J, Yamagata T (2008) Current status of ENSO prediction skill in coupled ocean–atmosphere models. Clim Dyn 31(6):647–664

    Article  Google Scholar 

  • Jolliffe IT, Stephenson DB (2008) Proper scores for probability forecasts can never be equitable. Mon Weather Rev 136:1505–1510

    Article  Google Scholar 

  • Kalnay E, Dalcher A (1987) Forecasting forecast skill. Mon Weather Rev 115(2):349–356

    Article  Google Scholar 

  • Kirtman BP, Min D (2009) Multimodel ensemble ENSO prediction with CCSM and CFS. Mon Weather Rev 137:2908–2930

    Article  Google Scholar 

  • Kirtman BP, Min D, Infanti JM, Kinter JL III, Paolino DA, Zhang Q, Van Den Dool H, Saha S, Mendez MP, Becker E, Peng P, Tripp P, Huang J, DeWitt DG, Tippett MK, Barnston AG, Li S, Rosati A, Schubert SD, Rienecker M, Suarez M, Li ZE, Marshak J, Lim YK, Tribbia J, Pegion K, Merryfield WJ, Denis B, Wood EF (2014) The North American multimodel ensemble: phase-1 seasonal-to-interannual prediction; phase-2 toward developing intraseasonal prediction. Bull Am Meteorol Soc 95(4):585–601

    Article  Google Scholar 

  • Kumar A, Peng P, Chen M (2014) Is there a relationship between potential and actual skill? Mon Weather Rev 142(6):2220–2227

    Article  Google Scholar 

  • Kumar A, Hu ZZ, Jha B, Peng P (2017) Estimating ENSO predictability based on multi-model hindcasts. Clim Dyn 48(1–2):39–51

    Article  Google Scholar 

  • Lorenz EN (1965) A study of the predictability of a 28-variable atmospheric model. Tellus 17(3):321–333

    Article  Google Scholar 

  • Lorenz EN (1969) The predictability of a flow which possesses many scales of motion. Tellus 21(3):289–307

    Article  Google Scholar 

  • Lorenz EN (1982) Atmospheric predictability experiments with a large numerical model. Tellus 34(6):505–513

    Article  Google Scholar 

  • Merryfield WJ, Lee WS, Boer GJ (2013) The Canadian seasonal to interannual prediction system. Part I: models and initialization. Mon Weather Rev 141:2910–2945

    Article  Google Scholar 

  • Min Q, Su J, Zhang R, Rong X (2015) What hindered the El Niño pattern in 2014? Geophys Res Lett 42:6762–6770. doi:10.1002/2015GL064899

    Article  Google Scholar 

  • Murphy AH (1969) On the “ranked probability score”. J Appl Meteorol 8:988–989

    Article  Google Scholar 

  • Neena JM, Lee JY, Waliser D, Wang B (2014) Predictability of the Madden–Julian oscillation in the intraseasonal variability hindcast experiment (ISVHE). J Clim 27:4531–4543

    Article  Google Scholar 

  • Palmer TN, Tibaldi S (1988) On the prediction of forecast skill. Mon Weather Rev 116(12):2453–2480

    Article  Google Scholar 

  • Palmer TN, Williams PD (2008) Introduction. Stochastic physics and climate modelling. Philos Trans R Soc A Math Phys Eng Sci 366:2421–2427

    Google Scholar 

  • Pegion K, Kirtman BP (2008) The impact of air-sea interactions on the predictability of the tropical intraseasonal oscillation. J Clim 21:5870–5886

    Article  Google Scholar 

  • Pegion K, Sardeshmukh PD (2011) Prospects for improving subseasonal predictions. Mon Weather Rev 139(11):3648–3666

    Article  Google Scholar 

  • Reynolds RW, Rayner NA, Smith TM (2002) An improved in situ and satellite SST analysis for climate. J Clim 15(13):1609–1625

    Article  Google Scholar 

  • Rienecker MM, Suarez MJ, Gelaro R (2011) MERRA: NASA’s modern-era retrospective analysis for research and applications. J Clim 24:3624–3648

    Article  Google Scholar 

  • Saha S, Nadiga S, Thiaw C, Wang J, Wang W, Zhang Q, Van den Dool HM, Pan HL, Moorthi S, Behringer D, Stokes D, Peña M, Lord S, White G, Ebisuzaki W, Peng P, Xie P (2006) The NCEP climate forecast system. J Clim 19(15):3483–3517

    Article  Google Scholar 

  • Saha S, Moorthi S, Pan HL, Wu X, Wang J, Nadiga S, Tripp P, Kistler R, Woollen J, Behringer D, Liu H, Stokes D, Grumbine R, Gayno G, Wang J, Hou YT, Chuang HY, Juang HMH, Sela J, Iredell M, Treadon R, Kleist D, Van Delst P, Keyser D, Derber J, Ek M, Meng J, Wei H, Yang R, Lord S, Van Den Dool H, Kumar A, Wang W, Long C, Chelliah M, Xue Y, Huang B, Schemm JK, Ebisuzaki W, Lin R, Xie P, Chen M, Zhou S, Higgins W, Zou CZ, Liu Q, Chen Y, Han Y, Cucurull L, Reynolds RW, Rutledge G, Goldberg M (2010) The NCEP climate forecast system reanalysis. Bull Am Meteorol Soc 91(8):1015–1057

    Article  Google Scholar 

  • Saha S, Moorthi S, Wu X, Wang J, Nadiga S, Tripp P, Behringer D, Hou YT, Chuang HY, Iredell M, Ek M, Meng J, Yang R, Mendez MP, Van Den Dool H, Zhang Q, Wang W, Chen M, Becker E (2014) The NCEP climate forecast system version 2. J Clim 27(6):2185–2208

    Article  Google Scholar 

  • Sanchez C, Williams KD (2016) Improved stochastic physics schemes for global weather and climate models. Q J R Meteorol Soc 142:147–159

    Article  Google Scholar 

  • Shi W, Schaller N, MacLeod D, Palmer TN, Weisheimer A (2015) Impact of hindcast length on estimates of seasonal climate predictability. Geophys Res Lett 42:1554–1559. doi:10.1002/2014GL062829

    Article  Google Scholar 

  • Shukla J (1981) Dynamical predictability of monthly means. J Atmos Sci 38:2547–2572

    Article  Google Scholar 

  • Slingo J, Palmer T (2011) Uncertainty in weather and climate prediction. Philos Trans R Soc A Math Phys Eng Sci 369(1956):4751–4767

    Article  Google Scholar 

  • Su J, Xiang B, Wang B, Li T (2014) Abrupt termination of the 2012 Pacific warming and its implication on ENSO prediction. Geophys Res Lett 41:9058–9064. doi:10.1002/2014GL062380

    Article  Google Scholar 

  • Tang Y, Lin H, Moore AM (2008) Measuring the potential predictability of ensemble climate predictions. J Geophys Res 113(D04):108

    Google Scholar 

  • Trenberth KE (1997) The definition of el nino. Bull Am Meteorol Soc 78(12):2771–2777

    Article  Google Scholar 

  • Vecchi GA, Delworth T, Gudgel R, Kapnick S (2014) On the seasonal forecasting of regional tropical cyclone activity. J Clim 27:7994–8016

    Article  Google Scholar 

  • Vernieres G, Rienecker M, Kovach CL, Robin ad Keppenne, (2013) Technical report series on global modeling and data assimilation, vol 30. Tech. rep, NASA

  • Waliser DE, Lau KM, Stern W (2003) Potential predictability of the Madden–Julian oscillation. Bull Am Meteorol Soc 84(1):33–50

    Article  Google Scholar 

  • Weisheimer A, Doblas-Reyes FJ, Palmer TN, Alessandri A, Arribas A, Déqué M, Keenlyside N, MacVean M, Navarra A, Rogel P (2009) ENSEMBLES: a new multi-model ensemble for seasonal-to-annual predictions–skill and progress beyond DEMETER in forecasting tropical Pacific SSTs. Geophys Res Lett 36(21):L21 (711)

  • Whitaker JS, Loughe AF (1998) The relationship between ensemble spread and ensemble mean skill. Mon Weather Rev 126:3292–3302. doi:10.1175/1520-0493(1998)126<3292:TRBESA>2.0.CO;2

    Article  Google Scholar 

  • Winkler RL, Murphy AH (1968) “Good” probability assessors. J Appl Meteorol 7(5):751–758

    Article  Google Scholar 

  • Wobus RL, Kalnay E (1995) Three years of operational prediction of forecast skill at NMC. Mon Weather Rev 123(7):2132–2148

    Article  Google Scholar 

  • Xue Y, Chen M, Kumar A, Hu ZZ, Wang W (2013) Prediction skill and bias of tropical Pacific sea surface temperatures in the NCEP climate forecast system version 2. J Clim 26:5358–5378

    Article  Google Scholar 

Download references

Acknowledgements

Constructive comments from three anonymous reviewers helped to improve a previous version of this manuscript. We are also grateful to Dr. Laurie Ternary for her assistance with LaTeX. This study was supported by NOAA’s Climate Program Office’s Modeling, Analysis, Predictions, and Projections Program, Grant #NA15OAR4310072. We acknowledge the agencies that support the NMME system, and we thank the climate modeling groups (Environment Canada, NASA, NCAR, NOAA/GFDL, NOAA/NCEP, and University of Miami) for producing and making available their model output. NOAA/NCEP, NOAA/CTB, and NOAA/CPO jointly provided coordinating support and led development of the NMME system. Additional support was provided by the National Science Foundation (AGS-1338427), National Aeronautics and Space Administration (NNX14AM19G), the National Oceanic and Atmospheric Administration (NA14OAR4310160). The views expressed herein are those of the authors and do not necessarily reflect the views of these agencies.

Author information

Authors and Affiliations

  1. Department of Atmospheric, Oceanic, and Earth Sciences and Center for Ocean-Land-Atmosphere Studies, George Mason University, Fairfax, VA, USA

    Kathy Pegion, Timothy DelSole & Teresa Cicerone

  2. NOAA/Climate Prediction Center and Innovim, LLC, College Park, MD, USA

    Emily Becker

Authors
  1. Kathy Pegion
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Timothy DelSole
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Emily Becker
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Teresa Cicerone
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Kathy Pegion.

Additional information

This paper is a contribution to the special collection on the North American Multi-Model Ensemble (NMME) seasonal prediction experiment. The special collection focuses on documenting the use of the NMME system database for research ranging from predictability studies, to multi-model prediction evaluation and diagnostics, to emerging applications of climate predictability for subseasonal to seasonal predictions. This special issue is coordinated by Annarita Mariotti (NOAA), Heather Archambault (NOAA), Jin Huang (NOAA), Ben Kirtman (University of Miami) and Gabriele Villarini (University of Iowa).

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Pegion, K., DelSole, T., Becker, E. et al. Assessing the fidelity of predictability estimates. Clim Dyn 53, 7251–7265 (2019). https://doi.org/10.1007/s00382-017-3903-7

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00382-017-3903-7

Keywords

Search

Navigation