Skip to main content

Advertisement

SpringerLink
Log in

Evaluation of the skill of North-American Multi-Model Ensemble (NMME) Global Climate Models in predicting average and extreme precipitation and temperature over the continental USA

  • Published:
Climate Dynamics Aims and scope Submit manuscript

Abstract

This paper examines the forecasting skill of eight Global Climate Models from the North-American Multi-Model Ensemble project (CCSM3, CCSM4, CanCM3, CanCM4, GFDL2.1, FLORb01, GEOS5, and CFSv2) over seven major regions of the continental United States. The skill of the monthly forecasts is quantified using the mean square error skill score. This score is decomposed to assess the accuracy of the forecast in the absence of biases (potential skill) and in the presence of conditional (slope reliability) and unconditional (standardized mean error) biases. We summarize the forecasting skill of each model according to the initialization month of the forecast and lead time, and test the models’ ability to predict extended periods of extreme climate conducive to eight ‘billion-dollar’ historical flood and drought events. Results indicate that the most skillful predictions occur at the shortest lead times and decline rapidly thereafter. Spatially, potential skill varies little, while actual model skill scores exhibit strong spatial and seasonal patterns primarily due to the unconditional biases in the models. The conditional biases vary little by model, lead time, month, or region. Overall, we find that the skill of the ensemble mean is equal to or greater than that of any of the individual models. At the seasonal scale, the drought events are better forecast than the flood events, and are predicted equally well in terms of high temperature and low precipitation. Overall, our findings provide a systematic diagnosis of the strengths and weaknesses of the eight models over a wide range of temporal and spatial scales.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6

Similar content being viewed by others

References

  • Becker E, Van den Dool H, Zhang Q (2014) Predictability and forecast skill in NMME. J Clim 27(15):5891–5906. doi:10.1175/JCLI-D-13-00597.1

    Article  Google Scholar 

  • Boer GJ, Kharin VV, Merryfield WJ (2013) Decadal predictability and forecast skill. Clim Dyn 41(7–8):1817–1833. doi:10.1007/s00382-013-1705-0

    Article  Google Scholar 

  • Bradley AA, Schwartz SS (2011) Summary verification measures and their interpretation for ensemble forecasts. Mon Weather Rev 139(9):3075–3089. doi:10.1175/2010MWR3305.1

    Article  Google Scholar 

  • Bradley AA, Habib M, Schwartz SS (2015) Climate index weighting of ensemble streamflow forecasts using a simple bayesian approach. Water Res Res 51(9):1–49. doi:10.1002/2014WR016811

    Article  Google Scholar 

  • Daly C, Gibson WP, Taylor GH, Johnson GL, Pasteris P (2002) A knowledge-based approach to the statistical mapping of climate. Clim Res 22(9):99–113

    Article  Google Scholar 

  • DelSole T, Tippett MK (2014) Comparing Forecast Skill. Mon Weather Rev 142(12):4658–4678. doi:10.1175/MWR-D-14-00045.1

    Article  Google Scholar 

  • Delsole T, Nattala J, Tippett MK (2014) Skill improvement from increased ensemble size and model diversity. Geophys Res Lett 41(20):7331–7342. doi:10.1002/2014GL060133

    Article  Google Scholar 

  • Delworth TL, Broccoli AJ, Rosati A, Stouffer RJ, Balaji V, Beesley JA, Coke WF, Dixon KW, Dunne J, Dunne KA, Durachta JW (2006) GFDL’s CM2 global coupled climate models. Part I: formulation and simulation characteristics. J Clim 19(5):643–674. doi:10.1175/JCLI3629.1

    Article  Google Scholar 

  • Donat MG, King AD, Overpeck JT, Alexander LV, Durre I, Karoly DJ (2016) Extraordinary heat during the 1930s US Dust Bowl and associated large-scale conditions. Clim Dyn 46(1–2):413–426. doi:10.1007/s00382-015-2590-5

    Article  Google Scholar 

  • Hagedorn R, Doblas-Reyes FJ, Palmer TN (2005) The rationale behind the success of multi-model ensembles in seasonal forecasting—I. Basic concept. Tellus A 57(3):219–233. doi:10.1111/j.1600-0870.2005.00103.x

    Article  Google Scholar 

  • Hashino T, Bradley AA, Schwartz SS (2007) Evaluation of bias-correction methods for ensemble streamflow volume forecasts. Hydrol Earth Syst Sci Discuss 3(2):561–594

    Article  Google Scholar 

  • Hijmans R (2015) Raster: geographic data analysis and modeling. R Package version 2.4-18. http://CRAN.R-project.org/package=raster

  • Hoerling M, Eischeid J, Kumar A, Leung R, Mariotti A, Mo K, Schubert S, Seagar R (2013) Causes and predictability of the 2012 great plains drought. Bull Am Meteorol Soc 95(2):269–282. doi:10.1175/BAMS-D-13-00055.1

    Article  Google Scholar 

  • Infanti JM, Kirtman BP (2014) Southeastern U.S. rainfall prediction in the North American multi-model ensemble. J Hydrometeorol 15(2):529–550. doi:10.1175/JHM-D-13-072.1

    Article  Google Scholar 

  • Infanti JM, Kirtman BP (2016) North American rainfall and temperature prediction response to the diversity of ENSO. Clim Dyn 46(9–10):3007–3023. doi:10.1007/s00382-015-2749-0

    Article  Google Scholar 

  • Jia L, Yang X, Vecchi GA, Gudgel RG, Delworth TL, Rosati A, Stern WF, Wittenberg AT, Krishnamurthy L, Zhang S, Msadek R, Kapnick S, Underwood S, Zeng Fanrong, Anderson WhitG, Balaji V, Dixon K (2015) Improved seasonal prediction of temperature and precipitation over land in a high-resolution GFDL climate model. J Clim 28(5):2044–2062. doi:10.1175/JCLI-D-14-00112.1

    Article  Google Scholar 

  • Jia L, Vecchi GA, Yang X, Gudgel RG, Delworth TL, Stern WF, Paffendorf K, Underwood SD, Zeng F (2016) The roles of radiative forcing, sea surface temperatures, and atmospheric and land initial conditions in U.S. summer warming episodes. J Clim 29(11):4121–4135. doi:10.1175/JCLI-D-15-0471.1

    Article  Google Scholar 

  • Kam J, Sheffield J, Yuan X, Wood EF (2014) Did a skillful prediction of sea surface temperatures help or hinder forecasting of the 2012 Midwestern US drought? Environ Res Lett 9(3):1–9. doi:10.1088/1748-9326/9/3/034005

    Article  Google Scholar 

  • Karl TR, Melillo JM, Peterson TC (eds) (2009) Global climate change impacts in the United States. Cambridge University Press, Cambridge

    Google Scholar 

  • Kirtman BP, Min D (2009) Multimodel ensemble ENSO prediction with CCSM and CFS. Mon Weather Rev 137(9):2908–2930. doi:10.1175/2009MWR2672.1

    Article  Google Scholar 

  • Kirtman BP, Min Du, Infanti JM, Kinter JL III, Paolino DA, Zhang Q, van den Dool H, Saha S, Pena Mendez M, Becker E, Peng P, Tripp P, Huang J, DeWitt DG, Tippett MK, Barnston AG, Li S, Rosati A, Schubert SD, Rienecker M, Suarez M, Li ZE, Marshak J, Lim Y-K, Tribbia J, Pegion K, Merryfield WJ, Denis B, Wood EF (2014) The North American multi-model ensemble: phase-1 seasonal-to-interannual prediction; phase-2 toward developing intraseasonal prediction. Bull Am Meteorol Soc 95(4):585–601. doi:10.1175/BAMS-D-12-00050.1

    Article  Google Scholar 

  • Koster RD, Sud YC, Guo Z, Dirmeyer PA, Bonan G, Oleson KW, Chan E, Verseghy D, Cox P, Davies H, Kowalczyk E (2006) GLACE: the global land–atmosphere coupling experiment. Part I: overview. J Hydrometeol 7(4):590–610. doi:10.1175/JHM510.1

    Article  Google Scholar 

  • Kumar A, Chen M, Hoerling M, Eischeid J (2013) Do extreme climate events require extreme forcings? Geophys Res Lett 40(13):3440–3445. doi:10.1002/grl.50657

    Article  Google Scholar 

  • Kumar A, Peng P, Chen M (2014) Is there a relationship between potential and actual skill? Mon Weather Rev 142(6):2220–2227. doi:10.1175/MWR-D-13-00287.1

    Article  Google Scholar 

  • Kunkel KE, Karl TR, Brooks H, Kossin J, Lawrimore JH, Arndt D, Bosart L, Changnon D, Cutter SL, Doesken N, Emanuel K, Groisman PY, Katz RW, Knutson T, O’brien J, Paciorek CJ, Peterson TC, Redmond K, Robinson D, Trapp J, Vose R, Weaver S, Wehner M, Wolter K, Wuebbles D (2013) Monitoring and understanding trends in extreme storms: state of knowledge. Bull Am Meteorol Soc 94(4):499–514. doi:10.1175/BAMS-D-11-00262.1

    Article  Google Scholar 

  • Lavers DA, Villarini G (2013) Atmospheric rivers and flooding over the central United States. J Clim 26(20):7829–7836

    Article  Google Scholar 

  • Lawrence DM, Oleson KW, Flanner MG, Fletcher CG, Lawrence PJ, Levis S, Swenson SC, Bonan GB (2012) The CCSM4 land simulation, 1850–2005: assessment of surface climate and new capabilities. J Clim 25(7):2240–2260

    Article  Google Scholar 

  • Luo LF, Wood EF (2008) Use of Bayesian merging techniques in a multimodel seasonal hydrologic ensemble prediction system for the eastern United States. J Hydrometeorol 9(5):866–884

    Article  Google Scholar 

  • Luo LF, Wood EF, Pan M (2007) Bayesian merging of multiple climate model forecasts for seasonal hydrological predictions. J Geophys Res Atmos. doi:10.1029/2006JD007655

    Article  Google Scholar 

  • Ma F, Ye A, Deng X, Zhou Z, Liu X, Duan Q, Xu J, Miao C, Di Z, Gong W (2015a) Evaluating the skill of NMME seasonal precipitation ensemble predictions for 17 hydroclimatic regions in continental China. Int J Climatol 36(1):132–144. doi:10.1002/joc.4333

    Article  Google Scholar 

  • Ma F, Yuan X, Ye A (2015b) Seasonal drought predictability and forecast skill over China. J Geophys Res Atmos 120(16):8264–8275. doi:10.1002/2015JD023185

    Article  Google Scholar 

  • Materia S, Borrelli A, Bellucci A et al (2014) Impact of atmosphere and land surface initial conditions on seasonal forecasts of global surface temperature. J Clim 27(24):9253–9271. doi:10.1175/JCLI-D-14-00163.1

    Article  Google Scholar 

  • McCabe GJ, Palecki MA, Betancourt JL (2004) Pacific and Atlantic Ocean influences on multidecadal drought frequency in the United States. Proc Natl Acad Sci 101(12):4136–4141. doi:10.1073/pnas.0306738101

    Article  Google Scholar 

  • Merryfield WJ, Lee W-S, Boer GJ, Kharin VV, Scinocca JF, Flato GM, Ajayamohan RS, Fyfe JC, Tang Y, Polavarapu S (2013) The Canadian seasonal to interannual prediction system. Part I: models and initialization. Mon Weather Rev 141(8):2910–2945. doi:10.1175/MWR-D-12-00216.1

    Article  Google Scholar 

  • Misra V, Li H (2014) The seasonal climate predictability of the Atlantic Warm Pool and its teleconnections. Geophys Res Lett 41(2):661–666. doi:10.1002/2013GL058740

    Article  Google Scholar 

  • Mo K (2011) Drought onset and recovery over the United States. J Geophys Res. doi:10.1029/2011JD016168

    Article  Google Scholar 

  • Mo KC, Lettenmaier DP (2014) Hydrologic prediction over the conterminous United States using the national multi-model ensemble. J Hydrometeorol 15(4):1457–1472. doi:10.1175/JHM-D-13-0197.1

    Article  Google Scholar 

  • Mo KC, Lyon B (2015) Global meteorological drought prediction using the North American multi-model ensemble. J Hydrometeorol 16(3):1409–1424. doi:10.1175/JHM-D-14-0192.1

    Article  Google Scholar 

  • Molod A, Takacs L, Suarez M, Bacmeister J, Song I-S, Eichmann A (2012) The GEOS-5 atmospheric general circulation model: mean climate and development from MERRA to Fortuna. In: Technical report series on global model data assimilation, vol 28. NASA Goddard Space Flight Cent., Greenbelt, p 175

  • Murphy AH, Winkler RL (1992) Diagnostic verification of probability forecasts. Int J Forecast 7(4):435–455. doi:10.1016/0169-2070(92)90028-8

    Article  Google Scholar 

  • National Research Council (US) (2010) Committee on assessment of intraseasonal to interannual climate prediction and predictability. In: Assessment of intraseasonal to interannual climate prediction and predictability. National Academies Press

  • Pierce D (2014) ncdf4: Interface to Unidata netCDF (version 4 or earlier) format data files. R Package Version 1.12. http://dwpierce.com/software

  • Roundy JK, Wood E (2015) The attribution of land-atmosphere interactions on the seasonal predictability of drought. J Hydrometeorol 16(2):793–810. doi:10.1175/JHM-D-14-0121.1

    Article  Google Scholar 

  • Roundy JK, Yuan X, Schaake J, Wood EF (2015) A framework for diagnosing seasonal prediction through Canonical event analysis. Mon Weather Rev 143(6):2404–2418. doi:10.1175/MWR-D-14-00190.1

    Article  Google Scholar 

  • Saha S, Moorthi S, Wu X, Wang J, Nadiga S, Tripp P, Behringer D, Hou Y-T, Chuang H-Y, Iredell M, Ek M, Meng J, Yang R, Peña Mendez M, van den Dool H, Zhang Q, Wang W, Chen M, Becker E (2014) The NCEP climate forecast system version 2. J Clim 27(6):2185–2208. doi:10.1175/JCLI-D-12-00823.1

    Article  Google Scholar 

  • Seager R, Hoerling M (2014) Atmosphere and ocean origins of North American droughts. J Clim 27(12):4581–4606. doi:10.1175/JCLI-D-13-00329.1

    Article  Google Scholar 

  • Seager R, Goddard L, Nakamura J, Henderson N, Lee DE (2014) Dynamical causes of the 2010/11 Texas-Northern Mexico drought. J Hydrometeorol 15(1):39–68. doi:10.1175/JHM-D-13-024.1

    Article  Google Scholar 

  • Seneviratne TC, Davin EL, Hirschi M, Jaeger EB, Lehner I, Orlowsky B, Teuling AJ (2010) Investigating soil moisture-climate interactions in a changing climate: a review. Earth Sci Rev 99(3):125–161

    Article  Google Scholar 

  • Shukla S, Safeeq M, AghaKouchak A, Guan K, Funk C (2015) Temperature impacts on the water year 2014 drought in California. Geophys Res Lett 42(11):4384–4393. doi:10.1002/2015GL063666

    Article  Google Scholar 

  • Stedinger JR, Vogel RM, Foufoula-Georgiou E (1993) Chapter 18, frequency analysis of extreme events. In: Maidment DR (ed) Handbook of Hydrology. McGrawHill Book Company, New York

    Google Scholar 

  • Thober S, Kumar R, Sheffield J, Mai J, Schäfer D, Samaniego L (2015) Seasonal soil moisture drought prediction over Europe using the North American multi-model ensemble (NMME). J Hydrometeorol 16(6):2329–2344. doi:10.1175/JHM-D-15-0053.1

    Article  Google Scholar 

  • Tian D, Martinez CJ, Graham WD, Hwang S (2014) Statistical downscaling multi-model forecasts for seasonal precipitation and surface temperature over the Southeastern United States. J Clim 27(22):8384–8411. doi:10.1175/JCLI-D-13-00481.1

    Article  Google Scholar 

  • Trenberth, K.E., & Guillemot, C.J. (1996). Physical processes involved in the 1988 drought and 1993 floods in North America. Journal of Climate. 9(6), 1288–1298. doi: 10.1175/1520-0442(1996)009<1288:PPIITD>2.0.CO;2

    Article  Google Scholar 

  • Vecchi GA, Delworth T, Gudgel R, Kapnick S, Rosati A, Wittenberg A, Zeng F, Anderson W, Balaji V, Dixon K, Jia L, Kim H-S, Krishnamurthy L, Msadek R, Stern WF, Underwood SD, Villarini G, Yang X, Zhang S (2014) On the seasonal forecasting of regional tropical cyclone activity. J Clim 27(21):7994–8016. doi:10.1175/JCLI-D-14-00158.1

    Article  Google Scholar 

  • Vernieres G, Rienecker MM, Kovach R, Keppenne CL (2012) The GEOS-iODAS: description and evaluation. In: GEOS5 technical report NASA/TM-2012-104606, vol 30. 61 pp. [Available online at http://gmao.gsfc.nasa.gov/pubs/docs/Vernieres589.pdf.]

  • Wang H (2014) Evaluation of monthly precipitation forecasting skill of the National Multi-model Ensemble in the summer season. Hydrol Process 28(15):4472–4486. doi:10.1002/hyp.9957

    Article  Google Scholar 

  • Wolter K, Timlin MS (2011) El Nino/Southern Oscillation behaviour since 1871 as diagnosed in an extended multivariate ENSO index (MEI.ext). Int J Climatol 31(7):1074–1087. doi:10.1002/joc.2336

    Article  Google Scholar 

  • Wood EF, Schubert SD, Wood AW, Peters-Lidard CD, Mo KC, Mariotti A, Pulwarty RS (2015) Prospects for advancing drought understanding, monitoring, and prediction. J Hydrometeorol 16(4):1636–1657. doi:10.1175/JHM-D-14-0164.1

    Article  Google Scholar 

  • Younas W, Tang Y (2013) PNA predictability at various time scales. J Clim 26(22):9090–9114. doi:10.1175/JCLI-D-12-00609.1

    Article  Google Scholar 

  • Yuan X, Wood EF (2013) Multimodel seasonal forecasting of global drought onset. Geophys Res Lett 40(18):4900–4905. doi:10.1002/grl.50949

    Article  Google Scholar 

  • Zhang S, Harrison MJ, Rosati A, Wittenberg A (2007) System design and evaluation of coupled ensemble data assimilation for global oceanic climate studies. Mon Weather Rev 135(10):3541–3564. doi:10.1175/MWR3466.1

    Article  Google Scholar 

Download references

Acknowledgments

The authors thank the NMME program partners and acknowledge the help of NCEP, IRI and NCAR personnel in creating, updating and maintaining the NMME archive, with the support of NOAA, NSF, NASA and DOE. This study was supported by NOAA’s Climate Program Office’s Modeling, Analysis, Predictions, and Projections Program, Grant#NA15OAR4310073. Gabriele Villarini also acknowledges financial support from the USACE Institute for Water Resources and from Grant/Cooperative Agreement Number G11 AP20079 from the United States Geological Survey. Its contents are solely the responsibility of the authors and do not necessarily represent the official views of NOAA, USACE or of the USGS.

Author information

Authors and Affiliations

  1. IIHR-Hydroscience and Engineering, The University of Iowa, Iowa City, IA, USA

    Louise J. Slater, Gabriele Villarini & Allen A. Bradley

Authors
  1. Louise J. Slater
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Gabriele Villarini
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Allen A. Bradley
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Louise J. Slater.

Ethics declarations

Conflict of interest

The authors declare that they have no conflict of interest.

Additional information

This paper is a contribution to the special collection on the North American Multi-Model Ensemble (NMME) seasonal prediction experiment. The special collection focuses on documenting the use of the NMME system database for research ranging from predictability studies, to multi-model prediction evaluation and diagnostics, to emerging applications of climate predictability for subseasonal to seasonal predictions. This special issue is coordinated by Annarita Mariotti (NOAA), Heather Archambault (NOAA), Jin Huang (NOAA), Ben Kirtman (University of Miami) and Gabriele Villarini (University of Iowa).

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Slater, L., Villarini, G. & Bradley, A. Evaluation of the skill of North-American Multi-Model Ensemble (NMME) Global Climate Models in predicting average and extreme precipitation and temperature over the continental USA. Clim Dyn 53, 7381–7396 (2019). https://doi.org/10.1007/s00382-016-3286-1

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00382-016-3286-1

Keywords

Search

Navigation