Evaluation of the skill of North-American Multi-Model Ensemble (NMME) Global Climate Models in predicting average and extreme precipitation and temperature over the continental USA

Abstract

This paper examines the forecasting skill of eight Global Climate Models from the North-American Multi-Model Ensemble project (CCSM3, CCSM4, CanCM3, CanCM4, GFDL2.1, FLORb01, GEOS5, and CFSv2) over seven major regions of the continental United States. The skill of the monthly forecasts is quantified using the mean square error skill score. This score is decomposed to assess the accuracy of the forecast in the absence of biases (potential skill) and in the presence of conditional (slope reliability) and unconditional (standardized mean error) biases. We summarize the forecasting skill of each model according to the initialization month of the forecast and lead time, and test the models’ ability to predict extended periods of extreme climate conducive to eight ‘billion-dollar’ historical flood and drought events. Results indicate that the most skillful predictions occur at the shortest lead times and decline rapidly thereafter. Spatially, potential skill varies little, while actual model skill scores exhibit strong spatial and seasonal patterns primarily due to the unconditional biases in the models. The conditional biases vary little by model, lead time, month, or region. Overall, we find that the skill of the ensemble mean is equal to or greater than that of any of the individual models. At the seasonal scale, the drought events are better forecast than the flood events, and are predicted equally well in terms of high temperature and low precipitation. Overall, our findings provide a systematic diagnosis of the strengths and weaknesses of the eight models over a wide range of temporal and spatial scales.

This is a preview of subscription content, log in to check access.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6

References

  1. Becker E, Van den Dool H, Zhang Q (2014) Predictability and forecast skill in NMME. J Clim 27(15):5891–5906. doi:10.1175/JCLI-D-13-00597.1

    Article  Google Scholar 

  2. Boer GJ, Kharin VV, Merryfield WJ (2013) Decadal predictability and forecast skill. Clim Dyn 41(7–8):1817–1833. doi:10.1007/s00382-013-1705-0

    Article  Google Scholar 

  3. Bradley AA, Schwartz SS (2011) Summary verification measures and their interpretation for ensemble forecasts. Mon Weather Rev 139(9):3075–3089. doi:10.1175/2010MWR3305.1

    Article  Google Scholar 

  4. Bradley AA, Habib M, Schwartz SS (2015) Climate index weighting of ensemble streamflow forecasts using a simple bayesian approach. Water Res Res 51(9):1–49. doi:10.1002/2014WR016811

    Article  Google Scholar 

  5. Daly C, Gibson WP, Taylor GH, Johnson GL, Pasteris P (2002) A knowledge-based approach to the statistical mapping of climate. Clim Res 22(9):99–113

    Article  Google Scholar 

  6. DelSole T, Tippett MK (2014) Comparing Forecast Skill. Mon Weather Rev 142(12):4658–4678. doi:10.1175/MWR-D-14-00045.1

    Article  Google Scholar 

  7. Delsole T, Nattala J, Tippett MK (2014) Skill improvement from increased ensemble size and model diversity. Geophys Res Lett 41(20):7331–7342. doi:10.1002/2014GL060133

    Article  Google Scholar 

  8. Delworth TL, Broccoli AJ, Rosati A, Stouffer RJ, Balaji V, Beesley JA, Coke WF, Dixon KW, Dunne J, Dunne KA, Durachta JW (2006) GFDL’s CM2 global coupled climate models. Part I: formulation and simulation characteristics. J Clim 19(5):643–674. doi:10.1175/JCLI3629.1

    Article  Google Scholar 

  9. Donat MG, King AD, Overpeck JT, Alexander LV, Durre I, Karoly DJ (2016) Extraordinary heat during the 1930s US Dust Bowl and associated large-scale conditions. Clim Dyn 46(1–2):413–426. doi:10.1007/s00382-015-2590-5

    Article  Google Scholar 

  10. Hagedorn R, Doblas-Reyes FJ, Palmer TN (2005) The rationale behind the success of multi-model ensembles in seasonal forecasting—I. Basic concept. Tellus A 57(3):219–233. doi:10.1111/j.1600-0870.2005.00103.x

    Article  Google Scholar 

  11. Hashino T, Bradley AA, Schwartz SS (2007) Evaluation of bias-correction methods for ensemble streamflow volume forecasts. Hydrol Earth Syst Sci Discuss 3(2):561–594

    Article  Google Scholar 

  12. Hijmans R (2015) Raster: geographic data analysis and modeling. R Package version 2.4-18. http://CRAN.R-project.org/package=raster

  13. Hoerling M, Eischeid J, Kumar A, Leung R, Mariotti A, Mo K, Schubert S, Seagar R (2013) Causes and predictability of the 2012 great plains drought. Bull Am Meteorol Soc 95(2):269–282. doi:10.1175/BAMS-D-13-00055.1

    Article  Google Scholar 

  14. Infanti JM, Kirtman BP (2014) Southeastern U.S. rainfall prediction in the North American multi-model ensemble. J Hydrometeorol 15(2):529–550. doi:10.1175/JHM-D-13-072.1

    Article  Google Scholar 

  15. Infanti JM, Kirtman BP (2016) North American rainfall and temperature prediction response to the diversity of ENSO. Clim Dyn 46(9–10):3007–3023. doi:10.1007/s00382-015-2749-0

    Article  Google Scholar 

  16. Jia L, Yang X, Vecchi GA, Gudgel RG, Delworth TL, Rosati A, Stern WF, Wittenberg AT, Krishnamurthy L, Zhang S, Msadek R, Kapnick S, Underwood S, Zeng Fanrong, Anderson WhitG, Balaji V, Dixon K (2015) Improved seasonal prediction of temperature and precipitation over land in a high-resolution GFDL climate model. J Clim 28(5):2044–2062. doi:10.1175/JCLI-D-14-00112.1

    Article  Google Scholar 

  17. Jia L, Vecchi GA, Yang X, Gudgel RG, Delworth TL, Stern WF, Paffendorf K, Underwood SD, Zeng F (2016) The roles of radiative forcing, sea surface temperatures, and atmospheric and land initial conditions in U.S. summer warming episodes. J Clim 29(11):4121–4135. doi:10.1175/JCLI-D-15-0471.1

    Article  Google Scholar 

  18. Kam J, Sheffield J, Yuan X, Wood EF (2014) Did a skillful prediction of sea surface temperatures help or hinder forecasting of the 2012 Midwestern US drought? Environ Res Lett 9(3):1–9. doi:10.1088/1748-9326/9/3/034005

    Article  Google Scholar 

  19. Karl TR, Melillo JM, Peterson TC (eds) (2009) Global climate change impacts in the United States. Cambridge University Press, Cambridge

    Google Scholar 

  20. Kirtman BP, Min D (2009) Multimodel ensemble ENSO prediction with CCSM and CFS. Mon Weather Rev 137(9):2908–2930. doi:10.1175/2009MWR2672.1

    Article  Google Scholar 

  21. Kirtman BP, Min Du, Infanti JM, Kinter JL III, Paolino DA, Zhang Q, van den Dool H, Saha S, Pena Mendez M, Becker E, Peng P, Tripp P, Huang J, DeWitt DG, Tippett MK, Barnston AG, Li S, Rosati A, Schubert SD, Rienecker M, Suarez M, Li ZE, Marshak J, Lim Y-K, Tribbia J, Pegion K, Merryfield WJ, Denis B, Wood EF (2014) The North American multi-model ensemble: phase-1 seasonal-to-interannual prediction; phase-2 toward developing intraseasonal prediction. Bull Am Meteorol Soc 95(4):585–601. doi:10.1175/BAMS-D-12-00050.1

    Article  Google Scholar 

  22. Koster RD, Sud YC, Guo Z, Dirmeyer PA, Bonan G, Oleson KW, Chan E, Verseghy D, Cox P, Davies H, Kowalczyk E (2006) GLACE: the global land–atmosphere coupling experiment. Part I: overview. J Hydrometeol 7(4):590–610. doi:10.1175/JHM510.1

    Article  Google Scholar 

  23. Kumar A, Chen M, Hoerling M, Eischeid J (2013) Do extreme climate events require extreme forcings? Geophys Res Lett 40(13):3440–3445. doi:10.1002/grl.50657

    Article  Google Scholar 

  24. Kumar A, Peng P, Chen M (2014) Is there a relationship between potential and actual skill? Mon Weather Rev 142(6):2220–2227. doi:10.1175/MWR-D-13-00287.1

    Article  Google Scholar 

  25. Kunkel KE, Karl TR, Brooks H, Kossin J, Lawrimore JH, Arndt D, Bosart L, Changnon D, Cutter SL, Doesken N, Emanuel K, Groisman PY, Katz RW, Knutson T, O’brien J, Paciorek CJ, Peterson TC, Redmond K, Robinson D, Trapp J, Vose R, Weaver S, Wehner M, Wolter K, Wuebbles D (2013) Monitoring and understanding trends in extreme storms: state of knowledge. Bull Am Meteorol Soc 94(4):499–514. doi:10.1175/BAMS-D-11-00262.1

    Article  Google Scholar 

  26. Lavers DA, Villarini G (2013) Atmospheric rivers and flooding over the central United States. J Clim 26(20):7829–7836

    Article  Google Scholar 

  27. Lawrence DM, Oleson KW, Flanner MG, Fletcher CG, Lawrence PJ, Levis S, Swenson SC, Bonan GB (2012) The CCSM4 land simulation, 1850–2005: assessment of surface climate and new capabilities. J Clim 25(7):2240–2260

    Article  Google Scholar 

  28. Luo LF, Wood EF (2008) Use of Bayesian merging techniques in a multimodel seasonal hydrologic ensemble prediction system for the eastern United States. J Hydrometeorol 9(5):866–884

    Article  Google Scholar 

  29. Luo LF, Wood EF, Pan M (2007) Bayesian merging of multiple climate model forecasts for seasonal hydrological predictions. J Geophys Res Atmos. doi:10.1029/2006JD007655

    Article  Google Scholar 

  30. Ma F, Ye A, Deng X, Zhou Z, Liu X, Duan Q, Xu J, Miao C, Di Z, Gong W (2015a) Evaluating the skill of NMME seasonal precipitation ensemble predictions for 17 hydroclimatic regions in continental China. Int J Climatol 36(1):132–144. doi:10.1002/joc.4333

    Article  Google Scholar 

  31. Ma F, Yuan X, Ye A (2015b) Seasonal drought predictability and forecast skill over China. J Geophys Res Atmos 120(16):8264–8275. doi:10.1002/2015JD023185

    Article  Google Scholar 

  32. Materia S, Borrelli A, Bellucci A et al (2014) Impact of atmosphere and land surface initial conditions on seasonal forecasts of global surface temperature. J Clim 27(24):9253–9271. doi:10.1175/JCLI-D-14-00163.1

    Article  Google Scholar 

  33. McCabe GJ, Palecki MA, Betancourt JL (2004) Pacific and Atlantic Ocean influences on multidecadal drought frequency in the United States. Proc Natl Acad Sci 101(12):4136–4141. doi:10.1073/pnas.0306738101

    Article  Google Scholar 

  34. Merryfield WJ, Lee W-S, Boer GJ, Kharin VV, Scinocca JF, Flato GM, Ajayamohan RS, Fyfe JC, Tang Y, Polavarapu S (2013) The Canadian seasonal to interannual prediction system. Part I: models and initialization. Mon Weather Rev 141(8):2910–2945. doi:10.1175/MWR-D-12-00216.1

    Article  Google Scholar 

  35. Misra V, Li H (2014) The seasonal climate predictability of the Atlantic Warm Pool and its teleconnections. Geophys Res Lett 41(2):661–666. doi:10.1002/2013GL058740

    Article  Google Scholar 

  36. Mo K (2011) Drought onset and recovery over the United States. J Geophys Res. doi:10.1029/2011JD016168

    Article  Google Scholar 

  37. Mo KC, Lettenmaier DP (2014) Hydrologic prediction over the conterminous United States using the national multi-model ensemble. J Hydrometeorol 15(4):1457–1472. doi:10.1175/JHM-D-13-0197.1

    Article  Google Scholar 

  38. Mo KC, Lyon B (2015) Global meteorological drought prediction using the North American multi-model ensemble. J Hydrometeorol 16(3):1409–1424. doi:10.1175/JHM-D-14-0192.1

    Article  Google Scholar 

  39. Molod A, Takacs L, Suarez M, Bacmeister J, Song I-S, Eichmann A (2012) The GEOS-5 atmospheric general circulation model: mean climate and development from MERRA to Fortuna. In: Technical report series on global model data assimilation, vol 28. NASA Goddard Space Flight Cent., Greenbelt, p 175

  40. Murphy AH, Winkler RL (1992) Diagnostic verification of probability forecasts. Int J Forecast 7(4):435–455. doi:10.1016/0169-2070(92)90028-8

    Article  Google Scholar 

  41. National Research Council (US) (2010) Committee on assessment of intraseasonal to interannual climate prediction and predictability. In: Assessment of intraseasonal to interannual climate prediction and predictability. National Academies Press

  42. Pierce D (2014) ncdf4: Interface to Unidata netCDF (version 4 or earlier) format data files. R Package Version 1.12. http://dwpierce.com/software

  43. Roundy JK, Wood E (2015) The attribution of land-atmosphere interactions on the seasonal predictability of drought. J Hydrometeorol 16(2):793–810. doi:10.1175/JHM-D-14-0121.1

    Article  Google Scholar 

  44. Roundy JK, Yuan X, Schaake J, Wood EF (2015) A framework for diagnosing seasonal prediction through Canonical event analysis. Mon Weather Rev 143(6):2404–2418. doi:10.1175/MWR-D-14-00190.1

    Article  Google Scholar 

  45. Saha S, Moorthi S, Wu X, Wang J, Nadiga S, Tripp P, Behringer D, Hou Y-T, Chuang H-Y, Iredell M, Ek M, Meng J, Yang R, Peña Mendez M, van den Dool H, Zhang Q, Wang W, Chen M, Becker E (2014) The NCEP climate forecast system version 2. J Clim 27(6):2185–2208. doi:10.1175/JCLI-D-12-00823.1

    Article  Google Scholar 

  46. Seager R, Hoerling M (2014) Atmosphere and ocean origins of North American droughts. J Clim 27(12):4581–4606. doi:10.1175/JCLI-D-13-00329.1

    Article  Google Scholar 

  47. Seager R, Goddard L, Nakamura J, Henderson N, Lee DE (2014) Dynamical causes of the 2010/11 Texas-Northern Mexico drought. J Hydrometeorol 15(1):39–68. doi:10.1175/JHM-D-13-024.1

    Article  Google Scholar 

  48. Seneviratne TC, Davin EL, Hirschi M, Jaeger EB, Lehner I, Orlowsky B, Teuling AJ (2010) Investigating soil moisture-climate interactions in a changing climate: a review. Earth Sci Rev 99(3):125–161

    Article  Google Scholar 

  49. Shukla S, Safeeq M, AghaKouchak A, Guan K, Funk C (2015) Temperature impacts on the water year 2014 drought in California. Geophys Res Lett 42(11):4384–4393. doi:10.1002/2015GL063666

    Article  Google Scholar 

  50. Stedinger JR, Vogel RM, Foufoula-Georgiou E (1993) Chapter 18, frequency analysis of extreme events. In: Maidment DR (ed) Handbook of Hydrology. McGrawHill Book Company, New York

    Google Scholar 

  51. Thober S, Kumar R, Sheffield J, Mai J, Schäfer D, Samaniego L (2015) Seasonal soil moisture drought prediction over Europe using the North American multi-model ensemble (NMME). J Hydrometeorol 16(6):2329–2344. doi:10.1175/JHM-D-15-0053.1

    Article  Google Scholar 

  52. Tian D, Martinez CJ, Graham WD, Hwang S (2014) Statistical downscaling multi-model forecasts for seasonal precipitation and surface temperature over the Southeastern United States. J Clim 27(22):8384–8411. doi:10.1175/JCLI-D-13-00481.1

    Article  Google Scholar 

  53. Trenberth, K.E., & Guillemot, C.J. (1996). Physical processes involved in the 1988 drought and 1993 floods in North America. Journal of Climate. 9(6), 1288–1298. doi: 10.1175/1520-0442(1996)009<1288:PPIITD>2.0.CO;2

    Article  Google Scholar 

  54. Vecchi GA, Delworth T, Gudgel R, Kapnick S, Rosati A, Wittenberg A, Zeng F, Anderson W, Balaji V, Dixon K, Jia L, Kim H-S, Krishnamurthy L, Msadek R, Stern WF, Underwood SD, Villarini G, Yang X, Zhang S (2014) On the seasonal forecasting of regional tropical cyclone activity. J Clim 27(21):7994–8016. doi:10.1175/JCLI-D-14-00158.1

    Article  Google Scholar 

  55. Vernieres G, Rienecker MM, Kovach R, Keppenne CL (2012) The GEOS-iODAS: description and evaluation. In: GEOS5 technical report NASA/TM-2012-104606, vol 30. 61 pp. [Available online at http://gmao.gsfc.nasa.gov/pubs/docs/Vernieres589.pdf.]

  56. Wang H (2014) Evaluation of monthly precipitation forecasting skill of the National Multi-model Ensemble in the summer season. Hydrol Process 28(15):4472–4486. doi:10.1002/hyp.9957

    Article  Google Scholar 

  57. Wolter K, Timlin MS (2011) El Nino/Southern Oscillation behaviour since 1871 as diagnosed in an extended multivariate ENSO index (MEI.ext). Int J Climatol 31(7):1074–1087. doi:10.1002/joc.2336

    Article  Google Scholar 

  58. Wood EF, Schubert SD, Wood AW, Peters-Lidard CD, Mo KC, Mariotti A, Pulwarty RS (2015) Prospects for advancing drought understanding, monitoring, and prediction. J Hydrometeorol 16(4):1636–1657. doi:10.1175/JHM-D-14-0164.1

    Article  Google Scholar 

  59. Younas W, Tang Y (2013) PNA predictability at various time scales. J Clim 26(22):9090–9114. doi:10.1175/JCLI-D-12-00609.1

    Article  Google Scholar 

  60. Yuan X, Wood EF (2013) Multimodel seasonal forecasting of global drought onset. Geophys Res Lett 40(18):4900–4905. doi:10.1002/grl.50949

    Article  Google Scholar 

  61. Zhang S, Harrison MJ, Rosati A, Wittenberg A (2007) System design and evaluation of coupled ensemble data assimilation for global oceanic climate studies. Mon Weather Rev 135(10):3541–3564. doi:10.1175/MWR3466.1

    Article  Google Scholar 

Download references

Acknowledgments

The authors thank the NMME program partners and acknowledge the help of NCEP, IRI and NCAR personnel in creating, updating and maintaining the NMME archive, with the support of NOAA, NSF, NASA and DOE. This study was supported by NOAA’s Climate Program Office’s Modeling, Analysis, Predictions, and Projections Program, Grant#NA15OAR4310073. Gabriele Villarini also acknowledges financial support from the USACE Institute for Water Resources and from Grant/Cooperative Agreement Number G11 AP20079 from the United States Geological Survey. Its contents are solely the responsibility of the authors and do not necessarily represent the official views of NOAA, USACE or of the USGS.

Author information

Affiliations

Authors

Corresponding author

Correspondence to Louise J. Slater.

Ethics declarations

Conflict of interest

The authors declare that they have no conflict of interest.

Additional information

This paper is a contribution to the special collection on the North American Multi-Model Ensemble (NMME) seasonal prediction experiment. The special collection focuses on documenting the use of the NMME system database for research ranging from predictability studies, to multi-model prediction evaluation and diagnostics, to emerging applications of climate predictability for subseasonal to seasonal predictions. This special issue is coordinated by Annarita Mariotti (NOAA), Heather Archambault (NOAA), Jin Huang (NOAA), Ben Kirtman (University of Miami) and Gabriele Villarini (University of Iowa).

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Slater, L., Villarini, G. & Bradley, A. Evaluation of the skill of North-American Multi-Model Ensemble (NMME) Global Climate Models in predicting average and extreme precipitation and temperature over the continental USA. Clim Dyn 53, 7381–7396 (2019). https://doi.org/10.1007/s00382-016-3286-1

Download citation

Keywords

  • Seasonal forecasting
  • NMME
  • Flood
  • Drought
  • Multi-model ensemble
  • Model biases