Skip to main content

Evaluation of CMIP5 ability to reproduce twentieth century regional trends in surface air temperature and precipitation over CONUS

Abstract

The ability of the 5th phase of the Coupled Model Intercomparison Project (CMIP5) to reproduce twentieth-century climate trends over the seven CONUS regions of the National Climate Assessment is evaluated. This evaluation is carried out for summer and winter for three time periods, 1895–1939, 1940–1979, and 1980–2005. The evaluation includes all 206 CMIP5 historical simulations from 48 unique models and their multi-model ensemble (MME), as well as a gridded in situ dataset of surface air temperature and precipitation. Analysis is performed on both individual members and the MME, and considers reproducing the correct sign of the trends by the members as well as reproducing the trend values. While the MME exhibits some trend bias in most cases, it reproduces historical temperature trends with reasonable fidelity for summer for all time periods and all regions, including at the CONUS scale, except the Northern Great Plains from 1895 to 1939 and Southeast during 1980–2005. Likewise, for DJF, the MME reproduces historical temperature trends across all time periods over all regions, including at the CONUS scale, except the Southeast from 1895 to 1939 and the Midwest during 1940–1979. Model skill was highest across all of the seven regions during JJA and DJF for the 1980–2005 period. The quantitatively best result is seen during DJF in the Southwest region with at least 74% of the ensemble members correctly reproducing the observed trend across all of the time periods. No clear trends in MME precipitation were identified at these scales due to high model precipitation variability.

This is a preview of subscription content, access via your institution.

Fig. 1

Adapted from Janssen et al. (2014)

Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10

References

  1. Baek H-J, Lee J, Lee H-S, Hyun Y-K, Cho C, Kwon W-T, Marzin C, Gan SY, Kim MJ, Choi DH, Lee J, Lee J, Boo K-O, Kang H-S, Byun Y-H (2013) Climate change in the 21st century simulated by HadGEM2-AO under representative concentration pathways. Asia Pac J Atmos Sci 49(5):603–618. https://doi.org/10.1007/s13143-013-0053-7

    Article  Google Scholar 

  2. Barnett TP, Pierce DW, Hidalgo HG, Bonfils C, Santer BD, Das T, Bala G, Wood AW, Nozawa T, Mirin AA, Cayan DR, Dettinger MD (2008) Human-induced changes in the hydrology of the western United States. Science 319(5866):1080–1083. Retrieved from http://science.sciencemag.org/content/319/5866/1080.abstract

    Article  Google Scholar 

  3. Bentsen M, Bethke I, Debernard JB, Iversen T, Kirkevåg A, Seland Ø, Drange H, Roelandt C, Seierstad IA, Hoose C, Kristjánsson JE (2013) The Norwegian earth system model, NorESM1-M—part 1: description and basic evaluation of the physical climate. Geosci Model Dev 6(3):687–720. https://doi.org/10.5194/gmd-6-687-2013

    Article  Google Scholar 

  4. Bukovsky MS (2012) Temperature trends in the NARCCAP regional climate models. J Clim 25(11):3985–3991. https://doi.org/10.1175/JCLI-D-11-00588.1

    Article  Google Scholar 

  5. Cayan DR, Tyree M, Kunkel KE, Castro C, Gershunov A, Barsugli J, Ray AJ, Overpeck J, Anderson A, Russell J, Rajagopalan B, Rangwala I, Duffy P, Barlow M (2013) Future climate: projected average. In: Garfin G, Jardine A, Merideth R, Black M, LeRoy S (eds) Assessment of climate change in the southwest United States: a report prepared for the national climate assessment. Island Press, Washington, DC, pp 101–125. https://doi.org/10.5822/978-1-61091-484-0_6

    Chapter  Google Scholar 

  6. Chylek P, Li J, Dubey MK, Wang M, Lesins G (2011) Observed and model simulated 20th century Arctic temperature variability: Canadian earth system model CanESM2. Atmos Chem Phys Discuss 11(8):22893–22907. https://doi.org/10.5194/acpd-11-22893-2011

    Article  Google Scholar 

  7. Collier M, Uhe P (2012) CMIP5 datasets from the ACCESS1.0 and ACCESS1.3 coupled climate models. CAWCR technical report no. 059. ISBN: 20978-1-922173-29-4

  8. Collins WD, Rasch PJ, Boville BA, Hack JJ, Williamson DL, Kiehl JT, Briegleb B, Bitz C, Lin SJ, Zhang M, Dai Y (2004) Description of the NCAR community atmosphere model (CAM 3.0). Ncar/Tn-464+ Str (June), 214

  9. Collins WJ, Bellouin N, Doutriaux-Boucher M, Gedney N, Halloran P, Hinton T, Hughes J, Jones CD, Joshi M, Liddicoat S, Martin G, O’Connor F, Rae J, Senior C, Sitch S, Totterdell I, Wiltshire A, Woodward S (2011) Development and evaluation of an earth-system model—HadGEM2. Geosci Model Dev 4(4):1051–1075. https://doi.org/10.5194/gmd-4-1051-2011

    Article  Google Scholar 

  10. Crowley TJ (2000) Causes of climate change over the past 1000 years. Science 289(5477):270–277. Retrieved from http://science.sciencemag.org/content/289/5477/270.abstract

    Article  Google Scholar 

  11. Delworth TL, Broccoli AJ, Rosati A, Stouffer RJ, Balaji V, Beesley JA, Cooke WF, Dixon KW, Dunne J, Dunne KA, Durachta JW, Findell KL, Ginoux P, Gnanadesikan A, Gordon CT, Griffies SM, Gudgel R, Harrison MJ, Held IM, Hemler RS, Horowitz LW, Klein SA, Knutson TR, Kushner PJ, Langenhorst AR, Lee H-C, Lin S-J, Lu J, Malyshev SL, Milly PCD, Ramaswamy V, Russell J, Schwarzkopf MD, Shevliakova E, Sirutis JJ, Spelman MJ, Stern WF, Winton M, Wittenberg AT, Wyman B, Zeng F, Zhang R (2006) GFDL’s CM2 global coupled climate models. Part I: formulation and simulation characteristics. J Clim 19(5):643–674. https://doi.org/10.1175/JCLI3629.1

    Article  Google Scholar 

  12. Donner LJ, Wyman BL, Hemler RS, Horowitz LW, Ming Y, Zhao M, Golaz J-C, Ginoux P, Lin S-J, Schwarzkopf MD, Austin J, Alaka G, Cooke WF, Delworth TL, Freidenreich SM, Gordon CT, Griffies SM, Held IM, Hurlin WJ, Klein SA, Knutson TR, Langenhorst AR, Lee H-C, Lin Y, Magi BI, Malyshev SL, Milly PCD, Naik V, Nath MJ, Pincus R, Ploshay JJ, Ramaswamy V, Seman CJ, Shevliakova E, Sirutis JJ, Stern WF, Stouffer RJ, Wilson RJ, Winton M, Wittenberg AT, Zeng F (2011) The dynamical core, physical parameterizations, and basic simulation characteristics of the atmospheric component AM3 of the GFDL global coupled model CM3. J Clim 24(13):3484–3519. https://doi.org/10.1175/2011JCLI3955.1

    Article  Google Scholar 

  13. Dufresne J-L, Foujols M-A, Denvil S, Caubel A, Marti O, Aumont O, Balkanski Y, Bekki S, Bellenger H, Benshila R, Bony S, Bopp L, Braconnot P, Brockmann P, Cadule P, Cheruy F, Codron F, Cozic A, Cugnet D, de Noblet N, Duvel J-P, Ethé C, Fairhead L, Fichefet T, Flavoni S, Friedlingstein P, Grandpeix J-Y, Guez L, Guilyardi E, Hauglustaine D, Hourdin F, Idelkadi A, Ghattas J, Joussaume S, Kageyama M, Krinner G, Labetoulle S, Lahellec A, Lefebvre M-P, Lefevre F, Levy C, Li ZX, Lloyd J, Lott F, Madec G, Mancip M, Marchand M, Masson S, Meurdesoif Y, Mignot J, Musat I, Parouty S, Polcher J, Rio C, Schulz M, Swingedouw D, Szopa S, Talandier C, Terray P, Viovy N, Vuichard N (2013) Climate change projections using the IPSL-CM5 earth system model: from CMIP3 to CMIP5. Clim Dyn 40(9):2123–2165. https://doi.org/10.1007/s00382-012-1636-1

    Article  Google Scholar 

  14. Dunne JP, John JG, Shevliakova E, Stouffer RJ, Krasting JP, Malyshev SL, Milly PC, Sentman LT, Adcroft AJ, Cooke W, Dunne KA, Griffies SM, Hallberg RW, Harrison MJ, Levy H, Wittenberg AT, Phillips PJ, Zadeh N (2013) GFDL’s ESM2 global coupled climate–carbon earth system models. Part II: carbon system formulation and baseline simulation characteristics. J Clim 26(7):2247–2267. https://doi.org/10.1175/JCLI-D-12-00150.1

    Article  Google Scholar 

  15. Easterling DR, Kunkel KE, Arnold JR, Knutson T, LeGrande AN, Leung LR, Vose RS, Waliser DE, Wehner MF (2017) Precipitation change in the United States. In: Wuebbles DJ, Fahey DW, Hibbard KA, Dokken DJ, Stewart BC, Maycock TK (eds) Climate science special report: fourth national climate assessment, vol I. U.S. Global Change Research Program, pp 207–230. https://doi.org/10.7930/j0h993cc

  16. Fogli PG, Iovino D (2014) CMCC–CESM–NEMO: toward the new CMCC earth system model. In: CMCC research paper, no 248. https://doi.org/10.2139/ssrn.2603176

  17. Gordon H, Farrell SO, Collier M, Dix M, Rotstayn L, Kowalczyk E, Hirst T, Watterson I (2010) The CSIRO Mk3.5 Climate Model. In: CAWCR Technical Report, no 21. CAWCR, Melbourne, pp 1–74

    Google Scholar 

  18. Hazeleger W, Wang X, Severijns C, Ştefănescu S, Bintanja R, Sterl A, Wyser K, Semmler T, Yang S, Van den Hurk B (2012) EC-Earth V2. 2: description and validation of a new seamless earth system prediction model. Clim Dyn 39(11):2611–2629

    Article  Google Scholar 

  19. Hogg RV, Tanis EA (2009) Probability and statistical inference. Pearson Educational International, London

    Google Scholar 

  20. IPCC (2013) Climate change 2013: the physical science basis. Contribution of Working Group I to the Fifth Assessment Report of the Intergovernmental Panel on Climate Change. Cambridge University Press, Cambridge

    Google Scholar 

  21. Janssen E, Wuebbles D, Kunkel K (2014) Observational and model based trends and projections of extreme precipitation over the contiguous United States. Earth’s Future. https://doi.org/10.1002/2013EF000185.Received

    Article  Google Scholar 

  22. Janssen E, Sriver RL, Wuebbles DJ, Kunkel KE (2016) Seasonal and regional variations in extreme precipitation event frequency using CMIP5. Geophys Res Lett 43(10):5385–5393. https://doi.org/10.1002/2016GL069151

    Article  Google Scholar 

  23. Ji D, Wang L, Feng J, Wu Q, Cheng H, Zhang Q, Yang J, Dong W, Dai Y, Gong D, Zhang RH, Wang X, Liu J, Moore JC, Chen D, Zhou M (2014) Description and basic evaluation of Beijing Normal University earth system model (BNU-ESM) version 1. Geosci Model Dev 7(5):2039–2064. https://doi.org/10.5194/gmd-7-2039-2014

    Article  Google Scholar 

  24. Karl TR, Knight RW, Easterling DR, Quayle RG (1996) Indices of climate change for the United States. Bull Am Meteorol Soc 77(2):279–292. https://doi.org/10.1175/1520-0477(1996)077%3c0279:IOCCFT%3e2.0.CO;2

    Article  Google Scholar 

  25. Knutson TR, Delworth TL, Dixon KW, Held IM, Lu J, Ramaswamy V, Schwarzkopf MD, Stenchikov G, Stouffer RJ (2006) Assessment of twentieth-century regional surface temperature trends using the GFDL CM2 coupled models. J Clim 19(9):1624–1651. https://doi.org/10.1175/JCLI3709.1

    Article  Google Scholar 

  26. Knutson TR, Zeng F, Wittenberg AT (2013) Multimodel assessment of regional surface temperature trends: CMIP3 and CMIP5 twentieth-century simulations. J Clim 26(22):8709–8743. https://doi.org/10.1175/JCLI-D-12-00567.1

    Article  Google Scholar 

  27. Kumar S III, Kinter J, Dirmeyer PA, Pan Z, Adams J (2013a) Multidecadal climate variability and the “warming hole” in North America: results from CMIP5 twentieth- and twenty-first-century climate simulations. J Clim 26(11):3511–3527. https://doi.org/10.1175/JCLI-D-12-00535.1

    Article  Google Scholar 

  28. Kumar S, Merwade V, Kinter JL, Niyogi D (2013b) Evaluation of temperature and precipitation trends and long-term persistence in CMIP5 twentieth-century climate simulations. J Clim 26(12):4168–4185. https://doi.org/10.1175/JCLI-D-12-00259.1

    Article  Google Scholar 

  29. Kunkel KE, Liang XZ (2005) GCM simulations of the climate in the central United States. J Clim 18(7):1016–1031. https://doi.org/10.1175/JCLI-3309.1

    Article  Google Scholar 

  30. Kunkel KE, Easterling DR, Redmond K, Hubbard K (2003) Temporal variations of extreme precipitation events in the United States: 1895–2000. Geophys Res Lett. https://doi.org/10.1029/2003GL018052

    Article  Google Scholar 

  31. Kunkel KE, Liang X-Z, Zhu J, Lin Y (2006) Can CGCMs simulate the twentieth-century “warming hole” in the central United States? J Clim 19(17):4137–4153. https://doi.org/10.1175/JCLI3848.1

    Article  Google Scholar 

  32. Kunkel K, Stevens LE, Stevens SE, Sun L, Janssen E, Wuebbles D, Dobson JG (2013) Regional climate trends and scenarios for the U. S. National Climate Assessment part 9. Climate of the contiguous United States (January), 77. Retrieved from http://www.nesdis.noaa.gov/technical_reports/149_Climate_Scenarios.html

  33. Kunkel KE, Vose RS, Stevens LE, Knight RW (2015) Is the monthly temperature climate of the United States becoming more extreme? Geophys Res Lett 42(2):629–636. https://doi.org/10.1002/2014GL062035

    Article  Google Scholar 

  34. Li L, Lin P, Yu Y, Wang B, Zhou T, Liu L, Liu J, Bao Q, Xu S, Huang W, Xia K, Pu Y, Dong L, Shen S, Liu Y, Hu N, Liu M, Sun W, Shi X, Zheng W, Wu B, Song M, Liu H, Zhang X, Wu G, Xue W, Huang X, Yang G, Song Z, Qiao F (2013) The flexible global ocean-atmosphere-land system model, grid-point version 2: FGOALS-g2. Adv Atmos Sci 30(3):543–560. https://doi.org/10.1007/s00376-012-2140-6

    Article  Google Scholar 

  35. Marsh DR, Mills MJ, Kinnison DE, Lamarque J-F, Calvo N, Polvani LM (2013) Climate change from 1850 to 2005 simulated in CESM1(WACCM). J Clim 26(19):7372–7391. https://doi.org/10.1175/JCLI-D-12-00558.1

    Article  Google Scholar 

  36. Meehl GA, Tebaldi C (2004) More intense, more frequent, and longer lasting heat waves in the 21st century. Science 305(5686):994–997. Retrieved from http://science.sciencemag.org/content/305/5686/994.abstract

    Article  Google Scholar 

  37. Meehl GA, Arblaster JM, Branstator G (2012) Mechanisms contributing to the warming hole and the consequent U.S. east–west differential of heat extremes. J Clim 25(18):6394–6408. https://doi.org/10.1175/JCLI-D-11-00655.1

    Article  Google Scholar 

  38. Melillo JM, Richmond TC, Yohe GW, US National Climate Assessment (2014) Climate change impacts in the United States: the third national climate assessment. US Global Change Research Program, vol 841. https://doi.org/10.7930/j0z31WJ2

  39. Menne MJ, Durre I, Vose RS, Gleason BE, Houston TG (2012) An overview of the global historical climatology network-daily database. J Atmos Ocean Technol 29(7):897–910. https://doi.org/10.1175/JTECH-D-11-00103.1

    Article  Google Scholar 

  40. Pope VD, Gallani ML, Rowntree PR, Stratton RA (2000) The impact of new physical parametrizations in the Hadley Centre climate model: HadAM3. Clim Dyn 16(2):123–146. https://doi.org/10.1007/s003820050009

    Article  Google Scholar 

  41. Qiao F, Song Z, Bao Y, Song Y, Shu Q, Huang C, Zhao W (2013) Development and evaluation of an earth system model with surface gravity waves. J Geophys Res Oceans 118(9):4514–4524

    Article  Google Scholar 

  42. Sakamoto TT, Komuro Y, Nishimura T, Ishii M, Tatebe H, Shiogama H, Hasegawa A, Toyoda T, Mori M, Suzuki T, Imada Y, Nozawa T, Takata K, Mochizuki T, Ogochi K, Emori S, Hasumi H, Kimoto M (2012) MIROC4h—a new high-resolution atmosphere-ocean coupled general circulation model. J Meteorol Soc Jpn Ser II 90(3):325–359. https://doi.org/10.2151/jmsj.2012-301

    Article  Google Scholar 

  43. Schmidt GA, Kelley M, Nazarenko L, Ruedy R, Russell GL, Aleinov I, Bauer M, Bauer SE, Bhat MK, Bleck R, Canuto V, Chen Y-H, Cheng Y, Clune TL, Del Genio A, de Fainchtein R, Faluvegi G, Hansen JE, Healy RJ, Kiang NY, Koch D, Lacis AA, LeGrande AN, Lerner J, Lo KK, Matthews EE, Menon S, Miller RL, Oinas V, Oloso AO, Perlwitz JP, Puma MJ, Putman WM, Rind D, Romanou A, Sato M, Shindell DT, Sun S, Syed RA, Tausnev N, Tsigaridis K, Unger N, Voulgarakis A, Yao M-S, Zhang J (2014) Configuration and assessment of the GISS ModelE2 contributions to the CMIP5 archive. J Adv Model Earth Syst 6(1):141–184. https://doi.org/10.1002/2013MS000265

    Article  Google Scholar 

  44. Scoccimarro E, Gualdi S, Bellucci A, Sanna A, Fogli PG, Manzini E, Vichi M, Oddo P, Navarra A (2011) Effects of tropical cyclones on ocean heat transport in a high-resolution coupled general circulation model. J Clim 24(16):4368–4384. https://doi.org/10.1175/2011JCLI4104.1

    Article  Google Scholar 

  45. Stevens B, Giorgetta M, Esch M, Mauritsen T, Crueger T, Rast S, Salzmann M, Schmidt H, Bader J, Block K, Brokopf R, Fast I, Kinne S, Kornblueh L, Lohmann U, Pincus R, Reichler T, Roeckner E (2013) Atmospheric component of the MPI-M earth system model: ECHAM6. J Adv Model Earth Syst 5(2):146–172. https://doi.org/10.1002/jame.20015

    Article  Google Scholar 

  46. Taylor KE, Stouffer RJ, Meehl GA (2012) An overview of CMIP5 and the experiment design. Bull Am Meteorol Soc 93(4):485–498. https://doi.org/10.1175/BAMS-D-11-00094.1

    Article  Google Scholar 

  47. Voldoire A, Sanchez-Gomez E, y Mélia D, Decharme B, Cassou C, Sénési S, Valcke S, Beau I, Alias A, Chevallier M, Déqué M, Deshayes J, Douville H, Fernandez E, Madec G, Maisonnave E, Moine M-P, Planton S, Saint-Martin D, Szopa S, Tyteca S, Alkama R, Belamari S, Braun A, Coquart L, Chauvin F (2013) The CNRM-CM5.1 global climate model: description and basic evaluation. Clim Dyn 40(9):2091–2121. https://doi.org/10.1007/s00382-011-1259-y

    Article  Google Scholar 

  48. Volodin EM, Dianskii NA, Gusev AV (2010) Simulating present-day climate with the INMCM4.0 coupled model of the atmospheric and oceanic general circulations. Izv Atmos Ocean Phys 46(4):414–431. https://doi.org/10.1134/S000143381004002X

    Article  Google Scholar 

  49. Vose RS, Applequist S, Squires M, Durre I, Menne CJ, Williams CN, Fenimore C, Gleason K, Arndt D (2014) Improved historical temperature and precipitation time series for U.S. climate divisions. J Appl Meteorol Climatol 53(5):1232–1251. https://doi.org/10.1175/JAMC-D-13-0248.1

    Article  Google Scholar 

  50. Watanabe M, Suzuki T, O’ishi R, Komuro Y, Watanabe S, Emori S, Takemura T, Chikira M, Ogura T, Sekiguchi M, Takata K, Yamazaki D, Yokohata T, Nozawa T, Hasumi H, Tatebe H, Kimoto M (2010) Improved climate simulation by MIROC5: mean states, variability, and climate sensitivity. J Clim 23(23):6312–6335. https://doi.org/10.1175/2010JCLI3679.1

    Article  Google Scholar 

  51. Watanabe S, Hajima T, Sudo K, Nagashima T, Takemura T, Okajima H, Nozawa T, Kawase H, Abe M, Yokohata T, Ise T, Sato H, Kato E, Takata K, Emori S, Kawamiya M (2011) MIROC-ESM 2010: model description and basic results of CMIP5-20c3m experiments. Geosci Model Dev 4(4):845–872. https://doi.org/10.5194/gmd-4-845-2011

    Article  Google Scholar 

  52. Wu T, Song L, Li W, Wang Z, Zhang H, Xin X, Zhang Zhang L, Li J, Wu F, Liu Y, Zhang F, Shi X, Chu M, Zhang J, Fang Y, Wang F, Lu Y, Liu X, Wei M, Liu Q, Zhou W, Dong M, Zhao Q, Ji J, Li L, Zhou M (2014) An overview of BCC climate system model development and application for climate change studies. J Meteorol Res 28(1):34–56. https://doi.org/10.1007/s13351-014-3041-7

    Article  Google Scholar 

  53. Wuebbles DJ, Kunkel K, Wehner M, Zobel Z (2014) Severe weather in United States under a changing climate. Eos Trans Am Geophys Union 95(18):149–150. https://doi.org/10.1002/2014EO180001

    Article  Google Scholar 

  54. Wuebbles DJ, Fahey DW, Hibbard KA, Dokken DJ, Stewart BC, Maycock TK (eds) (2017) US Global Change Research Program, Washington, DC, USA. https://doi.org/10.7930/j0j964j6

  55. Yukimoto S (2011) Meteorological research institute earth system model version 1 (MRI-ESM1): model description

  56. Yukimoto S, Adachi Y, Hosaka M, Sakami T, Yoshimura H, Hirabara M, Tanaka TY, Shindo E, Tsujino H, Deushi M, Mizuta R, Yabu S, Obata A, Nakano H, Koshiro T, Ose T, Kitoh A (2012) A new global climate model of the meteorological research institute: MRI-CGCM3—model description and basic performance. J Meteorol Soc Jpn Ser II 90A:23–64. https://doi.org/10.2151/jmsj.2012-A02

    Article  Google Scholar 

Download references

Acknowledgements

We would like to acknowledge the World Climate Research Programme’s Working Group on Coupled Modelling, which is responsible for CMIP, and we thank the climate modeling groups (listed in “Appendix B” of this paper) for producing and making available their model output. For CMIP the U.S. Department of Energy’s Program for Climate Model Diagnosis and Intercomparison provides coordinating support and led development of software infrastructure in partnership with the Global Organization for Earth System Science Portals. The primary author would also like to acknowledge California State University, Los Angeles NASA DIRECT-STEM program and director, Dr. Hengchun Ye for funding and support. This research was carried out at the Jet Propulsion Laboratory, California Institute of Technology, under a contract with the National Aeronautics and Space Administration. Support for this project was provided by NASA National Climate Assessment 11-NCA 11-0028. Kenneth Kunkel was supported by NOAA through the Cooperative Institute for Climate and Satellites—North Carolina under Cooperative Agreement NA14NES432003.

Author information

Affiliations

Authors

Corresponding author

Correspondence to Jinny Lee.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary material 1 (DOCX 886 kb)

Appendices

Appendix A

Methodology equations

For each NCA region, the seasonal mean time series from the reference data is represented as:

$$x\left( t \right), \quad \left( {t = 1, 2, \ldots , m} \right)$$
(1)

where t the year from the starting year in each time block. m number of years in the time block.

The regionally-averaged time series from ensemble member i is defined as:

$$y_{i} \left( t \right), \quad \left( {t = 1, 2, \ldots , m} \right)$$
(2)

In addition, the regionally-averaged time series from the MME is represented as:

$$Y\left( t \right), \quad \left( {t = 1, 2, \ldots , m} \right)$$
(3)

where the ensemble average of N simulations is calculated using the following equation:

$$Y\left( t \right) = \frac{1}{N}\mathop \sum \limits_{k = 1}^{N} y_{i} \left( t \right)$$
(4)

See Table 2 for the number of simulations (N) used to calculate Y(t) in each time block.

The seasonal mean trend for the reference data, \(\alpha_{ref}\) [K year−1], is defined as the least square fit for a linear regression model:

$$x = \alpha_{ref} \times t + \beta_{ref}$$
(5)

The linear trend, \(\alpha_{yi}\) [K year−1], for ensemble members, \(y_{i} \left( t \right)\), and the ensemble linear trend, \(\alpha_{Y}\), for MME, \(Y\left( t \right)\), is calculated in the same manner for three time blocks (1895–1939, 1940–1979, 1980–2005). The choice of the time blocks is based on the observed warming and cooling trends, and closely mimics those in Kunkel et al. (2006).

The performance metric of the simulated trends in each region are:

  1. (a)

    trend bias of the MME, \(\alpha_{Y} - \alpha_{ref}\),

  2. (b)

    trend biases of ensemble members, \(\alpha_{yi} - \alpha_{ref}\),

  3. (c)

    percentage of the ensemble members reproducing the same sign (±) trend as the observed trend, and

  4. (d)

    percentage of the ensemble members whose trend biases are small relative to standard errors of the observed and simulated trends.

For (a) and (b), the following null hypothesis is tested per time block per region.

$$H_{o} : \alpha_{ref} = \alpha_{Y} \quad {\text{for}}\; ( {\text{a}}).$$
(6)
$$H_{o} : \alpha_{ref} = \alpha_{yi} \quad {\text{for}}\; ( {\text{b}}).$$
(7)

For the reference, linear trend calculation, the standard error of \(\alpha_{ref}\) is defined as (Hogg and Tanis 2009):

$$s_{ref} = \sqrt {\frac{{\mathop \sum \nolimits_{k = 1}^{m} \left( {x_{k} - \hat{x}_{k} } \right)^{2} /\left( {m - 2} \right)}}{{\mathop \sum \nolimits_{k = 1}^{m} \left( {t_{k} - \bar{t}} \right)^{2} }}}$$
(8)

where

$$\hat{x}_{k} = \alpha_{ref} \times k + \beta_{ref}$$
(9)

and

$$\bar{t} = \frac{1}{m}\mathop \sum \limits_{k = 1}^{m} k$$
(10)

It should be noted that (a), the trend bias of the MME, has high dependence on some models that contribute many ensemble members. For instance, there are models that contribute as little as one simulation or as many as 25 different ensemble members. In the case of a model contributing large ensemble simulations, this particular model will bear greater weight to the overall regional mean because each simulation is weighted equally when calculating a MME (Eq. 4). Considering the unequal weights of models in \(\alpha_{Y} ,\) the standard error of \(\alpha_{Y}\) was computed by randomly selecting N individual model trends with replacement (called bootstrapping) and computing the mean of that selection. We repeated this sampling 1000 times to obtain standard deviation across the 1000 random ensemble trends and use it as \(\alpha_{Y}\)’s standard error \((s_{Y} )\). We compared \(Y's\) bias \((\alpha_{Y} - \alpha_{ref} )\) with \(s_{ref}\) and \(s_{ref}\) to test the null hypothesis (6).

To assess the statistical significance of (b), the trend bias for simulation i, (\(\alpha_{yi} - \alpha_{ref}\)), it is reasonable to assume that \(\alpha_{ref}\) and \(\alpha_{yi}\) likely have unequal variances. Therefore, the Welch’s t test statistic \(\left( {T_{i} } \right)\) is used to estimate the statistical significance of (\(\alpha_{yi} - \alpha_{ref}\)). \(T_{i}\) is defined as (Hogg and Tanis 2009):

$$T_{i} = \frac{{\alpha_{yi} - \alpha_{ref} }}{{\sqrt {\frac{{s_{ref}^{2} + s_{yi}^{2} }}{m - 2}} }}$$
(11)

Using the Welch–Satterthwaite equation, the degrees of freedom, \(f_{i}\), for \(T_{i}\) can be approximated by:

$$f_{i} \approx \frac{{\left( {\frac{{s_{ref}^{2} + s_{yi}^{2} }}{m - 2}} \right)^{2} }}{{\frac{{s_{ref}^{4} + s_{yi}^{4} }}{{\left( {m - 2} \right)^{2} \left( {m - 3} \right)^{2} }}}}$$
(12)

Let \(C^{{f_{i} }}\) be the cumulative density function of a student’s t-distribution with \(f_{i}\) be the number of degrees of freedom. Then,

$$p_{i} = C^{{f_{i} }} \left( {T_{i} } \right)$$
(12)

and using \(p_{i}\), the confidence level \((d_{i} )\) of \(\alpha_{yi} - \alpha_{ref}\) can be calculated and used as a metric:

$${\text{when}}\;p_{i} < 0.5,\;d_{i} = \left( {1 - 2 \times p_{i} } \right)*100\left[ \% \right]$$
(13)
$${\text{when}}\;p_{i} > 0.5,\;d_{i} = \left( {2 \times p_{i} - 1} \right)*100\left[ \% \right]$$
(14)
$${\text{when}}\;p_{i} = 0.5,\; \alpha_{ref} = \alpha_{yi} ,\;{\text{therefore}}\;d_{i} = 0\%$$
(15)

The null hypothesis \((H_{o} )\) is rejected if \(T_{i}\) and \(p_{i}\) are too small (indicating \(\alpha_{yi} \ll \alpha_{ref}\)), or too large (indicating \(\alpha_{yi} \gg \alpha_{ref}\)). In this case, \(\alpha_{yi}\) is statistically different from \(\alpha_{ref}\) at a confidence level of \(d_{i}\). We calculated \(d_{i}\) of the 206 simulations for each period and region, and show a fraction of simulations whose trend biases are not statistically significant with 90% confidence level. In other words, the fraction represents how many simulations reproduce observed trends considering standard errors of the trends.

Part (c) calculated the total percentage of N simulations in which the \(\alpha_{yi}\) and \(\alpha_{ref}\) have the same sign. If the product of the two trends are greater than 0, than the two carry the same warming (cooling) trend. The total tally count is divided by the number of simulations and multiplied by 100 to produce a percentage as follows:

$$f = \frac{{\mathop \sum \nolimits_{i = 1}^{N} {\text{X}}_{i} }}{N}*100\;{\text{where}}\;{\text{X}}_{i} = \left\{ {\begin{array}{*{20}l} 1 \hfill & {{\text{if}}\;\alpha_{yi} \cdot \alpha_{ref} \ge 0} \hfill \\ 0 \hfill & { {\text{if}}\;\alpha_{yi} \cdot \alpha_{ref} < 0} \hfill \\ \end{array} } \right.$$
(16)

In a similar manner, part (d) also produces a fraction examines the magnitude of the warming (cooling) trend. If a trend of a given simulation and its standard deviation ranges intersects with the reference and its standard error [as calculated with the equation from Hogg and Tanis (2009)], a tally is given. The total tally count is divided by the number of simulations and multiplied by 100 to produce a percentage as follows:

$$f = \frac{{\mathop \sum \nolimits_{i = 1}^{N} {\text{X}}_{i} }}{N}*100\;{\text{where}}\;{\text{X}}_{i} = \left\{ {\begin{array}{*{20}l} 0 \hfill & { {\text{if}}\;(\alpha_{yi} \pm 1\sigma ) \cap (\alpha_{ref} \pm SE) = \emptyset } \hfill \\ 1 \hfill & {\text{otherwise}} \hfill \\ \end{array} } \right.$$
(17)

Appendix B

Summary of CMIP5 historical simulation dataset

Modeling center
(Country)
Model Simulations References
Commonwealth Scientific and Industrial Research Organization Bureau of Meteorology
(Australia)
ACCESS1-0 r1i1p1
r2i1p1
Collier and Uhe (2012)
ACCESS1-3 r1i1p1
r2i1p1
r3i1p1
Beijing Climate Center
(China)
bcc-csm1-1 r1i1p1
r2i1p1
r3i1p1
Wu et al. (2014)
bcc-csm1-1-m r1i1p1
r2i1p1
r3i1p1
Beijing Normal University
(China)
BNU-ESM r1i1p1 Ji et al. (2014)
Canadian Center for Climate Modeling and Analysis
(Canada)
CanCM4 r1i1p1
r2i1p1
r3i1p1
r4i1p1
r5i1p1
r6i1p1
r7i1p1
r8i1p1
r9i1p1
r10i1p1
Chylek et al. (2011)
CanESM2 r1i1p1
r2i1p1
r3i1p1
r4i1p1
r5i1p1
National Center for Atmospheric Research
(USA)
CCSM4 r1i1p1
r2i1p1
r3i1p1
r4i1p1
r5i1p1
r6i1p1
Collins et al. (2004)
CESM1-BGC r1i1p1
CESM1-CAM5 r1i1p1
r2i1p1
r3i1p1
CESM1-FASTCHEM r1i1p1
r2i1p1
r3i1p1
CESM1-WACCM r1i1p1 Marsh et al. (2013)
r2i1p1
r3i1p1
r4i1p1
Centro Euro-Mediterraneo sui Cambiamenti Climatici
(Italy)
CMCC-CESM r1i1p1 Fogli and Iovino (2014)
CMCC-CM r1i1p1 Scoccimarro et al. (2011)
CMCC-CMS r1i1p1
Center National de Recherches Meteorologiques
Center Europeen de Recherche et de Formation Avancee en Calcul Scientifique
(France)
CNRM-CM5 r1i1p1
r2i1p1
r3i1p1
r4i1p1
r5i1p1
r6i1p1
r7i1p1
r8i1p1
r9i1p1
r10i1p1
Voldoire et al. (2013)
CNRM-CM5-2 r1i1p1
Commonwealth Scientific and Industrial Research Organization
Queensland Climate Change Center of Excellence
(Australia)
CSIRO-Mk3-6-0 r1i1p1
r2i1p1
r3i1p1
r4i1p1
r5i1p1
r6i1p1
r7i1p1
r8i1p1
r9i1p1
r10i1p1
Gordon et al. (2010)
EC-EARTH Consortium published at Irish Center of High-End Computing
(Netherlands/Ireland)
EC-EARTH r1i1p1 Hazeleger et al. (2012)
r2i1p1
r6i1p1
r7i1p1
r8i1p1
r9i1p1
r11i1p1
r12i1p1
r13i1p1
r14i1p1
Institute of Atmospheric Physics
Chinese Academy of Sciences
(China)
FGOALS-g2 r1i1p1 Li et al. (2013)
r2i1p1
r3i1p1
r4i1p1
r5i1p1
The First Institute of Oceanography, SOA
(China)
FIO r1i1p1
r2i1p1
r3i1p1
Qiao et al. (2013)
Geophysical Fluid Dynamics Laboratory
(USA)
GFDL-CM2p1 r1i1p1
r2i1p1
r3i1p1
r4i1p1
r5i1p1
r6i1p1
r7i1p1
r8i1p1
r9i1p1
r10i1p1
Delworth et al. (2006)
GFDL-CM3 r1i1p1
r2i1p1
r3i1p1
r4i1p1
r5i1p1
Donner et al. (2011)
GFDL-ESM2G r1i1p1 Dunne et al. (2013)
GFDL-ESM2 M r1i1p1
NASA/GISS
(USA)
GISS-E2-H r1i1p1
r2i1p1
r3i1p1
r4i1p1
r5i1p1
Schmidt et al. (2014)
r6i1p1
GISS-E2-H-CC r1i1p1
GISS-E2-R r1i1p1
r2i1p1
r3i1p1
r4i1p1
r5i1p1
r6i1p1
r1i1p2
r2i1p2
r3i1p2
r4i1p2
r5i1p2
r6i1p2
r1i1p3
r2i1p3
r3i1p3
r4i1p3
NASA/GISS
(USA)
GISS-E2-R r5i1p3
r6i1p3
Schmidt et al. (2014)
r1i1p121
r1i1p122
r1i1p124
r1i1p125
r1i1p126
r1i1p127
r1i1p128
GISS-E2-R-CC r1i1p2
Met Office Hadley Center
(UK)
HadCM3 r1i1p1
r2i1p1
r3i1p1
r4i1p1
r5i1p1
r6i1p1
r7i1p1
r8i1p1
r9i1p1
r10i1p1
Pope et al. (2000)
HadGEM2-CC r1i1p1  
HadGEM2-ES r1i1p1
r2i1p1
r3i1p1
r4i1p1
r5i1p1
National Institute of Meteorological Research
Korea Meteorological Administration
(South Korea)
HadGEM2-AO r1i1p1 Collins et al. (2011)
Baek et al. (2013)
Russian Academy of Sciences
Institute of Numerical Mathematics
(Russia)
inmcm4 r1i1p1 Volodin et al. (2010)
Institut Pierre Simon Laplace
(France)
IPSL-CM5A-LR r1i1p1
r2i1p1
r3i1p1
r4i1p1
r5i1p1
r6i1p1
Dufresne et al. (2013)
IPSL-CM5A-MR r1i1p1
r2i1p1
r3i1p1
IPSL-CM5B-LR r1i1p1
Atmosphere and Ocean Research Institute
(The University of Tokyo)
National Institute for Environmental Studies
Japan Agency for Marine-Earth Science and Technology
(Japan)
MIROC-ESM r1i1p1
r2i1p1
r3i1p1
Watanabe et al. (2011)
MIROC-ESM-CHEM r1i1p1
MIROC4 h r1i1p1
r2i1p1
r3i1p1
Sakamoto et al. (2012)
MIROC5 r1i1p1
r2i1p1
r3i1p1
r4i1p1
r5i1p1
Watanabe et al. (2010)
Max Planck Institute for Meteorology
(Germany)
MPI-ESM-LR r1i1p1
r2i1p1
r3i1p1
Stevens et al. (2013)
MPI-ESM-MR r1i1p1
r2i1p1
r3i1p1
MPI-ESM-P r1i1p1
r2i1p1
Meteorological Research Institute
(Japan)
MRI-CGCM3 r1i1p1
r2i1p1
r3i1p1
Yukimoto et al. (2012)
MRI-ESM1 r4i1p2
r5i1p2
Yukimoto (2011)
Bjerknes Center for Climate Research
Norwegian Meteorological Institute
(Norway)
NorESM1-M r1i1p1
r2i1p1
r3i1p1
Bentsen et al. (2013)
NorESM1-ME r1i1p1

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Lee, J., Waliser, D., Lee, H. et al. Evaluation of CMIP5 ability to reproduce twentieth century regional trends in surface air temperature and precipitation over CONUS. Clim Dyn 53, 5459–5480 (2019). https://doi.org/10.1007/s00382-019-04875-1

Download citation

Keywords

  • CMIP5
  • Model evaluation
  • Surface air temperature
  • Multi-model ensemble