Evaluation of CMIP5 ability to reproduce twentieth century regional trends in surface air temperature and precipitation over CONUS

Abstract

The ability of the 5th phase of the Coupled Model Intercomparison Project (CMIP5) to reproduce twentieth-century climate trends over the seven CONUS regions of the National Climate Assessment is evaluated. This evaluation is carried out for summer and winter for three time periods, 1895–1939, 1940–1979, and 1980–2005. The evaluation includes all 206 CMIP5 historical simulations from 48 unique models and their multi-model ensemble (MME), as well as a gridded in situ dataset of surface air temperature and precipitation. Analysis is performed on both individual members and the MME, and considers reproducing the correct sign of the trends by the members as well as reproducing the trend values. While the MME exhibits some trend bias in most cases, it reproduces historical temperature trends with reasonable fidelity for summer for all time periods and all regions, including at the CONUS scale, except the Northern Great Plains from 1895 to 1939 and Southeast during 1980–2005. Likewise, for DJF, the MME reproduces historical temperature trends across all time periods over all regions, including at the CONUS scale, except the Southeast from 1895 to 1939 and the Midwest during 1940–1979. Model skill was highest across all of the seven regions during JJA and DJF for the 1980–2005 period. The quantitatively best result is seen during DJF in the Southwest region with at least 74% of the ensemble members correctly reproducing the observed trend across all of the time periods. No clear trends in MME precipitation were identified at these scales due to high model precipitation variability.

Appendices

Appendix A

1.1 Methodology equations

For each NCA region, the seasonal mean time series from the reference data is represented as:

$$x\left( t \right), \quad \left( {t = 1, 2, \ldots , m} \right)$$

(1)

where t the year from the starting year in each time block. m number of years in the time block.

The regionally-averaged time series from ensemble member i is defined as:

$$y_{i} \left( t \right), \quad \left( {t = 1, 2, \ldots , m} \right)$$

(2)

In addition, the regionally-averaged time series from the MME is represented as:

$$Y\left( t \right), \quad \left( {t = 1, 2, \ldots , m} \right)$$

(3)

where the ensemble average of N simulations is calculated using the following equation:

$$Y\left( t \right) = \frac{1}{N}\mathop \sum \limits_{k = 1}^{N} y_{i} \left( t \right)$$

(4)

See Table 2 for the number of simulations (N) used to calculate Y(t) in each time block.

The seasonal mean trend for the reference data, $\alpha_{ref}$ [K year⁻¹], is defined as the least square fit for a linear regression model:

$$x = \alpha_{ref} \times t + \beta_{ref}$$

(5)

The linear trend, $\alpha_{yi}$ [K year⁻¹], for ensemble members, $y_{i} \left( t \right)$, and the ensemble linear trend, $\alpha_{Y}$, for MME, $Y\left( t \right)$, is calculated in the same manner for three time blocks (1895–1939, 1940–1979, 1980–2005). The choice of the time blocks is based on the observed warming and cooling trends, and closely mimics those in Kunkel et al. (2006).

The performance metric of the simulated trends in each region are:

• (a) trend bias of the MME, $\alpha_{Y} - \alpha_{ref}$,

• (b) trend biases of ensemble members, $\alpha_{yi} - \alpha_{ref}$,

• (c) percentage of the ensemble members reproducing the same sign (±) trend as the observed trend, and

• (d) percentage of the ensemble members whose trend biases are small relative to standard errors of the observed and simulated trends.

For (a) and (b), the following null hypothesis is tested per time block per region.

$$H_{o} : \alpha_{ref} = \alpha_{Y} \quad {\text{for}}\; ( {\text{a}}).$$

(6)

$$H_{o} : \alpha_{ref} = \alpha_{yi} \quad {\text{for}}\; ( {\text{b}}).$$

(7)

For the reference, linear trend calculation, the standard error of $\alpha_{ref}$ is defined as (Hogg and Tanis 2009):

$$s_{ref} = \sqrt {\frac{{\mathop \sum \nolimits_{k = 1}^{m} \left( {x_{k} - \hat{x}_{k} } \right)^{2} /\left( {m - 2} \right)}}{{\mathop \sum \nolimits_{k = 1}^{m} \left( {t_{k} - \bar{t}} \right)^{2} }}}$$

(8)

where

$$\hat{x}_{k} = \alpha_{ref} \times k + \beta_{ref}$$

(9)

and

$$\bar{t} = \frac{1}{m}\mathop \sum \limits_{k = 1}^{m} k$$

(10)

It should be noted that (a), the trend bias of the MME, has high dependence on some models that contribute many ensemble members. For instance, there are models that contribute as little as one simulation or as many as 25 different ensemble members. In the case of a model contributing large ensemble simulations, this particular model will bear greater weight to the overall regional mean because each simulation is weighted equally when calculating a MME (Eq. 4). Considering the unequal weights of models in $\alpha_{Y} ,$ the standard error of $\alpha_{Y}$ was computed by randomly selecting N individual model trends with replacement (called bootstrapping) and computing the mean of that selection. We repeated this sampling 1000 times to obtain standard deviation across the 1000 random ensemble trends and use it as $\alpha_{Y}$’s standard error $(s_{Y} )$. We compared $Y's$ bias $(\alpha_{Y} - \alpha_{ref} )$ with $s_{ref}$ and $s_{ref}$ to test the null hypothesis (6).

To assess the statistical significance of (b), the trend bias for simulation i, ($\alpha_{yi} - \alpha_{ref}$), it is reasonable to assume that $\alpha_{ref}$ and $\alpha_{yi}$ likely have unequal variances. Therefore, the Welch’s t test statistic $\left( {T_{i} } \right)$ is used to estimate the statistical significance of ($\alpha_{yi} - \alpha_{ref}$). $T_{i}$ is defined as (Hogg and Tanis 2009):

$$T_{i} = \frac{{\alpha_{yi} - \alpha_{ref} }}{{\sqrt {\frac{{s_{ref}^{2} + s_{yi}^{2} }}{m - 2}} }}$$

(11)

Using the Welch–Satterthwaite equation, the degrees of freedom, $f_{i}$, for $T_{i}$ can be approximated by:

$$f_{i} \approx \frac{{\left( {\frac{{s_{ref}^{2} + s_{yi}^{2} }}{m - 2}} \right)^{2} }}{{\frac{{s_{ref}^{4} + s_{yi}^{4} }}{{\left( {m - 2} \right)^{2} \left( {m - 3} \right)^{2} }}}}$$

(12)

Let $C^{{f_{i} }}$ be the cumulative density function of a student’s t-distribution with $f_{i}$ be the number of degrees of freedom. Then,

$$p_{i} = C^{{f_{i} }} \left( {T_{i} } \right)$$

(12)

and using $p_{i}$, the confidence level $(d_{i} )$ of $\alpha_{yi} - \alpha_{ref}$ can be calculated and used as a metric:

$${\text{when}}\;p_{i} < 0.5,\;d_{i} = \left( {1 - 2 \times p_{i} } \right)*100\left[ \% \right]$$

(13)

$${\text{when}}\;p_{i} > 0.5,\;d_{i} = \left( {2 \times p_{i} - 1} \right)*100\left[ \% \right]$$

(14)

$${\text{when}}\;p_{i} = 0.5,\; \alpha_{ref} = \alpha_{yi} ,\;{\text{therefore}}\;d_{i} = 0\%$$

(15)

The null hypothesis $(H_{o} )$ is rejected if $T_{i}$ and $p_{i}$ are too small (indicating $\alpha_{yi} \ll \alpha_{ref}$), or too large (indicating $\alpha_{yi} \gg \alpha_{ref}$). In this case, $\alpha_{yi}$ is statistically different from $\alpha_{ref}$ at a confidence level of $d_{i}$. We calculated $d_{i}$ of the 206 simulations for each period and region, and show a fraction of simulations whose trend biases are not statistically significant with 90% confidence level. In other words, the fraction represents how many simulations reproduce observed trends considering standard errors of the trends.

Part (c) calculated the total percentage of N simulations in which the $\alpha_{yi}$ and $\alpha_{ref}$ have the same sign. If the product of the two trends are greater than 0, than the two carry the same warming (cooling) trend. The total tally count is divided by the number of simulations and multiplied by 100 to produce a percentage as follows:

$$f = \frac{{\mathop \sum \nolimits_{i = 1}^{N} {\text{X}}_{i} }}{N}*100\;{\text{where}}\;{\text{X}}_{i} = \left\{ {\begin{array}{*{20}l} 1 \hfill & {{\text{if}}\;\alpha_{yi} \cdot \alpha_{ref} \ge 0} \hfill \\ 0 \hfill & { {\text{if}}\;\alpha_{yi} \cdot \alpha_{ref} < 0} \hfill \\ \end{array} } \right.$$

(16)

In a similar manner, part (d) also produces a fraction examines the magnitude of the warming (cooling) trend. If a trend of a given simulation and its standard deviation ranges intersects with the reference and its standard error [as calculated with the equation from Hogg and Tanis (2009)], a tally is given. The total tally count is divided by the number of simulations and multiplied by 100 to produce a percentage as follows:

$$f = \frac{{\mathop \sum \nolimits_{i = 1}^{N} {\text{X}}_{i} }}{N}*100\;{\text{where}}\;{\text{X}}_{i} = \left\{ {\begin{array}{*{20}l} 0 \hfill & { {\text{if}}\;(\alpha_{yi} \pm 1\sigma ) \cap (\alpha_{ref} \pm SE) = \emptyset } \hfill \\ 1 \hfill & {\text{otherwise}} \hfill \\ \end{array} } \right.$$

(17)

Appendix B

Summary of CMIP5 historical simulation dataset

Modeling center (Country)	Model	Simulations	References
Commonwealth Scientific and Industrial Research Organization Bureau of Meteorology (Australia)	ACCESS1-0	r1i1p1 r2i1p1	Collier and Uhe (2012)
	ACCESS1-3	r1i1p1 r2i1p1 r3i1p1
Beijing Climate Center (China)	bcc-csm1-1	r1i1p1 r2i1p1 r3i1p1	Wu et al. (2014)
	bcc-csm1-1-m	r1i1p1 r2i1p1 r3i1p1
Beijing Normal University (China)	BNU-ESM	r1i1p1	Ji et al. (2014)
Canadian Center for Climate Modeling and Analysis (Canada)	CanCM4	r1i1p1 r2i1p1 r3i1p1 r4i1p1 r5i1p1 r6i1p1 r7i1p1 r8i1p1 r9i1p1 r10i1p1	Chylek et al. (2011)
	CanESM2	r1i1p1 r2i1p1 r3i1p1 r4i1p1 r5i1p1
National Center for Atmospheric Research (USA)	CCSM4	r1i1p1 r2i1p1 r3i1p1 r4i1p1 r5i1p1 r6i1p1	Collins et al. (2004)
	CESM1-BGC	r1i1p1
	CESM1-CAM5	r1i1p1 r2i1p1 r3i1p1
	CESM1-FASTCHEM	r1i1p1 r2i1p1 r3i1p1
	CESM1-WACCM	r1i1p1	Marsh et al. (2013)
		r2i1p1 r3i1p1 r4i1p1
Centro Euro-Mediterraneo sui Cambiamenti Climatici (Italy)	CMCC-CESM	r1i1p1	Fogli and Iovino (2014)
	CMCC-CM	r1i1p1	Scoccimarro et al. (2011)
	CMCC-CMS	r1i1p1
Center National de Recherches Meteorologiques Center Europeen de Recherche et de Formation Avancee en Calcul Scientifique (France)	CNRM-CM5	r1i1p1 r2i1p1 r3i1p1 r4i1p1 r5i1p1 r6i1p1 r7i1p1 r8i1p1 r9i1p1 r10i1p1	Voldoire et al. (2013)
	CNRM-CM5-2	r1i1p1
Commonwealth Scientific and Industrial Research Organization Queensland Climate Change Center of Excellence (Australia)	CSIRO-Mk3-6-0	r1i1p1 r2i1p1 r3i1p1 r4i1p1 r5i1p1 r6i1p1 r7i1p1 r8i1p1 r9i1p1 r10i1p1	Gordon et al. (2010)
EC-EARTH Consortium published at Irish Center of High-End Computing (Netherlands/Ireland)	EC-EARTH	r1i1p1	Hazeleger et al. (2012)
		r2i1p1
		r6i1p1
		r7i1p1
		r8i1p1
		r9i1p1
		r11i1p1
		r12i1p1
		r13i1p1
		r14i1p1
Institute of Atmospheric Physics Chinese Academy of Sciences (China)	FGOALS-g2	r1i1p1	Li et al. (2013)
		r2i1p1
		r3i1p1
		r4i1p1
		r5i1p1
The First Institute of Oceanography, SOA (China)	FIO	r1i1p1 r2i1p1 r3i1p1	Qiao et al. (2013)
Geophysical Fluid Dynamics Laboratory (USA)	GFDL-CM2p1	r1i1p1 r2i1p1 r3i1p1 r4i1p1 r5i1p1 r6i1p1 r7i1p1 r8i1p1 r9i1p1 r10i1p1	Delworth et al. (2006)
	GFDL-CM3	r1i1p1 r2i1p1 r3i1p1 r4i1p1 r5i1p1	Donner et al. (2011)
	GFDL-ESM2G	r1i1p1	Dunne et al. (2013)
	GFDL-ESM2 M	r1i1p1
NASA/GISS (USA)	GISS-E2-H	r1i1p1 r2i1p1 r3i1p1 r4i1p1 r5i1p1	Schmidt et al. (2014)
		r6i1p1
	GISS-E2-H-CC	r1i1p1
	GISS-E2-R	r1i1p1 r2i1p1 r3i1p1 r4i1p1 r5i1p1 r6i1p1
		r1i1p2 r2i1p2 r3i1p2 r4i1p2 r5i1p2 r6i1p2
		r1i1p3 r2i1p3 r3i1p3 r4i1p3
NASA/GISS (USA)	GISS-E2-R	r5i1p3 r6i1p3	Schmidt et al. (2014)
		r1i1p121
		r1i1p122
		r1i1p124
		r1i1p125
		r1i1p126
		r1i1p127
		r1i1p128
	GISS-E2-R-CC	r1i1p2
Met Office Hadley Center (UK)	HadCM3	r1i1p1 r2i1p1 r3i1p1 r4i1p1 r5i1p1 r6i1p1 r7i1p1 r8i1p1 r9i1p1 r10i1p1	Pope et al. (2000)
	HadGEM2-CC	r1i1p1
	HadGEM2-ES	r1i1p1 r2i1p1 r3i1p1 r4i1p1 r5i1p1
National Institute of Meteorological Research Korea Meteorological Administration (South Korea)	HadGEM2-AO	r1i1p1	Collins et al. (2011) Baek et al. (2013)
Russian Academy of Sciences Institute of Numerical Mathematics (Russia)	inmcm4	r1i1p1	Volodin et al. (2010)
Institut Pierre Simon Laplace (France)	IPSL-CM5A-LR	r1i1p1 r2i1p1 r3i1p1 r4i1p1 r5i1p1 r6i1p1	Dufresne et al. (2013)
	IPSL-CM5A-MR	r1i1p1 r2i1p1 r3i1p1
	IPSL-CM5B-LR	r1i1p1
Atmosphere and Ocean Research Institute (The University of Tokyo) National Institute for Environmental Studies Japan Agency for Marine-Earth Science and Technology (Japan)	MIROC-ESM	r1i1p1 r2i1p1 r3i1p1	Watanabe et al. (2011)
	MIROC-ESM-CHEM	r1i1p1
	MIROC4 h	r1i1p1 r2i1p1 r3i1p1	Sakamoto et al. (2012)
	MIROC5	r1i1p1 r2i1p1 r3i1p1 r4i1p1 r5i1p1	Watanabe et al. (2010)
Max Planck Institute for Meteorology (Germany)	MPI-ESM-LR	r1i1p1 r2i1p1 r3i1p1	Stevens et al. (2013)
	MPI-ESM-MR	r1i1p1 r2i1p1 r3i1p1
	MPI-ESM-P	r1i1p1 r2i1p1
Meteorological Research Institute (Japan)	MRI-CGCM3	r1i1p1 r2i1p1 r3i1p1	Yukimoto et al. (2012)
	MRI-ESM1	r4i1p2 r5i1p2	Yukimoto (2011)
Bjerknes Center for Climate Research Norwegian Meteorological Institute (Norway)	NorESM1-M	r1i1p1 r2i1p1 r3i1p1	Bentsen et al. (2013)
	NorESM1-ME	r1i1p1