Finding plausible and diverse variants of a climate model. Part II: development and validation of methodology

Abstract

The usefulness of a set of climate change projections largely depends on how well it spans a range of outcomes consistent with known uncertainties. Here, we present exploratory work towards developing a strategy to select variants of a state-of-the-art but expensive climate model suitable for climate projection studies. The strategy combines information from a set of relatively cheap, idealized perturbed parameter ensemble (PPE) and CMIP5 multi-model ensemble (MME) experiments, and uses two criteria as the basis to select model variants for a PPE suitable for future projections: (a) acceptable model performance at two different timescales, and (b) maintaining diversity in model response to climate change. This second part of a pair of papers builds upon Part I in which we established a strong relationship between model errors at weather and climate timescales across a PPE for a variety of key variables. This relationship is used to filter out parts of parameter space that do not give credible simulations of present day climate, while minimizing the impact on ranges in forcings and feedbacks that drive model responses to climate change. We use statistical emulation to explore the parameter space thoroughly, and demonstrate that about 90% can be filtered out without affecting diversity in global-scale climate change responses. This leads to the identification of plausible parts of parameter space from which model variants can be selected for projection studies. We selected and ran 50 variants from the plausible parameter combinations and validated the emulator predictions. Comparisons with the CMIP5 MME demonstrate that our approach can produce a set of plausible model variants that span a relatively wide range in model response to climate change. We also highlight how the prior expert-specified ranges for uncertain model parameters are constrained as a result of our methodology, and discuss recommendations for future work.

Appendices

Appendix 1: experimental design details

• (i) Transpose-AMIP (TAMIP)

  
  

  Our design for this experiment is based on the second phase of Transpose-Atmospheric Model Intercomparison Project (TAMIP-II) activity as part the fifth phase of Coupled Model Intercomparison Project (CMIP5)(http://www.transpose-amip.info; Williams et al. 2013). The TAMIP-II simulations are 5 days in length and the state variables are initialized from 16 Met Office Analyses (global analyses of observations using the Met Office Numerical Weather Prediction Model). The 16 analyses were sampled by picking four dates from each season between October 2008 and August 2009, and exploring the diurnal cycle within each season by sampling start times of 000Z, 0600Z, 1200Z, and 1800Z once. This approach allows a relatively inexpensive examination of error growth from observed initial conditions on relatively short time scales (see Sect. 2.1), and facilitates evaluation of the processes responsible for the origin of model biases in our perturbed climate model variants. Since the atmospheric initial states did not include aerosol information (see Sect. 3), aerosols were represented in the TAMIP experiments by the use of a simple prescribed climatology from HadGEM2-ES (Collins et al. 2011) whereas the other experiments all use an interactive scheme predicting aerosol concentrations from precursor emissions.

• (ii) ATMOS

  
  

  ATMOS runs are 10 years long (1999–2008) and are used to assess model performance on climate time scales. We chose this recent period to provide maximum overlap with observational datasets. Since the vegetation response to changing climate conditions is relatively slow (at least 20–30 years), we decided not to use a dynamic vegetation scheme in these simulations, instead prescribing the same observed vegetation distribution used in previous HadGEM2 simulations lacking an interactive representation of the earths carbon cycle. The statistical method used to quantify model error in this study requires an estimate of the component of uncertainty due to internal variability (Sect. 5). This was obtained by running 16 ATMOS simulations using the standard version of HadGEM3-A, from 16 quasi-independent initial conditions obtained from different model variants in our ensemble.

• (iii) ATMOS + $4\times \hbox {CO}_2$

  
  

  This experiment has the same set-up as the ATMOS experiment above but is run for one year with instantaneous quadrupling of $\hbox {CO}_2$. It is designed to estimate $\hbox {CO}_2$ forcing in individual model variants, and its uncertainties across the PPE. The $4 \times \hbox {CO}_2$ forcing is the change in the net TOA radiation in this run relative to the first year of the ATMOS run. This experimental setup is the same as that used in CFMIP simulations included in CMIP5 (Expt 6.5, see http://cmip-pcmdi.llnl.gov/cmip5/docs/Taylor_CMIP5_design.pdf). Our results can therefore be compared with those from these other models, with the caveat that our forcing estimates for individual PPE variants are subject to larger uncertainty from internal variability than those for individual CMIP5 models, since the latter use the longer 30 year simulation period recommended in the CFMIP/CMIP5 experimental design. However, this sampling issue is expected to play a smaller role in determining the overall range of PPE estimates, facilitating comparison against the CMIP5 range (see discussion of Fig. 5 in Sect. 4.2).

• (iv) ATMOS + patterned + 4K SST (amipFuture)

  
  

  This experiment has the same set-up as the ATMOS experiment above but is run for 5 years with a prescribed + 4K SST pattern (Expt 6.6 in http://cmip-pcmdi.llnl.gov/cmip5/docs/Taylor_CMIP5_design.pdf, referred to as `amipFuture'). This experiment is used to examine the strength of climate feedbacks and associated uncertainties, estimated by prescribing climate change through changes in SST (see Webb and Lock 2013). The imposed SST pattern is obtained from coupled model simulations of 13 CMIP3 AOGCMs at the time of CO₂ quadrupling from simulations driven by a 1% per year increase in CO₂ concentration. Sea-ice extents are not perturbed in this setup. Feedbacks are quantified as the difference between net TOA radiation in the + 4K SST experiment and the corresponding ATMOS experiment, divided by the surface temperature response. The corresponding CMIP5 simulations were 30 years in length, so as in (iii) above. Internal variability is expected to contribute a larger sampling uncertainty to feedback estimates in individual PPE simulations. However, the impact on the cross-ensemble range, in comparison to that of CMIP5 (see Fig. 5 and Sect. 4.2), is expected to be smaller.

• (v) ATMOS + preindustrial aerosols

  
  

  The ATMOS experiments in (ii)–(iv) above include the CLASSIC aerosol module of Bellouin et al. (2011), which predicts aerosol concentrations of certain species (sulphate, soot, biomass, organic carbon) interactively, from emissions of their natural and anthropogenic precursors. In addition, we ran 1-year pre-industrial aerosol simulations including only natural (i.e. 1860) emissions, with an experimental setup otherwise identical to the simulation using the standard ATMOS protocol of (ii). This experiment allowed us to obtain a rough estimate of anthropogenic aerosol forcing and associated uncertainties by computing changes in net TOA radiation in the run with pre-industrial emissions, relative to the first year of the corresponding ATMOS run from (ii).

Appendix 2: variables and verification datasets

Verification datasets used to assess TAMIP and ATMOS performance for the variables listed include gridded station and satellite observations and reanalysis data and are tabulated in Table 3. In each case, we selected one verification dataset (Obs 1) to calculate model performance scores, and used the second one (Obs 2; if available) to determine an estimate of observational error.

Table 3 Variables and verification datasets for model evaluation. References: ERAI (Dee et al. 2011), NCEP 20CR (Compo et al. 2011), GPCP (Adler et al. 2003), CMAP (Xie and Arkin 1996), TRMM (Huffman et al. 2007), HadSLP2 (Allan and Ansell 2006), MERRA (Rienecker et al. 2011), ISCCP-D2 (Rossow and Schier 1999), CERES (Loeb et al. 2009)

Appendix 3: leave-one-out (LOO) validation for TAMIP and ATMOS emulators

The leave-one-out (LOO) cross validation scores for TAMIP and ATMOS emulators for the six assessment variables are shown in Figs. 15 and 16, respectively. We use Gaussian Process emulators (see Part I for more information) as described in Lee et al. (2011) but the emulators are built for global MSEs of individual variables for TAMIP and ATMOS separately instead of emulating individual grid points. The $\hbox {R}^2$-values of the LOO estimates plotted against the values of the left-out member are at least 0.7 for all TAMIP MSEs, and at least 0.6 for the ATMOS MSEs. Overall, high number of model failures in the ATMOS case due to instabilities in HadGEM3-GA4.0 meant limited training data (80) available for building ATMOS emulators. As a result, the ATMOS emulators are clearly less reliable than TAMIP. The main issue is a poor fit for high MSEs for pr, ta850, and ua250, perhaps due to a lack of training data at the high end. Also note that one of the outliers in these plots is from the same member of the ensemble which had low values for all the parameters perturbed and high MSEs. Another possibility is that we define the Gaussian Process to have a linear term and correlation length scale for each parameter, and this means a large number of parameters relative to the ensemble size. Originally, we used these emulators for both parts I and II to select 50 parameter combinations and run the validation ensemble. Since then, we improved the method to build the emulator by reducing the number of linear terms using a stepwise algorithm prior to fitting the Gaussian Process; potentially a similar reduction could be used for the correlation terms. In part I we have used the improved emulators but have retained the original emulators here for consistency with the validation ensemble.

Fig. 15

The leave-one-out cross validation plots for the TAMIP emulators for the six assessment variables (columns). In the first row, the ensemble members are ordered according to their fitted MSEs (transformed space) indicated by the dark blue line, the blue lines are 95% confidence intervals for TAMIP, and blue and red dots show actual values that are either within or outside the confidence intervals. The second row shows the comparison between the exact and fitted values for TAMIP

Fig. 16

Same as Fig. 15 but for ATMOS

Whilst the fit for these high MSEs for pr, ta850, and ua250 is poor in absolute terms, it does not affect the ranking too much. For example, five of the six members with the worst precipitation MSEs in the ATMOS experiment are predicted to have high MSEs by the emulator. These five would still be ruled out by the filtering. The other member of these six is predicted to have a moderate MSE value and may well have been accepted by the filtering. It may then have been selected to be run as an ATMOS experiment, then evaluated as a poor performer, and so not made it to the stage where it would be run in coupled mode. This is not ideal because it would have wasted a run of the ATMOS experiment. If we had been intending to run a coupled ensemble, rather than simply demonstrate the selection algorithm, then we would have had to have run a larger ATMOS ensemble to build better emulators and reduce the risk of this happening.