CMIP5: a Monte Carlo assessment of changes in summertime precipitation characteristics under RCP8.5-sensitivity to annual cycle fidelity, overconfidence, and gaussianity

Abstract

Using 5-day averaged precipitation from all initial condition realizations of 33 CMIP5 models for the Historical and RCP8.5 scenarios, we performed an assessment of summer precipitation in terms of amount, onset date, withdrawal date, and length of season using probability distributions of interannual anomalies. Climate change projections were generated using all models, one model per modelling group to account for overconfidence, and sub-selecting models on annual cycle fidelity. Compared to using all models, sub-selecting on annual cycle fidelity has a large impact on the climate change perturbation of the fractional change in precipitation, with differences between the two projections of up to ± 50%, especially in the tropics and subtropics. Sensitivity testing indicates that the Gaussian t-test and the non-parametric Mann–Whitney U-test (the latter using Monte Carlo sampling) yield consistent results for assessing where the climate change perturbation is significant at the 1% level, even in cases where skewness and excess kurtosis indicate non-Gaussian behavior. Similarly, in terms of climate change induced perturbations to below-normal, normal, and above-normal categorical probabilities, the Gaussian results are typically consistent with the non-parametric estimates. These sensitivity results promote the use of Gaussian statistics to present global maps of the lower-bound and upper-bound of the climate change response, given that the non-parametric calculation of confidence intervals would otherwise not be tractable in a desktop computing environment due to its CPU intensive requirement.

Appendix: methods

1.1 A.1 Terminology

In Sperber and Annamalai (2014) we used the term “duration” to represent the time from onset through withdrawal. However, herein we prefer the term “accumulation time” to avoid confusion with other climate change studies in which “duration” is defined differently (e.g., Kitoh et al. 2013). Herein, a shorter (longer) accumulation time means that it takes less (more) time under RCP.5 compared to the Historical simulation to achieve the fractional accumulation that occurred between observed pentads 31 and 55 (Fig. 1a–c).

1.2 A.2 Annual cycle fidelity sub-selection

The development of optimal strategies for combining climate projections is still in its “infancy” (Knutti et al. 2010), with three approaches typically used in the literature. Approach 1, using all available models with equal weighting, is the dominant method of generating projections (e.g., Kitoh et al. 2013). Numerous reasons exist for using all models, with (a) this being the most straightforward method to employ, (b) recognizing the effort made by all modelling groups to produce the necessary simulations, and (c) the assumption that using all models results in the best estimate of the climatological mean state projections by extension of the fact that the multi-model mean in present-day simulations tends to produce better agreement with observed climatologies than any individual model (Gleckler et al. 2008). Approach 2 addresses model independence considerations by producing projections that use one model version per modelling group (Fischer et al. 2014) to address the issue of overconfidence (Knutti et al. 2010), though this does not address the impact of model interdependence between different modelling groups (Sanderson et al. 2015). Approach 3 is physically based, with the sub-selection based on model fidelity in representing the mode of variability in question, with a uniform subset of models being used at all gridpoints. For example, this approach has been taken with respect to ENSO (Annamalai et al. 2007; van Oldenborgh et al. 2005) and the global monsoon (Wang et al. 2014). Approach 4, used in this paper, is a variant of Approach 3, but we sub-select models based on their annual cycle fidelity at each gridpoint, since it has been demonstrated that no single CMIP5 model performs well in terms of onset, withdrawal, and accumulation time of fractional precipitation over all monsoon domains (Sperber and Annamalai 2014). Spatial heterogeneity of the annual cycle of precipitation within monsoon domains has also been found, with Bombardi and Carvalho (2009) finding that the annual cycle is better represented over central and eastern Brazil compared to the Amazon and northwest South America in 10 CMIP3 models. Annual cycle errors have also been found over the Sahel (Vizy et al. 2013) and Australia (Coleman et al. 2011) in CMIP simulations. Even though results suggest that sub-selecting on model quality only weakly relates to the projected change (Knutti et al. 2010), our conservative approach is to sub-select models that are the physically realistic in terms of the timing of summer precipitation and its interannual variability in their historical simulations, since these quantities are directly relevant to the projected precipitation characteristics we are evaluating. In this way, we exclude models that have unrealistic annual cycle phasing of summer precipitation, and unrealistic levels of interannual variability, which might otherwise compromise the projections.

To subset the models we evaluate the onset pentad and accumulation time of summer rainfall in the Historical simulations. Consideration is also given to the standard deviations of the onset and accumulation times, with the goal of not including models whose Historical simulations are excessively under-dispersive or over-dispersive compared to GPCP and CMAP. Relevant to the discussion of model quality vs. climate change perturbations for area-averaged data (Sect. 5.1), in Table 2 are the means and standard deviations (σ) of the onset pentad and accumulation time (pentads) from GPCP, CMAP, and the CMIP5 Historical simulations for India (AIR), Sahel, Northern tropical Australia (Aus), North American Monsoon (NAM), South American Monsoon (SAM), and the southern subtropical Atlantic Ocean (SAtl).

The iterative procedure for sub-selecting models is based on performance of the CMIP5 Historical simulations relative to an “Optimal Range” for onset time, accumulation time, and their variances derived from GPCP and CMAP (1979–2004) with the:

• Initial “Optimal Range” from observations:

  • Onset time: mean ± 1σ.

  • Accumulation time: mean ± 1σ.

  • Onset variance: model variance must be within a factor of 1/2.52–2.73 of the observed variance (limits correspond to an F-test at the 1% level for 26 years of observations and 39 years of model data).

  • Accumulation time variance: model variance must be within a factor of 1/2.52–2.73 of the observed variance.

• If a model fails any of the criteria, the optimal ranges of the onset time and accumulation time are broadened by an additional ± 0.1σ up to a

  • maximum range = observed mean ± 1σ ± 1 pentad.

• If a model still fails any of the criteria, the maximum ranges of the onset time and accumulation time are used, and the variance range is incremented by 0.1 to a

  • maximum range = 1/2.79–3.

• If at any iteration 6 or more models meet the sub-selection criteria, the iterative process is stopped and the climate change impact for this subset of models is calculated.

• If the criteria are not met after relaxing the “Optimal Range” to their maximum ranges, climate change is not assessed at these gridpoints and the output is set to missing.

Note: GPCP has missing data at high northern latitudes and off the coast of Antarctica (shaded gray in Fig. 1d), hence no analysis is performed at these gridpoints.

1.3 A.3 Statistical analysis based on the Gaussian approach

We assess (1) changes in the mean by using a two-sided t-test at the 1% significance level, (2) determine the associated 99% confidence interval, and (3) evaluate changes in the probabilities of below-normal, normal, and above-normal tercile categories (Leith 1973) of the PDFs of interannual precipitation anomalies in the RCP8.5 simulations relative to those from the Historical simulations. For the case of having one initial condition realization per model we could calculate the perturbation to the tercile categories under RCP8.5 relative to the Historical simulations directly from the anomalies without assuming an underlying statistical model. However, because we have differing numbers of realizations from the individual models, using the full suite of anomalies in such an empirical approach would bias the probability estimates of the climate change signal toward models with more than one realization.

To ensure equal weighting for each model, for statistical purposes we assume the anomalies for each model to be Gaussian distributed. For each model realization we calculate Historical anomalies for each year with respect to that realization mean. For the RCP8.5 realizations performed using that model, the anomalies are calculated with respect to the ensemble mean of its available Historical realizations. For the evaluation of fractional changes in precipitation, we also divide the anomalies by the Historical ensemble mean. We then calculate the mean, variance, and the PDF of the Historical anomalies (mean = 0 by definition) and of the RCP8.5 anomalies. Then, over all models, we calculate the ensemble averages of the means, variances, and the PDFs so that each model is equally weighted. Support for this approach is given in Brown et al. (2013), who also calculated the PDFs for each model and averaged them together to create the multi-model mean PDF. We then use the resulting ensemble mean and variance to perform the t-test to assess if the mean of RCP8.5 is statistically different from the mean of the Historical simulations (see Sect. 5). Under the assumption that the anomalies are Gaussian distributed, using Simpson’s Rule we analytically calculate the tercile categorical probabilities in the RCP8.5 simulations relative to those defined from the Historical simulations. These probabilistic changes in RCP8.5 are assessed using cutoff thresholds for three equally probable categories determined from the Historical anomalies (Leith 1973). Using the Historical anomalies, the below-normal category is defined as −  6 standard deviations to −  0.43073 standard deviations, the normal category spans ± 0.43073 standard deviations, and the above-normal category is defined for 0.43073 standard deviations to 6 standard deviations. The pooled variance is used for calculating the categorical changes. Irrespective of the actual shapes of the multi-model PDFs seen in the accompanying histograms, all statistics are calculated using the Gaussian assumption. Hereafter, we refer to this procedure as the “Gaussian approach”. In the Gaussian approach, for each realization we calculate the effective degrees of freedom by taking serial correlation into account, since serial correlation inflates the standard deviation when the lag 1 autocorrelation is greater than zero (Wilks 2006). The total degrees of freedom equals the sum of the effective degrees of freedom using all realizations.

1.4 A.4 Non-parametric analysis using the Monte Carlo sampling approach

While the calculation of the afore-mentioned Gaussian approach is computationally very fast, it is also instructive to evaluate the climate change signals using a non-parametric approach, since Sardeshmukh et al. (2015) reported that seasonal mean GPCP precipitation exhibits strong skewness, especially in regions of descent. Though Sawilowsky and Blair (1992) demonstrate that the two-tailed t-test yields conservative results for large sample sizes (~ 60 or more samples) when performed at the 1% significance level, even for PDFs that are highly skewed, non-Gaussianity of the PDFs will impact the tercile probabilities for the below-normal, normal, and above-normal categories. Thus, we also report results using a Monte Carlo sampling using non-parametric statistics. In the cases in which multiple realizations are available for a given model (Table 1) we randomly select one Historical realization and one RCP8.5 realization. The random selection is performed separately for the Historical and for the RCP8.5 realizations, and at each gridpoint for the spatial patterns.

Based on the anomalies from the selected models (33 models, 18 models, or the annual cycle fidelity subset), for each Monte Carlo sample the two-sided Mann–Whitney U-test is performed (Mann and Whitney 1947), with the climate change signal being the median of the RCP anomalies minus the median of the Historical anomalies and with the associated P-value being retained as a measure of the statistical significance of the difference. The Mann–Whitney U-test is a “location” test, a non-parametric alternative to the t-test, with the null hypothesis being that the two batches of data (Historical and RCP8.5) are drawn from the same PDF. It operates by comparing the joint ranking of anomalies of the Historical versus RCP8.5 simulations. When anomalies in both sets of data are equal, the average rank is used to calculate a corrected standard deviation (https://en.wikipedia.org/wiki/Mann–Whitney_U_test). Working in rank-space guards against significant ensemble average changes that might arise due to anomalies that are outliers (Wilks 2006). The benefit of this test is that it is known to have higher power compared to the t-test in the presence of skewed distributions. It has been traditionally used to detect shifts in the distribution of the variables under different conditions (e.g., Basistha et al. 2009; Ersek et al. 2012; Jacob et al. 2014; Taxak et al. 2014). Although a very recent paper has raised warnings about the use of the Mann–Whitney procedure as a test of medians (Divine et al. 2018), the test still remains a good non-parametric alternative to the t-test, given that a pre-check is performed on the equality of the shape (e.g., variance) of the distributions to be compared. Before applying the Mann–Whitney U-test herein, we performed an F-test on the global gridded. For example, for the fractional change in precipitation we find that over most of the globe we cannot reject the null hypothesis that the Historical and RCP variances are equal at the 1% significance level for ≥ 70% of the models (not shown). The main exceptions are the equatorial central/eastern Pacific, poleward of about 80°N, and the interior of Antarctica over the eastern hemisphere where the null hypothesis cannot be rejected for ≤ 40% of the models. As part of the Monte Carlo testing we also perform a t-test in which we calculate the effective degrees of freedom by taking serial correlation into account. The t-test results can be compared to the non-parametric Mann–Whitney U-test results, and to the Gaussian approach obtained using the Section A.3 methodology. We also empirically calculate the tercile probabilities of the RCP8.5 anomalies relative to the Historical thresholds.

Using all Monte Carlo samples we separately average the results from the Mann–Whitney U-test, the t-test, and the empirically calculated RCP8.5 tercile probabilities. In the case of 33 models, the number of Historical (RCP8.5) realizations results in 4.53 × 10⁹ (2.04 × 10⁶) independent combinations. However, the Monte Carlo sampling is only repeated 2000 times, which balances CPU time (~ 36 h for 1 variable for the global gridpoint data on a top-of-the-line 2018 MacBook Pro using 1 processor) vs. consistency of the results to the number of Monte Carlo samples. In the sensitivity test in which a subset of models is retained based on annual cycle fidelity, the limited number of models at some gridpoints limits the number of possible Monte Carlo samples to less than 2000. Sensitivity of the results to the number of Monte Carlo samples is explored in Sect. 5.1. Additionally, the Monte Carlo analysis has been repeated using a different “seed” to initialize the random number generator, thus allowing us to assess the consistency of our results to the selection of independent combinations of realizations.

Note: While a confidence interval for the change in the median can be estimated non-parametrically, the present desktop computing environment precludes this calculation for the global gridpoint analysis. We do present non-parametric confidence intervals for select area-averaged time series, but this is limited to at most 2000 Monte Carlo samples using at most 14 models (39 [40] yearly anomalies for each Historical [RCP8.5] realization from each model) from the annual cycle fidelity subset. In this instance, for one area-averaged dataset, the CPU time is ~ 50 h. For the non-parametric confidence interval calculation using more data becomes even more problematic, since the CPU time scales as the square of the number of possible differences between the RCP and Historical anomalies due to the sorting needed to rank the differences.