Finding plausible and diverse variants of a climate model. Part 1: establishing the relationship between errors at weather and climate time scales

Abstract

The main aim of this two-part study is to use a perturbed parameter ensemble (PPE) to select plausible and diverse variants of a relatively expensive climate model for use in climate projections. In this first part, the extent to which climate biases develop at weather forecast timescales is assessed with two PPEs, which are based on 5-day forecasts and 10-year simulations with a relatively coarse resolution (N96) atmosphere-only model. Both ensembles share common parameter combinations and strong emergent relationships are found for a wide range of variables between the errors on two timescales. These relationships between the PPEs are demonstrated at several spatial scales from global (using mean square errors), to regional (using pattern correlations), and to individual grid boxes where a large fraction of them show positive correlations. The study confirms more robustly than in previous studies that investigating the errors on weather timescales provides an affordable way to identify and filter out model variants that perform poorly at short timescales and are likely to perform poorly at longer timescales too. The use of PPEs also provides additional information for model development, by identifying parameters and processes responsible for model errors at the two different timescales, and systematic errors that cannot be removed by any combination of parameter values.

Appendix

For this study, the global MSE of the average forecast error over 16 start dates is used to measure the performance of the TAMIP experiments. An alternative measure of forecast performance would have been to calculate the MSEs for each start date and then average those. We refer to this alternative metric as the mean-individual-forecast-error MSE. We how below how the latter relates to the “mean-error MSE” used in the paper.

Let ${m_{ij}}$ be the 5-day forecast value for the ith start date at the jth grid point, ${o_{ij}}$ be the verifying observation, and let the overbar indicate an average over the index denoted by the dot. Then the “mean-error MSE” used in the paper is

$$MSE=\frac{1}{N}~\mathop \sum \limits_{{j=1}}^{N} {\left( {\overline {{{m_{ \cdot j}}}} - ~\overline {{{o_{ \cdot j}}}} } \right)^2}.$$

(1)

The alternative mean-individual-forecast-error MSE measure is the left-hand side of Eq. (2), which can be expanded:

$$\frac{1}{R}\mathop \sum \limits_{{i=1}}^{R} \mathop \sum \limits_{{j=1}}^{N} {\left( {{m_{ij}} - ~{o_{ij}}} \right)^2}=\frac{1}{R}\mathop \sum \limits_{{i=1}}^{R} \mathop \sum \limits_{{j=1}}^{N} {\left( {\left( {{m_{ij}} - ~\overline {{{m_{ \cdot j}}}} } \right) - ~\left( {{o_{ij}} - ~\overline {{{o_{ \cdot j}}}} } \right)+\left( {\overline {{{m_{ \cdot j}}}} - ~\overline {{{o_{ \cdot j}}}} } \right)} \right)^2}.$$

(2)

Defining the deviation of the 5-day forecast value of the ith start date and jth grid point from the ensemble mean 5-day forecast as $\Delta {m_{ij}}=~~{m_{ij}} - ~\overline {{{m_{ \cdot j}}}}$ and the deviation of the observation 5 days from the ith start date at the jth grid point from the average of the values across the 16 start dates $\Delta {o_{ij}}=~~{o_{ij}} - \overline {{~{o_{ \cdot j}}}}$, we then get

$$\frac{1}{R}\mathop \sum \limits_{{i=1}}^{R} \mathop \sum \limits_{{j=1}}^{N} {\left( {{m_{ij}} - ~{o_{ij}}} \right)^2}=~MSE+\frac{1}{R}\mathop \sum \limits_{{i=1}}^{R} \mathop \sum \limits_{{j=1}}^{N} {\left( {\Delta {m_{ij}} - ~\Delta {o_{ij}}} \right)^2}+\frac{1}{R}\mathop \sum \limits_{{i=1}}^{R} \mathop \sum \limits_{{j=1}}^{N} \left( {\Delta {m_{ij}} - ~\Delta {o_{ij}}} \right)\left( {\overline {{~{m_{ \cdot j}}}} - ~\overline {{~{o_{ \cdot j}}}} } \right).$$

(3)

This shows that the mean-individual-forecast-error MSE has three components: the mean-error MSE from Eq. (1), and two extra terms that measure forecasting performance dependent on the given initial state of the weather. We refer to the sum of these two extra terms as the initial-condition-dependent MSE. An important component for these two extra terms are the 16 differences between the modelled and observed deviations from their respective mean. The two terms are the average of the MSEs of the 16 differences; and a term based on the correlation of the 16 differences with the averaged difference across the 16 start dates. Across the PPE, there is a high correlation between the mean-error MSE and the mean-individual-forecast-error MSE (see Fig. 14) suggesting that for climate models, the mean forecast error dominates. This supports our choice to focus on the mean-error MSE in the paper. In Fig. 15 we plot differences between the two scores against the mean-error MSE, showing that for many variables (other than surface air temperature and precipitation), these two terms are uncorrelated. This suggests that the dominance of mean-error MSE does obscure potentially interesting contributions from the mean-individual-forecast-error MSE that could provide potentially useful information.

Fig. 14

Scatterplots of the “mean-error MSE” (x-axis) against the “mean-individual-forecast-error MSE” (y-axis)

Fig. 15

Scatterplots of the “mean-error MSE” (x-axis) against the difference between the “mean-individual-forecast-error MSE” and “mean-error MSE” (y-axis)

A sensitivity analysis (Fig. 16) of the initial-condition-dependent MSE shows that parameters in the convection scheme dominate, even for variables like zonally averaged relative humidity on pressure levels and cloud area fraction, which tend to be dominated by the cloud radiation and microphysics parameters when considering the mean-error MSE. This would suggest that using the mean-individual-forecast-error MSE to filter parameter space would mainly add value for constraining the convection parameters.

Fig. 16

Sensitivity analysis showing the fraction of variance of the response surface by the emulators of the “mean-error MSE” (lower triangle) against the difference between the “mean-individual-forecast-error MSE” and “mean-error MSE” (upper triangle) for several variables (y-axis). A key is shown in far-right column