Decadal predictability and forecast skill

Abstract

The “potential predictability” of the climate system is the upper limit of available forecast skill and can be characterized by the ratio p of the predictable variance to the total variance. While the potential predictability of the actual climate system is unknown its analog q may be obtained for a model of the climate system. The usual correlation skill score r and the mean square skill score M are functions of p in the case of actual forecasts and potential correlation ρ and potential mean square skill score $\mathcal{M}$ are the same functions of q in the idealized model context. In the large ensemble limit the connection between model-based potential predictability and skill scores is particularly straightforward with $q=\rho^{2}=\mathcal{M}.$ Decadal predictions of annual mean temperature produced with the Canadian Centre for Climate Modelling and Analysis coupled climate model are analyzed for information on decadal climate predictability and actual forecast skill. Initialized forecast results are compared with the results of uninitialized climate simulations. Model-based values of potential predictability q and potential correlation skill ρ are obtained and ρ is compared with the actual forecast correlation skill r. The skill of externally forced and internally generated components of the variability are separately estimated. As expected, ρ > r and both decline with forecast range τ, at least for the first five years. The decline of skill is associated mainly with the decline of the skill of the internally generated component. The potential and actual skill of a forecast of time-averaged temperature depends on the averaging period. The skill of uninitialized simulations is low for short averaging times and increases as averaging time increases. By contrast, skill is high at short averaging times for forecasts initialized from observations and declines as averaging times increase to about three years, then increases somewhat at longer averaging times. The skills of the initialized forecasts and uninitialized simulations begin to converge for longer averaging times. The potential correlation skill ρ of the externally forced component of temperature is largest at tropical latitudes and the skill of the internally generated component is largest over the North Atlantic, parts of the Southern Ocean and to some extent the North Pacific. Potential skill over extratropical land is somewhat weaker than over oceans. The distribution of actual correlation skill r is broadly similar to that of potential skill for the externally forced component but less so for the internally generated component. Differences in potential and actual skill suggest where improvements in the forecast system might be found.

Appendix

The statistical assumptions made and the estimates of the population parameters that enter the expressions are reviewed here. Estimates apply at forecast range τ so this subscript is omitted. Retaining subscripts f and i, which identify the forced and internally generated components, the observations, forecasts and simulations are represented as

$$ \begin{aligned} X_{j}&=\mu+(\chi_{f}+\chi_{i}+x)_{j} \\ Y_{\alpha j}&=\nu+(\psi_{f}+\psi_{i}+y_{\alpha})_{j} \\ U_{\alpha j}&=\eta+(\varphi_{f}+\varphi_{i,\alpha}+u_{\alpha})_{j} \end{aligned} $$

where, as before, α labels the ensemble member and j labels the start date (and applies to all terms within the brackets).

1.1 Statistical assumptions

The population parameters for the observations and forecasts are assumed to consist of a mean, a deterministic externally forced component, and the remaining potentially predictable and noise components in the form Y = ν + ψ _f  + ψ _i  + y. The assumed statistical relationships are indicated by the expectations of the various first and second order terms that arise. For the forecasts these are

$$ \begin{aligned} E(\nu,\psi_{f},\psi_{i},y_{\alpha})_{j}&=(\nu,\psi_{f},0,0)_{j} \\ E\nu(\psi_{f},\psi_{i},y_{\alpha})_{j}&=(\nu\psi_{f},0,0)_{j} \\ E\psi_{f,j}(\psi_{i},y_{\alpha})_{k}&=(0,0) \\ E\psi_{i,j}(\psi_{i},y_{\alpha})_{k}&=(\sigma_{\psi_{i}}^{2}\varrho(j-k),0) \\ Ey_{\alpha j}y_{\beta k}&=\delta_{k}^{j}\delta_{\beta}^{\alpha}\sigma_{y}^{2} \end{aligned} $$

(10)

for ϱ(j − k) the lagged autocorrelation of ψ _i . Results are also similar for the simulations U where, however, $E\varphi_{i,\alpha j}\varphi_{i,\beta k}=\delta_{\beta}^{\alpha}\sigma_{\varphi_{i}}^{2}{\varrho_{\varphi_{i}}}(j-k)$ since the internally generated component is independent from one ensemble member to another in that case. The relationships for products of observed with forecast and simulated terms are indicated by

$$ \begin{aligned} E(\chi_{f}+\chi_{i}+x)_{j}(\psi_{f}+\psi_{i}+y_{\alpha})_{j}&=\chi_{f}\psi_{f}+Cov\chi_{i}\psi_{i} \\ E(\chi_{f}+\chi_{i}+x)_{j}(\varphi_{f}+\varphi_{i,\alpha}+u_{\alpha})_{j}&=\chi_{f}\varphi_{f} \\ \end{aligned} $$

1.2 Sample statistics

Estimates of the population variances and covariances are obtained from sample variances and covariances. The simplified notation in Sect. 4 is used, the mean over all start dates is indicated by an overbar, and the mean over all ensemble members is represented by braces as

$$ \begin{aligned} (\overline{X},\overline{Y},\overline{U})&= \frac{1}{n}\sum_{j}(X_{j},Y_{\alpha j},U_{\alpha j}) \\ (\overline{Y}_{a},\overline{U}_{a})&=(\{\overline{Y}\},\{\overline{U}\})= \frac{1}{m}\sum_{\alpha}(\overline{Y},\overline{U})= \frac{1}{nm}\sum_{\alpha j}(Y_{\alpha j},U_{\alpha j}) \\ \end{aligned} $$

Second order sample statistics are obtained as

$$ \begin{aligned} \overline{(X-\overline{X})^{2}}&=\overline{X^{2}}-\overline{X}^{2}=S_{X}^{2} \\ \{\overline{(Y-\overline{Y}_{a})^{2}}\}&=\{\overline{(Y-Y_{a})^{2}}\}+\overline{(Y_{a}-\overline{Y}_{a})^{2}}=S_{y}^{2}+S_{\psi}^{2} \\ \{\overline{(U-\overline{U}_{a})^{2}}\}&=\{\overline{(U-U_{a})^{2}}\}+\overline{(U_{a}-\overline{U}_{a})^{2}}=S_{u*}^{2}+S_{\varphi_{f}}^{2} \\ \overline{(X-\overline{X})(Y_{a}-\overline{Y}_{a})}&=\overline{XY_{a}}-\overline{X}\overline{Y}_{a}=S_{\chi\psi}^{2} \\ \overline{(X-\overline{X})(U_{a}-\overline{U}_{a})}&=\overline{XU_{a}}-\overline{X}\overline{U}_{a}=S_{\chi\varphi}^{2} \end{aligned} $$

where

$$ \begin{aligned} S_{y}^{2}&=\{\overline{(Y-Y_{a})^{2}}\}&=\{\overline{Y^{2}}\}-\overline{Y_{a}^{2}} \\ S_{\psi}^{2}&=\overline{(Y_{a}-\overline{Y}_{a})^{2}}&=\overline{Y_{a}^{2}}-\overline{Y}_{a}^{2} \\ S_{u*}^{2}&=\{\overline{(U-U_{a})^{2}}\}&=\{\overline{U^{2}}\}-\overline{U_{a}^{2}} \\ S_{\varphi_{f}}^{2}&=\overline{(U_{a}-\overline{U}_{a})^{2}}&=\overline{U_{a}^{2}}-\overline{U}_{a}^{2} \\ \end{aligned} $$

The sample statistics are used to estimate the population parameters in (3).

The expectations of these statistics follow, for example, as

$$ \begin{aligned} E\{\overline{Y^{2}}\}&=\frac{1}{nm} \sum_{\alpha j}E(\nu+\psi_{f}+\psi_{i}+y_{\alpha})_{j}^{2} \\ &=\nu^{2}+\sigma_{\psi_{f}}^{2}+\sigma_{\psi_{i}}^{2}+\sigma_{y}^{2} =\nu^{2}+\sigma_{\psi}^{2}+\sigma_{y}^{2} \end{aligned} $$

where the deterministic forced component has zero mean when averaged over all start dates, i.e. $\overline{\psi}_{f}=\frac{1}{n}\sum\nolimits_{j}\psi_{f,j}=0,$ and where cross-product terms vanish following (10). The other quantities are evaluated similarly with

$$ \begin{aligned} E\overline{Y_{a}^{2}}&=\frac{1}{n}\sum_{j}\frac{1}{m^{2}} \sum_{\alpha\beta}E(\nu+\psi_{f}+\psi_{i}+y_{\alpha})_{j} (\nu+\psi_{f}+\psi_{i}+y_{\beta})_{j}\\ & =\nu^{2}+\sigma_{\psi}^{2}+\sigma_{y}^{2}/m \end{aligned} $$

and

$$ \begin{aligned} E\overline{Y}_{a}^{2}&= \frac{1}{n^{2}m^{2}}\sum_{\alpha j}\sum_{\beta k}E(\nu+\psi_{f}+\psi_{i}+y_{\alpha})_{j} (\nu+\psi_{f}+\psi_{i}+y_{\beta})_{k}\\ &=\nu^{2}+\sigma_{\psi_{i}}^{2}\gamma/n+\sigma_{y}^{2}/nm \end{aligned} $$

where

$$ \frac{1}{n^{2}} \sum_{jk}E\psi_{i,j}\psi_{i,k}=\frac{1}{n^{2}} \sigma_{\psi_{i}}^{2}\sum_{jk}\varrho(j-k)=\sigma_{\psi_{i}} ^{2}\gamma/n $$

and $\gamma=1+2\sum\nolimits_{l=1}^{n}(1-\frac{l}{n})\varrho(l)=1+\epsilon$ (e.g. Boer 2004).

For the simulations

$$ \begin{aligned} E\{\overline{U^{2}}\}&=\eta^{2}+\sigma_{\varphi_{f}}^{2}+(\sigma_{\varphi_{i}}^{2}+\sigma_{u}^{2}) \\ E\overline{U_{a}^{2}}&=\eta^{2}+\sigma_{\varphi_{f}}^{2}+(\sigma_{\varphi_{i}}^{2}+\sigma_{u}^{2})/m \\ E\overline{U}_{a}^{2}&=\eta^{2}+(\sigma_{\varphi_{i}}^{2}\gamma+\sigma_{u}^{2})/nm \end{aligned} $$

For cross-products

$$ \begin{aligned} E\overline{XY_{a}}&= \frac{1}{n}\sum_{j}E(\mu+\chi_{f}+\chi_{i}+x)_{j} \left(\nu+\psi_{f}+\psi_{i}+ \frac{1}{m}\sum_{\alpha}y_{\alpha}\right)_{j}\\ &=\mu\nu+Cov\chi_{f}\psi_{f}+Cov\chi_{i}\psi_{i} \\ E\overline{X}\overline{Y}_{a}&=\frac{1}{n^{2}} \sum_{jk}E(\mu+\chi_{i}+x)_{j}\left(\nu+\psi_{i}+\frac{1}{m} \sum_{\alpha}y_{\alpha}\right)_{k} \\ &=\mu\nu+Cov\chi_{i}\psi_{i}/n \\ \end{aligned} $$

and

$$ \begin{aligned} E\overline{XU_{a}}&=\mu\eta+Cov\chi_{f}\varphi_{f} \\ E\overline{X}\overline{U}_{a}&=\mu\eta \end{aligned} $$

Expected values of the sample statistics follow as

$$ \begin{aligned} ES_{y}^{2}&=\left(\frac{m-1}{m}\right)\sigma_{y}^{2} \\ ES_{u^{*}}^{2}&=\left(\frac{m-1}{m}\right)\left(\sigma_{\varphi_{i}}^{2}+\sigma_{u}^{2}\right) \\ ES_{\psi}^{2}&=\sigma_{\psi_{f}}^{2}+\left(\frac{n-\gamma}{n}\right)\sigma_{\psi_{i}}^{2}+\left(\frac{n-1}{nm}\right)\sigma_{y}^{2} \\ ES_{\varphi_{f}}^{2}&=\sigma_{\varphi_{f}}^{2}+\left(\frac{n-\gamma}{nm}\right)\sigma_{\varphi_{i}}^{2}+\left(\frac{n-1}{nm}\right)\sigma_{u}^{2} \\ ES_{\chi\psi}^{2}&=Cov\chi_{f}\psi_{f}+\left(\frac{n-1}{n}\right)Cov\chi_{i}\psi_{i} \\ ES_{\chi\varphi}^{2}&=Cov\chi_{f}\varphi_{f}\approx Cov\chi_{f}\psi_{f} \end{aligned} $$

1.3 Statistical estimates of variances and covariances

Unbiased estimates, indicated by carets, of the variances and covariances for the forecasts follow with

$$ \hat{\sigma}_{y}^{2}=\left(\frac{m}{m-1}\right)S_{y}^{2} $$

and the unbiased estimate for σ ²_ψ is obtained as an approximation with

$$ \hat{\sigma}_{\psi}^{2} = \left(\frac{n}{n-1}\right)S_{\psi}^{2}-\left(\frac{1}{m-1}\right)S_{y}^{2} -\left(\frac{1}{n-1}\right)\left(\sigma_{\psi_{f}}^{2}-\epsilon\sigma_{\psi_{i}}^{2}\right) \approx\left(\frac{n}{n-1}\right)S_{\psi}^{2}-\left(\frac{1}{m-1}\right)S_{y}^{2} $$

For $\epsilon$ generally <2 as seen in Boer (2004), the neglected term $\xi=-(\frac{1}{n-1})(\sigma_{\psi_{f}}^{2} -\epsilon\sigma_{\psi_{i}}^{2})$ is a small fraction of the total with

$$ \frac{\xi}{\sigma_{\psi}^{2}}=-\frac{\sigma_{\psi_{f}}^{2}-\epsilon \sigma_{\psi_{i}}^{2}}{(n-1)q\sigma_{Y}^{2}}=\frac{\epsilon q_{i}-q_{f}}{(n-1)q}=0.0256\frac{\epsilon q_{i}-q_{f}}{q} $$

for n = 40 as is the case here. Similarly

$$ \begin{aligned} \hat{\sigma}_{X}^{2}&=\left(\frac{n}{n-1}\right)S_{X}^{2}-\left(\frac{1}{n-1}\right) \left(\sigma_{\chi_{f}}^{2}-\epsilon\sigma_{\chi_{i}}^{2}\right) \approx\left(\frac{n}{n-1}\right)S_{X}^{2} \\ \hat{C}ov\chi\psi&=\left(\frac{n}{n-1}\right)S_{\chi\psi}^{2} -Cov\chi_{f}\psi_{f}/(n-1)\approx\left(\frac{n}{n-1}\right) S_{\chi\psi}^{2} \\ \end{aligned} $$

and for the simulations

$$ \begin{aligned} \hat{\sigma}_{\varphi_{i}}^{2}+\hat{\sigma}_{u}^{2}&= \left(\frac{m}{m-1}\right) S_{u^{*}}^{2} \\ \hat{\sigma}_{\varphi_{f}}^{2}&= \left(\frac{n}{n-1}\right)S_{\varphi_{f}}^{2} -\left(\frac{1}{m-1}\right)S_{u*}^{2}-\left(\frac{1}{n-1}\right)\left(\sigma_{\varphi_{f}}^{2} -\epsilon\sigma_{\varphi_{i}}^{2}/m\right)\approx\left(\frac{n}{n-1}\right) S_{\varphi_{f}}^{2}-\left(\frac{1}{m-1}\right)S_{u*}^{2} \\ \hat{C}ov\chi\varphi&= \hat{C}ov\chi_{f}\varphi_{f}= S_{\chi\varphi}^{2} \end{aligned} $$

As noted in the text, Covχ _i ψ _i is loosely approximated by taking $Cov\chi_{f}\psi_{f}\approx Cov\chi_{f}\varphi_{f}$ whence

$$ \hat{C}ov\chi_{i}\psi_{i}\approx\left(\frac{n}{n-1}\right)S_{\chi\psi}^{2}-S_{\chi\varphi}^{2} $$

and, finally,

$$ \begin{aligned} \hat{\sigma}_{Y}^{2}&= \hat{\sigma}_{\psi}^{2}+\hat{\sigma}_{y}^{2} \\ \hat{\sigma}_{Y_{a}}^{2}&=\hat{\sigma}_{\psi}^{2}+\hat{\sigma}_{y}^{2}/m. \end{aligned} $$