Uncertainty component estimates in transient climate projections

Precision of estimators in a single time or time series approach


Quantifying model uncertainty and internal variability components in climate projections has been paid a great attention in the recent years. For multiple synthetic ensembles of climate projections, we compare the precision of uncertainty component estimates obtained respectively with the two Analysis of Variance (ANOVA) approaches mostly used in recent works: the popular Single Time approach (STANOVA), based on the data available for the considered projection lead time and a time series based approach (QEANOVA), which assumes quasi-ergodicity of climate outputs over the available simulation period. We show that the precision of all uncertainty estimates is higher when more members are used, when internal variability is smaller and/or the response-to-uncertainty ratio is higher. QEANOVA estimates are much more precise than STANOVA ones: QEANOVA simulated confidence intervals are roughly 3–5 times smaller than STANOVA ones. Except for STANOVA when less than three members is available, the precision is rather high for total uncertainty and moderate for internal variability estimates. For model uncertainty or response-to-uncertainty ratio estimates, the precision is low for QEANOVA to very low for STANOVA. In the most unfavorable configurations (small number of members, large internal variability), large over- or underestimation of uncertainty components is thus very likely. In a number of cases, the uncertainty analysis should thus be preferentially carried out with a time series approach or with a local-time series approach, applied to all predictions available in the temporal neighborhood of the target prediction lead time.

This is a preview of subscription content, log in to check access.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6


  1. Bosshard T, Carambia M, Goergen K, Kotlarski S, Krahe P, Zappa M, Schaer C (2013) Quantifying uncertainty sources in an ensemble of hydrological climate-impact projections. Water Resour Res 49(3):1523–1536

    Article  Google Scholar 

  2. Bracegirdle TJ, Turner J, Hosking JS, Phillips T (2014) Sources of uncertainty in projections of twenty-first century westerly wind changes over the Amundsen Sea, West Antarctica, in CMIP5 climate models. Clim Dyn 43(7–8):2093–2104

    Article  Google Scholar 

  3. Braun M, Caya D, Frigon A, Slivitzky M (2012) Internal variability of the Canadian RCMs hydrological variables at the basin scale in Quebec and Labrador. J Hydrometeorol 13(2):443–462

    Article  Google Scholar 

  4. Charlton-Perez AJ, Hawkins E, Eyring V, Cionni I, Bodeker GE, Kinnison DE, Akiyoshi H, Frith SM, Garcia R, Gettelman A, Lamarque JF, Nakamura T, Pawson S, Yamashita Y, Bekki S, Braesicke P, Chipperfield MP, Dhomse S, Marchand M, Mancini E, Morgenstern O, Pitari G, Plummer D, Pyle JA, Rozanov E, Scinocca J, Shibata K, Shepherd TG, Tian W, Waugh DW (2010) The potential to narrow uncertainty in projections of stratospheric ozone over the 21st century. Atmos Chem Phys 10(19):9473–9486

    Article  Google Scholar 

  5. Cleveland SW (1979) Robust locally weighted regression and smoothing scatterplots. J Am Stat Assoc 74(368):829–836

    Article  Google Scholar 

  6. Deser C, Phillips A, Bourdette V, Teng H (2012) Uncertainty in climate change projections: the role of internal variability. Clim Dyn 38(3–4):527–546

    Article  Google Scholar 

  7. Evin G, Eckert N, Hingray B, Verfaillie D, Morin S, Lafaysse M, Blanchet J (2018) Traiter l’incertitude des projections climatiques: l’exemple des conditions meteorologiques et d’enneigement dans les Alpes francaises. Schweizerische Zeitschrift fur Forstwesen 169(4):203–209

    Article  Google Scholar 

  8. Gelman A (2005) Analysis of variance—Why it is more important than ever. Ann Stat 33(1):1–53

    Article  Google Scholar 

  9. Giuntoli J, Vidal JP, Prudhomme C, Hannah DM (2015) Future hydrological extremes: the uncertainty from multiple global climate and global hydrological models. Earth Syst Dyn 16:267–285

    Article  Google Scholar 

  10. Hartley HO, Rao JNK (1967) Maximum-likelihood estimation for the mixed analysis of variance model. Biometrika 54(1/2):93–108

    Article  Google Scholar 

  11. Hawkins E, Sutton R (2009) The potential to narrow uncertainty in regional climate predictions. Bull Am Meteorol Soc 90(8):1095–1107

    Article  Google Scholar 

  12. Hawkins E, Sutton R (2011) The potential to narrow uncertainty in projections of regional precipitation change. Clim Dyn 37(1–2):407–418

    Article  Google Scholar 

  13. Hingray B, Blanchet J (2018) Uncertainty components estimates in transient climate projections. bias of moment-based estimators in the single time and time series approaches. Tech. rep., IGE, Univ-Grenoble Alpes, Grenoble, FR. https://hal.archives-ouvertes.fr/hal-01738218

  14. Hingray B, Saïd M (2014) Partitioning internal variability and model uncertainty components in a multimodel multimember ensemble of climate projections. J Clim 27(17):6779–6798

    Article  Google Scholar 

  15. Hingray B, Mezghani A, Buishand TA (2007) Development of probability distributions for regional climate change from uncertain global mean warming and an uncertain scaling relationship. Hydrol Earth Syst Sci 11(3):1097–1114

    Article  Google Scholar 

  16. Jacob D, Petersen J, Eggert B, Alias A, Christensen OB, Bouwer LM, Braun A, Colette A, Deque M, Georgievski G, Georgopoulou E, Gobiet A, Menut L, Nikulin G, Haensler A, Hempelmann N, Jones C, Keuler K, Kovats S, Kroener N, Kotlarski S, Kriegsmann A, Martin E, van Meijgaard E, Moseley C, Pfeifer S, Preuschmann S, Radermacher C, Radtke K, Rechid D, Rounsevell M, Samuelsson P, Somot S, Soussana JF, Teichmann C, Valentini R, Vautard R, Weber B, Yiou P (2014) EURO-CORDEX: new high-resolution climate change projections for European impact research. Reg Environ Change 14(2):563–578

    Article  Google Scholar 

  17. Johns T, Royer JF, Höschel I, Huebener H, Roeckner E, Manzini E, May W, Dufresne JL, Otteå O, Vuuren D, Salas y Melia D, Giorgetta M, Denvil S, Yang S, Fogli P, Körper J, Tjiputra J, Stehfest E, Hewitt C (2011) Climate change under aggressive mitigation: the ensembles multi-model experiment. Clim Dyn 37(9–10):1975–2003

    Article  Google Scholar 

  18. Kendon EJ, Rowell DP, Jones RG, Buonomo E (2008) Robustness of future changes in local precipitation extremes. J Clim 21(17):4280–4297

    Article  Google Scholar 

  19. Kew SF, Selten FM, Lenderink G, Hazeleger W (2011) Robust assessment of future changes in extreme precipitation over the Rhine basin using a GCM. Hydrol Earth Syst Sci 15(4):1157–1166

    Article  Google Scholar 

  20. Lafaysse M, Hingray B, Mezghani A, Gailhard J, Terray L (2014) Internal variability and model uncertainty components in a multireplicate multimodel ensemble of hydrometeorological projections : the Alpine Durance basin. Water Resour Res 50:3317–3341

    Article  Google Scholar 

  21. Montgomery DC (2012) Design and analysis of experiments, 8th edn. Wiley, Hoboken

    Google Scholar 

  22. Northrop PJ, Chandler RE (2014) Quantifying sources of uncertainty in projections of future climate. J Clim 27(23):8793–8808

    Article  Google Scholar 

  23. Olonscheck D, Notz D (2017) Consistently estimating internal climate variability from climate model simulations. J Clim 30(23):9555–9573. https://doi.org/10.1175/JCLI-D-16-0428.1

    Article  Google Scholar 

  24. Paeth H, Vogt G, Paxian A, Hertig E, Seubert S, Jacobeit J (2017) Quantifying the evidence of climate change in the light of uncertainty exemplified by the Mediterranean hot spot region. Global Planet Change 151(SI):144–151. https://doi.org/10.1016/j.gloplacha.2016.03.003

    Article  Google Scholar 

  25. van Pelt SC, Beersma JJ, Buishand TA, van den Hurk BJJM, Schellekens J (2015) Uncertainty in the future change of extreme precipitation over the Rhine basin: the role of internal climate variability. Clim Dyn 44(7–8):1789–1800

    Article  Google Scholar 

  26. Räisänen J (2001) CO2-induced climate change in CMIP2 experiments: quantification of agreement and role of internal variability. J Clim 14(9):2088–2104

    Article  Google Scholar 

  27. Reintges A, Martin T, Latif M, Keenlyside NS (2017) Uncertainty in twenty-first century projections of the Atlantic Meridional Overturning Circulation in CMIP3 and CMIP5 models. Clim Dyn 49(5–6):1495–1511. https://doi.org/10.1007/s00382-016-3180-x

    Article  Google Scholar 

  28. Vidal JP, Hingray B, Magand C, Sauquet E, Ducharne A (2016) Hierarchy of climate and hydrological uncertainties in transient low-flow projections. Hydrol Earth Syst Sci 20(9):3651–3672. https://doi.org/10.5194/hess-20-3651-2016

    Article  Google Scholar 

  29. Yip S, Ferro CAT, Stephenson DB, Hawkins E (2011) A simple, coherent framework for partitioning uncertainty in climate predictions. J Clim 24(17):4634–4643

    Article  Google Scholar 

Download references


We thank the three anonymous reviewers for their constructive suggestions which helped to significantly improve the content of our manuscript.

Author information




BH designed the analysis, developed the local-QEANOVA and the synthetic simulations. JB derived the theoretical expressions for unbiased estimators of uncertainty components and wrote the appendixes. All authors contributed to write the manuscript and discuss results.

Corresponding author

Correspondence to Benoit Hingray.

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary material 1 (PDF 1038 KB)


Appendix 1: QEANOVA Estimates of uncertainty components

We summarize the theoretical developments of the QEANOVA approach to achieve unbiased estimators of uncertainty components for a simplified configuration where the trend model is a simple linear function of time. The full developments are given in Hingray and Blanchet (2018) for the general configuration where the trend model is a linear combination of L functions of time. Here, we first consider the case where the number of members differs from one chain to the other; the simplified equations obtained when all chains have the same number are then given. For the sake of conciseness, we omit the subscript “\(QE\)” related to the QEANOVA approach.


We first consider the raw projections Y(gmt) with \(M_g\) members for each of the G chains, assuming that, for all \(t_s \le t \le t_f\):

$$\begin{aligned} Y(g,m,t)=\lambda (g,t) + \nu (g,m,t), \end{aligned}$$

where \(\lambda (g,t)\) is the trend model expressed as a linear function of time: \(\lambda (g,t)=\varLambda _{g1}+\varLambda _{g2}(t-t_s)\) and where the \(\nu (g,m,t)\) are independent and homoscedastic random variables (with variance \(\sigma _{\nu _g}^2\)). Second let consider the change variable at future prediction lead time \(t \in [t_c,t_f]\):

$$\begin{aligned} X(g,m,t)=Y(g,m,t)-Y(g,m,t_c)=\alpha (g,t)+\eta (g,m,t), \end{aligned}$$

where \(t_c\) is the reference period. We have thus \(\alpha (g,t)=\varLambda _{g2}(t-t_c)\) and \(\eta (g,m,t)=\nu (g,m,t)-\nu (g,m,t_c)\).

Unbiased estimation of the model parameters for the raw variable Y.

We discretize \([t_s,t_f]\) into T time steps (from \(t_s=t_1\) to \(t_f=t_T\)) and write \(t_c\) as the Kth time step (i.e. \(t_c=t_K\)). We are interested in the future prediction lead time \(t_k \in [t_K,t_T]\). Let consider the regression model (10) for a particular g. Unbiased estimators of the regression parameters \((\varLambda _{g1},\varLambda _{g2})\) are given by the least square estimates

$$\begin{aligned} ({\hat{\varLambda }}_{g1}, {\hat{\varLambda }}_{g2} )' ={\mathbb {V}}\; {\mathbb {R}}' \; \left( \frac{1}{M_g} \sum _{m=1}^{M_g} Y(g,m,t_1),\ldots ,\frac{1}{M_g} \sum _{m=1}^{M_g} Y(g,m,t_T)\right) ' \end{aligned}$$

where “”’ denotes the transpose , \({\mathbb {R}}\) is the \(T \times 2\) matrix of covariates whose kth row is \((1,t_k-t_1)\), for \(1 \le k \le T\), and \(\mathbb V= ({\mathbb {R}}' {\mathbb {R}})^{-1}\). Covariance matrix of the estimators \(({\hat{\varLambda }}_{g1}, {\hat{\varLambda }}_{g2})\) is given by \(\widehat{\sigma _{\nu _g}^2} M_g^{-1} {\mathbb {V}}\) where \(\widehat{\sigma _{\nu _g}^2}\) is an unbiased estimator of \(\sigma _{\nu _g}^2\) given by

$$\begin{aligned} \widehat{\sigma _{\nu _g}^2} = \frac{1}{TM_g-L} \sum _{m=1}^{M_g} \sum _{k=1}^T \left\{ Y(g,m,t_k)-{\hat{\lambda }}_{QE}(g,t_k)\right\} ^2 \end{aligned}$$

where \(L=2\) and \({\hat{\lambda }}_{QE}(g,t_k)={\hat{\varLambda }}_{g1}-{\hat{\varLambda }}_{g2}(t_k-t_1)\).

In particular, an unbiased estimator of \(\varLambda _{g2}^2\) is

$$\begin{aligned} \widehat{\varLambda _{g2}^2}={\hat{\varLambda }}_{g2}^2-\widehat{\sigma _{\nu _g}^2} M_g^{-1} V_{22}, \end{aligned}$$

where \(V_{22}\) is the element (2, 2) of \({\mathbb {V}}\). Considering \(t_1,\ldots ,t_T\) regularly spaced on \([t_s; t_f]\), we have \(V_{22}=12(T-1)/\{T(T+1)(t_T-t_1)^2\}\).

Unbiased estimation of the sample variance of the \(\alpha\)s in the change variable.

Given (11) and (13), an unbiased estimator of the sample variance of \(\alpha (g,t_k)\), i.e. of \(s_\alpha ^2(t_k)=\frac{1}{G-1}\sum _{g=1}^G \{\alpha (g,t_k)\}^2\) , is

$$\begin{aligned} \widehat{s_\alpha ^2}(t_k)=s_{{\hat{\alpha }}}^2(t_k)-\frac{12}{T}\frac{T-1}{T+1}\left( \frac{t_k-t_K}{t_T-t_1}\right) ^2 \left( \frac{1}{G} \sum _{g=1}^G \frac{\widehat{\sigma _{\nu _g}^2}}{M_g} \right) , \end{aligned}$$


$$\begin{aligned} s_{{\hat{\alpha }}}^2(t_k)=\frac{(t_k-t_1)^2}{G-1}\sum _{g=1}^G {\hat{\varLambda }}_{g2}^2. \end{aligned}$$

When all GCMs have the same number of runs (M), this expression reduces to:

$$\begin{aligned} {\widehat{s}}_{\alpha }^2(t_k) = s_{{\hat{\alpha }}}^2(t_k) -\frac{A(t_k,{\mathcal {C}})}{M}{\widehat{\sigma }}_{\eta }^2, \end{aligned}$$

where \({\widehat{\sigma }}_{\eta }^2\) is an unbiased estimator of internal variability variance for X

$$\begin{aligned} {\widehat{\sigma }}_{\eta }^2= \frac{2}{G} \sum _{g=1}^G {\widehat{\sigma _{\nu _g}^2}} \end{aligned}$$

and where

$$\begin{aligned} A(t_k,{\mathcal {C}})= \frac{6 (T-1)}{T(T+1)} \left( \frac{t_k-t_K}{t_T-t_1}\right) ^2 \end{aligned}$$

Appendix 2: Estimates with a local QEANOVA approach

We here summarize the expressions of the different uncertainty estimators obtained with a local-QEANOVA approach. The full developments, similar to those presented in appendix A for the QEANOVA approach, are detailed in Hingray and Blanchet (2018).

Model: a regression model is still considered to estimate the response function \(\lambda (g, t)\) for Y in Eq. 10 but \(\lambda (g, t)\) is assumed to be only locally linear in time, in the neighborhoods of \(t_c\) and \(t_e\) respectively, i.e. on \([t_c-\omega ,t_c+\omega ]\) and \([t_e-\omega ,t_e+\omega ]\), where \(t_e \in [t_c,t_f]\) is the future prediction lead time under consideration. \(\lambda (g, t)\) can thus be expressed as

$$\begin{aligned} \lambda (g,t)=\left\{ \begin{array}{cc} \lambda _{c}(g,t)=\varLambda _{g1,c} + (t-t_c) \varLambda _{g2,c} &{} \text{ for } t_{c}-\omega \le t \le t_{c}+\omega ,\\ \lambda _{e}(g,t)=\varLambda _{g1,e} + (t-t_e) \varLambda _{g2,e} &{} \text{ for } t_{e}-\omega \le t \le t_{e}+\omega . \end{array} \right. \end{aligned}$$

The change variable X(gmt), for \(t_e-\omega \le t \le t_e+\omega\), in Eq. 11 is such that \(\alpha (g,t)=(\varLambda _{g1,e}-\varLambda _{g1,c})+(t-t_e)\varLambda _{g2,e}\).

Each interval \([t_c-\omega ,t_c+\omega ]\) and \([t_e-\omega ,t_e+\omega ]\) is discretized into \(T^{\star }\) regular periods of length \(dt=2 \omega /(T^*-1)\), with \(T^{\star }\) odd, giving respectively the sequences \(t_1,\ldots ,t_{T^*}\) and \(t_{T^*+1},\ldots ,t_{2T^*}\). The values of Y for these different times are further considered to estimate the regression coefficients of the linear trend models in Eq. 17. For the illustration given in Sect. 5.3, \(dt = \omega =20yrs\) and \(T^*=3\).

Unbiased estimators of model uncertainty and internal variability variance.

Following Hingray and Blanchet (2018), an unbiased estimator of model uncertainty variance, i.e. of the sample variance of \(\alpha (g,t_e)\), is

$$\begin{aligned} \widehat{s_\alpha ^2}(t_e) = s_{{\hat{\alpha }}}^2(t_e) - \frac{4}{T} \left( \frac{1}{G}\sum _{g=1}^G \frac{ \widehat{\sigma _{\nu _g}^2}}{ M_g}\right) \end{aligned}$$

where \(T=2T^*\) is the total number of time steps considered in the analysis and where \(\widehat{\sigma _{\nu _g}^2}\) is an unbiased estimator of \(\sigma _{\nu _g}^2\) given by

$$\begin{aligned} \widehat{\sigma _{\nu _g}^2} & =\frac{1}{TM_g-4} \sum _{m=1}^{M_g} \left[ \sum _{j=1}^{T^*} \left\{ Y(g,m,t_j)-{\hat{\lambda }}_{c}(g,t_j)\right\} ^2 \right. \end{aligned}$$
$$\begin{aligned}&\quad + \sum _{j=T^*+1}^{2T^*} \left. \left\{ Y(g,m,t_j)-{\hat{\lambda }}_{e}(g,t_j)\right\} ^2\right] \end{aligned}$$

with \({\hat{\lambda }}_{c}(g,t_j)={\hat{\varLambda }}_{g1,c}+{\hat{\varLambda }}_{g2,c}(t_j-t_c)\) and \({\hat{\lambda }}_{e}(g,t_j)={\hat{\varLambda }}_{g1,e}+{\hat{\varLambda }}_{g2,e}(t_j-t_e)\) where \({\hat{\varLambda }}_{g1,c}, {\hat{\varLambda }}_{g2,c}, {\hat{\varLambda }}_{g1,e}\) and \({\hat{\varLambda }}_{g2,e}\) are the regression coefficients of the two linear models in Eq. 17.

When all GCMs have the same number of runs (M), the expression of model uncertainty in Eq. 18 reduces to

$$\begin{aligned} \widehat{s_\alpha ^2}(t_e) = s_{{\widehat{\alpha }}}^2(t_e) - \frac{1}{MT^*} {\hat{\sigma }}_{\eta }^2 \end{aligned}$$

where \({\widehat{\sigma }}_{\eta }^2\) is an unbiased estimator of internal variability variance for X

$$\begin{aligned} {\widehat{\sigma }}_{\eta }^2= \frac{2}{G} \sum _{g=1}^G \widehat{\sigma _{\nu _g}^2}. \end{aligned}$$

Appendix 3: Simulation of MMEs

Each MME is simulated for Y(gmt) assuming that, for all \(t_s \le t \le t_f\):

$$\begin{aligned} Y(g,m,t)=\lambda (g,t) + \nu (g,m,t), \end{aligned}$$

where the climate response function \(\lambda (g,t)\) is a linear function of time and where the \(\nu (g,m,t)\) are independent and homoscedastic random variables (with variance \(\sigma _{\nu _g}^2\)). For convenience we further assume that \(\lambda (g,t)\) can be decomposed as \(\lambda (g,t)=w(t)+d(g,t)\), for \(t=t_1,\ldots ,T\) where the mean climate response w(t) of the G chains and the deviations d(gt) of chain g are linear functions of time, expressed as: \(w(t)=B+P.(t-t_1)/(t_e-t_C)\) and \(d(g,t)=D(g).(t-t_1)/(t_e-t_C)\) with the constraint \(\sum _{g=1}^{ G }D(g)=0\).

For graphical simplification purposes, each MME for Y is constructed so that the parameters \((\sigma ^2_\nu , P, D(g), g=1,\ldots ,G )\) lead, for the change variable X at time \(t=t_e\), to a prescribed value of the Response-to-Uncertainty ratio [R2U(\(t_e\))] and to a prescribed value of the fractional variance \(F_\eta (t_e)\) due to internal variability.

For X, we have by definition \(\varphi (g,t)=\lambda (g,t)-\lambda (g,t_C)\) and \(\eta (g,m,t) = \nu (g,m,t)-\nu (g,m,t_C)\). We have thus for the change variable \(\mu (t)\) = \(w(t)-w(t_C)\), \(\alpha (g,t) = d(g,t)-d(g,t_C)\) and in turn \(\mu (t) = P.(t-t_C)/(t_e-t_C)\) and \(\alpha (g,t) = D(g).(t-t_C)/(t_e-t_C)\).

The theoretical values for \(\mu (t_e)\) and \(s^2_\alpha (t_e)\) are thus as follows : \(\mu (t_e) = P\) and \(s^2_\alpha (t_e)={\mathbb V\text {ar}}(D(g))\). Fixing P to 1, we thus simply require in turn

$$\begin{aligned}&\sigma ^2_X(t_e) = \frac{1}{[R2U(t_e)]^2}; \end{aligned}$$
$$\begin{aligned}&\sigma ^2_\nu (t_e) = \frac{1}{2} \sigma ^2_\eta (t_e) = \frac{1}{2} F_\eta (t_e).\sigma ^2_X(t_e); \end{aligned}$$
$$\begin{aligned}&{\mathbb V\text {ar}}(D(g))=s^2_\alpha (t_e)=(1-F_\eta (t_e)).\sigma ^2_X(t_e). \end{aligned}$$

For each MME simulation, the deviations of the different chains, \(D(g), g=1,\ldots ,G\), are obtained from a sample of G realizations in a normal distribution. These realizations are scaled so that their mean is zero and their variance corresponds to the prescribed value \(s^2_\alpha (t_e)\).

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Hingray, B., Blanchet, J., Evin, G. et al. Uncertainty component estimates in transient climate projections. Clim Dyn 53, 2501–2516 (2019). https://doi.org/10.1007/s00382-019-04635-1

Download citation


  • Uncertainty sources
  • Climate projections
  • Internal variability
  • Model uncertainty
  • Scenario uncertainty
  • Precision of estimates