Skip to main content

Ensemble optimisation, multiple constraints and overconfidence: a case study with future Australian precipitation change


Future climate is typically projected using multi-model ensembles, but the ensemble mean is unlikely to be optimal if models’ skill at reproducing historical climate is not considered. Moreover, individual climate models are not independent. Here, we examine the interplay between the benefits of optimising an ensemble for the performance of its mean and the the effect this has on ensemble spread as an uncertainty estimate. Using future Australian precipitation change as a case study, we perform optimal subset selection based on present-day precipitation, sea surface temperature and/or 500 hPa eastward wind climatologies. We use either one, two, or all three variables as predictors. Out-of-sample projection skill is assessed using a model-as-truth approach (rather than observations). For multiple variables, multi-objective optimisation is used to obtain Pareto-optimal subsets (an ensemble of model subsets), to gauge the uncertainty in optimisation arising from the multiple constraints. We find that the spread of climate model subset averages typically under-represents the true projection uncertainty (overconfidence), but that the situation can be significantly improved using mixture distributions for uncertainty estimation. The single best predictor, present-day precipitation, gives the most accurate results but is still overconfident—a consequence of calibrating too specifically. It is only when all three constraints are used that projection skill is improved and overconfidence is eliminated, but at the cost of a poorer best estimate relative to one predictor. We thus identify an important trade-off between accuracy and precision, depending on the number of predictors, which is likely relevant for any subset selection or weighting strategy.

This is a preview of subscription content, access via your institution.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9


Download references


We would like to thank Karsten Lehmann for help with the mathematical programming solver Gurobi. Thanks to Jan Sedláček for providing access to the CMIP5 archive based at ETHZ. Nadja Herger, Steven Sherwood, and Oliver Angélil acknowledge the support of the Australian Research Council Centre of Excellence for Climate System Science (CE110001028). Gab Abramowitz and Scott A. Sisson acknowledge the Australian Research Council Centre of Excellence for Climate Extremes (CE170100023). Scott A. Sisson acknowledges the support of the Australian Research Council Centre of Excellence for Mathematical and Statistical Frontiers (CE140100049). ETH Zurich acknowledges support by the European Union’s Horizon 2020 research and innovation programme under Grant agreement no. 641816 (CRESCENDO) We acknowledge the World Climate Research Programme’s Working Group on Coupled Modelling, which is responsible for CMIP, and we thank the climate modeling groups (listed in Table S1 in the SI) for producing and making available their model output. For CMIP the US Department of Energy’s Program for Climate Model Diagnosis and Intercomparison provides coordinating support and led development of software infrastructure in partnership with the Global Organization for Earth System Science Portals. CMIP data can be obtained from We use Python for data analysis and visualisation.

Author information

Authors and Affiliations


Corresponding author

Correspondence to Nadja Herger.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary material 1 (pdf 3494 KB)

Rights and permissions

Reprints and Permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Herger, N., Abramowitz, G., Sherwood, S. et al. Ensemble optimisation, multiple constraints and overconfidence: a case study with future Australian precipitation change. Clim Dyn 53, 1581–1596 (2019).

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: