Uncertainty Quantification Using Multiple Models—Prospects and Challenges

  • Reto KnuttiEmail author
  • Christoph Baumberger
  • Gertrude Hirsch Hadorn
Part of the Simulation Foundations, Methods and Applications book series (SFMA)


Model evaluation for long-term climate predictions must be done on quantities other than the actual prediction, and a comprehensive uncertainty quantification is impossible. An ad hoc alternative is provided by coordinated model intercomparisons which typically use a “one model one vote” approach. The problem with such an approach is that it treats all models as independent and equally plausible. Reweighting all models of the ensemble for performance and dependence seems like an obvious way to improve on model democracy, yet there are open questions on what constitutes a “good” model, how to define dependency, how to interpret robustness, and how to incorporate background knowledge. Understanding those issues have the potential to increase confidence in model predictions in modeling efforts outside of climate science where similar challenges exist.


Ensemble modeling Idealization Model independence Robustness Structural model uncertainty Uncertainty quantification 



We thank Claus Beisbart, Keith Beven, Benedikt Knüsel, Wendy Parker, and Marius Wälchli for discussions and comments which have improved the quality of this manuscript. This project was supported by the European Union’s Horizon 2020 research and innovation program under grant agreement No 641816 (CRESCENDO). We acknowledge the World Climate Research Programme’s Working Group on Coupled Modelling, which is responsible for CMIP, and we thank the climate modeling groups for producing and making available their model output. For CMIP, the U.S. Department of Energy’s Program for Climate Model Diagnosis and Intercomparison provides coordinating support and led the development of software infrastructure in partnership with the Global Organization for Earth System Science Portals.


  1. Abramowitz, G., & Gupta, H. (2008). Toward a model space and model independence metric. Geophysical Research Letters, 35(5), 1–4. Scholar
  2. Abramowitz, G., & Bishop, C. H. (2015). Climate model dependence and the ensemble dependence transformation of CMIP projections. Journal of Climate, 28, 2332–2348. Scholar
  3. Annan, J. D., & Hargreaves, J. C. (2010). Reliability of the CMIP3 ensemble. Geophysical Research Letters, 37(2), 1–5. Scholar
  4. Annan, J. D., & Hargreaves, J. C. (2011). Understanding the CMIP3 multimodel ensemble. Journal of Climate, 24(16), 4529–4538. Scholar
  5. Annan, J., & Hargreaves, J. (2016). On the meaning of independence in climate science. Earth System Dynamics Discussions, 1–17.
  6. Baumberger, C., Knutti, R., & Hirsch Hadorn, G. (2017). Building confidence in climate model projections: An analysis of inferences from fit. Wiley Interdisciplinary Reviews: Climate Change, 8(3), e454. Scholar
  7. Bellouin, N., et al. (2011). The HadGEM2 family of Met Office Unified Model climate configurations. Geoscientific Model Development, 4(3), 723–757.
  8. Boé, J., Hall, A., & Qu, X. (2009). September sea-ice cover in the Arctic Ocean projected to vanish by 2100. Nature Geoscience, 2(5), 341–343. (Nature Publishing Group). Scholar
  9. Caldwell, P. M., et al. (2014). Statistical significance of climate sensitivity predictors obtained by data mining. Geophysical Research Letters, 41(5), 1803–1808. Scholar
  10. Claussen, M., et al. (2002). Earth system models of intermediate complexity: Closing the gap in the spectrum of climate system models. Climate Dynamics, 18(7), 579–586. Scholar
  11. Cox, P. M., et al. (2013). Sensitivity of tropical carbon to climate change constrained by carbon dioxide variability. Nature, 494(7437), 341–344. (Nature Publishing Group). Scholar
  12. Deser, C., et al. (2012). Communication of the role of natural variability in future North American climate. Nature Climate Change, 2(11), 775–779. Scholar
  13. Edwards, P. N. (2011). History of climate modeling. Wiley Interdisciplinary Reviews: Climate Change, 2(1), 128–139. Scholar
  14. Elgin, C. Z. (2017). True enough. Project MUSE: The MIT Press.CrossRefGoogle Scholar
  15. Eyring, V., et al. (2016). Overview of the coupled model intercomparison project phase 6 (CMIP6) experimental design and organization. Geoscientific Model Development, 9(5), 1937–1958. Scholar
  16. Fischer, E. M., & Knutti, R. (2016) Observed heavy precipitation increase confirms theory and early models. Nature Climate Change, 6(11), 986–991. (Nature Publishing Group). Scholar
  17. Fitelson, B. (2001). A Bayesian account of independent evidence with applications. Philosophy of Science, 68(S3), S123–S140. Scholar
  18. Flato, G. M. (2011). Earth system models: An overview. Wiley Interdisciplinary Reviews: Climate Change, 2(6), 783–800. Scholar
  19. Gent, P. R., et al. (1995). Parameterizing eddy-induced tracer transports in ocean circulation models. Journal of Physical Oceanography, 25(4), 463–474. (American Meteorological Society).CrossRefGoogle Scholar
  20. Gleckler, P. J., Taylor, K. E., & Doutriaux, C. (2008). Performance metrics for climate models. Journal of Geophysical Research, 113(D6), 1–20. Scholar
  21. Hall, A., & Qu, X. (2006). Using the current seasonal cycle to constrain snow albedo feedback in future climate change. Geophysical Research Letters, 33(3), L03502. Scholar
  22. Harrison, S. P., et al. (2015). Evaluation of CMIP5 palaeo-simulations to improve climate projections. Nature Climate Change, 5(8), 735–743. Scholar
  23. Held, I. M. (2005). The gap between simulation and understanding in climate modeling. Bulletin of the American Meteorological Society, 86(11), 1609. Scholar
  24. Jun, M., Knutti, R., & Nychka, D. W. (2008). Spatial analysis to quantify numerical model bias and dependence. Journal of the American Statistical Association, 103(483), 934–947. Scholar
  25. Kay, J. E., Holland, M. M., & Jahn, A. (2011). Inter-annual to multi-decadal Arctic sea ice extent trends in a warming world. Geophysical Research Letters, 38(15), 2–7. Scholar
  26. Knutti, R. (2008). Should we believe model predictions of future climate change? Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences, 366(1885), 4647–4664. Scholar
  27. Knutti, R. (2010). The end of model democracy? Climatic Change, 102(3–4), 395–404. Scholar
  28. Knutti, R. (2018). Climate model confirmation: From philosophy to predicting climate in the real world. In Climate modelling (pp. 325–359). Cham: Springer International Publishing. Scholar
  29. Knutti, R., & Sedláček, J. (2012). Robustness and uncertainties in the new CMIP5 climate model projections. Nature Climate Change, 3(4), 369–373. (Nature Publishing Group). Scholar
  30. Knutti, R., Furrer, R., et al. (2010a). Challenges in combining projections from multiple climate models. Journal of Climate, 23(10), 2739–2758. Scholar
  31. Knutti, R., Abramowitz, G., et al. (2010b). Good practice guidance paper on assessing and combining multi model climate projections, meeting report of the intergovernmental panel on climate change expert meeting on assessing and combining multi model climate projections. In T. F. Stocker, et al. (Eds.), IPCC working group I technical support unit. Switzerland: University of Bern, Bern.Google Scholar
  32. Knutti, R., Masson, D., & Gettelman, A. (2013). Climate model genealogy: Generation CMIP5 and how we got there. Geophysical Research Letters, 40(6), 1194–1199. Scholar
  33. Knutti, R., et al. (2017). A climate model projection weighting scheme accounting for performance and interdependence. Geophysical Research Letters, 44(4), 1–10. Scholar
  34. Lenhard, J., & Winsberg, E. (2010). Holism, entrenchment, and the future of climate model pluralism. Studies in History and Philosophy of Science Part B: Studies in History and Philosophy of Modern Physics, 41(3), 253–262. (Elsevier). Scholar
  35. Lenton, T. M., et al. (2008). Tipping elements in the Earth’s climate system. Proceedings of the National Academy of Sciences of the United States of America. Scholar
  36. Levins, R. (1966). The strategy of model building in population biology. American Naturalist, 421–431.
  37. Lloyd, E. A. (2009). I—Elisabeth A. Lloyd: Varieties of support and confirmation of climate models. Aristotelian Society Supplementary Volume, 83(1), 213–232. Scholar
  38. Lloyd, E. (2010). Confirmation and robustness of climate models. Philosophy of Science, 77(5), 971–984. Retrieved July 7, 2014, from Scholar
  39. Mahlstein, I., & Knutti, R. (2012). September Arctic sea ice predicted to disappear near 2 °C global warming above present. Journal of Geophysical Research, 117(D6), 1–11. Scholar
  40. Masson, D., & Knutti, R. (2011a). Climate model genealogy. Geophysical Research Letters, 38(8), L08703. Scholar
  41. Masson, D., & Knutti, R. (2011b). Spatial-scale dependence of climate model performance in the CMIP3 ensemble. Journal of Climate, 24(11), 2680–2692. Scholar
  42. Masson, D., & Knutti, R. (2013). Predictor screening, calibration, and observational constraints in climate model ensembles: An illustration using climate sensitivity. Journal of Climate, 26(3), 887–898. Scholar
  43. Massonnet, F., et al. (2012). Constraining projections of summer Arctic sea ice. The Cryosphere, 6(6), 1383–1394. Scholar
  44. Mazzocchi, F., & Pasini, A. (2017). Climate model pluralism beyond dynamical ensembles. Wiley Interdisciplinary Reviews: Climate Change, 8(6), e477. Scholar
  45. McFarlane, N. (2011). Parameterizations: Representing key processes in climate models without resolving them. Wiley Interdisciplinary Reviews: Climate Change, 2(4), 482–497. Scholar
  46. Medhaug, I., et al. (2017). Reconciling controversies about the “global warming hiatus”. Nature, 545(7652), 41–47. (Nature Publishing Group). Scholar
  47. Notz, D., & Stroeve, J. (2016). Observed Arctic sea-ice loss directly follows anthropogenic CO2 emission. Science, 354(6313), 747–750. Scholar
  48. Oreskes, N., Shrader-Frechette, K., & Belitz, K. (1994). Verification, validation, and confirmation of numerical models in the earth sciences. Science, 263(5147), 641. AAAS. Retrieved June 4, 2014, from;263/5147/641.
  49. Overland, J. E., & Wang, M. (2013). When will the summer Arctic be nearly sea ice free? Geophysical Research Letters, 40(10), 2097–2101. Scholar
  50. Parker, W. S. (2006). Understanding pluralism in climate modeling. Foundations of Science, 11(4), 349–368. (Springer).
  51. Parker, W. S. (2009). Confirmation and adequacy-for-purpose in climate modelling. Aristotelian Society Supplementary, 83(1), 233–249. Scholar
  52. Parker, W. S. (2010). Predicting weather and climate: Uncertainty, ensembles and probability. Studies in History and Philosophy of Science Part B: Studies in History and Philosophy of Modern Physics, 41(3), 263–272. (Elsevier). Scholar
  53. Parker, W. S. (2011). When climate models agree: The significance of robust model predictions. Philosophy of Science, 78(4), 579–600. Scholar
  54. Parker, W. S. (2013). Ensemble modeling, uncertainty and robust predictions. Wiley Interdisciplinary Reviews: Climate Change, 4(3), 213–223. Scholar
  55. Pennell, C., & Reichler, T. (2011). On the effective number of climate models. Journal of Climate, 24(9), 2358–2367. Scholar
  56. Saffioti, C., et al. (2016). Reconciling observed and modeled temperature and precipitation trends over Europe by adjusting for circulation variability. Geophysical Research Letters, 43(15), 8189–8198. Scholar
  57. Sanderson, B. M., & Knutti, R. (2012). On the interpretation of constrained climate model ensembles. Geophysical Research Letters, 39(16), L16708. Scholar
  58. Sanderson, B. M., Knutti, R., & Caldwell, P. (2015a). A representative democracy to reduce interdependency in a multimodel ensemble. Journal of Climate, 28(13), 5171–5194. Scholar
  59. Sanderson, B. M., Knutti, R., & Caldwell, P. (2015b). Addressing interdependency in a multimodel ensemble by interpolation of model properties. Journal of Climate, 28(13), 5150–5170. Scholar
  60. Sanderson, B. M., Wehner, M., & Knutti, R. (2017). Skill and independence weighting for multi-model assessments. Geoscientific Model Development, 10(6), 2379–2395. Scholar
  61. Schaller, N., et al. (2011). Analyzing precipitation projections: A comparison of different approaches to climate model evaluation. Journal of Geophysical Research, 116(D10), 1–14. Scholar
  62. Schneider, T., et al. (2017). Climate goals and computing the future of clouds. Nature Climate Change, 7(1), 3–5. (Nature Publishing Group). Scholar
  63. Schupbach, J. N. (2016). Robustness analysis as explanatory reasoning. The British Journal for the Philosophy of Science, 69(February), axw008.
  64. Screen, J. A., & Francis, J. A. (2016). Contribution of sea-ice loss to Arctic amplification is regulated by Pacific Ocean decadal variability. Nature Climate Change, 6(9), 856–860. Scholar
  65. Stainforth, D. A., et al. (2005). Uncertainty in predictions of the climate response to rising levels of greenhouse gases. Nature, 433(7024), 403–406. Scholar
  66. Stegenga, J., & Menon, T. (2017). Robustness and independent evidence. 84(July), 414–435.
  67. Swart, N. C., et al. (2015). Influence of internal variability on Arctic sea-ice trends. Nature Climate Change, 5(2), 86–89. (Nature Publishing Group). Scholar
  68. Tebaldi, C., et al. (2004). Regional probabilities of precipitation change: A Bayesian analysis of multimodel simulations. Geophysical Research Letters, 31(24), 1–5. Scholar
  69. Vuuren, D. P., et al. (2011). The representative concentration pathways: An overview. Climatic Change, 109(1–2), 5–31. Scholar
  70. Waugh, D. W., & Eyring, V. (2008). Quantitative performance metrics for stratospheric-resolving chemistry-climate models. Atmospheric Chemistry and Physics, 8(18), 5699–5713. Scholar
  71. Weigel, A. P., et al. (2010). Risks of model weighting in multimodel climate projections. Journal of Climate, 23(15), 4175–4191. Scholar
  72. Weisberg, M. (2006). Robustness analysis. Philosophy of Science, 73(December), 730–742.MathSciNetCrossRefGoogle Scholar
  73. Wimsatt, W. C. (2012). Robustness, reliability, and overdetermination (1981). In L. Soler et al. (Eds.), Characterizing the robustness of science: After the practice turn in philosophy of science (pp. 61–87). Dordrecht: Springer Netherlands. Scholar
  74. Woodward, J. (2006). Some varieties of robustness. Journal of Economic Methodology, 13(2), 219–240. Scholar

Copyright information

© Springer Nature Switzerland AG 2019

Authors and Affiliations

  • Reto Knutti
    • 1
    Email author
  • Christoph Baumberger
    • 1
  • Gertrude Hirsch Hadorn
    • 1
  1. 1.ETH ZurichZürichSwitzerland

Personalised recommendations