Climate Dynamics

, Volume 38, Issue 11–12, pp 2513–2542

Multivariate probabilistic projections using imperfect climate models part I: outline of methodology

  • David M. H. Sexton
  • James M. Murphy
  • Mat Collins
  • Mark J. Webb


We demonstrate a method for making probabilistic projections of climate change at global and regional scales, using examples consisting of the equilibrium response to doubled CO2 concentrations of global annual mean temperature and regional climate changes in summer and winter temperature and precipitation over Northern Europe and England-Wales. This method combines information from a perturbed physics ensemble, a set of international climate models, and observations. Our approach is based on a multivariate Bayesian framework which enables the prediction of a joint probability distribution for several variables constrained by more than one observational metric. This is important if different sets of impacts scientists are to use these probabilistic projections to make coherent forecasts for the impacts of climate change, by inputting several uncertain climate variables into their impacts models. Unlike a single metric, multiple metrics reduce the risk of rewarding a model variant which scores well due to a fortuitous compensation of errors rather than because it is providing a realistic simulation of the observed quantity. We provide some physical interpretation of how the key metrics constrain our probabilistic projections. The method also has a quantity, called discrepancy, which represents the degree of imperfection in the climate model i.e. it measures the extent to which missing processes, choices of parameterisation schemes and approximations in the climate model affect our ability to use outputs from climate models to make inferences about the real system. Other studies have, sometimes without realising it, treated the climate model as if it had no model error. We show that omission of discrepancy increases the risk of making over-confident predictions. Discrepancy also provides a transparent way of incorporating improvements in subsequent generations of climate models into probabilistic assessments. The set of international climate models is used to derive some numbers for the discrepancy term for the perturbed physics ensemble, and associated caveats with doing this are discussed.


Uncertainty Probabilistic climate projections Climate prediction Probability Bayesian Metrics Observational constraints Climate sensitivity Discrepancy Model inadequacy 

Supplementary material

382_2011_1208_MOESM1_ESM.pdf (1.6 mb)
Supplementary material 1 PDF (1675 KB)


  1. Adler RF, Huffman GJ, Chang A, Ferraro R, Xie P, Janowiak J, Rudolf B, Schneider U, Curtis S, Bolvin D, Gruber A, Susskind J, Arkin P (2003) The version 2 global precipitation climatology project (GPCP) monthly precipitation analysis (1979–present). J Hydrometeorol 4:1147–1167CrossRefGoogle Scholar
  2. Allan RJ, Ansell TJ (2006) A new globally complete monthly historical gridded mean sea level pressure data set (HadSLP2): 1850–2003. J Clim 19:5816–5842CrossRefGoogle Scholar
  3. Allen MR, Tett SFB (1999) Checking for model consistency in optimal finger printing. Clim Dyn 15:419–434CrossRefGoogle Scholar
  4. Allen MR, Stott PA, Mitchell JFB, Schnur R, Delworth TL (2000) Quantifying the uncertainty in forecasts of anthropogenic climate change. Nature 407:617–620CrossRefGoogle Scholar
  5. Aumann HH, Chahine MT, Gautier C, Goldberg MD, Kalnay E, McMillin LM, Revercomb H, Rosenkranz PW, Smith WL, Staelin DH, Strow LL, Susskind J (2003) AIRS/AMSU/HSB on the aqua mission: design, science objectives, data products, and processing systems. IEEE Trans Geosci Remote Sens 41:253–264CrossRefGoogle Scholar
  6. Barnett DN, Brown SJ, Murphy JM, Sexton DMH, Webb MJ (2006) Quantifying uncertainty in changes in extreme event frequency in response to doubled CO2 using a large ensemble of GCM simulations. Clim Dyn 26:489–511CrossRefGoogle Scholar
  7. Berry DI, Kent EC (2009) A new air-sea interaction gridded dataset from ICOADS with uncertainty estimates. Bull Am Meteorol Soc 90:645–656CrossRefGoogle Scholar
  8. Bony S, Dufresne JL (2005) Marine boundary layer clouds at the heart of cloud feedback uncertainties in climate models. Geophys Res Lett 32:L20806CrossRefGoogle Scholar
  9. Box GE (1980) Sampling and Bayes’ inference in scientific modelling and robustness. J R Stat Soc 143:383–430CrossRefGoogle Scholar
  10. Brohan P, Kennedy J, Harris I, Tett SFB, Jones PD (2006) Uncertainty estimates in regional and global observed temperature changes: a new dataset from 1850. J Geophys Res 111:D12106. doi:10.1029/2005JD006548 CrossRefGoogle Scholar
  11. Collins M, Booth BBB, Bhaskaran B, Harris G, Murphy JM, Sexton DMH, Webb MJ (2010) A comparison of perturbed physics and multi-model ensembles: model errors, feedbacks and forcings. Clim Dyn doi:10.1007/s00382-010-0808-0
  12. da Silva AM, Young CC, Levitus S (1994) Atlas of surface marine data, vol 1: algorithms and procedures. Tech Rep 6, U.S. Department of Commerce, NOAA, NESDISGoogle Scholar
  13. Draper NR, Smith H (1998) Applied regression analysis. Wiley Series in Probability and Statisics, 3rd edn. Wiley, New YorkGoogle Scholar
  14. Evans M, Swartz T (2000) Approximating integrals via Monte Carlo and deterministic methods, vol 20 of Oxford Statistical Science Series, 1st edn. Oxford University Press, OxfordGoogle Scholar
  15. Frame DJ, Booth BBB, Kettleborough JA, Stainforth DA, Gregory JM, Collins M, Allen MR (2005) Constraining climate forecasts: the role of prior assumptions. Geophys Res Lett 32:L09702. doi:10.1029/2004GL022241 CrossRefGoogle Scholar
  16. Furrer R, Sain SR, Nychka D, Meehl GA (2007) Multivariate Bayesian analysis of atmosphere-ocean general circulation models. Environ Ecol Stat 13:249–266CrossRefGoogle Scholar
  17. Giorgi F, Francisco R (2000) Uncertainties in regional climate change predictions. A regional analysis of ensemble simulations with the HadCM2 GCM. Clim Dyn 16:169–182CrossRefGoogle Scholar
  18. Giorgi F, Mearns L (2003) Probability of regional climate change calculated using the reliability ensemble average (REA) method. Geophys Res Lett 30:1629–1632CrossRefGoogle Scholar
  19. Goldstein M, Rougier J (2004) Probabilistic formulations for transferring inferences from mathematical models to physical systems. SIAM J Sci Comput 26:467–487CrossRefGoogle Scholar
  20. Goldstein M, Rougier J (2006) Bayes linear calibrated prediction for complex systems. J Am Stat Assoc 101:1132–1143. doi:10.1198/016214506000000203 CrossRefGoogle Scholar
  21. Greene A, Goddard L, Lall U (2006) Probabilistic multimodel regional temperature change projections. J Clim 19:4326–4343CrossRefGoogle Scholar
  22. Harris GR, Sexton DMH, Booth BBB, Collins M, Murphy JM, Webb MJ (2006) Frequency distributions of transient regional climate change from perturbed physics ensembles of general circulation model simulations. Clim Dyn 27:357–375. doi:10.1007/s00382-006-0142-8 CrossRefGoogle Scholar
  23. Harrison EP, Minnis P, Barkstrom BR, Ramanathan V, Cess RD, Gibson GG (1990) Seasonal variation of cloud radiative forcing derived from the Earth Radiation Budget Experiment. J Geophys Res 95:18687–18703CrossRefGoogle Scholar
  24. Josey SA, Kent EC, Oakley D, Taylor PK (1996) A new global air-sea heat and momentum climatology. Int WOCE Newsl 24:3–5Google Scholar
  25. Joshi MM, Webb MJ, Maycock AC, Collins M (2010) Stratospheric water vapour and high climate sensitivity in a version of HadSM3 climate model. Atmos Chem Phys Discuss 10:7161–7167CrossRefGoogle Scholar
  26. Jun M, Knutti R, Nychka D (2008) Spatial analysis to quantify numerical model bias and dependence: how many climate models are there? J Am Stat Assoc 103:934–947CrossRefGoogle Scholar
  27. Kennedy M, O’Hagan A (2001) Bayesian calibration of computer models. J R Stat Soc B:425–464CrossRefGoogle Scholar
  28. Knutti R (2010) The end of model democracy? Clim Change doi:10.10007/s10584-010-9800-2
  29. Knutti R, Meehl GA, Allen MR, Stainforth DA (2006) Constraining climate sensitivity from the seasonal cycle in surface temperature. J Clim 19:4224–4233CrossRefGoogle Scholar
  30. Lopez A, Tebaldi C, New M, Stainforth DA, Allen MR, Kettleborough JA (2006) Two approaches to quantify uncertainty in global temperature changes under different forcing scenarios. J Clim 19:4785–4796. doi:10.1175/JCLI3895.1 Google Scholar
  31. McAvaney BJ, Le Treut H (2003) The cloud feedback intercomparison project: (CFMIP). CLIVAR exchanges—supplementary contributionsGoogle Scholar
  32. Meehl GA, Stocker TF, Collins WD, Friedlingstein P, Gaye AT, Gregory JM, Kitoh A, Knutti R, Murphy JM, Noda A, Raper SCB, Watterson IG, Weaver AJ, Zhao Z (2007) Global climate projections. In: Solomon S, Qin D, Manning M, Chen Z, Marquis M, Averyt KB, Tignor M, Miller HL (eds) Climate change 2007: the physical science basis. Contribution of Working Group I to the 4th Assessment Report of the Intergovernmental Panel on Climate Change. Cambridge University PressGoogle Scholar
  33. Morse A, Prentice C, Carter T (2009) ENSEMBLES: climate change and its impacts: summary of research and results from the ENSEMBLES project. Met Office, Hadley Centre for Climate Prediction and Research, FitzRoy Road, Exeter, chapter 9 Assessments of climate change impacts, pp 107–129Google Scholar
  34. Murphy JM, Sexton DMH, Barnett DN, Jones GS, Webb MJ, Collins M, Stainforth DA (2004) Quantification of modelling uncertainties in a large ensemble of climate change simulations. Nature 430:768–772CrossRefGoogle Scholar
  35. Murphy J, Booth B, Collins M, Harris G, Sexton D, Webb M (2007) A methodology for probabilistic predictions of regional climate change from perturbed physics ensembles. Phil Trans R Soc Lond 365:2133CrossRefGoogle Scholar
  36. Murphy JM, Sexton DMH, Jenkins GJ, Booth BBB, Brown CC, Clark RT, Collins M, Harris GR, Kendon EJ, Betts RA, Brown SJ, Humphrey KA, McCarthy MP, McDonald RE, Stephens A, Wallace C, Warren R, Wilby R, Wood RA (2009) UK climate projections science report: climate change projections. Met Office Hadley Centre, ExeterGoogle Scholar
  37. New M, Lopez A, Dessai S, Wilby R (2007) Challenges in using probabilistic climate change information for impact assessments: an example from the water sector. Phil Trans R Soc Lond 365:2117–2132CrossRefGoogle Scholar
  38. Nounou MN, Bakshi BR, Goel PK, Shen XT (2002) Process modeling by Bayesian latent variable regression. Ambio 48:1775–1793Google Scholar
  39. O’Hagan A, Forster J (2004) Bayesian inference, vol 2b of Kendall’s Advanced Theory of Statistics, 2nd edn. Edward Arnold, LondonGoogle Scholar
  40. Piani C, Frame DJ, Stainforth DA, Allen MR (2005) Constraints on climate change from a multi-thousand member ensemble of simulations. Geophys Res Lett 32:L23825. doi:10.1029/2005GL024452
  41. Pierce DW, Barnett TP, Santer BD, Gleckler PJ (2009) Selecting global climate models for regional climate change studies. Proc Natl Acad Sci USA 106:8441–8446Google Scholar
  42. Pope VD, Gallani ML, Rowntree PR, Stratton RA (2000) The impact of new physical parametrizations in the Hadley Centre climate model–HadAM3. Clim Dyn 16:123–146CrossRefGoogle Scholar
  43. Press WH, Teukolsky SA, Vetterling WT, Flannery BP (1992) Numerical recipes in Fortran: the art of scientific computing, 2nd edn. Cambridge University Press, CambridgeGoogle Scholar
  44. Randall DA, Wood RA, Bony S, Colman R, Fichefet T, Fyfe J, Kattsov V, Pitman A, Shukla J, Srinivasan J, Stouffer RJ, Sumi A, Taylor KE (2007) Climate models and their evaluation Climate Change 2007. In: Solomon S, Qin D, Manning M, Chen Z, Marquis M, Averyt KB, Tignor M, Miller HL (eds) The physical science basis. Contribution of Working Group I to the 4th Assessment Report of the Intergovernmental Panel on Climate Change. Cambridge University PressGoogle Scholar
  45. Rayner NA, Parker DE, Horton EB, Folland CK, Alexander LV, Rowell DP, Kent EC, Kaplan A (2003) Global analyses of SST, sea ice and night marine air temperature since the late nineteenth century. J Geophys Res 108:4407. doi:10.1029/2002JD002670 Google Scholar
  46. Rayner N, Brohan P, Parker DE, Folland CK, Kennedy J, Vanicek M, Ansell T, Tett SFB (2006) Improved analyses of changes and uncertainties in sea surface temperature measured in situ since the mid-nineteenth century. J Clim 19:446–469CrossRefGoogle Scholar
  47. Reichler T, Kim J (2008) How well do coupled models simulate today’s climate? Bull Am Meteorol Soc 89:303–311. doi:10.1175/BAMS-89-3-303 CrossRefGoogle Scholar
  48. Robert CP, Casella G (2004) Monte Carlo statistical methods, 2nd edn. Springer, New YorkGoogle Scholar
  49. Rodwell MJ, Palmer TN (2007) Using numerical weather prediction to assess climate models. Q J R Meteorol Soc 133:129–146. doi:10.1002/qj.23 CrossRefGoogle Scholar
  50. Rossow WB, Zhang YC (1995) Calculation of surface and top of atmosphere radiative fluxes from physical quantities based on ISCCP data sets. 2, validation and first results. J Geophys Res 100:1167–1197CrossRefGoogle Scholar
  51. Rougier J (2007) Probabilistic inference for future climate using an ensemble of climate model evaluations. Clim Change 81:247–264CrossRefGoogle Scholar
  52. Rougier J, Sexton DMH (2007) Inference in ensemble experiments. Phil Trans R Soc Lond 365:2133–2143. doi:10.1098/rsta.2007.2071 CrossRefGoogle Scholar
  53. Rougier J, Sexton DMH, Murphy JM, Stainforth DA (2009) Analyzing the climate sensitivity of the HadSM3 climate model using ensembles from different but related experiments. J Clim 22:1327–1353CrossRefGoogle Scholar
  54. Sanderson BM (2011) A multi-model study of parametric uncertainty in response to rising greenhouse gases concentrations. J Clim 24:1362–1377 (submitted)Google Scholar
  55. Sanderson BM, Piani C, Ingram WJ, Stone DA, Allen MR (2008) Towards constraining climate sensitivity by linear analysis of feedback patterns in thousands of perturbed-physics GCM simulations. Clim Dyn 30:175–190. doi:10.1007/s00382-007-0280-7 CrossRefGoogle Scholar
  56. Stainforth DA, Aina T, Christensen C, Collins M, Frame DJ, Kettleborough JA, Knight S, Martin A, Murphy J, Piani C, Sexton D, Smith LA, Spicer RA, Thorpe AJ, Allen MR (2005) Uncertainty in predictions of the climate response to rising levels of greenhouse gases. Nature 433:403–406CrossRefGoogle Scholar
  57. Stott PA, Kettleborough JA (2002) Origins and estimates of uncertainty in predictions of twenty first century temperature rise. Nature 416:723–726CrossRefGoogle Scholar
  58. Stott PA, Mitchell JFB, Allen MR, Delworth TL, Gregory JM, Meehl GA, Santer BD (2006) Observational constraints on past attributable warming and predictions of future global warming. J Clim 19:3055–3069CrossRefGoogle Scholar
  59. Tebaldi C, Lobell DB (2008) Towards probabilistic projections of climate change impacts on global crop yields. Geophys Res Lett 35:L08705. doi:10.1029/2008GL033423
  60. Tebaldi C, Sanso B (2009) Joint projections of temperature and precipitation change from multiple climate models: a hierarchical bayesian approach. J R Stat Soc 172:83–106CrossRefGoogle Scholar
  61. Tebaldi C, Mearns LO, Nychka D, Smith RW (2004) Regional probabilities of precipitation change: a Bayesian analysis of multimodel simulations. Geophys Res Lett 31:L24213. doi:10.1029/2004GL021276
  62. Uppala SM, Kållberg PW, Simmons AJ, Andrae U, daCosta Bechtold V, Fiorino M, Gibson JK, Haseler J, Hernandez A, Kelly GA, Li X, Onogi K, Saarinen S, Sokka N, Allan RP, Andersson E, Arpe K, Balmaseda MA, Beljaars ACM, van de Berg L, Bidlot J, Bormann N, Caires S, Chevallier F, Dethof A, Dragosavac M, Fisher M, Fuentes M, Hagemann S, Hólm E, Hoskins BJ, Isaksen L, Janssen PAEM, Jenne R, McNally AP, Mahfouf J-F, Morcrette J-J, Rayner NA, Saunders RW, Simon P, Sterl A, Trenberth KE, Untch A, Vasiljevic D, Viterbo P, Woollen J (2005) The ERA-40 re-analysis. Q J R Meteorol Soc 131:2961–3012. doi:10.1256/qj.04.176 CrossRefGoogle Scholar
  63. Watterson IG (2008) Calculation of probability density functions for temperature and precipitation change under global warming. J Geophys Res 116:D07101. doi:10.1029/2007JD009254
  64. Webb MJ, Senior CA, Sexton DMH, Ingram WJ, Williams KD, Ringer MA, McAvaney BJ, Colman R, Soden BJ, Gudgel R, Knutson T, Emori S, Ogura T, Tsushima Y, Andronova NG, Li B, Musat I, Bony S, Taylor KE (2006) On the contribution of local feedback mechanisms to the range of climate sensitivity in two GCM ensembles. Clim Dyn 27:17–38. doi:10.1007/s00382-006-0111-2 CrossRefGoogle Scholar
  65. Wielicki B, Raj M, Sau S (1996) Clouds and the Earth’s Radiant Energy System (CERES): an earth observing system experiment. Bull Am Meteorol Soc 77:853–868Google Scholar
  66. Williams KD, Webb MJ (2009) A quantitative performance assessment of cloud regimes in climate models. Clim Dyn 33:141–157. doi:10.1007/s00382-008-0443-1 CrossRefGoogle Scholar
  67. Williams KD, Ringer MA, Senior CA, Webb MJ, McAvaney BJ, Andronova N, Bony S, Dufresne J-L, Emori S, Gudgel R, Knutson T, Li B, Lo K, Musat I, Wegner J, Slingo A, Mitchell JFB (2006) Evaluation of a component of the cloud response to climate change in an intercomparison of climate models. Clim Dyn 26:145–165. doi:10.1007/s00382-005-0067-7 CrossRefGoogle Scholar
  68. Wylie DP, Menzel WP (1999) Eight years of high cloud statistics using HIRS. J Clim 12(1):170–184CrossRefGoogle Scholar
  69. Xie P, Arkin PA (1996) Analyses of global monthly precipitation using gauge observations, satelite estimates and numerical model predections. J Clim 9:840–858CrossRefGoogle Scholar
  70. Yokohata T, Webb MJ, Collins M, Williams KD, Yoshimori M, Hargreaves JD, Annan JD (2010) Structural similarities and differences in climate responses to CO2 increase between two perturbed physics ensembles. J Clim 23:1392–1410. doi:10.1175/2009JCLI2917.1 CrossRefGoogle Scholar

Copyright information

© Crown copyright 2011

Authors and Affiliations

  • David M. H. Sexton
    • 1
  • James M. Murphy
    • 1
  • Mat Collins
    • 1
  • Mark J. Webb
    • 1
  1. 1.Met Office Hadley CentreExeterUK

Personalised recommendations