Skip to main content

Advertisement

Springer Nature Link
Log in

A comparison of metrics for assessing state-of-the-art climate models and implications for probabilistic projections of climate change

  • Published:
Climate Dynamics Aims and scope Submit manuscript

Abstract

A major task of climate science are reliable projections of climate change for the future. To enable more solid statements and to decrease the range of uncertainty, global general circulation models and regional climate models are evaluated based on a 2 × 2 contingency table approach to generate model weights. These weights are compared among different methodologies and their impact on probabilistic projections of temperature and precipitation changes is investigated. Simulated seasonal precipitation and temperature for both 50-year trends and climatological means are assessed at two spatial scales: in seven study regions around the globe and in eight sub-regions of the Mediterranean area. Overall, 24 models of phase 3 and 38 models of phase 5 of the Coupled Model Intercomparison Project altogether 159 transient simulations of precipitation and 119 of temperature from four emissions scenarios are evaluated against the ERA-20C reanalysis over the 20th century. The results show high conformity with previous model evaluation studies. The metrics reveal that mean of precipitation and both temperature mean and trend agree well with the reference dataset and indicate improvement for the more recent ensemble mean, especially for temperature. The method is highly transferrable to a variety of further applications in climate science. Overall, there are regional differences of simulation quality, however, these are less pronounced than those between the results for 50-year mean and trend. The trend results are suitable for assigning weighting factors to climate models. Yet, the implications for probabilistic climate projections is strictly dependent on the region and season.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+
from $39.99 /Month
  • Starting from 10 chapters or articles per month
  • Access and download chapters and articles from more than 300k books and 2,500 journals
  • Cancel anytime
View plans

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11

Similar content being viewed by others

Explore related subjects

Discover the latest articles, books and news in related subjects, suggested using machine learning.

References

  • Armistead TW (2013) H. L. Wagner’s unbiased hit rate and the assessment of categorical forecasting accuracy. Weather Forecast 28:802–814. doi:10.1175/WAF-D-12-00047.1

    Article  Google Scholar 

  • Ayar PV, Vrac M, Bastin S, Carreau J, Déqué M, Gallardo C (2016) Intercomparison of statistical and dynamical downscaling models under the EURO- and MED-CORDEX initiative framework: present climate evaluations. Clim Dyn 46:1301–1329. doi:10.1007/s00382-015-2647-5

    Article  Google Scholar 

  • Babak O, Deutsch CV (2009) Statistical approach to inverse distance interpolation. Stoch Environ Res Risk Assess 23:543–553. doi:10.1007/s00477-008-0226-6

  • Bishop CH, Abramowitz G (2013) Climate model dependence and the replicate Earth paradigm. Clim Dyn 41:885–900. doi:10.1007/s00382-012-1610-y

    Article  Google Scholar 

  • Bortz J, Lienert GA, Boehnke K (2008) Verteilungsfreie Methoden in der Biostatistik, 3rd edn. Springer Berlin Heidelberg, Berlin, Heidelberg

    Google Scholar 

  • Clark MP, Wilby RL, Gutmann ED et al (2016) Characterizing uncertainty of the hydrologic impacts of climate change. Curr Clim Change Rep 2:55–64. doi:10.1007/s40641-016-0034-x

    Article  Google Scholar 

  • Diffenbaugh NS, Giorgi F (2012) Climate change hotspots in the CMIP5 global climate model ensemble. Clim Change 114:813–822. doi:10.1007/s10584-012-0570-x

    Article  Google Scholar 

  • Dittus AJ, Karoly DJ, Lewis SC et al (2016) A multiregion model evaluation and attribution study of historical changes in the area affected by temperature and precipitation extremes. J Clim 29:8285–8299. doi:10.1175/JCLI-D-16-0164.1

    Article  Google Scholar 

  • Donat MG, Alexander L V., Herold N, Dittus AJ (2016) Temperature and precipitation extremes in century-long gridded observations, reanalyses, and atmospheric model simulations. J Geophys Res Atmos 121:11,174–11,189. doi:10.1002/2016JD025480

    Article  Google Scholar 

  • Done J, Davis CA, Weisman M (2004) The next generation of NWP: explicit forecasts of convection using the weather research and forecasting (WRF) model. Atmos Sci Lett 5:110–117. doi:10.1002/asl.72

    Article  Google Scholar 

  • Doolittle MH (1885) The verification of predictions. Amer Meteor J 2:327–329

    Google Scholar 

  • Doolittle MH (1888) Association ratios. Bull Philos Soc Wash 10:83–96

    Google Scholar 

  • Eum H-I, Gachon P, Laprise R (2014) Developing a likely climate scenario from multiple regional climate model simulations with an optimal weighting factor. Clim Dyn 43:11–35. doi:10.1007/s00382-013-2021-4

    Article  Google Scholar 

  • Flato G, Marotzke J, Abiodun B et al (2013) Evaluation of climate models. In: Climate change 2013: the physical science basis. Contribution of working group I to the fifth assessment report of the intergovernmental panel on climate change. Cambridge University Press, Cambridge, p 866

    Google Scholar 

  • Gart JJ, Zweifel JR (1967) On the bias of various estimators of the logit and its variance with application to quantal bioassay. Biometrika 54:181. doi:10.2307/2333861

    Article  Google Scholar 

  • Ghelli A, Primo C (2009) On the use of the extreme dependency score to investigate the performance of an NWP model for rare events. Meteorol Appl 16:537–544. doi:10.1002/met.153

    Article  Google Scholar 

  • Gilbert GF (1884) Finley’s tornado predictions. Amer Meteor J 1:166–172

    Google Scholar 

  • Gill PG, Buchanan P (2014) An ensemble based turbulence forecasting system. Meteorol Appl 21:12–19. doi:10.1002/met.1373

    Article  Google Scholar 

  • Gillett NP, Annan J, Hargreaves J et al (2015) Weighting climate model projections using observational constraints. Philos Trans A Math Phys Eng Sci 373:L02703. doi:10.1098/rsta.2014.0425

    Article  Google Scholar 

  • Giorgi F (2006) Climate change hot-spots. Geophys Res Lett 33:L08707. doi:10.1029/2006GL025734

    Article  Google Scholar 

  • Giorgi F, Lionello P (2008) Climate change projections for the Mediterranean region. Glob Planet Change 63:90–104. doi:10.1016/j.gloplacha.2007.09.005

    Article  Google Scholar 

  • Giorgi F, Jones C, Asrar G (2009) Addressing climate information needs at the regional level: the CORDEX framework. WMO Bull 58:175–183

    Google Scholar 

  • Gleckler PJ, Taylor KE, Doutriaux C (2008) Performance metrics for climate models. J Geophys Res 113:D06104. doi:10.1029/2007JD008972

    Article  Google Scholar 

  • Grose MR, Brown JN, Narsey S, Brown JR, Murphy BF, Langlais C, Gupta AS, Moise AF, Irving DB (2014) Assessment of the CMIP5 global climate model simulations of the western tropical Pacific climate system and comparison to CMIP3. Int J Climatol 34 (12):3382–3399

    Article  Google Scholar 

  • Haughton N, Abramowitz G, Pitman A, Phipps SJ (2015) Weighting climate model ensembles for mean and variance estimates. Clim Dyn 45:3169–3181. doi:10.1007/s00382-015-2531-3

    Article  Google Scholar 

  • Hawkins E, Smith RS, Gregory JM, Stainforth DA (2016) Irreducible uncertainty in near-term climate projections. Clim Dyn 46:3807–3819. doi:10.1007/s00382-015-2806-8

    Article  Google Scholar 

  • Haylock MR, Hofstra N, Klein Tank AMG et al (2008) A European daily high-resolution gridded data set of surface temperature and precipitation for 1950–2006. J Geophys Res 113:D20119. doi:10.1029/2008JD010201

    Article  Google Scholar 

  • Heidke P (1926) Berechnung des Erfolges und der Güte der Windstärkevorhersagen im Sturmwarnungdienst (Calculation of the success and goodness of strong wind forecasts in the storm warning service). Geogr Ann Stockholm 8:301–349

    Google Scholar 

  • Hewitson BC, Crane RG (2006) Consensus between GCM climate change projections with empirical downscaling: precipitation downscaling over South Africa. Int J Climatol 26:1315–1337. doi:10.1002/joc.1314

    Article  Google Scholar 

  • Huang D-Q, Zhu J, Zhang Y-C, Huang A-N (2013) Uncertainties on the simulated summer precipitation over Eastern China from the CMIP5 models. J Geophys Res Atmos 118:9035–9047. doi:10.1002/jgrd.50695

    Article  Google Scholar 

  • Jacob D, Petersen J, Eggert B et al (2014) EURO-CORDEX: new high-resolution climate change projections for European impact research. Reg Environ Chang 14:563–578. doi:10.1007/s10113-013-0499-2

    Article  Google Scholar 

  • Knutti R (2010) The end of model democracy? Clim Change 102:395–404. doi:10.1007/s10584-010-9800-2

    Article  Google Scholar 

  • Knutti R, Sedláček J (2012) Robustness and uncertainties in the new CMIP5 climate model projections. Nat Clim Chang 3:369–373. doi:10.1038/nclimate1716

    Article  Google Scholar 

  • Knutti R, Furrer R, Tebaldi C et al (2010) Challenges in combining projections from multiple climate models. J Clim 23:2739–2758. doi:10.1175/2009JCLI3361.1

    Article  Google Scholar 

  • Knutti R, Masson D, Gettelman A (2013) Climate model genealogy: generation CMIP5 and how we got there. Geophys Res Lett 40:1194–1199. doi:10.1002/grl.50256

    Article  Google Scholar 

  • Koutroulis AG, Grillakis MG, Tsanis IK, Papadimitriou L (2016) Evaluation of precipitation and temperature simulation performance of the CMIP3 and CMIP5 historical experiments. Clim Dyn 47:1881–1898. doi:10.1007/s00382-015-2938-x

    Article  Google Scholar 

  • Kumar S, Merwade V, Kinter JL et al (2013) Evaluation of temperature and precipitation trends and long-term persistence in CMIP5 twentieth-century climate simulations. J Clim 26:4168–4185. doi:10.1175/JCLI-D-12-00259.1

    Article  Google Scholar 

  • Li G, Xie S-P (2014) Tropical biases in CMIP5 multimodel ensemble: the excessive equatorial Pacific cold tongue and double ITCZ problems*. J Clim 27(4):1765–1780

    Article  Google Scholar 

  • Liu B, Chen J, Chen X et al (2013) Uncertainty in determining extreme precipitation thresholds. J Hydrol 503:233–245. doi:10.1016/j.jhydrol.2013.09.002

    Article  Google Scholar 

  • Miao C, Duan Q, Sun Q et al (2014) Assessment of CMIP5 climate models and projected temperature changes over Northern Eurasia. Environ Res Lett 9:55007. doi:10.1088/1748-9326/9/5/055007

    Article  Google Scholar 

  • Mitchell TD, Jones PD (2005) An improved method of constructing a database of monthly climate observations and associated high-resolution grids. Int J Climatol 25(6):693–712

    Article  Google Scholar 

  • Moss RH, Edmonds JA, Hibbard KA et al (2010) The next generation of scenarios for climate change research and assessment. Nature 463:747–756. doi:10.1038/nature08823

    Article  Google Scholar 

  • Nakicenovic N, Alcamo J, Davis G et al (2000) Special report on emissions scenarios†¯: a special report of Working Group III of the Intergovernmental Panel on Climate Change. Cambridge University Press, USA

    Google Scholar 

  • Paeth H, Girmes R, Menz G, Hense A (2006) Improving seasonal forecasting in the low latitudes. Mon Weather Rev 134:1859–1879. doi:10.1175/MWR3149.1

    Article  Google Scholar 

  • Paeth H, Vogt G, Paxian A et al (2016) Quantifying the evidence of climate change in the light of uncertainty exemplified by the Mediterranean hot spot region. Glob Planet Change. doi:10.1016/j.gloplacha.2016.03.003

    Google Scholar 

  • Paxian A, Hertig E, Vogt G, Seubert S, Jacobeit J, Paeth H. (2013) Greenhouse gas-related predictability of regional climate model trends in the Mediterranean area. Int J Climatol 34:2293–2307. doi:10.1002/joc.3838

    Article  Google Scholar 

  • Perkins SE, Pitman AJ, Holbrook NJ et al (2007) Evaluation of the AR4 climate models’ simulated daily maximum temperature, minimum temperature, and precipitation over australia using probability density functions. J Clim 20:4356–4376. doi:10.1175/JCLI4253.1

    Article  Google Scholar 

  • Pettigrew HM, Gart JJ, Thomas DG (1986) The bias and higher cumulants of the logarithm of a binomial variate. Biometrika 73:425–435. doi:10.1093/biomet/73.2.425

    Article  Google Scholar 

  • Pielke RA, Wilby RL (2012) Regional climate downscaling: what’s the point? Eos Trans Am Geophys Union 93:52–53. doi:10.1029/2012EO050008

    Article  Google Scholar 

  • Pierce CS (1884) The numerical measure of the success of predictions. Science 4(93):453–454

    Article  Google Scholar 

  • Poli P, Hersbach H, Dee DP et al (2016) ERA-20C: an atmospheric reanalysis of the twentieth century. J Clim 29:4083–4097. doi:10.1175/JCLI-D-15-0556.1

    Article  Google Scholar 

  • Power SB, Delage F, Colman R, Moise A (2012) Consensus on twenty-first-century rainfall projections in climate models more widespread than previously thought. J Clim 25:3792–3809. doi:10.1175/JCLI-D-11-00354.1

    Article  Google Scholar 

  • Randall DA, Bony RA, S. W et al (2007) Cilmate models and their evaluation. In: Solomon S, Qin D, Manning M et al (eds) Climate change 2007: the physical science basis. contribution of working group I to the fourth assessment report of the intergovernmental panel on climate change. Cambridge University Press, Cambridge, p 662

    Google Scholar 

  • Reichler T, Kim J (2008) How well do coupled models simulate today’s climate? Bull Am Meteorol Soc 89:303–311. doi:10.1175/BAMS-89-3-303

    Article  Google Scholar 

  • Ring C, Mannig B, Pollinger F, Paeth H (2016) Uncertainties in the simulation of precipitation in selected regions of humid and dry climate. Int J Climatol 36:3521–3538. doi:10.1002/joc.4573

    Article  Google Scholar 

  • Rummukainen M (2010) State-of-the-art with regional climate models. Wiley Interdiscip Rev Clim Chang 1:82–96. doi:10.1002/wcc.8

    Article  Google Scholar 

  • Sanderson BM, Knutti R, Caldwell P et al (2015) A representative democracy to reduce interdependency in a multimodel ensemble. J Clim 28:5171–5194. doi:10.1175/JCLI-D-14-00362.1

    Article  Google Scholar 

  • Schulzweida U, Kornblueh L, Quast R (2009) CDO User’s Guide. Climate data operators. Version 1.4.1. In: MPI Meteorol. https://www.rsmas.miami.edu/users/rajib/cdo.pdf. Accessed 28 Dec 2016

  • Sheffield J, Barrett AP, Colle B et al (2013) North American Climate in CMIP5 experiments. Part I: evaluation of historical simulations of continental and regional climatology. J Clim 26:9209–9245

    Article  Google Scholar 

  • Stephenson DB (2000) Use of the “odds ratio” for diagnosing forecast skill. Weather Forecast 15:221–232. doi:10.1175/1520-0434(2000)015<0221:UOTORF>2.0.CO;2

    Article  Google Scholar 

  • Tebaldi C, Knutti R, Allen MR et al (2007) The use of the multi-model ensemble in probabilistic climate projections. Philos Trans R Soc A Math Phys Eng Sci 365:2053–2075. doi:10.1098/rsta.2007.2076

    Article  Google Scholar 

  • Thornes JE, Stephenson DB (2001) How to judge the quality and value of weather forecast products. Meteorol Appl 8:S1350482701003061. doi:10.1017/S1350482701003061

    Article  Google Scholar 

  • Von Storch H, Zwiers FW (1999) Statistical analysis in climate research, 1st edn. Cambridge University Press, Cambridge

    Book  Google Scholar 

  • Wilkinson JM (2017) A technique for verification of convection-permitting NWP model deterministic forecasts of lightning activity. Weather Forecast 32:97–115. doi:10.1175/WAF-D-16-0106.1

    Article  Google Scholar 

  • Wilks DS (2006) Statistical methods in the atmospheric sciences, 2. Elsevier, Amsterdam

    Google Scholar 

  • Woodcock F (1976) The evaluation of yes/no forecasts for scientific and administrative purposes. Mon Weather Rev 104:1209–1214. doi:10.1175/1520-0493(1976)104<1209:TEOYFF>2.0.CO;2

    Article  Google Scholar 

  • Wright AN, Schwartz MW, Hijmans RJ, Bradley Shaffer H (2016) Advances in climate models from CMIP3 to CMIP5 do not change predictions of future habitat suitability for California reptiles and amphibians. Clim Change 134:579–591. doi:10.1007/s10584-015-1552-6

    Article  Google Scholar 

Download references

Acknowledgements

We thank the Program for Climate Model Diagnosis and Intercomparison (PCMDI) and the World Climate Research Programme (WCRP) for providing the CMIP3, CMIP5 and CORDEX datasets used in this study. Furthermore, we are grateful for the provided observational data and reanalyses by the Climate Research Unit (CRU), the EU-FP6 project ENSEMBLES, the data providers in the ECA&D project and the European Centre for Medium-Range Weather Forecasts (ECMWF), respectively. This study was conducted within the COMEPRO-Project funded by the Deutsche Forschungsgemeinschaft (DFG).

Author information

Authors and Affiliations

  1. Institute of Geography and Geology, University of Wuerzburg, Am Hubland, 97074, Wuerzburg, Germany

    Christoph Ring, Felix Pollinger & Heiko Paeth

  2. Institute of Geography, University of Augsburg, Alter Postweg 118, 86159, Augsburg, Germany

    Irena Kaspar-Ott, Elke Hertig & Jucundus Jacobeit

Authors
  1. Christoph Ring
    View author publications

    Search author on:PubMed Google Scholar

  2. Felix Pollinger
    View author publications

    Search author on:PubMed Google Scholar

  3. Irena Kaspar-Ott
    View author publications

    Search author on:PubMed Google Scholar

  4. Elke Hertig
    View author publications

    Search author on:PubMed Google Scholar

  5. Jucundus Jacobeit
    View author publications

    Search author on:PubMed Google Scholar

  6. Heiko Paeth
    View author publications

    Search author on:PubMed Google Scholar

Corresponding author

Correspondence to Christoph Ring.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Ring, C., Pollinger, F., Kaspar-Ott, I. et al. A comparison of metrics for assessing state-of-the-art climate models and implications for probabilistic projections of climate change. Clim Dyn 50, 2087–2106 (2018). https://doi.org/10.1007/s00382-017-3737-3

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00382-017-3737-3

Keywords