Evaluation of methods for selecting climate models to simulate future hydrological change

  • Andrew C. RossEmail author
  • Raymond G. Najjar


A challenge for climate impact studies is the selection of a limited number of climate model projections among the dozens that are typically available. Here, we examine the impacts of methods for climate model selection on projections of runoff change for five different watersheds across the conterminous USA. The results from an ensemble of 29 global climate models and 29 corresponding hydrological model simulations are compared with the results that would have been obtained by applying six different selection methods to the climate model data and using only the selected models to drive the hydrological model. We evaluate each selection method based on whether the runoff projections produced by the method meet the method’s objective and on whether the results are sensitive to the number of models chosen. The Katsavounidis–Kuo–Zhang (KKZ) method, which is intended to maximize the spread in the projected climate change, was the only method that met both criteria for suitability. Although the KKZ method generally performed well, the results from both it and the other methods varied somewhat unpredictably based on region and number of models chosen. This study shows that the methods and models used in similar top–down studies should be carefully chosen and that the results obtained should be interpreted with caution.


Climate model ensembles Hydrological change Model selection Model uncertainty 



We thank the anonymous reviewers and editor for comments that improved this manuscript.We also thank the World Climate Research Programme’s Working Group on Coupled Modelling, the U.S. Department of Energy’s Program for Climate Model Diagnosis and Intercomparison, the Global Organization for Earth System Science Portals, and the climate modeling groups (Table S1) for their roles in CMIP, as well as the providers of the downscaled hydrology projections.

Funding information

Funding was provided by the National Science Foundation (CBET-1360286), Pennsylvania Sea Grant (NA10OAR4170063), and National Oceanic and Atmospheric Administration’s National Centers for Coastal Ocean Science (NA16NOS4780207 to Virginia Institute of Marine Science).

Supplementary material

10584_2019_2512_MOESM1_ESM.pdf (69 kb)
(PDF 69.2 KB)
10584_2019_2512_MOESM2_ESM.csv (2 kb)
(CSV 1.73 KB)


  1. Abramowitz G, Bishop CH (2015) Climate model dependence and the ensemble dependence transformation of CMIP projections. J Clim 28(6):2332–2348CrossRefGoogle Scholar
  2. Abramowitz G, Herger N, Gutmann E, Hammerling D, Knutti R, Leduc M, Lorenz R, Pincus R, Schmidt GA (2019) ESD Reviews: Model dependence in multi-model climate ensembles: weighting, sub-selection and out-of-sample testing. Earth Syst Dyn 10(1):91–105CrossRefGoogle Scholar
  3. Al Aamery N, Fox JF, Snyder M (2016) Evaluation of climate modeling factors impacting the variance of streamflow. J Hydrol 542:125–142CrossRefGoogle Scholar
  4. Arnell NW, Gosling SN (2013) The impacts of climate change on river flow regimes at the global scale. J Hydrol 486:351–364CrossRefGoogle Scholar
  5. Bastola S, Murphy C, Sweeney J (2011) The role of hydrological modelling uncertainties in climate change impact assessments of Irish river catchments. Adv Water Resour 34(5):562–576CrossRefGoogle Scholar
  6. Bishop CH, Abramowitz G (2013) Climate model dependence and the replicate Earth paradigm. Clim Dyn 41:885–900CrossRefGoogle Scholar
  7. Brekke LD, Dettinger MD, Maurer EP, Anderson M (2008) Significance of model credibility in estimating climate projection distributions for regional hydroclimatological risk assessments. Clim Chang 89:371–394CrossRefGoogle Scholar
  8. Brekke L, Thrasher BL, Maurer EP, Pruitt T (2013) Downscaled CMIP3 and CMIP5 climate projections: release of downscaled CMIP5 climate projections, comparison with preceding information, and summary of user needs. Technical report, U.S. Department of the Interior, Bureau of Reclamation Technical Services Center, DenverGoogle Scholar
  9. Brekke L, Wood A, Pruitt T (2014) Downscaled CMIP3 and CMIP5 climate and hydrology projections: release of hydrology projections, comparison with preceding information, and summary of user needs. Technical report, U.S. Department of the Interior, Bureau of Reclamation Technical Services Center, DenverGoogle Scholar
  10. Cannon AJ (2015a) Selecting GCM scenarios that span the range of changes in a multimodel ensemble: application to CMIP5 climate extremes indices. J Clim 28 (3):1260–1267CrossRefGoogle Scholar
  11. Cannon AJ, Sobie SR, Murdock TQ (2015b) Bias correction of GCM precipitation by quantile mapping: how well do methods preserve changes in quantiles and extremes?. J Clim 28(17):6938–6959CrossRefGoogle Scholar
  12. Casajus N, Périé C, Logan T, Lambert M-C, de Blois S, Berteaux D (2016) An objective approach to select climate scenarios when projecting species distribution under climate change. PLoS ONE:11Google Scholar
  13. Chai T, Draxler RR (2014) Root mean square error (RMSE) or mean absolute error (MAE)? – arguments against avoiding RMSE in the literature. Geosci Model Dev 7(3):1247–1250CrossRefGoogle Scholar
  14. Chen J, Brissette FP, Lucas-Picher P (2016) Transferability of optimally-selected climate models in the quantification of climate change impacts on hydrology. Clim Dyn 47:3359–3372CrossRefGoogle Scholar
  15. Chen J, Brissette FP, Lucas-Picher P, Caya D (2017) Impacts of weighting climate models for hydro-meteorological climate change studies. J Hydrol 549:534–546CrossRefGoogle Scholar
  16. Frontier S (1976) ÉTude de la décroissance des valeurs propres dans une analyse en composantes principales: Comparaison avec le modèle du bâton brisé. J Exp Mar Biol Ecol 25:67–75CrossRefGoogle Scholar
  17. Gibson JR, Najjar RG (2000) The response of Chesapeake Bay salinity to climate-induced changes in streamflow. Limnol Oceanogr 45(8):1764–1772CrossRefGoogle Scholar
  18. Hagemann S, Chen C, Clark DB, Folwell S, Gosling SN, Haddeland I, Hanasaki N, Heinke J, Ludwig F, Voß F, Wiltshire AJ (2013) Climate change impact on available water resources obtained using multiple global climate and hydrology models. Earth Syst Dyn 4:129–144CrossRefGoogle Scholar
  19. Hartigan JA, Wong MA (1979) Algorithm AS 136: a k-means clustering algorithm. J R Stat Soc Ser C (Appl Stat) 28(1):100–108Google Scholar
  20. Held IM, Soden BJ (2006) Robust responses of the hydrological cycle to global warming. J Clim 19(21):5686–5699CrossRefGoogle Scholar
  21. Herger N, Abramowitz G, Knutti R, Angélil O, Lehmann K, Sanderson BM (2017) Selecting a climate model subset to optimise key ensemble properties. Earth Syst Dyn Discuss 2017:1–24Google Scholar
  22. Hirabayashi Y, Mahendran R, Koirala S, Konoshima L, Yamazaki D, Watanabe S, Kim H, Kanae S (2013) Global flood risk under climate change. Nat Clim Chang 3:816–821CrossRefGoogle Scholar
  23. Holman IP, Allen DM, Cuthbert MO, Goderniaux P (2012) Towards best practice for assessing the impacts of climate change on groundwater. Hydrogeol J 20:1–4CrossRefGoogle Scholar
  24. Houle D, Bouffard A, Duchesne L, Logan T, Harvey R (2012) Projections of future soil temperature and water content for three southern quebec forested sites. J Clim 25:7690–7701CrossRefGoogle Scholar
  25. Hubert L, Arabie P (1985) Comparing partitions. J Classif 2:193–218CrossRefGoogle Scholar
  26. Jackson DA (1993) Stopping rules in principal components analysis: a comparison of heuristical and statistical approaches. Ecol 74(8):2204–2214CrossRefGoogle Scholar
  27. Johnson TE, Butcher JB, Parker A, Weaver CP (2012) Investigating the sensitivity of U.S. streamflow and water quality to climate change: U.S. EPA Global Change Research Program’s 20 Watersheds Project. J Water Resour Plan Manag 138 (5):453–464CrossRefGoogle Scholar
  28. Justić D, Rabalais NN, Turner RE (2005) Coupling between climate variability and coastal eutrophication: evidence and outlook for the northern Gulf of Mexico. J Sea Res 54(1):25–35CrossRefGoogle Scholar
  29. Katsavounidis I, Kuo C, Zhang Z (1994) A new initialization technique for generalized Lloyd iteration. IEEE Signal Process Lett 1(10):144–146CrossRefGoogle Scholar
  30. Kaufmann L, Rousseeuw PJ (1990) Finding groups in data: an introduction to cluster analysis. Wiley, New YorkCrossRefGoogle Scholar
  31. Kerkhoff C, Künsch HR, Schär C (2015) A Bayesian hierarchical model for heterogeneous RCM–GCM multimodel ensembles. J Clim 28(15):6249–6266CrossRefGoogle Scholar
  32. Knutti R, Furrer R, Tebaldi C, Cermak J, Meehl GA (2010) Challenges in combining projections from multiple climate models. J Clim 23:2739–2758CrossRefGoogle Scholar
  33. Knutti R, Masson D, Gettelman A (2013) Climate model genealogy: generation CMIP5 and how we got there. Geophys Res Lett 40:1194–1199CrossRefGoogle Scholar
  34. Krzysztofowicz R (2001) The case for probabilistic forecasting in hydrology. J Hydrol 249(1-4):2–9CrossRefGoogle Scholar
  35. Leduc M, Laprise R, De elía R, Šeparović L (2016) Is institutional democracy a good proxy for model independence?. J Clim 29:8301–8316CrossRefGoogle Scholar
  36. Li H, Sheffield J, Wood EF (2010) Bias correction of monthly precipitation and temperature fields from Intergovernmental Panel on Climate Change AR4 models using equidistant quantile matching. J Geophys Res 115:D10101CrossRefGoogle Scholar
  37. Liang X, Lettenmaier DP, Wood EF, Burges SJ (1994) A simple hydrologically based model of land-surface water and energy fluxes for general-circulation models. J Geophys Res Atmosph 99(D7):14415–14428CrossRefGoogle Scholar
  38. Liang X, Wood EF, Lettenmaier DP (1996) Surface soil moisture parameterization of the VIC-2l model: evaluation and modification. Glob Planet Chang 13:195–206CrossRefGoogle Scholar
  39. Maraun D (2013) Bias correction, quantile mapping, and downscaling: revisiting the inflation issue. J Clim 26(6):2137–2143CrossRefGoogle Scholar
  40. Maraun D, Shepherd TG, Widmann M, Zappa G, Walton D, Gutiérrez JM, Hagemann S, Richter I, Soares PMM, Hall A, Mearns LO (2017) Towards process-informed bias correction of climate change simulations. Nat Clim Chang 7 (11):764–773CrossRefGoogle Scholar
  41. Masson D, Knutti R (2011) Climate model genealogy. Geophys Res Lett:38Google Scholar
  42. Maurer EP, Wood AW, Adam JC, Lettenmaier DP (2002) A long-term hydrologically based dataset of land surface fluxes and states for conterminous United States. J Clim 15:3237–3251CrossRefGoogle Scholar
  43. Maurer EP, Pierce DW (2014) Bias correction can modify climate model simulated precipitation changes without adverse effect on the ensemble mean. Hydrol Earth Syst Sci 18(3):915–925CrossRefGoogle Scholar
  44. McSweeney CF, Jones RG, Lee RW, Rowell DP (2015) Selecting CMIP5 GCMs for downscaling over multiple regions. Clim Dyn 44:3237–3260CrossRefGoogle Scholar
  45. Melsen LA, Addor N, Mizukami N, Newman AJ, Torfs PJJF, Clark MP, Uijlenhoet R, Teuling AJ (2018) Mapping (dis)agreement in hydrologic projections. Hydrol Earth Syst Sci 22(3):1775–1791CrossRefGoogle Scholar
  46. Mendlik T, Gobiet A (2016) Selecting climate simulations for impact studies based on multivariate patterns of climate change. Clim Chang 135:381–393CrossRefGoogle Scholar
  47. Milligan GW, Cooper MC (1986) A study of the comparability of external criteria for hierarchical cluster analysis. Multivar Behav Res 21:441–485CrossRefGoogle Scholar
  48. Milly P, Wetherald RT, Dunne KA, Delworth TL (2002) Increasing risk of great floods in a changing climate. Nature 415:514–517CrossRefGoogle Scholar
  49. Najafi MR, Moradkhani H (2013) A hierarchical Bayesian approach for the analysis of climate change impact on runoff extremes. Hydrol Process 28(26):6292–6308CrossRefGoogle Scholar
  50. Nijssen B, Lettenmaier DP, Liang X, Wetzel SW, Wood EF (1997) Streamflow simulation for continental-scale river basins. Water Resour Res 33(4):711–724CrossRefGoogle Scholar
  51. Nijssen B, O’Donnell GM, Hamlet AF, Lettenmaier DP (2001) Hydrologic sensitivity of global rivers to climate change. Clim Chang 50:143–175CrossRefGoogle Scholar
  52. Ott I, Duethmann D, Liebert J, Berg P, Feldmann H, Ihringer J, Kunstmann H, Merz B, Schaedler G, Wagner S (2013) High-resolution climate change impact analysis on medium-sized river catchments in Germany: an ensemble assessment. J Hydrometeorol 14:1175–1193CrossRefGoogle Scholar
  53. Pierce DW, Cayan DR, Maurer EP, Abatzoglou JT, Hegewisch KC (2015) Improved bias correction techniques for hydrological simulations of climate change. J Hydrometeorol 16(6):2421–2442CrossRefGoogle Scholar
  54. Rabalais NN, Turner RE, Diaz RJ, Justić D (2009) Global change and eutrophication of coastal waters. ICES J Mar Sci 66:1528–1537CrossRefGoogle Scholar
  55. Ramos MH, van Andel SJ, Pappenberger F (2013) Do probabilistic forecasts lead to better decisions? Hydrol Earth Syst Sci 17(6):2219–2232CrossRefGoogle Scholar
  56. Riahi K, Rao S, Krey V, Cho C, Chirkov V, Fischer G, Kindermann G, Nakicenovic N, Rafaj P (2011) RCP 8.5-a scenario of comparatively high greenhouse gas emissions. Clim Chang 109:33–57CrossRefGoogle Scholar
  57. Rousseeuw PJ (1987) Silhouettes: a graphical aid to the interpretation and validation of cluster analysis. J Comput Appl Math 20:53–65CrossRefGoogle Scholar
  58. Sanderson BM, Knutti R, Caldwell P (2015) A representative democracy to reduce interdependency in a multimodel ensemble. J Clim 28:5171–5194CrossRefGoogle Scholar
  59. Schewe J, Heinke J, Gerten D, Haddeland I, Arnell NW, Clark DB, Dankers R, Eisner S, Fekete BM, Colón-González FJ, Gosling SN, Kim H, Liu X, Masaki Y, Portmann FT, Satoh Y, Stacke T, Tang Q, Wada Y, Wisser D, Albrecht T, Frieler K, Piontek F, Warszawski L, Kabat P (2014) Multimodel assessment of water scarcity under climate change. Proc Natl Acad Sci 111(9):3245–3250CrossRefGoogle Scholar
  60. Sriver RL, Forest CE, Keller K (2015) Effects of initial conditions uncertainty on regional climate variability: an analysis using a low-resolution CESM ensemble. Geophys Res Lett 42(13):5468–5476CrossRefGoogle Scholar
  61. Steinschneider S, McCrary R, Mearns LO, Brown C (2015) The effects of climate model similarity on probabilistic climate projections and the implications for local, risk-based adaptation planning. Geophys Res Lett 42:5014–5022CrossRefGoogle Scholar
  62. Tebaldi C, Knutti R (2007) The use of the multi-model ensemble in probabilistic climate projections. Philosophical Transactions of the Royal Society A: Mathematical. Phys Eng Sci 365:2053–2075CrossRefGoogle Scholar
  63. Teng J, Vaze J, Chiew FHS, Wang B, Perraud J-M (2012) Estimating the relative uncertainties sourced from GCMs and hydrological models in modeling climate change impact on runoff. J Hydrometeorol 13:122–139CrossRefGoogle Scholar
  64. Terando A, Keller K, Easterling WE (2012) Probabilistic projections of agro-climate indices in North America. Journal of Geophysical Research Atmospheres:117Google Scholar
  65. Vetter T, Reinhardt J, Flörke M, Griensven A, Hattermann F, Huang S, Koch H, Pechlivanidis IG, Plötner S, Seidou O, Su B, Vervoort RW, Krysanova V (2017) Evaluation of sources of uncertainty in projected hydrological changes under climate change in 12 large-scale river basins. Clim Chang:141Google Scholar
  66. Vicuna S, Maurer EP, Joyce B, Dracup JA, Purkey D (2007) The sensitivity of California water resources to climate change scenarios. J Amer Water Resour Assoc (JAWRA) 43(2):482–498CrossRefGoogle Scholar
  67. Wang H-M, Chen J, Cannon AJ, Xu C-Y, Chen H (2018) Transferability of climate simulation uncertainty to hydrological climate change impacts. Hydrol Earth Syst Sci 22:3739–3759CrossRefGoogle Scholar
  68. Weigel AP, Knutti R, Liniger MA, Appenzeller C (2010) Risks of model weighting in multimodel climate projections. J Clim 23:4175–4191CrossRefGoogle Scholar
  69. Weiland FCS, van Beek LPH, Weerts AH, Bierkens MFP (2012) Extracting information from an ensemble of GCMs to reliably assess future global runoff change. J Hydrol 412-413:66–75CrossRefGoogle Scholar
  70. Whetton P, Macadam I, Bathols J, O’Grady J (2007) Assessment of the use of current climate patterns to evaluate regional enhanced greenhouse response patterns of climate models. Geophys Res Lett:34Google Scholar
  71. Wilby RL, Dessai S (2010) Robust adaptation to climate change. Weather 65 (7):180–185CrossRefGoogle Scholar
  72. Wilcke RAI, Bärring L (2016) Selecting regional climate scenarios for impact modelling studies. Environ Modell Softw 78:191–201CrossRefGoogle Scholar
  73. Willmott CJ, Matsuura K (2005) Advantages of the mean absolute error (MAE) over the root mean square error (RMSE) in assessing average model performance. Clim Res 30:79–82CrossRefGoogle Scholar
  74. Willmott CJ, Matsuura K, Robeson SM (2009) Ambiguities inherent in sums-of-squares-based error statistics. Atmos Environ 43(3):749–752CrossRefGoogle Scholar
  75. Wood AW, Leung LR, Sridhar V, Lettenmaier DP (2004) Hydrologic implications of dynamical and statistical approaches to downscaling climate model outputs. Clim Chang 62:189–216CrossRefGoogle Scholar
  76. Zubler EM, Fischer AM, Fröb F, Liniger MA (2016) Climate change signals of CMIP5 general circulation models over the Alps—impact of model selection. International Journal of Climatology:3088–3104CrossRefGoogle Scholar

Copyright information

© Springer Nature B.V. 2019

Authors and Affiliations

  1. 1.Department of Meteorology and Atmospheric ScienceThe Pennsylvania State UniversityPennsylvaniaUSA
  2. 2.Princeton University Program in Atmospheric and Oceanic SciencesPrincetonUSA

Personalised recommendations