Abstract
Previous studies applied a large variety of performance metrics to evaluate the global and regional climate model simulations; however, there is still a huge debate to justify the rationale for the chosen metrics. The performance of sixty Euro-CORDEX temperature and precipitation simulations was investigated in temporal and spatial approaches through various common metrics over Türkiye. In addition, several mutual information (MI) methods based on the information theory were evaluated as state-of-the-art alternative metrics and compared with the applied common metrics. Based on the average of outputs over the ensemble of the driving models for the annual temperature, the MBE, MASE, MRAE, and NSE are presenting a similar pattern on the rank of the simulations. The MPI with 0.3 on MBE, NCC with 2.7, 2.9, and − 10.4 on MASE, MRAE, and NSE, respectively, and IPSL-LR and NOAA with 0.1 on the modified KGE represented the least errors. The ICHEC with respective 15, 0.1, 1.01, and − 4 for the MBE, MASE, MRAE, and NSE presented the lowest errors for the similar above-mentioned analysis except for the precipitation. The MPI and CNRM with 0.37, 0.37, and 0.08 obtained the highest outcomes on the KNN, mixed and noisy KNN, respectively. We conclude that the ability of mutual information to capture nonlinear relationships is very beneficial. Finally, it is also suggested to undertake these analyses for other hydroclimatic variables for future studies to gain a comprehensive insight into the performance of MI methods.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
Data availability
The datasets used and/or analyzed during the current study are available from the corresponding author upon reasonable request. All the GCM-RCM data used in this work are available at the Federated ESGF‐CoG Nodes (e.g., http://esgf-data.dkrz.de).
References
Altinsoy H, Kurnaz L (2021) A new methodology for climate model selection and application to temperature of Europe. Theor Appl Climatolhttps://doi.org/10.1007/s00704-021-03622-z
BickiciArikan B, Kahya E (2019) Homogeneity revisited: analysis of updated precipitation series in Turkey. Theor Appl Climatol 135:211–220. https://doi.org/10.1007/s00704-018-2368-x
Botchkarev A (2018) Performance metrics (error measures) in machine learning regression, forecasting and prognostics: properties and typology. http://arxiv.org/abs/1809.03006. Accessed Jan 2021
Brunner L, McSweeney C, Ballinger AP et al (2020) Comparing methods to constrain future European climate projections using a consistent framework. J Clim 33:8671–8692. https://doi.org/10.1175/jcli-d-19-0953.1
Chai T, Draxler RR (2014) Root mean square error (RMSE) or mean absolute error (MAE)? -arguments against avoiding RMSE in the literature. Geosci Model Dev 7:1247–1250. https://doi.org/10.5194/gmd-7-1247-2014
Chen M, Feixas M, Viola I, Bardera A, Shen HW, & Sbert M (2016) Information theory tools for visualization (1st ed.). A K Peters/CRC Press. https://doi.org/10.1201/9781315369228
Christensen OB, Kjellström E (2020) Partitioning uncertainty components of mean climate and climate change in a large ensemble of European regional climate model projections. Clim Dyn 54:4293–4308. https://doi.org/10.1007/s00382-020-05229-y
Cinquini L, Crichton D, Mattmann C, et al (2014) The Earth System Grid Federation: an open infrastructure for access to distributed geospatial data. Futur Gener Comput Syst 36:400–417. https://www.sciencedirect.com/science/article/pii/S0167739X13001477. Accessed Jan 2021
Cornes RC, van der Schrier G, van den Besselaar EJM, Jones PD (2018) An ensemble version of the E-OBS temperature and precipitation data sets. J Geophys Res Atmos 123:9391–9409. https://doi.org/10.1029/2017JD028200
Cover TM, Thomas JA (2005) Entropy, relative entropy, and mutual information. In: Elements of Information Theory. John Wiley & Sons, Ltd, 13–55. https://doi.org/10.1002/047174882X.ch2
Di Luca A, de Elía R, Laprise R (2015) Challenges in the quest for added value of regional climate dynamical downscaling. Curr Clim Chang Reports 1:10–21. https://doi.org/10.1007/s40641-015-0003-9
Ellenburg WL, Cruise JF, Singh VP (2018) The role of evapotranspiration in streamflow modeling – an analysis using entropy. J Hydrol 567:290–304. https://doi.org/10.1016/j.jhydrol.2018.09.048
Ertugrul M, Ozel HB, Varol T et al (2019) Investigation of the relationship between burned areas and climate factors in large forest fires in theÇanakkaleregion. Environ Monit Assess 191:737. https://doi.org/10.1007/s10661-019-7946-6
Fernández J, Frías MD, Cabos WD et al (2019) Consistency of climate change projections from multiple global and regional model intercomparison projects. Clim Dyn 52:1139–1156. https://doi.org/10.1007/s00382-018-4181-8
Franses PH (2016) A note on the mean absolute scaled error. Int J Forecast 32:20–22. https://doi.org/10.1016/j.ijforecast.2015.03.008
Gao W, Kannan S, Oh S, Viswanath P (2017) Estimating mutual information for discrete-continuous mixtures. Adv Neural Inf Process Syst 5987–5998
Gibson PB, Waliser DE, Lee H et al (2019) Climate model evaluation in the presence of observational uncertainty: precipitation indices over the contiguous United States. J Hydrometeorol 20:1339–1357. https://doi.org/10.1175/JHM-D-18-0230.1
Giorgi F (2019) Thirty years of regional climate modeling: where are we and where are we going next? J Geophys Res Atmos 124:5696–5723. https://doi.org/10.1029/2018JD030094
Giorgi F, Jones C, Asrar GR (2009) Addressing Climate Information Needs at the Regional Level : the CORDEX Framework 58:175–183
Gupta HV, Kling H, Yilmaz KK, Martinez GF (2009) Decomposition of the mean squared error and NSE performance criteria: implications for improving hydrological modelling. J Hydrol 377:80–91. https://www.sciencedirect.com/science/article/pii/S0022169409004843. Accessed Nov 2021
Güttler I, Stilinović T, Srnec L, et al (2020) Performance of RegCM4 simulations over Croatia and adjacent climate regions. Int J Climatol 1–20. https://doi.org/10.1002/joc.6552
Han Y, Jiao J, Weissman T (2015) Adaptive estimation of Shannon entropy. 2015 IEEE International Symposium on Information Theory (ISIT) 1372–1376
Herger N, Abramowitz G, Knutti R et al (2018) Selecting a climate model subset to optimise key ensemble properties. Earth Syst Dyn 9:135–151. https://doi.org/10.5194/esd-9-135-2018
Hyndman RJ, Koehler AB (2006) Another look at measures of forecast accuracy. Int J Forecast 22:679–688. https://doi.org/10.1016/j.ijforecast.2006.03.001
IPCC (2021) Climate change 2021: the physical science basis. Contribution of Working Group I to the Sixth Assessment Report of the Intergovernmental Panel on Climate Change. In: Masson-Delmotte V, Zhai P, Pirani A, Connors SL, Péan C, Berger S, Caud N, Chen Y, Goldfarb L, Gomis MI, Huang M, Leitzell K, Lonnoy E, Matthews JBR, Maycock TK, Waterfield T, Yelekçi O, Yu R, Zhou B (eds) Cambridge University Press. In Press. https://doi.org/10.1017/9781009157896
Ito R, Shiogama H, Nakaegawa T (2020) Takayabu I (2020) Uncertainties in climate change projections covered by the ISIMIP and CORDEX model subsets from CMIP5. Geosci Model Dev 13:859–872. https://doi.org/10.5194/gmd-13-859-2020
Jacob D, Teichmann C, Sobolowski S et al (2020) Regional climate downscaling over Europe: perspectives from the EURO-CORDEX community. Reg Environ Chang 20:51. https://doi.org/10.1007/s10113-020-01606-9
Jiao J, Gao W, Han Y (2018) The nearest neighbor information estimator is adaptively near minimax rate-optimal. In: Bengio S, Wallach H, Larochelle H et al (eds) Advances in Neural Information Processing Systems. Curran Associates, Inc. https://proceedings.neurips.cc/paper/2018/file/e9fd7c2c6623306db59b6aef5c0d5cac-Paper.pdf/. Accessed Aug 2021
Jr WJG, Giorgi F, Timbal B, et al (2016) WCRP COordinated Regional Downscaling EXperiment ( CORDEX ): a diagnostic MIP for CMIP6. 4087–4095. https://doi.org/10.5194/gmd-9-4087-2016
Jury MW, Prein AF, Truhetz H, Gobiet A (2015) Evaluation of CMIP5 models in the context of dynamical downscaling over Europe. J Clim 28:5575–5582. https://doi.org/10.1175/JCLI-D-14-00430.1
Kling H, Fuchs M, Paulin M (2012) Runoff conditions in the upper Danube basin under an ensemble of climate change scenarioshttps://doi.org/10.1016/j.jhydrol.2012.01.011
Knoben WJM, Freer JE, Woods RA (2019) Technical note: inherent benchmark or not? Comparing Nash-Sutcliffe and Kling-Gupta efficiency scores. Hydrol Earth Syst Sci Discuss 1–7. https://doi.org/10.5194/hess-2019-327
Knutti R (2010) The end of model democracy? Clim Change 102:395–404. https://doi.org/10.1007/s10584-010-9800-2
Kotlarski S, Keuler K, Christensen OB et al (2014) Regional climate modeling on European scales: a joint standard evaluation of the EURO-CORDEX RCM ensemble. Geosci Model Dev 7:1297–1333. https://doi.org/10.5194/gmd-7-1297-2014
Kraskov A, Stögbauer H, Grassberger P (2004) Estimating mutual information. Phys Rev E 69:66138. https://doi.org/10.1103/PhysRevE.69.066138
Kvalseth TO (1987) Entropy and correlation: some comments. IEEE Trans Syst Man Cybern 17:517–519. https://doi.org/10.1109/TSMC.1987.4309069
Leonenko N, Pronzato L, Savani V (2006) Estimation of entropies and divergences via nearest neighbors. In: ProbaStat 2006. Smolenice, Slovakia, pp 265–273. https://hal.archives-ouvertes.fr/hal-00322783. Accessed Aug 2021
McSweeney CF, Jones RG, Lee RW, Rowell DP (2015) Selecting CMIP5 GCMs for downscaling over multiple regions. Clim Dyn 44:3237–3260. https://doi.org/10.1007/s00382-014-2418-8
Meehl GA, Covey C, Delworth T et al (2007) The WCRP CMIP3 multimodel dataset: a new era in climatic change research. Bull Am Meteorol Soc 88:1383–1394. https://doi.org/10.1175/BAMS-88-9-1383
Merrifield AL, Brunner L, Lorenz R et al (2020) An investigation of weighting schemes suitable for incorporating large ensembles into multi-model ensembles. Earth Syst Dyn 11:807–834. https://doi.org/10.5194/esd-11-807-2020
Nash JE, Sutcliffe J V. (1970) River flow forecasting through conceptual models part I - a discussion of principles. J Hydrol 10:282–290. http://www.sciencedirect.com/science/article/pii/0022169470902556. Accessed June 2018
Nikulin G, Jones C, Giorgi F et al (2012) Precipitation climatology in an ensemble of CORDEX-Africa regional climate simulations. J Clim 25:6057–6078. https://doi.org/10.1175/JCLI-D-11-00375.1
NOAA (2020) National centers for environmental information. state of the climate: global climate report for august 2020. Published online September 2020, retrieved on October 8, 2020. https://www.ncdc.noaa.gov/sotc/global/202008. Accessed Oct 2020
ONeill BC, Tebaldi C, van Vuuren D, et al (2016) The Scenario Model Intercomparison Project (ScenarioMIP) for CMIP6. Geosci Model Dev Discuss 1–35. https://doi.org/10.5194/gmd-2016-84
Paninski L (2003) Estimation of entropy and mutual information. Neural Comput 15:1191–1253. https://doi.org/10.1162/089976603321780272
Paninski L, Yajima M (2008) Undersmoothed kernel entropy estimators. IEEE Trans Inf Theory 54:4384–4388. https://doi.org/10.1109/TIT.2008.928251
Papalexiou SM, Rajulapati CR, Clark MP, Lehner F (2020) Robustness of CMIP6 historical global mean temperature simulations: trends, long-term persistence, autocorrelation, and distributional shape. Earth’s Futur 8:e2020EF001667. https://agupubs.onlinelibrary.wiley.com/doi/abs/10.1029/2020EF001667
Prein AF, Gobiet A, Truhetz H et al (2016) Precipitation in the EURO-CORDEX 0.11∘ and 0.44∘ simulations: high resolution, high benefits? Clim Dyn 46:383–412. https://doi.org/10.1007/s00382-015-2589-y
Russo E, Kirchner I, Pfahl S et al (2019) Sensitivity studies with the regional climate model COSMO-CLM 5.0 over the CORDEX Central Asia Domain. Geosci Model Dev 12:5229–5249. https://doi.org/10.5194/gmd-12-5229-2019
Schaefli B, Gupta HV (2007) Do Nash values have value? Hydrol Process 21:2075–2080. https://doi.org/10.1002/hyp.6825
Shannon CE (1948) The mathematical theory of communications I and II Bell Syst. Tech J 27:379–423
Silver EA, Pyke DF, Thomas DJ (2016) Chapter 2 Frameworks for Inventory Management and Production Planning and Scheduling. Invent Prod Manag Supply Chain 23–72. https://doi.org/10.1201/9781315374406-3
Singh PV (2011) Hydrologic synthesis using entropy theory: review. J Hydrol Eng 16:421–433. https://doi.org/10.1061/(ASCE)HE.1943-5584.0000332
Smiatek G, Kunstmann H, Senatore A (2016) EURO-CORDEX regional climate model analysis for the Greater Alpine Region: performance and expected future change. J Geophys Res 121:7710–7728. https://doi.org/10.1002/2015JD024727
Sørland SL, Schär C, Lüthi D, Kjellström E (2018) Bias patterns and climate change signals in GCM-RCM model chains. Environ Res Lett 13:074017. https://iopscience.iop.org/article/10.1088/1748-9326/aacc77
Taguela TN, Vondou DA, Moufouma-Okia W et al (2020) CORDEX multi-RCM hindcast over Central Africa: evaluation within observational uncertainty. J Geophys Res Atmos 125:1–21. https://doi.org/10.1029/2019JD031607
Taylor KE, Stouffer RJ, Meehl GA (2012) An overview of CMIP5 and the experiment design. Bull Am Meteorol Soc 93:485–498. https://doi.org/10.1175/BAMS-D-11-00094.1
TSMS: Turkish State Meteorological Service Annual Precipitation Report, In Turkish 2022. https://mgm.gov.tr/FILES/iklim/yillikiklim/2022-iklim-raporu.pdf. Accessed April 2023
Valiant G, Valiant P (2011) Estimating the unseen: an n/Log(n)-sample estimator for entropy and support size, shown optimal via new CLTs. In: Proceedings of the Forty-Third Annual ACM Symposium on Theory of Computing. Association for Computing Machinery, New York, NY, USA, 685–694 https://doi.org/10.1145/1993636.1993727
van Vuuren DP, Edmonds J, Kainuma M et al (2011) The representative concentration pathways: an overview. Clim Change 109:5–31. https://doi.org/10.1007/s10584-011-0148-z
Vautard R, Kadygrov N, Iles C et al (2020) Evaluation of the large EURO-CORDEX regional climate model ensemble. J Geophys Res Atmos 126(17):e2019JD032344. https://doi.org/10.1029/2019JD032344
WMO (2022): WMO Provisional State of the Global Climate 2022. https://library.wmo.int/doc_num.php?explnum_id=11359. Accessed April 2023
Wu Y, Yang P (2016) Minimax rates of entropy estimation on large alphabets via best polynomial approximation. IEEE Trans Inf Theory 62:3702–3720. https://doi.org/10.1109/TIT.2016.2548468
Acknowledgements
The first author would like to thank the Turkish State Meteorological Service (TSMS) for providing the observational precipitation and temperature data. The authors would like to extend their sincere appreciation to the anonymous reviewers for their invaluable feedback and constructive comments.
Author information
Authors and Affiliations
Contributions
Conceptualization: Saeed Vazifehkhah, Ercan Kahya, Amir Delju.
Investigation: Saeed Vazifehkhah, Ercan Kahya, Weihao Gao, Amir Delju.
Data curation: Saeed Vazifehkhah.
Formal analysis: Saeed Vazifehkhah, Weihao Gao.
Visualization: Saeed Vazifehkhah.
Writing—original draft: Saeed Vazifehkhah.
Writing—review and edit: Saeed Vazifehkhah, Weihao Gao, Ercan Kahya, and Amir Delju.
Corresponding author
Ethics declarations
Competing interests
The authors declare no competing interests.
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Supplementary Information
Below is the link to the electronic supplementary material.
Rights and permissions
Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
About this article
Cite this article
Vazifehkhah, S., Kahya, E., Gao, W. et al. Beyond the ordinary metrics on the evaluation of historical Euro-CORDEX simulations over Türkiye: the mutual information approach. Theor Appl Climatol 153, 829–851 (2023). https://doi.org/10.1007/s00704-023-04492-3
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s00704-023-04492-3