Estimation of the Continuous Ranked Probability Score with Limited Information and Applications to Ensemble Weather Forecasts
 2.3k Downloads
 4 Citations
Abstract
The continuous ranked probability score (CRPS) is a much used measure of performance for probabilistic forecasts of a scalar observation. It is a quadratic measure of the difference between the forecast cumulative distribution function (CDF) and the empirical CDF of the observation. Analytic formulations of the CRPS can be derived for most classical parametric distributions, and be used to assess the efficiency of different CRPS estimators. When the true forecast CDF is not fully known, but represented as an ensemble of values, the CRPS is estimated with some error. Thus, using the CRPS to compare parametric probabilistic forecasts with ensemble forecasts may be misleading due to the unknown error of the estimated CRPS for the ensemble. With simulated data, the impact of the type of the verified ensemble (a random sample or a set of quantiles) on the CRPS estimation is studied. Based on these simulations, recommendations are issued to choose the most accurate CRPS estimator according to the type of ensemble. The interest of these recommendations is illustrated with real ensemble weather forecasts. Also, relationships between several estimators of the CRPS are demonstrated and used to explain the differences of accuracy between the estimators.
Keywords
Continuous ranked probability score Estimation Forecast comparison Ensemble weather forecasts1 Introduction
List of distributions whose closedform CRPS exists and were used in this study
Distribution  Original reference 

Beta: \(Y\sim Beta(\alpha ,\beta )\)  Taillardat et al. (2016) 
Gamma: \(Y\sim Gamma(\alpha ,\beta )\)  Möller and Scheuerer (2013) 
Gaussian mixture: \(Y\sim \sum _{i=1}^p \omega _i {\mathcal {N}}(\mu _i,\sigma _i)\), with  Grimit et al. (2006) 
\(\sum _{i=1}^p\omega _i=1, \quad \omega _{i=1,\ldots ,p} > 0\)  
Generalized extreme value: \(Y\sim GEV(\mu , \sigma , \xi )\)  Friederichs and Thorarinsdottir (2012) 
Generalized Pareto: \(Y\sim GPD(\mu , \sigma , \xi )\)  Friederichs and Thorarinsdottir (2012) 
Lognormal: \(\ln (Y) \sim {\mathcal {N}}(\mu ,\sigma )\)  Baran and Lerch (2015) 
Normal: \(Y\sim {\mathcal {N}}(\mu , \sigma )\)  Gneiting et al. (2005) 
Squareroot truncated normal: \(\sqrt{Y}\sim {\mathcal {N}}^0(\mu , \sigma )\)  Hemri et al. (2014) 
Truncated normal: \(Y\sim {\mathcal {N}}^0(\mu , \sigma )\)  Thorarinsdottir and Gneiting (2010) 
Usually, the instantaneous CRPS is averaged in space and/or time over several pairs of forecast/observation. Candille (2003) and Ferro et al. (2008) showed that when the ensemble is a random sample from F, the usual estimator of the instantaneous CRPS based on Eq. (INT), introduced later, is biased: its expectation over an infinite number of forecast/observation pairs does not give the right theoretical value. This bias stems from the limited information about F contained in an ensemble with finite size M. Several solutions have been proposed to remove this bias. Ferro (2014) introduced the notion of fair score and a formula to correct the bias in the estimation of the averaged CRPS. Müller et al. (2005) proposed two solutions to the same problem of biased estimation of the ranked probability score (RPS), the version of the CRPS for ordinal random variables. Adapted to the CRPS, their first solution would be to use an absolute value instead of a square inside the integral in Eq. (INT). As demonstrated in Appendix A, this score for an ensemble is minimized if all the members \(x_i\) equal the median of F, which is obviously not the purpose of an ensemble. Their second solution is to compute the RPS skill score against some ensemble of size M whose RPS is estimated by bootstrapping past observations. Although interesting, this solution does not allow assessing the absolute performance of the ensemble, but only the performance relative to this bootstrapped ensemble.
This study aims at improving heuristically the estimation of the average CRPS of a forecast CDF under limited information. The information is limited in two ways: (i) the CDF is known only through an ensemble as defined above, and (ii) the average CRPS is computed over a finite number of forecast/observation pairs. The problem is not to estimate the unknown forecast CDF F, but to estimate the CRPS of F under limited information about F. To improve the estimation with this limited information, the usual strategy is to correct the empirical mean score, as in Ferro (2014) or Müller et al. (2005). Here the approach is to improve the estimation of each term of the average, that is, the estimation of the instantaneous CRPS \(\hbox {crps}(F,y)\).
The rest of this paper is organized as follows. Section 2 reviews several estimators of the instantaneous CRPS proposed in the literature and demonstrates relationships among them. In particular, it is shown that the four proposed estimators reduce to two only. In Sect. 3, synthetic data are used to study the variations in accuracy of these two CRPS estimators, with the size M of the ensemble and the way this ensemble is built. These simulations lead to recommendations on the best estimation of the CRPS. Section 4 illustrates issues in CRPS estimation with two real meteorological data sets. Improvements in the inference obtained by following the recommendations from Sect. 3 are shown on these data. Section 5 gives a summary of the recommendations to get an accurate estimation of the instantaneous CRPS, concludes and discusses the results.
2 Review of Available Estimators of the CRPS
3 Study with Simulated Data
The accuracy of the two instantaneous CRPS estimators presented above, \(\widehat{\hbox {crps}}_{\mathrm{PWM}}(M, y)\) and \(\widehat{\hbox {crps}}_{\mathrm{INT}}(M, y)\), is studied with synthetic forecast/observation pairs. The forecast CDF F is chosen such that the theoretical CRPS \(\hbox {crps}(F,y)\) can be exactly computed with a closedform expression (see Table 1 for a list of such distributions). To mimic actual situations when F is not fully known, two types of ensembles are built from this forecast CDF. The two types of ensembles successively used in the remaining of this section are random ensembles and ensembles of quantiles, defined later. The estimators are then computed and compared to the theoretical value.
3.1 CRPS Estimation with a Random Ensemble
3.1.1 Methodology
3.1.2 Results
The results are presented for a standard normal forecast CDF F. For the sake of simplicity the CDF of the observation is also standard normal (\(G=F\)).
Since the ensemble is random, the estimated CRPS is also a random variable that depends on the observation y and the members \(x_{i=1,\ldots ,M}\). In order to study the variability of the estimated CRPS with the ensemble only, the observation is first held constant (with a value of − 0.0841427, for each n in Protocol 1), while \(N=1000\) ensembles of M members are drawn from F. The impact of M on the accuracy of the estimated CRPS is assessed by observing Protocol 1 with different ensemble sizes M.
These behaviours for the instantaneous and the averaged estimates remain true for every distribution listed in Table 1, every parameter value and even if the G and F are different (not shown).
The added value of these simulations to the results of Ferro (2014) is to show the behaviour of \(\widehat{\hbox {crps}}_{\mathrm{PWM}}\) for small ensemble sizes M and finite numbers of forecast/observation pairs. The poor scaling of this estimator’s variability with the ensemble size has been empirically shown, which had never been done, to the best of our knowledge. Finding a formula for the variability of \(\widehat{\hbox {crps}}_{\mathrm{PWM}}\) would be interesting to quantify the estimation uncertainty for practical purposes. Theoretical error bounds have been demonstrated but are not usable in practice since they require to know the forecast distribution (not shown).
The conclusion of these simulations is that, for a random ensemble, the estimation of the instantaneous CRPS is not very accurate whatever estimator is used, but the averaged CRPS can be estimated with a good accuracy. The unbiasedness of \(\widehat{\hbox {crps}}_{\mathrm{PWM}}\) for random ensembles stems from the use of estimators that are unbiased for independent samples from the underlying distribution F. In practice, if one seeks to estimate the potential performance of an ensemble with an infinite number of members, one should use the PWM estimator of the CRPS. The integral estimator of the CRPS assesses the global performance of the actual ensemble, and should be used for actual performance verification.
3.2 CRPS Estimation with an Ensemble of Quantiles
3.2.1 Methodology
An ensemble of M quantiles of orders \(\tau _{i=1,\ldots ,M}\in [0;1]\) is a set of M values \(x_{i=1,\ldots ,M}\) such that: \(x_i=F^{1}(\tau _i) \,\forall i \in \{1,\ldots ,M\}\). Contrasting with a random ensemble, the orders \(\tau _i\) associated to the members \(x_i\) are known.

regular ensemble (reg): it is the ensemble of the M quantiles of orders \(\tau _i\), with \(\tau _i \in \{\frac{1}{M}, \frac{2}{M},\ldots ,\frac{M1}{M},\frac{M0.1}{M}\}\) of F. The last order is not 1 to prevent infinite values.

optimal ensemble (opt): it is the set of M quantiles of orders \(\tau _i \in \{\frac{0.5}{M},\frac{1.5}{M},\ldots ,\frac{M0.5}{M}\}\) of F. This ensemble is called “optimal” because Bröcker (2012) showed that this set of quantiles minimizes the expectation of the CRPS of an ensemble over an infinite number of forecast/observation pairs, when using Eq. (eINT).
3.2.2 Results
These conclusions hold for all the tried distributions and the set of parameters values for each distribution (not shown). As for the poor performance of \(\widehat{\hbox {crps}}_{\mathrm{PWM}}\) with an ensemble of quantiles, let us recall that \(\widehat{\hbox {crps}}_{\mathrm{PWM}}\) is a sum of terms that are unbiased estimators of their population counterpart when computed with a random sample, which is not the case of an ensemble of quantiles. The computation of \(\widehat{\hbox {crps}}_{\mathrm{INT}}\) uses the approximation of the forecast distribution as a stepwise CDF, with a fixed stairstep height \(\frac{1}{M}\). The difference in estimation accuracy with the type of quantiles comes from the position of the stair steps. With regular quantiles, the stepwise CDF is always located below the forecast CDF. With optimal quantiles, the associated quantiles are shifted leftward, making the stair steps sometimes above F and sometimes below. This better approximates the forecast CDF F than with regular quantiles, thus improves the estimation of the CRPS.
3.2.3 Influence of Ties in an Ensemble of Quantiles
A way to address this issue of equal quantiles is to remove the ties by interpolation. The first considered case is when the implementation of the quantile regression method do not propose to know the available quantiles. Protocol 3 is modified as follows at lines 3 and 4: after computing the quantiles with ties, linear interpolation is done between unique values to recover the number of required regular or optimal quantiles, as explained in Fig. 6. As shown on the right side of Fig. 8, this interpolation results in a better estimation accuracy, even though the curves are less smooth than when all orders are available (compare with Fig. 5). The best CRPS estimation is now obtained with \(\widehat{\hbox {crps}}_{\mathrm{INT}}\) and regular quantiles, with at least \(M=30\) regular quantiles to get a sufficient accuracy. This behavior barely depends on the chosen distribution and parameter value, but requiring 100 regular quantiles seems to be the minimal number to get satisfactory accuracy, whatever the forecast distribution F is used (not shown). If the available quantiles and orders can be produced by the implementation of the quantile regression method, similar linear interpolation can be done relatively to the associated points, that is, the black dots in Fig. 6. Figure 9 shows that this linear interpolation nearly fully reproduces the good accuracy obtained when all orders are available. The best estimation strategy is again to use \(\widehat{\hbox {crps}}_{\mathrm{INT}}\) with optimal quantiles, albeit with a slightly worst accuracy than the one reached without ties.
The influence of the number of available orders \(N_\tau \) and the kind of postprocessing on \(\widehat{\hbox {crps}}_{\mathrm{INT}}\) is crucial as shown in Fig. 10. If the number of available quantiles is too low, no matter the postprocessing of the quantile ensemble, the estimated CRPS will not converge to the true value due to insufficient information about F. The number of available quantiles necessary to achieve a good accuracy depends on the complexity of the forecast distribution: a gaussian mixture with many different modes requires more available quantiles to be accurately described (not shown here).
Summary of recommendations to estimate the CRPS
Type of ensemble  Condition  Recommendation 

Random  The purpose is to assess the performance of an infinite ensemble  Use average \(\widehat{\hbox {crps}}_{\mathrm{PWM}}\) 
The purpose is to assess the performance of the actual ensemble  Use average \(\widehat{\hbox {crps}}_{\mathrm{INT}}\)  
Quantiles  All orders available  Use average \(\widehat{\hbox {crps}}_{\mathrm{INT}}\) with optimal quantiles 
\(N_\tau \lesssim 30\)  Use average \(\widehat{\hbox {crps}}_{\mathrm{INT}}\) with care  
\(N_\tau \gtrsim 30\) and available quantiles unknown  Use average \(\widehat{\hbox {crps}}_{\mathrm{INT}}\) with linearly interpolated regular quantiles between unique quantiles  
\(N_\tau \gtrsim 30\) and available quantiles known  Use average \(\widehat{\hbox {crps}}_{\mathrm{INT}}\) with linearly interpolated optimal quantiles between available quantiles 
4 Real Data Examples
With two real data sets, issues resulting from the uncertainty in the estimation of the instantaneous CRPS are illustrated. The practical benefits of the recommendations listed in Table 2 are highlighted.
4.1 Raw and Calibrated Ensemble Forecast Data Sets
The first forecast data set consists in four NWP ensembles from the TIGGE project (Bougeault et al. 2010). Tenmeter high wind speed forecasts have been extracted from four operational ensemble models issued by meteorological forecast services: the US National Centers for Environmental Prediction (NCEP), the Canadian Meteorological Center (CMC), the European Center for mediumrange weather forecasts (ECMWF) and MétéoFrance (MF). Those ensembles have respectively 21, 21, 51 and 35 members. The study domain is France with a grid size of 0.5\(^\circ \) (about 50 km), for a total of 267 grid points. Available forecast leadtimes are every 6 h. The period goes from 2011 to 2014.
The second forecast data set is composed of two versions of each ensemble calibrated with statistical postprocessing methods. In order to improve the forecast performance, each ensemble has been postprocessed thanks to two statistical methods: nonhomogeneous regression [NR, Gneiting et al. (2005)] and quantile regression forests [QRF, Meinshausen (2006)]. In NR, the forecast probability distribution F is supposed to be some known distribution: here the square root of forecast wind speed follows a truncated normal distribution whose mean and variance depend on the ensemble forecast. This is similar to the work of Hemri et al. (2014), who also gives the closed form expression of the instantaneous CRPS for this case. QRF is nonparametric and yields a set of quantiles \(x_i\) with chosen orders \(\tau _i\). This study uses a simplified version of the model proposed in Taillardat et al. (2016). Since QRF is nonparametric, the CRPS has to be estimated with limited information. Furthermore, QRF cannot yield every order and may lead to many ties among predicted quantiles, as seen in Fig. 7. To the best of our knowledge, no implementation of QRF in R allows knowing the available quantiles. Postprocessing was done separately for each of the 267 grid points, each ensemble and each lead time. The regression was trained with crossvalidation: 3 years were used as training data, the fourth one being used as test data. The four possible combinations of three training years and one test year were tested. The raw ensembles can be seen as random ensembles whereas the ensembles calibrated with QRF are ensembles of quantiles as defined above. The observation comes from a wind speed analysis made at MétéoFrance, presented in Zamo et al. (2016).
4.2 Issues Estimating the CRPS of Real Data
4.3 Issues on the Choice Between QRF and NR
5 Conclusions
A review of four estimators of the instantaneous CRPS when the forecast CDF is known through a set of values have been done. Among these four estimators proposed in the literature, only two, called the integral estimator and the probability weighted moment estimator, are not equal. Furthermore, a relationship between these two estimators has been demonstrated and generalizes to the instantaneous CRPS of any ensemble, a relationship established by Ferro et al. (2008) for the average CRPS of a random ensemble. With simulated data, the accuracy of the two estimators has been studied, when the forecast CDF is known with a limited information and the number of forecast/observation pairs is finite. The study leads to recommendations on the best CRPS estimator depending on the type of ensemble, whether random or a set of quantiles. For a random ensemble, the best estimator of the CRPS is the PWM estimator \(\widehat{\hbox {crps}}_{\mathrm{PWM}}\) if one wants to assess the performance of the ensemble of infinite size, whereas the integral estimator \(\widehat{\hbox {crps}}_{\mathrm{INT}}\) must be used to assess the performance of the ensemble with its current size. For an ensemble of quantiles, ties introduced by quantile regression methods strongly affect the estimation accuracy, and removing these ties by an interpolation step is paramount to allow a good estimation accuracy. If the number of available quantiles is too low (say, \(N_{\tau } \le 30\)) all the studied estimators exhibit a strong bias. But if the number of available quantiles is larger, the best estimation is obtained by computing the integral estimator \(\widehat{\hbox {crps}}_{\mathrm{INT}}\) with linearly interpolated quantiles, between the available quantiles if they are known or between the unique quantiles otherwise.
The established relationships between the estimators proposed in the literature have been linked to previous results. These relationships also explain why an estimator is more accurate for one type of ensemble and not for the other. The PWM estimator performs better on random ensembles because it is based on estimators that are unbiased for independent samples from the true underlying distribution. On the other hand, the integral estimator gives a good estimate when computed with optimal quantiles. This is because regular weights are associated to the members in the estimator formula but, when using optimal quantiles, the associated quantiles are shifted to better approximate the underlying forecast CDF.
The important consequences on the choice of method of estimation of the CRPS has also been illustrated on real meteorological data with raw ensembles and calibrated ensembles. As an example, the comparison of several calibrated ensembles may be mislead by a poor estimate of the average CRPS of ensembles of quantiles.
Notes
Acknowledgements
Both authors are grateful to Pr. Liliane Bel (AgroParisTech, France) and Pr. Tilmann Gneiting (Heidelberg Institute for Theoretical Studies, Germany) for their useful comments on this paper. Part of the work of Philippe Naveau has been supported by the ANRDADA, LEFEINSUMultirisk, AMERISKA, A2C2, CHAVANA and Extremoscope projects. Part of the work was done when Philippe Naveau was visiting the IMAGENCAR group in Boulder, Colorado, USA.
References
 Baran S, Lerch S (2015) Lognormal distribution based ensemble model output statistics models for probabilistic windspeed forecasting. Q J R Meteorol Soc 141:2289–2299CrossRefGoogle Scholar
 Bougeault P, Toth Z, Bishop C, Brown B, Burridge D, Chen DH, Ebert B, Fuentes M, Hamill TM, Mylne K et al (2010) The THORPEX interactive grand global ensemble. Bull Am Meteorol Soc 91(8):1059CrossRefGoogle Scholar
 Brier G (1950) Verification of forecasts expressed in terms of probability. Mon Weather Rev 78(1):1–3CrossRefGoogle Scholar
 Bröcker J (2012) Evaluating raw ensembles with the continuous ranked probability score. Q J R Meteorol Soc 138(667):1611–1617CrossRefGoogle Scholar
 Candille G (2003) Validation des systèmes de prévisions météorologiques probabilistes. PhD thesis, Paris 6Google Scholar
 Emery X, Lantuéjoul C (2006) Tbsim: a computer program for conditional simulation of threedimensional gaussian random fields via the turning bands method. Comput Geosci 32(10):1615–1628CrossRefGoogle Scholar
 Ferro CAT (2014) Fair scores for ensemble forecasts. Q J R Meteorol Soc 140(683):1917–1923CrossRefGoogle Scholar
 Ferro CA, Richardson DS, Weigel AP (2008) On the effect of ensemble size on the discrete and continuous ranked probability scores. Meteorol Appl 15(1):19–24CrossRefGoogle Scholar
 Friederichs P, Thorarinsdottir TL (2012) Forecast verification for extreme value distributions with an application to probabilistic peak wind prediction. Environmetrics 23(7):579–594CrossRefGoogle Scholar
 Furrer R, Naveau P (2007) Probability weighted moments properties for small samples. Stat Probab Lett 77(2):190–195CrossRefGoogle Scholar
 Gneiting T (2011) Quantiles as optimal point forecasts. Int J Forecast 27(2):197–207CrossRefGoogle Scholar
 Gneiting T, Raftery A (2007) Strictly proper scoring rules, prediction, and estimation. J Am Stat Assoc 102(477):359–378CrossRefGoogle Scholar
 Gneiting T, Raftery A, Westveld A III, Goldman T (2005) Calibrated probabilistic forecasting using ensemble model output statistics and minimum CRPS estimation. Mon Weather Rev 133(5):1098–1118CrossRefGoogle Scholar
 Greenwood JA, Landwehr JM, Matalas NC, Wallis JR (1979) Probability weighted moments: definition and relation to parameters of several distributions expressable in inverse form. Water Resour Res 15(5):1049–1054CrossRefGoogle Scholar
 Grimit E, Gneiting T, Berrocal V, Johnson N (2006) The continuous ranked probability score for circular variables and its application to mesoscale forecast ensemble verification. Q J R Meteorol Soc 132(621C):2925–2942CrossRefGoogle Scholar
 Hemri S, Scheuerer M, Pappenberger F, Bogner K, Haiden T (2014) Trends in the predictive performance of raw ensemble weather forecasts. Geophys Res Lett 41(24):9197–9205CrossRefGoogle Scholar
 Hersbach H (2000) Decomposition of the continuous ranked probability score for ensemble prediction systems. Weather Forecast 15(5):559–570CrossRefGoogle Scholar
 Hosking J (1990) Lmoments: analysis and estimation of distributions using linear combinations of order statistics. J Roy Stat Soc: Ser B (Methodol) 52(1):105–124Google Scholar
 Jolliffe I, Stephenson D (2011) Forecast verification: a Practioner’s guide in atmospheric science, 2nd edn. Wiley, LondonCrossRefGoogle Scholar
 Koenker R (2005) Quantile regression, vol 38. Cambridge University Press, CambridgeCrossRefGoogle Scholar
 Matheson JE, Winkler RL (1976) Scoring rules for continuous probability distributions. Manag Sci 22(10):1087–1096CrossRefGoogle Scholar
 Meinshausen N (2006) Quantile regression forests. J Mach Learn Res 7:983–999Google Scholar
 Möller D, Scheuerer M (2013) Postprocessing of ensemble forecasts for wind speed over Germany. PhD thesis, Diploma thesis, Faculty of Mathematics and Computer Science, Heidelberg University. http://www.rzuser.uniheidelberg.de/~kd4/files/Moeller2013.pdf
 Müller W, Appenzeller C, DoblasReyes F, Liniger M (2005) A debiased ranked probability skill score to evaluate probabilistic ensemble forecasts with small ensemble sizes. J Clim 18(10):1513–1523CrossRefGoogle Scholar
 Pirot G, Straubhaar J, Renard P (2014) Simulation of braided river elevation model time series with multiplepoint statistics. Geomorphology 214:148–156CrossRefGoogle Scholar
 Rasmussen PF (2001) Generalized probability weighted moments: application to the generalized pareto distribution. Water Resour Res 37(6):1745–1751CrossRefGoogle Scholar
 Székely GJ, Rizzo ML (2013) Energy statistics: a class of statistics based on distances. J Stat Plan Inference 143(8):1249–1272CrossRefGoogle Scholar
 Taillardat M, Mestre O, Zamo M, Naveau P (2016) Calibrated ensemble forecasts using quantile regression forests and ensemble model output statistics. Mon Weather Rev 144(6):2375–2393CrossRefGoogle Scholar
 Takeuchi I, Le Q, Sears T, Smola A (2006) Nonparametric quantile estimation. J Mach Learn Res 7:1231–1264Google Scholar
 Thorarinsdottir TL, Gneiting T (2010) Probabilistic forecasts of wind speed: ensemble model output statistics by using heteroscedastic censored regression. J R Stat Soc: Ser A (Stat Soc) 173(2):371–388CrossRefGoogle Scholar
 Wang Q (1996) Direct sample estimators of L moments. Water Resour Res 32(12):3617–3619CrossRefGoogle Scholar
 Weijs SV, Van Nooijen R, Van De Giesen N (2010) Kullback–Leibler divergence as a forecast skill score with classic reliability–resolution–uncertainty decomposition. Mon Weather Rev 138(9):3387–3399CrossRefGoogle Scholar
 White H (1992) Nonparametric estimation of conditional quantiles using neural networks. In: Page C, LePage R (eds) Computing science and statistics. Springer, New York, NY, pp 190–199CrossRefGoogle Scholar
 Winkler R, Muñoz J, Cervera J, Bernardo J, Blattenberger G, Kadane J, Lindley D, Murphy A, Oliver R, RíosInsua D (1996) Scoring rules and the evaluation of probabilities. Test 5(1):1–60CrossRefGoogle Scholar
 Yin G, Mariethoz G, McCabe MF (2016) Gapfilling of landsat 7 imagery using the direct sampling method. Remote Sens 9(1):12CrossRefGoogle Scholar
 Zamo M, Bel L, Mestre O, Stein J (2016) Improved gridded windspeed forecasts by statistical postprocessing of numerical models with block regression. Weather Forecast 31(6):1929–1945CrossRefGoogle Scholar
Copyright information
Open AccessThis article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made.