Using maximum likelihood to derive various distance-based goodness-of-fit indicators for hydrologic modeling assessment

Abstract

Currently used goodness-of-fit (GOF) indicators (i.e. efficiency criteria) are largely empirical and different GOF indicators emphasize different aspects of model performance; a thorough assessment of model skill may require the use of robust skill matrices. In this study, based on the maximum likelihood method, a statistical measure termed BC-GED error model is proposed, which firstly uses the Box–Cox (BC) transformation method to remove the heteroscedasticity of model residuals, and then employs the generalized error distribution (GED) with zero-mean to fit the distribution of model residuals after BC transformation. Various distance-based GOF indicators can be explicitly expressed by the BC-GED error model for different values of the BC transformation parameter λ and GED kurtosis coefficient β. Our study proves that (1) the shape of error distribution implied in the GOF indicators affects the model performance on high or low flow discharges because large error-power (β) value can cause low probability of large residuals and small β value will lead to high probability of zero value; (2) the mean absolute error could balance consideration of low and high flow value as its assumed error distribution (i.e. Laplace distribution, where β = 1) is the turning point of GED derivative at zero value. The results of a study performed in the Baocun watershed via comparison of the SWAT model-calibration results using six distance-based GOF indicators show that even though the formal BC-GED is theoretically reasonable, the calibrated model parameters do not always correspond to high performance of model-simulation results because of imperfection of the hydrologic model. However, the derived distance-based GOF indicators using the maximum likelihood method offer an easy way of choosing GOF indicators for different study purposes and developing multi-objective calibration strategies.

Appendices

Appendix 1: Maximum likelihood of BC-GED model

The derivative of logarithmic likelihood function at maximum point should equal to zero, so from the likelihood function Eq. (6), we can derive:

$$ \frac{{\partial \left( {l\left( {\theta \left| {obs} \right.} \right)} \right)}}{\partial \sigma } = - \frac{n}{\sigma } + \frac{{c(\beta )^{\beta } \beta }}{{\sigma^{\beta + 1} }}\sum\limits_{1}^{n} {\left| {e_{i} } \right|^{\beta } = 0} $$

(16)

By solving Eq. (16), we get:

$$ \sigma = c(\beta )\sqrt[\beta ]{\beta }\sqrt[\beta ]{{\frac{{\sum\nolimits_{1}^{n} {\left| {e_{i} } \right|^{\beta } } }}{n}}} = \sqrt[\beta ]{\beta }\sqrt {\frac{\Gamma [3/\beta ]}{\Gamma [1/\beta ]}} \sqrt[\beta ]{{\frac{{\sum\nolimits_{1}^{n} {\left| {e_{i} } \right|^{\beta } } }}{n}}} $$

(17)

By substituting Eqs. (17) into (6), we get:

$$ l\left( {\theta \left| {obs} \right.} \right)_{max} = - n\ln \frac{{\sigma \sqrt[\beta ]{\varepsilon }}}{\omega (\beta )} = - n\ln \left[ {\frac{2\Gamma [1/\beta ]}{\beta }\sqrt[\beta ]{{\varepsilon \beta \frac{{\sum\nolimits_{1}^{n} {\left| {e_{i} } \right|^{\beta } } }}{n}}}} \right] $$

(18)

where ε is the base of the natural logarithm (ε ≈ 2.718).

Equation (17) is a method to estimate the standard deviation (σ) in Eq. (5). In the following sections, we will separately prove the unbiasedness, consistency and efficiency of the estimation method (Lehmann and Casella 1998).

• (1) Unbiasedness

The mathematic expectation of Eq. (17) is:

$$ E(\sigma ) = \sqrt[\beta ]{\beta }\sqrt {\frac{\Gamma [3/\beta ]}{\Gamma [1/\beta ]}} \sqrt[\beta ]{{E\left( {\frac{{\sum\nolimits_{1}^{n} {\left| {e_{i} } \right|^{\beta } } }}{n}} \right)}} = \sqrt[\beta ]{\beta }\sqrt {\frac{\Gamma [3/\beta ]}{\Gamma [1/\beta ]}} \sqrt[\beta ]{{E\left( {\left| e \right|^{\beta } } \right)}} $$

(19)

If e _i follows the generalized error distribution (GED) with zero-mean, the Eq. (19) can be rewritten as:

$$ \begin{aligned} E(\sigma ) & = \sqrt[\beta ]{\beta }\sqrt {\frac{\Gamma [3/\beta ]}{\Gamma [1/\beta ]}} \sqrt[\beta ]{{\int_{ - \infty }^{ + \infty } {\left| x \right|^{\beta } } \frac{\omega (\beta )}{\sigma }exp\left( { - c(\beta )^{\beta } \frac{{\left| x \right|^{\beta } }}{{\sigma^{\beta } }}} \right)dx}} \\ & = \sqrt[\beta ]{\beta }\sqrt {\frac{\Gamma [3/\beta ]}{\Gamma [1/\beta ]}} \sqrt[\beta ]{{\frac{1}{\beta }\left( {\sigma \sqrt {\frac{\Gamma [1/\beta ]}{\Gamma [3/\beta ]}} } \right)^{\beta } }} \\ & = \sigma \\ \end{aligned} $$

(20)

So, Eq. (17) is the unbiased estimator of σ in Eq. (5).

• (2) Consistency

The variance of random variable ($\sigma^{\beta }$) estimated by Eq. (17) is:

$$ \begin{aligned} {\text{Var}}\left( {\sigma^{\beta } } \right) & {\text{ = Var}}\left( {\sqrt {\frac{\Gamma [3 /\beta ]}{\Gamma [1 /\beta ]}}^{\beta } } \right)\beta \left( {\frac{{\sum\nolimits_{1}^{n} {\left| {e_{i} } \right|^{\beta } } }}{n}} \right) \\ & = \left( {\frac{\Gamma [3 /\beta ]}{\Gamma [1 /\beta ]}} \right)^{\beta } \frac{{\beta^{2} }}{n}{\text{Var}}\left( {\left| e \right|^{\beta } } \right) \\ & = \left( {\frac{\Gamma [3 /\beta ]}{\Gamma [1 /\beta ]}} \right)^{2\beta } \frac{{\beta^{2} }}{n}\left( {E\left( {\left| e \right|^{2\beta } } \right) - E\left( {\left| e \right|^{\beta } } \right)^{2} } \right) \\ & = \frac{\beta }{n}\sigma^{2\beta } \\ \end{aligned} $$

(21)

According to the delta method that uses second-order Taylor expansions to approximate the variance of a random variable (Powell 2007), we get the approximate variance of Eq. (17):

$$ {\text{Var}}(\sigma ) = {\text{Var}}\left( {\sqrt[\beta ]{{\sigma^{\beta } }}} \right) \approx \left( {\frac{1}{\beta }\left( {\sigma^{\beta } } \right)^{1/\beta - 1} } \right)^{2} {\text{Var}}\left( {\sigma^{\beta } } \right) = \frac{1}{{\beta^{2} }}\sigma^{2 - 2\beta } \frac{\beta }{n}\sigma^{2\beta } = \frac{{\sigma^{2} }}{\beta n} $$

(22)

Equation (22) shows that the variance of Eq. (17) approaches zero as the number of samples (n) approaches infinity. So, Eq. (17) is the consistent estimator of σ in Eq. (5).

• (3) Efficiency

The definition of the standard deviation of model residuals is:

$$ \sigma = \sqrt {\frac{{\sum\nolimits_{1}^{n} {e_{i}^{2} } }}{n}} $$

(23)

The variance of random variable ($\sigma^{2}$) estimated by Eq. (23) is:

$$ \begin{aligned} {\text{Var}}(\sigma^{2} ) & = \frac{1}{n}\left( {E\left( {e^{4} } \right) - E\left( {e^{2} } \right)^{2} } \right) \\ & = \frac{1}{n}\frac{{\Gamma [1 /\beta ]\Gamma [5 /\beta ] -\Gamma [3 /\beta ]^{2} }}{{\Gamma [3 /\beta ]^{2} }}\sigma^{4} \\ \end{aligned} $$

(24)

According to the delta method (Powell 2007), we get the approximate variance of Eq. (23):

$$ \begin{aligned} {\text{Var}}(\sigma ) & = {\text{Var}}\left( {\sqrt[2]{{\sigma^{2} }}} \right) \approx \left( {\frac{1}{2}\left( {\sigma^{2} } \right)^{ - 1/2} } \right)^{2} {\text{Var}}\left( {\sigma^{2} } \right) \\ & = \frac{1}{4}\left( {\frac{\Gamma [1/\beta ]\Gamma [5/\beta ]}{{\Gamma [3/\beta ]^{2} }} - 1} \right)\frac{{\sigma^{2} }}{n} \\ \end{aligned} $$

(25)

By comparing the Eq. (25) with the Eq. (22), we get:

$$ \frac{1}{4}\left( {\frac{\Gamma [1/\beta ]\Gamma [5/\beta ]}{{\Gamma [3/\beta ]^{2} }} - 1} \right) \ge \frac{1}{\beta } $$

(26)

where the equality holds if and only if β = 2 (Gaussian distribution).

Equation (26) demonstrates that the variance of Eq. (2) is less than Eq. (8). So, Eq. (2) is more efficient or precise than Eq. (8) to estimate the value of σ in Eq. (5).

Appendix 2: Diagnosis of the autocorrelation of model residuals

See Fig. 10.

Fig. 10

Diagnosis of the autocorrelation of model residuals for six goodness-of-fit indicators where the dash line is the statistically zero band ($\pm \,{{1.96} \mathord{\left/ {\vphantom {{1.96} {\sqrt n }}} \right. \kern-0pt} {\sqrt n }}$)