Feature importance measures for hydrological applications: insights from a virtual experiment

Abstract

Discriminating the role of input variables in a hydrological system or in a multivariate hydrological study is particularly useful. Nowadays, emerging tools, called feature importance measures, are increasingly being applied in hydrological applications. In this study, we propose a virtual experiment to fully understand the functionality and, most importantly, the usefulness of these measures. Thirteen importance measures related to four general classes of methods are quantitatively evaluated to reproduce a benchmark importance ranking. This benchmark ranking is designed using a linear combination of ten random variables. Synthetic time series with varying distribution, cross-correlation, autocorrelation and random noise are simulated to mimic hydrological scenarios. The obtained results clearly suggest that a subgroup of three feature importance measures (Shapley-based feature importance, derivative-based measure, and permutation feature importance) generally provide reliable rankings and outperform the remaining importance measures, making them preferable in hydrological applications.

Appendices

Appendix

A Feature importance measures in detail

We introduce here the notation and the main notions useful for defining the measures used in this work. Let Y and $X=(X_1,\dots ,X_d)$ be a random variable and a random vector on the probability space $(\Omega ,{\mathcal {B}}(\Omega ), {\mathbb {P}})$, with $X \in {\mathcal {X}}_d \subseteq {\mathbb {R}}^d$, $Y \in {\mathcal {Y}}_Y \subseteq {\mathbb {R}}$, the cumulative distribution function ${\mathbb {P}}_X$ and ${\mathbb {P}}_Y$, and respective density functions $p_X$ and $p_Y$. The vector ${\textbf{X}}$ can be written as $\textbf{X}=(X_i,{\textbf{X}}_{-i})$, where $\textbf{X}_{-i}=\{X_l,l=1,\dots ,d,l\ne i\}$. We denote the corresponding observed value of $X_i$ as ${\textbf{x}}=\{x_i^{(1)},...,x_i^{(N)}\}'$ and the corresponding jth observation as $\textbf{x}^{(j)}=\{x_1^{(j)},...,x_d^{(j)}\}$ associated with the corresponding target value $y^{(j)} \in {\mathcal {Y}}$. For the computation of the ML feature importance measure, we adopt an ML model ${\widehat{g}}$ (a linear model) to approximate the unknown model. The chosen ML model is fitted on the training set $\{(\textbf{x}^{(j)},y^{(j)})\}_{j=1}^N$. We use $L({\widehat{g}})=\mathbb E[{\mathcal {L}}(Y,{\widehat{g}}({\textbf{X}}))]$ to refer to the generalization error of a trained ML model, where $\mathcal L:Y\times {\mathbb {R}}^d \rightarrow {\mathbb {R}}_{+}$ is the loss function. The notion of Shapley value arises from cooperative game theory. Given a group of players $D=(1,\dots ,d)$ who can join the coalitions $K\subseteq D$. The total number of possible coalitions is $2^d$. We denote by $v:2^d\rightarrow {\mathbb {R}}_{+}$ the value function that assigns the reward to coalitions. The reward of player i is given by

$$\begin{aligned} \phi _i(v)=\sum _{K\subseteq D \setminus \{i\}}\frac{|K|!(|D|-|K|-1)!}{|D|!}[ v(K\cup {i})-v(K)]. \end{aligned}$$

(5)

Regarding the ALE function, its estimation requires to partition the support of the feature of interest in K not overlapping intervals ${\mathcal {X}}_i^k=[z_k,z_{k-1}]$ with $k=1,\dots ,K$. The estimate of the ALE function of feature $X_i$ can be written as

$$\begin{aligned} \widehat{ALE}_{j}(x_{j})=\sum _{k=1}^{K}\frac{1}{N_{i}^k} \sum _{j: \textbf{x}_{j}\in {\mathcal {X}}_i^k}\left[ \widehat{g}\left(z_{i}^k,\textbf{x}_{-i}^{(j)}\right)- \widehat{g}\left(z_{i}^{k-1},\textbf{x}_{-i}^{(j)}\right)\right] , \end{aligned}$$

(6)

for each $x_i \in \left[z_{j}^0,z_{j}^K\right]$, where $z_{i}^0=\min \left\{x_i^{(1)},...,x_i^{(N)}\right\}$ and $z_{i}^K=\max \left\{x_i^{(1)},...,x_i^{(N)}\right\}$. The term $N_i^k$ denotes the number of data points in the kth interval for the feature $X_i$. We now report the feature importance measures employed in our analysis.

• 1. Correlation methods

  • Pearson correlation coefficient $r_i$ of feature $X_i$ is defined as follows

    $$\begin{aligned} r_i= & {} \frac{{\mathbb {E}}(YX_i)-\mathbb E(X_i)E(Y)}{\sqrt{E(X_i^2)-({\mathbb {E}}(X_i))^2}\sqrt{\mathbb E(Y^2)-({\mathbb {E}}(Y))^2}}\nonumber \\{} & {} = \frac{\text {cov}(Y, X_i)}{\sigma _{X_i}\sigma _{Y}}, \end{aligned}$$

    (7)

    where $\text {cov}(Y,X_i)$ denotes the covariance between Y and $X_i$, while $\sigma _{X_i}$ and $\sigma _Y$ denote the standard deviations of $X_i$ and Y, respectively.

  • Spearman’s rank correlation coefficient $\rho _i$ of feature $X_i$ can be written as

    $$\begin{aligned} \rho _i=\frac{\text {cov}(\text {Rank}(Y), \text {Rank}(X_i))}{\sigma _{\text {Rank}(X_i)}\sigma _{\text {Rank}(Y)}}. \end{aligned}$$

    (8)

    Note that the formula above is similar to that of Pearson’s coefficient, but it is applied to rank features.

  • Kendall rank correlation coefficient $\tau _i$ of feature $X_i$ can be written as

    $$\begin{aligned} \tau _i=\frac{n_c-n_d}{\sqrt{n_0-n_c}\sqrt{n_0-n_c}}, \end{aligned}$$

    (9)

    where $n_c$ is the number of concordant pairs, $n_d$ is the number of discordant pairs, $n_0$ is the total number of data pairs and $n_1$ is the total number of data pairs with different values for both features.

• 2. Regression-based methods

  • Standardized regression coefficient SCR$_i$ of the feature $X_i$ is defined as

    $$\begin{aligned} \text {SCR}_i=\frac{\beta _i\sigma _{X_i}}{\sigma _Y}, \end{aligned}$$

    (10)

    where $\beta _i$ is the coefficient of $X_i$ in the linear model.

• 3. ML feature importance measures

  • Permutation feature importance PFI$_i$ of feature $X_i$ is defined as

    $$\begin{aligned} \text {PFI}_i = {\mathbb {E}}[{\mathcal {L}}(Y,{\widehat{g}}(X_i^{\pi }, \textbf{X}_{-i})]-{\mathbb {E}}[{\mathcal {L}}(Y,{\widehat{g}}(X_i, \textbf{X}_{-i})], \end{aligned}$$

    (11)

    where $X_i^\pi$ has the same marginal distribution of $X_i$ and ${\mathbb {E}}[\cdot ]$ is the expectation operator.

  • Permute and Relearn feature importance $\text {VI}_j^{\pi \text {L}}$ of feature $X_i$ is defined as

    $$\begin{aligned} \text {VI}_i^{\pi \text {L}}= {\mathbb {E}}[{\mathcal {L}}(Y, \widehat{g}(X_i, \textbf{X}_{-i}))]-{\mathbb {E}}[{\mathcal {L}}(Y, \widehat{g}^{\pi ,i}(X_i, \textbf{X}_{-i}))], \end{aligned}$$

    (12)

    where ${\widehat{g}}^{i, \pi }$ is the ML model retrained after the permutation of $X_i$.

  • Shapley-based feature importance $\text {SbFI}_i$ of feature $X_i$ is defined as

    $$\begin{aligned} \text {SbFI}_i= & {} \sum _{K\subseteq D \setminus \{i\}}\frac{|K|!(|D|-|K|-1)!}{|D|!}\nonumber \\{} & {} [{\widehat{g}}_K(\textbf{x}_{K\cup {i}})-g_K(\textbf{x}_{K})], \end{aligned}$$

    (13)

    where ${\widehat{g}}_K ({\textbf{x}}_K )=E_{{\textbf{X}}_C } [\widehat{g}(x_K,{\textbf{X}}_C )]$, with C being the complement of K.

  • Shapley feature importance $\text {SFIMP}_i$ of feature $X_i$ is defined using a value function based on the model generalization error, i.e.,

    $$\begin{aligned} \text {SFIMP}_i= &{} \sum _{K\subseteq D \setminus \{i\}}\frac{|K|!(|D|-|K|-1)!}{|D|!}\nonumber \\{} & {} \left[{\widehat{L}}_{K\cup {i}}({\widehat{g}})- \widehat{L}_{K}({\widehat{g}})\right], \end{aligned}$$

    (14)

    where ${\widehat{L}}_{K}({\widehat{g}})=\frac{1}{N}\sum _{j=1}^N \sum _{k=j}^N {\mathcal {L}}\left({\widehat{g}}( x_K^{(j)}, {\textbf{x}}_C^{(k)}), y^{(j)}\right)$.

  • Derivative-based measure $\kappa _i^{\text {ALE}}$ of feature $X_i$ is defined as

    $$\begin{aligned} \kappa _i^{\text {ALE}}(x_i)= & {} \frac{1}{K}\sum _k {\mathbb {E}}\nonumber \\{} & {} \left[ \frac{{\widehat{g}}\left(z_i^k, \textbf{x}_{-i}^{(j)}\right)-\widehat{g}\left(z_i^{k-1}, \textbf{x}_{-i}^{(j)}\right)}{z_i^k-z_i^{k-1}}\right] ^2\frac{\sigma _{X_i}^2}{\sigma _Y^2}. \end{aligned}$$

    (15)

  • ALE-based feature importance $\text {FI}^{\text {ALE}}$ of feature $X_i$ is defined as

    $$\begin{aligned} \text {FI}^{\text {ALE}}_i= \sqrt{{\mathbb {V}}(\text {ALE}_i(X_i))}, \end{aligned}$$

    (16)

    where ${\mathbb {V}}[\cdot ]$ denotes the variance operator.

• 4. SA methods

  • Variance-based sensitivity measure of feature $X_i$ is defined as

    $$\begin{aligned} \eta _i^2=\frac{{\mathbb {V}}[Y] -{\mathbb {E}}_{\textbf{X}_{-i}}[{\mathbb {V}}_{X_i}[Y \mid X_i]]}{{\mathbb {V}}[Y]}. \end{aligned}$$

    (17)

  • Density-based sensitivity measure $\delta _i$ of feature $X_i$ can be written as

    $$\begin{aligned} \delta _i=\frac{1}{2}{\mathbb {E}}_{X_i}\left[ \int _{{\mathcal {Y}}} |p_Y(y)-p_{Y\mid X_i}(y) |dy\right] , \end{aligned}$$

    (18)

    where $p_Y$ and $p_{Y\mid X_i}$ are the marginal output density and the conditional density, respectively.

  • Cumulative distribution-based sensitivity measure $\beta ^{\text {KS}}_i$ of feature $X_i$ can be written as

    $$\begin{aligned} \beta ^{\text {KS}}_i={\mathbb {E}}_{X_i}\left[ \sup _{{\mathcal {Y}}}\left|{\mathbb {P}}_Y(y)-{\mathbb {P}}_{Y\mid X_i }(y)\right|dy\right] , \end{aligned}$$

    (19)

    where ${\mathbb {P}}_Y$ and ${\mathbb {P}}_{Y\mid X_i}$ are the cdfs.

B Tables

In this Section, we report the mean values of the CPI index calculated on the 100 simulations generated for the 30 case studies with $N =$ 100, 1000, and 10000 for the four classes of methods used.

Table 8 CPI estimates for all case studies reported in Table 2 for $N=$ 100, 1000 and 10000 using the Correlation Methods and Regression-based method

Table 9 CPI estimates for all case studies reported in Table 2 for $N=$ 100, 1000 and 10000 using the ML feature importance measures

Table 10 CPI estimates for all case studies reported in Table 2 for $N=$ 100, 1000 and 10000 using the SA measures