Model Optimization in Imbalanced Regression

Abstract

Imbalanced domain learning aims to produce accurate models in predicting instances that, though underrepresented, are of utmost importance for the domain. Research in this field has been mainly focused on classification tasks. Comparatively, the number of studies carried out in the context of regression tasks is negligible. One of the main reasons for this is the lack of loss functions capable of focusing on minimizing the errors of extreme (rare) values. Recently, an evaluation metric was introduced: Squared Error Relevance Area (SERA). This metric posits a bigger emphasis on the errors committed at extreme values while also accounting for the performance in the overall target variable domain, thus preventing severe bias. However, its effectiveness as an optimization metric is unknown. In this paper, our goal is to study the impacts of using SERA as an optimization criterion in imbalanced regression tasks. Using gradient boosting algorithms as proof of concept, we perform an experimental study with 36 data sets of different domains and sizes. Results show that models that used SERA as an objective function are practically better than the models produced by their respective standard boosting algorithms at the prediction of extreme values. This confirms that SERA can be embedded as a loss function into optimization-based learning algorithms for imbalanced regression scenarios.

Appendices

A SERA numerical approximation

SERA and its derivatives are approximated by the trapezoidal rule with a uniform grid of T equally spaced intervals with a step of 0.001, as follows.

$$\begin{aligned} \begin{aligned} SERA&= \int _0^1 SER_{t}\, dt \\&\approx \frac{1}{2T} \sum _{k=1}^T \left( SER_{t_{k-1}} + SER_{t_{k}}\right) \\&= \frac{1}{2T} \left( SER_{t_{0}} + 2SER_{t_{1}} +...+ 2SER_{t_{T-1}} + SER_{t_{T}}\right) \\&= \frac{1}{T} \bigg (\sum _{k=1}^{T-1} SER_{t_{k}} + \frac{SER_{t_{0}} + SER_{t_{T}}}{2}\bigg ) \\&= \frac{1}{T} \bigg (\frac{1}{2}\!\sum _{y_i \in \mathcal {D}^{t_0}}(\hat{y}_{i} - y_i)^2 +\!\sum _{y_i \in \mathcal {D}^{t_1}}(\hat{y}_{i} - y_i)^2 + ... \\&\qquad +\!\sum _{y_i \in \mathcal {D}^{t_{T-1}}}(\hat{y}_{i} - y_i)^2 + \frac{1}{2}\!\sum _{y_i \in \mathcal {D}^{t_T}}(\hat{y}_{i} - y_i)^2 \bigg ) \end{aligned} \end{aligned}$$

(9)

Similarly, the derivative of SERA w.r.t. a given prediction $\hat{y}_j$ is obtained by

$$\begin{aligned} \begin{aligned} \frac{\partial SERA}{\partial \hat{y}_j}&\approx \frac{1}{T}\frac{\partial }{\partial \hat{y}_j} \bigg (\frac{1}{2}\!\sum _{y_i \in \mathcal {D}^{t_0}}(\hat{y}_{i} - y_i)^2 +\!\sum _{y_i \in \mathcal {D}^{t_1}}(\hat{y}_{i} - y_i)^2 + ...\\ {}&\qquad +\!\sum _{y_i \in \mathcal {D}^{t_{T-1}}}(\hat{y}_{i} - y_i)^2 + \frac{1}{2}\!\sum _{y_i \in \mathcal {D}^{t_T}}(\hat{y}_{i} - y_i)^2 \bigg )\\&=\frac{1}{T} \bigg (\sum _{y_i \in \mathcal {D}^{t_0}} (\hat{y}_i - y_i)\delta _{ij} + 2\sum _{y_i \in \mathcal {D}^{t_1}} (\hat{y}_i - y_i)\delta _{ij} + ... \\&\qquad +\!\sum _{y_i \in \mathcal {D}^{t_{T-1}}} (\hat{y}_i - y_i)\delta _{ij} + \frac{1}{2}\!\sum _{y_i \in \mathcal {D}^{t_T}} (\hat{y}_i - y_i)\delta _{ij}\bigg ) \\&= \frac{1}{T} \bigg ((\hat{y}_j - y_j) \bigg |_{y_j \in \mathcal {D}^{t_0}} + 2(\hat{y}_j - y_j) \bigg |_{y_j \in \mathcal {D}^{t_1}} + ... \\&\qquad + 2(\hat{y}_j - y_j) \bigg |_{y_j \in \mathcal {D}^{t_{T-1}}} + (\hat{y}_j - y_j) \bigg |_{y_j \in \mathcal {D}^{t_T}} \bigg ) \\&= \frac{1}{T} \bigg ((\hat{y}_j - y_j) \bigg |_{y_j \in \mathcal {D}^{t_0}} + 2\sum _{k=1}^{T-1} (\hat{y}_j - y_j) \bigg |_{y_j \in \mathcal {D}^{t_k}} + (\hat{y}_j - y_j) \bigg |_{y_j \in \mathcal {D}^{t_T}} \bigg ) \\&= \frac{1}{T} \bigg (\boldsymbol{1}\left( y_j \in \mathcal {D}^{t_0}\right) + 2\sum _{k=1}^{T-1} \boldsymbol{1}\left( y_j \in \mathcal {D}^{t_k}\right) + \boldsymbol{1}\left( y_j \in \mathcal {D}^{t_T} \right) \bigg ) (\hat{y}_j - y_j) \end{aligned} \end{aligned}$$

(10)

Note that any given instance $y_j$ will always have at least zero relevance, i.e. $\phi (y_j) \ge 0$, so the first term of Eq. (10) will always be taken into account. Nevertheless, not all the summation terms will be considered for cases where $\phi (y_j) < 1$. With this in mind, we define

$$\begin{aligned} n_j = \sum _{k=1}^{K_j} \boldsymbol{1}\left( y_j \in \mathcal {D}^{t_k}\right) ~~ \end{aligned}$$

(11)

where $n_j$ is the number of times the instance $y_j$ contributes to SERA derivative, where $K_j \in \left[ 1, T-1\right]$. Equation (10) becomes then

$$\begin{aligned} \frac{\partial SERA}{\partial \hat{y}_j} \approx \frac{1}{T} \left( 1 + 2 n_j + \boldsymbol{1}\left( y_j \in \mathcal {D}^{t_T} \right) \right) (\hat{y}_j - y_j)~~ \end{aligned}$$

(12)

In this context, the second derivative for a given prediction $\hat{y}_j$ is obtained by

$$\begin{aligned} \frac{\partial ^2 SERA}{\partial \hat{y}_j^2} \approx \frac{1}{T} \left( 1 + 2 n_j + \boldsymbol{1}\left( y_j \in \mathcal {D}^{t_T} \right) \right) ~~ \end{aligned}$$

(13)

We now proceed to a study on the degree of error committed by using the approximations above (Eqs. (12), (13)) against the use of the trapezoidal rule directly on Eqs. (4) and (5). For that, we resort to the predictions obtained for $\text {XGBoost}^{S}$. The rationale is the following: 1) for each data set, we compute the first and second derivatives using both methods; 2) we calculate the absolute difference between the results obtained from both methods; 3) we average these differences over all instances.

The results obtained using this evaluation are depicted in the left box plot of Fig. 5.

Fig. 5.

Left: Absolute error differences for the first and second derivatives between our approximations and the trapezoidal rule. Right: Execution time (in seconds) of the first and second derivative for a given data set, under our approximation and the trapezoidal rule.

Results show that both first and second derivative approximations have a minor error ($\approx 10^{-12}$). Sim On the right box plot of Fig. 5, we also show the difference in execution time using both methods for each data set. Here, the execution time is measured as the time taken to evaluate both the first and second derivatives. As we can see, there is a non-negligible difference between our approximation and the trapezoidal rule as the size of a data set increases.

B Tables of Results

In this section, we report the MSE and SERA results obtained in out-of-sample of each data set.

Table 3. MSE results in out-of-sample, with the best models per data set in bold. Model (#wins): $\text {XGBoost}^{M}$ (27), $\text {XGBoost}^{S}$ (2), $\text {LGBM}^{M}$ (6), $\text {LGBM}^{S}$ (1).

Table 4. SERA results in out-of-sample, with the best models per data set in bold. Model (#wins): $\text {XGBoost}^{M}$ (16), $\text {XGBoost}^{S}$ (14), $\text {LGBM}^{S}$ (5), $\text {LGBM}^{M}$ (1).