Clustering and Weighted Scoring in Geometric Space Support Vector Machine Ensemble for Highly Imbalanced Data Classification

The goal of the supervised classification is to build a mathematical model of a real-life problem using a labeled dataset. This mathematical model is used to assign the class label to each new recognized object, which, in general, does not belong to the training set. The individual classification model is called a base classifier. Ensemble methods are a vastly used approach to improve the possibilities of base classifiers by building a more stable and accurate ensemble of classifiers (EoC) [23, 28]. In general, the procedure for building EoC consists of three steps: generation, selection, and integration [18]. An imbalanced data problem occurs when the prior probability of classes in a given dataset is very diverse. There are many real-life problems in which we deal with imbalanced data [11, 25], e.g., network intrusion detection [2, 14], source code fault detection [8], or in general fraud detection [1].

There exist two main approaches to solve the problem of imbalanced data: a data-level [9, 26] and an algorithm-level solution [29]. EoC is one of the approaches to solve the imbalanced data classification problem which improve classification measure compared to single models and is highly competitive and robust to imbalanced data [10, 13, 16]. The use of not only voting in the EoC integration phase is one of the directions to solve a problem with imbalanced data [15]. Therefore, this article concerns about calculating the weighted scoring function to be applied in the weighted voting process.

In the process of EoC generation, we use the K-Means clustering algorithm [5] for each class label separately. The base linear classifier – Support Vector Machine [7] – is trained on cluster combination. The weighted scoring function takes into account the distance of a classified object from the decision boundary and cluster centroids used to learn the proper base classifier. Regardless of the number of learning objects in a given cluster, the cluster centroid is always determined. The proposed method for determining the scoring function is, therefore, insensitive to the number of objects defining the cluster. As shown in the article, the proposed approach is useful for imbalanced data.

The paper is structured as follows: Sect. 2 introduces the base concept of EoC and presents the proposed algorithm. In Sect. 3, the experiments that were carried out are presented, while results and the discussion appear in Sect. 4. Finally, we conclude the paper in Sect. 5.

The experimental evaluation conducted for the needs of verification of the method proposed in the following work was based of 30 highly imbalanced datasets contained in the keel repository [3]. Datasets selected for the study are characterized by an imbalance ratio (the proportion between minority and majority class) ranging from 1:9 up to 1:40. Besides, due to the preliminary nature of the conducted research, the pool of datasets includes only binary classification problems.

The basis of the used division methodology was Stratified K-Fold Crossvalidation with \(k=5\), necessary to ensure the presence of minority class patterns in each of the analyzed training subsets. Statistical tests, for both pair and rank tests, were carried out using the Wilcoxon test with the significance level \(\alpha =0.05\) [4].

In the construction of each EoC, in order to limit the number of the presented tables and readability of the analysis, each time we construct the ensemble by dividing classes into two clusters, thus building a pool of four members. In the case of data as strongly imbalanced as those from the selected databases, often only a few (literally four or five) minority class objects remain in a single fold so that a more substantial number would treat almost every minority object as a separate cluster.

The whole experimental evaluation was performed in Python, using the scikit-learn api [20] to implement the cws method and is publicly available on the git repository¹. As metrics for the conducted analysis, due to the imbalanced nature of the classification problem, three aggregate measures (balanced accuracy score, F1-score, and G-mean) and three base measures constituting their calculation (precision, recall, and specificity) were applied, using their implementation included in the stream-learn package [17].

For the readability of the analysis, the full results of the experiment, along with the presentation of the relation between the algorithms resulting from the conducted paired tests, are presented only for the balanced accuracy score (Table 1) and recall (Table 2) metrics.

As may be observed, for aggregate metrics (results are consistent for both balanced accuracy and G-mean, only in F1-score presenting a slightly smaller scale of differences) the use of majority voting (cmv) for EoC diversified with clustering, often leads to deterioration of the classification quality even concerning a single base classifier. Integration by support accumulation (csa) performs slightly better, due to taking into consideration the certainty (support) of the decisions of each classifier but ignoring their areas of competence. The use of areas of competence present in the cws method allows for substantial improvement in classification results, often leading to a statistically significant advantage. The primary source of advantage in results is a significant improvement in the recall metric in this approach.

The design of the recognition algorithm dedicated to imbalanced data is almost always based on the calibration of factors measurable by the base classification metrics. As in the case of the cws method, we try to increase the recall so that the inevitable reduction of precision or specificity will further give us a significant statistical advantage in the chosen aggregate metric, selected to define the cost of the incorrect classification that is relevant to us. In this case, the cws method turns out to be much better than the other methods of EoC integration with a pool diversified by clusters and allows a statistically significant improvement of the base method in the case of highly imbalanced data.

This paper presented a new clustering and weighted scoring algorithm dedicated to constructing EoC. We proposed that the scoring function should take into account the distance from the decision boundary of each base classifier and the cluster centroids necessary to learn this classifier. In the proposed weighting scoring function the distance to the decision boundary and sum of the distances to the centroids have the same weight. The proposed approach applies to imbalanced datasets because each cluster centroid can be calculated regardless of the number of objects in this cluster.

Comprehensive experiments are presented on thirty examples of highly imbalanced datasets. The obtained results show that the proposed algorithm is better than other algorithms in the context of statistical tests and some performance measures. In particular, in the case of the balanced accuracy and G-mean classification measures, the proposed in this paper method is statistically significantly better than all others methods used in the experiments.

A possible future work is to be considered: other distance measures to calculate the distance to cluster centroids, the impact of the use of heterogeneous base classifiers in the proposed method of building EoC or another scoring weighting method. In particular, we suggest that the distance from the decision boundary can be scaled or weighting factors regarding the distance of the object from the decision boundary and the distance from the cluster centroids can be introduced. Additionally, we can assign weights for cluster centroids depending on the number of objects that were used to determine these centroids.