HTSS: a hyper-heuristic training set selection method for imbalanced data sets

Original Article
  • 75 Downloads

Abstract

Imbalanced data sets are those in which data samples have uneven distribution amongst the classes. When classifying such data, classical classifiers encounter problem; hence, this problem has become a challenging issue in the field of machine learning. To weaken this problem, we propose a novel hyper-heuristic algorithm, called HTSS, to select the best training samples in this paper. In other words, the best training sample subset is chosen with the goal of enhancing the performance of classifier when confronting imbalanced data. To do so, some local search algorithms and a choice function are incorporated with a global search algorithm to improve its effectiveness. The global search used in this paper is binary quantum inspired gravitational search algorithm (BQIGSA) which is a recently proposed meta-heuristic search for optimization of binary encoded problems. Experiments are performed on 75 imbalanced data sets, and G-mean and AUC measures are employed for evaluation. The results of comparing the proposed method with other state of the art algorithms show the superiority of the proposed HTSS method.

Keywords

Imbalanced data Under-sampling methods Training set selection Meta-heuristic algorithms Hyper-heuristic algorithms Binary Quantum-Inspired Gravitational Search Algorithm 

1 Introduction

Classification of imbalanced data is an important challenge in the field of machine learning and data mining, and has attracted researcher’s attentions to a great extent. Imbalanced data sets are those in which the cardinalities of one or more classes are much smaller than the others. The classes with more data samples are called majority while the minority classes have less samples. The ratio of majority class cardinality, \(N_{maj}\), to minority class cardinality, \(N_{min}\), is called imbalance ratio (IR) and is obtained as:
$$\begin{aligned} IR=\frac{N_{maj} }{N_{min}} \end{aligned}$$
(1)
Imbalanced data sets occur in different fields in real-world applications such as oil spills detection from satellite images [1], fraud detection [2], image annotation [3], and text classification [4]. Such data sets are an obstacle to classical learning algorithms, which are constructed for balanced data sets, since the samples of majority classes dominate the algorithm and the effect of minority samples is low. As a result, the accuracy of minority class is much less. This happens while in most cases and from the learning point of view, the minority class is of more importance and misclassifying the minority samples leads to high risk. Consequently, the problem of learning from imbalanced data sets should be addressed and many methods have been proposed in this regard, which can be mainly categorized as three groups: data preprocessing, algorithm modification and ensemble learning methods. Data preprocessing methods try to balance the data sets while algorithm modification approaches deal with the learning algorithm. Ensemble learning methods are also solutions which fuse the results of different learning algorithms for improvement.

In this paper, we propose a data preprocessing method in which the best training samples are selected and the redundant ones are thrown away. Such approach can be helpful since there may be some noisy or redundant samples in the training set from both majority and minority classes deteriorating the performance of the classifier. We consider binary class data sets, but the method can be applied on multi-class data sets too. Meta-heuristic algorithms are helpful for this aim because they do not need any information about the data samples; however, they may get stuck in local optima especially when the search space is large. Adding some local search can solve this problem; hence, in our proposed method, we design a hyper-heuristic algorithm that receives the training data as input and returns the modified training set as output. Hyper-heuristic algorithms are optimization tools constructed to enhance the performance of meta-heuristic algorithms. In this algorithm, we utilize a global search and dynamically select a local search in each iteration and the local search algorithms are chosen in each iteration according to their previous performance. By defining the local search algorithms properly, there can be tradeoff between exploitation and exploration abilities of the algorithm and the probability of reaching the global optima is increased [5]. Binary quantum inspired gravitational search algorithm (BQIGSA), which is a meta-heuristic proposed recently to improve the binary version of gravitational search algorithm (BGSA), is employed in our method as a global search and 4 general local search algorithms are incorporated to it [6].

The rest of this paper is as follows. Section 2 provides a review about some works related to imbalanced data classification, an explanation and a brief history about Hyper-heuristic algorithms as well as description of BQIGSA. In Sect. 3, the proposed method is explained and Sect. 4 gives the experimental results. Eventually, in Sect. 5, conclusion is drawn.

2 Related works and background

2.1 Imbalanced learning methods

As was mentioned in the previous section, methods proposed to solve the problem of imbalance data classification fall under three main categories: data preprocessing methods, Algorithm modification Methods, and Ensemble learning. In the following, methods of each category are reviewed briefly with the main focus on data preprocessing methods, which is the category our proposed method belongs to.

2.1.1 Data preprocessing methods

Data preprocessing methods, also called external methods, aim to change the distribution of the data for balancing it [7]. The main benefit of these methods is that after preprocessing, any classical classification algorithm can be employed [8]. Also, such methods are usually easy to implement. External methods are categorized into three classes: under-sampling, over-sampling and hybrid methods.

Under-sampling methods remove a number of data samples from the majority class for balancing the number of samples that belong to both classes. According to the process of removal, different methods of under-sampling have been proposed. The simplest method of under-sampling is to remove majority class samples randomly until a balance is achieved, called Random Under-Sampling (RUS) [9]. To do under-sampling more efficiently, many rules have been proposed in the literature some of which are Tomek Links [10], Condensed Nearest Neighbor Rule (CNN) [11], One-Sided Selection (OSS) [12], Edited Nearest Neighbor rule (ENN) [13], Neighborhood Cleaning Rule (NCL) [14], and Class Purity Maximization (CPM) . Evolutionary algorithms (EAs) are also used for under-sampling. In [7], a taxonomy of evolutionary under-sampling methods in imbalanced data is provided. Moreover, the authors in [15] and [16] utilize genetic algorithm to select the best training samples with the goal of enhancing the classifier’s performance. The main drawback of under-sampling methods is that eliminating some majority class samples may result in losing some important information [17].

On the contrary, over-sampling techniques add a number of samples to the minority class to make the data set balanced. New added samples can be replication of the existing minority samples or newly generated ones. Some over-sampling methods are Random Over-Sampling (ROS), which replicates the minority class samples randomly [9], RAnked Minority Oversampling (RAMO) [18], ADaptive Synthetic Sampling Technique (ADASYN) [19] and Synthetic Minority Oversampling Technique (SMOTE) [20]. SMOTE is a popular over-sampling algorithm which is the basis of many other methodologies. In this method, first, a minority class sample is selected randomly. Then, its k nearest neighbors are found using Euclidean distance and a minority class sample is chosen randomly from these neighbors. Lastly, the difference between this random sample and the primary sample is multiplied by a number randomly selected from the interval [0, 1]; next, it is added to the primary sample to form the new sample. This process is continued for m randomly selected minority samples until the desired cardinality for minority class is achieved. Modified SMOTE (MSMOTE) [21] is one of the methods originated from SMOTE. In [22], Majority Weighted Minority Oversampling Technique (MWMOTE) is introduced which uses SMOTE algorithm for generating new samples. However, it first finds critical samples named hard-to-learn informative minority class samples, and then applies SMOTE on them. PDF estimation-based over-sampling method, PDFOS, which uses PDF estimation for over-sampling is introduced in [23]. It is worth mentioning that over-sampling techniques usually encounter the problem of over-fitting because of presenting new synthetic samples [17].

Hybrid methods are incorporation of under-sampling and over-sampling methods to reduce the over-fitting effect [24]. Some hybrid methods are SMOTE-Tomek links (SMOTE-TL) [25], SMOTE-ENN [24], Borderline-SMOTE1 [26], Borderline-SMOTE2 [26], Safe-Level-SMOTE [27] and SMOTE-RSB [24]. In [28], a combination of over-sampling and under-sampling methods called SUNDO is proposed in which the loss of information is reduced.

2.1.2 Algorithm modification methods

Algorithm modification methods are also called internal approaches since they focus on the learning algorithms. In other words, these methods try to design new classifiers or modify the existing ones in such a way that they acquire the ability of dealing with imbalanced data sets. Cost-sensitive approaches, which belong to this category, assign misclassification costs to samples of each class, so the distribution of data is changed and the classifier tries to reduce the total cost [29]. Because the minority class is of more importance, the misclassification cost assigned to minority class samples is higher. Although setting costs solves the imbalance problem, finding suitable costs is difficult [30]. In [31], a cost-sensitive neural network is proposed in which the threshold can move based on the given cost and the cost is defined automatically using receiver operating characteristic curve. Gravitational Fixed Radius Nearest Neighbors Algorithm (GFRNN) is proposed in [32] to be an improved version of k Nearest Neighbor (kNN) for imbalanced data sets. Two other enhancements are presented in [33, 34]. Authors in [35] propose Neighbors Progressive Competition Algorithm (NPC), which employs a new grading method for training samples. Some other examples of algorithmic level approaches are presented in [29, 36, 37, 38, 39]

2.1.3 Ensemble learning methods

Apart from the two previously mentioned categories, ensemble learning is put forward as an efficient solution to tackle the problem of imbalanced data classification. These methods are based on employing multiple classifiers and incorporating their results. Ensemble learning methods are usually combined with the previous mentioned approaches to be efficiently used [40, 41, 42]. Bagging and boosting methods are within this group that use preprocessing at their first step, so they may suffer from over-fitting and under-fitting [43]. In [43], a methodology is proposed which converts the imbalanced data set into some balanced ones first and then uses multiple classifiers and an ensemble rule to combine their results. In [44], ensemble methods related to the class imbalance problem are reviewed. Furthermore, in [17], a new methodology of sampling as well as use of ensemble of SVM for classification of imbalanced data are suggested in order to prevent over-fitting and under-fitting. In [45], genetic algorithm is combined with ensemble learning which results in an algorithm called EUSBoost. Lim et al. proposed a new method that generates synthetic minority samples using clustering methods and then the corresponding parameters are optimized using an EA. Afterward, it is combined with ensemble methods [46].

2.2 Hyper-heuristic algorithm

As was mentioned before, hyper-heuristic algorithms are developed to enhance the performance of meta-heuristic algorithms so that they will become more applicable. Such algorithms, which can be considered as high level heuristics, aim at finding a proper sequence of low-level heuristics to be applied on a problem so as to reach more promising solutions. In each step of this algorithm, a low-level heuristic is selected and applied to the problem, i.e., the search space is on low-level heuristics rather than the problem’s solution [47].

A hyper-heuristic algorithm is constitute of two layers. The first layer includes some low-level heuristics, called local search, which provide some approaches to solve the problem. Although meta-heuristic algorithms are considered as low level heuristics in some cases, most of the times they are simpler methods. Local search can explore a small region around the current solution. The second layer is like a black box which means it is designed regardless of the problem at hand. The main responsibility of this layer is to accept or not accept the low level heuristic. Usually, the black box is a choice function which manages the local search selection during the search procedure and in each step, according to search region and the past performance of local search, a local search is selected and applied to the problem [48].

Over the last few years, there have been many methods proposed based on hyper-heuristics. A simulated annealing hyper-heuristic is proposed in [49]. An improved genetic algorithm based on hyper-heuristic is presented in [50]. Subsequently, an evolutionary hyper-heuristic is proposed in [51]. An ant colony-based hyper-heuristic, PSO-based hyper-heuristic and a threshold accepting hyper-heuristic algorithm are suggested in [47, 52] and [53], respectively. Recently, a new definition of hyper-heuristic is introduced in [54] in which it is defined as a learning procedure for either selection or generation of heuristics; therefore, there exist two classes of hyper-heuristics including heuristic selection and heuristic generation. While heuristic selection methods concentrate on choosing the already existing heuristics, heuristic generation approaches aim at generation of new heuristics based on the existing ones.

2.3 Binary quantum inspired gravitational search algorithm (BQIGSA)

Binary quantum inspired gravitational search algorithm is a population-based meta-heuristic which was proposed by Nezamabadi-pour in 2015 [6]. This algorithm retain the main structure of gravitational search algorithm (GSA), but incorporates its principles with the concepts of quantum computing (QC). The smallest unit of quantum information is called Q-bit that can be in the states “1”, “0” or any superposition of these two. Each Q-bit is represented by a pair \(\left( \upalpha ,\upbeta \right) \), where \(\alpha \) and \(\beta \) are complex numbers denoting the probability amplitudes of the corresponding states [55]. \(\left| \alpha \right| ^{2}\) and \(\left| \beta \right| ^{2}\) are probabilities of finding the Q-bit in the states “0” and “1”, respectively, and the condition \(\left| \alpha \right| ^{2}+\left| \beta \right| ^{2}=1\) should be satisfied for normalizing the states.

In BQIGSA, each object is shown as a string of n Q-bits as:
$$\begin{aligned} q_i \left( t \right)= & {} \left[ {q_i^1 \left( t \right) ,q_i^2 \left( t \right) ,\,\ldots ,\,q_i^n \left( t \right) } \right] \nonumber \\= & {} \left[ {\left. {{\begin{array}{l} {\alpha _i^1 \left( t \right) } \\ {\beta _i^1 \left( t \right) } \\ \end{array} }} \right| \left. {{\begin{array}{l} {\alpha _i^2 \left( t \right) } \\ {\beta _i^2 \left( t \right) } \\ \end{array} }} \right| \ldots \left| {{\begin{array}{l} {\alpha _i^n \left( t \right) } \\ {\beta _i^n \left( t \right) } \\ \end{array} }} \right. } \right] \end{aligned}$$
(2)
where \(\left| {\alpha _d} \right| ^{2}+\left| {\beta _d} \right| ^{2}=1\), \(d=1,2,\ldots ,n\), and \(i=1,2,\ldots ,NP\). n is the dimension of the search space and NP denotes the size of population, i.e., the number of objects in the swarm; hence, the swarm’s structure is given as:
$$\begin{aligned} Q\left( t \right) =\left\{ {q_1 \left( t \right) ,q_2 \left( t \right) ,\ldots ,q_{NP} \left( t \right) } \right\} \end{aligned}$$
(3)
An \(\hbox {n}\) Q-bits system can show \(2^{n}\) states simultaneously, so an observation process is needed to be applied to Q-bit objects to obtain a single state. After this process, current binary solutions \(SW\left( t \right) =\left\{ {X_1 \left( t \right) ,\ldots ,\,X_i \left( t \right) ,\ldots ,\,X_{NP} \left( t \right) } \right\} \) are achieved, where \(X_i \left( t \right) =\left( {x_i^1 \left( t \right) ,x_i^2 \left( t \right) ,\ldots ,x_i^n \left( t \right) } \right) \).
The observation rule is:
$$\begin{aligned} {\begin{array}{ll} {If} &{} {rand\left[ {0, 1} \right] <(\alpha _i^d \left( t \right) )^{2}} \\ {Then} &{} {x_i^d \left( t \right) =0} \\ {Else} &{} {x_i^d \left( t \right) =1} \\ \end{array} } \end{aligned}$$
(4)
The best solutions found after calculating the fitness values of objects through iterations are kept in the set \(SB\left( t \right) =\left\{ {B_1 \left( t \right) ,B_2 \left( t \right) ,\ldots ,B_{NP} \left( t \right) } \right\} \), where \(B_i \left( t \right) =\left( b_i^1 \left( t \right) ,b_i^2\right. \left. \left( t \right) ,\ldots ,b_i^n \left( t \right) \right) \). In the first iteration, all the members of SW are transferred to SB and for the others, each object of S is replaced with the corresponding object of SB if its fitness value is better.
BQIGSA starts with the initialization step in which a swarm \(Q\left( t \right) \) containing NP Q-bit objects in an n dimensional space is generated randomly. This procedure should fulfill the normalization rule mentioned before. Afterward, Q-bit objects of swarm \(Q\left( t \right) \) should be updated which is done employing Q-gates in quantum computing. Q-gates receive \(\alpha \) and \(\beta \) (Q-bit’s state) as inputs and change them in such a way that the normalization condition is fulfilled at the end. A modified rotation Q-gate (RQ-gate) is utilized in BQIGSA aiming at boosting the exploration capability of the search. The formula of RQ-gate for updating the Q-bits is as follows:
$$\begin{aligned}&\left[ {{\begin{array}{c} {\alpha _i^d \left( {t+1} \right) } \\ {\beta _i^d \left( {t+1} \right) } \\ \end{array} }} \right] =\left[ {{\begin{array}{cc} \cos \left( \Delta \theta _i^d \right) &{} -\sin \left( \Delta \theta _i^d \right) \\ \sin \left( \Delta \theta _i^d \right) &{} \cos \left( \Delta \theta _i^d \right) \\ \end{array} }} \right] \left[ {{\begin{array}{c} {\alpha _i^d \left( t \right) } \\ {\beta _i^d \left( t \right) } \\ \end{array} }} \right] \quad \nonumber \\&d =1,2,\ldots ,n \end{aligned}$$
(5)
Different from GSA in which linear velocity is used, in RQ-gate, the angular velocity should be adopted. The angular velocity determines the amount of movement toward 0 or 1. Linear velocity is converted to angular velocity using the following formula:
$$\begin{aligned} \omega =\frac{v}{r} \end{aligned}$$
(6)
where \(\omega \), v, and r denote angular velocity, linear velocity and the radius of the circular system, respectively. It is clear that when \(r=1\), the angular velocity equals to the linear velocity, so the following equations can be used for calculating it:
$$\begin{aligned}&a_i^d \left( t \right) =\mathop {\sum }\limits _{j\in kbest} G\left( t \right) \frac{M_j \left( t \right) }{R_{ij} \left( t \right) +\varepsilon }\left( {b_j^d \left( t \right) -x_i^d \left( t \right) } \right) \end{aligned}$$
(7)
$$\begin{aligned}&\omega _i^d \left( {t+1} \right) =rand_i \times \omega _i^d \left( t \right) +a_i^d \left( t \right) \end{aligned}$$
(8)
where \(a_i^d \left( t \right) \) and \(M_j \left( t \right) \) show the acceleration of the ith object in dimension d and the mass of object j, respectively, at iteration t. The gravitational constant at iteration \(\hbox {t}\) is represented by \(G\left( t \right) \) that is initialized by \(\hbox {G}_0 \) at the first iteration and decreased by time according to \(\hbox {G}=\hbox {G}_0 \left( {1-0.95\frac{\hbox {t}}{\hbox {T}}} \right) \) (\(\hbox {T}\) is the total number of iterations). kbest is a set that contains the first k best solutions of SB, i.e., the objects with largest masses. k equals to NP at the beginning and is decreased linearly by time until it reaches 1 in the end. \(R_{ij} \left( t\right) \) denotes the normalized Hamming distance between objects i and j in a binary space and is calculated as follows:
$$\begin{aligned} R_{ij} \left( t \right) =\frac{\mathop {\sum }\nolimits _{l=1}^n \left| {x_i^l \left( t \right) -x_j^l \left( t \right) } \right| }{n} \end{aligned}$$
(9)
The mass, \(M_i \left( t \right) \), is calculated based on the following formula for each object of SB:
$$\begin{aligned} M_i \left( t \right) =\frac{fit_i \left( t \right) -worst\left( t \right) }{\mathop {\sum }\nolimits _{j=1}^N \left( {fit_j \left( t \right) -worst\left( t \right) } \right) } \end{aligned}$$
(10)
where \(fit_i\) denotes the fitness of the ith object and \(\hbox {worst}\left( t \right) \) is the minimum (maximum) fitness value among the fitness values of NP objects at t for maximization (minimization) problem.
In a circular system, \(\Delta \theta \) defines the amount of displacement which is obtained as follows:
$$\begin{aligned} \Delta \theta =\omega \cdot \Delta t \end{aligned}$$
(11)
Since our system is discrete, \(\Delta t\) is 1, so \(\Delta \theta _i^d\) is computed with respect to a quarter \(\theta _i^d\) is in as:
$$\begin{aligned} \Delta \theta _i^d =\left\{ {{\begin{array}{ll} {\omega _i^d \left( {t+1} \right) }&{} {\hbox {if }\left( {\alpha _i^d \left( {t+1} \right) \beta _i^d \left( {t+1} \right) } \right) \ge 0} \\ {-\omega _i^d \left( {t+1} \right) }&{} {\hbox {if }\left( {\alpha _i^d \left( {t+1} \right) \beta _i^d \left( {t+1} \right) } \right) <0} \\ \end{array} }} \right. \end{aligned}$$
(12)
where \(\Delta \theta _d \) is rotation angle of the \(d-\)th Q-bit toward either 0 or 1 state. After that, Eq. 5 is utilized to update the states of Q-bits and then the iteration counter is increased by 1. This procedure is repeated until the stopping criterion is satisfied.

3 The proposed hyper-heuristic training set selection method

As was mentioned before, our goal is to improve the performance of BQIGSA in training set selection problem. This is done using it as a global search in a hyper-heuristic framework and adding some local search to provide a balance between exploitation and exploration abilities. In this section, the preliminaries of the proposed training set selection method, the framework of the suggested hyper-heuristic algorithm and the employed local searches are explained, respectively.

3.1 Preliminaries

In the training phase of this algorithm, the best training samples are selected from both minority and majority classes using the proposed hyper-heuristic algorithm. As was mentioned previously, the samples are selected from both classes since they both may include redundant samples. Then, after training the classifier by the new training set, test samples can be classified. It is assumed we have the primary training set,TR, containing n samples with m features. Each sample is presented as pair \(\left( {O_j ,\,l_j } \right) \) where \(O_j ,\,j=1,2,\ldots ,n,\) shows the j-th training sample and \(l_j\) is its corresponding label which can be assigned either 0 or 1 for majority or minority class samples, respectively.

To feed the training data to the hyper-heuristic algorithm, a binary representation is used in which the data set is shown as a string of n elements (one element for each sample in TR) and each element can be assigned either 0 or 1 value. When an element is assigned 0, it means that the corresponding sample should be omitted from the training set while if it is 1, the sample is retained. This string, that is represented as set \(X_i \left( t \right) \) in BQIGSA algorithm, is depicted in Fig. 1 where \(N_{maj}\) and \(N_{min}\) are the numbers of majority and minority class samples, respectively. It should be noticed that \(N_{maj} +N_{min} =n\).

3.2 The proposed hyper-heuristic framework

The proposed algorithm consists of two layers. The first layer includes a meta-heuristic combined with some low-level heuristics or local search algorithms which are simple algorithms exploring a small region of the search space and can improve the current solution by making small changes. The second layer, on the other hand, is a high-level heuristic with the responsibility of investigating the search space globally. Also, there exists a choice function, as a supervisor, that manages the selection of local searches due to the previous performance of them in the iterations. Using several local searches increase the flexibility of the algorithm to reach a good performance. Fig. 2 illustrates this framework.
Fig. 1

Object representation

Fig. 2

Framework of the proposed hyper-heuristic algorithm

Fig. 3

Pseudo-code of the proposed hyper-heuristic algorithm

Table 1

Characteristics of 75 imbalanced data sets used in the experiment

Data sets

# Samples

# Features

IR

Abalone9_18

731

8

16.4

Abalone19

4174

8

129.44

cleveland-0_vs_4

177

13

12.62

dermatology6

358

34

16.9

ecoli1

336

7

3.36

ecoli2

336

7

5.46

ecoli3

336

7

8.6

ecoli4

336

7

15.8

ecoli-0-1-4-6_vs_5

280

6

13

ecoli-0-1-4-7_vs_2-3-5-6

336

7

10.59

ecoli-0-1-4-7_vs_5-6

332

6

12.28

ecoli-0-1_vs_2-3-5

244

7

9.17

ecoli-0-1_vs_5

240

6

11

ecoli-0-2-3-4_vs_5

202

7

9.1

ecoli-0-2-6-7_vs_3-5

224

7

9.18

ecoli-0-3-4-6_vs_5

205

7

9.25

ecoli-0-3-4-7_vs_5-6

257

7

9.28

ecoli-0-3-4_vs_5

200

7

9

ecoli-0-4-6_vs_5

203

6

9.15

ecoli-0-6-7_vs_3-5

222

7

9.09

ecoli-0-6-7_vs_5

220

6

10

ecoli_0_1_3_7_vs_2_6

281

7

39.14

ecoli_0_vs_1

220

7

1.86

glass0

214

9

2.06

glass1

214

9

1.82

glass2

214

9

11.59

glass4

214

9

15.47

glass5

214

9

22.78

glass6

214

9

6.38

glass-0-1-4-6_vs_2

205

9

11.06

glass-0-1-5_vs_2

172

9

9.12

glass-0-4_vs_5

92

9

9.22

glass-0-6_vs_5

108

9

11

glass_0_1_6_vs_2

192

9

10.29

glass_0_1_6_vs_5

184

9

19.44

Haberman

306

3

2.78

led7digit-0-2-4-5-6-7-8-9_vs_1

443

7

10.97

new_thyroid1

215

5

5.14

new_thyroid2

215

5

5.14

page_blocks0

5472

10

8.79

page_blocks_1_3_vs_4

472

10

15.86

Pima

768

8

1.87

poker-8-9_vs_6

1485

10

58.4

poker-8_vs_6

1477

10

85.88

poker-9_vs_7

244

10

29.5

segment0

2308

19

6.02

shuttle-2_vs_5

3316

9

66.67

shuttle-6_vs_2-3

230

9

22

shuttle_c0_vs_c4

1829

9

13.87

shuttle_c2_vs_c4

129

9

20.5

vehicle0

846

18

3.25

vehicle1

846

18

2.9

vehicle2

846

18

2.88

vehicle3

846

18

2.99

vowel0

988

13

9.98

winequality-red-4

1599

11

29.17

winequality-red-8_vs_6-7-5

855

8

46.5

winequality-white-3-9_vs_5

1482

11

58.28

winequality-white-3_vs_7

900

11

44

winequality-white-9_vs_4

168

11

32.6

Wisconsin

683

9

1.86

yeast1

1484

8

2.46

yeast3

1484

8

8.1

yeast4

1484

8

28.1

yeast5

1484

8

32.73

yeast6

1484

8

41.4

yeast-0-2-5-6_vs_3-7-8-9

1004

8

9.14

yeast-0-2-5-7-9_vs_3-6-8

1004

8

9.14

yeast-0-3-5-9_vs_7-8

506

8

9.12

yeast_0_5_6_7_9_vs_4

528

8

9.35

yeast_1_2_8_9_vs_7

947

8

30.57

yeast_1_4_5_8_vs_7

693

8

22.1

yeast_1_vs_7

459

7

14.3

yeast_2_vs_4

514

8

9.08

yeast_2_vs_8

482

8

23.1

In this algorithm, after finding the SB set by BQIGSA in each iteration, choice function selects a local search and applies it to all objects of the population. This process is continued after reaching a stopping condition. The used local search algorithms are explained in detail in the following section. Moreover, roulette wheel is employed as choice function. Roulette wheel works based on probabilities and assigns a probability to each local search based on its fitness value using the following equation:
$$\begin{aligned} p_i =\frac{\Delta Fit_i }{\mathop {\sum }\nolimits _{j=1}^{nl} \Delta Fit_j }\quad \qquad \qquad i=1,2,\ldots ,nl \end{aligned}$$
(13)
where \(\Delta Fit_i\) shows the difference between the fitness values before and after applying the i-th local search. More precisely, in the first iteration of the proposed algorithm, equal probabilities are assigned to the local search algorithms and roulette wheel selects one of them for each object of population to be applied to it. Then, the probabilities are updated using Eq.13. \(\Delta Fit_i \) is the best improvement that the i-th local search has achieved in the population. In the next iteration, the updated probabilities are employed for local search selection and the process is continued. It should be stated that when \(\Delta Fit\) is lower than 0.001 or the local search is not used in an iteration, \(\Delta Fit\) is set to 0.001 to avoid its elimination.
Cumulative probability is utilized in implementation of roulette wheel. Cumulative probability for the i-th local search is computed by summing the selection probabilities of the first to i-th local search algorithms as:
$$\begin{aligned} C_i =\mathop {\sum }\limits _{j=1}^i p_{j} \end{aligned}$$
(14)
where \(C_i\) denotes the cumulative probability of the i-th local search. In this selection method, a section equal to the probability of each local search in the interval 0 to 1 is assigned to it. Subsequently, a random number in interval (0, 1) with uniform distribution is chosen. Then, it is determined which local search relates to the section this random number is located, and roulette wheel selects that. The pseudo-code of the proposed method is shown in Fig 3.
Fig. 4

Confusion matrix of a two-class problem

3.3 Local search algorithms

As was mentioned before, local search algorithms explore a small region around the current solution to increase the exploitation ability of the global search algorithm. We use 4 local searches to construct our hyper-heuristic algorithm that are explained in the following.
  • Swap Dimension (SW) This algorithm is a mutational heuristic in which 2 dimensions of each object are selected randomly and their values are exchanged.

  • Simple Mutation (MU) In this algorithm, one dimension of each object is selected randomly and then inversed.

  • Dimensional Mutation (DM) In this mutational heuristic, one dimension of each object is selected randomly and then inversed with the probability of 0.5.

  • Improved Mutation (IM) In this method, three dimensions are randomly chosen and complemented simultaneously.

4 Experimental results

To evaluate our memetic algorithms, we use 75 imbalanced data sets that are taken from the KEEL Repository [50]. The main characteristics of these data sets are presented in Table 1 showing the number of samples (#Samples), number of features (#Features) and the ratio of majority class samples to minority ones (IR). Five cross-validation (5-Fcv) is employed to partition these data sets and our classifier is C4.5 decision tree.

4.1 Assessment metric and fitness function

In imbalanced data classification problems, accuracy measure cannot be utilized as assessment metric anymore since the number of minority samples is much lower than majority ones and ignoring them has less effect than ignoring majority sample. Hence, another metric should be employed to consider the classifier’s accuracy on both classes simultaneously. A common performance measure which fulfills this requirement is Geometric Mean (G-mean). This measure enables us to maximize the accuracy of both majority and minority classes by increasing the value of this metric. G-mean is computed according to confusion matrix, which is presented for a two class problem in Fig. 4 as:
$$\begin{aligned} G\_mean=\sqrt{\left( {\frac{TP}{TP+FN}} \right) \cdot \left( {\frac{TN}{TN+FP}} \right) } \end{aligned}$$
(15)
The area under the Receiver Operating Characteristic (ROC) curve, which is called AUC, is another well-known and proper evaluation metric for imbalanced data sets. ROC curve visualizes the tradeoff between true-positive (TP) rate and false-negative (FP) rate, and AUC shows the probability according to which noise and signal plus noise are distinguished in a correct way. In other word, AUC can show the classifier’s performance by a single number and higher AUC result proves better performance of the classifier. To calculate AUC, the area under ROC is obtained as:
$$\begin{aligned} AUC=\frac{1+TP_{rate} -FP_{rate} }{2} \end{aligned}$$
(16)
We employ G-mean as fitness function in our proposed method as:
$$\begin{aligned} fitness\left( t \right) =G\_mean\left( t \right) \end{aligned}$$
(17)
Table 2

Parameters of the algorithms

Algorithm

Parameters

C4.5

Prune \(=\) True, Confidence level \(=\) 0.25

Minimum number of item-sets per leaf \(=\) 2

Confidence \(=\) Laplace Smoothing

NCL & OSS

Number of neighbors \(=\) 5

ADASYN

Number of neighbors \(=\) 5, Alpha \(=\) 0.5, Mu \(=\) 0.5

SMOTE + ENN

Number of neighbors ENN \(=\) 3, Number of neighbors SMOTE \(=\) 5

Type of SMOTE \(=\) both, Quantity \(=\) Balance

Distance \(=\) Heterogeneous Value Deference Metric (HVDM)

SMOTE + TL

Number of neighbors SMOTE \(=\) 5, Type of SMOTE \(=\) both, Quantity \(=\) Balance

Distance \(=\) Heterogeneous Value Dierence Metric (HVDM)

HTSS

Pop \(=\) 50, Eval \(=\) 10000, \(\hbox {G}_{0} = 0.125\)

4.2 Comparative algorithms and parameters settings

We first compare the proposed HTSS algorithm with cases where only one local search algorithm is employed constantly on 75 data sets and then the results are compared to 7 state-of-the-art methods comprising three under-sampling methods, RUS [9], NCL [14], and OSS [12], two over-sampling methods ROS [9] and ADASYN [19], and two popular hybrid methods, SMOTE-ENN [24] and SMOTE-TL [25]. All the algorithms are run in KEEL. Table 2 shows the parameters of C4.5 classifier, the comparative algorithms and the proposed method.
Table 3

The average G-mean and AUC results of applying the algorithms on test sets

Data sets

Evaluation measures

HTSS

BQIGSA+ IM

BQIGSA+ MU

BQIGSA+ DM

BQIGSA+ SW

ROS [27]

Abalone9_18

GM_Test

0.8190 (± 0.14)

0.8084 (± 0.16)

0.7966 (± 0.16)

0.8012 (± 0.14)

0.8117 (± 0.14)

0.5575 (± 0.20)

AUC_Test

0.8370 (± 0.12)

0.8306 (± 0.05)

0.8187 (± 0.06)

0.8251 (± 0.14)

0.8297 (± 0.12)

0.6601 (± 0.14)

Abalone19

GM_Test

0.3910 (± 0.12)

0.4397 (± 0.12)

0.3696 (± 0.17)

0.4359 (± 0.10)

0.4312 (± 0.11)

0.2362 (± 0.19)

AUC_Test

0.5890 (± 0.04)

0.5887 (± 0.04)

0.5831 (± 0.03)

0.5895 (± 0.04)

0.6023 (± 0.04)

0.5873 (± 0.03)

cleveland-0_vs_4

GM_Test

0.8529 (± 0.12)

0.8638 (± 0.12)

0.7170 (± 0.07)

0.8503 (± 0.12)

0.8295 (± 0.10)

0.4407 (± 0.38)

AUC_Test

0.8627 (± 0.11)

0.8740 (± 0.09)

0.7479 (± 0.06)

0.8576 (± 0.11)

0.8409 (± 0.09)

0.6554 (± 0.17)

Dermatology6

GM_Test

0.9807 (± 0.03)

0.9717 (± 0.02)

0.9717 (± 0.05)

0.9717 (± 0.03)

0.9717 (± 0.05)

0.9985 (± 0.00)

AUC_Test

0.9819 (± 0.03)

0.9860 (± 0.00)

0.9735 (± 0.01)

0.9819 (± 0.03)

0.9735 (± 0.05)

0.9985 (± 0.00)

ecoli1

GM_Test

0.8964 (± 0.26)

0.8647 (± 0.05)

0.8531 (± 0.08)

0.8847 (± 0.05)

0.8813 (± 0.06)

0.8923 (± 0.04)

AUC_Test

0.8977 (± 0.06)

0.9081 (± 0.03)

0.8573 (± 0.05)

0.8860 (± 0.05)

0.8811 (± 0.06)

0.8939 (± 0.04)

ecoli2

GM_Test

0.9194 (± 0.06)

0.9015 (± 0.07)

0.8987 (± 0.07)

0.9119 (± 0.05)

0.9111 (± 0.06)

0.8622 (± 0.04)

AUC_Test

0.9217 (± 0.06)

0.9125 (± 0.06)

0.9026 (± 0.06)

0.9137 (± 0.05)

0.9130 (± 0.05)

0.8672 (± 0.04)

ecoli3

GM_Test

0.8261 (± 0.05)

0.8340 (± 0.06)

0.8344 (± 0.06)

0.8330 (± 0.04)

0.8160 (± 0.06)

0.7543 (± 0.12)

AUC_Test

0.8349 (± 0.04)

0.8486 (± 0.03)

0.8432 (± 0.03)

0.8397 (± 0.04)

0.8267 (± 0.05)

0.7811 (± 0.09)

ecoli4

GM_Test

0.8622 (± 0.10)

0.8265 (± 0.11)

0.8201 (± 0.11)

0.8444 (± 0.10)

0.8495 (± 0.10)

0.8200 (± 0.10)

AUC_Test

0.8768 (± 0.08)

0.8947 (± 0.09)

0.9397 (± 0.09)

0.8595 (± 0.08)

0.8638 (± 0.08)

0.8405 (± 0.08)

ecoli-0-1-4-6_vs_5

GM_Test

0.8970 (± 0.12)

0.8531 (± 0.11)

0.8529 (± 0.12)

0.9013 (± 0.10)

0.8800 (± 0.13)

0.7974 (± 0.18)

AUC_Test

0.8981 (± 0.10)

0.8942 (± 0.12)

0.8731 (± 0.13)

0.9673 (± 0.09)

0.8923 (± 0.11)

0.8307 (± 0.13)

ecoli-0-1-4-7_vs_2-3-5-6

GM_Test

0.8762 (± 0.06)

0.8371 (± 0.07)

0.8194 (± 0.03)

0.8630 (± 0.06)

0.8603 (± 0.07)

0.8150 (± 0.09)

AUC_Test

0.8826 (± 0.05)

0.8910 (± 0.07)

0.8305 (± 0.04)

0.8695 (± 0.05)

0.8687 (± 0.06)

0.8304 (± 0.08)

ecoli-0-1-4-7_vs_5-6

GM_Test

0.8919 (± 0.09)

0.8584 (± 0.09)

0.8296 (± 0.12)

0.8909 (± 0.05)

0.8857 (± 0.08)

0.8008 (± 0.10)

AUC_Test

0.8987 (± 0.08)

0.9006 (± 0.03)

0.8444 (± 0.06)

0.8965 (± 0.05)

0.8927 (± 0.08)

0.8187 (± 0.08)

ecoli-0-1_vs_2-3-5

GM_Test

0.8985 (± 0.10)

0.9216 (± 0.06)

0.8139 (± 0.13)

0.8984 (± 0.10)

0.8979 (± 0.07)

0.6898 (± 0.19)

AUC_Test

0.9085 (± 0.08)

0.9144 (± 0.05)

0.8348 (± 0.09)

0.8984 (± 0.08)

0.954 (± 0.07)

0.7468 (± 0.13)

ecoli-0-1_vs_5

GM_Test

0.9658 (± 0.03)

0.9008 (± 0.07)

0.9074 (± 0.06)

0.9366 (± 0.04)

0.9566 (± 0.03)

0.7604 (± 0.08)

AUC_Test

0.9670 (± 0.03)

0.9229 (± 0.08)

0.9121 (± 0.04)

0.9398 (± 0.04)

0.9583 (± 0.03)

0.7863 (± 0.06)

ecoli-0-2-3-4_vs_5

GM_Test

0.8856 (± 0.10)

0.8208 (± 0.11)

0.8259 (± 0.09)

0.8796 (± 0.10)

0.8795 (± 0.11)

0.8205 (± 0.06)

AUC_Test

0.8959 (± 0.09)

0.8698 (± 0.09)

0.8427 (± 0.08)

0.8874 (± 0.14)

0.8913 (± 0.09)

0.8334 (± 0.05)

ecoli-0-2-6-7_vs_3-5

GM_Test

0.8106 (± 0.12)

0.7554 (± 0.12)

0.7604 (± 0.12)

0.8258 (± 0.11)

0.8070 (± 0.12)

0.7664 (± 0.11)

AUC_Test

0.8298 (± 0.10)

0.8149 (± 0.12)

0.7857 (± 0.12)

0.8203 (± 0.09)

0.8277 (± 0.11)

0.7927 (± 0.10)

ecoli-0-3-4-6_vs_5

GM_Test

0.9193 (± 0.04)

0.9248 (± 0.09)

0.8618 (± 0.05)

0.9173 (± 0.05)

0.8929 (± 0.07)

0.8100 (± 0.10)

AUC_Test

0.9244 (± 0.04)

0.9110 (± 0.08)

0.8703 (± 0.03)

0.9113 (± 0.05)

0.9009 (± 0.06)

0.8283 (± 0.08)

ecoli-0-3-4-7_vs_5-6

GM_Test

0.8811 (± 0.08)

0.8491 (± 0.11)

0.8407 (± 0.10)

0.8642 (± 0.09)

0.8733 (± 0.08)

0.8264 (± 0.08)

AUC_Test

0.8885 (± 0.07)

0.8726 (± 0.09)

0.8516 (± 0.06)

0.8509 (± 0.08)

0.8798 (± 0.07)

0.8405 (± 0.07)

ecoli-0-3-4_vs_5

GM_Test

0.9028 (± 0.11)

0.8820 (± 0.10)

0.8692 (± 0.10)

0.8905 (± 0.12)

0.9019 (± 0.11)

0.9246 (± 0.05)

AUC_Test

0.9139 (± 0.09)

0.9014 (± 0.09)

0.8819 (± 0.09)

0.9060 (± 0.10)

0.9120 (± 0.09)

0.9277 (± 0.05)

ecoli-0-4-6_vs_5

GM_Test

0.9391 (± 0.04)

0.8665 (± 0.06)

0.8649 (± 0.07)

0.9083 (± 0.04)

0.9095 (± 0.04)

0.7772 (± 0.07)

AUC_Test

0.9417 (± 0.03)

0.9318 (± 0.07)

0.8721 (± 0.06)

0.9112 (± 0.04)

0.9121 (± 0.03)

0.7950 (± 0.05)

ecoli-0-6-7_vs_3-5

GM_Test

0.8420 (± 0.23)

0.8399 (± 0.23)

0.8269 (± 0.23)

0.8321 (± 0.21)

0.8248 (± 0.23)

0.8508 (± 0.20)

AUC_Test

0.8750 (± 0.17)

0.8771 (± 0.05)

0.8617 (± 0.05)

0.8629 (± 0.16)

0.8600 (± 0.16)

0.8800 (± 0.15)

ecoli-0-6-7_vs_5

GM_Test

0.8519 (± 0.10)

0.8600 (± 0.10)

0.8393 (± 0.10)

0.8415 (± 0.09)

0.8470 (± 0.11)

0.8012 (± 0.18)

AUC_Test

0.8646 (± 0.08)

0.8725 (± 0.11)

0.8538 (± 0.09)

0.8529 (± 0.08)

0.8600 (± 0.09)

0.8350 (± 0.14)

ecoli_0_1_3_7_vs_2_6

GM_Test

0.7318 (± 0.42)

0.7184 (± 0.42)

0.7184 (± 0.41)

0.7202 (± 0.42)

0.7285 (± 0.42)

0.5321 (± 0.44)

AUC_Test

0.8400 (± 0.21)

0.8409 (± 0.22)

0.8263 (± 0.20)

0.8281 (± 0.21)

0.8366 (± 0.21)

0.7390 (± 0.21)

ecoli_0_vs_1

GM_Test

0.9864 (± 0.01)

0.9756 (± 0.01)

0.9756 (± 0.01)

0.9751 (± 0.01)

0.9763 (± 0.01)

0.9795 (± 0.01)

AUC_Test

0.9874 (± 0.00)

0.9867 (± 0.01)

0.9760 (± 0.01)

0.9757 (± 0.01)

0.9767 (± 0.01)

0.9797 (± 0.01)

glass0

GM_Test

0.8491 (± 0.05)

0.8303 (± 0.04)

0.8208 (± 0.04)

0.8331 (± 0.02)

0.8369 (± 0.03)

0.7474 (± 0.10)

AUC_Test

0.8517 (± 0.04)

0.8540 (± 0.05)

0.8236 (± 0.02)

0.8354 (± 0.02)

0.8396 (± 0.03)

0.7596 (± 0.08)

glass1

GM_Test

0.8497 (± 0.02)

0.8297 (± 0.02)

0.8264 (± 0.03)

0.8347 (± 0.02)

0.8352 (± 0.03)

0.7494 (± 0.04)

AUC_Test

0.8512 (± 0.02)

0.8331 (± 0.04)

0.8276 (± 0.05)

0.8373 (± 0.02)

0.8371 (± 0.03)

0.7543 (± 0.04)

glass2

GM_Test

0.7874 (± 0.17)

0.7236 (± 0.18)

0.6442 (± 0.22)

0.7860 (± 0.13)

0.7278 (± 0.15)

0.5639 (± 0.29)

AUC_Test

0.8485 (± 0.12)

0.8364 (± 0.16)

0.7177 (± 0.16)

0.8160 (± 0.11)

0.7645 (± 0.12)

0.6892 (± 0.10)

glass4

GM_Test

0.8997 (± 0.13)

0.8823 (± 0.12)

0.8474 (± 0.18)

0.8965 (± 0.13)

0.8871 (± 0.16)

0.7459 (± 0.37)

AUC_Test

0.9147 (± 0.11)

0.9182 (± 0.19)

0.8751 (± 0.22)

0.9104 (± 0.11)

0.9068 (± 0.13)

0.8491 (± 0.18)

glass5

GM_Test

0.9425 (± 0.10)

0.9340 (± 0.12)

0.9340 (± 0.12)

0.9321 (± 0.10)

0.9336 (± 0.12)

0.9389 (± 0.11)

AUC_Test

0.8498 (± 0.09)

0.9127 (± 0.09)

0.9427 (± 0.09)

0.9394 (± 0.09)

0.9423 (± 0.10)

0.9475 (± 0.09)

glass6

GM_Test

0.9087 (± 0.04)

0.8712 (± 0.04)

0.8720 (± 0.04)

0.8889 (± 0.05)

0.8841 (± 0.06)

0.8615 (± 0.09)

AUC_Test

0.9122 (± 0.04)

0.8946 (± 0.03)

0.8784 (± 0.03)

0.8902 (± 0.04)

0.8894 (± 0.06)

0.8729 (± 0.08)

glass-0-1-4-6_vs_2

GM_Test

0.8414 (± 0.02)

0.7697 (± 0.02)

0.7507 (± 0.07)

0.8039 (± 0.06)

0.8066 (± 0.06)

0.5914 (± 0.09)

AUC_Test

0.8681 (± 0.02)

0.8053 (± 0.01)

0.7961 (± 0.06)

0.8155 (± 0.05)

0.8188 (± 0.05)

0.6626 (± 0.06)

glass-0-1-5_vs_2

GM_Test

0.7986 (± 0.22)

0.8036 (± 0.22)

0.6931 (± 0.16)

0.7999 (± 0.20)

0.7655 (± 0.20)

0.4098 (± 0.35)

AUC_Test

0.8157 (± 0.19)

0.8227 (± 0.16)

0.7292 (± 0.13)

0.8134 (± 0.18)

0.7812 (± 0.18)

0.6163 (± 0.15)

glass-0-4_vs_5

GM_Test

0.9940 (± 0.01)

0.9940 (± 0.01)

0.9940 (± 0.01)

0.9940 (± 0.01)

0.9940 (± 0.01)

0.9940 (± 0.01)

AUC_Test

0.9941 (± 0.01)

0.9941 (± 0.01)

0.9941 (± 0.01)

0.9941 (± 0.01)

0.9941 (± 0.01)

0.9941 (± 0.01)

glass-0-6_vs_5

GM_Test

0.9714 (± 0.11)

0.9535 (± 0.09)

0.9500 (± 0.07)

0.9731 (± 0.03)

0.9583 (± 0.07)

0.9949 (± 0.01)

AUC_Test

0.9789 (± 0.10)

0.9586 (± 0.04)

0.9547 (± 0.03)

0.9742 (± 0.03)

0.9630 (± 0.06)

0.9949 (± 0.01)

glass_0_1_6_vs_2

GM_Test

0.7510 (± 0.16)

0.7758 (± 0.13)

0.6480 (± 0.15)

0.7249 (± 0.14)

0.6872 (± 0.16)

0.3050 (± 0.25)

AUC_Test

0.8040 (± 0.12)

0.7510 (± 0.08)

0.6850 (± 0.05)

0.7581 (± 0.10)

0.7292 (± 0.13)

0.5461 (± 0.07)

glass_0_1_6_vs_5

GM_Test

0.9317 (± 0.11)

0.8654 (± 0.11)

0.8759 (± 0.16)

0.9199 (± 0.11)

0.9033 (± 0.13)

0.7942 (± 0.39)

AUC_Test

0.9395 (± 0.09)

0.9083 (± 0.10)

0.8914 (± 0.10)

0.9298 (± 0.10)

0.9150 (± 0.11)

0.8942 (± 0.19)

Haberman

GM_Test

0.6704 (± 0.05)

0.6708 (± 0.05)

0.6541 (± 0.05)

0.6699 (± 0.05)

0.6646 (± 0.04)

0.5205 (± 0.07)

AUC_Test

0.6747 (± 0.05)

0.6757 (± 0.09)

0.6588 (± 0.09)

0.6701 (± 0.08)

0.6698 (± 0.04)

0.5510 (± 0.05)

led7digit-0-2-4-5-6-7-8-9_vs_1

GM_Test

0.9025 (± 0.08)

0.8996 (± 0.08)

0.8807 (± 0.09)

0.8950 (± 0.08)

0.8960 (± 0.07)

0.8598 (± 0.09)

AUC_Test

0.9051 (± 0.07)

0.8936 (± 0.08)

0.8857 (± 0.09)

0.8976 (± 0.07)

0.8979 (± 0.07)

0.8647 (± 0.08)

new_thyroid1

GM_Test

0.9779 (± 0.03)

0.9551 (± 0.02)

0.9447 (± 0.05)

0.9640 (± 0.02)

0.9554 (± 0.04)

0.9595 (± 0.03)

AUC_Test

0.9685 (± 0.03)

0.9614 (± 0.02)

0.9456 (± 0.03)

0.9646 (± 0.02)

0.9562 (± 0.04)

0.9603 (± 0.03)

new_thyroid2

GM_Test

0.9831 (± 0.03)

0.9645 (± 0.03)

0.9645 (± 0.04)

0.9799 (± 0.01)

0.9724 (± 0.03)

0.9526 (± 0.03)

AUC_Test

0.9736 (± 0.03)

0.9746 (± 0.02)

0.9654 (± 0.03)

0.9806 (± 0.01)

0.9731 (± 0.03)

0.9543 (± 0.03)

page_blocks0

GM_Test

0.9550 (± 0.01)

0.9547 (± 0.01)

0.9562 (± 0.00)

0.9560 (± 0.00)

0.9556 (± 0.00)

0.9312 (± 0.01)

AUC_Test

0.9553 (± 0.01)

0.9550 (± 0.00)

0.9564 (± 0.00)

0.9562 (± 0.00)

0.9558 (± 0.00)

0.9324 (± 0.01)

page_blocks_1_3_vs_4

GM_Test

0.9989 (± 0.00)

0.9989 (± 0.00)

0.9989 (± 0.00)

0.9989 (± 0.00)

0.9989 (± 0.00)

0.9977 (± 0.00)

AUC_Test

0.9989 (± 0.00)

0.9989 (± 0.00)

0.9989 (± 0.00)

0.9989 (± 0.00)

0.9989 (± 0.00)

0.9977 (± 0.00)

Pima

GM_Test

0.7549 (± 0.01)

0.7559 (± 0.01)

0.7561 (± 0.01)

0.7543 (± 0.01)

0.7544 (± 0.01)

0.6948 (± 0.03)

AUC_Test

0.7558 (± 0.01)

0.7519 (± 0.01)

0.7572 (± 0.01)

0.7556 (± 0.01)

0.7555 (± 0.01)

0.7014 (± 0.02)

poker-8-9_vs_6

GM_Test

0.7599 (± 0.09)

0.7210 (± 0.08)

0.7334 (± 0.08)

0.7253 (± 0.09)

0.7219 (± 0.09)

0.2643 (± 0.21)

AUC_Test

0.7563 (± 0.08)

0.7330 (± 0.03)

0.7390 (± 0.03)

0.7322 (± 0.08)

0.7290 (± 0.08)

0.5456 (± 0.04)

poker-8_vs_6

GM_Test

0.9630 (± 0.05)

0. 9571 (± 0.12)

0.9694 (± 0.05)

0.9486 (± 0.07)

0.9489 (± 0.04)

0.1140 (± 0.22)

AUC_Test

0.9663 (± 0.04)

0.9275 (± 0.07)

0.9722 (± 0.05)

0.9549 (± 0.06)

0.7517 (± 0.03)

0.5230 (± 0.06)

poker-9_vs_7

GM_Test

0.6032 (± 0.49)

0.5261 (± 0.46)

0.4749 (± 0.38)

0.5832 (± 0.39)

0.5229 (± 0.43)

0.1414 (± 0.28)

AUC_Test

0.7971 (± 0.23)

0.7614 (± 0.15)

0.7023 (± 0.14)

0.7607 (± 0.20)

0.7197 (± 0.20)

0.5457 (± 0.10)

segment0

GM_Test

0.9863 (± 0.00)

0.9869 (± 0.00)

0.9846 (± 0.00)

0.9878 (± 0.00)

0.9874 (± 0.00)

0.9900 (± 0.00)

AUC_Test

0.9863 (± 0.00)

0.9869 (± 0.00)

0.9847 (± 0.00)

0.9879 (± 0.00)

0.9875 (± 0.00)

0.9901 (± 0.00)

shuttle-2_vs_5

GM_Test

1.0000 (± 0.00)

1.0000 (± 0.00)

1.0000 (± 0.00)

1.0000 (± 0.00)

1.0000 (± 0.00)

1.0000 (± 0.00)

AUC_Test

1.0000 (± 0.00)

1.0000 (± 0.00)

1.0000 (± 0.00)

1.0000 (± 0.00)

1.0000 (± 0.00)

1.0000 (± 0.00)

shuttle-6_vs_2-3

GM_Test

1.0000 (± 0.00)

1.0000 (± 0.00)

1.0000 (± 0.00)

1.0000 (± 0.00)

1.0000 (± 0.00)

0.9977 (± 0.00)

AUC_Test

1.0000 (± 0.00)

1.0000 (± 0.00)

1.0000 (± 0.00)

1.0000 (± 0.00)

1.0000 (± 0.00)

0.9977 (± 0.00)

shuttle_c0_vs_c4

GM_Test

1.0000 (± 0.00)

1.0000 (± 0.00)

1.0000 (± 0.00)

1.0000 (± 0.00)

1.0000 (± 0.00)

0.9997 (± 0.00)

AUC_Test

1.0000 (± 0.00)

1.0000 (± 0.00)

1.0000 (± 0.00)

1.0000 (± 0.00)

1.0000 (± 0.00)

0.9997 (± 0.00)

shuttle_c2_vs_c4

GM_Test

0.9414 (± 0.13)

0.9414 (± 0.13)

0.9414 (± 0.13)

0.9414 (± 0.13)

0.9414 (± 0.13)

1.0000 (± 0.00)

AUC_Test

0.9500 (± 0.11)

0.9500 (± 0.01)

0.9500 (± 0.01)

0.9500 (± 0.01)

0.9500 (± 0.01)

1.0000 (± 0.00)

vehicle0

GM_Test

0.9634 (± 0.02)

0.9617 (± 0.03)

0.9622 (± 0.03)

0.9624 (± 0.02)

0.9627 (± 0.02)

0.9189 (± 0.02)

AUC_Test

0.9655 (± 0.02)

0.9648 (± 0.01)

0.9627 (± 0.01)

0.9637 (± 0.02)

0.9630 (± 0.02)

0.9202 (± 0.02)

vehicle1

GM_Test

0.7652 (± 0.04)

0.7561 (± 0.03)

0.7611 (± 0.03)

0.7682 (± 0.03)

0.7562 (± 0.04)

0.6986 (± 0.04)

AUC_Test

0.7673 (± 0.04)

0.7584 (± 0.01)

0.7629 (± 0.01)

0.7699 (± 0.03)

0.7585 (± 0.03)

0.7131 (± 0.04)

vehicle2

GM_Test

0.9755 (± 0.01)

0.9934 (± 0.00)

0.9934 (± 0.00)

0.9739 (± 0.01)

0.9753 (± 0.01)

0.9439 (± 0.01)

AUC_Test

0.9756 (± 0.01)

0.9789 (± 0.01)

0.9746 (± 0.01)

0.9741 (± 0.01)

0.9755 (± 0.01)

0.9444 (± 0.01)

vehicle3

GM_Test

0.7899 (± 0.01)

0.7788 (± 0.01)

0.7775 (± 0.01)

0.7837 (± 0.00)

0.7730 (± 0.01)

0.7057 (± 0.00)

AUC_Test

0.7900 (± 0.01)

0.7789 (± 0.00)

0.7791 (± 0.00)

0.7848 (± 0.00)

0.7743 (± 0.01)

0.7166 (± 0.00)

vowel0

GM_Test

0.9807 (± 0.03)

0.9736 (± 0.03)

0.9664 (± 0.04)

0.9800 (± 0.02)

0.9768 (± 0.03)

0.9542 (± 0.05)

AUC_Test

0.9815 (± 0.03)

0.9720 (± 0.03)

0.9681 (± 0.04)

0.9806 (± 0.02)

0.9775 (± 0.03)

0.9566 (± 0.05)

winequality-red-4

GM_Test

0.6455 (± 0.02)

0.6438 (± 0.03)

0.6221 (± 0.03)

0.6355 (± 0.02)

0.6119 (± 0.04)

0.2294 (± 0.19)

AUC_Test

0.6844 (± 0.01)

0.6710 (± 0.03)

0.6773 (± 0.03)

0.6713 (± 0.01)

0.6713 (± 0.02)

0.5308 (± 0.03)

winequality-red-8_vs_6-7

GM_Test

0.4863 (± 0.24)

0.5596 (± 0.19)

0.5548 (± 0.19)

0.5484 (± 0.26)

0.5211 (± 0.14)

0.1969 (± 0.24)

AUC_Test

0.6555 (± 0.10)

0.6701 (± 0.07)

0.6701 (± 0.06)

0.6822 (± 0.10)

0.6817 (± 0.07)

0.5386 (± 0.05)

winequality-white-3-9_vs_5

GM_Test

0.5646 (± 0.15)

0.5706 (± 0.15)

0.5502 (± 0.14)

0.5537 (± 0.15)

0.5553 (± 0.14)

0.2660 (± 0.21)

AUC_Test

0.6633 (± 0.09)

0.6731 (± 0.06)

0.6525 (± 0.06)

0.6560 (± 0.09)

0.6567 (± 0.09)

0.5490 (± 0.05)

winequality-white-3_vs_7

GM_Test

0.5976 (± 0.33)

0.3909 (± 0.20)

0.3965 (± 0.20)

0.4253 (± 0.35)

0.5599 (± 0.33)

0.2971 (± 0.24)

AUC_Test

0.7197 (± 0.13)

0.6291 (± 0.07)

0.6205 (± 0.07)

0.6431 (± 0.15)

0.7042 (± 0.13)

0.5676 (± 0.05)

winequality-white-9_vs_4

GM_Test

0.7430 (± 0.42)

0.1156 (± 0.30)

0.2121 (± 0.48)

0.5434 (± 0.43)

0.6090 (± 0.42)

0.5840 (± 0.47)

AUC_Test

0.8503 (± 0.22)

0.7724 (± 0.20)

0.6364 (± 0.22)

0.8471 (± 0.22)

0.8344 (± 0.22)

0.7844 (± 0.23)

Wisconsin

GM_Test

0.9613 (± 0.01)

0.9607 (± 0.01)

0.9772 (± 0.02)

0.9638 (± 0.01)

0.9604 (± 0.01)

0.9532 (± 0.02)

AUC_Test

0.9614 (± 0.01)

0.9677 (± 0.01)

0.9567 (± 0.01)

0.9639 (± 0.01)

0.9604 (± 0.01)

0.9534 (± 0.00)

yeast1

GM_Test

0.7304 (± 0.01)

0.7258 (± 0.01)

0.7251 (± 0.01)

0.7289 (± 0.01)

0.7264 (± 0.00)

0.6925 (± 0.02)

AUC_Test

0.7378 (± 0.01)

0.7281 (± 0.01)

0.7274 (± 0.01)

0.7319 (± 0.01)

0.7284 (± 0.00)

0.6980 (± 0.02)

yeast3

GM_Test

0.9317 (± 0.02)

0.9268 (± 0.02)

0.9288 (± 0.02)

0.9291 (± 0.02)

0.9221 (± 0.02)

0.8853 (± 0.04)

AUC_Test

0.9289 (± 0.02)

0.9273 (± 0.01)

0.9290 (± 0.01)

0.9279 (± 0.02)

0.9223 (± 0.02)

0.8884 (± 0.03)

yeast4

GM_Test

0.7621 (± 0.08)

0.7760 (± 0.07)

0.7705 (± 0.08)

0.7788 (± 0.07)

0.7636 (± 0.07)

0.6275 (± 0.09)

AUC_Test

0.7803 (± 0.06)

0.7858 (± 0.04)

0.7854 (± 0.04)

0.7972 (± 0.05)

0.7859 (± 0.05)

0.6929 (± 0.06)

yeast5

GM_Test

0.9401 (± 0.03)

0.9367 (± 0.03)

0.9254 (± 0.04)

0.9364 (± 0.04)

0.9342 (± 0.04)

0.8722 (± 0.07)

AUC_Test

0.9418 (± 0.03)

0.9389 (± 0.05)

0.9287 (± 0.05)

0.9388 (± 0.04)

0.9366 (± 0.04)

0.8815 (± 0.05)

yeast6

GM_Test

0.8212 (± 0.11)

0.8079 (± 0.13)

0.7975 (± 0.15)

0.8086 (± 0.13)

0.8031 (± 0.12)

0.7771 (± 0.08)

AUC_Test

0.8394 (± 0.10)

0.8095 (± 0.07)

0.8241 (± 0.07)

0.8292 (± 0.11)

0.8218 (± 0.10)

0.8015 (± 0.07)

yeast-0-2-5-6_vs_3-7-8-9

GM_Test

0.8286 (± 0.03)

0.8228 (± 0.01)

0.8263 (± 0.01)

0.8263 (± 0.03)

0.8205 (± 0.04)

0.7346 (± 0.04)

AUC_Test

0.8330 (± 0.03)

0.8285 (± 0.03)

0.8282 (± 0.03)

0.8312 (± 0.04)

0.8257 (± 0.04)

0.7550 (± 0.03)

yeast-0-2-5-7-9_vs_3-6-8

GM_Test

0.9228 (± 0.04)

0.9222 (± 0.03)

0.9241 (± 0.04)

0.9221 (± 0.03)

0.9170 (± 0.03)

0.9004 (± 0.05)

AUC_Test

0.9247 (± 0.03)

0.9219 (± 0.03)

0.9259 (± 0.03)

0.9239 (± 0.03)

0.9192 (± 0.03)

0.9042 (± 0.04)

yeast-0-3-5-9_vs_7-8

GM_Test

0.7836 (± 0.08)

0.7866 (± 0.07)

0.7695 (± 0.09)

0.7791 (± 0.08)

0.7796 (± 0.08)

0.5689 (± 0.08)

AUC_Test

0.7922 (± 0.07)

0.7963 (± 0.02)

0.7813 (± 0.04)

0.7873 (± 0.07)

0.7902 (± 0.07)

0.6394 (± 0.05)

yeast_0_5_6_7_9_vs_4

GM_Test

0.7921 (± 0.09)

0.7905 (± 0.09)

0.7875 (± 0.09)

0.7887 (± 0.11)

0.7891 (± 0.09)

0.6210 (± 0.06)

AUC_Test

0.8098 (± 0.08)

0.8093 (± 0.09)

0.8006 (± 0.09)

0.8031 (± 0.09)

0.8012 (± 0.08)

0.6693 (± 0.04)

yeast_1_2_8_9_vs_7

GM_Test

0.6971 (± 0.15)

0.6820 (± 0.08)

0.6736 (± 0.15)

0.6880 (± 0.15)

0.6841 (± 0.15)

0.5856 (± 0.10)

AUC_Test

0.7309 (± 0.09)

0.7246 (± 0.06)

0.7259 (± 0.06)

0.7344 (± 0.10)

0.7306 (± 0.10)

0.6669 (± 0.06)

yeast_1_4_5_8_vs_7

GM_Test

0.6351 (± 0.10)

0.6447 (± 0.09)

0.6622 (± 0.09)

0.6487 (± 0.11)

0.6323 (± 0.12)

0.4099 (± 0.23)

AUC_Test

0.6841 (± 0.07)

0.6893 (± 0.05)

0.7005 (± 0.05)

0.6925 (± 0.08)

0.6832 (± 0.08)

0.5872 (± 0.09)

yeast_1_vs_7

GM_Test

0.7742 (± 0.06)

0.7842 (± 0.06)

0.7862 (± 0.06)

0.7705 (± 0.03)

0.7784 (± 0.08)

0.5510 (± 0.09)

AUC_Test

0.7818 (± 0.05)

0.7996 (± 0.07)

0.8019 (± 0.07)

0.7891 (± 0.06)

0.7930 (± 0.06)

0.6352 (± 0.05)

yeast_2_vs_4

GM_Test

0.9333 (± 0.04)

0.9462 (± 0.03)

0.8803 (± 0.06)

0.9352 (± 0.04)

0.9325 (± 0.04)

0.7971 (± 0.05)

AUC_Test

0.9348 (± 0.04)

0.8999 (± 0.04)

0.8856 (± 0.05)

0.9363 (± 0.04)

0.9337 (± 0.04)

0.8131 (± 0.04)

yeast_2_vs_8

GM_Test

0.7968 (± 0.12)

0.7841 (± 0.14)

0.7914 (± 0.12)

0.7947 (± 0.11)

0.7753 (± 0.14)

0.7460 (± 0.16)

AUC_Test

0.8178 (± 0.10)

0.8031 (± 0.10)

0.8156 (± 0.10)

0.8187 (± 0.09)

0.8069 (± 0.11)

0.7880 (± 0.12)

Average

GM_Test

0.8489

0.8264

0.8133

0.8392

0.8355

0.7220

AUC_Test

0.8687

0.8477

0.8447

0.8631

0.8567

0.7925

Data sets

Evaluation measures

ADASYN [29]

SMOTE+TL [35]

SMOTE+ENN [24]

RUS [20]

OSS [23]

NCL [25]

Abalone9_18

GM_Test

0.6528 (± 0.15)

0.6710 (± 0.14)

0.7442 (± 0.13)

0.6817 (± 0.09)

0.5100 (± 0.12)

0.6296 (± 0.15)

AUC_Test

0.6924 (± 0.12)

0.6986 (± 0.11)

0.7606 (± 0.11)

0.6892 (± 0.09)

0.6144 (± 0.08)

0.7021 (± 0.09)

Abalone19

GM_Test

0.2341 (± 0.19)

0.4498 (± 0.07)

0.4174 (± 0.05)

0.6104 (± 0.09)

0.0620 (± 0.12)

0.0000 (± 0.00)

AUC_Test

0.5286 (± 0.03)

0.5850 (± 0.03)

0.5694 (± 0.02)

0.6247 (± 0.09)

0.4815 (± 0.03)

0.5000 (± 0.00)

cleveland-0_vs_4

GM_Test

0.7487 (± 0.18)

0.7339 (± 0.19)

0.7449 (± 0.19)

0.7005 (± 0.06)

0.6117 (± 0.32)

0.3087 (± 0.40)

AUC_Test

0.7886 (± 0.14)

0.7729 (± 0.16)

0.7792 (± 0.16)

0.7122 (± 0.06)

0.7107 (± 0.16)

0.6082 (± 0.19)

Dermatology6

GM_Test

0.9644 (± 0.05)

0.9644 (± 0.05)

0.9687 (± 0.05)

0.9644 (± 0.05)

0.9704 (± 0.05)

0.9985 (± 0.00)

AUC_Test

0.9661 (± 0.04)

0.9661 (± 0.05)

0.9705 (± 0.04)

0.9661 (± 0.05)

0.9720 (± 0.05)

0.9985 (± 0.00)

ecoli1

GM_Test

0.8833 (± 0.04)

0.8858 (± 0.03)

0.8695 (± 0.02)

0.8803 (± 0.03)

0.8859 (± 0.04)

0.8674 (± 0.04)

AUC_Test

0.8879 (± 0.03)

0.8878 (± 0.03)

0.8720 (± 0.02)

0.8852 (± 0.03)

0.8889 (± 0.04)

0.8700 (± 0.03)

ecoli2

GM_Test

0.9016 (± 0.05)

0.8931 (± 0.04)

0.8755 (± 0.05)

0.8668 (± 0.03)

0.8815 (± 0.04)

0.8697 (± 0.07)

AUC_Test

0.9022 (± 0.05)

0.8943 (± 0.04)

0.8770 (± 0.05)

0.8693 (± 0.03)

0.8839 (± 0.04)

0.8778 (± 0.06)

ecoli3

GM_Test

0.8807 (± 0.02)

0.8474 (± 0.06)

0.8315 (± 0.05)

0.8410 (± 0.01)

0.7474 (± 0.06)

0.8138 (± 0.03)

AUC_Test

0.8814 (± 0.03)

0.8501 (± 0.02)

0.8375 (± 0.04)

0.8453 (± 0.01)

0.7599 (± 0.04)

0.8238 (± 0.03)

ecoli4

GM_Test

0.8662 (± 0.10)

0.8606 (± 0.13)

0.9205 (± 0.06)

0.8557 (± 0.03)

0.8345 (± 0.11)

0.7782 (± 0.16)

AUC_Test

0.8762 (± 0.09)

0.8762 (± 0.11)

0.9230 (± 0.06)

0.8612 (± 0.03)

0.8414 (± 0.10)

0.8139 (± 0.12)

ecoli-0-1-4-6_vs_5

GM_Test

0.8293 (± 0.13)

0.7889 (± 0.17)

0.7866 (± 0.18)

0.7551 (± 0.12)

0.7554 (± 0.15)

0.7703 (± 0.16)

AUC_Test

0.8442 (± 0.11)

0.8211 (± 0.13)

0.8173 (± 0.14)

0.7596 (± 0.12)

0.7903 (± 0.11)

0.8057 (± 0.05)

ecoli-0-1-4-7_vs_2-3-5-6

GM_Test

0.8325 (± 0.06)

0.8867 (± 0.04)

0.8785 (± 0.07)

0.7411 (± 0.05)

0.8615 (± 0.04)

0.8026 (± 0.06)

AUC_Test

0.8393 (± 0.06)

0.8910 (± 0.04)

0.8840 (± 0.06)

0.7527 (± 0.05)

0.8655 (± 0.03)

0.8201 (± 0.05)

ecoli-0-1-4-7_vs_5-6

GM_Test

0.8531 (± 0.09)

0.8546 (± 0.06)

0.8087 (± 0.05)

0.8039 (± 0.07)

0.8178 (± 0.12)

0.8084 (± 0.12)

AUC_Test

0.8608 (± 0.08)

0.8607 (± 0.06)

0.8175 (± 0.04)

0.8069 (± 0.07)

0.8343 (± 0.10)

0.8318 (± 0.10)

ecoli-0-1_vs_2-3-5

GM_Test

0.8428 (± 0.08)

0.8632 (± 0.18)

0.8145 (± 0.17)

0.7145 (± 0.16)

0.8748 (± 0.06)

0.8632 (± 0.06)

AUC_Test

0.8513 (± 0.07)

0.8890 (± 0.13)

0.8445 (± 0.13)

0.7518 (± 0.12)

0.8759 (± 0.06)

0.8699 (± 0.06)

ecoli-0-1_vs_5

GM_Test

0.7977 (± 0.10)

0.8514 (± 0.13)

0.8497 (± 0.13)

0.8063 (± 0.12)

0.7783 (± 0.17)

0.7769 (± 0.17)

AUC_Test

0.8136 (± 0.08)

0.8659 (± 0.11)

0.8636 (± 0.12)

0.8113 (± 0.11)

0.8090 (± 0.13)

0.8113 (± 0.12)

ecoli-0-2-3-4_vs_5

GM_Test

0.8597 (± 0.10)

0.9121 (± 0.11)

0.8812 (± 0.11)

0.8314 (± 0.13)

0.8548 (± 0.10)

0.7898 (± 0.08)

AUC_Test

0.8696 (± 0.09)

0.9196 (± 0.09)

0.8892 (± 0.10)

0.8418 (± 0.12)

0.8670 (± 0.08)

0.8085 (± 0.07)

ecoli-0-2-6-7_vs_3-5

GM_Test

0.7637 (± 0.11)

0.7995 (± 0.09)

0.7761 (± 0.10)

0.7706 (± 0.09)

0.7717 (± 0.16)

0.8190 (± 0.14)

AUC_Test

0.7828 (± 0.10)

0.8104 (± 0.08)

0.7929 (± 0.09)

0.7877 (± 0.08)

0.8053 (± 0.11)

0.8426 (± 0.12)

ecoli-0-3-4-6_vs_5

GM_Test

0.8458 (± 0.08)

0.8344 (± 0.08)

0.9047 (± 0.06)

0.9099 (± 0.06)

0.7973 (± 0.11)

0.7886 (± 0.07)

AUC_Test

0.8567 (± 0.07)

0.8452 (± 0.07)

0.9067 (± 0.06)

0.9121 (± 0.06)

0.8121 (± 0.10)

0.8087 (± 0.05)

ecoli-0-3-4-7_vs_5-6

GM_Test

0.8023 (± 0.11)

0.8619 (± 0.11)

0.8345 (± 0.19)

0.8664 (± 0.06)

0.8638 (± 0.08)

0.7614 (± 0.08)

AUC_Test

0.8189 (± 0.09)

0.8772 (± 0.09)

0.8658 (± 0.13)

0.8731 (± 0.06)

0.8684 (± 0.07)

0.7892 (± 0.06)

ecoli-0-3-4_vs_5

GM_Test

0.8207 (± 0.13)

0.9518 (± 0.06)

0.9042 (± 0.06)

0.8817 (± 0.05)

0.8124 (± 0.16)

0.7740 (± 0.16)

AUC_Test

0.8333 (± 0.12)

0.9527 (± 0.05)

0.9083 (± 0.06)

0.8861 (± 0.05)

0.8388 (± 0.12)

0.8083 (± 0.12)

ecoli-0-4-6_vs_5

GM_Test

0.8382 (± 0.18)

0.8966 (± 0.07)

0.8995 (± 0.08)

0.8767 (± 0.08)

0.8339 (± 0.08)

0.8178 (± 0.10)

AUC_Test

0.8592 (± 0.15)

0.8983 (± 0.07)

0.9010 (± 0.07)

0.8870 (± 0.07)

0.8451 (± 0.07)

0.8364 (± 0.09)

ecoli-0-6-7_vs_3-5

GM_Test

0.7201 (± 0.36)

0.8304 (± 0.20)

0.7980 (± 0.19)

0.8292 (± 0.11)

0.7134 (± 0.36)

0.8621 (± 0.21)

AUC_Test

0.8150 (± 0.18)

0.8574 (± 0.15)

0.8250 (± 0.14)

0.8450 (± 0.09)

0.8125 (± 0.17)

0.8925 (± 0.15)

ecoli-0-6-7_vs_5

GM_Test

0.8058 (± 0.11)

0.6944 (± 0.15)

0.6999 (± 0.15)

0.7735 (± 0.11)

0.7709 (± 0.16)

0.8859 (± 0.05)

AUC_Test

0.8200 (± 0.10)

0.7224 (± 0.12)

0.7275 (± 0.13)

0.7825 (± 0.11)

0.7975 (± 0.13)

0.8925 (± 0.04)

ecoli_0_1_3_7_vs_2_6

GM_Test

0.3249 (± 0.40)

0.5233 (± 0.43)

0.5143 (± 0.42)

0.7588 (± 0.07)

0.3902 (± 0.32)

0.7401 (± 0.38)

AUC_Test

0.6226 (± 0.18)

0.7226 (± 0.21)

0.7172 (± 0.20)

0.7899 (± 0.05)

0.5428 (± 0.21)

0.8481 (± 0.20)

ecoli_0_vs_1

GM_Test

0.9578 (± 0.01)

0.9758 (± 0.02)

0.9830 (± 0.02)

0.9794 (± 0.01)

0.9607 (± 0.03)

0.9576 (± 0.02)

AUC_Test

0.9587 (± 0.01)

0.9760 (± 0.02)

0.9832 (± 0.02)

0.9796 (± 0.01)

0.9617 (± 0.03)

0.9585 (± 0.02)

glass0

GM_Test

0.7845 (± 0.10)

0.7885 (± 0.06)

0.8000 (± 0.04)

0.8130 (± 0.06)

0.7538 (± 0.06)

0.7769 (± 0.05)

AUC_Test

0.7884 (± 0.01)

0.7972 (± 0.06)

0.8076 (± 0.05)

0.8205 (± 0.06)

0.7661 (± 0.06)

0.7865 (± 0.04)

glass1

GM_Test

0.7235 (± 0.02)

0.7533 (± 0.06)

0.7109 (± 0.05)

0.7208 (± 0.06)

0.6941 (± 0.06)

0.6609 (± 0.07)

AUC_Test

0.7271 (± 0.02)

0.7581 (± 0.06)

0.7176 (± 0.05)

0.7259 (± 0.06)

0.7134 (± 0.03)

0.6712 (± 0.07)

glass2

GM_Test

0.7071 (± 0.09)

0.4806 (± 0.39)

0.6727 (± 0.34)

0.6154 (± 0.08)

0.6838 (± 0.17)

0.3381 (± 0.29)

AUC_Test

0.7304 (± 0.07)

0.6455 (± 0.20)

0.7556 (± 0.18)

0.6663 (± 0.05)

0.7185 (± 0.14)

0.5557 (± 0.11)

glass4

GM_Test

0.7288 (± 0.37)

0.9072 (± 0.10)

0.9072 (± 0.10)

0.8271 (± 0.13)

0.9115 (± 0.10)

0.6522 (± 0.35)

AUC_Test

0.8243 (± 0.19)

0.9151 (± 0.09)

0.9151 (± 0.09)

0.8346 (± 0.12)

0.9200 (± 0.09)

0.7699 (± 0.18)

glass5

GM_Test

0.7802 (± 0.39)

0.9699 (± 0.02)

0.9189 (± 0.10)

0.9338 (± 0.03)

0.7900 (± 0.39)

0.9975 (± 0.00)

AUC_Test

0.8804 (± 0.19)

0.9707 (± 0.02)

0.9280 (± 0.09)

0.9365 (± 0.03)

0.8902 (± 0.19)

0.9975 (± 0.00)

glass6

GM_Test

0.8436 (± 0.05)

0.8709 (± 0.07)

0.9135 (± 0.07)

0.8808 (± 0.05)

0.8011 (± 0.03)

0.8910 (± 0.05)

AUC_Test

0.8527 (± 0.05)

0.8798 (± 0.06)

0.9171 (± 0.06)

0.8824 (± 0.05)

0.8153 (± 0.02)

0.8950 (± 0.05)

glass-0-1-4-6_vs_2

GM_Test

0.6944 (± 0.12)

0.7523 (± 0.15)

0.7699 (± 0.15)

0.6063 (± 0.10)

0.5369 (± 0.28)

0.4059 (± 0.33)

AUC_Test

0.7114 (± 0.11)

0.7699 (± 0.14)

0.7913 (± 0.13)

0.6496 (± 0.11)

0.6406 (± 0.10)

0.6128 (± 0.13)

glass-0-1-5_vs_2

GM_Test

0.7063 (± 0.11)

0.6546 (± 0.14)

0.4817 (± 0.29)

0.5332 (± 0.27)

0.1831 (± 0.36)

0.4806 (± 0.39)

AUC_Test

0.7314 (± 0.09)

0.6787 (± 0.13)

0.6174 (± 0.15)

0.6110 (± 0.15)

0.5129 (± 0.21)

0.6599 (± 0.18)

glass-0-4_vs_5

GM_Test

0.9687 (± 0.02)

0.9634 (± 0.02)

0.9813 (± 0.01)

0.9940 (± 0.01)

0.994 0 (± 0.01)

0.9940 (± 0.01)

AUC_Test

0.9694 (± 0.01)

0.9643 (± 0.02)

0.9816 (± 0.01)

0.9941 (± 0.01)

0.9941 (± 0.01)

0.9941 (± 0.01)

glass-0-6_vs_5

GM_Test

0.9632 (± 0.04)

0.9794 (± 0.02)

0.9813 (± 0.01)

0.8979 (± 0.04)

0.9314 (± 0.04)

0.9949 (± 0.01)

AUC_Test

0.9650 (± 0.04)

0.9800 (± 0.02)

0.9800 (± 0.02)

0.9039 (± 0.03)

0.9349 (± 0.04)

0.9449 (± 0.01)

glass_0_1_6_vs_2

GM_Test

0.4748 (± 0.25)

0.5652 (± 0.32)

0.5076 (± 0.29)

0.6325 (± 0.12)

0.3077 (± 0.25)

0.5081 (± 0.28)

AUC_Test

0.5804 (± 0.10)

0.6702 (± 0.18)

0.6254 (± 0.16)

0.6500 (± 0.13)

0.5200 (± 0.06)

0.6288 (± 0.13)

glass_0_1_6_vs_5

GM_Test

0.9219 (± 0.11)

0.8651 (± 0.14)

0.9101 (± 0.10)

0.9405 (± 0.03)

0.9913 (± 0.00)

0.9942 (± 0.00)

AUC_Test

0.9300 (± 0.09)

0.8800 (± 0.12)

0.9185 (± 0.09)

0.9428 (± 0.02)

0.9914 (± 0.00)

0.9942 (± 0.00)

Haberman

GM_Test

0.6277 (± 0.09)

0.5886 (± 0.03)

0.6374 (± 0.04)

0.5711 (± 0.06)

0.6269 (± 0.04)

0.6141 (± 0.07)

AUC_Test

0.6374 (± 0.08)

0.5924 (± 0.02)

0.6490 (± 0.04)

0.6091 (± 0.04)

0.6360 (± 0.04)

0.6305 (± 0.06)

led7digit-0-2-4-5-6-7-8-9_vs_1

GM_Test

0.8660 (± 0.06)

0.8846 (± 0.08)

0.8591 (± 0.06)

0.8621 (± 0.09)

0.8730 (± 0.06)

0.8693 (± 0.07)

AUC_Test

0.8719 (± 0.05)

0.8883 (± 0.07)

0.8609 (± 0.06)

0.8636 (± 0.09)

0.8777 (± 0.05)

0.8775 (± 0.06)

new_thyroid1

GM_Test

0.9541 (± 0.02)

0.9571 (± 0.02)

0.9507 (± 0.03)

0.9136 (± 0.03)

0.9418 (± 0.04)

0.9377 (± 0.06)

AUC_Test

0.9551 (± 0.02)

0.9579 (± 0.02)

0.9519 (± 0.02)

0.9158 (± 0.02)

0.9436 (± 0.04)

0.9404 (± 0.05)

new_thyroid2

GM_Test

0.9716 (± 0.05)

0.9426 (± 0.03)

0.9161 (± 0.01)

0.9066 (± 0.06)

0.9368 (± 0.03)

0.9393 (± 0.03)

AUC_Test

0.9722 (± 0.01)

0.9440 (± 0.03)

0.9186 (± 0.01)

0.9071 (± 0.06)

0.9380 (± 0.03)

0.9404 (± 0.03)

page_blocks0

GM_Test

0.9314 (± 0.00)

0.9399 (± 0.00)

0.9374 (± 0.01)

0.9460 (± 0.00)

0.9380 (± 0.01)

0.9367 (± 0.00)

AUC_Test

0.9314 (± 0.00)

0.9401 (± 0.00)

0.9375 (± 0.01)

0.9461 (± 0.00)

0.9382 (± 0.01)

0.9372 (± 0.00)

page_blocks_1_3_vs_4

GM_Test

0.9794 (± 0.00)

0.9805 (± 0.01)

0.9863 (± 0.00)

0.9408 (± 0.04)

0.9840 (± 0.01)

0.9954 (± 0.00)

AUC_Test

0.9797 (± 0.00)

0.9808 (± 0.00)

0.9865 (± 0.00)

0.9435 (± 0.04)

0.9842 (± 0.01)

0.9955 (± 0.00)

Pima

GM_Test

0.7028 (± 0.03)

0.6893 (± 0.02)

0.7084 (± 0.04)

0.7229 (± 0.02)

0.6406 (± 0.05)

0.7034 (± 0.01)

AUC_Test

0.7048 (± 0.03)

0.6956 (± 0.03)

0.7115 (± 0.04)

0.7264 (± 0.01)

0.6653 (± 0.03)

0.7163 (± 0.01)

poker-8-9_vs_6

GM_Test

0.6145 (± 0.15)

0.4823 (± 0.39)

0.4836 (± 0.29)

0.6930 (± 0.17)

0.5400 (± 0.34)

0.0000 (± 0.00)

AUC_Test

0.6758 (± 0.09)

0.6631 (± 0.18)

0.6258 (± 0.14)

0.6986 (± 0.17)

0.6946 (± 0.18)

0.5000 (± 0.00)

poker-8_vs_6

GM_Test

0.4141 (± 0.21)

0.2453 (± 0.30)

0.1085 (± 0.21)

0.2589 (± 0.21)

0.0000 (± 0.00)

0.0000 (± 0.00)

AUC_Test

0.5742 (± 0.07)

0.5343 (± 0.11)

0.4747 (± 0.06)

0.4688 (± 0.06)

0.4948 (± 0.01)

0.5000 (± 0.00)

poker-9_vs_7

GM_Test

0.2766 (± 0.33)

0.5997 (± 0.32)

0.5997 (± 0.32)

0.4301 (± 0.36)

0.4219 (± 0.36)

0.1383 (± 0.27)

AUC_Test

0.5530 (± 0.14)

0.7139 (± 0.15)

0.7160 (± 0.15)

0.5952 (± 0.19)

0.6129 (± 0.17)

0.5393 (± 0.09)

segment0

GM_Test

0.9814 (± 0.00)

0.9898 (± 0.00)

0.9903 (± 0.00)

0.9786 (± 0.00)

0.9809 (± 0.00)

0.9824 (± 0.00)

AUC_Test

0.9815 (± 0.00)

0.9898 (± 0.00)

0.9903 (± 0.00)

0.9787 (± 0.00)

0.9810 (± 0.00)

0.9825 (± 0.00)

shuttle-2_vs_5

GM_Test

1.0000 (± 0.00)

0.9993 (± 0.00)

0.9993 (± 0.00)

0.9913 (± 0.01)

0.9846 (± 0.01)

1.0000 (± 0.00)

AUC_Test

1.0000 (± 0.00)

0.9993 (± 0.00)

0.9993 (± 0.00)

0.9915 (± 0.01)

0.9849 (± 0.01)

1.0000 (± 0.00)

shuttle-6_vs_2-3

GM_Test

0.9977 (± 0.00)

0.9977 (± 0.00)

0.9977 (± 0.00)

0.9237 (± 0.03)

0.9954 (± 0.00)

0.9931 (± 0.00)

AUC_Test

0.9977 (± 0.00)

0.9977 (± 0.00)

0.9977 (± 0.00)

0.9272 (± 0.03)

0.9954 (± 0.00)

0.9931 (± 0.00)

shuttle_c0_vs_c4

GM_Test

0.9988 (± 0.00)

0.9997 (± 0.00)

0.9997 (± 0.00)

1.0000 (± 0.00)

1.0000 (± 0.00)

0.9997 (± 0.00)

AUC_Test

0.9988 (± 0.00)

0.9997 (± 0.00)

1.0000 (± 0.00)

1.0000 (± 0.00)

1.0000 (± 0.00)

0.9997 (± 0.00)

shuttle_c2_vs_c4

GM_Test

0.9959 (± 0.00)

1.0000 (± 0.00)

1.0000 (± 0.00)

0.9833 (± 0.03)

1.0000 (± 0.00)

1.0000 (± 0.00)

AUC_Test

0.9960 (± 0.00)

1.0000 (± 0.00)

1.0000 (± 0.00)

0.9840 (± 0.03)

1.0000 (± 0.00)

1.0000 (± 0.00)

vehicle0

GM_Test

0.9388 (± 0.00)

0.9243 (± 0.01)

0.9279 (± 0.01)

0.9358 (± 0.01)

0.9230 (± 0.01)

0.9426 (± 0.01)

AUC_Test

0.9393 (± 0.00)

0.9248 (± 0.01)

0.9281 (± 0.01)

0.9363 (± 0.01)

0.9234 (± 0.01)

0.9432 (± 0.01)

vehicle1

GM_Test

0.6936 (± 0.03)

0.7557 (± 0.02)

0.7552 (± 0.02)

0.6968 (± 0.06)

0.7472 (± 0.03)

0.7363 (± 0.00)

AUC_Test

0.6979 (± 0.03)

0.7588 (± 0.03)

0.7541 (± 0.02)

0.7039 (± 0.06)

0.7517 (± 0.04)

0.7382 (± 0.00)

vehicle2

GM_Test

0.9391 (± 0.01)

0.9503 (± 0.00)

0.9521 (± 0.00)

0.9363 (± 0.01)

0.9415 (± 0.01)

0.9478 (± 0.01)

AUC_Test

0.9392 (± 0.01)

0.9507 (± 0.00)

0.9522 (± 0.00)

0.9368 (± 0.01)

0.9417 (± 0.01)

0.9480 (± 0.01)

vehicle3

GM_Test

0.7311 (± 0.01)

0.7298 (± 0.01)

0.7154 (± 0.01)

0.6994 (± 0.02)

0.7091 (± 0.02)

0.7642 (± 0.03)

AUC_Test

0.7313 (± 0.01)

0.7321 (± 0.01)

0.7158 (± 0.01)

0.7042 (± 0.02)

0.7155 (± 0.02)

0.7681 (± 0.03)

vowel0

GM_Test

0.9687 (± 0.01)

0.9877 (± 0.01)

0.9877 (± 0.01)

0.9429 (± 0.05)

0.9133 (± 0.08)

0.9558 (± 0.05)

AUC_Test

0.9688 (± 0.01)

0.9877 (± 0.01)

0.9877 (± 0.01)

0.9438 (± 0.04)

0.9188 (± 0.07)

0.9583 (± 0.05)

winequality-red-4

GM_Test

0.5207 (± 0.06)

0.5739 (± 0.08)

0.4023 (± 0.21)

0.5639 (± 0.03)

05994 (± 0.11)

0.3094 (± 0.17)

AUC_Test

0.6072 (± 0.03)

0.6274 (± 0.05)

0.5513 (± 0.07)

0.5823 (± 0.02)

0.4860 (± 0.01)

0.5548 (± 0.04)

winequality-red-8_vs_6-7

GM_Test

0.5467 (± 0.31)

0.6780 (± 0.16)

0.5619 (± 0.30)

0.5748 (± 0.06)

0.1023 (± 0.20)

0.0000 (± 0.00)

AUC_Test

0.6736 (± 0.16)

0.7206 (± 0.12)

0.6801 (± 0.14)

0.5956 (± 0.06)

0.5119 (± 0.02)

0.4880 (± 0.01)

winequality-white-3-9_vs_5

GM_Test

0.2095 (± 0.26)

0.3765 (± 0.19)

0.4155 (± 0.22)

0.5513 (± 0.11)

0.1170 (± 0.23)

0.3563 (± 0.17)

AUC_Test

0.5311 (± 0.07)

0.5553 (± 0.06)

0.5788 (± 0.08)

0.5692 (± 0.10)

0.5241 (± 0.05)

0.5755 (± 0.04)

winequality-white-3_vs_7

GM_Test

0.4267 (± 0.22)

0.4591 (± 0.26)

0.4620 (± 0.27)

0.5902 (± 0.07)

0.4392 (± 0.36)

0.0991 (± 0.19)

AUC_Test

0.5914 (± 0.07)

0.6198 (± 0.12)

0.6255 (± 0.12)

0.6136 (± 0.04)

0.6289 (± 0.15)

0.5198 (± 0.04)

winequality-white-9_vs_4

GM_Test

0.3771 (± 0.46)

0.5549 (± 0.45)

0.5549 (± 0.45)

0.7606 (± 0.16)

0.3704 (± 0.45)

0.8000 (± 0.39)

AUC_Test

0.6629 (± 0.23)

0.7475 (± 0.22)

0.7445 (± 0.22)

0.8020 (± 0.12)

0.6597 (± 0.22)

0.9000 (± 0.19)

Wisconsin

GM_Test

0.9512 (± 0.00)

0.9563 (± 0.01)

0.9393 (± 0.02)

0.9511 (± 0.01)

0.9079 (± 0.06)

0.9540 (± 0.01)

AUC_Test

0.9515 (± 0.00)

0.9563 (± 0.01)

0.9397 (± 0.01)

0.9512 (± 0.01)

0.9131 (± 0.05)

0.9541 (± 0.01)

yeast1

GM_Test

0.6919 (± 0.04)

0.6865 (± 0.02)

0.7064 (± 0.03)

0.7086 (± 0.03)

0.6570 (± 0.01)

0.6953 (± 0.00)

AUC_Test

0.6958 (± 0.04)

0.6820 (± 0.02)

0.7096 (± 0.02)

0.7126 (± 0.03)

0.6670 (± 0.01)

0.7059 (± 0.00)

yeast3

GM_Test

0.9107 (± 0.01)

0.9136 (± 0.03)

0.9068 (± 0.01)

0.9018 (± 0.03)

0.9207 (± 0.01)

0.9019 (± 0.01)

AUC_Test

0.9114 (± 0.01)

0.9139 (± 0.03)

0.9082 (± 0.00)

0.9022 (± 0.02)

0.9214 (± 0.01)

0.9035 (± 0.01)

yeast4

GM_Test

0.7094 (± 0.03)

0.7435 (± 0.09)

0.7695 (± 0.05)

0.8169 (± 0.05)

0.4683 (± 0.25)

0.6178 (± 0.15)

AUC_Test

0.7342 (± 0.02)

0.7644 (± 0.07)

0.7830 (± 0.04)

0.8194 (± 0.05)

0.6345 (± 0.09)

0.6966 (± 0.08)

yeast5

GM_Test

0.9254 (± 0.03)

0.9709 (± 0.02)

0.9719 (± 0.02)

0.9371 (± 0.03)

0.8956 (± 0.06)

0.8990 (± 0.02)

AUC_Test

0.9270 (± 0.03)

0.9711 (± 0.02)

0.9722 (± 0.02)

0.9375 (± 0.03)

0.9013 (± 0.05)

0.9034 (± 0.02)

yeast6

GM_Test

0.7797 (± 0.13)

0.8337 (± 0.09)

0.8334 (± 0.08)

0.8067 (± 0.07)

0.7383 (± 0.15)

0.7376 (± 0.13)

AUC_Test

0.8057 (± 0.11)

0.8455 (± 0.07)

0.8452 (± 0.07)

0.8113 (± 0.07)

0.7708 (± 0.11)

0.7784 (± 0.10)

yeast-0-2-5-6_vs_3-7-8-9

GM_Test

0.7016 (± 0.04)

0.7419 (± 0.05)

0.7387 (± 0.03)

0.7795 (± 0.03)

0.7194 (± 0.01)

0.7120 (± 0.07)

AUC_Test

0.7223 (± 0.04)

0.7551 (± 0.04)

0.7520 (± 0.03)

0.7878 (± 0.03)

0.7474 (± 0.00)

0.7452 (± 0.05)

yeast-0-2-5-7-9_vs_3-6-8

GM_Test

0.8984 (± 0.03)

0.9161 (± 0.03)

0.9094 (± 0.02)

0.8931 (± 0.02)

0.8766 (± 0.05)

0.8355 (± 0.03)

AUC_Test

0.8999 (± 0.03)

0.9179 (± 0.03)

0.9110 (± 0.02)

0.8963 (± 0.02)

0.8801 (± 0.05)

0.8465 (± 0.03)

yeast-0-3-5-9_vs_7-8

GM_Test

0.6822 (± 0.09)

0.7030 (± 0.09)

0.6759 (± 0.05)

0.6863 (± 0.05)

0.5219 (± 0.12)

0.5998 (± 0.07)

AUC_Test

0.6967 (± 0.08)

0.7141 (± 0.08)

0.6902 (± 0.04)

0.6896 (± 0.05)

0.6283 (± 0.04)

0.6593 (± 0.04)

yeast_0_5_6_7_9_vs_4

GM_Test

0.7036 (± 0.09)

0.7326 (± 0.11)

0.7295 (± 0.08)

0.7781 (± 0.06)

0.5906 (± 0.10)

0.7183 (± 0.09)

AUC_Test

0.7252 (± 0.08)

0.7505 (± 0.09)

0.7454 (± 0.06)

0.7857 (± 0.06)

0.6583 (± 0.06)

0.7480 (± 0.07)

yeast_1_2_8_9_vs_7

GM_Test

0.6319 (± 0.14)

0.4818 (± 0.07)

0.5379 (± 0.09)

0.4884 (± 0.25)

0.4182 (± 0.23)

0.1619 (± 0.19)

AUC_Test

0.6858 (± 0.09)

0.5804 (± 0.03)

0.6121 (± 0.05)

0.6050 (± 0.07)

0.6128 (± 0.08)

0.5273 (± 0.03)

yeast_1_4_5_8_vs_7

GM_Test

0.3359 (± 0.17)

0.4220 (± 0.22)

0.4523 (± 0.25)

0.5261 (± 0.09)

0.078 (± 0.15)

0.0000 (± 0.00)

AUC_Test

0.5192 (± 0.05)

0.5633 (± 0.07)

0.5868 (± 0.09)

0.5739 (± 0.07)

0.5098 (± 0.01)

0.4924 (± 0.01)

yeast_1_vs_7

GM_Test

0.6389 (± 0.09)

0.6842 (± 0.10)

0.6057 (± 0.17)

0.6531 (± 0.05)

0.5627 (± 0.08)

0.4764 (± 0.24)

AUC_Test

0.6754 (± 0.07)

0.7064 (± 0.08)

0.6567 (± 0.13)

0.6602 (± 0.05)

0.6379 (± 0.03)

0.6289 (± 0.07)

yeast_2_vs_4

GM_Test

0.8616 (± 0.05)

0.9164 (± 0.05)

0.8779 (± 0.05)

0.8824 (± 0.01)

0.8878 (± 0.06)

0.8592 (± 0.06)

AUC_Test

0.8652 (± 0.04)

0.9176 (± 0.04)

0.8827 (± 0.05)

0.8855 (± 0.01)

0.8918 (± 0.06)

0.8648 (± 0.05)

yeast_2_vs_8

GM_Test

0.5320 (± 0.28)

0.8277 (± 0.11)

0.8129 (± 0.19)

0.7028 (± 0.06)

0.7352 (± 0.12)

0.2722 (± 0.35)

AUC_Test

0.6534 (± 0.13)

0.8447 (± 0.09)

0.8436 (± 0.15)

0.7347 (± 0.06)

0.7623 (± 0.08)

0.5967 (± 0.14)

Average

GM_Test

0.7592

0.7893

0.7818

0.7864

0.7152

0.7018

AUC_Test

0.8040

0.8233

0.8195

0.8031

0.7845

0.7896

4.3 Non-parametric statistical test for statistical analysis

To evaluate HTSS algorithm statistically, Friedman test is employed [56]. To concisely explain this test, firstly, the competing algorithms are considered to be equivalent in terms of performance (null-hypothesis), and the main goal is to understand whether this hypothesis can be rejected or not. If yes, it means that a discrepancy exists among the algorithms; hence, the alternative hypothesis is accepted. p value is calculated to determine the discrepancy and if the p value is less than or equal to a specified significance level, the null hypothesis is rejected. In this experiment, \(\alpha = 0.05\) is the considered level of significance. To find out the controlling method, rankings are calculated at the end of this step and the method with lower rank is determined. At the second step, a post hoc procedure can be implemented with the goal of finding the differences between methods. We use Holm’s test as our post hoc procedure to calculate the adjusted p values [57].
Table 4

Average ranking obtained by Friedman test over the results of Table 2

Algorithm

Ranking

HTSS

2.72

BQIGSA + DM

3.94

BQIGSA + SW

4.58

BQIGSA + IM

4.76

BQIGSA + MU

5.91

SMOTE_TL

6.34

SMOTE_TL

6.78

RUS

7.61

ADASYN

8.36

ROS

8.88

NCL

8.89

OSS

9.18

4.4 Analysis of results

The average obtained classification G-mean as well as AUC on 30 runs of each algorithm over 75 test sets is reported in Table 3. The standard deviation over 5 folds is given in front of the results of each data set and best results are shown in boldface style. The Friedman test is then applied on the G-mean results of Table 3, and the ranking and p value results are shown in Tables 4 and 5, respectively. When analyzing the obtained results, it can be seen that by considering G-mean, HTSS is the winner algorithm for 34 data sets. Also, it has obtained the second and the third ranks for 14 and 18 data sets, respectively. Hence, HTSS has achieved the first to third ranks in 88% of the data sets. To explain the same for the AUC measure, it is observed that HTSS has gained the first, second and third ranks for 32, 21 and 9 data sets, respectively and, overall, the first to third ranks for 82% of them. It is worth to mention that in the cases where HTSS has not achieved the first rank, none of the other 11 algorithms is constantly the winner and the winning cases are divided between them. Furthermore, the best average G-mean and AUC results have been obtained for HTSS overall. After that, BQIGSA + SW, and BQIGSA + DM have resulted in the second and third average G-mean and AUC results, respectively. Moreover, it is seen that BQIGSA with local search algorithms has better ranks than other methods after HTSS. Table 4 demonstrates that the best ranking has attained for HTSS with the rank of 2.72. In the following, BQIGSA + SW and BQIGSA + DM stand on the second and third places, respectively, with the ranks of 3.94 and 4.58. By looking at Table 5, it is observed that the achieved p values also demonstrate the significant superiority of HTSS over the other algorithms resulting in p values less than 0.05 for them.
Table 5

Post hoc comparison for \(\upalpha = 0.05\)

I

Algorithm

p

Holm

1

BQIGSA + DM

0.039327

0.05

2

BQIGSA + SW

0.001583

0.025

3

BQIGSA + IM

0.000531

0.016667

4

BQIGSA + MU

0

0.0125

5

SMOTE_TL

0

0.01

6

SMOTE_TL

0

0.008333

7

RUS

0

0.007143

8

ADASYN

0

0.00625

9

ROS

0

0.005556

10

NCL

0

0.005

11

OSS

0

0.004545

As was mentioned, adding a local search to global search algorithm can improve its performance and increase the exploitation ability. Besides, by adding several local search algorithm to a pool and selecting dynamically from them during the search procedure, the flexibility of the algorithm will be increased which results in a controlled tradeoff between exploitation and exploration. Consequently, the search will be enhanced. From the presented results, it can be understood that constituting a hyper-heuristic algorithm can improve the performance of global search considerably in training set selection problem for imbalanced data sets.

5 Conclusion

In this paper, a novel hyper-heuristic algorithm for training set selection in imbalanced data sets was introduced with the goal of boosting the performance of the classifier. In designing HTSS, we used 4 local search algorithms and roulette wheel as the choice function. In our experiment, we test the proposed method on 75 imbalanced data sets taken from the KEEL Repository while being classified by C4.5 classifier. We compared HTSS with the state where each local search was utilized solely and then the results were compared to 7 other state of the art preprocessing methods. The experimental results demonstrated the best performance of HTSS among other algorithms which confirms the effectiveness of the proposed method. The results of non-parametric statistical tests prove it too.

References

  1. 1.
    Kubat, M., Holte, R.C., Matwin, S.: Machine learning for the detection of oil spills in satellite radar images. Mach. Learn. 30, 195–215 (1998)CrossRefGoogle Scholar
  2. 2.
    Cieslak, D.A., Chawla, N.V., Striegel, A.: Combating imbalance in network intrusion datasets. In: GrC, pp. 732–737 (2006)Google Scholar
  3. 3.
    Zhang, D., Islam, M.M., Lu, G.: A review on automatic image annotation techniques. Pattern Recogn. 45, 346–362 (2012)CrossRefGoogle Scholar
  4. 4.
    Cardie, C., Howe, N.: Improving minority class prediction using case-specific feature weights. In: ICML, pp. 57–65 (1997)Google Scholar
  5. 5.
    Burke, E., Kendall, G., Newall, J., Hart, E., Ross, P., Schulenburg, S.: Hyper-heuristics: an emerging direction in modern search technology. Handbook of Meta-Heuristics, pp. 457–474 (2003)Google Scholar
  6. 6.
    Nezamabadi-pour, H.: A quantum-inspired gravitational search algorithm for binary encoded optimization problems. Eng. Appl. Artif. Intell. 40, 62–75 (2015)CrossRefGoogle Scholar
  7. 7.
    García, S., Herrera, F.: Evolutionary undersampling for classification with imbalanced datasets: proposals and taxonomy. Evol. Comput. 17, 275–306 (2009)CrossRefGoogle Scholar
  8. 8.
    Gao, M., Hong, X., Chen, S., Harris, C.J.: A combined SMOTE and PSO based RBF classifier for two-class imbalanced problems. Neurocomputing 74, 3456–3466 (2011)CrossRefGoogle Scholar
  9. 9.
    Yu, D.-J., Hu, J., Tang, Z.-M., Shen, H.-B., Yang, J., Yang, J.-Y.: Improving protein-ATP binding residues prediction by boosting SVMs with random under-sampling. Neurocomputing 104, 180–190 (2013)CrossRefGoogle Scholar
  10. 10.
    Tomek, I.: Two modifications of CNN. IEEE Trans. Syst. Man Cybern. 6, 769–772 (1976)MathSciNetMATHGoogle Scholar
  11. 11.
    Hart, P.: The condensed nearest neighbor rule (Corresp.). IEEE Trans. Inf. Theory 14, 515–516 (1968)CrossRefGoogle Scholar
  12. 12.
    Kubat, M., Matwin, S.: Addressing the curse of imbalanced training sets: one-sided selection. In: ICML, pp. 179–186 (1997)Google Scholar
  13. 13.
    Wilson, D.L.: Asymptotic properties of nearest neighbor rules using edited data. IEEE Trans. Syst. Man Cybern. 2, 408–421 (1972)MathSciNetCrossRefMATHGoogle Scholar
  14. 14.
    Laurikkala, J.: Improving identification of difficult small classes by balancing class distribution. In: Quaglini, S., Barahona, P., Andreassen, S. (eds.) Artificial Intelligence in Medicine. Lecture Notes in Computer Science, vol. 2101, pp. 63–66. Springer, Berlin (2001)Google Scholar
  15. 15.
    García, S., Fernández, A., Herrera, F.: Enhancing the effectiveness and interpretability of decision tree and rule induction classifiers with evolutionary training set selection over imbalanced problems. Appl. Soft Comput. 9, 1304–1314 (2009)CrossRefGoogle Scholar
  16. 16.
    Garcı, S., Triguero, I., Carmona, C.J., Herrera, F.: Evolutionary-based selection of generalized instances for imbalanced classification. Knowl.-Based Syst. 25, 3–12 (2012)CrossRefGoogle Scholar
  17. 17.
    Jian, C., Gao, J., Ao, Y.: A new sampling method for classifying imbalanced data based on support vector machine ensemble. Neurocomputing 193, 115–122 (2016)CrossRefGoogle Scholar
  18. 18.
    Chen, S., He, H., Garcia, E.A.: RAMOBoost: ranked minority oversampling in boosting. IEEE Trans. Neural Netw. 21, 1624–1642 (2010)CrossRefGoogle Scholar
  19. 19.
    He, H., Bai, Y., Garcia, E.A., Li, S.: ADASYN: adaptive synthetic sampling approach for imbalanced learning. In: IEEE International Joint Conference on Neural Networks, 2008. IJCNN 2008 (IEEE World Congress on Computational Intelligence), pp. 1322–1328 (2008)Google Scholar
  20. 20.
    Chawla, N.V., Bowyer, K.W., Hall, L.O., Kegelmeyer, W.P.: SMOTE: synthetic minority over-sampling technique. J. Artif. Intell. Res. 16, 321–357 (2002)MATHGoogle Scholar
  21. 21.
    Hu, S., Liang, Y., Ma, L., He, Y.: MSMOTE: improving classification performance when training data is imbalanced. In: Second International Workshop on Computer Science and Engineering, 2009. WCSE’09, pp. 13–17 (2009)Google Scholar
  22. 22.
    Barua, S., Islam, M.M., Yao, X., Murase, K.: MWMOTE-majority weighted minority oversampling technique for imbalanced data set learning. IEEE Trans. Knowl. Data Eng. 26, 405–425 (2014)CrossRefGoogle Scholar
  23. 23.
    Gao, M., Hong, X., Chen, S., Harris, C.J., Khalaf, E.: PDFOS: PDF estimation based over-sampling for imbalanced two-class problems. Neurocomputing 138, 248–259 (2014)CrossRefGoogle Scholar
  24. 24.
    Ramentol, E., Caballero, Y., Bello, R., Herrera, F.: SMOTE-RSB*: a hybrid preprocessing approach based on oversampling and undersampling for high imbalanced data-sets using SMOTE and rough sets theory. Knowl. Inf. Syst. 33, 245–265 (2012)CrossRefGoogle Scholar
  25. 25.
    Batista, G.E., Prati, R.C., Monard, M.C.: A study of the behavior of several methods for balancing machine learning training data. ACM Sigkdd Explor. Newsl 6, 20–29 (2004)CrossRefGoogle Scholar
  26. 26.
    Han, H., Wang, W.-Y., Mao, B.-H.: Borderline-SMOTE: a new over-sampling method in imbalanced data sets learning. Advances in Intelligent Computing, pp. 878–887 (2005)Google Scholar
  27. 27.
    Bunkhumpornpat, C., Sinapiromsaran, K., Lursinsap, C.: Safe-level-smote: Safe-level-synthetic minority over-sampling technique for handling the class imbalanced problem. Advances in Knowledge Discovery and Data Mining, pp. 475–482 (2009)Google Scholar
  28. 28.
    Cateni, S., Colla, V., Vannucci, M.: A method for resampling imbalanced datasets in binary classification tasks for real-world problems. Neurocomputing 135, 32–41 (2014)CrossRefGoogle Scholar
  29. 29.
    Yang, C.-Y., Yang, J.-S., Wang, J.-J.: Margin calibration in SVM class-imbalanced learning. Neurocomputing 73, 397–411 (2009)CrossRefGoogle Scholar
  30. 30.
    Krawczyk, B., Woźniak, M., Schaefer, G.: Cost-sensitive decision tree ensembles for effective imbalanced classification. Appl. Soft Comput. 14, 554–562 (2014)CrossRefGoogle Scholar
  31. 31.
    Krawczyk, B., Woźniak, M.: Cost-sensitive neural network with roc-based moving threshold for imbalanced classification. In: International Conference on Intelligent Data Engineering and Automated Learning, pp. 45–52 (2015)Google Scholar
  32. 32.
    Zhu, Y., Wang, Z., Gao, D.: Gravitational fixed radius nearest neighbor for imbalanced problem. Knowl.-Based Syst. 90, 224–238 (2015)CrossRefGoogle Scholar
  33. 33.
    Nikpour, B., Shabani, M., Nezamabadi-pour, H.: Proposing new method to improve gravitational fixed nearest neighbor algorithm for imbalanced data classification. In: 2017 2nd Conference on Swarm Intelligence and Evolutionary Computation (CSIEC), pp. 6–11 (2017)Google Scholar
  34. 34.
    Shabani, M., Nikpour, B., Nezamabadi-pour, H.: An improvement to gravitational fixed radius nearest neighbor for imbalaced problem. Artificial Intelligence and Signal Processing (AISP) (2017)Google Scholar
  35. 35.
    Saryazdi, S., Nikpour, B., Nezamabadi-pour, H.: NPC: Neighbors Progressive Competition Algorithm for Classification of Imbalanced Data Sets, arXiv preprint arXiv:1711.10934 (2017)
  36. 36.
    Domingos, P.: Metacost: a general method for making classifiers cost-sensitive. In: Proceedings of the Fifth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 155–164 (1999)Google Scholar
  37. 37.
    Elkan, C.: The foundations of cost-sensitive learning. In: International Joint Conference on Artificial Intelligence, pp. 973–978 (2001)Google Scholar
  38. 38.
    Ting, K.M.: An instance-weighting method to induce cost-sensitive trees. IEEE Trans. Knowl. Data Eng. 14, 659–665 (2002)CrossRefGoogle Scholar
  39. 39.
    Zhou, Z.-H., Liu, X.-Y.: Training cost-sensitive neural networks with methods addressing the class imbalance problem. IEEE Trans. Knowl. Data Eng. 18, 63–77 (2006)CrossRefGoogle Scholar
  40. 40.
    Lin, S.-C., Yuan-chin, I.C., Yang, W.-N.: Meta-learning for imbalanced data and classification ensemble in binary classification. Neurocomputing 73, 484–494 (2009)CrossRefGoogle Scholar
  41. 41.
    Ghazikhani, A., Monsefi, R., Yazdi, H.S.: Ensemble of online neural networks for non-stationary and imbalanced data streams. Neurocomputing 122, 535–544 (2013)CrossRefGoogle Scholar
  42. 42.
    Yin, Q.-Y., Zhang, J.-S., Zhang, C.-X., Ji, N.-N.: A novel selective ensemble algorithm for imbalanced data classification based on exploratory undersampling. Mathematical Problems in Engineering, vol. 2014 (2014)Google Scholar
  43. 43.
    Sun, Z., Song, Q., Zhu, X., Sun, H., Xu, B., Zhou, Y.: A novel ensemble method for classifying imbalanced data. Pattern Recogn. 48, 1623–1637 (2015)CrossRefGoogle Scholar
  44. 44.
    Galar, M., Fernandez, A., Barrenechea, E., Bustince, H., Herrera, F.: A review on ensembles for the class imbalance problem: bagging-, boosting-, and hybrid-based approaches. IEEE Trans. Syst. Man Cybern. Part C (Appl. Rev.) 42, 463–484 (2012)CrossRefGoogle Scholar
  45. 45.
    Galar, M., Fernández, A., Barrenechea, E., Herrera, F.: EUSBoost: enhancing ensembles for highly imbalanced data-sets by evolutionary undersampling. Pattern Recogn. 46, 3460–3471 (2013)CrossRefGoogle Scholar
  46. 46.
    Lim, P., Goh, C.K., Tan, K.C.: Evolutionary cluster-based synthetic oversampling ensemble (eco-ensemble) for imbalance learning. IEEE Trans. Cybern. (2017)Google Scholar
  47. 47.
    Koulinas, G., Kotsikas, L., Anagnostopoulos, K.: A particle swarm optimization based hyper-heuristic algorithm for the classic resource constrained project scheduling problem. Inf. Sci. 277, 680–693 (2014)CrossRefGoogle Scholar
  48. 48.
    Burke, E.K., Gendreau, M., Hyde, M., Kendall, G., Ochoa, G., Özcan, E., et al.: Hyper-heuristics: a survey of the state of the art. J. Oper. Res. Soc. 64, 1695–1724 (2013)Google Scholar
  49. 49.
    Dowsland, K.A., Soubeiga, E., Burke, E.: A simulated annealing based hyperheuristic for determining shipper sizes for storage and transportation. Eur. J. Oper. Res. 179, 759–774 (2007)CrossRefMATHGoogle Scholar
  50. 50.
    Alcala-Fdez, J., Sanchez, L., Garcia, S., del Jesus, M., Ventura, S., Garrell, J., Otero, J., Romero, C., Bacardit, J., Rivas, V.: Keel: a software tool to assess evolutionary algorithms for data mining problems. Soft Comput 13(3), 307–318 (2009)CrossRefGoogle Scholar
  51. 51.
    Gascón-Moreno, J., Salcedo-Sanz, S., Saavedra-Moreno, B., Carro-Calvo, L., Portilla-Figueras, A.: An evolutionary-based hyper-heuristic approach for optimal construction of group method of data handling networks. Inf. Sci. 247, 94–108 (2013)MathSciNetCrossRefGoogle Scholar
  52. 52.
    Burke, E., Kendall, G., O’Brien, R., Redrup, D., Soubeiga, E.: An ant algorithm hyper-heuristic. In: Proceedings of the Fifth Meta-heuristics International Conference (MIC’03), pp. 1–10 (2003)Google Scholar
  53. 53.
    Koulinas, G.K., Anagnostopoulos, K.P.: Construction resource allocation and leveling using a threshold accepting-based hyperheuristic algorithm. J. Constr. Eng. Manag. 138, 854–863 (2011)CrossRefGoogle Scholar
  54. 54.
    Burke, E.K., Hyde, M., Kendall, G., Ochoa, G., Özcan, E., Woodward, J.R.: A classification of hyper-heuristic approaches. In: Handbook of Meta-heuristics. Springer, pp. 449–468 (2010)Google Scholar
  55. 55.
    Nielsen, M.A., Chuang, I.L.: Quantum computation and quantum information. Quantum 546, 1231 (2000)MATHGoogle Scholar
  56. 56.
    García, S., Fernández, A., Luengo, J., Herrera, F.: Advanced nonparametric tests for multiple comparisons in the design of experiments in computational intelligence and data mining: experimental analysis of power. Inf. Sci. 180, 2044–2064 (2010)CrossRefGoogle Scholar
  57. 57.
    Holm, S.: A simple sequentially rejective multiple test procedure. Scand. J. Stat 6(2), 65–70 (1979)Google Scholar

Copyright information

© Springer International Publishing AG, part of Springer Nature 2018

Authors and Affiliations

  1. 1.Intelligent Data Processing Laboratory (IDPL), Department of Electrical EngineeringShahid Bahonar University of KermanKermanIran

Personalised recommendations