Semi-random partitioning of data into training and test sets in granular computing context

Machine learning is a branch of artificial intelligence, which is increasingly used in the big data era for the purpose of knowledge discovery and predictive modelling. The former purpose generally means that a model is learned from data and some previously unknown patterns can be extracted from the model (Liu et al. 2016). The latter purpose means that a model is learned from data and the model is then used to predict on any new data instances. For both knowledge discovery and predictive modelling, it is essential to partition a data set into a training set and a test set (Liu et al. 2016). In particular, for the purpose of knowledge discovery, the training set is used for a machine learning algorithm to discover any new patterns, and the test set is then used to validate the degree to which the patterns truly exist and are trustable. In contrast, for the purpose of predictive modelling, the training set is used for a machine learning algorithm to build a model, and the test set is then used to evaluate against the predictive accuracy of the model.

In the context of partitioning a data set into a training set and a test set, it has been critical to decide effectively on which part of the data set is selected as the training set, and which part is selected for the test set (Liu et al. 2017). In the traditional machine learning, it is a normal practice that researchers and practitioners choose to do the data partitioning in a fully random way. This way of partitioning, however, leads to two major issues: (a) class imbalance and (b) sample representativeness issues.

The first issue of class imbalance (Longadge et al. 2013; Ali et al. 2015) is known to affect many classifiers’ performance (Sotiropoulos and Tsihrintzis 2017). Randomly partitioning the data, however, can lead to class imbalance in the training and the test set, even when there is no imbalance in the overall data set. For example, let us consider a 2-class (e.g., positive class and negative class) data set with a balanced distribution of instances across classes, i.e., 50% of the instances belong to the positive class and 50% of the instances belong to the negative class. When the data set is partitioned by selecting training/test instances randomly, it is likely that the class balance of the data set will be broken, which would lead, for example, to more than 50% of the training instances belonging to the positive class and more than 50% of the test instances belonging to the negative class, i.e., the training set has more positive instances than negative ones, while the test set has the opposite situation.

The second issue is about sample representativeness and the fact that the random partitioning may lead to high dissimilarity between training and test instances. In the context of student learning, the training instances are like the revision questions and the test instances are like the exam questions. To test effectively the performance of student learning, the revision questions should be representative with respect to the learning content covered in the exam questions. The random partitioning of data, however, can result in the case that the training instances are dissimilar to the test instances, which corresponds to the situation that students are tested on what they have not yet learned. Such a situation not only leads to a poor performance, but also to a poor judgment of the learner capability. Thus, in the context of predictive modelling, some algorithms may be judged as not being suitable for a particular problem due to a poor performance, when in reality the poor results are not due to the algorithm, but to the representativeness of the training sample.

To address the two issues mentioned above, we propose, in this paper, a semi-random data partitioning framework in the setting of granular computing, towards effective selection of training and test instances. In particular, we focus on dealing with the class imbalance issue and provide a brief proposal towards dealing effectively with the sample representativeness issue.

The rest of this paper is organized as follows: Sect. 2 provides theoretical preliminaries on data partitioning and granular computing concepts. In Sect. 3, we present a multi-granularity data partitioning framework for controlling effectively the partitioning of data into a training set and a test set, towards overcoming the class imbalance and sample representativeness issues. In Sect. 4, we report an experimental study on controlling the class balance of the training and test sets; the results are discussed critically and comparatively. In Sect. 5, we highlight the contributions of this paper and provide further directions towards dealing effectively with the issue of sample representativeness, as well as how to use our framework to change the class balance in the training set for highly imbalanced data sets to further address poor performance due to class imbalance.

In this section, we provide theoretical preliminaries on data partitioning and granular computing. In particular, we describe two ways of machine learning experimentation through data partitioning, namely cross-validation and partitioning into training/test sets. In addition, we describe the concepts of information granules and information granularity which are used in the proposed framework described in Sect. 3.

2.2 Granular computing

Granular computing has been an increasingly popular approach for in-depth processing of information. It is aimed at structural thinking at the philosophical level, as well as at structural problem solving at the practical level (Yao 2005). In general, granular computing involves two operations, namely, granulation and organization. The former operation means to decompose a whole into several parts, whereas the latter operation means to integrate several parts into a whole. From computer science perspective, granulation corresponds to the top-down approach and organization corresponds to the bottom-up approach. The nature of granular computing involves two commonly used concepts, namely, granule and granularity.

In the context of information granule, a granule is defined as “a small particle, especially, one of numerous particles forming a larger unit”, according to the Merriam-Webster Dictionary (Merriam-Webster 2016). In practice, there have been various examples of granules in broad application areas.

In the setting of set theory, a set of any formalism can be viewed as a granule, since a set is a collection of elements. In this context, each element is viewed as a particle. Different formalisms of sets include deterministic sets (Liu et al. 2016), probabilistic sets (Liu et al. 2016), fuzzy sets (Zadeh 2015), and rough sets (Pedrycz 2011).

In the area of computer science, a granule can act as a class due to the fact that a class is a group of objects which are highly similar to each other. An object can also be viewed as a granule, since each object involves a number of attributes, each of which is considered as a particle. Moreover, a granule can also act as a cluster due to the fact that clustering is another way of grouping objects.

In the area of natural languages, a document could be organized in different forms of text units, such as chapters, sections, paragraphs, sentences, and words. In this context, each form of text units can be viewed as a special type of granule. Moreover, each word is viewed as the finest granule due to the fact that a word consists of letters, each of which is viewed as a particle (Liu and Cocea 2017b).

The concept of information granules is also popularly involved in other application areas, such as image processing, machine learning, and rule-based systems. More details on information granules can be found in Pedrycz (2011), and Pedrycz and Chen (2011, 2015a, b).

In the context of information granularity, information granules can be located in different levels of granularity. In set theory, a set S may have several subsets (\(S_1, S_2,\ldots , S_n\)) and each subset may also have several subsubsets (\(S_{1.1}, S_{1.2},\ldots ,S_{1.m},\ldots ,S_{n.1}, S_{n.2},\ldots ,S_{n.m}\)). In this context, the set S is a granule in the top level of granularity, the subsets (\(S_1, S_2,\ldots ,S_n\)) are in the middle level of granularity, and the subsubsets (\(S_{1.1}, S_{1.1},\ldots ,S_{1.m},\ldots ,S_{n.1}, S_{n.2},\ldots ,S_{n.m}\)) are in the bottom level of granularity. In computer science, a class can be specialized into several subclasses through information granulation. In addition, subclasses can be generalized into a super class through information organization.

Open image in new window

In natural language processing, a document can be managed in a granular structure, as illustrated in Fig. 1. In particular, the complexity of a text instance (granule) can be reduced through top-down decomposition (granulation) to enable text units (granules) in different levels of granularity (such as paragraphs, sentences, and words) to be processed separately. In addition, the outcomes for processing text units in the same level of granularity can be combined through bottom-up aggregation (organization) towards deriving the outcome for processing larger text units in a higher level of granularity.

In real applications, techniques of granular computing have been involved very often in other popular areas, such as artificial intelligence (Wilke and Portmann 2016; Yao 2005; Skowron et al. 2016), computational intelligence  (Dubois and Prade 2016; Yao 2005; Kreinovich 2016; Livi and Sadeghian 2016), and machine learning (Min and Xu 2016; Peters and Weber 2016; Liu and Cocea 2017a; Antonelli et al. 2016).

Furthermore, ensemble learning is also a subject that involves applications of granular computing concepts (Liu and Cocea 2017a). In particular, ensemble learning approaches, such as Bagging, involve information granulation through decomposing a training set into a number of overlapping samples and combining the predictions made from different classifiers towards classifying a test instance; a similar perspective has also been stressed and discussed in Hu and Shi (2009). Section 3 will show how granular computing concepts can be used towards more effective partitioning of data for machine learning experimentation.

In this section, we propose a multi-granularity framework for effective control of the partitioning of a data set into a training set and a test set. We also justify how the proposed approach can address the class imbalance and sample representativeness issues that can arise from random partitioning.

3.1 Key features

The multi-granularity framework for semi-random data partitioning is illustrated in Fig. 2. In particular, this framework involves three levels of granularity as outlined below:

1. 1.

   Level 1 Data Partitioning is done randomly on the basis of the original data set towards getting a training set and a test set.

    
2. 2.

   Level 2 The original data set is divided into a number of subsets, with each subset containing a class of instances. Within each subset (i.e., all instances with a particular class label), data partitioning into training and test sets is done randomly. The training and test sets for the whole data set are obtained by merging all the training and test subsets, respectively.

    
3. 3.

   Level 3 Based on the subsets obtained in Level 2, each of them is divided again into a number of subsubsets, where each of the subsubsets contains a subclass (of the corresponding class) of instances. The data partitioning is done randomly within each subsubset. The training and test sets for the whole data set are obtained by merging all the training and test subsubsets, respectively.

    

Open image in new window

In this multi-granularity framework, Level 2 is aimed at addressing the class imbalance issue, i.e., to control the distribution of instances by class within the training and test sets. Level 3 is aimed at addressing the issue of sample representativeness, i.e., it is to avoid the case that the training instances are highly dissimilar to the test instances following the data partitioning.

In the setting of granular computing, the proposed framework involves explicitly both granulation and organization. In particular, granulation is involved through the operation that a data set is divided into a number of subsets and each subset is divided into a training subset and a test subset (Level 2), or further divided into subsubsets and then split into training and test subsubsets (Level 3). In addition, organization is involved by integrating the training subsets or subsubsets into a whole training set, and the test subsets or subsubsets into a whole test set. In addition, in each level of the granularity as shown in Fig. 2, a set of data is viewed as a granule, which also has hierarchical relationships with sets of data (granules) located in other levels of granularity.

Table 1

Sampling probability by stratified sampling

Weight	Probability for class 1 (%)	Probability for class 2 (%)	Probability for class 3 (%)
Training set: 70%	28	28	14
Test set: 30%	12	12	6

Table 2

Sampling probability by semi-random partitioning

Weight	Probability for class 1 (%)	Probability for class 2 (%)	Probability for class 3 (%)
Training set: 70%	70	70	70
Test set: 30%	30	30	30

In this section, we report two experimental studies. In particular, the first study involves comparing our proposed approach of semi-random data partitioning with the stratified sampling approach. The second study is to validate the effectiveness of the strategy of semi-random data partitioning involved in Level 2 of the multi-granularity framework proposed in Sect. 3. In particular, we compare the strategy of the semi-random data partitioning with the one of the traditional random data partitioning, in terms of class frequency distribution within the training and test sets, as well as the influence of this distribution on classification performance.

The experimental studies are conducted using 12 UCI data sets (Lichman 2013). The characteristics of the data sets are shown in Table 3. All the chosen data sets are either balanced or slightly imbalanced, except for the ‘anneal’ and ‘autos’ data sets, in terms of class frequency distribution. For using both balanced or slightly imbalanced data sets, the aim is to show that it is necessary to manage to keep the balance level of both the training and test sets as close to the balance level of the original data set as possible, towards avoiding any impact on the learning performance of the algorithms and on the classification performance of the learned classifiers. The imbalanced data sets, i.e., ‘anneal’ and ‘autos’, as well as the ‘segment’ balanced data set, have a larger number of classes, while the other nine data sets have two or three classes. These will allow us to analyze the results in terms of number of classes, as well.

Three popular machine learning algorithms, i.e., the C4.5 decision tree learning algorithm (Quinlan 1993), Naive Bayes (Rish 2001), and K-nearest neighbour (Liu et al. 2016), are used for validation, since these three algorithms are all sensitive to class imbalance (Longadge et al. 2013).

Regarding the first experimental study, the results are shown in Tables 4, 5, and  6. In these three tables, SS stands for stratified sampling and SR stands for semi-random partitioning.

Table 4 shows that the proposed semi-random partitioning outperforms stratified sampling in 9 out of 12 cases, and the two approaches perform the same in the other 3 cases, in terms of overall accuracy of classification. In addition, the proposed semi-random partitioning outperforms stratified sampling in terms of precision and recall with respect to each single class in most cases.

Table 5 shows that the proposed semi-random partitioning outperforms stratified sampling in 9 out of 12 cases, and the two approaches perform the same in 2 out of the other 3 cases, in terms of overall accuracy of classification. In addition, the proposed semi-random partitioning outperforms stratified sampling in terms of precision and recall with respect to each single class in most cases.

Table 6 shows that the proposed semi-random partitioning outperforms stratified sampling in 7 out of 12 cases, and the two approaches perform the same in 3 out of the other 5 cases, in terms of overall accuracy of classification. In addition, the proposed semi-random partitioning outperforms stratified sampling in terms of precision and recall with respect to each single class in most cases.

Regarding the second experimental study, Table 7 displays the original distribution of instances across classes for each data set in terms of frequency (designated by #) and percentages (designated by %). For example, the anneal data set (first row in Table 7) has 6 classes, and in the original distribution, class 1 has 8 instances (representing 1% of all instances), class 2 has 99 instances (representing 11% of the data), and so on. The same information is also displayed for the training and test sets used with the semi-random partitioning approach. The percentage numbers have been rounded to integers for ease of comparison. The loss of precision due to this rounding means that the sum across all classes may not be precisely 100%. In addition, when the number of instances is low, a small difference in the number of instances may lead to a much bigger difference in the percentages values.

Open image in new window

Open image in new window

Open image in new window

In terms of accuracy, the results show three situations: (a) the semi-random partitioning leads to the same accuracy as random partitioning, i.e., ‘anneal’, ‘kr-vs-kp’, and ‘segment’; (b) the semi-random partitioning displays small improvements in accuracy (up to 3%), i.e., ‘autos’, ‘credit-a’, ‘heart-statlog’, ‘sonar’, ‘tae’, and ‘vote’; (c) the semi-random partitioning displays large improvements in accuracy (5% or more), i.e., iris (7%), labor (23%), and wine (5%).

Figures 3, 4,  5 display the class distribution, as well as the precision and recall for the experiments with C4.5 on all data sets (4 per graph). The class distribution for the whole data set is represented by the middle bar for every class; the distribution into the training and test sets for random partitioning is represented by the bars on the left, while the ones for semi-random partitioning are represented by the bars on the right. The lines with the square points represent the values for precision—yellow for random partitioning and brown for semi-random partitioning; the lines with the triangle points represent the values for recall—blue for random partitioning and green for semi-random partitioning. The left axis on the graphs represents the number of instances (or class frequency), while the right axis represents the values for precision and recall, with a range from 0 to 1.

For the data sets where the accuracy is the same for both random and semi-random partitioning, i.e., ‘anneal’, ‘kr-vs-kp’, and ‘segment’, the class distribution (see Table 8; Figs. 3, 4) is very similar for both random and semi-random partitioning. For the ‘kr-vs-kp’ data set, although the test set is more balanced (and the training one more imbalanced) compared with the original distribution, the change is very small, especially for the training set where the change is of 1%. For this data set, we also observed that the majority class in the training set becomes the minority class in the test set—the difference, however, is very small, i.e., 2%. Given the large size of this data set and only a slight imbalance in the distribution of classes, it is not surprising that such a small change in distribution does not impact the results.

For the data sets where the accuracy is slightly higher when semi-random partitioning is used, i.e., ‘autos’, ‘credit-a’, ‘heart-statlog’, ‘sonar’, ‘tae’, and ‘vote’, the random partitioning has different effects on the class distribution within the training and test sets.

For the data sets where the accuracy is considerably higher when using semi-random partitioning, i.e., iris (7%), labor (23%), and wine (5%), we notice that the random partitioning leads to class distribution imbalance in the training sets, and something in the test sets as well (i.e., iris and wine). The difference in results is likely to be due to the class imbalance issue, as well as sample representativeness (e.g., class 1 of the ‘wine’ data set has similar distribution for both random and semi-random partitioning, but different precision results).

Table 12 displays the experimental results for Naive Bayes (NB), including recall and precision per class, and accuracy—for random (R) and semi-random (SR) partitioning. Figures 6, 7,  8 display the precision and recall results, as well as the class distribution, with the similar structure as for the previous graphs (with the C4.5 results).

Open image in new window

Open image in new window

Open image in new window

For the data sets displaying lower accuracy for the semi-random partitioning, i.e., ‘segment’ and ‘wine’, the difference in accuracy compared with random partitioning is very small, i.e., 1% for ‘segment’ and 2% for ‘wine’. For the ‘segment’ data set, there is a small change in the class distribution with random partitioning; for the classes where the change results in more instances in the training set, the recall values are higher, while for the classes where the change results in more instances in the test set, the precision is higher; for the classes where there is little or no change, the difference in results may be due to sample representativeness. For the ‘wine’ data set, the random partitioning results in a more balanced distribution across classes in the training set, which may explain the better performance.

The accuracy for random and semi-random partitioning is the same on the ‘sonar’ data set, for which the random partitioning leads to a more imbalanced training set, with the same effect as above, i.e., when there are more instances in the training set, the recall is higher, while when there are more instances in the test set, the precision is higher.

For 6 data sets, i.e., ‘anneal’, ‘autos’, ‘credit-a’, ‘iris’, ‘kr-vs-kp’, and ‘vote’, the semi-random partitioning has up to 4% better accuracy than random partitioning. For the ‘anneal’, ‘credit-a’, and ‘kr-vs-kp’, the class distribution is very similar for random and semi-random partitioning—thus, the small difference is likely to be due to sample representativeness. For the ‘autos’, ‘iris’, and ‘vote’, the random partitioning leads to more class imbalance, which may affect the results.