Abstract
In this study, a novel multi-classifier ensemble method based on dynamic weights is proposed to reduce the interference of unreliable decision information and improve the accuracy of fusion decision. The algorithm defines decision credibility to describe the real-time importance of the classifier to the current target, combines this credibility with the reliability calculated by the classifier on the training data set and dynamically assigns the fusion weight to the classifier. Compared with other methods, the contribution of different classifiers to fusion decision in acquiring weights is fully evaluated in consideration of the capability of the classifier to not only identify different sample regions but also output decision information when identifying specific targets. Experimental results on public face databases show that the proposed method can obtain higher classification accuracy than that of single classifier and some popular fusion algorithms. The feasibility and effectiveness of the proposed method are verified.
Similar content being viewed by others
Explore related subjects
Discover the latest articles, news and stories from top researchers in related subjects.Avoid common mistakes on your manuscript.
1 Introduction
The advent of the information age has increased the amount of data from scientific research in various fields of social life. Realising massive data storage and intelligent processing and fully utilising the knowledge and value contained in data have been the main tasks of the academia and the industry [5, 16, 30, 56,57,58]. Machine learning is a mainstream intelligent data processing technology and is an important way to achieve the above-mentioned goals. The continuous development of science and technology has facilitated the development of machine learning and its application to various fields, such as 3D modeling [55] and forecasting and analysis of stock trend [52, 54, 59]. Classification technology is an indispensable part of machine learning. Machine learning solves classification problems by finding a fully good function, that is, the classifier, by use of a specific learning algorithm in the hypothetical space to simulate the actual classification function. The widely used single classifier models are forward feedback artificial (BP) neural network [40], support vector machine (SVM) [10] and decision tree [27]. BP neural network is a non-linear mapping system with a forward structure without a feedback one. This model can fit the input–output relationship of nonlinear systems effectively and have been used by many scholars in this field. SVM is a supervised learning model. This model can transform linear inseparable samples into high-dimensional space by use of a nonlinear mapping algorithm to make the samples linearly separable and can effectively deal with binary classification problems. Unlike the two previous models, the construction of the decision tree does not depend on specific domain knowledge. This model uses the attribute selection metric to determine the topology amongst various feature attributes and thus complete the classification decision. Although each of these classifier models has achieved good results, all of them present poor generalisation capability and of classification accuracy that cannot meet users’ requirements when noise exists in the input sample or the distribution of the training and classification sets is inconsistent. Ensemble learning is a new paradigm of machine learning. This model integrates multiple classifiers in accordance with specific integration rules to solve the classification problem and can thus improve the accuracy and generalisation capability of system prediction. Ensemble learning is widely used in various fields, such as human activity recognition [33,34,35, 39], human motion tracking [11, 32], prediction [36, 44], water quality supervision [37, 38] and face or emotion discrimination [4, 14, 20].
Multi-classification ensemble can be divided into static ensemble and dynamic ensemble depending on whether weights are adaptively assigned to each classifier when different query samples are identified. The classifier that participates in the static integration is no longer changed after its determination. Castrillón-Santana et al. [47] focused on gender classification based on face images and evaluated multi-expert systems using three different fusion protocols. However, the said methods ignore the importance of classifiers when identifying different query samples and assign the same weight value to each expert. Adaboost [43] is a widely used static weighted ensemble algorithm. The weights of the base classifiers are evaluated before the query samples arrive, and these base classifiers are integrated by linear weighting to obtain the final decision. Kuncheva [25] proposed a method based on probability weighted voting. Chen [7] proposed a method based on training error to calculate the weight coefficient in completing the SAR image classification. The aforementioned static methods enable the classification model to statistically achieve the best outcome; however, a corresponding change cannot be made with the change in the input information and the abnormal output of the classifier. Such condition is obviously unreasonable. By contrast, dynamic methods can determine the adaptive weights on the basis of the characteristics of the sample; these weights can effectively emphasise the decision-making contribution made by excellent classifiers and suppress the influence of unreliable information output by use of high-deviation classifiers on classification. Cheon [8] assigned weights following the Hausdorff distance between query and gallery image sequences. Zhang [60] considered the performance of each classifier in different sample environments and then proposed a dynamic adaptive weighted voting method. Considering the correlation and reliability of the query descriptors, Sun [49] calculated the weights on the basis of the sparse coefficients of the query sample in different classes. The results showed that the weights change with the change in the number of query sample. Most of the existing weight allocation methods assign weights to the classifier from a certain aspect, and none of the methods either based on the statistical performance of the classifier or the characteristic distribution of the query sample comprehensively consider the identification performance of the classifier for different sample regions and samples. Each classifier presents different recognition capabilities for different samples owing to the different characteristics of these samples, and the recognition performance of the same classifier in different regions of the sample space also differs. Therefore, we determine whether the weight of the classifier in the fusion process can be dynamically adjusted by estimating the recognition performance of the classifier on different sample regions and samples, thereby reducing the interference of unreliable information on the decision making and achieving good ensemble performance. This study provides the following contributions.
-
A dynamic weighted multi-classifier ensemble method that defines reliability and credibility is proposed. Weights can be dynamically changed with the query samples, and the importance of the classifier in the fusion process can be described fully.
-
The reliability of the classifier is defined on the basis of its recognition capability, which is determined by the prior knowledge gained during the training process. Fuzzy set theory and cloud model are utilised to evaluate this criterion for base classifiers.
-
The credibility of the classifier is obtained by calculating the posterior probability distribution, which is used to determine the separability feature of the classifier. A mapping function is used to map the separability into the credibility of the classifier.
Compared with single classifier and current popular fusion methods, our method can more effectively reduce the effect of unreliable instance information in the training phase of the classifier and can therefore fuse the decisions in classification efficiently and improve the overall performance of the integrated method.
The rest of the paper is organised as follows. Section 2 reviews the traditional multi-classifier ensemble method. Section 3 presents the proposed method of multi-classifier ensemble based on dynamic weights. Section 4 demonstrates the application of the proposed algorithm in face recognition. Section 5 provides the experimental results and analysis. Section 6 contains the discussion. Finally, Section 7 elaborates the conclusions of the study.
2 Related work
On the basis of the output of the classifier, the decision information can be divided into three levels [53]: abstract, ranked and measurement levels. At the abstract level, each classifier directly outputs the category labels. Common ensemble methods include voting [42], weighted voting and behavioural knowledge space. The ranked level sorts the output results of the classifier. This type of output usually occurs in problems with a large number of categories. The commonly used ensemble methods of ranked level are highest rank method, Borda count method [19] and logistic regression. At the measurement level, the output of each classifier is the probability, credibility or distance measure of the class. The measurement level is more informative than the abstract and ranked levels. The measurement level also contains a richer amount of decision information compared with that of the two other levels. The corresponding ensemble methods include Max/Min/ Sum, linear combination method, D-S evidence theory [17] and fuzzy integral method [9, 26]. The multi-classifier ensemble method based on measurement level is defined as follows.
We define a classification space S, which contains M different classes, for K classifiers. Furthermore, S = C1 ∪ ⋯ ∪ CM, Cm indicates the m-th class, 1 ≤ m ≤ M, E = {e1, e2, ⋯eK} is a collection of base classifiers and pkm(x) represents the posterior probability of query sample x discriminated by classifier ek as belonging to class m. Thus, the subsequent decision output matrix can be obtained by Eq. (1) as follows:
The multi-classifier ensemble obtains the final classification decision by merging the recognition outputs of each classifier in different categories. The decision output of the multi-classifier system on class Cm is obtained by Eq. (2) as follows:
We select the category with the maximum posterior probability sum as the class to which query sample x belongs and complete the final decision. In particular, the decision result of the joint classification is as follows:
To highlight the importance of different classifiers in the fusion process, Eq. (2) is modified as follows:
In Eq. (4), ωk represents the weight of classifier ek. High value of ωk means high importance of the classifier in the fusion process.
3 Multi-classifier ensemble based on dynamic weights
In this study, we propose a multi-classifier weighted ensemble method based on measurement level fusion. Figure 1 presents the framework of our approach that comprises three major components. (1) In the training phase, the fuzzy density of the classifier is computed firstly and then inputted into an inverse cloud generator in accordance with the correct and incorrect decisions of the corresponding instances. As a result, the reliability of the classifier can be modelled on the basis of the expected value, entropy and hyper entropy digital eigenvalues. (2) The separability of the classifier is obtained by calculating the posterior probability distribution when a specific target is identified. Thereafter, a mapping function is used to map the separability into the credibility of the classifier. (3) The dynamic weights of the classifier are obtained on the basis of reliability and decision credibility. We utilise the dynamic weights to implement dynamic weighted integration. The three major components are discussed in detail in the following sections.
3.1 Statistic of fuzzy density
We aim to obtain the reliability of the classifier. For this purpose, we use different learning algorithms firstly to train the data set for constructing the base classifier and then use confusion matrix \( {CM}_{M\times M}^k \) to compute the identification of the known class samples made by base classifier ek. The expression is as follows:
where k = 1, 2, ⋯, K and \( {n}_{ml}^k \) is the number of samples in class Cm identified by base classifier ek as class Cl. If m = l, then \( {n}_{ml}^k \) is the number of samples in class Cm correctly identified by base classifier ek; if m ≠ l, then \( {n}_{ml}^k \) is the number of samples in class Cm incorrectly identified by base classifier ek as class Cl.
For K different base classifiers, we can obtain K confusion matrixes CM1, CM2, ⋯, CMK to determine the recognition capability of classifier ek(k = 1, 2, ⋯, K) to Cm(1 ≤ m ≤ M). We use fuzzy integral theory to compute the fuzzy density of the classifier by confusion matrix CMk. Fuzzy density can be calculated by
In Eq. (6), \( \frac{n_{mm}^k}{\sum \limits_{l=1}^M{n}_{ml}^k} \) is the ratio that class Cm is correctly recognised by classifier ek; \( \frac{n_{nm}^k}{\sum \limits_{l=1}^M{n}_{nl}^k} \) is the ratio that class Cn is incorrectly recognised by classifier ek as class Cm, n ≠ m. Large value of \( \frac{n_{mm}^k}{\sum \limits_{l=1}^M{n}_{ml}^k} \) means small value of \( \frac{n_{nm}^k}{\sum \limits_{l=1}^M{n}_{nl}^k} \) and large value of \( {g}_m^k \), thereby indicating that the recognition capability of classifier ek to class Cm is strong.
The recognition capability of classifier ek to identify M classes can be represented by fuzzy density vector \( {G}^k=\left({g}_1^k,{g}_2^k,\cdots, {g}_M^k\right) \). K different base classifiers make up fuzzy density matrix GM as follows:
3.2 Basic concept and eigenvalue acquisition of cloud model
The cloud model is introduced to describe the reliability of the identification results made by the classifier in different sample regions. The cloud model is a type of uncertain transformation model between qualitative concept and quantitative data proposed by Li [29]. The model combines the fuzziness in fuzzy set theory with the randomness in probability theory to characterise a whole concept. The model also occupies an important position in performance evaluation, recognition and other areas [12, 31, 48]. The cloud model consists of three eigenvalues: expected value Ex, entropy En and hyper entropy He. Ex is the central value of qualitative language concept domain and reflects the situation of gravity of cloud drop group. En is a measure of the ambiguity of the qualitative concept. On the one hand, it reflects the range of the concept that can be accepted in the domain space (i.e. the ambiguity). On the other hand, the points reflected in the domain space can represent the probability of this concept, thereby indicating the randomness of cloud droplets of the qualitative concept. He represents the aggregation of cloud droplets in the domain space and reflects the discrepancy in the cloud droplets and the random change in membership. In the number field space, He represents the cohesion of the uncertainty of all points of the language value. The cloud generator can be in forward and reverse modes. The inverse cloud generator is a large number of cloud droplet distributions Drop(x, μ) in the known cloud and determines the three numerical eigenvalues Ex, En and He of the normal cloud. The output ensemble problem of the classifier completes the quantitative to qualitative conversion. Therefore, this study focuses only on the inverse cloud generator, and the fuzzy density value of the classifier in different sample regions is used as a cloud droplet and input into the inverse cloud generator to generate a cloud that characterises the reliability of the classifier. The three digital eigenvalues (Exk, Enk, Hek) of classifier ek(k = 1, 2, ⋯, K) are calculated as follows:
-
(1)
The mean of Gk is given by
-
(2)
The variance of Gk is given by
-
(3)
The expected value of classifier ek is given by
-
(4)
The entropy of classifier ek is given by
-
(5)
The hyper entropy of classifier ek is given by
Fuzzy density reflects the confidence of the classifier for each category and can effectively describe the recognition performance of the classifier to different sample regions. The cloud model for describing the reliability of the classifier is used because the three digital features (Ex, En, He), which are generated by the cloud model using the fuzzy density of the classifier, can reflect not only the average performance of the classifier but also the stability and randomness of the change of the classifier with the changes in the number of sample. Ex represents the average recognition performance of the classifier, and large expected value means good performance of the classifier. En represents the degree of discretisation of the classifier’s capability to identify the sample and reflects the stability of the classifier with the change in the number of sample. Small value of entropy means better stability (i.e. the performance of the classifier). He represents the degree of deviation (i.e. randomness) from the normal operation of the classifier. Therefore, small value of hyper entropy means good performance of the classifier and that the recognition performance of each classifier in different sample regions is converted to the corresponding cloud model.
3.3 Reliability of the classifier based on the cloud model
This study defines reliability from two aspects: subjective and objective. The subjective reliability of the classifier is determined by the first digital feature of the cloud model, that is,
Good average recognition performance of the classifier means high subjective reliability. The objective reliability is determined by the sum of the entropy and hyper entropy of the cloud model. The entropy characterises the fuzziness of the classifier, and the hyper entropy characterises the randomness of the classifier. Therefore, this study considers using the entropy and hyper entropy of the cloud to construct the optimisation model for calculation.
We let εk = Enk + Hek be the sum of the entropy and hyper entropy of the k-th classifier and construct the optimisation model with the smallest sum of the entropy and hyper entropy of all classifiers. In particular, we set the sum of the fuzziness and randomness of all classifiers to be relatively minimal. Wang [50] considered the three eigenvalues of the normal cloud corresponding to different orders of magnitude. Therefore, the square of the objective reliability is used in the current study to constrain the entropy and hyper entropy in constructing the optimisation model of the entropy and hyper entropy based on the cloud model shown in Eq. (14).
We solve Eq. (14) and construct the Lagrangian function as Eq. (15).
According to the existence conditions of extremum, we obtain
where k = 1, 2, ⋯K. Combining the two equations in Eq. (16), we obtain
The reliability \( {\alpha}_k^{\ast } \) of classifier ek is synthetically determined by subjective and objective reliabilities as follows:
Each classifier can use the three eigenvalues obtained from the cloud model to obtain reliability. For K different base classifiers, we obtain \( \zeta =\left\{{\alpha}_1^{\ast}\right.,{\alpha}_2^{\ast },\cdots \left.,{\alpha}_K^{\ast}\right\} \), and the value in ζ is normalised by Eq. (19).
In Eq. (19), \( {\alpha}^{\mathrm{min}}=\underset{1\le {k}^{\hbox{'}}\le K}{\min}\left\{{\alpha}_{k^{\hbox{'}}}^{\ast}\right\} \) and \( {\alpha}^{\mathrm{max}}=\underset{1\le {k}^{\hbox{'}}\le K}{\max}\left\{{\alpha}_{k^{\hbox{'}}}^{\ast}\right\} \). The normalised reliability αk ∈ [0.5, 1] of the classifier ek is obtained. The reliability of the classifier is proportional to the expected value and is inversely proportional to the entropy and hyper entropy of the classifier. Therefore, the requirement that the classifier should characterise the recognition of different regions is satisfied. The reliability of the classifier can be used as the static weight of the classifier to realise the multi-classifier static weighting ensemble. Considering the characteristics of the sample itself, we aim to identify the real-time recognition performance of the classifier with respect to the currently tested sample and reduce the interference of the unreliable decision information to the classification. For this purpose, the decision credibility of the classifier is defined by measuring the separability amongst the posterior probabilities of the classifier.
3.4 Decision credibility of classifier
Under normal circumstances, the classifier can easily identify the category of the query sample when the posterior probability value obtained by the classifier is highly concentrated in a certain class. At this time, the possibility of misjudgement is low and the credibility of the classifier is high. When the distribution of the posterior probability is uniform, the classifier can exhibit difficulty in discriminating the class of the query sample. Furthermore, the possibility of misjudgement is high and the credibility of the classifier is low. In this study, we use the posterior probability distribution to define the decision credibility of the classifier in measuring the real-time importance of the classifier in the fusion process. The decision credibility βk of classifier ek is solved as follows:
-
(1)
The set of posterior probability values obtained by entering query sample x into classifier ek is denoted asPk = {pk1, pk2, ⋯, pkM}.
-
(2)
The components of the posterior probability value Pk are normalised to [0, 1] and then we obtain \( {P}_k^{\ast }=\left\{{p}_{k1}^{\ast },{p}_{k2}^{\ast },\cdots, {p}_{kM}^{\ast}\right\} \). We let \( {p}_{kj}^{\ast } \) be the largest posterior probability value in \( {P}_k^{\ast } \) and then define that separability τk is the sum between the difference of the other posterior probability values in \( {P}_k^{\ast } \) and \( {p}_{kj}^{\ast } \). Specifically, the difference is calculated first and then the sum is obtained using Eq. (20).
According to Eq. (20), the separability τk of the classifier is only related to \( {p}_{kj}^{\ast } \), and high proportion of \( {p}_{kj}^{\ast } \) in the output component of classifier ek means large value of τk and separability τk ∈ [0, 1]. When τk = 1, the maximum decision output value obtained by classifier ek is 1 and the other output components are 0. At this point, classifier ek maximises the probability of sample x being correctly identified and the decision credibility reaches a high value. When τk = 0, the posterior probability values of classifier ek are equal. At this time, the probability of misjudgement is high and the decision credibility of classifier ek is low.
-
(3)
To emphasise the contribution of large τk to the decision making and suppress the interference of small τk, a function is used to map τk to the decision credibility. When the value of τk is small, the assignment of decision credibility is as low as possible. As the value of τk increases, the decision credibility of the reliable decision output to the fusion decision rapidly enlarges. The mapping value approaches 1 when the value of τk increases to a certain range. To meet the requirements, this study uses the curve fitting in the numerical analysis to construct the sigmoid function in describing the change relationship between τk and decision credibility βk as shown in Eq. (21). Each parameter in Eq. (21) is the optimal value obtained after several simulation experiments.
In Eq. (21), βk ∈ (0.5, 1) and k = 1, 2, ⋯, K. Figure 2 shows that, when the proportion of the maximum posteriori probability of the classifier is small, the value of τk is also small as reflected by the curve; the mapping value βk is also low. As separability increases, the decision credibility βk of the classifier ek also gradually increases. When τk > 0.3, the growth degree of decision credibility βk increases; when the proportion of the maximum posteriori probability of the classifier is large (i.e. when τk > 0.65), the credibility of the classifier is extremely high. In addition, the value of decision credibility βk is stable and approaches 1.
3.5 Description of the proposed algorithm
In this study, a novel method of multi-classifier ensemble based on dynamic weights is proposed. The acquisition of weights is divided into four parts: (1) using different learning algorithms to train data sets to generate base classifiers; (2) using the cross-validation technique to estimate the recognition capability of the classifier in different sample regions, thereby obtaining the three eigenvalues of the cloud model and calculating the reliability of the classifier; (3) using function mapping to obtain the decision credibility of the classifier in accordance with the separability of the output components of the base classifier; (4) the corresponding dynamic weight is assigned to each classifier in accordance with reliability and decision credibility. The obtained weight vector and the output of each base classifier are used to make an integrated judgement to the tested samples. The specific processes are described below:
-
Input: data set Tr comprising the samples with known class information, Tr includes M different categories, in which each category contains n samples, a collection of learning algorithms L = {l1, l2, ⋯lK}) and the sample x to be tested.
-
Output: the class of sample x to be tested.
-
(1)
Using the learning algorithm lk(k = 1, 2, ⋯, K) to train data set Tr, we obtain the base classifier ek(k = 1, 2, ⋯, K); the stratified cross-validation technique is used to estimate the classification performance of the base classifier ek, and the obtained confusion matrix is denoted as \( {CM}_{M\times M}^k \).
-
(2)
By calculating the confusion matrix, we obtain the fuzzy density \( {g}_m^k\left(1\le m\le M\right) \) of base classifier ek in different sample regions.
-
(3)
We input \( {g}_m^k \) as a cloud droplet to the inverse cloud generator, obtain the three eigenvalues (Exk, Enk, Hek) of the cloud model and calculate the reliability αk of base classifier ek.
-
(4)
According to the posterior probability values of sample x to be tested belonging to each class obtained by base classifier ek, we calculate the decision credibility βk of the classifier using Eqs. (20) and (21).
-
(5)
We calculate the weight of classifier ek using Eq. (22) as follows:
-
(6)
We integrate the output components of each base classifier as follows:
-
(7)
The result of the integrated discrimination of sample x to be identified is
3.6 Analysis of computational complexity
Computational complexity focuses on the resources required to calculate the problem, such as time and spatial resources. The spatial resources required by the algorithm can meet the requirements of the general computer program. The time complexity of the base classifier for the training process is denoted as O(Classifiertraining). We calculate the confusion matrix of each base classifier and denote the time complexity of fuzzy density as O(MKn2), where M represents the number of pattern categories, K represents the number of base classifiers and n represents the number of each training samples. The time complexity of decision recognition is denoted as O(MK). Other processes are ignored. The total time complexity is denoted as O(O(ClassifierLearn) + n2MK + MK).
4 Application of the proposed algorithm in face recognition
4.1 Feature description of face image
Face recognition [6, 13, 24, 28, 41, 51] uses computers to verify the identity of a face. Face recognition is used to promote many related disciplines and related fields, such as emotion recognition, image process [45] and face detection [3, 22]. Face recognition consists of two steps: feature description and target classification. Commonly used image features include colour, shape and texture features. Amongst them, the texture features based on statistics are favoured by domestic and foreign scholars because of their simplicity and effectiveness; the representative methods are local binary pattern [61], central symmetric local binary pattern [18], local directional pattern [21], local gradient patterns [23], modified census transform [15] and local mean pattern [46]. In this study, the symmetric local graph structure (SLGS) [1] algorithm is used to extract the feature of face images. SLGS is an improvement of the local graph structure [2] operator and is also a local texture description algorithm based on statistics. Compared with the traditional binary pattern, SLGS is no longer limited to the circular neighbourhood of the central pixel, has a range of comparison that can be extended to the comparison between the neighbourhood points and obtains a richer texture feature. However, when describing the face texture feature, the SLGS algorithm only focuses on the texture information in the horizontal neighbourhood direction whilst ignoring the information in other neighbourhood directions. Accordingly, texture information in other directions is usually lost. The description of the facial features is insufficiently comprehensive. Therefore, SLGS is extended from the directional perspective by adding diagonal and vertical directions and defining 45° SLGS, 90° SLGS and 135° SLGS; the original SLGS referred to herein is 0° SLGS. Figure 3 shows the calculation of the SLGS coding in different directions. Figure 4 shows the SLGS reconstruction diagrams in different directions, including illumination, expression, gestures and occlusion changes. The reconstruction diagram clearly indicates the difference amongst the four different directions of SLGS in the texture feature description.
4.2 Face databases
The stability and validity of the proposed algorithm is tested by conducting simulated experiments on publicly available face databases including ORL, Yale A, FERET and AR databases. These databases cover various conditions of partial occlusion, complex illumination and different expressions and poses. The ORL database contains different images of 40 people; 10 face images are stored for each person, thereby comprising a total of 400 images, including poses, facial expressions and facial ornaments (e.g. glasses) and other changes. Each image is a 112 × 92 greyscale image. Some of the images in the ORL database are shown in Fig. 5a.
The Yale A database published by Yale University contains 11 images each from 15 individuals with lighting, occlusion and facial emotion variations. Some of the images in the Yale A database are shown in Fig. 5b.
The FERET database contains different images of 200 people, including frontal face images; face images with expression changes; face images under different illumination conditions; face images with left turn and right turn at 15°; face images with left turn and right turn at 30°. Some of the images in the FERET database are shown in Fig. 5c.
The AR database consists of more than 4000 frontal face images of 126 persons (70 men and 56 women). Each person has up to 26 images taken in two sessions. The first session contains 13 images, including different facial expressions, illumination variations and occlusions. The second session duplicates the first session two weeks later. To verify the robustness of the method to illumination, occlusion and expression changes, this study divides the AR database into four parts on the basis of different conditions that contain normal, occlusion, illumination and expression. As shown in Fig. 5d, the images numbered 1–4 are normal face images (the expression is neutral and without illumination changes and occlusion limits), those numbered 5–16 are face images under occlusion conditions and those numbered 23–26 are face images with expression changes. In the experiment, the images in the four face databases are pre-processed to 96 × 96 resolution.
5 Experiments
In the multi-classifier system, this study constructs the base classifier by separately using the nearest neighbour rule and BP neural network combined with the SLGS features in different directions. For the nearest neighbour classifier, the Euclidean distance between the query sample and the training samples in different classes is calculated firstly and then the posterior probability of the nearest neighbour classifier is constructed using the distance. BP neural network is composed of the input, hidden and output layers. In this study, the output of the BP classifier is in the form of posterior probability. The indicator of the performance of the classifier includes recognition accuracy, Kappa coefficient and efficiency. In this study, static ensemble is the fusion result obtained using reliability as the weight of the classifier. Dynamic ensemble refers to the fusion result obtained using reliability and decision credibility as the weight of the classifier. All experiments and comparisons are conducted under the same evaluation protocol.
5.1 Recognition rate comparison of each classifier under different blocks
In the experiment, five face images are selected randomly from ten face images of each individual and used as training set on the ORL database. In the Yale A database, five training samples of each person are randomly selected and used for training and the other images for testing. In the FERET database, four training samples of each person are randomly selected and used for training and the other images for testing. In the AR database, under occlusion conditions, we randomly select six pieces of numbers 5–16 for training and the other as testing samples. Under illumination conditions, three pieces of numbers 17–22 are randomly selected for training and the other as testing samples. Under the condition of expression changes, we randomly select two pieces of numbers 23–26 as facial expression image for training and the other as testing samples. Figure 6 shows the recognition results of the classifiers trained by four different algorithms and the proposed ensemble methods under different blocks.
Overall, in different face databases, the recognition performance firstly shows a rising trend and then a flat or descending trend with the increase in the number of blocks. Compared with the single classifier, the proposed multi-classifier fusion methods present good performance advantages regardless of their use in static and dynamic ensemble. Their recognition rates are also much higher than that of single classifier prior to fusion. Among them, the classification effect of the dynamic ensemble is better than that of the static ensemble. For the feature description algorithm, large number of blocks in a certain range means good details of the spatial texture structure described and good performance of the classifier. Given the increase in the number of blocks, the number of the pixels in different sub blocks gradually reduces and the operator’s feature description capability gradually weakens. Consequently, the classifier’s identification performance firstly rises and then declines. Figure 6 also shows that the classification performance of the classifier formed by different characteristics of the same training sample differs. By taking the FERET database as an example, the recognition performance of 0° SLGS + NN is in a state of equality with 90° SLGS + NN on the FERET database, whereas the recognition performance of 0° SLGS + NN is inferior to that of the 45° SLGS + NN. Such a scenario is also reflected from the side that the recognition capability of different classifiers to identify the same sample area differs. This study complements the missing information of original SLGS from the perspective of direction, and the classifiers trained by four different texture description operators construct the integrated system with the difference. In addition, the importance of the classifier in the fusion decision is described on the basis of the prior knowledge and posteriori output of each classifier. The three eigenvalues of the cloud model can fully describe the reliability of the classifier in different sample regions and can fully utilise the classification advantages of the classifier. The decision credibility can effectively describe the real-time performance of the classifier’s recognition performance with the change in the number of sample to be tested. Thus, the final dynamic ensemble achieves good recognition performance that is higher than that of static ensemble and single classifier. The increase in the number of blocks can also increase time complexity. Notably, this study aims to find a compromise between “simplicity” and “effectiveness.” In Fig. 6, when the number of blocks is 6 × 6, the system obtains the best recognition performance with a small time cost. Therefore, this parameter is used in the subsequent experiments to address the corresponding face image.
Kappa coefficient to further evaluate the optimal classification accuracy of dynamic and static ensembles. Kappa coefficient represents the proportion of the error reduction produced by the evaluated classification to that of the completely random classification. Large Kappa coefficient value means good performance of the classifier. In the contrast experiment, we select 13 images of each person as training samples on the AR database and the remaining samples as testing samples. The number of training samples on other face databases remains unchanged. Table 1 shows the comparison of the Kappa coefficients of static and dynamic ensemble methods.
Table 1 shows that the Kappa coefficients of the dynamic weighted ensemble method are higher than those of the static method on different databases. It can be seen from the definition of Kappa coefficient that the higher the Kappa value, the higher the accuracy of classification. Obviously, the method of dynamic weighted ensemble is better than the static method. In the process of weight acquisition, compared with the static method, the dynamic method not only considers the recognition ability of the classifier in different sample areas, but also takes the difference of the samples into consideration and considers the recognition result of the current target by this classifier as an important indicator in measuring the importance of it. Thereby, the acquired weight can effectively reduce the interference of unreliable decision information to fusion and thus obtaining improved classification performance.
5.2 Recognition rate comparison of each classifier under different number of training samples
In further verifying the effectiveness of the proposed algorithm, 2–8 images per person in the ORL and Yale A databases are used for training and the remaining samples for testing. In the FERET database, 2–5 training samples are selected randomly and the other images are for testing. In the AR database, 9–14 training samples are randomly selected and the other images are for testing. Following the conclusion of experiment (Section 5.1), the images in four face databases are blocked into 6 × 6 blocks. We repeat the experiments four times and calculate the average recognition rate for each database. The experimental results are shown in Fig. 7.
Figure 7 shows the curves of the recognition rates of the base classifiers based on four algorithms and the proposed multi-classifier fusion methods. These curves change with the increase in the number of training samples. In a certain range, large number of training samples results in complete classifier training and high recognition rate. In different face databases, the multi-classifier system composed of four different texture features and using NN or BP significantly improves after the decision fusion. Therefore, the method is valid. The reasons are that the four different texture features are portrayed from different directions in the feature description and that the classifier that comprises them is highly complementary. In the decision-making fusion, the dynamic ensemble exhibits a good description of the classifier’s reliability in different sample areas and real-time performance with the change in number of sample compared with the static ensemble. The former method also can be more targeted to play the recognition advantage of each classifier in different regions and different samples and ultimately improve the overall recognition rate. In the FERET and AR databases, the 0° SLGS + NN and 0°SLGS + BP classifiers present a lower performance than do those in the vertical and diagonal directions. By contrast, the performance of the four classifiers is comparable on the ORL and Yale A databases. The experimental results show that different directions of SLGS play different roles in describing the image texture features. Thus, SLGS should be extended from a directional perspective.
5.3 Recognition rate and identification time comparison of different multi-classifier ensemble methods
In this experiment, the optimal recognition rate obtained by the proposed dynamic ensemble method is compared with the recognition rate obtained by several typical fusion methods (e.g. voting method, maximum rule, D-S fusion rule and fuzzy integral method). The number of training samples of the ORL and Yale A databases is 5, that of the FERET database is 4, that of the AR database is 13 and the remaining samples are used as testing samples. The comparison results are shown in Table 2.
The experimental data in Table 2 indicate that the recognition performance of the maximum rule method, D-S fusion method, fuzzy integral method and dynamic ensemble method is better than that of the voting method. The voting method is a hard-grading standard, which considers that the input pattern either belongs to a certain class, or not. The voting method is an abstract-level fusion method, while the other fusion algorithms are belonging to metric-level fusion which contains more decision information.Thus they can achieve higher recognition rates than voting method. In the ORL and Yale A databases, the recognition performance of the maximum rule method, D-S fusion method and fuzzy integral method is close to that of the proposed method. On the contrary, the difference is significant in the AR database. Such a difference mainly occurs because the changes in illumination, occlusion and expression are complex in the AR database, thereby possibly causing different degrees of interference and reducing the performance of the fusion decision. D-S evidence theory is based on the assumption that different pieces of evidence should remain independent, thereby requiring that the classifiers should be independent of one another. However, in practice, complete independence is difficult. Fuzzy integral avoids the limitation of D-S evidence theory that the evidence needs to be completely independent, thus, it can effectively deal with the uncertainty of the classification decision. However, fuzzy integral generally makes the training performance index of the classifier as the basis for determining the fuzzy density, which no longer change once determined. When the classifier outputs unreliable information to the decision-making layer, decision-making interference occurs. Thus, the integrated classification effect becomes undesirable. Multi-classifier weighted ensemble can effectively emphasise the decision-making contribution made by excellent classifiers and suppress the influence of unreliable information output by high-deviation classifiers on classification. Thus it can reduce the interference of unreliable decision factors and still maintain its fusion decision-making performance above or close to that of the optimal classifier in the multi-classifier system. Therefore, the proposed method exhibits a stronger resistance to unreliable decision-making interference than common fusion methods do in the fusion decision making.
With regard to process time, that of different algorithms from fast to slow is in the following order: maximum rule method, dynamic ensemble method, D-S fusion method, fuzzy integral method and the voting method. The voting method consumes much time to find the nearest sample, especially when the training set is large. The fuzzy integral involves an iterative process and thus consumes much time when dealing with multiple types or large sample problems. The D-S fusion method presents increased time complexity during the construction of its basic probability distribution function, which reduces the classification efficiency. Given that the weights of the dynamic ensemble method are assigned by defining reliability and credibility, its reliability can be obtained by offline training and its credibility can be obtained by dispersion mapping of the classifier with negligible the time consumption for mapping. As a result, the dynamic integration method exhibits good efficiency in classification, which is only slightly below that of the maximum rule method. Considering the classification accuracy and efficiency of the proposed method in face recognition, its feasibility and effectiveness are verified.
To better illustrate the advantages of our method, the optimal experimental results of the multi-classifier dynamic ensemble method in face recognition are compared with those of some existing face recognition methods.
As shown in Table 3, the proposed multi-classifier dynamic ensemble method achieves a good classification performance. The classifier adjusts its weight by analysing its recognition capability in different regions of the sample set and the decision information of the output, thereby emphasising the “contribution” of the excellent classifier to the final classification result. Accordingly, some error-prone divisions can be corrected and the interference of unreliable decision output to fusion can be decreased.
6 Discussion
In a multi-classifier system, the base classifier should satisfy the difference and complementarity. If all the classifiers incorrectly identify the sample, then the ensemble will be meaningless. Therefore, a classifier set with difference and complementarity should be comprehensively constructed. In classifying different query samples, the performance of the classifiers shows a certain difference. If a numerical value is used to measure its importance, then the value should reflect not only the identification performance of the base classifier in the entire sample space but also the adaptability of the base classifier, which changes with the change in the number of query sample. This study aims to dynamically adjust the weights of each base classifier. Consequently, the weight of the classifier providing the correct classification information increases and the weight of the classifier providing incorrect information decreases. The weight exhibits strong adaptability and can be updated as the query sample changes.
7 Conclusions
In this study, we analyse the advantages and disadvantages of existing weighted ensemble methods on the basis of the study on ensemble learning. A dynamic weight assignment method that defines the reliability and credibility of the classifier is proposed. Compared with general methods, two aspects, namely, the performance on the training set and the decision information output when identifying specific target, are considered in acquiring the weights of the classifier. This method can thus describe the importance of the classifier comprehensively. The comparison between the results of simulation experiment on face recognition and those of the existing popular ensemble methods proves that the proposed method can effectively reduce the interference of unreliable decision information and efficiently improve the recognition performance of the classification system. The current study on multi-classifier ensemble method is of practical value to face recognition problem, of methodological significance to other fields in addition to machine learning and possesses a wide application prospect.
Issues related to multi-classifier ensemble methods, such as constructing and selecting efficient base classifiers and obtaining effective results of the base classifier, should be explored. In this study,the proposed method is only verified on standard face databases,corresonding optimisation and adjustment should be made according to the specific circustances when solving other pattern classification problems.Meanwhile, the base classifiers constructed in this paper are homogeneous, this would inevitably lead to the overlapping of wrong samples, which will affect the classification system to make accurate decision.The next research focus is how to screen the classifier by measuring the difference between the classifiers so as to obtain a better classification result.
References
Abdullah MFA, Sayeed MS, Muthu KS et al (2014) Face recognition with symmetric local graph structure (SLGS). Expert Syst Appl 41(14):6131–6137
Abusham EEA, Bashir HK (2011) Face recognition using local graph structure (LGS). Human-computer interaction. Interaction techniques and environments. Springer Berlin Heidelberg, pp 169–175
Ban Y, Kim SK, Kim S et al (2014) Face detection based on skin color likelihood. Pattern Recogn 47(4):1573–1585
Bejani M, Gharavian D, Charkari NM (2014) Audiovisual emotion recognition using ANOVA feature selection method and multi-classifier neural networks. Neural Comput & Applic 24(2):399–412
Bhimani J, Mi N, Leeser M et al (2017) FiM: performance prediction model for parallel computation in iterative data processing applications. IEEE International Conference on Cloud Computing
Chakraborti T, Chatterjee A (2014) A novel binary adaptive weight GSA based feature selection for face recognition using local gradient patterns, modified census transform, and local binary patterns. Eng Appl Artif Intell 33(1):80–90
Chen B, Wang S, Jiao LC (2015) PolSAR image classification method based on weighted majority vote ensemble. Huazhong Univ of Sci & Tech(Natural Science Edition) 43(3):79–82
Cheon Y, Kim D (2009) Natural facial expression recognition using differential-AAM and manifold learning. Pattern Recogn 42(7):1340–1350
Cho SB, Kim JH (1995) Combining multiple neural network by fuzzy integral for robust classification. IEEE Trans Syst Man Cybern 25(2):380–384
Codella N, Cai J, Abedini M et al (2015) Deep learning, sparse coding, and SVM for melanoma recognition in dermoscopy images. Machine learning in medical imaging, pp 118–126
Cui J, Liu Y, Xu Y et al (2013) Tracking generic human motion via fusion of low- and high-dimensional approaches. IEEE Trans Syst Man Cybern Syst 43(4):996–1002
Dai J, Liu H, Wang M et al (2017) Research and application of an uncertainty concept model: Time-varying cloud model. IEEE, International Conference on Cloud Computing and Big Data Analysis
Ding C, Choi J, Tao D et al (2016) Multi-directional multi-level dual-cross patterns for robust face recognition. IEEE Trans Pattern Anal Mach Intell 38(3):518–531
Eleftheriadis S, Rudovic O, Pantic M (2014) Discriminative shared Gaussian processes for multiview and view-invariant facial expression recognition. IEEE Trans Image Process 24(1):189–204
Froba B, Ernst A (2004) Face detection with the modified census transform. Proceedings of 6th International Conference on Automatic Face and Gesture Recognition. Seoul, South Korea: IEEE 91–96
Gao H, Yang Z, Bhimani J et al (2017) AutoPath: harnessing parallel execution paths for efficient resource allocation in multi-stage big data frameworks. International Conference on Computer Communications and Networks
Guo K, Li W (2011) Combination rule of D-S evidence theory based on the strategy of cross merging between evidences. Expert Syst Appl 38(10):13360–13366
Heikkilä M, Pietikäinen M, Schmid C (2009) Description of interest regions with local binary patterns. Pattern Recogn 42(3):425–436
Hu ZP, Li J, Zh SH (2013) Sub-modular sparse representation algorithm for robust pattern recognition based on Borda voted weighting. Chin J Sci Instrum 34(10):2309–2315
Huang ZH, Li WJ, Wang J et al (2015) Face recognition based on pixel-level and feature-level fusion of the top-level’s wavelet sub-bands. Inf Fusion 22:95–104
Jabid T, Kabir MH, Chae O (2010) Local directional pattern (LDP) —a robust image descriptor for object recognition. Proceedings of 7th International Conference on Advanced Video and Signal Based Surveillance. Boston, MA: IEEE, pp 482–487
Jin J, Xu B, Wang Y (2015) A face detection and location method based on feature binding. Signal Process Image Commun 36:179–189
Jun B, Kim D (2012) Robust face detection using local gradient patterns and evidence accumulation. Pattern Recogn 45(9):3304–3316
Kumar R, Banerjee A, Vemuri BC et al (2011) Maximizing all margins: pushing face recognition with kernel plurality. International Conference on Computer Vision. IEEE Computer Society, pp 2375–2382
Kuncheva LI, Rodríguez JJ (2014) A weighted voting framework for classifiers ensembles. Knowl Inf Syst 38(2):259–275
Kwak KC, Pedrycz W (2005) Face recognition: a study in information fusion using fuzzy integral. Pattern Recogn Lett 26(6):719–733
Lee C-C, Mower E, Busso C et al (2011) Emotion recognition using a hierarchical binary decision tree approach. Speech Comm 53(9):1162–1171
Li X, Song A (2013) Letters: fuzzy MSD based feature extraction method for face recognition. Neurocomputing 122:266–271
Li DY, Meng HJ, Shi XM (1995) Membership clouds and membership cloud generators. Computer R & D 32(6):15–20
Li G, Zhang Z, Wang L et al (2017) One-class collaborative filtering based on rating prediction and ranking prediction. Knowl-Based Syst 124:46–54
Liu HJ, Liu Z, Jiang WL et al (2010) Approach based on cloud model and vector neural network for emitter identification. Acta Electron Sin 38(12):2797–2804
Liu Y, Cui J, Zhao H et al (2012) Fusion of low-and high-dimensional approaches by trackers sampling for generic human motion tracking. International Conference on Pattern Recognition. IEEE, pp 898–901
Liu Y, Nie L, Han L et al (2015) Action2Activity: recognizing complex activities from sensor data. International Conference on Artificial Intelligence. AAAI Press, pp 1617–1623
Liu Y, Nie L, Liu L et al (2016) From action to activity: sensor-based activity recognition. Neurocomputing 181:108–115
Liu L, Cheng L, Liu Y et al (2016) Recognizing complex activities by a probabilistic interval-based model. Thirtieth AAAI Conference on Artificial Intelligence. AAAI Press, pp 1266–1272
Liu Y, Zhang L, Nie LQ et al (2016) Fortune teller: predicting your career path. Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence. AAAI-16, pp 201–207
Liu Y, Liang Y, Liu S et al (2016) Predicting urban water quality with ubiquitous data. arXiv:161009462
Liu Y, Zheng Y, Liang YX et al. Urban water quality prediction based on multi-task multi-view learning. Proceedings of the 25th International Joint Conference on Artificial Intelligence
Lu Y, Wei Y, Liu L et al (2017) Towards unsupervised physical activity recognition using smartphone accelerometers. Multimed Tools Appl 76(8):10701–10719
Luo Y, Chen S, He X et al (2013) Alphanumeric character recognition based on BP neural network classification and combined features. Int J Comput Intell Syst 6(6):1108–1115
Naik MK, Panda R (2016) A novel adaptive cuckoo search algorithm for intrinsic discriminant analysis based face recognition. Appl Soft Comput 38(C):661–675
Orrite C, Rodríguez M, Martínez F et al (2008) Classifier ensemble generation for the majority vote rule. Progress in Pattern Recognition, Image Analysis and Applications 5197:340–347
Owusu E, Zhan Y, Mao QR (2014) A neural-AdaBoost based facial expression recognition system. Expert Syst Appl 41(7):3383–3390
Preoţiuc-Pietro D, Liu Y, Hopkins D et al (2017) Beyond binary labels: political ideology prediction of twitter users. Meeting of the Association for Computational Linguistics, pp 729–740
Ren F, Li B, Chen Q (2013) Single parameter logarithmic image processing for edge detection. Ieice Trans Inform Syst 96(11):2437–2449
Ren FJ, Li YQ, Xu LF et al (2016) Face recognition method based on local mean pattern description and double weighted decision fusion for classification. J Image Graph 21(5):565–573
Santana MC, Marsico MD, Nappi M et al (2016) MEG: texture operators for multi-expert gender classification. Comput Vis Image Underst 156:4–18
Song YJ, Li DY, Yang XZ et al (2000) Reliability evaluation of electronic products based on cloud models. Acta Electron Sin 28(12):74–76 68
Sun B, He J (2015) Weighted joint sparse representation-based classification method for robust alignment-free face recognition. J Electron Imaging 24(1):013018
Wang XF, Xiao MS (2010) Approach of group decision making based on normal distribution interval number with incomplete information. Control and Decision 25(10):1494–1498
Wright J, Yang AY, Ganesh A et al (2009) Robust face recognition via sparse representation. IEEE Trans Pattern Anal Mach Intell 31(2):210
Xu Q (2013) A novel machine learning strategy based on two-dimensional numerical models in financial engineering. Math Probl Eng (2):1–6
Xu L, Krzyzak A, Suen CY (1992) Methods of combining multiple classifiers and their applications to handwriting recognition. IEEE Trans Cybern 22(3):418–435
Xu Q, Wu J, Chen Q (2014) A novel mobile personalized recommended method based on money flow model for stock exchange. Math Probl Eng 2014. https://doi.org/10.1155/2014/353910
Xu Q, Wang Z, Wang F et al (2017) Thermal comfort research on human CT data modeling. Multimed Tools Appl. https://doi.org/10.1007/s11042-017-4537-9
Yang Z, Awasthi M, Ghosh M et al (2017) A fresh perspective on total cost of ownership models for flash storage in datacenters. IEEE International Conference on Cloud Computing Technology and Science
Yang Z, Tai J, Bhimani J et al (2017) GReM: dynamic SSD resource allocation in virtualized storage systems with heterogeneous IO workloads. PERFORMANCE Computing and Communications Conference. IEEE
Yang Z, Wang J, Evans D et al (2017) AutoReplica: automatic data replica manager in distributed caching and data processing systems. PERFORMANCE Computing and Communications Conference. IEEE
Yang J, Li J, Liu S (2017) A novel technique applied to the economic investigation of recommender system. Multimed Tools Appl (8):1–16
Zhang YS, Guo J (2012) Word sence disambiguation based on ensemble classifier with dynamic weight adaptation. J Chin Inform Process 26(1):3–9
Zhang JY, Zhao HP, Chen S (2014) Face recognition based on weighted local binary pattern with adaptive threshold. J Electron Inform Technol 36(6):1327–1333
Acknowledgements
This research has been partially supported by National Natural Science Foundation of China under Grant Nos. 61432004, 61672202 and 61502141 and by JSPS KAKENHI under Grant No. 15H01712.
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made.
About this article
Cite this article
Ren, F., Li, Y. & Hu, M. Multi-classifier ensemble based on dynamic weights. Multimed Tools Appl 77, 21083–21107 (2018). https://doi.org/10.1007/s11042-017-5480-5
Received:
Revised:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11042-017-5480-5