A semi-hard voting combiner scheme to ensemble multi-class probabilistic classifiers

Abstract

Ensembling of probabilistic classifiers is a technique that has been widely applied in classification, allowing to build a new classifier combining a set of base classifiers. Of the different schemes that can be used to construct the ensemble, we focus on the simple majority vote (MV), which is one of the most popular combiner schemes, being the foundation of the meta-algorithm bagging. We propose a non-trainable weighted version of the simple majority vote rule that, instead of assign weights to each base classifier based on their respective estimated accuracies, uses the confidence level CL, which is the standard measure of the degree of support that each one of the base classifiers gives to its prediction. In the binary case, we prove that if the number of base classifiers is odd, the accuracy of this scheme is greater than that of the majority vote. Moreover, through a sensitivity analysis, we show in the multi-class setting that its resilience to the estimation error of the probabilities assigned by the classifiers to each class is greater than that of the average scheme. We also consider another simple measure of the degree of support that incorporates additional knowledge of the probability distribution over the classes, namely the modified confidence level MCL. The usefulness for bagging of the proposed weighted majority vote based on CL or MCL is checked through a series of experiments with different databases of public access, resulting that it outperforms the simple majority vote in the sense of a statistically significant improvement regarding two performance measures: Accuracy and Matthews Correlation Coefficient (MCC), while holding up against the average combiner, which majority vote does not, being less computationally demanding.

Appendices

Appendix : A: Properties of MCL

Proposition 5

For r ≥ 2,

• a)
  $$\frac{1}{r}\le \textup{CL} \le \textup{MCL} \le \textup{CL}+(1-\textup{CL}) \frac{r \textup{CL}-1}{r-1}\le 1 .$$

• b) If r > 2, fixed $\widetilde {\textup {CL}}$, MCL is an increasing function of CL, while fixed CL it is a straight line of non-positive slope as function of $\widetilde {\textup {CL}}$, then decreasing. Moreover, if $\widetilde {\textup {CL}}\le 1/r$,

  $$ \textup{CL}\le \textup{CL} + (1-\textup{CL}) \frac{r \textup{CL}-1}{r}\le {\textup{MCL}} $$

Proof

• a) The first inequality is evident by definition of CL. Second inequality is obvious, being strict except if CL= 1 or $\widetilde {\text {CL}}=$CL. Third inequality is due to the fact that

$$\widetilde{\text{CL}}\ge \frac{1-\text{CL}}{r-1} ,$$

which is obvious by definition of $\widetilde {\text {CL}}$, since a total probability of 1 −CL is divided among the r − 1 non-maximum values, from which $\widetilde {\text {CL}}$ is defined, in turn, as the maximum.

Finally, we prove that $\text {CL}+(1-\text {CL}) \frac {r \text {CL}-1}{r-1}\le 1$. Indeed, simple algebraic manipulations show that this inequality is equivalent to (1 −CL)² ≥ 0, what is obviously fulfilled. This inequality is strict except if CL= 1.

• b) Fixed $\widetilde {\text {CL}}$, MCL as function of x = CL is $g(x)=-x^{2}+(2+\widetilde {\text {CL}}) x-\widetilde {\text {CL}}$, which is strictly increasing since its first derivative is $g^{\prime }(x)=-2 x+(2+\widetilde {\text {CL}})$, which is > 0 for 0 < x ≤ 1. On the other hand, fixed CL, as function of $z=\widetilde {\text {CL}}$ MCL is h(z) = −(1 −CL)z + CL + (1 −CL)CL.

□

Corollary 1

For r ≥ 2,

$$\textup{CL}=1 \Longleftrightarrow \textup{MCL}=1$$

Proof

By definition of MCL, if CL= 1 then MCL= CL = 1.

The reverse implication is also true. Indeed, MCL= 1 implies by Proposition 5 a) that

$$\text{CL}+(1-\text{CL}) \frac{r \text{CL}-1}{r-1}=1 ,$$

which is equivalent to (1 −CL)² = 0, implying CL = 1. □

Appendix B: Complementary tables

Table 13 Comparison between MCL-MV, CL-MV, majority vote and average combiner schemes, with the different choices for the number of bags in bagging, attending to Accuracy and MCC, for different datasets

Table 14 Continuation of Table 13

Table 15 Comparison between MCL-MV, CL-MV, majority vote and average combiner schemes, with the different choices for the number of bags in bagging, attending to the mean running times

Table 16 Average over the runs of the averages over the folds, for the metrics Accuracy and MCC, with the different combiner ensembles used for bagging purpose, for all the datasets