Online AdaBoost-based methods for multiclass problems

Abstract

Boosting is a technique forged to transform a set of weak classifiers into a strong ensemble. To achieve this, the components are trained with different data samples and the hypotheses are aggregated in order to perform a better prediction. The use of boosting in online environments is a comparatively new activity, inspired by its success in offline environments, which is emerging to meet new demands. One of the challenges is to make the methods handle significant amounts of information taking into account computational constraints. This paper proposes two new online boosting methods: the first aims to perform a better weight distribution of the instances to closely match the behavior of AdaBoost.M1 whereas the second focuses on multiclass problems and is based on AdaBoost.M2. Theoretical arguments were used to demonstrate their convergence and also that both methods retain the main features of their traditional counterparts. In addition, we performed experiments to compare the accuracy as well as the memory usage of the proposed methods against other approaches using 20 well-known datasets. Results suggest that, in many different situations, the proposed algorithms maintain high accuracies, outperforming the other tested methods.

Appendices

Appendix A. Proof for Theorem 3

Proof

Considering that

$$\begin{aligned} F(x,y) = \sum ^{T}_{t=1} \alpha _t \left( [[y = h_t(x)]] - [[y \ne h_t(x)]]\right) , \end{aligned}$$

(10)

where

$$\begin{aligned} \alpha _t = \frac{1}{2} \times ln \left( \frac{1-\varepsilon _t}{\varepsilon _t}\right) . \end{aligned}$$

(11)

Taking into account that

$$\begin{aligned} w_{t+1,i}&= \frac{w_{t,i} \times \beta ^{1-[[h_t(x_i) \ne y_i]]}_{t}}{Z_t}&\end{aligned}$$

(12)

$$\begin{aligned}&= \frac{w_{t,i} \times e^{-\alpha _t\left( [[y_i = h_t(x_i)]] - [[y_i \ne h_t(x_i)]]\right) }}{Z_t}, \end{aligned}$$

(13)

we can say that, after passing through all classifiers t, the final distribution of instance i ($w_{T+1,i}$) will be defined by

$$\begin{aligned} w_{T+1,i}&= w_{1,i} \times \frac{e^{-\alpha _1\left( [[y_i = h_1(x_i)]] - [[y_i \ne h_1(x_i)]]\right) }}{Z_1} \times \cdots \times \frac{e^{-\alpha _T\left( [[y_i = h_T(x_i)]] - [[y_i \ne h_T(x_i)]]\right) }}{Z_T}\nonumber \\&= \frac{w_{1,i} \times exp\left( -\sum ^{T}_{t=1}\alpha _{t}([[y_i = h_t(x_i)]] - [[y_i \ne h_t(x_i)]])\right) }{\prod ^{T}_{t=1}Z_t}. \end{aligned}$$

(14)

Combining Eq. (10) with Eq. (14), we obtain

$$\begin{aligned} w_{T+1,i} = \frac{w_{1,i} \times e^{-F(x_i,y_i)}}{\prod ^{T}_{t=1}Z_t}. \end{aligned}$$

(15)

To initiate the construction of the Eq. (8), consider that, given an instance x, an arbitrary class y has not been chosen as a response by the committee ($h_f(x) \ne y$). Knowing this, we can say that there is another class ($\ell$) that satisfies the following inequalities:

$$\begin{aligned} \sum ^{T}_{t=1}\alpha _t[[y = h_t(x)]] \le \sum ^{T}_{t=1}\alpha _t[[\ell = h_t(x)]] \le \sum ^{T}_{t=1}\alpha _t[[y \ne h_t(x)]]. \end{aligned}$$

(16)

It is important to note that using $\alpha _t$ in the weighting of the committee’s final response will have the same effect as $\beta _t$, used in OABM1. In addition, the latter inequality takes advantage of the fact that $\alpha _t \ge 0$ since $\varepsilon _t < 1/2$. Thus, $h_f(x) \ne y$ implies that $F(x,y) \le 0$. Therefore,

$$\begin{aligned}{}[[h_f(x) \ne y]] \le e^{-F(x,y)}. \end{aligned}$$

(17)

Observing the Eq. (15), the Eq. (17), and taking into account that the training error (left side of the inequality in Eq. (8)) is defined by the sum of the weights of instances incorrectly classified by $h_f$, we have to:

$$\begin{aligned} Pr_{i \sim w_1}[h_f(x_i) \ne y_i]&= \sum ^{N}_{i=1}w_{1,i} \times [[h_f(x_i) \ne y_i]]\nonumber \\&\le \sum ^{N}_{i=1} w_{1,i} \times e^{-F(x_i,y_i)} \end{aligned}$$

(18)

$$\begin{aligned}&= \sum ^{N}_{i=1}w_{T+1,i} \times \prod ^{T}_{t=1}Z_t. \end{aligned}$$

(19)

The Eq. (18) takes into account the definition of (17), while Eq. (19) uses (15). However, since all instances are not known a priori in an online version, the iteration in N will be done in a gradual way. Therefore,

$$\begin{aligned} Pr_{i \sim w_1}[h_f(x_i) \ne y_i] \le \sum ^{N}_{i=1} \sum ^{i}_{j=1} \left( w_{T+1,j} \times \prod ^{T}_{t=1} Z_t \right) , \end{aligned}$$

(20)

where the iteration in j represents the incremental inclusions of new instances. Finally, to get (8), the value of $Z_t$, present in (20), needs to be defined. Rewriting the Eq. (12) more completely and in accordance with Line 14 of Algorithm 3, we obtain:

$$\begin{aligned} w_{t+1,i}&= \left( w_{t,i} \times \beta ^{1-[[h_t(x_i) \ne y_i]]}_{t} \right) \big / \left( \tfrac{\xi c_{t}}{cw_{t}} \times \beta _{t}^{1} + \tfrac{\xi w_{t}}{cw_{t}} \times \beta _{t}^{0} \right) \end{aligned}$$

(21)

$$\begin{aligned}&= \left( w_{t,i} \times e^{-\alpha _t\left( [[y_i = h_t(x_i)]] - [[y_i \ne h_t(x_i)]]\right) } \right) \big / \left( \tfrac{\xi c_{t}}{cw_{t}} \times e^{-\alpha _t} + \tfrac{\xi w_{t}}{cw_{t}} \times e^{\alpha _t} \right) . \end{aligned}$$

(22)

Given that $\varepsilon _t = \xi w_{t}/cw_{t}$, the denominator of 22 can be expressed by:

$$\begin{aligned} Z_t&= (1-\varepsilon _t) \times e^{-\alpha _t} + \varepsilon _t \times e^{\alpha _t}\end{aligned}$$

(23)

$$\begin{aligned}&= (1/2 + \gamma _t) \times e^{-\alpha _t} + (1/2 - \gamma _t) \times e^{\alpha _t} \end{aligned}$$

(24)

$$\begin{aligned}&= \sqrt{1-4\gamma ^{2}_{t}}, \end{aligned}$$

(25)

where the Eqs. (24) and (25) use the definitions of $\gamma _t$ and $\alpha _t$, respectively. In this way, by plugging the value of $Z_t$ (present in Eq. 25) into (20), we get the main equation (8), completing this demonstration. $\square$

Appendix B. Proof for Theorem 4

Proof

Considering that

$$\begin{aligned} Pr(x,y) = \tfrac{1}{2} \left( 1 - h_t(x,y_i) + h_t(x,y)\right) \end{aligned}$$

(26)

and that

$$\begin{aligned} F(x,y) = \sum ^{T}_{t=1} \alpha _t \left( h_t(x,y_i) - h_t(x,y)\right) , \end{aligned}$$

(27)

where

$$\begin{aligned} \alpha _t = \frac{1}{2} \times ln \left( \frac{1-\varepsilon _t}{\varepsilon _t}\right) . \end{aligned}$$

(28)

Taking into account that

$$\begin{aligned} w_{t+1,i}&= \frac{w_{t,i} \times \beta ^{(1/2)(1 + h_t(x_i,y_i) - h_t(x_i,y))}_{t}}{Z_t} \end{aligned}$$

(29)

$$\begin{aligned}&\approx \frac{w_{t,i} \times e^{-\alpha _t\left( h_t(x_i,y_i) - h_t(x_i,y)\right) }}{Z_t}, \end{aligned}$$

(30)

we can say that, after passing through all classifiers t, the final distribution of instance i ($w_{T+1,i}$) will be defined by

$$\begin{aligned} w_{T+1,i}&= w_{1,i} \times \frac{e^{-\alpha _1\left( h_1(x_i,y_i) - h_1(x_i,y)\right) }}{Z_1} \times \cdots \times \frac{e^{-\alpha _T\left( h_T(x_i,y_i) - h_T(x_i,y)\right) }}{Z_T} \nonumber \\&= \frac{w_{1,i} \times exp\left( -\sum ^{T}_{t=1}\alpha _{t}(h_t(x_i,y_i) - h_t(x_i,y))\right) }{\prod ^{T}_{t=1}Z_t}. \end{aligned}$$

(31)

Combining the Eq. (27) with the Eq. (31), we obtain

$$\begin{aligned} w_{T+1,i} = \frac{w_{1,i} \times e^{-F(x_i,y_i)}}{\prod ^{T}_{t=1}Z_t}. \end{aligned}$$

(32)

Whereas, giving an instance x, an arbitrary class y has not been chosen as a response by the committee ($h_f(x) \ne y$), then

$$\begin{aligned} \sum ^{T}_{t=1}\alpha _t\left( h_t(x,y_i) - h_t(x,y)\right) \le \sum ^{T}_{t=1}\alpha _t\left( h_t(x,y_i) - h_t(x,h_f(x))\right) . \end{aligned}$$

(33)

Thus, $h_f(x) \ne y$ implies that $F(x,y) \le 0$. This statement can be justified by the following fact: according to the Line 38 of OABM2 (Algorithm 4), the class chosen by the committee will be the one with the highest plausibility among the classifiers, making $F(x,y) \le F(x,h_f(x)) \le 0$. Therefore,

$$\begin{aligned} Pr(x,h_f(x)) \le e^{-F(x,y)}. \end{aligned}$$

(34)

Taking into account that the training error (left side of the inequality in Eq. 9) is defined by the sum of the distribution weights multiplied by the error probability of $h_f$ (for each instance) (Freund and Schapire 1997), we have:

$$\begin{aligned} Pr_{i \sim w_1}[h_f(x_i) \ne y_i]&= \sum ^{N}_{i=1}w_{1,i} \times Pr(x_i,h_f(x_i)). \end{aligned}$$

(35)

In addition, assuming that all $k-1$ iterations that occur in OABM2 are equally important, we can redefine the Eq. (35) as follows:

$$\begin{aligned} Pr_{i \sim w_1}[h_f(x_i) \ne y_i]&= \sum ^{N}_{i=1}w_{1,i} \times (k-1) \times Pr(x_i,h_f(x_i)). \end{aligned}$$

(36)

Then, replacing the definitions of the Eq. (32) and the Eq. (34) in (36) we get:

$$\begin{aligned} Pr_{i \sim w_1}[h_f(x_i) \ne y_i]&\le \sum ^{N}_{i=1} w_{1,i} \times (k-1) \times e^{-F(x_i,y_i)} \end{aligned}$$

(37)

$$\begin{aligned}&= \sum ^{N}_{i=1}w_{T+1,i} \times (k-1) \times \prod ^{T}_{t=1}Z_t. \end{aligned}$$

(38)

The Eq. (37) takes into account the definition of (34), while the Eq. (38) uses (32). However, since all instances are not known a priori in an online version, the iteration in N will be done in a gradual way. So,

$$\begin{aligned} Pr_{i \sim w_1}[h_f(x_i) \ne y_i] \le \sum ^{N}_{i=1} \sum ^{i}_{j=1} \left( w_{T+1,j} \times (k-1) \times \prod ^{T}_{t=1} Z_t \right) , \end{aligned}$$

(39)

where the iteration in j represents the incremental inclusions of new instances. Finally, to get (9), the value of $Z_t$, present in (39), needs to be defined. Rewriting the Eq. (29) more completely and in accordance with Line 35 of Algorithm 4, we obtain

$$\begin{aligned} w_{t+1,i}&\approx \left( w_{t,i} \times \beta ^{(1/2)(1 + h_t(x_i,y_i) - h_t(x_i,y))}_{t} \right) \Big / \left( \textstyle \sum _{i=1}^{N} w_{t,i} \times \beta ^{(1/2)(1 + h_t(x_i,y_i) - h_t(x_i,y))}_{t} \right) \end{aligned}$$

(40)

$$\begin{aligned}&\approx \left( w_{t,i} \times e^{-\alpha _t\left( h_t(x_i,y_i) - h_t(x_i,y)\right) }_{t} \right) \Big / \left( \textstyle \sum _{i=1}^{N} w_{t,i} \times e^{-\alpha _t\left( h_t(x_i,y_i) - h_t(x_i,y)\right) }_{t} \right) , \end{aligned}$$

(41)

with $Z_t$ corresponding to the denominators of Eqs. (40) and (41). Since the hypothesis of OABM2 is in the range $[-1,+1]$, $Z_t$ can be rearranged in the same way as described in subsection 3.1 of Schapire and Singer (1999) and in subsection 9.2.3 of Schapire and Freund (2012e). For this purpose, assume that u and r are defined by:

$$\begin{aligned} u&= h_t(x,y_i) - h_t(x,y) \end{aligned}$$

(42)

$$\begin{aligned} r&= \sum _{i=1}^{N} w_{t,i} (h_t(x_i,y_i) - h_t(x_i,y)). \end{aligned}$$

(43)

Thus,

$$\begin{aligned} Z_t&= \sum _{i=1}^{N} w_{t,i} \times e^{-\alpha _t\left( h_t(x_i,y_i) - h_t(x_i,y)\right) }_{t} \nonumber \\&= \sum _{i=1}^{N} w_{t,i} \times e^{-\alpha _t u_i}_{t}\end{aligned}$$

(44)

$$\begin{aligned}&\approx \sum _{i=1}^{N} w_{t,i} \left[ \left( \frac{1+u_i}{2}\right) e^{-\alpha _t} + \left( \frac{1-u_i}{2}\right) e^{\alpha _t} \right] \nonumber \\&= \left( \frac{e^{\alpha _t}+e^{-\alpha _t}}{2} \right) - \left( \frac{e^{\alpha _t}-e^{-\alpha _t}}{2} \right) r. \end{aligned}$$

(45)

By the definition of $\alpha _t$ and $\gamma _t$, and knowing that $\varepsilon _t = (1-r)/2$ (Schapire and Singer 1999), its possible replace them in 45 and get to:

$$\begin{aligned} Z_t \approx \sqrt{1-4\gamma ^{2}_{t}}. \end{aligned}$$

(46)

Plugging the value of $Z_t$ (from Eq. 46) into (39), we get the main equation (9), completing this demonstration.$\square$