DCA for online prediction with expert advice

Abstract

We investigate DC (Difference of Convex functions) programming and DCA (DC Algorithm) for a class of online learning techniques, namely prediction with expert advice, where the learner’s prediction is made based on the weighted average of experts’ predictions. The problem of predicting the experts’ weights is formulated as a DC program for which an online version of DCA is investigated. The two so-called approximate/complete variants of online DCA based schemes are designed, and their regrets are proved to be logarithmic/sublinear. The four proposed algorithms for online prediction with expert advice are furthermore applied to online binary classification. Experimental results tested on various benchmark datasets showed their performance and their superiority over three standard online prediction with expert advice algorithms—the well-known weighted majority algorithm and two online convex optimization algorithms.

Appendices

Appendix 1: DC decomposition of ${f}^{(i)}_t$

We present the following proposition to get a DC decomposition of ${f}^{(i)}_t$.

Proposition 3

Let $a > 0$, b, c and x be three constants and a given vector, respectively. The function

$$\begin{aligned} f(w) = \max \{0, \min \{a, b\langle w, x \rangle + c\}\}, \end{aligned}$$

(22)

is a DC function with DC components

$$\begin{aligned}g(w) = \max \{0, b\langle w, x \rangle + c\}~\text { and } ~ h(w) = \max \{0, b\langle w, x \rangle + c - a \}.\end{aligned}$$

Proof

Knowing from [54] that if $f = g - h$ is a DC function, then the function

$$\begin{aligned} \max \{0, g - h\} = \max \{g,h\} - h \end{aligned}$$

(23)

is DC too. We see that the function

$$\begin{aligned}\min \{a, b\langle w, x \rangle + c\} = (b\langle w, x \rangle + c) - \max \{0,b\langle w, x \rangle + c - a \}\end{aligned}$$

is DC. Therefore, applying (23) for $g = b\langle w, x \rangle + c$ and $h = \max \{0,b\langle w, x \rangle + c - a \}$, we get immediately a DC decomposition of f, that is,

$$\begin{aligned}&f(w) = \max \{{b\langle w, x \rangle + c,\max \{0,b\langle w, x \rangle + c - a \}}\} \\&\quad - \max \{0,b\langle w, x \rangle + c - a \}. \end{aligned}$$

However, since $a > 0$, we have

$$\begin{aligned}\max \{{b\langle w, x \rangle + c,\max \{0,b\langle w, x \rangle + c - a \}}\} = \max \{0,b\langle w, x \rangle + c\}.\end{aligned}$$

Hence, we complete the proof. $\square$

Appendix 2: Proof of Lemma 1

Proof

First, we observe that when $\tilde{\mathtt {p}}_{t-1} = 0$ or $\tau ^{(1)}_{t-1} > \langle w^{t-1}, \tilde{\mathtt {p}}_{t-1} \rangle$ (resp. $\tau ^{(2)}_{t-1} < \langle w^{t-1}, \tilde{\mathtt {p}}_{t-1} \rangle$), our algorithms do not update the weight, and thus, the result in Lemma 1 is straightforward. Hence, we need to consider only the cases where, at each step $t \in \mathcal {M}$,

$$\begin{aligned} \tau ^{(1)}_{t-1} = \langle w^{t-1}, \tilde{\mathtt {p}}_{t-1} \rangle ~(\mathrm {resp. }~ \tau ^{(2)}_{t-1} = \langle w^{t-1}, \tilde{\mathtt {p}}_{t-1} \rangle ) ~\mathrm {and}\,~ \tilde{\mathtt {p}}_{t-1} \ne 0. \end{aligned}$$

(24)

Since $u^*$ satisfies (18) and $\tilde{\mathtt {p}}_t \ne 0$, Assumption 1 (i) is satisfied for DC functions (8). We see that $\Vert u^* - w^t \Vert _2 \ne 0$ for all t because if there is some step t such that $u^* = w^t$, then $\langle u^*,\tilde{\mathtt {p}}_t \rangle = \langle w^t,\tilde{\mathtt {p}}_t \rangle$, which contradicts (18).

Next, we verify Assumptions 1 (ii)–(iv) for the DC functions $f^{(1)}_t$.

$\bullet$ Let us define the function $\overline{g}^{(1)}_t := {g}^{(1)}_t - \langle z^t, \cdot \rangle .$ From (24), we derive

$$\begin{aligned} \overline{g}^{(1)}_t(w^t)-\overline{g}^{(1)}_t(u^*) = \frac{\rho - \langle w^t, \tilde{\mathtt {p}}_t \rangle }{\rho - \tau ^{(1)}_{t}} = 1 > 0. \end{aligned}$$

Thus, there exists a positive number $\alpha \le \min \limits _{t \in \mathcal {M}}{\frac{2 }{ \Vert u^*-w^t\Vert _2^2}}$ such that Assumption 1 (ii) is satisfied.

$\bullet$ For any $\beta \ge 0$, we have

$$\begin{aligned} h^{(1)}_t(u^*)-h^{(1)}_t(w^t) - \langle z^t,u^*-w^t \rangle = 0 \le \frac{\beta }{2} \Vert u^*-w^t \Vert _2^2. \end{aligned}$$

Thus, Assumption 1 (iii) is satisfied.

$\bullet$ Assumption 1 (iv) is also verified with $\gamma \le \min \limits _{t \in \mathcal {M}}{\frac{2(\langle u^*, \tilde{\mathtt {p}}_t \rangle -\rho )}{(\rho - \tau ^{(1)}_{t})\Vert u^*-w^t\Vert _2^2}}$ since

$$\begin{aligned}&{g}^{(1)}_t(u^*)-{g}^{(1)}_t(w^t) - \langle r^t, u^*-w^t \rangle = \frac{\langle u^*, \tilde{\mathtt {p}}_t \rangle -\rho }{\rho - \tau ^{(1)}_{t}}\\&\quad \ge \frac{\gamma }{2} \Vert u^*-w^t \Vert _2^2. \end{aligned}$$

Similarly, as for the DC functions $f_t^{(i)}$ ($i=2,3$), Assumptions 1 (i)–(iv) are satisfied if

$$\begin{aligned}\alpha\le & {} \min \limits _{t \in \mathcal {M}}{\frac{2}{\Vert u^*-w^t\Vert _2^2}}, \beta \ge 0, \\ \gamma\le & {} \min \limits _{t \in \mathcal {M}}{\frac{2}{\Vert u^*-w^t\Vert _2^2}\left( \dfrac{\langle u^*,\tilde{\mathtt {p}}_t \rangle -\rho }{\rho -\tau ^{(i)}_t} - \delta ^{(i)}\right) }.\end{aligned}$$

The proof of Lemma 1 is established. $\square$

Appendix 3: Proof of Theorem 1

Proof

First of all, we analyze the regret bound of ODCA-SG.

From the definition (17), we have

$$\begin{aligned}&\mathsf {Regret}^T_{\text {ODCA-SG}} = \sum \limits _{t=1}^T f_t(w^t) - \min \limits _{w \in \mathcal {S}} \sum \limits _{t=1}^T f_t(w)\nonumber \\&\quad \quad \le \sum \limits _{t=1}^T \left[ f_t(w^t) - \min \limits _{w \in \mathcal {S}} f_t(w) \right] . \end{aligned}$$

(25)

It derives from Assumption 1 (i) that

$$\begin{aligned}&f_t(w^t) - \min \limits _{w \in \mathcal {S}} f_t(w) = f_t(w^t) - f_t(u^*) \nonumber \\&\quad = [\overline{g}_t(w^t) - \overline{g}_t(u^*)] + [h_t(u^*)-h_t(w^t) - \langle z^t,u^*-w^t \rangle ], \end{aligned}$$

(26)

where the convex function $\overline{g}_t := g_t - \langle z^t, \cdot \rangle$ for $t=1,\ldots ,T$.

From (25), (26) and Assumptions 1 (ii)–(iii) with the choice $\beta = \alpha$, we obtain

$$\begin{aligned}&\mathsf {Regret}^T_{\text {ODCA-SG}} {\le } 2\sum \limits _{t=1}^T [\overline{g}_t(w^t) - \overline{g}_t(u^*)] \nonumber \\&\quad = 2\sum \limits _{t=1}^T [\overline{g}_t(w^{t,0}) - \overline{g}_t(u^*)] \end{aligned}$$

(27)

$$\begin{aligned}&\quad \le 2\sum \limits _{t=1}^T \left\{ \sum \limits _{k=0}^{K_t-2} [\overline{g}_t(w^{t,k}) - \overline{g}_t(w^{t,k+1})] + [ \overline{g}_t(w^{t,K_t-1})-\overline{g}_t(u^*)]\right\} \nonumber \\&\quad \le 2\sum \limits _{t=1}^T \left\{ \sum \limits _{k=0}^{K_t-2} \langle s^{t,k}, w^{t,k}-w^{t,k+1} \rangle + \langle s^{t,K_t-1}, w^{t,K_t-1} - u^* \rangle \right\} . \end{aligned}$$

(28)

The last inequality holds as $s^{t,k} \in \partial \overline{g}_t(w^{t,k})$, $k=0,1,\ldots ,K_t-1$.

Similarly to Theorem 3.1 in [30], we can derive from (13) an upper bound of $\langle s^{t,K_t-1}, w^{t,K_t-1} - u^* \rangle$ as follows:

$$\begin{aligned}&\langle s^{t,K_t-1}, w^{t,K_t-1} - u^* \rangle \nonumber \\&\quad \le \frac{ \Vert w^{t,K_t-1} - u^* \Vert _2^2 - \Vert w^{t,K_t} - u^* \Vert _2^2 }{2\eta _t} + \frac{\eta _t}{2} \Vert s^{t,K_t-1} \Vert _2^2. \end{aligned}$$

(29)

Combining (19) and the fact that

$$\begin{aligned}&\Vert w^{t,K_t-1} - u^* \Vert _2 \le \Vert w^{t,K_t-1} - w^{t,0} \Vert _2 + \Vert w^{t,0} - u^* \Vert _2 \\&\quad \le \sum _{k=0}^{K_t-2} \Vert w^{t,k+1} - w^{t,k} \Vert _2 + \Vert w^{t} - u^* \Vert _2\\&\quad \le \sum _{k=0}^{K_t-2} \eta _t \Vert s^{t,k}\Vert _2 + \Vert w^{t} - u^* \Vert _2 \\&\quad \le \eta _t (K-1) L + \Vert w^{t} - u^* \Vert _2, \end{aligned}$$

we yield

$$\begin{aligned} \Vert w^{t,K_t-1} - u^* \Vert _2^2 \le \Vert w^{t} - u^* \Vert _2^2 + 3 \eta _t^2 (K-1)^2 L^2. \end{aligned}$$

(30)

It implies

$$\begin{aligned}&\langle s^{t,K_t-1}, w^{t,K_t-1} - u^* \rangle \nonumber \\&\quad \le \frac{\Vert w^{t} - u^* \Vert _2^2 - \Vert w^{t+1} - u^* \Vert _2^2 }{2\eta _t} + \frac{\eta _t}{2} (3K^2-6K+4) L^2. \end{aligned}$$

(31)

Similarly, we get

$$\begin{aligned}&\sum \limits _{k=0}^{K_t-2} \langle s^{t,k}, w^{t,k}-w^{t,k+1} \rangle \nonumber \\&\quad \le \sum \limits _{k=0}^{K_t-2} \left[ \frac{ \Vert w^{t,k} - w^{t,k+1} \Vert _2^2}{2\eta _t} + \frac{\eta _t}{2} \Vert s^{t,k} \Vert _2^2 \right] \nonumber \\&\quad \le \sum \limits _{k=0}^{K_t-2} \left[ \eta _t \Vert s^{t,k} \Vert _2^2 \right] \le \eta _t (K-1) L^2. \end{aligned}$$

(32)

We deduce from (28), (31), (32) that

$$\begin{aligned}&\mathsf {Regret}^T_{\text {ODCA-SG}} \\&\quad \le 2 \sum _{t=1}^T \left[ \frac{\Vert w^{t} - u^* \Vert _2^2 - \Vert w^{t+1} - u^* \Vert _2^2 }{2\eta _t} + \frac{\eta _t}{2} (3K^2-4K+2) L^2 \right] \\&\quad \le 2 \sum _{t=1}^T \left[ \Vert w^{t} - u^* \Vert _2^2 \left( \frac{1}{2\eta _t} - \frac{1}{2\eta _{t-1}} \right) + \frac{\eta _t}{2} (3K^2-4K+2) L^2 \right] , \end{aligned}$$

where, by convention, $\frac{1}{\eta _0}:=0$.

Let us define $\eta _t := \dfrac{1}{\sqrt{t}\sqrt{3K^2-4K+2}}$ for all $t=1,\ldots ,T$. We have

$$\begin{aligned} \mathsf {Regret}^T_{\text {ODCA-SG}}&\le 2\sqrt{3K^2-4K+2}\left[ \frac{L^2\sqrt{T}}{2} + \frac{L^2}{2} \left( 2\sqrt{T}\right) \right] \\&\le 3L^2\sqrt{T}\sqrt{3K^2-4K+2}. \end{aligned}$$

As for ODCA-SGk, since Assumption 1 (iv) is also satisfied, we can derive from (27) that

$$\begin{aligned}&\mathsf {Regret}^T_{\text {ODCA-SG}k} \le 2 \sum _{t=1}^T \left[ \langle s^t, w^t-u^* \rangle - \frac{\gamma }{2}\Vert u^*-w^t \Vert ^2 \right] \\&\quad \le 2 \sum _{t=1}^T \left[ \Vert w^{t} - u^* \Vert _2^2 \left( \frac{1}{2\eta _t} - \frac{1}{2\eta _{t-1}} - \frac{\gamma }{2}\right) + \frac{\eta _t}{2} L^2 \right] . \end{aligned}$$

Defining $\eta _t := \frac{1}{\gamma t}$ for all $t=1,\ldots ,T$, we obtain

$$\begin{aligned} \mathsf {Regret}^T_{\text {ODCA-SG}k} \le \frac{L^2\left( 1+\log (T)\right) }{\gamma }. \end{aligned}$$

The proof of Theorem 1 is established. $\square$

Appendix 4: Proof of Proposition 1

Proof

(i) From the condition (7) and Theorem 1, we have, for any $w \in \mathcal {S}$,

$$|\mathcal {M}| \le \sum \limits _{t \in \mathcal {M}} f_t(w^t) \le \sum \limits _{t \in \mathcal {M}} f_t(w) + 3L^2\sqrt{|\mathcal {M}|}\sqrt{3K^2 -4K+2}.$$

Here, $|\mathcal {M}|$ is the number of the steps in $\mathcal {M}$.

It implies that $|\mathcal {M}| \le \overline{a} + \overline{c}\sqrt{|\mathcal {M}|}$. Thus, the proof of (i) is complete.

(ii) From the definition of $\gamma _{\mathrm {SG}}$, we derive that for any $w \in \mathcal {S}$,

$$\begin{aligned} |\mathcal {M}| \le \sum _{t \in \mathcal {M}} f_t(w^t) \le \sum _{t \in \mathcal {M}} f_t(w) + \frac{L^2\left( 1+\log (|\mathcal {M}|)\right) }{\gamma _{\mathrm {SG}}}. \end{aligned}$$

(33)

It implies

$$\begin{aligned} |\mathcal {M}| \le \overline{a} + \overline{b}\left( 1+\log (|\mathcal {M}|)\right) . \end{aligned}$$

Considering the strictly convex function $r : (0,+\infty ) \rightarrow \mathbb {R}$,

$$\begin{aligned}r(x) = x - \overline{a} - \overline{b}\left( 1+\log (x)\right) .\end{aligned}$$

Since $\lim \nolimits _{x \rightarrow 0^+}r(x) = \lim \nolimits _{x \rightarrow +\infty }r(x) = +\infty$ and $r(\overline{b}) \le 0$, equation $r(x) = 0$ has two roots $\overline{x}_1$, $\overline{x}_2$ such that $0 < \overline{x}_2 \le \overline{b} \le \overline{x}_1$. The proof of (ii) is established. $\square$

Appendix 5: Euclidean projection onto the probability simplex

Appendix 6: Description of the experts

We give here a description of the five experts used in the numerical experiments. They are well-known online classification algorithms: perceptron [50, 57, 62], relaxed online maximum margin algorithm [47], approximate maximal margin classification algorithm [23], passive-aggressive learning algorithm [14], classic online gradient descent algorithm [69].

Note that, in this paper, we consider the outcome space $\mathcal {Y} = \{0,1\}$ and the prediction label $\tilde{p}_{i,t} = \mathbb {1}_{\langle u_i, x \rangle \ge 0}(x_t) \in \{0,1\}$, where $u_i$ is the linear classifier of ith expert. Therefore, in the description below, the $\{0,1\}$ label is used instead to $\{-1,1\}$ which are often used in linear classification algorithms.

First, the perceptron algorithm is known as the earliest, simplest approach for online binary linear classification [57].

Second, the relaxed online maximum margin algorithm [47] is an incremental algorithm for classification using a linear threshold function. It can be seen as a relaxed version of the algorithm that searches for the separating hyperplane which maximizes the minimum distance from previous instances classified correctly.

Third, the approximate maximal margin classification algorithm [23] consists in approximating the maximal margin hyperplane with respect to $\ell _p$-norm ($p \ge 2$) for a set of linearly separable data. The proposed algorithm in [23] is called Approximate Large Margin Algorithm.

Fourth, the passive-aggressive learning algorithm [14] computes the classifier based on analytical solution to simple constrained optimization problem which minimizes the distance from the current classifier $u^t$ to the half-space of vectors which are of the zero hinge-loss on the current sample.

Finally, the classic online gradient descent algorithm [69] uses the gradient descent method for minimizing the hinge-loss function.