Stochastic Online Kernel Selection with Instantaneous Loss in Random Feature Space

Abstract

Online kernel selection is critical to online kernel learning. However, the time complexity of existing online kernel selection algorithms of each round is linear with respect to the number of examples already arrived. This is not efficient for online learning. To address this issue, we propose a novel stochastic online kernel selection algorithm via the random feature mapping and using the instantaneous loss. This algorithm has only constant time complexity at each round and theoretical guarantee. Formally, the algorithm first maps the arriving example into the random feature space. Then the algorithm updates the kernel parameter and the weights of the classifier simultaneously using SGD (stochastic gradient descent) to minimize the instantaneous loss. We also prove that the algorithm enjoys a sub-linear regret bound. Experimental results on benchmark datasets demonstrate that the proposed algorithm is effective and efficient.

Appendix

In this appendix, we provide the proof details of Theorem 1.

Proof

Let $f_{*}(\varvec{x})= (\varvec{w}^{*})^\top \phi (\varvec{x},\gamma ^{*})$ be the optimal classifier in the random feature space that minimizes the expected loss. The desired inequality can be rewritten as

$$\begin{aligned} \begin{aligned}&\sum _{t=1}^{T}(\ell (\varvec{w}_{t}^{\top }\phi (\varvec{x}_{t},\gamma _{t}),y_t) - \ell _{t}(g^{*}(\varvec{x}_t), y_t))\\&\quad = \sum _{t=1}^{T}(\ell _{t}(\varvec{w}_{t}^{\top }\phi (\varvec{x}_{t},\gamma _{t}),y_t) - \ell _{t}((\varvec{w}^{*})^{\top }\phi (\varvec{x}_{t},\gamma _{t}),y_t) \\&\qquad +\sum _{t=1}^{T}(\ell _{t}((\varvec{w}^{*})^{\top }\phi (\varvec{x}_{t},\gamma _{t}),y_t) - \ell _{t}((\varvec{w}^{*})^{\top }\phi (\varvec{x}_{t},\gamma ^*),y_t))\\&\qquad +\sum _{t=1}^{T}(\ell _{t}((\varvec{w}^{*})^{\top }\phi (\varvec{x}_{t},\gamma ^*) - \ell _{t}(g^{*}(\varvec{x}_t), y_t))\\&\quad = A + B + C. \end{aligned} \end{aligned}$$

First of all, consider $\ell _t$ as a function of $\gamma$. From the convexity of the loss function, we obtain

$$\begin{aligned} \begin{aligned} \ell _t(\gamma _{t})-\ell _t(\gamma ^{*})\le&\nabla \ell _{t}(\gamma _{t})(\gamma _{t}-\gamma ^{*})\\ =&\frac{(\gamma _{t}-\gamma ^{*})^{2}-(\gamma _{t+1}-\gamma ^{*})^{2}}{2\eta }+ \frac{\eta (\nabla \ell _{t}(\gamma _{t}))^{2}}{2}. \end{aligned} \end{aligned}$$

Summing the above over $t=1,\ldots T$ leads to

$$\begin{aligned} \begin{aligned} B \le&\frac{(\gamma _{1}-\gamma ^{*})^{2}-(\gamma _{T+1}-\gamma ^{*})^{2}}{2\eta } + \frac{\eta }{2}\sum _{t=1}^{T}(\nabla \ell _{t}(\gamma _{t}))^{2} \\ \le&\frac{(\gamma ^{*})^2}{2\eta } + \frac{\eta }{2}L_1^{2}T, \end{aligned} \end{aligned}$$

where $L_1=\max _{t\in [T]}\Vert \nabla \ell _{t}(\gamma _{t})\Vert ^{2}$. We adopt a similar procedure and it suffices to show that

$$\begin{aligned} A\le \frac{\Vert \varvec{w}^{*}\Vert ^2 }{2\eta } + \frac{\eta }{2}L_2^{2}T \;\mathrm {and}\;L_2=\max _{t\in [T]}\Vert \nabla \ell _{t}(\varvec{w}_{t})\Vert ^{2}. \end{aligned}$$

From the result of  [13], we get with probability at least

$$\begin{aligned} 1-2^8\left( \frac{\gamma _pR}{\epsilon '}\right) ^2\exp \left( \frac{-D\epsilon ^2}{4(d+2)}\right) , \end{aligned}$$

$$\begin{aligned} \Vert \varvec{w}^{*}\Vert ^2 \le (1+\epsilon ')\Vert g^{*}\Vert _1^2 \;\mathrm {and}\; C\le \epsilon 'LT\Vert g^*\Vert _1. \end{aligned}$$

Recalling the definition of M, we can derive $\varvec{M}[j]\sim \mathcal {N}(0, \Vert \varvec{x}\Vert ^{2}).$ This easily leads to the upper bound of $|\nabla \ell (\gamma )|$, i.e.

$$\begin{aligned} |\nabla \ell (\gamma )|\le \sqrt{\frac{2}{D}}\left( \sum _{j=1}^{D/2}|\varvec{w}[j]\varvec{M}[j]| + \sum _{j=D/2+1}^{D}|\varvec{w}[j]\varvec{M}[j-D/2]|\right) . \end{aligned}$$

By the property of Gaussian variable, we therefore obtain

$$\begin{aligned} \mathbb {E}\left[ \left| \varvec{w}[j]\varvec{M}[j]\right| \right] \le \Vert \varvec{x}\Vert ^{2}(\varvec{w}[j])^{2}\sqrt{\frac{2}{\pi }} \;\mathrm {and}\; \mathbb {E}\left[ |\nabla \ell (\gamma )|\right] \le \frac{2\Vert \varvec{w}\Vert ^2\Vert \varvec{x}\Vert ^{2}}{\sqrt{\pi D}}. \end{aligned}$$

By Chernoff inequality, we have, with probability at least

$$\begin{aligned} 1-\exp (-\frac{2\epsilon ^{2}C}{3}\sqrt{\frac{D}{\pi }}), \end{aligned}$$

$$\begin{aligned} |\nabla \ell (\gamma _t)|\le (1+\epsilon )\frac{2C}{\sqrt{\pi D}}, \;\mathrm {for}\; \forall t\in \{1,\ldots ,T\}, \end{aligned}$$

where $C=\max _{t\in \{1,\ldots ,T\}}\Vert \varvec{w}_t\Vert ^2\Vert \varvec{x}_t\Vert ^{2}.$ From the basic relationship between the sine and the cosine, it follows that $\Vert \nabla \ell (\varvec{w})\Vert _2^2=1.$

We now conclude our proof.    $\square$