Improved Update Rule and Sampling of Stochastic Gradient Descent with Extreme Early Stopping for Support Vector Machines

Abstract

We propose three techniques for improving accuracy and speed of margin stochastic gradient descent support vector machines (MSGDSVM). The first technique is to use sampling with full replacement. The second technique is to use the new update rule derived from the squared hinge loss function. The third technique is to limit the number of values for tuning of the margin hyperparameter M. We also provide theoretical analysis of a novel optimization problem for the proposed update rule. The first two techniques improve accuracy of MSGDSVM and the last one speed of tuning. Experiments show that the proposed method achieves superior accuracy compared to MSGDSVM for binary and multiclass classification, with similar generalization performance to sequential minimal optimization (SMO) and is faster than MSGDSVM.

M. Orchel—Independent Researcher.

9 Appendix

We provide a derivation of the update rule (6).

Proof

The proof is based on the primal problem OP 4. The technique of a proof is similar as presented in [10]. First, we compute the gradient of the objective function in (15) and we get

$$\begin{aligned} \begin{gathered} \frac{\partial W}{\partial \boldsymbol{w}} = \boldsymbol{w} + \sum _{i=1}^n 2\left( M - y_if\left( \boldsymbol{x_i}\right) \ge 0\ ? \ 1 - y_i\boldsymbol{w}\cdot \varphi \left( \boldsymbol{x_i}\right) : 0\right) \\ \cdot \left( M - y_i\boldsymbol{w}\cdot \varphi \left( \boldsymbol{x_i}\right) \ge 0 \ ? \ -y_i\varphi \left( \boldsymbol{x_i}\right) \ :\ 0\right) . \end{gathered} \end{aligned}$$

Because we have the same condition in the last two factors, we can write

$$\begin{aligned} \frac{\partial W}{\partial \boldsymbol{w}} = \boldsymbol{w} + 2\sum _{i=1}^n M - y_i\boldsymbol{w}\cdot \varphi \left( \boldsymbol{x_i}\right) \ge 0 \ ? \ -\left( 1 - y_i\boldsymbol{w}\cdot \varphi \left( \boldsymbol{x_i}\right) \right) y_i\varphi \left( \boldsymbol{x_i}\right) \ :\ 0 . \end{aligned}$$

After substitution (18), we get

$$\begin{aligned} \begin{gathered} \sum _{i=1}^n y_i\alpha _i\varphi \left( \boldsymbol{x_i}\right) + 2\sum _{i=1}^n M - y_i\sum _{j=1}^n y_j\alpha _j\varphi \left( \boldsymbol{x_j}\right) \cdot \varphi \left( \boldsymbol{x_i}\right) \ge 0 \ ? \\ \ -\Bigl (1 - y_i\sum _{j=1}^n y_j\alpha _j\varphi \left( \boldsymbol{x_j}\right) \cdot \varphi \left( \boldsymbol{x_i}\right) \Bigr )y_i\varphi \left( \boldsymbol{x_i}\right) \ :\ 0 . \end{gathered} \end{aligned}$$

For the stochastic update, we approximate the above formula (which should be equal to 0 for the optimal solution), so we have

$$\begin{aligned} \begin{gathered} ny_k\alpha _k\varphi \left( \boldsymbol{x_k}\right) + 2n \Bigl (M - y_k\sum _{j=1}^n y_j\alpha _j\varphi \left( \boldsymbol{x_j}\right) \cdot \varphi \left( \boldsymbol{x_k}\right) \ge 0\Bigr ) \ ? \\ \ -\Bigl (1 - y_k\sum _{j=1}^n y_j\alpha _j\varphi \left( \boldsymbol{x_j}\right) \cdot \varphi \left( \boldsymbol{x_k}\right) \Bigr )y_k\varphi \left( \boldsymbol{x_k}\right) \ :\ 0 = 0 . \end{gathered} \end{aligned}$$

We can generate an update term as for the ordinary iteration method by transforming the equation into a fixed point form for $\alpha _k$ by dividing by $ny_k\varphi \left( \boldsymbol{x_k}\right)$. We assume that $\varphi \left( \boldsymbol{x_k}\right) \ne 0$ for any coefficient. For the RBF kernel, it means that each component of $\boldsymbol{x}$ should be different from 0. We move all terms except the first one to the right, and we get

$$\begin{aligned} \begin{gathered} \alpha _k \leftarrow 2\Bigl (M - y_k\sum _{j=1}^n y_j\alpha _j\varphi \left( \boldsymbol{x_j}\right) \cdot \varphi \left( \boldsymbol{x_k}\right) \ge 0\Bigr ) \ ? \\ \ 1 - y_k\sum _{j=1}^n y_j\alpha _j\varphi \left( \boldsymbol{x_j}\right) \cdot \varphi \left( \boldsymbol{x_k}\right) \ :\ 0 . \end{gathered} \end{aligned}$$

We can skip multiplier 2, because it will not affect the final decision boundary. Assuming also initialization with zero and extreme early stopping (updating each parameter maximally one time), we get

$$\begin{aligned} \begin{gathered} \alpha _k \leftarrow \left( M - y_kf_{k-1}\left( \boldsymbol{x_k}\right) \ge 0\right) \ ? \ 1 - y_kf_{k-1}\left( \boldsymbol{x_k}\right) \ :\ 0 . \end{gathered} \end{aligned}$$

By incorporating the additional assumption that the weight is positive, we get

$$\begin{aligned} \begin{gathered} \alpha _k \leftarrow \left( M - y_kf_{k-1}\left( \boldsymbol{x_k}\right) \ge 0\right) \ ? \ \max \left\{ 0, 1 - y_kf_{k-1}\left( \boldsymbol{x_k}\right) \right\} \ :\ 0 . \end{gathered} \end{aligned}$$

By returning to the original representation with $\beta _j$ weights, we get (6).