Accelerated proximal stochastic dual coordinate ascent for regularized loss minimization

Abstract

We introduce a proximal version of the stochastic dual coordinate ascent method and show how to accelerate the method using an inner-outer iteration procedure. We analyze the runtime of the framework and obtain rates that improve state-of-the-art results for various key machine learning optimization problems including SVM, logistic regression, ridge regression, Lasso, and multiclass SVM. Experiments validate our theoretical findings.

Appendix: Proofs of iteration bounds for Prox-SDCA

The proof technique follows that of Shalev-Shwartz and Zhang [26], but with the required generality for handling general strongly convex regularizers and smoothness/Lipschitzness with respect to general norms. The proof of Theorem 1 is almost identical to the proof of Theorem 1 in Shalev-Shwartz and Zhang [25], except that we do not upper bound $\mathbb {E}[D(\alpha ^*)-D(\alpha ^{(0)})]$ by 1.

Proof of Theorem 2

Denote $\epsilon _D^{(t)} := D(\alpha ^*)-D(\alpha ^{(t)})$. Define $t_0 = \lceil \frac{n}{s} \log (2\epsilon _D^{(0)}/\epsilon _D) \rceil$. The proof of Theorem 1 implies that for every $t$, $\mathbb {E}[\epsilon _D^{(t)}] \le \epsilon _D^{(0)}\,e^{-\frac{st}{n}}$. By Markov’s inequality, with probability of at least $1/2$ we have $\epsilon _D^{(t)} \le 2\epsilon _D^{(0)}\,e^{-\frac{st}{n}}$. Applying it for $t=t_0$ we get that $\epsilon _D^{(t_0)} \le \epsilon _D$ with probability of at least $1/2$. Now, lets apply the same argument again, this time with the initial dual sub-optimality being $\epsilon _D^{(t_0)}$. Since the dual is monotonically non-increasing, we have that $\epsilon _D^{(t_0)} \le \epsilon _D^{(0)}$. Therefore, the same argument tells us that with probability of at least $1/2$ we would have that $\epsilon _D^{(2t_0)} \le \epsilon _D$. Repeating this $\lceil \log _2(1/\delta ) \rceil$ times, we obtain that with probability of at least $1-\delta$, for some $k$ we have that $\epsilon _D^{(kt_0)} \le \epsilon _D$. Since the dual is monotonically non-decreasing, the claim about the dual sub-optimality follows.

Next, for the duality gap, using the inequality

$$\begin{aligned} \mathbb {E}_t[D(\alpha ^{(t)})-D(\alpha ^{(t-1)})] \ge \frac{s}{n}\, (P(w^{(t-1)})-D(\alpha ^{(t-1)})). \end{aligned}$$

from Lemma 1 of Shalev-Shwartz and Zhang [25].

We have that for every $t$ such that $\epsilon _D^{(t-1)} \le \epsilon _D$ we have

$$\begin{aligned} P(w^{(t-1)})-D(\alpha ^{(t-1)}) \le \frac{n}{s} \, \mathbb {E}[D(\alpha ^{(t)})-D(\alpha ^{(t-1)})] \le \frac{n}{s} \epsilon _D. \end{aligned}$$

This proves the second claim of Theorem 2.

For the last claim, suppose that at round $T_0$ we have $\epsilon _D^{(T_0)} \le \epsilon _D$. Let $T = T_0 + n/s$. It follows that if we choose $t$ uniformly at random from $\{T_0,\ldots ,T-1\}$, then $\mathbb {E}[ P(w^{(t)})-D(\alpha ^{(t)})] \le \epsilon _D$. By Markov’s inequality, with probability of at least $1/2$ we have $P(w^{(t)})-D(\alpha ^{(t)}) \le 2\epsilon _D$. Therefore, if we choose $\log _2(2/\delta )$ such random $t$, with probability $\ge 1-\delta /2$, at least one of them will have $P(w^{(t)})-D(\alpha ^{(t)}) \le 2\epsilon _D$. Combining with the first claim of the theorem, choosing $\epsilon _D = \epsilon _P/2$, and applying the union bound, we conclude the proof of the last claim of Theorem 2. $\square$