New results on subgradient methods for strongly convex optimization problems with a unified analysis

Abstract

We develop subgradient- and gradient-based methods for minimizing strongly convex functions under a notion which generalizes the standard Euclidean strong convexity. We propose a unifying framework for subgradient methods which yields two kinds of methods, namely, the proximal gradient method (PGM) and the conditional gradient method (CGM), unifying several existing methods. The unifying framework provides tools to analyze the convergence of PGMs and CGMs for non-smooth, (weakly) smooth, and further for structured problems such as the inexact oracle models. The proposed subgradient methods yield optimal PGMs for several classes of problems and yield optimal and nearly optimal CGMs for smooth and weakly smooth problems, respectively.

Appendix

In order to complete the proof of Theorem 5.3, we need to obtain upper bounds for $1/S_k$ and $\sum _{i=0}^kS_i/S_k$ for the sequence $\{S_k\}_{k \ge 0}$ defined by (46). Since $\lambda _{k+1}=S_{k+1}-S_k$, writing $r:=\frac{\sigma _f\sigma _d}{L-\bar{\sigma }_f\sigma _d}\ge 0$, the sequence $\{S_k\}_{k\ge 0}$ in (46) is determined by the recurrence

$$\begin{aligned} S_0=1,\quad (S_{k+1}-S_k)^2=S_{k+1}(1+rS_k),\quad k \ge 0 \end{aligned}$$

(52)

where the root of the equation in $S_{k+1}$ takes the largest one, namely,

$$\begin{aligned} S_{k+1}=\frac{1+(2+r)S_k+\sqrt{(1+(2+r)S_k)^2-4S_k^2}}{2}. \end{aligned}$$

(53)

The essentials of lemmas below are the same as [11, Lemma 4–7] excepting the replacement of $\mu /L$ in the article by an arbitrary $r\ge 0$.

Lemma 7.1

For any sequence $\{S_k\}_{k \ge 0}$ defined by (52) for $r \ge 0$, we have

$$\begin{aligned} \frac{1}{S_k} \le \min \left\{ \frac{4}{(k+1)(k+4)}, \left( \frac{2}{2+r+\sqrt{r^2+4r}}\right) ^{k}\right\} ,\quad \forall k \ge 0. \end{aligned}$$

Proof

Since $S_{k+1}\ge S_k$, we have

$$\begin{aligned} \sqrt{S_{k+1}}-\sqrt{S_{k}} = \frac{S_{k+1}-S_{k}}{\sqrt{S_{k+1}}+\sqrt{S_{k}}} \ge \frac{S_{k+1}-S_{k}}{2\sqrt{S_{k+1}}} \mathop {=}\limits ^{(52)} \frac{1}{2}\sqrt{1+rS_k}\ge \frac{1}{2} \end{aligned}$$

(54)

which shows $\sqrt{S_k}\ge \frac{k}{2}+\sqrt{S_0}=\frac{k+2}{2}$ for all $k \ge 0$. Then, we have

$$\begin{aligned} S_{k}-S_0= & {} \sum _{i=0}^{k-1}(S_{i+1}-S_i)\mathop {=}\limits ^{(52)}\sum _{i=0}^{k-1}\sqrt{S_{i+1}(1+rS_i)}\ge \sum _{i=0}^{k-1}\sqrt{S_{i+1}}\\\ge & {} \sum _{i=0}^{k-1}\frac{i+3}{2}=\frac{k(k+5)}{4} \end{aligned}$$

which gives $S_k \ge S_0+\frac{k(k+5)}{4}=\frac{(k+1)(k+4)}{4}$. On the other hand, using (53) yields that

$$\begin{aligned} \frac{S_{k+1}}{S_k}= & {} \frac{\frac{1}{S_k}+2+r+\sqrt{\left( \frac{1}{S_k}+(2+r)\right) ^2-4}}{2} \ge \frac{2+r+\sqrt{(2+r)^2-4}}{2} \nonumber \\= & {} \frac{2+r+\sqrt{r^2+4r}}{2} \end{aligned}$$

(55)

for all $k \ge 0$. Hence, we have $S_k\ge S_0\left( \frac{2+r+\sqrt{r^2+4r}}{2}\right) ^k=\left( \frac{2+r+\sqrt{r^2+4r}}{2}\right) ^k$. $\square$

Remark 7.2

The linear convergence factor $\frac{2}{2+r+\sqrt{r^2+4r}}$ in the above lemma satisfies

$$\begin{aligned} 1-\sqrt{\frac{r}{r+1}} \le \frac{2}{2+r+\sqrt{r^2+4r}} \le \left( 1+\frac{1}{2}\sqrt{r}\right) ^{-2}. \end{aligned}$$

In fact, since

$$\begin{aligned} \left( 1-\sqrt{\frac{r}{r+1}}\right) ^{-1}= & {} \frac{\sqrt{r+1}}{\sqrt{r+1}-\sqrt{r}}=\sqrt{r+1}(\sqrt{r+1}+\sqrt{r})\\= & {} \frac{2+2r+\sqrt{4r^2+4r}}{2}, \end{aligned}$$

we obtain

$$\begin{aligned} \left( 1+\frac{1}{2}\sqrt{r}\right) ^{2}= & {} \frac{2+r/2+\sqrt{4r}}{2} \le \frac{2+r+\sqrt{r^2+4r}}{2} \\\le & {} \frac{2+2r+\sqrt{4r^2+4r}}{2}=\left( 1-\sqrt{\frac{r}{r+1}}\right) ^{-1}. \end{aligned}$$

Note that if $\bar{\sigma }_f=\sigma _f$ and $r=\frac{\sigma _f\sigma _d}{L-\bar{\sigma }_f\sigma _d}$, then $\sqrt{\frac{r}{r+1}}=\sqrt{\frac{\sigma _f\sigma _d}{L}}$.

Lemma 7.3

The sequence $\{S_k\}_{k \ge 0}$ defined by (52) for $r>0$ satisfies

$$\begin{aligned} \frac{\sum _{i=0}^kS_i}{S_k} \le \frac{1+\sqrt{1+4r^{-1}}}{2}\le 1+\sqrt{\frac{1}{r}},\quad \forall k \ge 0. \end{aligned}$$

Proof

Notice that $\gamma := \frac{1+\sqrt{1+4r^{-1}}}{2}$ satisfies

$$\begin{aligned} \left( 1-\frac{1}{\gamma }\right) ^{-1}=\frac{\gamma }{\gamma -1}=\frac{\sqrt{1+4r^{-1}}+1}{\sqrt{1+4r^{-1}}-1}=\frac{(\sqrt{1+4r^{-1}}+1)^2}{4r^{-1}}=\frac{2+r+\sqrt{r^2+4r}}{2}. \end{aligned}$$

Therefore, we obtain $\frac{S_k}{S_{k+1}}\le 1-\frac{1}{\gamma }$ by (55). Now the result follows by induction: if $\sum _{i=0}^kS_i/S_k\le \gamma$ holds for some $k \ge 0$, we have

$$\begin{aligned} \frac{\sum _{i=0}^{k+1}S_i}{S_{k+1}} = 1+\frac{S_k}{S_{k+1}}\frac{\sum _{i=0}^kS_i}{S_k} \le 1+\frac{\gamma -1}{\gamma }\cdot \gamma =\gamma . \end{aligned}$$

This proves the first inequality; the second can be verified from $\sqrt{1+4r^{-1}}\le 1+2\sqrt{r^{-1}}$. $\square$

Note that the result of Lemma 7.3 is the same as [11, Lemma 5] because $1+\frac{2\sqrt{r^{-1}}}{\sqrt{r}+\sqrt{r+4}}=\frac{1+\sqrt{1+4r^{-1}}}{2}$.

Lemma 7.4

Let $\{S_k\}_{k \ge 0}$ be defined as Lemma 7.3 and $\{T_k\}_{k \ge 0}$ be defined by (52) with $r:=0$, namely $T_0:=1$ and $T_{k+1}:=\frac{1+2T_k+\sqrt{1+4T_k}}{2}$ for $k \ge 0$. Then, we have

$$\begin{aligned} \frac{\sum _{i=0}^kS_i}{S_k} \le \frac{\sum _{i=0}^kT_i}{T_k},\quad \forall k \ge 0. \end{aligned}$$

Proof

Due to the identity

$$\begin{aligned} \frac{\sum _{i=0}^kS_i}{S_k} = 1+\sum _{i=0}^{k-1}\frac{S_i}{S_{k}} = 1+\sum _{i=0}^{k-1}\prod _{j=i}^{k-1}\frac{S_j}{S_{j+1}}, \quad k \ge 0, \end{aligned}$$

it is enough to show that $\frac{S_k}{S_{k+1}} \le \frac{T_k}{T_{k+1}}$ for every $k \ge 0$. Notice that we have

$$\begin{aligned} \frac{S_{k+1}}{S_k} = \frac{\frac{1+rS_k}{S_k}+2+\sqrt{\left( \frac{1+rS_k}{S_k}+2\right) ^2-4}}{2},\quad \frac{T_{k+1}}{T_k} = \frac{\frac{1}{T_k}+2+\sqrt{\left( \frac{1}{T_k}+2\right) ^2-4}}{2}, \end{aligned}$$

(56)

which suggests us to prove $\frac{1+rS_k}{S_k} \ge \frac{1}{T_k}$ for $k \ge 0$. It is true for $k=0$ by $S_0=T_0$. If it holds for $k \ge 0$, then, writing $\alpha :=\frac{1+rS_k}{S_k} \ge \beta := \frac{1}{T_k}$, we obtain

$$\begin{aligned} \frac{1+rS_{k+1}}{S_{k+1}}\ge & {} \frac{1+rS_k}{S_{k+1}} = \frac{S_k}{S_{k+1}}\alpha \mathop {=}\limits ^{(56)} \frac{2\alpha }{\alpha +2+\sqrt{(\alpha +2)^2-4}}\\\ge & {} \frac{2\beta }{\beta +2+\sqrt{(\beta +2)^2-4}} \mathop {=}\limits ^{(56)} \frac{T_k}{T_{k+1}}\beta =\frac{1}{T_{k+1}} \end{aligned}$$

since $S_{k+1} \ge S_k$ and $x \mapsto \frac{2x}{x+2+\sqrt{(x+2)^2-4}}=\frac{2}{1+2x^{-1}+\sqrt{1+4x^{-1}}}$ is non-decreasing on $(0,\infty )$. Hence, we claim $\frac{1+rS_k}{S_k} \ge \frac{1}{T_k}$ for all $k \ge 0$ and therefore the proof is completed. $\square$

Lemma 7.5

Let $\{T_k\}_{k \ge 0}$ be a sequence defined by (52) with $r:=0$, namely $T_0:=1$ and $T_{k+1}:=\frac{1+2T_k+\sqrt{1+4T_k}}{2}$ for $k \ge 0$. Then, we have

$$\begin{aligned} \frac{\sum _{i=0}^kT_i}{T_k} \le \frac{1}{3}k +\frac{1}{6}\log (k+2)+1,\quad \forall k \ge 0. \end{aligned}$$

Proof

The case $k =0$ is obvious. Assume that the assertion is true for some $k \ge 0$. Putting $U_k:=\frac{1}{3}k +\frac{1}{6}\log (k+2)+1$, we have

$$\begin{aligned} \frac{\sum _{i=0}^{k+1}T_i}{T_{k+1}} = 1+\frac{T_k}{T_{k+1}}\frac{\sum _{i=0}^kT_i}{T_k} \le 1+\frac{T_k}{T_{k+1}}U_k. \end{aligned}$$

Hence, it remains to show $1+\frac{T_k}{T_{k+1}}U_k \le U_{k+1}$ for $k \ge 0$. For that, we analyze the sequence $t_0:=1,~t_{k+1}:=T_{k+1}-T_k$ for $k \ge 0$ (namely, $T_k=\sum _{i=0}^k t_i$). The recurrence relation of $T_k$ implies $t_{k}^2=(T_{k}-T_{k-1})^2=T_{k}$ and

$$\begin{aligned} t_{k+1}=T_{k+1}-T_k\mathop {=}\limits ^{(53)} \frac{1+\sqrt{1+4T_k}}{2} = \frac{1+\sqrt{1+4t_k^2}}{2},\quad \forall k \ge 0. \end{aligned}$$

Analyzing the difference $t_{k+1}-t_k$ shows for $k \ge 0$ that

$$\begin{aligned} t_{k+1}-t_k= & {} \frac{1+\sqrt{1+4t_k^2}-2t_k}{2}=\frac{1}{2}+\frac{1}{2\left( \sqrt{1+4t_k^2} +2t_k\right) } \\\le & {} \frac{1}{2}+\frac{1}{2\left( \sqrt{4t_k^2}+2t_k\right) } = \frac{1}{2}+\frac{1}{8t_k}. \end{aligned}$$

Since Lemma 7.1 yields $t_k=\sqrt{T_k}\ge \sqrt{(k+1)(k+4)/4}\ge (k+2)/2$ for $k \ge 0$, we obtain

$$\begin{aligned} t_{k+1}\le & {} t_0+\frac{k+1}{2}+\frac{1}{8}\sum _{i=0}^{k}\frac{1}{t_i}\le \frac{k}{2}+\frac{3}{2}+\frac{1}{8}\sum _{i=0}^{k}\frac{2}{i+2}\\\le & {} \frac{k}{2}+\frac{3}{2}+\frac{1}{4}\log (k+2)=\frac{3}{2}U_{k} \end{aligned}$$

for all $k \ge 0$. Finally, this upper bound of $t_k$ concludes that

$$\begin{aligned} \frac{U_k}{1+U_k-U_{k+1}}= & {} \frac{3U_k}{2+\frac{1}{2}\log \frac{k+2}{k+3}}\ge \frac{3}{2}U_k \\\ge & {} t_{k+1}=\frac{t_{k+1}^2}{t_{k+1}}=\frac{T_{k+1}}{T_{k+1}-T_k}. \end{aligned}$$

Taking the inverse and multiplying by $U_k$ for both sides yield $1+\frac{T_k}{T_{k+1}}U_k \le U_{k+1}$.   $\square$