Restricted strong convexity and its applications to convergence analysis of gradient-type methods in convex optimization

Abstract

A great deal of interest of solving large-scale convex optimization problems has, recently, turned to gradient method and its variants. To ensure rates of linear convergence, current theory regularly assumes that the objective functions are strongly convex. This paper goes beyond the traditional wisdom by studying a strictly weaker concept than the strong convexity, named restricted strongly convexity which was recently proposed and can be satisfied by a much broader class of functions. Utilizing the restricted strong convexity, we derive rates of linear convergence for (in)exact gradient-type methods. Besides, we obtain two by-products: (1) we rederive rates of linear convergence of inexact gradient method for a class of structured smooth convex optimizations; (2) we improve the rate of linear convergence for the linearized Bregman algorithm.

Appendix

We select the parameter $\theta$ and step size $h$ in (15e) to minimize the upper bound. Let $r=\frac{\nu }{L}, h>0$. As we need to deal with the second term in (15e), two cases are studied below depending on the sign of $h^2-\frac{2\theta h}{L}$:

Case A: $h^2-\frac{2\theta h}{L}\le 0$, i.e., $h\in (0, \frac{2\theta }{L}], \theta \in [0,1]$. Applying the Cauchy-Schwartz inequality to RSI, we get

$$\begin{aligned} \Vert \nabla f(x^{(k)})-\nabla f(x_{\mathrm {prj}}^{(k)})\Vert ^2\ge \nu ^2 \Vert x^{(k)}-x_{\mathrm {prj}}^{(k)}\Vert ^2. \end{aligned}$$

(42)

From $h^2-\frac{2\theta h}{L}\le 0$ and (15e), we derive that

$$\begin{aligned} \Vert x^{(k+1)}-x_{\mathrm {prj}}^{(k+1)}\Vert ^2&\le (1-2(1-\theta )\nu h)\Vert x^{(k)}- x_{\mathrm {prj}}^{(k)}\Vert ^2+\nu ^2\left( h^2-\frac{2\theta h}{L}\right) \Vert x^{(k)}-x_{\mathrm {prj}}^{(k)}\Vert ^2,\quad \quad \end{aligned}$$

(43a)

$$\begin{aligned}&= \left( \nu ^2h^2 -2\left( (1-\theta )\nu +\frac{\theta \nu ^2}{L}\right) h+1\right) \Vert x^{(k)}-x_{\mathrm {prj}}^{(k)}\Vert ^2,\end{aligned}$$

(43b)

$$\begin{aligned}&\triangleq f_1(\theta , h)\Vert x^{(k)}-x_{\mathrm {prj}}^{(k)}\Vert ^2. \end{aligned}$$

(43c)

Let $h_0=\frac{\theta }{L}+\frac{(1-\theta )}{\nu }$, which is the minimum point of the quadratic function $f_1(\theta , h)$ over variable $h$ for each fixed $\theta$. To determine whether such $h_0$ is included in the interval $(0, \frac{2\theta }{L}]$, we consider $h_0=\frac{\theta }{L}+\frac{(1-\theta )}{\nu }=\frac{2\theta }{L}$ and get $\theta = \frac{1}{1+r}$. Now, we split the interval $[0,1]$ into $[\frac{1}{1+r}, 1]$ and $[0, \frac{1}{1+r})$. If $\theta \in [\frac{1}{1+r}, 1]$, we have $\frac{2\theta }{L}\ge h_0$ which means the point $h_0\in (0, \frac{2\theta }{L}]$. Thus,

$$\begin{aligned} \min _{h\le \frac{2\theta }{L}, \frac{1}{1+r}\le \theta \le 1} f_1(\theta ,h)=\min _{\frac{1}{1+r}\le \theta \le 1} f_1(\theta , h_0)=\min _{\frac{1}{1+r}\le \theta \le 1} 1-(1-(1+r)\theta )^2= 1-r^2, \end{aligned}$$

where the minimum value $1-r^2$ is obtained at $\theta =1$ and $h=h_0=\frac{1}{L}$. If $\theta \in [0, \frac{1}{1+r})$, we have $\frac{2\theta }{L}< h_0$ which means the point $h_0\notin (0, \frac{2\theta }{L}]$. By monotone decreasing of $f_1(\theta , h)$ on the interval $h\le \frac{2\theta }{L}$ for each fixed $\theta$, we have

$$\begin{aligned} \min _{h\le \frac{2\theta }{L},0\le \theta < \frac{1}{1+r}} f_1(\theta , h)=\min _{ 0\le \theta < \frac{1}{1+r}} f_1\left( \theta , \frac{2\theta }{L}\right) =\min _{ 0\le \theta < \frac{1}{1+r}} 1-4\theta (1-\theta )r= 1-r \end{aligned}$$

where the minimum value $1-r$ is obtained at $\theta =\frac{1}{2}$ and $h=\frac{2\theta }{L}=\frac{1}{L}$; note that $\frac{1}{2}\in [0, \frac{1}{1+r})$ since $r<1$. Therefore, on the intervals $h\in (0, \frac{2\theta }{L}]$ and $\theta \in [0,1]$, the minimum value $1-r$ of $f_1(\theta , h)$ is obtained at $(\theta , h)=(\frac{1}{2},\frac{1}{L})$.

Case B: $h^2-\frac{2\theta h}{L}\ge 0$, i.e., $h\in [\frac{2\theta }{L}, +\infty ), \theta \in [0,1]$. Applying the Cauchy-Schwartz inequality to (11) in Lemma 2, we get

$$\begin{aligned} \Vert \nabla f(x^{(k)})-\nabla f(x_{\mathrm {prj}}^{(k)})\Vert ^2\le L^2 \Vert x^{(k)}-x_{\mathrm {prj}}^{(k)}\Vert ^2. \end{aligned}$$

(44)

From $h^2-\frac{2\theta h}{L}\ge 0$ and (15e), we derive that

$$\begin{aligned} \Vert x^{(k+1)}-x_{\mathrm {prj}}^{(k+1)}\Vert ^2&\le (1-2(1-\theta )\nu h)\Vert x^{(k)}- x_{\mathrm {prj}}^{(k)}\Vert ^2+L^2\left( h^2-\frac{2\theta h}{L}\right) \Vert x^{(k)}-x_{\mathrm {prj}}^{(k)}\Vert ^2,\quad \quad \quad \end{aligned}$$

(45a)

$$\begin{aligned}&= (L^2h^2 -2(\theta L+(1-\theta )\nu ) h+1)\Vert x^{(k)}-x_{\mathrm {prj}}^{(k)}\Vert ^2,\end{aligned}$$

(45b)

$$\begin{aligned}&\triangleq f_2(\theta , h)\Vert x^{(k)}-x_{\mathrm {prj}}^{(k)}\Vert ^2. \end{aligned}$$

(45c)

Let $h_1=\frac{\theta L+(1-\theta )\nu }{L^2}$, which is the minimum point of the quadratic function $f_2(\theta , h)$ over variable $h$ for each fixed $\theta$. Similarly, we split the interval $[0,1]$ into $(\frac{r}{1+r}, 1]$ and $[0, \frac{r}{1+r}]$. If $\theta \in (\frac{r}{1+r}, 1]$, we have $\frac{2\theta }{L}> h_1$ which means $h_1\notin [\frac{2\theta }{L}, +\infty )$. By monotone increasing of $f_2(\theta , h)$ on the interval $h\ge \frac{2\theta }{L}$ for each fixed $\theta$, we have

$$\begin{aligned} \min _{h\ge \frac{2\theta }{L}, \frac{r}{1+r} <\theta \le 1} f_2(\theta , h)=\min _{\frac{r}{1+r}<\theta \le 1}f_2\left( \theta , \frac{2\theta }{L}\right) =\min _{\frac{r}{1+r} <\theta \le 1} 1-4\theta (1-\theta )r = 1-r, \end{aligned}$$

where the minimum value $1-r$ is obtained at $\theta =1/2$ and $h=\frac{2\theta }{L}=\frac{1}{L}$; note that $\frac{1}{2}\in (\frac{r}{1+r}, 1]$ since $r<1$. If $\theta \in [0, \frac{r}{1+r}]$, we have $\frac{2\theta }{L}\le h_1$ which means $h_1\in [\frac{2\theta }{L}, +\infty )$. Thus,

$$\begin{aligned} \min _{h\ge \frac{2\theta }{L},0\le \theta \le \frac{r}{1+r}} f_2(\theta , h)&=\min _{ 0\le \theta \le \frac{r}{1+r}} f_2(\theta , h_1)\\&=\min _{ 0\le \theta \le \frac{r}{1+r}} 1-\left( \frac{\theta L+(1-\theta )\nu }{L}\right) ^2=1-\left( \frac{2\nu }{L+\nu }\right) ^2, \end{aligned}$$

where the minimum value is obtained at $\theta =\frac{r}{1+r}$ and $h=h_1$. After simple calculations, it holds $r=\frac{\nu }{L}>\left( \frac{2\nu }{L+\nu }\right) ^2$ and hence $1-r< 1-\left( \frac{2\nu }{L+\nu }\right) ^2$. Therefore, on the intervals $h\in [\frac{2\theta }{L}, +\infty )$ and $\theta \in [0,1]$, the minimum value $1-r$ of $f_2(\theta , h)$ is obtained at $(\theta , h)=(\frac{1}{2},\frac{1}{L})$ as well.