Exact Worst-Case Convergence Rates of the Proximal Gradient Method for Composite Convex Minimization

Abstract

We study the worst-case convergence rates of the proximal gradient method for minimizing the sum of a smooth strongly convex function and a non-smooth convex function, whose proximal operator is available. We establish the exact worst-case convergence rates of the proximal gradient method in this setting for any step size and for different standard performance measures: objective function accuracy, distance to optimality and residual gradient norm. The proof methodology relies on recent developments in performance estimation of first-order methods, based on semidefinite programming. In the case of the proximal gradient method, this methodology allows obtaining exact and non-asymptotic worst-case guarantees that are conceptually very simple, although apparently new. On the way, we discuss how strong convexity can be replaced by weaker assumptions, while preserving the corresponding convergence rates. We also establish that the same fixed step size policy is optimal for all three performance measures. Finally, we extend recent results on the worst-case behavior of gradient descent with exact line search to the proximal case.

Appendices

Appendix A: Details of the Proof of Theorem 3.1

In this section, we provide some details on the verification of the proof of Theorem 3.1 (convergence in distance to optimality). The proofs of Theorem 3.2 (convergence in residual gradient norm) and Theorem 3.3 (convergence in function value) follow the same lines, although the results in function values are technically more involved (we advise the reader to use appropriate computer algebra software to preserve his sanity).

As the proofs for both regimes (small and large step sizes) follow the same lines, we only consider the case $0\le \gamma \le \frac{2}{L+\mu }$ here.

The goal is to prove that the inequality

$$\begin{aligned} \left( 1-\gamma \mu \right) ^2 {\left\| x_k-x_\star \right\| ^2}\ge & {} {\left\| x_{k+1}-x_\star \right\| ^2} +\gamma ^2 {\left\| g_\star +s_{k+1}\right\| ^2}\nonumber \\&+\frac{\gamma (2-\gamma (L+\mu ))}{L-\mu } {\left\| \mu {(x_k -x_\star )} - g_k+ g_\star \right\| ^2}, \end{aligned}$$

(7)

can be obtained by performing a weighted sum of the following inequalities:

$$\begin{aligned}&\begin{array}{ll} f_\star &{}\ge f_k+{\left\langle g_k, x_\star -x_k\right\rangle }+\frac{1}{2L}{\left\| g_k-g_\star \right\| ^2} \\ &{}\quad +\frac{\mu }{2(1-\mu /L)}{\left\| x_k-x_\star -\frac{{1}}{L}(g_k-g_\star )\right\| ^2} \end{array} \quad&:2\gamma \rho (\gamma ), \\&\begin{array}{ll} f_k&{}\ge f_\star +{\left\langle g_\star , x_k-x_\star \right\rangle }+\frac{1}{2L}{\left\| g_k-g_\star \right\| ^2} \\ &{}\quad +\frac{\mu }{2(1-\mu /L)}{\left\| x_k-x_\star -\frac{{1}}{L}(g_k-g_\star )\right\| ^2} \end{array} \quad&:2\gamma \rho (\gamma ),\\&\begin{array}{ll} h_\star \ge h_{k+1}+{\left\langle s_{k+1}, x_\star -x_{k+1}\right\rangle }&\end{array}\quad&:2\gamma ,\\&\begin{array}{ll}h_{k+1}\ge h_\star + {\left\langle s_\star , x_{k+1}-x_\star \right\rangle }&\end{array} \quad&:2\gamma . \end{aligned}$$

For simplicity, let us first sum the previous inequalities two by two:

$$\begin{aligned}&\begin{array}{ll} 0&{}\ge -{\left\langle g_\star -g_k, x_\star -x_k\right\rangle } +\frac{1}{L}{\left\| g_k-g_\star \right\| ^2} \\ &{}\quad +\frac{\mu }{1-\mu /L}{\left\| x_k-x_\star -\frac{{1}}{L}(g_k-g_\star )\right\| ^2} \end{array} \quad&:2\gamma \rho (\gamma ), \\&\begin{array}{ll} 0\ge -{\left\langle s_\star -s_{k+1}, x_\star -x_{k+1}\right\rangle }&\end{array}\quad&:2\gamma . \end{aligned}$$

We proceed by showing that (7) can be obtained by reformulation of the following expression:

$$\begin{aligned} 0\ge & {} 2\gamma \rho (\gamma ) \left[ {\left\langle g_k-g_\star , x_\star -x_k\right\rangle } +\frac{1}{L} {\left\| g_k-g_\star \right\| ^2} \right. \nonumber \\&\left. +\frac{\mu }{1-\mu /L} {\left\| x_k-x_\star -\frac{{1}}{L}(g_k-g_\star )\right\| ^2}\right] \nonumber \\&+\,2\gamma {\left\langle s_{k+1}-s_\star , x_\star -x_{k+1}\right\rangle }. \end{aligned}$$

(8)

For doing that, we simply verify that the expression (7) minus the expression (8) is identically zero; that is:

$$\begin{aligned} 0= & {} 2\gamma \rho (\gamma ) \left[ {\left\langle g_k-g_\star , x_\star -x_k\right\rangle } +\frac{1}{L}{\left\| g_k-g_\star \right\| ^2} \right. \nonumber \\&\left. +\frac{\mu }{1-\mu /L}{\left\| x_k-x_\star -\frac{{1}}{L}(g_k-g_\star )\right\| ^2}\right] \nonumber \\&+\,2\gamma {\left\langle s_{k+1}-s_\star , x_\star -x_{k+1}\right\rangle }\nonumber \\&-\left[ -\left( 1-\gamma \mu \right) ^2 {\left\| x_k-x_\star \right\| ^2}+{\left\| x_{k+1}-x_\star \right\| ^2} +\gamma ^2 {\left\| g_\star +s_{k+1}\right\| ^2}\right] \nonumber \\&-\frac{\gamma (2-\gamma (L+\mu ))}{L-\mu } {\left\| \mu {(x_k -x_\star )} - g_k+ g_\star \right\| ^2}, \end{aligned}$$

(9)

with $x_{k+1}=x_k-\frac{1}{L}\left( g_k+s_{k+1}\right)$ and $s_\star =-g_\star$.

Finally, one can simply verify the equality (9) by expanding (7) and (8), which are both equal to:

$$\begin{aligned} \begin{aligned} 0\ge \frac{2}{L-\mu }&\Bigg ((\gamma -\gamma ^2\mu ) {\left\| g_k\right\| ^2} +{(\gamma ^2\mu +\gamma ^2 L-2 \gamma ){\left\langle g_k, g_\star \right\rangle } }\\&+{( \gamma ^2 L-\gamma ^2\mu ) {\left\langle g_k, s_{k+1}\right\rangle }}+{(\gamma ^2 \mu ^2+\gamma ^2 \mu L-\gamma L -\gamma \mu ) {\left\langle g_k, x_k\right\rangle }}\\&+{(- \gamma ^2 \mu ^2-\gamma ^2\mu L +\gamma L+\gamma \mu ){\left\langle g_k, x_\star \right\rangle }}+{(\gamma -\gamma ^2 \mu ) {\left\| g_\star \right\| ^2} }\\&+{( \gamma ^2 L-\gamma ^2 \mu ){\left\langle g_\star , s_{k+1}\right\rangle }}+{(2\gamma \mu - \gamma ^2 \mu ^2-\gamma ^2 \mu L ){\left\langle g_\star , x_k\right\rangle }}\\&+{( \gamma ^2 \mu ^2+\gamma ^2\mu L-2 \gamma \mu ) {\left\langle g_\star , x_\star \right\rangle }}+{( \gamma ^2 L-\gamma ^2 \mu ) {\left\| s_{k+1}\right\| ^2}}\\&+{(\gamma \mu -\gamma L ) {\left\langle s_{k+1}, x_k\right\rangle }}+{(\gamma L-\gamma \mu ) {\left\langle s_{k+1}, x_\star \right\rangle }}\\&+{(\gamma \mu L -\gamma ^2 \mu ^2 L) {\left\| x_k\right\| ^2}}+{(2 \gamma ^2 \mu ^2 L-2 \gamma \mu L) {\left\langle x_k, x_\star \right\rangle }}\\&+{( \gamma \mu L-\gamma ^2 \mu ^2 L) {\left\| x_\star \right\| ^2}}\Bigg ). \end{aligned} \end{aligned}$$

$\square$

Note that all proofs presented in the symbolic validation code rely on the exact same idea: the equivalences between pairs of expressions are verified by checking their differences being identically zero.

Appendix B: Details on Lower Bounds for Mixed Performance Measures

In this section, we provide details for obtaining the lower bounds marked with ($\star$) in Table 2 (i.e., those that do not come from purely quadratic functions).

First, consider two constants $0<\mu \le L<\infty$ and the following one-dimensional quadratic minimization problem

$$\begin{aligned} \min _{x\ge 0} \left( \frac{\mu }{2} x^2+cx\right) . \end{aligned}$$

The function $f(x)=\frac{\mu }{2} x^2+cx$ is clearly L-smooth and $\mu$-strongly convex with unique optimal point $x_\star =0$ over the nonnegative reals. A corresponding composite problem can be written as $F(x)=f(x)+i_{\ge 0}(x)$ with $i_{\ge 0}(.)$ being the indicator function for the nonnegative real half-line.

Second, consider the starting point $x_0\ge 0$ and a number of iterations $N\in \mathbb {N}$. Using the proximal gradient method with step size $\frac{1}{L}$ results in the following rule for the iterates, assuming c is small enough (i.e., such that $x_k\ge 0$ for all $0\le k \le N$):

$$\begin{aligned} x_{k+1}&=x_k-\frac{1}{L} \nabla f(x_k),\\&=\left( 1-\frac{\mu }{L}\right) x_k-\frac{c}{L}. \end{aligned}$$

Solving the recurrence equation provides us with the following rule for the iterates:

$$\begin{aligned}x_{k}= {\frac{(1-\kappa )^N (c+\mu x_0)-c}{\mu }}, \end{aligned}$$

with $\kappa {:=}\frac{\mu }{L}$ the (inverse) condition number, and the corresponding values:

$$\begin{aligned}&\nabla f(x_N)={(1-\kappa )^N (c+\mu x_0)},\\&f(x_N)-f(x_\star )={\frac{(1-\kappa )^{2 N} (c+\mu x_0)^2}{2 \mu }.} \end{aligned}$$

By optimizing over c, we get the following extreme cases:

$\diamond$ :

(quadratic optimization—maximize $f(x_N)-f(x_\star )$ with respect to c) $c=\frac{\mu x_0}{(1-\kappa )^{-2N}-1}$ results in $F(x_N)-F(x_\star )=\frac{\mu }{2} \frac{{\left\| x_0-x_\star \right\| }^2}{\rho ^{-2N}-1}$,

$\diamond$ :

(linear optimization—maximize $|\nabla f(x_N)|$ with respect to c by enforcing equality in the constraint $x_N\ge 0$; that is, we impose $x_N=0$) $c=\frac{\mu x_0}{(1-\kappa )^{-N}-1}$ results in $||\tilde{\nabla } F(x_k)||^2=\frac{\mu ^2{\left\| x_0-x_\star \right\| }^2}{(\rho ^{-N}-1)^2}$,

$\diamond$ :

(linear optimization—maximize $|\nabla f(x_N)|$ with respect to c by enforcing equality in the constraint $x_N\ge 0$; that is, we impose $x_N=0$) $c=\frac{\mu x_0}{(1-\kappa )^{-N}-1}$ (or equivalently $c=\frac{\sqrt{2\mu } \sqrt{F(x_0)-F_\star }}{\sqrt{(1-\kappa )^{-2N}-1}}$) results in $||\tilde{\nabla } F(x_N)||^2={2\mu \frac{F(x_0)-F_\star }{\rho ^{-2N}-1}}$,

which match the entries marked ($\star$) in Table 2.