On the Proximal Gradient Algorithm with Alternated Inertia

Abstract

In this paper, we investigate attractive properties of the proximal gradient algorithm with inertia. Notably, we show that using alternated inertia yields monotonically decreasing functional values, which contrasts with usual accelerated proximal gradient methods. We also provide convergence rates for the algorithm with alternated inertia, based on local geometric properties of the objective function. The results are put into perspective by discussions on several extensions (strongly convex case, non-convex case, and alternated extrapolation) and illustrations on common regularized optimization problems.

Appendix

For the sake of completeness, we provide short and direct proofs of known lemmas recalled in Sect. 2.2.

Proof of Lemma 2.2

Let $x\in \mathbb {R}^n$; by definition, we have

$$\begin{aligned} \mathsf {T}_\gamma (x)&= \mathop {\hbox {argmin}}\limits _w \left( \gamma g(w) + \frac{1}{2} \left\| w- \left( x - \gamma \nabla f(x) \right) \right\| ^2 \right) \\&= \mathop {\hbox {argmin}}\limits _w \left( \underbrace{ f(x) + g(w) + \langle w-x ; \nabla f(x) \rangle + \frac{1}{2\gamma } \left\| w- x \right\| ^2 }_{s_x(w)} \right) \end{aligned}$$

and, as it is defined as the minimizer of $\frac{1}{\gamma }$-strongly convex surrogate function $s_x$, we have for any $y\in \mathbb {R}^n$ that $s_x ( \mathsf {T}_\gamma (x)) + \frac{1}{2\gamma } \Vert \mathsf {T}_\gamma (x) - y\Vert ^2 \le s_x (y)$ so

$$\begin{aligned}&f(x) + g(\mathsf {T}_\gamma (x) ) + \langle \mathsf {T}_\gamma (x) -x ; \nabla f(x) \rangle + \frac{\left\| \mathsf {T}_\gamma (x) - x \right\| ^2 }{2\gamma } + \frac{\left\| \mathsf {T}_\gamma (x) - y \right\| ^2 }{2\gamma } \nonumber \\&\quad \le f(x) + g(y) + \langle y-x ; \nabla f(x) \rangle + \frac{\left\| y- x \right\| ^2 }{2\gamma }. \end{aligned}$$

(14)

Now we use (i) the descent lemma on L-smooth function f (see [23, Th. 18.15]) to show that

$$\begin{aligned} f( \mathsf {T}_\gamma (x) ) \le f(x) + \langle \mathsf {T}_\gamma (x) -x ; \nabla f(x) \rangle + \frac{L}{2} \left\| \mathsf {T}_\gamma (x) - x \right\| ^2 \end{aligned}$$

(15)

and (ii) the convexity of f to have

$$\begin{aligned} f(x) + \langle y-x ; \nabla f(x) \rangle \le f(y) . \end{aligned}$$

(16)

Using Eq. (15) on the left-hand side of (14) and Eq. (16) on the right-hand side, we get

$$\begin{aligned}&f(\mathsf {T}_\gamma (x)) + g(\mathsf {T}_\gamma (x) ) + \frac{ (1-\gamma L) \left\| \mathsf {T}_\gamma (x) - x \right\| ^2 }{2\gamma } + \frac{\left\| \mathsf {T}_\gamma (x) - y \right\| ^2 }{2\gamma } \\&\quad \le f(y) + g(y) + \frac{\left\| y- x \right\| ^2 }{2\gamma }. \end{aligned}$$

$\square$

Proof of Lemma 2.3

Let $x\in \mathbb {R}^n$, and let $y = \mathsf {T}_\gamma (x) \in {{\mathrm{argmin}}}_w \left( \gamma g(w) + \frac{1}{2} \left\| w- \left( x - \gamma \nabla f(x) \right) \right\| ^2 \right)$, then

$$\begin{aligned}&0 \in \gamma \partial g(y) + y - x + \gamma \nabla f(x) ~~~~ \Leftrightarrow ~~~~ 0 \in \nabla f(y) + \partial g(y) + \nabla f(x) \\&\quad - \nabla f(y) + \frac{1}{\gamma }(y-x) \end{aligned}$$

so $\nabla f(y) - \nabla f(x) + \frac{1}{\gamma }(x-y) \in \partial F(y)$, thus we have ${{\mathrm{dist}}}(0,\partial F(y)) \le \Vert \nabla f(y) - \nabla f(x) + \frac{1}{\gamma }(x-y) \Vert \le \left( L + \frac{1}{\gamma }\right) \Vert x-y\Vert$. $\square$

Lemma

(Non-convex version of Lemma 2.2) Let Assumption 1 hold but with g possibly non-convex, and take $\gamma >0$. Then, for any $x,y\in \mathbb {R}^n$,

$$\begin{aligned} F( \mathsf {T}_\gamma (x) ) + \frac{ (1-\gamma L)}{2\gamma } \left\| \mathsf {T}_\gamma (x) - x \right\| ^2 \le F(y) + \frac{1}{2\gamma } \left\| x - y \right\| ^2. \end{aligned}$$

Proof

Let $x\in \mathbb {R}^n$; by definition, we have, as in the proof of Lemma 2.2,

$$\begin{aligned} \mathsf {T}_\gamma (x)&= \mathop {\hbox {argmin}}\limits _w \left( \gamma g(w) + \frac{1}{2} \left\| w- \left( x - \gamma \nabla f(x) \right) \right\| ^2 \right) \\&= \mathop {\hbox {argmin}}\limits _w \left( \underbrace{ f(x) + g(w) + \langle w-x ; \nabla f(x) \rangle + \frac{1}{2\gamma } \left\| w- x \right\| ^2 }_{s_x(w)} \right) \end{aligned}$$

and, as it is defined as a minimizer of (non-necessarily convex) surrogate function $s_x$, we have for any $y\in \mathbb {R}^n$ that $s_x ( \mathsf {T}_\gamma (x)) \le s_x (y)$ (which differs from the convex case of Lemma 2.2) thus

$$\begin{aligned}&f(x) + g(\mathsf {T}_\gamma (x) ) + \langle \mathsf {T}_\gamma (x) -x ; \nabla f(x) \rangle + \frac{\left\| \mathsf {T}_\gamma (x) - x \right\| ^2 }{2\gamma } \nonumber \\&\quad \le f(x) + g(y) + \langle y-x ; \nabla f(x) \rangle + \frac{\left\| y- x \right\| ^2 }{2\gamma }. \end{aligned}$$

(17)

The proof then follows the same lines as that of Lemma 2.2.$\square$