Convergence of a relaxed inertial proximal algorithm for maximally monotone operators

Abstract

In a Hilbert space ${\mathcal {H}}$, given $A{:}\;{\mathcal {H}}\rightarrow 2^{\mathcal {H}}$ a maximally monotone operator, we study the convergence properties of a general class of relaxed inertial proximal algorithms. This study aims to extend to the case of the general monotone inclusion $Ax \ni 0$ the acceleration techniques initially introduced by Nesterov in the case of convex minimization. The relaxed form of the proximal algorithms plays a central role. It comes naturally with the regularization of the operator A by its Yosida approximation with a variable parameter, a technique recently introduced by Attouch–Peypouquet (Math Program Ser B, 2018. https://doi.org/10.1007/s10107-018-1252-x) for a particular class of inertial proximal algorithms. Our study provides an algorithmic version of the convergence results obtained by Attouch–Cabot (J Differ Equ 264:7138–7182, 2018) in the case of continuous dynamical systems.

Appendices

Appendix A. Yosida regularization

A set-valued mapping A from ${\mathcal {H}}$ to ${\mathcal {H}}$ assigns to each $x\in {\mathcal {H}}$ a set $A(x)\subset {\mathcal {H}}$, hence it is a mapping from ${\mathcal {H}}$ to $2^{\mathcal {H}}$. Every set-valued mapping $A:{\mathcal {H}}\rightarrow 2^{\mathcal {H}}$ can be identified with its graph defined by

$$\begin{aligned}{\mathrm{gph}}A=\{(x,u)\in {\mathcal {H}}\times {\mathcal {H}}: \, u\in A(x)\}.\end{aligned}$$

The set $\{x\in {\mathcal {H}}:\ 0\in A(x)\}$ of the zeros of A is denoted by ${\mathrm{zer}}A$. An operator $A:{\mathcal {H}}\rightarrow 2^{\mathcal {H}}$ is said to be monotone if for any (x, u), $(y,v)\in {\mathrm{gph}}A$, one has $\langle y-x, v-u\rangle \ge 0$. It is maximally monotone if there exists no monotone operator whose graph strictly contains ${\mathrm{gph}}A$. If a single-valued operator $A:{\mathcal {H}}\rightarrow {\mathcal {H}}$ is continuous and monotone, then it is maximally monotone, cf.  [13, Proposition 2.4].

Given a maximally monotone operator A and $\lambda >0$, the resolvent of A with index $\lambda$ and the Yosida regularization of A with parameter $\lambda$ are defined by

$$\begin{aligned} J_{\lambda A} = \left( I + \lambda A \right) ^{-1}\qquad \hbox {and}\qquad A_{\lambda } = \frac{1}{\lambda } \left( I- J_{\lambda A} \right) , \end{aligned}$$

respectively. The operator $J_{\lambda A}: {\mathcal {H}}\rightarrow {\mathcal {H}}$ is nonexpansive and everywhere defined (indeed it is firmly non-expansive). Moreover, $A_{\lambda }$ is $\lambda$-cocoercive: for all $x, y \in {\mathcal {H}}$ we have

$$\begin{aligned} \langle A_{\lambda }y - A_{\lambda }x, y-x\rangle \ge \lambda \Vert A_{\lambda }y - A_{\lambda }x \Vert ^2 . \end{aligned}$$

This property immediately implies that $A_{\lambda }: {\mathcal {H}}\rightarrow {\mathcal {H}}$ is $\frac{1}{\lambda }$-Lipschitz continuous. Another property that proves useful is the resolvent equation (see, for example, [13, Proposition 2.6] or [8, Proposition 23.6])

$$\begin{aligned} (A_\lambda )_{\mu }= A_{(\lambda +\mu )}, \end{aligned}$$

which is valid for any $\lambda , \mu >0$. This property allows to compute simply the resolvent of $A_\lambda$ by

$$\begin{aligned} J_{\mu A_\lambda } = \frac{\lambda }{\lambda + \mu }I + \frac{\mu }{\lambda + \mu }J_{(\lambda + \mu )A}, \end{aligned}$$

for any $\lambda , \mu >0$. Also note that for any $x \in {\mathcal {H}}$, and any $\lambda >0$   $A_\lambda (x) \in A (J_{\lambda A}x)= A( x - \lambda A_\lambda (x)).$ Finally, for any $\lambda >0$, $A_{\lambda }$ and A have the same solution set, ${\mathrm{zer}}A_{\lambda }= {\mathrm{zer}}A$. For a detailed presentation of the maximally monotone operators and the Yosida approximation, the reader can consult [8] or [13].

Appendix B. Some auxiliary results

In this section, we present some auxiliary lemmas that are used throughout the paper.

Lemma B.1

Let $(a_k)$, $(\alpha _k)$ and $(w_k)$ be sequences of real numbers satisfying

$$\begin{aligned} a_{i+1}\le \alpha _i a_i+w_i \quad \text{ for } \text{ every } i\ge 1. \end{aligned}$$

(67)

Assume that $\alpha _i\ge 0$ for every $i\ge 1$.

• (i) For every $k\ge 1$, we have

  $$\begin{aligned} \sum _{i=1}^k a_i\le t_{1,k}a_1+\sum _{i=1}^{k-1} t_{i+1,k} w_i, \end{aligned}$$

  (68)

  where the double sequence $(t_{i,k})$ is defined by (13).

• (ii) Under $(K_0)$, assume that the sequence $(t_i)$ defined by (15) satisfies $\sum _{i=1}^{+\infty }t_{i+1}(w_i)_+<~+\infty$. Then the series $\sum _{i\ge 1}(a_i)_+$ is convergent, and

  $$\begin{aligned}\sum _{i= 1}^{+\infty }(a_i)_+\le t_1(a_1)_+ +\sum _{i=1}^{+\infty } t_{i+1} (w_i)_+.\end{aligned}$$

Proof

$\mathrm{{(i)}}$ Recall from Lemma 2.4$\mathrm{{(i)}}$ that $\alpha _i t_{i+1,k}=t_{i,k}-1$ for every $i\ge 1$ and $k\ge i+1$. Multiplying inequality (67) by $t_{i+1,k}$ gives

$$\begin{aligned}t_{i+1,k} a_{i+1}\le (t_{i,k}-1) a_i+t_{i+1,k}w_i,\end{aligned}$$

or equivalently

$$\begin{aligned}a_i\le (t_{i,k}a_i-t_{i+1,k} a_{i+1}) +t_{i+1,k}w_i.\end{aligned}$$

By summing from $i=1$ to $k-1$, we deduce that

$$\begin{aligned}\sum _{i=1}^{k-1} a_i\le t_{1,k}a_1-t_{k,k}a_k+\sum _{i=1}^{k-1} t_{i+1,k} w_i.\end{aligned}$$

Since $t_{k,k}=1$, inequality (68) follows immediately.

$\mathrm{{(ii)}}$ Taking the positive part of each member of (67), we find

$$\begin{aligned}(a_{i+1})_+\le \alpha _i (a_i)_+ +(w_i)_+ .\end{aligned}$$

By applying $\mathrm{{(i)}}$ with $(a_i)_+$ (resp. $(w_i)_+$) in place of $a_i$ (resp. $w_i$), we obtain for every $k\ge 1$

$$\begin{aligned} \sum _{i=1}^k (a_i)_+\le & {} t_{1,k}(a_1)_+ +\sum _{i=1}^{k-1} t_{i+1,k} (w_i)_+ \le t_{1}(a_1)_+ +\sum _{i=1}^{+\infty } t_{i+1} (w_i)_+<+\infty , \end{aligned}$$

because $t_{i+1,k} \le t_{i+1}$, and $\sum _{i=1}^{+\infty } t_{i+1} (w_i)_+<+\infty$ by assumption. Then just let k tend to $+ \infty$. $\square$

Given a bounded sequence $(x_k)$ of a Banach space $({\mathcal {X}},\Vert .\Vert )$, the next lemma gives basic properties of the averaged sequence $(\widehat{x}_k)$ defined by (57).

Lemma B.2

Let $({\mathcal {X}},\Vert .\Vert )$ be a Banach space and let $(x_k)$ be a bounded sequence of ${\mathcal {X}}$. Given a sequence $(\tau _{i,k})_{i,k\ge 1}$ of nonnegative numbers satisfying (55)–(56), let $(\widehat{x}_k)$ be the averaged sequence defined by $\widehat{x}_k=\sum _{i=1}^{+\infty }\tau _{i,k}x_i$. Then we have

• (i) The sequence $(\widehat{x}_k)$ is well-defined, bounded and $\sup _{k\ge 1}\Vert \widehat{x}_k\Vert \le \sup _{k\ge 1}\Vert x_k\Vert$.

• (ii) If $(x_k)$ converges toward $\overline{x^{}}\in {\mathcal {X}}$, then the sequence $(\widehat{x}_k)$ is also convergent and $\lim _{k\rightarrow +\infty }\widehat{x}_k=\overline{x^{}}$.

Proof

$\mathrm{{(i)}}$ Set $M=\sup _{k\ge 1}\Vert x_k\Vert <+\infty$. In view of (55), observe that for every $k\ge 1$,

$$\begin{aligned} \sum _{i=1}^{+\infty }\tau _{i,k}\Vert x_i\Vert \le M\, \sum _{i=1}^{+\infty }\tau _{i,k}=M. \end{aligned}$$

(69)

Since the space ${\mathcal {X}}$ is complete, we classically deduce that the series $\sum _{i\ge 1}\tau _{i,k}x_i$ is convergent. From the definition of $\widehat{x}_k$, we then have $\Vert \widehat{x}_k\Vert \le \sum _{i=1}^{+\infty }\tau _{i,k}\Vert x_i\Vert ,$ and hence $\Vert \widehat{x}_k\Vert \le M$ in view of (69).

$\mathrm{{(ii)}}$ Assume that $(x_k)$ converges toward $\overline{x^{}}\in {\mathcal {X}}$. By using (55), we have for every $k\ge 1$,

$$\begin{aligned} \Vert \widehat{x}_k-\overline{x^{}}\Vert= & {} \left\| \sum _{i=1}^{+\infty }\tau _{i,k}(x_i-\overline{x^{}})\right\| \le \sum _{i=1}^{+\infty }\tau _{i,k}\Vert x_i-\overline{x^{}}\Vert . \end{aligned}$$

Fix $\varepsilon >0$, and let $K\ge 1$ such that $\Vert x_i-\overline{x^{}}\Vert \le \varepsilon$ for every $i\ge K$. From the above inequality, we obtain

$$\begin{aligned} \Vert \widehat{x}_k-\overline{x^{}}\Vert\le & {} \left( \sup _{i\in \{1,\ldots ,K\}}\Vert x_i-\overline{x^{}}\Vert \right) \left( \sum _{i=1}^K\tau _{i,k}\right) +\varepsilon \, \sum _{i=K+1}^{+\infty }\tau _{i,k} \le M\,\sum _{i=1}^K\tau _{i,k} +\varepsilon , \end{aligned}$$

with $M=\sup _{i\ge 1}\Vert x_i-\overline{x^{}}\Vert <+\infty$. Taking the upper limit as $k\rightarrow +\infty$, we deduce from (56) that

$$\begin{aligned}\limsup _{k\rightarrow +\infty }\Vert \widehat{x}_k-\overline{x^{}}\Vert \le \varepsilon .\end{aligned}$$

Since this is true for every $\varepsilon >0$, we conclude that $\lim _{k\rightarrow +\infty }\Vert \widehat{x}_k-\overline{x^{}}\Vert =0$. $\square$