Convergence of a Relaxed Inertial Forward–Backward Algorithm for Structured Monotone Inclusions

Abstract

In a Hilbert space ${{\mathcal {H}}}$, we study the convergence properties of a class of relaxed inertial forward–backward algorithms. They aim to solve structured monotone inclusions of the form $Ax + Bx \ni 0$ where $A:{{\mathcal {H}}}\rightarrow 2^{{\mathcal {H}}}$ is a maximally monotone operator and $B:{{\mathcal {H}}}\rightarrow {{\mathcal {H}}}$ is a cocoercive operator. We extend to this class of problems the acceleration techniques initially introduced by Nesterov, then developed by Beck and Teboulle in the case of structured convex minimization (FISTA). As an important element of our approach, we develop an inertial and parametric version of the Krasnoselskii–Mann theorem, where joint adjustment of the inertia and relaxation parameters plays a central role. This study comes as a natural extension of the techniques introduced by the authors for the study of relaxed inertial proximal algorithms. An illustration is given to the inertial Nash equilibration of a game combining non-cooperative and cooperative aspects.

Appendices

Appendix A: Yosida Regularization

Let $A:{{\mathcal {H}}}\rightarrow 2^{{\mathcal {H}}}$ be a maximally monotone operator. For any $\mu >0$, its resolvent with index $\mu$ is defined by $J_{\mu A} = \left( I + \mu A \right) ^{-1}$. The operator $J_{\mu A}: {{\mathcal {H}}}\rightarrow {{\mathcal {H}}}$ is nonexpansive and everywhere defined (indeed it is firmly non-expansive). The Yosida regularization of A with parameter $\mu$ is defined by $A_{\mu } = \frac{1}{\mu } \left( I- J_{\mu A} \right)$. The operator $A_{\mu }$ is $\mu$-cocoercive: for all $x, y \in {{\mathcal {H}}}$ we have

$$\begin{aligned} \langle A_{\mu }y - A_{\mu }x, y-x\rangle \ge \mu \Vert A_{\mu }y - A_{\mu }x \Vert ^2 . \end{aligned}$$

This property immediately implies that $A_{\mu }: {{\mathcal {H}}}\rightarrow {{\mathcal {H}}}$ is $\frac{1}{\mu }$-Lipschitz continuous. Another property that proves useful is the resolvent equation (see, for example [18, Proposition 2.6] or [14, Proposition 23.6])

$$\begin{aligned} (A_\mu )_{\theta }= A_{(\theta +\mu )}, \end{aligned}$$

which is valid for any $\theta , \mu >0$. This property allows to compute simply the resolvents of $A_\mu$ by

$$\begin{aligned} J_{\theta A_\mu } = \frac{\mu }{\theta + \mu }I + \frac{\theta }{\theta + \mu }J_{(\theta + \mu )A}, \end{aligned}$$

for any $\theta , \mu >0$. Also note that for any $x \in {{\mathcal {H}}}$, and any $\mu >0$

$$\begin{aligned} A_\mu (x) \in A (J_{\mu A}x)= A( x - \mu A_\mu (x)). \end{aligned}$$

(65)

Finally, for any $\mu >0$, $A_{\mu }$ and A have the same solution set, ${\mathrm{zer}}A_{\mu }= {\mathrm{zer}}A$. For a detailed presentation of the properties of the maximally monotone operators and the Yosida approximation, see References [14, 18].

Appendix B: Properties of the Operator $M_{A,B,\mu }$

The cocoercivity properties of the operator $M_{A,B,\mu }$ which is defined by $M_{A,B,\mu }(x)=\frac{1}{\mu } \left( x- J_{\mu A}( x-\mu B(x))\right)$ play a central role in our analysis.

Lemma B.1

Let A be a maximally monotone operator. Suppose that B is $\lambda$-cocoercive, and that $\mu \in ]0, 2\lambda [$. Then the operator $M_{A,B,\mu }$ is $\beta$-cocoercive with $\beta = \mu \left( 1-\frac{\mu }{4\lambda }\right)$.

Proof

The proof is based on the link between the cocoercivity property and the $\alpha$-averaged property. Recall that $T: {{\mathcal {H}}}\rightarrow {{\mathcal {H}}}$ is $\alpha$-averaged if $T= (1-\alpha )I + \alpha R$  for some nonexpansive operator $R: {{\mathcal {H}}}\rightarrow {{\mathcal {H}}}$, and $\alpha \in ]0,1[$. The operators $J_{\mu A}$ and $I-\mu B$ are respectively $\alpha _1:= \frac{1}{2}$ and $\alpha _2:= \frac{\mu }{2 \lambda }$ averaged, see References [14, Corollary 23.8], resp. [14, Proposition 4.33]. By [24, Proposition 2.4] their composition $J_{\mu A}(I-\mu B)$ is $\alpha$-averaged with $\alpha = \frac{\alpha _1 + \alpha _2 -2 \alpha _1 \alpha _2}{1- \alpha _1 \alpha _2}$, which yields $\alpha = \frac{1}{2(1-\frac{\mu }{4\lambda } )}$. Hence $J_{\mu A}(I-\mu B) =(1-\alpha )I + \alpha R$ for some nonexpansive operator R, which gives $M_{A,B,\mu } = \frac{\alpha }{\mu } (I-R)$. Since $I-R$ is $\frac{1}{2}$-cocoercive, see for example [3, Lemma 2.3], we finally obtain that $M_{A,B,\mu }$ is $\frac{1}{2}\times \frac{\mu }{\alpha }= \mu \left( 1-\frac{\mu }{4\lambda }\right)$-cocoercive. $\square$

Lemma B.2

Let $A:{{\mathcal {H}}}\rightarrow 2^{{\mathcal {H}}}$ be a maximally monotone operator and let $B: {{\mathcal {H}}}\rightarrow {{\mathcal {H}}}$ be a $\lambda$-cocoercive operator for some $\lambda >0$. Given $(z,q)\in \mathrm{gph}(A+B)$ and $\theta \in [0,1[$, we have for every $x\in {{\mathcal {H}}}$ and $\mu >0$,

$$\begin{aligned} \langle M_{A,B,\mu }(x), x-z \rangle\ge & {} \left( \mu -\frac{\mu ^2}{4(1-\theta )\lambda }\right) \Vert M_{A,B,\mu }(x)\Vert ^2+ \theta \lambda \Vert B(x)-B(z)\Vert ^2\nonumber \\&+\langle q, x-\mu M_{A,B,\mu }(x)-z\rangle . \end{aligned}$$

(66)

In particular, we obtain for $\theta =0$

$$\begin{aligned} \langle M_{A,B,\mu }(x), x-z \rangle\ge & {} \left( \mu -\frac{\mu ^2}{4\lambda }\right) \Vert M_{A,B,\mu }(x)\Vert ^2\nonumber \\&+ \langle q, x-\mu M_{A,B,\mu }(x)-z\rangle . \end{aligned}$$

(67)

Now assume that ${\mathrm{zer}}(A+B)\ne \emptyset$ and let $z\in {\mathrm{zer}}(A+B)$. For every $x\in {{\mathcal {H}}}$, $\mu >0$ and $\theta \in [0,1[$, the following inequality holds true

$$\begin{aligned} \langle M_{A,B,\mu }(x), x-z \rangle\ge & {} \left( \mu -\frac{\mu ^2}{4(1-\theta )\lambda }\right) \Vert M_{A,B,\mu }(x)\Vert ^2\nonumber \\&+\, \theta \lambda \Vert B(x)-B(z)\Vert ^2. \end{aligned}$$

(68)

Proof

First observe that for every $x\in {{\mathcal {H}}}$ and $\mu >0$, $M_{A,B,\mu }(x)=A_\mu (x-\mu B(x))+B(x).$ Using the classical property (65) of the Yosida approximation, this leads to

$$\begin{aligned} M_{A,B,\mu }(x)-B(x)\in A\left( x-\mu M_{A,B,\mu }(x)\right) . \end{aligned}$$

(69)

Since $(z,q)\in \mathrm{gph}(A+B)$, we have $q-B(z)\in A(z)$. The monotonicity of A then yields

$$\begin{aligned} \left\langle M_{A,B,\mu }(x)-B(x)-(q-B(z)), x-\mu M_{A,B,\mu }(x)-z\right\rangle \ge 0. \end{aligned}$$

Hence

$$\begin{aligned} \langle M_{A,B,\mu }(x), x-z\rangle\ge & {} \mu \Vert M_{A,B,\mu }(x)\Vert ^2+\langle B(x)-B(z), x-z\rangle \nonumber \\&-\,\mu \langle B(x)-B(z), M_{A,B,\mu }(x)\rangle \nonumber \\&+\,\langle q, x-\mu M_{A,B,\mu }(x)-z\rangle . \end{aligned}$$

(70)

The $\lambda$-cocoercivity of B gives

$$\begin{aligned} \langle B(x)-B(z), x-z\rangle \ge \lambda \Vert B(x)-B(z)\Vert ^2. \end{aligned}$$

(71)

On the other hand, by using Cauchy-Schwarz inequality, we obtain

$$\begin{aligned} \langle B(x)-B(z), M_{A,B,\mu }(x)\rangle\le & {} \Vert B(x)-B(z)\Vert \Vert M_{A,B,\mu }(x)\Vert \nonumber \\\le & {} (1-\theta )\frac{\lambda }{\mu }\Vert B(x)-B(z)\Vert ^2\nonumber \\&+ \frac{\mu }{4(1-\theta )\lambda }\Vert M_{A,B,\mu }(x)\Vert ^2. \end{aligned}$$

(72)

By combining (70), (71) and (72), we immediately find (66). Inequality (67) (resp. (68)) is obtained by taking $\theta =0$ (resp. $q=0$) in (66). $\square$

Lemma B.3

Let $A:{{\mathcal {H}}}\rightarrow 2^{{\mathcal {H}}}$ be a maximally monotone operator, and let $B: {{\mathcal {H}}}\rightarrow {{\mathcal {H}}}$ be a $\lambda$-cocoercive operator for some $\lambda >0$. Let $(\xi _n)$ be a sequence of ${{\mathcal {H}}}$, and let $(\lambda _n)$ be a sequence of positive numbers. Assume that $\xi _n\rightharpoonup \overline{x^{}}$ weakly in ${{\mathcal {H}}}$ and that

$$\begin{aligned} \lim _{n\rightarrow +\infty }\Vert M_{A,B,\lambda _{n}}(\xi _n)\Vert =0 \quad \hbox {and}\quad \lim _{n\rightarrow +\infty }\Vert \lambda _{n}M_{A,B,\lambda _{n}}(\xi _n)\Vert =0. \end{aligned}$$

(73)

Then we have $\overline{x^{}}\in {\mathrm{zer}}(A+B)$.

Proof

By using inclusion (69) with $x=\xi _n$ and $\mu =\lambda _n$, we obtain

$$\begin{aligned} M_{A,B,\lambda _n}(\xi _n)-B(\xi _n)\in A(\xi _n-\lambda _n M_{A,B,\lambda _n}(\xi _n)) \end{aligned}$$

and hence

$$\begin{aligned}&M_{A,B,\lambda _{n}}(\xi _n)+B(\xi _n-\lambda _{n} M_{A,B,\lambda _{n}}(\xi _n))-B(\xi _n)\nonumber \\&\quad \in (A+B)\left( \xi _n-\lambda _{n} M_{A,B,\lambda _{n}}(\xi _n)\right) . \end{aligned}$$

(74)

Since the operator B is $\frac{1}{\lambda }$-Lipschitz continuous, we have

$$\begin{aligned}&\Vert M_{A,B,\lambda _{n}}(\xi _n)+B(\xi _n-\lambda _{n} M_{A,B,\lambda _{n}}(\xi _n))-B(\xi _n)\Vert \\&\quad \le \Vert M_{A,B,\lambda _{n}}(\xi _n)\Vert +\Vert B(\xi _n-\lambda _{n} M_{A,B,\lambda _{n}}(\xi _n))-B(\xi _n)\Vert \\&\quad \le \Vert M_{A,B,\lambda _{n}}(\xi _n)\Vert +\frac{1}{\lambda }\Vert \lambda _{n}M_{A,B,\lambda _{n}}(\xi _n)\Vert . \end{aligned}$$

We then deduce from (73) that the left member of (74) tends to 0 strongly in ${{\mathcal {H}}}$ as $n\rightarrow +\infty$. By invoking again (73), we see that $\xi _n-\lambda _{n} M_{A,B,\lambda _{n}}(\xi _n)$ converges weakly to $\overline{x^{}}$ as $n\rightarrow +\infty$. Taking the limit as $n\rightarrow +\infty$ in (74), we conclude that $0\in (A+B)(\overline{x^{}})$, due to the graph-closedness of the maximally monotone operator $A+B$ for the weak-strong topology in ${{\mathcal {H}}}\times {{\mathcal {H}}}$. $\square$

We now study the variations of the function $(x,\mu )\mapsto \mu M_{A,B,\mu }(x)$.

Lemma B.4

Let $A:{{\mathcal {H}}}\rightarrow 2^{{\mathcal {H}}}$ be a maximally monotone operator, and let $B: {{\mathcal {H}}}\rightarrow {{\mathcal {H}}}$ be a $\lambda$-cocoercive operator for some $\lambda >0$. Assume that ${\mathrm{zer}}(A+B)\ne \emptyset$. Let $\gamma$, $\delta \in ]0,2\lambda [$, and x, $y\in {{\mathcal {H}}}$. Then, for each $z\in {\mathrm{zer}}(A+B)$, we have

$$\begin{aligned} \Vert \gamma M_{A,B,\gamma }(x)-\delta M_{A,B,\delta }(y)\Vert \le 2\Vert x-y\Vert +2\frac{|\gamma -\delta |}{\gamma }\Vert x-z\Vert . \end{aligned}$$

Proof

The proof follows the lines of Reference [10, Lemma A.4]. By using successively the definition of $M_{A,B,\gamma }$, the resolvent identity [14, Proposition 23.28 (i)], and the nonexpansive property of the resolvent, we obtain

$$\begin{aligned} \Vert \gamma M_{A,B,\gamma }(x)- & {} \delta M_{A,B,\delta }(y)\Vert \le \Vert x-y\Vert +\Vert J_{\gamma A}(x-\gamma B(x))-J_{\delta A}(y-\delta B(y))\Vert \nonumber \\= & {} \Vert x-y\Vert +\left\| J_{\delta A}\left( \frac{\delta }{\gamma }(x-\gamma B(x))+\left( 1-\frac{\delta }{\gamma }\right) J_{\gamma A}(x-\gamma B(x))\right) \right. \nonumber \\&\left. -J_{\delta A}(y-\delta B(y))\right\| \nonumber \\\le & {} \Vert x-y\Vert +\left\| \frac{\delta }{\gamma }(x-\gamma B(x))\right. \nonumber \\&\left. +\left( 1-\frac{\delta }{\gamma }\right) J_{\gamma A}(x-\gamma B(x)) -(y-\delta B(y))\right\| \nonumber \\= & {} \Vert x-y\Vert +\left\| (x-\delta B(x))-(y-\delta B(y))\right. \nonumber \\&\left. +\left( 1-\frac{\delta }{\gamma }\right) (J_{\gamma A}(x-\gamma B(x))-x)\right\| \nonumber \\\le & {} \Vert x-y\Vert +\Vert (x-\delta B(x))-(y-\delta B(y))\Vert \nonumber \\&+\left| 1-\frac{\delta }{\gamma }\right| \Vert J_{\gamma A}(x-\gamma B(x))-x\Vert . \end{aligned}$$

(75)

Since $\delta \in ]0, 2\lambda [$, the operator $I-\delta B$ is $\frac{\delta }{2\lambda }$-averaged, see Reference [14, Proposition 4.33]. This guarantees that the operator $I-\delta B$ is nonexpansive, hence

$$\begin{aligned} \Vert (x-\delta B(x))-(y-\delta B(y))\Vert \le \Vert x-y\Vert . \end{aligned}$$

(76)

On the other hand, observe that for $z\in {\mathrm{zer}}(A+B)$,

$$\begin{aligned} x-J_{\gamma A}(x-\gamma B(x))=x-J_{\gamma A}(x-\gamma B(x))-(z-J_{\gamma A}(z-\gamma B(z))). \end{aligned}$$

From the triangle inquality combined with the nonexpansive property of the resolvent, we deduce that

$$\begin{aligned} \Vert x-J_{\gamma A}(x-\gamma B(x))\Vert\le & {} \Vert x-z\Vert +\Vert J_{\gamma A}(x-\gamma B(x))-J_{\gamma A}(z-\gamma B(z))\Vert \nonumber \\\le & {} \Vert x-z\Vert +\Vert (x-\gamma B(x))-(z-\gamma B(z))\Vert \nonumber \\\le & {} 2\Vert x-z\Vert \quad \hbox { because }I-\gamma B\hbox { is nonexpansive.} \end{aligned}$$

(77)

By putting together (75), (76) and (77), we conclude that

$$\begin{aligned} \Vert \gamma M_{A,B,\gamma }(x)-\delta M_{A,B,\delta }(y)\Vert \le 2\Vert x-y\Vert +2\left| 1-\frac{\delta }{\gamma }\right| \Vert x-z\Vert . \end{aligned}$$

$\square$

Appendix C: Some Auxiliary Results

Lemma C.1

(Attouch-Cabot [6]) Let $(a_k)$, $(\alpha _k)$ and $(w_k)$ be sequences of real numbers satisfying

$$\begin{aligned} a_{k+1}\le \alpha _k a_k+w_k \quad \hbox {for every } k\ge 1. \end{aligned}$$

Assume that $\alpha _k\ge 0$ for every $k\ge 1$. Let $(t_i)$ and $(t_{i,k})$ be the sequences respectively defined by (6) and (7).

• (i) For every $k\ge 1$, we have

  $$\begin{aligned} \sum _{i=1}^k a_i\le t_{1,k}a_1+\sum _{i=1}^{k-1} t_{i+1,k} w_i. \end{aligned}$$

• (ii) Under $(K_0)$, assume that $\sum _{i=1}^{+\infty }t_{i+1}[w_i]_+<~+\infty$. Then the series $\sum _{i\ge 1}[a_i]_+$ is convergent, and

  $$\begin{aligned} \sum _{i= 1}^{+\infty }[a_i]_+\le t_1 [a_1]_+ +\sum _{i=1}^{+\infty } t_{i+1} [w_i]_+. \end{aligned}$$

The proof is omitted and the reader is referred to Reference [6, Lemma B.1]. It makes use of Reference [6, Lemma 2.4], which corresponds to Lemma 2.1.

The next lemma gives basic properties of the averaging process (58), see Reference [6, Lemma B.2] for the proof.

Lemma C.2

(Attouch-Cabot [6]) Let $({{\mathcal {X}}},\Vert .\Vert )$ be a Banach space and let $(x_k)$ be a bounded sequence of ${{\mathcal {X}}}$. Given a sequence $(\tau _{i,k})_{i,k\ge 1}$ of nonnegative numbers satisfying (56) and (57), let $({\widehat{x}}_k)$ be the averaged sequence defined by ${\widehat{x}}_k=\sum _{i=1}^{+\infty }\tau _{i,k}x_i$. Then we have

• (i) The sequence $({\widehat{x}}_k)$ is well-defined, bounded and $\sup _{k\ge 1}\Vert {\widehat{x}}_k\Vert \le \sup _{k\ge 1}\Vert x_k\Vert$.

• (ii) If $(x_k)$ converges toward $\overline{x^{}}\in {{\mathcal {X}}}$, then the sequence $({\widehat{x}}_k)$ is also convergent and $\lim _{k\rightarrow +\infty }{\widehat{x}}_k=\overline{x^{}}$.