Splitting Methods with Variable Metric for Kurdyka–Łojasiewicz Functions and General Convergence Rates

Abstract

We study the convergence of general descent methods applied to a lower semi-continuous and nonconvex function, which satisfies the Kurdyka–Łojasiewicz inequality in a Hilbert space. We prove that any precompact sequence converges to a critical point of the function, and obtain new convergence rates both for the values and the iterates. The analysis covers alternating versions of the forward–backward method with variable metric and relative errors. As an example, a nonsmooth and nonconvex version of the Levenberg–Marquardt algorithm is detailed.

Appendix

1.1 Proofs of Theorems 3.1 and 3.2

The argument is a straightforward adaptation of the ideas in the proof of [17, Lemma 2.6]. One first proves the following:

Lemma 7.1

Let $\mathbf {H}_1$ and $\mathbf {H}_2$ hold and fix $k\in \mathbb {N}$. If $x^k$ and $x^{k+1}$ belong to $\underline{\Gamma }_\eta (x^*,\delta )$, then

$$\begin{aligned} 2\Vert x^{k+1}-x^k\Vert&\le \Vert x^k-x^{k-1}\Vert \\&+\frac{1}{a_kb_k}\big [\varphi (f(x^k)-f(x^*)) -\varphi (f(x^{k+1})-f(x^*))\big ]+\epsilon _{k}.\nonumber \end{aligned}$$

(24)

For next results, we introduce the following auxiliary property (automatically fulfilled under the hypotheses of Theorems 3.1 and 3.2), which includes a stability of the sequence $(x^k)_{k\in \mathbb {N}}$ with respect to the point $x^*$, along with a sufficiently close initialization.

$\underline{\mathbf {S}(x^*,\delta ,\rho )}$: There exist $\delta >\rho >0$ such that

• (i) For each $k\in \mathbb {N}$, if $\frac{}{}\!x^0,\dots ,x^k\in \underline{\Gamma }_\eta (x^*,\rho )$, then $x^{k+1}\in \underline{\Gamma }_\eta (x^*,\delta )$;

• (ii) The initial point $x^0$ belongs to $\Gamma _\eta (x^*,\rho )$ and

  $$\begin{aligned} \Vert x^*-x^{0}\Vert +2\sqrt{\frac{f(x^{0})-f(x^*)}{a_0}}+ M\varphi (f(x^{0})-f(x^*))+\sum _{i=1}^{+\infty }\epsilon _i < \rho . \end{aligned}$$

  (25)

Then, we have the following estimation:

Lemma 7.2

Let $\mathbf {H}_1$, $\mathbf {H}_2$, $\mathbf {H}_3$ and $\mathbf {S}(x^*,\delta ,\rho )$ hold, and $M:=\sup _{k\in \mathbb {N}^*}\frac{1}{a_kb_k}<+\infty$. Then, for all $K\in \mathbb {N}^*$, we have $x^K\in \underline{\Gamma }_\eta (x^*,\rho )$ and

$$\begin{aligned}&\sum _{k=1}^{K}\Vert x^{k+1}-x^k\Vert +\Vert x^{K+1}-x^K\Vert \\&\quad \le \Vert x^1-x^0\Vert +M\big [\varphi (f(x^1)-f( x^*))-\varphi (f(x^{K+1})-f(x^*))\big ]+\sum _{k=1}^{K}\epsilon _k. \end{aligned}$$

The basic asymptotic properties are given by the following result:

Proposition 7.1

Let $\mathbf {H}_1$, $\mathbf {H}_2$, $\mathbf {H}_3$ and $\mathbf {S}(x^*,\delta ,\rho )$ hold. Then, $x^k\in \underline{\Gamma }_\eta (x^*,\rho )$ for all $k$, the sequence has finite length and converges to some $\overline{x}$. Moreover, $\liminf \limits _{k\rightarrow \infty }\Vert \partial f(x^k) \Vert _- =0$, and $f(\overline{x})\le \lim \limits _{k\rightarrow \infty }f(x^k)=f(x^*)$.

Proof

Capture, convergence and finite length follow from Lemma 7.2 and $\mathbf {H}_3$. Next, since $(b_k)\notin \ell ^1$ and $\sum _{k=1}^\infty b_{k+1}\Vert \partial f(x^k) \Vert _- \le \sum _{k=1}^\infty \Vert x^{k+1}-x^k\Vert +\sum _{k=1}^\infty \varepsilon _{k+1}<\infty$, we obtain $\liminf _{k\rightarrow \infty }\Vert \partial f(x^k) \Vert _- =0$. Finally, observe that that $\lim _{k\rightarrow \infty }f(x^k)$ exists because $f(x^k)$ is decreasing, and bounded from below by $f(x^*)$; and the lower semi-continuity of $f$ implies $f(\overline{x})\le \lim _{k\rightarrow \infty }f(x^k)$. If $\lim _{k\rightarrow \infty }f(x^k)=\beta >f(x^*)$, the KŁ inequality and the fact that $\varphi '$ is decreasing imply $\varphi '(\beta -f(x^*))\Vert \partial f(x^k) \Vert _- \ge \varphi '(f(x^k)-f(x^*))\Vert \partial f(x^k) \Vert _- \ge 1$ for all $k\in \mathbb {N}$, which is impossible because $\liminf _{k\rightarrow \infty }\Vert \partial f(x^k) \Vert _- =0$, whence $\beta = f(x^*)$. $\square$

We are now in position to complete the proofs of Theorems 3.1 and 3.2.

Proof of Theorem 3.1

Let $x^{n_k}\rightarrow x^*$ with $f(x^{n_k})\rightarrow f(x^*)$ as $k\rightarrow \infty$. Since $f(x^k)$ is nonincreasing and admits a limit point, we deduce that $f(x^k)\downarrow f(x^*)$. In particular, we have $f(x^*)\le f(x^k)$ for all $k\in \mathbb {N}$. The function $f$ satisfies the KŁ inequality on $\Gamma _\eta (x^*,\delta )$ with desingularising function $\varphi$. Let $K_0\in \mathbb {N}$ be sufficiently large so that $f(x^{K})-f(x^*)<\min \{\eta ,\underline{a}\delta ^2\}$, and pick $\rho >0$ such that $f(x^K)-f(x^*)<\underline{a}(\delta -\rho )^2$.

Hence, $f(x^*)\le f(x^{k+1})<f(x^*)+\eta$ for all $k\ge K$ and

$$\begin{aligned} \Vert x^{k+1}-x^k\Vert \le \sqrt{\frac{f(x^k)-f(x^{k+1})}{a_k}}\le \sqrt{\frac{f(x^K)-f(x^*)}{\underline{a}}}< \delta -\rho , \end{aligned}$$

which implies part (i) of $\mathbf {S}(x^*,\delta ,\rho )$. Now take $K\ge K_0$ such that

$$\begin{aligned} \Vert x^*-x^{K}\Vert +2\sqrt{\frac{f(x^{K})-f(x^*)}{a_{n_K}}}+M\varphi (f(x^{K})-f(x^*))+\sum _{k=K+1}^{+\infty }\epsilon _{k}<\rho . \end{aligned}$$

The sequence $(y^k)_{k\in \mathbb {N}}$, defined by $y^k:=x^{K+k}$ for all $k\in \mathbb {N}$, satisfies the hypotheses of Proposition 7.1. Finally, since the whole sequence $(y^k)_{k\in \mathbb {N}}$ is $f$-convergent toward $x^*$ and $\liminf \limits _{k\rightarrow \infty }\Vert \partial f(y^k) \Vert _-=0$, we conclude that $x^*$ must be critical using Lemma 2.1. $\square$

Proof of Theorem 3.2

Since $f$ has the KŁ property in $x^*$, there is a strict local upper level set $\Gamma _{\eta }(x^*,\delta )$, where the KŁ inequality holds with $\varphi$ as a desingularising function. Take $\rho <\frac{3}{4}\delta$ and then $\gamma <\frac{1}{3}\rho$. If necessary, shrink $\eta$ so that $2\sqrt{\dfrac{\eta }{\underline{a}}} + M \varphi (\eta ) < \dfrac{2\rho }{3}$. This is possible since $\varphi$ is continuous in $0$ with $\varphi (0)=0$. Let $x^0\in \underline{\Gamma }_\eta (x^*,\gamma )\subset \underline{\Gamma }_\eta (x^*,\rho )$. It suffices to verify that $\mathbf {S}(x^*,\delta ,\rho )$ is fulfilled and use Proposition 7.1. For (i), let us suppose that $x^0,\dots ,x^k$ lie in $\underline{\Gamma }_\eta (x^*,\rho )$ and prove that $x^{k+1} \in \underline{\Gamma }_\eta (x^*,\delta )$. Since $x^*$ is a global minimum, from $\mathbf {H}_1$ and the fact that $\left( f(x^k)\right) _{k\in \mathbb {N}}$ is decreasing, we have

$$\begin{aligned} f(x^*)\!+\!\underline{a} \Vert x^{k+1}\!-\!x^k \Vert ^2\!\le \! f(x^{k+1})\!+\!\underline{a} \Vert x^{k+1}-x^k \Vert ^2\le f(x^k)\!\le \! f(x^0)< f(x^*) + \eta . \end{aligned}$$

It follows that $\Vert x^{k+1}-x^*\Vert \le \Vert x^{k+1}-x^k\Vert +\Vert x^k-x^*\Vert <\sqrt{\frac{\eta }{\underline{a}}}+\rho <\frac{4}{3}\rho <\delta$, and so $x^{k+1}\in \underline{\Gamma }_\eta (x^*,\delta )$. Finally,

$$\begin{aligned} \Vert x^0-x^*\Vert \!+\!2\sqrt{\frac{f(x^0)\!-\!f(x^*)}{a_0}}\!+\!M\varphi (f(x^0)-f(x^*))<\frac{1}{3}\rho +2\sqrt{\dfrac{\eta }{\underline{a}}} + M \varphi (\eta ) \!<\!\rho , \end{aligned}$$

which is precisely (ii). $\square$

1.2 Proof of Proposition 4.1

Proof

Since $X_i^k=Y_i^k +S_i^k$, we can rewrite the algorithm as

$$\begin{aligned} y_i^{k+1} \in \text{ prox }^{A_{i,k}}_{g_i} (y_i^k - A_{i,k}^{-1} \nabla _i h(Y_i^k + S_i^k) + r_i^k + s_i^k). \end{aligned}$$

(26)

We start by showing that $\mathbf {H}_1$ is satisfied. Let $i=1..p$ be fixed; using the definition of the proximal operator $\text{ prox }_{g_i}^{A_{i,k}}$ in (26) and developing the squared norms gives

$$\begin{aligned}&g_i (y_{i}^k) - g_i (y_{i}^{k+1}) \ge \frac{1}{2} \Vert y_i^{k+1} - y_i^k\Vert _{A_{i,k}}^2 + \langle y_i^{k+1} - y_i^k, \nabla _i h(Y_i^k + S_i^k)\rangle \nonumber \\&\quad \quad - \langle y_i^{k+1} - y_i^{k}, r_i^k +s_i^k\rangle _{A_{i,k}}. \end{aligned}$$

(27)

Using $\mathbf {HE}_3$ in (27), the latter results in

$$\begin{aligned} g_i (y_{i}^k) - g_i (y_{i}^{k+1}) \ge \frac{1}{2} \Vert y_i^{k+1} - y_i^k\Vert _{\rho A_{i,k}}^2 + \langle y_i^{k+1} - y_i^k, \nabla _i h(Y_i^k + S_i^k)\rangle . \end{aligned}$$

(28)

We introduce $\tilde{h}_{i,k} : y_i \in H_i \mapsto (y_1^{k+1},..,y_{i-1}^{k+1},y_i,y_{i+1}^k,..,y_p^k) \in \mathbb {R}$ for fixed $k\in \mathbb {N}$ and $i=1,\dots ,p$. It satisfies $\tilde{h}_{i,k} (y_i^k) = h(Y_i^k)$, $\tilde{h}_{i,k} (y_i^{k+1}) = h(Y_{i+1}^k)$ and $\nabla \tilde{h}_{i,k} (y_i^k) = \nabla _i h (Y_i^{k})$. Applying the descent lemma to $\tilde{h}_{i,k}$, we obtain

$$\begin{aligned} h(Y_{i+1}^k) - h(Y_i^k) - \langle y_i^{k+1} - y_i^k , \nabla _i h(Y_i^k) \rangle \le \frac{L}{2} \Vert y_i^{k+1} - y_i^k \Vert ^2. \end{aligned}$$

(29)

Then, combining (28) and (29), we get

$$\begin{aligned}&\frac{1}{2} \Vert y_i^{k+1} - y_i^k\Vert _{\rho A_{i,k} - Lid_{H_i}}^2 + \langle y_i^{k+1} - y_i^k, \nabla _i h(Y_i^k + S_i^k) - \nabla _i h(Y_i^k) \rangle \nonumber \\&\quad \le g_i (y_{i}^k) - g_i (y_{i}^{k+1}) + h(Y_i^k) - h(Y_{i+1}^k), \end{aligned}$$

(30)

where $\rho A_{i,k} - Lid_{H_i}$ remains coercive, since $\rho \alpha _k > L$. Using successively the Cauchy–Schwartz inequality, the Lipschitz property of $\nabla _i h$ (see Remark 4.2) and $\mathbf {HE}_1$, one gets

$$\begin{aligned} \langle y_i^{k+1} - y_i^k, \nabla _i h(Y_i^k + S_i^k) - \nabla _i h(Y_i^k) \rangle \ge - \frac{\sigma L}{2\sqrt{p}} \Vert y_i^{k+1} - y_i^k \Vert ^2. \end{aligned}$$

By inserting this estimation in (30), we deduce that

$$\begin{aligned} g_i (y_{i}^k) - g_i (y_{i}^{k+1}) + h(Y_i^k) - h(Y_{i+1}^k) \ge \frac{1}{2} \Vert y_i^{k+1} - y_i^k\Vert _{\rho A_{i,k} - L(\sigma \frac{\sqrt{p}}{p} +1)id_{H_i}}^2. \end{aligned}$$

(31)

We deduce that $\mathbf {H}_1$ is fulfilled with $a_k=\frac{\rho \alpha _k - L(\sigma \frac{\sqrt{p}}{p} +1)}{2}$, by summing (31) over $i$:

$$\begin{aligned} f(Y^k) - f(Y^{k+1})&= \sum \limits _{i=1}^{p} g_i(y_i^k) -g_i(y_i^{k+1}) + h(Y_i^k) - h(Y_{i+1}^k) \\&\ge \frac{1}{2} \sum \limits _{i=1}^{p} \Vert y_i^{k+1} - y_i^k\Vert _{\rho A_{i,k} - L({\sigma }\frac{\sqrt{p}}{p} +1)id_{H_i}}^2. \end{aligned}$$

To prove $\mathbf {H}_2$, fix $i=1,\dots ,p$ and use Fermat’s first-order condition in (26) to get

$$\begin{aligned} 0 \in \partial g_i(y_i^{k+1}) + \left\{ A_{i,k}(y_i^{k+1} - y_i^k) - A_{i,k}(r_i^k + s_i^k) + \nabla _i h(Y_i^k + S_i^k) \right\} . \end{aligned}$$

(32)

Define $w_i^{k+1} := \nabla _i h(Y^k) - \nabla _i h(Y_i^k + S_i^k) - A_{i,k}(y_i^{k+1} - y_i^k) + A_{i,k}(r_i^k + s_i^k)$, which lies in $\partial g_i(y_i^{k+1}) + \nabla _i h(Y^{k+1})$, by (32). The triangle inequality gives

$$\begin{aligned} \Vert w_i^{k+1} \Vert \le \ \beta _k \left( \Vert y_i^{k+1} - y_i^k \Vert + \Vert r_i^k \Vert + \Vert s_i^k \Vert \right) + \Vert \nabla _i h(Y_i^k + S_i^k) - \nabla _i h(Y^{k+1}) \Vert .\nonumber \\ \end{aligned}$$

(33)

Using the error estimations from (HE), and the $\sqrt{p} L$-Lipschitz continuity of $\nabla _i h$ in (33), we get

$$\begin{aligned} \Vert w_i^{k+1} \Vert \le \ (\beta _k (1 + \sigma ) + \sqrt{p}L\sigma ) \Vert y_i^{k+1} - y_i^k \Vert + \sqrt{p}L \Vert Y^{k+1} - Y^k \Vert + \beta _k \mu _k.\nonumber \\ \end{aligned}$$

(34)

Define now $W^{k+1} := (w_1^{k+1},\ldots ,w_p^{k+1}) \in \partial f (Y^{k+1})$ (recall the definition of $w_i^{k+1}$). Then, by summing (34) over $i=1 \ldots p$, we have (using $\sqrt{p} \le p \le p^2$)

$$\begin{aligned} \Vert W^{k+1} \Vert \ \le \ \sum \limits _{i=1}^{p} \Vert w_i^{k+1} \Vert \ \le \ p\beta _k \mu _k + p^2 (\beta _k+L) (1 + \sigma )\Vert Y^{k+1} - Y^k \Vert . \end{aligned}$$

Hence, $\mathbf {H}_2$ is verified with $b_{k+1}=\frac{1}{p^2 (1+\sigma )(\beta _k +L)}$ and $\epsilon _{k+1}= \frac{\beta _k \mu _k}{p(1+\sigma )(\beta _k +L)}$.

Now we just need to check that the hypotheses $\mathbf {H}_3$ are satisfied with our hypotheses on $\alpha _k$, $\beta _k$ and $\mu _k$. Clearly, $\mathbf {H}_3(i)$ holds since we have supposed that $\alpha _k \ge \underline{\alpha } > (\sigma \frac{\sqrt{p}}{p} +1) \frac{L}{\rho }$. Then, $\mathbf {H}_3(ii)$ asks that $b_k \notin \ell ^1$, which is equivalent to $\frac{1}{\beta _k +L} \notin \ell ^1$ in our context. This holds since we have supposed that $\frac{1}{\beta _k} \notin \ell ^1$. Hypothesis $\mathbf {H}_3(iii)$ is satisfied because $\frac{\beta _k}{\alpha _{k+1}}$ is supposed to be bounded. Finally, $\mathbf {H}_3(iv)$ asks the summability of $\frac{\beta _k \mu _k}{\beta _k +L}$ which is bounded by $\mu _k \in \ell ^1$. $\square$