A Bregman-Style Improved ADMM and its Linearized Version in the Nonconvex Setting: Convergence and Rate Analyses

Abstract

This work explores a family of two-block nonconvex optimization problems subject to linear constraints. We first introduce a simple but universal Bregman-style improved alternating direction method of multipliers (ADMM) based on the iteration framework of ADMM and the Bregman distance. Then, we utilize the smooth performance of one of the components to develop a linearized version of it. Compared to the traditional ADMM, both proposed methods integrate a convex combination strategy into the multiplier update step. For each proposed method, we demonstrate the convergence of the entire iteration sequence to a unique critical point of the augmented Lagrangian function utilizing the powerful Kurdyka–Łojasiewicz property, and we also derive convergence rates for both the sequence of merit function values and the iteration sequence. Finally, some numerical results show that the proposed methods are effective and encouraging for the Lasso model.

Appendix

Proof

(i) Firstly, from the definition of $\mathcal {L_\beta (\cdot )}$ and (5c), one has

$$\begin{aligned} \mathcal {L}_\beta (w^{k+1})-\mathcal {L}_\beta (x^{k+1},y^{k+1},\lambda ^{k})= & {} (\lambda ^{k}-\lambda ^{k+1})^{\top } (Ax^{k+1}+y^{k+1}-b)\nonumber \\= & {} \frac{1}{ \gamma \beta + (1- \gamma ) \alpha }\Vert \lambda ^{k+1}-\lambda ^{k}\Vert ^{2}. \end{aligned}$$

(45)

Secondly, the expression $\mathcal {L}_{\beta }(\cdot )$ (2) provides us with

$$\begin{aligned}{} & {} \quad \mathcal {L}_{\beta }(x^{k+1}, y^{k+1},\lambda ^{k})-\mathcal {L}_{\beta }(x^{k+1}, y^{k},\lambda ^{k}) \nonumber \\{} & {} = g(y^{k+1})- g(y^{k}) - \langle \lambda ^{k}, y^{k+1}- y^{k}\rangle +\frac{\beta }{2} \Vert A x^{k+1}+ y^{k+1}-b\Vert ^{2} \nonumber \\{} & {} \quad - \frac{\beta }{2} \Vert A x^{k+1}+y^{k}-b\Vert ^{2} \nonumber \\{} & {} = g(y^{k+1})- g(y^{k}) - \langle \lambda ^{k} - \beta (A x^{k+1}+ y^{k+1}-b), y^{k+1}- y^{k}\rangle \nonumber \\{} & {} \quad - \frac{\beta }{2} \Vert y^{k+1}-y^{k}\Vert ^{2}. \end{aligned}$$

(46)

On the other hand, by the $l_{g}-$Lipschitz continuity of $\nabla g$ and Lemma 1, we have

$$\begin{aligned} g(y^{k+1})-g(y^{k})- \nabla g(y^{k})^{\top } (y^{k+1}-y^{k}) \leqslant \frac{l_{g}}{2} \Vert y^{k+1}-y^{k}\Vert ^{2}. \end{aligned}$$

(47)

Therefore, by combining the above relations (46) and (47), along with (39b) and (5c), we conclude that

$$\begin{aligned}{} & {} \quad \mathcal {L}_{\beta }(x^{k+1}, y^{k+1},\lambda ^{k})-\mathcal {L}_{\beta }(x^{k+1}, y^{k},\lambda ^{k}) \nonumber \\{} & {} \leqslant - \frac{\beta - l_{g}}{2} \Vert y^{k+1}-y^{k}\Vert ^{2} + \langle \nabla g(y^{k}) -\lambda ^{k} + \beta (A x^{k+1}+ y^{k+1}-b), y^{k+1}- y^{k}\rangle \nonumber \\{} & {} = - \frac{\beta - l_{g}}{2} \Vert y^{k+1}-y^{k}\Vert ^{2} + \langle (\beta -\alpha ) (A x^{k+1}+ y^{k+1}-b) \nonumber \\{} & {} \quad - \tau _k (y^{k+1}-y^{k}), y^{k+1}- y^{k}\rangle \nonumber \\{} & {} = - \frac{\beta - l_{g}}{2} \Vert y^{k+1}-y^{k}\Vert ^{2} + \frac{\alpha -\beta }{ \gamma \beta + (1- \gamma ) \alpha } \langle \lambda ^{k+1}- \lambda ^{k}, y^{k+1}- y^{k}\rangle \nonumber \\{} & {} \quad - \tau _k \Vert y^{k+1}-y^{k}\Vert ^2 \nonumber \\{} & {} \leqslant - \left( \frac{ 2\tau _k+ \beta - l_{g}}{2} - \frac{|\alpha -\beta |}{2 [\gamma \beta + (1- \gamma ) \alpha ]}\right) \Vert y^{k+1}-y^{k}\Vert ^{2} \nonumber \\{} & {} \quad + \frac{|\alpha -\beta |}{2 [\gamma \beta + (1- \gamma ) \alpha ]} \Vert \lambda ^{k+1}- \lambda ^{k}\Vert ^{2}. \end{aligned}$$

(48)

Thirdly, since $x^{k+1}$ is a minimizer of the $x-$subproblem (5a), then using the strong convexity of $\triangle _{\psi }$, we obtain

$$\begin{aligned} \mathcal {L}_\beta (x^{k+1},y^{k},\lambda ^{k})-\mathcal {L}_\beta (w^{k})\leqslant -\triangle _{\psi }(x^{k+1},x^{k}) \leqslant -\frac{\sigma _{\psi }}{2}\Vert x^{k+1}-x^{k}\Vert ^{2}. \end{aligned}$$

(49)

Summing up relations (45), (48) and (49), it follows that

$$\begin{aligned}{} & {} \quad \mathcal {L}_\beta (w^{k+1})-\mathcal {L}_\beta (w^{k})\nonumber \\{} & {} \leqslant -\frac{\sigma _{\psi }}{2}\Vert x^{k+1}-x^{k}\Vert ^{2}-\left( \frac{2\tau _k+ \beta - l_{g}}{2} - \frac{|\alpha -\beta |}{2 [\gamma \beta + (1- \gamma ) \alpha ]}\right) \Vert y^{k+1}-y^{k}\Vert ^{2} \nonumber \\{} & {} \quad +\frac{2 + |\alpha -\beta |}{2 [\gamma \beta + (1- \gamma ) \alpha ]}\Vert \lambda ^{k+1}-\lambda ^{k}\Vert ^{2}. \end{aligned}$$

(50)

By recalling (5c) and the optimality condition (39b), we have

$$\begin{aligned} \lambda ^{k+1}=\nabla g(y^{k}) + \tau _k (y^{k+1}-y^{k}) + \gamma (\alpha - \beta ) (Ax^{k+1}+y^{k+1}-b), \end{aligned}$$

(51)

and using Assumption 2 (i), one has

$$\begin{aligned}{} & {} \quad \Vert \lambda ^{k+1}-\lambda ^{k}\Vert \nonumber \\{} & {} \leqslant (\tau _k + \gamma |\alpha - \beta |) \Vert y^{k+1}-y^{k}\Vert + (\tau _k+ l_{g}) \Vert y^{k}-y^{k-1}\Vert \nonumber \\{} & {} \quad + \gamma |\alpha - \beta | \Vert A(x^{k+1} - x^{k})\Vert , \end{aligned}$$

(52)

which further implies

$$\begin{aligned}{} & {} \quad \Vert \lambda ^{k+1}-\lambda ^{k}\Vert ^2 \\{} & {} \leqslant 3 (\tau _k + \gamma |\alpha - \beta |)^2 \Vert y^{k+1}-y^{k}\Vert ^2 + 3(\tau _k+ l_{g})^2 \Vert y^{k}-y^{k-1}\Vert ^2 \\{} & {} \quad + 3\gamma ^2 (\alpha - \beta )^2 \lambda _{\max }(A^{\top }A) \Vert x^{k+1} - x^{k}\Vert ^2. \end{aligned}$$

Finally, from the relation above, (50) and $0 < \tau _1 \leqslant \tau _k \leqslant \tau _2$, we obtain

$$\begin{aligned}{} & {} \quad \mathcal {L}_\beta (w^{k+1})-\mathcal {L}_\beta (w^{k})\\{} & {} \leqslant -\left( \frac{\sigma _{\psi }}{2} - \gamma ^2 (\alpha - \beta )^2 \lambda _{\max }(A^{\top }A) \cdot \frac{6 + 3 |\alpha -\beta |}{2 [\gamma \beta + (1- \gamma ) \alpha ]} \right) \Vert x^{k+1}-x^{k}\Vert ^{2} \\{} & {} \quad - \left( \frac{2\tau _1+ \beta - l_{g}}{2} - \frac{|\alpha -\beta |}{2 [\gamma \beta + (1- \gamma ) \alpha ]} - (\tau _2 + \gamma |\alpha - \beta |)^2 \cdot \right. \\{} & {} \quad \left. \frac{6 + 3 |\alpha -\beta |}{2 [\gamma \beta + (1- \gamma ) \alpha ]} \right) \Vert y^{k+1}-y^{k}\Vert ^{2} \\{} & {} \quad +(\tau _2+ l_{g})^2 \cdot \frac{6 + 3 |\alpha -\beta |}{2 [\gamma \beta + (1- \gamma ) \alpha ]} \Vert y^{k}-y^{k-1}\Vert ^{2}, \end{aligned}$$

and letting $\varrho =(\tau _2+ l_{g})^2 \cdot \frac{6 + 3 |\alpha -\beta |}{2 [\gamma \beta + (1- \gamma ) \alpha ]}$, then

$$\begin{aligned}{} & {} \quad \left[ \mathcal {L}_\beta (w^{k+1})+\varrho \Vert y^{k+1}-y^{k}\Vert ^{2}\right] -\left[ \mathcal {L}_\beta (w^{k})+\varrho \Vert y^{k}-y^{k-1}\Vert ^{2}\right] \\{} & {} \leqslant -\left( \frac{\sigma _{\psi }}{2} - \gamma ^2 (\alpha - \beta )^2 \lambda _{\max }(A^{\top }A) \cdot \frac{6 + 3 |\alpha -\beta |}{2 [\gamma \beta + (1- \gamma ) \alpha ]} \right) \Vert x^{k+1}-x^{k}\Vert ^{2} \\{} & {} \quad - \left( \frac{ 2\tau _1+ \beta - l_{g}}{2} - \frac{|\alpha -\beta |}{2 [\gamma \beta + (1- \gamma ) \alpha ]} - (\tau _2 + \gamma |\alpha - \beta |)^2 \cdot \right. \\{} & {} \quad \left. \frac{6 + 3 |\alpha -\beta |}{2 [\gamma \beta + (1- \gamma ) \alpha ]} \right) \Vert y^{k+1}-y^{k}\Vert ^{2} \\{} & {} \quad +(\tau _2+ l_{g})^2 \cdot \frac{6 + 3 |\alpha -\beta |}{2 [\gamma \beta + (1- \gamma ) \alpha ]} \Vert y^{k+1}-y^{k}\Vert ^{2}. \end{aligned}$$

Hence,

$$\begin{aligned} \hat{\mathcal {L}}_\beta (\hat{w}^{k+1})\leqslant \hat{\mathcal {L}}_\beta (\hat{w}^{k})-\delta (\Vert x^{k+1}-x^{k}\Vert ^2+\Vert y^{k+1}-y^{k}\Vert ^2). \end{aligned}$$

From Assumption 2 (ii) and (iii), we have $\delta >0$. This together with relation above further indicates that $\{\hat{\mathcal {L}}_\beta (\hat{w}^{k})\}$ is monotonically decreasing.

(ii) Since the cluster point set of $\{w^{k}\}$ is nonempty, we obtain that the sequence $\{\hat{w}^{k}\}$ has at least one cluster point. Let $\hat{w}^{*}$ be a cluster point of {$\hat{w}^{k}$} and let $\{\hat{w}^{k_{j}}\}$ be the subsequence converging to it, i.e., $\lim \limits _{j \rightarrow +\infty }\hat{w}^{k_{j}}= \hat{w}^{*}.$ To proceed, similar to the proof of Lemma 4 (ii), the desired conclusion holds.

Proof

• (i) By the definitions of $\varOmega$ and $\hat{\varOmega }$, they are trivial.

• (ii) Combining Lemma 5 (ii) and the definition of $w^{k}$ and $\hat{w}^{k}$, we easily get the result.

• (iii) Let $w^{*}\in \varOmega$. Then, there exists a subsequence $\{w^{k_{j}}\}$ of $\{w^{k}\}$ converging to $w^{*}$. Lemma 5 (ii) implies that $\lim \limits _{j \rightarrow +\infty }w^{k_{j}+1} = \lim \limits _{j \rightarrow +\infty }w^{k_{j}} =w^*$. By taking limit $j \rightarrow +\infty$ in (5c), we have $\lambda ^{*}=\lambda ^{*}- [\gamma \beta + (1- \gamma ) \alpha ](Ax^*+y^*-b)$ with $[\gamma \beta + (1- \gamma ) \alpha ] >0$, which implies

  $$\begin{aligned} Ax^*+y^*-b=0. \end{aligned}$$

  (53)

  Thus, $(x^{*}, y^{*})$ is a feasible point of (1). In view of $x^{k_{j}+1}$ is a minimizer of the $x-$subproblem (5a), it holds that

  $$\begin{aligned}{} & {} f(x^{k_{j}+1})-\left\langle \lambda ^{k_{j}}, A x^{k_{j}+1}\right\rangle +\frac{\beta }{2}\left\| A x^{k_{j}+1}+y^{k_{j}}-b\right\| ^{2}+\triangle _{\psi }(x^{k_{j}+1},x^{k_{j}}) \\{} & {} \quad \leqslant f(x^{*})-\left\langle \lambda ^{k_{j}}, A x^{*}\right\rangle +\frac{\beta }{2}\left\| A x^{*}+y^{k_{j}}-b\right\| ^{2}+\triangle _{\psi }(x^{*},x^{k_{j}}). \end{aligned}$$

  It is equivalent to

  $$\begin{aligned} \mathcal {L}_\beta (x^{k_{j}+1},y^{k_{j}},\lambda ^{k_{j}})+\triangle _{\psi }(x^{k_{j}+1},x^{k_{j}}) \leqslant \mathcal {L}_\beta (x^{*},y^{k_{j}},\lambda ^{k_{j}})+\triangle _{\psi }(x^{*},x^{k_{j}}), \end{aligned}$$

  which implies

  $$\begin{aligned}{} & {} \quad \hat{\mathcal {L}}_\beta (x^{k_{j}+1},y^{k_{j}},\lambda ^{k_{j}},y^{k_{j}-1})- \varrho \Vert y^{k_{j}}-y^{k_{j}-1}\Vert ^2\\{} & {} \leqslant \hat{\mathcal {L}}_\beta (x^{*},y^{k_{j}},\lambda ^{k_{j}},y^{k_{j}-1})-\triangle _{\psi }(x^{k_{j}+1},x^{k_{j}})+\triangle _{\psi }(x^{*},x^{k_{j}})\\{} & {} \quad -\varrho \Vert y^{k_{j}}-y^{k_{j}-1}\Vert ^2\\{} & {} \leqslant \hat{\mathcal {L}}_\beta (x^{*},y^{k_{j}},\lambda ^{k_{j}},y^{k_{j}-1})+\triangle _{\psi }(x^{*},x^{k_{j}})-\varrho \Vert y^{k_{j}}-y^{k_{j}-1}\Vert ^2. \end{aligned}$$

  Taking limit along the convergent subsequence and using the continuity of ${\hat{\mathcal {L}}_\beta (\cdot )}$ with respect to $(y, \lambda , \hat{y}),$ we have

  $$\begin{aligned} \limsup _{j \rightarrow +\infty }\hat{\mathcal {L}}_\beta (\hat{w}^{k_{j}+1})=\limsup _{j\rightarrow +\infty }\hat{\mathcal {L}}_\beta (x^{k_{j}+1},y^{k_{j}},\lambda ^{k_{j}},y^{k_{j}-1})\leqslant \hat{\mathcal {L}}_\beta (\hat{w}^{*}). \end{aligned}$$

  This, together with lower semicontinuity of ${\mathcal {\hat{L}}_\beta (\cdot )}$, yields that $\lim \limits _{j \rightarrow +\infty }\hat{\mathcal {L}}_\beta (\hat{w}^{k_{j}+1})$ $=\hat{\mathcal {L}}_\beta (\hat{w}^{*}).$ Along with the monotonity of $\{\hat{\mathcal {L}}_\beta (\hat{w}^{k})\}$, further shows that the whole sequence $\{\hat{\mathcal {L}}_\beta (\hat{w}^{k})\}$ is convergent. Therefore, in views of $\hat{\mathcal {L}}_\beta (\hat{w}^{k}) \leqslant \hat{\mathcal {L}}_\beta (\hat{w}^{0})< +\infty$, we have

  $$\begin{aligned} +\infty > \hat{\mathcal {L}}_\beta (\hat{w}^{0}) \geqslant \lim \limits _{k\rightarrow +\infty }\hat{\mathcal {L}}_\beta (\hat{w}^{k})=\inf \limits _{k}\hat{\mathcal {L}}_\beta (\hat{w}^{k})= \hat{\mathcal {L}}_\beta (\hat{w}^{*}). \end{aligned}$$

  Hence, $\hat{\mathcal {L}}_{\beta }(\hat{w}^{*}) \equiv \lim \limits _{k\rightarrow + \infty } \hat{\mathcal {L}}_{\beta }(\hat{w}^{k}) < +\infty$ for all $\hat{w}^{*} \in \hat{\varOmega }$.

• (iv) Similar to the proof analysis of Theorem 1 (iii), the desired result holds.

Proof

From Theorem 5 (iii), it follows that $\lim \limits _{k\rightarrow +\infty }\hat{\mathcal {L}}_\beta (\hat{w}^{k})$ $= \hat{\mathcal {L}}_\beta (\hat{w}^{*})$ for all $\hat{w}^{*}\in \hat{\varOmega }$. We consider two cases as follows:

• (A) If there exists an integer $k_{0}$ such that $\hat{\mathcal {L}}_\beta (\hat{w}^{k_{0}})= \hat{\mathcal {L}}_\beta (\hat{w}^{*})$. From Lemma 5 (i), for any $k \geqslant k_{0}$, we have

  $$\begin{aligned}{} & {} \delta (\Vert x^{k+1}-x^{k}\Vert ^{2}+\Vert y^{k+1}-y^{k}\Vert ^{2})\leqslant \hat{\mathcal {L}}_\beta (\hat{w}^{k})- \hat{\mathcal {L}}_\beta (\hat{w}^{k+1})\leqslant \hat{\mathcal {L}}_\beta (\hat{w}^{k_{0}})\\{} & {} \quad - \hat{\mathcal {L}}_\beta (\hat{w}^{*})=0, \end{aligned}$$

  thus, $x^{k+1}=x^{k}$ and $y^{k+1}=y^{k}$ for all $k \geqslant k_{0}$. Associated with (52), for any $k \geqslant k_{0}+1$, it follows that $w^{k+1}=w^{k}$ and the assertion holds.

• (B) Let us assume $\hat{\mathcal {L}}_\beta (\hat{w}^{k})> \hat{\mathcal {L}}_\beta (\hat{w}^{*})$ for all k. From $\lim \limits _{k\rightarrow +\infty }d(\hat{w}^{k},\hat{\varOmega })=0$, we infer that for $\varepsilon >0$, there exists $k_{1}>0$, such that $d(\hat{w}^{k},\hat{\varOmega })<\varepsilon$ for any $k>k_{1}$. Again, it follows from $\lim \limits _{k\rightarrow +\infty }\hat{\mathcal {L}}_\beta (\hat{w}^{k})= \hat{\mathcal {L}}_\beta (\hat{w}^{*})$ that for $\eta >0$, there exists $k_{2}>0$, such that $\hat{\mathcal {L}}_\beta (\hat{w}^{k})< \hat{\mathcal {L}}_\beta (\hat{w}^{*})+\eta$ for all $k>k_{2}$. Consequently, for $\varepsilon ,\eta >0$, when $k>\tilde{k}=\max \{k_{1},k_{2}\},$ we have

  $$\begin{aligned} d(\hat{w}^{k},\hat{\varOmega })<\varepsilon ,~~\hat{\mathcal {L}}_\beta (\hat{w}^{*})<\hat{\mathcal {L}}_\beta (\hat{w}^{k})< \hat{\mathcal {L}}_\beta (\hat{w}^{*})+\eta . \end{aligned}$$

  From Theorem 5, we know that $\hat{\mathcal {L}}_\beta (\cdot )$ is constant on $\hat{\varOmega }$, and $\hat{\varOmega }$ is a nonempty compact set. Thus, it follows from Lemma 3 that $\mathcal {\hat{L}}_\beta (\cdot )$ satisfies the uniformized KŁ property, further implies

  $$\begin{aligned} \varphi '(\hat{\mathcal {L}}_\beta (\hat{w}^{k})-\hat{\mathcal {L}}_\beta (\hat{w}^{*}))d(0,\partial \hat{\mathcal {L}}_\beta (\hat{w}^{k}))\geqslant 1,\ \forall \ k>\tilde{k}. \end{aligned}$$

  (54)

Next, we attempt to provide upper estimate in terms of the iterates for limiting subdifferential of $\hat{\mathcal {L}}_\beta (\hat{w}^{k})$. Taking the subdifferential of $\hat{\mathcal {L}}_\beta (\cdot )$ at $\hat{w}^{k}$ with respect to x, one has

$$\begin{aligned} \partial _{x}\mathcal {\hat{L}}_\beta (\hat{w}^{k})= & {} \partial f(x^{k})-A^{\top } \lambda ^{k}+\beta A^{\top } (Ax^{k}+y^{k}-b) \nonumber \\&\ni&A^{\top } (\lambda ^{k-1}-\lambda ^{k})+\beta A^{\top } (y^{k}-y^{k-1})-[\nabla \psi (x^{k})-\nabla \psi (x^{k-1})],\nonumber \\ \end{aligned}$$

(55)

where the second relation follows from the optimality condition (39a). Similarly, we obtain for y that

$$\begin{aligned} \partial _{y}\mathcal {\hat{L}}_\beta (\hat{w}^{k})= & {} \nabla g(y^{k})-\lambda ^{k}+\beta (Ax^{k}+y^{k}-b) +2 \varrho (y^{k}-y^{k-1}) \nonumber \\= & {} \nabla g(y^{k})-\nabla g(y^{k-1}) + (2 \varrho -\tau _{k})(y^{k}-y^{k-1}) \nonumber \\{} & {} + [\beta - \gamma (\alpha - \beta )] (Ax^{k}+y^{k}-b) \nonumber \\= & {} \nabla g(y^{k})-\nabla g(y^{k-1}) + (2 \varrho -\tau _{k})(y^{k}-y^{k-1}) \nonumber \\{} & {} + \frac{\gamma (\alpha - \beta ) - \beta }{\gamma \beta + (1- \gamma ) \alpha } (\lambda ^{k}-\lambda ^{k-1}), \end{aligned}$$

(56)

where the second equality follows from (51) and the final equality utilizes (5c). In addition,

$$\begin{aligned} \partial _{\lambda }\mathcal {\hat{L}}_\beta (\hat{w}^{k})=-(Ax^{k}+y^{k}-b)= \frac{1}{\gamma \beta + (1- \gamma ) \alpha } (\lambda ^{k}-\lambda ^{k-1}) \end{aligned}$$

(57)

and

$$\begin{aligned} \partial _{\hat{y}}\mathcal {\hat{L}}_\beta (\hat{w}^{k})=-2 \varrho (y^{k}-y^{k-1}). \end{aligned}$$

(58)

In view of $\nabla g$ and $\nabla \psi$ are Lipschitz continuous, then combining with relations (55)-(58), there exists $\zeta _{1}>0$ such that

$$\begin{aligned} d(0,\partial \mathcal {\hat{L}}_\beta (\hat{w}^{k}))\leqslant \zeta _{1}(\Vert x^{k}-x^{k-1}\Vert +\Vert y^{k}-y^{k-1}\Vert +\Vert \lambda ^{k}-\lambda ^{k-1}\Vert ). \end{aligned}$$

On the other hand, from (52), there exists $\zeta _{2}>0$ such that

$$\begin{aligned} \Vert \lambda ^{k}-\lambda ^{k-1}\Vert \leqslant \zeta _{2} (\Vert x^{k} - x^{k-1}\Vert + \Vert y^{k}-y^{k-1}\Vert + \Vert y^{k-1}-y^{k-2}\Vert ). \end{aligned}$$

Let $\zeta = \zeta _{1} (1+ \zeta _{2})$, and from the two inequalities above, we obtain

$$\begin{aligned} d(0,\partial \mathcal {\hat{L}}_\beta (\hat{w}^{k}))\leqslant \zeta (\Vert x^{k}-x^{k-1}\Vert +\Vert y^{k}-y^{k-1}\Vert +\Vert y^{k-1}-y^{k-2}\Vert ). \end{aligned}$$

(59)

Finally, we start to study the convergence of the entire sequence $\{{w}^{k}\}$. Denote $\varTheta _{p,q}=\varphi (\hat{\mathcal {L}}_\beta (\hat{w}^{p})-\hat{\mathcal {L}}_\beta (\hat{w}^{*}))-\varphi (\hat{\mathcal {L}}_\beta (\hat{w}^{q})-\hat{\mathcal {L}}_\beta (\hat{w}^{*})).$ Taking into account $\hat{\mathcal {L}}_\beta (\hat{w}^{k})-\hat{\mathcal {L}}_\beta (\hat{w}^{k+1})=(\hat{\mathcal {L}}_\beta (\hat{w}^{k})-\hat{\mathcal {L}}_\beta (\hat{w}^{*}))-(\hat{\mathcal {L}}_\beta (\hat{w}^{k+1})-\hat{\mathcal {L}}_\beta (\hat{w}^{*}))$ and the concavity of $\varphi$, we get

$$\begin{aligned} \varTheta _{k,k+1} \geqslant \varphi '(\hat{\mathcal {L}}_\beta (\hat{w}^{k})-\hat{\mathcal {L}}_\beta (\hat{w}^{*}))(\mathcal {\hat{L}}_\beta (\hat{w}^{k})-\mathcal {\hat{L}}_\beta (\hat{w}^{k+1})). \end{aligned}$$

From this, (54), (59) and $\varphi '(\hat{\mathcal {L}}_\beta (\hat{w}^{k})-\hat{\mathcal {L}}_\beta (\hat{w}^{*}))>0$, it follows that

$$\begin{aligned} \hat{\mathcal {L}}_\beta (\hat{w}^{k})-\hat{\mathcal {L}}_\beta (\hat{w}^{k+1})\leqslant & {} \frac{1}{\varphi '(\hat{\mathcal {L}}_\beta (\hat{w}^{k})-\hat{\mathcal {L}}_\beta (\hat{w}^{*}))} \cdot \varTheta _{k,k+1}\\\leqslant & {} d(0,\partial \mathcal {\hat{L}}_\beta (\hat{w}^{k})) \cdot \varTheta _{k,k+1} \\\leqslant & {} \zeta (\Vert x^{k}-x^{k-1}\Vert +\Vert y^{k}-y^{k-1}\Vert +\Vert y^{k-1}-y^{k-2}\Vert ) \varTheta _{k,k+1}. \end{aligned}$$

From Lemma 5 (i) and the inequality above, one has

$$\begin{aligned}{} & {} \quad \delta (\Vert x^{k+1}-x^{k}\Vert ^{2}+\Vert y^{k+1}-y^{k}\Vert ^{2})\\{} & {} \leqslant \zeta (\Vert x^{k}-x^{k-1}\Vert +\Vert y^{k}-y^{k-1}\Vert +\Vert y^{k-1}-y^{k-2}\Vert )\varTheta _{k,k+1},\ \forall \ k>\tilde{k}, \end{aligned}$$

and hence

$$\begin{aligned}{} & {} \quad (2\Vert x^{k+1}-x^{k}\Vert ^{2}+2\Vert y^{k+1}-y^{k}\Vert ^{2})^{\frac{1}{2}}\\{} & {} \leqslant (\Vert x^{k}-x^{k-1}\Vert +\Vert y^{k}-y^{k-1}\Vert +\Vert y^{k-1}-y^{k-2}\Vert )^{\frac{1}{2}}\sqrt{\frac{2\zeta }{\delta }\varTheta _{k,k+1}}. \end{aligned}$$

Using inequality $a+b\leqslant \sqrt{2(a^{2}+b^{2})}$, the inequality above further gives that

$$\begin{aligned}{} & {} \quad \Vert x^{k+1}-x^{k}\Vert +\Vert y^{k+1}-y^{k}\Vert \\{} & {} \leqslant (\Vert x^{k}-x^{k-1}\Vert +\Vert y^{k}-y^{k-1}\Vert +\Vert y^{k-1}-y^{k-2}\Vert )^{\frac{1}{2}}\sqrt{\frac{2\zeta }{\delta }\varTheta _{k,k+1}}, \end{aligned}$$

which implies

$$\begin{aligned}{} & {} \quad 4(\Vert x^{i+1}-x^{i}\Vert +\Vert y^{i+1}-y^{i}\Vert )\\{} & {} \leqslant 2(\Vert x^{i}-x^{i-1}\Vert +\Vert y^{i}-y^{i-1}\Vert +\Vert y^{i-1}-y^{i-2}\Vert )^{\frac{1}{2}}\cdot \sqrt{\frac{8\zeta }{\delta }\varTheta _{i,i+1}}\\{} & {} \leqslant \Vert x^{i}-x^{i-1}\Vert +\Vert y^{i}-y^{i-1}\Vert +\Vert y^{i-1}-y^{i-2}\Vert +\frac{8\zeta }{\delta }\varTheta _{i,i+1},\ \forall \ i>\tilde{k}, \end{aligned}$$

where the last inequality from the relation $2\sqrt{ab}\leqslant a+b$ $(a, b \geqslant 0)$. Summing the inequalities above from $i=k(\geqslant \tilde{k}+1)$ to $i= N$, we have

$$\begin{aligned}{} & {} \quad 3\sum \limits _{i=k}^{N}\Vert x^{i+1}-x^{i}\Vert +2\sum \limits _{i=k}^{N}\Vert y^{i+1}-y^{i}\Vert \\{} & {} \leqslant 2\Vert {y}^{k}-{y}^{k-1}\Vert -2\Vert y^{N+1}-y^{N}\Vert +\Vert {x}^{k}-{x}^{k-1}\Vert -\Vert x^{N+1}-x^{N}\Vert \\{} & {} +\Vert {y}^{k-1}-{y}^{k-2}\Vert -\Vert y^{N}-y^{N-1}\Vert +\frac{8\zeta }{\delta }\varTheta _{k,N+1}. \end{aligned}$$

This, along with $\varphi (\hat{\mathcal {L}}_\beta (\hat{w}^{N+1})-\hat{\mathcal {L}}_\beta (\hat{w}^{*}))>0$, further implies

$$\begin{aligned}{} & {} \quad 3\sum \limits _{i=k}^{N}\Vert x^{i+1}-x^{i}\Vert +2\sum \limits _{i=k}^{N}\Vert y^{i+1}-y^{i}\Vert \\{} & {} \leqslant 2\Vert {y}^{k}-{y}^{k-1}\Vert +\Vert {x}^{k}-{x}^{k-1}\Vert +\Vert {y}^{k-1}-{y}^{k-2}\Vert +\frac{8\zeta }{\delta } (\varphi (\hat{\mathcal {L}}_\beta (\hat{w}^{k})-\hat{\mathcal {L}}_\beta (\hat{w}^{*}))). \end{aligned}$$

Thus,

$$\begin{aligned}{} & {} \quad 3\sum \limits _{i=k}^{+\infty }\Vert x^{i+1}-x^{i}\Vert +2\sum \limits _{i=k}^{+\infty }\Vert y^{i+1}-y^{i}\Vert \nonumber \\{} & {} \leqslant 2\Vert {y}^{k}-{y}^{k-1}\Vert +\Vert {x}^{k}-{x}^{k-1}\Vert +\Vert {y}^{k-1}-{y}^{k-2}\Vert +\frac{8\zeta }{\delta } (\varphi (\hat{\mathcal {L}}_\beta (\hat{w}^{k})-\hat{\mathcal {L}}_\beta (\hat{w}^{*}))).\nonumber \\ \end{aligned}$$

(60)

Consider $k=\tilde{k}+1$ in the relation above, one has

$$\begin{aligned}{} & {} \quad 3\sum \limits _{i=\tilde{k}+1}^{+\infty }\Vert x^{i+1}-x^{i}\Vert +2\sum \limits _{i=\tilde{k}+1}^{+\infty }\Vert y^{i+1}-y^{i}\Vert \\{} & {} \leqslant 2\Vert {y}^{\tilde{k}+1}-{y}^{\tilde{k}}\Vert +\Vert {x}^{\tilde{k}+1}-{x}^{\tilde{k}}\Vert +\Vert {y}^{\tilde{k}}-{y}^{\tilde{k}-1}\Vert +\frac{8\zeta }{\delta } (\varphi (\hat{\mathcal {L}}_\beta (\hat{w}^{\tilde{k}+1})-\hat{\mathcal {L}}_\beta (\hat{w}^{*}))). \end{aligned}$$

This immediately shows that $\sum \limits _{k=0}^{+\infty }\Vert x^{k+1}-x^{k}\Vert <+\infty$ and $\sum \limits _{k=0}^{+\infty }\Vert y^{k+1}-y^{k}\Vert <+\infty .$ Further, from this and relation (52), we obtain $\sum \limits _{k=0}^{+\infty }\Vert \lambda ^{k+1}-\lambda ^{k}\Vert <+\infty .$ The remaining proof is similar to that of Theorem 2 and so omitted here.

Proof

From (59) for any $k\geqslant 2$, we have

$$\begin{aligned} \frac{1}{3 \zeta ^2}\Vert \varpi ^{k}\Vert ^2 \leqslant \Vert x^{k}-x^{k-1}\Vert ^2+\Vert y^{k}-y^{k-1}\Vert ^2+\Vert y^{k-1}-y^{k-2}\Vert ^2, \end{aligned}$$

(61)

where $\varpi ^{k}\in \partial \mathcal {\hat{L}}_\beta (\hat{w}^{k}).$ By (41), there exists a $k_{0} \geqslant 2$ such that

$$\begin{aligned} \Vert x^{k}-x^{k-1}\Vert ^2+\Vert y^{k}-y^{k-1}\Vert ^2 +\Vert y^{k-1}-y^{k-2}\Vert ^2 \leqslant \frac{1}{\delta } (e_{k-2}-e_{k}), \ \forall \ k \geqslant k_{0}.\nonumber \\ \end{aligned}$$

(62)

Combining (61) and (62) leads to

$$\begin{aligned} \frac{1}{3 \zeta ^2}\Vert \varpi ^{k}\Vert ^2 \leqslant \frac{1}{\delta } (e_{k-2}-e_{k}). \end{aligned}$$

(63)

Let $\varepsilon >0$ arbitrary small. Since $\hat{w}^{k}$ converges to $\hat{w}^{*}$, there exists a $k_1 \geqslant 0$ such that $d(\hat{w}^k, \hat{w}^{*})< \varepsilon$ for any $k>k_{1}$. By the fact that $\mathcal {\hat{L}}_\beta (\hat{w})$ satisfies the KŁ property at $\hat{w}^{*}$, $\mathcal {\hat{L}}_\beta (\hat{w}^{k})$ monotonically decreasing and $\mathcal {\hat{L}}_\beta (\hat{w}^{k}) \rightarrow \mathcal {\hat{L}}_\beta (\hat{w}^{*})$ as $k \rightarrow + \infty$, then there exist $k_2 \geqslant 0$, KŁ-exponent $\theta \in [0, 1)$ and $c_{l}>0$ such that $(\mathcal {\hat{L}}_\beta (\hat{w}^{k}) - \mathcal {\hat{L}}_\beta (\hat{w}^{*}))^{\theta } \leqslant c_{l} \cdot d(0,\partial \hat{\mathcal {L}}_\beta (\hat{w}^{k}))$ for all $k \geqslant k_2$. This follows that

$$\begin{aligned} e_{k}^{2 \theta } \leqslant c_{l}^{2} \Vert \varpi ^{k}\Vert ^2 \ \ \textrm{with} \ \ \varpi ^{k}\in \partial \mathcal {\hat{L}}_\beta (\hat{w}^{k}), \ \forall \ k \geqslant k_2. \end{aligned}$$

This, together with (63), yields

$$\begin{aligned} \frac{\delta }{3 c_{l}^{2} \zeta ^2} e_{k}^{2 \theta } \leqslant e_{k-2}-e_{k}. \end{aligned}$$

Denote $q = \frac{\delta }{3 c_{l}^{2} \zeta ^2}$, we obtain (42) where $\hat{k} =\max \{k_1, k_2\}$.

• (i) Let $\theta = 0$. If $e_{k} >0$ for $k \geqslant \hat{k}$, we have $\alpha \leqslant e_{k-2}-e_{k}$. As $k \rightarrow +\infty$, the right- hand-side approaches to zero, then $0 <\alpha \leqslant 0$, which leads to a contradiction. Since $e_{k}$ must be equal to zero for $k \geqslant \hat{k}$. Hence, there exists $\tilde{k} \geqslant \hat{k}$ such that $e_{k}=0$ for all $k \geqslant \tilde{k}$.

• (ii) If $\theta \in (0, \frac{1}{2}]$, then $2 \theta -1 < 0$. Let $k \geqslant \hat{k}$ be fixed. In view of $\{e_{i}\}_{i\geqslant \hat{k}}$ is monotonically decreasing, and so $e_{i} \leqslant e_{\hat{k}}$ for $i= \hat{k}, \hat{k}+1, \cdots , k$, and from (42), which implies that

  $$\begin{aligned} q e_{\hat{k}}^{2 \theta -1}e_{k} \leqslant q e_{k}^{2 \theta -1}e_{k} \leqslant e_{k-2}-e_{k}, \ \mathrm{i.e.}, \ e_{k} \leqslant \frac{1}{1+ q e_{\hat{k}}^{2 \theta -1}} e_{k-2}. \end{aligned}$$

  We rearrange this to obtain two cases:

  • (ii-A) If $k-\hat{k}$ is odd, then

    $$\begin{aligned} e_{k} \leqslant \frac{e_{k-2}}{1+ q e_{\hat{k}}^{2 \theta -1}} \leqslant \frac{e_{k-4}}{(1+ q e_{\hat{k}}^{2 \theta -1})^2} \leqslant \cdots \leqslant \frac{e_{\hat{k}-1}}{(1+ q e_{\hat{k}}^{2 \theta -1})^{\frac{k-\hat{k}+1}{2}}}. \end{aligned}$$

  • (ii-B) If $k-\hat{k}$ is even, then

    $$\begin{aligned} e_{k} \leqslant \frac{e_{k-2}}{1+ q e_{\hat{k}}^{2 \theta -1}} \leqslant \frac{e_{k-4}}{(1+ q e_{\hat{k}}^{2 \theta -1})^2} \leqslant \cdots \leqslant \frac{e_{\hat{k}}}{(1+ q e_{\hat{k}}^{2 \theta -1})^{\frac{k-\hat{k}}{2}}}. \end{aligned}$$

    Let $\tau :=\left( \frac{1}{1+ \alpha e_{\hat{k}}^{2 \theta -1}}\right) ^{1/2} \in (0, 1)$. Hence

    $$\begin{aligned} e_{k} \leqslant \frac{\max \{e_{j}: 0 \leqslant j \leqslant \hat{k}\}}{\tau ^{\hat{k}-k}} = \frac{\max \{e_{j}: 0 \leqslant j \leqslant \hat{k}\}}{\tau ^{\hat{k}}} \tau ^{k} = O(\tau ^{k}), \ \forall \ k \geqslant \hat{k}. \end{aligned}$$

• (iii) Let $\theta \in (\frac{1}{2}, 1)$, and then $1- 2\theta < 0$. Rearrange (42) to obtain

  $$\begin{aligned} q \leqslant (e_{k-2}-e_{k}) e_{k}^{-2\theta }, \ \forall \ k \geqslant \hat{k}. \end{aligned}$$

  (64)

  Now, define $h(s) = s^{-2 \theta }$ for $s \in [0, +\infty )$. Clearly, h is monotonically decreasing as $h^{\prime } (s) = -2 \theta s^{-(1+ 2 \theta )} <0$. This further gives $h(e_{k-2}) \leqslant h(e_{k})$ for all $k \geqslant 2$ as $e_{k}$ is monotonically decreasing, and so $h(e_{k-2}) \leqslant h(s), \ s \in [e_{k}, e_{k-2}]$. We consider two cases as follows:

  • (iii-A) If $h(e_{k}) \leqslant 2\,h(e_{k-2})$ for all $k \geqslant \hat{k},$ then, together with (64), we have

    $$\begin{aligned} q \leqslant 2 (e_{k-2}-e_{k}) h(e_{k-2})= & {} 2 h(e_{k-2}) \int ^{e_{k-2}}_{e_{k}} 1 \textrm{d} s \leqslant 2 \int ^{e_{k-2}}_{e_{k}} h(s) \textrm{d} s \\= & {} 2 \int ^{e_{k-2}}_{e_{k}} s^{-2 \theta } \textrm{d} s \\= & {} \frac{2}{1-2 \theta } (e_{k-2}^{1-2\theta } - e_{k}^{1-2\theta }), \end{aligned}$$

    where $1-2\theta < 0$. Rearrange to get

    $$\begin{aligned} 0 < \frac{q (2\theta -1)}{2} \leqslant e_{k}^{1-2\theta } - e_{k-2}^{1-2\theta }. \end{aligned}$$

    Denote $\hat{\mu } = \frac{q (2\theta -1)}{2} >0$ and $\nu = 1- 2 \theta < 0$, one has

    $$\begin{aligned} 0< \hat{\mu } \leqslant e_{k}^{\nu } - e_{k-2}^{\nu }, \ \forall \ k \geqslant \hat{k}. \end{aligned}$$

    (65)

  • (iii-B) Considering the case where $h(e_{k}) \geqslant 2 h(e_{k-2})$, $e_{k}^{-2\theta } \geqslant 2 e_{k-2}^{-2\theta }$. Rearranging this gives $\frac{1}{2} e_{k-2}^{2\theta } \geqslant e_{k}^{2\theta }$, which by raising both sides to the power $\frac{1}{2\theta }$ and setting $\hat{q}= (1/2)^{\frac{1}{2\theta }}$ leads to $\hat{q} e_{k-2} \geqslant e_{k}$. Since $\nu = 1-2 \theta < 0$, we have $\hat{q}^{\nu } e^{\nu }_{k-2} \leqslant e^{\nu }_{k}$, and then $(\hat{q}^{\nu } -1) e^{\nu }_{k-2} \leqslant e^{\nu }_{k} - e^{\nu }_{k-2}$. In view of $\hat{q}^{\nu } -1>0$ and $e_{p} \searrow 0$ as $p \rightarrow + \infty$, there exists a $\bar{\mu } >0$ such that $(\hat{q}^{\nu } -1) e^{\nu }_{k-2} > \bar{\mu }$ for all $k \geqslant \hat{k}$. Therefore, we obtain

    $$\begin{aligned} 0< \bar{\mu } \leqslant e_{k}^{\nu } - e_{k-2}^{\nu }, \ \forall \ k \geqslant \hat{k}. \end{aligned}$$

    (66)

    By selecting $\mu =\min \{\hat{\mu }, \ \bar{\mu }\} >0$ and combining (65) and (66), we conclude that

    $$\begin{aligned} 0< \mu \leqslant e_{k}^{\nu } - e_{k-2}^{\nu }, \ \forall \ k \geqslant \hat{k}. \end{aligned}$$

    Summing inequality above from $\hat{k}$ to some $k (\geqslant \hat{k})$, we have

    $$\begin{aligned} \sum _{i=\hat{k}}^{k} (e_{i}^{\nu } - e_{i-2}^{\nu }) = (e_{k}^{\nu } + e_{k-1}^{\nu }) - (e_{\hat{k}-1}^{\nu } + e_{\hat{k}-2}^{\nu }) \geqslant \mu (k-\hat{k}+1). \end{aligned}$$

    This, along with $e_{k-1} \geqslant e_{k}$ (for all k) and $\nu <0$, implies that

    $$\begin{aligned} \frac{\mu }{2} (k-\hat{k}+1) \leqslant e_{k}^{\nu }- e_{\hat{k}-2}^{\nu } \leqslant e_{k}^{\nu }, \ \forall \ k \geqslant \hat{k}. \end{aligned}$$

    Hence,

    $$\begin{aligned} e_{k}^{\nu } \leqslant \left[ \frac{\mu }{2} (k-\hat{k}+1)\right] ^{1/\nu } = \left[ \frac{\mu }{2} (k-\hat{k}+1)\right] ^{1/(1- 2 \theta )}=O(k^{1/(1- 2 \theta )}). \end{aligned}$$

    So, the claim (iii) is proved, and the proof is complete.

Proof

It follows from Lemma 5 (i) that $\{e_{k}\}_{k \geqslant 0}$ is monotonically decreasing. Then, by utilizing (62) and $a+b+c \leqslant \sqrt{3(a^2 +b^2 +c^2)}$, we can deduce that for all $k \geqslant 2$,

$$\begin{aligned} \Vert x^{k}-x^{k-1}\Vert +\Vert y^{k}-y^{k-1}\Vert +\Vert y^{k-1}-y^{k-2}\Vert \leqslant \sqrt{\frac{3}{\delta } (e_{k-2}-e_{k})} \leqslant \sqrt{\frac{3}{\delta } e_{k-2}}.\nonumber \\ \end{aligned}$$

(67)

On the other hand, we have

$$\begin{aligned} \sum \limits _{i=k}^{N} \Vert x^{i+1}-x^{i}\Vert \geqslant \sum \limits _{i=k}^{N}(\Vert x^{i}-x^{*}\Vert - \Vert x^{i+1}-x^{*}\Vert ) = \Vert x^{k}-x^{*}\Vert - \Vert x^{N+1}-x^{*}\Vert . \end{aligned}$$

Letting $N \rightarrow + \infty$ in the relation above, it implies that

$$\begin{aligned} \sum \limits _{i=k}^{+ \infty } \Vert x^{i+1}-x^{i}\Vert \geqslant \Vert x^{k}-x^{*}\Vert - \lim \limits _{N \rightarrow + \infty } \Vert x^{N+1}-x^{*}\Vert = \Vert x^{k}-x^{*}\Vert . \end{aligned}$$

(68)

Similarly,

$$\begin{aligned} \sum \limits _{i=k}^{+ \infty } \Vert y^{i+1}-y^{i}\Vert \geqslant \Vert y^{k}-y^{*}\Vert . \end{aligned}$$

(69)

To proceed, it is immediately shown from (60), (68) and (69) that

$$\begin{aligned} \Vert x^{k}-x^{*}\Vert +\Vert y^{k}-y^{*}\Vert\leqslant & {} (\Vert {y}^{k}-{y}^{k-1}\Vert +\frac{1}{2}\Vert {x}^{k}-{x}^{k-1}\Vert + \frac{1}{2}\Vert {y}^{k-1}-{y}^{k-2}\Vert )\\{} & {} +\frac{4\zeta }{\delta } \varphi (e_{k}). \end{aligned}$$

Exploiting (67), the inequality above leads to

$$\begin{aligned} \Vert x^{k}-x^{*}\Vert +\Vert y^{k}-y^{*}\Vert\leqslant & {} \frac{4\zeta }{\delta } \varphi (e_{k}) + \sqrt{\frac{3}{\delta } e_{k-2}} \nonumber \\= & {} O(\max \{\varphi (e_{k}), \ \sqrt{e_{k-2}}\}). \end{aligned}$$

(70)

By taking into account (6), (51) and the $l_g-$Lipschitz continuity of $\nabla g$, we obtain

$$\begin{aligned} \Vert \lambda ^{k}-\lambda ^{*}\Vert{} & {} \leqslant \gamma |\alpha - \beta | \Vert A(x^{k} - x^{*})\Vert + (\tau _k + \gamma |\alpha - \beta |) \Vert y^{k}-y^{*}\Vert \\{} & {} \quad + (\tau _k+ l_{g}) \Vert y^{k-1}-y^{*}\Vert , \end{aligned}$$

which implies that

$$\begin{aligned} \Vert \lambda ^{k}-\lambda ^{*}\Vert= & {} O(\Vert x^{k}-x^{*}\Vert ) + O(\Vert y^{k}-y^{*}\Vert ) + O(\Vert y^{k-1}-y^{*}\Vert ) \\= & {} O(\max \{\varphi (e_{k}), \ \sqrt{e_{k-2}}\}). \end{aligned}$$

Together with (70), one has

$$\begin{aligned} \Vert w^{k}-w^{*}\Vert = O(\max \{\varphi (e_{k}), \ \sqrt{e_{k-2}}\}), \ \forall \ k \geqslant \hat{k}, \end{aligned}$$

and relation (43) holds true. The remaining proof is similar to that of Theorem 4 and so omitted here.