A Method of Inertial Regularized ADMM for Separable Nonconvex Optimization Problems

Abstract

The alternating direction method of multipliers (ADMM) is an effective algorithm for solving optimization problems with separable structures. Recently, inertial technique has been widely used in various algorithms to accelerate its convergence speed and enhance the numerical performance. There are a lot of convergence analyses for solving the convex optimization problems by combining inertial technique with ADMM, while the research on the nonconvex cases is still in its infancy. In this paper, we propose an algorithm framework of inertial regularized ADMM (iRADMM) for a class of two-block nonconvex optimization problems. Under some assumptions, we establish the subsequential and global convergence of the proposed method. Furthermore, we apply the iRADMM to solve the signal recovery, image reconstruction and SCAD penalty problem. The numerical results demonstrate the efficiency of the iRADMM algorithm and also illustrate the effectiveness of the introduced inertial term.

6 Appendix A.

1.1 A.1 Proof of Lemma 5

Proof

From the definition of the augmented Lagrangian function ${\mathcal {L}}_{\beta }(\cdot )$, it follows that

$$\begin{aligned}{} & {} {\mathcal {L}}_{\beta }(x^{k+1},y^{k+1},p^{k+1})-{\mathcal {L}}_{\beta }(x^{k+1},y^{k+1},p^{k})\nonumber \\{} & {} \quad =\langle p^{k}-p^{k+1},Ax^{k+1}+By^{k+1}-b\rangle =\frac{1}{\beta }\Vert p^{k}-p^{k+1}\Vert ^{2}.\nonumber \\ \end{aligned}$$

(23)

It is obvious that

$$\begin{aligned}{} & {} {\mathcal {L}}_{\beta }(x^{k+1},y^{k+1},p^{k})-{\mathcal {L}}_{\beta }(x^{k+1},\bar{y}^{k},p^{k})\nonumber \\{} & {} \quad = g(y^{k+1})-g(\bar{y}^{k})+\langle p^{k},B(\bar{y}^{k}-y^{k+1})\rangle \nonumber \\{} & {} \qquad +\frac{\beta }{2}\Vert Ax^{k+1}+By^{k+1}-b\Vert ^{2}-\frac{\beta }{2}\Vert Ax^{k+1}+B\bar{y}^{k}-b\Vert ^{2}.\nonumber \\ \end{aligned}$$

(24)

From the optimality condition of y-subproblem, which implies that

$$\begin{aligned} \begin{aligned} \nabla g(y^{k+1})=B^{\top }p^{k+1}. \end{aligned} \end{aligned}$$

(25)

Since $\nabla g$ is Lipschitz continuous with modulus $L>0$, it follows from Lemma 2 and (25) obtain

$$\begin{aligned}{} & {} g(y^{k+1})-g(\bar{y}^{k})\le \langle \nabla g(y^{k+1}),y^{k+1}-\bar{y}^{k}\rangle +\frac{L}{2}\Vert y^{k+1}-\bar{y}^{k}\Vert ^{2}\nonumber \\{} & {} \quad =\langle p^{k+1},B(y^{k+1}-\bar{y}^{k})\rangle +\frac{L}{2}\Vert y^{k+1}-\bar{y}^{k}\Vert ^{2}. \end{aligned}$$

(26)

By substituting (26) into (24), we obtain

$$\begin{aligned}{} & {} {\mathcal {L}}_{\beta }(x^{k+1},y^{k+1},p^{k})-{\mathcal {L}}_{\beta }(x^{k+1},\bar{y}^{k},p^{k})\nonumber \\{} & {} \quad \le \langle p^{k}-p^{k+1},B(\bar{y}^{k}-y^{k+1})\rangle +\frac{\beta }{2}\Vert Ax^{k+1}+By^{k+1}-b\Vert ^{2}\nonumber \\{} & {} \qquad -\frac{\beta }{2}\Vert Ax^{k+1}+B\bar{y}^{k}-b\Vert ^{2}+\frac{L}{2}\Vert y^{k+1}-\bar{y}^{k}\Vert ^{2}, \end{aligned}$$

(27)

and $p^{k+1}=p^{k}-\beta (Ax^{k+1}+By^{k+1}-b),$ we have

$$\begin{aligned} \begin{aligned}\begin{array}{l} \frac{\beta }{2}\Vert Ax^{k+1}+By^{k+1}-b\Vert ^{2}=\frac{1}{2\beta }\Vert p^{k}-p^{k+1}\Vert ^{2}. \end{array} \end{aligned}\end{aligned}$$

(28)

Furthermore, $Ax^{k+1}+B\bar{y}^{k}-b=Ax^{k+1}+By^{k+1}-b+B(\bar{y}^{k}-y^{k+1}),$ thus

$$\begin{aligned}{} & {} \langle p^{k}-p^{k+1},B(\bar{y}^{k}-y^{k+1})\rangle -\frac{\beta }{2}\Vert Ax^{k+1}+B\bar{y}^{k}-b\Vert ^{2}\nonumber \\{} & {} \quad =-\frac{1}{2\beta }\Vert p^{k}-p^{k+1}\Vert ^{2}-\frac{\beta \mu _{B^{\top }}}{2}\Vert \bar{y}^{k}-y^{k+1}\Vert ^{2}. \end{aligned}$$

(29)

By substituting (28),(29) into (27), we have

$$\begin{aligned}{} & {} {\mathcal {L}}_{\beta }(x^{k+1},y^{k+1},p^{k})-{\mathcal {L}}_{\beta }(x^{k+1},\bar{y}^{k},p^{k})\nonumber \\{} & {} \quad \le -\frac{\beta \mu _{B^{\top }}-L}{2}\Vert y^{k+1}-\bar{y}^{k}\Vert ^{2}. \end{aligned}$$

(30)

Since $x^{k+1}$ is the minimum value of x-subproblem for iterative scheme (7), we have

$$\begin{aligned}{} & {} {\mathcal {L}}_{\beta }(x^{k+1},\bar{y}^{k},p^{k})\le {\mathcal {L}}_{\beta }(x^{k},\bar{y}^{k},p^{k})\nonumber \\{} & {} \quad +\frac{1}{2}\Vert x^{k}-\bar{x}^{k}\Vert ^{2}_{S}-\frac{1}{2}\Vert x^{k+1}-\bar{x}^{k}\Vert ^{2}_{S}. \end{aligned}$$

(31)

On the other hand

$$\begin{aligned}{} & {} {\mathcal {L}}_{\beta }(x^{k},\bar{y}^{k},p^{k})-{\mathcal {L}}_{\beta }(x^{k},y^{k},p^{k})\nonumber \\{} & {} \quad =g(\bar{y}^{k})-g(y^{k})-\langle p^{k},B(\bar{y}^{k}-y^{k})\rangle \nonumber \\{} & {} \qquad +\frac{\beta }{2}\Vert Ax^{k}+B\bar{y}^{k}-b\Vert ^{2}-\frac{\beta }{2}\Vert Ax^{k}+By^{k}-b\Vert ^{2}. \end{aligned}$$

(32)

By the same reason we can obtain

$$\begin{aligned}{} & {} g(\bar{y}^{k})-g(y^{k})\le \langle \nabla g(y^{k}),\bar{y}^{k}-y^{k}\rangle \nonumber \\{} & {} \quad +\frac{L}{2}\Vert \bar{y}^{k}-y^{k}\Vert ^{2}=\langle p^{k},B(\bar{y}^{k}-y^{k})\rangle +\frac{L}{2}\Vert \bar{y}^{k}-y^{k}\Vert ^{2}.\nonumber \\ \end{aligned}$$

(33)

It is easy to verify that

$$\begin{aligned} \begin{aligned}\begin{array}{l} \frac{\beta }{2}\Vert Ax^{k}+By^{k}-b\Vert ^{2}=\frac{1}{2\beta }\Vert p^{k-1}-p^{k}\Vert ^{2}. \end{array} \end{aligned}\end{aligned}$$

(34)

Furthermore, we know

$$\begin{aligned}{} & {} \frac{\beta }{2}\Vert Ax^{k}+B\bar{y}^{k}-b\Vert ^{2}\nonumber \\{} & {} \quad =\frac{\beta }{2}\Vert \frac{1}{\beta }(p^{k-1}-p^{k})+B(\bar{y}^{k}-y^{k})\Vert ^{2}\nonumber \\{} & {} \quad \le \frac{1}{\beta }\Vert p^{k-1}-p^{k}\Vert ^{2}+\beta \lambda _{B^{\top }}\Vert \bar{y}^{k}-y^{k}\Vert ^{2}, \end{aligned}$$

(35)

where the inequality using of $(a+b)^{2}\le 2(a^{2}+b^{2}).$ By substitution (33)–(35) into (32)

$$\begin{aligned}{} & {} {\mathcal {L}}_{\beta }(x^{k},\bar{y}^{k},p^{k})-{\mathcal {L}}_{\beta }(x^{k},y^{k},p^{k})\nonumber \\{} & {} \quad \le (\beta \lambda _{B^{\top }}+\frac{L}{2})\Vert y^{k}-\bar{y}^{k}\Vert ^{2}+\frac{1}{2\beta }\Vert p^{k-1}-p^{k}\Vert ^{2}. \end{aligned}$$

(36)

Summing up (23),(30), (31), and (36), we obtain

$$\begin{aligned}{} & {} {\mathcal {L}}_{\beta }(w^{k+1})-{\mathcal {L}}_{\beta }(w^{k})\le \frac{1}{\beta }\Vert p^{k}-p^{k+1}\Vert ^{2}\nonumber \\{} & {} \quad +\frac{1}{2\beta }\Vert p^{k-1}-p^{k}\Vert ^{2}-\frac{\beta \mu _{B^{\top }}-L}{2}\Vert y^{k+1}-\bar{y}^{k}\Vert ^{2}\nonumber \\{} & {} \quad +(\beta \lambda _{B^{\top }}+\frac{L}{2})\Vert \bar{y}^{k}-y^{k}\Vert ^{2}\nonumber \\{} & {} \quad +\frac{1}{2}\Vert x^{k}-\bar{x}^{k}\Vert ^{2}_{S}-\frac{1}{2}\Vert x^{k+1}-\bar{x}^{k}\Vert ^{2}_{S}. \end{aligned}$$

(37)

Assumption 1(v) and (7) imply that $p^{k+1}-p^{k}\in Im B.$ It follows from Lemma 3 and (25) that

$$\begin{aligned}{} & {} \Vert p^{k+1}-p^{k}\Vert ^{2}\le \frac{1}{\mu _{B^{\top }}}\Vert B^{\top }(p^{k+1}-p^{k})\Vert ^{2}\nonumber \\{} & {} \quad =\frac{1}{\mu _{B^{\top }}}\Vert \nabla g(y^{k+1})-\nabla g(y^{k})\Vert ^{2}\nonumber \\{} & {} \quad \le \frac{L^{2}}{\mu _{B^{\top }}}\Vert y^{k+1}-y^{k}\Vert ^{2}. \end{aligned}$$

(38)

Since $\Vert y^{k+1}-y^{k}\Vert ^{2}\le 2(\Vert y^{k+1}-\bar{y}^{k}\Vert ^{2}+\Vert \bar{y}^{k}-y^{k}\Vert ^{2}),$ we have

$$\begin{aligned}{} & {} \frac{3}{2\beta }\Vert p^{k+1}-p^{k}\Vert ^{2}\le \frac{3L^{2}}{\beta \mu _{B^{\top }}}\Vert y^{k+1}-\bar{y}^{k}\Vert ^{2}\nonumber \\{} & {} \quad +\frac{3L^{2}}{\beta \mu _{B^{\top }}}\Vert \bar{y}^{k}-y^{k}\Vert ^{2}. \end{aligned}$$

(39)

Combining to $\bar{x}^{k+1}=x^{k+1}+\theta (x^{k+1}-\bar{x}^{k})$, $\bar{y}^{k+1}=y^{k+1}+\eta (y^{k+1}-\bar{y}^{k})$, and (37), it can be obtained through simplification and arrangement

$$\begin{aligned}{} & {} {\mathcal {L}}_{\beta }(w^{k+1})+\frac{1}{2\beta }\Vert p^{k}-p^{k+1}\Vert ^{2}\nonumber \\{} & {} \qquad +\delta \Vert y^{k+1}-\bar{y}^{k+1}\Vert ^{2}+\frac{1}{2}\Vert x^{k+1}-\bar{x}^{k+1}\Vert ^{2}_{S}\nonumber \\{} & {} \quad \le {\mathcal {L}}_{\beta }(w^{k})+\frac{1}{2\beta }\Vert p^{k-1}-p^{k}\Vert ^{2}\nonumber \\{} & {} \qquad +\delta \Vert y^{k}-\bar{y}^{k}\Vert ^{2}+\frac{1}{2}\Vert x^{k}-\bar{x}^{k}\Vert ^{2}_{S}\nonumber \\{} & {} \qquad -(M_{0}-\eta ^{2}\delta )\Vert y^{k+1}-\bar{y}^{k}\Vert ^{2}-\frac{1-\theta ^{2}}{2}\Vert x^{k+1}-\bar{x}^{k}\Vert ^{2}_{S}. \end{aligned}$$

where $\delta =\beta \lambda _{B^{\top }}+\frac{L}{2}+\frac{3L^{2}}{\beta \mu _{B^{\top }}},M_{0}=\frac{\beta \mu _{B^{\top }}-L}{2}-\frac{3L^{2}}{\beta \mu _{B^{\top }}}.$ Let $\hat{{\mathcal {L}}}_{\beta }(\hat{w}^{k})={\mathcal {L}}_{\beta }(w^{k})+\frac{1}{2\beta }\Vert p^{k-1}-p^{k}\Vert ^{2}+\delta \Vert y^{k}-\bar{y}^{k}\Vert ^{2}+\frac{1}{2}\Vert x^{k}-\bar{x}^{k}\Vert ^{2}_{S}.$ Thus, we have

$$\begin{aligned}{} & {} \hat{{\mathcal {L}}}_{\beta }(\hat{w}^{k+1})-\hat{{\mathcal {L}}}_{\beta }(\hat{w}^{k})\nonumber \\{} & {} \quad \le -\frac{1-\theta ^{2}}{2}\Vert x^{k+1}-\bar{x}^{k}\Vert ^{2}_{S}-M\Vert y^{k+1}-\bar{y}^{k}\Vert ^{2}. \end{aligned}$$

where $M=M_{0}-\eta ^2\delta ,$ from Assumption 1(iv), we know $M>0.$ This completes the proof. $\square$

1.2 A.2 Proof of Lemma 6

Proof

Since $\{w^{k}\}$ is bounded, $\{\hat{w}^{k}\}$ is bounded, which has at least one cluster point. Let $\hat{w}^{*}$ be a cluster point of $\{\hat{w}^{k}\}$, then the subsequence $\{\hat{w}^{k_{j}}\}$ to converge it, i.e., $\lim \limits _{j\rightarrow +\infty }\hat{w}^{k_{j}}=\hat{w}^{*}$. Since f is the lower semicontinuous and g is the continuous, then $\hat{{\mathcal {L}}}_{\beta }(\cdot )$ is the lower semicontinuous, and hence

$$\begin{aligned} \liminf \limits _{j\rightarrow +\infty }\hat{{\mathcal {L}}}_{\beta }(\hat{w}^{k_{j}})\ge \hat{{\mathcal {L}}}_{\beta }(\hat{w}^{*}). \end{aligned}$$

(40)

Consequently, $\{\hat{{\mathcal {L}}}_{\beta }(\hat{w}^{k_{j}})\}$ is bounded from below, which, together with the fact that $\{\hat{{\mathcal {L}}}_{\beta }(\hat{w}^{k})\}$ is nonincreasing, means that $\{\hat{{\mathcal {L}}}_{\beta }(\hat{w}^{k_{j}})\}$ is convergent. Moreover, $\{\hat{{\mathcal {L}}}_{\beta }(\hat{w}^{k})\}$ is convergent and $\hat{{\mathcal {L}}}_{\beta }(\hat{w}^{k})\ge \hat{{\mathcal {L}}}_{\beta }(\hat{w}^{*})$. Rearranging terms of (9), summing up for $k=0,\cdot \cdot \cdot n$, we have

$$\begin{aligned}{} & {} \sum \limits _{k=0}^{n}\left( \frac{1-\theta ^{2}}{2}\Vert x^{k+1}-\bar{x}^{k}\Vert ^{2}_{S}+M\Vert y^{k+1}-\bar{y}^{k}\Vert ^{2}\right) \\{} & {} \quad \le \hat{{\mathcal {L}}}_{\beta }(\hat{w}^{0})-\hat{{\mathcal {L}}}_{\beta }(\hat{w}^{n+1})\le \hat{{\mathcal {L}}}_{\beta }(\hat{w}^{0})-\hat{{\mathcal {L}}}_{\beta }(\hat{w}^{*})<+\infty . \end{aligned}$$

Since $\theta \in [0,1),M>0$, we have $\sum \limits _{k=0}^{+\infty }\Vert x^{k+1}-\bar{x}^{k}\Vert ^{2}_{S}<+\infty$, $\sum \limits _{k=0}^{+\infty }\Vert y^{k+1}-\bar{y}^{k}\Vert ^{2}<+\infty$. It is apparent from the inequality property that

$$\begin{aligned}{} & {} \Vert x^{k+1}-x^{k}\Vert ^{2}_{S}\le 2\left( \Vert x^{k+1}-\bar{x}^{k}\Vert ^{2}_{S}+\Vert \bar{x}^{k}-x^{k}\Vert ^{2}_{S}\right) \nonumber \\{} & {} \quad \le 2\left( \Vert x^{k+1}-\bar{x}^{k}\Vert ^{2}_{S}+\theta ^{2}\Vert x^{k}-\bar{x}^{k-1}\Vert ^{2}_{S}\right) . \end{aligned}$$

(41)

$$\begin{aligned}{} & {} \quad \Vert y^{k+1}-y^{k}\Vert ^{2}\le 2\left( \Vert y^{k+1}-\bar{y}^{k}\Vert ^{2}+\Vert \bar{y}^{k}-y^{k}\Vert ^{2}\right) \nonumber \\{} & {} \quad \le 2\left( \Vert y^{k+1}-\bar{y}^{k}\Vert ^{2}+\eta ^{2}\Vert y^{k}-\bar{y}^{k-1}\Vert ^{2}\right) . \end{aligned}$$

(42)

From (41) and (42), we have $\sum \limits _{k=0}^{+\infty }\Vert x^{k+1}-x^{k}\Vert ^{2}<+\infty ,$ $\sum \limits _{k=0}^{+\infty }\Vert y^{k+1}-y^{k}\Vert ^{2}<+\infty$, it follows from (38) that $\sum \limits _{k=0}^{+\infty }\Vert p^{k+1}-p^{k}\Vert ^{2}<+\infty .$ Therefore, we obtain $\sum \limits _{k=0}^{+\infty }\Vert w^{k+1}-w^{k}\Vert ^{2}<+\infty .$ $\square$

1.3 A.4 Proof of Lemma 7

Proof

From the definition of the augment Lagrangian function ${\mathcal {L}}_{\beta }(\cdot ),$ it follows

$$\begin{aligned} \begin{aligned} \left\{ \begin{array}{l} \partial _{x}\hat{{\mathcal {L}}}_{\beta }(\hat{w}^{k+1})=\partial f(x^{k+1})-A^{\top }p^{k+1}+\beta A^{\top }(Ax^{k+1}\\ \quad +By^{k+1}-b)+S(x^{k+1}-\bar{x}^{k+1}),\\ \partial _{y}\hat{{\mathcal {L}}}_{\beta }(\hat{w}^{k+1})=\nabla g(x^{k+1})-B^{\top }p^{k+1}+\beta B^{\top }(Ax^{k+1}\\ \quad +By^{k+1}-b)+2\delta (y^{k+1}-\bar{y}^{k+1}),\\ \partial _{p}\hat{{\mathcal {L}}}_{\beta }(\hat{w}^{k+1})=\frac{2}{\beta }(p^{k+1}-p^{k}),\\ \partial _{\bar{x}}\hat{{\mathcal {L}}}_{\beta }(\hat{w}^{k+1})=S(\bar{x}^{k+1}-x^{k+1}),\\ \partial _{\bar{y}}\hat{{\mathcal {L}}}_{\beta }(\hat{w}^{k+1})=2\delta (\bar{y}^{k+1}-x^{k+1}),\\ \partial _{\bar{p}}\hat{{\mathcal {L}}}_{\beta }(\hat{w}^{k+1})=\frac{1}{\beta }(p^{k}-p^{k+1}). \end{array} \right. \end{aligned} \end{aligned}$$

This together with optimality condition (8) yields

$$\begin{aligned} \begin{aligned} \left\{ \begin{array}{l} A^{\top }(p^{k}-p^{k+1})+\beta A^{\top }B(y^{k+1}-\bar{y}^{k})-(1+\theta )S(x^{k+1}\\ \quad -\bar{x}^{k})\in \partial _{x}\hat{{\mathcal {L}}}_{\beta }(\hat{w}^{k+1}),\\ B^{\top }(p^{k}-p^{k+1})+2\delta (y^{k+1}-\bar{y}^{k+1})\in \partial _{y}\hat{{\mathcal {L}}}_{\beta }(\hat{w}^{k+1}),\\ \frac{2}{\beta }(p^{k+1}-p^{k})=\partial _{p}\hat{{\mathcal {L}}}_{\beta }(\hat{w}^{k+1}),\\ \theta S(x^{k+1}-\bar{x}^{k})=\partial _{\bar{x}}\hat{{\mathcal {L}}}_{\beta }(\hat{w}^{k+1}),\\ 2\delta \eta (y^{k+1}-\bar{y}^{k})=\partial _{\bar{y}}\hat{{\mathcal {L}}}_{\beta }(\hat{w}^{k+1}),\\ \frac{1}{\beta }(p^{k}-p^{k+1})\in \partial _{\bar{p}}\hat{{\mathcal {L}}}_{\beta }(\hat{w}^{k+1}). \end{array} \right. \end{aligned}\end{aligned}$$

(43)

From Lemma 1, we obtain $(\varepsilon _{1}^{k+1},\varepsilon _{2}^{k+1},\varepsilon _{3}^{k+1},\varepsilon _{4}^{k+1},\varepsilon _{5}^{k+1},\varepsilon _{6}^{k+1})^{\top }\in \partial \hat{{\mathcal {L}}}_{\beta }(\hat{w}^{k+1})$. Furthermore, from (43), there exists a real number $\zeta _{0}$ such that

$$\begin{aligned}{} & {} \Vert (\varepsilon _{1}^{k+1},\varepsilon _{2}^{k+1},\varepsilon _{3}^{k+1},\varepsilon _{4}^{k+1},\varepsilon _{5}^{k+1},\varepsilon _{6}^{k+1})\Vert ^{2}\nonumber \\{} & {} \quad \le \zeta _{0}(\Vert x^{k+1}-\bar{x}^{k}\Vert +\Vert y^{k+1}-\bar{y}^{k}\Vert +\Vert p^{k}-p^{k+1}\Vert ).\nonumber \\ \end{aligned}$$

(44)

It follows from (39) that there exists $\zeta _{1}>0$ such that

$$\begin{aligned} \begin{aligned} \begin{array}{l} \Vert p^{k+1}-p^{k}\Vert \le \zeta _{1}(\Vert y^{k+1}-\bar{y}^{k}\Vert +\Vert y^{k}-\bar{y}^{k-1}\Vert ),k\ge 1. \end{array} \end{aligned}\nonumber \\\end{aligned}$$

(45)

Thus, combining (44) and (45), there exists $\zeta >0$ such that

$$\begin{aligned}{} & {} d(0,\partial \hat{{\mathcal {L}}}_{\beta }(\hat{w}^{k+1}))\le \Vert (\varepsilon _{1}^{k+1},\varepsilon _{2}^{k+1},\varepsilon _{3}^{k+1},\varepsilon _{4}^{k+1},\varepsilon _{5}^{k+1},\varepsilon _{6}^{k+1})\Vert \\{} & {} \quad \le \zeta (\Vert x^{k+1}-\bar{x}^{k}\Vert +\Vert y^{k+1}-\bar{y}^{k}\Vert +\Vert y^{k}-\bar{y}^{k-1}\Vert ),k\ge 1. \end{aligned}$$

This completes the proof. $\square$