A class of accelerated GADMM-based method for multi-block nonconvex optimization problems

Abstract

To improve the computational efficiency, based on the generalized alternating direction method of multipliers (GADMM), we consider a class of accelerated method for solving multi-block nonconvex and nonsmooth optimization problems. First, we linearize the smooth part of the objective function and add proximal terms in subproblems, resulting in the proximal linearized GADMM. Then, we introduce an inertial technique and give the inertial proximal linearized GADMM. The convergence of the regularized augmented Lagrangian function sequence is proved under some appropriate assumptions. When some component functions of the objective function are convex, we use the error bound condition and obtain that the sequences generated by the algorithms locally converge to the critical point in a R-linear rate. Moreover, we apply the proposed algorithms to SCAD and robust PCA problems to verify the efficiency of the algorithms.

Appendix. The proof of (34)

Here we give the proof of the inequality (34).

Proof

Let ${\bar{w}}^{k+1}=({\bar{x}}^{k+1},{\bar{y}}^{k+1},{\bar{\lambda }}^{k+1})\in \text {crit}\mathcal {L}_{\beta }$, then

$$\begin{aligned} \sum \limits _{i=1}^{n}{{{A}_{i}}{\bar{x}_{i}}^{k+1}}+B{\bar{y}^{k+1}}-b=0. \end{aligned}$$

By the definition of $\mathcal {L}_\beta (\cdot )$ (2), it holds that

$$\begin{aligned} \begin{aligned}&\mathcal {L}_\beta \left( w^{k+1}\right) -\mathcal {L}_\beta \left( \bar{w}^{k+1}\right) \\ =&\left[ \sum \limits _{i=1}^{n}{{{f}_{i}}({x}_{i}^{k+1})}+g(y^{k+1})-\left\langle \lambda ^{k+1} ,\sum \limits _{i=1}^{n}{{{A}_{i}}{{x}_{i}^{k+1}}}+By^{k+1}-b \right\rangle \right. \\&\left. +\frac{\beta }{2}{{\left\| \sum \limits _{i=1}^{n}{{{A}_{i}}{{x}_{i}^{k+1}}}+By^{k+1}-b \right\| }^{2}}\right] \\&-\left[ \sum \limits _{i=1}^{n}{{f}_{i}}({\bar{x}_{i}^{k+1}})+g(\bar{y}^{k+1})-\left\langle \bar{\lambda }^{k+1} ,\sum \limits _{i=1}^{n}{{{A}_{i}}{\bar{x}_{i}^{k+1}}}+B\bar{y}^{k+1}-b \right\rangle \right. \\&\left. +\frac{\beta }{2}{{\left\| \sum \limits _{i=1}^{n}{{{A}_{i}}{\bar{x}_{i}^{k+1}}}+B\bar{y}^{k+1}-b \right\| }^{2}}\right] . \end{aligned} \end{aligned}$$

(A1)

From the convexity of $f_i$, we get

$$\begin{aligned} \begin{aligned} {{{f}_{i}}({x}_{i}^{k+1})}-{{f}_{i}}({\bar{x}_{i}^{k+1}})\le&\left\langle {A}_{i}^{\top }{{\lambda }^{k}}-\beta {A}_{i}^{\top }(\sum \limits _{j=1}^{i}{{{A}_{j}}{x}_{j}^{k+1}+}\sum \limits _{j=i+1}^{n}{{{A}_{j}}{x}_{j}^{k}+}B{{y}^{k}}-b)\right. \\&\left. -F_i\Delta x_i^{k+1}, x_i^{k+1}-\bar{x}_i^{k+1}\right\rangle . \end{aligned} \end{aligned}$$

(A2)

And from the Lipschitz continuity of $\nabla g$ and $B^{\top }\lambda ^{k+1}=\nabla g(y^{k})+Q\Delta y^{k+1}$, we have

$$\begin{aligned} \begin{aligned} g(y^{k+1})-g(\bar{y}^{k+1})\le&\left\langle \nabla g(y^{k+1})-\nabla g(y^{k})+B^{\top }\lambda ^{k+1}\right. \\&\left. -Q\Delta y^{k+1}, y^{k+1}-\bar{y}^{k+1}\right\rangle +\frac{L_g}{2}{\left\| y^{k+1}-\bar{y}^{k+1} \right\| }^2. \end{aligned} \end{aligned}$$

(A3)

Combining (A1), (A2) and (A3), it follows that

$$\begin{aligned}{} & {} \mathcal {L}_\beta \left( w^{k+1}\right) -\mathcal {L}_\beta \left( \bar{w}^{k+1}\right) \\ \le{} & {} \sum \limits _{i=1}^n \left\langle \lambda ^k,A_i(x_i^{k+1}-\bar{x}_i^{k+1})\right\rangle -\sum \limits _{i=1}^n\left\langle F_i\Delta x_i^{k+1},x_i^{k+1}-\bar{x}_i^{k+1}\right\rangle \\{} & {} -\beta \sum \limits _{i=1}^n \left\langle \sum \limits _{j=1}^n A_jx_j^{k+1}+By^k-b-\sum \limits _{j=i+1}^nA_j\Delta x_j^{k+1},A_i(x_i^{k+1}-\bar{x}_i^{k+1})\right\rangle \\{} & {} +\left\langle \nabla g(y^{k+1})-\nabla g(y^{k}),y^{k+1}-\bar{y}^{k+1}\right\rangle -\left\langle Q\Delta y^{k+1},y^{k+1}-\bar{y}^{k+1}\right\rangle \\{} & {} +\frac{L_g}{2}{\left\| y^{k+1}-\bar{y}^{k+1} \right\| }^2+\frac{\beta }{2}{{\left\| \sum \limits _{i=1}^{n}{{{A}_{i}}{{x}_{i}^{k+1}}}+By^{k+1}-b \right\| }^{2}}\\{} & {} \underbrace{+\left\langle B^{\top }\lambda ^{k+1},y^{k+1}-\bar{y}^{k+1}\right\rangle -\left\langle \lambda ^{k+1} ,\sum \limits _{i=1}^{n}{{{A}_{i}}{{x}_{i}^{k+1}}}+By^{k+1}-b \right\rangle }\\ ={} & {} -\left\langle \lambda ^{k+1} ,\sum \limits _{i=1}^{n}{{{A}_{i}}{{x}_{i}^{k+1}}}+By^{k+1}-b-By^{k+1}+B\bar{y}^{k+1} \right\rangle \\{} & {} +\left\langle \lambda ^k,\sum \limits _{i=1}^nA_i(x_i^{k+1}-\bar{x}_i^{k+1})\right\rangle -\sum \limits _{i=1}^n\left\langle F_i\Delta x_i^{k+1},x_i^{k+1}-\bar{x}_i^{k+1}\right\rangle \\{} & {} -\beta \sum \limits _{i=1}^n \left\langle \sum \limits _{j=1}^n A_jx_j^{k+1}+By^k-b-\sum \limits _{j=i+1}^nA_j\Delta x_j^{k+1},A_i(x_i^{k+1}-\bar{x}_i^{k+1})\right\rangle \\{} & {} +\left\langle \nabla g(y^{k+1})-\nabla g(y^{k}),y^{k+1}-\bar{y}^{k+1}\right\rangle \\{} & {} -\left\langle Q\Delta y^{k+1},y^{k+1}-\bar{y}^{k+1}\right\rangle +\frac{L_g}{2}{\left\| y^{k+1}-\bar{y}^{k+1} \right\| }^2\\{} & {} +\frac{\beta }{2}{{\left\| \sum \limits _{i=1}^{n}{{{A}_{i}}{{x}_{i}^{k+1}}}+By^{k+1}-b \right\| }^{2}}. \end{aligned}$$

Considering that $\sum \nolimits _{i=1}^{n}{{{A}_{i}}{\bar{x}_{i}}^{k+1}}+B{\bar{y}^{k+1}}-b=0$ and ${{\lambda }^{k+1}}={{\lambda }^{k}}-\beta (\sum \limits _{i=1}^{n}{{{A}_{i}}}{x}_{i}^{k+1}+B{{y}^{k+1}}-b)+\beta (1-s)(\sum \limits _{i=1}^{n}{{{A}_{i}}}{x}_{i}^{k+1}+B{{y}^{k}}-b)$, we have

$$\begin{aligned}{} & {} \mathcal {L}_\beta \left( w^{k+1}\right) -\mathcal {L}_\beta \left( \bar{w}^{k+1}\right) \nonumber \\{} & {} \!=\left\langle \lambda ^k-\lambda ^{k+1},\!\sum \limits _{i=1}^n A_i(x_i^{k+1}\!-\!\bar{x}_i^{k+1})\right\rangle \!+\!\beta \sum \limits _{i=1}^n\left\langle \!\sum \limits _{j=i+1}^nA_j\Delta x_j^{k+1}, A_i(x_i^{k+1}\!-\!\bar{x}_i^{k+1})\right\rangle \nonumber \\{} & {} -\frac{1}{1-s}\left\langle \lambda ^{k+1}-\lambda ^{k}+\beta (\sum \limits _{i=1}^{n}{{{A}_{i}}{{x}_{i}^{k+1}}}+By^{k+1}-b),\sum \limits _{i=1}^n A_i(x_i^{k+1}-\bar{x}_i^{k+1})\right\rangle \nonumber \\{} & {} -\sum \limits _{i=1}^n\left\langle F_i\Delta x_i^{k+1},x_i^{k+1}-\bar{x}_i^{k+1}\right\rangle +\left\langle \nabla g(y^{k+1})-\nabla g(y^{k}),y^{k+1}-\bar{y}^{k+1}\right\rangle \nonumber \\{} & {} -\left\langle Q\Delta y^{k+1},y^{k+1}-\bar{y}^{k+1}\right\rangle +\frac{L_g}{2}{\left\| y^{k+1}-\bar{y}^{k+1} \right\| }^2\nonumber \\{} & {} +\frac{\beta }{2}{{\left\| \sum \limits _{i=1}^{n}{{{A}_{i}}{{x}_{i}^{k+1}}}+By^{k+1}-b \right\| }^{2}}. \end{aligned}$$

(A4)

Further, from $s\beta \sum \nolimits _{i=1}^nA_ix_i^{k+1}=\lambda ^k-\lambda ^{k+1}-\beta (By^{k+1}-b)+\beta (1-s)(By^k-b)$, we obtain following two equalities

$$\begin{aligned}&\sum \limits _{i=1}^{n}A_ix_i^{k+1}-\sum \limits _{i=1}^{n}A_i\bar{x}_i^{k+1}\\&=\frac{1}{s\beta }(\lambda ^k-\lambda ^{k+1})-\frac{1}{s}(By^{k+1}-B\bar{y}^{k+1})+\frac{1-s}{s}(By^k-B\bar{y}^{k+1}), \end{aligned}$$

and

$$\begin{aligned} \sum \limits _{i=1}^{n}A_ix_i^{k+1}+By^{k+1}-b=\frac{1}{s\beta }(\lambda ^k-\lambda ^{k+1})-\frac{1-s}{s}(By^{k+1}-By^k). \end{aligned}$$

Substituting the above two equations into (A4), then we obtain (34).$\square$