Generalized alternating direction method of multipliers: new theoretical insights and applications

Abstract

Recently, the alternating direction method of multipliers (ADMM) has received intensive attention from a broad spectrum of areas. The generalized ADMM (GADMM) proposed by Eckstein and Bertsekas is an efficient and simple acceleration scheme of ADMM. In this paper, we take a deeper look at the linearized version of GADMM where one of its subproblems is approximated by a linearization strategy. This linearized version is particularly efficient for a number of applications arising from different areas. Theoretically, we show the worst-case ${\mathcal {O}}(1/k)$ convergence rate measured by the iteration complexity ($k$ represents the iteration counter) in both the ergodic and a nonergodic senses for the linearized version of GADMM. Numerically, we demonstrate the efficiency of this linearized version of GADMM by some rather new and core applications in statistical learning. Code packages in Matlab for these applications are also developed.

Appendices

Appendices

We show that our analysis in Sects. 3 and 4 can be extended to the case where both the $\mathbf {x}$- and $\mathbf {y}$-subproblems in (3) are linearized. The resulting scheme, called doubly linearized version of the GADMM (“DL-GADMM” for short), reads as

$$\begin{aligned} \mathbf {x}^{t+1}&= ~ \mathop {{\mathrm {argmin}}}_{\mathbf {x}\in {\mathcal {X}}}\Big \{f_1(\mathbf {x})-\mathbf {x}^T\mathbf {A}^T{\gamma }^t + \frac{\rho }{2}\Vert \mathbf {A}\mathbf {x}+\mathbf {B}\mathbf {y}^t-\mathbf {b}\Vert ^2 + \frac{1}{2}\Vert \mathbf {x}- \mathbf {x}^t \Vert ^2_{\mathbf {G}_1}\Big \}, \quad \nonumber \\ \mathbf {y}^{t+1}&= ~ \mathop {{\mathrm {argmin}}}_{\mathbf {y}\in {\mathcal {Y}}} \Big \{f_2(\mathbf {y}) -\mathbf {y}^T\mathbf {B}^T{\gamma }^t + \frac{\rho }{2} \left\| \alpha \mathbf {A}\mathbf {x}^{t+1}\right. \nonumber \\&\left. +(1-\alpha )(\mathbf {b}-\mathbf {B}\mathbf {y}^t) +\mathbf {B}\mathbf {y}-\mathbf {b}\right\| ^2 + \frac{1}{2}\Vert \mathbf {y}-\mathbf {y}^t\Vert ^2_{\mathbf {G}_2}\Big \}, \nonumber \\ {\gamma }^{t+1}&= ~ {\gamma }^t-\rho \Big (\alpha \mathbf {A}\mathbf {x}^{t+1} +(1-\alpha )(\mathbf {b}-\mathbf {B}\mathbf {y}^t) + \mathbf {B}\mathbf {y}^{t+1}-\mathbf {b}\Big ), \end{aligned}$$

(79)

where the matrices $\mathbf {G}_1\in {\mathbb {R}}^{n_1\times n_1}$ and $\mathbf {G}_2\in {\mathbb {R}}^{n_2\times n_2}$ are both symmetric and positive definite.

For further analysis, we define two matrices, which are analogous to $\mathbf {H}$ and $\mathbf {Q}$ in (11), respectively, as

$$\begin{aligned} \begin{aligned} \mathbf {H}_2&= \begin{pmatrix} \mathbf {G}_1 &{}\quad 0 &{}\quad 0\\ 0 &{}\quad \frac{\rho }{\alpha }\mathbf {B}^T\mathbf {B}+\mathbf {G}_2 &{}\quad \frac{1-\alpha }{\alpha }\mathbf {B}^T\\ 0 &{}\quad \frac{1-\alpha }{\alpha }\mathbf {B}&{}\quad \frac{1}{\alpha \rho }\mathbf {I}_n \end{pmatrix}, \\ \mathbf {Q}_2&= \begin{pmatrix} \mathbf {G}_1 &{}\quad 0 &{}\quad 0\\ 0 &{}\quad \rho \mathbf {B}^T\mathbf {B}+\mathbf {G}_2 &{}\quad (1-\alpha )\mathbf {B}^T\\ 0 &{}\quad -\mathbf {B}&{}\quad \frac{1}{\rho }\mathbf {I}_n \end{pmatrix}. \end{aligned} \end{aligned}$$

(80)

Obviously, we have

$$\begin{aligned} \mathbf {Q}_2 = \mathbf {H}_2\mathbf {M}, \end{aligned}$$

(81)

where $\mathbf {M}$ is defined in (10). Note that the equalities (8) and (9) still hold.

1.1 A worst-case ${\mathcal {O}}(1/k)$ convergence rate in the ergodic sense for (79)

We first establish a worst-case ${\mathcal {O}}(1/k)$ convergence rate in the ergodic sense for the DL-GADMM (79). Indeed, using the relationship (81), the resulting proof is nearly the same as that in Sect. 3 for the L-GADMM (4). We thus only list two lemmas (analogous to Lemmas 1 and 2) and one theorem (analogous to Theorem 2) to demonstrate a worst-case ${\mathcal {O}}(1/k)$ convergence rate in the ergodic sense for (79), and omit the details of proofs.

Lemma 7

Let the sequence $\{\mathbf {w}^t\}$ be generated by the DL-GADMM (79) with $\alpha \in (0,2)$ and the associated sequence $\{\widetilde{\mathbf {w}}^t\}$ be defined in (7). Then we have

$$\begin{aligned} \begin{aligned} f(\mathbf {u}) - f(\widetilde{\mathbf {u}}^t) +\left( \mathbf {w}-\widetilde{\mathbf {w}}^t\right) ^TF(\widetilde{\mathbf {w}}^t) \ge \left( \mathbf {w}-\widetilde{\mathbf {w}}^t\right) ^T\mathbf {Q}_2\left( \mathbf {w}^{t}-\widetilde{\mathbf {w}}^t\right) , \;\; \forall \mathbf {w}\in \varOmega , \end{aligned} \end{aligned}$$

(82)

where $\mathbf {Q}_2$ is defined in (80).

Lemma 8

Let the sequence $\{\mathbf {w}^t\}$ be generated by the DL-GADMM (79) with $\alpha \in (0,2)$ and the associated sequence $\{\widetilde{\mathbf {w}}^t\}$ be defined in (7). Then for any $\mathbf {w}\in \varOmega$, we have

$$\begin{aligned}&\left( \mathbf {w}-\widetilde{\mathbf {w}}^t\right) ^T\mathbf {Q}_2\left( \mathbf {w}^t-\widetilde{\mathbf {w}}^t\right) \nonumber \\&\quad = \frac{1}{2}\left( \Vert \mathbf {w}-\mathbf {w}^{t+1}\Vert _{\mathbf {H}_2}^2 - \Vert \mathbf {w}-\mathbf {w}^t\Vert _{\mathbf {H}_2}^2\right) + \frac{1}{2}\Vert \mathbf {x}^t-\widetilde{\mathbf {x}}^{t}\Vert _{\mathbf {G}_1}^2\nonumber \\&\quad \quad + \frac{1}{2}\Vert \mathbf {y}^t - \widetilde{\mathbf {y}}^t\Vert _{\mathbf {G}_2}^2 + \frac{2-\alpha }{2\rho }\Vert {\gamma }^t-\widetilde{{\gamma }}^t\Vert ^2. \end{aligned}$$

(83)

Theorem 7

Let $\mathbf {H}_2$ be given by (80) and $\{\mathbf {w}^t\}$ be the sequence generated by the DL-GADMM (79) with $\alpha \in (0,2)$. For any integer $k>0$, let $\widehat{\mathbf {w}}_k$ be defined by

$$\begin{aligned} \widehat{\mathbf {w}}_k = \frac{1}{k+1} \sum _{t=0}^k \widetilde{\mathbf {w}}^t, \end{aligned}$$

(84)

where $\widetilde{\mathbf {w}}^t$ is defined in (7). Then, $\widehat{\mathbf {w}}_k\in \varOmega$ and

$$\begin{aligned} f(\widehat{\mathbf {u}}_k) -f(\mathbf {u}) + \left( \widehat{\mathbf {w}}_{k}-\mathbf {w}\right) ^T F(\mathbf {w}) \le \frac{1}{2(k+1)}\Vert \mathbf {w}-\mathbf {w}^0\Vert _{\mathbf {H}_2}^2,\quad \forall \mathbf {w}\in \varOmega . \end{aligned}$$

1.2 A worst-case ${\mathcal {O}}(1/k)$ convergence rate in a nonergodic sense for (79)

Next, we prove a worst-case ${\mathcal {O}}(1/k)$ convergence rate in a nonergodic sense for the DL-GADMM (79). Note that Lemma 4 still holds by replacing $\mathbf {H}$ with $\mathbf {H}_2$. That is, if $\Vert \mathbf {w}^t-\mathbf {w}^{t+1}\Vert _{\mathbf {H}_2}^2 = 0$, $\widetilde{\mathbf {w}}^t$ defined in (7) is an optimal solution point to (5). Thus, for the sequence $\{\mathbf {w}^t\}$ generated by the DL-GADMM (79), it is reasonable to measure the accuracy of an iterate by $\Vert \mathbf {w}^t-\mathbf {w}^{t+1}\Vert _{\mathbf {H}_2}^2$.

Proofs of the following two lemmas are analogous to those of Lemmas 5 and 6, respectively. We thus omit them.

Lemma 9

Let the sequence $\{\mathbf {w}^t\}$ be generated by the DL-GADMM (79) with $\alpha \in (0,2)$ and the associated $\{ \widetilde{\mathbf {w}}^t\}$ be defined in (7); the matrix $\mathbf {Q}_2$ be defined in (80). Then, we have

$$\begin{aligned} \left( \widetilde{\mathbf {w}}^t - \widetilde{\mathbf {w}}^{t+1}\right) ^T\mathbf {Q}_2 \left[ \left( \mathbf {w}^t-\mathbf {w}^{t+1}\right) - \left( \widetilde{\mathbf {w}}^t-\widetilde{\mathbf {w}}^{t+1}\right) \right] \ge 0. \end{aligned}$$

Lemma 10

Let the sequence $\{\mathbf {w}^t\}$ be generated by the DL-GADMM (79) with $\alpha \in (0,2)$ and the associated $\{ \widetilde{\mathbf {w}}^t\}$ be defined in (7); the matrices $\mathbf {M}$, $\mathbf {H}_2$, $\mathbf {Q}_2$ be defined in (10) and (80). Then, we have

$$\begin{aligned} \begin{aligned}&\left( \mathbf {w}^t-\widetilde{\mathbf {w}}^t\right) ^T\mathbf {M}^T\mathbf {H}_2\mathbf {M}\left[ \left( \mathbf {w}^t-\widetilde{\mathbf {w}}^t\right) -\left( \mathbf {w}^{t+1}- \widetilde{\mathbf {w}}^{t+1}\right) \right] \\&\quad \ge \frac{1}{2} \left\| \left( \mathbf {w}^t-\widetilde{\mathbf {w}}^t\right) - \left( \mathbf {w}^{t+1} -\widetilde{\mathbf {w}}^{t+1}\right) \right\| ^2_{\left( \mathbf {Q}_2^T+\mathbf {Q}_2\right) }. \end{aligned} \end{aligned}$$

Based on the above two lemmas, we see that the sequence $\{\Vert \mathbf {w}^t-\mathbf {w}^{t+1}\Vert _{\mathbf {H}_2}\}$ is monotonically non-increasing. That is, we have the following theorem.

Theorem 8

Let the sequence $\{\mathbf {w}^t\}$ be generated by the DL-GADMM (79) and the matrix $\mathbf {H}_2$ be defined in (80). Then, we have

$$\begin{aligned} \Vert \mathbf {w}^{t+1} - \mathbf {w}^{t+2}\Vert _{\mathbf {H}_2}^2 \le \Vert \mathbf {w}^t-\mathbf {w}^{t+1}\Vert _{\mathbf {H}_2}^2. \end{aligned}$$

Note that for the DL-GADMM (79), the $\mathbf {y}$-subproblem is also proximally regularized, and we can not extend the inequality (31) to this new case. This is indeed the main difficulty for proving a worst-case ${\mathcal {O}}(1/k)$ convergence rate in a nonergodic sense for the DL-GADMM (79). A more elaborated analysis is needed. Let us show one lemma first to bound the left-hand side in (31).

Lemma 11

Let $\{\mathbf {y}^t\}$ be the sequence generated by the DL-GADMM (79) with $\alpha \in (0,2)$. Then, we have

$$\begin{aligned} \left( \mathbf {y}^t - \mathbf {y}^{t+1}\right) \mathbf {B}^T\left( {\gamma }^t-{\gamma }^{t+1}\right) \ge \frac{1}{2} \Vert \mathbf {y}^t-\mathbf {y}^{t+1}\Vert _{\mathbf {G}_2}^2 -\frac{1}{2}\Vert \mathbf {y}^{t-1}-\mathbf {y}^t\Vert _{\mathbf {G}_2}^2. \end{aligned}$$

(85)

Proof

It follows from the optimality condition of the $\mathbf {y}$-subproblem in (79) that

$$\begin{aligned} f_2(\mathbf {y}) - f_2(\mathbf {y}^{t+1}) + \left( \mathbf {y}- \mathbf {y}^{t+1}\right) ^T \big [-\mathbf {B}^T{\gamma }^{t+1} + \mathbf {G}_2(\mathbf {y}^{t+1}-\mathbf {y}^t)\big ] \ge 0, \quad \forall \mathbf {y}\in {\mathcal {Y}}. \end{aligned}$$

(86)

Similarly, we also have,

$$\begin{aligned} f_2(\mathbf {y}) - f_2(\mathbf {y}^{t}) + (\mathbf {y}- \mathbf {y}^{t})^T \big [-\mathbf {B}^T{\gamma }^{t} + \mathbf {G}_2(\mathbf {y}^{t}-\mathbf {y}^{t-1})\big ] \ge 0, \quad \forall \mathbf {y}\in {\mathcal {Y}}. \end{aligned}$$

(87)

Setting $\mathbf {y}= \mathbf {y}^{t}$ in (86) and $\mathbf {y}=\mathbf {y}^{t+1}$ in (87), and summing them up, we have

$$\begin{aligned} \begin{aligned} \left( \mathbf {y}^t - \mathbf {y}^{t+1}\right) \mathbf {B}^T\left( {\gamma }^t-{\gamma }^{t+1}\right)&\ge (\mathbf {y}^{t+1}-\mathbf {y}^t)\mathbf {G}_2 (\mathbf {y}^{t+1} - \mathbf {y}^t + \mathbf {y}^{t-1} -\mathbf {y}^{t})\\&\ge \Vert \mathbf {y}^t\!{-}\mathbf {y}^{t{+}1}\Vert _{\mathbf {G}_2}^2 \!{-}\! \frac{1}{2}\Vert \mathbf {y}^t-\mathbf {y}^{t+1}\Vert _{\mathbf {G}_2}^2 {-} \frac{1}{2}\Vert \mathbf {y}^{t-1}-\mathbf {y}^{t}\Vert _{\mathbf {G}_2}^2\\&= \frac{1}{2}\Vert \mathbf {y}^t-\mathbf {y}^{t+1}\Vert _{\mathbf {G}_2}^2 - \frac{1}{2}\Vert \mathbf {y}^{t-1}-\mathbf {y}^{t}\Vert _{\mathbf {G}_2}^2, \end{aligned} \end{aligned}$$

where the second inequality holds by the fact that $\mathbf {a}^T\mathbf {b}\ge - \frac{1}{2}(\Vert \mathbf {a}\Vert ^2 + \Vert \mathbf {b}\Vert ^2)$. The assertion (85) is proved. $\square$

Two more lemmas should be proved in order to establish a worst-case ${\mathcal {O}}(1/k)$ convergence rate in a nonergodic sense for the DL-GADMM (79).

Lemma 12

The sequence $\{\mathbf {w}^t\}$ generated by the DL-GADMM (79) with $\alpha \in (0,2)$ and the associated $\{ \widetilde{\mathbf {w}}^t\}$ be defined in (7), then we have

$$\begin{aligned} \begin{aligned} c_\alpha \left( \Vert \mathbf {x}^t-\widetilde{\mathbf {x}}^{t}\Vert _{\mathbf {G}_1}^2+\Vert \mathbf {y}^t - \widetilde{\mathbf {y}}^t\Vert _{\mathbf {G}_2}^2 + \frac{\alpha }{\rho }\Vert {\gamma }^t-\widetilde{{\gamma }}^t\Vert ^2\right) \le \Vert \mathbf {w}^t - \widetilde{\mathbf {w}}^t\Vert _{\mathbf {Q}_2^T + \mathbf {Q}_2 -\mathbf {M}^T\mathbf {H}_2\mathbf {M}}^2 \end{aligned} \end{aligned}$$

(88)

where $c_\alpha$ is defined in (37).

Proof

By the definition of $\mathbf {Q}_2$, $\mathbf {M}$ and $\mathbf {H}_2$, we have

$$\begin{aligned} \begin{aligned}&\ \ \Vert \mathbf {w}^t - \widetilde{\mathbf {w}}^t\Vert _{\mathbf {Q}_2^T + \mathbf {Q}_2 -\mathbf {M}^T\mathbf {H}_2\mathbf {M}}^2\\&\quad =\Vert \mathbf {x}^t-\widetilde{\mathbf {x}}^{t}\Vert _{\mathbf {G}_1}^2+\Vert \mathbf {y}^t - \widetilde{\mathbf {y}}^t\Vert _{\mathbf {G}_2}^2 + \frac{2-\alpha }{\rho }\Vert {\gamma }^t-\widetilde{{\gamma }}^{t}\Vert ^2\\&\quad \ge \min \left\{ \frac{2-\alpha }{\alpha }, 1\right\} \left( \Vert \mathbf {x}^t-\widetilde{\mathbf {x}}^{t}\Vert _{\mathbf {G}_1}^2+\Vert \mathbf {y}^t - \widetilde{\mathbf {y}}^t\Vert _{\mathbf {G}_2}^2 + \frac{\alpha }{\rho }\Vert {\gamma }^t-\widetilde{{\gamma }}^t\Vert ^2\right) , \end{aligned} \end{aligned}$$

which implies the assertion (88) immediately. $\square$

In the next lemma, we refine the bound of $(\mathbf {w}-\widetilde{\mathbf {w}}^t)^T\mathbf {Q}_2(\mathbf {w}^{t}-\widetilde{\mathbf {w}}^t)$ in (82). The refined bound consists of the terms $\Vert \mathbf {w}-\mathbf {w}^{t+1}\Vert _{\mathbf {H}_2}^2$ recursively, which is favorable for establishing a worst-case ${\mathcal {O}}(1/k)$ convergence rate in a nonergodic sense for the DL-GADMM (79).

Lemma 13

Let $\{\mathbf {w}^t\}$ be the sequence generated by the DL-GADMM (79) with $\alpha \in (0,2)$. Then, $\widetilde{\mathbf {w}}^t\in \varOmega$ and

$$\begin{aligned} \begin{aligned} f(\mathbf {u}) - f(\mathbf {u}^t) + (\mathbf {w}-\widetilde{\mathbf {w}})^TF(\mathbf {w})&\ge \frac{1}{2}\big (\Vert \mathbf {w}-\mathbf {w}^{t+1}\Vert ^2_{\mathbf {H}_2} - \Vert \mathbf {w}- \mathbf {w}^t\Vert _{\mathbf {H}_2}^2\big )\\&\quad + \frac{1}{2}\Vert \mathbf {w}^t - \widetilde{\mathbf {w}}^t\Vert _{\mathbf {Q}_2^T + \mathbf {Q}_2 -\mathbf {M}^T\mathbf {H}_2\mathbf {M}}^2, \quad \forall \mathbf {w}\in \varOmega , \end{aligned} \end{aligned}$$

(89)

where $\mathbf {M}$ is defined in (10), and $\mathbf {H}_2$ and $\mathbf {Q}_2$ are defined in (80).

Proof

By the identity $\mathbf {Q}_2(\mathbf {w}^t - \widetilde{\mathbf {w}}^t) = \mathbf {H}_2(\mathbf {w}^t - \mathbf {w}^{t+1})$, it holds that

$$\begin{aligned} \left( \mathbf {w}-\widetilde{\mathbf {w}}^t\right) ^T \mathbf {Q}_2 \left( \mathbf {w}^t - \widetilde{\mathbf {w}}^t\right) = \left( \mathbf {w}-\widetilde{\mathbf {w}}^t\right) ^T \mathbf {H}_2\left( \mathbf {w}^t-\mathbf {w}^{t+1}\right) , \quad \forall \mathbf {w}\in \varOmega . \end{aligned}$$

Setting $\mathbf {a}=\mathbf {w}$, $\mathbf {b}=\widetilde{\mathbf {w}}^t$, $\mathbf {c}=\mathbf {w}^t$ and $\mathbf {d}= \mathbf {w}^{t+1}$ in the identity

$$\begin{aligned} (\mathbf {a}-\mathbf {b})^T\mathbf {H}_2(\mathbf {c}-\mathbf {d}) {=} \frac{1}{2}\left( \Vert \mathbf {a}{-}\mathbf {d}\Vert ^2_{\mathbf {H}_2}-\Vert \mathbf {a}{-}\mathbf {c}\Vert ^2_{\mathbf {H}_2}\right) {+} \frac{1}{2}\left( \Vert \mathbf {c}{-}\mathbf {b}\Vert ^2_{\mathbf {H}_2}-\Vert \mathbf {d}-\mathbf {b}\Vert ^2_{\mathbf {H}_2}\right) , \end{aligned}$$

we have

$$\begin{aligned} \begin{aligned}&2\left( \mathbf {w}-\widetilde{\mathbf {w}}^t\right) \mathbf {Q}_2\left( \mathbf {w}^t-\widetilde{\mathbf {w}}^t\right) \\&\quad = \Vert \mathbf {w}-\mathbf {w}^{t+1}\Vert _{\mathbf {H}_2}^2 - \Vert \mathbf {w}-\mathbf {w}^{t}\Vert _{\mathbf {H}_2}^2 + \Vert \mathbf {w}^t-\widetilde{\mathbf {w}}^{t}\Vert _{\mathbf {H}_2}^2 - \Vert \mathbf {w}^{t+1}-\widetilde{\mathbf {w}}^{t}\Vert _{\mathbf {H}_2}^2. \end{aligned} \end{aligned}$$

(90)

Meanwhile, we have

$$\begin{aligned} \begin{aligned} \Vert \mathbf {w}^t-\widetilde{\mathbf {w}}^{t}\Vert _{\mathbf {H}_2}^2 - \Vert \mathbf {w}^{t+1}-\widetilde{\mathbf {w}}^{t}\Vert _{\mathbf {H}_2}^2&= \Vert \mathbf {w}^t-\widetilde{\mathbf {w}}^{t}\Vert _{\mathbf {H}_2}^2 - \Vert (\mathbf {w}^t-\widetilde{\mathbf {w}}^{t}) - \left( \mathbf {w}^t-\mathbf {w}^{t+1}\right) \Vert _{\mathbf {H}_2}^2\\&= \Vert \mathbf {w}^t- \widetilde{\mathbf {w}}^{t}\Vert _{\mathbf {H}_2}^2 - \Vert (\mathbf {w}^t-\widetilde{\mathbf {w}}^{t}) - \mathbf {M}(\mathbf {w}^t-\widetilde{\mathbf {w}}^{t})\Vert _{\mathbf {H}_2}^2\\&= \left( \mathbf {w}^t-\widetilde{\mathbf {w}}^t\right) (2\mathbf {H}_2\mathbf {M}- \mathbf {M}^T\mathbf {H}\mathbf {M})\left( \mathbf {w}^t-\widetilde{\mathbf {w}}^t\right) \\&= \left( \mathbf {w}^t-\widetilde{\mathbf {w}}^t\right) (\mathbf {Q}^T_2+ \mathbf {Q}_2 -\mathbf {M}^T\mathbf {H}\mathbf {M})\left( \mathbf {w}^t-\widetilde{\mathbf {w}}^t\right) , \end{aligned} \end{aligned}$$

where the last equality comes from the identity $\mathbf {Q}_2 = \mathbf {H}_2\mathbf {M}$.

Substituting the above identity into (90), we have, for all $\mathbf {w}\in \varOmega$,

$$\begin{aligned} 2\left( \mathbf {w}\!-\!\widetilde{\mathbf {w}}^t\right) \mathbf {Q}_2\left( \mathbf {w}^t\!-\!\widetilde{\mathbf {w}}^t\right) \!=\! \Vert \mathbf {w}\!-\!\mathbf {w}^{t+1}\Vert _{\mathbf {H}_2}^2 \!-\! \Vert \mathbf {w}\!-\!\mathbf {w}^{t}\Vert _{\mathbf {H}_2}^2 \!+\! \Vert \mathbf {w}^t\! -\! \widetilde{\mathbf {w}}^t\Vert _{\mathbf {Q}_2^T \!+\! \mathbf {Q}_2 \!-\!\mathbf {M}^T\mathbf {H}_2\mathbf {M}}^2 \end{aligned}$$

Plugging this identity into (82), our claim follows immediately. $\square$

Then, we show the boundedness of the sequence $\{\mathbf {w}^t\}$ generated by the DL-GADMM (79), which essentially implies the convergence of $\{\mathbf {w}^t\}$.

Theorem 9

Let $\{\mathbf {w}^t\}$ be the sequence generated by the DL-GADMM (79) with $\alpha \in (0,2)$. Then, it holds that

$$\begin{aligned} \sum _{t=0}^\infty \Vert \mathbf {w}^{t} -\widetilde{\mathbf {w}}^t\Vert _{\mathbf {Q}^T_2+\mathbf {Q}_2-\mathbf {M}^T\mathbf {H}_2\mathbf {M}}^2 \le \Vert \mathbf {w}^0-\mathbf {w}^*\Vert _{\mathbf {H}_2}^2 , \end{aligned}$$

(91)

where $\mathbf {H}_2$ is defined in (80).

Proof

Setting $\mathbf {w}= \mathbf {w}^*$ in (89), we have

$$\begin{aligned} \begin{aligned} f(\mathbf {u}^*) - f(\mathbf {u}^t) + \left( \mathbf {w}^*-\widetilde{\mathbf {w}}^t\right) ^TF(\mathbf {w}^*)&\ge \frac{1}{2}\big (\Vert \mathbf {w}^*-\mathbf {w}^{t+1}\Vert ^2_{\mathbf {H}_2} - \Vert \mathbf {w}^* - \mathbf {w}^t\Vert _{\mathbf {H}_2}^2\big )\\&\quad + \frac{1}{2}\Vert \mathbf {w}^t - \widetilde{\mathbf {w}}^t\Vert _{\mathbf {Q}_2^T + \mathbf {Q}_2 -\mathbf {M}^T\mathbf {H}_2\mathbf {M}}^2. \end{aligned} \end{aligned}$$

Then, recall (5), we have

$$\begin{aligned} \Vert \mathbf {w}^{t} -\widetilde{\mathbf {w}}^t\Vert _{\mathbf {Q}^T_2+\mathbf {Q}_2-\mathbf {M}^T\mathbf {H}_2\mathbf {M}}^2\le \Vert \mathbf {w}^{t} -\mathbf {w}^*\Vert _{\mathbf {H}_2}^2 - \Vert \mathbf {w}^{t+1} -\mathbf {w}^*\Vert _{\mathbf {H}_2}^2. \end{aligned}$$

It is easy to see that $\mathbf {Q}^T_2+\mathbf {Q}_2-\mathbf {M}^T\mathbf {H}_2\mathbf {M}\succeq {\varvec{0}}$. Thus, it holds

$$\begin{aligned} \sum _{t=0}^\infty \Vert \mathbf {w}^{t} -\widetilde{\mathbf {w}}^t\Vert _{\mathbf {Q}^T_2+\mathbf {Q}_2-\mathbf {M}^T\mathbf {H}_2\mathbf {M}}^2 \le \Vert \mathbf {w}^0-\mathbf {w}^*\Vert _{\mathbf {H}_2}^2, \end{aligned}$$

which completes the proof. $\square$

Finally, we establish a worst-case ${\mathcal {O}}(1/k)$ convergence rate in a nonergodic sense for the DL-GADMM (79).

Theorem 10

Let the sequence $\{\mathbf {w}^t\}$ be generated by the scheme DL-GADMM (79) with $\alpha \in (0,2)$. It holds that

$$\begin{aligned} \Vert \mathbf {w}^k-\mathbf {w}^{k+1}\Vert _{\mathbf {H}_2}^2 = {\mathcal {O}}(1/k). \end{aligned}$$

(92)

Proof

By the definition of $\mathbf {H}_2$ in (80), we have

$$\begin{aligned} \ \Vert \mathbf {w}^t \!-\! \mathbf {w}^{t+1}\Vert ^2_{\mathbf {H}_2} \!&= \! \Vert \mathbf {x}^t \!-\! \widetilde{\mathbf {x}}^t\Vert ^2_{\mathbf {G}_1} \!+\!\Vert \mathbf {y}^t \!-\! \widetilde{\mathbf {y}}^t\Vert ^2_{\mathbf {G}_2}\!+\! \frac{1}{\alpha \rho } \left( \Vert \rho \mathbf {B}\left( \mathbf {y}^t \!-\! \mathbf {y}^{t+1}\right) \Vert ^2 \right. \nonumber \\&\!+\, \Vert {\gamma }^t\!-\!{\gamma }^{t+1}\Vert ^2 \left. + 2(1 - \alpha )\rho \left( \mathbf {y}^t - \mathbf {y}^{t+1}\right) ^T\mathbf {B}^T\left( {\gamma }^t-{\gamma }^{t+1}\right) \right) \nonumber \\&= \Vert \mathbf {x}^t - \widetilde{\mathbf {x}}^t\Vert ^2_{\mathbf {G}_1} +\Vert \mathbf {y}^t - \widetilde{\mathbf {y}}^t\Vert ^2_{\mathbf {G}_2}+\frac{\alpha }{\rho }\Vert {\gamma }^t-\widetilde{{\gamma }}^t\Vert ^2 \nonumber \\&-2\left( \mathbf {y}^t - \mathbf {y}^{t+1}\right) ^T\mathbf {B}^T\left( {\gamma }^t-{\gamma }^{t+1}\right) . \end{aligned}$$

(93)

Using (85, 88, 91) and (93), we obtain

$$\begin{aligned} \begin{aligned} \sum _{t=1}^k \Vert \mathbf {w}^t-\mathbf {w}^{t+1}\Vert _{\mathbf {H}_2}^2&\le \frac{1}{c_\alpha }\sum _{t=1}^k\Vert \mathbf {w}^t - \widetilde{\mathbf {w}}^t\Vert _{\mathbf {Q}_2^T + \mathbf {Q}_2 -\mathbf {M}^T\mathbf {H}_2\mathbf {M}}^2\\&\quad \quad +\sum _{t=1}^k\left( \Vert \mathbf {y}^{t-1}-\mathbf {y}^t\Vert _{\mathbf {G}_2}^2-\Vert \mathbf {y}^t-\mathbf {y}^{t+1}\Vert _{\mathbf {G}_2}^2\right) \\&\le \frac{1}{c_\alpha }\Vert \mathbf {w}^0-\mathbf {w}^*\Vert _{\mathbf {H}_2}^2+\Vert \mathbf {y}^{0}-\mathbf {y}^1\Vert _{\mathbf {G}_2}^2. \end{aligned} \end{aligned}$$

By Theorem 8, the sequence $\{\Vert \mathbf {w}^t-\mathbf {w}^{t+1}\Vert _{\mathbf {H}_2}^2\}$ is non-increasing. Thus, we have

$$\begin{aligned} \begin{aligned} k \Vert \mathbf {w}^k - \mathbf {w}^{k+1} \Vert _{\mathbf {H}_2}^2\le&\sum _{t=1}^k \Vert \mathbf {w}^t-\mathbf {w}^{t+1}\Vert _{\mathbf {H}_2}^2 \\ \le&\frac{1}{c_\alpha }\Vert \mathbf {w}^0-\mathbf {w}^*\Vert _{\mathbf {H}_2}^2+\Vert \mathbf {y}^{0}- \mathbf {y}^1\Vert _{\mathbf {G}_2}^2, \end{aligned} \end{aligned}$$

and the assertion (92) is proved. $\square$

Recall that for the sequence $\{\mathbf {w}^t\}$ generated by the DL-GADMM (79), it is reasonable to measure the accuracy of an iterate by $\Vert \mathbf {w}^t-\mathbf {w}^{t+1}\Vert _{\mathbf {H}_2}^2$. Thus, Theorem 10 demonstrates a worst-case ${\mathcal {O}}(1\!/k)$ convergence rate in a nonergodic sense for the DL-GADMM (79).