On the Global and Linear Convergence of the Generalized Alternating Direction Method of Multipliers

Abstract

The formulation

$$\begin{aligned} \min _{x,y} ~f(x)+g(y),\quad \text{ subject } \text{ to } Ax+By=b, \end{aligned}$$

where f and g are extended-value convex functions, arises in many application areas such as signal processing, imaging and image processing, statistics, and machine learning either naturally or after variable splitting. In many common problems, one of the two objective functions is strictly convex and has Lipschitz continuous gradient. On this kind of problem, a very effective approach is the alternating direction method of multipliers (ADM or ADMM), which solves a sequence of f/g-decoupled subproblems. However, its effectiveness has not been matched by a provably fast rate of convergence; only sublinear rates such as O(1 / k) and $O(1/k^2)$ were recently established in the literature, though the O(1 / k) rates do not require strong convexity. This paper shows that global linear convergence can be guaranteed under the assumptions of strong convexity and Lipschitz gradient on one of the two functions, along with certain rank assumptions on A and B. The result applies to various generalizations of ADM that allow the subproblems to be solved faster and less exactly in certain manners. The derived rate of convergence also provides some theoretical guidance for optimizing the ADM parameters. In addition, this paper makes meaningful extensions to the existing global convergence theory of ADM generalizations.

Appendices

Appendix 1: Proof of Lemma 2.1

Proof

• i) The optimality conditions of the subproblems of Algorithm 2 yield (2.6) and (2.7).

• ii) By the convexity of f (1.16), combining the optimality conditions (1.13) and (2.6) yield (2.8). Similarly, (2.9) follows from the convexity of g (1.17) and the optimality conditions (1.14) and (2.7).

• iii) Equation (2.10) follows directly from (1.15) and (2.1).

• iv) By adding (2.8) and (2.9) and using (2.10), we have

  $$\begin{aligned}&\frac{1}{\beta }\langle {\uplambda }^{k}-\hat{{\uplambda }},~\hat{{\uplambda }}-{\uplambda }^*-\beta A(x^k-x^{k+1})\rangle + \langle x^{k+1}-x^*, ~\hat{P}(x^k-x^{k+1})\rangle \nonumber \\&\qquad + \langle y^{k+1}-y^*, ~Q(y^k-y^{k+1})\rangle \nonumber \\&\quad \ge \nu _f\Vert x^{k+1}-x^{*}\Vert ^2+\nu _g\Vert y^{k+1}-y^*\Vert ^2, \end{aligned}$$

  (7.1)

  which can be simplified as

  $$\begin{aligned} (\hat{u}-u^*)^TG_1(u^k-\hat{u})\ge & {} \langle A(x^k-x^{k+1}),~{\uplambda }^k-\hat{{\uplambda }}\rangle \nonumber \\&+\,\nu _f\Vert x^{k+1}-x^{*}\Vert ^2+\nu _g\Vert y^{k+1}-y^*\Vert ^2. \end{aligned}$$

  (7.2)

  By rearranging the terms, we have

  $$\begin{aligned} (u^k-u^*)^TG_1(u^k-\hat{u})\ge & {} \Vert u^k-\hat{u}\Vert _{G_1}^2+\langle A(x^k-x^{k+1}),{\uplambda }^k-\hat{{\uplambda }}\rangle \nonumber \\&+\,\nu _f\Vert x^{k+1}-x^{*}\Vert ^2+\nu _g\Vert y^{k+1}-y^*\Vert ^2. \end{aligned}$$

  (7.3)

  Then, from the identity $\Vert a-c\Vert _G^2-\Vert b-c\Vert _G^2=2(a-c)^TG(a-b)-\Vert a-b\Vert _G^2$ and (2.5), it follows that

  $$\begin{aligned} \Vert u^k-u^*\Vert _G^2-\Vert u^{k+1}-u^*\Vert _G^2=2(u^k-u^*)^TG_1(u^k-\hat{u})-\Vert G_0(u^k-\hat{u})\Vert ^2_G. \end{aligned}$$

  (7.4)

  Substituting (7.3) into the right-hand side of the above equation, we have

  $$\begin{aligned} \Vert u^k-u^*\Vert _G^2-\Vert u^{k+1}-u^*\Vert _G^2\ge & {} 2\Vert u^k-\hat{u}\Vert _{G_1}^2-\Vert u^k-\hat{u}\Vert ^2_{G_1G_0}\nonumber \\&+\,2\langle A(x^k-x^{k+1}),{\uplambda }^k-\hat{{\uplambda }}\rangle \nonumber \\&+\,2\nu _f\Vert x^{k+1}-x^{*}\Vert ^2+2\nu _g\Vert y^{k+1}-y^*\Vert ^2, \end{aligned}$$

  (7.5)

  and thus (2.11) follows immediately.

Appendix 2: Proof of Theorem 2.1

Proof

• i) By the Cauchy–Schwarz inequality, we have

  $$\begin{aligned} 2({\uplambda }^k-\hat{{\uplambda }})^TA(x^k-x^{k+1})\ge -\frac{1}{\rho }\Vert A(x^k-x^{k+1})\Vert ^2-\rho \Vert {\uplambda }^k-\hat{{\uplambda }}\Vert ^2,~\forall \rho >0. \end{aligned}$$

  (8.1)

  Substituting (8.1) into (2.11) and using $\frac{1}{\gamma }({\uplambda }^k-{\uplambda }^{k+1})={\uplambda }^k-\hat{{\uplambda }}$, we have

  $$\begin{aligned}&\Vert u^k-u^*\Vert _G^2-\Vert u^{k+1}-u^*\Vert _G^2\nonumber \\&\quad \ge ~\Vert x^k-x^{k+1}\Vert _{\hat{P}-\frac{1}{\rho }A^TA}^2+\Vert y^k-y^{k+1}\Vert _Q^2+\left( \frac{2-\gamma }{\beta }-\rho \right) \frac{1}{\gamma ^2}\Vert {\uplambda }^k-{\uplambda }^{k+1}\Vert ^2\nonumber \\&\qquad +\,2\nu _f\Vert x^{k+1}-x^{*}\Vert ^2+2\nu _g\Vert y^{k+1}-y^*\Vert ^2,\quad \forall \rho >0. \end{aligned}$$

  (8.2)

  To show that (2.14) holds for a certain $\eta >0$, we only need $\hat{P}-\frac{1}{\rho }A^TA\succ 0$ and $\frac{2-\gamma }{\beta }-\rho >0$ for a certain $\rho >0$, which is true if and only if we have $\hat{P}\succ \frac{\beta }{2-\gamma }A^T A$ or, equivalently, (2.13).

• ii) For $P=\mathbf {0}$, we first derive a lower bound for the cross term $({\uplambda }^k-\hat{{\uplambda }})^TA(x^k-x^{k+1})$. Applying (2.6) at two consecutive iterations with $P=\mathbf {0}$ and in light of the definition of $\hat{{\uplambda }}$, we have

  $$\begin{aligned} \left\{ \begin{array}{l} A^T[{\uplambda }^{k-1}-\beta (Ax^{k}+By^{k}-b)] \in \partial f(x^{k}),\\ A^T\hat{{\uplambda }} \in \partial f(x^{k+1}). \end{array} \right. \end{aligned}$$

  (8.3)

  The difference of the two terms on the left in (8.3) is

  $$\begin{aligned}&A^T\left[ {\uplambda }^{k-1}-\beta (Ax^{k}+By^{k}-b)-\hat{{\uplambda }}\right] \nonumber \\&\quad =A^T\left( {\uplambda }^k-\hat{{\uplambda }}\right) -(1-\gamma )\beta A^T\left( Ax^{k}+By^{k}-b\right) . \end{aligned}$$

  (8.4)

  By (8.3), (8.4) and (1.16), we get

  $$\begin{aligned}&\langle A^T({\uplambda }^k-\hat{{\uplambda }}),x^k-x^{k+1}\rangle - \langle (1-\gamma )\beta A^T(Ax^{k}+By^{k}-b),x^k-x^{k+1}\rangle \nonumber \\&\quad \ge \nu _f\Vert x^k-x^{k+1}\Vert ^2, \end{aligned}$$

  (8.5)

  to which applying the Cauchy–Schwarz inequality gives

  $$\begin{aligned}&({\uplambda }^k-\hat{{\uplambda }})^T A(x^k-x^{k+1})\nonumber \\&\quad \ge \langle \sqrt{\beta } (Ax^{k}+By^{k}-b), (1-\gamma )\sqrt{\beta }A(x^k-x^{k+1})\rangle +\nu _f\Vert x^k-x^{k+1}\Vert ^2\nonumber \\&\quad \ge -\frac{\beta }{2\rho }\Vert Ax^{k}+By^{k}-b\Vert ^2-\frac{(1-\gamma )^2\beta \rho }{2}\Vert A(x^k-x^{k+1})\Vert ^2\nonumber \\&\qquad +\nu _f\Vert x^k-x^{k+1}\Vert ^2,\quad \forall \rho >0. \end{aligned}$$

  (8.6)

  Substituting (8.6) into (2.11) and using $\hat{P}=P+\beta A^T A=\beta A^T A$ and the definition of $\hat{{\uplambda }}$, we have

  $$\begin{aligned}&\Vert u^k-u^*\Vert _G^2+\frac{\beta }{\rho }\Vert Ax^{k}+By^{k}-b\Vert ^2 \nonumber \\&\quad \ge ~\Vert u^{k+1}-u^*\Vert _G^2+\frac{\beta }{\rho }\Vert Ax^{k+1}+By^{k+1}-b\Vert ^2\nonumber \\&\qquad +\,\beta \left( 2-\gamma -\frac{1}{\rho }\right) \Vert Ax^{k+1} +By^{k+1}-b\Vert ^2 +\beta \left( 1-(1-\gamma )^2\rho \right) \Vert A(x^k-x^{k+1})\Vert ^2\nonumber \\&\qquad +\,\Vert y^k-y^{k+1}\Vert ^2_Q+2\nu _f\Vert x^k-x^{k+1}\Vert ^2+2\nu _f\Vert x^{k+1}-x^*\Vert ^2+2\nu _g\Vert y^{k+1}-y^*\Vert ^2. \end{aligned}$$

  (8.7)

  To prove such $\eta >0$ exists for (2.16), we only need the existence of $\rho >0$ such that $2-\gamma -\frac{1}{\rho }>0$ and $1-(1-\gamma )^2\rho >0$, which holds if and only if $2-\gamma >(1-\gamma )^2$ or, equivalently, $\gamma \in (0,\frac{1+\sqrt{5}}{2})$. In this case of $P=\mathbf {0}$, if we set $\gamma =1$, (8.6) reduces to $({\uplambda }^k-\hat{{\uplambda }})^T A(x^k-x^{k+1})\ge \nu _f\Vert x^k-x^{k+1}\Vert ^2$, which substituting into (2.11) gives (2.17) with $\eta = 1$.