Parallel Multi-Block ADMM with o(1 / k) Convergence

Abstract

This paper introduces a parallel and distributed algorithm for solving the following minimization problem with linear constraints:

$$\begin{aligned} \text {minimize} ~~&f_1(\mathbf{x}_1) + \cdots + f_N(\mathbf{x}_N)\\ \text {subject to}~~&A_1 \mathbf{x}_1 ~+ \cdots + A_N\mathbf{x}_N =c,\\&\mathbf{x}_1\in {\mathcal {X}}_1,~\ldots , ~\mathbf{x}_N\in {\mathcal {X}}_N, \end{aligned}$$

where $N \ge 2$, $f_i$ are convex functions, $A_i$ are matrices, and ${\mathcal {X}}_i$ are feasible sets for variable $\mathbf{x}_i$. Our algorithm extends the alternating direction method of multipliers (ADMM) and decomposes the original problem into N smaller subproblems and solves them in parallel at each iteration. This paper shows that the classic ADMM can be extended to the N-block Jacobi fashion and preserve convergence in the following two cases: (i) matrices $A_i$ are mutually near-orthogonal and have full column-rank, or (ii) proximal terms are added to the N subproblems (but without any assumption on matrices $A_i$). In the latter case, certain proximal terms can let the subproblem be solved in more flexible and efficient ways. We show that $\Vert {\mathbf {x}}^{k+1} - {\mathbf {x}}^k\Vert _M^2$ converges at a rate of o(1 / k) where M is a symmetric positive semi-definte matrix. Since the parameters used in the convergence analysis are conservative, we introduce a strategy for automatically tuning the parameters to substantially accelerate our algorithm in practice. We implemented our algorithm (for the case ii above) on Amazon EC2 and tested it on basis pursuit problems with >300 GB of distributed data. This is the first time that successfully solving a compressive sensing problem of such a large scale is reported.

Appendix: On o(1 / k) Convergence Rate of ADMM

The convergence of the standard two-block ADMM has been long established in the literature [14, 16]. Its convergence rate has been actively studied; see [9, 10, 12, 17, 23,24,25, 28] and the references therein. In the following, we briefly review the convergence analysis for ADMM ($N=2$) and then improve the O(1 / k) convergence rate established in [24] slightly to o(1 / k) by using the same technique as in Sect. 2.2.

As suggested in [24], the quantity $\Vert {\mathbf {w}}^k-{\mathbf {w}}^{k+1}\Vert _H^2$ can be used to measure the optimality of the iterations of ADMM , where

$$\begin{aligned} {\mathbf {w}}:=\begin{pmatrix} \mathbf{x}_2\\ \lambda \end{pmatrix}, ~H:=\begin{pmatrix} \rho A_2^\top A_2 &{}\quad \\ \quad &{}\frac{1}{\rho }{\mathbf {I}}\end{pmatrix}, \end{aligned}$$

and ${\mathbf {I}}$ is the identity matrix of size $m\times m$. Note that $\mathbf{x}_1$ is not part of ${\mathbf {w}}$ because $\mathbf{x}_1$ can be regarded as an intermediate variable in the iterations of ADMM, whereas $(\mathbf{x}_2,\lambda )$ are the essential variables [3]. In fact, if $\Vert {\mathbf {w}}^k-{\mathbf {w}}^{k+1}\Vert _H^2=0$ then ${\mathbf {w}}^{k+1}$ is optimal. The reasons are as follows. Recall the subproblems of ADMM:

$$\begin{aligned} \mathbf{x}_1^{k+1}&= \hbox {arg min}_{\mathbf{x}_1}~f_1(\mathbf{x}_1)+\frac{\rho }{2} \left\| A_1\mathbf{x}_1+A_2\mathbf{x}_2^k-\lambda ^k/\rho \right\| ^2, \end{aligned}$$

(6.1)

$$\begin{aligned} \mathbf{x}_2^{k+1}&= \hbox {arg min}_{\mathbf{x}_2}~f_2(\mathbf{x}_2)+\frac{\rho }{2}\left\| A_1\mathbf{x}^{k+1}_1+A_2 \mathbf{x}_2-\lambda ^k/\rho \right\| ^2. \end{aligned}$$

(6.2)

By the formula for $\lambda ^{k+1}$, their optimality conditions can be written as:

$$\begin{aligned} A_1^\top \lambda ^{k+1}-\rho A_1^\top A_2\left( \mathbf{x}_2^k-\mathbf{x}_2^{k+1}\right)&\in \partial f\left( \mathbf{x}_1^{k+1}\right) , \end{aligned}$$

(6.3)

$$\begin{aligned} A_2^\top \lambda ^{k+1}&\in \partial f_2\left( \mathbf{x}_2^{k+1}\right) . \end{aligned}$$

(6.4)

In comparison with the KKT conditions (1.14a) and (1.14b), we can see that $\mathbf{u}^{k+1}=\left( \mathbf{x}^{k+1}_1,\mathbf{x}^{k+1}_2,\lambda ^{k+1}\right)$ is a solution of (1.1) if and only if the following holds:

$$\begin{aligned}&{\mathbf {r}}^{k+1}_p:=A_1\mathbf{x}_1^{k+1}+A_2\mathbf{x}_2^{k+1}-c = 0 \quad \text{(primal } \text{ feasibility) }, \end{aligned}$$

(6.5)

$$\begin{aligned}&{\mathbf {r}}^{k+1}_d:=\rho A_1^\top A_2\left( \mathbf{x}_2^k-\mathbf{x}_2^{k+1}\right) = 0 \quad \quad \text{(dual } \text{ feasibility) }. \end{aligned}$$

(6.6)

By the update formula for $\lambda ^{k+1}$, we can write ${\mathbf {r}}_p$ equivalently as

$$\begin{aligned} {\mathbf {r}}^{k+1}_p = \frac{1}{\rho }\left( \lambda ^k-\lambda ^{k+1}\right) . \end{aligned}$$

(6.7)

Clearly, if $\Vert {\mathbf {w}}^k-{\mathbf {w}}^{k+1}\Vert _H^2=0$ then the optimality conditions (6.5) and (6.6) are satisfied, so ${\mathbf {w}}^{k+1}$ is a solution. On the other hand, if $\Vert {\mathbf {w}}^k-{\mathbf {w}}^{k+1}\Vert _H^2$ is large, then ${\mathbf {w}}^{k+1}$ is likely to be far away from being a solution. Therefore, the quantity $\Vert {\mathbf {w}}^k-{\mathbf {w}}^{k+1}\Vert _H^2$ can be viewed as a measure of the distance between the iteration ${\mathbf {w}}^{k+1}$ and the solution set. Furthermore, based on the variational inequality (1.15) and the variational characterization of the iterations of ADMM, it is reasonable to use the quadratic term $\Vert {\mathbf {w}}^k-{\mathbf {w}}^{k+1}\Vert _H^2$ rather than $\Vert {\mathbf {w}}^k-{\mathbf {w}}^{k+1}\Vert _H$ to measure the convergence rate of ADMM (see [24] for more details).

The work [24] proves that $\Vert {\mathbf {w}}^k-{\mathbf {w}}^{k+1}\Vert _H^2$ converges to zero at a rate of O(1 / k). The key steps of the proof are to establish the following properties:

• the sequence $\{{\mathbf {w}}^k\}$ is contractive:

  $$\begin{aligned} \Vert {\mathbf {w}}^k-{\mathbf {w}}^*\Vert ^2_H-\Vert {\mathbf {w}}^{k+1}-{\mathbf {w}}^*\Vert ^2_H\ge \Vert {\mathbf {w}}^{k}-{\mathbf {w}}^{k+1}\Vert ^2_H, \end{aligned}$$

  (6.8)

• the sequence $\Vert {\mathbf {w}}^k-{\mathbf {w}}^{k+1}\Vert _H^2$ is monotonically non-increasing:

  $$\begin{aligned} \Vert {\mathbf {w}}^k-{\mathbf {w}}^{k+1}\Vert ^2_H\le \Vert {\mathbf {w}}^{k-1}-{\mathbf {w}}^k\Vert ^2_H. \end{aligned}$$

  (6.9)

The contraction property (6.8) has been long established and its proof dates back to [14, 16]. Inspired by [24], we provide a shorter proof for (6.9) than the one in [24].

Proof (Proof of (6.9))

Let $\Delta \mathbf{x}_i^{k+1}=\mathbf{x}_i^k-\mathbf{x}_i^{k+1}$ and $\Delta \lambda ^{k+1}=\lambda ^k-\lambda ^{k+1}$. By Lemma 1.1, i.e., (1.17), the optimality condition 6.3 at the k-th and $(k+1)$-th iterations yields: $\langle \Delta \mathbf{x}_1^{k+1},~A_1^\top \Delta \lambda ^{k+1}-\rho A_1^\top A_2(\Delta \mathbf{x}_2^k-\Delta \mathbf{x}_2^{k+1})\rangle \ge 0.$ Similarly for (6.4), we obtain $\langle \Delta \mathbf{x}_2^{k+1},~A_2^\top \Delta \lambda ^{k+1}\rangle \ge 0.$ Adding the above two inequalities together, we have

$$\begin{aligned} \left( A_1\Delta \mathbf{x}_1^{k+1}+A_2\Delta \mathbf{x}_2^{k+1}\right) ^\top \Delta \lambda ^{k+1}-\rho \left( A_1\Delta \mathbf{x}_1^{k+1}\right) ^\top A_2\left( \Delta \mathbf{x}_2^k-\Delta \mathbf{x}_2^{k+1}\right) \ge 0. \end{aligned}$$

(6.10)

Using the equality according to (6.7):

$$\begin{aligned} A_1\Delta \mathbf{x}_1^{k+1}+A_2\Delta \mathbf{x}_2^{k+1} = \frac{1}{\rho }\left( \Delta \lambda ^k-\Delta \lambda ^{k+1}\right) , \end{aligned}$$

(6.11)

(6.10) becomes $\frac{1}{\rho }\left( \Delta \lambda ^k-\Delta \lambda ^{k+1}\right) ^\top \Delta \lambda ^{k+1}- \left( \Delta \lambda ^k-\Delta \lambda ^{k+1}-\rho A_2\Delta \mathbf{x}_2^{k+1}\right) ^ \top A_2\left( \Delta \mathbf{x}_2^k\right. \left. -\Delta \mathbf{x}_2^{k+1}\right) \ge 0.$ After rearranging the terms, we get

$$\begin{aligned}&\left( \sqrt{\rho }A_2\Delta \mathbf{x}_2^k+\frac{1}{\sqrt{\rho }}\Delta \lambda ^k\right) ^ \top \left( \sqrt{\rho }A_2\Delta \mathbf{x}_2^{k+1}+\frac{1}{\sqrt{\rho }} \Delta \lambda ^{k+1}\right) -\left( A_2\Delta \mathbf{x}_2^k\right) ^\top \Delta \lambda ^k \nonumber \\&\quad -\left( A_2\Delta \mathbf{x}_2^{k+1}\right) ^\top \Delta \lambda ^{k+1} \nonumber \\&\quad \ge \frac{1}{\rho }\left\| \Delta \lambda ^{k+1}\right\| ^2+\rho \left\| A_2\Delta \mathbf{x}_2^{k+1}\right\| ^2 =\Vert {\mathbf {w}}^k-{\mathbf {w}}^{k+1}\Vert _H^2. \end{aligned}$$

(6.12)

By the Cauchy-Schwarz inequality, we have $(a_1+b_1)^\top (a_2+b_2)\le (\Vert a_1+b_1\Vert ^2+\Vert a_2+b_2\Vert ^2)/2,$ or equivalently, $(a_1+b_1)^\top (a_2+b_2)-a_1^\top b_1-a_2^\top b_2 \le (\Vert a_1\Vert ^2+\Vert b_1\Vert ^2+\Vert a_2\Vert ^2+\Vert b_2\Vert ^2)/2.$ Applying this inequality to the left-hand side of (6.12), we have

$$\begin{aligned} \Vert {\mathbf {w}}^k-{\mathbf {w}}^{k+1}\Vert _H^2\le & {} \left( \rho \Vert A_2\Delta \mathbf{x}_2^{k}\Vert ^2+\frac{1}{\rho }\Vert \Delta \lambda ^{k}\Vert ^2+\rho \Vert A_2\Delta \mathbf{x}_2^{k+1} \Vert ^2+\frac{1}{\rho }\Vert \Delta \lambda ^{k+1}\Vert ^2\right) /2\\= & {} \left( \Vert {\mathbf {w}}^{k-1}-{\mathbf {w}}^{k}\Vert _H^2+\Vert {\mathbf {w}}^k-{\mathbf {w}}^{k+1}\Vert _H^2\right) /2, \end{aligned}$$

and thus (6.9) follows immediately.

We are now ready to improve the convergence rate from O(1 / k) to o(1 / k).

Theorem 6.1

The sequence $\{{\mathbf {w}}^k\}$ generated by Algorithm 2 (for $N=2$) converges to a solution ${\mathbf {w}}^*$ of problem (1.1) in the H-norm, i.e., $\Vert {\mathbf {w}}^k-{\mathbf {w}}^*\Vert _H^2\rightarrow 0$, and $\Vert {\mathbf {w}}^k-{\mathbf {w}}^{k+1}\Vert ^2_H = o(1/k)$. Therefore,

$$\begin{aligned} \left\| A_1\mathbf{x}_1^k- A_1\mathbf{x}_1^{k+1}\right\| ^2 + \left\| A_2\mathbf{x}_2^k - A_2\mathbf{x}_2^{k+1}\right\| ^2 + \left\| \lambda ^k- \lambda ^{k+1}\right\| ^2 = o(1/k). \end{aligned}$$

(6.13)

Proof

Using the contractive property of the sequence $\{{\mathbf {w}}^k\}$ (6.8) along with the optimality conditions, the convergence of $\Vert {\mathbf {w}}^k-{\mathbf {w}}^*\Vert _H^2\rightarrow 0$ follows from the standard analysis for contraction methods [19].

By (6.8), we have

$$\begin{aligned} \sum _{k=1}^{n} \Vert {\mathbf {w}}^k-{\mathbf {w}}^{k+1}\Vert ^2_H \le \Vert {\mathbf {w}}^{1}-{\mathbf {w}}^*\Vert ^2_H - \Vert {\mathbf {w}}^{n+1}-{\mathbf {w}}^*\Vert ^2_H,~\forall n. \end{aligned}$$

(6.14)

Therefore, $\sum _{k=1}^{\infty } \Vert {\mathbf {w}}^k-{\mathbf {w}}^{k+1}\Vert ^2_H<\infty$. By (6.9), $\Vert {\mathbf {w}}^k-{\mathbf {w}}^{k+1}\Vert ^2_H$ is monotonically non-increasing and nonnegative. So Lemma 1.1 indicates that $\Vert {\mathbf {w}}^k-{\mathbf {w}}^{k+1}\Vert ^2_H = o(1/k)$, which further implies that $\Vert A_2\mathbf{x}_2^k - A_2\mathbf{x}_2^{k+1}\Vert ^2 = o(1/k)$ and $\Vert \lambda ^k- \lambda ^{k+1}\Vert ^2 = o(1/k)$. By (6.11), we also have $\Vert A_1\mathbf{x}_1^k- A_1\mathbf{x}_1^{k+1}\Vert ^2= o(1/k)$. Thus (6.13) follows immediately. $\square$

Remark 6.1

The proof technique based on Lemma 1.1 can be applied to improve some other existing convergence rates of O(1 / k) (e.g., [8, 21]) to o(1 / k) as well.