Linear Convergence Rates for Variants of the Alternating Direction Method of Multipliers in Smooth Cases

Abstract

In the present paper, we propose a novel convergence analysis of the alternating direction method of multipliers, based on its equivalence with the overrelaxed primal–dual hybrid gradient algorithm. We consider the smooth case, where the objective function can be decomposed into one differentiable with Lipschitz continuous gradient part and one strongly convex part. Under these hypotheses, a convergence proof with an optimal parameter choice is given for the primal–dual method, which leads to convergence results for the alternating direction method of multipliers. An accelerated variant of the latter, based on a parameter relaxation, is also proposed, which is shown to converge linearly with same asymptotic rate as the primal–dual algorithm.

Appendices

Appendix A: Proof of Theorem 3.1 and of Corollary 3.1

We proceed analogously to [7, Section 5.2], but we do not specify any parameter unless needed. This proof is also inspired by the one found in [9], which does not allow $\theta \ne 1$. For now, we only assume that $0<\theta \le 1$.

We suppose that $\mathcal {L}$ satisfy Assumption (S2). Let $(y_n,\xi _n,\bar{\xi }_n)_{n}$ a sequence generated by Algorithm 1. For any $N\ge 1$, points $\xi _{n+1}$ and $y_{n+1}$ can be seen as minimizers of convex problem, thus, the first-order optimality condition yields, respectively, $-(\xi _{n+1}-\xi _n)/\tau - K^*y_{n+1} \in \partial G(\xi _{n+1})$ and $-(y_{n+1}-y_n)/\sigma + K\bar{\xi }_n \in \partial H^*(y_{n+1})$. Using the definition of strong convexity, we get for any $(\xi ,y)\in Z\times Y$ (after expanding the scalar products and summing the two inequalities for G and $H^*$)

$$\begin{aligned} \mathcal {G}(\xi _{n+1},\xi ;y,y_{n+1})\le & {} \frac{\Vert \xi -\xi _n\Vert ^2}{2\tau } - (1+\tau \tilde{\gamma })\frac{\Vert \xi -\xi _{n+1}\Vert ^2}{2\tau } - \frac{\Vert \xi _n-\xi _{n+1}\Vert ^2}{2\tau }\quad \nonumber \\&+\, \frac{\Vert y-y_n\Vert ^2}{2\sigma } - (1+\sigma \tilde{\delta })\,\frac{\Vert y-y_{n+1}\Vert ^2}{2\sigma } - \frac{\Vert y_n-y_{n+1}\Vert ^2}{2\sigma }\nonumber \\&+\langle K(\xi _{n+1}-\xi ),y_{n+1}-y_n\rangle -\langle K(\xi _{n+1}-\bar{\xi }_n),y_{n+1}-y\rangle .\qquad \qquad \end{aligned}$$

(65)

For any $N\ge 1$ and $(\xi ,y)\in Z\times Y$, we set $\Delta _n = \Vert \xi -\xi _n\Vert ^2/(2\tau ) + \Vert y-y_n\Vert ^2/(2\sigma )$. Let us first prove the following lemma:

Lemma A.1

Let $\mathcal {L}$ satisfy Assumption (S2) and $(y_n,\xi _n,\bar{\xi }_n)_{n}$ a sequence generated by Algorithm 1. Then, for any integer $N\ge 1$, $\tau ,\sigma >0$ and $0<\omega \le \theta$, we have for any $(\xi ,y)\in Z\times Y$

$$\begin{aligned} \mathcal {G}(\xi _n,\xi ;y,y_n) \le \Delta _n -\frac{\Delta _{n+1}}{\omega } +\omega \,\frac{\Vert \xi _{n-1}-\xi _n\Vert ^2}{2\tau } +\frac{\Vert \xi _n-\xi _{n+1}\Vert ^2}{2\tau } \\ +\omega \,\langle K(\xi _{n-1}-\xi _n),y-y_n\rangle -\langle K(\xi _n-\xi _{n+1}),y-y_{n+1}\rangle .\nonumber \end{aligned}$$

(66)

Proof

Replacing $\bar{\xi }_n$ by $\xi _n+\theta \,(\xi _n-\xi _{n-1})$ in (65), we get after simplification

$$\begin{aligned} \mathcal {G}(\xi _{n+1},\xi ;y,y_{n+1})&\le \frac{\Vert \xi -\xi _n\Vert ^2}{2\tau } - (1+\tau \tilde{\gamma })\,\frac{\Vert \xi -\xi _{n+1}\Vert ^2}{2\tau } - \frac{\Vert \xi _n-\xi _{n+1}\Vert ^2}{2\tau }\quad \nonumber \\&\quad + \frac{\Vert y-y_n\Vert ^2}{2\sigma } - (1+\sigma \tilde{\delta })\,\frac{\Vert y-y_{n+1}\Vert ^2}{2\sigma } - \frac{\Vert y_n-y_{n+1}\Vert ^2}{2\sigma }\nonumber \\&\quad +\theta \,\langle K(\xi _{n-1}-\xi _n),y-y_{n+1}\rangle -\langle K(\xi _n-\xi _{n+1}),y-y_{n+1}\rangle .\nonumber \\ \end{aligned}$$

(67)

Now, we define $\tau \tilde{\gamma }= \mu >0$ and $\sigma \tilde{\delta }= \mu '>0$. Let us bound the scalar products in (67). Introducing $0<\omega \le \theta$, we have

$$\begin{aligned} \theta \,\langle K(\xi _{n-1}-\xi _n),y-y_{n+1}\rangle&= \omega \,\langle K(\xi _{n-1}-\xi _n),y-y_n\rangle \nonumber \\&\quad +\,\omega \,\langle K(\xi _{n-1}-\xi _n),y_n-y_{n+1}\rangle \end{aligned}$$

(68)

Let us have a closer look at the last two terms. Let $\alpha >0$. Since $\langle K\Xi ,Y\rangle \le L_K(\alpha \Vert \Xi \Vert ^2+\Vert Y\Vert ^2/\alpha )/2$, the majoration (67) becomes, after simplification,

$$\begin{aligned}&\mathcal {G}(\xi _{n+1},\xi ;y,y_{n+1}) \le \Delta _n - (1+\mu )\,\Delta _{n+1} +\left( \frac{\omega L_K\sigma }{\alpha } - 1\right) \frac{\Vert y_n-y_{n+1}\Vert ^2}{2\sigma }\nonumber \\&\quad +\,\!\left( \!\!\frac{(\theta -\omega )L_K\sigma }{\alpha }+\mu -\mu '\!\right) \!\!\frac{\Vert y-y_{n+1}\Vert ^2}{2\sigma } +\theta L_K\alpha \tau \frac{\Vert \xi _{n-1}-\xi _n\Vert ^2}{2\tau } - \frac{\Vert \xi _n-\xi _{n+1}\Vert ^2}{2\tau }\nonumber \\&\quad +\,\omega \,\langle K(\xi _{n-1}-\xi _n),y-y_n\rangle -\langle K(\xi _n-\xi _{n+1}),y-y_{n+1}\rangle .\qquad \end{aligned}$$

(69)

Choose $\alpha = \omega L_K\sigma$. Hence, $\omega L_K/\alpha =1/\sigma$, so that the $\Vert y_n-y_{n+1}\Vert ^2$ term cancels. Since $1+\mu = 1/\omega + 1+\mu -1/\omega$, one gets

$$\begin{aligned}&\mathcal {G}(\xi _{n+1},\xi ;y,y_{n+1}) \le \Delta _n-\frac{\Delta _{n+1}}{\omega } +\omega \theta L_K^2\tau \sigma \frac{\Vert \xi _n-\xi _{n-1}\Vert ^2}{2\tau } - \frac{\Vert \xi _n-\xi _{n+1}\Vert ^2}{2\tau }\quad \nonumber \\&\quad +\, \left( \frac{1}{\omega }-\mu -1\right) \frac{\Vert \xi -\xi _{n+1} \Vert ^2}{2\tau } + {\left( \frac{\theta -\omega }{\omega }+\frac{1}{\omega }-\mu '-1\right) }\frac{\Vert y-y_{n+1} \Vert ^2}{2\sigma }\nonumber \\&\quad +\,\omega \,\langle K(\xi _{n-1}-\xi _n),y-y_n\rangle -\langle K(\xi _n-\xi _{n+1}),y-y_{n+1}\rangle . \end{aligned}$$

(70)

We can now set conditions on $\omega$, $\theta$, $\tau$ and $\sigma$. First, choose $\theta$, $\tau$ and $\sigma$ so that $\theta L_K^2\tau \sigma \le 1$. Then, choose $\theta$ so that both the parentheses are nonpositive, which implies that $1/(\mu +1)\le \omega$ and $(\theta +1)/(\mu '+2) \le \omega$. Since $\omega \le \theta$, we get the wanted inequality. $\square$

Let us now prove Corollary 3.1 and then Theorem 3.1. Set $\xi ^{-1} = \xi ^0$. Multiplying (15) by $1/\omega ^n$ and summing between $n=0$ and $n=N-1$ yield

$$\begin{aligned} \sum _{n=1}^N\frac{1}{\omega ^{n-1}}\,\mathcal {G}(\xi _n,\xi ;y,y_n)&\le \Delta _0-\frac{\Delta _N}{\omega ^N} - \frac{1}{\omega ^{N-1}}\,\frac{\Vert \xi _{N-1}-\xi _N\Vert ^2}{2\tau }\nonumber \\&\quad -\frac{1}{\omega ^{N-1}}\,\langle K(\xi _{N-1}-\xi _N),y-y_N\rangle . \end{aligned}$$

(71)

One has, for any $\beta >0$,

$$\begin{aligned} -\langle K(\xi _{N-1}-\xi _N),y-y_N\rangle \le L_K \left( \frac{\beta }{2}\,\Vert \xi _{N-1}-\xi _N \Vert ^2 +\frac{1}{2\beta }\,\Vert y-y_N\Vert ^2\right) \end{aligned}$$

(72)

and thus the right-hand side of (71) is bounded from above by

$$\begin{aligned} \Delta _0-\frac{1}{\omega ^N}\,\Delta _N + \frac{1}{\omega ^{N-1}}(L_K\beta \tau - 1)\,\frac{\Vert \xi _{N-1}-\xi _N\Vert ^2}{2\tau } + \frac{L_K}{\omega ^{N-1}}\frac{1}{2\beta }\,\Vert y_N-y\Vert ^2. \end{aligned}$$

(73)

Choosing $\beta = 1/(L_K\tau )$, which cancels the $\Vert \xi _{N-1}-\xi _N\Vert ^2$ term, we have, since $\omega L_K^2\tau \sigma \le \theta L_K^2\tau \sigma \le 1$ and $\mathcal {G}(\xi _n,\xi ^*;y^*,y_n)\ge 0$ for any $N\ge 1$,

$$\begin{aligned} 0&\le \frac{\Vert \xi ^*-\xi _N \Vert ^2}{2\tau } +(1-\omega L_K^2\tau \sigma )\frac{\Vert y^*-y_N\Vert ^2}{2\sigma } +\sum _{n=1}^N\frac{\omega ^N}{\omega ^{n-1}}\,\mathcal {G}(\xi _n,\xi ^*;y^*,y_n) \nonumber \\&\le \omega ^N\left( \frac{\Vert \xi ^*-\xi _0 \Vert ^2}{2\tau } + \frac{\Vert y^*-y_0 \Vert ^2}{2\sigma }\right) . \end{aligned}$$

(74)

This inequality proves the linear convergence of the iterates (Corollary 3.1). We can now complete the proof of Theorem 3.1. Now, dividing (74) by $\omega ^NT_N$ and using convexity gives Theorem 3.1.$\square$

Appendix B: Equivalence Between Algorithm 4 and Algorithm 2

Let us prove that the iterations of Algorithm 4 are equivalent to (41). Let $n\ge 0$. By optimality in (37), one has for any $x\in X$

$$\begin{aligned} g(x_{n+1}) + \langle Ax_{n+1},y_n\rangle + \frac{\Vert Ax_{n+1}-z_n\Vert ^2}{2\tau } \le g(x) + \langle Ax,y_n\rangle + \frac{\Vert Ax-z_n\Vert ^2}{2\tau } \end{aligned}$$

(75)

For any $\xi \in Y$, we define

$$\begin{aligned} g_A(\xi ):=\left\{ \begin{array}{ll} \inf \{g(x) : Ax=\xi \}, &{} \quad \mathrm {if}\,\{x\in X : Ax=\xi \}\ne \emptyset ,\\ +\infty , &{} \quad \mathrm {otherwise}. \end{array}\right. \end{aligned}$$

(76)

Let $\xi \in Y$. Suppose that $\{x\in X : Ax=\xi \}\ne \emptyset$. Then, taking the infimum over $\{x\in X : Ax=\xi \}$, we have

$$\begin{aligned} g(x_{n+1}) + \langle Ax_{n+1},y_n\rangle + \frac{\Vert Ax_{n+1}-z_n\Vert ^2}{2\tau } \le g_A(\xi ) + \langle \xi ,y_n\rangle + \frac{\Vert \xi -z_n\Vert ^2}{2\tau }. \end{aligned}$$

(77)

Let us define $\xi _{n+1}:=Ax_{n+1}$. Since $g_A(\xi _{n+1})\le g(x_{n+1})$, one has

$$\begin{aligned} g_A(\xi _{n+1}) + \langle \xi _{n+1},y_n\rangle + \frac{\Vert \xi _{n+1}-z_n\Vert ^2}{2\tau } \le g_A(\xi ) + \langle \xi ,y_n\rangle + \frac{\Vert \xi -z_n\Vert ^2}{2\tau } \end{aligned}$$

(78)

for any $\xi \in Y$, including those such that $\{x\in X : Ax=\xi \}$ is empty. Thus

$$\begin{aligned} g_A(\xi _{n+1}) + \langle \xi _{n+1},y_n\rangle + \frac{\Vert \xi _{n+1}-z_n\Vert ^2}{2\tau } = \min _{\xi \in Y}\left\{ g_A(\xi ) + \langle \xi ,y_n\rangle + \frac{\Vert \xi -z_n\Vert ^2}{2\tau }\right\} . \end{aligned}$$

(79)

In particular, one can check that $\xi _{n+1} = \mathrm {prox}_{\tau g_A}(z_n-\tau \,y_n)$. Let us write the y-update (39). We have

$$\begin{aligned} y_{n+1} = y_n + \frac{\xi _{n+1} - z_{n+1}}{\tau '}&\Longleftrightarrow z_{n+1} = \xi _{n+1} - \tau '\,(y_{n+1}-y_n)\end{aligned}$$

(80)

$$\begin{aligned}&\Longleftrightarrow z_{n+1}+\tau '\,y_{n+1} = \xi _{n+1} + \tau '\,y_n. \end{aligned}$$

(81)

Let us define $\bar{y}_{n+1}$ as in (30). Equation (80) implies that, for any $n\ge 1$, one has $z_n-\tau \,y_n = \xi _n - \tau '\,(y_n-y_{n-1})-\tau \,y_n = \xi _n-\tau \,(y_n + (\tau '/\tau )\,(y_n-y_{n-1}))$. Injecting the latter equality in the expression of $\xi _{n+1}$ yields (28). The z-update (38) can be rewritten as a proximal step

$$\begin{aligned} z_{n+1} = \mathrm {prox}_{\tau ' h}(\xi _{n+1} + \tau '\,y_n) = \mathrm {prox}_{\tau ' h}(z_{n+1}+\tau '\,y_{n+1}). \end{aligned}$$

(82)

Using Moreau’s decomposition [18], we eventually obtain (29).