Pointwise and Ergodic Convergence Rates of a Variable Metric Proximal Alternating Direction Method of Multipliers

Abstract

In this paper, we obtain global pointwise and ergodic convergence rates for a variable metric proximal alternating direction method of multipliers for solving linearly constrained convex optimization problems. We first propose and study nonasymptotic convergence rates of a variable metric hybrid proximal extragradient framework for solving monotone inclusions. Then, the convergence rates for the former method are obtained essentially by showing that it falls within the latter framework. To the best of our knowledge, this is the first time that global pointwise (resp. pointwise and ergodic) convergence rates are obtained for the variable metric proximal alternating direction method of multipliers (resp. variable metric hybrid proximal extragradient framework).

Appendix A Proofs of Theorems 3.1 and 3.2

We start by presenting the following two Lemmas.

Lemma A.1

For any $z^*,z,z_+,\tilde{z}\in \mathscr {Z}$ and $M\in \mathscr {M}^{\mathscr {Z}}_{+}$, we have

$$\begin{aligned}&\Vert z^*-z\Vert _{\mathscr {Z},\,M}^2 - \Vert z^*-z_+\Vert _{\mathscr {Z},\,M}^2 = \Vert z-\tilde{z}\Vert _{\mathscr {Z},M}^2\\&\quad -\Vert z_+-\tilde{z}\Vert _{\mathscr {Z},M}^2 +2\langle \tilde{z}-z^*, M(z-z_+) \rangle _{\mathscr {Z}}. \end{aligned}$$

Proof

Direct calculations yield

$$\begin{aligned} \Vert z^*-z\Vert _{\mathscr {Z},M}^2-\Vert z^*-z_+\Vert _{\mathscr {Z},M}^2&=2\langle z_+-z^*, M(z-z_+)\rangle _{\mathscr {Z}}+ \Vert z_+-z\Vert _{\mathscr {Z},\,M}^2\\&= 2\langle z_+-\tilde{z}, M(z-z_+)\rangle _{\mathscr {Z}}\\&\quad +\,2\langle \tilde{z}-z^*, M(z-z_+)\rangle _{\mathscr {Z}}+ \Vert z_+-z\Vert _{\mathscr {Z},M}^2 \\&= 2\langle \tilde{z}-z^*, M(z-z_+)\rangle _{\mathscr {Z}}\\&\quad + \,\Vert \tilde{z}-z\Vert _{\mathscr {Z},M}^2- \Vert \tilde{z}-z_{+}\Vert _{\mathscr {Z},M}^2.\; \end{aligned}$$

$\square$

Lemma A.2

Let $\{z_k\}$, $\{M_k\}$, $\{\tilde{z}_k\}$ and $\{\eta _k\}$ be generated by the variable metric HPE framework. For every $k \ge 1$ and $z^* \in T^{-1}(0):$

• (a) we have

  $$\begin{aligned} \Vert z^*-z_{k}\Vert _{\mathscr {Z},M_k}^2\le \Vert z^*-z_{k-1}\Vert _{\mathscr {Z},M_k}^2 +\eta _{k-1}-\eta _{k}-(1 - \sigma ) \Vert z_{k-1}-\tilde{z}_k\Vert _{\mathscr {Z},M_k}^2; \end{aligned}$$

• (b) we have

  $$\begin{aligned}&\Vert z^*-z_{k}\Vert _{\mathscr {Z},M_k}^2+\eta _k+(1-\sigma ) \displaystyle \sum _{i=1}^k\Vert z_{i-1}-\tilde{z}_i\Vert _{\mathscr {Z},M_i}^2 \\&\quad \le C_P (\Vert z^*-z_{0}\Vert _{\mathscr {Z},M_0}^2 + \eta _{0})\,, \end{aligned}$$

  where $C_P$ and $M_0$ are as in (11) and condition C1, respectively.

Proof

• (a) From Lemma A.1 with $(z,z_+,\tilde{z})=(z_{k-1},z_k,\tilde{z}_k)$ and $M=M_k$, (12) and (13), we obtain

  $$\begin{aligned}&\Vert z^*-z_{k-1}\Vert _{\mathscr {Z},M_k}^2 - \Vert z^*-z_{k}\Vert _{\mathscr {Z},M_k}^2 +\eta _{k-1}\\&\quad \ge (1-\sigma ) \Vert z_{k-1}-\tilde{z}_k\Vert _{\mathscr {Z},M_k}^2+\eta _k + 2\langle \tilde{z}_k-z^*, r_k\rangle . \end{aligned}$$

  Hence, (a) follows from the above inequality, the fact that $0 \in T(z^*)$ and $r_k \in T(\tilde{z}_k)$ (see (12)), and the monotonicity of T.

• (b) Using (a), (3) and condition C1, we find

  $$\begin{aligned}&\Vert z^*-z_{k}\Vert _{\mathscr {Z},M_k}^2 \le (1+c_{k-1})\Vert z^*-z_{k-1}\Vert _{\mathscr {Z},\,M_{k-1}}^2\\&\quad +\,\eta _{k-1}-\eta _{k}-(1 - \sigma ) \Vert z_{k-1}-\tilde{z}_k\Vert _{\mathscr {Z},\,M_k}^2. \end{aligned}$$

  Thus, the result follows by applying the above inequality recursively and by using (11).$\square$

We are now ready to prove Theorem 3.1.

Proof of Theorem 3.1:

First, note that the desired inclusion holds due to (12). Now, using (2) and (13), we obtain, respectively,

$$\begin{aligned} \begin{aligned}&\Vert {z}_{k-1}-z_k\Vert _{\mathscr {Z},M_k}^2\le 2\left( \Vert {z_{k-1}}- {\tilde{z}}_k\Vert _{\mathscr {Z},M_k}^2+\Vert {\tilde{z}}_k-z_k\Vert _{\mathscr {Z},M_k}^2\right) ,\\&\Vert {\tilde{z}}_k-z_k\Vert _{\mathscr {Z},M_k}^2 \le \sigma \Vert {z_{k-1}}-\tilde{z}_{k}\Vert _{\mathscr {Z},M_k}^2+\eta _{k-1}-\eta _k. \end{aligned} \end{aligned}$$

Combining the above inequalities, we find

$$\begin{aligned} \Vert {z_{k-1}}- {z}_k\Vert _{\mathscr {Z},M_k}^2\le 2\left[ (1+\sigma ) \Vert {z_{k-1}}- {\tilde{z}}_k\Vert _{\mathscr {Z},M_k}^2+\eta _{k-1}-\eta _k\right] , \end{aligned}$$

which in turn, combined with Lemma A.2(b), yields

$$\begin{aligned} \sum _{i=1}^k \Vert {z_{i-1}}- {z}_i\Vert _{\mathscr {Z},M_i}^2 \le \frac{2(1+\sigma )C_P (\Vert z^*-z_{0}\Vert _{\mathscr {Z},M_0}^2 +\eta _{0})+ 2(1-\sigma )\eta _{0}}{ (1 - \sigma )}, \end{aligned}$$

(68)

for all $z^*\in T^{-1}(0)$. Now, from (11), we obtain $M_i\preceq C_P M_0$ for every $i\ge 1$. Thus, it follows from (12) and Proposition 2.1 that

$$\begin{aligned} \sum _{i=1}^k \Vert r_i\Vert ^2_{\mathscr {Z}}= \sum _{i=1}^k\Vert M_i(z_{i-1}-z_i)\Vert ^2_{\mathscr {Z}}\le {C_P\Vert M_0\Vert } \sum _{i=1}^k\Vert {z_{i-1}}- {z}_i\Vert ^2_{\mathscr {Z},M_i}, \end{aligned}$$

which, combined with the fact that $\sum _{i=1}^k\,t_i\ge k\min _{i=1,\dots , k} \{t_i\}$ and the definition in (14), proves (15). $\square$

Before proceeding to the proof of the ergodic convergence of the variable metric HPE framework, let us first present an auxiliary result.

Proposition A.1

Let $\{z_k\}$, $\{M_k\}$ and $\{\eta _k\}$ be generated by the variable metric HPE framework and consider $\{\tilde{z}_k^a\}$ and $\{\varepsilon _k^a\}$ as in (18). Then, for every $k\ge 1$,

$$\begin{aligned} \varepsilon _k^a \le \frac{1}{2k}\left( \eta _{0}+\Vert \tilde{z}^a_{k}-z_{0}\Vert _{\mathscr {Z},M_{0}}^2+ \sum _{i=1}^{k}c_{i-1}\Vert \tilde{z}^a_{k}-z_{i-1}\Vert _{\mathscr {Z},M_{i-1}}^2\right) , \end{aligned}$$

(69)

where $\{c_k\}$ is given in condition C1.

Proof

Using Lemma A.1 with $(z^*,z,z_+,\tilde{z})=(\tilde{z}^a_{k},z_{i-1},z_i,\tilde{z}_i)$ and $M=M_i$, (12) and (13), we find, for every $i=1,\dots ,k$,

$$\begin{aligned}&\Vert \tilde{z}^a_{k}-z_{i-1}\Vert _{\mathscr {Z},M_i}^2-\Vert \tilde{z}^a_{k}-z_{i}\Vert _{\mathscr {Z},M_i}^2 +\eta _{i-1}\\&\quad \ge (1-\sigma )\Vert \tilde{z}_i-z_{i-1}\Vert _{\mathscr {Z},M_i}^2+ \eta _i+2\langle r_i, \tilde{z}_i - \tilde{z}^a_{k}\rangle \\&\quad \ge \eta _i+2\langle r_i,\tilde{z}_i-\tilde{z}^a_{k}\rangle , \end{aligned}$$

where the second inequality is due to the fact that $1-\sigma \ge 0$. Hence, using condition C1 and simple calculations, we obtain

$$\begin{aligned} \Vert \tilde{z}^a_{k}-z_{i}\Vert _{\mathscr {Z},M_i}^2\le & {} (1+c_{i-1})\Vert \tilde{z}^a_{k}-z_{i-1}\Vert _{\mathscr {Z},M_{i-1}}^2\\&+\,\eta _{i-1}-\eta _{i}- 2\langle r_i, \tilde{z}_i -\tilde{z}^a_{k}\rangle \quad \forall i=1,\ldots ,k. \end{aligned}$$

Summing up the last inequality from $i=1$ to $i=k$ and using the definition of $\varepsilon _k^a$ in (18), we have

$$\begin{aligned} 0\le \Vert \tilde{z}^a_{k}-z_k\Vert _{\mathscr {Z},M_k}^2\le & {} \sum _{i=1}^{k}c_{i-1}\Vert \tilde{z}^a_{k}-z_{i-1}\Vert _{\mathscr {Z},M_{i-1}}^2\\&+\,\Vert \tilde{z}^a_{k}-z_{0}\Vert _{\mathscr {Z},M_{0}}^2 +\eta _{0}-2 k\,\varepsilon _k^a, \end{aligned}$$

which clearly gives (69). $\square$

Proof of Theorem 3.2:

Note first that the desired inclusion and the first inequality in (20) follow from (12), (18) and Theorem 2.1(a). Take $z^*\in T^{-1}(0)$. Now, let us prove the second inequality in (20), which will follow by bounding the term in the right-hand side of (69). Note that, using the convexity of $\Vert \cdot \Vert _{M_{i-1}}^2$, inequality (2) and (18), we find

$$\begin{aligned} \Vert \tilde{z}_k^a-z_{i-1}\Vert _{\mathscr {Z},M_{i-1}}^2\le & {} \frac{1}{k}\sum _{j=1}^{k} \Vert \tilde{z}_j-z_{i-1}\Vert _{\mathscr {Z},M_{i-1}}^2\nonumber \\\le & {} \frac{2}{k} \sum _{j=1}^{k}\left( \Vert \tilde{z}_j-z_j\Vert _{\mathscr {Z},M_{i-1}}^2+ \Vert z_j-z_{i-1}\Vert _{\mathscr {Z},M_{i-1}}^2\right) . \end{aligned}$$

(70)

From (11), we have $M_{i-1}\preceq C_PM_j$ for all $j=1,\ldots , k$. Hence, using Proposition 2.1, inequality (13), Lemma A.2(b) and (14), we find

$$\begin{aligned} \sum _{j=1}^k\,\Vert \tilde{z}_j-z_j\Vert _{\mathscr {Z},M_{i-1}}^2&\le C_P\sum _{j=1}^k\,\Vert \tilde{z}_j-z_j\Vert _{\mathscr {Z},M_{j}}^2 \nonumber \\&\le {C_P}\sum _{j=1}^k\,\left( \sigma \Vert \tilde{z}_j-z_{j-1}\Vert ^2_{\mathscr {Z},M_j}+\eta _{j-1}-\eta _j\right) \nonumber \\&\le \dfrac{\sigma }{1-\sigma }C_p^2(d_0^2+\eta _0)+C_P\eta _0. \end{aligned}$$

(71)

On the other hand, using (2), $M_{i-1}\preceq C_P M_j$ for all $j=1,\ldots , k$, Proposition 2.1, Lemma A.2(b) and (14), we obtain

$$\begin{aligned} \sum _{j=1}^k\,\Vert z_j-z_{i-1}\Vert ^2_{\mathscr {Z},M_{i-1}}&\le 2 \sum _{j=1}^k\,\left( \Vert z_j-z^*\Vert _{\mathscr {Z},M_{i-1}}^2+\Vert z^*-z_{i-1}\Vert _{\mathscr {Z},M_{i-1}}^2\right) \nonumber \\&\le 2 \sum _{j=1}^k\,\left( C_P\Vert z_j-z^*\Vert _{\mathscr {Z},M_{j}}^2+ \Vert z^*-z_{i-1}\Vert _{\mathscr {Z},M_{i-1}}^2\right) \nonumber \\&\le 2(1+C_P)C_P(d_0^2+\eta _0)k. \end{aligned}$$

(72)

It follows from inequalities (70)–(72) and the fact that $k\ge 1$ that

$$\begin{aligned} \Vert \tilde{z}_k^a-z_{i-1}\Vert _{\mathscr {Z},M_{i-1}}^2\le \left( \frac{\sigma C_P}{1-\sigma }+2(1+C_P)\right) 2C_P(d_0^2+\eta _0)+2C_P\eta _0, \end{aligned}$$

which, combined with Proposition A.1 and the first condition in (10), yields

$$\begin{aligned} \varepsilon ^a_{k}\le & {} \frac{1}{2k}\left[ 2C_P(1+C_S)\left( \frac{\sigma C_P}{1-\sigma }+2(1+C_P)\right) (d_0^2+\eta _0)\right. \\&\left. +\left( 1+2(1+C_S)C_P\right) \eta _0\right] . \end{aligned}$$

Therefore, the second inequality in (20) now follows from definition of $\widehat{\mathscr {E}}$ and simple calculations.

To finish the proof of the theorem, it remains to prove (19). Assume first that $k\ge 2$. Using (18) and simple calculations, we have

$$\begin{aligned} k\,r^a_{k}= & {} \sum _{i=1}^k\,r_i=M_1(z_0-z^*)-M_k(z_k-z^*)\nonumber \\&+\sum _{i=1}^{k-1} (M_{i+1}-M_i)(z_i-z^*). \end{aligned}$$

(73)

Since $M_k\preceq C_P M_0$ and $M_1\preceq C_P M_0$ (see (11)), we obtain from Proposition 2.1 that

$$\begin{aligned} \Vert M_k(z_k-z^*)\Vert _{\mathscr {Z}}&\le \sqrt{C_P\Vert M_0\Vert }\Vert z_k-z^*\Vert _{\mathscr {Z}, M_k}, \end{aligned}$$

(74)

$$\begin{aligned} \Vert M_1(z_0-z^*)\Vert _{\mathscr {Z}}&\le \sqrt{C_P\Vert M_0\Vert }\Vert z_0-z^*\Vert _{\mathscr {Z}, M_1}\nonumber \\&\le C_p\sqrt{\Vert M_0\Vert }\Vert z_0-z^*\Vert _{\mathscr {Z},M_0}. \end{aligned}$$

(75)

Next step is to estimate the general term in the summation in (73). To do this, first note that using condition C1, we find

$$\begin{aligned} 0\preceq L_i:=M_{i+1}-M_i + c_iM_{i+1}\preceq c_i (2+c_i)M_i,\quad \forall \, i=1,\ldots , k-1, \end{aligned}$$

(76)

and so

$$\begin{aligned} \Vert (M_{i+1}-M_i)(z_i-z^*)\Vert _{\mathscr {Z}}&=\Vert (L_i-c_iM_{i+1})(z_i-z^*)\Vert _{\mathscr {Z}}\nonumber \\&\le \Vert L_i(z_i-z^*)\Vert _{\mathscr {Z}}+c_i\Vert M_{i+1}(z_i-z^*)\Vert _{\mathscr {Z}}. \end{aligned}$$

(77)

It follows from the last inequality in (76) and (11) that $L_i\preceq c_i(2+c_i)M_i$ and $M_i\preceq C_P M_0$. Hence, we have

$$\begin{aligned} \Vert L_i(z_i-z^*)\Vert ^2_{\mathscr {Z}}&=\langle {L_i(L_i^{1/2}(z_i-z^*))},{L_i^{1/2}(z_i-z^*)}\rangle \nonumber \\&\le c_i(2+c_i)\langle {M_i(L_i^{1/2}(z_i-z^*))},{L_i^{1/2}(z_i-z^*)}\rangle \nonumber \\&\le c_i(2+c_i)C_P\langle {M_0(L_i^{1/2}(z_i-z^*))},{L_i^{1/2}(z_i-z^*)}\rangle \nonumber \\&\le c_i(2+c_i)C_P\Vert M_0 \Vert \Vert z_i-z^*\Vert ^2_{\mathscr {Z},L_i}\nonumber \\&\le c_i^2(2+c_i)^2C_P\Vert M_0 \Vert \Vert z_i-z^*\Vert ^2_{\mathscr {Z},M_i}. \end{aligned}$$

(78)

Again, using the facts that $M_{i+1}\preceq C_P M_0$ and $M_{i+1}\preceq (1+c_i)M_i$ (see (11)), and Proposition 2.1, we obtain

$$\begin{aligned} \Vert M_{i+1}(z_i-z^*)\Vert _{\mathscr {Z}}&\le \sqrt{C_p\Vert M_0\Vert }\Vert z_i-z^*\Vert _{\mathscr {Z}, M_{i+1}} \nonumber \\&\le \sqrt{C_P\Vert M_0\Vert (1+c_i)}\Vert z_i-z^*\Vert _{\mathscr {Z}, M_i}. \end{aligned}$$

(79)

Hence, using (11) and (77)–(79), we find

$$\begin{aligned} \Vert (M_{i+1}-M_i)(z_i-z^*)\Vert _{\mathscr {Z},M_k}&\le c_i\sqrt{C_p\Vert M_0\Vert }\left( 1+(1+c_i)+\sqrt{1+c_i}\,\right) \Vert z_i-z^*\Vert _{\mathscr {Z}, M_i}\nonumber \\&\le c_i\sqrt{C_P\Vert M_0\Vert }\left( 1+C_P+\sqrt{C_P}\right) \Vert z_i-z^*\Vert _{\mathscr {Z}, M_i}.\nonumber \\ \end{aligned}$$

(80)

Finally, using the definition of $d_0$ in (14), (73)–(75), (80) and Lemma A.2(b), we conclude that

$$\begin{aligned} k\Vert r^a_k\Vert _{\mathscr {Z}}&\le \Vert M_1(z_0-z^*)\Vert _{\mathscr {Z}}+\Vert M_k(z_k-z^*)\Vert _{\mathscr {Z}}\\&\quad +\sum _{i=1}^{k-1}\,\Vert (M_{i+1}-M_i)(z_i-z^*)\Vert _{\mathscr {Z}}\\&\le \left( C_P+\sqrt{C_P}+C_S\sqrt{C_P}\left( 1+C_P+\sqrt{C_P}\right) \right) \\&\quad \sqrt{\Vert M_0\Vert }\max _{i=0,\ldots ,k}\,\Vert z_i-z^*\Vert _{\mathscr {Z},M_i}\\&\le \sqrt{C_P\Vert M_0\Vert }\left( C_P+\sqrt{C_P}+C_S\sqrt{C_P}\left( 1+C_P+\sqrt{C_P}\right) \right) \sqrt{d_0^2+\eta _0}\\&\le \left( (1+C_S)(1+\sqrt{C_P})C_P+C_SC_P^2\right) \sqrt{\Vert M_0\Vert }\sqrt{d_0^2+\eta _0} \end{aligned}$$

which gives (19) for the case $k\ge 2$. Note now that by (11), we have $M_1\preceq C_PM_0$ and so, using the second identity in (18) with $k=1$, Proposition 2.1, Lemma A.2(b) and (14), we find

$$\begin{aligned} \Vert r_1^a\Vert _{\mathscr {Z}}=\Vert M_1(z_0-z_1)\Vert _{\mathscr {Z}}&\le \sqrt{C_P\Vert M_0\Vert }\Vert z_0-z_1\Vert _{\mathscr {Z},M_1}\\&\le \sqrt{C_P\Vert M_0\Vert }( \Vert z_0-z^*\Vert _{\mathscr {Z},M_1}+\Vert z_1-z^*\Vert _{\mathscr {Z},M_1})\\&\le \sqrt{C_P\Vert M_0\Vert }( \sqrt{C_P}\Vert z_0-z^*\Vert _{\mathscr {Z},M_0}+\Vert z_1-z^*\Vert _{\mathscr {Z},M_1})\\&\le (C_P+\sqrt{C_P})\sqrt{\Vert M_0\Vert }\max _{i=0,1}\,\Vert z_i-z^*\Vert _{\mathscr {Z},M_i}\\&\le (C_P+\sqrt{C_P})\sqrt{C_P\Vert M_0\Vert }\sqrt{d_0^2+\eta _0}\,, \end{aligned}$$

which, in turn, gives (19) for $k=1$. $\square$