A block symmetric Gauss–Seidel decomposition theorem for convex composite quadratic programming and its applications

Abstract

For a symmetric positive semidefinite linear system of equations $\mathcal{Q}{{\varvec{x}}}= {{\varvec{b}}}$, where ${{\varvec{x}}}= (x_1,\ldots ,x_s)$ is partitioned into s blocks, with $s \ge 2$, we show that each cycle of the classical block symmetric Gauss–Seidel (sGS) method exactly solves the associated quadratic programming (QP) problem but added with an extra proximal term of the form $\frac{1}{2}\Vert {{\varvec{x}}}-{{\varvec{x}}}^k\Vert _\mathcal{T}^2$, where $\mathcal{T}$ is a symmetric positive semidefinite matrix related to the sGS decomposition of $\mathcal{Q}$ and ${{\varvec{x}}}^k$ is the previous iterate. By leveraging on such a connection to optimization, we are able to extend the result (which we name as the block sGS decomposition theorem) for solving convex composite QP (CCQP) with an additional possibly nonsmooth term in $x_1$, i.e., $\min \{ p(x_1) + \frac{1}{2}\langle {{\varvec{x}}},\,\mathcal{Q}{{\varvec{x}}}\rangle -\langle {{\varvec{b}}},\,{{\varvec{x}}}\rangle \}$, where $p(\cdot )$ is a proper closed convex function. Based on the block sGS decomposition theorem, we extend the classical block sGS method to solve CCQP. In addition, our extended block sGS method has the flexibility of allowing for inexact computation in each step of the block sGS cycle. At the same time, we can also accelerate the inexact block sGS method to achieve an iteration complexity of $O(1/k^2)$ after performing k cycles. As a fundamental building block, the block sGS decomposition theorem has played a key role in various recently developed algorithms such as the inexact semiproximal ALM/ADMM for linearly constrained multi-block convex composite conic programming (CCCP), and the accelerated block coordinate descent method for multi-block CCCP.

Appendix: Proof of part (b) of Proposition 2

To begin the proof, we state the following lemma from [35].

Lemma 2

Suppose that $\{u_k\}$ and $\{\lambda _k\}$ are two sequences of nonnegative scalars, and $\{ s_k\}$ is a nondecreasing sequence of scalars such that $s_0\ge u_0^2$. Suppose that for all $k\ge 1$, the inequality $u_k^2 \le s_k + 2\sum _{i=1}^k \lambda _i u_i$ holds. Then for all $k\ge 1$, $u_k \le \bar{\lambda }_k + \sqrt{ s_k + \bar{\lambda }_k^2},$ where $\bar{\lambda }_k = \sum _{i=1}^k \lambda _i$.

Proof

In this proof, we let $\Delta ^j = \Delta (\tilde{\varvec{\delta }}^j,\varvec{\delta }^j)$. Note that under the assumption that $t_j=1$ for all $j\ge 1$, $\widetilde{{{\varvec{x}}}}^j = {{\varvec{x}}}^{j-1}$. Note also that from (30), we have that $\Vert {\hat{\mathcal{Q}}}^{-1/2}\Delta ^j\Vert \le M \epsilon _j$, where M is given as in Proposition 2.

For $j\ge 1$, from the optimality of ${{\varvec{x}}}^j$ in (29), one can show that for all $x\in \mathcal{X}$

$$\begin{aligned} F({{\varvec{x}}}) - F({{\varvec{x}}}^j)\ge & {} \frac{1}{2}\Vert {{\varvec{x}}}- {{\varvec{x}}}^j\Vert ^2_{\mathcal{Q}} - \langle \mathcal{T}_{\mathcal{Q}}({{\varvec{x}}}^j - {{\varvec{x}}}^{j-1}),\,{{\varvec{x}}}- {{\varvec{x}}}^j\rangle + \langle \Delta ^j,\,{{\varvec{x}}}- {{\varvec{x}}}^j\rangle \nonumber \\\ge & {} \frac{1}{2}\Vert {{\varvec{x}}}- {{\varvec{x}}}^j\Vert ^2_{\mathcal{Q}} + \frac{1}{2}\Vert {{\varvec{x}}}- {{\varvec{x}}}^j\Vert ^2_{\mathcal{T}_{\mathcal{Q}}} - \frac{1}{2}\Vert {{\varvec{x}}}- {{\varvec{x}}}^{j-1}\Vert ^2_{\mathcal{T}_{\mathcal{Q}}} + \langle \Delta ^j,\,{{\varvec{x}}}- {{\varvec{x}}}^j\rangle \nonumber \\\ge & {} \frac{1}{2}\Vert {{\varvec{x}}}- {{\varvec{x}}}^j\Vert ^2_{{\hat{\mathcal{Q}}}} - \frac{1}{2}\Vert {{\varvec{x}}}- {{\varvec{x}}}^{j-1}\Vert ^2_{{\hat{\mathcal{Q}}}} + \langle \Delta ^j,\,{{\varvec{x}}}- {{\varvec{x}}}^j\rangle \nonumber \\= & {} \frac{1}{2}\Vert {{\varvec{x}}}^j - {{\varvec{x}}}^{j-1}\Vert _{{\hat{\mathcal{Q}}}}^2 + \langle {{\varvec{x}}}^{j-1}-{{\varvec{x}}},\,{\hat{\mathcal{Q}}}({{\varvec{x}}}^j-{{\varvec{x}}}^{j-1})\rangle + \langle \Delta ^j,\,{{\varvec{x}}}-{{\varvec{x}}}^j\rangle .\nonumber \\ \end{aligned}$$

(41)

Let ${{\varvec{e}}}^j = {{\varvec{x}}}^j-{{\varvec{x}}}^*$. By setting ${{\varvec{x}}}= {{\varvec{x}}}^{j-1}$ and ${{\varvec{x}}}={{\varvec{x}}}^*$ in (41), we get

$$\begin{aligned} F({{\varvec{x}}}^{j-1}) - F({{\varvec{x}}}^j)\ge & {} \frac{1}{2} \Vert {{\varvec{e}}}^j-{{\varvec{e}}}^{j-1}\Vert _{\hat{\mathcal{Q}}}^2 + \langle \Delta ^j,\,{{\varvec{e}}}^{j-1}-{{\varvec{e}}}^j\rangle , \end{aligned}$$

(42)

$$\begin{aligned} F({{\varvec{x}}}^*) - F({{\varvec{x}}}^j)\ge & {} \frac{1}{2} \Vert {{\varvec{e}}}^j\Vert _{\hat{\mathcal{Q}}}^2 -\frac{1}{2} \Vert {{\varvec{e}}}^{j-1}\Vert _{\hat{\mathcal{Q}}}^2 - \langle \Delta ^j,\,{{\varvec{e}}}^j\rangle . \end{aligned}$$

(43)

By multiplying $j-1$ to (42) and combining with (43), we get

$$\begin{aligned} (a_j + b_j^2)\le & {} (a_{j-1}+b_{j-1}^2) - (j-1)\Vert {{\varvec{e}}}^j-{{\varvec{e}}}^{j-1}\Vert _{\hat{\mathcal{Q}}}^2 + 2\langle \Delta ^j,\, j {{\varvec{e}}}^j-(j-1){{\varvec{e}}}^{j-1} \rangle \nonumber \\\le & {} (a_{j-1}+b_{j-1}^2) + 2 \Vert {\hat{\mathcal{Q}}}^{-1/2}\Delta ^j\Vert \Vert j {{\varvec{e}}}^j-(j-1){{\varvec{e}}}^{j-1} \Vert _{\hat{\mathcal{Q}}}\nonumber \\\le & {} (a_{j-1}+b_{j-1}^2) + 2\Vert {\hat{\mathcal{Q}}}^{-1/2}\Delta ^j\Vert (j b_j + (j-1)b_{j-1}) \nonumber \\\le & {} \cdots \nonumber \\\le & {} a_1 + b_1^2 + 2 \sum _{i=2}^j M\epsilon _i (i b_i +(i-1)b_{i-1}) \nonumber \\\le & {} { b_0^2 + 2 \sum _{i=1}^j 2Mi \epsilon _i b_i }, \end{aligned}$$

(44)

where $a_j = 2j [F({{\varvec{x}}}^j)-F({{\varvec{x}}}^*)]$ and $b_j = \Vert {{\varvec{e}}}^j\Vert _{\hat{\mathcal{Q}}}$. Note that the last inequality follows from (43) with $j=1$, the fact that the sequence $\{\epsilon _i\}$ is non-increasing, and some simple manipulations.

To summarize, we have $b_j^2 \le b_0^2 + 2 \sum _{i=1}^j 2M i\epsilon _i b_i$. By applying Lemma 2, we get

$$\begin{aligned} b_j \;\le \; \bar{\lambda }_j + \sqrt{ b_0^2 + \bar{\lambda }_j^2} \;\le \; b_0 + 2\bar{\lambda }_j, \end{aligned}$$

where $\bar{\lambda }_j = \sum _{i=1}^j \lambda _i$ with $\lambda _i = 2M i \epsilon _i$. Applying the above result to (44), we get

$$\begin{aligned} a_j \;\le \; b_0^2 + 2 \sum _{i=1}^j \lambda _i (2\bar{\lambda }_i + b_0) \; \le \; (b_0 + 2\bar{\lambda }_j)^2. \end{aligned}$$

From here, the required result in Part (b) of Proposition 2 follows. $\square$