Faster first-order primal-dual methods for linear programming using restarts and sharpness

Abstract

First-order primal-dual methods are appealing for their low memory overhead, fast iterations, and effective parallelization. However, they are often slow at finding high accuracy solutions, which creates a barrier to their use in traditional linear programming (LP) applications. This paper exploits the sharpness of primal-dual formulations of LP instances to achieve linear convergence using restarts in a general setting that applies to alternating direction method of multipliers (ADMM), primal-dual hybrid gradient method (PDHG) and extragradient method (EGM). In the special case of PDHG, without restarts we show an iteration count lower bound of $\Omega (\kappa ^2 \log (1/\epsilon ))$, while with restarts we show an iteration count upper bound of $O(\kappa \log (1/\epsilon ))$, where $\kappa$ is a condition number and $\epsilon$ is the desired accuracy. Moreover, the upper bound is optimal for a wide class of primal-dual methods, and applies to the strictly more general class of sharp primal-dual problems. We develop an adaptive restart scheme and verify that restarts significantly improve the ability of PDHG, EGM, and ADMM to find high accuracy solutions to LP problems.

Appendices

Proof of Lemma 2

Proof

Let $U \Sigma V^{\top }$ be a singular value decomposition of H. Recall U and V are orthogonal matrices ($U^{\top } U = U U^{\top } = {{\textbf {I}}}$ and $\Vert U z \Vert = \Vert z \Vert$), and $\Sigma$ is diagonal. First, by assumption there exists some $\bar{z}$ such that $H \bar{z} = h$. This implies that for $\bar{p} = V^{\top } \bar{z}$ we have $U^{\top } h = U^{\top } (U \Sigma V^{\top }) \bar{z} = \Sigma V^{\top } \bar{z} = \Sigma \bar{p}$. Consider some $z \in {\mathbb {R}}^{n}$. Define $p = V^{\top } z$,

$$\begin{aligned} p^{*}_i = {\left\{ \begin{array}{ll} \Sigma _{ii}^{-1} (U^{\top } h)_i &{}\quad \Sigma _{ii} \ne 0 \\ z_i &{}\quad \text {otherwise} \end{array}\right. } \end{aligned}$$

and $z^{\star }= V p^{*}$. If $\Sigma _{ii} \ne 0$ then by definition of $p^{*}$, $(\Sigma p^{*})_i = (U^{\top } h)_i$. If $\Sigma _{ii} = 0$ then $(U^{\top } h)_i = (\Sigma \bar{p})_i = \Sigma _{ii} \bar{p}_i = 0 = \Sigma _{ii} p^{*}_i = (\Sigma p^{*})_i$. Hence $\Sigma p^{*} = U^{\top } h$. Therefore,

$$\begin{aligned} \Vert z^{\star }- z \Vert= & {} \Vert V^{\top } z^{\star }- V^{\top } z \Vert \\= & {} \Vert p^{*} - p \Vert \\\le & {} \frac{1}{{\sigma _{\min }^+}{(H)}} \Vert \Sigma p^{*} - \Sigma p \Vert \\= & {} \frac{1}{{\sigma _{\min }^+}{(H)}} \Vert (U^{\top } h) - \Sigma p \Vert \\= & {} \frac{1}{{\sigma _{\min }^+}{(H)}} \Vert h - U \Sigma V^{\top } z \Vert \\= & {} \frac{1}{{\sigma _{\min }^+}{(H)}} \Vert h - H z \Vert . \end{aligned}$$

$\square$

Proof of Proposition 3 for EGM

Proof

Let $z^{\star }= \min _{z \in Z^{\star }} \Vert z^{t} - z \Vert$. By definition of $\hat{z}^t$ and that F is $L$-Lipschitz, we have

$$\begin{aligned}&F(z^{t-1})^{\top } (\hat{z}^t - z^{t-1}) + \frac{1}{2 \eta } \Vert \hat{z}^t - z^{t-1} \Vert ^2 \nonumber \\&\quad \le F(z^{t-1})^{\top } (z^{\star }- z^{t-1}) + \frac{1}{2 \eta } \Vert z^{\star }- z^{t-1} \Vert ^2 \nonumber \\&\quad = F(z^{\star })^{\top } (z^{\star }- z^{t-1}) + (F(z^{t-1}) - F(z^{\star }))^{\top } (z^{\star }- z^{t-1}) + \frac{1}{2 \eta } \Vert z^{\star }- z^{t-1} \Vert ^2 \ . \end{aligned}$$

(57)

Note that

$$\begin{aligned} \begin{aligned} F(z^{t-1})^{\top } (\hat{z}^t - z^{t-1}) =F(z^{\star })^{\top } (\hat{z}^t - z^{t-1}) + (F(z^{t-1}) - F(z^{\star }))^{\top } (\hat{z}^t - z^{\star }) + \\ (F(z^{t-1}) - F(z^{\star }))^{\top } (z^{\star }- z^{t-1}) \ . \end{aligned} \end{aligned}$$

(58)

Substituting (58) into the LHS of (57) and cancelling terms yields

$$\begin{aligned} \begin{aligned}&F(z^{\star })^{\top } (\hat{z}^t - z^{t-1}) + (F(z^{t-1}) - F(z^{\star }))^{\top } (\hat{z}^t - z^{\star }) + \frac{1}{2 \eta } \Vert \hat{z}^t - z^{t-1} \Vert ^2 \\&\le F(z^{\star })^{\top } (z^{\star }- z^{t-1}) + \frac{1}{2 \eta } \Vert z^{\star }- z^{t-1} \Vert ^2\ . \end{aligned} \end{aligned}$$

(59)

By $F(z^{\star })^{\top } (z^{\star }- z^{t-1}) \le F(z^{\star })^{\top } (\hat{z}^t - z^{t-1})$, (59), F is $L$-Lipschitz, and the triangle inequality one gets

$$\begin{aligned} 0&\ge F(z^{\star })^{\top } (z^{\star }- z^{t-1}) - F(z^{\star })^{\top } (\hat{z}^t - z^{t-1}) \nonumber \\&\ge \frac{1}{2 \eta } \Vert \hat{z}^t - z^{t-1} \Vert ^2 + (F(z^{t-1}) - F(z^{\star }))^{\top } (\hat{z}^t - z^{\star }) - \frac{1}{2 \eta } \Vert z^{\star }- z^{t-1} \Vert ^2 \nonumber \\&\ge \frac{1}{2 \eta } \Vert \hat{z}^t - z^{t-1} \Vert ^2 - L\Vert z^{t-1} - z^{\star }\Vert \Vert \hat{z}^t - z^{\star }\Vert - \frac{1}{2 \eta } \Vert z^{\star }- z^{t-1} \Vert ^2 \nonumber \\&= \frac{1}{2 \eta } \Vert \hat{z}^t - z^{t-1} \Vert ^2 - L\Vert z^{t-1} - z^{\star }\Vert \Vert z^{t-1} - z^{\star }+ \hat{z}^t - z^{t-1} \Vert - \frac{1}{2 \eta } \Vert z^{\star }- z^{t-1} \Vert ^2 \nonumber \\&\ge \frac{1}{2 \eta } \Vert \hat{z}^t - z^{t-1} \Vert ^2 - L\Vert z^{t-1} - z^{\star }\Vert \Vert \hat{z}^t - z^{t-1} \Vert - \left( L+ \frac{1}{2 \eta } \right) \Vert z^{\star }- z^{t-1} \Vert ^2 \nonumber \\&= \left( \frac{1}{2\eta } \Vert \hat{z}^t - z^{t-1} \Vert - \left( \frac{1}{2\eta }+L\right) \Vert z^{\star }- z^{t-1} \Vert \right) \left( \Vert \hat{z}^t - z^{t-1} \Vert +\Vert z^{\star }- z^{t-1} \Vert \right) \ \end{aligned}$$

(60)

where the first inequality uses $F(z^{\star })^{\top } (z^{\star }- z^{t-1}) \le F(z^{\star })^{\top } (\hat{z}^t - z^{t-1})$, the second inequality uses (59), the third inequality uses that F is $L$-Lipschitz, and the final inequality uses the triangle inequality.

Notice that $\Vert \hat{z}^t - z^{t-1} \Vert +\Vert z^{\star }- z^{t-1} \Vert > 0$ otherwise the result trivially holds, thus by (60) we get $\frac{1}{2\eta } \Vert \hat{z}^t - z^{t-1} \Vert - \left( \frac{1}{2\eta }+L\right) \Vert z^{\star }- z^{t-1} \Vert \le 0$, rearranging yields

$$\begin{aligned} \Vert \hat{z}^t - z^{t-1} \Vert \le (1+ 2\eta L) \Vert z^{\star }- z^{t-1} \Vert \le 3 \Vert z^{\star }- z^{t-1} \Vert \, \end{aligned}$$

where the last inequality uses that $\eta \le 1 / L$ from the premise of Proposition 1. $\square$

Proof of Lemma 6

Proof

Recall for ADMM the norm is $\Vert z \Vert = \sqrt{z^{\top } M z}$ with

$$\begin{aligned} M= \begin{pmatrix} 0 &{}\quad 0 &{}\quad 0 \\ 0 &{}\quad \eta V^{\top } V&{}\quad 0 \\ 0 &{}\quad 0 &{}\quad \frac{1}{\eta } {{\textbf {I}}}\end{pmatrix} = \begin{pmatrix} 0 &{}\quad 0 &{} \quad 0 \\ 0 &{}\quad \eta {{\textbf {I}}}&{}\quad 0 \\ 0 &{}\quad 0 &{}\quad \frac{1}{\eta } {{\textbf {I}}}\end{pmatrix}\, \end{aligned}$$

where we ustilize $V=I$ in (28). Suppose that $\eta = 1$. Consider $z \in Z$, $r \in (0, 2R]$. Let

$$\begin{aligned} v = \begin{pmatrix} 0 \\ (c + y)^{-} \\ x_{V} - x_{U} \end{pmatrix} \text { and note that } F(z) = \begin{pmatrix} -y \\ c + y \\ x_{U} - x_{V} \end{pmatrix}. \end{aligned}$$

Let $z_1 = z + r \frac{v}{\Vert v \Vert }$, then $z_1\in Z$ by the definition of Z. It follows from (21) that

$$\begin{aligned} \begin{aligned} \rho _r(z)&\ge \frac{F(z)^{\top } (z - z_1)}{r} = -\frac{1}{\Vert v \Vert } F(z)^{\top } v = \left\| v \right\| _{2} \ . \end{aligned} \end{aligned}$$

(61)

Consider any optimal solution $z^{\star }= (x^{\star }_{V}, x^{\star }_{V}, y^{\star })$, define $\hat{z} = (x^{\star }_{V}, x^{\star }_{V}, y) \in Z^{\star }$. Let $z_2= z - \mu (z - \hat{z})$ for $\mu = \min \left\{ \frac{r}{\Vert z - \hat{z} \Vert }, 1 \right\}$, then $\Vert z_2-z\Vert \le \frac{r}{\Vert z - \hat{z} \Vert }\Vert z - \hat{z} \Vert \le r$. Meanwhile, we have $z_2 = (1- \mu ) z + \mu \hat{z}$, thus $z_2\in Z$ by convexity. We conclude that $z_2 \in W_{r}(z)$. Therefore, it follows from (21) that

$$\begin{aligned} \begin{aligned} \rho _r(z)&\ge \frac{1}{r} \min \left\{ \frac{r}{\Vert z - \hat{z} \Vert }, 1 \right\} F(z)^{\top } (z - \hat{z}) \\&= \min \left\{ \frac{1}{\Vert z - \hat{z} \Vert }, \frac{1}{r} \right\} \begin{pmatrix} -y \\ c+y \\ x_{U} - x_{V} \end{pmatrix} \cdot \begin{pmatrix} x_U - x^{\star }_V \\ x_{V} - x^{\star }_V \\ y \end{pmatrix} \\&= \min \left\{ \frac{1}{\Vert z - \hat{z} \Vert }, \frac{1}{r} \right\} c^{\top } (x_{V} - x^{\star }_{V}) \ . \end{aligned} \end{aligned}$$

(62)

Substituting $r \in (0, 2R]$ and $\Vert z - \hat{z} \Vert \le \Vert z - z^{\star }\Vert \in [0,2R]$ into (62) and noticing $\rho _r(z)\ge 0$ yields

$$\begin{aligned} \begin{aligned} \rho _r(z)&\ge \frac{1}{2R} (c^{\top } x_{V} - c^{\top } x^{\star }_{V})^+ \ . \end{aligned} \end{aligned}$$

(63)

Combining (61) and (63), we deduce there exists $K'$ and $h'$ such that

$$\begin{aligned} (4R^2 +1)\rho _r(z)^2&\ge ((c^{\top } x_{V} - c^{\top } x^{\star }_{V})^+)^2 + \Vert (c + y)^{-} \Vert _2^2 + \Vert x_V - x_U \Vert _2^2 \\&= ((c^{\top } x_{V} - c^{\top } x^{\star }_{V})^+)^2 + \Vert (c + y)^{-} \Vert _2^2\\&\qquad + \Vert x_V - x_U \Vert _2^2 + \Vert x_U^{-} \Vert _2^2 + \Vert A x_{V} - b \Vert _2^2\\&= \Vert (h'-K' z )^+\Vert ^2_2 \ge \frac{1}{H(K')^2}\mathbf{dist{}}(z,Z^{\star })^2 \ , \end{aligned}$$

where the last inequality is from duality, Hoffman condition (Equation (26)), and the fact that $\Vert z \Vert _2 \ge \Vert z \Vert$. Taking the square root obtains the result for $\eta = 1$.

Next, we consider the case $\eta \ne 1$. Let us denote the corresponding normalized duality gap as $\rho _r^{\eta }$ and distance as $\mathbf{dist{}}^{\eta }$ then with $\theta = \max \{ \eta , 1/\eta \}$ we get

$$\begin{aligned} \rho _{r}^{\eta }(z)\ge & {} \frac{\rho _{\theta r}^{1}(z)}{\theta } \ge \frac{1}{\theta H(K') \sqrt{4 R^2 + 1}} \textbf{dist}^{1}(z, Z^{\star })\\\ge & {} \frac{1}{\theta ^2 H(K') \sqrt{4 R^2 + 1}} \textbf{dist}^{\eta }(z, Z^{\star }) . \end{aligned}$$

$\square$

Proof of Proposition 10

The next lemma is used in the proof of Proposition 10.

Lemma 7

Consider the sequence $\{ z^{n, 0} \}_{i=0}^{\infty }$, $\{ \tau ^{n} \}_{i=1}^{\infty }$ generated by Algorithm 1 with the adaptive restart scheme and $\beta <\frac{1}{q+3}$. Suppose (6) satisfies Property 3, and there exists $a \in (0,\infty )$ s.t. $\rho _r(z) \ge \frac{a}{1 + \Vert z \Vert } \mathbf{dist{}}(z, Z^{\star })$. Then $z^{n,0}$ stays in a bounded region, in particular, $z^{n,0} \in W_{R}(z^{0,0})$ for all $n \in {\mathbb {N}}$ with $R = \frac{2C (q + 3)^3}{\beta a (1 - \beta (q+3))} \mathbf{dist{}}( z^{0, 0}, Z^{\star }) (1 + \Vert z^{0, 0} \Vert + \mathbf{dist{}}(z^{0,0}, Z^{\star }))$.

Proof

First, notice that it holds for any $n>0$ that $\mathbf{dist{}}(z^{n, 0}, Z^{\star }) \le \mathbf{dist{}}(z^{n-1, 0}, Z^{\star }) + \Vert z^{n, 0} - z^{n-1,0} \Vert \le (q + 3) \mathbf{dist{}}(z^{n-1, 0}, Z^{\star })$ where the first inequality uses the triangle inequality, and the second inequality is from Property ii.. Recursively applying this inequality yields

$$\begin{aligned} \mathbf{dist{}}(z^{n, 0}, Z^{\star }) \le (q + 3)^n \mathbf{dist{}}(z^{0, 0}, Z^{\star }) \ . \end{aligned}$$

(64)

Thus, we have

$$\begin{aligned} \Vert z^{N, 0} - z^{0, 0} \Vert&\le \sum _{n=0}^{N-1} \Vert z^{n+1, 0} - z^{n,0} \Vert \le (q + 3) \sum _{n=0}^{N-1} \mathbf{dist{}}(z^{n, 0}, Z^{\star }) \nonumber \\&\le \mathbf{dist{}}(z^{0,0}, Z^{\star }) \sum _{n=0}^{N-1} (q+3)^{n+1}\nonumber \\&= \mathbf{dist{}}(z^{0,0}, Z^{\star }) \frac{(q + 3)^{N + 1} - (q+3)}{q+2} \nonumber \\&\le \frac{(q + 3)^{N + 1}}{2} \mathbf{dist{}}(z^{0,0}, Z^{\star }) \ , \end{aligned}$$

(65)

where the first inequality uses the triangle inequality, the second inequality is from Property ii., the third inequality utilizes (64) for $n=0,\ldots ,N-1$, the equality uses the formula for the sum of a geometric series, and the last inequality uses $q\ge 0$.

Furthermore, we have

$$\begin{aligned} \rho _{\Vert z^{n, 0} - z^{n-1, 0} \Vert }(z^{n,0})&\le \beta ^{n-1} \rho _{\Vert z^{1, 0} - z^{0, 0} \Vert }( z^{1,0} ) \le \beta ^{n-1} 2 C \Vert z^{1, 0} - z^{0, 0} \Vert \nonumber \\&\le \beta ^{n-1} 2 C (q + 2) \mathbf{dist{}}( z^{0, 0}, Z^{\star }) \ , \end{aligned}$$

(66)

where the first inequality recursively uses (30), the second inequality is from Property i. and $\tau ^{0} \ge 1$, and the last inequality uses Property ii..

Therefore, it holds that

$$\begin{aligned} \begin{aligned} \Vert z^{n+1,0} - z^{n,0} \Vert&\le (q + 2) \mathbf{dist{}}(z^{n,0}, Z^{\star }) \\&\le (q + 2) \frac{\rho _{\Vert z^{n, 0} - z^{n-1, 0} \Vert }(z^{n,0})}{a} \left( 1 + \Vert z^{n,0} \Vert \right) \\&\le \beta ^{n-1} \frac{2 C (q + 2)^2 \mathbf{dist{}}( z^{0, 0}, Z^{\star })}{a} \left( 1 + \Vert z^{n,0} \Vert \right) \\&\le \beta ^{n-1} \frac{2 C (q + 2)^2 \mathbf{dist{}}( z^{0, 0}, Z^{\star })}{a} \left( 1 + \Vert z^{0, 0} \Vert + \Vert z^{n,0} - z^{0,0} \Vert \right) \\&\le \beta ^{n-1} \frac{2 C (q + 2)^2 \mathbf{dist{}}( z^{0, 0}, Z^{\star })}{a}\\&\qquad \left( 1 + \Vert z^{0, 0} \Vert + \frac{(q + 3)^{n+1}}{2} \mathbf{dist{}}(z^{0,0}, Z^{\star }) \right) \\&\le (\beta (q+3))^{n} \frac{C (q + 3)^3}{\beta a} \mathbf{dist{}}( z^{0, 0}, Z^{\star }) \\&\qquad \left( 1 + \Vert z^{0, 0} \Vert + \mathbf{dist{}}(z^{0,0}, Z^{\star }) \right) \ , \end{aligned} \end{aligned}$$

(67)

where the first inequality uses Property ii., the second inequality is from that $\rho _r(z) \ge \frac{a}{1 + \Vert z \Vert } \mathbf{dist{}}(z, Z^{\star })$, the third inequality uses (66), the fourth inequality uses the triangle inequality, the fifth inequality uses (65), and the last inequality uses $q+3 \ge 2$.

Therefore, we have

$$\begin{aligned} \Vert z^{N, 0} - z^{0, 0} \Vert&\le \sum _{n=0}^{N-1} \Vert z^{n+1,0} - z^{n,0} \Vert \\&\le \sum _{n=0}^{N-1} (\beta (q+3))^{n} \frac{C (q + 3)^3}{\beta a} \mathbf{dist{}}( z^{0, 0}, Z^{\star })\\&\qquad \left( 1 + \Vert z^{0, 0} \Vert + \mathbf{dist{}}(z^{0,0}, Z^{\star }) \right) \\&\le \frac{C (q + 3)^3}{\beta a (1 - \beta (q+3))} \mathbf{dist{}}( z^{0, 0}, Z^{\star }) \left( 1 + \Vert z^{0, 0} \Vert + \mathbf{dist{}}(z^{0,0}, Z^{\star }) \right) \ , \end{aligned}$$

where the first inequality is the triangle inequality, the second inequality uses (67), and the last inequality is from the bound on the sum of a geometric series by noticing $\beta (q+3)<1$. This finishes the proof. $\square$

Proof of Proposition 10for EGM and ADMM appled to LP. Recall that the Lagrangian form of an LP instance is $\alpha$-sharp on $S(R)=\{z \in Z: \Vert z \Vert \le R\}$ with $\alpha =\frac{1}{H(K)\sqrt{1+4R^2}}$ (see Lemma 5) and the ADMM form of a LP instance is $\alpha$-sharp on $S(R)=\{z \in Z: \Vert z \Vert \le R\}$ with $\alpha =\frac{1}{ \max \{ \eta ^2, 1/\eta ^2 \} H(K')\sqrt{1+4R^2}}$ (see Lemma 6). Thus the sharpness constant $\alpha$ satisfies the condition in Lemma 7 with $a=\frac{1}{2\,H(K)}$ for standard form LP (Lemma 5) and and $a=\frac{1}{2 \max \{ \eta ^2, 1/\eta ^2 \} H(K')}$ for ADMM form LP. We finish the proof by setting $R = \frac{2C (q + 3)^3}{\beta a (1 - \beta (q+3))} \mathbf{dist{}}( z^{0, 0}, Z^{\star }) (1 + \Vert z^{0, 0} \Vert + \mathbf{dist{}}(z^{0,0}, Z^{\star }))$ as stated in Lemma 7. $\square$

Proof of Corollary 2

Proof

Let $\Vert z \Vert _M = \sqrt{\Vert x \Vert ^2 - 2 \eta y^{\top } A x + \Vert y \Vert ^2}$. Recall that Property 3 holds for $\Vert \cdot \Vert _M$ with $q = 0$ and $C = 2 / \eta$ (refer to Table 1). Define $\hat{B}_{r}(z) = \{ z \in Z: \Vert z \Vert _{M} \le r \}$ then with $\hat{r} = r \sqrt{\frac{1}{1 - \eta {\sigma _{\max }}{(A)}}}$ for any $z \in Z$ we get

$$\begin{aligned} \rho _r(z)&= \frac{\max _{\hat{z}\in W_{r}(z)}\{ \mathcal {L}(x, \hat{y}) - \mathcal {L}(\hat{x}, y)\} }{r} \\&\le \frac{\max _{\hat{z}\in \hat{B}_{\hat{r}}(z)}\{ \mathcal {L}(x, \hat{y}) - \mathcal {L}(\hat{x}, y)\} }{r} \\&= \sqrt{\frac{1}{1 - \eta {\sigma _{\max }}{(A)}}} \frac{\max _{\hat{z}\in \hat{B}_{\hat{r}}(z)}\{ \mathcal {L}(x, \hat{y}) - \mathcal {L}(\hat{x}, y)\} }{\hat{r}} \\&\le \frac{1}{\eta } \sqrt{\frac{1}{1 - \eta {\sigma _{\max }}{(A)}}} \frac{2 \hat{r}}{t} = \frac{1}{\eta (1 - \eta {\sigma _{\max }}{(A)})} \frac{2 r}{t} \end{aligned}$$

where the first inequality uses that $W_{r}(z) \subseteq \hat{B}_{\hat{r}}(z)$ by Proposition 7, the second inequality uses Property i.. This establishes $C = \frac{2}{\eta (1 - \eta {\sigma _{\max }}{(A)})}$. Moreover, with $z^{\star }\in {\mathop {\textrm{argmin}}\limits _{z \in Z^{\star }}} \Vert z^{0} - z \Vert _M$ we have

$$\begin{aligned} \Vert \bar{z}^{t} - z^{0} \Vert ^2 \le \frac{\Vert \bar{z}^{t} - z^{0} \Vert _M^2}{1 - \eta {\sigma _{\max }}{(A)}} \le \frac{4 \Vert z^{\star }- z^{0} \Vert _{M}^2}{1 - \eta {\sigma _{\max }}{(A)}} \le \frac{4 (1 + \eta {\sigma _{\max }}{(A)})}{1 - \eta {\sigma _{\max }}{(A)}} \Vert z^{\star }- z^{0} \Vert ^2 \end{aligned}$$

where the first inequality uses by Proposition 7, the second inequality uses Property ii., and the third inequality uses Proposition 7. This establishes $q = 4 \frac{1 + \eta {\sigma _{\max }}{(A)}}{1 - \eta {\sigma _{\max }}{(A)}} - 2$. $\square$

Linear time algorithm for linear trust region problems

Theorem 4

Algorithm 2 exactly solves (49), the trust region problem with linear objective, Euclidean norm, and bounds.

Proof

If the algorithm returns from line 1, (51) is unbounded, and the algorithm returns $\hat{z}(\infty )$. Otherwise, in each iteration of the while loop (line 5), the algorithm maintains the invariants

• $\mathcal {I} = \{i:\;\lambda _{\text {lo}}< \hat{\lambda }_i < \lambda _{\text {hi}}\}$

• For $\lambda _{\text {lo}}< \lambda < \lambda _{\text {hi}}$, $\Vert \hat{z}(\lambda ) - z \Vert ^2 = f_{\text {lo}}+ \lambda ^2 f_{\text {hi}}+ \sum _{i \in \mathcal {I}} (\hat{z}(\lambda )_i - z_i)^2$

• $\Vert \hat{z}(\lambda _{\text {lo}}) - z\Vert ^2 \le r^2 \le \Vert \hat{z}(\lambda _{\text {hi}}) - z\Vert ^2$

The while loop (line 5) is finite, since on each iteration $|\mathcal {I}|$ is reduced by at least a factor of 2. When the while loop exits (with $\mathcal {I} = \emptyset$), these invariants mean that the final $\lambda _{\text {mid}}$ computed on line 18 optimizes (51) and the returned $\hat{z}(\lambda _{\text {mid}})$ solves (49).    $\square$

Theorem 5

Algorithm 2 runs in $O(m+n)$ time.

Proof

The work outside the while loop (line 5) is clearly $O(m+n)$. In each pass through the while loop, the median (line 6) can be found in $O(|\mathcal {I}|)$ time ([9]), and the rest of the loop also takes $O(|\mathcal {I}|)$ time. Since at least half of the elements of $\mathcal {I}$ are removed in each iteration, and initially $|\mathcal {I}| \le m+n$, the total time in the while loop is at most $O(\sum _{i=0}^\infty \frac{m+n}{2^i}) = O(m+n)$ by the formula for the sum of an infinite geometric series. $\square$

Numerical results for restarted EGM and ADMM

In this section, we repeat the numerical experiments of Sect. 7 with EGM and ADMM instead of PDHG. We tune the primal weight for EGM and the step size $\eta$ for ADMM using the same procedure as with PDHG (see Sect. 7).

Fig. 3

Plots show normalized duality gap (in $\log$ scale) versus number of iterations for (restarted) EGM. Results from left to right, top to bottom for qap10, qap15, nug08-3rd, and nug20. Each plot compares no restarts, adaptive restarts, and the three best fixed restart lengths. For the restart schemes we evaluate the normalized duality gap at the average, for no restarts we evaluate it at the last iterate as the average performs much worse. A star indicates the last iteration that the active set changed before the iteration limit was reached. Series markers indicate when restarts occur

Figure 3 (EGM) is almost identical to Fig. 2 (PDHG), which verifies again the improved performance of restarted algorithms. Furthermore, note that one iteration of EGM is about twice as expensive as a PDHG iteration, because EGM requires four matrix–vector multiplications per iteration, while PDHG requires two matrix–vector multiplications.

Indeed, to see the difference between EGM and PDHG requires a close examination of Table 4. Recall that termination criteria is only checked every 30 iterations, which causes the iteration counts to be identical in several instances.

Fig. 4

Plots show normalized duality gap (in $\log$ scale) versus number of iterations for (restarted) ADMM. Results from left to right, top to bottom for qap10, qap15, nug08-3rd, and nug20. Each plot compares no restarts, adaptive restarts, and the three best fixed restart lengths. For the restart schemes we evaluate the normalized duality gap at the average, for no restarts we evaluate it at the last iterate as the average performs much worse. A star indicates the last iteration that the active set changed before the iteration limit was reached. Series markers indicate when restarts occur

Figure 4 shows the performance of restarted ADMM for the same problems, which verifies again the improved performance of restarted algorithms. We observe that restarts do appear to make performance slower for nug08-3rd but this appears to be a very easy problem—the number of iterations for all methods is very low.