A Unified Primal-Dual Algorithm Framework for Inequality Constrained Problems

Abstract

In this paper, we propose a unified primal-dual algorithm framework based on the augmented Lagrangian function for composite convex problems with conic inequality constraints. The new framework is highly versatile. First, it not only covers many existing algorithms such as PDHG, Chambolle–Pock, GDA, OGDA and linearized ALM, but also guides us to design a new efficient algorithm called Semi-OGDA (SOGDA). Second, it enables us to study the role of the augmented penalty term in the convergence analysis. Interestingly, a properly selected penalty not only improves the numerical performance of the above methods, but also theoretically enables the convergence of algorithms like PDHG and SOGDA. Under properly designed step sizes and penalty term, our unified framework preserves the ${\mathcal {O}}(1/N)$ ergodic convergence while not requiring any prior knowledge about the magnitude of the optimal Lagrangian multiplier. Linear convergence rate for affine equality constrained problem is also obtained given appropriate conditions. Finally, numerical experiments on linear programming, $\ell _1$ minimization problem, and multi-block basis pursuit problem demonstrate the efficiency of our methods.

Appendix

Theorem 7

(Moreau’s decomposition theorem [27]) Let ${\mathcal {K}}\subset {\mathbb {R}}^n$ be a closed convex cone and ${\mathcal {K}}^\circ$ be its polar cone. For $x,y,z\in {\mathbb {R}}^n$, the following statements are equivalent:

• $z=x+y,x\in {\mathcal {K}}, y\in {\mathcal {K}}^\circ$ and $\langle x,y\rangle =0$,

• $x={\mathcal {P}}_{{\mathcal {K}}}(z)$ and $y={\mathcal {P}}_{{\mathcal {K}}^\circ }(z)$.

The proof of basic lemmas is presented as follows. It is worth noting that when ${\mathcal {K}}={\mathbb {R}}^{n}_{\le 0}$, we have ${\mathcal {P}}_{+}\left( \cdot \right) =\left[ \cdot \right] _{+}$ being the standard entry-wise positive part, and ${\mathcal {P}}_{-}\left( \cdot \right) =\left[ \cdot \right] _{-}$.

Proof of Lemma 6

By the definition of ${\mathcal {P}}_{+}\left( \cdot \right)$ and ${\mathcal {P}}_{-}\left( \cdot \right)$, we have

$$\begin{aligned} \left\| w-w' \right\| ^2 =&\left\| {\mathcal {P}}_{+}\left( w\right) -{\mathcal {P}}_{+}\left( w'\right) -{\mathcal {P}}_{-}\left( w\right) +{\mathcal {P}}_{-}\left( w'\right) \right\| ^2\\ =&\left\| {\mathcal {P}}_{+}\left( w\right) -{\mathcal {P}}_{+}\left( w'\right) \right\| ^2 + \left\| {\mathcal {P}}_{-}\left( w\right) -{\mathcal {P}}_{-}\left( w'\right) \right\| ^2\\&+2\left\langle {{\mathcal {P}}_{+}\left( w\right) },{{\mathcal {P}}_{-}\left( w'\right) }\right\rangle +2\left\langle {{\mathcal {P}}_{+}\left( w'\right) },{{\mathcal {P}}_{-}\left( w\right) }\right\rangle , \end{aligned}$$

where the second equality is due to the fact that $\left\langle {{\mathcal {P}}_{+}\left( w\right) },{{\mathcal {P}}_{-}\left( w\right) }\right\rangle =\left\langle {{\mathcal {P}}_{+}\left( w'\right) },{{\mathcal {P}}_{-}\left( w'\right) }\right\rangle =0$. Hence, the case $a>b$ is proven by noting that $\left\langle {{\mathcal {P}}_{+}\left( w\right) },{{\mathcal {P}}_{-}\left( w'\right) }\right\rangle \ge 0$ and $\left\langle {{\mathcal {P}}_{+}\left( w'\right) },{{\mathcal {P}}_{-}\left( w\right) }\right\rangle \ge 0$. As of the case $a\ge b$, we have

$$\begin{aligned}&a^2\left\| w-w' \right\| ^2-(2a-b)\left[ 2a\left\langle { {\mathcal {P}}_{+}\left( w\right) },{ {\mathcal {P}}_{-}\left( w'\right) }\right\rangle +b\left\| {\mathcal {P}}_{+}\left( w\right) -{\mathcal {P}}_{+}\left( w'\right) \right\| ^2\right] \\ =&(a-b)^2\left\| {\mathcal {P}}_{+}\left( w\right) -{\mathcal {P}}_{+}\left( w'\right) \right\| ^2+a^2\left\| {\mathcal {P}}_{-}\left( w\right) -{\mathcal {P}}_{-}\left( w'\right) \right\| ^2\\&-2a(a-b)\left\langle {{\mathcal {P}}_{+}\left( w\right) },{{\mathcal {P}}_{-}\left( w'\right) }\right\rangle +2a^2\left\langle {{\mathcal {P}}_{+}\left( w'\right) },{{\mathcal {P}}_{-}\left( w\right) }\right\rangle \\ =&\left\| (a-b)\left( {\mathcal {P}}_{+}\left( w\right) -{\mathcal {P}}_{+}\left( w'\right) \right) +a\left( {\mathcal {P}}_{-}\left( w\right) -{\mathcal {P}}_{-}\left( w'\right) \right) \right\| ^2\\&+2a(2a-b)\left\langle {{\mathcal {P}}_{+}\left( w'\right) },{{\mathcal {P}}_{-}\left( w\right) }\right\rangle \\ \ge&0, \end{aligned}$$

which completes the proof of the first inequality.

For the second inequality, we consider

$$\begin{aligned}&\left\| u-u'\right\| ^2+\left\| v-v'\right\| ^2-\left\| r-r'\right\| ^2\\ =&\left\| w+v-w'-v'\right\| ^2+\left\| v-v'\right\| ^2-\left\| {\mathcal {P}}_{+}\left( w\right) +v-{\mathcal {P}}_{+}\left( w'\right) -v'\right\| ^2\\ =&\left\| w-w'\right\| ^2+\left\| v-v'\right\| ^2-\left\langle {v-v'},{{\mathcal {P}}_{-}\left( w\right) -{\mathcal {P}}_{-}\left( w'\right) }\right\rangle -\left\| {\mathcal {P}}_{+}\left( w\right) -{\mathcal {P}}_{+}\left( w'\right) \right\| ^2\\ \ge&\left\| {\mathcal {P}}_{-}\left( w\right) -{\mathcal {P}}_{-}\left( w'\right) \right\| ^2+\left\| v-v'\right\| ^2-\left\langle {v-v'},{{\mathcal {P}}_{-}\left( w\right) -{\mathcal {P}}_{-}\left( w'\right) }\right\rangle \\ \ge&0. \end{aligned}$$

Lemma 10

When ${\mathcal {K}}=\{x|x\le 0\}$ and $\rho >0$, for any $i\in [m]$, the dual update rule of (14) is

$$\begin{aligned} y_i^{k+1}= {\left\{ \begin{array}{ll} 0, &{} \omega _i>0\ \text {and}\ \kappa \nu _i\ge -\omega _i,\\ \omega _i, &{} \omega _i\le 0\ \text {and}\ \nu _i\ge \omega _i,\\ \frac{\omega _i+\kappa \nu _i}{\kappa +1}, &{}\text {otherwise}. \end{array}\right. } \end{aligned}$$

(64)

Proof

According to the dual update rule in (14), $y^{k+1}$ is the solution of the following equation:

$$\begin{aligned} y = -\left[ \omega -\kappa \left[ \nu -y\right] _-\right] _-. \end{aligned}$$

We can give the solution of the equation through category discussion:

• 1. If $\nu _i\ge y_i$, $\left[ \nu _i-y_i\right] _-=0$ and hence $y_i = -\left[ \omega _i\right] _-$.

• 2. If $\nu _i<y_i$, the equation becomes $y_i = -\left[ \omega _i+\kappa (\nu _i-y_i)\right] _-$, then we consider the following two cases:

  • When $\omega _i+\kappa \nu _i\ge 0$, it holds that $\omega _i+\kappa (\nu _i-y_i)\ge 0$ due to the fact that $\kappa \ge 0$ and $y_i\le 0$. Thus, we obtain $y_i=0$.

  • When $\omega _i+\kappa \nu _i< 0$, the equation has a solution if and only if $\omega _i+\kappa (\nu _i-y_i)\le 0$, that is $y_i = \omega _i+\kappa (\nu _i-y_i)$. Thus, the solution is $y_i=\frac{\omega _i+\kappa \nu _i}{\kappa +1}$.

By further simplification, we get (64). $\square$

Lemma 11

For any $a\in \mathbb {R}^n$ and $b\in -{\mathcal {K}}$, it holds that $\Vert {\mathcal {P}}_{+}\left( a+b\right) \Vert \ge \Vert {\mathcal {P}}_{+}\left( a\right) \Vert$.

Proof

Write $c={\mathcal {P}}_{+}\left( a+b\right) \in {\mathcal {K}}^{\circ }, d={\mathcal {P}}_{-}\left( a+b\right) \in -{\mathcal {K}}$, then $a+b=c-d$. By definition of ${\mathcal {P}}_{{\mathcal {K}}}$, we have

$$\begin{aligned} \left\| a-{\mathcal {P}}_{{\mathcal {K}}}(a)\right\| =\min _{x\in {\mathcal {K}}}\Vert a-x\Vert \le \left\| a-(-b-d)\right\| =\Vert c\Vert . \end{aligned}$$

Hence, $\left\| {\mathcal {P}}_{+}\left( a\right) \right\| =\left\| a+{\mathcal {P}}_{-}\left( a\right) \right\| =\left\| a-{\mathcal {P}}_{{\mathcal {K}}}(a)\right\| \le \Vert c\Vert =\left\| {\mathcal {P}}_{+}\left( a+b\right) \right\|$. $\square$