On Iteration Complexity of a First-Order Primal-Dual Method for Nonlinear Convex Cone Programming

Abstract

Nonlinear convex cone programming (NCCP) models have found many practical applications. In this paper, we introduce a flexible first-order primal-dual algorithm, called the variant auxiliary problem principle (VAPP), for solving NCCP problems when the objective function and constraints are convex but may be nonsmooth. At each iteration, VAPP generates a nonlinear approximation of the primal augmented Lagrangian model. The approximation incorporates both linearization and a distance-like proximal term, and then the iterations of VAPP are shown to possess a decomposition property for NCCP. Motivated by recent applications in big data analytics, there has been a growing interest in the convergence rate analysis of algorithms with parallel computing capabilities for large scale optimization problems. We establish O(1/t) convergence rate towards primal optimality, feasibility and dual optimality. By adaptively setting parameters at different iterations, we show an $O(1/t^2)$ rate for the strongly convex case. Finally, we discuss some issues in the implementation of VAPP.

Appendix

A$_1$: Proof of Lemma 1(Descent inequalities of generalized distance function)

Step 1. Estimate $L(u^{k+1},q^k)-L(u,q^k)$.

For the primal subproblem (22) of VAPP, the unique solution $u^{k+1}$ is characterized by the following variational inequality:

$$\begin{aligned}&\langle \nabla G(u^{k}),u-u^{k+1}\rangle +J(u)-J(u^{k+1})\nonumber \\&\qquad +\langle q^k,\nabla \varOmega (u^{k})(u-u^{k+1})+\varPhi (u)-\varPhi (u^{k+1})\rangle \nonumber \\&\qquad +\frac{1}{\varepsilon ^k}\langle \nabla K(u^{k+1})-\nabla K(u^k), u-u^{k+1}\rangle \ge 0, \forall u\in {\varvec{U}}, \end{aligned}$$

(A1)

which follows that

$$\begin{aligned} L(u^{k+1},q^k)-L(u,q^k)= & {} (G+J)(u^{k+1})-(G+J)(u)+\langle q^k, \varTheta (u^{k+1})-\varTheta (u)\rangle \nonumber \\\leqslant & {} \underbrace{G(u^{k+1})-G(u)+\langle \nabla G(u^{k}),u-u^{k+1}\rangle }_{\varLambda _1}\nonumber \\&+\underbrace{\langle q^k,\varOmega (u^{k+1})-\varOmega (u)+\nabla \varOmega (u^{k})(u-u^{k+1})\rangle }_{\varLambda _2}\nonumber \\&+\underbrace{\frac{1}{\varepsilon ^k}\langle \nabla K(u^{k+1})-\nabla K(u^k), u-u^{k+1}\rangle }_{\varLambda _3}. \end{aligned}$$

(A2)

By the convexity of G, we estimate term $\varLambda _1$ in (A2).

$$\begin{aligned} \varLambda _1= & {} G(u^k)-G(u)+\langle \nabla G(u^{k}),u-u^{k}\rangle +\big (G(u^{k+1})-G(u^{k})\nonumber \\&\quad -\langle \nabla G(u^{k}),u^{k+1}-u^{k}\rangle \big )\nonumber \\\leqslant & {} G(u^{k+1})-G(u^{k})-\langle \nabla G(u^{k}),u^{k+1}-u^{k}\rangle . \end{aligned}$$

(A3)

Since $\varOmega (u)$ is ${\varvec{ C}}$-convex, $q^k\in {\varvec{ C}}^*$, then $\langle q^k,\varOmega (u)\rangle$ is convex and

$$\begin{aligned} \varLambda _2= & {} \langle q^k,\varOmega (u^{k})-\varOmega (u)+\nabla \varOmega (u^{k})(u-u^{k})\rangle +\big (\langle q^k,\varOmega (u^{k+1})-\varOmega (u^{k})\nonumber \\&-\nabla \varOmega (u^{k})(u^{k+1}-u^{k})\rangle \big )\nonumber \\\leqslant & {} \langle q^k,\varOmega (u^{k+1})-\varOmega (u^{k})-\nabla \varOmega (u^{k})(u^{k+1}-u^{k})\rangle . \end{aligned}$$

(A4)

Since $K(\cdot )$ satisfies Assumption 2, simple algebraic operation follows that

$$\begin{aligned} \varLambda _3= & {} \frac{1}{\varepsilon ^k}\langle \nabla K(u^{k+1})-\nabla K(u^k),u-u^{k+1}\rangle \nonumber \\= & {} \frac{1}{\varepsilon ^k}\big [D(u,u^k)-D(u,u^{k+1})-D(u^{k+1},u^k)\big ]. \end{aligned}$$

(A5)

Take $\varLambda _1$, $\varLambda _2$ and $\varLambda _3$ into (A2), we have

$$\begin{aligned} L(u^{k+1},q^k)-L(u,q^k)\leqslant & {} \frac{1}{\varepsilon ^k}D(u,u^k)-\frac{1}{\varepsilon ^{k}}D(u,u^{k+1})-\frac{1}{\varepsilon ^k}\bigg \{D(u^{k+1},u^{k})\;\\&-\varepsilon ^k\bigg [\big (G(u^{k+1})-G(u^{k})-\langle \nabla G(u^{k}),u^{k+1}-u^{k}\rangle \big )\\&+\langle q^k,\varOmega (u^{k+1})-\varOmega (u^{k})-\nabla \varOmega (u^{k})(u^{k+1}-u^{k})\rangle \bigg ]\bigg \}. \end{aligned}$$

Multiply $\varepsilon ^k$ on both side of the above inequality, and we have that

$$\begin{aligned}&\varepsilon ^k[L(u^{k+1},q^k)-L(u,q^k)]\nonumber \\&\leqslant D(u,u^k)-D(u,u^{k+1})-\varDelta ^k(u^k,u^{k+1})-\frac{\varepsilon ^k\gamma }{2}\Vert \varTheta (u^{k})-\varTheta (u^{k+1})\Vert ^2. \end{aligned}$$

(A6)

Step 2. Estimate $L(u^{k+1},p)-L(u^{k+1},q^k).$

We first derive two inequalities. By the property of projection (16) with $u=p^k+\gamma \varTheta (u^{k+1})$, $v=p$, $\forall p\in {\varvec{C}}^*$, we have

$$\begin{aligned} \frac{1}{\gamma }\langle p-p^{k+1}, p^k+\gamma \varTheta (u^{k+1})-p^{k+1}\rangle \leqslant 0. \end{aligned}$$

(A7)

Using Proposition 1 with $u=\gamma \varTheta (u^{k+1})$, $v=\gamma \varTheta (u^k)$, and $w=p^k$, we have

$$\begin{aligned}&2\langle p^{k+1}-q^k,\gamma \varTheta (u^{k+1})\rangle \leqslant \Vert \gamma \varTheta (u^{k+1})\nonumber \\&\quad -\gamma \varTheta (u^k)\Vert ^2+\Vert p^{k+1}-p^k\Vert ^2-\Vert q^k-p^k\Vert ^2. \end{aligned}$$

(A8)

Statement (ii) follows from (A7) and (A8):

$$\begin{aligned}&L(u^{k+1},p)-L(u^{k+1},q^k)\nonumber \\= & {} \langle p-q^k,\varTheta (u^{k+1})\rangle \nonumber \\= & {} \langle p-p^{k+1},\varTheta (u^{k+1})\rangle +\langle p^{k+1}-q^k,\varTheta (u^{k+1})\rangle \nonumber \\= & {} \frac{1}{\gamma }\langle p-p^{k+1},p^k+\gamma \varTheta (u^{k+1})-p^{k+1}\rangle +\frac{1}{\gamma }\langle p-p^{k+1},p^{k+1}-p^k\rangle \nonumber \\&+\langle p^{k+1}-q^k,\varTheta (u^{k+1})\rangle \nonumber \\\leqslant & {} \frac{1}{\gamma }\langle p-p^{k+1},p^{k+1}-p^k\rangle +\langle p^{k+1}-q^k,\varTheta (u^{k+1})\rangle \quad \text{(by } \text{ inequality }~(47))\nonumber \\\leqslant & {} \frac{1}{\gamma }\langle p-p^{k+1},p^{k+1}-p^{k}\rangle +\frac{1}{2\gamma }\Vert p^k-p^{k+1}\Vert ^2-\frac{1}{2\gamma }\Vert q^k-p^k\Vert ^2 \nonumber \\&+\frac{\gamma }{2}\Vert \varTheta (u^{k})-\varTheta (u^{k+1})\Vert ^2\quad \;\text{(by } \text{ inequality }~(48))\nonumber \\= & {} \frac{1}{2\gamma }\big [\Vert p-p^{k}\Vert ^2-\Vert p-p^{k+1}\Vert ^2\big ]-\frac{1}{2\gamma }\Vert q^k-p^k\Vert ^2 \nonumber \\&+\frac{\gamma }{2}\Vert \varTheta (u^{k})-\varTheta (u^{k+1})\Vert ^2. \end{aligned}$$

(A9)

Then, multiplying $\varepsilon ^k$ on both side of (A9), we obtain

$$\begin{aligned}&\varepsilon ^k[L(u^{k+1},p)-L(u^{k+1},q^k)]\nonumber \\&\leqslant \frac{\varepsilon ^k}{2\gamma }\big [\Vert p-p^{k}\Vert ^2-\Vert p-p^{k+1}\Vert ^2\big ]-\frac{\varepsilon ^k}{2\gamma }\Vert q^k-p^k\Vert ^2+\frac{\varepsilon ^k\gamma }{2}\Vert \varTheta (u^{k})-\varTheta (u^{k+1})\Vert ^2\nonumber \\&\leqslant \frac{\varepsilon ^k}{2\gamma }\Vert p-p^{k}\Vert ^2-\frac{\varepsilon ^{k+1}}{2\gamma }\Vert p-p^{k+1}\Vert ^2-\frac{\varepsilon ^{k}}{2\gamma }\Vert q^k-p^k\Vert ^2+\frac{\varepsilon ^k\gamma }{2}\Vert \varTheta (u^{k})-\varTheta (u^{k+1})\Vert ^2\nonumber \\&\text{(since } \varepsilon ^{k+1}\leqslant \varepsilon ^k). \end{aligned}$$

(A10)

Step 3. Estimate $L(u^{k+1},p)-L(u,q^k)$:

Summing (A6) and (A10), the desired result is coming.

A$_2$: Proof of Theorem 1(Convergence analysis for VAPP)

Take $u=u^{*}$ and $p=p^*$ in Lemma 1, then we have that

$$\begin{aligned}&\big [D(u^*,u^{k+1})+\frac{\varepsilon ^{k+1}}{2\gamma }\Vert p^*-p^{k+1}\Vert ^2\big ]-\big [D(u^*,u^k)+\frac{\varepsilon ^k}{2\gamma }\Vert p^*-p^{k}\Vert ^2\big ]\nonumber \\&\leqslant \varepsilon ^k[L(u^*,q^k)-L(u^{k+1},p^*)]-\big [\varDelta ^k(u^k,u^{k+1})+\frac{\varepsilon ^k}{2\gamma }\Vert q^k-p^k\Vert ^2\big ]\nonumber \\&\leqslant -\big [\varDelta ^k(u^k,u^{k+1})+\frac{\varepsilon ^k}{2\gamma }\Vert q^k-p^k\Vert ^2\big ]\quad \text{(since } (u^*,p^*) \text{ is } \text{ a } \text{ saddle } \text{ point }~(24))\nonumber \\&\leqslant -\bigg [\frac{\beta -\varepsilon ^k(B_G+B_{\varOmega }+\gamma \tau ^2)}{2}\Vert u^{k}-u^{k+1}\Vert ^2+\frac{\varepsilon ^k}{2\gamma }\Vert q^k-p^k\Vert ^2\bigg ]\nonumber \\&\qquad \qquad \qquad \quad \;\text{(from }~(26), \varDelta ^k(u,v)\ge \frac{\beta -\varepsilon ^k(B_G+B_{\varOmega }+\gamma \tau ^2)}{2}\Vert u-v\Vert ^2)\nonumber \\&\leqslant -\bigg [\frac{\beta -{\overline{\varepsilon }}(B_G+B_{\varOmega }+\gamma \tau ^2)}{2}\Vert u^{k}-u^{k+1}\Vert ^2+\frac{{\underline{\varepsilon }}}{2\gamma }\Vert q^k-p^k\Vert ^2\bigg ]\nonumber \\&\qquad \text{(since } {\underline{\varepsilon }}\leqslant \varepsilon ^k\leqslant {\overline{\varepsilon }} \text{ satisfy }~(21)). \end{aligned}$$

(A11)

Since $\{\varepsilon ^k\}$ satisfies (21), we conclude that the sequence $\{D(u^*,u^{k})+\frac{\varepsilon ^{k}}{2\gamma }\Vert p^*-p^{k}\Vert ^2\}$ is strictly decreasing, unless $u^k=u^{k+1}$ and $p^k=q^k$ or $p^k=p^{k+1}$. The rest of proof is similar to that of [24].

A$_3$: Proof of Theorem 2(Bifunction value estimation, primal optimality and feasibility for solving (P) by VAPP)

• (i) Note that the set ${\varvec{U}}\times {\varvec{C}}^*$ is convex, and the VAPP scheme guarantees that $(u^k,p^k)\in {\varvec{U}}\times {\varvec{C}}^*$, $\forall k\in {\mathbb {N}}$; thus we have $({\bar{u}}_t,{\bar{p}}_t)\in {\varvec{U}}\times {\varvec{C}}^*$. Since $\{\varepsilon ^k\}$ satisfies (21), then $\varDelta ^k(u^k,u^{k+1})\ge 0$. From Lemma 1, we have

  $$\begin{aligned}&\varepsilon ^k[L(u^{k+1},p)-L(u,q^k)]\leqslant \big [D(u,u^k)+\frac{\varepsilon ^k}{2\gamma }\Vert p-p^{k}\Vert ^2\big ]\\&\quad -\big [D(u,u^{k+1})+\frac{\varepsilon ^{k+1}}{2\gamma }\Vert p-p^{k+1}\Vert ^2\big ]. \end{aligned}$$

  Note that the bifunction $L(u',p)-L(u,p')$ is convex in $u'$ and linear in $p'$ for given $u\in {\varvec{U}}$, $p\in {\varvec{C}}^*$. Summing the above inequality over $k=0,1,\cdots ,t$, we obtain that

  $$\begin{aligned} L({\bar{u}}_{t},p)-L(u,{\bar{p}}_t)\leqslant & {} \frac{1}{\sum _{k=0}^{t}\varepsilon ^k}\sum _{k=0}^{t}\varepsilon ^k[L(u^{k+1},p)-L(u,q^k)]\\\leqslant & {} \frac{1}{{\underline{\varepsilon }}(t+1)}\left[ D(u,u^0)+\frac{\varepsilon ^0}{2\gamma }\Vert p-p^{0}\Vert ^2\right] ,\;\forall u\in {\varvec{ U}},p\in {\varvec{ C}}^*. \end{aligned}$$

• (ii) If $\Vert \varPi (\varTheta ({\bar{u}}_t))\Vert =0$, statement (ii) is obviously true.

  Otherwise, taking $u=u^*\in {\varvec{U}}$ and $p={\hat{p}}=\frac{(M_0+1)\varPi (\varTheta ({\bar{u}}_t))}{\Vert \varPi (\varTheta ({\bar{u}}_t))\Vert }\in {\varvec{C}}^*$ in statement (i) of this theorem, we have that

  $$\begin{aligned}&L({\bar{u}}_t,{\hat{p}})-L(u^*,{\bar{p}}_t)\nonumber \\= & {} (G+J)({\bar{u}}_{t})-(G+J)(u^*)\nonumber \\&+\langle \frac{(M_0+1)\varPi (\varTheta ({\bar{u}}_t))}{\Vert \varPi (\varTheta ({\bar{u}}_t))\Vert }, \varTheta ({\bar{u}}_{t})\rangle -\langle {\bar{p}}_t, \varTheta (u^*)\rangle \nonumber \\\ge & {} (G+J)({\bar{u}}_{t})-(G+J)(u^*)\nonumber \\&+\langle \frac{(M_0+1)\varPi (\varTheta ({\bar{u}}_t))}{\Vert \varPi (\varTheta ({\bar{u}}_t))\Vert }, \varTheta ({\bar{u}}_{t})\rangle \;\;\text{(since } \langle {\bar{p}}_t, \varTheta (u^*)\rangle \leqslant 0)\nonumber \\= & {} (G+J)({\bar{u}}_{t})-(G+J)(u^*)\nonumber \\&+\langle \frac{(M_0+1)\varPi (\varTheta ({\bar{u}}_t))}{\Vert \varPi (\varTheta ({\bar{u}}_t))\Vert }, \varPi (\varTheta ({\bar{u}}_t))+\varPi _{-{\varvec{C}}}(\varTheta ({\bar{u}}_t))\rangle \quad \text{(from }~(19))\nonumber \\= & {} (G+J)({\bar{u}}_{t})-(G+J)(u^*)+(M_0+1)\Vert \varPi (\varTheta ({\bar{u}}_t))\Vert \quad \text{(from }~(20)).\nonumber \\ \end{aligned}$$

  (A12)

  Combining statement (i) of this theorem, (A12) yields that

  $$\begin{aligned} (G+J)({\bar{u}}_{t})-(G+J)(u^*)+(M_0+1)\Vert \varPi (\varTheta ({\bar{u}}_t))\Vert\leqslant & {} \frac{D(u^*,u^0)+\frac{\varepsilon ^0}{2\gamma }\Vert {\hat{p}}-p^{0}\Vert ^2}{{\underline{\varepsilon }}(t+1)}\nonumber \\\leqslant & {} \frac{d_1}{{\underline{\varepsilon }}(t+1)}, \end{aligned}$$

  (A13)

  where $d_1=\max \nolimits _{\Vert p\Vert \leqslant M_0+1}\big [D(u^*,u^0)+\frac{\varepsilon ^0}{2\gamma }\Vert p-p^{0}\Vert ^2\big ]$. Moreover, taking $u={\bar{u}}_t$ in the right-hand side of saddle point inequality (5) yields that

  $$\begin{aligned} (G+J)({\bar{u}}_{t})-(G+J)(u^*)\ge & {} -\langle p^*,\varTheta ({\bar{u}}_t)\rangle \nonumber \\= & {} -\langle p^*,\varPi (\varTheta ({\bar{u}}_t))+\varPi _{-{\varvec{C}}}(\varTheta ({\bar{u}}_t))\rangle \quad \text{(since }~(19))\nonumber \\\ge & {} -\langle p^*,\varPi (\varTheta ({\bar{u}}_t))\rangle \quad \text{(since } \langle p^*,\varPi _{-{\varvec{C}}}(\varTheta ({\bar{u}}_t))\rangle \leqslant 0)\nonumber \\\ge & {} -\Vert p^*\Vert \Vert \varPi (\varTheta ({\bar{u}}_t))\Vert \nonumber \\\ge & {} -M_0\Vert \varPi (\varTheta ({\bar{u}}_t))\Vert \quad \text{(by } \Vert p^*\Vert \leqslant M_0). \end{aligned}$$

  (A14)

  Taking (A13) and (A14) together, we get that $\Vert \varPi (\varTheta ({\bar{u}}_t))\Vert \leqslant \frac{d_1}{{\underline{\varepsilon }}(t+1)}$.

• (iii) Since $(M_0+1)\Vert \varPi (\varTheta ({\bar{u}}_t))\Vert \ge 0$, from (A13) we have

  $$\begin{aligned} (G+J)({\bar{u}}_{t})-(G+J)(u^*)\leqslant \frac{d_1}{{\underline{\varepsilon }}(t+1)}. \end{aligned}$$

  Combining statement (ii) of this theorem and (A14), we obtain that

  $$\begin{aligned} (G+J)({\bar{u}}_{t})-(G+J)(u^*)\ge -\frac{M_0d_1}{{\underline{\varepsilon }}(t+1)}. \end{aligned}$$

A$_4:$ Proof of Lemma 2

Suppose the assertion of the lemma does not hold, that is, for any $\kappa >0$, there is $\Vert p^j\Vert \leqslant d_p$ so that all optimizers ${\hat{u}}(p^j)\in \arg \min \nolimits _{u\in {\varvec{U}}}L_\gamma (u,p^j)$ satisfy $\Vert {\hat{u}}(p^j)\Vert >\kappa$. Then, we construct a sequence $\{{\hat{u}}(p^j)\}$ such that $\Vert {\hat{u}}(p^j)\Vert \rightarrow +\infty$.

On the other hand, we observe that

$$\begin{aligned} L_\gamma ({\hat{u}}(p^j),p^j)= & {} (G+J)({\hat{u}}(p^j))+\varphi \big (\varTheta ({\hat{u}}(p^j)),p^j\big )\\= & {} (G+J)({\hat{u}}(p^j))+\max _{q\in {\varvec{C}}^*}\langle q,\varTheta ({\hat{u}}(p^j))\rangle -\frac{1}{2\gamma }\Vert q-p^j\Vert ^2\\\ge & {} (G+J)({\hat{u}}(p^j))-\frac{1}{2\gamma }\Vert p^j\Vert ^2\\\ge & {} (G+J)({\hat{u}}(p^j))-\frac{d_p^2}{2\gamma }. \end{aligned}$$

Since $\Vert {\hat{u}}(p^j)\Vert \rightarrow +\infty$, from the coercivity of $(G+J)(u)$, we have $\psi _\gamma (p^j)=L_\gamma ({\hat{u}}(p^j),p^j)\rightarrow +\infty$. However, from the boundness of $\{p^j\}$ and the continuity of $\psi _\gamma (\cdot )$, we conclude that $\psi _\gamma (p^j)$ is bounded, which follows one contradiction and assertion of lemma is provided.

A$_5:$ Proof of Theorem 3(Approximate saddle point and dual optimality for solving (P) by VAPP)

• (i) From statement (i) of Theorem 2, it is easy to have that, for any $(u,p)\in ({\varvec{ U}}\cap {\mathfrak {B}}^{u})\times ({\varvec{C}}^*\cap {\mathfrak {B}}^{p})$,

  $$\begin{aligned} L({\bar{u}}_{t},p)-L(u,{\bar{p}}_{t})\leqslant \frac{D(u,u^0)+\frac{\varepsilon ^0}{2\gamma }\Vert p-p^{0}\Vert ^2}{{\underline{\varepsilon }}(t+1)}\leqslant \frac{d_2}{{\underline{\varepsilon }}(t+1)}, \end{aligned}$$

  (A15)

  where $d_2=\max _{(u,p)\in ({\varvec{U}}\cap {\mathfrak {B}}^{u})\times ({\varvec{C}}^*\cap {\mathfrak {B}}^{p}))}\big [D(u,u^0)+\frac{\varepsilon ^0}{2\gamma }\Vert p-p^{0}\Vert ^2\big ]$.

  Since ${\bar{u}}_t\in {\varvec{U}}\cap {\mathfrak {B}}^{u}$, then taking $u={\bar{u}}_t$ in (A15), we obtain

  $$\begin{aligned} L({\bar{u}}_{t},p)-L({\bar{u}}_{t},{\bar{p}}_{t})\leqslant \frac{d_2}{{\underline{\varepsilon }}(t+1)}, \forall p\in {\varvec{C}}^*\cap {\mathfrak {B}}^{p}. \end{aligned}$$

  (A16)

  Similarly, by taking $p={\bar{p}}_t\in {\varvec{C}}^*\cap {\mathfrak {B}}^{p}$ in (A15), we obtain

  $$\begin{aligned} L({\bar{u}}_{t},{\bar{p}}_t)-L(u,{\bar{p}}_{t})\leqslant \frac{d_2}{{\underline{\varepsilon }}(t+1)}, \forall u\in {\varvec{U}}\cap {\mathfrak {B}}^{u}. \end{aligned}$$

  (A17)

• (ii) In the left-hand side of inequality in statement (i), taking $p=0$, we get $\langle {\bar{p}}_t,\varTheta ({\bar{u}}_t)\rangle \ge -\frac{d_2}{{\underline{\varepsilon }}(t+1)}$. Then, from (13), we have

  $$\begin{aligned} \varphi \big (\varTheta ({\bar{u}}_t),{\bar{p}}_t\big )\ge \langle {\bar{p}}_t,\varTheta ({\bar{u}}_t)\rangle \ge -\frac{d_2}{{\underline{\varepsilon }}(t+1)}. \end{aligned}$$

  (A18)

  On the other hand, for $p\in {\varvec{C}}^*\cap {\mathfrak {B}}^{p}$, we have

  $$\begin{aligned} \varphi \big (\varTheta ({\bar{u}}_t),p\big )= & {} \min _{\xi \in -{\varvec{C}}}\langle p,\varTheta ({\bar{u}}_t)-\xi \rangle +\frac{\gamma }{2}\Vert \varTheta ({\bar{u}}_t)-\xi \Vert ^2\qquad \text{(from }~(12))\nonumber \\\leqslant & {} \langle p,\varTheta ({\bar{u}}_t)-\varPi _{-{\varvec{C}}}(\varTheta ({\bar{u}}_t))\rangle +\frac{\gamma }{2}\Vert \varTheta ({\bar{u}}_t)-\varPi _{-{\varvec{C}}}(\varTheta ({\bar{u}}_t))\Vert ^2\nonumber \\\leqslant & {} \Vert p\Vert \cdot \Vert \varPi (\varTheta ({\bar{u}}_t))\Vert +\frac{\gamma }{2}\Vert \varPi (\varTheta ({\bar{u}}_t))\Vert ^2\nonumber \\\leqslant & {} \frac{r^pd_1}{{\underline{\varepsilon }}(t+1)}+\frac{\gamma (d_1)^2}{2{\underline{\varepsilon }}^2(t+1)^2}\nonumber \\&\text{(from } \text{ statment } \text{(ii) } \text{ of } \text{ Theorem }~2 \,\text{ and }\, {p}\in {\varvec{C}}^*\cap {\mathfrak {B}}^{p}). \end{aligned}$$

  (A19)

  Therefore, we get the left-hand side of inequality in statement (ii):

  $$\begin{aligned} L_{\gamma }({\bar{u}}_{t},p)-L_{\gamma }({\bar{u}}_{t},{\bar{p}}_{t})= & {} \varphi (\varTheta ({\bar{u}}_t),p)-\varphi (\varTheta ({\bar{u}}_t),{\bar{p}}_t)\nonumber \\\leqslant & {} \frac{r^pd_1+d_2}{{\underline{\varepsilon }}(t+1)}+\frac{\gamma (d_1)^2}{2{\underline{\varepsilon }}^2(t+1)^2}. \end{aligned}$$

  (A20)

  From (A18) and (A19), it also has that

  $$\begin{aligned} -\frac{d_2}{{\underline{\varepsilon }}(t+1)}\leqslant \langle {\bar{p}}_t,\varTheta ({\bar{u}}_t)\rangle \leqslant \varphi (\varTheta ({\bar{u}}_t),{\bar{p}}_t)\leqslant \frac{r^p d_1}{{\underline{\varepsilon }}(t+1)}+\frac{\gamma (d_1)^2}{2{\underline{\varepsilon }}^2(t+1)^2}, \end{aligned}$$

  which follows that

  $$\begin{aligned} \varphi (\varTheta ({\bar{u}}_t),{\bar{p}}_t)-\langle {\bar{p}}_t,\varTheta ({\bar{u}}_t)\rangle\leqslant & {} \frac{r^pd_1}{{\underline{\varepsilon }}(t+1)}+\frac{\gamma (d_1)^2}{2{\underline{\varepsilon }}^2(t+1)^2}-(-\frac{d_2}{{\underline{\varepsilon }}(t+1)})\\= & {} \frac{r^pd_1+d_2}{{\underline{\varepsilon }}(t+1)}+\frac{\gamma (d_1)^2}{2{\underline{\varepsilon }}^2(t+1)^2}. \end{aligned}$$

  Then, for $u\in {\varvec{U}}\cap {\mathfrak {B}}^{u}$, we have

  $$\begin{aligned}&L_\gamma ({\bar{u}}_{t},{\bar{p}}_{t})\leqslant L({\bar{u}}_{t},{\bar{p}}_{t})+\frac{r^pd_1+d_2}{{\underline{\varepsilon }}(t+1)}+\frac{\gamma (d_1)^2}{2{\underline{\varepsilon }}^2(t+1)^2}\nonumber \\&\quad \leqslant L(u,{\bar{p}}_{t})+\frac{r^pd_1+2d_2}{{\underline{\varepsilon }}(t+1)}+\frac{\gamma (d_1)^2}{2{\underline{\varepsilon }}^2(t+1)^2}\text{(by } \text{ right-hand } \text{ side } \text{ of } \text{ statement } \text{(i)) }\nonumber \\&\quad \leqslant L_\gamma (u,{\bar{p}}_{t})+\frac{r^pd_1+2d_2}{{\underline{\varepsilon }}(t+1)}+\frac{\gamma (d_1)^2}{2{\underline{\varepsilon }}^2(t+1)^2}\quad \text{(from }~(13)), \end{aligned}$$

  (A21)

  which follows the right-hand side of inequality in statement (ii).

• (iii) For saddle point $(u^*,p^*)$, we have

  $$\begin{aligned} L_{\gamma }(u^*,p)\leqslant L_{\gamma }(u^*,p^*)\leqslant L_{\gamma }(u,p^*), \forall u\in {\varvec{U}}, p\in {\mathbb {R}}^m. \end{aligned}$$

  (A22)

  Taking $u={\bar{u}}_t$, $p={\bar{p}}_t$ in (A22), and taking $u={\hat{u}}({\bar{p}}_t)$, $p=p^*$ in statement (ii) of this theorem, we obtain the following two inequalities, respectively:

  $$\begin{aligned} L_{\gamma }(u^*,{\bar{p}}_t)\leqslant&L_{\gamma }(u^*,p^*)&\leqslant L_{\gamma }({\bar{u}}_t,p^*), \end{aligned}$$

  and

  $$\begin{aligned}&-\frac{r^pd_1+d_2}{{\underline{\varepsilon }}(t+1)}-\frac{\gamma (d_1)^2}{2{\underline{\varepsilon }}^2(t+1)^2}+L_{\gamma }({\bar{u}}_{t},p^*)\leqslant L_{\gamma }({\bar{u}}_{t},{\bar{p}}_{t})\leqslant L_{\gamma }({\hat{u}}({\bar{p}}_t),{\bar{p}}_{t})\\&\quad +\frac{r^pd_1+2d_2}{{\underline{\varepsilon }}(t+1)}+\frac{\gamma (d_1)^2}{2{\underline{\varepsilon }}^2(t+1)^2}. \end{aligned}$$

  Combining these two inequalities, the desired inequality is obtained:

  $$\begin{aligned}&-\frac{r^pd_1+d_2}{{\underline{\varepsilon }}(t+1)}-\frac{\gamma (d_1)^2}{2{\underline{\varepsilon }}^2(t+1)^2}+L_{\gamma }(u^*,p^*)\leqslant L_{\gamma }({\hat{u}}({\bar{p}}_t),{\bar{p}}_{t})\\&\quad +\frac{r^pd_1+2d_2}{{\underline{\varepsilon }}(t+1)}+\frac{\gamma (d_1)^2}{2{\underline{\varepsilon }}^2(t+1)^2}. \end{aligned}$$

  Therefore,

  $$\begin{aligned} \psi _\gamma (p^*)=L_{\gamma }(u^*,p^*)\leqslant & {} L_{\gamma }({\hat{u}}({\bar{p}}_t),{\bar{p}}_{t})+\frac{2r^pd_1+3d_2}{{\underline{\varepsilon }}(t+1)}+\frac{\gamma (d_1)^2}{{\underline{\varepsilon }}^2(t+1)^2}\nonumber \\= & {} \psi _\gamma ({\bar{p}}_t)+\frac{2r^pd_1+3d_2}{{\underline{\varepsilon }}(t+1)}+\frac{\gamma (d_1)^2}{{\underline{\varepsilon }}^2(t+1)^2}. \end{aligned}$$

  (A23)