Self-adaptive ADMM for semi-strongly convex problems

Abstract

In this paper, we develop a self-adaptive ADMM that updates the penalty parameter adaptively. When one part of the objective function is strongly convex i.e., the problem is semi-strongly convex, our algorithm can update the penalty parameter adaptively with guaranteed convergence. We establish various types of convergence results including accelerated convergence rate of $O(1/k^2),$ linear convergence and convergence of iteration points. This enhances various previous results because we allow the penalty parameter to change adaptively. We also develop a partial proximal point method with the subproblems being solved by our adaptive ADMM. This enables us to solve problems without semi-strongly convex property. Numerical experiments are conducted to demonstrate the high efficiency and robustness of our method.

Proof details

1.1 Proof of Lemma 1

Proof

From the optimality conditions in step 1 and 2, we have that

$$\begin{aligned}&-\Big (B^\top \lambda ^k+\beta _kB^\top (By^{k+1}+Cz^k-b)+P_k(y^{k+1}-y^k)\Big ) \in \partial f(y^{k+1}) \end{aligned}$$

(56)

$$\begin{aligned}&-\Big (C^\top \lambda ^k+\beta _kC^\top (By^{k+1}+Cz^{k+1}-b)+Q_k(z^{k+1}-z^k)\Big ) \in \partial g(z^{k+1}). \end{aligned}$$

(57)

From (56) and the convexity of f, we have that

$$\begin{aligned} f(y^{k+1})-f(y)&\le \langle B^\top \lambda ^k+\beta _kB^\top (By^{k+1}+Cz^k-b)+P_k(y^{k+1}\!-\!y^k),\,y\!-\!y^{k+1}\rangle \nonumber \\&=\langle \lambda ^k+\beta _k(By^{k+1}+Cz^k-b),\,By-By^{k+1}\rangle +\eta _{P_k}(y,y^k,y^{k+1}). \end{aligned}$$

(58)

Similarly, from (57) and (2), we have that

$$\begin{aligned}&g(z^{k+1})-g(z) \nonumber \\&\quad \le \langle \lambda ^k+\beta _k(By^{k+1}+Cz^{k+1}-b),\,Cz-Cz^{k+1}\rangle \nonumber \\&\qquad +\eta _{Q_k}(z,z^k,z^{k+1})-\frac{\sigma _g}{2}\Vert z^{k+1}-z\Vert ^{2}\nonumber \\&\quad =\langle \lambda ^k+\beta _k(By^{k+1}+Cz^k-b),Cz-Cz^{k+1}\rangle \nonumber \\&\qquad +\eta _{\beta _kC^\top C+Q_k}(z,z^k,z^{k+1})-\frac{\sigma _g}{2}\Vert z^{k+1}-z\Vert ^{2}. \end{aligned}$$

(59)

From (58), (59) we have that

$$\begin{aligned}&{\mathcal {F}}(x^{k+1})-{\mathcal {F}}(x)\nonumber \\&\quad \le \langle \lambda ^k+\beta _k(By^{k+1}+Cz^k-b),\,b-{\mathcal {A}}x^{k+1}\rangle \nonumber \\&\qquad +\eta _{\beta _kC^\top C+Q_k}(z,z^k,z^{k+1})+\eta _{P_k}(y,y^k,y^{k+1}) \nonumber \\&\quad \quad -\frac{\sigma _g}{2}\Vert z-z^{k+1}\Vert ^{2} \nonumber \\&\quad =\langle \lambda ^k+\beta _k({\mathcal {A}}x^{k+1}-b),b-{\mathcal {A}}x^{k+1}\rangle +\beta _k\langle C(z^k-z^{k+1}),b-{\mathcal {A}}x^{k+1}\rangle \nonumber \\&\quad \quad +\eta _{\beta _k C^\top C+Q_k}(z,z^k,z^{k+1}) +\eta _{P_k}(y,y^k,y^{k+1})-\frac{\sigma _g}{2}\Vert z-z^{k+1}\Vert ^2 \nonumber \\&\quad =\langle \lambda ^{k+1},b-{\mathcal {A}}x^{k+1}\rangle +(\gamma -1)\beta _k\Vert {\mathcal {A}}x^{k+1}\!-\!b\Vert ^2\!+\!\beta _k\langle C(z^k\!-\!z^{k+1}),b\!-\!{\mathcal {A}}x^{k+1}\rangle \nonumber \\&\quad \quad +\eta _{\beta _k C^\top C+Q_k}(z,z^k,z^{k+1}) +\eta _{P_k}(y,y^k,y^{k+1})-\frac{\sigma _g}{2}\Vert z-z^{k+1}\Vert ^2, \end{aligned}$$

(60)

where we have used step 3 to get the last equality. From (60), we have

$$\begin{aligned}&{\mathcal {L}}(x^{k+1},\lambda )-{\mathcal {L}}(x,\lambda )\nonumber \\&\quad \le \langle \lambda ^{k+1}-\lambda ,b-{\mathcal {A}}x^{k+1}\rangle +(\gamma -1)\beta _k\Vert {\mathcal {A}}x^{k+1}-b\Vert ^2\nonumber \\&\qquad +\beta _k\langle C(z^k-z^{k+1}),b-{\mathcal {A}}x^{k+1}\rangle \nonumber \\&\quad \quad +\eta _{\beta _k C^\top C+Q_k}(z,z^k,z^{k+1})+\eta _{P_k}(y,y^k,y^{k+1})-\frac{\sigma _g}{2}\Vert z-z^{k+1}\Vert ^2\nonumber \\&\quad =\Big \langle \lambda ^{k+1}-\lambda ,\frac{\lambda ^k-\lambda ^{k+1}}{\beta _k \gamma }\Big \rangle +(\gamma -1)\beta _k\Vert {\mathcal {A}}x^{k+1}-b\Vert ^2\nonumber \\&\qquad +\beta _k\langle C(z^k-z^{k+1}),b-{\mathcal {A}}x^{k+1}\rangle \nonumber \\&\quad \quad +\eta _{\beta _k C^\top C+Q_k}(z,z^k,z^{k+1}) +\eta _{P_k}(y,y^k,y^{k+1})-\frac{\sigma _g}{2}\Vert z-z^{k+1}\Vert ^2. \end{aligned}$$

(61)

Now, we need to estimate $\beta _k\langle C(z^k-z^{k+1}),b-{\mathcal {A}}x^{k+1}\rangle$. From (57), we know that

$$\begin{aligned}&-C^\top \lambda ^k-\beta _k C^\top ({\mathcal {A}}x^{k+1}-b)-Q_k(z^{k+1}-z^k)\in \partial g(z^{k+1})\\&-C^\top \lambda ^{k-1}-\beta _{k-1}C^\top ({\mathcal {A}}x^k-b)-Q_{k-1}(z^k-z^{k-1})\in \partial g(z^k) \end{aligned}$$

Combining the above two equations together with the strongly convexity of g, we get

$$\begin{aligned} \Big \langle \begin{array}{c} C^\top (\lambda ^{k-1}-\lambda ^k)-\beta _k C^\top ({\mathcal {A}}x^{k+1}-b)-Q_k(z^{k+1}-z^k) \\ +\beta _{k-1}C^\top ({\mathcal {A}}x^k-b)+Q_{k-1}(z^k-z^{k-1}) \end{array},z^{k+1}-z^k \Big \rangle \ge \sigma _g \Vert z^k-z^{k+1}\Vert ^2, \end{aligned}$$

which, together with step 3, implies that

$$\begin{aligned}&\sigma _g \Vert z^k-z^{k+1}\Vert ^2 \\&\quad \le \langle \lambda ^{k-1}+\beta _{k-1}({\mathcal {A}}x^k-b)-\lambda ^k-\beta _k({\mathcal {A}}x^{k+1}-b),C(z^{k+1}-z^k)\rangle \\&\quad \quad +\langle -Q_k(z^{k+1}-z^k)+Q_{k-1}(z^k-z^{k-1}),z^{k+1}-z^k\rangle \\&\quad =\langle (1-\gamma )\beta _{k-1}({\mathcal {A}}x^k-b)-\beta _k({\mathcal {A}}x^{k+1}-b),C(z^{k+1}-z^k)\rangle \\&\quad \quad +\langle -Q_{k-1}(z^{k+1}-z^k)+Q_{k-1}(z^k-z^{k-1}),z^{k+1}-z^k\rangle \\&\qquad -(\beta _k-\beta _{k-1})\Vert z^{k+1}-z^k\Vert ^2_Q. \end{aligned}$$

The above inequality implies that

$$\begin{aligned}&\beta _k \langle {\mathcal {A}}x^{k+1}-b,C(z^{k+1}-z^k)\rangle \\&\quad \le (\gamma -1)\langle \beta _{k-1}(b-{\mathcal {A}}x^k),C(z^{k+1}-z^k)\rangle -\Vert z^{k+1}-z^k\Vert ^2_{Q_{k}} \\&\quad \quad +\beta _{k-1}\langle Q(z^k-z^{k-1}),z^{k+1}-z^k\rangle -\sigma _g\Vert z^k-z^{k+1}\Vert ^2 \\&\quad \le \frac{(\gamma -1)\beta _{k-1}^2}{2\gamma \beta _k}\Vert {\mathcal {A}}x^k-b\Vert ^2 +\frac{(\gamma -1)\gamma \beta _k}{2}\Vert C(z^{k+1}-z^k)\Vert ^2-\Vert z^{k+1}-z^k\Vert ^2_{Q_k}\\&\quad \quad +\frac{\beta _{k-1}^2}{2\beta _k}\Vert z^k-z^{k-1}\Vert ^2_Q+\frac{\beta _k}{2}\Vert z^{k+1}-z^k\Vert ^2_Q-\sigma _g\Vert z^k-z^{k+1}\Vert ^2. \\&\quad =\frac{(\gamma -1)\beta _{k-1}^2}{2\gamma \beta _k}\Vert {\mathcal {A}}x^k-b\Vert ^2 +\frac{(1-\delta )\beta _k}{2}\Vert C(z^{k+1}-z^k)\Vert ^2+\frac{\beta _{k-1}^2}{2\beta _k}\Vert z^k-z^{k-1}\Vert ^2_Q \\&\quad \quad -\frac{\beta _k}{2}\Vert z^{k+1}-z^k\Vert ^2_Q-\sigma _g\Vert z^k-z^{k+1}\Vert ^2, \end{aligned}$$

In the above, we use the fact that $\gamma (\gamma -1) = 1-\delta$. Now, we plug the above inequality into (61), we get

$$\begin{aligned}&{\mathcal {L}}(x^{k+1},\lambda )-{\mathcal {L}}(x,\lambda )\nonumber \\&\quad \le \Big \langle \lambda ^{k+1}-\lambda ,\frac{\lambda ^k-\lambda ^{k+1}}{\beta _k\gamma }\Big \rangle +(\gamma -1)\beta _k\Vert {\mathcal {A}}x^{k+1}-b\Vert ^2 +\frac{(\gamma -1)\beta _{k-1}^2}{2\gamma \beta _k}\Vert {\mathcal {A}}x^k\!-\!b\Vert ^2\nonumber \\&\quad \quad +\eta _{P_k}(y,y^k,y^{k+1})+\eta _{\beta _k C^\top C+Q_k}(z,z^k,z^{k+1}) +\frac{(1-\delta )\beta _k}{2}\Vert C(z^{k+1}-z^k)\Vert ^2\nonumber \\&\quad \quad +\frac{\beta _{k-1}^2}{2\beta _k}\Vert z^k-z^{k-1}\Vert ^2_Q-\frac{\beta _k}{2}\Vert z^{k+1}-z^k\Vert ^2_Q-\sigma _g\Vert z^k-z^{k+1}\Vert ^2 -\frac{\sigma _g}{2}\Vert z-z^{k+1}\Vert ^2\nonumber \\&\quad \le \Big \langle \lambda ^{k+1}-\lambda ,\frac{\lambda ^k-\lambda ^{k+1}}{\beta _k\gamma } \Big \rangle +(\gamma -1)\beta _k\Vert {\mathcal {A}}x^{k+1}-b\Vert ^2 \!+\!\frac{(\gamma -1)\beta _{k-1}^2}{2\gamma \beta _k}\Vert {\mathcal {A}}x^k\!-\!b\Vert ^2\nonumber \\&\quad \quad +\eta _{P_k}(y,y^k,y^{k+1})+\xi _{\beta _k C^\top C+Q_k}(z,z^k,z^{k+1}) - \frac{\delta \beta _k}{2}\Vert C(z^{k+1}-z^k)\Vert ^2\nonumber \\&\quad \quad +\frac{\beta _{k-1}^2}{2\beta _k}\Vert z^k-z^{k-1}\Vert ^2_Q-\beta _k\Vert z^{k+1}-z^k\Vert ^2_Q-\sigma _g\Vert z^k-z^{k+1}\Vert ^2-\frac{\sigma _g}{2}\Vert z-z^{k+1}\Vert ^2. \end{aligned}$$

(62)

Note that from step 4, we can derive that

$$\begin{aligned}{} & {} \beta _k\left( \xi _{\beta _k C^\top C+Q_k}(z,z^k,z^{k+1})-\frac{(1-\epsilon )\sigma _g}{2}\Vert z-z^{k+1}\Vert ^2\right) \nonumber \\{} & {} \quad \le \frac{\beta _k^2}{2}\Vert z-z^k\Vert ^2_{C^\top C+Q}-\frac{\beta _{k+1}^2}{2}\Vert z-z^{k+1}\Vert ^2_{ C^\top C+Q}. \end{aligned}$$

(63)

Multiply (62) by $\beta _k$ and use the above inequality, we obtain that

$$\begin{aligned}&\beta _k\left( {\mathcal {L}}(x^{k+1},\lambda )-{\mathcal {L}}(x,\lambda )\right) \\&\quad \le \frac{1}{\gamma }\langle \lambda ^{k+1}-\lambda , \lambda ^k-\lambda ^{k+1}\rangle +(\gamma -1)\beta _k^2\Vert {\mathcal {A}}x^{k+1}\\&\qquad -b\Vert ^2+\frac{(\gamma -1)\beta _{k-1}^2}{2\gamma }\Vert {\mathcal {A}}x^k-b\Vert ^2 \\&\quad \quad +\beta _k\eta _{P_k}(y,y^k,y^{k+1}) +\frac{\beta _k^2}{2}\Vert z-z^k\Vert ^2_{C^\top C+Q}-\frac{\beta _{k+1}^2}{2}\Vert z-z^{k+1}\Vert ^2_{C^\top C+Q} \\&\quad \quad -\frac{\delta \beta _k^2}{2}\Vert C(z^{k+1}-z^k)\Vert ^2+\frac{\beta _{k-1}^2}{2}\Vert z^k-z^{k-1}\Vert ^2_Q -\beta _k^2\Vert z^{k+1}\\&\qquad -z^k\Vert ^2_Q -\sigma _g\beta _k\Vert z^k-z^{k+1}\Vert ^2\\&\quad \quad -\frac{\epsilon \sigma _g\beta _k}{2}\Vert z-z^{k+1}\Vert ^2 \\&\quad =\frac{1}{\gamma }\xi (\lambda ,\lambda ^k,\lambda ^{k+1}) -\frac{(2-\gamma )\beta _k^2}{2}\Vert {\mathcal {A}}x^{k+1}-b\Vert ^2+\frac{(\gamma -1)\beta _{k-1}^2}{2\gamma }\Vert A x^{k}-b\Vert ^2\\&\quad \quad +\beta _k\eta _{P_k}(y,y^k,y^{k+1})+\frac{\beta _k^2}{2}\Vert z-z^k\Vert ^2_{C^\top C+Q}-\frac{\beta _{k+1}^2}{2}\Vert z-z^{k+1}\Vert ^2_{C^\top C+Q} \\&\quad \quad -\frac{\delta \beta _k^2}{2}\Vert C(z^{k+1}-z^k)\Vert ^2+\frac{\beta _{k-1}^2}{2}\Vert z^k-z^{k-1}\Vert ^2_Q -\beta _k^2\Vert z^{k+1}\\&\qquad -z^k\Vert ^2_Q -\sigma _g\beta _k\Vert z^k-z^{k+1}\Vert ^2\\&\quad \quad -\frac{\epsilon \sigma _g\beta _k}{2}\Vert z-z^{k+1}\Vert ^2 \end{aligned}$$

where we have used the fact that $\langle \lambda _{k+1}-\lambda ,\,\lambda _k-\lambda _{k+1}\rangle = \frac{1}{2}\Vert \lambda -\lambda _k\Vert ^2 -\frac{1}{2}\Vert \lambda -\lambda _{k+1}\Vert ^2 -\frac{1}{2}\Vert \lambda _k-\lambda _{k+1}\Vert ^2$ and $\lambda ^{k+1}-\lambda ^{k}=\gamma \beta _k ({\mathcal {A}}x^{k+1}-b).$ Note that since $\gamma \in (1,\frac{1+\sqrt{5}}{2}),$ $\delta = 1+\gamma -\gamma ^2 > 0$. Using the identity, $\frac{\gamma -1}{2\gamma }= \frac{(2-\gamma )}{2}-\frac{\delta }{2\gamma }$, we deduce that

$$\begin{aligned}&\beta _k\left( {\mathcal {L}}(x^{k+1},\lambda )-{\mathcal {L}}(x,\lambda )\right) +\frac{\delta \beta _{k-1}^2}{2\gamma }\Vert {\mathcal {A}}x^{k}-b\Vert ^2 \\&\quad \le \frac{1}{\gamma }\xi (\lambda ,\lambda ^k\lambda ^{k+1})+ \frac{(2-\gamma )\beta _{k-1}^2}{2}\Vert {\mathcal {A}}x^k-b\Vert ^2 -\frac{(2-\gamma )\beta _{k}^2}{2}\Vert {\mathcal {A}}x^{k+1}-b\Vert ^2 \\&\quad \quad +\beta _k\eta _{P_k}(y,y^k,y^{k+1}) +\frac{\beta _k^2}{2}\Vert z-z^k\Vert ^2_{C^\top C+Q}-\frac{\beta _{k+1}^2}{2}\Vert z-z^{k+1}\Vert ^2_{C^\top C+Q} \\&\quad \quad -\frac{\delta \beta _k^2}{2}\Vert C(z^{k+1}-z^k)\Vert ^2+\frac{\beta _{k-1}^2}{2}\Vert z^k-z^{k-1}\Vert ^2_Q -\beta _k^2\Vert z^{k+1}\\&\qquad -z^k\Vert ^2_Q-\sigma _g\beta _k\Vert z^k-z^{k+1}\Vert ^2\\&\quad \quad -\frac{\epsilon \sigma _g\beta _k}{2}\Vert z-z^{k+1}\Vert ^2. \end{aligned}$$

From here, one can readily get the required inequality in Lemma 1. $\square$

1.2 Proof of Lemma 2

Proof

Because $(x^*,\lambda ^*)$ is a KKT solution, we have that $0\in \partial _x {\mathcal {L}}(x^*,\lambda ^*)$. Since ${\mathcal {L}}(x,\lambda ^*)$ is a convex function of x, we then have that

$$\begin{aligned} {\mathcal {L}}(x^*,\lambda ^*)\le {\mathcal {L}}(x,\lambda ^*)\ \mathrm{for\ any}\ x, \end{aligned}$$

(64)

from which we get

$$\begin{aligned} -\langle \lambda ^*,\,{\mathcal {A}}x^k-b\rangle \le {\mathcal {F}}(x^k)-{\mathcal {F}}(x^*). \end{aligned}$$

(65)

Consider all $\lambda \in {\mathbb {R}}^m$ such that $\Vert \lambda \Vert \le \Vert \lambda ^*\Vert +1$ in ${\mathcal {L}}(x^k,\lambda )-{\mathcal {L}}(x^*,\lambda )\le h(k)D(\lambda )$, we have

$$\begin{aligned} {\mathcal {F}}(x^k)-{\mathcal {F}}(x^*)+(\Vert \lambda ^*\Vert +1)\Vert {\mathcal {A}}x^k-b\Vert \le h(k)\max _{\Vert \lambda \Vert \le \Vert \lambda ^*\Vert +1}D(\lambda ). \end{aligned}$$

(66)

Using (65) in (66), we get

$$\begin{aligned} \Vert {\mathcal {A}}x^k-b\Vert \le h(k)\max _{\Vert \lambda \Vert \le \Vert \lambda ^*\Vert +1}D(\lambda )=O(h(k)). \end{aligned}$$

(67)

Now, using (67) in (65) and (66) respectively, we get $| {\mathcal {F}}(x^k)-{\mathcal {F}}(x^*)|=O(h(k)).$ $\square$

1.3 Proof of Lemma 3

Proof

Substitute $(x^*,\lambda ^*)$ into (5), we get the following long inequality

$$\begin{aligned}&\beta _k\left( {\mathcal {L}}(x^{k+1},\lambda ^*)-{\mathcal {L}}(x^*,\lambda ^*) \right) +\overbrace{\frac{\delta \beta ^2_{k-1}}{2\gamma }\Vert {\mathcal {A}}x^k-b\Vert ^2}^{1}+\overbrace{\frac{\delta \beta ^2_k}{2}\Vert C(z^{k+1}-z^k)\Vert ^2}^{0}\nonumber \\&\quad \quad +\overbrace{\frac{\beta _k}{2}\Vert y^k-y^{k+1} \Vert ^2_{P_k}}^{0}+\frac{\beta _k^2}{2}\Vert z^k-z^{k+1}\Vert ^2_Q+\frac{1}{2\gamma }\Vert \lambda ^*\nonumber \\&\qquad -\lambda ^{k+1}\Vert ^2+\frac{(2-\gamma )\beta ^2_{k}}{2}\Vert {\mathcal {A}}x^{k+1}-b\Vert ^2\nonumber \\&\quad \quad +\frac{\beta ^2_{k+1}}{2}\Vert z^*-z^{k+1}\Vert ^2_{C^\top C+Q}+\frac{\beta _k^2}{2}\Vert z^{k+1}-z^k\Vert ^2_Q+\overbrace{\frac{\beta _{k+1}}{2}\Vert y^*-y^{k+1} \Vert ^2_{P_{k+1}}}^{0}\nonumber \\&\quad \le \frac{1}{2\gamma } \Vert \lambda ^*-\lambda ^k\Vert ^2+\frac{(2-\gamma )\beta _{k-1}^2}{2}\Vert {\mathcal {A}}x^k-b\Vert ^2+\frac{\beta _k^2}{2}\Vert z^*\nonumber \\&\qquad -z^k\Vert ^2_{C^\top C+Q}+\frac{\beta _{k-1}^2}{2}\Vert z^k-z^{k-1}\Vert ^2_Q\nonumber \\&\quad \quad +\overbrace{\frac{\beta _k}{2}\Vert y^*-y^k\Vert ^2_{P_k}}^{0}-\overbrace{\sigma _g \beta _k \Vert z^k-z^{k+1}\Vert ^2}^{2}-\overbrace{\frac{\epsilon \sigma _g\beta _k}{2}\Vert z^*-z^{k+1}\Vert ^2}^{3} \end{aligned}$$

(68)

Now, we apply several operations to the above inequality: 1, ignore terms under “0” since $P=0$; 2, move the term under “1” to the right hand side; 3, move the term under “2” to the left hand side and apply $\Vert z^{k+1}-z^k\Vert ^2_Q/\lambda _{\max }(Q)\le \Vert z^k-z^{k+1}\Vert ^2$; 4, move one half of “4” to the left hand side and apply $\Vert z^{k+1}-z^*\Vert ^2\le \Vert z^{k+1}-z^*\Vert ^2_{C^\top C+Q}/\lambda _{\max (C^\top C+Q)}.$ After all these operations, we will get the inequality (16). $\square$

1.4 Proof of Lemma 4

Proof

From step 2 and step 3 in IADMM, we have that

$$\begin{aligned} 0=\nabla g(z^{k+1})+C^\top \lambda ^{k+1}+(1-\gamma )\beta _k C^\top ({\mathcal {A}}x^{k+1}-b)+Q_k(z^{k+1}-z^k). \end{aligned}$$

Since $(y^*,z^*,\lambda ^*)$ is a KKT solution, we have $0=\nabla g(z^*)+C^\top \lambda ^*.$ Combining these two equations together with the Lipschitz continuity of $\nabla g$, we have

$$\begin{aligned}&\Vert C^\top (\lambda ^{k+1}-\lambda ^*)+(1-\gamma )\beta _k C^\top ({\mathcal {A}}x^{k+1}-b)+Q_k(z^{k+1}-z^k)\Vert ^2\nonumber \\&\quad =\Vert \nabla g(z^{k+1})-\nabla g(z^*)\Vert ^2\nonumber \\&\quad \le L_g^2\Vert z^{k+1}-z^*\Vert ^2. \end{aligned}$$

(69)

For $0<\alpha <\frac{1}{2}$, by using the inequality $\Vert u+v+w\Vert ^2 \ge (1-2\alpha )\Vert u\Vert ^2-\frac{1}{\alpha }\Vert v\Vert ^2-\frac{1}{\alpha }\Vert w\Vert ^2$, we have that

$$\begin{aligned}&\Vert C^\top (\lambda ^{k+1}-\lambda ^*)+(1-\gamma )\beta _k C^\top ({\mathcal {A}}x^{k+1}-b)+Q_k(z^{k+1}-z^k)\Vert ^2\\&\quad \ge (1-2\alpha )\Vert C^\top (\lambda ^{k+1}-\lambda ^*)\Vert ^2 -\frac{1}{\alpha }\Vert (1-\gamma ) \beta _k C^\top ({\mathcal {A}}x^{k+1}-b)\Vert ^2\nonumber \\&\qquad -\frac{1}{\alpha }\Vert Q_k(z^{k+1}-z^k)\Vert ^2\nonumber \\&\quad \ge (1-2\alpha )\lambda _{\min }(CC^\top )\Vert \lambda ^{k+1}-\lambda ^*\Vert ^2 -\frac{1}{\alpha }\lambda _{\max }(CC^\top )(1-\gamma )^2\beta _k^2\Vert {\mathcal {A}}x^{k+1}-b\Vert ^2\\&\quad \quad -\frac{1}{\alpha }\Vert Q_k(z^{k+1}-z^k)\Vert ^2. \end{aligned}$$

Plug this into (69), we get

$$\begin{aligned}&(1-2\alpha )\lambda _{\min }(CC^\top ) \Vert \lambda ^{k+1}-\lambda ^*\Vert ^2\\&\quad \le \frac{1}{\alpha }\lambda _{\max }(CC^\top )(1-\gamma )^2\beta _k^2\Vert {\mathcal {A}}x^{k+1}-b\Vert ^2\\&\qquad + \frac{1}{\alpha } \Vert Q_k(z^{k+1}-z^k)\Vert ^2+L_g^2\Vert z^{k+1}-z^*\Vert ^2\\&\quad \le \frac{1}{\alpha }\lambda _{\max }(CC^\top )(1-\gamma )^2\beta _k^2\Vert {\mathcal {A}}x^{k+1}-b\Vert ^2\\&\qquad +\frac{1}{\alpha }\lambda _{\max }(Q)\beta _k^2\Vert z^{k+1}-z^k\Vert ^2_Q+L_g^2\Vert z^{k+1}-z^*\Vert ^2. \end{aligned}$$

This completes the proof. $\square$