Local convergence analysis of augmented Lagrangian method for nonlinear semidefinite programming

Abstract

The augmented Lagrangian method (ALM) has gained tremendous popularity for its elegant theory and impressive numerical performance since it was proposed by Hestenes and Powell in 1969. It has been widely used in numerous efficient solvers to improve numerical performance to solve many problems. In this paper, without requiring the uniqueness of multipliers, the local (asymptotic Q-superlinear) Q-linear convergence rate of the primal-dual sequences generated by ALM for the nonlinear semidefinite programming is established by assuming the second-order sufficient condition and the semi-isolated calmness of the Karush–Kuhn–Tucker solution under some mild conditions.

Appendices

Appendix A: Proof of Lemma 1

For each $s\in \{1,\dots ,\bar{d}\}$, let $\varLambda _{\bar{\iota }_s\bar{\iota }_s}=\textrm{Diag}(\lambda _{\bar{\iota }_s}(A))$ and $\varXi _{\bar{\iota }_s\bar{\iota }_s}=\textrm{Diag}(\lambda _{\bar{\iota }_s}(A+H))$. We first show that (10) holds. If $\bar{d}=1$, i.e., $\lambda _1(\overline{A})=\dots =\lambda _n(\overline{A})$, the first equation in (10) trivially holds. Next we assume that $\bar{d}\ge 2$. From (9), we have for any ${\mathcal S}^n\ni H\rightarrow 0$,

$$\begin{aligned} \left[ \begin{array}{cccc} \varLambda _{\bar{\iota }_1\bar{\iota }_1} &{} 0 &{} \dots &{} 0\\ 0 &{} \varLambda _{\bar{\iota }_2\bar{\iota }_2} &{} \dots &{} 0\\ \vdots &{}\vdots &{}\ddots &{}\vdots \\ 0 &{} 0 &{}\dots &{} \varLambda _{\bar{\iota }_{\bar{d}}\bar{\iota }_{\bar{d}}} \end{array}\right] U+HU=U\left[ \begin{array}{cccc} \varXi _{\bar{\iota }_1\bar{\iota }_1} &{} 0 &{} \dots &{} 0\\ 0 &{} \varXi _{\bar{\iota }_2\bar{\iota }_2} &{} \dots &{} 0\\ \vdots &{}\vdots &{}\ddots &{}\vdots \\ 0 &{} 0 &{}\dots &{} \varXi _{\bar{\iota }_{\bar{d}}\bar{\iota }_{\bar{d}}} \end{array}\right] . \end{aligned}$$

It follows from $A\in \mathbb {B}_{r}(\overline{A})$ that $\lambda _i(A)\ne \lambda _j(A)$ whenever $i\in \bar{\iota }_t$, $j\in \bar{\iota }_s$ with $s\ne t$. It is easy to see that for all $i\in \bar{\iota }_s$, $j\in \bar{\iota }_t$ with $s\ne t$, $U_{ij}=\frac{(HU)_{ij}}{\varLambda _{ii}-\varXi _{jj}}$. Then we have for all $\Vert H\Vert \le \omega :=r/6$,

$$\begin{aligned} \frac{\Vert U_{\bar{\iota }_{s} \bar{\iota }_{t}}\Vert }{\Vert H\Vert }\le \sum _{i\in \bar{\iota }_s, j\in \bar{\iota }_t}\frac{1}{(\varLambda _{ii}-\varXi _{jj})^2}\le \sum _{i\in \bar{\iota }_k, j\in \bar{\iota }_l}\frac{1}{(|v_i(\overline{A})-v_j(\overline{A})|-2r-2\omega )^2}:=q, \end{aligned}$$

where $\omega$ and q are independent of A. Hence we obtain that

$$\begin{aligned} U_{\bar{\iota }_{s} \bar{\iota }_{t}}={O}(\Vert H\Vert ) \quad \forall \, 1 \le s \ne t \le \bar{d}, \end{aligned}$$

where ${O}(\Vert H\Vert )$ is uniform for all $A\in \mathbb {B}_{r}(\overline{A})$. By using the fact that U is orthogonal, we obtain directly that the second equation in (10) holds. In order to prove (11), we consider the SVD of each $U_{\bar{\iota }_{s} \bar{\iota }_{s}}$, $s=1,\dots ,\bar{d}$. Fix $s\in \{1,\dots ,\bar{d}\}$. Let W and V be in ${\mathcal O}^{|\bar{\iota }_s|}$ such that $U_{\bar{\iota }_{s} \bar{\iota }_{s}}=W\varSigma V^T$, where $\varSigma$ is a nonnegative diagonal matrix. From (10), we obtain that for all $A\in \mathbb {B}_{r}(\overline{A})$,

$$\begin{aligned} W\varSigma ^2W^T=I_{|\bar{\iota }_{s}|}+{O}(\Vert H\Vert ^2), \end{aligned}$$

which is equivalent to

$$\begin{aligned} \varSigma ^2=W^TW+{O}(\Vert H\Vert ^2)=I_{|\bar{\iota }_{s}|}+{O}(\Vert H\Vert ^2). \end{aligned}$$

Since $\varSigma$ is a nonnegative diagonal matrix, we may conclude that

$$\begin{aligned} \varSigma =\textrm{Diag}(1+{O}(\Vert H\Vert ^2),\dots ,1+{O}(\Vert H\Vert ^2)). \end{aligned}$$

Therefore, from $U_{\bar{\iota }_{s} \bar{\iota }_{s}}=W\varSigma V^T$, we have $U_{\bar{\iota }_{s} \bar{\iota }_{s}}=WV^T+O(\Vert H\Vert ^2)$. Since $WV^T\in {\mathcal O}^{|\bar{\iota }_{s}|}$, we know that for all $A\in \mathbb {B}_{r}(\overline{A})$, (11) holds. Next, we shall show (12) holds. For each $s\in \{1,\dots ,\bar{d}\}$ by comparing the s-th diagonal block of both sides of (9), we obtain that

$$\begin{aligned} U^T_{\bar{\iota }_{s}}(\varLambda (A)+H)U_{\bar{\iota }_{s}}=\varXi _{\bar{\iota }_s\bar{\iota }_s}. \end{aligned}$$

(78)

Fix $s\in \{1,\dots ,\bar{d}\}$. From (10) and (78), we know that

$$\begin{aligned} U_{\bar{\iota }_{s}}^T\varLambda (A)U_{\bar{\iota }_{s}}&=\left[ \begin{array}{ccc} {O}(\Vert H\Vert )&U_{\bar{\iota }_{s} \bar{\iota }_{s}}&{O}(\Vert H\Vert )\end{array}\right] \left[ \begin{array}{ccc} \varLambda _1(A)&{}0&{}0\\ 0&{}\varLambda (A)_{\bar{\iota }_{s} \bar{\iota }_{s}}&{}0\\ 0&{}0&{}\varLambda _2(A) \end{array}\right] \left[ \begin{array}{c} {O}(\Vert H\Vert )\\ U_{\bar{\iota }_{s} \bar{\iota }_{s}}\\ {O}(\Vert H\Vert ) \end{array}\right] \\&={O}(\Vert H\Vert ^2)\varLambda _1(A)+U_{\bar{\iota }_{s} \bar{\iota }_{s}}^T\varLambda (A)_{\bar{\iota }_{k} \bar{\iota }_{k}}U_{\bar{\iota }_{k} \bar{\iota }_{k}}+{O}(\Vert H\Vert ^2)\varLambda _2(A). \end{aligned}$$

It follows that

$$\begin{aligned} \varXi _{\bar{\iota }_{k} \bar{\iota }_{k}}-\big ({O}(\Vert H\Vert ^2)\varLambda _1(A)+U_{\bar{\iota }_{k} \bar{\iota }_{s}}^T\varLambda (A)_{\bar{\iota }_{s} \bar{\iota }_{s}}U_{\bar{\iota }_{s} \bar{\iota }_{s}}+{O}(\Vert H\Vert ^2)\varLambda _2(A)\big )\\ =U_{\bar{\iota }_{s} \bar{\iota }_{s}}^TH_{\bar{\iota }_{s} \bar{\iota }_{s}}U_{\bar{\iota }_{s} \bar{\iota }_{s}}+{O}(\Vert H\Vert ^2). \end{aligned}$$

Since $U_{\bar{\iota }_{s} \bar{\iota }_{s}}=Q_s+{O}(\Vert H\Vert ^2)$ and $\Vert \varLambda (A)\Vert \le \Vert \varLambda (\overline{A})\Vert +r$, we obtain that

$$\begin{aligned} Q_s^TH_{\bar{\iota }_{s} \bar{\iota }_{s}}Q_k=\varXi _{\bar{\iota }_{s} \bar{\iota }_{s}}-Q_s^T\varLambda (A)_{\bar{\iota }_{s} \bar{\iota }_{s}}Q_s+{O}(\Vert H\Vert ^2). \end{aligned}$$

Hence (12) holds with the uniform ${O}(\Vert H\Vert ^2)$ for all $A\in \mathbb {B}_{r}(\overline{A})$. The proof is completed.

Appendix B: Proof of Proposition 1

Firstly, we show (14) holds for the case that $A=\varLambda (A)$. For any $H\in {\mathcal S}^n$, denote $Z=A+H$. Let $U\in {\mathcal O}^n$ (depending on H) be such that

$$\begin{aligned} \varLambda (A)+H=U\varLambda (Z)U^T. \end{aligned}$$

(79)

Let $\omega >0$ be any fixed number such that $0\le \omega \le \frac{{\lambda }_{|\alpha |}(\overline{A})-r}{2}$ if $\alpha \ne \varnothing$ and be any fixed positive number otherwise. Then, define the following continuous scalar function

$$\begin{aligned} f(t):=\left\{ \begin{array}{ll} t &{} \text{ if } \quad t>\omega \\ 2 t-\omega &{} \text{ if } \quad \frac{\omega }{2}<t<\delta \\ 0 &{} \text{ if } \quad t<\frac{\omega }{2}. \end{array}\right. \end{aligned}$$

Therefore, we have

$$\begin{aligned} \left\{ \lambda _{1}(A), \ldots , \lambda _{|\alpha |}(A)\right\} \in (\omega ,+\infty ) \quad \text{ and } \quad \left\{ \lambda _{|\alpha |+1}(A), \ldots , \lambda _{n}(A)\right\} \in (-\infty , \frac{\omega }{2}). \end{aligned}$$

For the scalar function f, let $F:{\mathcal S}^n\rightarrow {\mathcal S}^n$ be the corresponding Löwner’s operator, i.e., for any $W\in {\mathcal S}^n$,

$$\begin{aligned} F(W):=\sum _{i=1}^nf(\lambda _i(W))P_iP_i^T, \end{aligned}$$

where $P\in {\mathcal O}^n(W)$. Since f is real analytic on the open set $(-\infty ,\frac{\omega }{2})\cup (\omega ,+\infty )$, It is well-known that for H sufficiently close to zero,

$$\begin{aligned} F(A+H)-F(A)-F'(A)H=O(\Vert H\Vert ^2) \end{aligned}$$

(80)

and

$$\begin{aligned} F'(A)H=\left[ \begin{array}{ccc} H_{\alpha \alpha } &{} H_{\alpha \beta } &{} \varSigma _{\alpha \gamma } \circ H_{\alpha \gamma } \\ H_{\alpha \beta }^{T} &{} 0 &{} 0 \\ \varSigma _{\alpha \gamma }^{T} \circ H_{\alpha \gamma }^{T} &{} 0 &{} 0 \end{array}\right] , \end{aligned}$$

where $O(\Vert H\Vert ^2)$ is independent of A for any $A\in \mathbb {B}_r(\overline{A})$ and $\varSigma \in {\mathcal S}^n$ is given by

$$\begin{aligned} \varSigma _{ij}=\frac{\max \{\lambda _i(A),0\}-\max \{\lambda _j(A),0\}}{\lambda _i(A)-\lambda _j(A)},\quad i,j=1,\dots ,n. \end{aligned}$$

Let $R(\cdot ):=\varPi _{{\mathcal S}_+^n}(\cdot )-F(\cdot )$. By the definition of f, we know that $F(A)=\varPi _{{\mathcal S}_+^n}(A)$, which implies that $R(A)=0$. Meanwhile, it is clear that the matrix valued function R is directionally differentiable at A, and the directional derivative of R for any given direction $H\in {\mathcal S}^n$ is given by

$$\begin{aligned} R^{\prime }(A; H)=\varPi _{\mathcal {S}_{+}^{n}}^{\prime }(A; H)-F^{\prime }(A) H=\left[ \begin{array}{ccc} 0 &{} 0 &{} 0 \\ 0 &{} \varPi _{\mathcal {S}_{+}^{|\beta |}}( H_{\beta \beta }) &{} 0 \\ 0 &{} 0 &{} 0 \end{array}\right] . \end{aligned}$$

By the Lipschitz continuity of $\lambda (\cdot )$, we know that for H sufficiently close to zero,

$$\begin{aligned} \left\{ \lambda _{1}(Z), \ldots , \lambda _{|\alpha |}(Z)\right\} \in (\omega ,+\infty ), \quad \left\{ \lambda _{|\alpha |+1}(Z), \ldots , \lambda _{|\beta |}(Z)\right\} \in (-\infty , \frac{\omega }{2}) \end{aligned}$$

and

$$\begin{aligned} \{\lambda _{|\beta |+1}(Z), \dots , \lambda _{n}(Z)\} \in (-\infty ,0). \end{aligned}$$

Therefore, by the definition of F, we know that for H sufficiently close to zero,

$$\begin{aligned} R(A+H)=\varPi _{{\mathcal S}_+^n}(A+H)-F(A+H)=U\left[ \begin{array}{ccc} 0 &{} 0 &{} 0 \\ 0 &{} \varPi _{\mathcal {S}_{+}^{|\beta |}}(\varLambda (Z)_{\beta \beta }) &{} 0 \\ 0 &{} 0 &{} 0 \end{array}\right] U^T. \end{aligned}$$

Since $U\in {\mathcal O}^n(Z)$, we know from Lemma 1 that for any ${\mathcal S}^n\ni H\rightarrow 0$, there exists an orthogonal matrix $Q\in {\mathcal O}^{|\beta |}$ such that

$$\begin{aligned} U_{\beta }=\left[ \begin{array}{c} {O}(\Vert H\Vert )\\ U_{\beta \beta }\\ {O}(\Vert H\Vert ) \end{array}\right] \quad \textrm{and}\quad U_{\beta \beta }=Q+{O}(\Vert H\Vert ^2), \end{aligned}$$

(81)

Therefore, by noting that $\varPi _{\mathcal {S}_{+}^{|\beta |}}(\varLambda (Z)_{\beta \beta })={O}(\Vert H\Vert )$ and ${O}(\Vert H\Vert )$ is uniform for $A\in \mathbb {B}_r(\overline{A})$ with $\pi (\overline{A})=\pi (A)$, we obtain from the above discussion that

$$\begin{aligned}&R(A+H)-R(A)-R'(A;H)\\&=\left[ \begin{array}{ccc} 0 &{} 0 &{} 0 \\ 0 &{} Q\varPi _{\mathcal {S}_{+}^{|\beta |}}(\varLambda (Z)_{\beta \beta })Q^T-\varPi _{\mathcal {S}_{+}^{|\beta |}}(H_{\beta \beta }) &{} 0 \\ 0 &{} 0 &{} 0 \end{array}\right] +{O}(\Vert H\Vert ^2) \end{aligned}$$

By (79) and (81), we know that

$$\begin{aligned} \varLambda (Z)_{\beta \beta }=U_{\beta }^{T}\varLambda (A)U_{\beta }+U_{\beta }^{T}HU_{\beta }\\ = U_{\beta \beta }^{T}H_{\beta \beta }U_{\beta \beta }+{O}(\Vert H\Vert ^{2})= Q^{T}H_{\beta \beta }Q+{O}(\Vert H\Vert ^{2}). \end{aligned}$$

Since $Q\in {\mathcal O}^{|\beta |}$, we have

$$\begin{aligned} H_{\beta \beta }=Q\varLambda (Z)_{\beta \beta }Q^{T}+{O}(\Vert H\Vert ^{2}), \end{aligned}$$

where ${O}(\Vert H\Vert )$ is uniform for $A\in \mathbb {B}_r(\overline{A})$ with $\pi (\overline{A})=\pi (A)$. Combining this with the globally Lipschitz continuity of $\varPi _{{\mathcal S}_{+}^{|\beta |}}(\cdot )$ and $\varPi _{{\mathcal S}_{+}^{|\beta |}}(Q \varLambda (Z)_{\beta \beta }Q^{T})=Q\varPi _{{\mathcal S}_{+}^{|\beta |}}( \varLambda (Z)_{\beta \beta })Q^{T}$, we obtain that

$$\begin{aligned} Q\varPi _{{\mathcal S}_{+}^{|\beta |}}( \varLambda (Z)_{\beta \beta })Q^{T}-\varPi _{{\mathcal S}_{+}^{|\beta |}}(H_{\beta \beta }) ={O}(\Vert H\Vert ^{2}). \end{aligned}$$

Therefore,

$$\begin{aligned} R(A+H)-R(A)-R'(A;H)={O}(\Vert H\Vert ^{2}). \end{aligned}$$

(82)

By combining (80) and (82), we know that for any ${\mathcal S}^{n}\ni H\rightarrow 0$,

$$\begin{aligned} \varPi _{{\mathcal S}_{+}^{n}}(\varLambda (A)+H)-\varPi _{{\mathcal S}_{+}^{n}}(\varLambda (A))-\varPi '_{{\mathcal S}_{+}^{n}}(\varLambda (A);H)={O}(\Vert H\Vert ^{2})\, \end{aligned}$$

(83)

and ${O}(\Vert H\Vert ^2)$ is uniform for $A\in \mathbb {B}_r(\overline{A})$ with $\pi (\overline{A})=\pi (A)$. Next, we consider the case that $A={P}^{T}\varLambda (A){P}$. Re-write (79) as

$$\begin{aligned} \varLambda (A)+{P}^{T}H{P}={P}^{T}U\varLambda (Z)U^{T}{P}. \end{aligned}$$

Let $\widetilde{H}:={P}^{T}H{P}$. Then, we have $\varPi _{{\mathcal S}^{n}_{+}}(A+H)={P}\,\varPi _{{\mathcal S}^{n}_{+}}(\varLambda (A)+\widetilde{H}){P}^{T}\,.$ Therefore, since ${P}\in {\mathcal O}^{n}$, we know from (83) and (5) that for any ${\mathcal S}^{n}\ni H\rightarrow 0$, (14) holds.

Appendix C: Proof of Lemma 2

We first consider the case where A is diagonal. For notational simplicity, let $\varLambda =\varLambda (A)$ and $\varXi =\varLambda ({A}+H)$. From (13), we have $AU+HU=U\varXi$, which implies

$$\begin{aligned} \varLambda _{\bar{\iota }_s\bar{\iota }_s}U_{\bar{\iota }_s\bar{\iota }_t}+(HU)_{\bar{\iota }_s\bar{\iota }_t}=U_{\bar{\iota }_s\bar{\iota }_t}\varXi _{\bar{\iota }_s\bar{\iota }_t}. \end{aligned}$$

It follows that

$$\begin{aligned} \varLambda _{\bar{\iota }_s\bar{\iota }_s}U_{\bar{\iota }_s\bar{\iota }_t}+\sum _{j=1}^{\bar{d}}H_{\bar{\iota }_s\bar{\iota }_j}U_{\bar{\iota }_j\bar{\iota }_t}=U_{\bar{\iota }_s\bar{\iota }_t}\varXi _{\bar{\iota }_t\bar{\iota }_t}. \end{aligned}$$

This, together with Lemma 1 shows that

$$\begin{aligned} U_{\bar{\iota }_s\bar{\iota }_t}&=\varSigma ^{st}\circ \sum _{j=1}^{\bar{d}}H_{\bar{\iota }_s\bar{\iota }_j}U_{\bar{\iota }_j\bar{\iota }_t} =\varSigma ^{st}\circ H_{\bar{\iota }_s\bar{\iota }_t}Q_t+{O}(\Vert H\Vert ^2) \end{aligned}$$

(84)

where $(\varSigma ^{st})_{ij}=1/((\varXi _{\bar{\iota }_t\bar{\iota }_t})_i-(\varLambda _{\bar{\iota }_s\bar{\iota }_s})_j)$. It is easy to see that $1/\big ((\varXi _{\bar{\iota }_t\bar{\iota }_t})_i-(\varLambda _{\bar{\iota }_s\bar{\iota }_s})_j\big )=1/\big ((\varLambda _{\bar{\iota }_t\bar{\iota }_t})_i-(\varLambda _{\bar{\iota }_s\bar{\iota }_s})_j\big )+{O}(\Vert H\Vert )$. Combining this with (84), we have

$$\begin{aligned} U_{\bar{\iota }_s\bar{\iota }_t}=\varTheta ^{st}\circ H_{\bar{\iota }_s\bar{\iota }_t}Q_t+{O}(\Vert H\Vert ^2),\;\textrm{with}\;O(\Vert H\Vert ^2)\;\mathrm{uniform\;for\;all}\;A\in \mathbb {B}_r(\overline{A}). \end{aligned}$$

Next we consider $A=P^T\varLambda (A)P^T$. Re-write (13) as

$$\begin{aligned} \varLambda (A)+{P}^{T}H{P}={P}^{T}U\varLambda (A+H)U^{T}{P}. \end{aligned}$$

Let $\widetilde{H}:={P}^{T}H{P}$. Since P is an orthogonal matrix, the following proof is the same as the diagonal case. Thus we have completed the proof.