Complexity analysis of interior-point methods for second-order stationary points of nonlinear semidefinite optimization problems

Abstract

We propose a primal-dual interior-point method (IPM) with convergence to second-order stationary points (SOSPs) of nonlinear semidefinite optimization problems, abbreviated as NSDPs. As far as we know, the current algorithms for NSDPs only ensure convergence to first-order stationary points such as Karush–Kuhn–Tucker points, but without a worst-case iteration complexity. The proposed method generates a sequence approximating SOSPs while minimizing a primal-dual merit function for NSDPs by using scaled gradient directions and directions of negative curvature. Under some assumptions, the generated sequence accumulates at an SOSP with a worst-case iteration complexity. This result is also obtained for a primal IPM with a slight modification. Finally, our numerical experiments show the benefits of using directions of negative curvature in the proposed method.

Proofs of Lemma 5.2

1.1 Proof of Lemma 5.2 (1)

Proof

Notice that

$$\begin{aligned} \Vert X^{-1}_{\ell }- X^{-1}\Vert _{\textrm{F}} =\Vert X^{-1}_{\ell }\left( {X_{\ell }- X}\right) X^{-1}\Vert _{\textrm{F}} \le \Vert X^{-1}_{\ell }\Vert _{\textrm{F}}\Vert X^{-1}\Vert _{\textrm{F}} \Vert X_{\ell }-X\Vert _{\textrm{F}}. \end{aligned}$$

Since $\Vert X^{-1}\Vert _{\textrm{F}}\le \Vert X^{-1}_{\ell }\Vert _{\textrm{F}}$ follows from Proposition 5.1 (1), inequality (8) yields $\bigl \Vert X^{-1}_{\ell }-X^{-1}\bigr \Vert _{\textrm{F}} \le 2 L_0 \Vert X^{-1}_{\ell }\Vert _{\textrm{F}}^2 \Vert x^{\ell }-x\Vert .$ The proof is complete. $\square$

1.2 Proof of Lemma 5.2 (2)

Proof

Using (17), we obtain

$$\begin{aligned}&\Vert \nabla _x \psi _{\mu ,\nu } \left( {x^{\ell }, Z_\ell }\right) - \nabla _x \psi _{\mu ,\nu } \left( {x, Z_\ell }\right) \Vert \le \underbrace{\Vert \nabla f(x^{\ell })-\nabla f(x)\Vert }_{(A)} \\&\;+\underbrace{\nu \Vert \left( {\mathcal {A}^*(x^{\ell })Z_\ell - \mathcal {A}^*(x)Z_\ell }\right) \Vert }_{(B)} +\underbrace{\left( {1+\nu }\right) \mu \Vert \mathcal {A}^*(x^{\ell })X^{-1}_{\ell }- \mathcal {A}^*(x)X^{-1}\Vert }_{(C)} . \end{aligned}$$

In what follows, we evaluate upper bounds of (A), (B),  and (C).

Evaluation of (A) :  From (3), we have

$$\begin{aligned} (A) \le L_1 \Vert x^{\ell }- x\Vert . \end{aligned}$$

(A-1)

Evaluation of (B) : 

$$\begin{aligned} \begin{aligned} (B)&\le \nu \sum _{i=1}^n|\,\textrm{tr}\left( {Z_\ell \left( {\mathcal {A}_i\left( {x^{\ell }}\right) -\mathcal {A}_i\left( {x}\right) }\right) }\right) | \\&\le \nu \sum _{i=1}^n \Vert Z_\ell \Vert _{\textrm{F}}\Vert \mathcal {A}_i\left( {x^{\ell }}\right) -\mathcal {A}_i\left( {x}\right) \Vert _{\textrm{F}}\le \nu L_1 \Vert Z_\ell \Vert _{\textrm{F}} \Vert x^{\ell }- x\Vert , \end{aligned} \end{aligned}$$

(A-2)

where the third inequality follows from (9).

Evaluation of (C) :  Let

$$\begin{aligned} (D)&{:}{=}\left( {1+\nu }\right) \mu \Vert \left( {\mathcal {A}^*(x^{\ell })-\mathcal {A}^*(x)}\right) X^{-1}_{\ell }\Vert ,\\ (E)&{:}{=}\left( {1+\nu }\right) \mu \Vert \mathcal {A}^*(x)\left( {X^{-1}_{\ell }-X^{-1}}\right) \Vert . \end{aligned}$$

As $(C)\le (D) + (E)$, we will evaluate upper bounds of (D) and (E). With regard to (D), we obtain from (9) that

$$\begin{aligned} \begin{aligned} (D)&\le \left( {1+\nu }\right) \mu \Vert X^{-1}_{\ell }\Vert _{\textrm{F}}\sum _{i=1}^n \Vert \mathcal {A}^*(x^{\ell })- \mathcal {A}^*(x)\Vert _{\textrm{F}} \\&\le \left( {1+\nu }\right) \mu L_1 \Vert X^{-1}_{\ell }\Vert _{\textrm{F}}\Vert x^{\ell }- x\Vert . \end{aligned} \end{aligned}$$

Similarly, it follows from (5), that

$$\begin{aligned} (E)&\le \left( {1+\nu }\right) \mu \sum _{i=1}^n |\,\textrm{tr}\left( {\mathcal {A}_i\left( {x^{\ell }}\right) \left( {X^{-1}_{\ell }-X^{-1}}\right) }\right) | \\&\le \left( {1+\nu }\right) \mu \sum _{i=1}^n \Vert \mathcal {A}_i\left( {x}\right) \Vert _{\textrm{F}} \Vert X^{-1}_{\ell }-X^{-1}\Vert _{\textrm{F}} \\&\le 2 \left( {1+\nu }\right) \mu L_0\sum _{i=1}^n \Vert \mathcal {A}_i\left( {x}\right) \Vert _{\textrm{F}} \Vert X^{-1}_{\ell }\Vert _{\textrm{F}}^2 \Vert x^{\ell }- x\Vert \quad \quad (\text {by Lemma }5.2 (1)) \\&\le 2 \left( {1+\nu }\right) \mu L_0^2 \Vert X^{-1}_{\ell }\Vert _{\textrm{F}}^2 \Vert x^{\ell }- x\Vert .\quad \quad \qquad \qquad \qquad \qquad \quad (\text {by }(6)) \end{aligned}$$

Therefore

$$\begin{aligned} (C) \le \left( {\left( {1+\nu }\right) \mu L_1 \Vert X^{-1}_{\ell }\Vert _{\textrm{F}} + 2 \left( {1+\nu }\right) \mu L_0^2 \Vert X^{-1}_{\ell }\Vert _{\textrm{F}}^2}\right) \Vert x^{\ell }- x\Vert . \end{aligned}$$

(A-3)

Lastly, from (A-1)-(A-3), we obtain

$$\begin{aligned} \begin{aligned}&\Vert \nabla _x \psi _{\mu ,\nu } \left( {x^{\ell }, Z_\ell }\right) -\nabla _x \psi _{\mu ,\nu } \left( {x, Z_\ell }\right) \Vert \le \\&\quad \left( {L_1 + \nu L_1\Vert Z_\ell \Vert _{\textrm{F}}+ 2\left( {1+\nu }\right) \mu L_0^2\Vert X^{-1}_{\ell }\Vert _{\textrm{F}}^2 + \left( {1+\nu }\right) \mu L_1 \Vert X^{-1}_{\ell }\Vert _{\textrm{F}}}\right) \Vert x^{\ell }- x\Vert . \end{aligned} \end{aligned}$$

The proof is complete. $\square$

1.3 Proof of Lemma 5.2 (3)

Proof

Note that

$$\begin{aligned} \Vert \nabla _Z \psi _{\mu ,\nu } \left( {x^{\ell }, Z_\ell }\right) - \nabla _Z \psi _{\mu ,\nu } \left( {x^{\ell }, Z}\right) \Vert _{\textrm{F}}&= \mu \nu \Vert Z^{-1} - Z_\ell ^{-1}\Vert _{\textrm{F}} \\&\le \mu \nu \Vert Z_\ell ^{-1}\Vert _{\textrm{F}}\Vert Z_\ell - Z\Vert _{\textrm{F}} \Vert Z^{-1}\Vert _{\textrm{F}}, \end{aligned}$$

where the equality follows from (18). Since $\Vert Z^{-1}\Vert _{\textrm{F}} \le 2\Vert Z_\ell ^{-1}\Vert _{\textrm{F}}$ from Proposition 5.1 (2), we have

$$\begin{aligned} \Vert \nabla _Z \psi _{\mu ,\nu } \left( {x^{\ell }, Z_\ell }\right) - \nabla _Z \psi _{\mu ,\nu } \left( {x^{\ell }, Z}\right) \Vert _{\textrm{F}} \le 2\mu \nu \Vert Z_\ell ^{-1}\Vert _{\textrm{F}}^2 \Vert Z_\ell - Z\Vert _{\textrm{F}}. \end{aligned}$$

The proof is complete. $\square$

1.4 Proof of Lemma 5.2 (4)

Proof

From (19), we have

$$\begin{aligned} (\nabla ^2_{xx} \psi _{\mu ,\nu } \left( {x^{\ell }, Z_\ell }\right) )_{ij} - (\nabla ^2_{xx}\psi _{\mu ,\nu } \left( {x, Z_\ell }\right) )_{ij} {=} (A_{ij}) + (B_{ij}) + (C_{ij}), \end{aligned}$$

where

$$\begin{aligned} (A_{ij})&:=\left( {\nabla ^2 f(x^{\ell })}\right) _{ij}-\left( {\nabla ^2 f\bigl (x\bigr )}\right) _{ij}, \\ (B_{ij})&:=\left( {1+\nu }\right) \mu \Bigl ( \,\textrm{tr}\left( {\mathcal {A}_i(x^{\ell }) X_{\ell }^{-1}\mathcal {A}_j(x^{\ell }) X_{\ell }^{-1}}\right) \\&\quad -\,\textrm{tr}\left( {\mathcal {A}_i(x)X^{-1}\mathcal {A}_j(x)X^{-1}}\right) \Bigr ), \\ (C_{ij})&:= \,\textrm{tr}\left( {\left( {\left( {1+\nu }\right) \mu X^{-1}- \nu Z_\ell }\right) \frac{\partial X}{\partial x_i}{x_j}\left( {x}\right) }\right) \\&\quad -\,\textrm{tr}\left( {\left( {\left( {1+\nu }\right) \mu X_{\ell }^{-1}- \nu Z_\ell }\right) \frac{\partial X}{\partial x_i}{x_j}\left( {x^{\ell }}\right) }\right) . \end{aligned}$$

In what follows, we evaluate $(A_{ij})$, $(B_{ij})$, and $(C_{ij})$.

Evaluation of $(A_{ij})$: From (4), we have

$$\begin{aligned} \Vert \nabla ^2 f(x^{\ell }) - \nabla ^2 f(x)\Vert \le L_2 \Vert x^{\ell }-x\Vert . \end{aligned}$$

(A-4)

Evaluation of $(B_{ij})$: $(B_{ij})$ can be written as

$$\begin{aligned} (B_{ij})&=\underbrace{\left( {1+\nu }\right) \mu \left( { \,\textrm{tr}\left( {\mathcal {A}_i(x^{\ell }) X^{-1}_{\ell }\mathcal {A}_j(x^{\ell })X^{-1}_{\ell }}\right) - \,\textrm{tr}\left( {\mathcal {A}_i(x) X^{-1}\mathcal {A}_j(x)X^{-1}_{\ell }}\right) }\right) }_{(D_{ij})} \\&\quad + \underbrace{ \left( {1+\nu }\right) \mu \left( { \,\textrm{tr}\left( {\mathcal {A}_i(x) X^{-1}\mathcal {A}_j(x) X^{-1}_{\ell }}\right) -\,\textrm{tr}\left( {\mathcal {A}_i(x)X^{-1}\mathcal {A}_j(x)X^{-1}}\right) }\right) }_{(E_{ij})}. \end{aligned}$$

Here, $(D_{ij})$ can be bounded as

$$\begin{aligned} |(D_{ij})|&\le \left( {1+\nu }\right) \mu \Vert X_{\ell }^{-1}\Vert _{\textrm{F}} \Vert \mathcal {A}_i(x^{\ell }) X_{\ell }^{-1}\mathcal {A}_j(x^{\ell })-\mathcal {A}_i(x) X^{-1}\mathcal {A}_j(x)\Vert _{\textrm{F}}\\&\le \underbrace{\left( {1+\nu }\right) \mu \Vert X_{\ell }^{-1}\Vert _{\textrm{F}} \Vert \mathcal {A}_i(x^{\ell }) X_{\ell }^{-1}\mathcal {A}_j(x^{\ell })-\mathcal {A}_i(x) X_{\ell }^{-1}\mathcal {A}_j(x)\Vert _{\textrm{F}}}_{(F_{ij})} \\&\quad + \underbrace{ \left( {1+\nu }\right) \mu \Vert X_{\ell }^{-1}\Vert _{\textrm{F}} \Vert \mathcal {A}_i(x) X_{\ell }^{-1}\mathcal {A}_j(x)-\mathcal {A}_i(x) X^{-1}\mathcal {A}_j(x)\Vert _{\textrm{F}}}_{(G_{ij})}. \end{aligned}$$

In the above, $(F_{ij})$ can be bounded as

$$\begin{aligned} \begin{aligned} (F_{ij})&\le \left( {1+\nu }\right) \mu \Vert X_{\ell }^{-1}\Vert _{\textrm{F}} \Vert \mathcal {A}_i(x^{\ell }) X_{\ell }^{-1}\mathcal {A}_j(x^{\ell })-\mathcal {A}_i(x) X_{\ell }^{-1}\mathcal {A}_j(x^{\ell })\Vert _{\textrm{F}}\\&\quad +\left( {1+\nu }\right) \mu \Vert X_{\ell }^{-1}\Vert _{\textrm{F}} \Vert \mathcal {A}_i(x) X_{\ell }^{-1}\mathcal {A}_j(x^{\ell })-\mathcal {A}_i(x) X_{\ell }^{-1}\mathcal {A}_j(x)\Vert _{\textrm{F}}\\&\le \left( {1+\nu }\right) \mu \Vert X_{\ell }^{-1}\Vert _{\textrm{F}} \Vert \mathcal {A}_i(x^{\ell }) - \mathcal {A}_i(x) \Vert _{\textrm{F}} \Vert X_{\ell }^{-1}\mathcal {A}_j(x^{\ell })\Vert _{\textrm{F}}\\&\quad + \left( {1+\nu }\right) \mu \Vert X_{\ell }^{-1}\Vert _{\textrm{F}}\Vert \mathcal {A}_i(x)X_{\ell }^{-1}\Vert _{\textrm{F}} \Vert \mathcal {A}_j(x^{\ell })-\mathcal {A}_j(x)\Vert _{\textrm{F}}. \end{aligned} \end{aligned}$$

Taking the sum of $(F_{ij})$ over i, j together with (5) and (9) yields

$$\begin{aligned} \begin{aligned} \sum _{i=1}^n \sum _{j=1}^n (F_{ij})&\le 2\left( {1+\nu }\right) \mu L_0 L_1{\Vert X^{-1}_{\ell }\Vert _{\textrm{F}}^2} \Vert x^{\ell }-x\Vert . \end{aligned} \end{aligned}$$

(A-5)

For $(G_{ij})$, we obtain, from Lemma Lemma 5.2 (1),

$$\begin{aligned} \begin{aligned} (G_{ij})&\le \left( {1+\nu }\right) \mu \Vert X_{\ell }^{-1}\Vert _{\textrm{F}}\Vert \mathcal {A}_i(x)\Vert _{\textrm{F}}\Vert \mathcal {A}_j(x)\Vert _{\textrm{F}} \Vert X_{\ell }^{-1}- X^{-1}\Vert _{\textrm{F}} \\&\le 2\left( {1+\nu }\right) \mu L_0 \Vert X_{\ell }^{-1}\Vert _{\textrm{F}}^3\Vert \mathcal {A}_i(x)\Vert _{\textrm{F}}\Vert \mathcal {A}_j(x)\Vert _{\textrm{F}} \Vert x^{\ell }-x\Vert , \end{aligned} \end{aligned}$$

which together with (5) yields

$$\begin{aligned} \begin{aligned} \sum _{i=1}^n\sum _{j=1}^n (G_{ij})\le 2\left( {1+\nu }\right) \mu L_0^3 \Vert X^{-1}_{\ell }\Vert _{\textrm{F}}^3 \Vert x^{\ell }-x\Vert . \end{aligned} \end{aligned}$$

(A-6)

By combining inequalities (A-5) and (A-6), the sum of $(D_{ij})$ is bounded from above by

$$\begin{aligned} \begin{aligned}&\sum _{i=1}^n \sum _{j=1}^n |(D_{ij})|\\&\le 2\left( { \left( {1+\nu }\right) \mu L_0^3 \Vert X^{-1}_{\ell }\Vert _{\textrm{F}}^3+ \left( {1+\nu }\right) \mu L_0 L_1{\Vert X^{-1}_{\ell }\Vert _{\textrm{F}}^2} }\right) \Vert x^{\ell }-x\Vert . \end{aligned} \end{aligned}$$

(A-7)

With regard to $(E_{ij})$, we have

$$\begin{aligned} |\left( {E_{ij}}\right) | \le \left( {1+\nu }\right) \mu \Vert \mathcal {A}_i(x)\Vert _{\textrm{F}}\Vert \mathcal {A}_j(x)\Vert _{\textrm{F}}\Vert X^{-1}\Vert _{\textrm{F}} \Vert X_{\ell }^{-1}- X^{-1}\Vert _{\textrm{F}}. \end{aligned}$$

By taking the sum over i, j of $|\left( {E_{ij}}\right) |$ together with (5), Lemma 5.2 (1), and Proposition 5.1 (1), we have

$$\begin{aligned} \begin{aligned} \sum _{i=1}^n \sum _{j=1}^n |\left( {E_{ij}}\right) |&\le 4 \left( {1+\nu }\right) \mu L_0^3 \Vert X_{\ell }^{-1}\Vert _{\textrm{F}}^3 \Vert x^{\ell }- x\Vert . \end{aligned} \end{aligned}$$

(A-8)

Combining inequalities (A-7) and (A-8) yields

$$\begin{aligned} \begin{aligned}&\sum _{i=1}^n \sum _{j=1}^n |\left( {B_{ij}}\right) |\\&\le \left( {6\left( {1+\nu }\right) \mu L_0^3 \Vert X^{-1}_{\ell }\Vert _{\textrm{F}}^3+ 2\left( {1+\nu }\right) \mu L_0 L_1 {\Vert X_{\ell }^{-1}\Vert _{\textrm{F}}^2}}\right) \Vert x^{\ell }- x\Vert . \end{aligned} \end{aligned}$$

(A-9)

Evaluation of $(C_{ij})$: Note that $(C_{ij})$ can be written as

$$\begin{aligned} \left( {C_{ij}}\right)&=\underbrace{\nu \,\textrm{tr}\left( {Z_\ell \left( {\frac{\partial X}{\partial x_i}{x_j}\left( {x^{\ell }}\right) -\frac{\partial X}{\partial x_i}{x_j}\left( {x}\right) }\right) }\right) }_{(J_{ij})} \\&\quad +\underbrace{\left( {1+\nu }\right) \mu \,\textrm{tr}\left( {X^{-1}\frac{\partial X}{\partial x_i}{x_j}\left( {x}\right) -X^{-1}_{\ell }\frac{\partial X}{\partial x_i}{x_j}\left( {x^{\ell }}\right) }\right) }_{(K_{ij})}. \end{aligned}$$

By the Cauchy-Schwartz inequality, we have

$$\begin{aligned} \begin{aligned} \sum _{i=1}^n \sum _{j=1}^n|\left( {J_{ij}}\right) |&\le \nu \sum _{i=1}^n \sum _{j=1}^n\Vert Z_\ell \Vert _{\textrm{F}}\Vert \frac{\partial X}{\partial x_i}{x_j}\left( {x^{\ell }}\right) -\frac{\partial X}{\partial x_i}{x_j}\left( {x}\right) \Vert _{\textrm{F}} \\&\le \nu L_2 \Vert Z_\ell \Vert _{\textrm{F}} \Vert x^{\ell }-x\Vert . \end{aligned} \end{aligned}$$

(A-10)

With regard to the sum of $|(K_{ij})|$,

$$\begin{aligned}&\sum _{i=1}^n \sum _{j=1}^n |(K_{ij})|\nonumber \\ {}&= \sum _{i=1}^n \sum _{j=1}^n \left| \left( {1+\nu }\right) \mu \,\text {tr}\left( {\left( {X^{-1}-X^{-1}_{\ell }}\right) \frac{\partial X}{\partial x_i}{x_j}\left( {x}\right) }\right) \right| \nonumber \\ {}&\quad + \sum _{i=1}^n \sum _{j=1}^n\left| \left( {1+\nu }\right) \mu \,\text {tr}\left( {X^{-1}_{\ell }\left( {\frac{\partial X}{\partial x_i}{x_j}\left( {x}\right) -\frac{\partial X}{\partial x_i}{x_j}\left( {x^{\ell }}\right) }\right) }\right) \right| \nonumber \\ {}&\le 2\left( {1+\nu }\right) \mu L_0 L_1 {\Vert X^{-1}_{\ell }\Vert _{\text {F}}^2} \Vert x^{\ell }-x\Vert +\left( {1+\nu }\right) \mu L_2 \Vert X^{-1}_{\ell }\Vert _{\text {F}}\Vert x^{\ell }-x\Vert , \end{aligned}$$

(A-11)

where the inequality follows from (6), (7), and Lemma 5.2 (1). Moreover, combining (A-10) and (A-11) yields

$$\begin{aligned} \begin{aligned}&\sum _{i=1}^n \sum _{j=1}^n |\left( {C_{ij}}\right) | \\&\le \left( { \nu L_2 \Vert Z_\ell \Vert _{\textrm{F}} + 2\left( {1+\nu }\right) \mu L_0 L_1{\Vert X^{-1}_{\ell }\Vert _{\textrm{F}}^2}+\left( {1+\nu }\right) \mu L_2 \Vert X^{-1}_{\ell }\Vert _{\textrm{F}}}\right) \Vert x^{\ell }-x\Vert . \end{aligned} \end{aligned}$$

(A-12)

Lastly, we have

$$\begin{aligned}&\Vert \nabla ^2_{xx} \psi _{\mu ,\nu } \left( {x^{\ell }, Z_\ell }\right) - \nabla ^2_{xx} \psi _{\mu ,\nu } \left( {x, Z_\ell }\right) \Vert \\&\le \Vert \nabla ^2 f(x^{\ell })-\nabla ^2 f(x)\Vert + \Vert (B_{ij})_{i,j}\Vert + \Vert (C_{ij})_{i,j}\Vert \\&\le \Vert \nabla ^2 f(x^{\ell })-\nabla ^2 f(x)\Vert + \Vert (B_{ij})_{i,j}\Vert _{\textrm{F}}+ \Vert (C_{ij})_{i,j}\Vert _{\textrm{F}}\\&\le \Vert \nabla ^2 f(x^{\ell })-\nabla ^2 f(x)\Vert + \sum _{i=1}^n\sum _{j=1}^n\left| (B_{ij})\right| +\sum _{i=1}^n\sum _{j=1}^n\left| (C_{ij})\right| , \end{aligned}$$

which together with (A-4), (A-9), and (A-12) implies

$$\begin{aligned}&\Vert \nabla ^2_{xx} \psi _{\mu ,\nu } \left( {x^{\ell }, Z_\ell }\right) - \nabla ^2_{xx} \psi _{\mu ,\nu } \left( {x, Z_\ell }\right) \Vert \\&\le \biggl ( L_2+\nu L_2 \Vert Z_\ell \Vert _{\textrm{F}} \\&\quad \quad + \left( {1+\nu }\right) \mu \left( { L_2 \Vert X^{-1}_{\ell }\Vert _{\textrm{F}} + 4 L_1 L_0\Vert X^{-1}_{\ell }\Vert _{\textrm{F}}^2 + 6 L_0^3 \Vert X^{-1}_{\ell }\Vert _{\textrm{F}}^3}\right) \biggr )\\&\quad \cdot \Vert x^{\ell }- x\Vert \end{aligned}$$

The proof is complete. $\square$