An Inexact Proximal DC Algorithm with Sieving Strategy for Rank Constrained Least Squares Semidefinite Programming

Abstract

In this paper, the optimization problem of supervised distance preserving projection (SDPP) for data dimensionality reduction is considered, which is equivalent to a rank constrained least squares semidefinite programming (RCLSSDP). Due to the combinatorial nature of rank function, the rank constrained optimization problems are NP-hard in most cases. In order to overcome the difficulties caused by rank constraint, a difference-of-convex (DC) regularization strategy is employed, then RCLSSDP is transferred into a DC programming. For solving the corresponding DC problem, an inexact proximal DC algorithm with sieving strategy (s-iPDCA) is proposed, whose subproblems are solved by an accelerated block coordinate descent method. The global convergence of the sequence generated by s-iPDCA is proved. To illustrate the efficiency of the proposed algorithm for solving RCLSSDP, s-iPDCA is compared with classical proximal DC algorithm, proximal gradient method, proximal gradient-DC algorithm and proximal DC algorithm with extrapolation by performing dimensionality reduction experiment on COIL-20 database. From the computation time and the quality of solution, the numerical results demonstrate that s-iPDCA outperforms other methods. Moreover, dimensionality reduction experiments for face recognition on ORL and YaleB databases demonstrate that rank constrained kernel SDPP is efficient and competitive when comparing with kernel semidefinite SDPP and kernel principal component analysis in terms of recognition accuracy.

Appendices

Appendix A Derivations of (2) and (62)

Firstly, we give the derivation of (2). Let ${\mathbf {U}} = {\mathbf {P}}{\mathbf {P}}^{\top }$, then the SDP relaxation of (1) can be formulated as

$$\begin{aligned} \min J({\mathbf {U}})=\frac{1}{n} \sum _{i, j=1}^{n}{\mathbf {G}}_{i,j}\left( \langle {\mathbf {U}},({\varvec{x}}_{i} -{\varvec{x}}_{j})({\varvec{x}}_{i} -{\varvec{x}}_{j})^{\top }\rangle -\Vert {\varvec{y}}_{i}-{\varvec{y}}_{j}\Vert ^{2}\right) ^{2}. \end{aligned}$$

(A1)

Suppose the number of non-zero elements in i-th row of the graph matrix ${\mathbf {G}}$ is $n_i$ and ${\mathbf {G}}_{i,j_k}=1,k=1,\cdots ,n_i$, then (A1) can be simplified to

$$\begin{aligned} \min _{\mathbf {U}\in \mathcal {S}_+^d} J(\mathbf {U})=\frac{1}{n} \sum _{i}^{n}\sum _{ k=1}^{n_i}\left( \langle {\mathbf {U}},({\varvec{x}}_{i} -{\varvec{x}}_{j_k})({\varvec{x}}_{i} -{\varvec{x}}_{j_k})^{\top }\rangle -\Vert {\varvec{y}}_{i}-{\varvec{y}}_{j_k}\Vert ^{2}\right) ^{2}. \end{aligned}$$

(A2)

Let $l := \sum _{s=1}^{i-1}n_s+k$ be the index of the k-th non-zero element in i-th row of G. Let $\varvec{\tau }_l = {\varvec{x}}_{i} -{\varvec{x}}_{j_k}$ and $\varvec{b}_l = \Vert {\varvec{y}}_{i}-{\varvec{y}}_{j_k}\Vert ^2$. Then (A1) can be formulated as

$$\begin{aligned} \min _{\mathbf {U}\in \mathcal {S}_+^d} J(\mathbf {U})=\frac{1}{n} \sum _{l}^{p}( \langle \mathbf {U},\varvec{\tau }_l\varvec{\tau }_l^{\top }\rangle -\varvec{b}_l)^{2},\end{aligned}$$

(A3)

where $p=\sum _{i,j=1}^n {\mathbf {G}}_{i,j}$. Clearly, (A3) can be formulated into a least squares form:

$$\begin{aligned} \min _{{\mathbf {U}} \in {\mathcal {S}}^d_+} J({\mathbf {U}}) =\frac{1}{n} \Vert {\mathcal {A}}({\mathbf {U}})- {\varvec{b}}\Vert ^{2}, \end{aligned}$$

(A4)

where ${\mathcal {A}}:{\mathcal {S}}^d_+\rightarrow {\mathbb {R}}^p$ is a linear operator that can be explicitly represented as

$$\begin{aligned} {\mathcal {A}}({\mathbf {U}}) = \left[ \langle {\mathbf {A}}_1,{\mathbf {U}}\rangle ,\langle {\mathbf {A}}_2,{\mathbf {U}}\rangle ,\ldots ,\langle {\mathbf {A}}_p,{\mathbf {U}}\rangle \right] ^{\top }, {\mathbf {A}}_l= \varvec{\tau }_l\varvec{\tau }_l^{\top }, l = 1,2,\ldots ,p. \end{aligned}$$

Then ${\mathcal {A}}({\mathbf {U}})$ can be computed as

$$\begin{aligned} {\mathcal {A}}({\mathbf {U}}) = \left[ \varvec{\tau }_1^{\top }{\mathbf {U}}\varvec{\tau }_1,\varvec{\tau }_2^{\top }{\mathbf {U}}\varvec{\tau }_2,\ldots ,\varvec{\tau }_p^{\top }{\mathbf {U}}\varvec{\tau }_p\right] ^{\top }, \end{aligned}$$

its computation cost is $O((d^2+d)p)$.

It is noted that the least squares SDP in (62) can be obtained from (61) by the same way, then we omit its derivation.

Appendix B Derivation of (19)

The subproblem of Algorithm 3 can be formulated as

$$\begin{aligned} \min _{{\mathbf {U}}} \frac{1}{n}\Vert {\mathcal {A}}({\mathbf {U}})- {\varvec{b}}\Vert ^{2}+c\langle {\mathbf {U}},{\mathbf {I}}\rangle - \langle {\mathbf {U}}, {\mathbf {W}}^{k}\rangle +\delta _{{\mathcal {S}}^d_+}({\mathbf {U}}) + \frac{\alpha }{2}\Vert {\mathbf {U}}-{\mathbf {U}}^{k}\Vert _F^2. \end{aligned}$$

(B5)

By ignoring the constant term, (B5) can be simplified as

$$\begin{aligned} \min _{{\mathbf {U}}} \frac{1}{n}\Vert {\mathcal {A}}({\mathbf {U}})- {\varvec{b}}\Vert ^{2}-\langle {\mathbf {U}},{\varvec{\Phi }}_c^{k}\rangle +\delta _{{\mathcal {S}}^d_+}({\mathbf {U}}) + \frac{\alpha }{2}\Vert {\mathbf {U}}\Vert _F^2, \end{aligned}$$

(B6)

where ${\varvec{\Phi }}_c^{k} = {\mathbf {W}}^k+\alpha {\mathbf {U}}^k - c{\mathbf {I}}$. To obtain the dual problem, we introduce two auxiliary variables: ${\varvec{s}}$ and ${\mathbf {T}}$ such that (B6) is equivalent to

$$\begin{aligned} \begin{aligned} \min _{{\mathbf {U}}}&\frac{1}{n}\Vert {\varvec{s}}\Vert ^{2}-\langle {\mathbf {U}},{\varvec{\Phi }}_c^{k}\rangle +\delta _{{\mathcal {S}}^d_+}({\mathbf {T}}) + \frac{\alpha }{2}\Vert {\mathbf {U}}\Vert _F^2\\ \text {s.t.}&{\mathcal {A}}({\mathbf {U}})- {\varvec{b}} = {\varvec{s}}, {\mathbf {U}} = {\mathbf {T}}. \end{aligned} \end{aligned}$$

(B7)

The Lagrange function of (B7) can be expressed as

$$\begin{aligned} \begin{aligned} {\mathbb {L}} ({\mathbf {U}},{\mathbf {T}},{\varvec{s}};{\mathbf {Y}},{\varvec{z}}) =&\frac{1}{n}\Vert {\varvec{s}}\Vert ^{2}-\langle {\mathbf {U}},{\varvec{\Phi }}_c^{k}\rangle +\delta _{{\mathcal {S}}^d_+}({\mathbf {T}})+ \frac{\alpha }{2}\Vert {\mathbf {U}}\Vert _F^2\\&+{\varvec{z}}^{\top }({\mathcal {A}}({\mathbf {U}})- {\varvec{b}}-{\varvec{s}})+\langle {\mathbf {Y}},{\mathbf {T}}-{\mathbf {U}}\rangle . \end{aligned} \end{aligned}$$

(B8)

Then the objective function of the dual problem of (B5) can be obtained by

$$\begin{aligned} \begin{aligned}&\inf _{{\mathbf {U}},{\mathbf {T}},{\varvec{s}}} {\mathbb {L}}({\mathbf {U}},{\mathbf {T}},{\varvec{s}};{\mathbf {Y}},{\varvec{z}})\\&=-\frac{n}{4}\Vert {\varvec{z}}\Vert ^2-{\varvec{z}}^{\top }{\varvec{b}}-\frac{1}{2\alpha }\Vert {\mathcal {A}}^{*}({\mathbf {z}}) -{\varvec{\Phi }}_c^{k}-{\mathbf {Y}} \Vert _F^2-\delta _{{\mathcal {S}}_+^d}^*(-{\mathbf {Y}})\\&=-\frac{n}{4}\Vert {\varvec{z}}\Vert ^2-{\varvec{z}}^{\top }{\varvec{b}}-\frac{1}{2\alpha }\Vert {\mathcal {A}}^{*}({\mathbf {z}}) -{\varvec{\Phi }}_c^{k}-{\mathbf {Y}} \Vert _F^2-\delta _{{\mathcal {S}}_+^d}({\mathbf {Y}}), \end{aligned} \end{aligned}$$

(B9)

where the last equality uses the fact that $\delta _{{\mathcal {S}}_+^d}^*(-{\mathbf {Y}}) = \delta _{{\mathcal {S}}_-^d}(-{\mathbf {Y}}) = \delta _{{\mathcal {S}}_+^d}({\mathbf {Y}})$. Then the dual problem of (B6) can be expressed as

$$\begin{aligned} \begin{aligned} \max _{{\mathbf {Y}},{\varvec{z}}}-\frac{n}{4}\Vert {\varvec{z}}\Vert ^2-{\varvec{z}}^{\top }{\varvec{b}}-\frac{1}{2\alpha }\Vert {\mathcal {A}}^{*}({\mathbf {z}}) -{\varvec{\Phi }}_c^{k}-{\mathbf {Y}} \Vert _F^2-\delta _{{\mathcal {S}}_+^d}({\mathbf {Y}}). \end{aligned} \end{aligned}$$

(B10)

Thus the dual problem of (B5) can be equivalently formulated as the following minimization problem:

$$\begin{aligned} \begin{aligned} \min _{{\mathbf {Y}},{\varvec{z}}}\frac{n}{4}\Vert {\varvec{z}}\Vert ^2+{\varvec{z}}^{\top }{\varvec{b}}+\frac{1}{2\alpha }\Vert {\mathcal {A}}^{*}({\mathbf {z}}) -{\varvec{\Phi }}_c^{k}-{\mathbf {Y}} \Vert _F^2+\delta _{{\mathcal {S}}_+^d}({\mathbf {Y}}). \end{aligned} \end{aligned}$$

(B11)

Appendix C Proof of the Statements (1)–(5) in Proposition 7

For statement (1), since ${\mathbf {U}}^{k_{l+1}} = {\mathbf {V}}^{k+1}$ is the stability center generated in serious step, then the condition (18) holds, shown as

$$\begin{aligned} \Vert \varvec{\Delta }^{k_{l+1}}\Vert _F\le (1-\kappa )\alpha \Vert {\mathbf {U}}^{k_{l+1}}-{\mathbf {U}}^{k_l}\Vert _F. \end{aligned}$$

(B12)

Then we have

$$\begin{aligned} \kappa \alpha \Vert {\mathbf {U}}^{k_{l+1}}-{\mathbf {U}}^{k_l}\Vert _F^2\le \alpha \Vert {\mathbf {U}}^{k_{l+1}}-{\mathbf {U}}^{k_l}\Vert _F^2-\langle \varvec{\Delta }^{k_{l+1}},{\mathbf {U}}^{k_{l+1}}-{\mathbf {U}}^{k_l}\rangle . \end{aligned}$$

(B13)

Consequently,

$$\begin{aligned} \begin{aligned}&E({\mathbf {U}}^{k_{l+1}},{\mathbf {W}}^{k_l},{\mathbf {U}}^{k_l},\varvec{\Delta }^{k_{l+1}})\\&= f_c({\mathbf {U}}^{k_{l+1}})-\langle {\mathbf {U}}^{k_{l+1}},{\mathbf {W}}^{k_l}\rangle + g^*_c({\mathbf {W}}^{k_l})\\&\quad +\alpha \Vert {\mathbf {U}}^{k_{l+1}} -{\mathbf {U}}^{k_l}\Vert ^2_F-\langle \varvec{\Delta }^{k_{l+1}},{\mathbf {U}}^{k_{l+1}}-{\mathbf {U}}^{k_l}\rangle \\&= f_c({\mathbf {U}}^{k_{l+1}})-\langle {\mathbf {U}}^{k_{l+1}}-{\mathbf {U}}^{k_l},{\mathbf {W}}^{k_l}\rangle - g_c({\mathbf {U}}^{k_l})\\&\quad +\alpha \Vert {\mathbf {U}}^{k_{l+1}} -{\mathbf {U}}^{k_l}\Vert ^2_F-\langle \varvec{\Delta }^{k_{l+1}},{\mathbf {U}}^{k_{l+1}}-{\mathbf {U}}^{k_l}\rangle \\&\ge f_c({\mathbf {U}}^{k_{l+1}})- g_c({\mathbf {U}}^{k_{l+1}})+\alpha \Vert {\mathbf {U}}^{k_{l+1}} -{\mathbf {U}}^{k_l}\Vert ^2_F-\langle \varvec{\Delta }^{k_{l+1}},{\mathbf {U}}^{k_{l+1}}-{\mathbf {U}}^{k_l}\rangle \\&\ge f_c({\mathbf {U}}^{k_{l+1}})- g_c({\mathbf {U}}^{k_{l+1}})+\kappa \alpha \Vert {\mathbf {U}}^{k_{l+1}} -{\mathbf {U}}^{k_l}\Vert ^2_F\ge J_c({\mathbf {U}}^{k_{l+1}}), \end{aligned} \end{aligned}$$

where the second equality follows from the convexity of $g_c$ and the fact that ${\mathbf {W}}^{k_l}\in \partial g_c({\mathbf {U}}^{k_l})$, and the first inequality follows from the convexity of $g_c$.

For statement (2), since ${\mathbf {U}}^{k_{l+1}}\in {\mathcal {S}}_+^d$ is an inexact solution of (15) with inexact term $\varvec{\Delta }^{k_{l+1}}$, we have

$$\begin{aligned}{\mathbf {U}}^{k_{l+1}} = \text {arg min}_{{\mathbf {U}}} G_c^{k_l}({\mathbf {U}})-\langle \varvec{\Delta }^{k_{l+1}},{\mathbf {U}}\rangle .\end{aligned}$$

Since $G_c^{k_l}({\mathbf {U}})-\langle \varvec{\Delta }^{k_{l+1}},{\mathbf {U}}\rangle$ is strongly convex, then the following inequality holds:

$$\begin{aligned} \begin{aligned}&f_c({\mathbf {U}}^{k_{l+1}})-\langle {\mathbf {U}}^{k_{l+1}},{\mathbf {W}}^{k_l}\rangle +\frac{\alpha }{2}\Vert {\mathbf {U}}^{k_{l+1}}-{\mathbf {U}}^{k_l}\Vert _F^2-\langle \varvec{\Delta }^{k_{l+1}},{\mathbf {U}}^{k_{l+1}}\rangle \\&\le f_c({\mathbf {U}}^{k_l})-\langle {\mathbf {U}}^{k_l},{\mathbf {W}}^{k_l}\rangle -\langle \varvec{\Delta }^{k_{l+1}},{\mathbf {U}}^{k_l}\rangle -\frac{\alpha }{2}\Vert {\mathbf {U}}^{k_{l+1}}-{\mathbf {U}}^{k_l}\Vert _F^2. \end{aligned} \end{aligned}$$

(B14)

Thus, we have

$$\begin{aligned} \begin{aligned}&E({\mathbf {U}}^{k_{l+1}},{\mathbf {W}}^{k_l},{\mathbf {U}}^{k_l},\varvec{\Delta }^{k_{l+1}})\\&= f_c({\mathbf {U}}^{k_{l+1}})-\langle {\mathbf {U}}^{k_{l+1}},{\mathbf {W}}^{k_l}\rangle + g^*_c({\mathbf {W}}^{k_l})\\&\quad +\alpha \Vert {\mathbf {U}}^{k_{l+1}} -{\mathbf {U}}^{k_l}\Vert ^2_F -\langle \varvec{\Delta }^{k_{l+1}},{\mathbf {U}}^{k_{l+1}}-{\mathbf {U}}^{k_l}\rangle \\&\le f_c({\mathbf {U}}^{k_l})-\langle {\mathbf {U}}^{k_l},{\mathbf {W}}^{k_l}\rangle + g^*_c({\mathbf {W}}^{k_l})= f_c({\mathbf {U}}^{k_l}) - g_c({\mathbf {U}}^{k_l}), \end{aligned} \end{aligned}$$

where the last equality follows from the convexity of $g_c$ and the fact that ${\mathbf {W}}^{k_l}\in \partial g_c({\mathbf {U}}^{k_l})$. Similar to (B13), the following inequality holds:

$$\begin{aligned} \kappa \alpha \Vert {\mathbf {U}}^{k_l}-{\mathbf {U}}^{k_{l-1}}\Vert _F^2\le \alpha \Vert {\mathbf {U}}^{k_l}-{\mathbf {U}}^{k_{l-1}}\Vert _F^2-\langle \varvec{\Delta }^{k_{l}},{\mathbf {U}}^{k_l}-{\mathbf {U}}^{k_{l-1}}\rangle . \end{aligned}$$

(B15)

Consequently, we have

$$\begin{aligned} \begin{aligned}&E({\mathbf {U}}^{k_{l+1}},{\mathbf {W}}^{k_l},{\mathbf {U}}^{k_l},\varvec{\Delta }^{k_{l+1}}) \le f_c({\mathbf {U}}^{k_l}) - g_c({\mathbf {U}}^{k_l})\\&\le f_c({\mathbf {U}}^{k_l})-\langle {\mathbf {U}}^{k_l},{\mathbf {W}}^{k_{l-1}}\rangle + g^*_c({\mathbf {W}}^{k_{l-1}})\\&= E({\mathbf {U}}^{k_l},{\mathbf {W}}^{k_{l-1}},{\mathbf {U}}^{k_{l-1}},\varvec{\Delta }^{k_{l}})- \alpha \Vert {\mathbf {U}}^{k_l} -{\mathbf {U}}^{k_{l-1}}\Vert ^2_F+ \langle \varvec{\Delta }^{k_{l}},{\mathbf {U}}^{k_l}-{\mathbf {U}}^{k_{l-1}}\rangle \\&\le E({\mathbf {U}}^{k_l},{\mathbf {W}}^{k_{l-1}},{\mathbf {U}}^{k_{l-1}},\varvec{\Delta }^{k_{l-1}})- \kappa \alpha \Vert {\mathbf {U}}^{k_l} -{\mathbf {U}}^{k_{l-1}}\Vert ^2_F, \end{aligned} \end{aligned}$$

where the second inequality follows from the convexity of $g_c$ and the Young’s inequality applied to $g_c$. The last inequality is due to (B15).

For statement (3), we first note from Proposition 6 that $\left\{ {\mathbf {U}}^{k_l} \right\}$ is bounded. The boundedness of $\left\{ {\mathbf {W}}^{k_l} \right\}$ follows immediately from the finite-valued property and the convexity of $g_c$ and the fact that ${\mathbf {W}}^{k_l}\in \partial g_c({\mathbf {U}}^{k_l})$. The boundedness of $\left\{ \varvec{\Delta }^{k_{l}} \right\}$ is followed by the fact that $\lim _{l\rightarrow \infty }\epsilon _{k_{l+1}} = 0$. Then, the bounded sequence $\left\{ ({\mathbf {U}}^{k_{l+1}}, {\mathbf {W}}^{k_l}, {\mathbf {U}}^{k_l},\varvec{\Delta }^{k_{l+1}})\right\}$ has nonempty accumulation point set ${\varvec{\Omega }}$.

For statement (4), since $J_c$ is bounded below, from (54) and (55), we have that $\left\{ E({\mathbf {U}}^{k_{l+1}},{\mathbf {W}}^{k_l},{\mathbf {U}}^{k_l},\varvec{\Delta }^{k_{l+1}}) \right\}$ is nonincreasing and bounded below. Thus, the limit $\Upsilon = \lim _{l\rightarrow \infty }E({\mathbf {U}}^{k_{l+1}},{\mathbf {W}}^{k_l},{\mathbf {U}}^{k_l},\varvec{\Delta }^{k_{l+1}})$ exists. Next, we will prove that $E \equiv \Upsilon$ on ${\varvec{\Omega }}$. Take any $(\widehat{{\mathbf {U}}},\widehat{{\mathbf {W}}},\widehat{{\mathbf {U}}},\widehat{\varvec{\Delta }})\in {\varvec{\Omega }}$. Since the above limit exists, there exists a subset ${\mathcal {L}}^{\prime }\subset {\mathcal {L}}$ such that

$$\begin{aligned}\lim _{l\in {\mathcal {L}}^{\prime }}({\mathbf {U}}^{k_{l+1}},{\mathbf {W}}^{k_l},{\mathbf {U}}^{k_l},\varvec{\Delta }^{k_{l+1}}) =(\widehat{{\mathbf {U}}},\widehat{{\mathbf {W}}},\widehat{{\mathbf {U}}},\widehat{\varvec{\Delta }}).\end{aligned}$$

From the optimality of ${\mathbf {U}}^{k_{l+1}}$ and the feasibility of $\widehat{{\mathbf {U}}}$ for solving $\min _{{\mathbf {U}}}G_c^{k_l}({\mathbf {U}})$, we have

$$\begin{aligned} \begin{aligned}&f_c({\mathbf {U}}^{k_{l+1}})-\langle {\mathbf {U}}^{k_{l+1}},{\mathbf {W}}^{k_l}\rangle +\frac{\alpha }{2}\Vert {\mathbf {U}}^{k_{l+1}}-{\mathbf {U}}^{k_l}\Vert _F^2-\langle {\mathbf {U}}^{k_{l+1}},\varvec{\Delta }^{k_{l+1}}\rangle \\&\le f_c(\widehat{{\mathbf {U}}})-\langle \widehat{{\mathbf {U}}},{\mathbf {W}}^{k_l}\rangle +\frac{\alpha }{2}\Vert \widehat{{\mathbf {U}}}-{\mathbf {U}}^{k_l}\Vert _F^2-\langle \widehat{{\mathbf {U}}},\varvec{\Delta }^{k_{l+1}}\rangle . \end{aligned} \end{aligned}$$

(B16)

Rearranging terms in the above inequality, we obtain that

$$\begin{aligned} \begin{aligned}&f_c({\mathbf {U}}^{k_{l+1}})-\langle {\mathbf {U}}^{k_{l+1}}-\widehat{{\mathbf {U}}},{\mathbf {W}}^{k_l}+\varvec{\Delta }^{k_{l+1}}\rangle +\frac{\alpha }{2}\Vert {\mathbf {U}}^{k_{l+1}}-{\mathbf {U}}^{k_l}\Vert _F^2\\&\le f_c(\widehat{{\mathbf {U}}})+\frac{\alpha }{2}\Vert \widehat{{\mathbf {U}}}-{\mathbf {U}}^{k_l}\Vert _F^2. \end{aligned} \end{aligned}$$

(B17)

From the boundedness of $\left\{ {\mathbf {U}}^{k_l}\right\}$, $\left\{ {\mathbf {W}}^{k_l}\right\}$ and $\left\{ \varvec{\Delta }^{k_{l}}\right\}$, we have

$$\begin{aligned}\lim _{l\in {\mathcal {L}}^{\prime }}\langle {\mathbf {U}}^{k_{l+1}}-\widehat{{\mathbf {U}}},{\mathbf {W}}^{k_l}\rangle = 0, \lim _{l\in {\mathcal {L}}^{\prime }}\langle {\mathbf {U}}^{k_{l+1}}-\widehat{{\mathbf {U}}},\varvec{\Delta }^{k_{l+1}}\rangle = 0.\end{aligned}$$

Then, we have

$$\begin{aligned} \Upsilon= & {} \lim _{l\in {\mathcal {L}}^{\prime }}E({\mathbf {U}}^{k_{l+1}},{\mathbf {W}}^{k_l},{\mathbf {U}}^{k_l},\varvec{\Delta }^{k_{l+1}})\\= & {} \lim _{l\in {\mathcal {L}}^{\prime }}f_c({\mathbf {U}}^{k_{l+1}})-\langle {\mathbf {U}}^{k_{l+1}},{\mathbf {W}}^{k_l}\rangle + g^*_c({\mathbf {W}}^{k_l})\\&+\alpha \Vert {\mathbf {U}}^{k_{l+1}} -{\mathbf {U}}^{k_l}\Vert ^2_F -\langle \varvec{\Delta }^{k_{l+1}},{\mathbf {U}}^{k_{l+1}}-{\mathbf {U}}^{k_l}\rangle \\= & {} \lim _{l\in {\mathcal {L}}^{\prime }} f_c({\mathbf {U}}^{k_{l+1}})-\langle {\mathbf {U}}^{k_{l+1}}-\widehat{{\mathbf {U}}},{\mathbf {W}}^{k_l}+\varvec{\Delta }^{k_{l+1}}\rangle +\alpha \Vert {\mathbf {U}}^{k_{l+1}}-{\mathbf {U}}^{k_l}\Vert _F^2\\&-\langle {\mathbf {U}}^{k_{l+1}},{\mathbf {W}}^{k_l}\rangle + g^*_c({\mathbf {W}}^{k_l})-\langle \varvec{\Delta }^{k_{l+1}},{\mathbf {U}}^{k_{l+1}}-{\mathbf {U}}^{k_l}\rangle \\\le & {} \lim \sup _{l\in {\mathcal {L}}^{\prime }} f_c(\widehat{{\mathbf {U}}})+\frac{\alpha }{2}\Vert \widehat{{\mathbf {U}}}-{\mathbf {U}}^{k_l}\Vert _F^2+\frac{\alpha }{2}\Vert {\mathbf {U}}^{k_{l+1}}-{\mathbf {U}}^{k_l}\Vert _F^2 -\langle {\mathbf {U}}^{k_{l+1}},{\mathbf {W}}^{k_l}\rangle \\&+ g^*_c({\mathbf {W}}^{k_l})-\langle \varvec{\Delta }^{k_{l+1}},{\mathbf {U}}^{k_{l+1}}-{\mathbf {U}}^{k_l}\rangle \\= & {} \lim \sup _{l\in {\mathcal {L}}^{\prime }} f_c(\widehat{{\mathbf {U}}})+\frac{\alpha }{2}\Vert \widehat{{\mathbf {U}}}-{\mathbf {U}}^{k_l}\Vert _F^2+\frac{\alpha }{2}\Vert {\mathbf {U}}^{k_{l+1}}-{\mathbf {U}}^{k_l}\Vert _F^2 \\&-\langle {\mathbf {U}}^{k_{l+1}}-{\mathbf {U}}^{k_l},{\mathbf {W}}^{k_l}\rangle - g_c({\mathbf {U}}^{k_l})-\langle \varvec{\Delta }^{k_{l+1}},{\mathbf {U}}^{k_{l+1}}-{\mathbf {U}}^{k_l}\rangle \\= & {} f_c(\widehat{{\mathbf {U}}})-g_c(\widehat{{\mathbf {U}}}) =J_c(\widehat{{\mathbf {U}}}) \le E(\widehat{{\mathbf {U}}},\widehat{{\mathbf {W}}},\widehat{{\mathbf {U}}}, \widehat{\varvec{\Delta }}), \end{aligned}$$

where the fourth equality follows from the convexity of $g_c$ and ${\mathbf {W}}^{k_l}\in \partial g_c({\mathbf {U}}^{k_l})$, and the last inequality holds from (54) with l trending to infinity. Since E is lower semicontinuous, we also have

$$\begin{aligned}E(\widehat{{\mathbf {U}}},\widehat{{\mathbf {W}}},\widehat{{\mathbf {U}}}, \widehat{\varvec{\Delta }})= \lim \inf _{l\in {\mathcal {L}}^{\prime }}E({\mathbf {U}}^{k_{l+1}},{\mathbf {W}}^{k_l}, {\mathbf {U}}^{k_l},\varvec{\Delta }^{k_{l+1}})=\Upsilon .\end{aligned}$$

Thus, $E \equiv \Upsilon$ on ${\varvec{\Omega }}$.

For statement (5), since the subdifferential of the function E at the point $({\mathbf {U}}^{k_{l+1}},{\mathbf {W}}^{k_l},{\mathbf {U}}^{k_l},\varvec{\Delta }^{k_{l+1}})$ is

$$\begin{aligned} \begin{aligned}&\partial E({\mathbf {U}}^{k_{l+1}},{\mathbf {W}}^{k_l},{\mathbf {U}}^{k_l},\varvec{\Delta }^{k_{l+1}})\\&=\left[ \begin{array}{c} \nabla f_c({\mathbf {U}}^{k_{l+1}})- {\mathbf {W}}^{k_l}+ 2\alpha ({\mathbf {U}}^{k_{l+1}}-{\mathbf {U}}^{k_l})-\varvec{\Delta }^{k_{l+1}}+\partial \delta _{{\mathcal {S}}_+^d}({\mathbf {U}}^{k_{l+1}})\\ -{\mathbf {U}}^{k_{l+1}}+\partial g_{c}^{*}({\mathbf {W}}^{k_l}) \\ -\alpha ({\mathbf {U}}^{k_{l+1}}-{\mathbf {U}}^{k_l})+\varvec{\Delta }^{k_{l+1}}\\ {\mathbf {U}}^{k_l}-{\mathbf {U}}^{k_{l+1}} \end{array}\right] . \end{aligned} \end{aligned}$$

Since ${\mathbf {U}}^{k_{l+1}}$ is the optimal solution of (16), we have

$$\begin{aligned} {\mathbf {0}}\in \nabla f_c({\mathbf {U}}^{k_{l+1}})- {\mathbf {W}}^{k_l}+ \alpha ({\mathbf {U}}^{k_{l+1}}-{\mathbf {U}}^{k_l})-\varvec{\Delta }^{k_{l+1}} +\partial \delta _{{\mathcal {S}}_+^d}({\mathbf {U}}^{k_{l+1}}). \end{aligned}$$

Since ${\mathbf {U}}^{k_l}\in \partial g_{c}^{*}({\mathbf {W}}^{k_l})$, then

$$\begin{aligned} \left[ \begin{array}{c} \alpha ({\mathbf {U}}^{k_{l+1}}-{\mathbf {U}}^{k_l})\\ {\mathbf {U}}^{k_l}-{\mathbf {U}}^{k_{l+1}} \\ -\alpha ({\mathbf {U}}^{k_{l+1}}-{\mathbf {U}}^{k_l})+\varvec{\Delta }^{k_{l+1}}\\ {\mathbf {U}}^{k_l}-{\mathbf {U}}^{k_{l+1}} \end{array}\right] \in \partial E({\mathbf {U}}^{k_{l+1}},{\mathbf {W}}^{k_l},{\mathbf {U}}^{k_l},\varvec{\Delta }^{k_{l+1}}) \end{aligned}$$

(B18)

Since ${\mathbf {U}}^{k_{l+1}}$ is the stability center of serious step, so it satisfies the test (18), i.e., $\Vert \varvec{\Delta }^{k_{l+1}}\Vert _F\le (1-\kappa )\alpha \Vert {\mathbf {U}}^{k_{l+1}}-{\mathbf {U}}^{k_l}\Vert _F$. Thus there exists a constant $\rho$ such that the following inequality holds:

$$\begin{aligned} {\mathrm{dist}}({\mathbf {0}},\partial E({\mathbf {U}}^{k_{l+1}},{\mathbf {W}}^{k_l},{\mathbf {U}}^{k_l},\varvec{\Delta }^{k_{l+1}}))\le \rho \Vert {\mathbf {U}}^{k_{l+1}}-{\mathbf {U}}^{k_l}\Vert _F \end{aligned}$$

(B19)

This completes the proof. $\square$