A communication-efficient and privacy-aware distributed algorithm for sparse PCA

Abstract

Sparse principal component analysis (PCA) improves interpretability of the classic PCA by introducing sparsity into the dimension-reduction process. Optimization models for sparse PCA, however, are generally non-convex, non-smooth and more difficult to solve, especially on large-scale datasets requiring distributed computation over a wide network. In this paper, we develop a distributed and centralized algorithm called DSSAL1 for sparse PCA that aims to achieve low communication overheads by adapting a newly proposed subspace-splitting strategy to accelerate convergence. Theoretically, convergence to stationary points is established for DSSAL1. Extensive numerical results show that DSSAL1 requires far fewer rounds of communication than state-of-the-art peer methods. In addition, we make the case that since messages exchanged in DSSAL1 are well-masked, the possibility of private-data leakage in DSSAL1 is much lower than in some other distributed algorithms.

Appendices

Appendix A proof of lemma 1

Proof of Lemma 1

According to the definition of $\textrm{Proj}_{\mathcal {T}_{Z} \mathcal {S}_{n,p}} \left( \cdot \right)$, it follows that

$$\begin{aligned}{} & {} \left\| \textrm{Proj}_{\mathcal {T}_{Z} \mathcal {S}_{n,p}} \left( -A A^{\top }Z + R(Z)\right) \right\| ^2_{\textrm{F}}\\ = {}{} & {} \dfrac{1}{4}\left\| Z^{\top }\left( -A A^{\top }Z + R(Z)\right) - \left( -A A^{\top }Z + R(Z)\right) ^{\top }Z\right\| ^2_{\textrm{F}}\\{} & {} + \left\| {\textbf{P}}^{\perp }_{Z} \left( -A A^{\top }Z + R(Z)\right) \right\| ^2_{\textrm{F}}\\ = {}{} & {} \left\| {\textbf{P}}^{\perp }_{Z} \left( -A A^{\top }Z + R(Z)\right) \right\| ^2_{\textrm{F}}+ \dfrac{1}{4}\left\| Z^{\top }R(Z) - R(Z)^{\top }Z\right\| ^2_{\textrm{F}}, \end{aligned}$$

where $R(Z) \in \partial r(Z)$. The proof is completed.$\square$

Appendix B proof of proposition 2

Proof of Proposition 2

To begin with, we assume that $\left( Z, \{X_i\} \right)$ is a first-order stationary point. Then there exists $R(Z) \in \partial r(Z)$ such that

$$\begin{aligned} {\textbf{P}}^{\perp }_{Z} \left( - A A^{\top }Z + R(Z)\right) = 0, \end{aligned}$$

and $Z^{\top }R(Z)$ is symmetric. Let $\Theta = Z^{\top }R(Z) \in Z^{\top }\partial r(Z)$, $\Gamma _i = - X_i^{\top }A_i A_i^{\top }X_i$, and

$$\begin{aligned} \Lambda _i = - {{\textbf {P}}}^{\perp }_{X_i} A_i A_i^{\top }X_i X_i^{\top }- X_i X_i^{\top }A_i A_i^{\top }{{\textbf {P}}}^{\perp }_{X_i}, \end{aligned}$$

with $i=1,\dotsc ,d$. Then the matrices $\Theta$, $\Gamma _i$ and $\Lambda _i$ are symmetric and $\textrm{rank}\left( \Lambda _i\right) \le 2p$. Moreover, we can deduce that

$$\begin{aligned} A_i A_i^{\top }X_i + X_i \Gamma _i + \Lambda _i X_i =A_i A_i^{\top }X_i - X_iX_i^{\top }A_i A_i^{\top }X_i - {\textbf{P}}^{\perp }_{X_i} A_i A_i^{\top }X_i = 0, \end{aligned}$$

and

$$\begin{aligned} \begin{aligned} R(Z) + \sum \limits _{i=1}^d\Lambda _i Z - Z \Theta = {}&R(Z) - \sum \limits _{i=1}^d{\textbf{P}}^{\perp }_{Z} A_i A_i^{\top }Z - ZZ^{\top }R(Z) \\ = {}&{\textbf{P}}^{\perp }_{Z} \left( - A A^{\top }Z + R(Z)\right) = 0. \end{aligned} \end{aligned}$$

Hence, $\left( Z, \{X_i\} \right)$ satisfies the conditions in (9) under these specific choices of $\Theta$, $\Gamma _i$ and $\Lambda _i$.

Conversely, we now assume that there exist $R(Z) \in \partial r(Z)$ and symmetric matrices $\Theta$, $\Gamma _i$ and $\Lambda _i$ such that $\left( Z, \{X_i\} \right)$ satisfies the conditions in (9). It follows from the first and second equality in (9) that

$$\begin{aligned} \begin{aligned}&\sum \limits _{i=1}^d{\textbf{P}}^{\perp }_{X_i} A_i A_i^{\top }X_i X_i^{\top }= - \sum \limits _{i=1}^d{\textbf{P}}^{\perp }_{X_i} \left( X_i\Gamma _i + \Lambda _i X_i \right) X_i^{\top }= - \sum \limits _{i=1}^d{\textbf{P}}^{\perp }_{X_i} \Lambda _i X_i X_i^{\top }\\&= - {\textbf{P}}^{\perp }_{Z} \left( \sum \limits _{i=1}^d\Lambda _i Z \right) Z^{\top }= {\textbf{P}}^{\perp }_{Z}\left( R(Z) - Z\Theta \right) Z^{\top }= {\textbf{P}}^{\perp }_{Z} R(Z) Z^{\top }. \end{aligned} \end{aligned}$$

At the same time, since $X_i X_i^{\top }= Z Z^{\top }$, we have

$$\begin{aligned} \sum \limits _{i=1}^d{\textbf{P}}^{\perp }_{X_i} A_i A_i^{\top }X_i X_i^{\top }={}&\sum \limits _{i=1}^d{\textbf{P}}^{\perp }_{Z} A_i A_i^{\top }Z Z^{\top }= {\textbf{P}}^{\perp }_{Z} A A^{\top }Z Z^{\top }. \end{aligned}$$

Combining the above two equalities and orthogonality of Z, we arrive at

$$\begin{aligned} {\textbf{P}}^{\perp }_{Z} \left( - A A^{\top }Z + R(Z)\right) = 0. \end{aligned}$$

Left-multiplying both sides of the second equality in (9) by $Z^{\top }$, we obtain that

$$\begin{aligned} Z^{\top }R(Z) = \Theta - \sum \limits _{i=1}^dZ^{\top }\Lambda _i Z, \end{aligned}$$

which together with the symmetry of $\Lambda _i$ and $\Theta$ implies that $Z^{\top }R(Z)$ is also symmetric. This completes the proof. $\square$

Appendix C proof of lemma 3

Proof of Lemma 3

Since $( Z^{(k)}, \{X_i^{(k)}\} )$ is feasible, we know $X_i^{(k)} (X_i^{(k)})^{\top }{=} Z^{(k)} (Z^{(k)})^{\top }$ for $i=1,\dotsc ,d$.

Thus, it can be readily verified that

$$\begin{aligned} \begin{aligned} Q^{(k)} Z^{(k)} = {}&\sum \limits _{i=1}^d\left( \Lambda _i^{(k)} - \beta _i X_i^{(k)} (X_i^{(k)})^{\top }\right) Z^{(k)} \\ = {}&\sum \limits _{i=1}^d\left( - {\textbf{P}}^{\perp }_{Z^{(k)}} A_i A_i^{\top }Z^{(k)} (Z^{(k)})^{\top }- \beta _i Z^{(k)} (Z^{(k)})^{\top }\right) Z^{(k)} \\ = {}&- {\textbf{P}}^{\perp }_{Z^{(k)}} A_i A_i^{\top }Z^{(k)} - \left( \sum \limits _{i=1}^d\beta _i \right) Z^{(k)}, \end{aligned} \end{aligned}$$

which implies that

$$\begin{aligned} \textrm{Proj}_{\mathcal {T}_{Z^{(k)}} \mathcal {S}_{n,p}} \left( Q^{(k)}Z^{(k)}\right) = \textrm{Proj}_{\mathcal {T}_{Z^{(k)}} \mathcal {S}_{n,p}} \left( - A_i A_i^{\top }Z^{(k)}\right) . \end{aligned}$$

According to Theorem 4.1 in [57], the first-order optimality condition of (18) can be stated as:

$$\begin{aligned} 0 \in \textrm{Proj}_{\mathcal {T}_{Z^{(k)}} \mathcal {S}_{n,p}} \left( Q^{(k)} Z^{(k)} + \dfrac{1}{\eta } D^{(k)} + \partial r(Z^{(k)} + D^{(k)})\right) . \end{aligned}$$

Since $D^{(k)} = 0$ is the global minimizer of (18), we have

$$\begin{aligned} \begin{aligned} 0 \in {}&\textrm{Proj}_{\mathcal {T}_{Z^{(k)}} \mathcal {S}_{n,p}} \left( Q^{(k)} Z^{(k)} + \partial r(Z^{(k)})\right) \\ = {}&\textrm{Proj}_{\mathcal {T}_{Z^{(k)}} \mathcal {S}_{n,p}} \left( - A_i A_i^{\top }Z^{(k)} + \partial r(Z^{(k)})\right) . \end{aligned} \end{aligned}$$

We obtain the assertion of this lemma. $\square$

Appendix D convergence of algorithm 2

Now we prove Theorem 4 to establish the global convergence of Algorithm 2. In addition to the notations introduced in Sect. 1, we further adopt the followings throughout the theoretical analysis. The notations $\textrm{rank}\left( C\right)$ and $\sigma _{\min } \left( C\right)$ represent the rank and the smallest singular value of $C$, respectively. For $X, Y \in \mathcal {S}_{n,p}$, we define ${\mathbf {D_p}}\left( X,Y\right) := XX^{\top }- YY^{\top }$ and ${\mathbf {d_p}}\left( X,Y\right) := \left\| {\mathbf {D_p}}\left( X,Y\right) \right\| _{\textrm{F}}$, standing for, respectively, the projection distance matrix and its measurement.

To begin with, we provide a sketch of our proof. Suppose $\{Z^{(k)}\}$ is the iteration sequence generated by Algorithm 2, with $X_i^{(k)}$ and $\Lambda _i^{(k)}$ being the local variable and multiplier of the i-th agent at the k-th iteration, respectively. The proof includes the following main steps.

• 1. The sequence $\{Z^{(k)}\}$ is bounded and the sequence $\{ \mathcal {L}( Z^{(k)}, \{X_i^{(k)}\}, \{\Lambda _i^{(k)}\} ) \}$ is bounded from below.

• 2. The sequence $\{Z^{(k)}\}$ satisfies ${\textbf{d}^2_{\textbf{p}}}\left( Z^{(k+1)},X_i^{(k)}\right) \le 2 ( 1 -\underline{\sigma }^2 )$, and $\underline{\sigma }$ is a unified lower bound of the smallest singular values of the matrices $(X_i^k)^{\top }Z^{k+1}(i=1,\dotsc ,d)$.

• 3. The sequence $\{ \mathcal {L}( Z^{(k)}, \{X_i^{(k)}\}, \{\Lambda _i^{(k)}\} ) \}$ is monotonically non-increasing, and hence is convergent.

• 4. The sequence $\{Z^{(k)}\}$ has at least one accumulation point, and any accumulation point is a first-order stationary point of the sparse PCA problem (2).

Next we verify all the items in the above sketch by proving the following lemmas and corollaries.

Lemma 5

Suppose $\{Z^{(k)}\}$ is the iterate sequence generated by Algorithm 2. Let

$$\begin{aligned} g^{(k)} (D) = \left\langle Q^{(k)}Z^{(k)}, D\right\rangle + \dfrac{1}{2\eta } \left\| D\right\| ^2_{\textrm{F}}+ r(Z^{(k)} + D). \end{aligned}$$

Then the following relationship holds for any $k \in \mathbb {N}$,

$$\begin{aligned} g^{(k)} (0) - g^{(k)} (D^{(k)}) \ge \dfrac{1}{2\eta } \left\| D^{(k)}\right\| ^2_{\textrm{F}}. \end{aligned}$$

Proof

Since $g^{(k)}$ is strongly convex with modulus $\dfrac{1}{\eta }$, we have

$$\begin{aligned} g^{(k)} (\hat{D}) \ge g^{(k)} (D) + \left\langle \partial g^{(k)} (D), \hat{D} - D\right\rangle + \dfrac{1}{2\eta } \left\| \hat{D} - D\right\| ^2_{\textrm{F}}, \end{aligned}$$

(30)

for any $D, \hat{D} \in \mathbb {R}^{n\times p}$. In particular, if $\hat{D}, D \in \mathcal {T}_{Z^{(k)}} \mathcal {S}_{n,p}$, it holds that

$$\begin{aligned} \left\langle \partial g^{(k)} (D), \hat{D} - D\right\rangle = \left\langle \textrm{Proj}_{\mathcal {T}_{Z^{(k)}} \mathcal {S}_{n,p}} \left( \partial g^{(k)} (D)\right) , \hat{D} - D\right\rangle . \end{aligned}$$

It follows from the first-order optimality condition of (18) that $0 \in \textrm{Proj}_{\mathcal {T}_{Z^{(k)}} \mathcal {S}_{n,p}} \left( \partial g^{(k)}\right)$ ${ (D^{(k)})}$. Finally, taking $\hat{D} = 0$ and $D = D^{(k)}$ in (30) yields the assertion of this lemma. $\square$

Lemma 6

Suppose $Z \in \mathcal {S}_{n,p}$ and $D \in \mathcal {T}_{Z} \mathcal {S}_{n,p}$. Then it holds that

$$\begin{aligned} \left\| \textrm{Proj}_{\mathcal {S}_{n,p}}\left( Z + D\right) - Z\right\| _{\textrm{F}}\le \left\| D\right\| _{\textrm{F}}, \end{aligned}$$

and

$$\begin{aligned} \left\| \textrm{Proj}_{\mathcal {S}_{n,p}}\left( Z + D\right) - Z - D\right\| _{\textrm{F}}\le \dfrac{1}{2} \left\| D\right\| ^2_{\textrm{F}}. \end{aligned}$$

Proof

The proof can be found in, for example, [63]. For the sake of completeness, we provide a proof here. It follows from the orthogonality of Z and the skew-symmetry of $Z^{\top }D$ that $Z + D$ has full column rank. This yields that $\textrm{Proj}_{\mathcal {S}_{n,p}} (Z + D) = (Z + D)F^{-1}$, where $F = (I_p + D^{\top }D)^{1/2}$. Since $\textrm{Proj}_{\mathcal {S}_{n,p}} (Z + D) - Z = ( Z (I_p - F) + D ) F^{-1}$, we have

$$\begin{aligned} \begin{aligned} \left\| \textrm{Proj}_{\mathcal {S}_{n,p}}\left( Z + D\right) - Z\right\| ^2_{\textrm{F}}= {}&2\textrm{tr}\left( I_p - F^{-1}\right) - 2 \textrm{tr}\left( F^{-1}Z^{\top }D\right) = 2\textrm{tr}\left( I_p - F^{-1}\right) \\ = {}&2 \sum \limits _{j=1}^d\left( 1 - \left( 1 + \tilde{\sigma }_i^2 \right) ^{-1/2} \right) \le \sum \limits _{j=1}^d\tilde{\sigma }_i^2 = \left\| D\right\| ^2_{\textrm{F}}, \end{aligned} \end{aligned}$$

where $\tilde{\sigma }_1 \ge \cdots \ge \tilde{\sigma }_d \ge 0$ are the singular values of D. Similarly, it follows from the relationship $\textrm{Proj}_{\mathcal {S}_{n,p}} (Z + D) - Z - D = (Z + D) (F^{-1}- I_p)$ that

$$\begin{aligned} \begin{aligned} \left\| \text {Proj}_{\mathcal {S}_{n,p}}\left( Z + D\right) - Z - D\right\| _{\text {F}}^2= {}&\text {tr}\left( \left( I_p - F\right) ^2 \right) = \sum \limits _{j=1}^d\left( 1 - \left( 1 + \tilde{\sigma }_i^2\right) ^{1/2} \right) ^2 \\ \le {}&\dfrac{1}{4} \sum \limits _{j=1}^d\tilde{\sigma }_i^4 = \dfrac{1}{4} \left\| D\right\| _{\text {F}}^4, \end{aligned} \end{aligned}$$

which completes the proof. $\square$

Corollary 7

Suppose $\{Z^{(k)}\}$ is the iterate sequence generated by Algorithm 2 with the parameters satisfying Condition 1. Then for any $k \in \mathbb {N}$, it holds that

$$\begin{aligned} \mathcal {L}( Z^{(k)}, \{X_i^{(k)}\}, \{\Lambda _i^{(k)}\} ) - \mathcal {L}( Z^{(k+1)}, \{X_i^{(k)}\}, \{\Lambda _i^{(k)}\} ) \ge \bar{M} \left\| D^{(k)}\right\| ^2_{\textrm{F}}, \end{aligned}$$

(31)

where $\bar{M} > 0$ is a constant defined in Sect. 4.

Proof

Firstly, it can be readily verified that

$$\begin{aligned} \left\| Q^{(k)}\right\| _{\textrm{F}}\le \sum \limits _{i=1}^d\left\| Q_i^{(k)}\right\| _{\textrm{F}}\le \sum \limits _{i=1}^d\left( 2 \left\| A_i\right\| ^2_{\textrm{F}}+ \sqrt{p} \beta _i \right) . \end{aligned}$$

Let $\bar{q}^{(k)} (Z) = \textrm{tr}(Z^{\top }Q^{(k)} Z) / 2$ bethe smooth part of the objective function $q^{(k)} (Z)$ in (16). Since $\nabla \bar{q}^{(k)}$ is Lipschitz continuous with the corresponding Lipschitz constant $\left\| Q^{(k)}\right\| _{\textrm{F}}$, we have

$$\begin{aligned} \begin{aligned} \bar{q}^{(k)} (Z^{(k+1)}) - \bar{q}^{(k)} (Z^{(k)}) \le {}&\left\langle Q^{(k)} Z^{(k)}, Z^{(k+1)} - Z^{(k)}\right\rangle \\&+ \dfrac{1}{2} \left\| Q^{(k)}\right\| _{\textrm{F}}\left\| Z^{(k+1)} - Z^{(k)}\right\| ^2_{\textrm{F}}. \end{aligned} \end{aligned}$$

It follows from Lemma 6 that

$$\begin{aligned} \begin{aligned} \left\langle Q^{(k)} Z^{(k)}, Z^{(k+1)} - Z^{(k)} - D^{(k)}\right\rangle \le {}&\left\| Q^{(k)} Z^{(k)}\right\| _{\textrm{F}}\left\| Z^{(k+1)} - Z^{(k)} - D^{(k)}\right\| _{\textrm{F}}\\ \le {}&\sum \limits _{i=1}^d\left( \left\| A_i\right\| ^2_{\textrm{F}}+ \dfrac{\sqrt{p}}{2} \beta _i \right) \left\| D^{(k)}\right\| ^2_{\textrm{F}}, \end{aligned} \end{aligned}$$

and

$$\begin{aligned} \begin{aligned} \dfrac{1}{2} \left\| Q^{(k)}\right\| _{\textrm{F}}\left\| Z^{(k+1)} - Z^{(k)}\right\| ^2_{\textrm{F}}\le \sum \limits _{i=1}^d\left( \left\| A_i\right\| ^2_{\textrm{F}}+ \dfrac{\sqrt{p}}{2} \beta _i \right) \left\| D^k\right\| ^2_{\textrm{F}}. \end{aligned} \end{aligned}$$

Combing the above three inequalities, we can obtain that

$$\begin{aligned} \begin{aligned} \bar{q}^{(k)} (Z^{(k+1)}) - \bar{q}^{(k)} (Z^{(k)}) \le \left\langle Q^{(k)} Z^{(k)}, D^{(k)}\right\rangle + \sum \limits _{i=1}^d\left( 2\left\| A_i\right\| ^2_{\textrm{F}}+ \beta _i \sqrt{p} \right) \left\| D^{(k)}\right\| ^2_{\textrm{F}}. \end{aligned} \end{aligned}$$

It follows from Lemma 5 that

$$\begin{aligned} \begin{aligned}&\left\langle Q^{(k)} Z^{(k)}, D^{(k)}\right\rangle + r(Z^{(k)} + D^{(k)}) - r(Z^{(k)}) \\&= g^{(k)} (D^{(k)}) - g^{(k)} (0) - \dfrac{1}{2\eta } \left\| D^{(k)}\right\| ^2_{\textrm{F}}\le - \dfrac{1}{\eta } \left\| D^{(k)}\right\| ^2_{\textrm{F}}, \end{aligned} \end{aligned}$$

which infers that

$$\begin{aligned} \begin{aligned}&\bar{q}^{(k)} (Z^{(k+1)}) - \bar{q}^{(k)} (Z^{(k)}) + r(Z^{(k)} + D^{(k)}) - r(Z^{(k)}) \\&\le \sum \limits _{i=1}^d\left( 2\left\| A_i\right\| ^2_{\textrm{F}}+ \beta _i \sqrt{p} \right) \left\| D^{(k)}\right\| ^2_{\textrm{F}}- \dfrac{1}{\eta } \left\| D^{(k)}\right\| ^2_{\textrm{F}}. \end{aligned} \end{aligned}$$

This together with the Lipschitz continuity of r(Z) yields that

$$\begin{aligned} \begin{aligned}&q^{(k)} (Z^{(k+1)}) - q^{(k)} (Z^{(k)}) \\&= \bar{q}^{(k)} (Z^{(k+1)}) - \bar{q}^{(k)} (Z^{(k)}) + r(Z^{(k+1)}) - r(Z^{(k)}) \\&= \bar{q}^{(k)} (Z^{(k+1)}) - \bar{q}^{(k)} (Z^{(k)}) + r(Z^{(k)} + D^{(k)}) - r(Z^{(k)}) \\&\quad + r(Z^{(k+1)}) - r(Z^{(k)} + D^{(k)}) \\&\le \bar{q}^{(k)} (Z^{(k+1)}) - \bar{q}^{(k)} (Z^{(k)}) + r(Z^{(k)} + D^{(k)}) - r(Z^{(k)}) \\&\quad + \mu \sqrt{np} \left\| Z^{(k+1)} - Z^{(k)} - D^{(k)}\right\| _{\textrm{F}}\\&\le \sum \limits _{i=1}^d\left( 2\left\| A_i\right\| ^2_{\textrm{F}}+ \beta _i \sqrt{p} \right) \left\| D^{(k)}\right\| ^2_{\textrm{F}}- \dfrac{1}{\eta } \left\| D^{(k)}\right\| ^2_{\textrm{F}}+ \dfrac{\mu }{2} \sqrt{np} \left\| D^{(k)}\right\| ^2_{\textrm{F}}\\&= \left( \bar{M} - \dfrac{1}{\eta } \right) \left\| D^{(k)}\right\| ^2_{\textrm{F}}. \end{aligned} \end{aligned}$$

Here, $\bar{M} > 0$ is a constant defined in Sect. 4. According to Condition 1, we know that $\bar{M} - 1/\eta \le -\bar{M}$. Hence, we finally arrive at

$$\begin{aligned} \begin{aligned} \mathcal {L}( Z^{(k)}, \{X_i^{(k)}\}, \{\Lambda _i^{(k)}\}) - \mathcal {L}( Z^{(k+1)}, \{X_i^{(k)}\}, \{\Lambda _i^{(k)}\}) = {}&q^{(k)} (Z^{(k)}) - q^{(k)} (Z^{(k+1)}) \\ \ge {}&\bar{M} \left\| D^{(k)}\right\| ^2_{\textrm{F}}. \end{aligned} \end{aligned}$$

This completes the proof. $\square$

Lemma 8

Suppose $\{Z^{(k)}\}$ is the iterate sequence generated by Algorithm 2 with the parameters satisfying Condition 1. Then for any $k \in \mathbb {N}$, it can be verified that

$$\begin{aligned} {\textbf{d}^2_{\textbf{p}}}\left( Z^{(k+1)},X_i^{(k)}\right) \le \rho \sum \limits _{j=1}^d{\textbf{d}^2_{\textbf{p}}}\left( Z^{(k)},X_j^{(k)}\right) + \dfrac{8}{\beta _i} \left( \sqrt{p} \left\| A\right\| ^2_{\textrm{F}}+ \mu n p\right) , \end{aligned}$$

(32)

where $\rho \ge 1$ is a constant defined in Sect. 4.

Proof

The inequality (31) directly results in the following relationship.

$$\begin{aligned} {q}^{(k)} (Z^{(k)}) - {q}^{(k)} (Z^{(k+1)}) \ge 0. \end{aligned}$$

According to the definition of $q^{(k)}$, it follows that

$$\begin{aligned} \begin{aligned} 0 \le {}&\dfrac{1}{2} \textrm{tr}\left( (Z^{(k)})^{\top }Q^{(k)} Z^{(k)} \right) - \dfrac{1}{2} \textrm{tr}\left( (Z^{(k+1)})^{\top }Q^{(k)} Z^{(k+1)} \right) + r(Z^{(k)}) - r(Z^{(k+1)}) \\ \le {}&\dfrac{1}{2} \sum \limits _{j=1}^d\textrm{tr}\left( \left( \beta _j X_j^{(k)} (X_j^{(k)})^{\top }- \Lambda _j^{(k)} \right) {\mathbf {D_p}}\left( Z^{(k+1)},Z^{(k)}\right) \right) + 2 \mu n p. \end{aligned} \end{aligned}$$

By straightforward calculations, we can deduce that

$$\begin{aligned} \begin{aligned} \sum \limits _{j=1}^d\textrm{tr}\left( \Lambda _j^{(k)} {\mathbf {D_p}}\left( Z^{(k)},Z^{(k+1)}\right) \right) \le {}&\sum \limits _{j=1}^d\left\| \Lambda _j^{(k)}\right\| _{\textrm{F}}{\mathbf {d_p}}\left( Z^{(k+1)},Z^{(k)}\right) \\ \le {}&4 \sqrt{p} \sum \limits _{j=1}^d\left\| A_j\right\| ^2_{\textrm{F}}= 4\sqrt{p} \left\| A\right\| ^2_{\textrm{F}}, \end{aligned} \end{aligned}$$

and

$$\begin{aligned} \begin{aligned}&\sum \limits _{j=1}^d\beta _j \textrm{tr}\left( X_j^{(k)} (X_j^{(k)})^{\top }{\mathbf {D_p}}\left( Z^{(k+1)},Z^{(k)}\right) \right) \\&= \dfrac{1}{2} \sum \limits _{j=1}^d\beta _j {\textbf{d}^2_{\textbf{p}}}\left( Z^{(k)},X_j^{(k)}\right) - \dfrac{1}{2} \sum \limits _{j=1}^d\beta _j {\textbf{d}^2_{\textbf{p}}}\left( Z^{(k+1)},X_j^{(k)}\right) . \end{aligned} \end{aligned}$$

The above three inequalities yield that

$$\begin{aligned} \sum \limits _{j=1}^d\beta _j {\textbf{d}^2_{\textbf{p}}}\left( Z^{(k+1)},X_j^{(k)}\right) \le \sum \limits _{j=1}^d\beta _j {\textbf{d}^2_{\textbf{p}}}\left( Z^{(k)},X_j^{(k)}\right) + 8 \sqrt{p} \left\| A\right\| ^2_{\textrm{F}}+ 8 \mu n p, \end{aligned}$$

which further implies that

$$\begin{aligned} \begin{aligned} {\textbf{d}^2_{\textbf{p}}}\left( Z^{(k+1)},X_i^{(k)}\right) \le {}&\dfrac{1}{\beta _i}\sum \limits _{j=1}^d\beta _j {\textbf{d}^2_{\textbf{p}}}\left( Z^{(k+1)},X_j^{(k)}\right) \\ \le {}&\rho \sum \limits _{j=1}^d{\textbf{d}^2_{\textbf{p}}}\left( Z^{(k)},X_j^{(k)}\right) + \dfrac{8}{\beta _i}\left( \sqrt{p} \left\| A\right\| ^2_{\textrm{F}}+ \mu n p\right) . \end{aligned} \end{aligned}$$

This completes the proof. $\square$

Lemma 9

Suppose $Z^{(k+1)}$ is the $(k+1)$-th iterate generated by Algorithm 2 and satisfies the following condition:

$$\begin{aligned} {\textbf{d}^2_{\textbf{p}}}\left( Z^{(k+1)},X_i^{(k)}\right) \le 2 \left( 1 -\underline{\sigma }^2\right) , \end{aligned}$$

where $\underline{\sigma } \in (0,1)$ is a constant defined in Condition 1. Let the algorithm parameters satisfy Conditions 1 and 2. Then for any $i=1,\dotsc ,d$, it holds that

$$\begin{aligned} h_i^{(k)} ( X_i^{(k)} ) - h_i^{(k)} ( X_i^{(k+1)} ) \ge \frac{1}{4} \underline{\sigma }^2 c_i \beta _i {\textbf{d}^2_{\textbf{p}}}\left( Z^{(k+1)},X_i^{(k)}\right) , \end{aligned}$$

(33)

and

$$\begin{aligned} {\textbf{d}^2_{\textbf{p}}}\left( Z^{(k+1)},X_i^{(k+1)}\right) \le \left( 1 - c_i \underline{\sigma }^2\right) {\textbf{d}^2_{\textbf{p}}}\left( Z^{(k+1)},X_i^{(k)}\right) + \dfrac{12}{\beta _i} \sqrt{p} \left\| A_i\right\| ^2_{\textrm{F}}. \end{aligned}$$

(34)

Proof

It follows from Condition 2 that $\beta _i > c_i^{\prime } \left\| A_i\right\| _2^2$, which together with (25) yields that

$$\begin{aligned} h_i^{(k)} ( X_i^{(k)} ) - h_i^{(k)} ( X_i^{(k+1)} ) \ge \dfrac{c_i}{2\beta _i} \left\| {{\textbf {P}}}^{\perp }_{X_i^{(k)}} H_i^{(k)} X_i^{(k)} \right\| ^2_{\text {F}}. \end{aligned}$$

(35)

And it can be checked that

$$\begin{aligned} \begin{aligned}&{\textbf{P}}^{\perp }_{X_i^{(k)}} H_i^{(k)} X_i^{(k)} = {\textbf{P}}^{\perp }_{X_i^{(k)}} \left( A_i A_i^{\top }X_i^{(k)} + \Lambda _i^{(k)} X_i^{(k)} + \beta _i Z^{(k+1)}(Z^{(k+1)})^{\top }X_i^{(k)} \right) \\&= {\textbf{P}}^{\perp }_{X_i^{(k)}} \left( A_i A_i^{\top }X_i^{(k)} - {\textbf{P}}^{\perp }_{X_i^{(k)}} A_i A_i^{\top }X_i^{(k)} \right) -\beta _i {\textbf{P}}^{\perp }_{X_i^{(k)}} Z^{(k+1)}(Z^{(k+1)})^{\top }X_i^{(k)} \\&= -\beta _i {\textbf{P}}^{\perp }_{X_i^{(k)}} Z^{(k+1)}(Z^{(k+1)})^{\top }X_i^{(k)}. \end{aligned} \end{aligned}$$

(36)

Suppose $\hat{\sigma }_1, \dotsc , \hat{\sigma }_p$ are the singular values of $(X_i^{(k)})^{\top }Z^{(k+1)}$. It is clear that $0 \le \hat{\sigma }_i \le 1$ for any $i = 1, \dotsc , p$ due to the orthogonality of $X_i^{(k)}$ and $Z^{(k+1)}$. On the one hand, we have

$$\begin{aligned} {\textbf{d}^2_{\textbf{p}}}\left( Z^{(k+1)},X_i^{(k)}\right) = \left\| X_i^{(k)}(X_i^{(k)})^{\top }- Z^{(k+1)}(Z^{(k+1)})^{\top }\right\| ^2_{\textrm{F}}= 2\sum \limits _{j=1}^p \left( 1 - \hat{\sigma }_j^2 \right) . \end{aligned}$$

On the other hand, it follows from ${\textbf{d}^2_{\textbf{p}}}\left( Z^{(k+1)},X_i^{(k)}\right) \le 2 \left( 1 -\underline{\sigma }^2\right)$ that

$$\begin{aligned} \sigma _{\min }\left( (X_i^{(k)})^{\top }Z^{(k+1)} \right) \ge \underline{\sigma }. \end{aligned}$$

Let $Y_i^{(k)} = (X_i^{(k)})^{\top }Z^{(k+1)}(Z^{(k+1)})^{\top }X_i^{(k)}$. By straightforward calculations, we can derive that

$$\begin{aligned} \begin{aligned}&\left\| {\textbf{P}}^{\perp }_{X_i^{(k)}} Z^{(k+1)}(Z^{(k+1)})^{\top }X_i^{(k)}\right\| ^2_{\textrm{F}}= \textrm{tr}\left( Y_i^{(k)}\right) - \textrm{tr}\left( (Y_i^{(k)})^2\right) = \sum \limits _{j=1}^p \hat{\sigma }_j^2 \left( 1 - \hat{\sigma }_j^2 \right) \\&\ge \sum \limits _{j=1}^p \sigma _{\min }^2\left( (X_i^{(k)})^{\top }Z^{(k+1)} \right) \left( 1 - \hat{\sigma }_j^2 \right) \ge \dfrac{1}{2} \underline{\sigma }^2 {\textbf{d}^2_{\textbf{p}}}\left( Z^{(k+1)},X_i^{(k)}\right) . \end{aligned} \end{aligned}$$

(37)

Combining (35), (36) and (37), we acquire the assertion (33). Then it follows from the definition of $h_i^{(k)}$ that

$$\begin{aligned} \begin{aligned} c_i \underline{\sigma }^2 {\textbf{d}^2_{\textbf{p}}}\left( Z^{(k+1)},X_i^{(k)}\right) \le {}&2 \textrm{tr}\left( Z^{(k+1)}(Z^{(k+1)})^{\top }{\mathbf {D_p}}\left( X_i^{(k+1)},X_i^{(k)}\right) \right) \\&+ \dfrac{2}{\beta _i} \textrm{tr}\left( \left( A_i A_i^{\top }+ \Lambda _i^{(k)} \right) {\mathbf {D_p}}\left( X_i^{(k+1)},X_i^{(k)}\right) \right) . \end{aligned} \end{aligned}$$

By straightforward calculations, we can obtain that

$$\begin{aligned} \begin{aligned} \textrm{tr}\left( \left( A_i A_i^{\top }+ \Lambda _i^{(k)} \right) {\mathbf {D_p}}\left( X_i^{(k+1)},X_i^{(k)}\right) \right) \le {}&\left\| A_i A_i^{\top }+ \Lambda _i^{(k)} \right\| _{\textrm{F}}{\mathbf {d_p}}\left( X_i^{(k+1)},X_i^{(k)}\right) \\ \le {}&6 \sqrt{p}\left\| A_i\right\| ^2_{\textrm{F}}, \end{aligned} \end{aligned}$$

and

$$\begin{aligned} \begin{aligned}&\textrm{tr}\left( Z^{(k+1)}(Z^{(k+1)})^{\top }{\mathbf {D_p}}\left( X_i^{(k+1)},X_i^{(k)}\right) \right) \\&= \dfrac{1}{2} {\textbf{d}^2_{\textbf{p}}}\left( Z^{(k+1)},X_i^{(k)}\right) - \dfrac{1}{2} {\textbf{d}^2_{\textbf{p}}}\left( Z^{(k+1)},X_i^{(k+1)}\right) . \end{aligned} \end{aligned}$$

The above three relationships yield (34). We complete the proof. $\square$

Lemma 10

Suppose $\{Z^{(k)}\}$ is the iterate sequence generated by Algorithm 2 initiated from $Z^{(0)} \in \mathcal {S}_{n,p}$ with the parameters satisfying Conditions 1 and 2. Then for any $i=1,\dotsc ,d$ and $k \in \mathbb {N}$, it holds that

$$\begin{aligned} {\textbf{d}^2_{\textbf{p}}}\left( Z^{(k+1)},X_i^{(k)}\right) \le 2 \left( 1 -\underline{\sigma }^2\right) . \end{aligned}$$

(38)

Proof

We use mathematical induction to prove this lemma. To begin with, it follows from the inequality (32) that

$$\begin{aligned} \begin{aligned} {\textbf{d}^2_{\textbf{p}}}\left( Z^{(1)},X_i^{(0)}\right) \le {}&\rho \sum \limits _{j=1}^d{\textbf{d}^2_{\textbf{p}}}\left( Z^{(0)},X_j^{(0)}\right) + \dfrac{8}{\beta _i} \left( \sqrt{p} \left\| A\right\| ^2_{\textrm{F}}+ \mu n p\right) \\ = {}&\dfrac{8}{\beta _i} \left( \sqrt{p} \left\| A\right\| ^2_{\textrm{F}}+ \mu n p\right) \le 2 \left( 1 - \underline{\sigma }^2\right) , \end{aligned} \end{aligned}$$

under the relationship $\beta _i > 4 (\sqrt{p} \left\| A\right\| ^2_{\textrm{F}}+ \mu n p ) / ( 1 - \underline{\sigma }^2 )$ in Condition 2. Thus, the argument (38) directly holds for $( Z^{(1)}, \{X_i^{(0)}\} )$. Now, we assume the argument holds at $( Z^{(k+1)}, \{X_i^{(k)}\} )$, and investigate the situation at $( Z^{(k+2)}, \{X_i^{(k+1)}\} )$.

According to Condition 2, we have $12 \sqrt{p} \left\| A_i\right\| ^2_{\textrm{F}}/\beta _i < 2\left( 1 - \underline{\sigma }^2\right) c_i \underline{\sigma }^2$.

Since we assume that ${\textbf{d}^2_{\textbf{p}}}\left( Z^{(k+1)},X_i^{(k)}\right) \le 2 \left( 1 -\underline{\sigma }^2\right)$, it follows from the relationship (34) that

$$\begin{aligned} \begin{aligned} {\textbf{d}^2_{\textbf{p}}}\left( Z^{(k+1)},X_i^{(k+1)}\right) \le {}&\left( 1 - c_i \underline{\sigma }^2 \right) {\textbf{d}^2_{\textbf{p}}}\left( Z^{(k+1)},X_i^{(k)}\right) + \dfrac{12}{\beta _i} \sqrt{p} \left\| A_i\right\| ^2_{\textrm{F}}\\ \le {}&2\left( 1 - \underline{\sigma }^2\right) \left( 1 - c_i \underline{\sigma }^2\right) + 2\left( 1 - \underline{\sigma }^2\right) c_i\underline{\sigma }^2 = 2\left( 1 - \underline{\sigma }^2\right) , \end{aligned} \end{aligned}$$

which infers that $\sigma _{\min } \left( (X_i^{(k+1)})^{\top }Z^{(k+1)} \right) \ge \underline{\sigma }$. Similar to the proof of Lemma 9, we can acquire that

$$\begin{aligned} \left\| {\textbf{P}}^{\perp }_{X_i^{(k+1)}} Z^{(k+1)}(Z^{(k+1)})^{\top }X_i^{(k+1)} \right\| ^2_{\textrm{F}}\ge \dfrac{1}{2} \underline{\sigma }^2 {\textbf{d}^2_{\textbf{p}}}\left( Z^{(k+1)},X_i^{(k+1)}\right) . \end{aligned}$$

(39)

Combining the condition (26) and the equality (36), we have

$$\begin{aligned} \begin{aligned}&\left\| {\textbf{P}}^{\perp }_{X_i^{(k+1)}} H_i^{(k)} X_i^{(k+1)} \right\| _{\textrm{F}}\le \delta _i \left\| {\textbf{P}}^{\perp }_{X_i^{(k)}} H_i^{(k)} X_i^{(k)} \right\| _{\textrm{F}}\\&= \delta _i \beta _i \left\| {\textbf{P}}^{\perp }_{X_i^{(k)}} Z^{(k+1)}(Z^{(k+1)})^{\top }X_i^{(k)} \right\| _{\textrm{F}}\le \delta _i \beta _i {\mathbf {d_p}}\left( Z^{(k+1)},X_i^{(k)}\right) . \end{aligned} \end{aligned}$$

(40)

On the other hand, it follows from the triangular inequality that

$$\begin{aligned} \begin{aligned}&\left\| {{\textbf {P}}}^{\perp }_{X_i^{(k+1)}} H_i^{(k)} X_i^{(k+1)} \right\| _{\text {F}}\\ \ge {}&\left\| {{\textbf {P}}}^{\perp }_{X_i^{(k+1)}} \left( A_i A_i^{\top }+ \Lambda _i^{(k+1)} + \beta _i Z^{(k+1)}(Z^{(k+1)})^{\top }\right) X_i^{(k+1)} \right\| _{\text {F}}\\ {}&- \left\| {{\textbf {P}}}^{\perp }_{X_i^{(k+1)}} \left( \Lambda _i^{(k+1)} - \Lambda _i^{(k)} \right) X_i^{(k+1)} \right\| _{\text {F}}. \end{aligned} \end{aligned}$$

Combing the inequality (39), it can be verified that

$$\begin{aligned} \begin{aligned}&\left\| {\textbf{P}}^{\perp }_{X_i^{(k+1)}} \left( A_i A_i^{\top }+ \Lambda _i^{(k+1)} + \beta _i Z^{(k+1)}(Z^{(k+1)})^{\top }\right) X_i^{(k+1)} \right\| _{\textrm{F}}\\&= \beta _i \left\| {\textbf{P}}^{\perp }_{X_i^{(k+1)}} Z^{(k+1)}(Z^{(k+1)})^{\top }X_i^{(k+1)} \right\| _{\textrm{F}}\ge \dfrac{\sqrt{2}}{2}\underline{\sigma } \beta _i {\mathbf {d_p}}\left( Z^{(k+1)},X_i^{(k+1)}\right) . \end{aligned} \end{aligned}$$

Moreover, according to Lemma B.4 in [48], we have

$$\begin{aligned} \begin{aligned} \left\| {\textbf{P}}^{\perp }_{X_i^{(k+1)}} \left( \Lambda _i^{(k+1)} - \Lambda _i^{(k)} \right) X_i^{(k+1)} \right\| _{\textrm{F}}\le {}&\left\| \Lambda _i^{(k+1)} - \Lambda _i^{(k)} \right\| _{\textrm{F}}\\ \le {}&4\left\| A_i\right\| _2^2 {\mathbf {d_p}}\left( X_i^{(k+1)},X_i^{(k)}\right) . \end{aligned} \end{aligned}$$

Combing the above three inequalities, we further obtain that

$$\begin{aligned} \begin{aligned} \left\| {\textbf{P}}^{\perp }_{X_i^{(k+1)}} H_i^{(k)} X_i^{(k+1)} \right\| _{\textrm{F}}\ge {}&\dfrac{\sqrt{2}}{2}\underline{\sigma } \beta _i {\mathbf {d_p}}\left( Z^{(k+1)},X_i^{(k+1)}\right) \\&- 4\left\| A_i\right\| _2^2 {\mathbf {d_p}}\left( X_i^{(k+1)},X_i^{(k)}\right) . \end{aligned} \end{aligned}$$

Together with (40), this yields that

$$\begin{aligned} \begin{aligned}&\dfrac{\sqrt{2}}{2} \underline{\sigma } \beta _i {\mathbf {d_p}}\left( Z^{(k+1)},X_i^{(k+1)}\right) \\&\le \delta _i \beta _i {\mathbf {d_p}}\left( Z^{(k+1)},X_i^{(k)}\right) + 4\left\| A_i\right\| _2^2 {\mathbf {d_p}}\left( X_i^{(k+1)},X_i^{(k)}\right) \\&\le \left( \delta _i \beta _i + 4 \left\| A_i\right\| _2^2 \right) {\mathbf {d_p}}\left( Z^{(k+1)},X_i^{(k)}\right) + 4 \left\| A_i\right\| _2^2 {\mathbf {d_p}}\left( Z^{(k+1)},X_i^{(k+1)}\right) . \end{aligned} \end{aligned}$$

According to Conditions 1 and 2, we have $\sqrt{2} \underline{\sigma } \beta _i - 8 \left\| A_i\right\| _2^2 > 0$ and $\underline{\sigma } - 2\sqrt{\rho d} \delta _i > 0$. Thus, it can be verified that

$$\begin{aligned} \begin{aligned} {\mathbf {d_p}}\left( Z^{(k+1)},X_i^{(k+1)}\right) \le {}&\dfrac{ 2 ( \delta _i \beta _i + 4 \left\| A_i\right\| _2^2 ) }{ \sqrt{2} \underline{\sigma } \beta _i - 8 \left\| A_i\right\| _2^2 } {\mathbf {d_p}}\left( Z^{(k+1)},X_i^{(k)}\right) \\ \le {}&\sqrt{\dfrac{1}{2 \rho d}} {\mathbf {d_p}}\left( Z^{(k+1)},X_i^{(k)}\right) , \end{aligned} \end{aligned}$$

(41)

where the last inequality follows from the relationship $\beta > \dfrac{4 \left( 2 \sqrt{\rho d} + \sqrt{2}\right) \left\| A_i\right\| _2^2}{\underline{\sigma } - 2 \sqrt{\rho d} \delta _i}$ in Condition 2. This together with (32) and (38) yields that

$$\begin{aligned} \begin{aligned}&{\textbf{d}^2_{\textbf{p}}}\left( Z^{(k+2)},X_i^{(k+1)}\right) \le \rho \sum \limits _{j=1}^d{\textbf{d}^2_{\textbf{p}}}\left( Z^{(k+1)},X_j^{(k+1)}\right) + \dfrac{8}{\beta _i} \left( \sqrt{p} \left\| A\right\| ^2_{\textrm{F}}+ \mu n p\right) \\&\le \dfrac{1}{2d} \sum \limits _{j=1}^d{\textbf{d}^2_{\textbf{p}}}\left( Z^{(k+1)},X_j^{(k)}\right) + \left( 1 -\underline{\sigma }^2\right) \le \left( 1 -\underline{\sigma }^2\right) + \left( 1 -\underline{\sigma }^2\right) = 2 \left( 1 -\underline{\sigma }^2\right) , \end{aligned} \end{aligned}$$

since we assume that $\beta _i > 8 ( \sqrt{p} \left\| A\right\| ^2_{\textrm{F}}+ \mu n p ) / ( 1 - \underline{\sigma }^2 )$ in Condition 2. The proof is completed. $\square$

Corollary 11

Suppose $\{Z^{(k)}\}$ is the iterate sequence generated by Algorithm 2 initiated from $Z^{(0)} \in \mathcal {S}_{n,p}$, and the problem parameters satisfy Conditions 1 and 2. Then for any $k \in \mathbb {N}$, we can obtain that

$$\begin{aligned} \begin{aligned}&\mathcal {L}( Z^{(k+1)}, \{X_i^{(k)}\}, \{\Lambda _i^{(k)}\} ) - \mathcal {L}( Z^{(k+1)}, \{X_i^{(k+1)}\}, \{\Lambda _i^{(k)}\} ) \\&\ge \dfrac{1}{4} \underline{\sigma }^2 \sum \limits _{i=1}^dc_i\beta _i {\textbf{d}^2_{\textbf{p}}}\left( Z^{(k+1)},X_i^{(k)}\right) . \end{aligned} \end{aligned}$$

Proof

This corollary directly follows from Lemma 9 and Lemma 10. $\square$

Corollary 12

Suppose $\{Z^{(k)}\}$ is the iterate sequence generated by Algorithm 2 initiated from $Z^{(0)} \in \mathcal {S}_{n,p}$, and problem parameters satisfy Conditions 1 and 2. Then for any $k \in \mathbb {N}$, we can acquire that

$$\begin{aligned} \begin{aligned}&\mathcal {L}( Z^{(k+1)}, \{X_i^{(k+1)}\}, \{\Lambda _i^{(k)}\} ) - \mathcal {L}( Z^{(k+1)}, \{X_i^{(k+1)}\}, \{\Lambda _i^{(k+1)}\} ) \\&\ge - \dfrac{\sqrt{2 \rho d} + 1}{\rho d} \sum \limits _{i=1}^d\left\| A_i\right\| _2^2 {\textbf{d}^2_{\textbf{p}}}\left( Z^{(k+1)},X_i^{(k)}\right) . \end{aligned} \end{aligned}$$

Proof

According to the Cauchy-Schwarz inequality, we can show that

$$\begin{aligned} \begin{aligned}&\left| \left\langle \Lambda _i^{(k+1)} - \Lambda _i^{(k)}, {\mathbf {D_p}}\left( X_i^{(k+1)},Z^{(k+1)}\right) \right\rangle \right| \le \left\| \Lambda _i^{(k+1)} - \Lambda _i^{(k)} \right\| _{\textrm{F}}{\mathbf {d_p}}\left( Z^{(k+1)},X_i^{(k+1)}\right) \\&\le \sqrt{\dfrac{8}{\rho d}} \left\| A_i\right\| _2^2 {\mathbf {d_p}}\left( X_i^{(k+1)},X_i^{(k)}\right) {\mathbf {d_p}}\left( Z^{(k+1)},X_i^{(k)}\right) , \end{aligned} \end{aligned}$$

where the last inequality follows from Lemma B.4 in [48] and (41). In addition, we have

$$\begin{aligned} \begin{aligned} {\mathbf {d_p}}\left( X_i^{(k+1)},X_i^{(k)}\right) \le {}&{\mathbf {d_p}}\left( Z^{(k+1)},X_i^{(k+1)}\right) + {\mathbf {d_p}}\left( Z^{(k+1)},X_i^{(k)}\right) \\ \le {}&\dfrac{\sqrt{2 \rho d} + 1}{\sqrt{2 \rho d}} {\mathbf {d_p}}\left( Z^{(k+1)},X_i^{(k)}\right) , \end{aligned} \end{aligned}$$

which implies that

$$\begin{aligned} \begin{aligned}&\left\langle \Lambda _i^{(k+1)} - \Lambda _i^{(k)}, {\mathbf {D_p}}\left( X_i^{(k+1)},Z^{(k+1)}\right) \right\rangle \\&\ge - \dfrac{2 \left( \sqrt{2 \rho d} + 1 \right) }{\rho d} \left\| A_i\right\| _2^2 {\textbf{d}^2_{\textbf{p}}}\left( Z^{(k+1)},X_i^{(k)}\right) . \end{aligned} \end{aligned}$$

Combing the fact that

$$\begin{aligned} \begin{aligned}&\mathcal {L}( Z^{(k+1)}, \{X_i^{(k+1)}\}, \{\Lambda _i^{(k)}\} ) - \mathcal {L}( Z^{(k+1)}, \{X_i^{(k+1)}\}, \{\Lambda _i^{(k+1)}\} ) \\&= \dfrac{1}{2} \sum \limits _{i=1}^d\left\langle \Lambda _i^{(k+1)}-\Lambda _i^{(k)}, {\mathbf {D_p}}\left( X_i^{(k+1)},Z^{(k+1)}\right) \right\rangle , \end{aligned} \end{aligned}$$

we complete the proof. $\square$

Now based on these lemmas and corollaries, we can demonstrate the monotonic non-increasing of $\left\{ \mathcal {L}( \{X_i^k\}, Z^k, \{\Lambda _i^k\} ) \right\}$, which results in the global convergence of our algorithm.

Proposition 13

Suppose $\{Z^{(k)}\}$ is the iterate sequence generated by Algorithm 2 initiated from $Z^{(0)} \in \mathcal {S}_{n,p}$, and problem parameters satisfy Conditions 1 and 2. Then the sequence of augmented Lagrangian functions $\{ \mathcal {L}( Z^{(k)}, \{X_i^{(k)}\}, \{\Lambda _i^{(k)}\} ) \}$ is monotonically non-increasing, and for any $k \in \mathbb {N}$, it satisfies the following sufficient descent property:

$$\begin{aligned} \begin{aligned}&\mathcal {L}( Z^{(k)}, \{X_i^{(k)}\}, \{\Lambda _i^{(k)}\} ) - \mathcal {L}( Z^{(k+1)}, \{X_i^{(k+1)}\}, \{\Lambda _i^{(k+1)}\} ) \\&\ge \sum \limits _{i=1}^dJ_i {\textbf{d}^2_{\textbf{p}}}\left( Z^{(k+1)},X_i^{(k+1)}\right) + \bar{M} \left\| D^{(k)}\right\| ^2_{\textrm{F}}, \end{aligned} \end{aligned}$$

(42)

where $J_i = \dfrac{1}{2} \rho d \underline{\sigma }^2 c_i \beta _i - 2 (\sqrt{2 \rho d} + 1) \left\| A_i\right\| _2^2 > 0$ is a constant.

Proof

Combining Corollary 7, Corollary 11, and Corollary 12, we obtain that

$$\begin{aligned} \begin{aligned}&\mathcal {L}( Z^{(k)}, \{X_i^{(k)}\}, \{\Lambda _i^{(k)}\} ) - \mathcal {L}( Z^{(k+1)}, \{X_i^{(k+1)}\}, \{\Lambda _i^{(k+1)}\} ) \\&\ge \sum \limits _{i=1}^d\left( \dfrac{1}{4} \underline{\sigma }^2 c_i \beta _i - \dfrac{\sqrt{2 \rho d} + 1}{\rho d} \left\| A_i \right\| _2^2 \right) {\textbf{d}^2_{\textbf{p}}}\left( Z^{(k+1)},X_i^{(k)}\right) + \bar{M} \left\| D^{(k)}\right\| ^2_{\textrm{F}}. \end{aligned} \end{aligned}$$

Recalling the relationship $\beta _i > 4 ( \sqrt{2 \rho d} + 1) \left\| A_i\right\| _2^2 / (\rho d \underline{\sigma }^2 c_i)$ in Condition 2, we can conclude that $\mathcal {L}( Z^{(k)}, \{X_i^{(k)}\}, \{\Lambda _i^{(k)}\} ) \ge \mathcal {L}( Z^{(k+1)}, \{X_i^{(k+1)}\}, \{\Lambda _i^{(k+1)}\} )$. Hence, the sequence $\{\mathcal {L}( Z^{(k)}, \{X_i^{(k)}\}, \{\Lambda _i^{(k)}\} )\}$ is monotonically non-increasing. Finally, the above relationship together with (41) yields the assertion (42). The proof is finished. $\square$

Based on the above properties, we are ready to prove Theorem 4, which establishes the global convergence rate of our proposed algorithm.

Proof (Proof of Theorem 4)

The whole sequence $\{ Z^{(k)}, \{X_i^{(k)}\} \}$ is naturally bounded, since each of $X_i^{(k)}$ or $Z^{(k)}$ is orthogonal. Then it follows from the Bolzano-Weierstrass theorem that this sequence exists an accumulation point $\{Z^{*}, \{X_i^{*}\}\}$, where $Z^{*} \in \mathcal {S}_{n,p}$ and $X_i^{*} \in \mathcal {S}_{n,p}$. Moreover, the boundedness of $\{\Lambda _i^{(k)}\}$ results from the multipliers updating formula (15). Hence, the lower boundedness of $\{ \mathcal {L}( Z^{(k)}, \{ X_i^{(k)}\}, \{\Lambda _i^{(k)}\} ) \}$ is owing to the continuity of the augmented Lagrangian function. Namely, there exists a constant $\underline{L}$ such that $\mathcal {L}( Z^{(k)}, \{X_i^{(k)}\}, \{\Lambda _i^{(k)}\} ) \ge \underline{L}$ for all $k \in \mathbb {N}$.

It follows from the sufficient descent property (42) that

$$\begin{aligned}{} & {} \sum \limits _{k=1}^{K} \left\| D^{(k)}\right\| ^2_{\textrm{F}}\nonumber \\{} & {} \quad \le \bar{M}^{-1}\sum \limits _{k=1}^{K} \left( \mathcal {L}( Z^{(k)}, \{X_i^{(k)}\}, \{\Lambda _i^{(k)}\} ) - \mathcal {L}( Z^{(k+1)}, \{X_i^{(k+1)}\}, \{\Lambda _i^{(k+1)}\} ) \right) \nonumber \\{} & {} \quad = \bar{M}^{-1}\left( \mathcal {L}( Z^{(1)}, \{X_i^{(1)}\}, \{\Lambda _i^{(1)}\} ) - \mathcal {L}( Z^{(K+1)}, \{X_i^{(K+1)}\}, \{\Lambda _i^{(K+1)}\} ) \right) \nonumber \\{} & {} \quad \le \bar{M}^{-1}\left( \mathcal {L}( Z^{(1)}, \{X_i^{(1)}\}, \{\Lambda _i^{(1)}\} ) - \underline{L} \right) , \end{aligned}$$

(43)

and

$$\begin{aligned}{} & {} \sum \limits _{k = 1}^{K} {\textbf{d}^2_{\textbf{p}}}\left( Z^{(k)},X_i^{(k)}\right) \nonumber \\{} & {} \quad \le J_i^{-1}\sum \limits _{k=1}^{K} \left( \mathcal {L}( Z^{(k-1)}, \{X_i^{(k-1)}\}, \{\Lambda _i^{(k-1)}\} ) - \mathcal {L}( Z^{(k)}, \{X_i^{(k)}\}, \{\Lambda _i^{(k)}\} ) \right) \nonumber \\{} & {} \quad = J_i^{-1}\left( \mathcal {L}( Z^{(0)}, \{X_i^{(0)}\}, \{\Lambda _i^{(0)}\} ) - \mathcal {L}( Z^{(K)}, \{X_i^{(K)}\}, \{\Lambda _i^{(K)}\} ) \right) \nonumber \\{} & {} \quad \le J_i^{-1}\left( \mathcal {L}( Z^{(0)}, \{X_i^{(0)}\}, \{\Lambda _i^{(0)}\} ) - \underline{L} \right) . \end{aligned}$$

(44)

Upon taking the limit as $K \rightarrow \infty$, we obtain that

$$\begin{aligned} \sum \limits _{k = 1}^{\infty } \left\| D^{(k)}\right\| ^2_{\textrm{F}}< \infty \text { and } \sum \limits _{k = 1}^{\infty } {\textbf{d}^2_{\textbf{p}}}\left( Z^{(k)},X_i^{(k)}\right) < \infty , \end{aligned}$$

which further implies that

$$\begin{aligned} \lim \limits _{k\rightarrow \infty } \left\| D^{(k)} \right\| _{\textrm{F}}= 0 \text { and } \lim \limits _{k\rightarrow \infty } {\mathbf {d_p}}\left( Z^{(k)},X_i^{(k)}\right) = 0, \end{aligned}$$

respectively. Combing this with Lemma 3, we know that any accumulation point $Z^{*}$ of sequence $\{Z^{(k)}\}$ is a first-order stationary point of the problem (2).

Eventually, we prove the sublinear convergence rate. Indeed, it follows from the inequalities (43) and (44) that

$$\begin{aligned}{} & {} \min \limits _{k = 1, \dotsc , K} \left\{ \left\| D^{(k)}\right\| ^2_{\textrm{F}}+\dfrac{1}{d} \sum \limits _{i=1}^d{\textbf{d}^2_{\textbf{p}}}\left( Z^{(k)},X_i^{(k)}\right) \right\} \\{} & {} \quad \le \dfrac{1}{K} \sum \limits _{k = 1}^K \left\{ \left\| D^{(k)}\right\| ^2_{\textrm{F}}+\dfrac{1}{d} \sum \limits _{i=1}^d{\textbf{d}^2_{\textbf{p}}}\left( Z^{(k)},X_i^{(k)}\right) \right\} \le \dfrac{C}{K}, \end{aligned}$$

where

$$\begin{aligned} C= & {} \bar{M}^{-1}\left( \mathcal {L}( Z^{(1)}, \{X_i^{(1)}\}, \{\Lambda _i^{(1)}\} ) - \underline{L} \right) \\{} & {} + \left( \sum \limits _{i=1}^dJ_i^{-1}\right) d^{-1}\left( \mathcal {L}( Z^{(0)}, \{X_i^{(0)}\}, \{\Lambda _i^{(0)}\} ) - \underline{L} \right) \end{aligned}$$

is a positive constant. This completes the proof. $\square$