Solving sparse principal component analysis with global support

Abstract

Sparse principal component analysis with global support (SPCAgs), is the problem of finding the top-r leading principal components such that all these principal components are linear combinations of a common subset of at most k variables. SPCAgs is a popular dimension reduction tool in statistics that enhances interpretability compared to regular principal component analysis (PCA). Methods for solving SPCAgs in the literature are either greedy heuristics (in the special case of $r = 1$) with guarantees under restrictive statistical models or algorithms with stationary point convergence for some regularized reformulation of SPCAgs. Crucially, none of the existing computational methods can efficiently guarantee the quality of the solutions obtained by comparing them against dual bounds. In this work, we first propose a convex relaxation based on operator norms that provably approximates the feasible region of SPCAgs within a $c_1 + c_2 \sqrt{\log r} = O(\sqrt{\log r})$ factor for some constants $c_1, c_2$. To prove this result, we use a novel random sparsification procedure that uses the Pietsch-Grothendieck factorization theorem and may be of independent interest. We also propose a simpler relaxation that is second-order cone representable and gives a $(2\sqrt{r})$-approximation for the feasible region. Using these relaxations, we then propose a convex integer program that provides a dual bound for the optimal value of SPCAgs. Moreover, it also has worst-case guarantees: it is within a multiplicative/additive factor of the original optimal value, and the multiplicative factor is $O(\log r)$ or O(r) depending on the relaxation used. Finally, we conduct computational experiments that show that our convex integer program provides, within a reasonable time, good upper bounds that are typically significantly better than the natural baselines.

Appendices

Appendix

Additional concentration inequalities

We need the standard multiplicative Chernoff bound (see Theorem 4.4 [36]).

Lemma 3

(Chernoff Bound) Let $X_1,\ldots ,X_n$ be independent random variables taking values in [0, 1]. Then for any $\delta > 0$ we have

$$\begin{aligned} \Pr \bigg (\sum _i X_i > (1+\delta ) \mu \bigg ) \,<\, \bigg (\frac{e}{1+\delta }\bigg )^{(1+\delta ) \mu }, \end{aligned}$$

where $\mu = {\mathbb {E}}\sum _i X_i$.

We also need the one-sided Chebychev inequality, see for example Exercise 3.18 of [36].

Lemma 4

(One-sided Chebychev) For any random variable X with finite first and second moments

$$\begin{aligned} \Pr \bigg ( X \le {\mathbb {E}}X - t \bigg ) \,\le \, \frac{\text {Var}(X)}{\text {Var}(X) + t^2}. \end{aligned}$$

Scaling invariance of Theorem 4

Given any data matrix $\varvec{X} \in {\mathbb {R}}^{d \times M}$ with M samples, the sample covariance matrix is $\varvec{A} = \frac{1}{M} \varvec{X}\varvec{X}^{\top }$. Theorem 4 shows that

$$\begin{aligned} \text {opt}^{{\mathcal {F}}}(\varvec{A}) \le \text {ub}^{\mathcal {CR}i}(\varvec{A}) \le \rho _{\mathcal {CR}i}^2 \cdot \text {opt}^{{\mathcal {F}}}(\varvec{A}) + \underbrace{\sum _{j = 1}^d \frac{r \theta _j^2 \lambda _j(\varvec{A})}{4N^2}}_{\text {additive term}, ~ =: \text {add}(\varvec{A})} =: \text {aff}(\varvec{A}). \end{aligned}$$

Note that rescaling the data matrix $\varvec{X}$ to $\tilde{\varvec{X}} = c \cdot \varvec{X}$ for any constant $c > 0$ does not change the approximation ratios $\rho _{\mathcal {CR}i}^2$ for $i \in \{1, 1', 2\}$ and changes the terms $\text {opt} ^{{\mathcal {F}}}(\varvec{A})$ and $\text {add} (\varvec{A})$ quadratically: letting $\tilde{\varvec{A}} := \frac{1}{M} \tilde{\varvec{X}}\tilde{\varvec{X}}^{\top } = c^2 \varvec{A}$,

$$\begin{aligned} \text {opt}^{{\mathcal {F}}}(\tilde{\varvec{A}})&= \max _{\varvec{V}^{\top } \varvec{V} = \varvec{I}^r, \Vert \varvec{V}\Vert _0 \le k} \text {Tr}(\varvec{V}^{\top }\tilde{\varvec{A}} \varvec{V}) \\&= \max _{\varvec{V}^{\top } \varvec{V} = \varvec{I}^r, \Vert \varvec{V}\Vert _0 \le k} c^2 \cdot \text {Tr}(\varvec{V}^{\top } \varvec{A} \varvec{V}) = c^2 \cdot \text {opt}^{{\mathcal {F}}}(\varvec{A}), \\ \text {add}(\tilde{\varvec{A}})&= \sum _{j = 1}^d \frac{r \theta _j^2 \lambda _j(\tilde{\varvec{A}})}{4N^2} = \sum _{j = 1}^d \frac{r \theta _j^2 \lambda _j(c^2 \varvec{A})}{4N^2} = c^2 \cdot \sum _{j = 1}^d \frac{r \theta _j^2 \lambda _j(\varvec{A})}{4N^2} = c^2 \cdot \text {add}(\varvec{A}). \end{aligned}$$

In particular, this implies that the effective multiplicative ratio (emr) between $\text {opt}^{{\mathcal {F}}}(\varvec{A})$ and the affine upper bound $\text {aff} (\varvec{A})$ is invariant under rescaling:

$$\begin{aligned} \text {emr}(\varvec{A})&:= \frac{\text {aff}(\varvec{A})}{\text {opt}^{{\mathcal {F}}}(\varvec{A})} = \frac{\rho _{\mathcal {CR}i}^2 \cdot \text {opt}^{{\mathcal {F}}}(\varvec{A}) + \text {add}(\varvec{A})}{\text {opt}^{{\mathcal {F}}}(\varvec{A})} \\&= \frac{\rho _{\mathcal {CR}i}^2 \cdot c^2 \cdot \text {opt}^{{\mathcal {F}}}(\varvec{A}) + c^2 \cdot \text {add}(\varvec{A})}{ c^2 \cdot \text {opt}^{{\mathcal {F}}}(\varvec{A})} \\&= \frac{\rho _{\mathcal {CR}i}^2 \cdot \text {opt}^{{\mathcal {F}}}(\tilde{\varvec{A}}) + \text {add}(\tilde{\varvec{A}})}{ \text {opt}^{{\mathcal {F}}}(\tilde{\varvec{A}})} \\&= \frac{\text {aff}(\tilde{\varvec{A}})}{\text {opt}^{{\mathcal {F}}}(\tilde{\varvec{A}})} = \text {emr}(\tilde{\varvec{A}}). \end{aligned}$$

Greedy heuristic for SPCAgs

1.1 Complete pseudocode

1.2 Proof of Lemma 2

By optimality of $\varvec{V}^t_{S_t}$ we can see that $f(S_t) = f(S_t, \varvec{V}^t_{S_t})$ for all t. Thus, letting $\varvec{G}_t := \varvec{I}^k - \varvec{V}^t_{S_{t}} (\varvec{V}_{S_{t}}^t)^{\top }$ to simplify the notation, we have

$$\begin{aligned} f(S_{t-1})&= f(S_{t-1}, \varvec{V}^{t-1}_{S_{t-1}}) = \left\| \varvec{G}_t\, (\varvec{A}^{1/2})_{S_{t - 1}} \right\| _F^2 + \sum _{j \in S_{t - 1}^C} \left\| (\varvec{A}^{1/2})_{j} \right\| _2^2 \\&= \left\| \varvec{G}_t\, \varvec{A}^{1/2}_{S_{t}} \right\| _F^2 + \sum _{j \in S_t^C} \left\| \varvec{A}^{1/2}_{j} \right\| _2^2 + \underbrace{ \Delta ^{\text {in}}_{j^{\text {in}}} - \Delta ^{\text {out}}_{j^{\text {out}}} }_{> 0} \\&> \left\| \varvec{G}_t \varvec{A}^{1/2}_{S_{t}} \right\| _F^2 + \sum _{j \in S_t^C} \left\| \varvec{A}^{1/2}_{j} \right\| _2^2\\&= f(S_t, \varvec{V}^t_{S_t}) = f(S_t). \end{aligned}$$

1.3 Primal Heuristic Algorithm For Near-Maximal Determinant

Here we present another primal heuristic algorithm that finds a principal submatrix whose determinant is near-maximal.

We compare the performance of the greedy neighborhood search Algorithm 1, with the greedy heuristic (GH) 2 in Tables 7 and  8 where we report the relative gap defined as $\text {Gap} := \frac{\text {lb}_{\text {GH}}}{\text {lb}_{\text {GNS}}}$, where $\text {lb}_{\text {GH}}, \text {lb}_{\text {GNS}}$ denote the primal lower bounds of sparse PCA obtained from the greedy heuristic and the greedy neighborhood search respectively.

Table 7 Comparison between GH Algo 2 and GNS Algo 1 on artificial instances where $\text {Gap} := \frac{\text {lb}_{\text {GH}}}{\text {lb}_{\text {GNS}}}$

Table 8 Comparison between GH Algo 2 and GNS Algo 1 on real instances where $\text {Gap} := \frac{\text {lb}_{\text {GH}}}{\text {lb}_{\text {GNS}}}$

Based on the numerical results in Tables 7 and  8, the greedy neighborhood search (GNS) algorithm outperforms the greedy heuristic (GH) in every instance.

Techniques for reducing the running time of CIP

In practice, we want to reduce the running time of CIP. Here are the techniques that we used to enhance the efficiency in practice.

1.1 Threshold

The first technique is to reduce the number of SOS-II constraints in the set PLA. Let $\lambda _{\mathrm {TH}}$ be a threshold parameter that splits the eigenvalues $\{\lambda _j\}_{j = 1}^d$ of sample covariance matrix $\varvec{A}$ into two parts $J^+ = \{j: \lambda _j > \lambda _{\mathrm {TH}} \}$ and $J^- = \{j: \lambda _j \le \lambda _{\mathrm {TH}} \}$. The objective function $\text {Tr} \left( \varvec{V}^{\top } \varvec{A} \varvec{V} \right)$ satisfies

$$\begin{aligned} \text {Tr} \left( \varvec{V}^{\top } \varvec{A} \varvec{V} \right) \!= \!\sum _{j \in J^+} (\lambda _j \!-\! \lambda _{\mathrm {TH}}) \sum _{i = 1}^r g_{ji}^2 \!+\! \sum _{j \in J^-} (\lambda _j - \lambda _{\mathrm {TH}}) \sum _{i = 1}^r g_{ji}^2 + \lambda _{\mathrm {TH}} \sum _{j = 1}^d \sum _{i = 1}^r g_{ji}^2, \end{aligned}$$

in which the first term is convex, the second term is concave, and the third term satisfies

$$\begin{aligned} \lambda _{\mathrm {TH}} \sum _{j = 1}^d \sum _{i = 1}^r g_{ji}^2 \le r \lambda _{\mathrm {TH}} \end{aligned}$$

(threshold-term)

due to $\sum _{j = 1}^d \sum _{i = 1}^r g_{ji}^2 \le r$. Since maximizing a concave function is equivalent to convex optimization, we replace the second term by a new auxiliary variable s and the third term by its upper bound $r \lambda _{\mathrm {TH}}$ such that

$$\begin{aligned} \text {Tr} \left( \varvec{V}^{\top } \varvec{A} \varvec{V} \right) \le&~ \sum _{j \in J^+} (\lambda _j - \lambda _{\mathrm {TH}}) \sum _{i = 1}^r g_{ji}^2 - s + r \lambda _{\mathrm {TH}} \end{aligned}$$

(threshold-tech)

where

$$\begin{aligned} s \ge \sum _{j \in J^-} \underbrace{(\lambda _{\mathrm {TH}} - \lambda _j)}_{\ge 0} \sum _{i = 1}^r g_{ji}^2 \end{aligned}$$

(s-var)

is a convex constraint. We select a value of $\lambda _{\mathrm {TH}}$ so that $|J^+| = 3$. Therefore, it is sufficient to construct a piecewise-linear upper approximation for the quadratic terms $g_{ji}^2$ in the first term with $j \in J^+$, i.e., constraint set $\text {PLA}([J^+] \times [r])$. We thus, greatly reduce the number of SOS-II constraints from ${\mathcal {O}}( d \times r )$ to ${\mathcal {O}}( |J^+| \times r )$, i.e. in our experiments to 3r SOS-II constraints.

1.2 Cutting planes

Similar to classical integer programming, we can incorporate additional cutting planes to improve the efficiency.

Cutting plane for sparsity: The first family of cutting-planes is obtained as follows: Since $\Vert \varvec{V}\Vert _0 \le k$ and $\varvec{v}_1, \ldots , \varvec{v}_r$ are orthogonal, by Bessel inequality, we have

$$\begin{aligned}&\sum _{i = 1}^r g_{ji}^2 = \sum _{i = 1}^r (\varvec{a}_j^{\top } \varvec{v}_i)^2 = \varvec{a}_j^{\top } \varvec{V} \varvec{V}^{\top } \varvec{a}_j \le \theta _j^2, \end{aligned}$$

(sparse-g)

$$\begin{aligned}&\sum _{i = 1}^r \xi _{ji} \le \theta _j^2 \left( 1 + \frac{r}{4N^2} \right) . \end{aligned}$$

(sparse-xi)

We call these above cuts–sparse cut since $\theta _j$ is obtained from the row sparsity parameter k.

Cutting plane from objective value: The second type of cutting plane is based on the property: for any symmetric matrix, the sum of its diagonal entries are equal to the sum of its eigenvalues. Let $\varvec{A}_{j_1, j_1}, \ldots , \varvec{A}_{j_k, j_k}$ be the largest k diagonal entries of the sample covariance matrix $\varvec{A}$, we have

Proposition 1

The following are valid cuts for SPCAgs:

$$\begin{aligned} \sum _{j = 1}^d \lambda _j \sum _{i = 1}^r g_{ji}^2 \le \varvec{A}_{j_1, j_1} + \cdots + \varvec{A}_{j_k, j_k}. \end{aligned}$$

(cut-g)

When the splitting points $\{\gamma _{ji}^{\ell }\}_{\ell = - N}^N$ in SOS-II are set to be $\gamma _{ji}^{\ell } = \frac{\ell }{N} \cdot \theta _j$, we have:

$$\begin{aligned} \begin{array}{rcl} {\sum }_{j \in J^+} (\lambda _j - \lambda _{\mathrm {TH}}){\sum }_{i = 1}^r \xi _{ji} - s + g \lambda _{\mathrm {TH}} &{}\le &{} \varvec{A}_{j_1, j_1} + \cdots + \varvec{A}_{j_k, j_k} + {\sum }_{j \in J^+} \frac{r (\lambda _j - \phi ) \theta _j^2}{4 N^2} \\ g &{}\ge &{}{\sum }_{j = 1}^d {\sum }_{i = 1}^r g_{ji}^2. \end{array} \end{aligned}$$

(cut-xi)

1.3 Implemented version of CIP

Thus the implemented version of CIP is

$$\begin{aligned} \begin{array}{llll} \max &{} {\sum }_{j \in J^+} (\lambda _j - \lambda _{\mathrm {LB}}) {\sum }_{i = 1}^r \xi _{ji} - s + r \lambda _{\mathrm {LB}} \\ \text {s.t} &{} \varvec{V} \in \mathcal {CR}2 \\ &{} (g, \xi , \eta ) \in \text {PLA}([J^+] \times [r]) \\ &{} \text {(s-var), (sparse-g), (sparse-xi), (cut-g), (cut-xi)} \end{array} \end{aligned}$$

(CIP-impl)

1.4 Submatrix technique

Proposition 2

Let $X \in {\mathbb {R}}^{m \times n}$ and let $\theta$ be defined as

$$\begin{aligned} \theta := 2\text {max} _{\varvec{V}^1 \in {\mathbb {R}}^{m \times r}, \varvec{V}^2 \in {\mathbb {R}}^{n \times r}} ~ 2 \text {Tr} \left( (\varvec{V}^1)^{\top } \varvec{X} \varvec{V}^2 \right) \text { s.t. } (\varvec{V}^1)^{\top } \varvec{V}^1 + (\varvec{V}^2)^{\top } \varvec{V}^2 = \varvec{I}^r, \end{aligned}$$

then $\theta \le \sqrt{r} \Vert X\Vert _F$

Proof

$$\begin{aligned}&~ \max _{\varvec{V}^1, \varvec{V}^2} ~ 2 \text {Tr} \left( (\varvec{V}^1)^{\top } \varvec{X} \varvec{V}^2 \right) \text { s.t. } (\varvec{V}^1)^{\top } \varvec{V}^1 + (\varvec{V}^2)^{\top } \varvec{V}^2 = \varvec{I}^r, \\ \Leftrightarrow ~&~ \max _{\varvec{V}^1, \varvec{V}^2} ~ \text {Tr} \left( \begin{pmatrix} \varvec{V}^1)^{\top }&(\varvec{V}^2)^{\top } \end{pmatrix} \begin{pmatrix} 0 &{} \varvec{X} \\ \varvec{X}^{\top } &{} 0 \\ \end{pmatrix} \begin{pmatrix} \varvec{V}^1 \\ \varvec{V}^2 \\ \end{pmatrix} \right) \text { s.t. } (\varvec{V}^1)^{\top } \varvec{V}^1 + (\varvec{V}^2)^{\top } \varvec{V}^2 = \varvec{I}^r, \\ \Leftrightarrow ~&~ \max _{\varvec{V}} ~ \text {Tr}\left( \varvec{V}^{\top } \begin{pmatrix} 0 &{} \varvec{X} \\ \varvec{X}^{\top } &{} 0 \\ \end{pmatrix} \varvec{V} \right) \text { s.t. } \varvec{V}^{\top } \varvec{V} = \varvec{I}^r. \end{aligned}$$

Note that the final maximization problem is equal to

$$\begin{aligned}&\max _{\varvec{V}} ~ \text {Tr}\left( \varvec{V}^{\top } \begin{pmatrix} 0 &{} \varvec{X} \\ \varvec{X}^{\top } &{} 0 \\ \end{pmatrix} \varvec{V} \right) \text { s.t. } \varvec{V}^{\top } \varvec{V} = \varvec{I}^r \\&\quad \le \sum _{i = 1}^r \lambda _i \left( \begin{pmatrix} 0 &{} \varvec{X} \\ \varvec{X}^{\top } &{} 0 \\ \end{pmatrix} \right) , \end{aligned}$$

Next we verify that the eigenvalues of

$$\begin{aligned} \left( \begin{array}{cc} 0 &{} X \\ X^{\top }&{} 0 \end{array}\right) \end{aligned}$$

are ± singular values of X: Let $X = U\Sigma W^{\top }$. In particular, note that:

$$\begin{aligned} \begin{array}{rcccl} \left( \begin{array}{cc} 0 &{} U\Sigma W^{\top } \\ W\Sigma U^{\top }&{} 0 \end{array}\right) \left[ \begin{array}{c} u_i \\ w_i \end{array}\right] &{}=&{} \left[ \begin{array}{c} U\Sigma e_i \\ W\Sigma e_i\end{array}\right] &{}=&{} \sigma _i(X)\left[ \begin{array}{c} u_i \\ w_i \end{array}\right] \\ \left( \begin{array}{cc} 0 &{} U\Sigma W^{\top } \\ W\Sigma U^{\top }&{} 0 \end{array}\right) \left[ \begin{array}{c} u_i \\ -w_i \end{array}\right] &{}=&{} \left[ \begin{array}{c} -U\Sigma e_i \\ W\Sigma e_i\end{array}\right] &{}=&{} -\sigma _i(X)\left[ \begin{array}{c} u_i \\ -w_i \end{array}\right] . \end{array} \end{aligned}$$

Therefore, we have

$$\begin{aligned} \sum _{i = 1}^r \lambda _i \left( \begin{pmatrix} 0 &{} \varvec{X} \\ \varvec{X}^{\top } &{} 0 \\ \end{pmatrix} \right) = \sum _{i = 1}^r \sigma _i(\varvec{X}) \le \sqrt{r} \Vert \varvec{X}\Vert _F. \end{aligned}$$

$\square$