Sparse estimation via nonconcave penalized likelihood in factor analysis model

Abstract

We consider the problem of sparse estimation in a factor analysis model. A traditional estimation procedure in use is the following two-step approach: the model is estimated by maximum likelihood method and then a rotation technique is utilized to find sparse factor loadings. However, the maximum likelihood estimates cannot be obtained when the number of variables is much larger than the number of observations. Furthermore, even if the maximum likelihood estimates are available, the rotation technique does not often produce a sufficiently sparse solution. In order to handle these problems, this paper introduces a penalized likelihood procedure that imposes a nonconvex penalty on the factor loadings. We show that the penalized likelihood procedure can be viewed as a generalization of the traditional two-step approach, and the proposed methodology can produce sparser solutions than the rotation technique. A new algorithm via the EM algorithm along with coordinate descent is introduced to compute the entire solution path, which permits the application to a wide variety of convex and nonconvex penalties. Monte Carlo simulations are conducted to investigate the performance of our modeling strategy. A real data example is also given to illustrate our procedure.

Appendices

Appendix A: Derivation of complete-data penalized log-likelihood function in EM algorithm

In order to apply the EM algorithm, first, the common factors $\varvec{f}_n$ can be regarded as missing data and maximize the complete-data penalized log-likelihood function

$$\begin{aligned} {l}_{\rho }^{C} ({\varvec{\varLambda }},\varvec{\varPsi }) = \sum _{n=1}^N \log f(\varvec{x}_n,\varvec{f}_n) - N \sum _{i=1}^p\sum _{j=1}^m \rho P(|\lambda _{ij}|), \end{aligned}$$

where the density function $f(\varvec{x}_n,\varvec{f}_n)$ is defined by

$$\begin{aligned} f(\varvec{x}_n,\varvec{f}_n)&= \prod _{i=1}^p \left\{ (2\pi \psi _i)^{-1/2} \exp \left( - \frac{ (x_{ni}-\varvec{\lambda }_i^T\varvec{f}_n )^2}{2\psi _i} \right) \right\} \\&\cdot \ (2\pi )^{-m/2}\exp \left( - \frac{\Vert \varvec{f}_n \Vert ^2}{2} \right) \end{aligned}$$

Then, the expectation of ${l}_{\rho }^{C}({\varvec{\varLambda }},\varvec{\varPsi })$ can be taken with respect to the distributions $f(\varvec{f}_n | \varvec{x}_n,{\varvec{\varLambda }},\varvec{\varPsi })$,

$$\begin{aligned}&E[{l}_{\rho }^{C} ({\varvec{\varLambda }},\varvec{\varPsi })]= \\&-\frac{N(p+m)}{2} \log (2\pi ) - \frac{N}{2} \sum _{i=1}^p \log \psi _i \\&- \frac{1}{2} \sum _{n=1}^N\sum _{i=1}^p \frac{x_{ni}^2 - 2x_{ni}\varvec{\lambda }_i^TE[\varvec{F}_n|\varvec{x}_n]+ \varvec{\lambda }_i^T E[\varvec{F}_n\varvec{F}_n^T|\varvec{x}_n]\varvec{\lambda }_i}{\psi _i} \\&- \frac{1}{2} \hbox { tr} \left\{ \sum _{n=1}^N E[\varvec{F}_n\varvec{F}_n^T|\varvec{x}_n] \right\} - N \sum _{i=1}^p\sum _{j=1}^m \rho P(|\lambda _{ij}|) \end{aligned}$$

For given ${\varvec{\varLambda }}_\text {old}$ and $\varvec{\varPsi }_\text {old}$, the posterior $f(\varvec{f}_n | \varvec{x}_n,{\varvec{\varLambda }}_\text {old}, \varvec{\varPsi }_\text {old})$ is normally distributed with $E[\varvec{F}_n|\varvec{x}_n] = \varvec{M}^{-1}{\varvec{\varLambda }}_\text {old}^T\varvec{\varPsi }_\text {old}^{-1} \varvec{x}_n$ and $E[\varvec{F}_n\varvec{F}_n^T|\varvec{x}_n] = \varvec{M} ^{-1} + E[\varvec{F}_n|\varvec{x}_n] E[\varvec{F}_n|\varvec{x}_n] ^T$, where $\varvec{M} = {\varvec{\varLambda }}_\text {old}^T\varvec{\varPsi }_\text {old}^{-1}{\varvec{\varLambda }}_\text {old} + \varvec{I}_m$. Then, we have

$$\begin{aligned} \sum _{n=1}^N E[\varvec{F}_n]x_{ni}&= N\varvec{M}^{-1}{\varvec{\varLambda }}_\text {old}^T\varvec{\varPsi }_\text {old}^{-1}\varvec{s}_i,\\ \sum _{n=1}^N E[\varvec{F}_n\varvec{F}_n^T]&= N (\varvec{M} ^{-1} + \varvec{M}^{-1}{\varvec{\varLambda }}_\text {old}^T\varvec{\varPsi }_\text {old}^{-1}\varvec{S}\varvec{\varPsi }_\text {old}^{-1}{\varvec{\varLambda }}_\text {old}\varvec{M}^{-1}). \end{aligned}$$

Let $\varvec{M}^{-1}{\varvec{\varLambda }}_\text {old}^T\varvec{\varPsi }_\text {old}^{-1}\varvec{s}_i$ and $\varvec{M} ^{-1} + \varvec{M}^{-1}{\varvec{\varLambda }}_\text {old}^T\varvec{\varPsi }_\text {old}^{-1}\varvec{S}\varvec{\varPsi }_\text {old}^{-1}{\varvec{\varLambda }}_\text {old}\varvec{M}^{-1}$ be $\varvec{b}_i$ and $\varvec{A}$, respectively. Then, the expectation of ${l}_{\rho }^{C}({\varvec{\varLambda }},\varvec{\varPsi })$ in (7) can be derived.

Appendix B: proof of Lemma 1

The proof is by contradiction. Assume that $\hat{{\varvec{\varLambda }}}$ and $\hat{\varvec{\varPsi }}$ are the solution of (6) and the $j$th column of $\hat{{\varvec{\varLambda }}}$ has only one nonzero element, say, $\hat{\lambda }_{aj}$. Another parameter $\hat{{\varvec{\varLambda }}}^*$ and $\hat{\varvec{\varPsi }}^*$ are defined, where $\hat{{\varvec{\varLambda }}}^*$ is same as $\hat{{\varvec{\varLambda }}}$ but with the $aj$th element being zero and $\hat{\varvec{\varPsi }}^*$ is same as $\hat{\varvec{\varPsi }}$ but with the $j$th diagonal element being $\hat{\psi }_j+\hat{\lambda }_{aj}^2$. In this case, we have the same covariance structure, i.e., $\hat{{\varvec{\varLambda }}}\hat{{\varvec{\varLambda }}}^T+\hat{\varvec{\varPsi }}=\hat{{\varvec{\varLambda }}}^*\hat{{\varvec{\varLambda }}}^{*T}+\hat{\varvec{\varPsi }}^*$, which suggests $\ell (\hat{{\varvec{\varLambda }}}, \hat{\varvec{\varPsi }}) = \ell (\hat{{\varvec{\varLambda }}}^*, \hat{\varvec{\varPsi }}^*)$, whereas the penalty term of $\sum _{i=1}^p\sum _{j=1}^m \rho P(|\hat{\lambda }_{ij}|)$ is larger than $\sum _{i=1}^p\sum _{j=1}^m \rho P(|\hat{\lambda }^*_{ij}|)$. This means $\ell _{\rho }(\hat{{\varvec{\varLambda }}}, \hat{\varvec{\varPsi }}) < \ell _{\rho }(\hat{{\varvec{\varLambda }}}^*, \hat{\varvec{\varPsi }}^*)$, which contradicts the assumption that $\hat{{\varvec{\varLambda }}}$ and $\hat{\varvec{\varPsi }}$ are penalized maximum likelihood estimates.