Sparse Exploratory Factor Analysis

Abstract

Sparse principal component analysis is a very active research area in the last decade. It produces component loadings with many zero entries which facilitates their interpretation and helps avoid redundant variables. The classic factor analysis is another popular dimension reduction technique which shares similar interpretation problems and could greatly benefit from sparse solutions. Unfortunately, there are very few works considering sparse versions of the classic factor analysis. Our goal is to contribute further in this direction. We revisit the most popular procedures for exploratory factor analysis, maximum likelihood and least squares. Sparse factor loadings are obtained for them by, first, adopting a special reparameterization and, second, by introducing additional $\ell _1$-norm penalties into the standard factor analysis problems. As a result, we propose sparse versions of the major factor analysis procedures. We illustrate the developed algorithms on well-known psychometric problems. Our sparse solutions are critically compared to ones obtained by other existing methods.

Appendices

Appendix 1

Here, we find the gradient of the penalty term $P_\tau (Q)^\top P_\tau (Q)$ in (11) and (12), which can be then combined with the gradients of the objective functions of ML-, LS-, or GLS-EFA. Let start with

$$\begin{aligned} d(P(Q_\tau )^\top P_\tau (Q)) = 2 d(P_\tau (Q))^\top P_\tau (Q) = 2 d(P_\tau )^\top P_\tau , \end{aligned}$$

(14)

which requires the calculation of $d(P_\tau )$. At this point, we need an approximation of $\text{ sign }(x)$, and we employ the one already used in (Trendafilov & Jolliffe, 2006), which is $\text{ sign }(x) \approx \text{ tanh }(\gamma x)$ for some large $\gamma > 0$, or for short $\text{ th }(\gamma x)$. See also (Hage & Kleinsteuber, 2014; Luss & Teboulle, 2013). Then

$$\begin{aligned} 2(dP_\tau )= & {} (d{\mathbf {q}}_\tau ) \odot [1_r + \text{ th }(\gamma {\mathbf {q}}_\tau )] + {\mathbf {q}}_\tau \odot [1_r - \text{ th }^2(\gamma {\mathbf {q}}_\tau )] \odot \gamma (d{\mathbf {q}}_\tau ),\nonumber \\= & {} (d{\mathbf {q}}_\tau ) \odot \left\{ 1_r + \text{ th }(\gamma {\mathbf {q}}_\tau ) + \gamma {\mathbf {q}}_\tau \odot [1_r - \text{ th }^2(\gamma {\mathbf {q}}_\tau )] \right\} , \end{aligned}$$

(15)

where $1_r$ is a $r \times 1$ vector with unit entries. The next differential to be found is:

$$\begin{aligned} dq_\tau= & {} 1_p^\top \left\{ (dQ) \odot \text{ th }(\gamma Q) + Q \odot [1_{p \times r} - \text{ th }^2(\gamma Q)] \odot \gamma (dQ)\right\} \nonumber \\= & {} 1_p^\top \left\{ (dQ) \odot \{ \text{ th }(\gamma Q) + (\gamma Q) \odot [1_{p \times r} - \text{ th }^2(\gamma Q)\} \right\} , \end{aligned}$$

(16)

where $1_{p \times r}$ is a $p \times r$ matrix with unit entries.

Now we are ready to find the gradient $\nabla _Q$ of the penalty term with respect to Q. To simplify the notations, let

$$\begin{aligned} {{\mathbf {w}}} = 1_r + \text{ th }(\gamma {{\mathbf {q}}}_\tau ) + (\gamma {{\mathbf {q}}}_\tau ) \odot [1_r - \text{ th }^2(\gamma {{\mathbf {q}}}_\tau )], \end{aligned}$$

(17)

and

$$\begin{aligned} W = \text{ th }(\gamma Q) + (\gamma Q) \odot [1_{p \times r} - \text{ th }^2(\gamma Q)]. \end{aligned}$$

(18)

Going back to (14) and (15), we find that:

$$\begin{aligned} 2 (dP_\tau )^\top P_\tau= & {} \text{ trace } [(d{{\mathbf {q}}}_\tau ) \odot {{\mathbf {w}}} ]^\top P_\tau = \text{ trace } (d{{\mathbf {q}}}_\tau )^\top ({{\mathbf {w}}} \odot P_\tau ) \nonumber \\= & {} \text{ trace }\{ 1_p^\top [(dQ) \odot W] \}^\top ({{\mathbf {w}}} \odot P_\tau ) \nonumber \\= & {} \text{ trace } [(dQ)^\top \odot W^\top ] 1_p ({\mathbf {w}} \odot P_\tau ) \nonumber \\= & {} \text{ trace } (dQ)^\top \{W \odot [1_p ({\mathbf {w}} \odot P_\tau )]\}, \end{aligned}$$

(19)

making use of the identity $\text{ trace } (A \odot B) C = \text{ trace } A (B^\top \odot C)$. Thus, the gradient $\nabla _Q$ of the penalty term with respect to Q is:

$$\begin{aligned} \nabla _Q = W \odot [1_p ({\mathbf {w}} \odot P_\tau )]. \end{aligned}$$

(20)

Appendix 2

Here, we summarize some technical details related to the numerical solutions employed in the work.

The gradients of the ML-, LS- and GLS-EFA objective functions with respect to the unknowns $\{Q, D, \Psi \}$ are given in Trendafilov (2003) as the following block-matrix: $( -Y \textit{QD}^{2}, -Q^{T} Y Q \odot D, -Y \odot \Psi )$. For ML-EFA, one has $Y = 2 R_{ZZ}^{-1} (R-R_{ZZ}) R_{ZZ}^{-1}$, and for LS- and GLS-EFA, it changes to $Y = 4 (R - R_{ZZ})V^2$. Additionally, we need the gradient $\nabla _Q$ of the penalty term $P_\tau (Q)^\top P_\tau (Q)$ with respect to Q, which should be added to $-Y \textit{QD}^{2}$. Its derivation is given in details in Appendix.

The dynamical system approach employed in (Trendafilov, 2003) can be readily applied for solving (11) and (12). It involves numerical integration of matrix ordinary differential equations (ODE) for $\{Q, D, \Psi \}$ defined by their projected gradients. Particularly, it involves projected gradient dynamical system for Q on the Stiefel manifold of all $p \times r$ orthonormal matrices. There exist a number of specialized numerical methods for solving such problem listed in (Trendafilov, 2003), e.g. Del Buono & Lopez (2001) and etc. In contrast to the standard EFA alternating approaches (Jöreskog, 1977; Mulaik, 2010), the dynamical system approach gives matrix algorithms which produce simultaneous solution for $\{Q, D, \Psi \}$ exploiting the geometry of their specific matrix structures. Moreover, such algorithms are globally convergent, i.e. the convergence is reached independently of the starting (initial) point (Absil et al., 2008; Trendafilov, 2003).

The numerical ODE solvers currently available in MATLAB (MATLAB, 2014) are not suitable for solving large optimization problems. They track the whole trajectory defined by the ODE which is time-consuming and undesirable when the asymptotic state is of interest only. This limits the application of the proposed approach to solving (11) and (12) for rather small data sets.

An alternative way is to employ iterative algorithms directly working on matrix manifolds (Absil et al., 2008; Edelman et al., 1998; Wen & Yin, 2013). The listed above gradients can be readily used for solving (11) and (12) by employing MANOPT, a free MATLAB-based software for optimization on matrix manifolds (Boumal et al., 2014). The MANOPT code for solving (11) and (12) can be obtained from the authors upon request, and will be available online. Note that by choosing $\mu = 0$, one can obtain solutions for the standard ML-, LS- and GLS-EFA problems (9) and (10).