Computation of the maximum likelihood estimator in low-rank factor analysis

Abstract

Factor analysis is a classical multivariate dimensionality reduction technique popularly used in statistics, econometrics and data science. Estimation for factor analysis is often carried out via the maximum likelihood principle, which seeks to maximize the Gaussian likelihood under the assumption that the positive definite covariance matrix can be decomposed as the sum of a low-rank positive semidefinite matrix and a diagonal matrix with nonnegative entries. This leads to a challenging rank constrained nonconvex optimization problem, for which very few reliable computational algorithms are available. We reformulate the low-rank maximum likelihood factor analysis task as a nonlinear nonsmooth semidefinite optimization problem, study various structural properties of this reformulation; and propose fast and scalable algorithms based on difference of convex optimization. Our approach has computational guarantees, gracefully scales to large problems, is applicable to situations where the sample covariance matrix is rank deficient and adapts to variants of the maximum likelihood problem with additional constraints on the model parameters. Our numerical experiments validate the usefulness of our approach over existing state-of-the-art approaches for maximum likelihood factor analysis.

Appendix

Proposition 10

(See Sect. 6 in [10]) Suppose the function $g: \mathbf {E} \mapsto (-\infty , \infty )$ is convex, and the point x lies in interior of dom(g) with $\mathbf {E} \subset \mathfrak {R}^{m}$. Let $x^r,x \in \mathbf {E}$ and $\nu ^r$ be a subgradient of g evaluated at $x^r$ for $r\ge 1$. If $x^{r} \rightarrow x$ and $\nu ^{r} \rightarrow \nu$ as $r \rightarrow \infty$, then $\nu$ is a subgradient of g evaluated at x.

1.1 Proof of Proposition 9

Note that the objective function $f(\varvec{\phi })$ (see (22)) is unbounded above when $\phi _{i} \rightarrow 0$ for any $i \in [p]$—see also Proposition 3. This implies that there exists a $\alpha >0$ such that $\varvec{\phi }^{(k)} \in [\alpha , \tfrac{1}{\epsilon }]^p$ for all k. The boundedness of the sequence $\varvec{\phi }^{(k)}$ implies the existence of a limit point of $\varvec{\phi }^{(k)}$, say, $\varvec{\phi }^{*}$. Let $\varvec{\phi }^{(k_j)}$ be a subsequence such that $\varvec{\phi }^{(k_j)} \rightarrow \varvec{\phi }^{*}$. Note that for every k, $\varvec{\phi }^{(k+1)} \in {\hbox {arg min}}_{\varvec{\phi }\in {\mathcal {C}}} F(\varvec{\phi }; \varvec{\phi }^{(k)})$ is equivalent to

$$\begin{aligned} \left\langle \nabla f_{1}(\varvec{\phi }^{(k+1)}) - \partial f_{2} ( \varvec{\phi }^{(k)}), \varvec{\phi }- \varvec{\phi }^{(k+1)} \right\rangle \ge 0~~\forall \varvec{\phi }\in {{\mathcal {C}}}. \end{aligned}$$

(54)

Now consider the sequence $\varvec{\phi }^{(k_j)}$ as $k_j \rightarrow \infty$. Using the fact that $\varvec{\phi }^{(k+1)} - \varvec{\phi }^{(k)} \rightarrow \mathbf {0}$; it follows from the continuity of $\nabla f_{1}(\cdot )$ that: $\nabla f_{1}(\varvec{\phi }^{(k_j+1)}) \rightarrow \nabla f_{1}(\varvec{\phi }^{*})$ as $k_j \rightarrow \infty$.

Note that $\partial f_2(\varvec{\phi }^{(k_j)})$ (see (36)) is bounded as $\varvec{\phi }^{(k_j)} \in [\alpha , \tfrac{1}{\epsilon }]^p$. Passing onto a further subsequence $\{k'_j\}$ if necessary, it follows that $\partial f_2(\varvec{\phi }^{(k'_j)}) \rightarrow \vartheta$ (say). Using Proposition 10, we conclude that $\vartheta$ is a subgradient of $f_2$ evaluated at $\varvec{\phi }^*$. As $k'_j \rightarrow \infty$, the above argument along with (54) implies that:

$$\begin{aligned} \left\langle \nabla f_{1}(\varvec{\phi }^{*}) - \partial f_{2} ( \varvec{\phi }^{*}), \varvec{\phi }- \varvec{\phi }^{*} \right\rangle \ge 0~~\forall \varvec{\phi }\in {{\mathcal {C}}}, \end{aligned}$$

(55)

where, $\partial f_{2} ( \varvec{\phi }^{*})$ is a subgradient of $f_{2}$ evaluated at $\varvec{\phi }^{*}$. (55) implies that $\varvec{\phi }^*$ is a first order stationary point.

1.2 Representing $\partial {h(\varvec{\Psi },\mathbf {L})}/{\partial \mathbf {L}}=0$

Note that $\partial {h(\varvec{\Psi },\mathbf {L})}/{\partial \mathbf {L}}=0$ is equivalent to setting (12) to zero, which leads to $\mathbf {L}=\mathbf {S}\varvec{\Sigma }^{-1}\mathbf {L}$. By applying Sherman Woodbury formula on $(\varvec{\Psi }+ \mathbf {L}\mathbf {L}^\top )^{-1}$, we have the following:

$$\begin{aligned} \mathbf {L}= & {} \mathbf {S}\varvec{\Sigma }^{-1}\mathbf {L} \nonumber \\= & {} \mathbf {S}\left( \varvec{\Psi }^{-1} - \varvec{\Psi }^{-1}\mathbf {L}\left( \mathbf {I} + \mathbf {L}^\top \varvec{\Psi }^{-1}\mathbf {L} \right) ^{-1}\mathbf {L}^\top \varvec{\Psi }^{-1} \right) \mathbf {L} \end{aligned}$$

(56)

$$\begin{aligned}= & {} \mathbf {S}\varvec{\Psi }^{-1}\mathbf {L} - \mathbf {S}\varvec{\Psi }^{-1}\mathbf {L} \left( \left( \mathbf {I} + \mathbf {L}^\top \varvec{\Psi }^{-1}\mathbf {L} \right) ^{-1} \mathbf {L}^\top \varvec{\Psi }^{-1}\mathbf {L} \right) \end{aligned}$$

(57)

$$\begin{aligned}= & {} \mathbf {S}\varvec{\Psi }^{-1}\mathbf {L} - \mathbf {S}\varvec{\Psi }^{-1}\mathbf {L} \left( \mathbf {I} - \left( \mathbf {I} + \mathbf {L}^\top \varvec{\Psi }^{-1}\mathbf {L} \right) ^{-1} \right) \end{aligned}$$

(58)

$$\begin{aligned}= & {} \mathbf {S}\varvec{\Psi }^{-1}\mathbf {L}\left( \mathbf {I} + \mathbf {L}^\top \varvec{\Psi }^{-1}\mathbf {L} \right) ^{-1}, \end{aligned}$$

(59)

where, Eqn (56) follows from (8); Eqn (58) follows from (57) by using the observation that for a PSD matrix $\mathbf {B}$, we have the following identity: $(\mathbf {I}+\mathbf {B})^{-1}\mathbf {B} = \mathbf {I} - (\mathbf {I} + \mathbf {B})^{-1}$ (this can be verified by simple algebra).

Finally, we note that (13) follows by using the definition of $\mathbf {L}^*$ and $\mathbf {S}^*$ in (59) and doing some algebraic manipulations.