Statistics and Computing

, Volume 18, Issue 2, pp 109–123 | Cite as

ML estimation for factor analysis: EM or non-EM?

Article
First Online:
Received:
Accepted:

Abstract

To obtain maximum likelihood (ML) estimation in factor analysis (FA), we propose in this paper a novel and fast conditional maximization (CM) algorithm, which has quadratic and monotone convergence, consisting of a sequence of CM log-likelihood (CML) steps. The main contribution of this algorithm is that the closed form expression for the parameter to be updated in each step can be obtained explicitly, without resorting to any numerical optimization methods. In addition, a new ECME algorithm similar to Liu’s (Biometrika 81, 633–648, 1994) one is obtained as a by-product, which turns out to be very close to the simple iteration algorithm proposed by Lawley (Proc. R. Soc. Edinb. 60, 64–82, 1940) but our algorithm is guaranteed to increase log-likelihood at every iteration and hence to converge. Both algorithms inherit the simplicity and stability of EM but their convergence behaviors are much different as revealed in our extensive simulations: (1) In most situations, ECME and EM perform similarly; (2) CM outperforms EM and ECME substantially in all situations, no matter assessed by the CPU time or the number of iterations. Especially for the case close to the well known Heywood case, it accelerates EM by factors of around 100 or more. Also, CM is much more insensitive to the choice of starting values than EM and ECME.

Keywords

CM ECME EM Factor analysis Maximum likelihood estimation 
This is a preview of subscription content, log in to check access.

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. Bartholomew, D.J.: Latent Variable Models and Factor Analysis, 1st edn. Oxford University Press, New York (1987) MATHGoogle Scholar
  2. Chen, R.: An SAS/IML procedure for maximum likelihood factor analysis. Behav. Res. Methods Instrum. Comput. 35(3), 310–317 (2003) Google Scholar
  3. Dempster, A.P., Laird, N.M., Rubin, D.B.: Maximum likelihood from incomplete data using the EM algorithm (with discussion). J. R. Stat. Soc. B 39(1), 1–38 (1977) MATHMathSciNetGoogle Scholar
  4. Fletcher, R., Powell, M.J.D.: A rapidly convergent descent method for minimization. Comput. J. 2, 163–168 (1963) MathSciNetGoogle Scholar
  5. Hastie, T., Tibshirani, R., Friedman, J.: The Elements of Statistical Learning. Springer, New York (2001) MATHGoogle Scholar
  6. Jennrich, R.I., Bobinson, S.M.: A Newton-Raphson algorithm for maximum likelihood factor analysis. Psychometrika 34(1), 111–123 (1969) CrossRefMathSciNetGoogle Scholar
  7. Jöreskog, K.G.: Some contributions to maximum likelihood factor analysis. Psychometrika 32(4), 433–482 (1967) CrossRefGoogle Scholar
  8. Lange, K.: Numerical Analysis for Statisticians. Springer, New York (1999) MATHGoogle Scholar
  9. Lawley, D.N.: The estimation of factor loadings by the method of maximum likelihood. Proc. R. Soc. Edinb. A 60, 64–82 (1940) MathSciNetGoogle Scholar
  10. Lawley, D.N., Maxwell, A.E.: Factor Analysis as a Statistical Method, 2nd edn. Butterworths, London (1971) MATHGoogle Scholar
  11. Liu, C.: The ECME algorithm: a simple extention of EM and ECM with faster monotone convergence. Biometrika 81, 633–648 (1994) MATHCrossRefMathSciNetGoogle Scholar
  12. Liu, C., Rubin, D.B.: Maximum likelihood estimation of factor analysis using the ECME algorithm with complete and incomplete data. Stat. Sinica 8, 729–747 (1998) MATHMathSciNetGoogle Scholar
  13. Meng, X.-L., Rubin, D.B.: Maximum likelihood estimation via the ECM algorithm: a general framework. Biometrika 80(2), 267–278 (1993) MATHCrossRefMathSciNetGoogle Scholar
  14. Meng, X.L., van Dyk, D.A.: The EM algorithm—an old folk-song sung to a fast new tune. J. R. Stat. Soc. B 59(3), 511–567 (1997) MATHCrossRefGoogle Scholar
  15. Petersen, K.B., Winther, O., Hansen, L.K.: On the slow convergence of EM and VBEM in low-noise linear models. Neural Comput. 17(9), 1921–1926 (2005) MATHCrossRefMathSciNetGoogle Scholar
  16. Rubin, D.B., Thayer, T.T.: EM algorithms for ML factor analysis. Psychometrika 47, 69–76 (1982) MATHCrossRefMathSciNetGoogle Scholar
  17. Tipping, M.E., Bishop, C.M.: Probabilistic principal component analysis. J. R. Stat. Soc. Ser. B 61, 611–622 (1999) MATHCrossRefMathSciNetGoogle Scholar
  18. Wu, C.F.J.: On the convergence properties of the EM algorithm. Ann. Stat. 11(1), 95–103 (1983) MATHCrossRefGoogle Scholar
  19. Yu, P.L.H., Lam, K.F., Lo, S.M.: Factor analysis for ranked data with application to a job selection attitude survey. J. R. Stat. Soc. Ser. A 168(3), 583–597 (2005) MATHCrossRefMathSciNetGoogle Scholar

Copyright information

© Springer Science+Business Media, LLC 2007

Authors and Affiliations

  1. 1.Department of Statistics and Actuarial ScienceThe University of Hong KongFokfulamHong Kong
  2. 2.Department of StatisticsYunnan UniversityKunmingChina
  3. 3.Department of MathematicsSoutheast UniversityNanjingChina

Personalised recommendations