Skip to main content
Log in

Improving the convergence rate and speed of Fisher-scoring algorithm: ridge and anti-ridge methods in structural equation modeling

  • Published:
Annals of the Institute of Statistical Mathematics Aims and scope Submit manuscript

Abstract

In structural equation modeling (SEM), parameter estimates are typically computed by the Fisher-scoring algorithm, which often has difficulty in obtaining converged solutions. Even for simulated data with a correctly specified model, non-converged replications have been repeatedly reported in the literature. In particular, in Monte Carlo studies it has been found that larger factor loadings or smaller error variances in a confirmatory factor model correspond to a higher rate of convergence. However, studies of a ridge method in SEM indicate that adding a diagonal matrix to the sample covariance matrix also increases the rate of convergence for the Fisher-scoring algorithm. This article addresses these two seemingly contradictory phenomena. Using statistical and numerical analyses, the article clarifies why both approaches increase the rate of convergence in SEM. Monte Carlo results confirm the analytical results. Recommendations are provided on how to increase both the speed and rate of convergence in parameter estimation.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

Notes

  1. The implementation of the FS algorithm as described in Eqs. (2) and (3) is straightforward. Readers are welcome to contact the authors to obtain an electronic copy of the SAS IML code.

References

  • Anderson, J. C., Gerbing, D. W. (1984). The effect of sampling error on convergence, improper solutions, and goodness-of-fit indices for maximum likelihood confirmatory factor analysis. Psychometrika, 49, 155–173.

  • Bentler, P. M. (2008). EQS 6 structural equations program manual. Encino, CA: Multivariate Software.

    Google Scholar 

  • Bentler, P. M., Tanaka, J. S. (1983). Problems with EM algorithms for ML factor analysis. Psychometrika, 48, 247–251.

  • Bentler, P. M., Yuan, K.-H. (1999). Structural equation modeling with small samples: Test statistics. Multivariate Behavioral Research, 34, 181–197.

  • Boomsma, A. (1985). Nonconvergence, improper solutions, and starting values in LISREL maximum likelihood estimation. Psychometrika, 50, 229–242.

    Article  Google Scholar 

  • Browne, M. W. (1974). Generalized least squares estimators in the analysis of covariance structures. South African Statistical Journal, 8, 1–24.

    MathSciNet  MATH  Google Scholar 

  • Browne, M. W. (1984). Asymptotic distribution-free methods for the analysis of covariance structures. British Journal of Mathematical and Statistical Psychology, 37, 62–83.

    Article  MathSciNet  MATH  Google Scholar 

  • Dempster, A. P., Laird, N. M., Rubin, D. B. (1977). Maximum likelihood from incomplete data via the EM algorithm. Journal of the Royal Statistical Society B, 39, 1–38.

  • Everitt, B. S. (1987). Introduction to optimization methods and their application in statistics. London: Chapman & Hall.

    Book  MATH  Google Scholar 

  • Ferguson, T. (1996). A course in large sample theory. London: Chapman & Hall.

    Book  MATH  Google Scholar 

  • Golub, G. H., Van Loan, C. F. (1983). Matrix computations. Baltimore, MD: Johns Hopkins University Press.

  • Hu, L. T., Bentler, P. M., Kano, Y. (1992). Can test statistics in covariance structure analysis be trusted? Psychological Bulletin, 112, 351–362.

  • Huang, Y., Bentler, P. M. (2015). Behavior of asymptotically distribution free test statistics in covariance versus correlation structure analysis. Structural Equation Modeling, 22, 489–503.

  • Ichikawa, M., Konishi, S. (1995). Application of the bootstrap methods in factor analysis. Psychometrika, 60, 77–93.

  • Jackson, D. L. (2001). Sample size and number of parameter estimates in maximum likelihood confirmatory factor analysis: A Monte Carlo investigation. Structural Equation Modeling, 8, 205–223.

    Article  Google Scholar 

  • Jamshidian, M., Bentler, P. M. (1999). Using complete data routines for ML estimation of mean and covariance structures with missing data. Journal Educational and Behavioral Statistics, 23, 21–41.

  • Jöreskog, K. G., Sörbom, D. (1981). LISREL: Analysis of linear structural relationships by the method of maximum likelihood (version V). Chicago, IL: National Educational Resources Inc.

  • Kamada, A. (2011). Selecting a shrinkage parameter in structural equation modeling with a near singular covariance matrix by the GIC minimization method. Technical report 11-04, Statistical Research Group, Hiroshima University, Hiroshima. (http://www.math.sci.hiroshima-u.ac.jp/~stat/TR/TR11/TR11-04).

  • Kamada, A., Kano, Y. (2012). Statistical inference in structural equation modeling with a near singular covariance matrix. In Paper presented at The 2nd institute of mathematical statistics Asia Pacific Rim meeting. Tsukuba, Japan.

  • Nocedal, J., Wright, S. J. (1999). Numerical optimization (2nd ed.). New York: Springer.

  • Preacher, K. J., Wichman, A. L., MacCallum, R. C., Briggs, N. E. (2008). Latent growth curve modeling. Thousand Oaks, CA: Sage.

  • SAS Institute (1996). SAS/STAT Software: Changes and enhancements through release 6.11. Cary, NC: Author.

  • SAS Institute Inc. (2011). SAS/STAT 9.3 user’s guide. Cary, NC: Author.

  • Savalei, V., Falk, C. F. (2014). Robust two-stage approach outperforms robust full information maximum likelihood with incomplete nonnormal data. Structural Equation Modeling, 21, 280–302.

  • Schott, J. (2005). Matrix analysis for statistics (2nd ed.). New York: Wiley.

    MATH  Google Scholar 

  • Velicer, W. F., Fava, J. L. (1998). Effects of variable and subject sampling on factor pattern recovery. Psychological Methods, 3, 231–251.

  • Wu, C. F. (1983). On the convergence properties of the EM algorithm. Annals of Statistics, 11, 95–103.

    Article  MathSciNet  MATH  Google Scholar 

  • Yuan, K.-H., Bentler, P. M. (2000). Three likelihood-based methods for mean and covariance structure analysis with nonnormal missing data. Sociological Methodology, 30, 167–202.

  • Yuan, K.-H., Chan, W. (2008). Structural equation modeling with near singular covariance matrices. Computational Statistics and Data Analysis, 52, 4842–4858.

  • Yuan, K.-H., Hayashi, K. (2003). Bootstrap approach to inference and power analysis based on three statistics for covariance structure models. British Journal of Mathematical and Statistical Psychology, 56, 93–110.

Download references

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Ke-Hai Yuan.

Additional information

The research is supported by the National Science Foundation under Grant No. SES-1461355.

Appendix

Appendix

This appendix provides the details leading to the coefficient of variations (CV) of the sample covariances as given in (7), (8) and (9). With \(y_{j0}=y_j-\mu _j\) and \(y_{k0}=y_k-\mu _k\), the main work is to obtain \(\gamma _{jk}=\mathrm{Var}(y_{j0}y_{k0})\).

It follows from (5) that

$$\begin{aligned} y_{j0}y_{k0}=\lambda _j\lambda _k\xi _{j^*}\xi _{k^*} +\lambda _j\xi _{j^*}\varepsilon _{k}+\lambda _k\xi _{k^*}\varepsilon _{j} +\varepsilon _{j}\varepsilon _{k} \end{aligned}$$

and

$$\begin{aligned} y_{j0}^2y_{k0}^2= & {} \lambda _j^2\lambda _k^2\xi _{j^*}^2\xi _{k^*}^2+\lambda _j^2\xi _{j^*}^2\varepsilon _{k}^2+\lambda _k^2\xi _{k^*}^2\varepsilon _{j}^2+\varepsilon _{j}^2\varepsilon _{k}^2 +2\left( \lambda _j^2\lambda _k\xi _{j^*}^2\xi _{k^*}\varepsilon _{k}\right. \nonumber \\&\left. +\lambda _j\lambda _k^2\xi _{j^*}\xi _{k^*}^2\varepsilon _{j}+2\lambda _j\lambda _k\xi _{j^*}\xi _{k^*}\varepsilon _{j}\varepsilon _{k}+ \lambda _j\xi _{j^*}\varepsilon _{j}\varepsilon _{k}^2 +\lambda _k\xi _{k^*}\varepsilon _{j}^2\varepsilon _{k}\right) . \end{aligned}$$
(28)

Taking the expected value of (28) term by term, we have:

when \(j^*\ne k^*\),

$$\begin{aligned} E\left( y_{j0}^2y_{k0}^2\right) =\lambda _j^2\lambda _k^2E\left( \xi _{j*}^2\xi _{k^*}^2\right) +\lambda _j^2\psi _{kk}+\lambda _k^2\psi _{jj}+\psi _{jj}\psi _{kk}; \end{aligned}$$
(29)

when \(j^*=k^*\) but \(j\ne k\),

$$\begin{aligned} E\left( y_{j0}^2y_{k0}^2\right) =\lambda _j^2\lambda _k^2E\left( \xi _{j^*}^4\right) +\lambda _j^2\psi _{kk}+\lambda _k^2\psi _{jj}+\psi _{jj}\psi _{kk}; \end{aligned}$$
(30)

and when \(j=k\),

$$\begin{aligned} E\left( y_{j0}^4\right) =\lambda _j^4E\left( \xi _{j^*}^4\right) +6\lambda _j^2\psi _{jj} +E\left( \varepsilon _j^4\right) . \end{aligned}$$
(31)

Notice that \(\sigma _{jk}=\lambda _j\lambda _k\phi _{j^*k^*}\) when \(j^*\ne k^*\), it follows from (29) that

$$\begin{aligned} \gamma _{jk}=\lambda _j^2\lambda _k^2\left[ E\left( \xi _{j*}^2\xi _{k^*}^2\right) -\phi _{j^*k^*}^2\right] +\lambda _j^2\psi _{kk}+\lambda _k^2\psi _{jj}+\psi _{jj}\psi _{kk}. \end{aligned}$$
(32)

The CV in (7) is obtained using (32) and \(\mathrm{CV}_{jk}=\gamma _{jk}^{1/2}/(\sqrt{n}\sigma _{jk})\).

When \(j^*=k^*\) but \(j\ne k\), \(\sigma _{jk}=\lambda _j\lambda _k\), it follows from (30) that

$$\begin{aligned} \gamma _{jk}=\lambda _j^2\lambda _k^2\left[ E\left( \xi _{j^*}^4\right) -1\right] +\lambda _j^2\psi _{kk} +\lambda _k^2\psi _{jj}+\psi _{jj}\psi _{kk}. \end{aligned}$$
(33)

The CV in (8) directly follows from (33).

When \(j=k\), \(\sigma _{jj}=\lambda _j^2+\psi _{jj}\), it follows from (31) that

$$\begin{aligned} \gamma _{jj}=\lambda _j^4\left[ E\left( \xi _{j^*}^4\right) -1\right] +4\lambda _j^2\psi _{jj} +\left[ E\left( \varepsilon _j^4\right) -\psi _{jj}^2\right] . \end{aligned}$$
(34)

With (34), the result in (9) is obtained from \(\mathrm{CV}_{jj}=\gamma _{jj}^{1/2}/(\sqrt{n}\sigma _{jj})\).

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Yuan, KH., Bentler, P.M. Improving the convergence rate and speed of Fisher-scoring algorithm: ridge and anti-ridge methods in structural equation modeling. Ann Inst Stat Math 69, 571–597 (2017). https://doi.org/10.1007/s10463-016-0552-2

Download citation

  • Received:

  • Revised:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10463-016-0552-2

Keywords

Navigation