Advertisement

Psychometrika

, Volume 82, Issue 3, pp 533–558 | Cite as

Assessing the Size of Model Misfit in Structural Equation Models

  • Alberto Maydeu-OlivaresEmail author
Article

Abstract

When a statistically significant mean difference is found, the magnitude of the difference is judged qualitatively using an effect size such as Cohen’s d. In contrast, in a structural equation model (SEM), the result of the statistical test of model fit is often disregarded if significant, and inferences are drawn using “close” models retained based on point estimates of sample statistics (goodness-of-fit indices). However, when a SEM cannot be retained using a test of exact fit, all substantive inferences drawn from it are suspect. It is therefore important to determine the size of the model misfit. Standardized residual covariances and residual correlations provide standardized effect sizes of the misfit of SEM models. They can be summarized using the Standardized Root Mean Squared Residual (SRMSR) and the Correlation Root Mean Squared Residual (CRMSR) which can be used as overall effect sizes of the misfit. Statistical theory is provided that allows the construction of confidence intervals and tests of close fit based on the SRMSR and CRMSR. It is hoped that the use of standardized effect sizes of misfit will help reconcile current practices in SEM and elsewhere in statistics.

Keywords

goodness-of-fit RMSEA effect size 

References

  1. Alden, D. L., Steenkamp, J. B. E. M., & Batra, R. (2006). Consumer attitudes toward marketplace globalization: Structure, antecedents and consequences. International Journal of Research in Marketing, 23(3), 227–239. doi: 10.1016/j.ijresmar.2006.01.010.CrossRefGoogle Scholar
  2. Barrett, P. (2007). Structural equation modelling: Adjudging model fit. Personality and Individual Differences, 42(5), 815–824. doi: 10.1016/j.paid.2006.09.018.CrossRefGoogle Scholar
  3. Bentler, P. M. (1990). Comparative fit indexes in structural models. Psychological Bulletin, 107(2), 238–246. doi: 10.1037/0033-2909.107.2.238.CrossRefPubMedGoogle Scholar
  4. Bentler, P. M. (1995). EQS 5 [Computer Program]. Encino, CA: Multivariate Software Inc.Google Scholar
  5. Bollen, K. A. (1989). Structural equations with latent variables. New York: Wiley.CrossRefGoogle Scholar
  6. Bollen, K. A., & Arminger, G. (1991). Observational residuals in factor analysis and structural equation models. Sociological Methodology, 21, 235. doi: 10.2307/270937.CrossRefGoogle Scholar
  7. Bollen, K. A., & Long, J. S. (1993). Testing structural equation models. Newbury Park, CA: Sage.Google Scholar
  8. Browne, M. W. (2000). Cross-validation methods. Journal of Mathematical Psychology, 44(1), 108–132. doi: 10.1006/jmps.1999.1279.CrossRefPubMedGoogle Scholar
  9. Browne, M. W. (1982). Covariance structures. In D. M. Hawkins (Ed.), Topics in applied multivariate analysis (pp. 72–141). Cambridge: Cambridge University Press.CrossRefGoogle Scholar
  10. Browne, M. W. (1984). Asymptotically distribution-free methods for the analysis of covariance structures. British Journal of Mathematical and Statistical Psychology, 37(1), 62–83. doi: 10.1111/j.2044-8317.1984.tb00789.x.CrossRefPubMedGoogle Scholar
  11. Browne, M. W., & Arminger, G. (1995). Specification and estimation of mean- and covariance-structure models. In G. Arminger, C. C. Clogg, & M. E. Sobel (Eds.), Handbook of statistical modeling for the social and behavioral sciences (pp. 185–249). New York: Plenum.CrossRefGoogle Scholar
  12. Browne, M. W., & Cudeck, R. (1989). Single sample cross-validation indices for covariance structures. Multivariate Behavioral Research, 24(4), 445–455. doi: 10.1207/s15327906mbr2404_4.CrossRefPubMedGoogle Scholar
  13. Browne, M. W., & Cudeck, R. (1993). Alternative ways of assessing model fit. In K. A. Bollen & J. S. Long (Eds.), Testing structural equation models (pp. 136–162). Newbury Park, CA: Sage.Google Scholar
  14. Chen, F., Curran, P. J., Bollen, K. A., Kirby, J., & Paxton, P. (2008). An empirical evaluation of the use of fixed cutoff points in rmsea test statistic in structural equation models. Sociological Methods & Research, 36(4), 462–494. doi: 10.1177/0049124108314720.CrossRefGoogle Scholar
  15. Coffman, D., & Millsap, R. (2006). Evaluating latent growth curve models using individual fit statistics. Structural Equation Modeling, 13(1), 37–41. doi: 10.1207/s15328007sem1301_1.CrossRefGoogle Scholar
  16. Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Hillsdale, NJ: Lawrence Erlbaum.Google Scholar
  17. Cohen, J., Cohen, P., West, S. G., & Aiken, L. S. (2003). Applied multiple regression/correlation analysis for behavioral sciences (3rd ed.). Mahwah, NJ: Erlbaum.Google Scholar
  18. Edwards, J. R., & Lambert, L. S. (2007). Methods for integrating moderation and mediation: A general analytical framework using moderated path analysis. Psychological Methods, 12(1), 1–22. doi: 10.1037/1082-989X.12.1.1.CrossRefPubMedGoogle Scholar
  19. Hildreth, L. (2013). Residual analysis for structural equation modeling. Ph.D. dissertation, Statistics Department, Iowa State University.Google Scholar
  20. Hu, L., & Bentler, P. M. (1999). Cutoff criteria for fit indexes in covariance structure analysis: Conventional criteria versus new alternatives. Structural Equation Modeling, 6(1), 1–55. doi: 10.1080/10705519909540118.CrossRefGoogle Scholar
  21. Jöreskog, K. G. (1993). Testing structural equation models. In K. A. Bollen & J. S. Long (Eds.), Testing structural equation models (pp. 294–316). Newbury Park, CA: Sage.Google Scholar
  22. Jöreskog, K. G., & Sörbom, D. (1981). Analysis of linear structural relationships by maximum likelihood and least squares methods. Research report 81-8, Uppsala, Sweden.Google Scholar
  23. Jöreskog, K. G., & Sörbom, D. (1988). LISREL 7. A guide to the program and applications (2nd ed.). Chicago, IL: International Education Services.Google Scholar
  24. Kenny, D. A., & McCoach, D. B. (2003). Effect of the number of variables on measures of fit in structural equation modeling. Structural Equation Modeling: A Multidisciplinary Journal, 10(3), 333–351. doi: 10.1207/S15328007SEM1003_1.CrossRefGoogle Scholar
  25. Lee, S. Y., & Jennrich, R. I. (1979). A study of algorithms for covariance structure analysis with specific comparisons using factor analysis. Psychometrika, 44(1), 99–113. doi: 10.1007/BF02293789.CrossRefGoogle Scholar
  26. MacCallum, R. C., Wegener, D. T., Uchino, B. N., & Fabrigar, L. R. (1993). The problem of equivalent models in applications of covariance structure analysis. Psychological Bulletin, 114(1), 185–199. doi: 10.1037/0033-2909.114.1.185.CrossRefPubMedGoogle Scholar
  27. Marsh, H., Hau, K., & Grayson, D. (2005). Goodness of fit in structural equation models. In A. Maydeu-Olivares & J. J. Mcardle (Eds.), Contemporary psychometrics: A Festschrift for Roderick P. McDonald (pp. 275–340). Mahwah, NJ: Erlbaum.Google Scholar
  28. Marsh, H. W., Hau, K., & Wen, Z. (2004). In search of golden rules: Comment on hypothesis-testing approaches to setting cutoff values for fit indexes and dangers in overgeneralizing Hu and Bentler’s (1999) findings. Structural Equation Modeling: A Multidisciplinary Journal, 11(3), 320–341. doi: 10.1207/s15328007sem1103_2.CrossRefGoogle Scholar
  29. Maydeu-Olivares, A. (2006). Limited information estimation and testing of discretized multivariate normal structural models. Psychometrika, 71(1), 57–77. doi: 10.1007/s11336-005-0773-4.CrossRefGoogle Scholar
  30. Maydeu-Olivares, A. (2015). Evaluating fit in IRT models. In S. P. Reise & D. A. Revicki (Eds.), Handbook of item response theory modeling: Applications to typical performance assessment (pp. 111–127). New York: Routledge.Google Scholar
  31. Maydeu-Olivares, A., & Shi, D. (2017). Effect sizes of model misfit in structural equation models: Standardized residual covariances and residual correlations. Methodology: European Journal of Research Methods for the Behavioral and Social Sciences.Google Scholar
  32. McDonald, R. P., & Ho, M.-H. R. (2002). Principles and practice in reporting structural equation analyses. Psychological Methods, 7(1), 64–82. doi: 10.1037/1082-989X.7.1.64.CrossRefPubMedGoogle Scholar
  33. Muthén, B. (1984). A general structural equation model with dichotomous, ordered categorical, and continuous latent variable indicators. Psychometrika, 49(1), 115–132. doi: 10.1007/BF02294210.
  34. Ogasawara, H. (2001). Standard errors of fit indices using residuals in structural equation modeling. Psychometrika, 66(3), 421–436. doi: 10.1007/BF02294443.CrossRefGoogle Scholar
  35. Raykov, T. (2000). On sensitivity of structural equation modeling to latent relation misspecifications. Structural Equation Modeling, 7(4), 596–607. doi: 10.1207/S15328007SEM0704_4.CrossRefGoogle Scholar
  36. Reise, S. P., Scheines, R., Widaman, K. F., & Haviland, M. G. (2012). Multidimensionality and structural coefficient bias in structural equation modeling: A bifactor perspective. Educational and Psychological Measurement, 73(1), 5–26. doi: 10.1177/0013164412449831.CrossRefGoogle Scholar
  37. Rosseel, Y. (2012). lavaan: An R package for structural equation modeling. Journal of Statistical Software, 48(2), 1–36. doi: 10.18637/jss.v048.i02.CrossRefGoogle Scholar
  38. Saris, W. E., Satorra, A., & van der Veld, W. M. (2009). Testing structural equation models or detection of misspecifications? Structural Equation Modeling: A Multidisciplinary Journal, 16(4), 561–582. doi: 10.1080/10705510903203433.CrossRefGoogle Scholar
  39. Satorra, A. (1989). Alternative test criteria in covariance structure analysis: A unified approach. Psychometrika, 54(1), 131–151. doi: 10.1007/BF02294453.CrossRefGoogle Scholar
  40. Satorra, A. (2015). A comment on a paper by H. Wu and M. W. Browne (2014). Psychometrika, 80(3), 613–618. doi: 10.1007/s11336-015-9455-z.CrossRefPubMedGoogle Scholar
  41. Satorra, A., & Bentler, P. M. (1994). Corrections to test statistics and standard errors in covariance structure analysis. In A. Von Eye & C. C. Clogg (Eds.), Latent variable analysis. Applications for developmental research (pp. 399–419). Thousand Oaks, CA: Sage.Google Scholar
  42. Savalei, V. (2012). The relationship between root mean square error of approximation and model misspecification in confirmatory factor analysis models. Educational and Psychological Measurement, 72(6), 910–932. doi: 10.1177/0013164412452564.CrossRefGoogle Scholar
  43. Schmidt, F., & Hunter, J. (1997). Eight common but false objections to the discontinuation of significance testing in the analysis of research data. In L. L. Harlow, S. Mulaik, & J. H. Steiger (Eds.), What if there were no significance tests (pp. 37–64). Mahwah, NJ: Lawrence Erlbaum.Google Scholar
  44. Schott, J. R. (1997). Matrix analysis for statistics. New York: Wiley.Google Scholar
  45. Steiger, J. H. (1989). EzPATH: A supplementary module for SYSTAT and SYGRAPH [Computer program manual]. Evanston, IL: Systat Inc.Google Scholar
  46. Steiger, J. H. (1990). Structural model evaluation and modification: An interval estimation approach. Multivariate Behavioral Research, 25(2), 173–180. doi: 10.1207/s15327906mbr2502_4.CrossRefPubMedGoogle Scholar
  47. Steiger, J. H. (2000). Point estimation, hypothesis testing, and interval estimation using the RMSEA: Some comments and a reply to Hayduck and Glaser. Structural Equation Modeling, 7(2), 149–162. doi: 10.1207/S15328007SEM0702_1.CrossRefGoogle Scholar
  48. Steiger, J. H., & Fouladi, R. (1997). Noncentrality interval estimation and the evaluation of statistical models. In L. L. Harlow, S. A. Mulaik, & J. H. Steiger (Eds.), What if there were no significance tests (pp. 221–257). Mahwah, NJ: Erlbaum.Google Scholar
  49. Steiger, J. H., & Lind, J. C. (1980). Statistically-based tests for the number of common factors. Paper presented at the Annual Meeting of the Psychometric Society, Iowa.Google Scholar
  50. Tanaka, J. S., & Huba, G. J. (1989). A general coefficient of determination for covariance structure models under arbitrary GLS estimation. British Journal of Mathematical and Statistical Psychology, 42(2), 233–239. doi: 10.1111/j.2044-8317.1989.tb00912.x.CrossRefGoogle Scholar
  51. Wu, H., & Browne, M. W. (2015). Quantifying adventitious error in a covariance structure as a random effect. Psychometrika, 80(3), 571–600. doi: 10.1007/s11336-015-9451-3.CrossRefPubMedPubMedCentralGoogle Scholar
  52. Yuan, K.-H., & Bentler, P. M. (1997). Mean and covariance structure analysis: Theoretical and practical improvements. Journal of the American Statistical Association, 92, 767–774. doi: 10.1080/01621459.1997.10474029.CrossRefGoogle Scholar
  53. Yuan, K.-H., & Bentler, P. M. (1998). Normal theory based test statistics in structural equation modelling. The British Journal of Mathematical and Statistical Psychology, 51(2), 289–309. doi: 10.1111/j.2044-8317.1998.tb00682.x.CrossRefPubMedGoogle Scholar
  54. Yuan, K.-H., & Bentler, P. M. (1999). \(F\) tests for mean and covariance structure analysis. Journal of Educational and Behavioral Statistics, 24(3), 225–243. doi: 10.3102/10769986024003225.CrossRefGoogle Scholar
  55. Yuan, K.-H., & Hayashi, K. (2010). Fitting data to model: Structural equation modeling diagnosis using two scatter plots. Psychological Methods, 15(4), 335–351. doi: 10.1037/a0020140.CrossRefPubMedGoogle Scholar
  56. Yuan, K.-H., Marshall, L. L., & Bentler, P. M. (2003). Assessing the effect of model misspecifications on parameter estimates in structural equation models. Sociological Methodology, 33(1), 241–265. doi: 10.1111/j.0081-1750.2003.00132.x.CrossRefGoogle Scholar

Copyright information

© The Psychometric Society 2017

Authors and Affiliations

  1. 1.University of BarcelonaBarcelonaSpain
  2. 2.Department of PsychologyUniversity of South CarolinaColumbiaUSA

Personalised recommendations