Behavior Research Methods

, Volume 51, Issue 1, pp 409–428 | Cite as

RMSEA, CFI, and TLI in structural equation modeling with ordered categorical data: The story they tell depends on the estimation methods

  • Yan XiaEmail author
  • Yanyun Yang
Brief Report


In structural equation modeling, application of the root mean square error of approximation (RMSEA), comparative fit index (CFI), and Tucker–Lewis index (TLI) highly relies on the conventional cutoff values developed under normal-theory maximum likelihood (ML) with continuous data. For ordered categorical data, unweighted least squares (ULS) and diagonally weighted least squares (DWLS) based on polychoric correlation matrices have been recommended in previous studies. Although no clear suggestions exist regarding the application of these fit indices when analyzing ordered categorical variables, practitioners are still tempted to adopt the conventional cutoff rules. The purpose of our research was to answer the question: Given a population polychoric correlation matrix and a hypothesized model, if ML results in a specific RMSEA value (e.g., .08), what is the RMSEA value when ULS or DWLS is applied? CFI and TLI were investigated in the same fashion. Both simulated and empirical polychoric correlation matrices with various degrees of model misspecification were employed to address the above question. The results showed that DWLS and ULS lead to smaller RMSEA and larger CFI and TLI values than does ML for all manipulated conditions, regardless of whether or not the indices are scaled. Applying the conventional cutoffs to DWLS and ULS, therefore, has a pronounced tendency not to discover model–data misfit. Discussions regarding the use of RMSEA, CFI, and TLI for ordered categorical data are given.


Structural equation modeling Ordered categorical data Diagonally weighted least squares Unweighted least squares Maximum likelihood Fit index 


  1. Asparouhov, T., & Muthén, B. (2010). Simple second order chi-square correction. Retrieved from
  2. Barrett, P. (2007). Structural equation modelling: Adjudging model fit. Personality and Individual Differences, 42, 815–824.CrossRefGoogle Scholar
  3. Beauducel, A., & Herzberg, P. Y. (2006). On the performance of maximum likelihood versus means and variance adjusted weighted least squares estimation in CFA. Structural Equation Modeling, 13, 186–203.CrossRefGoogle Scholar
  4. De Beer, L. T., Pienaar, J., & Rothmann, S. (2014). Job burnout’s relationship with sleep difficulties in the presence of control variables: A self-report study. South African Journal of Psychology, 44, 454–466.CrossRefGoogle Scholar
  5. Bentler, P. M. (1990). Comparative fit indexes in structural models. Psychological Bulletin, 107, 238–246. CrossRefGoogle Scholar
  6. Bentler, P. M. (2008). EQS structural equation modeling software. Encino, CA: Multivariate Software.Google Scholar
  7. Bentler, P. M., & Bonett, D. G. (1980). Significance tests and goodness of fit in the analysis of covariance structures. Psychological Bulletin, 88, 588–606. CrossRefGoogle Scholar
  8. Brosseau-Liard, P. E., & Savalei, V. (2014). Adjusting incremental fit indexes for nonnormality. Multivariate Behavioral Research, 49, 460–470.CrossRefGoogle Scholar
  9. Brosseau-Liard, P. E., Savalei, V., & Li, L. (2012). An investigation of the sample performance of two nonnormality corrections for RMSEA. Multivariate Behavioral Research, 47, 904–930.CrossRefGoogle Scholar
  10. Browne, M. W., & Cudeck, R. (1993). Alternative ways of assessing model fit. In K. A. Bollen & J. S. Long (Eds.), Testing structural equation models (pp. 136–162). Newbury Park, CA: Sage.Google Scholar
  11. Cudeck, R., & Browne, M. W. (1992). Constructing a covariance matrix that yields a specified minimizer and a specified minimum discrepancy function value. Psychometrika, 57, 357–369.CrossRefGoogle Scholar
  12. Currier, J. M., & Holland, J. M. (2014). Involvement in abusive violence among Vietnam veterans: Direct and indirect associations with substance use problems and suicidality. Psychological Trauma: Theory, Research, Practice, and Policy, 6, 73–82.CrossRefGoogle Scholar
  13. DiStefano, C., & Morgan, G. B. (2014). A comparison of diagonal weighted least squares robust estimation techniques for ordinal data. Structural Equation Modeling, 21, 425–438.CrossRefGoogle Scholar
  14. Fernandez, S., & Moldogaziev, T. (2013). Employee empowerment, employee attitudes, and performance: Testing a causal model. Public Administration Review, 73, 490–506.CrossRefGoogle Scholar
  15. Forero, C. G., Maydeu-Olivares, A., & Gallardo-Pujol, D. (2009). Factor analysis with ordinal indicators: A Monte Carlo study comparing DWLS and ULS estimation. Structural Equation Modeling, 16, 625–641.CrossRefGoogle Scholar
  16. Garrido, L. E., Abad, F. J., & Ponsoda, V. (2016). Are fit indexes really fit to estimate the number of factors with categorical variables? Some cautionary findings via Monte Carlo simulation. Psychological Methods, 21, 93–111.CrossRefGoogle Scholar
  17. Hu, L., & Bentler, P. M. (1999). Cutoff criteria for fit indexes in covariance structure analysis: Conventional criteria versus new alternatives. Structural Equation Modeling, 6, 1–55. CrossRefGoogle Scholar
  18. Iglesias, K., Burnand, B., & Peytremann-Bridevaux, I. (2014). PACIC Instrument: Disentangling dimensions using published validation models. International Journal for Quality in Health Care, 26, 250–260.CrossRefGoogle Scholar
  19. Johnson, D. R., & Creech, J. C. (1983). Ordinal measures in multiple indicator models: A simulation study of categorization error. American Sociological Review, 48, 398–407.CrossRefGoogle Scholar
  20. Jöreskog, K. G., & Sörbom, D. (1993). LISREL 8: Structural equation modeling with the SIMPLIS command language. Chicago, IL: Scientific Software International.Google Scholar
  21. Koziol, N. A. (2010). Evaluating measurement invariance with censored ordinal data: A Monte Carlo comparison of alternative model estimators and scales of measurement. Unpublished master’s thesis, University of Nebraska, Lincoln, NE.Google Scholar
  22. Lai, K., & Green, S. B. (2016). The problem with having two watches: Assessment of fit when RMSEA and CFI disagree. Multivariate Behavioral Research, 51 220–239.CrossRefGoogle Scholar
  23. Li, C. H. (2014). The performance of MLR, USLMV, and WLSMV estimation in structural regression models with ordinal variables (Doctoral dissertation). Michigan State University, East Lansing, MI.Google Scholar
  24. MacInnis, M. J., Lanting, S. C., Rupert, J. L., & Koehle, M. S. (2013). Is poor sleep quality at high altitude separate from acute mountain sickness? Factor structure and internal consistency of the Lake Louise Score Questionnaire. High Altitude Medicine and Biology, 14, 334–337.CrossRefGoogle Scholar
  25. Marsh, H. W., Hau, K. T., & Wen, Z. (2004). In search of golden rules: Comment on hypothesis-testing approaches to setting cutoff values for fit indexes and dangers in overgeneralizing Hu and Bentler’s (1999) findings. Structural Equation Modeling, 11, 320–341.CrossRefGoogle Scholar
  26. Martínez-Rodríguez, S., Iraurgi, I., Gómez-Marroquin, I., Carrasco, M., Ortiz-Marqués, N., & Stevens, A. B. (2016). Psychometric properties of the Leisure Time Satisfaction Scale in family caregivers. Psicothema, 28, 207–213.Google Scholar
  27. Maydeu-Olivares, A., Fairchild, A. J., & Hall, A. G. (2017). Goodness of fit in item factor analysis: Effect of the number of response alternatives. Structural Equation Modeling, 24, 495–505.CrossRefGoogle Scholar
  28. McIntosh, C. N. (2007). Rethinking fit assessment in structural equation modelling: A commentary and elaboration on Barrett (2007). Personality and Individual Differences, 42, 859–867.CrossRefGoogle Scholar
  29. Muthén, B. O., du Toit, S. H. C., & Spisic, D. (1997). Robust inference using weighted least squares and quadratic estimating equations in latent variable modeling with categorical and continuous outcomes (Unpublished technical report). Retrieved from
  30. Muthén, L. K., & Muthén, B. O. (2015). Mplus user’s guide (7th ed.). Los Angeles, CA: Muthén & Muthén.Google Scholar
  31. Nguyen, T. Q., Poteat, T., Bandeen-Roche, K., German, D., Nguyen, Y. H., Vu, L. K., … Knowlton, A. R. (2016). The internalized homophobia scale for Vietnamese sexual minority women: Conceptualization, factor structure, reliability, and associations with hypothesized correlates. Archives of Sexual Behavior, 45, 1329–1346.CrossRefGoogle Scholar
  32. Nye, C. D., & Drasgow, F. (2011). Assessing goodness of fit: Simple rules of thumb simply do not work. Organizational Research Methods, 14, 548–570.CrossRefGoogle Scholar
  33. Olsson, U. (1979). Maximum likelihood estimation of the polychoric correlation coefficient. Psychometrika, 44, 443–460.CrossRefGoogle Scholar
  34. Pearson, K. (1904). Mathematical contributions to the theory of evolution XIII: On the theory of contingency and its relation to association and normal correlation (Drapers’ Co. Research Memoirs, Biometric Series, no. 1). Cambridge, UK: Cambridge University Press.Google Scholar
  35. Pettersen, C., Nunes, K. L., & Cortoni, F. (2016). Does the factor structure of the aggression questionnaire hold for sexual offenders? Criminal Justice and Behavior, 43, 811–829.CrossRefGoogle Scholar
  36. Rhemtulla, M., Brosseau-Liard, P. E., & Savalei, V. (2012). When can categorical variables be treated as continuous? A comparison of robust continuous and categorical SEM estimation methods under suboptimal conditions. Psychological Methods, 17, 354–373.CrossRefGoogle Scholar
  37. SAS Institute. (2015). SAS/STAT 14.1 user’s guide. Cary, NC: Author.Google Scholar
  38. Sass, D. A., Schmitt, T. A., & Marsh, H. W. (2014). Evaluating model fit with ordered categorical data with a measurement invariance framework: A comparison of estimators. Structural Equation Modeling, 21, 167–180.CrossRefGoogle Scholar
  39. Satorra, A., & Bentler, P. M. (1994). Corrections to test statistics and standard errors in covariance structure analysis. In A. von Eye & C. C. Clogg (Eds.), Latent variables analysis: Applications for developmental research (pp. 399–419). Thousand Oaks, CA: Sage.Google Scholar
  40. Savalei, V. (2014). Understanding robust corrections in structural equation modeling. Structural Equation Modeling, 21, 149–160.CrossRefGoogle Scholar
  41. Savalei, V., & Rhemtulla, M. (2013). The performance of robust test statistics with categorical data. British Journal of Mathematical and Statistical Psychology, 66, 201–223.CrossRefGoogle Scholar
  42. Stander, F. W., Mostert, K., & de Beer, L. T. (2014). Organisational and individual strengths use as predictors of engagement and productivity. Journal of Psychology in Africa, 24, 403–409.CrossRefGoogle Scholar
  43. Steiger, J. H. (1990). Structural model evaluation and modification: An interval estimation approach. Multivariate Behavioral Research, 25, 173–180.CrossRefGoogle Scholar
  44. Steiger, J. H., & Lind, J. C. (1980). Statistically based tests for the number of common factors. Paper presented at the Annual Meeting of the Psychometric Society, Iowa City, IA.Google Scholar
  45. Tallis, G. M. (1962). The maximum likelihood estimation of correlation from contingency tables. Biometrics, 18, 342–353.CrossRefGoogle Scholar
  46. Tucker, L. R., & Lewis, C. (1973). A reliability coefficient for maximum likelihood factor analysis. Psychometrika, 38, 1–10.CrossRefGoogle Scholar
  47. Xia, Y., Yung, Y. F., & Zhang, W. (2016). Evaluating the selection of normal-theory weight matrices in the Satorra–Bentler correction of chi-square and standard errors. Structural Equation Modeling, 23, 585–594.CrossRefGoogle Scholar
  48. Yang, Y., & Xia, Y. (2015). On the number of factors in exploratory factor analysis for ordered categorical data. Behavior Research Methods, 47, 756–772. CrossRefGoogle Scholar
  49. Yang-Wallentin, F., Jöreskog, K. G., & Luo, H. (2010). Confirmatory factor analysis of ordinal variables with misspecified models. Structural Equation Modeling, 17, 392–423.CrossRefGoogle Scholar
  50. Yu, C.-Y. (2002). Evaluating cutoff criteria of model-fit indexes for latent variable models with binary and continuous outcomes (Doctoral dissertation). University of California, Los Angeles, CA.Google Scholar
  51. Yuan, K. H., & Marshall, L. L. (2004). A new measure of misfit for covariance structure models. Behaviormetrika, 31, 67–90.CrossRefGoogle Scholar
  52. Zhang, W. (2008). A comparison of four estimators of a population measure of model fit in covariance structure analysis. Structural Equation Modeling, 15, 301–326.CrossRefGoogle Scholar

Copyright information

© Psychonomic Society, Inc. 2018

Authors and Affiliations

  1. 1.T. Denny Sanford School of Social and Family DynamicsArizona State UniversityTempeUSA
  2. 2.College of EducationFlorida State UniversityTallahasseeUSA

Personalised recommendations