Skip to main content
SpringerLink
Log in

Flexible cutoff values for fit indices in the evaluation of structural equation models

  • Methodological Paper
  • Published:
Journal of the Academy of Marketing Science Aims and scope Submit manuscript

Abstract

Researchers often struggle when applying ‘golden rules of thumb’ to evaluate structural equation models. This paper questions the notion of universal thresholds and calls for adjusted orientation points that account for sample size, factor loadings, the number of latent variables and indicators, as well as data (non-)normality. This research explores the need for flexible cutoffs and their accuracy in single- and two-index strategies. Study 1 reveals that many indices are biased; thus, rigid cutoffs can become imprecise. Flexible cutoff values are shown to compensate for the unique distorting patterns and prove to be particularly beneficial for moderate misspecification. Study 2 sheds further light on this ‘gray’ area of misspecification and disentangles the different sources of misspecification. Study 3 finally investigates the performance of flexible cutoffs for non-normal data. Having substantiated higher performance for flexible reference values, this paper provides to managers an easy-to-use tool that facilitates the determination of adequate cutoffs.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9

Similar content being viewed by others

Notes

  1. The definitions of Type I and Type II errors are based on Hu and Bentler (1999, p. 5) and Marsh et al. (2004, pp. 335-336) (see the Web Appendix for a more detailed explanation).

  2. Note that Hu and Bentler’s (1999) recommendations are often ‘inappropriately promoted’ (Marsh et al. 2004, p. 322) in textbooks or guidelines, citing single-index cutoffs while clearly promoting two-index cutoffs.

  3. Pairwise manipulation is necessary to ensure model identification and comparability (i.e., positive and constant degrees of freedom across all combinations).

  4. For example, for two five-indicator constructs, the initial (a1-a5 on A) and final specification (b1-b5 on A) are identical, forming the level MM = 0, because the meaning of the two constructs is reversed. For the same reason, the second (b1, a2-a5) and fourth (b1-b4, a5) specifications are equal (MM = 1), and only the third (a1-a3, b4-b5) specification entails the highest possible degree of misspecification (MM = 2).

  5. It should be noted that robust and scaled versions of the χ2 statistic and χ2 based fit indices are readily available in modern statistical software (e.g., LISREL, lavaan). Users of this software should check whether these corrections are automatically applied or can be applied when estimating fit indices (if non-normality may be an issue).

References

  • Anderson, J. C., & Gerbing, D. W. (1984). The effect of sampling error on convergence, improper solutions, and goodness-of-fit indices for maximum likelihood confirmatory factor analysis. Psychometrika, 49(2), 155–173.

    Article  Google Scholar 

  • Babakus, E., & Boller, G. W. (1992). An empirical assessment of the SERVQUAL scale. Journal of Business Research, 24(3), 253–268.

    Article  Google Scholar 

  • Bagozzi, R. P., & Yi, Y. (1988). On the evaluation of structural equation models. Journal of the Academy of Marketing Science, 16(1), 74–94.

    Article  Google Scholar 

  • Bagozzi, R. P., & Yi, Y. (1989). On the use of structural equation models in experimental designs. Journal of Marketing Research, 26(3), 271–284.

    Article  Google Scholar 

  • Bandalos, D. L. (2002). The effects of item parceling on goodness-of-fit and parameter estimate bias in structural equation modeling. Structural Equation Modeling, 9(1), 78–102.

    Article  Google Scholar 

  • Bearden, W. O., Sharma, S., & Teel, J. E. (1982). Sample size effects on chi square and other statistics used in evaluating causal models. Journal of Marketing Research, 19(4), 425–430.

    Article  Google Scholar 

  • Beauducel, A., & Wittmann, W. W. (2005). Simulation study on fit indexes in CFA based on data with slightly distorted simple structure. Structural Equation Modeling, 12(1), 41–75.

    Article  Google Scholar 

  • Bentler, P. M. (1990). Comparative fit indexes in structural models. Psychological Bulletin, 107(2), 238–246.

    Article  Google Scholar 

  • Bentler, P. M. (1995). EQS structural equations program manual. Encino: Multivariate Software.

    Google Scholar 

  • Bentler, P. M. (2007). On tests and indices for evaluating structural models. Personality and Individual Differences, 42(5), 825–829.

    Article  Google Scholar 

  • Bentler, P. M., & Bonett, D. G. (1980). Significance tests and goodness of fit in the analysis of covariance structures. Psychological Bulletin, 88(3), 588–606.

    Article  Google Scholar 

  • Bollen, K. A., Harden, J. J., Ray, S., & Zavisca, J. (2014). BIC and alternative Bayesian information criteria in the selection of structural equation models. Structural Equation Modeling, 21(1), 1–19.

    Article  Google Scholar 

  • Bone, P. F., Sharma, S., & Shimp, T. A. (1989). A bootstrap procedure for evaluating goodness-of-fit indices of structural equation and confirmatory factor models. Journal of Marketing Research, 26(1), 105–111.

    Article  Google Scholar 

  • Boomsma, A. (1985). Nonconvergence, improper solutions, and starting values in LISREL maximum likelihood estimation. Psychometrika, 50(2), 229–242.

    Article  Google Scholar 

  • Breivik, E., & Olsson, U. H. (2001). Adding variables to improve fit: The effect of model size on fit assessment in LISREL. In S. DuToit, R. Cudeck, & D. Sorbom (Eds.), Structural equation modeling: Present and future (pp. 169–194). Lincolnwood: Scientific Software International, Inc..

    Google Scholar 

  • Browne, M. W., & Cudeck, R. (1989). Single sample cross-validation indices for covariance structures. Multivariate Behavioral Research, 24(4), 445–455.

    Article  Google Scholar 

  • Chen, F., Curran, P. J., Bollen, K. A., Kirby, J., & Paxton, P. (2008). An empirical evaluation of the use of fixed cutoff points in RMSEA test statistic in structural equation models. Sociological Methods & Research, 36(4), 462–494.

    Article  Google Scholar 

  • Cheung, G. W., & Rensvold, R. B. (2001). The effects of model parsimony and sampling error on the fit of structural equation models. Organizational Research Methods, 4(3), 236–264.

    Article  Google Scholar 

  • Cohen, J. (1988). Statistical power analysis for the behavioral sciences. Hillsdale: Lawrence Erlbaum.

    Google Scholar 

  • Curran, P. J., West, S. G., & Finch, J. F. (1996). The robustness of test statistics to nonnormality and specification error in confirmatory factor analysis. Psychological Methods, 1(1), 16–29.

    Article  Google Scholar 

  • Curran, P. J., Bollen, K. A., Paxton, P., Kirby, J., & Chen, F. (2002). The noncentral chi-square distribution in misspecified structural equation models: Finite sample results from a Monte Carlo simulation. Multivariate Behavioral Research, 37(1), 1–36.

    Article  Google Scholar 

  • Enders, C. K. (2001). The impact of nonnormality on full information maximum-likelihood estimation for structural equation models with missing data. Psychological Methods, 6(4), 352–370.

    Article  Google Scholar 

  • Fan, X., & Sivo, S. A. (2005). Sensitivity of fit indexes to misspecified structural or measurement model components: Rationale of two-index strategy revisited. Structural Equation Modeling, 12(3), 343–367.

    Article  Google Scholar 

  • Fan, X., & Sivo, S. A. (2007). Sensitivity of fit indices to model misspecification and model types. Multivariate Behavioral Research, 42(3), 509–529.

    Article  Google Scholar 

  • Fouladi, R. T. (2000). Performance of modified test statistics in covariance and correlation structure analysis under conditions of multivariate nonnormality. Structural Equation Modeling, 7(3), 356–410.

    Article  Google Scholar 

  • Gagne, P., & Hancock, G. R. (2006). Measurement model quality, sample size, and solution propriety in confirmatory factor models. Multivariate Behavioral Research, 41(1), 65–83.

    Article  Google Scholar 

  • Gupta, S., Hanssens, D., Hauser, J. R., Lehmann, D., & Schmitt, B. (2014). Introduction to theory and practice in marketing conference special section of marketing science. Marketing Science, 33(1), 1–5.

    Article  Google Scholar 

  • Hau, K. T., & Marsh, H. W. (2004). The use of item parcels in structural equation modelling: Non-normal data and small sample sizes. British Journal of Mathematical and Statistical Psychology, 57(2), 327–351.

    Article  Google Scholar 

  • Hayduk, L., Cummings, G., Boadu, K., Pazderka-Robinson, H., & Boulianne, S. (2007). Testing! Testing! One, two, three–testing the theory in structural equation models! Personality and Individual Differences, 42(5), 841–850.

    Article  Google Scholar 

  • Heene, M., Hilbert, S., Draxler, C., Ziegler, M., & Bühner, M. (2011). Masking misfit in confirmatory factor analysis by increasing unique variances: A cautionary note on the usefulness of cutoff values of fit indices. Psychological Methods, 16(3), 319–336.

    Article  Google Scholar 

  • Herzog, W., Boomsma, A., & Reinecke, S. (2007). The model-size effect on traditional and modified tests of covariance structures. Structural Equation Modeling, 14(3), 361–390.

    Article  Google Scholar 

  • Hu, L., & Bentler, P. M. (1998). Fit indices in covariance structure modeling: Sensitivity to underparameterized model misspecification. Psychological Methods, 3(4), 424–453.

    Article  Google Scholar 

  • Hu, L., & Bentler, P. M. (1999). Cutoff criteria for fit indexes in covariance structure analysis: Conventional criteria versus new alternatives. Structural Equation Modeling, 6(1), 1–55.

    Article  Google Scholar 

  • Hulland, J., Baumgartner, H., & Smith, K. M. (2017). Marketing survey research best practices: Evidence and recommendations from a review of JAMS articles. Journal of the Academy of Marketing Science, 46(1), 92–108.

    Article  Google Scholar 

  • Hyndman, R. J., & Fan, Y. (1996). Sample quantiles in statistical packages. The American Statistician, 50(4), 361–365.

    Google Scholar 

  • Jackson, D. L. (2007). The effect of the number of observations per parameter in misspecified confirmatory factor analytic models. Structural Equation Modeling, 14(1), 48–76.

    Article  Google Scholar 

  • Jackson, D. L., Gillaspy, J. A. J., & Purc-Stephenson, R. (2009). Reporting practices in confirmatory factor analysis: An overview and some recommendations. Psychological Methods, 14(1), 6–23.

    Article  Google Scholar 

  • Jöreskog, K. G., & Sörbom, D. (1981). LISREL V: Analysis of linear structural relationships by maximum likelihood and least squares methods. University of Uppsala, Department of Statistics.

  • Jöreskog, K. G., & Sörbom, D. (1982). Recent developments in structural equation modeling. Journal of Marketing Research, 19(4), 404–416.

    Article  Google Scholar 

  • Kenny, D. A., & McCoach, D. B. (2003). Effect of the number of variables on measures of fit in structural equation modeling. Structural Equation Modeling, 10(3), 333–351.

    Article  Google Scholar 

  • Kenny, D. A., Kaniskan, B., & McCoach, D. B. (2015). The performance of RMSEA in models with small degrees of freedom. Sociological Methods & Research, 44(3), 486–507.

    Article  Google Scholar 

  • Kumar, V., Sharma, A., & Gupta, S. (2017). Accessing the influence of strategic marketing research on generating impact: Moderating roles of models, journals, and estimation approaches. Journal of the Academy of Marketing Science, 45(2), 164–185.

    Article  Google Scholar 

  • La Du, T. J., & Tanaka, J. S. (1989). Influence of sample size, estimation method, and model specification on goodness-of-fit assessments in structural equation models. Journal of Applied Psychology, 74(4), 625–635.

    Article  Google Scholar 

  • Lehmann, D. R. (2014). Introduction to the special issue on theory and practice in marketing. Journal of Marketing Research, 51(6), 645–646.

    Article  Google Scholar 

  • Lehmann, D. R., McAlister, L., & Staelin, R. (2013). Sophistication in research in marketing. Journal of Marketing, 75(4), 155–165.

    Article  Google Scholar 

  • Lombardi, L., & Pastore, M. (2012). Sensitivity of fit indices to fake perturbation of ordinal data: A sample by replacement approach. Multivariate Behavioral Research, 47(4), 519–546.

    Article  Google Scholar 

  • MacCallum, R. C., Widaman, K. F., Preacher, K. J., & Hong, S. (2001). Sample size in factor analysis: The role of model error. Multivariate Behavioral Research, 36(4), 611–637.

    Article  Google Scholar 

  • Marsh, H. W., Balla, J. R., & McDonald, R. P. (1988). Goodness-of-fit indexes in confirmatory factor analysis: The effect of sample size. Psychological Bulletin, 103(3), 391–410.

    Article  Google Scholar 

  • Marsh, H. W., Hau, K.-T., Balla, J. R., & Grayson, D. (1998). Is more ever too much? The number of indicators per factor in confirmatory factor analysis. Multivariate Behavioral Research, 33(2), 181–220.

    Article  Google Scholar 

  • Marsh, H. W., Hau, K.-T., & Wen, Z. (2004). In search of golden rules: Comment on hypothesis-testing approaches to setting cutoff values for fit indexes and dangers in overgeneralizing Hu and Bentler's (1999) findings. Structural Equation Modeling, 11(3), 320–341.

    Article  Google Scholar 

  • Miao, C. F., & Wang, G. (2017). Effects of work–family interface conflicts on salesperson behaviors: A double-edged sword. Journal of the Academy of Marketing Science, 45(5), 762–783.

    Article  Google Scholar 

  • Mishra, S., Umesh, U., & Stem Jr., D. E. (1993). Antecedents of the attraction effect: An information-processing approach. Journal of Marketing Research, 30(3), 331–349.

    Article  Google Scholar 

  • Moshagen, M. (2012). The model size effect in SEM: Inflated goodness-of-fit statistics are due to the size of the covariance matrix. Structural Equation Modeling, 19(1), 86–98.

    Article  Google Scholar 

  • Muthén, B., & Asparouhov, T. (2012). Bayesian structural equation modeling: A more flexible representation of substantive theory. Psychological Methods, 17(3), 313–335.

    Article  Google Scholar 

  • Muthén, B., & Kaplan, D. (1985). A comparison of some methodologies for the factor analysis of non-normal Likert variables. British Journal of Mathematical and Statistical Psychology, 38(2), 171–189.

    Article  Google Scholar 

  • Nasser, F., & Wisenbaker, J. (2003). A Monte Carlo study investigating the impact of item parceling on measures of fit in confirmatory factor analysis. Educational and Psychological Measurement, 63(5), 729–757.

    Article  Google Scholar 

  • Nye, C. D., & Drasgow, F. (2011). Assessing goodness of fit: Simple rules of thumb simply do not work. Organizational Research Methods, 14(3), 548–570.

    Article  Google Scholar 

  • Olsson, U. H., Foss, T., Troye, S. V., & Howell, R. D. (2000). The performance of ML, GLS, and WLS estimation in structural equation modeling under conditions of misspecification and nonnormality. Structural Equation Modeling, 7(4), 557–595.

    Article  Google Scholar 

  • Parasuraman, A., Zeithaml, V. A., & Berry, L. L. (1988). Servqual: A multiple-item scale for measuring consumer perc. Journal of Retailing, 64(1), 12–40.

    Google Scholar 

  • Podsakoff, P. M., MacKenzie, S. B., Lee, J.-Y., & Podsakoff, N. P. (2003). Common method biases in behavioral research: A critical review of the literature and recommended remedies. Journal of Applied Psychology, 88(5), 879–903.

    Article  Google Scholar 

  • Satorra, A., & Bentler, P. M. (2001). A scaled difference chi-square test statistic for moment structure analysis. Psychometrika, 66(4), 507–514.

    Article  Google Scholar 

  • Savalei, V. (2012). The relationship between root mean square error of approximation and model misspecification in confirmatory factor analysis models. Educational and Psychological Measurement, 72(6), 910–932.

    Article  Google Scholar 

  • Schermelleh-Engel, K., Moosbrugger, H., & Müller, H. (2003). Evaluating the fit of structural equation models: Tests of significance and descriptive goodness-of-fit measures. Methods of Psychological Research Online, 8(2), 23–74.

    Google Scholar 

  • Sedlmeier, P., & Gigerenzer, G. (1989). Do studies of statistical power have an effect on the power of studies? Psychological Bulletin, 105(2), 309–316.

    Article  Google Scholar 

  • Sharma, S., Mukherjee, S., Kumar, A., & Dillon, W. R. (2005). A simulation study to investigate the use of cutoff values for assessing model fit in covariance structure models. Journal of Business Research, 58(7), 935–943.

    Article  Google Scholar 

  • Sivo, S. A., Fan, X., Witta, E. L., & Willse, J. T. (2006). The search for" optimal" cutoff properties: Fit index criteria in structural equation modeling. Journal of Experimental Education, 74(3), 267–288.

    Article  Google Scholar 

  • Sleep, S., Bharadwaj, S., & Lam, S. K. (2015). Walking a tightrope: The joint impact of customer and within-firm boundary spanning activities on perceived customer satisfaction and team performance. Journal of the Academy of Marketing Science, 43(4), 472–489.

    Article  Google Scholar 

  • Steiger, J. H., & Lind, J. C. (1980). Statistically based tests for the number of common factors. Iowa City: Paper presented at the Annual Meeting of the Psychometric Society.

    Google Scholar 

  • Tucker, L. R., & Lewis, C. (1973). A reliability coefficient for maximum likelihood factor analysis. Psychometrika, 38(1), 1–10.

    Article  Google Scholar 

  • West, S. G., Taylor, A. B., & Wu, W. (2012). Model fit and model selection in structural equation modeling. In R. H. Hoyle (Ed.), Handbook of structural equation modeling (Vol. 1, pp. 209–231). New York: Guilford Press.

    Google Scholar 

  • Ximénez, C. (2009). Recovery of weak factor loadings in confirmatory factor analysis under conditions of model misspecification. Behavior Research Methods, 41(4), 1038–1052.

    Article  Google Scholar 

  • Zboja, J. J., Clark, R. A., & Haytko, D. L. (2016). An offer you can’t refuse: Consumer perceptions of sales pressure. Journal of the Academy of Marketing Science, 44(6), 806–821.

    Article  Google Scholar 

Download references

Acknowledgements

The authors thank Peter Kovacs for his help in programming the flexible cutoff tool. The authors are further grateful for the very helpful comments by the editor, the associate editor, three reviewers, and the marketing team of the Grenoble Ecole de Management.

Author information

Authors and Affiliations

  1. Department of Market Research, Clausthal University of Technology, 38678, Clausthal-Zellerfeld, Germany

    Thomas Niemand

  2. Department of Marketing, Grenoble Ecole de Management, Univ Grenoble Alpes ComUE, 12 Rue Pierre Semard, 38000, Grenoble, France

    Robert Mai

Authors
  1. Thomas Niemand
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Robert Mai
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Robert Mai.

Electronic supplementary material

ESM 1

(DOCX 314 kb)

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Niemand, T., Mai, R. Flexible cutoff values for fit indices in the evaluation of structural equation models. J. of the Acad. Mark. Sci. 46, 1148–1172 (2018). https://doi.org/10.1007/s11747-018-0602-9

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11747-018-0602-9

Keywords

Search

Navigation