Advertisement

Behavior Research Methods

, Volume 51, Issue 1, pp 172–194 | Cite as

Evaluating fit indices in a multilevel latent growth curve model: A Monte Carlo study

  • Hsien-Yuan HsuEmail author
  • John J. H. Lin
  • Susan Troncoso Skidmore
  • Minjung Kim
Article

Abstract

The multilevel latent growth curve model (MLGCM), which is subsumed by the multilevel structural equation modeling framework, has been advocated as a means of investigating individual and cluster trajectories. Still, how to evaluate the goodness of fit of MLGCMs has not been well addressed. The purpose of this study was to conduct a systematic Monte Carlo simulation to carefully investigate the effectiveness of (a) level-specific fit indices and (b) target-specific fit indices in an MLGCM, in terms of their independence from the sample size’s influence and their sensitivity to misspecification in the MLGCM that occurs in either the between-covariance, between-mean, or within-covariance structure. The design factors included the number of clusters, the cluster size, and the model specification. We used Mplus 7.4 to generate simulated replications and estimate each of the models. We appropriately controlled the severity of misspecification when we generated the simulated replications. The simulation results suggested that applying RMSEAT_S_COV, TLIT _ S _ COV, and SRMRB maximizes the capacity to detect misspecifications in the between-covariance structure. Moreover, the use of RMSEAPS _ B, CFIPS _ B, and TLIPS _ B is recommended for evaluating the fit of the between-mean structure. Finally, we found that evaluation of the within-covariance structure turned out to be unexpectedly challenging, because none of the within-level-specific fit indices (RMSEAPS _ W, CFIPS _ W, TLIPS _ W, and SRMRW) had a practically significant sensitivity.

Keywords

Fit index Model evaluation Multilevel latent growth curve model Multilevel structural equation modeling 

References

  1. Bentler, P. M. (1990). Comparative fit indexes in structural models. Psychological Bulletin, 107, 238–246.  https://doi.org/10.1037/0033-2909.107.2.238 CrossRefPubMedGoogle Scholar
  2. Bentler, P. M., & Dudgeon, P. (1996). Covariance structure analysis: Statistical practice, theory, and directions. Annual Review of Psychology, 47, 563–592.CrossRefPubMedGoogle Scholar
  3. Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd). Hillsdale: Erlbaum.Google Scholar
  4. Cohen, J. (1992). A power primer. Psychological Bulletin, 112, 155–159.  https://doi.org/10.1037/0033-2909.112.1.155 CrossRefPubMedGoogle Scholar
  5. Cohen, J., Cohen, P., West, S. G., & Aiken, L. S. (2003). Applied multiple regression/correlation analysis for the behavioral sciences (3rd). Mahwah: Erlbaum.Google Scholar
  6. Fan, X., & Sivo, S. A. (2005). Sensitivity of fit indexes to misspecified structural or measurement model components: Rationale of two-Index strategy revisited. Structural Equation Modeling, 12, 343–367.CrossRefGoogle Scholar
  7. Gerbing, D. W., & Anderson, J. C. (1992). Monte Carlo evaluations of goodness of fit indices for structural equation models. Sociological Methods and Research, 21, 132–160.CrossRefGoogle Scholar
  8. Hox, J. J. (2010). Multilevel analysis: Techniques and applications (2nd). New York: Routledge.CrossRefGoogle Scholar
  9. Hox, J. J., & Maas, C. J. M. (2001). The accuracy of multilevel structural equation modeling with pseudobalanced groups and small samples. Structural Equation Modeling, 8, 157–174.  https://doi.org/10.1207/S15328007SEM0802_1 CrossRefGoogle Scholar
  10. Hox, J. J., Maas, C. J. M., & Brinkhuis, M. J. S. (2010). The effect of estimation method and sample size in multilevel structural equation modeling. Statistica Neerlandica, 64, 157–170.  https://doi.org/10.1111/j.1467-9574.2009.00445.x CrossRefGoogle Scholar
  11. Hsu, H.-Y., Kwok, O., Acosta, S., & Lin, J.-H. (2015). Detecting misspecified multilevel SEMs using common fit indices: A Monte Carlo study. Multivariate Behavioral Research, 50, 197–215.  https://doi.org/10.1080/00273171.2014.977429 CrossRefPubMedGoogle Scholar
  12. Hsu, H.-Y., Lin, J.-H., Kwok, O.-M., Acosta, S., & Willson, V. (2016). The impact of intraclass correlation on the effectiveness of level-specific fit indices in multilevel structural equation modeling: A Monte Carlo study. Educational and Psychological Measurement, 77, 5–31.  https://doi.org/10.1177/0013164416642823 CrossRefPubMedPubMedCentralGoogle Scholar
  13. Hu, L., & Bentler, P. M. (1998). Fit indices in covariance structure modeling: Sensitivity to underparameterized model misspecification. Psychological Methods, 3, 424–453.  https://doi.org/10.1037/1082-989X.3.4.424 CrossRefGoogle Scholar
  14. Hu, L., & Bentler, P. M. (1999). Cutoff criteria for fit indexes in covariance structure analysis: Conventional criteria versus new alternatives. Structural Equation Modeling, 6, 1–55.  https://doi.org/10.1080/10705519909540118 CrossRefGoogle Scholar
  15. Ingels, S. J., Pratt, D. J., Herget, D. R., Dever, J. A., Fritch, L. B., Ottem, R., … Leinwand, S. (2013). Education Longitudinal Study of 2002: Base-year to first follow-up data file documentation (NCES 2014-361). Washington, DC: National Center for Education Statistics.Google Scholar
  16. Jackson, D. L. (2003). Revisiting sample size and number of parameter estimates: Some support for the N:q hypothesis. Structural Equation Modeling, 10, 128–141.CrossRefGoogle Scholar
  17. Kaplan, D. (2009). Structural equation modeling: Foundations and extensions (2nd). Thousand Oaks: Sage.CrossRefGoogle Scholar
  18. Kline, R. B. (2011). Principles and practice of structural equation modeling (3rd). New York: Guilford Press.Google Scholar
  19. Kwok, O.-M., West, S. G., & Green, S. B. (2007). The impact of misspecifying the within-subject covariance structure in multiwave longitudinal multilevel models: A Monte Carlo study. Multivariate Behavioral Research, 42, 557–592.  https://doi.org/10.1080/00273170701540537 CrossRefGoogle Scholar
  20. Leite, W. L., & Stapleton, L. M. (2011). Detecting growth shape misspecifications in latent growth models: An evaluation of fit indexes. Journal of Experimental Education, 79, 361–381.CrossRefGoogle Scholar
  21. Lohr, S. L. (2009). Sampling design and analysis (2nd). Boston: Brooks/Cole.Google Scholar
  22. Lüdtke, O., Marsh, H. W., Robitzsch, A., & Trautwein, U. (2011). A 2 × 2 taxonomy of multilevel latent contextual models: Accuracy–bias trade-offs in full and partial error correction models. Psychological Methods, 16, 444–467.  https://doi.org/10.1037/a0024376 CrossRefPubMedGoogle Scholar
  23. Ma, X., & Ma, L. (2004). Modeling stability of growth between mathematics and science achievement during middle and high school. Evaluation Review, 28, 104–122.  https://doi.org/10.1177/0193841X03261025 CrossRefPubMedGoogle Scholar
  24. Ma, X., & Wilkins, J. M. (2007). Mathematics coursework regulates growth in mathematics achievement. Journal for Research in Mathematics Education, 38, 230–257.Google Scholar
  25. Marsh, H. W., Balla, J. R., & McDonald, R. P. (1988). Goodness-of-fit indexes in confirmatory factor analysis: The effect of sample size. Psychological Bulletin, 103, 391–410.CrossRefGoogle Scholar
  26. Marsh, H. W., Hau, K.-T., & Grayson, D. (2005). Goodness of fit in structural equation models. In A. Maydeu-Olivares & J. J. McArdle (Eds.), Contemporary psychometrics: A festschrift for Roderick P. McDonald. Multivariate applications book series (pp. 275–340). Mahwah: Erlbaum.Google Scholar
  27. McDonald, R. P., & Marsh, H. W. (1990). Choosing a multivariate model: Noncentrality and goodness of fit. Psychological Bulletin, 107, 247–255.  https://doi.org/10.1037/0033-2909.107.2.247 CrossRefGoogle Scholar
  28. Miller, J. D., Kimmel, L., Hoffer, T. B., & Nelson, C. (2000). Longitudinal study of American youth: User’s manual. Evanston: Northwestern University, International Center for the Advancement of Scientific Literacy.Google Scholar
  29. Muthén, B. O. (1997). Latent variable growth modeling with multilevel data. In M. Berkane (Ed.), Latent variable modeling and applications to causality (pp. 149–161). New York: Springer.CrossRefGoogle Scholar
  30. Muthén, B. O. (2004). Latent variable analysis: Growth mixture modeling and related techniques for longitudinal data. In D. Kaplan (Ed.), Handbook of quantitative methodology for the social sciences (pp. 345–368). Newbury Park: Sage.Google Scholar
  31. Muthén, B. O., & Asparouhov, T. (2011). Beyond multilevel regression modeling: Multilevel analysis in a general latent variable framework. In J. Hox & J. K. Roberts (Eds.), The handbook of advanced multilevel analysis. New York: Routledge.Google Scholar
  32. Muthén, B. O., & Satorra, A. (1995). Complex sample data in structural equation modeling. Sociological Methodology, 25, 267–316.  https://doi.org/10.2307/271070 CrossRefGoogle Scholar
  33. Muthén, L. K., & Muthén, B. O. (1998–2017). Mplus user’s guide (8th). Los Angeles: Muthén & Muthén.Google Scholar
  34. Neuman, W. L. (2009). Social research methods: Qualitative and quantitative approaches (7th). Boston: Allyn & Bacon.Google Scholar
  35. Ryu, E. (2014). Model fit evaluation in multilevel structural equation models. Frontiers in Psychology, 5, 81:1–9.  https://doi.org/10.3389/fpsyg.2014.00081 Google Scholar
  36. Ryu, E., & West, S. G. (2009). Level-specific evaluation of model fit in multilevel structural equation modeling. Structural Equation Modeling, 16, 583–601.  https://doi.org/10.1080/10705510903203466 CrossRefGoogle Scholar
  37. Scheaffer, R. L., Mendenhall, W., III, & Ott, R. L. (2005). Elementary survey sampling (6th). Belmont: Thomson Brooks/Cole.Google Scholar
  38. Schermelleh-Engel, K., Kerwer, M., & Klein, A. G. (2014). Evaluation of model fit in nonlinear multilevel structural equation modeling. Frontiers in Psychology, 5, 181:1–11.  https://doi.org/10.3389/fpsyg.2014.00181 Google Scholar
  39. Tourangeau, K., Nord, C., Lê, T., Sorongon, A. G., Hagedorn, M. C., Daly, P., & Najarian, M. (2015). Early Childhood Longitudinal Study, kindergarten class of 2010–11 (ECLS-K:2011), user’s manual for the ECLS-K:2011 kindergarten data file and electronic codebook, public version (NCES 2015-074). Washington, DC: National Center for Education Statistics.Google Scholar
  40. Tourangeau, K., Nord, C., Lê, T., Sorongon, A. G., & Najarian, M. (2009). Early Childhood Longitudinal Study, kindergarten class of 1998–99 (ECLS-K), combined user’s manual for the ECLS-K eighth-grade and K–8 full sample data files and electronic codebooks (NCES 2009-004). Washington, DC: National Center for Education Statistics. Retrieved from http://nces.ed.gov/ecls/data/ECLSK_K8_Manual_part1.pdf Google Scholar
  41. Tucker, J. S., & Lewis, C. (1973). The reliability coefficient for maximum likelihood factor analysis. Psychometrika, 38, 1–10.CrossRefGoogle Scholar
  42. Widaman, K. F., & Thompson, J. S. (2003). On specifying the null model for incremental fit indices in structural equation modeling. Psychological Methods, 8, 16–37.  https://doi.org/10.1037/1082-989X.8.1.16 CrossRefPubMedGoogle Scholar
  43. Wu, J.-Y., Kwok, O., & Willson, V. L. (2015). Using design-based latent growth curve modeling with cluster-level predictor to address dependency. Journal of Experimental Education, 82, 431–454.  https://doi.org/10.1080/00220973.2013.876226 CrossRefGoogle Scholar
  44. Wu, W., & West, S. G. (2010). Sensitivity of fit indices to misspecification in growth curve models. Multivariate Behavioral Research, 45, 420–452.CrossRefPubMedGoogle Scholar
  45. Wu, W., West, S. G., & Taylor, A. B. (2009). Evaluating model fit for growth curve models: Integration of fit indices from SEM and MLM frameworks. Psychological Methods, 14, 183–201.  https://doi.org/10.1037/a0015858 CrossRefPubMedGoogle Scholar

Copyright information

© Psychonomic Society, Inc. 2018

Authors and Affiliations

  • Hsien-Yuan Hsu
    • 1
    Email author
  • John J. H. Lin
    • 2
  • Susan Troncoso Skidmore
    • 3
  • Minjung Kim
    • 4
  1. 1.Children’s Learning InstituteUniversity of Texas Health Science CenterHoustonUSA
  2. 2.Office of Institutional ResearchNational Central UniversityTaoyuan CityTaiwan
  3. 3.Department of Educational LeadershipSam Houston State UniversityHuntsvilleUSA
  4. 4.Department of Educational StudiesOhio State UniversityColumbusUSA

Personalised recommendations