Prevention Science

, Volume 16, Issue 7, pp 987–996 | Cite as

Sample Size Considerations in Prevention Research Applications of Multilevel Modeling and Structural Equation Modeling

Article

Abstract

When the goal of prevention research is to capture in statistical models some measure of the dynamic complexity in structures and processes implicated in problem behavior and its prevention, approaches such as multilevel modeling (MLM) and structural equation modeling (SEM) are indicated. Yet the assumptions that must be satisfied if these approaches are to be used responsibly raise concerns regarding their use in prevention research involving smaller samples. In this article, we discuss in nontechnical terms the role of sample size in MLM and SEM and present findings from the latest simulation work on the performance of each approach at sample sizes typical of prevention research. For each statistical approach, we draw from extant simulation studies to establish lower bounds for sample size (e.g., MLM can be applied with as few as ten groups comprising ten members with normally distributed data, restricted maximum likelihood estimation, and a focus on fixed effects; sample sizes as small as N = 50 can produce reliable SEM results with normally distributed data and at least three reliable indicators per factor) and suggest strategies for making the best use of the modeling approach when N is near the lower bound.

Keywords

Sample size Multilevel modeling Structural equation modeling 

References

  1. Aitkin, M. A., & Longford, N. T. (1986). Statistical modeling issues in school effectiveness studies. Journal of the Royal Statistical Society, 149, 1–26. Retrieved from http://www.jstor.org/stable/2981882.CrossRefGoogle Scholar
  2. Anderson, T. W. (1957). Maximum likelihood estimates for a multivariate normal distribution when some observations are missing. Journal of the American Statistical Association, 52, 200–203. Retrieved from http://www.jstor.org/stable/2280845.CrossRefGoogle Scholar
  3. Austin, P. C. (2010). Estimating multilevel logistic regression models when the number of clusters is low: A comparison of different statistical software procedures. The International Journal of Biostatistics, 6, 1–18. doi:10.2202/1557-4679.1195.Google Scholar
  4. Bandalos, D. L., & Gagné, P. (2012). Simulation methods in structural equation modeling. In R. H. Hoyle (Ed.), Handbook of structural equation modeling (pp. 92–108). New York: Guildford Press.Google Scholar
  5. Bauer, D. J. (2003). Estimating multilevel linear models as structural equation models. Journal of Educational and Behavioral Statistics, 28, 135–167. doi:10.3102/10769986028002135.CrossRefGoogle Scholar
  6. Bauer, D. J., & Sterba, S. K. (2011). Fitting multilevel models with ordinal outcomes: Performance of alternative specifications and methods of estimation. Psychological Methods, 16, 373–390. doi:10.1037/a0025813.PubMedCentralCrossRefPubMedGoogle Scholar
  7. Bearden, W. O., Sharma, S., & Teel, J. E. (1982). Sample size effects on chi square and other statistics used in evaluating causal models. Journal of Marketing Research, 19, 425–430. doi:10.2307/3151716.CrossRefGoogle Scholar
  8. Bentler, P. M. (1990). Comparative fit indexes in structural models. Psychological Bulletin, 107, 238–246. doi:10.1037/0033-2909.107.2.238.CrossRefPubMedGoogle Scholar
  9. Bollen, K. A. (1990). Overall fit in covariance structure models: Two types of sample size effects. Psychological Bulletin, 107, 256–259. doi:10.1037/0033-2909.107.2.256.CrossRefGoogle Scholar
  10. Bollen, K. A., & Curran, P. J. (2006). Latent curve models: A structural equation approach. Hoboken: Wiley.Google Scholar
  11. Bollen, K. A., & Noble, M. D. (2011). Structural equation models and the quantification of behavior. Proceedings of the National Academy of Sciences of the United States of America, 108, 15639–15646. doi:10.1073/pnas.1010661108.PubMedCentralCrossRefPubMedGoogle Scholar
  12. Breslow, N. E., & Clayton, D. G. (1993). Approximate inference in generalized linear mixed models. Journal of the American Statistical Association, 88, 9–25. Retrieved from http://www.jstor.org/stable/2290687.Google Scholar
  13. Curran, P. J. (2003). Have multilevel models been structural equation models all along? Multivariate Behavioral Research, 38, 529–569. doi:10.1207/s15327906mbr3804_5.CrossRefGoogle Scholar
  14. Curran, P. J., Lee, T., Howard, A. L., Lane, S., & MacCallum, R. (2012). Disaggregating within-person and between-person effects in multilevel and structural equation growth models. In J. R. Harring & G. R. Hancock (Eds.), Advances in longitudinal models in the social and behavioral sciences (pp. 217–253). Charlotte: Information Age Publishing.Google Scholar
  15. Demidenko, E. (2004). Mixed models: Theory and applications. Hoboken: Wiley.CrossRefGoogle Scholar
  16. Dempster, A. P., Laird, N. M., & Rubin, D. B. (1977). Maximum likelihood from incomplete data via the EM algorithm. Journal of the Royal Statistical Society, Series B, 44, 1–38. Retrieved from http://www.jstor.org/stable/2984875?origin=JSTOR-pdf.Google Scholar
  17. Duncan, S. C., Duncan, T. E., & Strycker, L. A. (2002). A multilevel analysis of neighborhood context and youth alcohol and drug problems. Prevention Science, 3, 125–133. doi:10.1023/A:1015483317310.CrossRefPubMedGoogle Scholar
  18. Fan, X., Thompson, B., & Wang, L. (1999). Effects of sample size, estimation methods, and model specification on structural equation modeling fit indexes. Structural Equation Modeling, 6, 56–83. doi:10.1080/10705519909540119.CrossRefGoogle Scholar
  19. Gagné, P., & Hancock, G. R. (2006). Measurement model quality, sample size, and solution propriety in confirmatory factor models. Multivariate Behavioral Research, 41, 65–83. doi:10.1207/s15327906mbr4101_5.CrossRefGoogle Scholar
  20. Gillmore, M. R., Hawkins, J. D., Catalano, R. F., Jr., Day, L. E., Moore, M., & Abbott, R. (1991). Structure of problem behaviors in preadolescence. Journal of Consulting and Clinical Psychology, 59, 499–506. doi:10.1037/0022-006X.59.4.499.CrossRefPubMedGoogle Scholar
  21. Goldstein, H. (1986). Multilevel mixed linear model analysis using iterative generalized least squares. Biometrika, 73, 43–56. doi:10.1093/biomet/73.1.43.CrossRefGoogle Scholar
  22. Hopkin, C. R., Hoyle, R. H., & Gottfredson, N. C. (2013). Maximizing the yield of small samples in prevention research: A review of general strategies and best practices. Manuscript submitted for publication.Google Scholar
  23. Hoyle, R. H. (2011). Structural equation modeling for social and personality psychology. London: Sage Publications.Google Scholar
  24. Hoyle, R. H., & Kenny, D. A. (1999). Sample size, reliability, and tests of statistical mediation. In R. H. Hoyle (Ed.), Statistical strategies for small sample research (pp. 195–222). Thousand Oaks: Sage Publications.Google Scholar
  25. Hu, L.-T., & Bentler, P. M. (1999). Cutoff criteria for fit indexes in covariance structure analysis: Conventional criteria versus new alternatives. Structural Equation Modeling, 6, 1–55. doi:10.1080/10705519909540118.CrossRefGoogle Scholar
  26. Jackson, D. L., Voth, J., & Frey, M. P. (2013). A note on sample size and solution propriety for confirmatory factor analytic models. Structural Equation Modeling, 20, 86–97. doi:10.1080/10705511.2013.742388.CrossRefGoogle Scholar
  27. Kaplan, D. (1995). Statistical power in structural equation modeling. In R. H. Hoyle (Ed.), Structural equation modeling: Concepts, issues, and applications (pp. 100–117). Newbury Park: Sage Publications.Google Scholar
  28. Kim, K. H. (2005). The relation among fit indexes, power, and sample size in structural equation modeling. Structural Equation Modeling, 12, 368–390. doi:10.1207/s15328007sem1203_2.CrossRefGoogle Scholar
  29. Kim, S.-Y. (2012). Sample size requirements in single- and multiphase growth mixture models: A Monte Carlo simulation. Structural Equation Modeling, 19, 457–476. doi:10.1080/10705511.2012.687672.CrossRefGoogle Scholar
  30. Kreft, I. G. G., & de Leeuw, J. (1988). Introducing multilevel modeling. Thousand Oaks: Sage Publications.Google Scholar
  31. Kreft, I. G. G., de Leeuw, J., & Aiken, L. S. (1995). The effect of different forms of centering in hierarchical linear models. Multivariate Behavioral Research, 30, 1–21. doi:10.1207/s15327906mbr3001_1.CrossRefGoogle Scholar
  32. Lüdtke, O., Marsh, H. W., Robitzsch, A., Trautwein, U., Asparouhov, T., & Muthén, B. (2008). The multilevel latent covariate model: A new, more reliable approach to group-level effects in contextual studies. Psychological Methods, 13, 203–229. doi:10.1037/a0012869.CrossRefPubMedGoogle Scholar
  33. Maas, C. J. M., & Hox, J. J. (2005). Sufficient sample sizes for multilevel modeling. Methodology: European Journal of Research Methods for the Behavioral and Social Sciences, 1, 85–91. doi:10.1027/1614-1881.1.3.86.CrossRefGoogle Scholar
  34. MacCallum, R. C., Browne, M. W., & Sugawara, H. M. (1996). Power analysis and determination of sample size for covariance structure modeling. Psychological Methods, 1, 130–149. Retrieved from http://doi.apa.org/journals/met/1/2/130.pdf.CrossRefGoogle Scholar
  35. MacCallum, R. C., Zhang, S., Preacher, K. J., & Rucker, D. D. (2002). On the practice of dichotomization of quantitative variables. Psychological Methods, 7, 19–40. doi:10.1037/1082-989X.7.1.19.CrossRefPubMedGoogle Scholar
  36. Marsh, H. W., Hau, K.-T., Balla, J. R., & Grayson, D. (1998). Is more ever too much? The number of indicators per factor in confirmatory factor analysis. Multivariate Behavioral Research, 33, 181–220. doi:10.1207/s15327906mbr3302_1.CrossRefGoogle Scholar
  37. McArdle, J. J., & Nesselroade, J. R. (2003). Growth curve analysis in contemporary psychological research. In J. Schinka & W. Velicer (Eds.), Comprehensive handbook of psychology: Research methods in psychology (pp. 447–480). New York: Wiley. doi:10.1002/0471264385.Google Scholar
  38. Meredith, W., & Tisak, J. (1990). Latent curve analysis. Psychometrika, 55, 107–122. doi:10.1007/BF02294746.CrossRefGoogle Scholar
  39. Noh, M., & Lee, Y. (2007). REML estimation for binary data in GLMMs. Journal of Multivariate Analysis, 98, 896–915. doi:10.1016/j.jmva.2006.11.009.CrossRefGoogle Scholar
  40. Rabe-Hesketh, S., Skrondal, A., & Pickles, A. (2002). Reliable estimation of generalized linear mixed models using adaptive quadrature. The Stata Journal, 2, 1–21. Retrieved from http://www.stata-journal.com/article.html?article=st0005.Google Scholar
  41. Raudenbush, S. W. (1997). Statistical analysis and optimal design for cluster randomized trials. Psychological Methods, 2, 173–185. doi:10.1037/1082-989X.2.2.173.CrossRefGoogle Scholar
  42. Raudenbush, S. W., & Bryk, A. S. (2002). Hierarchical linear models: Applications and data analysis methods. Thousand Oaks: Sage Publications.Google Scholar
  43. Raudenbush, S. W., & Liu, X. (2000). Statistical power and optimal design for multisite randomized trials. Psychological Methods, 5, 199–213. doi:10.1037//1082-989X.5.2.199.CrossRefPubMedGoogle Scholar
  44. Raudenbush, S. W., Bryk, A. S., & Congdon, R. (2004). HLM 6 for Windows [Computer software]. Lincolnwood: Scientific Software.Google Scholar
  45. Raudenbush, S. W., Spybrook, J., Congdon, R., Liu, X., & Martinez, A. (2011). Optimal Design Software for multi-level and longitudinal research (version 3.01) [Software]. Available from www.wtgrantfoundation.org.
  46. Rodríguez, G., & Goldman, N. (2001). Improved estimation procedures for multilevel models with binary response: A case study. Journal of the Royal Statistical Society: Series A (Statistics in Society), 164, 339–355. doi:10.1111/1467-985X.00206.CrossRefGoogle Scholar
  47. Schabenberger, O. (2005). Introducing the GLIMMIX procedure for generalized linear mixed models. Cary: SAS Institute. Retrieved from http://www2.sas.com/proceedings/sugi30/196-30.pdf.Google Scholar
  48. Schwarz, G. E. (1978). Estimating the dimension of a model. Annals of Statistics, 6, 461–464. Retrieved from http://www.jstor.org/stable/2958889.CrossRefGoogle Scholar
  49. Shiyko, M. P., Lanza, S. T., Tan, X., Li, R., & Shiffman, S. (2012). Using the time-varying effect model (TVEM) to examine dynamic associations between negative affect and self-confidence on smoking urges: Differences between successful quitters and relapsers. Prevention Science, 13, 288–299. doi:10.1007/s11121-011-0264-z.PubMedCentralCrossRefPubMedGoogle Scholar
  50. Simon, T. R., Ikeda, R. M., Smith, E. P., Reese, L. E., Rabiner, D. L., Miller-Johnson, S., et al. (2008). The multisite violence prevention project: Impact of a universal school-based violence prevention program on social-cognitive outcomes. Prevention Science, 9, 231–244. doi:10.1007/s11121-008-0101-1.PubMedCentralCrossRefGoogle Scholar
  51. Singer, J. D., & Willett, J. B. (2003). Applied longitudinal data analysis: Modeling change and event occurrence. New York: Oxford University Press.CrossRefGoogle Scholar
  52. Snijders, T. A. B., & Bosker, R. J. (2004). Multilevel analysis: An introduction to basic and advanced multilevel modeling (2nd ed.). London: Sage Publications.Google Scholar
  53. Steiger, J. H., & Lind, J. C. (1980). Statistically based tests for the number of common factors. Iowa City: Paper presented at the Meeting of the Psychometric Society.Google Scholar
  54. Tanaka, J. S. (1987). “How big is big enough?”: Sample size and goodness of fit in structural equation models with latent variables. Child Development, 58, 134–146. Retrieved from http://www.jstor.org/stable/1130296.CrossRefGoogle Scholar
  55. Weston, R., & Gore, P. A., Jr. (2006). A brief guide to structural equation modeling. The Counseling Psychologist, 34, 719–751. doi:10.1177/0011000006286345.CrossRefGoogle Scholar
  56. Whiteside, S. P., & Lynam, D. R. (2001). The five factor model and impulsivity: Using a structural model of personality to understand impulsivity. Personality and Individual Differences, 30, 669–689. doi:10.1016/S0191-8869(00)00064-7.CrossRefGoogle Scholar

Copyright information

© Society for Prevention Research 2014

Authors and Affiliations

  1. 1.Department of Psychology and NeuroscienceDuke UniversityDurhamUSA
  2. 2.Center for Developmental ScienceUniversity of North Carolina at Chapel HillChapel HillUSA

Personalised recommendations