Advertisement

Prevention Science

, Volume 20, Issue 3, pp 407–418 | Cite as

Sample Size Planning for Cluster-Randomized Interventions Probing Multilevel Mediation

  • Ben KelceyEmail author
  • Jessaca Spybrook
  • Nianbo Dong
Article

Abstract

Multilevel mediation analyses play an essential role in helping researchers develop, probe, and refine theories of action underlying interventions and document how interventions impact outcomes. However, little is known about how to plan studies with sufficient power to detect such multilevel mediation effects. In this study, we describe how to prospectively estimate power and identify sufficient sample sizes for experiments intended to detect multilevel mediation effects. We outline a simple approach to estimate the power to detect mediation effects with individual- or cluster-level mediators using summary statistics easily obtained from empirical literature and the anticipated magnitude of the mediation effect. We draw on a running example to illustrate several different types of mediation and provide an accessible introduction to the design of multilevel mediation studies. The power formulas are implemented in the R package PowerUpR and the PowerUp software (causalevaluation.org).

Keywords

Mediation Power Indirect effects Multilevel models Sample size determination 

Notes

Funding

This study was supported by grants from the National Science Foundation (Award Nos. 1437679, 1552535, 1437745, and 1437692). The opinions expressed herein are those of the authors and not the funding agencies.

Compliance with Ethical Standards

Conflict of Interest

The authors declare that they have no conflicts of interest.

Ethical Approval

This article does not contain any studies with human participants or animals performed by any of the authors.

Informed Consent

The manuscript does not report any studies with human participants or animals; no study was conducted requiring informed consent.

References

  1. Cox, K. & Kelcey, B. (2018). Optimal Sample Allocation in Group-randomized Studies of Multilevel Mediation with a Group-level Mediator. Journal of Experimental Education (in press).Google Scholar
  2. Dong, N., Kelcey, B., Spybrook, J., & Maynard, R. A. (2016). PowerUp!-Mediator: A tool for calculating statistical power for causally-defined mediation in cluster randomized trials. (Version 0.4) [Software]. Available from http://www.causalevaluation.org/. Accessed 1 May 2007.
  3. Gottfredson, D. C., Cook, T. D., Gardner, F. E., Gorman-Smith, D., Howe, G. W., Sandler, I. N., & Zafft, K. M. (2015). Standards of evidence for efficacy, effectiveness, and scale-up research in prevention science: Next generation. Prevention Science, 16, 893–926.CrossRefGoogle Scholar
  4. Kelcey, B., & Phelps, G. (2013). Considerations for Designing Group Randomized Trials of Professional Development with Teacher Knowledge Outcomes. Educational Evaluation and Policy Analysis, 35, 370–390.Google Scholar
  5. Kelcey, B., Dong, N., Spybrook, J., & Cox, K. (2017). Statistical power for causally-defined indirect effects in group-randomized trials. Journal of Educational and Behavioral Statistics. Google Scholar
  6. Kelcey, B., Dong, N., Spybrook, J., & Shen, Z. (2017a). Experimental Power for Indirect Effects in Group-randomized Studies with Group-level Mediators. Multivariate Behavioral Research, 52(6), 699–719.Google Scholar
  7. Kelcey, B., Dong, N., Spybrook, J. & Cox, K. (2017b). Statistical power for causally-defined indirect effects in group-randomized trials. Journal of Educational and Behavioral Statistics. Google Scholar
  8. Kelcey, B. & Shen, Z. (2018). Effective and Efficient Experimental Design 2-1-1 Mediation Studies. Journal of Experimental Education (in press).Google Scholar
  9. Krull, J. L., & MacKinnon, D. P. (2001). Multilevel modeling of individual and cluster level mediated effects. Multivariate Behavioral Research, 36, 249–277.CrossRefGoogle Scholar
  10. Phelps, G., Kelcey, B., Liu, S., & Jones, N. (2016). Informing Estimates of Program Effects for Studies of Mathematics Professional Development Using Teacher Content Knowledge Outcomes. Evaluation Review, 40, 383–409.CrossRefGoogle Scholar
  11. Pituch, K. A., & Stapleton, L. M. (2012). Distinguishing between cross- and cluster-level mediation processes in the cluster randomized trial. Sociological Methods & Research, 41, 630–670.CrossRefGoogle Scholar
  12. Preacher, K. J., & Selig, J. P. (2012). Advantages of Monte Carlo confidence intervals for indirect effects. Communication Methods and Measures, 6(2), 77–98.CrossRefGoogle Scholar
  13. Texas Christian University [TCU] (2005) Institute of Behavioral Research. The Organizational Readiness for Change: Treatment Staff Version (TCU ORC-S). Available at: www.ibr.tcu.edu.
  14. Sobel, M. E. (1982). Asymptotic confidence intervals for indirect effects in structural equation models. Sociological Methodology, 13, 290–312.CrossRefGoogle Scholar
  15. Schwartz, R. (2010). Motivational interviewing (patient-centered counseling) to address childhood obesity. Pediatric Annals, 39(3), 154–158.CrossRefGoogle Scholar
  16. Spybrook, J., Shi, R., & Kelcey, B. (2016). Progress in the Past Decade: An Examination of the Precision of Cluster Randomized Trials Funded by the U.S. Institute of Education Sciences. International Journal of Research & Method in Education, 39(3), 255–267.CrossRefGoogle Scholar
  17. Stapleton, L., Pituch, K., & Dion, E. (2015). Standardized effect size measures for mediation analysis in clusterrandomized trials. Journal of Experimental Education, 83(4), 547–582.CrossRefGoogle Scholar
  18. VanderWeele, T. J. (2010). Direct and indirect effects for neighborhood-based clustered and longitudinal data. Sociological Methods & Research, 38, 515–544.CrossRefGoogle Scholar
  19. Williams, N., & Glisson, C. (2014). Testing a theory or organizational culture, climate and youth outcomes in child welfare systems: A United States national study. Child Abuse and Neglect, 38, 4.CrossRefGoogle Scholar
  20. Zhang, Z., Zyphur, M., & Preacher, K. (2009). Testing multilevel mediation using hierarchical linear models: Problems and solutions. Organizational Research Methods, 12, 695–719.CrossRefGoogle Scholar

Copyright information

© Society for Prevention Research 2018

Authors and Affiliations

  1. 1.University of CincinnatiCincinnatiUSA
  2. 2.Western Michigan UniversityKalamazooUSA
  3. 3.University of North CarolinaChapel HillUSA

Personalised recommendations