Skip to main content
SpringerLink
Log in

Sample Size Planning for Cluster-Randomized Interventions Probing Multilevel Mediation

  • Published:
Prevention Science Aims and scope Submit manuscript

Abstract

Multilevel mediation analyses play an essential role in helping researchers develop, probe, and refine theories of action underlying interventions and document how interventions impact outcomes. However, little is known about how to plan studies with sufficient power to detect such multilevel mediation effects. In this study, we describe how to prospectively estimate power and identify sufficient sample sizes for experiments intended to detect multilevel mediation effects. We outline a simple approach to estimate the power to detect mediation effects with individual- or cluster-level mediators using summary statistics easily obtained from empirical literature and the anticipated magnitude of the mediation effect. We draw on a running example to illustrate several different types of mediation and provide an accessible introduction to the design of multilevel mediation studies. The power formulas are implemented in the R package PowerUpR and the PowerUp software (causalevaluation.org).

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2

Similar content being viewed by others

Notes

  1. Core assumptions include (a) stable unit treatment value assumption, (b) sequential ignorability, (c) consistency, (d) no downstream confounders, and (e) no treatment-by-mediator interaction.

References

  • Cox, K. & Kelcey, B. (2018). Optimal Sample Allocation in Group-randomized Studies of Multilevel Mediation with a Group-level Mediator. Journal of Experimental Education (in press).

  • Dong, N., Kelcey, B., Spybrook, J., & Maynard, R. A. (2016). PowerUp!-Mediator: A tool for calculating statistical power for causally-defined mediation in cluster randomized trials. (Version 0.4) [Software]. Available from http://www.causalevaluation.org/. Accessed 1 May 2007.

  • Gottfredson, D. C., Cook, T. D., Gardner, F. E., Gorman-Smith, D., Howe, G. W., Sandler, I. N., & Zafft, K. M. (2015). Standards of evidence for efficacy, effectiveness, and scale-up research in prevention science: Next generation. Prevention Science, 16, 893–926.

    Article  PubMed  PubMed Central  Google Scholar 

  • Kelcey, B., & Phelps, G. (2013). Considerations for Designing Group Randomized Trials of Professional Development with Teacher Knowledge Outcomes. Educational Evaluation and Policy Analysis, 35, 370–390.

  • Kelcey, B., Dong, N., Spybrook, J., & Cox, K. (2017). Statistical power for causally-defined indirect effects in group-randomized trials. Journal of Educational and Behavioral Statistics.

  • Kelcey, B., Dong, N., Spybrook, J., & Shen, Z. (2017a). Experimental Power for Indirect Effects in Group-randomized Studies with Group-level Mediators. Multivariate Behavioral Research, 52(6), 699–719.

  • Kelcey, B., Dong, N., Spybrook, J. & Cox, K. (2017b). Statistical power for causally-defined indirect effects in group-randomized trials. Journal of Educational and Behavioral Statistics.

  • Kelcey, B. & Shen, Z. (2018). Effective and Efficient Experimental Design 2-1-1 Mediation Studies. Journal of Experimental Education (in press).

  • Krull, J. L., & MacKinnon, D. P. (2001). Multilevel modeling of individual and cluster level mediated effects. Multivariate Behavioral Research, 36, 249–277.

    Article  CAS  PubMed  Google Scholar 

  • Phelps, G., Kelcey, B., Liu, S., & Jones, N. (2016). Informing Estimates of Program Effects for Studies of Mathematics Professional Development Using Teacher Content Knowledge Outcomes. Evaluation Review, 40, 383–409.

    Article  Google Scholar 

  • Pituch, K. A., & Stapleton, L. M. (2012). Distinguishing between cross- and cluster-level mediation processes in the cluster randomized trial. Sociological Methods & Research, 41, 630–670.

    Article  Google Scholar 

  • Preacher, K. J., & Selig, J. P. (2012). Advantages of Monte Carlo confidence intervals for indirect effects. Communication Methods and Measures, 6(2), 77–98.

    Article  Google Scholar 

  • Texas Christian University [TCU] (2005) Institute of Behavioral Research. The Organizational Readiness for Change: Treatment Staff Version (TCU ORC-S). Available at: www.ibr.tcu.edu.

  • Sobel, M. E. (1982). Asymptotic confidence intervals for indirect effects in structural equation models. Sociological Methodology, 13, 290–312.

    Article  Google Scholar 

  • Schwartz, R. (2010). Motivational interviewing (patient-centered counseling) to address childhood obesity. Pediatric Annals, 39(3), 154–158.

    Article  PubMed  Google Scholar 

  • Spybrook, J., Shi, R., & Kelcey, B. (2016). Progress in the Past Decade: An Examination of the Precision of Cluster Randomized Trials Funded by the U.S. Institute of Education Sciences. International Journal of Research & Method in Education, 39(3), 255–267.

    Article  Google Scholar 

  • Stapleton, L., Pituch, K., & Dion, E. (2015). Standardized effect size measures for mediation analysis in clusterrandomized trials. Journal of Experimental Education, 83(4), 547–582.

    Article  Google Scholar 

  • VanderWeele, T. J. (2010). Direct and indirect effects for neighborhood-based clustered and longitudinal data. Sociological Methods & Research, 38, 515–544.

    Article  CAS  Google Scholar 

  • Williams, N., & Glisson, C. (2014). Testing a theory or organizational culture, climate and youth outcomes in child welfare systems: A United States national study. Child Abuse and Neglect, 38, 4.

    Article  Google Scholar 

  • Zhang, Z., Zyphur, M., & Preacher, K. (2009). Testing multilevel mediation using hierarchical linear models: Problems and solutions. Organizational Research Methods, 12, 695–719.

    Article  Google Scholar 

Download references

Funding

This study was supported by grants from the National Science Foundation (Award Nos. 1437679, 1552535, 1437745, and 1437692). The opinions expressed herein are those of the authors and not the funding agencies.

Author information

Authors and Affiliations

  1. University of Cincinnati, Cincinnati, OH, 45221, USA

    Ben Kelcey

  2. Western Michigan University, Kalamazoo, MI, 49008, USA

    Jessaca Spybrook

  3. University of North Carolina, Chapel Hill, NC, 27599, USA

    Nianbo Dong

Authors
  1. Ben Kelcey
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Jessaca Spybrook
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Nianbo Dong
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Ben Kelcey.

Ethics declarations

Conflict of Interest

The authors declare that they have no conflicts of interest.

Ethical Approval

This article does not contain any studies with human participants or animals performed by any of the authors.

Informed Consent

The manuscript does not report any studies with human participants or animals; no study was conducted requiring informed consent.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Kelcey, B., Spybrook, J. & Dong, N. Sample Size Planning for Cluster-Randomized Interventions Probing Multilevel Mediation. Prev Sci 20, 407–418 (2019). https://doi.org/10.1007/s11121-018-0921-6

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11121-018-0921-6

Keywords

Search

Navigation