Sample size, effect size, and statistical power: a replication study of Weisburd’s paradox
- 606 Downloads
This study expands upon Weisburd’s work (1993) by reexamining the relationship between sample size and statistical power in criminological experiments. This inquiry, now known as the Weisburd paradox, postulates that increasing the sample size of experiments does not always lead to increases in statistical power. The current research also begins to explore the potential sources of the Weisburd paradox.
The effect sizes and statistical power are computed for the outcome measures (n = 402) of all experiments (n = 66) included in systematic reviews published by the Campbell Collaboration’s Crime and Justice Coordinating Group. The design sensitivity of these experiments is reviewed by sample size, as well as other factors that may explain the variation in effect sizes and statistical power across studies.
Effect sizes decline as the sample size of the experiment increases, whereas statistical power is unrelated to sample size but strongly associated with effect size. Disclosure of fidelity issues and publication bias is unrelated to statistical power and treatment effects. Variability in the dependent variable and sample demographics are significantly related to statistical power, but not to effect size.
The study finds support for the Weisburd paradox, as the ability to manipulate statistical power by increasing sample size is not as strong as statistical theory would suggest, and experiments with larger sample sizes generally produce smaller effects. It is believed that a relationship was not observed between sample size and statistical power because the sensitivity gained from increasing sample size is offset by effect size simultaneously decreasing.
KeywordsExperiments Statistical power Effect size Sample size Weisburd paradox
- Alexander, R. A., Barrett, G. V., Alliger, G. M., & Kenneth, P. C. (1986). Towards a general model of non-random sampling and the impact on population correlation: generalizations of Berkson’s fallacy and restriction of range. British Journal of Mathematical and Statistical Psychology, 39(1), 90–105.CrossRefGoogle Scholar
- Bellg, A. J., Borrelli, B., Resnick, B., Hecht, J., Minicucci, D. S., Ory, M., Ogedbe, G., Orwig, D., Ernst, D., & Czajkowski, S. (2004). Enhancing treatment fidelity in health behavior change studies: best practices and recommendations from the NIH behavior change consortium. Health Psychology, 23(5), 443–451.CrossRefGoogle Scholar
- Britt, C. L., & Weisburd, D. (2011). Statistical power. In A. R. Piquero & D. Weisburd (Eds.), Handbook of quantitative criminology (pp. 313–332). New York: Springer.Google Scholar
- Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Hillsdale: Erlbaum.Google Scholar
- Cook, T. D., & Campbell, D. T. (1979). Quasi-experimentation: Design and analysis issues for field settings. Chicago: Rand McNally.Google Scholar
- Dickersin, K. (2005). Publication bias: Recognizing the problem, understanding its origins and scope, and preventing harm. In H. Rothstein, A. J. Sutton, & M. Borenstein (Eds.), Publication bias in meta-analysis: Prevention, assessment and adjustments (pp. 11–34). Chichester: Wiley.Google Scholar
- Esbensen, F. (1991). Ethical considerations in criminal justice research. American Journal of Police, 10(2), 87–104.Google Scholar
- Farrington, D. P. (1983). Randomized experiments on crime and justice. In M. Tonry & N. Morris (Eds.), Crime and justice (pp. 257–308). Chicago: University of Chicago Press.Google Scholar
- Farrington, D. P., Gottfredson, D. C., Sherman, L. W., & Welsh, B. C. (2002). The Maryland scientific methods scale. In L. W. Sherman, D. P. Farrington, B. C. Welsh, & D. L. MacKenzie (Eds.), Evidence-based crime prevention (pp. 13–21). London: Routledge.Google Scholar
- Fienberg, S. E., & Tanur, J. M. (1986). The design and analysis of longitudinal surveys: Controversies and issues of cost and continuity. In R. Pearson & R. Boruch (Eds.), Survey research designs: Towards a better understanding of their costs and benefits (pp. 60–93). New York: Springer.CrossRefGoogle Scholar
- Gill, C. E. (2011). Missing links: how descriptive validity impacts the policy relevance of randomized controlled trials in criminology. Journal of Experimental Criminology, 7 (3), 201–224.Google Scholar
- Glazerman, S., Levy, D. M., & Myers, D. (2002). Non experimental replications of social experiments: A systematic review. Washington: Mathematics Policy Research.Google Scholar
- Harbord, R. M., & Higgins, J. P. (2008). Meta-regression in Stata. The Stata Journal, 8(4), 493–519.Google Scholar
- Lipsey, M. (1990). Design sensitivity: Statistical power for experimental research. Newbury Park: Sage.Google Scholar
- Lipsey, M. W. (2009). The primary factors that characterize effective interventions withjuvenile offenders: A meta-analytic overview.Victims and Offenders, 4, 124–147.Google Scholar
- Lösel, F., & Köferl, P. (1989). Evaluation research on correctional treatment in West Germany: A meta-analysis. In Criminal behavior and the justice system (pp. 334–355). Springer Berlin Heidelberg.Google Scholar
- McCord, J. (1978). A thirty-year followup of treatment effects. American Psychologist, 33 (3), 284–289.Google Scholar
- Müllen, B. (1989). Advanced BASIC meta-analysis. Hillsdale: Erlbaum.Google Scholar
- Shadish, W. R., Cook, T. D., & Campbell, D. T. (2002). Experimental and quasi-experimental designs for generalized causal inference. Boston: Houghton-Mifflin.Google Scholar
- Sharp, S. (1998). Meta-analysis regression. Stata Technical Bulletin 42: 16–22. In Stata Technical Bulletin Reprints, vol. 7, 148–155. College Station, TX: Stata Press.Google Scholar
- Sherman, L. W., Gottfredson, D. C., MacKenzie, D. L., Eck, J., Reuter, P., & Bushway, S. D. (1998). Preventing crime: What works, what doesn’t, what’s promising. Washington: U.S. National Institute of Justice.Google Scholar
- We, S. R., et al. (2012). Placebo effect was influenced by publication year in three-armed acupuncture trials. Complementary Therapies in Medicine 20.1, 83–92.Google Scholar
- Weisburd, D. (1993). Design sensitivity in criminal justice experiments: reassessing the relationship between sample size and statistical power. In M.Tonry & N. Morris (Eds.), Crime and Justice, Vol 17 (pp. 337–379). Chicago: University of Chicago Press.Google Scholar
- Weisburd, D., & Britt, C. (2007). Statistics in criminal justice (3rd ed.). New York: Springer.Google Scholar