Prevention Science

, Volume 16, Issue 7, pp 950–955 | Cite as

Maximizing the Yield of Small Samples in Prevention Research: A Review of General Strategies and Best Practices

  • Cameron R. Hopkin
  • Rick H. Hoyle
  • Nisha C. Gottfredson
Article

Abstract

The goal of this manuscript is to describe strategies for maximizing the yield of data from small samples in prevention research. We begin by discussing what “small” means as a description of sample size in prevention research. We then present a series of practical strategies for getting the most out of data when sample size is small and constrained. Our focus is the prototypic between-group test for intervention effects; however, we touch on the circumstance in which intervention effects are qualified by one or more moderators. We conclude by highlighting the potential usefulness of graphical methods when sample size is too small for inferential statistical methods.

Keywords

Small samples Maximizing statistical power Graphical methods 

References

  1. Bacchetti, P., Deeks, S. G., & McCune, J. M. (2011). Breaking free of sample size dogma to perform innovative translational research. Science Translational Medicine, 3, 87. doi:10.1126/scitranslmed.3001628.CrossRefGoogle Scholar
  2. Brown, C. H., Sloboda, Z., Faggiano, F., Teasdale, B., Keller, F., Burkhart, G., Vigna-Taglianti, F., Howe, G., Masyn, K., Wang, W., Muthén, B., Stephens, P., Grey, S., & Perrino, T. (2011). Methods for synthesizing findings on moderation effects across multiple randomized trials. Prevention Science, 14, 144–156. doi:10.1007/s11121-011-0207-8.CrossRefGoogle Scholar
  3. Carrig, M., Wirth, R. J., & Curran, P. J. (2004). A SAS macro for estimating and visualizing individual growth curves. Structural Equation Modeling: An Interdisciplinary Journal, 11, 132–149. doi:10.1207/S15328007SEM1101_9.CrossRefGoogle Scholar
  4. Cohen, J., Cohen, P., West, S., & Aiken, L. (2003). Applied multiple regression/correlation analysis for the behavioral sciences (3rd ed.). Mahwah: Erlbaum.Google Scholar
  5. Collins, L. M., Schafer, J. L., & Kam, C. (2001). A comparison of inclusive and restrictive strategies in modern missing data procedures. Psychological Methods, 6, 330–351. doi:10.1037/1082-989X.6.4.330.CrossRefPubMedGoogle Scholar
  6. Collins, L. M., Dziak, J. J., & Li, R. (2009). Design of experiments with multiple independent variables: A resource management perspective on complete and reduced factorial designs. Psychological Methods, 14, 202–224. doi:10.1037/a0015826.PubMedCentralCrossRefPubMedGoogle Scholar
  7. Collins, L. M., Baker, T. D., Mermelstein, R. J., Piper, M. E., Jorenby, D. E., Smith, S. S., Christiansen, B. A., Schlam, T. R., Cook, J. W., & Fiore, M. C. (2011). The multiphase optimization strategy for engineering effective tobacco use interventions. Annals of Behavioral Medicine, 41, 208–226. doi:10.1007/s12160-010-9253-x.PubMedCentralCrossRefPubMedGoogle Scholar
  8. Curran, P. J., & Hussong, A. M. (2009). Integrative data analysis: The simultaneous analysis of multiple data sets. Psychological Methods, 14, 81–100. doi:10.1037/a0015914.PubMedCentralCrossRefPubMedGoogle Scholar
  9. Dennis, M., Francis, D. J., Cirino, P. T., Schachar, R., Barnes, M. A., & Fletcher, J. M. (2009). Why IQ is not a covariate in cognitive studies of neurodevelopmental disorders. Journal of the International Neuropsychological Society, 15, 331–343. doi:10.1017/S1355617709090481.PubMedCentralCrossRefPubMedGoogle Scholar
  10. Dumas, J. E., Lynch, A. M., Laughlin, J. E., Phillips Smith, E., & Prinz, R. J. (2001). Promoting intervention fidelity. Conceptual issues, methods, and preliminary results from the EARLY ALLIANCE prevention trial. American Journal of Preventive Medicine, 20, 38–47. doi:10.1016/S0749-3797(00)00272-5.CrossRefPubMedGoogle Scholar
  11. Embretson, S. E. (1996). Item response theory models and spurious interaction effect in factorial ANOVA designs. Applied Psychological Measurement, 20, 201–212. doi:10.1177/014662169602000302.CrossRefGoogle Scholar
  12. Enders, C. K. (2010). Applied missing data analysis. New York: Guildford Press.Google Scholar
  13. Fok, C. C. T., Henry, D., & Allen, J. A (2015). Maybe small is too small a term: Introduction to advancing small sample prevention science. Prevention Science, in press.Google Scholar
  14. Friendly, M. (1995). Exploratory and graphical methods of data analysis [Online short course]. Retrieved from http://www.datavis.ca/courses/eda/. Accessed 6 Aug 2013.
  15. Graham, J. W. (2009). Missing data analysis: Making it work in the real world. Annual Review of Psychology, 60, 549–576. doi:10.1146/annurev.psych.58.110405.085530.CrossRefPubMedGoogle Scholar
  16. Hansen, W. B., & Collins, L. M. (1994). Seven ways to increase power without increasing N. NIDA Research Monograph. In L. M. Collins & L. A. Seitz (Eds.), Advances in data analysis for prevention intervention research (NIDA Research Monograph 142, NIH Publication No. 94–3599, pp. 184–195). Rockville, MD: National Institutes of Health.Google Scholar
  17. Hansen, W. B., Tobler, N. S., & Graham, J. W. (1990). Attrition in substance abuse prevention research: A meta-analysis of 85 longitudinally followed cohorts. Evaluation Review, 14, 677–685. doi:10.1177/0193841X9001400608.CrossRefGoogle Scholar
  18. Harlow, L. L., Mulaik, S. A., & Steiger, J. H. (Eds.). (1997). What if there were no significance tests? Mahwah: Erlbaum.Google Scholar
  19. Hawkins, N. G., Sanson-Fisher, R. W., Shakeshaft, A., D’Este, C., & Green, L. W. (2007). The multiple baseline design for evaluating population-based research. American Journal of Preventive Medicine, 33, 162–168. doi:10.1016/j.amepre.2007.03.020.CrossRefPubMedGoogle Scholar
  20. Hoyle, R. H. (Ed.). (1999). Statistical strategies for small sample research‬. Thousand Oaks: Sage Publications.Google Scholar
  21. Hoyle, R. H., & Gottfredson, N. C. (2015). Sample size considerations in prevention research applications of multilevel modeling and structural equation modeling. Prevention Science. doi:10.1007/s11121-014-0489-8.
  22. Kang, S., & Waller, G. (2005). Moderated multiple regression, spurious interaction effects, and IRT. Applied Psychological Measurement, 29, 87–105. doi:10.1177/0146621604272737.CrossRefGoogle Scholar
  23. Kratochwill, T. R., & Levin, J. R. (2010). Enhancing the scientific credibility of single-case intervention research: Randomization to the rescue. Psychological Methods, 15, 124–144. doi:10.1037/a0017736.CrossRefPubMedGoogle Scholar
  24. Nickerson, R. S. (2000). Null hypothesis significance testing: A review of an old and continuing controversy. Psychological Methods, 5, 241–301. doi:10.1037//1082-989X.S.2.241.CrossRefPubMedGoogle Scholar
  25. Post, F. H., Nielson, G. M., & Bonneau, G.-P. (Eds.) (2003). Data visualization: The state of the art. Boston: Kluwer Academic Publishers. Retrieved from http://www.springer.com/computer/image+processing/book/978-1-4020-7259-8?otherVersion=978-1-4613-5430-7
  26. Raudenbush, S. W. (1997). Statistical analysis and optimal design for cluster randomized trials. Psychological Methods, 2, 173–185. doi:10.1037/1082-989X.2.2.173.CrossRefGoogle Scholar
  27. Schafer, J. L. (1997). Analysis of incomplete multivariate data. London: Chapman & Hall.CrossRefGoogle Scholar
  28. Supplee, L. H., Kelly, B. C., MacKinnon, D. M., & Barofsky, M. Y. (2013). Subgroup analysis in prevention and intervention science [Special issue]. Prevention Science 14(2).Google Scholar
  29. von Hippel, P. T. (2013). The bias and efficiency of incomplete-data estimators in small univariate normal samples. Sociological Methods & Research, 42, 531–558. doi:10.1177/0049124113494582.CrossRefGoogle Scholar
  30. Wang, R., & Ware, J. H. (2013). Detecting moderator effects using subgroup analyses. Prevention Science, 14, 111–120. doi:10.1007/s11121-011-0221-x.PubMedCentralCrossRefPubMedGoogle Scholar
  31. Young, F. W. (1996). ViSta: The visual statistics system. Chapel Hill: Thurstone Psychometric Laboratory Research Memorandum 94-1(c). Retrieved from http://forrest.psych.unc.edu/research/index.html
  32. Young, F. W., & Bann, C. M. (1996). ViSta: A visual statistics system. In R. A. Stine & J. Fox (Eds.), Statistical computing environments for social research (pp. 207–236). Thousand Oaks: Sage Publications.Google Scholar
  33. Zand, D., Thomson, N. R., Dugan, M., Braun, J. A., Holterman-Hommes, P., & Hunter, P. L. (2006). Predictors of retention in an alcohol, tobacco, and other drug prevention study. Evaluation Review, 30, 209–222. doi:10.1177/0193841X05281160.CrossRefPubMedGoogle Scholar

Copyright information

© Society for Prevention Research 2015

Authors and Affiliations

  • Cameron R. Hopkin
    • 1
  • Rick H. Hoyle
    • 1
  • Nisha C. Gottfredson
    • 2
  1. 1.Department of Psychology and NeuroscienceDuke UniversityDurhamUSA
  2. 2.Center for Developmental Science, University of North Carolina at Chapel HillChapel HillUSA

Personalised recommendations