Skip to main content
Log in

When is the Story in the Subgroups?

Strategies for Interpreting and Reporting Intervention Effects for Subgroups

  • Published:
Prevention Science Aims and scope Submit manuscript

Abstract

This paper examines strategies for interpreting and reporting estimates of intervention effects for subgroups of a study sample. The paper considers: why and how subgroup findings are important for applied research, alternative ways to define subgroups, different research questions that motivate subgroup analyses, the importance of pre-specifying subgroups before analyses are conducted, the importance of using existing theory and prior research to distinguish between subgroups for whom study findings are confirmatory (hypothesis testing) as opposed to exploratory (hypothesis generating), and the conditions under which study findings should be considered confirmatory. Each issue is illustrated by selected empirical examples.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Subscribe and save

Springer+ Basic
EUR 32.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or Ebook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Similar content being viewed by others

Notes

  1. As with any attempt to collapse a continuous construct based on multiple considerations (like strength of scientific evidence) into a dichotomy (confirmatory versus exploratory findings or strong versus weak evidence), it is not possible to distinguish between the resulting categories in a way that fits all possible situations. Hence in practice, the operational distinction must remain somewhat vague and its application to specific cases will require professional judgment.

  2. Our recommendations are designed to minimize the risk of Type I error in statistical hypothesis testing about intervention effects for subgroups. We recognize that other things being equal, reducing the risk of Type I error (wrongly emphasizing a subgroup finding that is not real or important) increases the risk of Type II error (wrongly not emphasizing a subgroup finding that is real and important). We also acknowledge that there is no consensus about how to balance the tradeoff between these two types of errors.

  3. The study is being funded by the Administration for Children and Families and the Office of the Assistant Secretary for Planning and Evaluation of the U.S. Department of Health and Human Services.

  4. By conditional risk we mean the risk or probability of obtaining an impact estimate that is statistically significant when the true impact is zero.

References

  • Benjamini, Y., & Hochberg, Y. (1995). Controlling the false discovery rate: A practical and powerful approach to multiple testing. Journal of the Royal Statistical Society, 57, 289–300.

    Google Scholar 

  • Bloom, D. (2009). The Joyce Foundation’s transitional jobs reentry demonstration: Testing strategies to help former prisoners find and keep jobs and stay out of prison. Chicago, IL: The Joyce Foundation.

    Google Scholar 

  • Bloom, D., Redcross, C., Zweig, J., & Azurdia, G. (2007). Transitional jobs for ex-prisoners: Early impacts from a random assignment evaluation of the Center for Employment Opportunities (CEO) prisoner reentry program. New York: MDRC.

    Google Scholar 

  • Bloom, H. S., Thompson, S. L., & Unterman, R. (2010). Transforming the high school experience: How New York city’s new small schools are boosting student achievement and graduation rates. New York: MDRC.

    Google Scholar 

  • Council, N. R. (2007). Crime, parole, desistance from crime, and community integration. Committee on Community Supervision and Desistance from Crime. Washington, DC: The National Academies Press.

    Google Scholar 

  • Fournier, J. C., DeRubeis, R. J., Hollon, S. D., Dimidjian, S., Amsterdam, J. D., Shelton, R. C., et al. (2010). Antidepressant drug effects and depression severity: A patient-level meta-analysis. Journal of the American Medical Association, 303, 47–53.

    Article  PubMed  CAS  Google Scholar 

  • Kim, S., Leblanc, A., & Michalopoulos, C. (2009). Working toward Wellness: Early results from a telephonic care management program for medicaid recipients with depression. New York: MDRC.

    Google Scholar 

  • Kim, S., Leblanc, A., Simon, G., Walter, J., & Michalopoulos, C. (2010). Working toward Wellness: Eighteen-month results from a telephonic care management program for medicaid recipients with depression. New York: MDRC.

    Google Scholar 

  • Michalopoulos, C., & Schwartz, C. (2000). What works best for whom: Impacts of 20 welfare-to-work programs by subgroup. Washington, DC: U.S. Department of Health and Human Services, Office of the Assistant Secretary for Planning and Evaluation and Administration for Children and Families, and U.S. Department of Education.

    Google Scholar 

  • Redcross, C., Bloom, D., Azurdia, G., Zweig, J., & Pindus, N. (2007). Transitional jobs for ex-prisoners: Implementation, two-year impacts, and costs of the Center for Employment Opportunities (CEO) prisoner reentry program. New York: MDRC.

    Google Scholar 

  • Rothwell, P. M. (2005). Subgroup analysis in randomised control trials: Importance, indications and interpretation. The Lancet, 365, 176–186.

    Article  Google Scholar 

  • Schochet, P. M. (2008). Technical methods report: Guidelines for multiple testing in impact evaluations. Washington, DC: U.S. Department of Education, Institute of Education Sciences.

    Google Scholar 

  • Wells, K. B., Sherbourne, C., Schoenbaum, M., Duan, N., Meredith, L., Unutzer, J., et al. (2000). Impact of disseminating quality improvement programs for depression in managed primary care: A randomized controlled trial. Journal of American Medical Association, 283, 212–220.

    Article  CAS  Google Scholar 

  • Wells, K., Sherbourne, C., Schoenbaum, M., Ettner, S., Duan, N., Miranda, J., et al. (2004). Five-year impact of quality improvement for depression: Results of a group-level randomized controlled trial. Archives of General Psychiatry, 61, 378–386.

    Article  PubMed  Google Scholar 

Download references

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Howard S. Bloom.

Additional information

This paper was supported by funding from the W.T. Grant Foundation and the Judith Gueron Fund for Methodological Innovation in Social Policy Research at MDRC.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Bloom, H.S., Michalopoulos, C. When is the Story in the Subgroups?. Prev Sci 14, 179–188 (2013). https://doi.org/10.1007/s11121-010-0198-x

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11121-010-0198-x

Keywords

Navigation