Abstract
This paper examines strategies for interpreting and reporting estimates of intervention effects for subgroups of a study sample. The paper considers: why and how subgroup findings are important for applied research, alternative ways to define subgroups, different research questions that motivate subgroup analyses, the importance of pre-specifying subgroups before analyses are conducted, the importance of using existing theory and prior research to distinguish between subgroups for whom study findings are confirmatory (hypothesis testing) as opposed to exploratory (hypothesis generating), and the conditions under which study findings should be considered confirmatory. Each issue is illustrated by selected empirical examples.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
Notes
As with any attempt to collapse a continuous construct based on multiple considerations (like strength of scientific evidence) into a dichotomy (confirmatory versus exploratory findings or strong versus weak evidence), it is not possible to distinguish between the resulting categories in a way that fits all possible situations. Hence in practice, the operational distinction must remain somewhat vague and its application to specific cases will require professional judgment.
Our recommendations are designed to minimize the risk of Type I error in statistical hypothesis testing about intervention effects for subgroups. We recognize that other things being equal, reducing the risk of Type I error (wrongly emphasizing a subgroup finding that is not real or important) increases the risk of Type II error (wrongly not emphasizing a subgroup finding that is real and important). We also acknowledge that there is no consensus about how to balance the tradeoff between these two types of errors.
The study is being funded by the Administration for Children and Families and the Office of the Assistant Secretary for Planning and Evaluation of the U.S. Department of Health and Human Services.
By conditional risk we mean the risk or probability of obtaining an impact estimate that is statistically significant when the true impact is zero.
References
Benjamini, Y., & Hochberg, Y. (1995). Controlling the false discovery rate: A practical and powerful approach to multiple testing. Journal of the Royal Statistical Society, 57, 289–300.
Bloom, D. (2009). The Joyce Foundation’s transitional jobs reentry demonstration: Testing strategies to help former prisoners find and keep jobs and stay out of prison. Chicago, IL: The Joyce Foundation.
Bloom, D., Redcross, C., Zweig, J., & Azurdia, G. (2007). Transitional jobs for ex-prisoners: Early impacts from a random assignment evaluation of the Center for Employment Opportunities (CEO) prisoner reentry program. New York: MDRC.
Bloom, H. S., Thompson, S. L., & Unterman, R. (2010). Transforming the high school experience: How New York city’s new small schools are boosting student achievement and graduation rates. New York: MDRC.
Council, N. R. (2007). Crime, parole, desistance from crime, and community integration. Committee on Community Supervision and Desistance from Crime. Washington, DC: The National Academies Press.
Fournier, J. C., DeRubeis, R. J., Hollon, S. D., Dimidjian, S., Amsterdam, J. D., Shelton, R. C., et al. (2010). Antidepressant drug effects and depression severity: A patient-level meta-analysis. Journal of the American Medical Association, 303, 47–53.
Kim, S., Leblanc, A., & Michalopoulos, C. (2009). Working toward Wellness: Early results from a telephonic care management program for medicaid recipients with depression. New York: MDRC.
Kim, S., Leblanc, A., Simon, G., Walter, J., & Michalopoulos, C. (2010). Working toward Wellness: Eighteen-month results from a telephonic care management program for medicaid recipients with depression. New York: MDRC.
Michalopoulos, C., & Schwartz, C. (2000). What works best for whom: Impacts of 20 welfare-to-work programs by subgroup. Washington, DC: U.S. Department of Health and Human Services, Office of the Assistant Secretary for Planning and Evaluation and Administration for Children and Families, and U.S. Department of Education.
Redcross, C., Bloom, D., Azurdia, G., Zweig, J., & Pindus, N. (2007). Transitional jobs for ex-prisoners: Implementation, two-year impacts, and costs of the Center for Employment Opportunities (CEO) prisoner reentry program. New York: MDRC.
Rothwell, P. M. (2005). Subgroup analysis in randomised control trials: Importance, indications and interpretation. The Lancet, 365, 176–186.
Schochet, P. M. (2008). Technical methods report: Guidelines for multiple testing in impact evaluations. Washington, DC: U.S. Department of Education, Institute of Education Sciences.
Wells, K. B., Sherbourne, C., Schoenbaum, M., Duan, N., Meredith, L., Unutzer, J., et al. (2000). Impact of disseminating quality improvement programs for depression in managed primary care: A randomized controlled trial. Journal of American Medical Association, 283, 212–220.
Wells, K., Sherbourne, C., Schoenbaum, M., Ettner, S., Duan, N., Miranda, J., et al. (2004). Five-year impact of quality improvement for depression: Results of a group-level randomized controlled trial. Archives of General Psychiatry, 61, 378–386.
Author information
Authors and Affiliations
Corresponding author
Additional information
This paper was supported by funding from the W.T. Grant Foundation and the Judith Gueron Fund for Methodological Innovation in Social Policy Research at MDRC.
Rights and permissions
About this article
Cite this article
Bloom, H.S., Michalopoulos, C. When is the Story in the Subgroups?. Prev Sci 14, 179–188 (2013). https://doi.org/10.1007/s11121-010-0198-x
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11121-010-0198-x