Selection Effects and Prevention Program Outcomes
A primary goal of the paper is to provide an example of an evaluation design and analytic method that can be used to strengthen causal inference in nonexperimental prevention research. We used this method in a nonexperimental multisite study to evaluate short-term outcomes of a preventive intervention, and we accounted for effects of two types of selection bias: self-selection into the program and differential dropout. To provide context for our analytic approach, we present an overview of the counterfactual model (also known as Rubin's causal model or the potential outcomes model) and several methods derived from that model, including propensity score matching, the Heckman two-step approach, and full information maximum likelihood based on a bivariate probit model and its trivariate generalization. We provide an example using evaluation data from a community-based family intervention and a nonexperimental control group constructed from the Washington State biennial Healthy Youth Survey (HYS) risk behavior data (HYS n = 68,846; intervention n = 1,502). We identified significant effects of participant, program, and community attributes in self-selection into the program and program completion. Identification of specific selection effects is useful for developing recruitment and retention strategies, and failure to identify selection may lead to inaccurate estimation of outcomes and their public health impact. Counterfactual models allow us to evaluate interventions in uncontrolled settings and still maintain some confidence in the internal validity of our inferences; their application holds great promise for the field of prevention science as we scale up to community dissemination of preventive interventions.
KeywordsSelection effects Translational research Universal prevention Family-focused interventions Causal inference Observational research Nonexperimental research
This study was supported in part by the National Institute of Drug Abuse (grants R21 DA025139-01Al and R21 DA19758-01). We thank the Washington State Department of Health for providing the supplementary data sample, and we thank the program providers and families who participated in the program evaluation.
- Arendt, J.N. & Holm, A. (2006). Probit models with binary endogenous regressors (working paper 4/2006). Retrieved from Department of Business and Economics at the University of Southern Denmark website: http://static.sdu.dk/mediafiles/Files/Om_SDU/Institutter/Ivoe/Disc_papers/Disc_2006/dpbe4%202006%20pdf.pdf. Accessed 23 Oct 2012.
- Berinsky, A. (2004). Silent voices: Opinion polls and political representation in America. Princeton: Princeton University Press.Google Scholar
- Cook, T. D., & Steiner, P. M. (2010). Case matching and the reduction of selection bias in quasi experiments: The relative importance of pretest measures of outcome, of unreliable measurement, and of mode of data analysis. Psychological Methods, 15, 56–68. doi: 10.1037/a0018536.CrossRefPubMedGoogle Scholar
- Heckman, J. J., Ichimura, H., Smith, J., & Todd, P. (1996). Sources of selection bias in evaluating social programs: An interpretation of conventional measures and evidence on the effectiveness of matching as a program evaluation method. Proceedings of the National Acadamy of Science, 93, 13416–13420.CrossRefGoogle Scholar
- Kumpfer, K. L., Molgaard, V., & Spoth, R. (1996). The Strengthening Families Program for the prevention of delinquency and drug use. In R. D. V. Peters & R. J. McMahon (Eds.), Preventing childhood disorders, substance abuse, and delinquency. Banff International Behavioral Science Series (Vol. 3) (pp. 241–267). Thousand Oaks: Sage Publications.CrossRefGoogle Scholar
- Neyman, J. (1990). On the application of probability theory to agricultural experiments: Essay on principles. Section 9. Statistical Science, 5, 465–480.Google Scholar
- Pearl, J. (2000). Causality: Models, reasoning, and inference. Cambridge: Cambridge University Press.Google Scholar
- Roodman, D.M. (2007). CMP: Stata module to implement conditional (recursive) mixed process estimator. Statistical software components. http://ideas.repec.org/c/boc/bocode/s456882.html. Accessed 23 Oct 2012.
- Roodman, D. (2009). Estimating fully observed recursive mixed-process models with cmp. http://www.cgdev.org/content/publications/detail/1421516. Accessed 23 Oct 2012.
- Rosenman, R., Mandal, B., Tennekoon, V., & Hill, L.G. (2010). Estimating treatment effectiveness with sample selection (working paper 2010-05). Retrieved from School of Economic Sciences at Washington State University website: http://faculty.ses.wsu.edu/WorkingPapers/Rosenman/WP2010-5.pdf. Accessed 23 Oct 2012.
- Shadish, W. R., Cook, T. D., & Campbell, D. T. (2002). Experimental and quasi-experimental designs for generalized causal inference. Boston: Houghton, Mifflin and Company.Google Scholar
- US Department of Labor, Bureau of Labor Statistics (2012). http://www.bls.gov/data/#unemployment. Accessed 23 Oct 2012.
- Washington Office of Financial Management (2010). http://www.ofm.wa.gov/localdata/default.asp. Accessed 23 Oct 2012.
- Washington State Department of Health (2006). Washington State Healthy Youth Survey. http://www.doh.wa.gov/Portals/1/Documents/Pubs/WashingtonStateHYS2006.pdf. Accessed 23 Oct 2012.
- West, S. G., & Thoemmes, F. (2010). Campbell’s and Rubin’s perspectives on causal inference. Psychological Methods, 15, 18–37Google Scholar