Abstract
Objectives
A 2010 article by the Conduct Problems Prevention Research Group considers an important social and public policy problem: can early psychosocial intervention prevent delinquency? The article examines the effects of the Fast Track preventive intervention on youth arrests and self-reported delinquent behavior through age 19. The article reports that the intervention reduced court-recorded juvenile arrest activity as well as a range of other disparate effects.
Methods
This comment assesses the methodology employed in the article.
Results
The original article suffers from a range of methodological problems. First and foremost, the article includes a large number of statistical tests and highlights only a subset that are statistically significant. As this article demonstrates, these findings likely involve “false discoveries” or chance findings. Uncertainty about the study’s findings is increased still further by problems of randomization, the treatment of site, the handling of missing data, and the inclusion of a collider as a covariate in key analyses. A proper assessment of the study’s meaning and implications has been impeded by inaccuracies in how the study’s methodology has been described over time.
Conclusions
The original article offers chance findings, suffers from methodological errors and builds on a flawed study design. As a result, it is impossible to conclude that “that a comprehensive preventive intervention can prevent juvenile arrest rates”. What the intervention would accomplish were it implemented in a new community is unknown.
This is a preview of subscription content, log in via an institution to check access.
Notes
Only the values of observations close to this percentile contribute to its estimation.
In fairness to the authors, their description of randomization has become more accurate over time. Original study reports indicated that the study randomly assigned schools: “Because part of the intervention (described below) involved a school-based intervention, we assigned entire schools (n = 54) to either the intervention condition or the control condition” (Conduct Problems Prevention Research Group,1999, p. 634).
Indeed, observational methods such as propensity scores may balance observed factors more effectively than randomization (Freedman et al. 2009).
That study indicates that “Children's sum scores on the two screening measures (teacher and parent ratings of behavior problems) were averaged. Children who fell in the top 10 % at each site on the combined screen ("high risk") were invited to participate in the longitudinal study”. (Conduct Problems Prevention Research Group 1999, p. 634)
One irony of the study is that the ability to examine the high-risk interaction highlighted in the paper is that it involves comparing children originally targeted with those children added to the study to reach recruitment goals or for some other purpose. Note that Foster (2010) finds that the interaction between initial risk status does not generalize across outcomes.
The authors seem to reverse themselves in an erratum subsequently published in the journal.
This problem has been known for some time; the clearest recent statement can be found in Sobel (2008). In other words, the standard treatment of mediation assumes that—conditional on the covariates—detention is determined as if it is randomly assigned.
There are several archetypical examples used to illustrate this point. In one, the outcome of interest is the event that the golf course is wet. There are two causes—rain and a sprinkler system. The sprinkler system is on a timer and runs every other day, whether it rains or not. It runs even on rainy days. However, if one limits one’s analyses to when the golf course is wet, there is a spurious relationship between rain and the sprinkler running. In particular, on those days, the chance that the sprinkler ran on days when it did not rain is 100 %. If it rained, then the chance that the sprinkler ran is 50 %.
References
Baron, R. M., & Kenny, D. A. (1986). The moderator-mediator variable distinction in social psychological research: Conceptual, strategic, and statistical considerations. Journal of Personality and Social Psychology, 51(6), 1173–1182.
Benjamini, Y., & Hochberg, Y. (1995). Controlling the false discovery rate: a practical and powerful approach to multiple testing. Journal of the Royal Statistical Society. Series B (Methodological), 289–300.
Benjamini, Y., & Yekutieli, D. (2001). The control of the false discovery rate in multiple testing under dependency. The Annals of Statistics, 29(4), 1165–1188.
Cohen, M. A. (2005). The costs of crime and justice. New York: London.
Cole, S. R., & Hernan, M. A. (2008). Constructing inverse probability weights for marginal structural models. American Journal of Epidemiology, 168(6), 656–664.
Conduct Problems Prevention Research Group. (1999). Initial impact of the Fast Track prevention trial for conduct problems: I The High-risk sample. Journal of Consulting and Clinical Psychology, 67(5), 631–647.
Conduct Problems Prevention Research Group. (2010). Fast Track intervention effects on youth arrests and delinquency. Journal of Experimental Criminology, 6(2), 131–157.
Dixon, D. O., & Simon, R. (1991). Bayesian subset analysis. Biometrics, 47(3), 871–881.
Dixon, D. O., & Simon, R. (1992). Bayesian subset analysis in a colorectal cancer clinical trial. Statistics in Medicine, 11(1), 13–22.
Efron, B., & Gong, G. (1983). A leisurely look at the bootstrap, the jackknife, and cross-validation. The American Statistician, 37(1), 36–48.
Foster, E. M. (2010). Costs and effectiveness of the Fast Track intervention for antisocial behavior. The Journal of Mental Health Policy and Economics, 13(3), 101–119.
Foster, E. M., Jones, D., & Conduct Problems Prevention Research Group. (2006). Can a costly intervention be cost-effective?: An analysis of violence prevention. Archives of General Psychiatry, 63, 1284–1291.
Freedman, D., Collier, D., Sekhon, J. S., & Stark, P. B. (2009). Statistical models and causal inference: a dialogue with the social sciences. New York: Cambridge University Press.
Robins, J. M. (1997). Marginal structural models. Paper presented at the 1997 American Statistical Association, Section on Bayesian Statistical Science, Washington, DC.
Robins, J. M. (1999). Marginal structural models versus structural nested models as tools for causal inference. Statistical Models in Epidemiology: The Environment and Clinical Trials, 16, 95–134.
Robins, J. M., Blevins, D., Ritter, G., & Wulfsohn, M. (1992). G-estimation of the effect of prophylaxis therapy for Pneumocystis carinii pneumonia on the survival of AIDS patients. Epidemiology, 3(4), 319–336.
Sato, T., & Matsuyama, Y. (2003). Marginal structural models as a tool for standardization. Epidemiology, 14(6), 680–686.
Simon, R. (2002). Bayesian subset analysis: application to studying treatment-by-gender interactions. Statistics in Medicine, 21(19), 2909–2916.
Sobel, M. E. (2008). Identification of causal parameters in randomized studies with mediating variables. Journal of Educational and Behavioral Statistics, 33(2), 230–251.
Author information
Authors and Affiliations
Corresponding author
Additional information
This article is commentary for article “Fast track intervention effects on youth arrests and delinquency”, published in Volume 6, Issue 2, under doi:10.1007/s11292-010-9091-7.
Rights and permissions
About this article
Cite this article
Foster, E.M. Reassessing findings from the Fast Track study: problems of method and analysis. J Exp Criminol 9, 109–117 (2013). https://doi.org/10.1007/s11292-012-9172-x
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11292-012-9172-x