Can Machine Learning Improve Screening for Targeted Delinquency Prevention Programs?
- 31 Downloads
The cost-effectiveness of targeted delinquency prevention programs for children depends on the accuracy of the screening process. Screening accuracy is often poor, resulting in wasted resources and missed opportunities to avert negative outcomes. This study examined whether screening approaches based on logistic regression or machine learning algorithms could improve accuracy relative to traditional sum-score approaches when identifying boys in the 5th grade (N = 1012) who would be repeatedly arrested for violent and serious crimes from ages 13 to 30. Screening algorithms were developed that incorporated facets of teacher-reported externalizing problems and other known risk factors (e.g., peer rejection). The predictive performance of these algorithms was evaluated and compared in holdout (i.e., test) data using the area under the receiver operating curve (AUROC) and Brier score. Both the logistic and machine learning methods yielded AUROC superior to traditional sum-score screening approaches when a broad set of risk factors for future delinquency was considered. However, this improvement was modest and was not present when using item-level information from a composite scale assessing externalizing problems. Contrary to expectations, machine learning algorithms performed no better than simple logistic models. There was a large apparent advantage of machine learning that disappeared after appropriate cross-validation, underscoring the importance of careful evaluation of these methods. Results suggest that screening using logistic regression could improve the cost-effectiveness of targeted delinquency prevention programs in some cases, but screening using machine learning would confer no marginal benefit under currently realistic conditions.
KeywordsViolence Delinquency Prevention Machine learning
This research was funded by National Institute of Child Health and Human Development grant HD092094. Additional support was provided by grants from the National Institute on Drug Abuse (DA039772, DA009757, DA041713) and National Institute on Alcohol Abuse and Alcoholism (AA026768).
Compliance with Ethical Standards
Conflict of Interest
The authors declare that they have no conflicts of interest.
All procedures performed in studies involving human participants were in accordance with the ethical standards of the institutional and/or national research committee and with the 1964 Helsinki declaration and its later amendments or comparable ethical standards.
Informed consent/assent was obtained from all participants in this study.
- Achenbach, T. M. (1991). Manual for the teacher’s report form and 1991 profile. Burlington: University of Vermont, Department of Psychiatry.Google Scholar
- Bureau of Justice Statistics. (2015). Justice expenditure and exployment extracts, 2012 - Preliminary (no. NCJ 248628). U.S. Department of Justice.Google Scholar
- Davison, A. C., & Hinkley, D. V. (1997). Bootstrap Methods and Their Application. Cambridge: Cambridge University Press.Google Scholar
- Federal Bureau of Investigation. (2017). Uniform crime report: Crime in the United States, 2016. Department of Justice: Washington D.C..Google Scholar
- Hawes, S. W., Perlman, S. B., Byrd, A. L., Raine, A., Loeber, R., & Pardini, D. A. (2016). Chronic anger as a precursor to adult antisocial personality features: The moderating influence of cognitive control. Journal of Abnormal Psychology, 125, 64–74.Google Scholar
- Lochman, J. E., Boxmeyer, C. L., Powell, N. P., Barry, T. D., & Pardini, D. A. (2010). Anger control training for aggressive youths. In J. R. Weisz & A. E. Kazdin (Eds.), Evidence based psychotherapies for children and adolescents (2nd ed., pp. 227–242).Google Scholar
- Lochman, J. E., Dishion, T. J., Powell, N. P., Boxmeyer, C. L., Qu, L., & Sallee, M. (2015). Evidence-based preventive intervention for preadolescent aggressive children: One-year outcomes following randomization to group versus individual delivery. Journal of Consulting and Clinical Psychology, 83, 728–735.Google Scholar
- Mason, S. J., & Graham, N. E. (2002). Areas beneath the relative operating characteristics (ROC) and relative operating levels (ROL) curves: Statistical significance and interpretation. Quarterly Journal of the Royal Meteorological Society, 128, 2145–2166.Google Scholar
- O’Connell, M. E., Boat, T., & Warner, K. E. (Eds.). (2009). Preventing Mental, Emotional, and Behavioral Disorders Among Young People: Progress and Possibilities. Washington: National Academies Press.Google Scholar
- Petras, H., Chilcoat, H. D., Leaf, P. J., Ialongo, N. S., & Kellam, S. G. (2004). Utility of TOCA-R scores during the elementary school years in identifying later violence among adolescent males. Journal of the American Academy of Child & Adolescent Psychiatry, 43, 88–96.Google Scholar
- Petras, H., Buckley, J. A., Leoutsakos, J.-M. S., Stuart, E. A., & Ialongo, N. S. (2013). The use of multiple versus single assessment time points to improve screening accuracy in identifying children at risk for later serious antisocial behavior. Prevention Science, 14, 423–436.CrossRefGoogle Scholar
- R Core Team (2018). R: A language and environment for statistical computing. Vienna, Austria: R Foundating for Statistical Computing.Google Scholar