Abstract
Objectives
The purpose of this study is to apply an empirically derived effect size distribution to benchmark the practical magnitude of interventions aimed at reducing recidivism at the individual level.
Methods
We conducted a systematic review and meta-analysis of crime intervention evaluations. To establish a framework for benchmarking the magnitude of these interventions, we generated means, medians, tertiles, and interquartile ranges from these analyses.
Results
The results of the overall meta-analytic models revealed that crime intervention programs were associated with statistically significant reductions in recidivism regardless of outcome type (k = 74, n = 293, OR [odds ratio] = 1.42, SE = 0.05, p = 0.0001, 95% CI [1.30, 1.57]).
Conclusions
Overall, the results from the current study have several important implications for the crime prevention field. Most importantly, the study provided evidence that the tradition of using generalized guidelines for interpreting effect sizes as small/medium/large should be avoided given that they are devoid of context and ignore important variations in effects across interventions and outcomes. Moreover, this study provided an alternative framework to benchmark the practical magnitude of crime intervention programs aimed at reducing recidivism at the individual level.
This is a preview of subscription content, log in via an institution to check access.
Access this article
We’re sorry, something doesn't seem to be working properly.
Please try refreshing the page. If that doesn't work, please contact support so we can address the problem.
Similar content being viewed by others
Data availability
Much of the data that support the findings of this study are available from the corresponding author, SG, upon reasonable request. Restrictions apply to the availability of study quality data because it is not made publicly available by the National Institute of Justice. This data may be available from SG provided permission is granted by the National Institute of Justice.
Notes
A “crime intervention” is defined as any practice designed to result in less crime than would occur without the practice (Sherman et al. 1997).
“Violation” includes technical violations while an individual is on probation and parole.
In order to focus on contemporary research, CrimeSolutions originally reviewed studies published after 1980. In 2020, this date was pushed to 1990. However, programs that were reviewed prior to this change were not removed from the inventory of programs.
CS is an active and dynamic database of crime and crime-related programs that is continually updated/revised based upon the available information.
An odds ratio is a measure of association between an exposure and an outcome; it represents the odds that an outcome will occur given a particular exposure, compared with the odds of the outcome occurring in the absence of that exposure (Szumilas 2010).
These effects were derived from study IDs 42, 44, 47, and 48. See Additional 1 for study ID numbers.
CS uses a systematic methodology, similar to that proposed by Sherman et al. (1997), to rate the methodological quality of each study included in the database. The scores range from 0 to 3, with higher scores representing higher-quality studies. Using these ratings, we created a dichotomous measure of methodological quality for each study. Specifically, studies that scored 0 to 2 were categorized as lower quality, while studies that were rated greater than 2 were categorized as higher quality.
The two points that divide an ordered distribution into three parts, each containing one-third of the distribution.
The standardized mean difference effect size represents the difference in means between two groups (often represented as the difference in mean outcomes for an intervention and a comparison group) on a standardized scale, which allows for easy comparison of effects across different samples that may have different levels of variability in the outcome measure (Lipsey and Wilson 2001). Standardized mean differences are often calculated such that standardized mean differences of 0 indicate a null effect; those greater than 0 indicate a beneficial intervention effect, and those less than 0 indicate a harmful intervention effect.
A practice is a general category of programs, strategies, or procedures that share similar characteristics with regard to the issues they address and how they address them.
References
Aguinis, H., Beaty, J. C., Boik, R. J., & Pierce, C. A. (2005). Effect size and power in assessing moderating effects of categorical variables using multiple regression: a 30-year review. Journal of Applied Psychology, 90(1), 94–107. https://doi.org/10.1037/0021-9010.90.1.94
Albers, C., & Lakens, D. (2018). When power analyses based on pilot data are biased: Inaccurate effect size estimators and follow-up bias. Journal of Experimental Social Psychology, 74, 187–195. https://doi.org/10.1016/j.jesp.2017.09.004
Barnes, J. C., TenEyck, M. F., Pratt, T. C., & Cullen, F. T. (2020). How powerful is the evidence in criminology? On whether we should fear a coming crisis of confidence. Justice Quarterly, 37(3), 383–409. https://doi.org/10.1080/07418825.2018.1495252
Bloom, H. S., Hill, C. J., Black, A. R., & Lipsey, M. (2008). Performance trajectories and performance gaps as achievement effect-size benchmarks for educational interventions. Journal of Research on Educational Effectiveness, 1(4), 289–328. https://doi.org/10.1080/19345740802400072
Borenstein, M., Hedges, L. V., Higgins, J. P. T., & Rothstein, H. R. (2009). Part 2: Effect size and precision. In Introduction to meta-analysis (1st ed.). Wiley.
Bosco, F. A., Aguinis, H., Singh, K., Field, J. G., & Pierce, C. A. (2015). Correlational effect size benchmarks. Journal of Applied Psychology, 100(2), 431–449. https://doi.org/10.1037/a0038047
Chow, J., Zhao, H., Sandbank, M., Bottema–Beutel, K., & Woynaroski, T. (2022). Empirically derived effect size distributions of interventions for young children on the autism spectrum. Journal of Clinical Child & Adolescent Psychology, 0(0), 1–13. https://doi.org/10.1080/15374416.2021.2007485
Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Routledge.
Cohen, J. (1992). A power primer. Psychological Bulletin, 112(1), 155–159. https://doi.org/10.1037//0033-2909.112.1.155
Egger, M., Smith, G. D., Schneider, M., & Minder, C. (1997). Bias in meta-analysis detected by a simple, graphical test. British Medical Journal, 315(7109), 629–634. https://doi.org/10.1136/bmj.315.7109.629
Evidence-Based Policymaking Collaborative. (2016). Principles of evidence-based policymaking. Washington, D.C.: Urban Institute. https://www.urban.org/research/publication/principles-evidence-based-policymaking. Accessed 9 August 2022
Feucht, T. E., & Tyson, J. (2018). Advancing “what works” in justice: Past, present, and future work of federal justice research agencies. Justice Evaluation Journal, 1(2), 151–187. https://doi.org/10.1080/24751979.2018.1552083
Gatti, U., Tremblay, R. E., & Vitaro, F. (2009). Iatrogenic effect of juvenile justice. Journal of Child Psychology and Psychiatry, 50(8), 991–998. https://doi.org/10.1111/j.1469-7610.2008.02057.x
Gies, S., Healy, E., & Stephenson, R. (2020). The evidence of effectiveness: beyond the methodological standards. Justice Evaluation Journal. https://doi.org/10.1080/24751979.2020.1727296
Gopalakrishnan, S., & Ganeshkumar, P. (2013). Systematic reviews and meta-analysis: understanding the best evidence in primary healthcare. Journal of Family Medicine and Primary Care, 2(1), 9–14. https://doi.org/10.4103/2249-4863.109934
Griffin, T., & Miller, M. K. (2008). Child abduction, AMBER alert, and crime control theater. Criminal Justice Review, 33(2), 159–176. https://doi.org/10.1177/0734016808316778
Harris, D. N. (2009). Toward policy-relevant benchmarks for interpreting effect sizes: combining effects with costs. Educational Evaluation and Policy Analysis, 31(1), 3–29.
Hasselblad, V., & Hedges, L. V. (1995). Meta-analysis of screening and diagnostic tests. Psychological Bulletin, 117(1), 167–178. https://doi.org/10.1037/0033-2909.117.1.167
Hedberg, E. C. (2014, April 23). ROBUMETA: Stata module to perform robust variance estimation in meta-regression with dependent effect size estimates. Statistical Software Components. Boston, MA: Boston College Department of Economics. https://ideas.repec.org/c/boc/bocode/s457219.html. Accessed 9 August 2022
Hedges, L. V., Tipton, E., & Johnson, M. C. (2010). Robust variance estimation in meta-regression with dependent effect size estimates. Research Synthesis Methods, 1(1), 39–65. https://doi.org/10.1002/jrsm.5
Higgins, J. P. T., Eldridge, S., & Li, T. (2022). Chapter 23: Including variants on randomized trials. In J. P. T. Higgins, J. Thomas, J. Chandler, M. Cumpston, T. Li, M. J. Page, & V. A. Welch (Eds.), Cochrane handbook for systematic reviews of interventions version 6.3 (updated February 2022). London, England: Cochrane. Cochrane. www.training.cochrane.org/handbook
Hill, C. J., Bloom, H. S., Black, A. R., & Lipsey, M. (2007). Empirical benchmarks for interpreting effect sizes in research. Child Development Perspectives, 2(3), 172–177. https://doi.org/10.1111/j.1750-8606.2008.00061.x
Kraft, M. A. (2020). Interpreting effect sizes of education interventions. Educational Researcher, 49(4), 241–253. https://doi.org/10.3102/0013189X20912798
Lipsey, M., & Cullen, F. (2007). The effectiveness of correctional rehabilitation: a review of systematic reviews. Annual Review of Law and Social Science, 3(1), 297–320. https://doi.org/10.1146/annurev.lawsocsci.3.081806.112833
Lipsey, M., & Wilson, D. B. (2001). Practical meta-analysis. Sage Publications Inc.
Lipsey, M., Puzio, K., Yun, C., Hebert, M., Steinka–Fry, K., Cole, M., et al. (2012). Translating the statistical representation of the effects of education interventions into more readily interpretable forms (No. NCSER 2013–3000). Washington, D.C.: National Center for Special Education Research, Institute of Education Sciences, U.S. Department of Education. https://ies.ed.gov/ncser/pubs/20133000/pdf/20133000.pdf
Liverani, M., Hawkins, B., & Parkhurst, J. O. (2013). PLOS ONE, 8(10), e77404.
Loeffler, C. E., & Braga, A. A. (2022). Estimating the effects of shrinking the criminal justice system on criminal recidivism. Criminology & Public Policy, 21(3), 595–617. https://doi.org/10.1111/1745-9133.12588
Lovakov, A., & Agadullina, E. R. (2021). Empirically derived guidelines for effect size interpretation in social psychology. European Journal of Social Psychology, 51(3), 485–504. https://doi.org/10.1002/ejsp.2752
Matthay, E. C., Hagan, E., Gottlieb, L. M., Tan, M. L., Vlahov, D., Adler, N., & Glymour, M. M. (2021). Powering population health research: Considerations for plausible and actionable effect sizes. Social Science & Medicine—Population Health, 14, 100789. https://doi.org/10.1016/j.ssmph.2021.100789
McCartney, K., & Rosenthal, R. (2000). Effect size, practical importance, and social policy for children. Child Development, 71(1), 173–180. https://doi.org/10.1111/1467-8624.00131
Mears, D. P. (2007). Towards rational and evidence-based crime policy. Journal of Criminal Justice, 35(6), 667–682. https://doi.org/10.1016/j.jcrimjus.2007.09.003
Moher, D., Liberati, A., Tetzlaff, J., Altman, D. G., & PRISMA Group. (2009). Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. PLoS medicine, 6(7), e1000097. https://doi.org/10.1371/journal.pmed.1000097
Palmer, T. M., Sutton, A. J., Peters, J. L., & Moreno, S. G. (2008). Contour-enhanced funnel plots for meta-analysis. The STATA journal, 8(2), 242–254.
Paterson, T. A., Harms, P. D., Steel, P., & Credé, M. (2016). An assessment of the magnitude of effect sizes: Evidence from 30 years of meta-analysis in management. Journal of Leadership & Organizational Studies, 23(1), 66–81. https://doi.org/10.1177/1548051815614321
Pew–MacArthur Results First Initiative. (2014). Evidence-based policymaking: A guide for effective government. Washington, D.C.: Pew Charitable Trusts. https://www.pewtrusts.org/-/media/assets/2014/11/evidencebasedpolicymakingaguideforeffectivegovernment.pdf. Accessed 9 August 2022
Plonsky, L., & Oswald, F. L. (2014). How big is “big”? Interpreting effect sizes in L2 research. Language Learning, 64(4), 878–912. https://doi.org/10.1111/lang.12079
Schäfer, T., & Schwarz, M. A. (2019). The meaningfulness of effect sizes in psychological research: Differences between subdisciplines and the impact of potential biases. Frontiers in Psychology, 10, 813. https://doi.org/10.3389/fpsyg.2019.00813
Sherman, L. W., Gottfredson, D., MacKenzie, D., Eck, J., Reuter, P., & Bushway, S. (1997). Preventing crime: What works, what doesn’t, what’s promising: A report to the United States Congress. Washington, D.C.: U.S. Department of Justice, Office of Justice Programs, National Institute of Justice. https://nij.ojp.gov/library/publications/preventing-crime-what-works-what-doesnt-whats-promising-report-united-states. Accessed 9 August 2022
Spencer, T. D., Detrich, R., & Slocum, T. A. (2012). Evidence-based practice: a framework for making effective decisions. Education and Treatment of Children, 35(2), 127–151. https://doi.org/10.1353/etc.2012.0013
StataCorp. (2021). Stata Statistical Software: Release 17. Stata Corp LLC.
Szumilas, M. (2010). Explaining odds ratios. Journal of the Canadian Academy of Child and Adolescent Psychiatry, 19(3), 227–229.
Tanner-Smith, E. E., Durlak, J. A., & Marx, R. A. (2018). Empirically based mean effect size distributions for universal prevention programs targeting school-aged youth: a review of meta-analyses. Prevention Science, 19(8), 1091–1101. https://doi.org/10.1007/s11121-018-0942-1
Wakefield, S., Lee, H., & Wildeman, C. (2016). Tough on crime, tough on families? criminal justice and family life in America. ANNALS of the American Academy of Political and Social Science, 665(1), 8–21. https://doi.org/10.1177/0002716216637048
Wilson, D. B. (2022). The relative incident rate ratio effect size for count-based impact evaluations: When an odds ratio is not an odds ratio. Journal of Quantitative Criminology, 38(2), 323–341. https://doi.org/10.1007/s10940-021-09494-w
Funding
This work was supported by funding from the National Institute of Justice, Office of Justice Programs, US Department of Justice National Institute of Justice, under Contract Number 47QRAA20D002V. The content is solely the responsibility of the authors and does not necessarily represent the official views of the US Department of Justice.
Author information
Authors and Affiliations
Contributions
Conceptualization: SG, ETS, RG; data curation: SG, RG, LM; formal analysis: SG, ETS; funding acquisition: SG; investigation: SG, LM, ETS; methodology: SG, RG, ETS; project administration: SG; resources: SG; supervision: SG, ETS; writing—original draft: SG, ETS; writing—reviewing and editing: SG, LM, FM, RG, ETS.
Corresponding author
Ethics declarations
Ethical approval
This article does not contain any studies with human participats or animals performed by any of the authors.
Consent to participate
For this type of study, informed consent is not required.
Conflict of interest
DSG is the contractor for the National Institute of Justice’s CrimeSolutions Intervention Assessment and Content Development Services project.
Additional information
Publisher's note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Supplementary Information
Rights and permissions
Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
About this article
Cite this article
Gies, S.V., Nichols, L.M., Mojekwu, F. et al. Applying an empirically derived effect size distribution to benchmark the practical magnitude of interventions to reduce recidivism in the USA. J Exp Criminol (2023). https://doi.org/10.1007/s11292-023-09561-1
Accepted:
Published:
DOI: https://doi.org/10.1007/s11292-023-09561-1