Skip to main content
SpringerLink
Log in

Missing links: how descriptive validity impacts the policy relevance of randomized controlled trials in criminology

  • Published:
Journal of Experimental Criminology Aims and scope Submit manuscript

Abstract

Objectives

To assess quality of reporting of issues that may affect internal and external validity in randomized controlled trials (RCTs) in criminology, and explore the impact of reporting quality (descriptive validity) on the policy relevance of rigorous research.

Methods

Reporting indicators based on CONSORT standards from the health sciences are constructed and applied to a sample of 38 RCTs, covering a range of criminal justice interventions, published in journals between 2002 and 2008. A Descriptive Validity Matrix is constructed to visually convey information about reporting quality across a group of studies, based on the reporting indicators, to decision-makers.

Results

Criminological RCTs are moderately well-reported. The sample of studies show medium descriptive validity in reporting on elements relevant to internal validity, and high descriptive validity for items relevant to external validity. However, there was considerable variation in the quality of reporting on key issues, especially those related to implementation of the random assignment sequence, deviations from the planned study, and attrition of participants.

Conclusions

This study and the Descriptive Validity Matrix provide a useful framework for assessing descriptive validity. Although the indicators developed were not specific to criminology, and the analysis was limited to a small number of studies published in academic journals, this study is an important starting point for continued research and discussion on the relationship between implementation of field experimentation, reporting quality, and policymaking. The ability to report research clearly is as important as choosing the most rigorous research design for enhancing the objectives of evidence-based crime policy.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8

Similar content being viewed by others

Notes

  1. The third and fourth forms of validity described by Cook and Campbell (1979) are statistical conclusion validity (the relationship between cause and effect, and the ability of the research design to identify it), and construct validity (the adequacy of operational definitions and measurement).

  2. http://www.campbellcollaboration.org/ .

  3. I am grateful to an anonymous peer reviewer for this observation.

  4. This observation was also pointed out by an anonymous peer reviewer.

  5. ISI Web of Knowledge, 2007 Journal Citation Reports, Social Science Edition. Access provided through University of Pennsylvania Libraries, March 2009.

  6. The Journal of Experimental Criminology is an obvious starting point in a search for RCTs. However, because the journal was founded more recently (2005), it was too new at the time of this research to be included in the Journal Citation Reports.

  7. When information about the lead author was not presented within the publication, I obtained it through web searches of publicly-available details (e.g., faculty Web pages, biographies, and CVs).

  8. Perry, Weisburd, and Hewitt (2010) suggest that more rigorous practices may have developed in certain criminal justice domains, perhaps due to funding availability or the backgrounds of researchers who work in those areas.

  9. Lynette Feder, John Goldkamp, Denise Gottfredson, Doris MacKenzie, Edmund McGarrell, Jonathan Shepherd, Faye Taxman, Susan Turner, Patricia van Voorhis, and David Weisburd. A list of AEC Fellows is published at http://www.crim.upenn.edu/aec/fellows.htm.

  10. David Weisburd; Jonathan Shepherd and Peter van der Laan were co-authors of two additional studies. A list of Campbell Collaboration Crime and Justice Group steering committee members is published at http://www.campbellcollaboration.org/crime_and_justice_our_group/Who_s_involved_CJ.php.

References

* denotes a report of an experiment included in the study

  • Altman, D. G., Schultz, K. F., Moher, D., Egger, M., Davidoff, F., Elbourne, D., et al. (2001). The revised CONSORT statement for reporting randomized trials: Explanation and elaboration. Annals of Internal Medicine, 134(8), 663–694.

    Google Scholar 

  • *Armstrong, T. A. (2002). The effect of environment on the behavior of youthful offenders: A randomized experiment. Journal of Criminal Justice, 30, 19–28.

    Article  Google Scholar 

  • *Armstrong, T. A. (2003). The effect of moral reconation therapy on the recidivism of youthful offenders: A randomized experiment.

  • *Banks, D., & Gottfredson, D. C. (2004). Participation in drug treatment court and time to rearrest. Justice Quarterly, 21(3), 637–658.

    Article  Google Scholar 

  • Barnes, G. C., Ahlman, L., Gill, C., Sherman, L. W., Kurtz, E., & Malvestuto, R. (2010). Low-intensity community supervision for low-risk offenders: A randomized, controlled trial. Journal of Experimental Criminology, 6, 159–189.

    Article  Google Scholar 

  • Berk, R. A., & Rossi, P. H. (1999). Thinking about program evaluation (2nd ed.). Thousand Oaks, CA: Sage.

    Google Scholar 

  • *Biggam, F. H., & Power, K. G. (2002). A controlled, problem-solving, group-based intervention with vulnerable incarcerated young offenders. International Journal of Offender Therapy and Comparative Criminology, 46(6), 678–698.

    Article  Google Scholar 

  • *Bijleveld, C. (2007). Fare dodging and the strong arm of the law: An experimental evaluation of two different penalty schemes for fare evasion. Journal of Experimental Criminology, 3, 183–199.

    Article  Google Scholar 

  • *Blais, E., & Bacher, J. L. (2007). Situational deterrence and claim padding: Results from a randomized field experiment. Journal of Experimental Criminology, 3, 337–352.

    Article  Google Scholar 

  • Boruch, R. F. (1997). Randomized experiments for planning and evaluation: A practical guide. Thousand Oaks, CA: Sage.

    Google Scholar 

  • Boruch, R., Snyder, B., & DeMoya, D. (2000). The importance of randomized field trials. Crime and Delinquency, 46(2), 156–180.

    Article  Google Scholar 

  • *Bottcher, J., & Ezell, M. E. (2005). Examining the effectiveness of boot camps: A randomized experiment with a long-term follow up. Journal of Research in Crime and Delinquency, 42(3), 309–332.

    Article  Google Scholar 

  • Braga, A. A., & Bond, B. J. (2008). Policing crime and disorder hot spots: A randomized controlled trial. Criminology, 46(3), 577–607.

    Article  Google Scholar 

  • Brown, S. E. (1989). Statistical power and criminal justice research. Journal of Criminal Justice, 17, 115–122.

    Article  Google Scholar 

  • Cook, T. D., & Campbell, D. T. (1979). Quasi-experimentation: Design and analysis issues for field settings. Boston, MA: Houghton Mifflin.

    Google Scholar 

  • Farrington, D. P. (2003a). Methodological quality standards for evaluation research. Annals of the American Academy of Political and Social Science, 587 (May), 49-58.

    Google Scholar 

  • Farrington, D. P. (2003b). A short history of randomized experiments in criminology: A meager feast. Evaluation Review, 27(3), 218–227.

    Article  Google Scholar 

  • Farrington, D. P., Gottfredson, D. C., Sherman, L. W., & Welsh, B. C. (2006). The Maryland Scientific Methods Scale. In L. W. Sherman, D. P. Farrington, B. C. Welsh, & D. L. MacKenzie (Eds.), Evidence-based crime prevention (2nd ed., pp. 13–21). New York, NY: Routledge.

    Google Scholar 

  • Farrington, D. P., & Welsh, B. C. (2005). Randomized experiments in criminology: What have we learned in the last two decades? Journal of Experimental Criminology, 1, 9–38.

    Article  Google Scholar 

  • *Feder, L., & Dugan, L. (2002). A test of the efficacy of court-mandated counseling for domestic violence offenders: The Broward experiment. Justice Quarterly, 19(2), 343–375.

    Article  Google Scholar 

  • *Feske, U. (2008). Treating low-income and minority women with posttraumatic stress disorder: A pilot study comparing prolonged exposure and treatment as usual conducted by community therapists. Journal of Interpersonal Violence, 23(8), 1027–1040.

    Article  Google Scholar 

  • *Giblin, M. J. (2002). Using police officers to enhance the supervision of juvenile probationers: An evaluation of the Anchorage CAN program. Crime and Delinquency, 48(1), 116–137.

    Article  Google Scholar 

  • Goldkamp, J. S. (2008). Missing the target and missing the point: ‘Successful’ random assignment but misleading results. Journal of Experimental Criminology, 4, 83–115.

    Article  Google Scholar 

  • *Goldkamp, J. S., & White, M. D. (2006). Restoring accountability in pretrial release: the Philadelphia pretrial release supervision experiments. Journal of Experimental Criminology, 2, 143–181.

    Article  Google Scholar 

  • *Gondolf, E. W. (2007). Culturally-focused batterer counseling for African-American men. Criminology and Public Policy, 6(2), 341–366.

    Article  Google Scholar 

  • *Gottfredson, D. C., & Exum, M. L. (2002). The Baltimore City drug treatment court: One-year results from a randomized study. Journal of Research in Crime and Delinquency, 39(3), 337–356.

    Article  Google Scholar 

  • *Gottfredson, D. C., Najaka, S. S., & Kearley, B. (2003). Effectiveness of drug treatment courts: Evidence from a randomized trial. Criminology and Public Policy, 2(2), 171–196.

    Article  Google Scholar 

  • *Gottfredson, D. C., Najaka, S. S., Kearley, B. W., & Rocha, C. M. (2006). Long-term effects of participation in the Baltimore City drug treatment court: Results from an experimental study. Journal of Experimental Criminology, 2, 67–98.

    Article  Google Scholar 

  • *Haapanen, R., & Britton, L. (2002). Drug testing for youthful offenders on parole: An experimental evaluation. Criminology and Public Policy, 1(2), 217–244.

    Article  Google Scholar 

  • *Kilmer, B. (2008). Does parolee drug testing influence employment and education outcomes? Evidence from a randomized experiment with noncompliance. Journal of Quantitative Criminology, 24, 93–123.

    Article  Google Scholar 

  • *Kinlock, T. W., Gordon, M. S., Schwartz, R. P., & O’Grady, K. E. (2008). A study of methadone maintenance for male prisoners: 3-month postrelease outcomes. Criminal Justice and Behavior, 35(1), 34–47.

    Article  Google Scholar 

  • *Labriola, M., Rempel, M., & Davis, R. C. (2008). Do batterer programs reduce recidivism? Results from a randomized trial in the Bronx. Justice Quarterly, 25(2), 252–282.

    Article  Google Scholar 

  • *Lane, J., Turner, S., Fain, T., & Seghal, A. (2005). Evaluating an experimental intensive juvenile probation program: Supervision and official outcomes. Crime and Delinquency, 51(1), 26–52.

    Article  Google Scholar 

  • Lauritsen, J. L. (2006). Assessing problematic research: How can academic researchers help improve the quality of anticrime program evaluations? Journal of Experimental Criminology, 2, 363–373.

    Article  Google Scholar 

  • *LeSure-Lester, G. E. (2002). An application of cognitive-behavior principles in the reduction of aggression among abused African American adolescents. Journal of Interpersonal Violence, 17(4), 394–402.

    Article  Google Scholar 

  • Lipsey, M., Petrie, C., Weisburd, D., & Gottfredson, D. (2006). Improving evaluation of anti-crime programs: Summary of a National Research Council report. Journal of Experimental Criminology, 2, 271–307.

    Article  Google Scholar 

  • *Little, M., Kogan, J., Bullock, R., & van der Laan, P. (2004). ISSP: An experiment in multi-systemic responses to persistent young offenders known to children’s services. British Journal of Criminology, 44, 225–240.

    Article  Google Scholar 

  • Lösel, F., & Köferl, P. (1989). Evaluation research on correctional treatment in West Germany: A meta-analysis. In H. Wegener, F. Lösel, & J. Haisch (Eds.), Criminal behavior and the justice system: Psychological perspectives (pp. 334–355). New York, NY: Springer.

    Google Scholar 

  • Lum, C., Koper, C. S., and Telep, C. W. (in press). The Evidence-Based Policing Matrix. Journal of Experimental Criminology.

  • Lum, C., & Yang, S.-M. (2005). Why do evaluation researchers in crime and justice choose non-experimental methods? Journal of Experimental Criminology, 1, 191–213.

    Article  Google Scholar 

  • MacKenzie, D. L. (2000). Evidence-based corrections: Identifying what works. Crime and Delinquency, 46(4), 457–471.

    Article  Google Scholar 

  • *MacKenzie, D. L., Bierie, D. L., & Mitchell, O. (2007). An experimental study of a therapeutic boot camp: Impact on impulses, attitudes and recidivism. Journal of Experimental Criminology, 3, 221–246.

    Article  Google Scholar 

  • *Marlowe, D. B., Festinger, D. S., Lee, P. A., Schepise, M. M., Hazzard, J. E. R., Merrill, J. C., et al. (2003). Are judicial status hearings a key component of drug court? During-treatment data from a randomized trial. Criminal Justice and Behavior, 30(2), 141–162.

    Article  Google Scholar 

  • *Marques, J. K., Wiederanders, M., Day, D. M., Nelson, C., & van Ommeren, A. (2005). Effects of a relapse prevention program on sexual recidivism: Final results from California’s Sex Offender Treatment and Evaluation Project (SOTEP). Sexual Abuse: A Journal of Research and Treatment, 17(1), 79–107.

    Article  Google Scholar 

  • *McGarrell, E. F., & Hipple, N. K. (2007). Family group conferencing and re-offending among first-time juvenile offenders: The Indianapolis experiment. Justice Quarterly, 24(2), 221–246.

    Article  Google Scholar 

  • Moher, D., Jones, A., & Lepage, L. (2001). Use of the CONSORT statement and quality of reports of randomized trials: A comparative before-and-after evaluation. Journal of the American Medical Association, 285(15), 1992–1995.

    Article  Google Scholar 

  • *Moynahan, L., & Strømgren, B. (2005). Preliminary results of Aggression Replacement Training for Norwegian youth with aggressive behavior and with a different diagnosis. Psychology, Crime & Law, 11(4), 411–419.

    Article  Google Scholar 

  • Perry, A. E. (2010). Descriptive validity and transparent reporting in randomised controlled trials. In A. R. Piquero & D. Weisburd (Eds.), Handbook of quantitative criminology (pp. 333–352). New York, NY: Springer.

    Chapter  Google Scholar 

  • Perry, A. E., & Johnson, M. (2008). Applying the Consolidated Standards of Reporting Trials (CONSORT) to studies of mental health provision for juvenile offenders: A research note. Journal of Experimental Criminology, 4, 165–185.

    Article  Google Scholar 

  • Perry, A. E., Weisburd, D., & Hewitt, C. (2010). Are criminologists describing randomized controlled trials in ways that allow us to assess them? Findings from a sample of crime and justice trials. Journal of Experimental Criminology, 6, 245–262.

    Article  Google Scholar 

  • Petrosino, A., Kiff, P., & Lavenberg, J. (2006). Randomized field experiments published in the British Journal of Criminology, 1960-1994. Journal of Experimental Criminology, 2, 99–111.

    Article  Google Scholar 

  • *Phillips, L. A. (2004). Evaluating the effects of a brief program for moral education in a county jail. Journal of Offender Rehabilitation, 39(2), 59–72.

    Article  Google Scholar 

  • Plint, A. C., Moher, D., Morrison, A., Schultz, K., Altman, D. G., Hill, C., et al. (2006). Does the CONSORT checklist improve the quality of reports of randomised controlled trials? A systematic review. The Medical Journal of Australia, 185(5), 263–267.

    Google Scholar 

  • *Sacks, J. Y., Sacks, S., McKendrick, K., Banks, S., Schoeneberger, M., Hamilton, Z., et al. (2008). Prison therapeutic community treatment for female offenders: Profiles and preliminary findings for mental health and other variables (crime, substance use and HIV risk). Journal of Offender Rehabilitation, 46(3), 233–261.

    Article  Google Scholar 

  • Shadish, W. R., Cook, T. D., & Campbell, D. T. (2002). Experimental and quasi-experimental designs for generalized causal inference. Boston, MA: Houghton Mifflin.

    Google Scholar 

  • Shepherd, J. P. (2003). Explaining feast or famine in randomized field trials: Medical science and criminology compared. Evaluation Review, 27(3), 290–315.

    Article  Google Scholar 

  • Sherman, L. W., Farrington, D. P., Welsh, B. C., & MacKenzie, D. L. (Eds.). (2006). Evidence-based crime prevention (2nd ed.). New York, NY: Routledge.

    Google Scholar 

  • Sherman, L. W., Gottfredson, D., MacKenzie, D., Eck, J., Reuter, P., & Bushway, S. (1997). Preventing crime: What works, what doesn’t, what’s promising. Washington, D.C.: United States Department of Justice, National Institute of Justice. Retrieved from http://www.ncjrs.gov/works.

  • Sherman, L. W., Strang, H., Angel, C., Woods, D., Barnes, G. C., Bennett, S., et al. (2005). Effects of face-to-face restorative justice on victims of crime in four randomized, controlled trials. Journal of Experimental Criminology, 1, 367–395.

    Article  Google Scholar 

  • *Stickle, W. P., Connell, N. M., Wilson, D. M., & Gottfredson, D. (2008). An experimental evaluation of teen courts. Journal of Experimental Criminology, 4, 137–163.

    Article  Google Scholar 

  • *Sullivan, C. M., Bybee, D. I., & Allen, N. E. (2002). Findings from a community-based program for battered women and their children. Journal of Interpersonal Violence, 17(9), 915–936.

    Article  Google Scholar 

  • *Taxman, F. S., & Thanner, M. (2006). Risk, need, and responsivity (RNR): It all depends. Crime and Delinquency, 52(1), 28–51.

    Article  Google Scholar 

  • *van Voorhis, P., Spruance, L. M., Ritchey, P. N., Listwan, S. J., & Seabrook, R. (2004). The Georgia Cognitive Skills experiment: A replication of Reasoning and Rehabilitation. Criminal Justice and Behavior, 31(3), 282–305.

    Article  Google Scholar 

  • *Vannoy, S. D., & Hoyt, W. T. (2004). Evaluation of an anger therapy intervention for incarcerated adult males. Journal of Offender Rehabilitation, 39(2), 39–57.

    Article  Google Scholar 

  • *Watt, K., Shepherd, J., & Newcombe, R. (2008). Drunk and dangerous: A randomised controlled trial of alcohol brief intervention for violent offenders. Journal of Experimental Criminology, 4, 1–19.

    Article  Google Scholar 

  • Weisburd, D. (2003). Ethical practice and evaluation of interventions in crime and justice: The moral imperative for randomized trials. Evaluation Review, 27(3), 336–354.

    Article  Google Scholar 

  • *Weisburd, D., Einat, T., & Kowalski, M. (2008). The miracle of the cells: An experimental study of interventions to increase payment of court-ordered financial obligations. Criminology and Public Policy, 7(1), 9–36.

    Article  Google Scholar 

  • Weisburd, D., Lum, C. M., & Petrosino, A. (2001). Does research design affect study outcomes in criminal justice? Annals of the American Academy of Social and Political Science, 578 (November), 50-70.

    Google Scholar 

  • Weisburd, D., Morris, N. A., & Ready, J. (2008). Risk-focused policing at places: An experimental evaluation. Justice Quarterly, 25(1), 163–200.

    Article  Google Scholar 

  • Weisburd, D., Petrosino, A., & Mason, G. (1993). Design sensitivity in criminal justice experiments. Crime and Justice, 17, 337–379.

    Article  Google Scholar 

  • Weisburd, D., Wyckoff, L. A., Ready, J., Eck, J. E., Hinkle, J. C., & Gajewski, F. (2006). Does crime just move around the corner? A controlled study of spatial displacement and diffusion of crime control benefits. Criminology, 44(3), 549–592.

    Article  Google Scholar 

  • *White, M. D., Goldkamp, J. S., & Robinson, J. B. (2006). Acupuncture in drug treatment: Exploring its role and impact on participant behavior in the drug court setting. Journal of Experimental Criminology, 2, 45–65.

    Article  Google Scholar 

  • *Zhang, S. X., & Zhang, L. (2005). An experimental study of the Los Angeles County Repeat Offender Prevention Program: Its implementation and evaluation. Criminology and Public Policy, 4(2), 205–236.

    Article  Google Scholar 

Download references

Acknowledgments

I would like to thank Richard Berk, Bob Boruch, John MacDonald, Larry Sherman, David Weisburd, and the anonymous reviewers for their helpful comments and support in developing and improving this paper.

Author information

Authors and Affiliations

  1. Center for Evidence-Based Crime Policy, George Mason University, 4400 University Dr., MS 6D3, Fairfax, VA, 22030, USA

    Charlotte E. Gill

Authors
  1. Charlotte E. Gill
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Charlotte E. Gill.

Appendix

Appendix

Table 3
Full size table

Rights and permissions

Reprints and permissions

About this article

Cite this article

Gill, C.E. Missing links: how descriptive validity impacts the policy relevance of randomized controlled trials in criminology. J Exp Criminol 7, 201–224 (2011). https://doi.org/10.1007/s11292-011-9122-z

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11292-011-9122-z

Keywords

Search

Navigation