Advancing knowledge about replication in criminology

  • David P. FarringtonEmail author
  • Friedrich Lösel
  • Robert F. Boruch
  • Denise C. Gottfredson
  • Lorraine Mazerolle
  • Lawrence W. Sherman
  • David Weisburd


This article summarizes key points made in a session at the American Society of Criminology meeting in Philadelphia in November 2017, entitled “The replication issue in science and its relevance for criminology”, organized by Friedrich Lösel and Robert F. Boruch. In turn, this session was inspired by Friedrich Lösel’s (2018) article in this journal, based on his 2015 Joan McCord Award Lecture of the Academy of Experimental Criminology. In the present article, Friedrich Lösel introduces the topic of replication in criminology and summarizes his main arguments. Then, six leading criminologists present short papers on this topic. Robert F. Boruch points out the instability in social systems, David P. Farrington argues that systematic reviews are important, and Denise C. Gottfredson calls attention to the heterogeneity in conclusions across different studies. Lorraine Mazerolle reviews attempts to replicate experiments in procedural justice, Lawrence W. Sherman draws attention to enthusiasm bias in criminal justice experiments, and David Weisburd discusses the logic of null hypothesis significance testing and multi-center trials. Finally, some developments since November 2017 in research on replication in criminology are discussed.


Replication Systematic reviews Randomized experiments Policing Multi-center trials 


  1. Andrews, D. A., Bonta, J., & Wormith, S. (2011). The risk-need-responsivity (RNR) model: does adding the good lives model contribute to effective crime prevention? Criminal Justice and Behavior, 38, 735–755.CrossRefGoogle Scholar
  2. Antrobus, E., Alpert, G., & Rojek, J. (2015). Replicating experiments in criminology: Lessons learned from Richland. Unpublished short report.Google Scholar
  3. Antrobus, E., & Pilotto, A. (2016). Improving forensic responses to residential burglaries: results of a randomized controlled field trial. Journal of Experimental Criminology, 12(3), 319–345.CrossRefGoogle Scholar
  4. Archer, D., & Gartner, R. (1976). Violent acts and violent times: a comparative approach to postwar homicide. American Sociological Review, 41, 937–963.CrossRefGoogle Scholar
  5. Baker, M. (2016). Is there a reproducibility crisis? Nature, 533, 452–455.Google Scholar
  6. Bennett, S., Mazerolle, L., Hine, L., & Antrobus, E. (in press). The trials and tribulations of evidence-based procedural justice. In L. Huey & R. Mitchell (Eds.), Evidence-based policing: an introduction. Bristol, UK: Policy Press.Google Scholar
  7. Boruch, R. F., & the API Research Team. (2016). In search of Terra firma: administrative records on teachers’ positional instability across subjects, grades, and schools and the implications for deploying randomized controlled trials. Philadelphia: University of Pennsylvania Available from: Scholar
  8. Bottoms, A., & Tankebe, J. (2012). Beyond procedural justice: a dialogic approach to legitimacy in criminal justice. Journal of Criminal Law and Criminology, 102(1), 119–170.Google Scholar
  9. Campbell, D. T., & Stanley, J. C. (1963). Experimental and quasi-experimental designs for research on teaching. In N. L. Gage (Ed.), Handbook of research on teaching. Chicago: Rand McNally.Google Scholar
  10. Chan, W. (2017). Partially identified treatment effects for generalizability. Journal of Research on Educational Effectiveness, 10(3), 649–669.CrossRefGoogle Scholar
  11. Cohen, J. (1994). The earth is round (p<.05). American Psychologist, 49(12), 997–1003.CrossRefGoogle Scholar
  12. Conduct Problems Prevention Research Group. (2010). Fast track intervention effects on youth arrest and delinquency. Journal of Experimental Criminology, 6, 131–157.CrossRefGoogle Scholar
  13. Derzon, J. H. (2010). The correspondence of family features with problem, aggressive, criminal, and violent behavior: a meta-analysis. Journal of Experimental Criminology, 6, 263–292.CrossRefGoogle Scholar
  14. Dingler, H. (1923). Grundlagen der physik: synthetische prinzipien der mathematischen naturphilosophie [basics of physics: synthetical principles of mathematical natural philosophy]. Berlin: de Gruyter.Google Scholar
  15. Eckenrode, J., Campa, M., Luckey, D. W., Henderson, C. R., Cole, R., Kitzman, H., Anson, E., Sidora-Arocleo, K., Powers, J., & Olds, D. (2010). Long-term effects of prenatal and infancy nurse home visitation on the life course of youths: 19-year follow-up of a randomized trial. Archives of Pediatrics and Adolescent Medicine, 164, 9–15.CrossRefGoogle Scholar
  16. Eisner, M. (2009). No effects in independent prevention trials: can we reject the cynical view? Journal of Experimental Criminology, 5, 163–183.CrossRefGoogle Scholar
  17. Fagan, A. A., & Buchanan, M. (2016). What works in crime prevention? Comparison and critical review of three crime prevention registries. Criminology and Public Policy, 15, 617–649.CrossRefGoogle Scholar
  18. Farrington, D. P. (2000). Explaining and preventing crime: the globalization of knowledge. Criminology, 38, 1–24.CrossRefGoogle Scholar
  19. Farrington, D. P., Gaffney, H., Losel, F., & Ttofi, M. M. (2017a). Systematic reviews of the effectiveness of developmental prevention programs in reducing delinquency, aggression and bullying. Aggression and Violent Behavior, 33, 91–106.CrossRefGoogle Scholar
  20. Farrington, D. P., Gaffney, H., & Ttofi, M. M. (2017b). Systematic reviews of explanatory risk factors for violence, offending, and delinquency. Aggression and Violent Behavior, 33, 24–36.CrossRefGoogle Scholar
  21. Farrington, D. P., Ttofi, M. M., & Losel, F. A. (2016). Developmental and social prevention. In D. Weisburd, D. P. Farrington, & C. Gill (Eds.), What works in crime prevention and rehabilitation: lessons from systematic reviews (pp. 15–75). New York: Springer.CrossRefGoogle Scholar
  22. Farrington, D. P., & Welsh, B. C. (2013). Randomized experiments in criminology: What has been learned from long-term follow-ups? In B. C. Welsh, A. A. Braga, & G. J. N. Bruinsma (Eds.), Experimental criminology: prospects for advancing science and public policy (pp. 111–140). New York: Cambridge University Press.CrossRefGoogle Scholar
  23. Fixsen, D. L., Blasé, K. A., Naoom, S. F., & Wallace, F. (2009). Core implementation components. Research on Social Work Practice, 19, 531–540.CrossRefGoogle Scholar
  24. Fleiss, J. (1982). Multicentre clinical trials: Bradford Hill’s contributions and some subsequent developments. Statistics in Medicine, 1, 353–359.CrossRefGoogle Scholar
  25. Friendship, C., Mann, R. E., & Beech, A. R. (2003). Evaluation of the prison-based sex offender treatment programme (SOTP). London: Home Office Research and Statistics Directorate.Google Scholar
  26. Gandhi, A. G., Murphy-Graham, E., Petrosino, A., Chrismer, S. S., & Weiss, C. H. (2007). The devil is in the details: examining the evidence for “proven” school-based drug abuse prevention programs. Evaluation Review, 31, 43–74.CrossRefGoogle Scholar
  27. Garner, J., Fagan, J., & Maxwell, C. (1995). Published findings from the spouse assault replication program: a critical review. Journal of Quantitative Criminology, 11, 3–28.CrossRefGoogle Scholar
  28. Gelman, A., Skardhamar, T., & Aaltonen, M. (in press). Type M error might explain Weisburd’s paradox. Journal of Quantitative Criminology in press.Google Scholar
  29. Gottfredson, D. (2016). Why registries matter. Criminology and Public Policy, 15, 651–659.CrossRefGoogle Scholar
  30. Gottfredson, D. C., Cook, T. D., Gardner, F. E. M., Gorman-Smith, D., Howe, G. W., Sandler, I. N., & Zafft, K. M. (2015). Standards of evidence for efficacy, effectiveness, and scale-up research in prevention science: next generation. Prevention Science, 16, 893–926. Scholar
  31. Hill, B. (1962). Principles of medical statistics. New York: Oxford University Press.Google Scholar
  32. Hopewell, S., Clarke, M., Moher, D., Wager, E., Middleton, P., Altman, D. G., Schulz, K. F., & the CONSORT Group. (2002). CONSORT for reporting randomized trials in journal and conference abstracts: explanation and elaboration. PLoS Medicine, 5(1), e20. Scholar
  33. Ioannidis, J. P. A. (2005a). Contradicted and initially stronger effects in highly cited clinical research. Journal of the American Medical Association, 294(2), 218–228. Scholar
  34. Ioannidis, J. P. A. (2005b). Why most published research findings are false. PLoS Medicine, 2(8), e124. Scholar
  35. King, G., & Roberts, M. E. (2015). How robust standard errors expose methodological problems they do not fix, and what to do about it. Political Analysis, 23, 159–179.CrossRefGoogle Scholar
  36. Koehler, J. A., Lösel, F., Humphreys, D. K., & Akoensi, T. D. (2013). A systematic review and meta-analysis on the effects of young offender treatment programs in Europe. Journal of Experimental Criminology, 9, 19–43.CrossRefGoogle Scholar
  37. Langley, B. (2014). A randomised control trial comparing the effects of procedural justice to experienced utility theories in airport security stops (Master’s Thesis, University of Cambridge, UK). Retrieved from,%20B.%20A%20randomised%20control%20trial%20comparing%20the%20effects%20of%20procedural%20justice%20to%20experienced%20utility%20theories%20in%20airport%20security%20stops.pdf
  38. Lipsey, M. W. (2003). Those confounded moderators in meta-analysis: good, bad, and ugly. Annals of the American Academy of Political and Social Science, 587, 69–81.CrossRefGoogle Scholar
  39. Lipsey, M. W., & Cullen, F. T. (2007). The effectiveness of correctional rehabilitation: a review of systematic reviews. Annual Review of Law and Social Science, 3, 297–320.CrossRefGoogle Scholar
  40. Lösel, F. (2012a). Offender treatment and rehabilitation: what works? In M. Maguire, R. Morgan, & R. Reiner (Eds.), The Oxford handbook of criminology (5th ed., pp. 986–1016). Oxford: Oxford University Press.Google Scholar
  41. Lösel, F. (2012b). Entwicklungsbezogene prävention von gewalt und kriminalität: Ansätze und wirkungen [developmental prevention of violence and crime: approaches and effects]. Forensische Psychiatrie, Psychologie und Kriminologie, 6, 71–84.CrossRefGoogle Scholar
  42. Lösel, F. (2018). Evidence comes by replication, but needs differentiation: the reproducibility issue in science and its relevance for criminology. Journal of Experimental Criminology. Advance online publication.
  43. Lösel, F., & Bender, D. (2012). Child social skills training in the prevention of antisocial development and crime. In D. P. Farrington & B. C. Welsh (Eds.), The Oxford handbook of crime prevention (pp. 102–129). Oxford: Oxford University Press.Google Scholar
  44. Lösel, F., & Farrington, D. P. (2012). Direct protective and buffering protective factors in the development of youth violence. American Journal of Preventive Medicine, 43(2S1), S8–S23.CrossRefGoogle Scholar
  45. Lösel, F., & Schmucker, M. (2005). The effectiveness of treatment for sexual offenders: a comprehensive meta-analysis. Journal of Experimental Criminology, 1, 117–146.CrossRefGoogle Scholar
  46. Lösel, F., & Stemmler, M. (2012). Preventing child behavior problems in the Erlangen-Nuremberg development and prevention study: results from preschool to secondary school age. International Journal of Conflict and Violence, 6, 214–224.Google Scholar
  47. Lösel, F., Stemmler, M., & Bender, D. (2013). Long-term evaluation of a bimodal universal prevention program: effects from kindergarten to adolescence. Journal of Experimental Criminology, 9, 429–449.CrossRefGoogle Scholar
  48. Luhmann, N. (1997). Die moral des risikos und das risiko der moral [the morality of risk and the risk of morality]. In G. Bechmannn (Ed.), Risiko und gesellschaft [Risk and society]. Opladen: Leske and Budrich.Google Scholar
  49. Lum, C., Koper, C. S., Gill, C., Hibdon, J., Telep, C., & Robinson, L. (2016). An evidence-assessment of the recommendations of the President’s Task Force on 21st Century Policing—implementation and research priorities. Fairfax, VA: Center for Evidence-Based Crime Policy, George Mason University, And International Association of Chiefs of police.Google Scholar
  50. MacQueen, S., & Bradford, B. (2014). The Scottish Community Engagement Trial (ScotCET). In Scottish Institute for Policing Research, Annual Report 2013 (p. 18). Retrieved from
  51. MacQueen, S., & Bradford, B. (2015a). Enhancing public trust and police legitimacy during road traffic encounters: results from a randomised controlled trial in Scotland. Journal of Experimental Criminology, 11(3), 419–443. Scholar
  52. MacQueen, S., & Bradford, B. (2015b). Procedural justice in practice: findings from the Scottish community engagement trial (ScotCET). Scottish Justice Matters, 3(2), 11–12.Google Scholar
  53. MacQueen, S., & Bradford, B. (2017). Where did it all go wrong? Implementation failure—and more—in a field experiment of procedural justice policing. Journal of Experimental Criminology, 13(3), 321–345. Scholar
  54. Makel, M., Plucker, J., & Hegarty, B. (2012). Replications in psychology research: how often do they really occur? Perspectives on Psychological Science, 7, 537–542.CrossRefGoogle Scholar
  55. Maxwell, C. D., Garner, J. H., & Skogan, W. G. (in press). Collective efficacy and violence in Chicago neighborhoods: a reproduction. Journal of Contemporary Criminal Justice, in press.Google Scholar
  56. Mazerolle, L., Antrobus, E., Bennett, S., & Tyler, T. (2013a). Shaping citizen perceptions of police legitimacy: a randomized field trial of procedural justice. Criminology, 51(1), 33–64. Scholar
  57. Mazerolle, L., Bennett, S., Antrobus, E., & Eggins, E. (2012). Procedural justice, routine encounters and citizen perceptions of police: main findings from the Queensland community engagement trial (QCET). Journal of Experimental Criminology, 8(4), 343–367. Scholar
  58. Mazerolle, L., Bennett, S., Davis, J., Sargeant, E., & Manning, M. (2013b). Legitimacy in policing: a systematic review. Campbell Systematic Reviews, 2013, 1. Scholar
  59. McNeely, S., & Warner, J. J. (2015). Replication in criminology: necessary practice. European Journal of Criminology, 12, 581–597.CrossRefGoogle Scholar
  60. Mews, A., Di Bella, L., & Purver, M. (2017). Impact evaluation of the prison-based core sex offender treatment programme. London: UK Ministry of Justice. Available at:
  61. Mihalic, S. H., & Elliott, D. S. (2015). Evidence-based programs registry: blueprints for healthy youth development. Evaluation and Program Planning, 48, 124–131.CrossRefGoogle Scholar
  62. Miller, G. A. (1956). The magical number seven, plus or minus two: some limits on our capacity for processing information. Psychological Review, 63, 81–97.CrossRefGoogle Scholar
  63. Myers, W., Lloyd, K., Turanovic, J. J., & Pratt, T. C. (in press). Revisiting a criminological classic: the cycle of violence. Journal of Contemporary Criminal Justice, in press.Google Scholar
  64. Nagin, D. S., & Telep, C. W. (2017). Procedural justice and legal compliance. Annual Review of Law and Social Science, 13, 5–28. Scholar
  65. National Academies of Sciences, Engineering, and Medicine. (2017). Proactive policing: effects on crime and communities. Washington, DC: National Academies Press. Scholar
  66. Nosek, B. A., Spies, J. R., & Motyl, M. (2012). Scientific utopia: II. Restructuring incentives and practices to promote truth over publishability. Perspectives on Psychological Science, 7, 615–631. Scholar
  67. Open Science Collaboration. (2015). Estimating the reproducibility of psychological science. Science, 349, 4716-3–4716-8.CrossRefGoogle Scholar
  68. Perry, A. E., Weisburd, D., & Hewitt, C. (2010). Are criminologists describing randomized controlled trials in ways that allow us to assess them? Findings from a sample of crime and justice trials. Journal of Experimental Criminology, 6, 245–282.CrossRefGoogle Scholar
  69. Petrosino, A., & Soydan, H. (2005). The impact of program developers as evaluators on criminal recidivism: results from meta-analyses of experimental and quasi-experimental research. Journal of Experimental Criminology, 1, 435–450.CrossRefGoogle Scholar
  70. Popper, K. (1959). The logic of scientific discovery. London: Hutchinson.Google Scholar
  71. President’s Task Force on 21st Century Policing. (2015). Final report of the President’s task force on 21st century policing. Washington, DC: Office for Community Oriented Policing Services.Google Scholar
  72. Pridemore, W. A., Makel, M. C., & Plucker, J. A. (2018). Replication in criminology and the social sciences. Annual Review of Criminology, 1, 19–38.CrossRefGoogle Scholar
  73. Rozeboom, W. W. (1960). The fallacy of the null hypothesis significance test. Psychological Bulletin, 37, 416–428.CrossRefGoogle Scholar
  74. Sahin, N. M. (2014). Legitimacy, procedural justice, and police-citizen encounters: a randomized controlled trial of the impact of procedural justice on citizen perceptions of the police during traffic stops in Turkey (Doctoral dissertation). Retrieved from
  75. Sahin, N., Braga, A. A., Apel, R., & Brunson, R. K. (2017). The impact of procedurally-just policing on citizen perceptions of police during traffic stops: the Adana randomized controlled trial. Journal of Quantitative Criminology, 33(4), 701–726. Scholar
  76. Sampson, R. J., Raudenbush, S. W., & Earls, F. E. (1997). Neighborhoods and violent crime: a multilevel study of collective efficacy. Science, 277, 918–924.CrossRefGoogle Scholar
  77. Sanders, M. R. (2015). Management of conflict of interest in psychosocial research on parenting and family interventions. Journal of Child and Family Studies, 24, 832–841.CrossRefGoogle Scholar
  78. Savolainen, J., & VanEseltine, M. (in press). Replication and research integrity in criminology: Introduction to the special issue. Journal of Contemporary Criminal Justice, in press.Google Scholar
  79. Schmucker, M., & Lösel, F. (2015). The effects of sexual offender treatment on recidivism: An international meta-analysis of sound quality evaluations. Journal of Experimental Criminology, 11, 597–630.CrossRefGoogle Scholar
  80. Sherman, L. W. (1992). Policing domestic violence: experiments and dilemmas. New York: Free Press.Google Scholar
  81. Sherman, L. W., & Berk, R. A. (1984). The specific deterrent effects of arrest for domestic assault. American Sociological Review, 49(2), 261–272.CrossRefGoogle Scholar
  82. Sherman, L. W., van Mastrigt, S., Gade, C. N., & Strang, H. (forthcoming). External validity and partial random assignment of treatment providers to treatment delivery within small work groups. Cambridge, UK: Cambridge Institute of Criminology (under review).Google Scholar
  83. Sherman, L. W., Strang, H., Barnes, G., Woods, D. J., Bennett, S., Inkpen, N., Newbury-Birch, D., Rossner, M., Angel, C., Mearns, M., & Slothower, M. (2015). Twelve experiments in restorative justice: the Jerry Lee program of randomized trials of restorative justice conferences. Journal of Experimental Criminology, 11(4), 501–540.CrossRefGoogle Scholar
  84. Singh, J. P., Yang, S., Mulvey, E., & RAGEE group. (2015). Reporting guidance for violence risk assessment predictive validity studies: the RAGEE statement. Law and Human Behavior, 39, 15–22.CrossRefGoogle Scholar
  85. Stamatel, J. P., & Romans, S. (in press). The effects of wars on post-war homicide rates: a replication and extension of Archer and Gartner’s classic study. Journal of Contemporary Criminal Justice, in press.Google Scholar
  86. Stanley, K., Stjernsward, M., & Isley, M. (1981). The conduct of a cooperative clinical trial. New York: Springer-Verlag.CrossRefGoogle Scholar
  87. Steering Group of the Campbell Collaboration (2016). Campbell systematic reviews: policies and guidelines. Campbell Policies and Guidelines Series No. 1. doi:
  88. Stuart, E. A., Cole, S. R., Bradshaw, C. P., & Leaf, P. J. (2011). The use of propensity scores to assess the generalizability of results from randomized trials. Journal of the Royal Statistical Society: Series A (Statistics in Society), 174(2), 369–386.CrossRefGoogle Scholar
  89. Sullivan, L. H. (1896). The tall office building artistically considered. Lippincott's Magazine, 57(3), 406.Google Scholar
  90. Sullivan, L. H. (1956). The autobiography of an idea. New York: Dover Publications.Google Scholar
  91. Tong, L. S. J., & Farrington, D. P. (2006). How effective is the reasoning and rehabilitation programme in reducing offending? A meta-analysis of evaluations in four countries. Psychology, Crime and Law, 12, 3–24.CrossRefGoogle Scholar
  92. Ttofi, M. M., & Farrington, D. P. (2011). Effectiveness of school-based programs to reduce bullying: a systematic and meta-analytic review. Journal of Experimental Criminology, 7, 27–56.CrossRefGoogle Scholar
  93. Tyler, T. R. (2004). Enhancing police legitimacy. Annals of the American Academy of Political and Social Science, 593(1), 84–99. Scholar
  94. Valentine, J., Biglan, A., Boruch, R. F., Gonzalez-Castro, F., Collins, Linda, M., Flay, B. R., Kellam, S., Moscicki, E. K., & Schinke, S. P. (2011). Replication in prevention science. Prevention Science, 12, 103–117.CrossRefGoogle Scholar
  95. Vuchinich, S., Flay, B. R., Aber, L., & Bickman, L. (2013). Person mobility in the design and analysis of cluster randomized cohort prevention trials. Prevention Science, 13(3), 300–313.CrossRefGoogle Scholar
  96. Wareham, J., Smith, B. W., & Lambert, E. G. (2015). Rates and patterns of law enforcement turnover: a research note. Criminal Justice Policy Review, 26(4), 345–370. Scholar
  97. Weisburd, D., Farrington, D. P., & Gill, C. (2017). What works in crime prevention and rehabilitation: an assessment of systematic reviews. Criminology and Public Policy, 16(2), 416–449.CrossRefGoogle Scholar
  98. Weisburd, D., & Taxman, F. (2000). Developing a multicenter randomized trial in criminology: the case of HIDTA. Journal of Quantitative Criminology, 16(3), 315–340.CrossRefGoogle Scholar
  99. Welsh, B. C., Sullivan, C. J., & Olds, D. (2010). When early crime prevention goes to scale: a new look at the evidence. Prevention Science, 11, 115–125.CrossRefGoogle Scholar
  100. Widom, C. S. (1989). The cycle of violence. Science, 244, 160–166.CrossRefGoogle Scholar
  101. Wilson, D. B. (2016). Correctional programs. In D. Weisburd, D. P. Farrington, & C. Gill (Eds.), What works in crime prevention and rehabilitation: lessons from systematic reviews (pp. 193–217). New York: Springer.CrossRefGoogle Scholar
  102. Wilson, J. M., Dalton, E., Scheer, C., & Grammich, C. (2010). Police recruitment and retention for the new millennium: the state of knowledge. Santa Monica, CA: RAND Corporation.Google Scholar

Copyright information

© Springer Nature B.V. 2018

Authors and Affiliations

  • David P. Farrington
    • 1
    Email author
  • Friedrich Lösel
    • 1
    • 2
  • Robert F. Boruch
    • 3
  • Denise C. Gottfredson
    • 4
  • Lorraine Mazerolle
    • 5
  • Lawrence W. Sherman
    • 1
    • 4
  • David Weisburd
    • 6
    • 7
  1. 1.Cambridge UniversityCambridgeUK
  2. 2.University of Erlangen-NurembergErlangenGermany
  3. 3.University of PennsylvaniaPhiladelphiaUSA
  4. 4.University of MarylandCollege ParkUSA
  5. 5.University of QueenslandBrisbaneAustralia
  6. 6.George Mason UniversityFairfaxUSA
  7. 7.Hebrew UniversityJerusalemIsrael

Personalised recommendations