Skip to main content

Advertisement

Log in

Predictions put into practice: a quasi-experimental evaluation of Chicago’s predictive policing pilot

Journal of Experimental Criminology Aims and scope Submit manuscript

Abstract

Objectives

In 2013, the Chicago Police Department conducted a pilot of a predictive policing program designed to reduce gun violence. The program included development of a Strategic Subjects List (SSL) of people estimated to be at highest risk of gun violence who were then referred to local police commanders for a preventive intervention. The purpose of this study is to identify the impact of the pilot on individual- and city-level gun violence, and to test possible drivers of results.

Methods

The SSL consisted of 426 people estimated to be at highest risk of gun violence. We used ARIMA models to estimate impacts on city-level homicide trends, and propensity score matching to estimate the effects of being placed on the list on five measures related to gun violence. A mediation analysis and interviews with police leadership and COMPSTAT meeting observations help understand what is driving results.

Results

Individuals on the SSL are not more or less likely to become a victim of a homicide or shooting than the comparison group, and this is further supported by city-level analysis. The treated group is more likely to be arrested for a shooting.

Conclusions

It is not clear how the predictions should be used in the field. One potential reason why being placed on the list resulted in an increased chance of being arrested for a shooting is that some officers may have used the list as leads to closing shooting cases. The results provide for a discussion about the future of individual-based predictive policing programs.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6

Similar content being viewed by others

Notes

  1. This is only one component of the larger long-term collaboration which broadly explores whether and how crime can be predicted.

  2. Here, “numerical instability” refers to the likelihood that the estimates that an SSL member’s calculated risk of a specified thousands of times more likely to be killed is due more to statistical artifacts from fitting the quadratic curve than an accurate estimate.

  3. Initially, CPD said they would put all the highest-risk individuals on the list; however, they decided to vet the list through their Deployment Operations Center (DOC), who made some changes to who would appear on the list, and therefore, the 426 individuals did not represent the highest scoring individuals based on the model.

  4. Details of GVRS are available at http://directives.chicagopolice.org/directives/data/a7a57bf0-136d1d31-16513-6d1d-382b311ddf65fd3a.html

  5. It is important to note that Chicago has gone through transformative changes over this time period, including a new Superintendent in 2011 and the integration of COMPSTAT to provide oversight. In addition to changes in leadership and management style, CPD has implemented a large number of homicide reduction strategies during this time period, including the multiple changes to the Gang Violence Reduction Strategy, and gang call-ins across different districts starting in 2010.

  6. While there is always the possibility that the groups are different on unobservable variables, we have captured many of the important research-validated criminogenic factors. That is why we specify the approach reduces, rather than eliminates, bias.

  7. With the exception of the winter of 2012 which did not experience the same degree of a “cooling off” period.

  8. Most arrestees were not incapacitated for any significant period of time, but rather were booked into the Cook County jail and released within a few hours to a few days.

References

  • Abrahamse, A. F., Ebener, P. A., Greenwood, P. W., Fitzgerald, N., & Kosin, T. E. (1991). An experimental evaluation of the Phoenix repeat offender program. Justice Quarterly, 8(2), 141–168.

    Article  Google Scholar 

  • Aitchison, J., & Dunsmore, I. R. (1980). Statistical prediction analysis. CUP Archive.

  • Auerhahn, K. (1999). Selective incapacitation and the problem of prediction. Criminology, 37(4), 703–734.

    Article  Google Scholar 

  • Bar-Hillel, M. (1980). The base-rate fallacy in probability judgments. Acta Psychologica, 44(3), 211–233.

    Article  Google Scholar 

  • Beck, C., & McCue, C. (2009). Predictive policing: what can we learn from Wal-Mart and Amazon about fighting crime in a recession? Police Chief, 76(11), 18.

    Google Scholar 

  • Becker, G. S. (1993). Nobel lecture: the economic way of looking at behavior. Journal of Political Economy, 101, 385–409.

    Article  Google Scholar 

  • Benjamini, Y., & Hochberg, Y. (1995). Controlling the false discovery rate: a practical and powerful approach to multiple testing. Journal of the Royal Statistical Society: Series B: Methodological, 57, 289–300.

    MathSciNet  MATH  Google Scholar 

  • Berk, R. (2008). Forecasting methods in crime and justice. Annual Review of Law and Social Science, 4, 219–238.

    Article  Google Scholar 

  • Berk, R. (2011). Asymmetric loss functions for forecasting in criminal justice settings. Journal of Quantitative Criminology, 27(1), 107–123.

    Article  Google Scholar 

  • Berk, R., & Bleich, J. (2013). Statistical procedures for forecasting criminal behavior. Criminology and Public Policy, 12(3), 513–544.

    Article  Google Scholar 

  • Berk, R., Sherman, L., Barnes, G., Kurtz, E., & Ahlman, L. (2009). Forecasting murder within a population of probationers and parolees: a high stakes application of statistical learning. Journal of the Royal Statistical Society: Series A (Statistics in Society), 172(1), 191–211.

    Article  MathSciNet  Google Scholar 

  • Berry, M. J., & Linoff, G. S. (2004). Data mining techniques: for marketing, sales, and customer relationship management. Indianapolis, IN: John Wiley & Sons.

  • Blumstein, A. (1986). Criminal Careers and “Career Criminals” (Vol. 2). Washington, DC: National Academies.

  • Bordua, D. J., & Reiss, A. J., Jr. (1966). Command, control, and charisma: reflections on police bureaucracy. American Journal of Sociology, 72, 68–76.

    Article  Google Scholar 

  • Braga, A. (2005). Hot spots policing and crime prevention: a systematic review of randomized controlled trials. Journal of Experimental Criminology, 1(3), 317–342.

    Article  Google Scholar 

  • Braga, A., & Weisburd, D. L. (2010). Policing problem places: Crime hot spots and effective prevention. New York, NY: Oxford University Press on Demand.

  • Braga, A., & Weisburd, D. L. (2012). The effects of focused deterrence strategies on crime: a systematic review and meta-analysis of the empirical evidence. Journal of Research in Crime and Delinquency, 49(3), 323–358.

    Article  Google Scholar 

  • Braga, A., Papachristos, A. V., & Hureau, D. M. (2012). The effects of hot spots policing on crime: An updated systematic review and meta-analysis. Justice Quarterly, 1–31.

  • Bruinsma, G., & Weisburd, D. (2014). Encyclopedia of Criminology and Criminal Justice. New York: Springer.

    Book  Google Scholar 

  • Caldwell, M. F., Vitacco, M., & Van Rybroek, G. J. (2006). Are violent delinquents worth treating? A cost–benefit analysis. Journal of Research in Crime and Delinquency, 43(2), 148–168.

    Article  Google Scholar 

  • Chicago Police Department. (2014).Gang Violence Reuction Strategy: General Order G10-01. Chicago, IL.

  • Chinman, M., Imm, P., & Wandersman, A. (2004). Getting To Outcomes™ 2004. Santa Monica: Rand Corporation.

    Google Scholar 

  • Cohen, J., Gorr, W. L., & Olligschlaeger, A. M. (2007). Leading indicators and spatial interactions: a crime‐forecasting model for proactive police deployment. Geographical Analysis, 39(1), 105–127.

    Article  Google Scholar 

  • Cope, N. (2004). ‘Intelligence led policing or policing led intelligence?’Integrating volume crime analysis into policing. British Journal of Criminology, 44(2), 188–203.

    Article  Google Scholar 

  • Cornish, D. B., & Clarke, R. V. (2014). The reasoning criminal: Rational choice perspectives on offending. New Brunswick, NJ: Transaction Publishers.

  • Dvoskin, J. A., & Heilbrun, K. (2001). Risk assessment and release decision-making: Toward resolving the great debate. American Academy of Psychiatry and the Law, 29, 6–10.

    CAS  Google Scholar 

  • Eck, J., Chainey, S., Cameron, J., & Wilson, R. (2005). Mapping crime: Understanding hotspots (Vol. NCJ 209393). Washington, DC: National Institute of Justice.

    Google Scholar 

  • Erbentraut, J. (2014). Chicago’s controversial new police program prompts fear of racial profiling. The Huffington Post.

  • Foster, E. M., & Jones, D. (2006). Can a costly intervention be cost-effective?: an analysis of violence prevention. Archives of General Psychiatry, 63(11), 1284–1291.

    Article  PubMed  PubMed Central  Google Scholar 

  • Funk, M. J., Westreich, D., Wiesen, C., Stürmer, T., Brookhart, M. A., & Davidian, M. (2011). Doubly robust estimation of causal effects. American Journal of Epidemiology, 173(7), 761–767.

    Article  PubMed  PubMed Central  Google Scholar 

  • Gendreau, P., Little, T., & Goggin, C. (1996). A meta-anallysis of the predictors of adult offender recidivism: what works! Criminology, 34(4), 575–608.

    Article  Google Scholar 

  • Gorr, W., & Harries, R. (2003). Introduction to crime forecasting. International Journal of Forecasting, 19(4), 551–555.

    Article  Google Scholar 

  • Gottfredson, M., & Hirschi, T. (1986). The true value of Lambda would appear to be zero: an essay on career criminals, criminal careers, selective incapacitation, cohort studies, and related topics*. Criminology, 24(2), 213–234.

    Article  Google Scholar 

  • Greenwood, P. W., & Abrahamse, A. F. (1982). Selective incapacitation. Santa Monica: Rand Corporation.

    Google Scholar 

  • Groff, E. R., & La Vigne, N. G. (2002). Forecasting the future of predictive crime mapping. Crime Prevention Studies, 13, 29–58.

    Google Scholar 

  • Grove, W. M., & Meehl, P. E. (1996). Comparative efficiency of informal (subjective, impressionistic) and formal (mechanical, algorithmic) prediction procedures: the clinical–statistical controversy. Psychology, Public Policy, and Law, 2(2), 293.

    Article  Google Scholar 

  • Huber, P. J. (1973). Robust regression: asymptotics, conjectures and Monte Carlo. The Annals of Statistics, 1, 799–821.

    Article  MathSciNet  MATH  Google Scholar 

  • Hunt, P., Saunders, J., & Hollywood, J. S. (2014). Evaluation of the Shreveport Predictive Policing Experiment. Santa Monica: RAND Corporation.

    Google Scholar 

  • Kang, J. D., & Schafer, J. L. (2007). Demystifying double robustness: a comparison of alternative strategies for estimating a population mean from incomplete data. Statistical Science, 25, 523–539.

  • Kennedy, D. M. (1996). Pulling levers: chronic offenders, high-crime settings, and a theory of prevention. Valparaiso University Law Review, 31, 449.

    Google Scholar 

  • Kovandzic, T. V., Sloan, J. J., III, & Vieraitis, L. M. (2004). “Striking out” as crime reduction policy: the impact of “three strikes” laws on crime rates in US cities. Justice Quarterly, 21(2), 207–239.

    Article  Google Scholar 

  • Lewin, J., & Wernick, M. (2015). Chicago Police Department Data Analytics and Predictive Policing. Paper presented at the International Association of Chief’s of Police, Chicago, IL.

  • Lipsey, M. W. (1999). Can intervention rehabilitate serious delinquents? The Annals of the American Academy of Political and Social Science, 564(1), 142–166.

    Article  Google Scholar 

  • Litwack, T. R., & 2. (2001). Actuarial versus clinical assessments of dangerousness. Psychology, Public Policy, and Law, 7, 409.

    Article  Google Scholar 

  • Llenas, B. (2014). Brave New World of “Predictive Policing” Raises Specter of High-Tech Racial Profiling, Fox News Latino.

  • Loeber, R., & Farrington, D. P. (1998). Serious and violent juvenile offenders: Risk factors and successful interventions. Thousand Oaks, CA: Sage Publications.

  • Lum, C., Koper, C. S., & Telep, C. W. (2011). The evidence-based policing matrix. Journal of Experimental Criminology, 7(1), 3–26.

    Article  Google Scholar 

  • Martin, S. E., & Sherman, L. W. (1986). Selective apprehension: a police strategy for repeat offenders. Criminology, 24, 155.

    Article  Google Scholar 

  • Mazerolle, L. G., Kadleck, C., & Roehl, J. (1998). Controlling drug and disorder problems: the role of place managers. Criminology, 36(2), 371–404.

    Article  Google Scholar 

  • Mazerolle, L. G., Ready, J., Terrill, W., & Waring, E. (2000). Problem-oriented policing in public housing: the Jersey City evaluation. Justice Quarterly, 17(1), 129–158.

    Article  Google Scholar 

  • McCaffrey, D. F., Ridgeway, G., & Morral, A. R. (2004). Propensity score estimation with boosted regression for evaluating causal effects in observational studies. Psychological Methods, 9(4), 403.

    Article  PubMed  Google Scholar 

  • McCleary, R., Hay, R. A., Meidinger, E. E., & McDowall, D. (1980). Applied time series analysis for the social sciences. Beverly Hills: Sage Publications.

    Google Scholar 

  • McCord, J. (2003). Cures that harm: unanticipated outcomes of crime prevention programs. The Annals of the American Academy of Political and Social Science, 587(1), 16–30.

    Article  Google Scholar 

  • McGarrell, E. F., Chermak, S., Wilson, J. M., & Corsaro, N. (2006). Reducing homicide through a “lever‐pulling” strategy. Justice Quarterly, 23(02), 214–231.

    Article  Google Scholar 

  • Mohler, G. O., Short, M. B., Malinowski, S., Johnson, M., Tita, G. E., Bertozzi, A. L., & Brantingham, P. J. (2015). Randomized controlled field trials of predictive policing. Journal of the American Statistical Association, 110(512), 1399–1411.

  • Ohri, A. (2013). Forecasting and Time Series Models R for Business Analytics (pp. 241–258). Springer.

  • Papachristos, A. (2009). Murder by structure: dominance relations and the social structure of gang homicide. American Journal of Sociology, 115(1), 74–128.

    Article  MathSciNet  Google Scholar 

  • Papachristos, A. V., & Kirk, D. S. (2015). Changing the street dynamic. Criminology and Public Policy, 14(3), 525–558.

    Article  Google Scholar 

  • Papachristos, A., Braga, A., & Hureau, D. (2011). Six-degrees of violent victimization: Social networks and the risk of gunshot injury.

  • Papachristos, A., Braga, A., & Hureau, D. (2012). Social networks and the risk of gunshot injury. Journal of Urban Health, 89(6), 992–1003.

    Article  PubMed  PubMed Central  Google Scholar 

  • Pate, T., Bowers, R. A., & Parks, R. (1976). Three approaches to criminal apprehension in Kansas City: An evaluation report. Washington, DC: Police Foundation.

    Google Scholar 

  • Perry, W. L., McInnis, B., Price, C. C., Smith, S. C., & Hollywood, J. S. (2013). Predictive policing: The role of crime forecasting in law enforcement operations. Santa Monica, CA: Rand Corporation.

  • Quinsey, V. L., Harris, G. T., & Rice, M. E. (2000). Violent Offenders: Appraising and Managing Risk. Psychiatric Services, 51(3), 395

  • Ratcliffe, J. (2002). Intelligence-led policing and the problems of turning rhetoric into practice. Policing and Society, 12(1), 53–66.

    Article  MathSciNet  Google Scholar 

  • Ratcliffe, J. (2005). The effectiveness of police intelligence management: a New Zealand case study. Police Practice and Research, 6(5), 435–451.

    Article  Google Scholar 

  • Ratcliffe, J. H. (2012). Intelligence-led policing. New York, NY: Routledge.

  • Ratcliffe, J. H., & Guidetti, R. (2008). State police investigative structure and the adoption of intelligence-led policing. Policing: An International Journal of Police Strategies and Management, 31(1), 109–128.

    Article  Google Scholar 

  • Ridgeway, G. (2013). Linking prediction and prevention. Criminology and Public Policy, 12(3), 545–550.

    Article  Google Scholar 

  • Ridgeway, G., & MacDonald, J. M. (2009). Doubly robust internal benchmarking and false discovery rates for detecting racial bias in police stops. Journal of the American Statistical Association, 104(486), 661–668.

    Article  MathSciNet  CAS  MATH  Google Scholar 

  • Ridgeway, G., Braga, A. A., Tita, G., & Pierce, G. L. (2011). Intervening in gun markets: an experiment to assess the impact of targeted gun-law messaging. Journal of Experimental Criminology, 7(1), 103–109.

    Article  Google Scholar 

  • Ridgeway, G., McCaffrey, D., Morral, A., Burgette, L., & Griffin, B. A. (2014). Toolkit for Weighting and Analysis of Nonequivalent Groups: A tutorial for the twang package. R vignette. RAND.

  • Rosenbaum, P. R., & Rubin, D. B. (1983). The central role of the propensity score in observational studies for causal effects. Biometrika, 70(1), 41–55.

    Article  MathSciNet  MATH  Google Scholar 

  • Sailor, W., Dunlap, G., Sugai, G., & Horner, R. (2008). Handbook of positive behavior support. New York, NY: Springer.

  • Sherman, L. W. (1986). Policing communities: what works? Crime and justice, 343–386.

  • Sherman, L. W. (1992). The influence of criminology on criminal law: evaluating arrests for misdemeanor domestic violence. Journal of Criminal Law and Criminology, 83, 1–45.

    Article  Google Scholar 

  • Sherman, L. W., & Berk, R. A. (1984). The specific deterrent effects of arrest for domestic assault. American Sociological Review, 49, 261–272.

    Article  CAS  PubMed  Google Scholar 

  • Sherman, L. W., & Weisburd, D. (1995). General deterrent effects of police patrol in crime “hot spots”: a randomized, controlled trial. Justice Quarterly, 12(4), 625–648.

    Article  Google Scholar 

  • Sherman, L. W., Gottfredson, D., MacKenzie, D., Eck, J., Reuter, P., & Bushway, S. (1997). Preventing crime: What works, what doesn’t, what’s promising: A report to the United States Congress. Washington, DC: US Department of Justice, Office of Justice Programs.

    Google Scholar 

  • Silver, E., & Miller, L. L. (2002). A cautionary note on the use of actuarial risk assessment tools for social control. Crime and Delinquency, 48(1), 138–161.

    Article  Google Scholar 

  • Starr, S. B. (2014). Evidence-based sentencing and the scientific rationalization of discrimination. Stanford Law Review, 66, 803–953.

    Google Scholar 

  • Stroud, M. (2014). The Minority Report: Chicago’s new police computer predicts crimes, but is it racist? The Verge.

  • Tonry, M. (1987). Prediction and classification: legal and ethical issues. Crime and Justice, 9, 367–413.

    Article  Google Scholar 

  • Weisburd, D., & Mazerolle, L. G. (2000). Crime and disorder in drug hot spots: implications for theory and practice in policing. Police Quarterly, 3(3), 331–349.

    Article  Google Scholar 

  • Welsh, B. C., & Rocque, M. (2014). When crime prevention harms: a review of systematic reviews. Journal of Experimental Criminology, 10(3), 245–266.

    Article  Google Scholar 

  • Wright, K. N., Clear, T. R., & Dickson, P. (1984). Universal applicability of probation risk‐assessment instruments. Criminology, 22(1), 113–134.

    Article  Google Scholar 

  • Yang, M., Wong, S. C., & Coid, J. (2010). The efficacy of violence prediction: a meta-analytic comparison of nine risk assessment tools. Psychological Bulletin, 136(5), 740.

    Article  PubMed  Google Scholar 

  • Zikopoulos, P., & Eaton, C. (2011). Understanding big data: Analytics for enterprise class hadoop and streaming data. McGraw-Hill Osborne Media.

Download references

Acknowledgments

We would like to thank the Chicago Police Department and Dr. Miles Wernick from the Illinois Institute of Technology for their participation and support of this evaluation. We would also like to acknowledge research assistance provided by Sam Cooper and Alessandra Sienra-Canas. This publication was made possible by Award Number 2009-IJ-CX-K114 - Predictive Policing Analytic & Evaluation Research Support awarded by the National Institute of Justice, Office of Justice Programs. The opinions, findings, conclusions and recommendations expressed in this publication are those of the authors and do not necessarily reflect the views of the Department of Justice.

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Jessica Saunders.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Saunders, J., Hunt, P. & Hollywood, J.S. Predictions put into practice: a quasi-experimental evaluation of Chicago’s predictive policing pilot. J Exp Criminol 12, 347–371 (2016). https://doi.org/10.1007/s11292-016-9272-0

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11292-016-9272-0

Keywords

Navigation