Police departments around the country are implementing Early Intervention Systems (EIS) to identify officers who may be exhibiting problematic or unprofessional behaviors. The goal of EIS is to minimize officer misconduct and increase officer accountability. To evaluate whether EIS can actually differentiate “problem” from “non-problem” officers, we analyzed the performance of officers from incident reports of police–citizen interactions.
Using a blind scoring method, we evaluated performance from 1000 police reports; 500 randomly selected reports from EIS-flagged officers (treatment group) and 500 randomly selected reports from non-flagged officers (control group). Six hundred and sixty-seven reports contained relevant performance data. The interval-level metrics used to score officer performance were developed by Vila and colleagues (2016, 2018) to assess performance—expressed as a percentage—across a range of police–citizen encounters.
The overall performance score assigned to officers across all 667 incident reports was 80.46% (SD = 8.75%). When separated into EIS-flagged and non-EIS–flagged incidents, performance scores were 80.63% (SD = 8.58%) compared to 80.27% (SD = 8.95). There was not a statistically significant difference between EIS-flagged and non-EIS–flagged performance.
The EIS evaluated does not appear to be differentiating between problem behavior and non-problem behavior. This suggests that the “thresholds” used to identify problem officers are not working effectively.
This is a preview of subscription content, access via your institution.
Buy single article
Instant access to the full article PDF.
Price excludes VAT (USA)
Tax calculation will be finalised during checkout.
Alpert, G. P., & Walker, S. (2000). Police accountability and early warning systems: developing policies and programs. Justice Research and Policy, 2(2), 59–72. https://doi.org/10.3818/JRP.2.2.2000.59.
Ariel, B., Farrar, W., & Sutherland, A. (2015). The effect of police body-worn cameras on use of force and citizens’ complaints against the police: a randomized controlled trial. Journal of Quantitative Criminology, 31(3), 509–535.
Ariel, B., Sutherland, A., Henstock, D., Young, J., Drover, P., Sykes, J., et al. (2017). "Contagious accountability": a global multisite randomized controlled trial on the effect of police body-worn cameras on citizens' complaints against the police. Criminal Justice And Behavior, 44(2), 293–316.
Bazley, T. D., et al. (2009). Early intervention program criteria: evaluating officer use of force. Justice Quarterly, 26(1), 107–124. https://doi.org/10.1080/07418820801989742.
Brandl, S. G., et al. (2001). Who are the complaint-prone officers?: An examination of the relationship between police officers' attributes, arrest activity, assignment, and citizens' complaints about excessive force. Journal of Criminal Justice, 29(6), 521–529. https://doi.org/10.1016/S0047-2352(01)00114-3.
Harris, C. J. (2009). Exploring the relationship between experience and problem behaviors. Police Quarterly, 12(2), 192–213. https://doi.org/10.1177/1098611108327314.
Harris, C. J. (2016) “Towards a career view of police misconduct.” Aggression and Violent Behavior, vol. 31, 2016, pp. 219–228., doi:https://doi.org/10.1016/j.avb.2016.10.001.
Hedberg, E., Katz, C., & Choate, D. (2017). Body-worn cameras and citizen interactions with police officers: estimating plausible effects given varying compliance levels. Justice Quarterly, 34(4), 627–651.
Helsby, J., Stageman, D. L., et al. (2018). Early intervention systems: predicting adverse interactions between police and the public. Criminal Justice Policy Review, 29(2), 190–209. https://doi.org/10.1177/0887403417695380.
Independent Commission on the Los Angeles Police Department (1991), Report of the Independent Commission on the Los Angeles Police Department, Los Angeles, CA.
Jackson, B., et al. (2017). Managing officer behavioural risk using early intervention systems: addressing system design challenges for law enforcement and corrections environments. Policing : A Journal of Policy and Practice, 11(1), 103. https://doi.org/10.1093/police/paw028.
James, L., & James, S. (2017). Crisis intervention team (CIT) metrics: a novel method of measuring police performance during encounters with people in crisis. Mental Health and Addiction Research, 2(2), 1–4.
Kane, M., & Trochim, W. M. K. (2007). Concept mapping for planning and evaluation. Thousand Oaks, CA: Sage.
Kappeler, V. E., Sluder, R. D., & Alpert, G. P. (1998). Forces of deviance: understanding the dark side of policing. Prospect Heights, IL: Waveland Press.
Lersch, K. M. (2002). Are citizen complaints just another measure of officer productivity? An analysis of citizen complaints and officer activity measures. Police Practice and Research, 3(2), 135–147. https://doi.org/10.1080/15614260290033639.
Lersch, M., Kim, et al. (2006). Early intervention programs: an effective police accountability tool, or punishment of the productive? Policing: An International Journal of Police Strategies & Management, 29(1), 58–76. https://doi.org/10.1108/13639510610648485.
Mitchell, R., Ariel, B., Firpo, M., Fraiman, R., Del Castillo, F., Hyatt, J., et al. (2018). Measuring the effect of body-worn cameras on complaints in Latin America: The case of traffic police in Uruguay. Policing-An International Journal Of Police Strategies & Management, 41(4), 510–524.
Newman, J. (2015). Program that flags Chicago cops at risk of misconduct misses most officers. The Chicago Reporter.
Shjarback, J. A. (2015). Emerging early intervention systems: an agency-specific pre-post comparison of formal citizen complaints of use of force. Policing: A Journal of Policy and Practice, 9(4), 314–325. https://doi.org/10.1093/police/pav006.
Shjarback, J. A., Pyrooz, D. C., Wolfe, S. E., & Decker, S. H. (2017). De-policing and crime in the wake of Ferguson: racialized changes in the quantity and quality of policing among Missouri police departments. Journal of Criminal Justice, 50, 42–52.
Thurstone, L. L. (1959). Measurement of values. Chicago, IL: University of Chicago Press.
Thurstone, L. L., & Chave, E. J. (1929). The measurement of attitude: a psychophysical method and some experiments with a scale for measuring attitude toward the church. Chicago, IL: University of Chicago Press.
U.S. Commission on Civil Rights. (1981). Who is guarding the guardians? Washington, DC: Government Printing Office.
Vila, B., James, L., James, S. M., & Waggoner, L. B. (2012). “Final report: developing a common metric for evaluating police performance in deadly force situations”, Grant No. 2008IJCX0015. Washington, DC: National Institute of Justice.
Vila, B., James, L., & James, S. M. (2014). “Final report: empowering the strategic corporal: training young warfighters to be socially adept with strangers in any culture”, Grant No. W911NF-12-0039. Arlington, VA: Defense Advanced Research Projects Agency.
Vila, B., James, S. M., & James, L. (2016). Novel process for developing metrics that measure what police do. In Interservice/Industry Training, Simulation and Education Conference (I/ITSEC) proceedings, Orlando, Florida.
Vila, B., James, S., & James, L. (2018). How police officers perform in encounters with the public. Policing: An International Journal, 41(2), 215–232 https://doi.org/10.1108/PIJPSM-11-2016-0166.
Walker, S. (2010). Introduction to early intervention systems. In R. G. Dunham & G. P. Alpert (Eds.), Critical issues in police: Contemporary readings (6th ed., pp. 181–196). Long Grove, IL: Waveland Press, Inc..
Walker, S., & Alpert, G. P. (2002). Early warning systems as risk management for police. In K. M. Lersch (Ed.), Policing and Misconduct (pp. 219–230). Upper Saddle River, NJ: Prentice-Hall.
Walker, S., Alpert, G., & Kenney, D. (2000). Responding to the problem police officer: A national evaluation of early warning systems. Final report to the National Institute of Justice.
White, M. D., Gaub, J. E., & Todak, N. (2017). Exploring the potential for body-worn cameras to reduce violence in police–citizen encounters. Policing: A Journal of Policy and Practice, 12(1), 66–76.
Worden, R. E., Kim, M., Harris, C. J., Pratte, M. A., Dorn, S. E., & Hyland, S. S. (2013). Intervention with problem officers: an outcome evaluation of an EIS intervention. Criminal Justice and Behavior, 40(4), 409–437.
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
About this article
Cite this article
James, S., James, L. & Dotson, L. Evaluating the effectiveness of a police department’s early intervention system. J Exp Criminol 17, 457–471 (2021). https://doi.org/10.1007/s11292-019-09397-8
- Police behavior
- Early Intervention System
- Police accountability
- Performance metrics