Skip to main content
SpringerLink
Log in

Quasi-experimental designs for community-level public health violence reduction interventions: a case study in the challenges of selecting the counterfactual

Journal of Experimental Criminology volume 14, pages 155–185 (2018)Cite this article

Abstract

Objectives

We highlight the importance of documenting the step-by-step processes used for the selection of comparison areas when evaluating a community-level intervention that targets a large-scale community.

Methods

We demonstrate the proposed method using a propensity score matching framework for an impact analysis of the Cure Violence Public Health Model in Philadelphia. To select comparison communities, propensity score models are run using different levels of aggregation to define the intervention site. We discuss the trade-offs made.

Results

We find wide variation in documentation and explanation in the extant literature of the methods used to select comparison communities. The size of the unit of analysis at which a community is measured complicates the decision processes, and in turn, can affect the validity of the counterfactual.

Conclusions

It is important to carefully consider the unit of analysis for measurement of comparison communities. Assessing the geographic clustering of matched communities to mirror that of the treated community holds conceptual appeal and represents a strategy to consider when evaluating community-level interventions taking place at a large scale. Regardless of the final decisions made in the selection of the counterfactual, the field could benefit from more systematic diagnostic tools that document and guide the steps and decisions along the way, and ask: “could there have been another way of doing each step, and what difference would this have made?” Overall, across community-level evaluations that utilize quasi-experimental designs, documentation of the counterfactual selection process will provide a more fine-grained understanding of causal inference.

This is a preview of subscription content, access via your institution.

Access options

Buy single article

Instant access to the full article PDF.

39,95 €

Price includes VAT (Finland)

Fig. 1
Fig. 2
Fig. 3
Fig. 4

Notes

  1. Philadelphia CeaseFire originally emerged as a program in 2011 as a small pilot test to begin to build neighborhood support for full implementation of the Cure Violence Public Health Model. The pilot was comprised of three outreach workers and a supervisor, and their work was focused on one small hot spot within one Police Service Area in North Philadelphia.

  2. PSAs were established in 2009, after the new Police Commissioner (Charles Ramsey) was hired. Each PSA is headed by a police lieutenant, and includes an average of three sergeants and 39 officers who are responsible for patrolling the area. The idea is to increase police–community contact and officer involvement in the communities. The PSA model is considered a foundation of Philadelphia’s neighborhood policing strategy (Joyce 2016).

  3. Maps representing additional output with autocorrelated concentrated disadvantage included as a balancing variable are available from the authors upon request.

References

  • Abbott, A. (2001). Time matters: on theory and method. Chicago: University of Chicago Press.

    Google Scholar 

  • Apel, R. J., & Sweeten, G. (2010). Propensity score matching in criminology and criminal justice. In A. R. Piquero & D. Weisburd (Eds.), Handbook of quantitative criminology (pp. 543–562). New York: Springer.

    Chapter  Google Scholar 

  • Berk, R., Sherman, L., Barnes, G., Kurtz, E., & Ahlman, L. (2009). Forecasting murder within a population of probationers and parolees: A high stakes application of statistical learning. Journal of Royal Statistical Society, 172(1), 191–211.

    Article  Google Scholar 

  • Bernascco, W., & Block, R. (2009). Where offenders choose to attack: A discrete choice model of robberies in Chicago. Criminology, 47, 91–130.

    Article  Google Scholar 

  • Bjerregaard, B., & Lizotte, A. J. (1995). Gun ownership and gang membership. The Journal of Criminal Law and Criminology, 86(1), 37–58.

    Article  Google Scholar 

  • Boyle, D. J., Lanterman, J. L., Pascarella, J. E., & Cheng, C.-C. (2010). The impact of Newark’s Operation Ceasefire on trauma center gunshot wound admissions. Justice Research and Policy, 12(2), 105–123.

    Article  Google Scholar 

  • Braga, A. A. (2008). Pulling levers focused deterrence strategies and the prevention of gun homicide. Journal of Criminal Justice, 36(4), 332–343.

    Article  Google Scholar 

  • Braga, A. A., Hureau, D. M., & Papachristos, A. V. (2011). An ex post facto evaluation framework for place-based police interventions. Evaluation Review, 35(6), 592–626.

    Article  Google Scholar 

  • Braveman, P., Kumanyika, S., Fielding, J., & LaVeist, T. (2011). Health disparities and health equity: The issue is justice. American Journal of Public Health, 101(Suppl 1), S149–S155.

    Article  Google Scholar 

  • Browning, C. R., Byron, R. A., Calder, C. A., Krivo, L. J., & Kwan, M.-P. (2010). Commercial density, residential concentration, and crime: Land use patterns and violence in neighborhood context. Journal of Research in Crime and Delinquency, 47(3), 329–357.

    Article  Google Scholar 

  • Butts, J. A., Roman, C. G., Bostwick, L., & Porter, J. R. (2015). Cure Violence: a public health model to reduce gun violence. Annual Review of Public Health, 36, 39–53.

  • Caliendo, M., & Kopeining, S. (2008). Some practical guidance for the implementation of propensity score matching. Journal of Economic Surveys, 22(1), 31–72.

    Article  Google Scholar 

  • Centers for Disease Control and Prevention [CDC] (2010). Cooperative agreement program for the national academic centers of excellence in youth violence prevention. http://www.grants.gov/search/search.do?mode=VIEW&oppId=52740.

    Google Scholar 

  • Chatfield, C. (2004). The analysis of time series: an introduction. New York: CRC Press.

    Google Scholar 

  • Cohen, D. A., Inagami, S., & Finch, B. (2008). The built environment and collective efficacy. Health & Place, 14(2), 198–208 http://doi.org/10.1016/j.healthplace.2007.06.001.

    Article  Google Scholar 

  • Cohen, L., Davis, R., & Realini, A. (2016). Communities are not all created equal: strategies to prevent violence affecting youth in the United States. Journal of Public Health Policy, 37, S81–S94.

    Article  Google Scholar 

  • Cook, T. D., & Campbell, D. T. (1979). Quasi-experimentation: design and analysis for field settings. Chicago: Rand McNally.

    Google Scholar 

  • Corsaro, N., Brunson, R. K., & McGarrell, E. (2013). Problem-oriented policing and open-air drug markets examining the Rockford Pulling Levers Deterrence Strategy. Crime and Delinquency, 59(7), 1085–1107.

    Article  Google Scholar 

  • Corsaro, N., & Engel, R. S. (2015). Most challenging of contexts: Assessing the impact of focused deterrence on serious violence in New Orleans. Criminology and Public Policy, 14, 471–505.

    Article  Google Scholar 

  • Corsaro, N., & McGarrell, E. (2010). An evaluation of the Nashville Drug Market Initiative (DMI) Pulling Levers Strategy. East Lansing: Michigan State University, School of Criminal Justice.

    Google Scholar 

  • Engel, R. S., Tillyer, M. S., & Corsaro, N. (2013). Reducing gang violence using focused deterrence: evaluating the Cincinnati Initiative to Reduce Violence (CIRV). Justice Quarterly, 30, 403–439.

    Article  Google Scholar 

  • Farrell, A. D., Henry, D., Bradshaw, C., & Reischl, T. (2016). Designs for evaluating the community-level impact of comprehensive prevention programs: Examples from the CDC Centers of Excellence in Youth Violence Prevention. The Journal of Primary Prevention, 37, 165–188 http://doi.org/10.1007/s10935-016-0425-8.

    Article  Google Scholar 

  • Flay, B. R., Biglan, A., Boruch, R. F., Castro, F. G., Gottfredson, D., & Kellam, S. (2005). Standards of evidence: criteria for efficacy, effectiveness and dissemination. Prevention Science, 6, 151–175. https://doi.org/10.1007/s11121-005-5553-y.

    Article  Google Scholar 

  • Fox, A. M., Katz, C. M., Choate, D. E., & Hedberg, E. C. (2015). Evaluation of the Phoenix TRUCE Project: a replication of Chicago CeaseFire. Justice Quarterly, 32, 85–115.

    Article  Google Scholar 

  • Galster, G., Temkin, K., Walker, C., & Sawyer, N. (2004). Measuring the impacts of community development initiatives. Evaluation Review, 28(6), 1–38.

    Article  Google Scholar 

  • Haegerich, T. M., Mercy, J., & Weiss, B. (2013). What is the role of public health in gang-membership prevention? In T. Simon, N. Ritter, & R. Mahendra (Eds.), Changing course: preventing gang membership. Washington, D.C: U.S Department of Justice, Office of Justice Programs, and the Centers for Disease Control and Prevention.

    Google Scholar 

  • Haviland, A.M. & Nagin, D.S. (2007). Using group-based trajectory modeling in conjunction withpropensity scores to improve balance. Journal of Experimental Criminology, 3(1), 65-82.

  • Ho, D., Imai, K., King, G., & Stuart, E. A. (2007). Matching as nonparametric preprocessing for reducing model dependence in parametric causal inference. Political Analysis, 15, 199–236. http://gking.harvard.edu/files/abs/matchp-abs.shtml.

    Article  Google Scholar 

  • Holder, H. D., Treno, A. J., Saltz, R. F., & Grube, J. W. (1997). Recommendations and experiences for evaluation of community-level prevention programs. Evaluation Review, 21, 268–278.

    Article  Google Scholar 

  • Howell, J. C., & Decker, S. H. (1999). The youth gangs, drugs, and violence connection. Washington, DC: OJJDP Juvenile Justice Bulletin.

    Book  Google Scholar 

  • Imai, K., King, G., & Stuart, E. A. (2008). Misunderstandings among experimentalists and observationalists in causal inference. Journal of the Royal Statistical Society Series A, 171(2), 481–502.

    Article  Google Scholar 

  • Joyce, N. (2016). The Philadelphia Police Department: moving into the 21 st Century, 2008–2015. Philadelphia: Philadelphia Police Department Available: https://www.phillypolice.com/assets/directives/MovingThePhiladelphiaPoliceDepartmentIntoThe21stCentury.pdf. Accessed 13 Jan 2017.

    Google Scholar 

  • Ljung, G. M., & Box, G. E. P. (1978). Measure of lack of fit in time-series models. Biometrika, 65, 297–303.

    Article  Google Scholar 

  • Loughran, T. A., Wilson, T., Nagin, D. S., & Piquero, A. R. (2015). Evolutionary regression? Assessing the problem of hidden biases in criminal justice applications using propensity scores. Journal of Experimental Criminology, 11(4), 631–652.

    Article  Google Scholar 

  • Lubrano, A. (2014). Philadelphia rates highest among top 10 cities for deep poverty. In The Philadelphia Inquirer Available: http://www.philly.com/philly/news/20140925_Phila_s_deep_poverty_rate_highest_of_nation_s_10_most_populous_cities.html.

    Google Scholar 

  • Matjasko, J. L., Massetti, G. M., & Bacon, S. (2016). Implementing and evaluating comprehensive evidence-based approaches to youth violence prevention: partnering to create communities where youth are safe from violence. The Journal of Primary Prevention, 37, 109–119.

    Article  Google Scholar 

  • McGarrell, E. F., Chermak, S., Wilson, J. M., & Corsaro, N. (2006). Reducing homicide through a “lever-pulling” strategy. Justice Quarterly, 23(2), 214–231.

    Article  Google Scholar 

  • Meyer, B. D. (1995). Natural and quasi-experiments in economics. Journal of Business and Economic Statistics, 13(2), 151–162.

    Google Scholar 

  • Morenoff, J. D., & Sampson, R. J. (1997). Violent crime and the spatial dynamics of neighborhood transition: Chicago, 1970–1990. Social Forces, 76, 31–64.

    Article  Google Scholar 

  • Mozaffarian, D., Hemenway, D., & Ludwig, D. S. (2013). Curbing gun violence: lessons from public health successes. Journal of the American Medical Association, 309(6), 551–552. https://doi.org/10.1001/jama.2013.38.

    Article  Google Scholar 

  • Nagin, D. S., Cullen, F. T., & Jonson, C. L. (2009). Imprisonment and reoffending. In M. Tonry (Ed.), Crime and justice: a review of research (Vol. 38, pp. 115–200). Chicago: University of Chicago Press.

    Google Scholar 

  • Papachristos, A. V., Meares, T. L., & Fagan, J. (2007). Attention felons: evaluating Project Safe Neighborhoods in Chicago. Journal of Empirical Legal Studies, 4(2), 223–272.

    Article  Google Scholar 

  • Picard-Fritsche, S. & Cerniglia, L. (2013). Testing a public health approach to gun violence: An evaluation of Crown Heights Save Our Streets, a replication of the Cure Violence Model. Center for Court Innovation. http://www.courtinnovation.org/sites/default/files/documents/SOS_Evaluation.pdf. Accessed 2 February 2017.

  • Pratt, T.C. & Cullen, F.T. (2005). Assessing macro-level predictors and theories of crime: A meta-analysis.Crime and Justice, 32, 373-450.

  • Pawson, R., & Tilley, N. (1994). What works in evaluation research? British Journal of Criminology, 31(3), 291–306.

  • Penfold, R. B., & Zhang, F. (2013). Use of interrupted time series analysis in evaluating health care quality improvements. Academic Pediatrics, 13(6), S38–S44.

    Article  Google Scholar 

  • Pew. (2015). Philadelphia 2015: state of the city. Philadelphia: Pew Trusts Retrieved from http://www.pewtrusts.org/~/media/assets/2015/11/2015-state-of-the-city-report_web_v2.pdf.

    Google Scholar 

  • Picard-Fritsche, S., & Cerniglia, L. (2010). Testing a public health approach to gun violence: An evaluation of Crown Heights Save Our Streets, a replication of the Cure Violence Model. New York: Center for Court Innovation.

    Google Scholar 

  • Roman, C., Link, N., Mayes, L., & Hyatt, J. (2015). Gang-involvement in shootings: preliminary analysis of gang-involved shootings in Philadelphia, 2009–2015. Report to the Focused Deterrence Executive Committee. Philadelphia: Temple University.

    Google Scholar 

  • Rosenbaum, P.R. & Rubin, D.B. (1983). The central role of the propensity score in observational studiesfor causal effects. Biometrika, 70(1), 41-55.

  • Rosenbaum, P. R., & Rubin, D. B. (1985). Constructing a control group using multivariate matched sampling methods that incorporate the propensity score. The American Statistician, 39(1), 33–38.

    Google Scholar 

  • Satcher, D. (1995). Violence as a public health issue. Bulletin of the New York Academy of Medicine, 72(1), 46–56.

    Google Scholar 

  • Saunders, J., Lundberg, R., Braga, A. A., Ridgeway, G., & Miles, J. (2015). A synthetic control approach to evaluating place-based crime interventions. Journal of Quantitative Criminology, 31, 413–434. https://doi.org/10.1007/s10940-014-9226-5.

    Article  Google Scholar 

  • Skogan, W.G., Hartnett, S.M., Bump, N., & Dubois, J. (2008). Evaluation of CeaseFire Chicago. Report for Office of Justice Programs, US Department of Justice.

    Google Scholar 

  • Spergel, I. A., Wa, K. M., & Sosa, R. V. (2005a). Evaluation of the Mesa Gang Intervention Program (MGIP). Chicago: School of Social Service Administration, The University of Chicago.

    Google Scholar 

  • Spergel, I. A., Wa, K. M., & Sosa, R. V. (2005b). Evaluation of the Riverside Comprehensive Community-Wide Approach to Gang Prevention. Chicago: School of Social Service Administration, The University of Chicago.

    Google Scholar 

  • Stuart, E. A. (2010). Matching methods for causal inference: a review and a look forward. Statistical Science, 25(1), 1–21.

    Article  Google Scholar 

  • Taylor, R. B. (1995). The impact of crime on communities. The Annals of the American Academy of Political and Social Science, 539(1), 28–45.

    Article  Google Scholar 

  • Taylor, R. B., Ratcliffe, J. H., & Perenzin, A. (2015). Can we predict long-term community crime problems? The estimation of ecological continuity to model risk heterogeneity. Journal of Research in Crime and Delinquency, 52(3), 635–657.

    Article  Google Scholar 

  • Tita, G. E., Riley, K. J., Ridgeway, G., Grammich, C., Abrahamse, A., & Greenwood, P. W. (2003). Reducing gun violence: results from an Intervention in East Los Angeles. Santa Monica: RAND Corporation.

    Book  Google Scholar 

  • Virginia Commonwealth University. (2016). Philadelphia life expectancy methodology and data table. Richmond: Virginia Commonwealth University, Center on Society and Health http://societyhealth.vcu.edu/media/society-health/pdf/LE-Map-Philly-Methods.pdf. Accessed 2 Mar 2016.

    Google Scholar 

  • Webster, D. W., Whitehall, J. M., Vernick, J. S., & Parker, E. M. (2012). Evaluation of Baltimore’s Safe Streets Program: effects on attitudes, participants’ experiences, and gun violence. Baltimore: Johns Hopkins Bloomberg School of Public Health.

    Google Scholar 

  • Wellford, C., Pepper, J., & Petrie, C. (2005). Firearms and violence: a critical review. Washington: National Science Academies Press.

    Google Scholar 

  • Wilson, J. M., Chermak, S., & McGarrell, E. F. (2010). Community-based violence prevention: an assessment of Pittsburgh’s One Vision One Life Program. Santa Monica: RAND.

    Google Scholar 

Download references

Acknowledgements

Data collection and evaluation of Philadelphia CeaseFire was supported by Award No. 2012-PB-FX-K004, from the Office of Juvenile Justice and Delinquency Prevention, Office of Justice Programs, U.S. Department of Justice. The content of this paper, however, is solely the responsibility of the authors, and does not necessarily represent the official views of the U.S. Department of Justice.

Author information

Authors and Affiliations

  1. Department of Criminal Justice, Temple University, 1115 Polett Walk, 5th FL, Gladfelter Hall, Philadelphia, Pennsylvania, 19122, United States

    Caterina G. Roman & Hannah J. Klein

  2. Department of Criminal Justice, John Jay College of Criminal Justice, 524 W. 59th Street, New York, NY, 10019, United States

    Kevin T. Wolff

Authors
  1. Caterina G. Roman
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Hannah J. Klein
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Kevin T. Wolff
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Caterina G. Roman.

Rights and permissions

Reprints and Permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Roman, C.G., Klein, H.J. & Wolff, K.T. Quasi-experimental designs for community-level public health violence reduction interventions: a case study in the challenges of selecting the counterfactual. J Exp Criminol 14, 155–185 (2018). https://doi.org/10.1007/s11292-017-9308-0

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11292-017-9308-0

Keywords

  • Counterfactual
  • Evaluation
  • Place-based
  • Propensity score matching
  • Violence

Search

Navigation