Analyzing block randomized studies: the example of the Jersey City drug market analysis experiment
While block randomized designs have become more common in place-based policing studies, there has been relatively little discussion of the assumptions employed and their implications for statistical analysis. Our paper seeks to illustrate these assumptions, and controversy regarding statistical approaches, in the context of one of the first block randomized studies in criminal justice—the Jersey City Drug Market Analysis Project (DMAP).
Using DMAP data, we show that there are multiple approaches that can be used in analyzing block randomized designs, and that those approaches will yield differing estimates of statistical significance. We develop outcomes using both models with and without interaction, and utilizing both Type I and Type III sums-of-squares approaches. We also examine the impacts of using randomization inference, an approach for estimating p values not based on approximations using normal distribution theory, to adjust for possible small N biases in estimating standard errors.
The assumptions used for identifying the analytic approach produce a comparatively wide range of p values for the main DMAP program impacts on hot spots. Nonetheless, the overall conclusions drawn from our re-analysis remain consistent with the original analyses, albeit with more caution. Results were similar to the original analyses under different specifications supporting the identification of diffusion of benefits effects to nearby areas.
The major contribution of our article is to clarify statistical modeling in unbalanced block randomized studies. The introduction of blocking adds complexity to the models that are estimated, and care must be taken when including interaction effects in models, whether they are ANOVA models or regression models. Researchers need to recognize this complexity and provide transparent and alternative estimates of study outcomes.
KeywordsBlock randomized design Drug markets Policing
We would like to thank Anthony Braga, Donald Green, and Alese Wooditch for helpful comments on earlier drafts of this paper, and Tori Goldberg for help in preparing the manuscript.
- Ariel, B., & Farrington, D. P. (2010). Randomized block designs. In A. Piquero & D. Weisburd (Eds) Handbook of quantitative criminology (pp. 437–454). New York: Springer.Google Scholar
- Bayley, D. H. (1994). Police for the future (studies in crime and public policy). Oxford: Oxford University Press.Google Scholar
- Blattman, C., Green, D., Ortega, D., & Tobón, S. (2017). Pushing crime around the corner? Estimating experimental impacts of large-scale security interventions (no. w23941). National Bureau of Economic Research.Google Scholar
- Braga, A. A., & Weisburd, D. (2006). Problem-oriented policing: the disconnect between principles and practice. In D. Weisburd & A. A. Braga (Eds.), Police innovation: Contrasting perspectives (pp. 133–152). Cambridge: Cambridge University Press.Google Scholar
- Braga, A. A., Green, L. A., Weisburd, D. L., & Gajewski, F. (1994). Police perceptions of street-level narcotics activity: evaluating drug buys as a research tool. American Journal of Police, 13, 37.Google Scholar
- Braga, A., Papachristos, A., & Hureau, D. (2012). Hot spots policing effects on crime. Campbell Systematic Reviews, 8(8), 1–96.Google Scholar
- Braga, A.A., A. Papachristos, D. Hureau, & Turchan, B.. (2017). “Hot spots policing and crime control: an updated systematic review and meta-analysis.” American Society of Criminology Annual Meetings.Google Scholar
- Clarke, R. V., & Weisburd, D. (1994). Diffusion of crime control benefits: observations on the reverse of displacement. Crime Prevention Studies, 2, 165–184.Google Scholar
- Cohen, J. (1988). Statistical power analysis for the behavioral sciences 2nd edn. Hillsdale: Lawrence Erlbaum Associates.Google Scholar
- Feise, R.J. (2002). Do multiple outcome measures require p-value adjustment?. BMC Medical Research Methodology 2(8).Google Scholar
- Fisher, R. A. (1925). Statistical methods for research workers. Edinburgh: Oliver and Boyd.Google Scholar
- Gerber, A. S., & Green, D. P. (2012). Field experiments: design, analysis, and interpretation. New York: WW Norton.Google Scholar
- Gill, C. E., & Weisburd, D. (2013). Increasing equivalence in small-sample place-based experiments taking advantage of block randomization methods. Experimental criminology: Prospects for advancing. Science and Public Policy, 141–162.Google Scholar
- Goldstein, H. (1990). Problem-oriented policing. New York: McGraw-Hill.Google Scholar
- Gottfredson, M. R., & Hirschi, T. (1990). A general theory of crime. Stanford: Stanford University Press.Google Scholar
- Jaccard, J., & Turrisi, R. (2003). Interaction effects in multiple regression (no. 72). Thousand Oaks: Sage.Google Scholar
- Jaccard, J., Turrisi, R., & Wan, C. K. (1990). Implications of behavioral decision theory and social marketing for designing social action programs. In J. Edwards, R.S. Tindel, L. Heath, & E.J. Posavac (Eds.) In Social influence processes and prevention (pp. 103-142). Boston: Springer.Google Scholar
- Kirk, R. E. (1982). Experimental design. Hoboken: John Wiley & Sons, Inc.Google Scholar
- Lipsey, M. W. (1990). Design sensitivity: Statistical power for experimental research (Vol. 19). Thousand Oaks: Sage. Google Scholar
- Pierce, G., Spaar, S., & Briggs, L. (1988). The character of police work: strategic and tactical implications. Boston: NortheasternUniversity Press.Google Scholar
- Sherman, L. W. (1987). Repeat calls to police in minneapolic. Crime Control Reports No. 5. Washington, D.C.: Crime Control Institute.Google Scholar
- Venables, W. N. (1998). Exegeses on linear models. In S-Plus User’s Conference, Washington DC.Google Scholar
- Weisburd, D., & Green, L. (1991). Identifying and controlling drug markets: The drug market analysis program (phase 2). Funded Proposal to the National Institute of Justice.Google Scholar
- Weisburd, D., & Green, L. (1994). Defining the drug market: the case of Jersey City's DMAP system. Drugs and Crime: Evaluating Public Policy Initiatives. Newbury Park: Sage.(1995)." Policing drug hotspots: Findings from the Jersey City DMA." justice quarterly.Google Scholar
- Weisburd, D., Lum, C., & Yang, S. M. (2003). When can we conclude that treatments or programs ‘Don’t work’? The Annals of the American Academy of Social and Political. Sciences, 587(May), 31–48.Google Scholar