Abstract
Objectives
Propensity score methods rely on an untestable assumption of unconfoundedness for making causal inference. Yet, empirical applications using propensity scores in criminology routinely invoke this assumption without careful scrutiny.
Methods
We use a dataset with a wide range of observable, potential confounders, which allows us to evaluate recidivism outcomes for adolescent offenders who are sentenced to either placement or probation. We then systematically withhold important known confounders from the matching process to demonstrate the effectiveness of sensitivity checks in sizing up the robustness of these treatment effect estimates in the case where hidden biases clearly exist.
Results
We find important variability in the estimated treatment effect, and a large degree of imbalance in ‘unobserved’ covariates, which we did not explicitly control for. The hidden biases observed in our controlled analysis would have at least been suggested in an actual application by the low gamma statistics that attended our analysis, a statistic that is not reported in most criminological applications of propensity score analysis.
Conclusions
Researchers who use propensity score methods should openly discuss potential limitations of their analysis due to hidden bias and report bias sensitivity checks based on the gamma statistic when statistically significant treatment effects are reported.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Angrist, J. D. (2006). Instrumental variables methods in experimental criminological research: what, why and how. Journal of Experimental Criminology, 2(1), 23–44.
Apel, R. J., & Sweeten, G. (2010). Propensity score matching in criminology and criminal justice. In A. R. Piquero & W. David (Eds.), Handbook of quantitative criminology (pp. 543–562). New York: Springer.
Bales, W. D., & Piquero, A. R. (2012a). Assessing the impact of imprisonment on recidivism. Journal of Experimental Criminology, 8(1), 71–101.
Bales, W. D., & Piquero, A. R. (2012b). Racial/ethnic differentials in sentencing to incarceration. Justice Quarterly, 29(5), 742–773.
Becker, S. O., & Caliendo, M. (2007). Sensitivity analysis for average treatment effects. The Stata Journal, 2, 358–377.
Berk, R., Barnes, G., Ahlman, L., & Kurtz, E. (2010). When second best is good enough: A comparison between a true experiment and a regression discontinuity quasi-experiment. Journal of Experimental Criminology, 6(2), 191–208.
Bhati, A. S., & Piquero, A. R. (2007). Estimating the impact of incarceration on subsequent offending trajectories: Deterrent, criminogenic, or null effect? The Journal of Criminal Law and Criminology, 98(1), 207–253.
Cochran, J. C., Mears, D. P., & Bales, W. D. (2014). Assessing the Effectiveness of Correctional Sanctions. Journal of Quantitative Criminology, 30(2), 317–347.
Cornfield, J., Haenszel, W., Hammond, E., Lilienfield, A., Shimkin, M., & Wynder, E. (1959). Smoking and lung cancer: Recent evidence and a discussion of some questions. Journal of the National Cancer Institute, 22, 173–203.
Dehejia, R. (2005). Practical propensity score matching: a reply to Smith and Todd. Journal of Econometrics, 125(1), 355–364.
Dehejia, R. H., & Wahba, S. (1999). Causal effects in nonexperimental studies: Reevaluating the evaluation of training programs. Journal of the American Statistical Association, 94(448), 1053–1062.
Dehejia, R. H., & Wahba, S. (2002). Propensity score-matching methods for nonexperimental causal studies. Review of Economics and Statistics, 84(1), 151–161.
DiPrete, T. A., & Gangl, M. (2004). Assessing bias in the estimation of causal effects: Rosenbaum bounds on matching estimators and instrumental variables estimation with imperfect instruments. Sociological Methodology, 34(1), 271–310.
Durose, M. R., Cooper, A. D., & Snyder, H. N. (2014). Recidivism of prisoners released in 30 states in 2005: Patterns from 2005 to 2010. Bureau of Justice Statistics.
Heckman, J. J. (1979). Sample selection bias as a specification error. Econometrica: Journal of the Econometric Society, 153–161.
Heckman, J. J., Ichimura, H., & Todd, P. E. (1997). Matching as an econometric evaluation estimator: Evidence from evaluating a job training programme. The Review of Economic Studies, 64(4), 605–654.
Heckman, J. J., Ichimura, H., & Todd, P. (1998). Matching as an econometric evaluation estimator. The Review of Economic Studies, 65(2), 261–294.
Heckman, J., & Navarro-Lozano, S. (2004). Using matching, instrumental variables, and control functions to estimate economic choice models. Review of Economics and Statistics, 86(1), 30–57.
Heckman, J. J., & Robb, R., Jr. (1985). Alternative methods for evaluating the impact of interventions: An overview. Journal of Econometrics, 30(1), 239–267.
Holland, P. W. (1986). Statistics and causal inference. Journal of the American statistical Association, 81(396), 945–960.
Jolliffe, D., & Hedderman, C. (2012). Investigating the impact of custody on reoffending using propensity score matching. Crime & Delinquency. doi:10.1177/0011128712466007.
Kuo, Y. H. (2002). Extrapolation of correlation between 2 variables in 4 general medical journals. JAMA, 287(21), 2815–2817.
LaLonde, R. J. (1986). Evaluating the econometric evaluations of training programs with experimental data. The American Economic Review, 604–620.
Loughran, T. A., & Mulvey, E. P. (2010). Estimating treatment effects: Matching quantification to the question. In A. R. Piquero & W. David (Eds.), Handbook of quantitative criminology (pp. 163–180). New York: Springer.
Loughran, T. A., Mulvey, E. P., Schubert, C. A., Fagan, J., Piquero, A. R., & Losoya, S. H. (2009). Estimating a dose–response relationship between length of stay and future recidivism in serious juvenile offenders. Criminology, 47(3), 699–740.
Mulvey, E. P. (2012). Research on Pathways to Desistance [Maricopa County, AZ and Philadelphia County, PA]: subject measures, 2000–2010. ICPSR29961-v1. Inter-university consortium for political and social research [distributor]. Ann Arbor, MI, 20.
Mulvey, E. P., Schubert, C. A., & Piquero, A. (2014). Pathways to Desistance-Final Technical Report.
Mulvey, E. P., Steinberg, L., Fagan, J., Cauffman, E., Piquero, A. R., Chassin, L., Knight, G. P., Brame, R., Schubert, C., Hecker, T., & Losoya, S. H. (2004). Theory and research on desistance from antisocial activity among serious adolescent offenders. Youth Violence and Juvenile Justice, 2(3), 213–236.
Nagin, D. S., Cullen, F. T., & Jonson, C. L. (2009). Imprisonment and reoffending. Crime and Justice, 38(1), 115–200.
Pearl, J. (2009). Letter to the Editor: Remarks on the method of propensity score. Department of Statistics, UCLA.
Pearson, E. S. (1938). Karl Pearson. An appreciation of some aspects of his life and work. Cambridge: Cambridge University Press.
Porter, A. M. (1999). Misuse of correlation and regression in three medical journals. Journal of the Royal Society of Medicine, 92(3), 123–128.
Rosenbaum, P. R. (2002). Observational studies. New York: Springer.
Rosenbaum, P. R. (2005). Sensitivity analysis in observational studies. In B. S. Everitt & D. C. Howell (Eds.), Encyclopedia of Statistics in Behavioral Science (pp. 1809–1814). Chichester: John Wiley & Sons.
Rosenbaum, P. R., & Rubin, D. B. (1983). The central role of the propensity score in observational studies for causal effects. Biometrika, 70(1), 41–55.
Rosenbaum, P. R., & Rubin, D. B. (1984). Reducing bias in observational studies using subclassification on the propensity score. Journal of the American Statistical Association, 79(387), 516–524.
Rosenbaum, P. R., & Rubin, D. B. (1985). Constructing a control group using multivariate matched sampling methods that incorporate the propensity score. The American Statistician, 39(1), 33–38.
Rubin, D. B. (1974). Estimating causal effects of treatments in randomized and nonrandomized studies. Journal of Educational Psychology, 66(5), 688.
Rubin, D. B. (1977). Assignment to Treatment Group on the Basis of a Covariate. Journal of Educational and Behavioral Statistics, 2(1), 1–26.
Rubin, D. B. (2008). For Objective Causal Inference, Design Trumps Analysis. Annals of Applied Statistics, 2(3), 808–840.
Schubert, C. A., Mulvey, E. P., Steinberg, L., Cauffman, E., Losoya, S. H., Hecker, T., Chassin, L., & Knight, G. P. (2004). Operational lessons from the pathways to desistance project. Youth Violence and Juvenile Justice, 2(3), 237–255.
Shadish, W. R. (2013). Propensity score analysis: promise, reality and irrational exuberance. Journal of Experimental Criminology, 9(2), 129–144.
Shah, B. R., Laupacis, A., Hux, J. E., & Austin, P. C. (2005). Propensity score methods gave similar results to traditional regression modeling in observational studies: a systematic review. Journal of Clinical Epidemiology, 58(6), 550–559.
Smith, J. A., & Todd, P. E. (2005). Does matching overcome LaLonde's critique of nonexperimental estimators? Journal of Econometrics, 125(1), 305–353.
Thornberry, T. P., Krohn, M. D., Lizotte, A. J., Smith, C. A., & Porter, P. K. (1998). Taking stock: An overview of findings from the Rochester Youth Development Study. (Paper presented at the 54th Annual Meeting of the American Society of Criminology, Washington, DC)
Travis, J., Western, B., & Redburn, S. (Eds.). (2014). The growth of incarceration in the United States: Exploring causes and consequences. Washington, DC: The National Academies Press.
Winkelmayer, W. C., & Kurth, T. (2004). Propensity scores: help or hype? Nephrology Dialysis Transplantation, 19(7), 1671–1673.
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Loughran, T.A., Wilson, T., Nagin, D.S. et al. Evolutionary regression? Assessing the problem of hidden biases in criminal justice applications using propensity scores. J Exp Criminol 11, 631–652 (2015). https://doi.org/10.1007/s11292-015-9242-y
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11292-015-9242-y