Abstract
Objective
Conduct a multi-site, retrospective investigation of predictive bias and disparate impact post-implementation of the Indiana Risk Assessment System-Pretrial Assessment Tool (IRAS-PAT).
Methods
Black and White defendants who received IRAS-PAT assessments (n = 2,570) were matched to two comparison conditions (n = 1,527 and n = 3,107) of defendants who did not receive assessments. Area under the curve statistics and multivariable logistic regression models tested for predictive bias. Weighted, multivariable mixed-effects models examined effects of assessments on release decisions by race.
Results
IRAS-PAT assessments produced lower levels of predictive validity for Black defendants relative to White defendants. Although there were disparities in pretrial release rates, bond amount, and days in detention, there was no evidence that effects of pretrial risk assessments differed by race.
Conclusions
Even a tool producing biased assessments of risk can improve pretrial release outcomes for defendants irrespective of race and relative to practice as usual.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Angwin, J., Larson, J., Mattu, S., & Kirchner, L. (2016). Machine bias. ProPublica. https://www.propublica.org/article/machine-bias-risk-assessments-in-criminal-sentencing
Arnold, D., Dobbie, W., & Yang, C. S. (2018). Racial bias in bail decisions. The Quarterly Journal of Economics, 133(4), 1885–1932. https://doi.org/10.1093/qje/qjy012
Arnold Ventures. (2019). Public Safety Assessment FAQs (“PSA 101”). Houston, TX: Arnold Ventures. https://craftmediabucket.s3.amazonaws.com/uploads/Public-Safety-Assessment-101_190319_140124.pdf
Ayres, I., & Waldfogel, J. (1994). A market test for race discrimination in bail setting. Stanford Law Review, 46(5), 987. https://doi.org/10.2307/1229062
Bales, W. D., & Piquero, A. R. (2012). Racial/ethnic differentials in sentencing to incarceration. Justice Quarterly, 29(5), 742–773. https://doi.org/10.1080/07418825.2012.659674
Berk, R., Heidari, H., Jabbari, S., Kearns, M., & Roth, A. (2017). Fairness in criminal justice risk assessments: The state of the art. arXiv. https://arxiv.org/abs/1703.09207v2
Callis, R. R., & Kresin, M. (2016). Residential vacancies and homeownership in the second quarter. Washington, D. C.: U.S. Census Bureau, U.S. Department of Commerce.
Chouldechova, A. (2017). Fair prediction with disparate impact: A study of bias in recidivism prediction instruments. Big Data, 5(2), 153–163. https://doi.org/10.1089/big.2016.0047
Cleary, T. A. (1968). Test bias: Prediction of grades of Negro and White students in integrated colleges. Journal of Educational Measurement, 5(2), 115–124.
Cohen, T. H., & Reaves, B. A. (2007). Pretrial release of felony defendants in state courts (No. NCJ 214994) (p. 18). Washington, D.C.: Bureau of Justice Statistics, Office of Justice Programs, U.S. Department of Justice.
Cohen, T. H., & Lowenkamp, C. T. (2019). Revalidation of the Federal PTRA: Testing the PTRA for predictive biases. Criminal Justice and Behavior, 46(2), 234–260. https://doi.org/10.1177/0093854818810315
Cohen, T. H., Lowenkamp, C. T., & Hicks, W. E. (2018). Revalidating the Federal Pretrial Risk Assessment instrument (PTRA): A research summary. Federal Probation, 82(2), 7.
Copp, J. E., Blomberg, T. G., Casey, W., & Pesta, G. (2019). Validation of the Virginia Pretrial Risk Assessment Instrument-Revised in Palm Beach County, Florida: Estimates of overall predictive validity and assessment of predictive bias.
Corbett-Davies, S., Pierson, E., Feller, A., Goel, S., & Huq, A. (2017). Algorithmic decision making and the cost of fairness. In Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (pp. 797–806). New York, NY, USA: ACM Press. https://doi.org/10.1145/3097983.309809
Curran, P. J., & Hussong, A. M. (2009). Integrative data analysis: The simultaneous analysis of multiple data sets. Psychological Methods, 14(2), 81–100. https://doi.org/10.1037/a0015914
Danner, M. J. E., VanNostrand, M., & Spruance, L. M. (2016). Race and gender neutral pretrial risk assessment release recommendations, and supervision: VPRAI and Praxis revised. St. Petersburg, FL: Luminosity, Inc. https://www.ncsc.org/__data/assets/pdf_file/0021/1668/race-and-gender-neutral-pretrial-risk-assessment-release-recommendations-and-supervision.ashx.pdf
DeMichele, M., & Baumgartner, P. (2020). Bias testing of the Public Safety Assessment: error rate balance between Whites and Blacks for new arrests. Crime & Delinquency, 0011128720965116. https://doi.org/10.1177/0011128720965116
DeMichele, M., Baumgartner, P., Barrick, K., Comfort, M., Scaggs, S., & Misra, S. (2018). What do criminal justice professionals think about risk assessment at pretrial? (SSRN Scholarly Paper No. ID 3168490). Rochester, NY: Social Science Research Network. https://papers.ssrn.com/abstract=3168490. Accessed 3 September 2018
DeMichele, M., Baumgartner, P., Wenger, M., Barrick, K., & Comfort, M. (2020). Public safety assessment: Predictive utility and differential prediction by race in Kentucky. Criminology & Public Policy, 19(2), 409–431. https://doi.org/10.1111/1745-9133.12481
Demuth, S. (2006). Racial and ethnic differences in pretrial release decisions and outcomes: A comparison of Hispanic, Black, and White felony arrestees. Criminology, 41(3), 873–908. https://doi.org/10.1111/j.1745-9125.2003.tb01007.x
Demuth, S., & Steffensmeier, D. (2004). The impact of gender and race-ethnicity in the pretrial release process. Social Problems, 51(2), 222–242. https://doi.org/10.1525/sp.2004.51.2.222
Desmarais, S. L., Johnson, K. L., & Singh, J. P. (2016). Performance of recidivism risk assessment instruments in U.S. correctional settings. Psychological Services, 13(3).
Desmarais, S. L., Monahan, J., & Austin, J. (2021). The empirical case for pretrial risk assessment instruments. Criminal Justice and Behavior, 00938548211041651. https://doi.org/10.1177/00938548211041651
Desmarais, S. L., & Singh, J. P. (2013). Risk assessment instruments validated and implemented in correctional settings in the United States. Council of State Governments Justice Center.
Desmarais, S. L., Zottola, S. A., Duhart Clarke, S. E., & Lowder, E. M. (2020). Predictive validity of pretrial risk assessments: a systematic review of the literature. Criminal Justice and Behavior, 0093854820932959.https://doi.org/10.1177/0093854820932959
Desmarais, S. L., & Lowder, E. M. (2019). Pretrial risk assessment tools: A primer for judges, prosecutors, and defense attorneys. Safety and Justice Challenge, John D. and Catherine T. MacArthur Foundation. http://www.safetyandjusticechallenge.org/wp-content/uploads/2019/02/Pretrial-Risk-Assessment-Primer-February-2019.pdf
Flores, A. W., Bechtel, K., & Lowenkamp, C. T. (2016). False positives, false negatives, and false analyses: A rejoinder to “machine bias: There’s software used across the country to predict future criminals. And it’s biased against blacks.” Federal Probation, 80(2), 38–46.
Freiburger, T. L., Marcum, C. D., & Pierce, M. (2010). The impact of race on the pretrial decision. American Journal of Criminal Justice, 35(1–2), 76–86. https://doi.org/10.1007/s12103-009-9069-z
Gelman, A., Fagan, J., & Kiss, A. (2007). An analysis of the New York City Police Department’s “Stop-and-Frisk” policy in the context of claims of racial bias. Journal of the American Statistical Association, 102(479), 813–823. https://doi.org/10.1198/016214506000001040
Grove, W. M., Zald, D. H., Lebow, B. S., Snitz, B. E., & Nelson, C. (2000). Clinical versus mechanical prediction: A meta-analysis. Psychological Assessment, 12(1), 19–30.
Harcourt, B. E. (2015). Risk as a proxy for race: The dangers of risk assessment. Federal Sentencing Reporter, 27(4), 237–243. https://doi.org/10.1525/fsr.2015.27.4.237
Ho, D., Imai, K., King, G., & Stuart, E. (2011). MatchIt: Nonparametric preprocessing for parametric causal inference. Journal of Statistical Software, 42(8).
Holder, E. (2014). Attorney General Eric Holder Speaks at the National Association of Criminal Defense Lawyers 57th Annual Meeting. http://www.justice.gov/opa/speech/attorney-general-eric-holder-speaks-nationalassociation-criminal-defense-lawyers-57th
Katz, C. M., & Spohn, C. C. (1995). The effect of race and gender on bail outcomes: A test of an interactive model. American Journal of Criminal Justice, 19(2), 161–184. https://doi.org/10.1007/BF02885913
Kleinberg, J., Mullainathan, S., & Raghavan, M. (2016). Inherent trade-offs in the fair determination of risk scores. arXiv. 1609.05807
Kutateladze, B. L., Andiloro, N. R., Johnson, B. D., & Spohn, C. C. (2014). Cumulative disadvantage: Examining racial and ethnic disparity in prosecution and sentencing. Criminology, 52(3), 514–551. https://doi.org/10.1111/1745-9125.12047
Latessa, E. J., Lovins, B. K., & Makarios, M. (2013). Validation of the Indiana Risk Assessment System: Final report. Cincinnati, OH: University of Cincinnati, Center for Criminal Justice Research. https://www.in.gov/courts/iocs/files/prob-risk-iras-final.pdf
Latessa, E. J., Smith, P., Lemke, R., Makarios, M., & Lowenkamp, C. T. (2009). Creation and validation of the Ohio Risk Assessment System: Final report. Cincinnati, OH: Center for Criminal Justice Research, School of Criminal Justice, University of Cincinnati. http://www.uc.edu/content/dam/uc/ccjr/docs/reports/project_reports/ORAS_Final_Report.pdf. Accessed 9 April 2018
Latessa, E., Lovins, B., & Makorios, M. (2013). Validation of the Indiana Risk Assessment System: Final report. Cincinnati, OH: Center for Criminal Justice Research, School of Criminal Justice, University of Cincinnati.
Lattimore, P. K., Tueller, S., Levin-Rector, A., & Witwer, A. (2020). The prevalence of local criminal justice practices. Federal Probation, 84(1), 28–37.
Lin, Z. “Jerry,” Jung, J., Goel, S., & Skeem, J. (2020). The limits of human predictions of recidivism. Science Advances, 6(7), eaaz0652. https://doi.org/10.1126/sciadv.aaz0652
Lowder, E. M., Diaz, C. L., Grommon, E., & Ray, B. R. (2021). Effects of pretrial risk assessments on release decisions and misconduct outcomes relative to practice as usual. Journal of Criminal Justice, 73, 101754. https://doi.org/10.1016/j.jcrimjus.2020.101754
Lowder, E. M., Lawson, S. G., Grommon, E., & Ray, B. R. (2020). Five-county validation of the Indiana Risk Assessment System – Pretrial Assessment Tool (IRAS-PAT) using a local validation approach. Justice Quarterly. https://doi.org/10.1080/07418825.2020.1829006
Lowder, E. M., & Wilson, D. B. (2021). Pretrial risk assessment validation research: Range restriction and attenuation of predictive validity estimates. Law and Human Behavior, 45(4), 324–335. https://doi.org/10.1037/lhb0000458
Lowenkamp, C. T., VanNostrand, M., & Holsinger, A. (2013). The hidden costs of pretrial detention. Laura and John Arnold Foundation. https://nicic.gov/hidden-costs-pretrial-detention. Accessed 9 April 2018
Lowenkamp, C. T., & Whetzel, J. (2009). The development of an actuarial risk assessment instrument for U.S. pretrial services. Federal Probation, 73(2), 33–36.
Mamalian, C. (2011). State of the science of pretrial risk assessment. Pretrial Justice Institute. https://www.pretrial.org/download/risk-assessment/PJI%20State%20of%20the%20Science%20Pretrial%20Risk%20Assessment%20(2011).pdf. Accessed 9 April 2018
Marlowe, D. B., Ho, T., Carey, S. M., & Chadick, C. D. (2020). Employing standardized risk assessment in pretrial release decisions: Association with criminal justice outcomes and racial equity. Law and Human Behavior, 44(5), 361–376. https://doi.org/10.1037/lhb0000413
Meade, A. W., & Fetzer, M. (2009). Test bias, differential prediction, and a revised approach for determining the suitability of a predictor in a selection context. Organizational Research Methods, 12(4), 738–761. https://doi.org/10.1177/1094428109331487
Meade, A. W., & Tonidandel, S. (2010). Not seeing clearly with Cleary: What test bias analyses do and do not tell us. Industrial and Organizational Psychology, 3(2), 192–205. https://doi.org/10.1111/j.1754-9434.2010.01223.x
Meehl, P. E. (1954). Clinical versus statistical prediction: A theoretical analysis and a review of the evidence. University of Minnesota Press. https://psycnet.apa.org/doi/https://doi.org/10.1037/11281-000
Minow, M., Zittrain, J., & Bowers. (2019). Technical flaws of pretrial risk assessments raise grave concerns. Cambridge, MA: Berkman Klein Center. https://cyber.harvard.edu/story/2019-07/technical-flaws-pretrial-risk-assessments-raise-grave-concerns. Accessed 12 January 2021
Nagel, I. (1983). The legal/extra-legal controversy: Judicial decisions in pretrial release. Law & Society Review, 3, 481. https://doi.org/10.2307/3053590
National Institute of Corrections. (2017). Evidence-based decision making: A guide for pretrial officials. Washington, D. C. https://info.nicic.gov/ebdm/sites/info.nicic.gov.ebdm/files/ebdm-users-guide-pretrial.pdf
Office of Judicial Administration. (2020). MyCase-Odyssey Public Access. https://public.courts.in.gov/mycase/#/vw/Search. Accessed 3 June 2020
Pepe, M. S. (2003). The statistical evaluation of medical tests for classification and prediction. OUP Oxford.
Petee, T. A. (1994). Recommended for release on recognizance: Factors affecting pretrial release recommendations. The Journal of Social Psychology, 134(3), 375–382.
Pretrial Justice Institute. (2014). Implementing the recommendations of the national symposium on pretrial justice: The 2013 progress report. Pretrial Justice Institute. Accessed 8 April 2018
Pretrial Justice Institute. (2019). Scan of pretrial practices (No. No. 2019–10–02KN). Baltimore, MD. https://university.pretrial.org/viewdocument/scan-of-pretrial-practices-pji-20
Pretrial Justice Institute. (2020). Updated position on pretrial risk assessment tools. Baltimore, Maryland. https://www.pretrial.org/wp-content/uploads/Risk-Statement-PJI-2020.pdf
Redcross, C., Henderson, B., Miratrix, L., & Valentine, E. (2019). Evaluation of pretrial justice system reforms that use the Public Safety Assessment: Effects in Mecklenburg County, North Carolina. MDRC Center for Criminal Justice Research. https://www.mdrc.org/sites/default/files/PSA_Mecklenburg_Brief2.pdf
Rice, M. E., & Harris, G. T. (2005). Comparing effect sizes in follow-up studies: ROC Area, Cohen’s d, and r. Law and Human Behavior, 29(5), 615–620. https://doi.org/10.1007/s10979-005-6832-7
Sackett, P. R., & Wilk, S. L. (1994). Within-group norming and other forms of score adjustment in preemployment testing. American Psychologist, 49(11), 929–954. https://doi.org/10.1037/0003-066X.49.11.929
Sacks, M., Sainato, V. A., & Ackerman, A. R. (2015). Sentenced to pretrial detention: A study of bail decisions and outcomes. American Journal of Criminal Justice, 40(3), 661–681. https://doi.org/10.1007/s12103-014-9268-0
Schaefer, B. P., & Hughes, T. (2019). Examining judicial pretrial release decisions: The influence of risk assessments and race. Criminology, Criminal Justice, Law & Society, 20, 47.
Schlesinger, T. (2005). Racial and ethnic disparity in pretrial criminal processing. Justice Quarterly, 22(2), 170–192.
Shadish, W. R., Cook, T. D., & Campbell, D. T. (2001). Experimental and quasi-experimental designs for generalized causal inference (2nd ed.). Wadsworth Publishing.
Skeem, J. L., & Lowenkamp, C. T. (2016). Risk, race, and recidivism: Predictive bias and disparate impact. Criminology, 54(4), 680–712. https://doi.org/10.1111/1745-9125.12123
Skeem, J. L., & Lowenkamp, C. T. (2020). Using algorithms to address trade-offs inherent in predicting recidivism. Behavioral Sciences & the Law, 38(3), 259–278. https://doi.org/10.1002/bsl.2465
Starr, S. B. (2014). Evidence-based sentencing and the scientific rationalization of discrimination. Stanford Law Review, 66, 803.
Stevenson, M. (2018). Assessing risk assessment in action. Minnesota Law Review, 103, 303–384.
Stevenson, M. T., & Mayson, S. G. (2017). Bail reform: New directions for pretrial detention and release (SSRN Scholarly Paper No. ID 2939273). Rochester, NY: Social Science Research Network. https://papers.ssrn.com/abstract=2939273. Accessed 9 April 2018
Stuart, E. A., & Green, K. M. (2008). Using full matching to estimate causal effects in nonexperimental studies: Examining the relationship between adolescent marijuana use and adult outcomes. Developmental Psychology, 44(2), 395–406. https://doi.org/10.1037/0012-1649.44.2.395
Thigpen, M. L., & Keiser, G. (2008). Applying evidence-based practices in pretrial services (p. 41). National Institute of Corrections, U.S. Department of Justice.
U.S. Census Bureau. (2017). Table 1. Educational attainment of the population 18 years and over, by age, sex, race, and Hispanic origin: 2016. https://www.census.gov/data/tables/2016/demo/education-attainment/cps-detailed-tables.html
van Eijk, G. (2016). Socioeconomic marginality in sentencing: The built-in bias in risk assessment tools and the reproduction of social inequality. Punishment & Society, 19(4), 463–481. https://doi.org/10.1177/1462474516666282
Viljoen, J. L., Cochrane, D. M., & Jonnson, M. R. (2018). Do risk assessment tools help manage and reduce risk of violence and reoffending? A systematic review. Law and Human Behavior, 42(3), 181–214. https://doi.org/10.1037/lhb0000280
Viljoen, J. L., Jonnson, M. R., Cochrane, D. M., Vargen, L. M., & Vincent, G. M. (2019). Impact of risk assessment instruments on rates of pretrial detention, postconviction placements, and release: A systematic review and meta-analysis. Law and Human Behavior, 43(5), 397–420. https://doi.org/10.1037/lhb0000344
Vincent, G. M., & Viljoen, J. L. (2020). Racist algorithms or systemic problems? Risk assessments and racial disparities. Criminal Justice and Behavior, 47(12), 1576–1584. https://doi.org/10.1177/0093854820954501
Wooldredge, J., Frank, J., Goulette, N., & Travis, L. (2015). Is the impact of cumulative disadvantage on sentencing greater for black defendants? Criminology & Public Policy, 14(2), 187–223. https://doi.org/10.1111/1745-9133.12124
Wu, J. (2016). Racial/ethnic discrimination and prosecution: A meta-analysis. Criminal Justice and Behavior, 43(4), 437–458. https://doi.org/10.1177/0093854815628026
Yang, C. S. (2017). Toward an optimal bail system. New York University Law Review, 92, 1399.
Zottola, S. A., Desmarais, S. L., Lowder, E. M., & Duhart Clarke, S. E. (2021). Evaluating Fairness of Algorithmic Risk Assessment Instruments: The Problem With Forcing Dichotomies. Criminal Justice and Behavior, 00938548211040544. https://doi.org/10.1177/00938548211040544
Funding
This work was supported by the National Institute of Justice, Office of Justice Programs, US Department of Justice [2018-R2-CX-0023]. The opinions, findings, and conclusions or recommendations expressed in this manuscript are those of the author(s) and do not necessarily reflect those of the Department of Justice.
Author information
Authors and Affiliations
Contributions
Evan M. Lowder, Eric Grommon, and Bradley Ray conceptualized and designed the study. Data cleaning and analysis were performed by Evan M. Lowder and Carmen Diaz. The first draft of the manuscript was written by Evan M. Lowder and Carmen Diaz and all authors reviewed and commented on previous versions of the manuscript. All authors read and approved the final manuscript.
Corresponding author
Ethics declarations
Ethics approval
This study was approved by Institutional Review Boards at Indiana University (protocol # 1811208825) and George Mason University (protocol # 124370).
Consent to participate
The authors received a waiver of informed consent from Institutional Review Boards for this investigation.
Data availability
The datasets analyzed during the current study are available in the National Archives of Criminal Justice Data (NACJD) repository, https://www.icpsr.umich.edu/web/NACJD/studies/37829.
Conflict of interest
E. Lowder, E. Grommon, and B. Ray received funding from the Indiana Office of Court Services to evaluate its Pretrial Pilot Program. The others have no other financial or non-financial interests to disclose.
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
About this article
Cite this article
Lowder, E.M., Diaz, C.L., Grommon, E. et al. Differential prediction and disparate impact of pretrial risk assessments in practice: a multi-site evaluation. J Exp Criminol 19, 561–594 (2023). https://doi.org/10.1007/s11292-021-09492-9
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11292-021-09492-9