Now You See It, Now You Don’t: A Simulation and Illustration of the Importance of Treating Incomplete Data in Estimating Race Effects in Sentencing

Abstract

Objectives

Evaluate the impact of missing data on observed racial disparities in the likelihood of an incarceration sentence, given that complete case analysis in the common analytic approach used in criminological research.

Methods

Using a simulation study with data based on cases sentenced in the Court of Common Pleas in Pennsylvania between 2010 and 2019, we assess the differences in the likelihood of incarceration between similarly situated White and Black defendants based on varying sample sizes and patterns of missing data.

Results

Complete case analysis (CCA) of incomplete data can fail to provide unbiased estimates of the race effect, even with less than 10% of cases missing. The degree of bias introduced depends on the amount, pattern, assumptions, and treatment of missing data. Multiple imputation provides an established, valid methodology for the unbiased estimation of race effects when data are missing at random, and this holds across sample sizes and number of imputations.

Conclusions

The existence and magnitude of race effects on the likelihood of an incarceration sentence can vary greatly based on the degree, pattern, assumptions, and treatment of missing data. Limitations include that missing data mechanisms cannot be truly known outside of a data simulation. Future sentencing research should prioritize the identification, treatment, and reporting of missing data prior to isolating race effects, in line with calls from the field for more open science practices. Sensitivity analyses should also be prioritized.

Appendix

Simulated Data Set Construction

Categorical and Ordinal Variables

Crime type ($CrimeType$): Sampled from

$X$	Persons	Property (ref.)	Drug	DUI	Other
$P\left( {CrimeType = x} \right)$	0.157	0.254	0.228	0.238	0.123

Case disposition ($CaseDisp$): sampled from

$X$	Yes	Plea (ref.)
$P\left( {CaseDisp = x} \right)$	0.978	0.022

Offense gravity score ($OGS$): Sampled from 1,2,…,14 with the probability distribution

$X$	1	2	3	4	5	6	7
$P\left( {OGS = x} \right)$	0.295	0.099	0.264	0.031	0.156	0.048	0.036
	8	9	10	11	12	13	14
	0.020	0.015	0.021	0.007	0.003	0.001	< 0.001

Prior record score ($PRS$): Sampled from

$X$	None (ref.)	1/2/3	4/5	REVOC/RFEL
$P\left( {PRS = x} \right)$	0.486	0.303	0.182	0.030

Recommended minimum ($RecMin$): sampled from

$X$	Yes	No (ref.)
$P\left( {RecMin = x} \right)$	0.670	0.30

Sex ($Sex$): Sampled from

$X$	Male (ref.)	Female
$P\left( {Sex = x} \right)$	0.774	0.226

Race ($Race$): sampled from

$X$	White (ref.)	Black	Latino	Other
$P\left( {Race = x} \right)$	0.717	0.268	0.009	0.006

County ($County$): sampled from indexes 1,2,…,67

$X$ P (County = x)	Adams (Ref.) 0.0100	Allegheny 0.1177	Armstrong 0.0047	Beaver 0.0141	Bedford 0.0044
	Berks 0.0318	Blair 0.0148	Bradford 0.0052	Bucks 0.0474	Butler 0.0148
	Cambria 0.0116	Cameron 0.0004	Carbon 0.0088	Centre 0.0099	Chester 0.0290
	Clarion 0.0032	Clearfield 0.0036	Clinton 0.0039	Columbia 0.0038	Crawford 0.0081
	Cumberland 0.0201	Dauphin 0.0302	Delaware 0.0644	Elk 0.0022	Erie 0.0203
	Fayette 0.0166	Forest 0.0005	Franklin 0.0173	Fulton 0.0016	Greene 0.0025
	Huntington 0.0038	Indiana 0.0070	Jefferson 0.0037	Juniata 0.0019	Lackawanna 0.0163
	Lancaster 0.0266	Lawrence 0.0059	Lebanon 0.0116	Lehigh 0.0297	Luzerne 0.0239
	Lycoming 0.0152	McKean 0.0041	Mercer 0.0099	Mifflin 0.0047	Monroe 0.0124
	Montgomery 0.0636	Montour 0.0012	Northampton 0.0251	Northumberland 0.0078	Perry 0.0040
	Philadelphia 0.0567	Pike 0.0043	Potter 0.0016	Schuylkill 0.0123	Snyder 0.0035
	Somerset 0.0047	Sullivan 0.0004	Susquehanna 0.0024	Tioga 0.0025	Union 0.0028
	Venango 0.0053	Warren 0.0031	Washington 0.0098	Wayne 0.0031	Westmoreland 0.0317
	Wyoming 0.0029	York 0.0517

Year ($Year$): sampled from

X	2010	(Ref.)	2011	2012	2013	2014	2015	2016	2017	2018	2019
P (Y ear = x)	0.103		0.097	0.100	0.105	0.104	0.097	0.101	0.097	0.096	0.100

Quantitative Variable: Let $A_{i} = Age_{i} - min\left( {Age} \right)$ for ${\text{i }} = { }1, \ldots ,{\text{N}}$.

Draw $\overline{{A_{i} }} \sim Gamma\left( {\hat{a},\hat{b}} \right)$ where $\hat{a} = \frac{N - 1}{N}\frac{{\overline{{A^{2} }} }}{{\widehat{Var}\left( A \right)}}$ and $\hat{b} = \frac{N}{N - 1}\frac{{\widehat{Var}\left( A \right)}}{{\overline{A}}}$ is the scale parameter with $\overline{A} = \frac{1}{N}\mathop \sum \limits_{i = 1}^{N} A_{i}$ and $\widehat{Var}\left( A \right) = \frac{1}{N - 1}\mathop \sum \limits_{i = 1}^{N} \left( {A_{i} - \overline{A}} \right)^{2}$. The simulated age for individual $i$ is then $\overline{{Age_{i} }} = \overline{{A_{i} }} + min\left( {Age} \right)$. In the PCS data set, $\overline{A} = 22.57$ and $\widehat{Var}\left( A \right) = 132.434$.

Incarceration Outcome Variable: We used the estimated coefficients from the fitted logistic regression model (Eq. 1) using the PCS data as the “true” parameter values for the simulated population. We then sample sentencing decisions using the same logistic regression model evaluated on the simulated independent and control variables.

Missingness Model Parameters

The multinomial model uses two variables to determine the probability of an observation being missing: incarceration and whether the defendant’s race is Black. Two interaction variables were created for non-incarceration and race being Black and being incarcerated and race being Black. The intercept of the model is set for each of the eight missing data patterns to reflect the observed probability of that pattern in the Pennsylvania data. The parameters of the model vary by pattern and to induce over- or under-estimation of the race effect in the complete case analysis. See Table 1 for a list of the missing data patterns in the PA data.

MNAR Missingness

The missingness model assigns slightly more missingness than we saw in the administrative data while still having a relatively low number of incomplete cases.

$$P\left( {PatternrforSubjecti} \right) = \frac{{exp(\alpha_{r} + \beta_{1r} INCAR_{i} + \beta_{2r} Black_{i} + \beta_{3r} Z_{1i} + \beta_{4r} Z_{2i} )}}{{1 + \mathop \sum \nolimits_{r = 1}^{8} exp\left( {\alpha_{r} + \beta_{1r} INCAR_{i} + \beta_{2r} Black_{i} + \beta_{3r} Z_{1i} + \beta_{4r} Z_{2i} } \right)}}$$

(5)

To over-estimate the race effect, missingness in race is conditional on race being Black and incarceration so we set the parameters of the model to be $exp\left( \beta \right) = \left( {10, 0.1, 0.001, 100} \right){\prime} \in R^{4}$. The first element in $\beta$ emphasizes missingness based on incarceration, the second de-emphasizes missingness based on being Black while the third element corresponding to $Z_{1}$ greatly de-emphasizes missingness for defendants who aren’t incarcerated and are Black and the fourth corresponding to $Z_{2}$ greatly emphasizes missingness for individuals who are incarcerated and are Black.

$$P\left( {PatternrforSubjecti} \right) = \frac{{exp\left( {\alpha_{r} + \beta_{1r} INCAR_{i} + \beta_{2r} Black_{i} + \beta_{3r} Z_{1i} } \right)}}{{1 + \mathop \sum \nolimits_{r = 1}^{8} exp(\alpha_{r} + \beta_{1r} INCAR_{i} + \beta_{2r} Black_{i} + \beta_{3r} Z_{1i} )}}$$

(6)

To under-estimate the effect, missingness in race is again conditional on race being Black and incarceration, so we set the parameters of the model to be $exp\left( \beta \right) = \left( {5,5, 10} \right){\prime} \in R^{3}$. The first element in $\beta$ emphasizes missingness based on incarceration, the second element emphasizes missingness based on being Black while the third element greatly emphasizes missingness for defendants who aren’t incarcerated and are Black corresponding to $Z_{1}$.

MAR Missingness

We used similar parameter settings for the MAR simulations as we did for the MNAR, but now missingness based on race is only for patterns 1,3,5, and 7 which are no missing variables, missing in age, missing in recommended minimum, and missing in both age and recommended minimum. This maintains the MAR assumption since the missingness is not contingent on race for the patterns where race is missing. However, since we are performing the analyses with complete case analysis, we are still systematically excluding data from our analysis in such a way that will dramatically bias the results based on the values of race and incarceration.

Slightly different parameter values are used for patterns 3 and 5 compared to patterns 1 and 7. See Table 1 for a list of the missing data patterns.

$$P\left( {PatternrforSubjecti} \right) = \frac{{exp(\alpha_{r} + \beta_{1r} INCAR_{i} + \beta_{2r} Black_{i} + \beta_{3r} Z_{1i} )}}{{1 + \mathop \sum \nolimits_{r = 1}^{8} exp\left( {\alpha_{r} + \beta_{1r} INCAR_{i} + \beta_{2r} Black_{i} + \beta_{3r} Z_{1i} } \right)}}$$

(7)

To get an over-estimated race effect estimate with CCA, we de-emphasize missingness based on $Z_{1}$ for patterns where race is observed.

$$P\left( {PatternrforSubjecti} \right) = \frac{{exp(\alpha_{r} + \beta_{1r} Z_{1i} + \beta_{2r} Z_{2i} )}}{{1 + \mathop \sum \nolimits_{r = 1}^{8} exp(\alpha_{r} + \beta_{1r} Z_{1i} + \beta_{2r} Z_{2i} )}}$$

(8)

To under-estimate the effect with CCA, we emphasize $Z_{1}$ and de-emphasize $Z_{2}$ for patterns where race is observed (Tables 3 , 4 , 5 and 6 ).

Table 3 The parameters for the missingness model for CCA over-estimates with MNAR missing values

Table 4 The parameters for the missingness model for CCA over-estimates with MNAR missing values

Table 5 The parameters for the missingness model for CCA over-estimates with MAR missing values

Table 6 The parameters for the missingness model for CCA under-estimates with MAR missing values