Dude, Where’s My Treatment Effect? Errors in Administrative Data Linking and the Destruction of Statistical Power in Randomized Experiments

Abstract

Objective

The increasing availability of large administrative datasets has led to an exciting innovation in criminal justice research—using administrative data to measure experimental outcomes in lieu of costly primary data collection. We demonstrate that this type of randomized experiment can have an unfortunate consequence: the destruction of statistical power. Combining experimental data with administrative records to track outcomes of interest typically requires linking datasets without a common identifier. In order to minimize mistaken linkages, researchers often use stringent linking rules like “exact matching” to ensure that speculative matches do not lead to errors in an analytic dataset. We show that this, seemingly conservative, approach leads to underpowered experiments, leaves real treatment effects undetected, and can therefore have profound implications for entire experimental literatures.

Methods

We derive an analytic result for the consequences of linking errors on statistical power and show how the problem varies across combinations of relevant inputs, including linking error rate, outcome density and sample size.

Results

Given that few experiments are overly well-powered, even small amounts of linking error can have considerable impact on Type II error rates. In contrast to exact matching, machine learning-based probabilistic matching algorithms allow researchers to recover a considerable share of the statistical power lost under stringent data-linking rules.

Conclusion

Our results demonstrate that probabilistic linking substantially outperforms stringent linking criteria. Failure to implement linking procedures designed to reduce linking errors can have dire consequences for subsequent analyses and, more broadly, for the viability of this type of experimental research.

Appendices

Appendix 1: Computational Details

In this appendix we provide additional details for how statistical power can be computed under two possible states of the world: (1) in the absence of linking errors and (2) in the presence of linking errors. We use the derivations in this appendix to empirically demonstrate the effect of linking errors on statistical power in a hypothetical experiment in “Empirical Example” section of the paper.

For illustrative purposes, we will assume that a roster of individuals involved in a treatment program is being linked to arrest data to measure whether the program reduced the likelihood of arrest. Additionally, we will assume a record-linkage algorithm was run on the arrest data and that there existed a unique identifier allowing us to measure when predicted links between two records represented true and false matches and when predicted non-links represtented true and false non-matches.

We motivate the derivation by introducing a framework—a confusion matrix—that governs the incidence of linking errors in the arrest. Each row of the confusion matrix represents the incidence of an actual class (true non-match and true match) while each column represents the instances in a predicted class (predicted non-link and predicted link). The matrix thus allows us to understand the extent to which the algorithm is successful in classifying that two records belong to the same person.

In the following confusion matrix, $y^*$ represents the true state of the world and y represents the observed state of the world after linking. The cells provide counts of the number of true negatives, false negatives, false positives and true positives, respectively in linking the data.

	$y^*$ = 0	$y^*$ = 1
y = 0	TN	FN
y = 1	FP	TP

The diagonal entries of the matrix correspond to an alignment of the true and observed states of the world—observations for which $y^*$ = y = 0 are true negatives and observations for which $y^*$ = y = 1 are true positives. The off-diagonal entries provide us with the number of linking errors. In particular, the 2,1 element of the matrix provides the number of false positive links—this is the number of times in which an observation which is truly $y^*$ = 0 is mistakenly linked to y = 1. Similarly, the 1,2 element of the matrix provides the number of false negative links where an observation that is truly $y^*$ = 1 is mistakenly linked to a record for which y = 0.

The matrix allows us to compute four different rates capturing the success of a given linking strategy: the true positive and true negative rate and the false positive and false negative rate.

$$\begin{aligned} TPR&= \frac{TP}{TP+FN} \\ TNR&= \frac{TN}{TN+FP} \\ FPR&= 1-TNR \\ FNR&= 1-TPR \end{aligned}$$

The true positive rate (TPR) is defined as the number of linked positives divided by the number of true positives (TP+FN). Likewise the true negative rate (TNR) is the number of linked negatives divided by the number of true negatives (TN+FP). The corresponding false positive and false negative link rates are obtained by subtracting each of these quantities from 1. As we show in “Derivation of Estimated Treatment Effects, Standard Errors and Statistical Power” section of the paper, estimated treatment effects will be attenuated under linking errors and the attenuation will be proportional to 1-FPR-FNR. So long as FPR+FNR < 1, this will be strict attenuation towards zero but if FPR+FNR exceeds 1 then there can be a change in the sign of the bias.

To appreciate how this works, assume that the arrest dataset contained N = 10,000 records and that after running the matching algorithm, the following confusion matrix was generated:

	$y^*$ = 0	$y^*$ = 1
y = 0	TN=3000	FN=1000
y = 1	FP=2000	TP=4000

The error rates for the matching algorithm can be computed as:

• FNR = $\frac{1000}{4000 + 1000}$ = 0.20

• FPR = $\frac{2000}{3000 + 2000}$ = 0.40

Assume that to test the effectiveness of the treatment program, 1500 individuals where randomized, with one-half in the treatment group (p = 0.5), the control group mean was $\frac{1}{3}$, and that the treatment effect was $\tau = -\frac{1}{15}$.

In the true state of the world, there are 250 individuals arrested in the control group and 200 in the treatment group, reflecting the fact that $\tau = -\frac{1}{15}$. We next apply the error rates from the matching algorithm to the control and treatment groups respectively to generate the following confusion matrices:

Control group

	$y^*$ = 0	$y^*$ = 1
y = 0	TN=300	FN=50
y = 1	FP=200	TP=200

Treatment group

	$y^*$ = 0	$y^*$ = 1
y = 0	TN=330	FN=40
y = 1	FP=220	TP=160

Let $y^*_{T=0}$ be the true number of individuals arrested in the control group and $y_{T=0}$ be the observed number of individuals arrested in the control group. We see that $y^*_{T=0}=50+200=250$ and $y_{T=0}=200+200=400$.

Let $y^*_{T=1}$ be the true number of individuals arrested in the treatment group and $y_{T=1}$ be the observed number of individuals arrested in the treatment group. We see that $y^*_{T=1}=40+160=200$ and $y_{T=1}=220+160=380$.

The observed treatment effect can be computed as $\bar{y}_{T=1} - \bar{y}_{T=0} = \frac{380}{750} - \frac{400}{750} = -\frac{2}{75}$, which is equivalent to $\tau * (TPR - FPR) = -\frac{1}{15} * (0.8 - 0.4) = -\frac{2}{75}$

In order to compute statistical power to detect a given potential treatment effect, we need to compute a standard error which is computed according to:

$$\begin{aligned} var(\hat{\tau }) = \frac{1}{p(1-p)} \frac{\sigma ^2}{N} \end{aligned}$$

The square root of this quantity is the standard error around the estimated treatment effect. N and p are simply the sample size and the proportion treated but we will need to compute $\varsigma$ which is the mean square error from a regression of either $y^*$ or y on D, depending on which state of the world we are in. We show how to compute $\sigma ^2$ in absence and presence of linking errors in "Appendix 2".

We can then compute statistical power according to:

$$\begin{aligned} \beta = {{\Phi }} \left[ {-~{\Phi ^{-1}}} \left( \frac{\alpha }{2}\right) - \frac{\tau _h}{\sigma _{\tau _h}} \right] \end{aligned}$$

Carrying through the numerical example from our confusion table, power to detect a treatment effect of 20% in these data is 81% in the true state of the world and just 72% in the state of the world with linking errors. What would have been an adequately well-powered experiment is no longer well-powered in the presence of modest linking errors.

Appendix 2: Deriving Outcome Variance

In this section we show how to compute the residual sum of squares with a binary outcome and binary treatment in order to compute the $\sigma ^2$. Let $\bar{y}_C$ equal the control group mean and $\tau$ the treatment effect:

$$\begin{aligned} \sum _i (y_i - \hat{y}_i)^2&= \sum _i (y_i - \bar{y}_C - \tau T_i)^2 \end{aligned}$$

We can decompose the above equation into four mutually exclusive groups determined by whether an individual is in the treatment or control group, and whether their associated outcome is $y=0$ or $y=1$.

$$\begin{aligned}&\sum _{i \in \{i | T_i=0, y_i=0\}} ( - \bar{y}_C)^2 + \sum _{i \in \{i | T_i=0, y_i=1\}} (1 - \bar{y}_C)^2 + \sum _{i \in \{i | T_i=1, y_i=0\}} (- \bar{y}_C - \tau )^2 + \sum _{i \in \{i | T_i=1, y_i=1\}} (1 - \bar{y}_C - \tau )^2\\&\quad = N_{C,0}~\bar{y}_C^2 + N_{C,1} + N_{C,1}~\bar{y}_C^2 - N_{C,1}~2\bar{y}_C + N_{T,0}\bar{y}_T^2 + N_{T,1} + N_{T,1}~\bar{y}_T^2 - N_{T,1}~2\bar{y}_T\\&\quad = n_C~\bar{y}_C^2 + N_{C,1} - N_{C,1}~2\bar{y}_C + n_T\bar{y}_T^2 + N_{T,1} - N_{T,1}~2\bar{y}_T\\&\quad = N_{C,1}~\bar{y}_C + N_{C,1} - N_{C,1}~2\bar{y}_C + N_{T,1}\bar{y}_T + N_{T,1} - N_{T,1}~2\bar{y}_T\\&\quad = N_{C,1} (\bar{y}_C + 1 - 2\bar{y}_C) + N_{T,1}(\bar{y}_T + 1 - 2\bar{y}_T)\\&\quad = N_{C,1} (1 - \bar{y}_C) + N_{T,1}(1 - \bar{y}_T) \end{aligned}$$

Appendix 3: Maximizing RSS

We now show why Eq. 10 is maximized when the control group mean, $\bar{y}_C$, plus the treatment effect, $\tau$, equal 0.5. Let $N_{T,1}$ equal the number of individuals in the treatment group with $y=1$ and $N_{T,0}$ equal the number of individuals in the treatment group with $y=0$. Note that $N_{T,1} = (\bar{y}_C+ \tau ) N_T$ and $N_{T,0} = N_T (1 - (\bar{y}_C+\tau ))$.

$$\begin{aligned} \sum _i (y_i - \hat{y}_i)^2&= \sum _i (y_i - (\bar{y}_C + \tau T_i))^2 \end{aligned}$$

For a given control group mean we will take derivatives with respect to $\tau$, which means we will only consider individuals in the treatment group. We can decompose the previous equation into:

$$\begin{aligned} \sum _{i \in T} (y_i - \hat{y}_i)^2&= N_{T,0} (- \bar{y}_C - \tau )^2 + N_{T,1} (1 - \bar{y}_C - \tau )^2\\&= N_T (1 - (\bar{y}_C+\tau )) (- \bar{y}_C - \tau )^2 + N_T(\bar{y}_C+ \tau ) (1 - \bar{y}_C - \tau )^2\\&= N_T (1 - \bar{y}_C -\tau ) (\bar{y}_C + \tau )^2 + N_T(\bar{y}_C + \tau ) (1 - \bar{y}_C - \tau )^2\\&= N_T (1 - \bar{y}_C -\tau ) (\bar{y}_C+ \tau ) (\bar{y}_C + \tau + 1 - \alpha - \tau )\\&= N_T (1 - \bar{y}_C -\tau ) (\bar{y}_C + \tau ) \end{aligned}$$

Let $\kappa = N_T (1 - \bar{y}_C -\tau ) (\bar{y}_C + \tau )$, then taking derivatives with respect to $\tau$:

$$\begin{aligned} \frac{d\kappa }{d\tau }&= N_T (1 -2\bar{y}_C - 2\tau ) \end{aligned}$$

Setting the previous equation to zero and solving for $\tau$ leads to

$$\begin{aligned} \bar{y}_C + \tau = 0.5 \end{aligned}$$

Appendix 4: Proof for Power Attenuation

In this section we show that even when the standard error estimated under linking error is smaller than the standard error estimated under no error, statistical power will still be larger for the latter scenario. Let $\eta$ be True Positive Rate, $\omega$ the False Positive Rate, $tau^*$ the true treatment effect, $\sigma _{\tau ^*}$ the true standard error, $\hat{\tau }$ the observed treatment effect, and $\sigma _{\hat{\tau }}$ the observed standard error. Note that $0 \le \eta \le 1$ and $0 \le \omega \le 1$. In order to show that

$$\begin{aligned} \frac{\tau ^*}{\sigma _{\tau ^*}} > \frac{\hat{\tau }}{\sigma _{\hat{\tau }}} \end{aligned}$$

we use the result from Eq. 7 to substitute for the observed treatment effect to get

$$\begin{aligned} \frac{\tau ^*}{\sigma _{\tau ^*}} > \frac{(\eta -\omega )\tau ^* }{\sigma _{\hat{\tau }}} \end{aligned}$$

and then rearrange terms to get the following:

$$\begin{aligned} \sigma _{\hat{\tau }} > (\eta -\omega )\sigma _{\tau _h} \end{aligned}$$

It is straightforward to show that this is equivalent to

$$\begin{aligned} \sqrt{\widehat{RSS}} > (\eta -\omega ) \sqrt{RSS^*} \end{aligned}$$

Or that

$$\begin{aligned} \widehat{RSS} - (\eta -\omega )^2 RSS^* > 0 \end{aligned}$$

where $\widehat{RSS}$ is the residual sum of squares with linking error and $RSS^*$ is the residual sum of squares without linking error.

In the following, $N_{j,1}^{*}$ represents the number of observations for which the true value of y, $y^{*}=1$ and $N_{j,0}^{*}$ represents the number of observations for which the true value of y, $y^{*}=0$. This allows us to write the last inequality above as

$$\begin{aligned} \sum _{j \in \{T,C\}} (\eta ~N_{j,1}^{*} + \omega ~N_{j,0}^{*}) \left( 1 - \frac{\eta ~N_{j,1}^{*} + \omega ~N_{j,0}^{*}}{N_j^{*}} \right) - (\eta -\omega )^2 \sum _{j \in \{T,C\}} N_{j,1}^{*} \left( 1 - \frac{N_{j,1}^{*}}{N_j^{*}}\right)&> 0 \implies \\ \sum _{j \in \{T,C\}} (\eta ~N_{j,1}^{*} + \omega ~N_{j,0}^{*}) \left( \frac{ (1-\eta ) N_{j,1}^{*}+ (1-\omega ) N_{j,0}^{*} }{N_{j,1}^{*}+N_{j,0}^{*}} \right) - (\eta -\omega )^2 \left( \frac{N_{j,1^{*}}~N_{j,0}^{*}}{N_{j,1}^{*}+N_{j,0}^{*}}\right)&> 0 \implies \\ \sum _{j \in \{T,C\}} \frac{\eta (1-\eta ) N_{j,1}^{*2}+ \omega (1-\omega ) N_{j,0}^{*2} + N_{j,1}^{*} N_{j,0}^{*} \left[ \eta (1-\omega ) + (1-\eta ) \omega - (\eta - \omega )^2 \right] }{N_{j,1}^{*}+N_{j,0}^{*}}&> 0 \implies \\ \sum _{j \in \{T,C\}} \frac{\eta (1-\eta ) N_{j,1}^{*2}+ \omega (1-\omega ) N_{j,0}^{*2} + N_{j,1}^{*} N_{j,0}^{*} \left[ \eta + \omega - \eta ^2 - \omega ^2\right] }{N_{j,1}^{*}+N_{j,0}^{*}}&> 0 \end{aligned}$$

All terms in the numerator of the last inequality are greater than zero, satisfying the condition.

Appendix 5: Treatment Heterogeneity Correlated With Matching Error

In this section we demonstrate how treatment effect heterogeneity that’s correlated with linking error can impact coefficient estimates. Consider a dichotomous covariate G which takes on two values M and F. We assume that linking error rates within group are equal across treatment and control but that $TPR_{M} \ne TPR_{F}$ and $FPR_{M} \ne FPR_{F}$. Further, assume that $\tau _{M} \ne \tau _{F}$. We rewrite Eq. 4 as:

$$\begin{aligned} \hat{\tau } =&\sum _{j \in \{0,1\}} \sum _{g \in \{M,F\}} P(y_i = 1, y_i^{*}=j, G_i=g|T_i = 1) \\&-\, ~ P(y_i = 1, y_i^{*}=j, G_i=g|T_i = 0) \\ =&~ \sum _{j \in \{0,1\}} \sum _{g \in \{M,F\}} P(y_i = 1| y_i^{*}=j, G_i=g, T_i = 1) P(y_i^{*}=j| G_i=g, T_i = 1) P( G_i=g| T_i = 1) \\&-\, ~ P(y_i = 1| y_i^{*}=j, G_i=g, T_i = 0) P(y_i^{*}=j| G_i=g, T_i = 0) P( G_i=g| T_i = 0) \\ =&~ P(M) (TPR_{M} - FPR_{M}) \tau _M + P(F) (TPR_{F} - FPR_{F}) \tau _F \\ \end{aligned}$$

When both $\tau _M$ and $\tau _F$ are in the same direction, then linking error will only attenuate the pooled treatment effect in absolute value. However, if the signs of $\tau _M$ and $\tau _F$ are different, then the observed treatment effect may be greater than the true average treatment effect in absolute value.