Skip to main content
Log in

Difference-in-differences and matching on outcomes: a tale of two unobservables

  • Published:
Health Services and Outcomes Research Methodology Aims and scope Submit manuscript

Abstract

Difference-in-differences combined with matching on pre-treatment outcomes is a popular method for addressing non-parallel trends between a treatment and control group. However, previous simulations suggest that this approach does not always eliminate or reduce bias, and it is not clear when and why. Using Medicaid claims data from Oregon, we systematically vary the distribution of two key unobservables—fixed effects and the random error term—to examine how they affect bias of matching on pre-treatment outcomes levels or trends combined with difference-in-differences. We find that in most scenarios, bias increases with the standard deviation of the error term because a higher standard deviation makes short-term fluctuations in outcomes more likely, and matching cannot easily distinguish between these short-term fluctuations and more structural outcome trends. The fixed effect distribution may also create bias, but only when matching on pre-treatment outcome levels. A parallel-trend test on the matched sample does not reliably distinguish between successful and unsuccessful matching. Researchers using matching on pre-treatment outcomes to adjust for non-parallel trends should report estimates from both unadjusted and propensity-score matching adjusted difference-in-differences, compare results for matching on outcome levels and trends and examine outcome changes around intervention begin to assess remaining bias.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Fig. 1
Fig. 2
Fig. 3
Fig. 4

Similar content being viewed by others

Notes

  1. Other methods to address non-parallel trends include the synthetic control methods (Abadie et al. 2010) and lagged dependent variable regression. See O’Neill et al. (2016) for a comparison of these methods.

  2. Used as separate approaches, difference-in-differences versus matching on covariates have non-nested assumptions (Angrist and Pischke 2008; Imbens and Wooldridge 2009). Difference-in-differences requires parallel trends but allows for level effect imbalance between the treatment and control group. Matching requires all confounders to be balanced between the two groups but does not require parallel trends. When matching is applied to pre-treatment outcome levels, successful matching implies that all potential outcome trends are perfectly aligned. In this case, assumption (A1) is nested in (A2). Conversely, if level effects remain imbalanced after matching, assumption (A2) is violated but assumption (A1) may still hold, as long as trend effects are balanced.

  3. Simulations with alternative values for \(\alpha\) do not change our results. Intuitively, matching methods only use pre-intervention observations, which are assumed to be unaffected by the intervention effect.

  4. Assuming that the expected value of the time shock \(\lambda _t\) changes by a constant term implies a specific version of the Abadie et al. (2010) model with a constant slope for the expected outcomes. We make this restriction to simplify our simulations.

  5. For this model, matching is based on just one covariate, and therefore, propensity score matching does not have the advantage of condensing a large number of variables into one measure. We still use propensity score matching for this model so that all three matching models follow the same matching approach.

  6. We estimate the Monte Carlo standard error based on Koehler et al. (2009) using the following formula: \({\widehat{MCSE}} = \frac{1}{R}\sqrt{\sum _{r=1}^R({\hat{\beta }}_{2,r}-\overline{{\hat{\beta }}_{2}})^2}\), where \(R\) is the number of replications, \({\hat{\beta }}_{2,r}\) is the estimate of the treatment effect for the difference-in-differences model shown by Eq. (10) for replication \(r\), and \(\overline{{\hat{\beta }}_{2}} = \frac{1}{R}\sum _{r=1}^R{\hat{\beta }}_{2,r}\).

  7. The expected mean bias for the simple difference-in-differences is exactly 0.5 across all scenarios. To understand this result, note that \({\hat{\alpha }}\) in Eq. (10) can be written as follows (Angrist and Pischke 2008): \({\hat{\alpha }} = ({\bar{y}}_{Treat, t\ge t^*t} - {\bar{y}}_{Treat, t<t^*t}) - ({\bar{y}}_{Control, t\ge t^*t} - {\bar{y}}_{Control, t<t^*t})\), where \(\hat{y}\) denotes averages. The second difference is zero in expectations for all simulations because expected values of the level and trend effects for the comparison group are set to zero. Regarding the first difference, note that treatment group observations have an expected intercept and slope of 0.6 and 0.1, respectively. The slope implies that expected average outcome values are shifted upwards by \(0, 0.1, \ldots 0.4\) for the five pre-intervention periods and by \(0.5, 0.6, \ldots 0.9 = 2.5+(0, 0.1, \ldots 0.4)\) for the five post-intervention scenarios. It then follows that \((E({\hat{\alpha }}) = \alpha + 0.6 + 0.5 + (0 + 0.1 + 0.2 + 0.3 + 0.4)/5) - (0.6 + (0 + 0.1 + 0.2 + 0.3 + 0.4)/5) = \alpha + 0.5\).

  8. Specifically, the probability of being assigned to the treatment group is associated positively with pre-intervention trends for their third scenario, which suggests that level effects were similar between the two groups.

References

Download references

Acknowledgements

We would like to thank participants of the CHSE brownbag for helpful comments and suggestions.

Funding

Authors Lindner and McConnell received no outside funding for this study.

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Stephan Lindner.

Ethics declarations

Conflict of interest

Authors Lindner and McConnell declare that they have no conflict of interest.

Ethical approval

This article does not contain any studies with animals performed by any of the authors. All procedures performed in this study that involved human subsjects were in accordance with the ethical standards of the institutional and/or national research committee and with the 1964 Helsinki declaration and its later amendments or comparable ethical standards. Informed consent for this study was not required.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Lindner, S., McConnell, K.J. Difference-in-differences and matching on outcomes: a tale of two unobservables. Health Serv Outcomes Res Method 19, 127–144 (2019). https://doi.org/10.1007/s10742-018-0189-0

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10742-018-0189-0

Keywords

Navigation