Skip to main content
SpringerLink
Log in

Difference-in-Differences for Policy Evaluation

  • Living reference work entry
  • First Online:
Handbook of Labor, Human Resources and Population Economics

Abstract

Difference-in-differences is one of the most used identification strategies in empirical work in economics. This chapter reviews a number of important, recent developments related to difference-in-differences. First, this chapter reviews recent work pointing out limitations of two-way fixed-effects regressions (these are panel data regressions that have been the dominant approach to implementing difference-in-differences identification strategies) that arise in empirically relevant settings where there are more than two time periods, variation in treatment timing across units, and treatment effect heterogeneity. Second, this chapter reviews recently proposed alternative approaches that are able to circumvent these issues without being substantially more complicated to implement. Third, this chapter covers a number of extensions to these results, paying particular attention to (i) parallel trends assumptions that hold only after conditioning on observed covariates and (ii) strategies to partially identify causal effect parameters in difference-in-differences applications in cases where the parallel trends assumption may be violated.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Institutional subscriptions

References

  • Abadie A (2005) Semiparametric difference-in-differences estimators. Rev Econ Stud 72(1):1–19

    Article  Google Scholar 

  • Abadie A, Diamond A, Hainmueller J (2010) Synthetic control methods for comparative case studies: estimating the effect of California’s tobacco control program. J Am Stat Assoc 105(490):493–505

    Article  Google Scholar 

  • Angrist JD, Pischke J-S (2008) Mostly Harmless econometrics: an empiricist’s companion. Princeton University Press

    Book  Google Scholar 

  • Arel-Bundock V (2022) modelsummary: Summary tables and plots for statistical models and data: beautiful, customizable, and publication-ready. R package version 0.9.6. https://CRAN.R-project.org/package=modelsummary

  • Arkhangelsky D, Athey S, Hirshberg DA, Imbens GW, Wager S (2021) Synthetic difference-in-differences. Am Econ Rev 111(12):4088–4118

    Article  Google Scholar 

  • Athey S, Imbens G (2006) Identification and inference in nonlinear difference-in-differences models. Econometrica 74(2):431–497

    Article  Google Scholar 

  • Athey S, Imbens GW (2022) Design-based analysis in difference-in-differences settings with staggered adoption. J Econ 226(1):62–79

    Article  Google Scholar 

  • Athey S, Bayati M, Doudchenko N, Imbens G, Khosravi K (2021) Matrix completion methods for causal panel data models. J Am Stat Assoc:1–15

    Google Scholar 

  • Bai J, Ng S (2021) Matrix completion, counterfactuals, and factor analysis of missing data. J Am Stat Assoc 116:1746

    Article  Google Scholar 

  • Baker AC, Larcker DF, Wang CCY (2022) How much should we trust staggered difference-in-differences estimates? J Financ Econ 144(2):370–395. Elsevier

    Google Scholar 

  • Bergé L (2018) Efficient estimation of maximum likelihood models with multiple fixed-effects: the R package FENmlm. CREA Discussion Papers 13

    Google Scholar 

  • Blundell R, Costa Dias M (2009) Alternative approaches to evaluation in empirical microeconomics. J Hum Resour 44(3):565–640

    Google Scholar 

  • Bonhomme S, Sauder U (2011) Recovering distributions in difference-in-differences models: a comparison of selective and comprehensive schooling. Rev Econ Stat 93(2):479–494

    Article  Google Scholar 

  • Borusyak K, Jaravel X, Spiess J (2021) Revisiting event study designs: robust and efficient estimation. Working Paper

    Google Scholar 

  • Botosaru I, Gutierrez FH (2018) Difference-in-differences when the treatment status is observed in only one period. J Appl Econ 33(1):73–90

    Article  Google Scholar 

  • Botosaru I, Muris C (2017) Binarization for panel models with fixed effects. Working Paper

    Google Scholar 

  • Busso M, DiNardo J, McCrary J (2014) New evidence on the finite sample properties of propensity score reweighting and matching estimators. Rev Econ Stat 96(5):885–897

    Article  Google Scholar 

  • Butts K (2021a) did2s: two-stage difference-in-differences Following Gardner (2021). https://github.com/kylebutts/did2s/

  • Butts K (2021b) didimputation: difference-in-differences estimator from Borusyak, Jaravel, and Spiess (2021). https://github.com/kylebutts/didimputation

  • Caetano C, Callaway B, Payne R, Rodrigues H (2022) Difference in differences with time-varying covariates. Working Paper

    Google Scholar 

  • Callaway B (2021) Bounds on distributional treatment effect parameters using panel data with an application on job displacement. J Econ 222.2:861–881

    Article  Google Scholar 

  • Callaway B (2022). pte: panel treatment effects. R package version 0.0.0.9000. https://github.com/bcallaway11/pte

  • Callaway B, Karami S (2022) Treatment effects in interactive fixed effects models with a small number of time periods. J Econ (forthcoming). https://doi.org/10.1016/j.jeconom.2022.02.001

  • Callaway B, Li T (2019) Quantile treatment effects in difference in differences models with panel data. Quant Econ 10(4):1579–1618

    Article  Google Scholar 

  • Callaway B, Li T (2021) Policy evaluation during a pandemic. Working Paper

    Google Scholar 

  • Callaway B, Sant’Anna PHC (2021) Difference-in-differences with multiple time periods. J Econ 225(2):200–230

    Article  Google Scholar 

  • Callaway B, Sant’Anna PHC (2022) did: difference in differences. R package version 2.1.1. https://bcallaway11.github.io/did/

  • Callaway B, Li T, Oka T (2018) Quantile treatment effects in difference in differences models under dependence restrictions and with only two time periods. J Econ 206(2):395–413

    Article  Google Scholar 

  • Callaway B, Goodman-Bacon A, Sant’Anna PHC (2021) Difference-in-differences with a continuous treatment. Working Paper

    Google Scholar 

  • Card D (1990) The impact of the Mariel boatlift on the Miami labor market. Ind Labor Relat Rev 43(2):245–257

    Article  Google Scholar 

  • Card D, Krueger A (1994) Minimum wages and employment: a case study of the fast-food industry in New Jersey and Pennsylvania. Am Econ Rev 84(4):772

    Google Scholar 

  • Cengiz D, Dube A, Lindner A, Zipperer B (2019) The effect of minimum wages on low-wage jobs. Q J Econ 134(3):1405–1454

    Article  Google Scholar 

  • Chang N-C (2020) Double/debiased machine learning for difference-in-differences models. Econ J 23(2):177–191

    Google Scholar 

  • Chernozhukov V, Fernandez-Val I, Hahn J, Newey W (2013) Average and quantile effects in nonseparable panel models. Econometrica 81(2):535–580

    Article  Google Scholar 

  • Chernozhukov V, Chetverikov D, Demirer M, Duflo E, Hansen C, Newey W, Robins J (2018) Double/debiased machine learning for treatment and structural parameters. Econ J 21(1):C1–C68

    Article  Google Scholar 

  • Conley TG, Taber CR (2011) Inference with “difference in differences” with a small number of policy changes. Rev Econ Stat 93(1):113–125

    Article  Google Scholar 

  • Cunningham S (2021) Causal inference: the mixtape. Yale University Press

    Book  Google Scholar 

  • Currie J, Kleven H, Zwiers E (2020) Technology and big data are changing economics: mining text to track methods. AEA Pap Proc 110:42–48

    Article  Google Scholar 

  • de Chaisemartin C, D’Haultfœuille X (2018) Fuzzy differences-in-differences. Rev Econ Stud 85(2):999–1028

    Article  Google Scholar 

  • de Chaisemartin C, D’Haultfœuille X (2020) Two-way fixed effects estimators with heterogeneous treatment effects. Am Econ Rev 110(9):2964–2996

    Article  Google Scholar 

  • de Chaisemartin, Clément, D’Haultfœuille X (2021a) Two-way fixed effects and differences-in-differences with heterogeneous treatment effects: a survey. Available at SSRN

    Google Scholar 

  • de Chaisemartin C, D’Haultfœuille X (2021b) Two-way fixed effects regressions with several treatments. Working Paper

    Google Scholar 

  • Dube A, William Lester T, Reich M (2016) Minimum wage shocks, employment flows, and labor market frictions. J Labor Econ 34(3):663–704

    Article  Google Scholar 

  • Ferman B (2019) Matching estimators with few treated and many control observations. arXiv preprint arXiv:1909.05093

    Google Scholar 

  • Ferman B, Pinto C (2021) Synthetic controls with imperfect pretreatment fit. Quant Econ 12(4):1197–1221

    Article  Google Scholar 

  • Flack E, Jee E (2022) bacondecomp: Goodman-Bacon Decomposition. R package version 0.1.3. https://github.com/evanjack/bacondecomp/issues

  • Freyaldenhoven S, Hansen C, Shapiro JM (2019) Pre-event trends in the panel event-study design. Am Econ Rev 109(9):3307–3338

    Article  Google Scholar 

  • Gardner J (2021) Two-stage difference in differences. Working Paper

    Google Scholar 

  • Gobillon L, Magnac T (2016) Regional policy evaluation: interactive fixed effects and synthetic controls. Rev Econ Stat 98(3):535–551

    Article  Google Scholar 

  • Goodman-Bacon A (2021) Difference-in-differences with variation in treatment timing. J Econ 225(2):254–277

    Article  Google Scholar 

  • Hagemann A (2019) Placebo inference on treatment effects when the number of clusters is small. J Econ 213(1):190–209

    Article  Google Scholar 

  • Heckman J, Hotz VJ (1989) Choosing among alternative nonexperimental methods for estimating the impact of social programs: the case of manpower training. J Am Stat Assoc 84(408):862–874

    Article  Google Scholar 

  • Heckman J, Ichimura H, Todd P (1997) Matching as an econometric evaluation estimator: evidence from evaluating a job training programme. Rev Econ Stud 64(4):605–654

    Article  Google Scholar 

  • Hong S-H (2013) Measuring the effect of napster on recorded music sales: difference-in-differences estimates under compositional changes. J Appl Econ 28(2):297–324

    Article  Google Scholar 

  • Hsiao C, Steve Ching H, Wan SK (2012) A panel data approach for program evaluation: measuring the benefits of political and economic integration of Hong Kong with mainland China. J Appl Econ 27(5):705–740

    Article  Google Scholar 

  • Imbens G, Kallus N, Mao X (2021) Controlling for unmeasured confounding in panel data using minimal bridge functions: from two-way fixed effects to factor models. Working Paper

    Google Scholar 

  • Ishimaru S (2022) What do we get from a two-way fixed effects estimator? Implications from a general numerical equivalence”. Working Paper

    Google Scholar 

  • Kang JDY, Schafer JL (2007) Demystifying double robustness: a comparison of alternative strategies for estimating a population mean from incomplete data. Stat Sci 22(4):523–539

    Google Scholar 

  • Lee M-J, Lee S (2021) Difference in differences and ratio in ratios for limited dependent variables. Working Paper

    Google Scholar 

  • Liu L, Wang Y, Xu Y (2021) A practical guide to counterfactual estimators for causal inference with time-series cross-sectional data”. Working Paper

    Google Scholar 

  • Manski CF, Pepper JV (2018) How do right-to-carry laws affect crime rates? Coping with ambiguity using bounded-variation assumptions. Rev Econ Stat 100(2):232–244

    Article  Google Scholar 

  • Marcus M, Sant’Anna PHC (2021) The role of parallel trends in event study settings: an application to environmental economics. J Assoc Environ Resour Econ 8(2):235–275

    Google Scholar 

  • Meer J, West J (2016) Effects of the minimum wage on employment dynamics. J Hum Resour 51(2):500–522

    Article  Google Scholar 

  • Mora R, Reggio I (2019) Alternative diff-in-diffs estimators with several pretreatment periods. Econ Rev 38(5):465–486

    Article  Google Scholar 

  • Rambachan A, Roth J (2020) Design-based uncertainty for quasi-experiments”. Working Paper

    Google Scholar 

  • Rambachan A, Roth J (2021a) An honest approach to parallel trends”. Working Paper

    Google Scholar 

  • Rambachan A, Roth J (2021b) HonestDiD: robust inference in difference-in-differences and event study designs. R package version 0.2.0. https://github.com/asheshrambachan/HonestDiD

  • Robins JM, Rotnitzky A, Zhao LP (1994) Estimation of regression coefficients when some regressors are not always observed. J Am Stat Assoc 89(427):846–866

    Article  Google Scholar 

  • Roth J (2020) Pre-test with caution: event-study estimates after testing for parallel trends. Working Paper

    Google Scholar 

  • Roth J, Sant’Anna PHC (2021) When is parallel trends sensitive to functional form? Working Paper

    Google Scholar 

  • Roth J, Sant’Anna PHC, Bilinski A, Poe J (2022) What’s trending in difference-in-differences? A synthesis of the recent econometrics literature. Working Paper

    Google Scholar 

  • Sant’Anna PHC, Zhao J (2020) Doubly robust difference-in-differences estimators. J Econ 219(1):101–122

    Article  Google Scholar 

  • Scharfstein DO, Rotnitzky A, Robins JM (1999) Adjusting for nonignorable drop-out using semiparametric nonresponse models. J Am Stat Assoc 94(448):1096–1120

    Article  Google Scholar 

  • Słoczyński T (2020) Interpreting OLS estimands when treatment effects are heterogeneous: smaller groups get larger weights. Rev Econ Stat:1–27

    Google Scholar 

  • Słoczyński T, Wooldridge JM (2018) A general double robustness result for estimating average treatment effects. Economet Theor 34(1):112–133

    Article  Google Scholar 

  • Sun L, Abraham S (2021) Estimating dynamic treatment effects in event studies with heterogeneous treatment effects. J Econ 225(2):175–199

    Article  Google Scholar 

  • Wooldridge JM (2005) Fixed-effects and related estimators for correlated random-coefficient and treatment-effect panel data models. Rev Econ Stat 87(2):385–390

    Article  Google Scholar 

  • Wooldridge J (2021) Two-way fixed effects, the two-way Mundlak regression, and difference-in-differences estimators. Working Paper

    Google Scholar 

  • Xu Y (2017) Generalized synthetic control method: causal inference with interactive fixed effects models. Polit Anal 25(1):57–76

    Article  Google Scholar 

Download references

Acknowledgments

Responsible Section Editor: Enrico Rettore.

This chapter has benefited from the valuable comments of the editors, anonymous referees, John Gardner, Jon Roth, and Pedro Sant’Anna. There is no conflict of interest.

Author information

Authors and Affiliations

  1. Department of Economics, University of Georgia, Athens, GA, USA

    Brantly Callaway

Authors
  1. Brantly Callaway
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Brantly Callaway .

Editor information

Editors and Affiliations

  1. UNU-MERIT & Princeton University, Maastricht, The Netherlands

    Klaus F. Zimmermann

Section Editor information

  1. Department of Economics and management, University of Padova, Padova, Italy

    Enrico Rettore

Rights and permissions

Reprints and permissions

Copyright information

© 2022 Springer Nature Switzerland AG

About this entry

Check for updates. Verify currency and authenticity via CrossMark

Cite this entry

Callaway, B. (2022). Difference-in-Differences for Policy Evaluation. In: Zimmermann, K.F. (eds) Handbook of Labor, Human Resources and Population Economics. Springer, Cham. https://doi.org/10.1007/978-3-319-57365-6_352-1

Download citation

  • DOI: https://doi.org/10.1007/978-3-319-57365-6_352-1

  • Received:

  • Accepted:

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-319-57365-6

  • Online ISBN: 978-3-319-57365-6

  • eBook Packages: Springer Reference Economics and FinanceReference Module Humanities and Social SciencesReference Module Business, Economics and Social Sciences

Publish with us

Policies and ethics

Search

Navigation