Skip to main content
SpringerLink
Log in

An introduction to the difference-in-differences design in education policy research

  • Published:
Asia Pacific Education Review Aims and scope Submit manuscript

Abstract

Education researchers often encounter scenarios where an abrupt policy change occurs within or across jurisdictions or populations that affect key student outcomes. Difference-in-differences is a research design analysts can use to estimate causal effects of these “natural experiments.” This article introduces education researchers to the difference-in-differences approach. We provide an overview of the standard two-by-two design, which examines changes in outcomes over time between a treated and an unaffected group. Then, we discuss the key assumptions and threats to validity that researchers must address to have maximum confidence in their results. We also discuss commonly used alternative specifications that provide researchers with a more nuanced approach to estimating policy effects. Throughout the paper, we illustrate the difference-in-differences approach using the example of the implementation of an after-school tutoring program to increase economically disadvantaged students’ academic performance. In supplementary material, we provide Stata code to help assist researchers adapt the approach to their own scholarship.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3

Similar content being viewed by others

References

  • Abadie, A., Athey, S., Imbens, G. W., & Wooldridge, J. M. (2023). When should you adjust standard errors for clustering? The Quarterly Journal of Economics, 138(1), 1–35. https://doi.org/10.1093/qje/qjac038

    Article  Google Scholar 

  • Athey, S., & Imbens, G. W. (2022). Design-based analysis in difference-in-differences settings with staggered adoption. Journal of Econometrics, 226(1), 62–79.

    Article  Google Scholar 

  • Baker, A. C., Larcker, D. F., & Wang, C. C. Y. (2022). How much should we trust staggered difference-in-differences estimates? Journal of Financial Economics, 144(2), 370–395. https://doi.org/10.1016/j.jfineco.2022.01.004

    Article  Google Scholar 

  • Bertrand, M., Duflo, E., & Mullainathan, S. (2004). How much should we trust differences-in-differences estimates? The Quarterly Journal of Economics, 119(1), 249–275.

    Article  Google Scholar 

  • Borusyak, K., Jaravel, X., & Spiess, J. (2023). Revisiting event study designs: Robust and efficient estimation. arXiv. https://doi.org/10.48550/arXiv.2108.12419

    Article  Google Scholar 

  • Bray, M. (2010). Researching shadow education: Methodological challenges and directions. Asia Pacific Education Review, 11(1), 3–13. https://doi.org/10.1007/s12564-009-9056-6

    Article  Google Scholar 

  • Callaway, B., & Sant’Anna, P. H. (2021). Difference-in-differences with multiple time periods. Journal of Econometrics, 225(2), 200–230.

    Article  Google Scholar 

  • Cengiz, D., Dube, A., Lindner, A., & Zipperer, B. (2019). The effect of minimum wages on low-wage jobs. The Quarterly Journal of Economics, 134(3), 1405–1454. https://doi.org/10.1093/qje/qjz014

    Article  Google Scholar 

  • de Chaisemartin, C., D’Haultfoeuille, X., & Deeb, A. (2019). TWOWAYFEWEIGHTS: Stata module to estimate the weights and measure of robustness to treatment effect heterogeneity attached to two-way fixed effects regressions [Computer software]. Statistical Software Components S458611, Boston College Department of Economics.

  • Chetty, R., Friedman, J. N., Hilger, N., Saez, E., Schanzenbach, D. W., & Yagan, D. (2011). How does your kindergarten classroom affect your earnings? Evidence from project STAR. The Quarterly Journal of Economics, 126(4), 1593–1660.

    Article  Google Scholar 

  • Corral, D., & Ward, J. D. (2023). Calibrating costs: Do tuition reset policies affect diverse student enrollment at private baccalaureate colleges? The Review of Higher Education. https://doi.org/10.1353/rhe.0.a909286

    Article  Google Scholar 

  • Dang, H. A. (2007). The determinants and impact of private tutoring classes in Vietnam. Economics of Education Review, 26(6), 683–698.

    Article  Google Scholar 

  • de Chaisemartin, C., & D’Haultfœuille, X. (2020). Two-way fixed effects estimators with heterogeneous treatment effects. American Economic Review, 110(9), 2964–2996. https://doi.org/10.1257/aer.20181169

    Article  Google Scholar 

  • Dettmann, E., Giebler, A., & Weyh, A. (2020). Flexpaneldid: A Stata toolbox for causal analysis with varying treatment time and duration (SSRN Scholarly Paper No. 3692458). https://doi.org/10.2139/ssrn.3692458.

  • Furquim, F., Corral, D., & Hillman, N. (2020). A primer for interpreting and designing difference-in-differences studies in higher education research. In L. W. Perna (Ed.), Higher Education: Handbook of Theory and Research (Vol. 35, pp. 667–723). Springer. https://doi.org/10.1007/978-3-030-11743-6_5-1

    Chapter  Google Scholar 

  • Goodman-Bacon, A. (2021). Difference-in-differences with variation in treatment timing. Journal of Econometrics, 225(2), 254–277. https://doi.org/10.1016/j.jeconom.2021.03.014

    Article  Google Scholar 

  • Huntington-Klein, N. (2023). Controls in difference-in-differences don’t just work [Substack newsletter]. Data, On Average. https://nickchk.substack.com/p/controls-in-difference-in-differences.

  • Huntington-Klein, N. (2022). The effect: An introduction to research design and causality. CRC Press.

    Google Scholar 

  • Jackson, C. K., Johnson, R. C., & Persico, C. (2016). The effects of school spending on educational and economic outcomes: Evidence from school finance reforms. Quarterly Journal of Economics, 131(1), 157–218.

    Article  Google Scholar 

  • Jayachandran, S. (2014). Incentives to teach badly: After-school tutoring in developing countries. Journal of Development Economics, 108, 190–205.

    Article  Google Scholar 

  • Jia, X., Zhang, X., Jing, J., Zhou, M., Li, H., Friesen, D., & Ma, Y. (2022). The impact of nonboarding on the development of disadvantaged boarding students in western rural China. Asia Pacific Education Review, 23(1), 131–150. https://doi.org/10.1007/s12564-022-09742-z

    Article  Google Scholar 

  • Jung, H. (2018). A late bird or a good bird? The effect of 9 o’clock attendance policy on student’s achievement. Asia Pacific Education Review, 19(4), 511–529.

    Article  Google Scholar 

  • Kim, E., Goodman, J., & West, M. R. (2021a). Kumon in: The recent, rapid rise of private tutoring centers. Brown University Annenberg EdWorkingPaper (21–367), 1–35. https://www.edworkingpapers.com/sites/default/files/ai21-367.pdf.

  • Kim, T., Lim, S., Yang, M., & Park, S. J. (2021b). Making sense of schooling during COVID-19: Crisis as opportunity in Korean schools. Comparative Education Review, 65(4), 617–639.

    Article  Google Scholar 

  • Kim, Y., Kester, K., & Han, S. (2022). Interesting, but are we convinced? The importance of methodological rigor at Asia Pacific Education Review. Asia Pacific Education Review, 23(2), 209–210.

    Article  Google Scholar 

  • Kim, Y., & Steiner, P. M. (2021). Gain scores revisited: A graphical models perspective. Sociological Methods & Research, 50(3), 1353–1375. https://doi.org/10.1177/0049124119826155

    Article  Google Scholar 

  • Lechner, M. (2010). The estimation of causal effects by difference-in-difference methods. Foundations and Trends® in Econometrics, 4(3), 165–224. https://doi.org/10.1561/0800000014

    Article  Google Scholar 

  • Li, L., Yang, Y., & Zhou, L. (2022). Can after-school tutoring improve the academic performance of students from low-income families? Trial evidence from economically backward areas in China. Asia Pacific Education Review. https://doi.org/10.1007/s12564-022-09803-3

    Article  Google Scholar 

  • Lu, F. (2014). Testing peer effects among college students: Evidence from an unusual admission policy change in China. Asia Pacific Education Review, 15, 257–270.

    Article  Google Scholar 

  • Miller, D. L. (2023). An introductory guide to event study models. Journal of Economic Perspectives, 37(2), 203–230. https://doi.org/10.1257/jep.37.2.203

    Article  Google Scholar 

  • Murnane, R. J., & Willett, J. B. (2011). Methods matter: Improving causal inference in educational and social science research. Oxford University Press.

    Google Scholar 

  • Park, H., Byun, S. Y., & Kim, K. K. (2011). Parental involvement and students’ cognitive outcomes in Korea: Focusing on private tutoring. Sociology of Education, 84(1), 3–22.

    Article  Google Scholar 

  • Roth, J. (2022). Pretest with caution: Event-study estimates after testing for parallel trends. American Economic Review: Insights, 4(3), 305–322. https://doi.org/10.1257/aeri.20210236

    Article  Google Scholar 

  • Roth, J., Sant’Anna, P. H. C., Bilinski, A., & Poe, J. (2023). What’s trending in difference-in-differences? A synthesis of the recent econometrics literature. Journal of Econometrics, 235(2), 2218–2244. https://doi.org/10.1016/j.jeconom.2023.03.008

    Article  Google Scholar 

  • Rubin, D. B. (2005). Causal inference using potential outcomes: Design, modeling, decisions. Journal of the American Statistical Association, 100(469), 322–331.

    Article  Google Scholar 

  • Shadish, W. R., Cook, T. D., & Campbell, D. T. (2002). Experimental and quasi-experimental designs for generalized causal inference. Wadsworth.

    Google Scholar 

  • Slavin, R. E. (2002). Evidence-based education policies: Transforming educational practice and research. Educational Researcher, 31(7), 15–21. https://doi.org/10.3102/0013189X031007015

    Article  Google Scholar 

  • St. Clair, T., & Cook, T. D. (2015). Difference-in-differences methods in public finance. National Tax Journal, 68(2), 319–338. https://doi.org/10.17310/ntj.2015.2.04

    Article  Google Scholar 

  • Sun, L., & Abraham, S. (2021). Estimating dynamic treatment effects in event studies with heterogeneous treatment effects. Journal of Econometrics, 225(2), 175–199. https://doi.org/10.1016/j.jeconom.2020.09.006

    Article  Google Scholar 

  • Wooldridge, J. M. (2021). Two-Way Fixed Effects, the two-way mundlak regression, and difference-in-differences estimators (SSRN Scholarly Paper No. 3906345). https://doi.org/10.2139/ssrn.3906345.

  • Zhang, W., & Bray, M. (2020). Comparative research on shadow education: Achievements, challenges, and the agenda ahead. European Journal of Education, 55(3), 322–341. https://doi.org/10.1111/ejed.12413

    Article  Google Scholar 

Download references

Funding

We received no funding to complete this project.

Author information

Authors and Affiliations

  1. Department of Leadership, Higher and Adult Education, University of Toronto, 252 Bloor Street West, Toronto, ON, M5S 1V6, Canada

    Daniel Corral

  2. Department of Educational Leadership and Policy Analysis, University of Missouri, Columbia, USA

    Minseok Yang

Authors
  1. Daniel Corral
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Minseok Yang
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

As the first author, Dr. Corral conceptualized the study. Both authors contributed to the study design, collaboratively wrote the first draft of the manuscript, and approved the final manuscript.

Corresponding author

Correspondence to Daniel Corral.

Ethics declarations

Conflicts of interest

We report no potential conflicts of interest.

Ethical approval

No ethics board approval was required given that this manuscript is based on simulated data.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Below is the link to the electronic supplementary material.

Supplementary file1 (DO 6 KB)

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Corral, D., Yang, M. An introduction to the difference-in-differences design in education policy research. Asia Pacific Educ. Rev. (2024). https://doi.org/10.1007/s12564-024-09959-0

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s12564-024-09959-0

Keywords

Search

Navigation