Abstract
Regression discontinuity (RD) designs have gained significant popularity as a quasi-experimental device for evaluating education programs and policies. In this paper, we present a comprehensive review of RD designs, focusing on the continuity-based framework, the most widely adopted RD framework. We first review the fundamental aspects of RD designs, drawing on potential outcomes and causal graphs. We then discuss the validity threats in RD designs, including manipulation, discreteness of the running variable, statistical power, and generalizability. Additionally, we provide an overview of the existing extensions to RD designs. To exemplify the application of RD methods, we analyze the effect of New Jersey’s pre-kindergarten program on children’s vocabulary test scores, using an educational dataset. Finally, we offer practical guidelines in the conclusion to promote the appropriate use of RD methods in educational research.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
Notes
When the functional form of the regression model is uncertain, it is recommended to adopt an overfitting strategy by including more polynomial and interaction terms than strictly necessary (Shadish et al., 2002).
Note that in fuzzy RD designs, it is recommended to select the bandwidth based on the outcome regression and then use the same bandwidth for the treatment regression. This recommendation is based on the observation that the treatment regression typically requires a wider bandwidth, as it is expected to exhibit a very flat relationship.
The backdoor criterion in causal graphs (Pearl, 1995) involves identifying and adjusting for variables that lie on “backdoor paths” between the treatment and outcome variables. By conditioning on these variables, non-causal paths are blocked, and this blocking allows for unbiased estimation of causal effects in observational studies.
References
Angrist, J. D., & Rokkanen, M. (2015). Wanna get away? regression discontinuity estimation of exam school effects away from the cutoff. Journal of the American Statistical Association, 110(512), 1331–1344. https://doi.org/10.1080/01621459.2015.1012259
Bergolo, M., & Galván, E. (2018). Intra-household behavioral responses to cash transfer programs. evidence from a regression discontinuity design. World Development, 103, 100–118. https://doi.org/10.1016/j.worlddev.2017.10.030
Brunner, E. J., Dougherty, S. M., & Ross, S. L. (2023). The effects of career and technical education: Evidence from the connecticut technical high school system. Review of Economics and Statistics. https://doi.org/10.1162/rest_a_01098
Bulus, M. (2021). Minimum detectable effect size computations for cluster-level regression discontinuity studies: Specifications beyond the linear functional form. Journal of Research on Educational Effectiveness, 15(1), 151–177. https://doi.org/10.1080/19345747.2021.1947425
Calonico, S., Cattaneo, M. D., Farrell, M. H., & Titiunik, R. (2019). Regression discontinuity designs using covariates. The Review of Economics and Statistics, 101(3), 442–451. https://doi.org/10.1162/rest_a_00760
Calonico, S., Cattaneo, M. D., Farrell, M. H., & Titiunik, R. (2023). Rdrobust: Robust data-driven statistical inference in regression-discontinuity designs [R package version 2.2]. https://CRAN.R-project.org/package=rdrobust.
Calonico, S., Cattaneo, M. D., & Titiunik, R. (2014). Robust nonparametric confidence intervals for regression-discontinuity designs: Robust nonparametric confidence intervals. Econometrica, 82(6), 2295–2326. https://doi.org/10.3982/ecta11757
Campbell, F. A., & Ramey, C. T. (1994). Effects of early intervention on intellectual and academic achievement: A follow-up study of children from low-income families. Child Development, 65(2), 684. https://doi.org/10.2307/1131410
Card, D., Lee, D. S., Pei, Z., & Weber, A. (2015). Inference on causal effects in a generalized regression kink design. Econometrica, 83(6), 2453–2483. https://doi.org/10.3982/ecta11224
Card, D., Lee, D. S., Pei, Z., & Weber, A. (2017). Regression kink design: Theory and practice. Advances in econometrics (pp. 341–382). Emerald Publishing Limited. https://doi.org/10.1108/s0731-905320170000038016
Carlson, D., & Knowles, J. E. (2016). The effect of english language learner reclassification on student ACT scores, high school graduation, and postsecondary enrollment: Regression discontinuity evidence from wisconsin. Journal of Policy Analysis and Management, 35(3), 559–586. https://doi.org/10.1002/pam.21908
Cattaneo, M. D., Idrobo, N., & Titiunik, R. (2019). A practical introduction to regression discontinuity designs: Extensions. Cambridge University Press. https://doi.org/10.4135/9781412993869
Cattaneo, M. D., Idrobo, N., & Titiunik, R. (2019). A practical introduction to regression discontinuity designs: Foundations. Cambridge University Press. https://doi.org/10.4135/9781412993869
Cattaneo, M. D., Frandsen, B. R., & Titiunik, R. (2015). Randomization inference in the regression discontinuity design: An application to party advantages in the u.s. senate. Journal of Causal Inference, 3(1), 1–24. https://doi.org/10.1515/jci-2013-0010
Cattaneo, M. D., Jansson, M., & Ma, X. (2018). Manipulation testing based on density discontinuity. The Stata Journal: Promoting communications on statistics and Stata, 18(1), 234–261. https://doi.org/10.1177/1536867x1801800115
Cattaneo, M. D., Jansson, M., & Ma, X. (2020). Simple local polynomial density estimators. Journal of the American Statistical Association, 115(531), 1449–1455. https://doi.org/10.1080/01621459.2019.1635480
Cattaneo, M. D., Jansson, M., & Ma, X. (2023). Rddensity: Manipulation testing based on density discontinuity [R package version 2.4]. https://CRAN.R-project.org/package=rddensity.
Cattaneo, M. D., Keele, L., Titiunik, R., & Vazquez-Bare, G. (2020). Extrapolating treatment effects in multi-cutoff regression discontinuity designs. Journal of the American Statistical Association, 116(536), 1941–1952. https://doi.org/10.1080/01621459.2020.1751646
Cattaneo, M. D., Titiunik, R., & Vazquez-Bare, G. (2019). Power calculations for regression-discontinuity designs. The Stata Journal: Promoting communications on statistics and Stata, 19(1), 210–245. https://doi.org/10.1177/1536867x19830919
Cattaneo, M. D., & Titiunik, R. (2022). Regression discontinuity designs. Annual Review of Economics, 14(1), 821–851. https://doi.org/10.1146/annurev-economics-051520-021409
Cattaneo, M. D., Titiunik, R., & Vazquez-Bare, G. (2017). Comparing inference approaches for RD designs: A reexamination of the effect of head start on child mortality (B. S. Barnow, Ed.). Journal of Policy Analysis and Management, 36(3), 643–681. https://doi.org/10.1002/pam.21985
Cook, T. D. (2008). Waiting for life to arrive: A history of the regression-discontinuity design in psychology, statistics and economics. Journal of Econometrics, 142(2), 636–654. https://doi.org/10.1016/j.jeconom.2007.05.002
Coyne, M. D., Oldham, A., Dougherty, S. M., Leonard, K., Koriakin, T., Gage, N. A., Burns, D., & Gillis, M. (2018). Evaluating the effects of supplemental reading intervention within an MTSS or RTI reading reform initiative using a regression discontinuity design. Exceptional Children, 84(4), 350–367. https://doi.org/10.1177/0014402918772791
Dong, Y. (2014). Regression discontinuity applications with rounding errors in the running variable. Journal of Applied Econometrics, 30(3), 422–446. https://doi.org/10.1002/jae.2369
Dong, Y., & Lewbel, A. (2015). Identifying the effect of changing the policy threshold in regression discontinuity models. Review of Economics and Statistics, 97(5), 1081–1092. https://doi.org/10.1162/rest_a_00510
Figlio, D., Holden, K. L., & Ozek, U. (2018). Do students benefit from longer school days? regression discontinuity evidence from florida’s additional hour of literacy instruction. Economics of Education Review, 67, 171–183. https://doi.org/10.1016/j.econedurev.2018.06.003
Figlio, D., & Özek, U. (2023). The unintended consequences of test-based remediation (tech. rep.). NBER Working Paper No. w30831. https://doi.org/10.2139/ssrn.4320579.
Goldberger, A. S. (1972). Selection bias in evaluating treatment effects: Some formal illustrations. University of Wisconsin-Madison.
Grembi, V., Nannicini, T., & Troiano, U. (2016). Do fiscal rules matter? American Economic Journal: Applied Economics, 8(3), 1–30. https://doi.org/10.1257/app.20150076
Hahn, J., Todd, P., & van der Klaauw, W. (2001). Identification and estimation of treatment effects with a regression-discontinuity design. Econometrica, 69(1), 201–209. https://doi.org/10.1111/1468-0262.00183
Hausman, C., & Rapson, D. S. (2018). Regression discontinuity in time: Considerations for empirical applications. Annual Review of Resource Economics, 10(1), 533–552. https://doi.org/10.1146/annurev-resource-121517-033306
Heissel, J. A., & Ladd, H. F. (2018). School turnaround in north carolina: A regression discontinuity analysis. Economics of Education Review, 62, 302–320. https://doi.org/10.1016/j.econedurev.2017.08.001
Hjalmarsson, R. (2009). Juvenile jails: A path to the straight and narrow or to hardened criminality? The Journal of Law and Economics, 52(4), 779–809. https://doi.org/10.1086/596039
Imbens, G., & Kalyanaraman, K. (2012). Optimal bandwidth choice for the regression discontinuity estimator. The Review of Economic Studies, 79(3), 933–959. https://doi.org/10.1093/restud/rdr043
Imbens, G., & Lemieux, T. (2008). Regression discontinuity designs: A guide to practice. Journal of Econometrics, 142(2), 615–635. https://doi.org/10.1016/j.jeconom.2007.05.001
Imbens, G., & Rubin, D. B. (2015). Causal inference in statistics, social, and biomedical sciences. Cambridge University Press. https://doi.org/10.1017/cbo9781139025751
Keele, L. J., & Titiunik, R. (2015). Geographic boundaries as regression discontinuities. Political Analysis, 23(1), 127–155. https://doi.org/10.1093/pan/mpu014
Lee, D. S. (2008). Randomized experiments from non-random selection in U.S. house elections. Journal of Econometrics, 142(2), 675–697. https://doi.org/10.1016/j.jeconom.2007.05.004
Lee, D. S., & Card, D. (2008). Regression discontinuity inference with specification error. Journal of Econometrics, 142(2), 655–674. https://doi.org/10.1016/j.jeconom.2007.05.003
Lee, D. S., & Lemieux, T. (2010). Regression discontinuity designs in economics. Journal of Economic Literature, 48(2), 281–355. https://doi.org/10.1257/jel.48.2.281
Lee, M. G., & Soland, J. G. (2022). Does reclassification change how english learners feel about school and themselves? evidence from a regression discontinuity design. Educational Evaluation and Policy Analysis, 45(1), 27–51. https://doi.org/10.3102/01623737221097419
Li, F., Mercatanti, A., Mäkinen, T., & Silvestrini, A. (2021). A regression discontinuity design for ordinal running variables: Evaluating central bank purchases of corporate bonds. The Annals of Applied Statistics, 15(1), 304–322. https://doi.org/10.1214/20-AOAS1396
Linden, A., & Adams, J. L. (2012). Combining the regression discontinuity design and propensity score-based weighting to improve causal inference in program evaluation. Journal of Evaluation in Clinical Practice, 18(2), 317–325. https://doi.org/10.1111/j.1365-2753.2011.01768.x
Ludwig, J., & Miller, D. L. (2007). Does head start improve children’s life chances? evidence from a regression discontinuity design. The Quarterly Journal of Economics, 122(1), 159–208. https://doi.org/10.1162/qjec.122.1.159
McCrary, J. (2008). Manipulation of the running variable in the regression discontinuity design: A density test. Journal of Econometrics, 142(2), 698–714. https://doi.org/10.1016/j.jeconom.2007.05.005
McEachin, A., Domina, T., & Penner, A. (2020). Heterogeneous effects of early algebra across california middle schools. Journal of Policy Analysis and Management, 39(3), 772–800. https://doi.org/10.1002/pam.22202
Mealli, F., & Rampichini, C. (2012). Evaluating the effects of university grants by using regression discontinuity designs. Journal of the Royal Statistical Society Series A: Statistics in Society, 175(3), 775–798. https://doi.org/10.1111/j.1467-985x.2011.01022.x
Melguizo, T., Sanchez, F., & Velasco, T. (2015). Credit for low-income students and access to and academic performance in higher education in colombia: A regression discontinuity approach. SSRN Electronic Journal. https://doi.org/10.2139/ssrn.2608642
Neyman, J. S. (1923). On the application of probability theory to agricultural experiments: Essay on principles. Section 9 (with discussion). Statistical Science, 4, 465–480.
Nielsen, H. S., Sørensen, T., & Taber, C. (2010). Estimating the effect of student aid on college enrollment: Evidence from a government grant policy reform. American Economic Journal: Economic Policy, 2(2), 185–215. https://doi.org/10.1257/pol.2.2.185
Nomi, T., & Raudenbush, S. W. (2016). Making a success of algebra for all. Educational Evaluation and Policy Analysis, 38(2), 431–451. https://doi.org/10.3102/0162373716643756
Otsu, T., Xu, K.-L., & Matsushita, Y. (2013). Estimation and inference of discontinuity in density. Journal of Business & Economic Statistics, 31(4), 507–524. https://doi.org/10.1080/07350015.2013.818007
Pearl, J. (1995). Causal diagrams for empirical research. Biometrika, 82(4), 669–688. https://doi.org/10.1093/biomet/82.4.669
Pearl, J. (2009). Causality: Models, reasoning, and inference. Cambridge University Press. https://doi.org/10.1017/CBO9780511803161
Reardon, S. F., & Robinson, J. P. (2012). Regression discontinuity designs with multiple rating-score variables. Journal of Research on Educational Effectiveness, 5(1), 83–104. https://doi.org/10.1080/19345747.2011.609583
Rokkanen, M. (2015). Exam schools, ability, and the effects of affirmative action: Latent factor extrapolation in the regression discontinuity design (tech. rep.). Discuss. Pap. 1415-03, Dep. Econ., Columbia Univ., New York.
Rubin, D. B. (1974). Estimating causal effects of treatments in randomized and nonrandomized studies. Journal of Educational Psychology, 66(5), 688–701. https://doi.org/10.1037/h0037350
Rubin, D. B. (1986). Comment: Which ifs have causal answers. Journal of the American Statistical Association, 81(396), 961–962. https://doi.org/10.2307/2289065
Schochet, P. (2009). Statistical power for regression discontinuity designs in education evaluations. Journal of Educational and Behavioral Statistics, 34(2), 238–266.
Schochet, P., Cook, T., Deke, J., Imbens, G., Lockwood, J. R., Porter, J., & Smith, J. Standards for regression discontinuity designs. 2010. http://ies.ed.gov/ncee/wwc/pdf/wwc%5C_rd.pdf.
Schwerdt, G., West, M. R., & Winters, M. A. (2017). The effects of test-based retention on student outcomes over time: Regression discontinuity evidence from florida. Journal of Public Economics, 152, 154–169. https://doi.org/10.1016/j.jpubeco.2017.06.004
Shadish, W. R., Cook, T. D., & Campbell, D. T. (2002). Experimental and quasi-experimental designs for generalized causal inference. Houghton Mifflin.
Steiner, P. M., & Cook, D. (2013). Matching and propensity scores. In T. Little (Ed.), The oxford handbook of quantitative methods (pp. 236–258). Oxford University Press.
Steiner, P. M., Kim, Y., Hall, C. E., & Su, D. (2017). Graphical models for quasi-experimental designs. Sociological Methods & Research, 46(2), 155–188. https://doi.org/10.1177/0049124115582272
Suk, Y., & Kim, Y. (2023). Fuzzy regression discontinuity designs with multiple control groups under one-sided noncompliance: Evaluating extended time accommodations. https://doi.org/10.31234/osf.io/sa96g
Suk, Y., Steiner, P. M., Kim, J.-S., & Kang, H. (2022). Regression discontinuity designs with an ordinal running variable: Evaluating the effects of extended time accommodations for english-language learners. Journal of Educational and Behavioral Statistics, 47(4), 459–484. https://doi.org/10.3102/10769986221090275
Tang, Y., Cook, T. D., Kisbu-Sakarya, Y., Hock, H., & Chiang, H. (2017). The comparative regression discontinuity (crd) design: An overview and demonstration of its performance relative to basic rd and the randomized experiment. In Regression discontinuity designs (pp. 237–279). Emerald Publishing Limited. https://doi.org/10.1108/s0731-905320170000038011.
Thistlethwaite, D. L., & Campbell, D. T. (1960). Regression-discontinuity analysis: An alternative to the ex post facto experiment. Journal of Educational Psychology, 51(6), 309–317. https://doi.org/10.1037/h0044319
Villamizar-Villegas, M., Pinzon-Puerto, F. A., & Ruiz-Sanchez, M. A. (2021). A comprehensive history of regression discontinuity designs: An empirical survey of the last 60 years. Journal of Economic Surveys, 36(4), 1130–1178. https://doi.org/10.1111/joes.12461
Wing, C., & Cook, T. D. (2013). Strengthening the regression discontinuity design using additional design elements: A within-study comparison. Journal of Policy Analysis and Management, 32(4), 853–877. https://doi.org/10.1002/pam.21721
Wong, V. C., Cook, T. D., Barnett, W. S., & Jung, K. (2007). An effectiveness-based evaluation of five state pre-kindergarten programs. Journal of Policy Analysis and Management, 27(1), 122–154. https://doi.org/10.1002/pam.20310
Wong, V. C., Steiner, P. M., & Cook, T. D. (2013). Analyzing regression-discontinuity designs with multiple assignment variables. Journal of Educational and Behavioral Statistics, 38(2), 107–141. https://doi.org/10.3102/1076998611432172
Acknowledgements
We would like to thank two editors for this special issue, Peter Steiner and Yongnam Kim, as well as an anonymous reviewer, for their useful comments that have improved the manuscript. We would also like to thank Vivian Wong for granting permission to use her data from {wong et al. 2007} for the purpose of demonstrating regression discontinuity designs in this work.
Funding
This work was partly supported by a grant from the American Educational Research Association which receives funds for its "AERA Grants Program" from the National Science Foundation under NSF award NSF-DRL #1749275. Opinions reflect those of the author and do not necessarily reflect those AERA or NSF.
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Conflicting interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
About this article
Cite this article
Suk, Y. Regression discontinuity designs in education: a practitioner’s guide. Asia Pacific Educ. Rev. (2024). https://doi.org/10.1007/s12564-024-09956-3
Received:
Revised:
Accepted:
Published:
DOI: https://doi.org/10.1007/s12564-024-09956-3