Propensity score analysis: promise, reality and irrational exuberance
- 1.3k Downloads
The aim of this work is to examine the promise that propensity scores can yield accurate effect estimates in nonrandomized experiments, review research on the realities of the conditions needed to meet this promise, and caution against irrational exuberance about their capacity to meet this promise.
A review of selected experimental work that illustrates both the promise and realities of propensity score analysis.
Propensity score analysis of nonrandomized experiments can yield the same results as randomized experiments. Those estimates depend on meeting the strong ignorability assumption that the available covariates well describe selection processes and on use of comparison groups that are from the same location with very similar focal characteristics. When those assumptions are not met, propensity scores may not yield accurate estimates.
The use of propensity score analysis has proliferated exponentially, especially in the last decade, but careful attention to its assumptions seems to be very rare in practice. Researchers and policymakers who rely on these extensive propensity score applications may be using evidence of largely unknown validity. All stakeholders should devote far more empirical attention to justifying that each study has met these assumptions.
KeywordsPropensity score Nonrandomized experiment Quasi-experiment
- Guo, S., & Fraser, M. W. (2010). Propensity score analysis: Statistical methods and applications. Thousand Oaks: Sage Publications.Google Scholar
- LaLonde, R. (1986). Evaluating the econometric evaluations of training programs with experimental data. American Economic Review, 76, 604–620.Google Scholar
- Light, R. J., Singer, J. D., & Willett, J. B. (1990). By design: Planning research in higher education. Cambridge: Harvard University Press.Google Scholar
- Luellen, J. (2007). A comparison of propensity score estimation and adjustment methods on simulated data (Unpublished doctoral dissertation). The University of Memphis, Memphis, TN.Google Scholar
- Popper, K. R. (1959). The logic of scientific discovery. New York: Basic Books.Google Scholar
- Renkewitz, R., Fuchs, H. M., & Fiedler, S. (2011). Is there evidence of publication biases in JDM research? Judgment and Decision Making, 6, 870–881.Google Scholar
- Shadish, W.R., Steiner, P.M., & Cook, T.D. (2008). Peikes, D.N., Moreno, L. & Orzol, S.M. (2008). Propensity score matching: A note of caution for evaluators of social programs. The American Statistician, 62, 222-231: Comment by Shadish, Steiner and Cook. Unpublished manuscript.Google Scholar