Abstract
In the literature on the effects of charging user fees for healthcare on health outcomes, Shinsuke Tanaka’s 2014 AEJ: Economic Policy paper evaluating South Africa’s 1994 user fee reform, “Does Abolishing User Fees Lead to Improved Health Status? Evidence from Post-Apartheid South Africa,” is regarded as a seminal piece of evidence demonstrating that eliminating user fees significantly improves population health. Tanaka finds that South Africa’s elimination of user fees for pregnant women and young children caused children’s weight-for-age z-scores (WAZ) to rise by 0.6, making it one of the most successful child nutrition interventions ever studied. However, evaluations of the same reform employing other methodologies have found that it had little to no effect on outcomes like healthcare utilization, making it difficult to understand how it might have had such large effects on WAZ. In this paper, I replicate, reanalyze, and extend Tanaka, 2014, considering its use of a two-way fixed effects model (TWFE) with additional covariates in the context of recent methods literature exposing the limitations of such designs. I show that Tanaka’s approach may not yield an unbiased causal estimate under the intended conditional parallel trends assumption but find that his results are robust to reanalysis with updated estimators. I then explore the implications of substantial treatment effect heterogeneity across subgroups within Tanaka’s sample for the external validity of his results, arguing that this heterogeneity severely undermines the case that his results generalize. I thereby recontextualize the debate over the effects of eliminating user fees in South Africa in terms of new thinking about TWFE and treatment effect heterogeneity, illustrating the practical implications of those advances in the process.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
Notes
This analysis can be replicated using data and code available under the author’s name on the Harvard Dataverse.
He later confirms this with a regression—see Table 16.
If the treatment did benefit boys’ nutritional status so much more than girls’, one wonders whether perhaps it was itself responsible (at least in part) for the rise in the proportion of boys specifically in the high-treatment group by skewing under-five mortality by sex.
To do this, I used the DRDID package Sant’Anna and Zhao wrote for R to accompany their paper (Sant’Anna and Zhao 2021).
Far more individuals in Tanaka’s data set had a maternity home in their community in 1993 than a hospital, pharmacy, or dispensary.
My WHZ results tell a similar story, exceeding Tanaka’s estimates by quite a bit for newborn children and appearing comparable for already born children.
Furthermore, one might think utilization metrics are less likely to be confounded in an analysis of this sort than WAZ, as utilization is likely a function of fewer variables.
References
Abadie A (2005) Semiparametric difference-in-differences estimators. Rev Econ Stud 72(1):1–19. https://doi.org/10.1111/0034-6527.00321
Ansah EK, Narh-Bana S, Asiamah S, Dzordzordzi V, Biantey K, Dickson K, Gyapong JO, Koram KA, Greenwood BM, Mills A, Whitty CJM (2009) Effect of removing direct payment for health care on utilisation and health outcomes in Ghanaian children: A randomised controlled trial. PLoS Med. https://doi.org/10.1371/journal.pmed.1000007
Bauhoff S, Hotchkiss DR, Smith O (2011) The impact of medical insurance for the poor in Georgia: a regression discontinuity approach. Health Econ 20(11):1362–1378. https://doi.org/10.1002/hec.1673
Bhutta ZA, Ahmed T, Black RE, Cousens S, Dewey K, Giugliani E, Haider BA, Kirkwood B, Morris SS, Sachdev H, Shekar M (2008) What works? Interventions for maternal and child undernutrition and survival. Lancet 371(9610):417–440. https://doi.org/10.1016/S0140-6736(07)61693-6
Bhutta ZA, Das JK, Rizvi A, Gaffey MF, Walker N, Horton S, Webb P, Lartey A, Black RE (2013) Evidence-based interventions for improvement of maternal and child nutrition: What can be done and at what cost? Lancet 382(9890):452–477. https://doi.org/10.1016/S0140-6736(13)60996-4
Brink AS, Koch SF (2015) Did primary healthcare user fee abolition matter? Reconsidering South Africa’s experience. Dev South Afr 32(2):170–192. https://doi.org/10.1080/0376835X.2014.984373
Caetano C, Callaway B, Payne S, Rodrigues HS (2022) Difference in differences with time-varying covariates. arXiv. https://doi.org/1048550/ARXIV.2202.02903
Cinelli C, Forney A, Pearl J (2022) A crash course in good and bad controls. Sociol Methods Res. https://doi.org/10.1177/00491241221099552
Clemens MA (2017) The meaning of failed replications: a review and proposal. J Econ Surv 31(1):326–342. https://doi.org/10.1111/joes.12139
Conti G, Ginja R (2017) Who benefits from free health insurance: evidence from Mexico (Working Paper W17/26). Institute for Fiscal, London, UK. http://hdl.handle.net/10419/200286
Das J, Holla A, Mohpal A, Muralidharan K (2016) Quality and accountability in health care delivery: audit-study evidence from primary care in India. Am Econ Rev 106(12):3765–3799. https://doi.org/10.1257/aer.20151138
Dzakpasu S, Powell-Jackson T, Campbell OM (2014) Impact of user fees on maternal health service utilization and related health outcomes: a systematic review. Health Policy Plan 29(2):137–150. https://doi.org/10.1093/heapol/czs142
Hamermesh DS (2007) Viewpoint: replication in economics. Can J Econ Rev Can d’économique 40(3):715–733. https://doi.org/10.1111/j.1365-2966.2007.00428.x
International Food Policy Research Institute (2001). University of Natal-Durban. & University of Wisconsin-Madison. KwaZulu-Natal Income Dynamics Study 1993–1998, Data First, University of Cape Town, 2, https://doi.org/10.25828/mqh7-e378
Ito T, Tanaka S (2018) Abolishing user fees, fertility choice, and educational attainment. J Dev Econ 130:33–44. https://doi.org/10.1016/j.jdeveco.2017.09.006
Keats EC, Das JK, Salam RA, Lassi ZS, Imdad A, Black RE, Bhutta ZA (2021) Effective interventions to address maternal and child malnutrition: an update of the evidence. Lancet Child Adolesc Health 5(5):367–384. https://doi.org/10.1016/S2352-4642(20)30274-1
Koch SF (2017) User fee abolition and the demand for public health care. S Afr J Econ 85(2):242–258. https://doi.org/10.1111/saje.12146
Koch SF, Racine JS (2016) Healthcare facility choice and user fee abolition: regression discontinuity in a multinomial choice setting. J R Stat Soc A Stat Soc 179(4):927–950. https://doi.org/10.1111/rssa.12161
Kowalski AE (2021) Reconciling seemingly contradictory results from the Oregon health insurance experiment and the Massachusetts health reform. Rev Econ Stat. https://doi.org/10.1162/rest_a_01069
Lagarde M, Palmer N (2011) The impact of user fees on access to health services in low- and middle-income countries. Cochrane Database Syst Rev. https://doi.org/10.1002/14651858.CD009094
Lamichhane P, Sharma A, Mahal A (2017) Impact evaluation of free delivery care on maternal health service utilisation and neonatal health in Nepal. Health Policy Plan 32(10):1427–1436. https://doi.org/10.1093/heapol/czx124
Leonard KL (2002) When both states and markets fail: asymmetric information and the role of NGOs in African health care. Int Rev Law Econ 22(1):61–80. https://doi.org/10.1016/S0144-8188(02)00069-8
Leonard KL, Masatu MC (2010) Using the Hawthorne effect to examine the gap between a doctor’s best possible practice and actual performance. J Dev Econ 93(2):226–234. https://doi.org/10.1016/j.jdeveco.2009.11.001
Matthay EC, Glymour MM (2022) Causal inference challenges and new directions for epidemiologic research on the health effects of social policies. Curr Epidemiol Rep. https://doi.org/10.1007/s40471-022-00288-7
McKinnon B, Harper S, Kaufman JS, Bergevin Y (2015) Removing user fees for facility-based delivery services: a difference-in-differences evaluation from ten sub-Saharan African countries. Health Policy Plan 30(4):432–441. https://doi.org/10.1093/heapol/czu027
Nguyen BT, Lo Sasso AT (2019) The effect of universal health insurance for children in Vietnam. Health Econ Policy Law 14(3):299–314. https://doi.org/10.1017/S1744133117000159
Nguyen H, Wang W (2013) The effects of free government health insurance among small children-evidence from the free care for children under six policy in Vietnam. Int J Health Plann Manag 28(1):3–15. https://doi.org/10.1002/hpm.2114
Nwokolo A (2020) Abolition of user fees and adult mental health: evidence from South Africa. https://www.semanticscholar.org/paper/Abolition-of-User-Fees-and-Adult-Mental-Health%3A-Nwokolo/a8027675b9851d408fc86b0c6517d764a81889d8
Qin VM, Hone T, Millett C, Moreno-Serra R, McPake B, Atun R, Lee JT (2019) The impact of user charges on health outcomes in low-income and middle-income countries: a systematic review. BMJ Glob Health. https://doi.org/10.1136/bmjgh-2018-001087
Roth J, Sant’Anna PH, Bilinski A, Poe J (2022) What’s trending in difference-in-differences? A synthesis of the recent econometrics literature. arXiv. https://doi.org/1048550/ARXIV.2201.01194
Sant’Anna PH, Zhao J (2020) Doubly robust difference-in-differences estimators. J Econ 219(1):101–122. https://doi.org/10.1016/j.jeconom.2020.06.003
Sant’Anna PH, Zhao J (2021) DRDID: doubly robust difference-in-differences estimators (Version 1.0.3). Nashville, TN. https://cran.r-project.org/web/packages/DRDID/index.html.
Sood N, Bendavid E, Mukherji A, Wagner Z, Nagpal S, Mullen P (2014) Government health insurance for people below poverty line in India: quasi-experimental evaluation of insurance and health outcomes. BMJ. https://doi.org/10.1136/bmj.g5114
Sood N, Wagner Z (2016) Impact of health insurance for tertiary care on postoperative outcomes and seeking care for symptoms: Quasi-experimental evidence from Karnataka, India. BMJ Open. https://doi.org/10.1136/bmjopen-2015-010512
Tanaka S (2014) Does abolishing user fees lead to improved health status? Evidence from post-Apartheid South Africa. Am Econ J Econ Pol 6(3):282–312. https://doi.org/10.1257/pol.6.3.282
Tanaka S (2019) Replication data for: Does abolishing user fees lead to improved health status? Evidence from post-Apartheid South Africa (data set). Inter-university Consortium for Political and Social Research. Ann Arbor, MI. https://doi.org/10.3886/E114870V1
The Lancet (2010) Striving for universal health coverage. Lancet 376(9755):1799. https://doi.org/10.1016/S0140-6736(10)62148-4
Yates R (2009) Universal health care and the removal of user fees. Lancet 373(9680):2078–2081. https://doi.org/10.1016/S0140-6736(09)60258-0
Zeldow B, Hatfield LA (2021) Confounding and regression adjustment in difference-in-differences studies. Health Serv Res 56(5):932–941. https://doi.org/10.1111/1475-6773.13666
Acknowledgements
I would like to thank Kevin Croke and Winnie Yip for their invaluable guidance, feedback, and support in taking this project from conception to publication. Any flaws in my work, of course, are mine alone.
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Conflict of interest
Jonah S. Goldberg declares that he has no conflict of interest. The author has no relevant financial or non-financial interests to disclose.
Ethical approval
This article does not report the results of any studies with human participants or animals.
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Appendix A
Appendix A
See Tables 9, 10, 11, 12, 13, 14, 15, 16 and 17.
Rights and permissions
Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
About this article
Cite this article
Goldberg, J.S. What we measure when we measure the effects of user fees: a replication, reanalysis, and extension of Tanaka, 2014. Empir Econ 65, 1981–2009 (2023). https://doi.org/10.1007/s00181-023-02400-2
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s00181-023-02400-2