Abstract
Researchers are often faced with evaluating the effect of a policy or program that was simultaneously initiated across an entire population of units at a single point in time, and its effects over the targeted population can manifest at any time period afterwards. In the presence of data measured over time, Bayesian time series models have been used to impute what would have happened after the policy was initiated, had the policy not taken place, in order to estimate causal effects. However, the considerations regarding the definition of the target estimands, the underlying assumptions, the plausibility of such assumptions, and the choice of an appropriate model have not been thoroughly investigated. In this paper, we establish useful estimands for the evaluation of large-scale policies. We discuss that imputation of missing potential outcomes relies on an assumption which, even though untestable, can be partially evaluated using observed data. We illustrate an approach to evaluate this key causal assumption and facilitate model elicitation based on data from the time interval before policy initiation and using classic statistical techniques. As an illustration, we study the Hospital Readmissions Reduction Program (HRRP), a US federal intervention aiming to improve health outcomes for patients with pneumonia, acute myocardial infraction, or congestive failure admitted to a hospital. We evaluate the effect of the HRRP on population mortality among the elderly across the US and in four geographic subregions, and at different time windows. We find that the HRRP increased mortality from pneumonia and acute myocardial infraction across at least one geographical region and time horizon, and is likely to have had a detrimental effect on public health.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
Data availability
The code and datasets generated and analysed during the current study are available from the corresponding author on reasonable request.
Notes
Looking at Fig. S.16 in the Online Resource, we believe that \(\hbox {PM}_{2.5}\) can be excluded from the regional analysis without concerns. Indeed, the inclusion probabilities of \(\hbox {PM}_{2.5}\) are small in all models; moreover, the sensitivity analysis performed at the national level shows robustness to different choices of predictors (namely, the effects in Table 2 are in line with the estimates resulting from a local linear trend and seasonal model without \(\hbox {PM}_{2.5}\), see panel (1) in Table 4).
References
Abadie, A., Gardeazabal, J.: The economic costs of conflict: a case study of the Basque Country. Am. Econ. Rev. 93(1), 113–132 (2003)
Abadie, A., Diamond, A., Hainmueller, J.: Synthetic control methods for comparative case studies: estimating the effect of California’s tobacco control program. J. Am. Stat. Assoc. 105, 493–505 (2010)
Abadie, A., Diamond, A., Hainmueller, J.: Comparative politics and the synthetic control method. Am. J. Polit. Sci. 59, 495–510 (2015)
Agresti, A.: Categorical Data Analysis. John Wiley & Sons, Hoboken (2013)
Angrist, J.D., Pischke, J.S.: Mostly Harmless Econometrics: An Empiricist’s Companion. Princeton University Press, Princeton (2008)
Antonelli, J., Beck, B.: Heterogeneous causal effects of neighborhood policing in New York city with staggered adoption of the policy (2020). arXiv preprint arXiv:2006.07681
Athey, S., Imbens, G.W.: Identification and inference in nonlinear difference-in-differences models. Econometrica 74(2), 431–497 (2006). https://doi.org/10.1111/j.1468-0262.2006.00668.x
Ben-Michael, E., Feller, A., Rothstein, J.: The augmented synthetic control method. J. Am. Stat. Assoc. 116(536), 1789–1803 (2021)
Bernal, J.L., Cummins, S., Gasparrini, A.: Interrupted time series regression for the evaluation of public health interventions: a tutorial. Int. J. Epidemiol. 46(1), 348–355 (2017)
Bojinov, I., Shephard, N.: Time series experiments and causal estimands: exact randomization tests and trading. J. Am. Stat. Assoc. 114(528), 1665–1682 (2019)
Brockwell, P.J., Davis, R.A.: Time Series: Theory and Methods. Springer science & business media, Berlin (2009)
Brodersen, K.H., Gallusser, F., Koehler, J., et al.: Inferring causal impact using Bayesian structural time-series models. Annals Appl. Stat. 9(1), 247–274 (2015). https://doi.org/10.1214/14-AOAS788
Card, D., Krueger, A.B.: Minimum wages and employment: A case study of the fast food industry in New Jersey and Pennsylvania. Tech. rep., Working Paper No. 4509, National Bureau of Economic Research (1993)
Casalino, L.P., Elster, A., Eisenberg, A., et al.: Will pay-for-performance and quality reporting affect health care disparities? These rapidly proliferating programs do not appear to be devoting much attention to the possible impact on disparities in health care. Health Aff. 26(Suppl2), w405–w414 (2007)
Coxe, S., West, S.G., Aiken, L.S.: The analysis of count data: a gentle introduction to Poisson regression and its alternatives. J. Pers. Assess. 91(2), 121–136 (2009)
Cruz, M., Bender, M., Ombao, H.: A robust interrupted time series model for analyzing complex health care intervention data. Stat. Med. 36(29), 4660–4676 (2017)
Desai, N.R., Ross, J.S., Kwon, J.Y., et al.: Association between hospital penalty status under the hospital readmission reduction program and readmission rates for target and nontarget conditions. J. Am. Med. Assoc. 316(24), 2647–2656 (2016). https://doi.org/10.1001/jama.2016.18533
Dharmarajan, K., Wang, Y., Lin, Z., et al.: Association of changing hospital readmission rates with mortality rates after hospital discharge. J. Am. Med. Assoc. 318(3), 270–278 (2017). https://doi.org/10.1001/jama.2017.8444
Doudchenko, N., Imbens, G.: Balancing, regression, difference-in-differences and synthetic control methods: A synthesis. no. 22791. National Bureau of Economic Research (2016)
Doudchenko, N., Imbens, G.W.: Balancing, Regression, Difference-In-Differences and Synthetic Control Methods: A Synthesis (2017). arXiv preprint arXiv:1610.07748v2
Fonarow, G.C., Konstam, M.A., Yancy, C.W.: The hospital readmission reduction program is associated with fewer readmissions, more deaths: time to reconsider. J. Am. Coll. Cardiol. 70(15), 1931–1934 (2017). https://doi.org/10.1016/j.jacc.2017.08.046
Gaughan, J., Gutacker, N., Grašič, K., et al.: Paying for efficiency: incentivising same-day discharges in the English NHS. J. Health Econ. 68(102), 226 (2019)
Gelman, A., Carlin, J.B., Stern, H.S., et al.: Bayesian Data Analysis. CRC Press, Boca Raton (2013)
Gelman, A., Meng, X.L., Stern, H.: Posterior predictive assessment of model fitness via realized discrepancies. Statistica sinica (1996)
Griffin, B.A., Schuler, M.S., Pane, J., et al.: Methodological considerations for estimating policy effects in the context of co-occurring policies. Health Serv. Outcomes Res. Methodol. pp 1–17 (2022)
Gupta, A., Allen, L.A., Bhatt, D.L., et al.: Association of the hospital readmissions reduction program implementation with readmission and mortality outcomesin heart failure. JAMA Cardiol. 3(1), 44–53 (2017). https://doi.org/10.1001/jamacardio.2017.4265
Huckfeldt, P., Escarce, J., Sood, N., et al.: Thirty-day postdischarge mortality among black and white patients 65 years and older in the medicare hospital readmissions reduction programpostdischarge mortality among black and white patients with medicarepostdischarge mortality among black and white P. JAMA Netw. Open 2(3), e190,634-e190,634 (2019). https://doi.org/10.1001/jamanetworkopen.2019.0634
Imbens, G.W., Rubin, D.B.: Causal Inference in Statistics, Social, and Biomedical Sciences. Cambridge University Press, Cambridge (2015)
Joynt, K.E., Jha, A.K.: Thirty-day readmissions—truth and consequences. N. Engl. J. Med. 366(15), 1366–1369 (2012). https://doi.org/10.1056/NEJMp1201598
Joynt, K.E., Orav, E.J., Jha, A.K.: Thirty-day readmission rates for medicare beneficiaries by race and site of care. J. Am. Med. Assoc. 305(7), 675 (2011). https://doi.org/10.1001/jama.2011.123
Khera, R., Dharmarajan, K., Wang, Y., et al.: Association of the hospital readmissions reduction program with mortality during and after hospitalization for acute myocardial infarction, heart failure, and pneumonia. JAMA Newt. Open 1(5), e182777–e182777 (2018). https://doi.org/10.1001/jamanetworkopen.2018.2777
McDowall, D., McCleary, R., Bartos, B.J.: Interrupted Time Series Analysis. Oxford University Press, Oxford (2019)
Medina-Ramón, M., Zanobetti, A., Cavanagh, D.P., et al.: Extreme temperatures and mortality: assessing effect modification by personal characteristics and specific cause of death in a multi-city case-only analysis. Environ. Health Perspect. 114(9), 1331–1336 (2006)
MedPAC.: The effects of the Hospital Readmissions Reduction Program. Tech. rep., (2018) http://www.medpac.gov/docs/default-source/reports/jun18_ch1_medpacreport_sec.pdf?sfvrsn=0
Menchetti, F., Cipollini, F., Mealli, F.: Combining counterfactual outcomes and ARIMA models for policy evaluation., forthcoming. The Econometrics Journal. Earlier version available at (2021) https://arxiv.org/abs/2103.06740
Meyer, B.D., Viscusi, W.K., Durbin, D.L.: Workers’ compensation and injury duration: evidence from a natural experiment. Am. Econ. Rev. 85(3), 322–340 (1995)
Miratrix, L.: Package simits: analysis via simulation of interrupted time series (ITS) data (2020)
Miratrix, L., Anderson, C., Henderson, B., et al.: Simulating for uncertainty with interrupted time series designs. Tech. rep., (2019) https://www.mdrc.org/sites/default/files/img/methods_for_ITS.pdf
O’Neill, S., Kreif, N., Grieve, R., et al.: Estimating causal effects: considering three alternatives to difference-in-differences estimation. Health Serv. Outcomes Res. Method. 16, 1–21 (2016)
Ryan, A.M., Burgess, J.F., Jr., Dimick, J.B.: Why we should not be indifferent to specification choices for difference-in-differences. Health Serv. Res. 50, 1211–1235 (2015)
Sandhu, A.T., Heidenreich, P.A.: Comparison of the change in heart failure readmission and mortality rates between hospitals subject to hospital readmission reduction program penalties and critical access hospitals. Am. Heart J. 209, 63–67 (2019). https://doi.org/10.1016/j.ahj.2018.12.002
Schaffer, A.L., Dobbins, T.A., Pearson, S.A.: Interrupted time series analysis using autoregressive integrated moving average (ARIMA) models: a guide for evaluating large-scale health interventions. BMC Med. Res. Methodol. 21(1), 1–12 (2021)
Wadhera, R.K., Joynt Maddox, K.E., Wasfy, J.H., et al.: Association of the hospital readmissions reduction program with mortality among medicare beneficiaries hospitalized for heart failure, acute myocardial infarction, and pneumonia. JAMA 320(24), 2542–2552 (2018). https://doi.org/10.1001/jama.2018.19232
Wagner, A.K., Soumerai, S.B., Zhang, F., et al.: Segmented regression analysis of interrupted time series studies in medication use research. J. Clin. Pharm. Ther. 27(4), 299–309 (2002)
Wasfy, J.H., Zigler, C.M., Choirat, C., et al.: Readmission rates after passage of the hospital readmissions reduction program: a pre-post analysis. Annals Internal Med. 166(5), 324–331 (2017). https://doi.org/10.7326/M16-0185
West, M., Harrison, J.: Bayesian Forecasting and Dynamic Models. Springer-Verlag, New York (2006)
Wheeler, A.P.: The effect of 311 calls for service on crime in DC at microplaces. Crime Delinq. 64(14), 1882–1903 (2018)
Wadhera, R.K., Joynt, Maddox, K.E., Kazi, D.S., et al.: Hospital revisits within 30 days after discharge for medical conditions targeted by the Hospital Readmissions Reduction Program in the United States: national retrospective analysis. BMJ 366(l4563) (2019). https://doi.org/10.1136/bmj.l4563
Ye, S., Wang, R., Zhang, B.: Comparison of estimation methods and sample size calculation for parameter-driven interrupted time series models with count outcomes. Health Serv. Outcomes Res. Methodol. pp 1–48 (2022)
Zigler, C.M., Papadogeorgou, G.: Bipartite causal inference with interference. Stat. Sci. Rev. J. Inst. Math. Stat. 36(1), 109 (2021)
Zuckerman, R.B., Sheingold, S.H., Orav, E.J., et al.: Readmissions, observation, and the hospital readmissions reduction program. N. Engl. J. Med. 374(16), 1543–1551 (2016). https://doi.org/10.1056/NEJMsa1513024
Acknowledgements
The authors would like to thank Francesca Dominici, Alessandra Mattei, and Joseph Antonelli for their constructive comments.
Funding
Funding was provided by National Institutes of Health R01 GM111339, R01 ES024332, R01 ES026217, P50MD010428, DP2MD012722, R01 ES028033, USEPA 83615601, and Health Effects Institute 4953-RFA14-3/16-4. The contents of this work are solely the responsibility of the grantee and do not necessarily represent the official views of the USEPA. Further, USEPA does not endorse the purchase of any commercial products or services mentioned in the publication. Research described in this article was conducted under contract to the Health Effects Institute (HEI), an organization jointly funded by the United States Environmental Protection Agency (EPA) (Assistance Award No.CR-83467701) and certain motor vehicle and engine manufacturers. The contents of this article do not necessarily reflect the views of HEI, or its sponsors, nor do they necessarily reflect the views and policies of the EPA or motor vehicle and engine manufacturers. Dr. Wasfy is supported by National Institutes of Health and Harvard Catalyst KL2 TR001100 and American Heart Association 18CDA34110215. Dr. Menchetti and Prof. Mealli thank the Department of Excellence 2018-2022 funding provided by the Italian Ministry of Education, University and Research (MIUR).
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Conflict of interest
The authors have no conflicts of interest to declare that are relevant to the content of this article.
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Supplementary Information
Rights and permissions
Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
About this article
Cite this article
Papadogeorgou, G., Menchetti, F., Choirat, C. et al. Evaluating federal policies using Bayesian time series models: estimating the causal impact of the hospital readmissions reduction program. Health Serv Outcomes Res Method 23, 433–451 (2023). https://doi.org/10.1007/s10742-022-00294-8
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10742-022-00294-8