Abstract
Two major statistical issues confronting comparative analyses of hospital outcomes are adequacy of case-mix adjustment and proper accounting for random variation. Hierarchical modeling has been proposed to improve precision and reduce the impact of random variation but becomes difficult to implement when there are numerous case-mix factors to control. In this paper we formulate the problem of hospital performance comparisons within the framework of potential outcomes and illustrate an approach to hospital comparisons which combines multiple category propensity score methods for the control of case-mix variations with hierarchical Bayesian modeling of case-mix adjusted summaries. The approach is similar to that proposed by Huang et al. (Health Serv Res 40:253–278, 2005) but extends their approach by using a Bayesian model to accommodate hospital level attributes and to facilitate joint modeling of performance for multiple outcomes. The analytical approach is illustrated by a comparison of 30 day post admission mortality risks for patients treated for acute myocardial infarction, pneumonia or stroke in 34 New Zealand public hospitals. In a small simulation study, reported in electronic supplementary material, hierarchical models outperformed non-hierarchical models, achieving both better credible interval coverage and shorter average interval lengths for measures of between hospital variation based on contrasts between the 90th and 10th percentiles of the mortality risk distribution. Simulation performance of hierarchical and non-hierarchical models in detecting unusual performance was similar.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Agency for Healthcare Research and Quality: Guide to Inpatient Quality Indicators. AHRQ Quality Indicators. Agency for Healthcare Research and Quality, Department of Health and Human Services, (2007)
Agresti, A.: Categorical Data Analysis. Wiley, New York (1990)
Austin, P.C.: Are (the log-odds of) hospital mortality rates normally distributed? Implications for studying variations in outcomes of medical care. J. Eval. Clin. Pract. 15, 514–523 (2009)
Austin, P.C., Alter, D.A., Tu, J.V.: The use of fixed and random-effects models for classifying hospitals as mortality outliers: A Monte Carlo assessment. Med. Decis. Mak. 23, 526–539 (2003)
Austin, P.C., Brunner, L.J.: Optimal Bayesian probability levels for hospital report cards. Health Serv. Outcomes Res. Methodol. 8, 80–97 (2008)
Begg, C.B., Gray, R.: Calculation of polytomous logistic regression parameters using individualised regressions. Biometrika 71, 11–18 (1984)
British Thoracic Society: Community-acquired pneumonia in adults in British hospitals in 1982–1983: a survey of aetiology, mortality, prognostic factors and outcome. Q. J. Med. 62, 195–220 (1986)
Bross, I.D.H.: Spurious effects from an extraneous variable. J. Chronic Dis. 19, 637–647 (1986)
Burgess, J.F., Christiansen, C.L., Michalak, S.E., Morris, C.N.: Medical profiling: improving standards and risk adjustments using hierarchical models. J. Health Econ. 19, 291–309 (2000)
Charlson, M.E., Pompei, P., Ales, K.L., McKenzie, C.R.: A new method of classifying prognostic comorbidity in longitudinal studies: development and validation. J. Chronic Dis. 40, 373–383 (1987)
Christiansen, C.L., Morris, C.N.: Improving the statistical approach to health care provider profiling. Ann. Intern. Med. 127, 764–768 (1997)
Cochran, W.G.: The effectiveness of adjustment by subclassification in removing bias in observational studies. Biometrics 24, 205–213 (1968)
Daniels, M.J.: A prior for the variance in hierarchical models. Can. J. Stat. 27, 567–578 (1999)
Deely, J.J., Smith, A.F.M.: Quantitative refinements for comparisons of institutional performance. J. R. Stat. Soc. A 161, 5–12 (1998)
Deyo, R.A., Cherkin, D.C., Ciol, M.A.: Adapting a clincial comorbidity index for use with ICD-9-CM administrative databases. J. Clin. Epidemiol. 45, 613–619 (1992)
Draper, D.: Discussion of the paper by Goldstein and Spiegelhalter. J. R. Stat. Soc. A 159, 416–418 (1996)
Elixhauser, A., Steiner, C., Harruis, D., Coffey, R.: Comorbidity measures for use with administrative data. Med. Care 36, 8–27 (1998)
Everson, P.J., Morris, C.N.: Inference for multivariate normal hierarchical models. J. R. Stat. Soc. B 62, 399–412 (2000)
Fine, M.J., Auble, T.E., Yealy, D.M., Hanusa, B.H., Weissfeld, L.A., Singer, D.E., Coley, C.M., Marrie, T.J., Kapoor, W.N.: A prediction rule to identify low-risk patients with community acquired pneumonia. N. Engl. J. Med. 336, 243–250 (1997)
Gelman, A., Carlin, J.B., Stern, H.S., Rubin, D.B.: Bayesian Data Analysis: Texts in Statistical Science, 2nd edn. Chapman & Hall/CRC, Boca Raton, Florida (2004)
Gelman, A., Hill, J.: Data Analysis using Regression and Multilevel/Hierarchical Models. Analytical Methods for Social Research. Cambridge University Press, New York (2007)
Gibberd, R., Hancock, S., Howley, P., Richards, K.: Using indicators to quantify the potential to improve the quality of health care. Int. J. Qual. Health Care 16(Supplement 1), i37–i43 (2004)
Goldstein, H., Spiegelhalter, D.: Statistical aspects of institutional performance (with discussion). J. R. Stat. Soc. A 159, 385–444 (1996)
Graham, P.: Bayesian inference for a generalized population attributable fraction: The impact of early vitamin A levels on chronic lung disease in very low birthweight infants. Stat. Med. 19, 937–956 (2000)
Greenland, S.: Putting background information about relative risks into conjugate prior distributions. Biometrics 57, 663–670 (2001)
Hansen, B.B.: The prognostic analogue of the propensity score. Biometrika 95(2), 481–488 (2008)
He, Y., Wolf, R.E., Normand, S.T.: Assessing geographical variations in hospital processes of care using multilevel item response models. Health Serv. Outcomes Res. Methodol. 10, 111–133 (2010)
Howley, P.P., Gibberd, R.W.: Using hierarchical models to analyse clinical indicators: a comparison of the gamma-Poisson and beta-binomial models. Int. J. Qual. Health Care 15, 319–329 (2003)
Huang, I., Frangakis, C., Dominici, F., Diette, G.B., Wu, A.W.: Application of a propensity score approach for risk adjustment in profiling multiple physician groups on asthma care. Health Serv. Res. 40, 253–278 (2005)
Iezzoni, L.I., Shwartz, M.S., Ash, A.S., Hughes, J.S., Daley, J., MacKiernan, Y.D.: Severity measurement methods and judging hospital death rates for pneumonia. Med. Care 34, 11–28 (1996)
Imbens, G.W.: The role of the propensity score in estimating dose-response functions. Biometrika 87, 706–710 (2000)
Krumholz, H.M., Normand, S.T., Spertus, J.A., Shahian, D.M., Brdley, E.H.: Measuring performance for treating heart attacks and heart failure: the case for outcomes measurement. Health Aff. 26, 75–85 (2007)
Krumholz, H.M., Wang, Y., Mattera, J.A., Wang, Y., Han, L.F., Ingber, M.J., Roman, S., Normand, S.T.: An administrative claims model suitable for profiling hospital performance based on 30-day mortality rates among patients with an acute myocardial infarction. Circulation 113, 1683–1692 (2006)
Lindley, D.V., Smith, A.F.M.: Bayes estimates for the linear model. J. R. Stat. Soc. B 34, 1–34 (1972)
McCandless, L.C., Gustafson, P., Austin, P.C.: Bayesian propensity score analysis for observational data. Stat. Med. 28, 94–112 (2009a)
McCandless, L.C., Gustafson, P., Austin, P.C., Levy, A.R.: Covariate balance in a Bayesian propensity score analysis of beta blocker therapy in heart failure patients. Epidemiol. Perspect. Innov. 6, 5 (2009b)
Natarajan, R., Kass, R.E.: Reference Bayesian methods for generalized linear mixed models. J. Am. Stat. Assoc. 95, 227–237 (2000)
Neyman, J.: On the application of probability theory to agricultural experiments. Essay on principles, section 9. Ann. Agric. Sci. 1–51 (1923). Reprinted (with introduction and discussion) in Stat. Sci. 5, 463–480 (1990) (Translated by D.M. Dabrowska and T.P. Speed)
Normand, S.L., Glickman, M., Gatsonis, C.: Statistical methods for profiling providers of medical care: issues and applications. J. Am. Stat. Assoc. 92, 803–814 (1997)
Ohlssen, D.I., Sharples, L.D., Spiegelhalter, D.J.: Flexible random-effects models using Bayesian semi-parametric models: applications to institutional comparisons. Stat. Med. 26, 2088–2112 (2007a)
Ohlssen, D.I., Sharples, L.D., Spiegelhalter, D.J.: A hierarchical modelling framework for identifying unusual performance in health care providers. J. R. Stat. Soc. A 170(4), 865–890 (2007b)
Pearl, J.: On the consistency rule in causal inference. Epidemiology 21, 872–875 (2010)
Racz, M.J., Sedransk, J.: Bayesian and frequentist methods for provider profiling using risk-adjusted assessments of medical outcomes. J. Am. Stat. Assoc. 105, 48–58 (2010)
Robins, J.M.: A new approach to causal inference in mortality studies with a sustained exposure period—application to control of the healthy worker s survivors effect. Math. Modell. 7, 1393–1512 (1986)
Robinson, J.: Development of Severity-Adjustment Models for Hospital Resource Utilization Data. The Centre for Health Systems Research and Analysis, University of Wisconsin-Madison, Wisconsin (2008)
Rosenbaum, P.: Model-based direct adjustment. J. Am. Stat. Assoc. 82, 387–394 (1987)
Rosenbaum, P., Rubin, D.B.: The central role of the propensity score in observational studies for causal effects. Biometrika 70, 41–55 (1983)
Rosenbaum, P., Rubin, D.B.: Constructing a control group using multivariate matched sampling methods that incorporate the propensity score. Am. Stat. 39, 33–38 (1985)
Rosenbaum, P.R., Rubin, D.B.: Reducing bias in observational studies using subclassification on the propensity score. J. Am. Stat. Assoc. 79, 516–524 (1984)
Rubin, D.B.: Bayesian inference for causal effects: the role of randomization. Ann. Stat. 7, 34–58 (1978)
Rubin, D.B.: Which ifs have causal answers? J. Am. Stat. Assoc. 81, 961–962 (1986)
Rubin, D.B.: Estimating causal effects from large data sets using propensity scores. Ann. Intern. Med. 127, 757–763 (1997)
Rutledge, R., Osler, T., Kromhout-Schiro, S.: Illness severity adjustment for outcomes analysis: validation of the ICISS methodology in all 821,455 patients hospitalised in North Carolina in 1996. Surgery 124, 187–196 (1998)
Salmond, C., Crampton, P.: NZDep2001 Index of Deprivation. Department of Public Health, Wellington School of Medicine and Health Sciences., Wellington, New Zealand (2002)
Spiegelhalter, D.J., Aylin, P., Best, N.G., Evans, S.J.W., Murray, G.D.: Commissioned analysis of surgical performance by using routine data: lessons from the Bristol inquiry. J. R. Stat. Soc. A 165, 1–31 (2002)
Spreeuwenberg, M.D., Bartak, A., Croon, M.A., Hagenaars, J.A., Busschbach, J.J.V., Andrea, H., Twisk, J., Stijnen, T.: The multiple propensity score as control for bias in the comparison of more than two treatment arms: an introduction from a case study in mental health. Med. Care 48, 166–174 (2010)
Stone, R.: The assumptions on which causal inferences rest. J. R. Stat. Soc. B 55, 455–466 (1993)
Thomas, J.W., Hofer, T.P.: Accuracy of risk-adjusted mortality rates as a measure of hospital quality of care. Med. Care 37, 83–92 (1999)
Tu, J.V., Austin, P.C., Walld, R., Roos, L., Agras, J., McDonald, M.M.: Development and validation of the Ontario acute myocardial infarction mortality prediction score. J. Am. College Cardiol. 37, 992–997 (2001)
Acknowledgement
This research was funded by the Health Research Council of New Zealand. We thank Zhaojing Gong for programming the Elixhauser algorithm for identifying comorbidities from secondary diagnosis codes.
Author information
Authors and Affiliations
Corresponding author
Electronic supplementary material
Below is the link to the electronic supplementary material.
Rights and permissions
About this article
Cite this article
Graham, P., Hider, P., Cumming, J. et al. Variation in New Zealand hospital outcomes: combining hierarchical Bayesian modeling and propensity score methods for hospital performance comparisons. Health Serv Outcomes Res Method 12, 1–28 (2012). https://doi.org/10.1007/s10742-012-0079-9
Received:
Revised:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10742-012-0079-9