Abstract
A commonly used method for evaluating a hospital’s performance on an outcome is to compare the hospital’s observed outcome rate to the hospital’s expected outcome rate given its patient (case) mix and service. The process of calculating the hospital’s expected outcome rate given its patient mix and service is called risk adjustment (Iezzoni 1997). Risk adjustment is critical for accurately evaluating and comparing hospitals’ performances since we would not want to unfairly penalize a hospital just because it treats sicker patients. The key to risk adjustment is accurately estimating the probability of an Outcome given patient characteristics. For cases with binary outcomes, the method that is commonly used in risk adjustment is logistic regression. In this paper, we consider ensemble of trees methods as alternatives for risk adjustment, including random forests and Bayesian additive regression trees (BART). Both random forests and BART are modern machine learning methods that have been shown recently to have excellent performance for prediction of outcomes in many settings. We apply these methods to carry out risk adjustment for the performance of neonatal intensive care units (NICU). We show that these ensemble of trees methods outperform logistic regression in predicting mortality among babies treated in NICU, and provide a superior method of risk adjustment compared to logistic regression.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Normand S-LT, Shahian DM (2007) Statistical and clinical aspects of hospital outcomes profiling. Institute of Mathematical Statistics. Stat Sci 22(2):206–226
Austin PC (2008) Bayes rules for optimally using Bayesian hierarchical regression models in provider profiling to identify high-mortality hospitals. BMC Med Res Methodol 8(1):30. BioMed Central Ltd
Berta P, Seghieri C, Vittadini G (2013) Comparing health outcomes among hospitals: the experience of the Lombardy Region. Health Care Manag Sci 16(3):245–257. Springer, US. doi:10.1007/s10729-013-9227-1
Farrell PJ, Groshen S,MacGibbon B, Tomberlin TJ (2010) Outlier detection for a hierarchical Bayes model in a study of hospital variation in surgical procedures. SAGE Publications. StatMethods Med Res 19(6):601–619
He Y, Selck F, Normand S-LT (2013) On the accuracy of classifying hospitals on their performance measures. Stat Med. doi:10.1002/sim.6012
Ieva F, Paganoni A (2014) Detecting and visualizing outliers in provider profiling via funnel plots and mixed effect models. Health Care Manag Sci. Springer, US. doi:10.1007/s10729-013-9264-9
Kalbfleisch JD, Wolfe RA (2013) On monitoring outcomes of medical providers. Stat Biosci 5(2):286–302. Springer US
Mohammed MA, Manktelow BN, Hofer TP (2012) Comparison of four methods for deriving hospital standardised mortality ratios from a single hierarchical logistic regression model. Stat Methods Med Res. doi:10.1177/0962280212465165
Paddock SM, Louis TA (2011) Percentile-based empirical distribution function estimates for performance evaluation of healthcare providers. J R Stat Soc Ser C Appl Stat 60(4):575–589. Wiley Online Library
Phibbs CS, Bronstein JM, Buxton E, Phibbs RH (1996) The effects of patient volume and level of care at the hospital of birth on neonatal mortality. J Am Med Assoc 276(13):1054–1059
Phibbs CS, Baker LC, Caughey AB, Danielsen B, Schmitt SK, Phibbs RH (2007) Level and volume of neonatal intensive care and mortality in very-low-birth-weight infants. N Engl J Med. Mass Med Soc 356(21):2165–2175
Racz MJ, Sedransk J (2010) Bayesian and frequentist methods for provider profiling using risk-adjusted assessments of medicaloutcomes. J Am Stat Assoc 105(489):48–58
Lorch SA, Baiocchi M, Ahlberg CE, Small DS (2012) The differential impact of delivery hospital on the outcomes of premature infants. Am Acad Pediatr 130(2):270–278
Breiman L, Friedman JH, Olshen RA, Stone CJ (1984) Classification and regression trees. Wadsworth & Brooks, Monterey
Quinlan JR (1986) Induction of decision trees. Mach Learn 1(1):81–106
Dietterich TG (2000) Ensemble methods in machine learning. In: Multiple classifier systems. Springer, pp 1–15
Breiman L (1996) Bagging predictors.Mach Learn 24(2):123–140
Freund Y, Schapire RE et al (1996) Experiments with a new boosting algorithm. In: ICML, vol 96, pp 148–156
Bernardo JM, Smith AFM (2009) Bayesian theory, vol 405.Wiley, Hoboken
Breiman L (2001) Random forests. Mach Learn 45(1):5–32
Cutler DR, Edwards TC Jr, Beard KH, Cutler A, Hess KT, Gibson J, Lawler JJ (2007) Random forests for classification in ecology. Ecol Eco Soc Am 88(11):2783–2792
Riddick G, Song H, Ahn S,Walling J, Borges-Rivera D, ZhangW, Fine HA (2011) Predicting in vitro drug sensitivity using Random Forests. Bioinformatics 27(2):220–224. Oxford Univ Press
Svetnik V, Liaw A, Tong C, Culberson JC, Sheridan RP, Feuston BP (2003) Random forest: a classification and regression tool for compound classification and QSAR modeling. J Chem Inform Comput Sci 43(6):1947–1958. ACS Publications
Chipman HA, George EI, McCulloch RE (2010) BART: Bayesian additive regression trees. Ann Appl Stat 4(1):266–298. Institute of Mathematical Statistics
Díaz-Uriarte R, De Andres SA (2006) Gene selection and classification of microarray data using random forest. BMC bioinforma 7(1):3. BioMed Central Ltd
Pasta DJ (2009) Learning when to be discrete: continuous vs. categorical predictors. SAS Global Forum, Washington, DC
Malley JD, Kruppa J, Dasgupta A, Malley KG, Ziegler A (2012) Probability machines: consistent probability estimation using non-parametric learning machines. Methods Inf Med 51(1):74
Little RJA, Rubin DB (2002) Statistical analysis with missing data. Wiley, Hoboken
Bostrom H (2007) Estimating class probabilities in random forests In: 6th international conference on machine learning and applications, 2007. ICMLA 2007. IEEE, pp 211–216
Provost F, Domingos P (2000) Well-trained PETs: improving probability estimation trees. Citeseer
Devroye L (1996) A probabilistic theory of pattern recognition, vol 31. Springer
Rosenbaum PR, Rubin DB (1983) The central role of the propensity score in observational studies for causal effects. Biometrika 70(1):41–55
Stuart EA (2010) Matching methods for causal inference: a review and a look forward. Stat Sci Rev J Inst Math Stat 25(1):1. NIH Public Access
Silber JH, Rosenbaum PR, Ross RN, Ludwig JM, Wang W, Niknam BA, Mukherjee N, Saynisch PA, Even-Shoshan O, Kelz RR, Fleisher LA (2013) Template matching for auditing hospital cost and quality. Health Services Research, in press. doi: 10.1111/1475-6773.12156
Iezzoni LI (1997) Risk adjustment for measuring health care outcomes. Health Administration Press, Chicago
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Liu, Y., Traskin, M., Lorch, S.A. et al. Ensemble of trees approaches to risk adjustment for evaluating a hospital’s performance. Health Care Manag Sci 18, 58–66 (2015). https://doi.org/10.1007/s10729-014-9272-4
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10729-014-9272-4