Health Care Management Science

, Volume 18, Issue 1, pp 58–66 | Cite as

Ensemble of trees approaches to risk adjustment for evaluating a hospital’s performance

  • Yang Liu
  • Mikhail Traskin
  • Scott A. Lorch
  • Edward I. George
  • Dylan Small


A commonly used method for evaluating a hospital’s performance on an outcome is to compare the hospital’s observed outcome rate to the hospital’s expected outcome rate given its patient (case) mix and service. The process of calculating the hospital’s expected outcome rate given its patient mix and service is called risk adjustment (Iezzoni 1997). Risk adjustment is critical for accurately evaluating and comparing hospitals’ performances since we would not want to unfairly penalize a hospital just because it treats sicker patients. The key to risk adjustment is accurately estimating the probability of an Outcome given patient characteristics. For cases with binary outcomes, the method that is commonly used in risk adjustment is logistic regression. In this paper, we consider ensemble of trees methods as alternatives for risk adjustment, including random forests and Bayesian additive regression trees (BART). Both random forests and BART are modern machine learning methods that have been shown recently to have excellent performance for prediction of outcomes in many settings. We apply these methods to carry out risk adjustment for the performance of neonatal intensive care units (NICU). We show that these ensemble of trees methods outperform logistic regression in predicting mortality among babies treated in NICU, and provide a superior method of risk adjustment compared to logistic regression.


Random forest BART Logistic regression Risk adjustment Neonatal mortality 


  1. 1.
    Normand S-LT, Shahian DM (2007) Statistical and clinical aspects of hospital outcomes profiling. Institute of Mathematical Statistics. Stat Sci 22(2):206–226CrossRefGoogle Scholar
  2. 2.
    Austin PC (2008) Bayes rules for optimally using Bayesian hierarchical regression models in provider profiling to identify high-mortality hospitals. BMC Med Res Methodol 8(1):30. BioMed Central LtdCrossRefGoogle Scholar
  3. 3.
    Berta P, Seghieri C, Vittadini G (2013) Comparing health outcomes among hospitals: the experience of the Lombardy Region. Health Care Manag Sci 16(3):245–257. Springer, US. doi: 10.1007/s10729-013-9227-1 CrossRefGoogle Scholar
  4. 4.
    Farrell PJ, Groshen S,MacGibbon B, Tomberlin TJ (2010) Outlier detection for a hierarchical Bayes model in a study of hospital variation in surgical procedures. SAGE Publications. StatMethods Med Res 19(6):601–619CrossRefGoogle Scholar
  5. 5.
    He Y, Selck F, Normand S-LT (2013) On the accuracy of classifying hospitals on their performance measures. Stat Med. doi: 10.1002/sim.6012
  6. 6.
    Ieva F, Paganoni A (2014) Detecting and visualizing outliers in provider profiling via funnel plots and mixed effect models. Health Care Manag Sci. Springer, US. doi: 10.1007/s10729-013-9264-9
  7. 7.
    Kalbfleisch JD, Wolfe RA (2013) On monitoring outcomes of medical providers. Stat Biosci 5(2):286–302. Springer USCrossRefGoogle Scholar
  8. 8.
    Mohammed MA, Manktelow BN, Hofer TP (2012) Comparison of four methods for deriving hospital standardised mortality ratios from a single hierarchical logistic regression model. Stat Methods Med Res. doi: 10.1177/0962280212465165
  9. 9.
    Paddock SM, Louis TA (2011) Percentile-based empirical distribution function estimates for performance evaluation of healthcare providers. J R Stat Soc Ser C Appl Stat 60(4):575–589. Wiley Online LibraryCrossRefGoogle Scholar
  10. 10.
    Phibbs CS, Bronstein JM, Buxton E, Phibbs RH (1996) The effects of patient volume and level of care at the hospital of birth on neonatal mortality. J Am Med Assoc 276(13):1054–1059CrossRefGoogle Scholar
  11. 11.
    Phibbs CS, Baker LC, Caughey AB, Danielsen B, Schmitt SK, Phibbs RH (2007) Level and volume of neonatal intensive care and mortality in very-low-birth-weight infants. N Engl J Med. Mass Med Soc 356(21):2165–2175Google Scholar
  12. 12.
    Racz MJ, Sedransk J (2010) Bayesian and frequentist methods for provider profiling using risk-adjusted assessments of medicaloutcomes. J Am Stat Assoc 105(489):48–58CrossRefGoogle Scholar
  13. 13.
    Lorch SA, Baiocchi M, Ahlberg CE, Small DS (2012) The differential impact of delivery hospital on the outcomes of premature infants. Am Acad Pediatr 130(2):270–278Google Scholar
  14. 14.
    Breiman L, Friedman JH, Olshen RA, Stone CJ (1984) Classification and regression trees. Wadsworth & Brooks, MontereyGoogle Scholar
  15. 15.
    Quinlan JR (1986) Induction of decision trees. Mach Learn 1(1):81–106Google Scholar
  16. 16.
    Dietterich TG (2000) Ensemble methods in machine learning. In: Multiple classifier systems. Springer, pp 1–15Google Scholar
  17. 17.
    Breiman L (1996) Bagging predictors.Mach Learn 24(2):123–140Google Scholar
  18. 18.
    Freund Y, Schapire RE et al (1996) Experiments with a new boosting algorithm. In: ICML, vol 96, pp 148–156Google Scholar
  19. 19.
    Bernardo JM, Smith AFM (2009) Bayesian theory, vol 405.Wiley, HobokenGoogle Scholar
  20. 20.
    Breiman L (2001) Random forests. Mach Learn 45(1):5–32CrossRefGoogle Scholar
  21. 21.
    Cutler DR, Edwards TC Jr, Beard KH, Cutler A, Hess KT, Gibson J, Lawler JJ (2007) Random forests for classification in ecology. Ecol Eco Soc Am 88(11):2783–2792Google Scholar
  22. 22.
    Riddick G, Song H, Ahn S,Walling J, Borges-Rivera D, ZhangW, Fine HA (2011) Predicting in vitro drug sensitivity using Random Forests. Bioinformatics 27(2):220–224. Oxford Univ PressCrossRefGoogle Scholar
  23. 23.
    Svetnik V, Liaw A, Tong C, Culberson JC, Sheridan RP, Feuston BP (2003) Random forest: a classification and regression tool for compound classification and QSAR modeling. J Chem Inform Comput Sci 43(6):1947–1958. ACS PublicationsCrossRefGoogle Scholar
  24. 24.
    Chipman HA, George EI, McCulloch RE (2010) BART: Bayesian additive regression trees. Ann Appl Stat 4(1):266–298. Institute of Mathematical StatisticsCrossRefGoogle Scholar
  25. 25.
    Díaz-Uriarte R, De Andres SA (2006) Gene selection and classification of microarray data using random forest. BMC bioinforma 7(1):3. BioMed Central LtdCrossRefGoogle Scholar
  26. 26.
    Pasta DJ (2009) Learning when to be discrete: continuous vs. categorical predictors. SAS Global Forum, Washington, DCGoogle Scholar
  27. 27.
    Malley JD, Kruppa J, Dasgupta A, Malley KG, Ziegler A (2012) Probability machines: consistent probability estimation using non-parametric learning machines. Methods Inf Med 51(1):74CrossRefGoogle Scholar
  28. 28.
    Little RJA, Rubin DB (2002) Statistical analysis with missing data. Wiley, HobokenGoogle Scholar
  29. 29.
    Bostrom H (2007) Estimating class probabilities in random forests In: 6th international conference on machine learning and applications, 2007. ICMLA 2007. IEEE, pp 211–216Google Scholar
  30. 30.
    Provost F, Domingos P (2000) Well-trained PETs: improving probability estimation trees. CiteseerGoogle Scholar
  31. 31.
    Devroye L (1996) A probabilistic theory of pattern recognition, vol 31. SpringerGoogle Scholar
  32. 32.
    Rosenbaum PR, Rubin DB (1983) The central role of the propensity score in observational studies for causal effects. Biometrika 70(1):41–55CrossRefGoogle Scholar
  33. 33.
    Stuart EA (2010) Matching methods for causal inference: a review and a look forward. Stat Sci Rev J Inst Math Stat 25(1):1. NIH Public AccessGoogle Scholar
  34. 34.
    Silber JH, Rosenbaum PR, Ross RN, Ludwig JM, Wang W, Niknam BA, Mukherjee N, Saynisch PA, Even-Shoshan O, Kelz RR, Fleisher LA (2013) Template matching for auditing hospital cost and quality. Health Services Research, in press. doi:  10.1111/1475-6773.12156
  35. 35.
    Iezzoni LI (1997) Risk adjustment for measuring health care outcomes. Health Administration Press, ChicagoGoogle Scholar

Copyright information

© Springer Science+Business Media New York 2014

Authors and Affiliations

  • Yang Liu
    • 1
  • Mikhail Traskin
    • 1
  • Scott A. Lorch
    • 2
  • Edward I. George
    • 3
  • Dylan Small
    • 3
  1. 1.Department of MathematicsUniversity of PennsylvaniaPhiladelphiaUSA
  2. 2.The Childrens Hospital of Philadelphia and School of MedicineUniversity of PennsylvaniaPhiladelphiaUSA
  3. 3.Department of StatisticsThe Wharton School, University of PennsylvaniaPhiladelphiaUSA

Personalised recommendations