Classifying Hospitals as Mortality Outliers: Logistic Versus Hierarchical Logistic Models

  • Roxana Alexandrescu
  • Alex Bottle
  • Brian Jarman
  • Paul Aylin
Systems-Level Quality Improvement
Part of the following topical collections:
  1. Topical Collection on Systems-Level Quality Improvement

Abstract

The use of hierarchical logistic regression for provider profiling has been recommended due to the clustering of patients within hospitals, but has some associated difficulties. We assess changes in hospital outlier status based on standard logistic versus hierarchical logistic modelling of mortality. The study population consisted of all patients admitted to acute, non-specialist hospitals in England between 2007 and 2011 with a primary diagnosis of acute myocardial infarction, acute cerebrovascular disease or fracture of neck of femur or a primary procedure of coronary artery bypass graft or repair of abdominal aortic aneurysm. We compared standardised mortality ratios (SMRs) from non-hierarchical models with SMRs from hierarchical models, without and with shrinkage estimates of the predicted probabilities (Model 1 and Model 2). The SMRs from standard logistic and hierarchical models were highly statistically significantly correlated (r > 0.91, p = 0.01). More outliers were recorded in the standard logistic regression than hierarchical modelling only when using shrinkage estimates (Model 2): 21 hospitals (out of a cumulative number of 565 pairs of hospitals under study) changed from a low outlier and 8 hospitals changed from a high outlier based on the logistic regression to a not-an-outlier based on shrinkage estimates. Both standard logistic and hierarchical modelling have identified nearly the same hospitals as mortality outliers. The choice of methodological approach should, however, also consider whether the modelling aim is judgment or improvement, as shrinkage may be more appropriate for the former than the latter.

Keywords

Standardised mortality ratios Logistic regression Hierarchical logistic regression 

Notes

Acknowledgments

The Dr Foster Unit is largely funded by a research grant from Dr Foster Intelligence, an independent healthcare information company and joint venture with the UK Department of Health. The Dr Foster Unit at Imperial is affiliated with the Centre for Patient Safety and Service Quality at Imperial College Healthcare NHS Trust and funded by the National Institute of Health Research. We are grateful for support from the NIHR Biomedical Research Centre funding scheme.

Conflict of interest

The authors declare that they have no conflict of interest.

Ethics

We have permission from the NIGB under Section 251 of the NHS Act 2006 (formerly Section 60 approval from the Patient Information Advisory Group) to hold confidential data and analyse them for research purposes. We have approval to use them for research and measuring quality of delivery of healthcare, from the South East Ethics Research Committee.

References

  1. 1.
    Jarman, B., Gault, S., Alves, B., et al., Explaining differences in English hospital death rates using routinely collected data. BMJ 318:1515–1520, 1999.CrossRefGoogle Scholar
  2. 2.
    Aylin, P., Bottle, A., Jen, M. H., et al., HSMR mortality indicators. http://www1.imperial.ac.uk/resources/3321CA24-A5BC-4A91-9CC9-12C74AA72FDC Accessed 10 May 2013.
  3. 3.
    Bottle, A., and Aylin, P., Intelligent information: A national system for monitoring clinical performance. Health Serv. Res. 43:10–31, 2008.CrossRefGoogle Scholar
  4. 4.
    Alharbi, O., Rabeneck, L., Paszat, L. F., et al., A population-based analysis of outpatient colonoscopy in adults assisted by an anaesthesiologist. Anaesthesiology 111:734–740, 2009.CrossRefGoogle Scholar
  5. 5.
    Krumholz, H. M., Brindis, R. G., Brush, J. E., et al., Standards for statistical models used for public reporting of health outcomes. Circulation 113:456–462, 2006.CrossRefGoogle Scholar
  6. 6.
    Cohen, L. H., Cardiac surgery in the adult. 3rd edition. http://cardiacsurgery.ctsnetbooks.org/ Accessed 20 March 2013.
  7. 7.
    DeLong, E., Hierarchical modeling: Its time has come. Am. Heart J. 145:16–18, 2003.CrossRefGoogle Scholar
  8. 8.
    Diez-Roux, A. V., Multilevel analysis in public health research. Annu. Rev. Public Health 21:171–192, 2000.CrossRefGoogle Scholar
  9. 9.
    Steele, F., Module 5: Introduction to multilevel modelling concepts. http://www.cmm.bristol.ac.uk/learning-training/ Accessed 1 August 2013.
  10. 10.
    Hox, J. J., Applied multilevel analysis. TT Publikaties Amsterdam, 1995. http://www.eric.ed.gov/PDFS/ED384636.pdf Accessed 1 August 2013.
  11. 11.
    Alexandrescu, R., Jen, M. H., Bottle, A., et al., Logistic vs. Hierarchical modeling: An analysis of a statewide inpatient sample. J. Am. Coll. Surg. 213:392–401, 2011.CrossRefGoogle Scholar
  12. 12.
    The Information centre. Hospital episode statistics. 2011. www.hesonline.nhs.uk/ Accessed 11 March 2013.
  13. 13.
    NHS Connecting for Health. Secondary users service. 2011. http://www.connectingforhealth.nhs.uk/systemsandservices/sus Accessed 11 March 2013.
  14. 14.
    HCUP CCS: Healthcare Cost and Utilization Project (HCUP). Clinical Classifications Software (CCS) for ICD-10. Agency for Healthcare and Quality, Rockville, MD 2009. http://www.hcup-us.ahrq.gov/toolssoftware/icd_10/ccs_icd_10.jsp Accessed 11 March 2013.
  15. 15.
    Sundararajan, V., Henderson, T., Perry, C., et al., New ICD-10 version of the Charlson comorbidity index predicted in-hospital mortality. J. Clin. Epidemiol. 57:1288–1294, 2004.CrossRefGoogle Scholar
  16. 16.
  17. 17.
    Wong, S., Flowers, J., and Lyratzopoulos, Y., Hospital admissions for myocardial infarction in the East of England, 1997/98–2007/08. Available from: http://www.erpho.org.uk/Download/Public/18587/1/Briefing%2020%20STEMI%204%20sides%20A4%20FINAL.pdf Accessed 1 August 2013.
  18. 18.
    Seccareccia, F., Perucci, C. A., D’Errigo, P., et al., The Italian CABG Outcome Study: Short-term outcomes in patients with coronary artery bypass graft surgery. Eur. J. Cardiothorac. Surg. 29:56–62, 2006.CrossRefGoogle Scholar
  19. 19.
    Grant, S. W., Grayson, A. D., Purkayastha, D., et al., Logistic risk model for mortality following elective abdominal aortic aneurysm repair. Br. J. Surg. 98:652–658, 2011.CrossRefGoogle Scholar
  20. 20.
    SAS Institute Inc, SAS/STAT® 9.2 user’s guide. The GLIMMIX procedure. SAS Institute Inc., Cary, NC, 2008.Google Scholar
  21. 21.
    Cohen, M. E., Dimick, J. B., Bilimoria, K. Y., et al., Risk adjustment in the American College of Surgeons National Surgical Quality Improvement Program: A comparison of logistic versus hierarchical modeling. J. Am. Coll. Surg. 209:687–693, 2009.CrossRefGoogle Scholar
  22. 22.
    Cohen, M. E., Ko, C. Y., Bilimoria, K. Y., et al., Optimizing ACS NSQIP modeling for evaluation of surgical quality and risk: Patient risk adjustment, procedure mix adjustment, shrinkage adjustment, and surgical focus. J. Am. Coll. Surg. 217:336–346, 2013.CrossRefGoogle Scholar
  23. 23.
    Glance, L. G., Dick, A., Osler, T. M., et al., Impact of changing the statistical methodology on hospital and surgeon ranking: The case of the New York State cardiac surgery report card. Med. Care 44:311–319, 2006.CrossRefGoogle Scholar
  24. 24.
    The Centre for Medicare and Medicaid Services. Statistical issues in assessing hospital performance. http://www.cms.gov/Medicare/Quality-Initiatives-Patient-Assessment-Instruments/HospitalQualityInits/Downloads/Statistical-Issues-in-Assessing-Hospital-Performance.pdf Accessed 1 August 2013.
  25. 25.
    Spiegelhalter, D. J., Funnel plots for comparing institutional performance. Stat. Med. 24:1185–1202, 2005.CrossRefMathSciNetGoogle Scholar
  26. 26.
    Cook, N. R., Use and misuse of the receiver operating characteristic curve in risk prediction. Circulation 115:928–935, 2007.CrossRefGoogle Scholar
  27. 27.
    Snijders, T. A. B., and Bosker, R. J., Multilevel analysis. An introduction to basic and advanced multilevel modeling. SAGE Publications, London, 1999.MATHGoogle Scholar
  28. 28.
    D’Errigo, P., Tosti, M. E., Fusco, D., et al., Use of hierarchical models to evaluate performance of cardiac surgery centres in the Italian CABG outcome study. BMC Med. Res. Methodol. 7:29, 2007.CrossRefGoogle Scholar
  29. 29.
    Hannan, E. L., Wu, C., DeLong, E. R., and Raudenbush, S. W., Predicting risk-adjusted mortality for CABG surgery: Logistic versus hierarchical logistic models. Med. Care 43:726–735, 2005.CrossRefGoogle Scholar
  30. 30.
    Austin, P. C., Alter, D. A., and Tu, J. V., The use of fixed- and random-effects models for classifying hospitals as mortality outliers: A Monte Carlo assessment. Med. Decis. Making 23:526–539, 2003.CrossRefGoogle Scholar
  31. 31.
    Austin, P. C., Tu, J. V., and Alter, D. A., Comparing hierarchical modeling with traditional logistic regression analysis among patients hospitalized with acute myocardial infarction: Should we be analyzing cardiovascular outcomes data differently? Am. Heart J. 145:27–35, 2003.CrossRefGoogle Scholar
  32. 32.
    Clark, D. E., Hannan, E. L., and Wu, C., Predicting risk adjusted mortality for trauma patients: Logistic versus multilevel logistic models. J. Am. Coll. Surg. 211:224–231, 2010.CrossRefGoogle Scholar
  33. 33.
    Tan, A., Freeman, J. L., and Freeman, D. H., Jr., Evaluating health care performance: Strengths and limitations of multilevel analysis. Biom. J. 49:707–718, 2007.CrossRefMathSciNetGoogle Scholar
  34. 34.
    Mohammed, M. A., Manktelow, B. N., and Hofer, T. P., Comparison of four methods for deriving hospital standardised mortality ratios from a single hierarchical logistic regression model. Stat. Methods Med. Res. Nov 6, 2012.Google Scholar
  35. 35.
    Bland, J. M., and Altman, D. G., Measuring agreement in method comparison studies. Stat. Methods Med. Res. 8:135–160, 1999.CrossRefGoogle Scholar

Copyright information

© Springer Science+Business Media New York 2014

Authors and Affiliations

  • Roxana Alexandrescu
    • 1
    • 2
  • Alex Bottle
    • 1
  • Brian Jarman
    • 1
  • Paul Aylin
    • 1
  1. 1.Dr. Foster Unit at Imperial College, Department of Primary Care and Public HealthImperial College LondonLondonUK
  2. 2.Department of Palliative Care, Policy and RehabilitationKing’s College LondonLondonUK

Personalised recommendations