Skip to main content


Log in

The performance of automated case-mix adjustment regression model building methods in a health outcome prediction setting

  • Published:
Health Care Management Science Aims and scope Submit manuscript


We have previously described a system for monitoring a number of healthcare outcomes using case-mix adjustment models. It is desirable to automate the model fitting process in such a system if monitoring covers a large number of outcome measures or subgroup analyses. Our aim was to compare the performance of three different variable selection strategies: “manual”, “automated” backward elimination and re-categorisation, and including all variables at once, irrespective of their apparent importance, with automated re-categorisation. Logistic regression models for predicting in-hospital mortality and emergency readmission within 28 days were fitted to an administrative database for 78 diagnosis groups and 126 procedures from 1996 to 2006 for National Health Services hospital trusts in England. The performance of models was assessed with Receiver Operating Characteristic (ROC) c statistics, (measuring discrimination) and Brier score (assessing the average of the predictive accuracy). Overall, discrimination was similar for diagnoses and procedures and consistently better for mortality than for emergency readmission. Brier scores were generally low overall (showing higher accuracy) and were lower for procedures than diagnoses, with a few exceptions for emergency readmission within 28 days. Among the three variable selection strategies, the automated procedure had similar performance to the manual method in almost all cases except low-risk groups with few outcome events. For the rapid generation of multiple case-mix models we suggest applying automated modelling to reduce the time required, in particular when examining different outcomes of large numbers of procedures and diseases in routinely collected administrative health data.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Fig. 1

Similar content being viewed by others


  1. Allison P (2001) Logistic regression using the SAS system: theory and application. Cary, NC: SAS Institute and New York: John Wiley

  2. Austin PC, Tu JV (2004) Automated variable selection methods for logistic regression produced unstable models for predicting acute myocardial infarction mortality. J Clin Epidemiol 57:1138–1146

    Article  Google Scholar 

  3. Austin PC, Tu JV (2004) Bootstrap methods for developing predictive models. Am Stat 58:131–137

    Article  Google Scholar 

  4. Aylin P, Bottle A, Majeed A (2007) Use of administrative data or clinical databases as predictors of risk of death in hospital: comparison of models. Br Med J 334:1044–51

    Article  Google Scholar 

  5. Bottle A, Aylin P (2008) Intelligent Information: a national system for monitoring clinical performance. Health Serv Res 43:10–31

    Article  Google Scholar 

  6. Cao R (1999) An overview of bootstrap methods for estimating and predicting in time series. TEST Official J Span Soc Stat Oper Res 8:95–166

    Google Scholar 

  7. Chen M-H, Ibrahim JG, Yiannoutsos C (1999) Prior elicitation, variable selection, and bayesian computation for logistic regression models. J R Stat Soc Ser B 61:223–242

    Article  Google Scholar 

  8. Cook NR (2007) Use and misuse of the receiver operating characteristic curve in risk prediction. Circulation 115:928–935

    Article  Google Scholar 

  9. Dreiseitl S, Ohno-Machado L (2002) Logistic regression and artificial neural network classification models: a methodology review. J Biomed Inform 35:352–359

    Article  Google Scholar 

  10. Efroymson MA (1960) Multiple regression analysis. In: Ralston A, Wilf HS (eds) Mathematical methods for digital computers. Wiley, New York

    Google Scholar 

  11. Hanley JA, McNeil BJ (1982) The meaning and use of the area under the receiver operating characteristics (ROC) curve. Radiology 143:29–36

    Google Scholar 

  12. Harrell FE Jr, Lee KL, Mark DB (1996) Multivariable prognostic models: issues in developing models, evaluating assumptions and adequacy, and measuring and reducing errors. Stat Med 15:361–387

    Article  Google Scholar 

  13. Hosmer DW, Lemeshow S (2000) Applied logistic regression, 2nd edn. Wiley, New York

    Book  Google Scholar 

  14. Ikeda M, Ishigaki T, Yamauchi K (2002) Relationship between Brier score and area under the binormal ROC curve. Comput Meth Programs Biomed 67:187–194

    Article  Google Scholar 

  15. Iezzoni LI (1997) Risk adjustment for measuring healthcare outcomes, 2nd edn. Health Administration, Chicago

    Google Scholar 

  16. Mackillop WJ, Quirt CF (1997) Measuring the accuracy of prognostic judgments in oncology. J Clin Epidemiol 50:21–29

    Article  Google Scholar 

  17. Mitchell TJ, Beauchamp JJ (1988) Bayesian variable selection in linear regression (with discussion). J Am Stat Assoc 83:1023–1036

    Article  Google Scholar 

  18. Moons K, Royston P, Vegouwe Y, Grobbee D, Altman D (2009) Prognosis and prognostic research: what, why, and how? Br Med J 338:1317–1320

    Article  Google Scholar 

  19. Nicholl J (2007) Case-mix adjustment in non-randomised observational evaluations: the constant risk fallacy. J Epidemiol Community Health 61:1010–1013

    Article  Google Scholar 

  20. SAS Institute Inc. SAS version 9.1.

  21. Signorini DF, Andrews PJD, Jones PA, Joanna MW, Miller JD (1999) Predicting survival using simple clinical variables: a case study in traumatic brain injury. J Neurol Neurosurg Psychiatry 66:20–25

    Article  Google Scholar 

  22. The Information centre (2008) Hospital Episode Statistics [Online]. [accessed March 2010]; Available from: URL

  23. Weisburg S (1985) Applied linear regression, 2nd edn. John Wiley and Sons inc. pp. 214–215

  24. Wilkinson L, Dallal GE (1981) Tests of significance in forward selection regression with an F-to enter stopping rule. Technometrics 23:377–380

    Article  Google Scholar 

Download references


MHJ, AB and PA and the Dr Foster Unit at Imperial are principally funded via a research grant by Dr Foster Intelligence, an independent healthcare information company. The Unit is affiliated with the Imperial Centre for Patient Safety and Service Quality at Imperial College Healthcare NHS Trust which is funded by the National Institute of Health Research and the Centre for Infection Prevention and Management funded by the UK Clinical Research Collaboration. The Department of Primary Care & Public Health at Imperial College is grateful for support from the NIHR Biomedical Research Centre scheme, the NIHR Collaboration for Leadership in Applied Health Research & Care (CLAHRC) Scheme.


We have permission from the NIGB under Section 251 of the NHS Act 2006 (formerly Section 60 approval from the Patient Information Advisory Group) to hold confidential data and analyse them for research purposes. Consent was given on behalf of patients since for national data, individual consent is considered unfeasible. We have ethical approval from the South East Research Ethics Committee.

Author information

Authors and Affiliations


Corresponding author

Correspondence to Min-Hua Jen.



Table 5 List of diagnosis groups

Rights and permissions

Reprints and permissions

About this article

Cite this article

Jen, MH., Bottle, A., Kirkwood, G. et al. The performance of automated case-mix adjustment regression model building methods in a health outcome prediction setting. Health Care Manag Sci 14, 267–278 (2011).

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: