Advertisement

Intensive Care Medicine

, Volume 40, Issue 4, pp 513–527 | Cite as

How to derive and validate clinical prediction models for use in intensive care medicine

  • José LabarèreEmail author
  • Renaud Bertrand
  • Michael J. Fine
Statistics for Intensivists

Abstract

Background

Clinical prediction models are formal combinations of historical, physical examination and laboratory or radiographic test data elements designed to accurately estimate the probability that a specific illness is present (diagnostic model), will respond to a form of treatment (therapeutic model) or will have a well-defined outcome (prognostic model) in an individual patient. They are derived and validated using empirical data and used to assist physicians in their clinical decision-making that requires a quantitative assessment of diagnostic, therapeutic or prognostic probabilities at the bedside.

Purpose

To provide intensivists with a comprehensive overview of the empirical development and testing phases that a clinical prediction model must satisfy before its implementation into clinical practice.

Results

The development of a clinical prediction model encompasses three consecutive phases, namely derivation, (external) validation and impact analysis. The derivation phase consists of building a multivariable model, estimating its apparent predictive performance in terms of both calibration and discrimination, and assessing the potential for statistical over-fitting using internal validation techniques (i.e. split-sampling, cross-validation or bootstrapping). External validation consists of testing the predictive performance of a model by assessing its calibration and discrimination in different but plausibly related patients. Impact analysis involves comparative research [i.e. (cluster) randomized trials] to determine whether clinical use of a prediction model affects physician practices, patient outcomes or the cost of healthcare delivery.

Conclusions

This narrative review introduces a checklist of 19 items designed to help intensivists develop and transparently report valid clinical prediction models.

Keywords

Clinical prediction models Clinical decision rule Prognosis Severity of illness index Intensive care 

Notes

Acknowledgments

This research received no specific grant from any funding agency in the public, commercial or not-for-profit sectors. On behalf of all authors, the corresponding author states that there is no conflict of interest. The authors thank Linda Northrup from English Solutions for her assistance in editing the manuscript.

Conflicts of interest

The authors have no conflict of interest to report.

References

  1. 1.
    Laupacis A, Sekar N, Stiell IG (1997) Clinical prediction rules. A review and suggested modifications of methodological standards. JAMA 277:488–494PubMedCrossRefGoogle Scholar
  2. 2.
    Moons KG, Royston P, Vergouwe Y, Grobbee DE, Altman DG (2009) Prognosis and prognostic research: what, why, and how? BMJ 338:375CrossRefGoogle Scholar
  3. 3.
    Steyerberg EW, Moons KG, van der Windt DA et al (2013) Prognosis Research Strategy (PROGRESS) 3: prognostic model research. PLoS Med 10:e1001381PubMedCentralPubMedCrossRefGoogle Scholar
  4. 4.
    Vincent JL, Moreno R (2010) Clinical review: scoring systems in the critically ill. Crit Care 14:207PubMedCentralPubMedCrossRefGoogle Scholar
  5. 5.
    Reilly BM, Evans AT (2006) Translating clinical research into clinical practice: impact of using prediction rules to make decisions. Ann Intern Med 144:201–209PubMedCrossRefGoogle Scholar
  6. 6.
    Moons KG, Altman DG, Vergouwe Y, Royston P (2009) Prognosis and prognostic research: application and impact of prognostic models in clinical practice. BMJ 338:b606PubMedCrossRefGoogle Scholar
  7. 7.
    Steyerberg EW (2009) Clinical prediction models: a practical approach to development, validation, and updating. Springer, New YorkCrossRefGoogle Scholar
  8. 8.
    McGinn TG, Guyatt GH, Wyer PC et al (2000) Users’ guides to the medical literature: XXII: how to use articles about clinical decision rules. Evidence-Based Medicine Working Group. JAMA 284:79–84PubMedCrossRefGoogle Scholar
  9. 9.
    Royston P, Moons KG, Altman DG, Vergouwe Y (2009) Prognosis and prognostic research: developing a prognostic model. BMJ 338:b604PubMedCrossRefGoogle Scholar
  10. 10.
    Altman DG, Vergouwe Y, Royston P, Moons KG (2009) Prognosis and prognostic research: validating a prognostic model. BMJ 338:b605PubMedCrossRefGoogle Scholar
  11. 11.
    Wasson JH, Sox HC, Neff RK, Goldman L (1985) Clinical prediction rules. Applications and methodological standards. N Engl J Med 313:793–799PubMedCrossRefGoogle Scholar
  12. 12.
    Moons KG, Kengne AP, Grobbee DE et al (2012) Risk prediction models: II. External validation, model updating, and impact assessment. Heart 98:691–698PubMedCrossRefGoogle Scholar
  13. 13.
    Moons KG, Kengne AP, Woodward M et al (2012) Risk prediction models: I. Development, internal validation, and assessing the incremental value of a new (bio)marker. Heart 98:683–690PubMedCrossRefGoogle Scholar
  14. 14.
    Stiell IG, Wells GA (1999) Methodologic standards for the development of clinical decision rules in emergency medicine. Ann Emerg Med 33:437–447PubMedCrossRefGoogle Scholar
  15. 15.
    Labarere J, Schuetz P, Renaud B et al (2012) Validation of a clinical prediction model for early admission to the intensive care unit of patients with pneumonia. Acad Emerg Med 19:993–1003PubMedCrossRefGoogle Scholar
  16. 16.
    Renaud B, Labarere J, Coma E et al (2009) Risk stratification of early admission to the intensive care unit of patients with no major criteria of severe community-acquired pneumonia: development of an international prediction rule. Crit Care 13:R54PubMedCentralPubMedCrossRefGoogle Scholar
  17. 17.
    Guyatt GH (2006) Determining prognosis and creating clinical decision rules. In: Haynes RB, Sackett DL, Guyatt GH, Tugwell P (eds) Clinical epidemiology: how to do clinical practice research. Lippincott Williams & Wilkins, New YorkGoogle Scholar
  18. 18.
    Altman DG (2009) Prognostic models: a methodological framework and review of models for breast cancer. Cancer Invest 27:235–243PubMedCrossRefGoogle Scholar
  19. 19.
    Randolph AG, Guyatt GH, Calvin JE, Doig G, Richardson WS (1998) Understanding articles describing clinical prediction tools. Evidence Based Medicine in Critical Care Group. Crit Care Med 26:1603–1612PubMedCrossRefGoogle Scholar
  20. 20.
    Altman DG (1991) Practical statistics for medical research. Chapman & Hall/CRC, LondonGoogle Scholar
  21. 21.
    Aujesky D, Obrosky DS, Stone RA et al (2005) Derivation and validation of a prognostic model for pulmonary embolism. Am J Respir Crit Care Med 172:1041–1046PubMedCentralPubMedCrossRefGoogle Scholar
  22. 22.
    Fine MJ, Auble TE, Yealy DM et al (1997) A prediction rule to identify low-risk patients with community-acquired pneumonia. N Engl J Med 336:243–250PubMedCrossRefGoogle Scholar
  23. 23.
    Royston P, Altman DG, Sauerbrei W (2006) Dichotomizing continuous predictors in multiple regression: a bad idea. Stat Med 25:127–141PubMedCrossRefGoogle Scholar
  24. 24.
    Altman DG, Lausen B, Sauerbrei W, Schumacher M (1994) Dangers of using “optimal” cutpoints in the evaluation of prognostic factors. J Natl Cancer Inst 86:829–835PubMedCrossRefGoogle Scholar
  25. 25.
    Steyerberg EW, Schemper M, Harrell FE (2011) Logistic regression modeling and the number of events per variable: selection bias dominates. J Clin Epidemiol 64:1464–1465 (author reply 1463–1464.)PubMedCrossRefGoogle Scholar
  26. 26.
    Sauerbrei W, Royston P, Binder H (2007) Selection of important variables and determination of functional form for continuous predictors in multivariable model building. Stat Med 26:5512–5528PubMedCrossRefGoogle Scholar
  27. 27.
    Harrell FE Jr (2001) Regression modelling strategies with applications to linear models, logistic regression, and survival analysis. Springer, New YorkGoogle Scholar
  28. 28.
    Vergouwe Y, Royston P, Moons KG, Altman DG (2010) Development and validation of a prediction model with missing predictor data: a practical approach. J Clin Epidemiol 63:205–214PubMedCrossRefGoogle Scholar
  29. 29.
    Altman DG, Bland JM (2007) Missing data. BMJ 334:424PubMedCentralPubMedCrossRefGoogle Scholar
  30. 30.
    Groenwold RH, White IR, Donders AR et al (2012) Missing covariate data in clinical research: when and when not to use the missing-indicator method for analysis. CMAJ 184:1265–1269PubMedCentralPubMedCrossRefGoogle Scholar
  31. 31.
    Groenwold RH, Donders AR, Roes KC, Harrell FE Jr, Moons KG (2012) Dealing with missing outcome data in randomized trials and observational studies. Am J Epidemiol 175:210–217PubMedCrossRefGoogle Scholar
  32. 32.
    Liublinska V, Rubin DB (2012) Re: “dealing with missing outcome data in randomized trials and observational studies”. Am J Epidemiol 176:357–358PubMedCrossRefGoogle Scholar
  33. 33.
    Janssen KJ, Donders AR, Harrell FE Jr et al (2010) Missing covariate data in medical research: to impute is better than to ignore. J Clin Epidemiol 63:721–727PubMedCrossRefGoogle Scholar
  34. 34.
    Concato J, Feinstein AR, Holford TR (1993) The risk of determining risk with multivariable models. Ann Intern Med 118:201–210PubMedCrossRefGoogle Scholar
  35. 35.
    Harrell FE Jr, Lee KL, Mark DB (1996) Multivariable prognostic models: issues in developing models, evaluating assumptions and adequacy, and measuring and reducing errors. Stat Med 15:361–387PubMedCrossRefGoogle Scholar
  36. 36.
    Peduzzi P, Concato J, Kemper E, Holford TR, Feinstein AR (1996) A simulation study of the number of events per variable in logistic regression analysis. J Clin Epidemiol 49:1373–1379PubMedCrossRefGoogle Scholar
  37. 37.
    Courvoisier DS, Combescure C, Agoritsas T, Gayet-Ageron A, Perneger TV (2011) Performance of logistic regression modeling: beyond the number of events per variable, the role of data structure. J Clin Epidemiol 64:993–1000PubMedCrossRefGoogle Scholar
  38. 38.
    Breiman L, Friedman JH, Olshen RA, Stone CJ (1984) Classification and regression trees. Chapman & Hall/CRC, New YorkGoogle Scholar
  39. 39.
    Marshall RJ (2001) The use of classification and regression trees in clinical epidemiology. J Clin Epidemiol 54:603–609PubMedCrossRefGoogle Scholar
  40. 40.
    Sun GW, Shook TL, Kay GL (1996) Inappropriate use of bivariable analysis to screen risk factors for use in multivariable analysis. J Clin Epidemiol 49:907–916PubMedCrossRefGoogle Scholar
  41. 41.
    Hosmer DW, Lemeshow S (2000) Applied logistic regression, 2nd edn. Wiley, New YorkCrossRefGoogle Scholar
  42. 42.
    Sullivan LM, Massaro JM, D’Agostino RB Sr (2004) Presentation of multivariate data for clinical use: the Framingham Study risk score functions. Stat Med 23:1631–1660PubMedCrossRefGoogle Scholar
  43. 43.
    Steyerberg EW, Vickers AJ, Cook NR et al (2010) Assessing the performance of prediction models: a framework for traditional and novel measures. Epidemiology 21:128–138PubMedCentralPubMedCrossRefGoogle Scholar
  44. 44.
    Rufibach K (2010) Use of Brier score to assess binary predictions. J Clin Epidemiol 63:938–939PubMedCrossRefGoogle Scholar
  45. 45.
    Vergouwe Y, Steyerberg EW, Eijkemans MJ, Habbema JD (2005) Substantial effective sample sizes were required for external validation studies of predictive logistic regression models. J Clin Epidemiol 58:475–483PubMedCrossRefGoogle Scholar
  46. 46.
    Justice AC, Covinsky KE, Berlin JA (1999) Assessing the generalizability of prognostic information. Ann Intern Med 130:515–524PubMedCrossRefGoogle Scholar
  47. 47.
    Altman DG, Royston P (2000) What do we mean by validating a prognostic model? Stat Med 19:453–473PubMedCrossRefGoogle Scholar
  48. 48.
    Steyerberg EW, Harrell FE Jr, Borsboom GJ et al (2001) Internal validation of predictive models: efficiency of some procedures for logistic regression analysis. J Clin Epidemiol 54:774–781PubMedCrossRefGoogle Scholar
  49. 49.
    Efron B, Tibshirani RJ (1993) An introduction to the bootstrap. Chapman & Hall/CRC, New YorkCrossRefGoogle Scholar
  50. 50.
    Bleeker SE, Moll HA, Steyerberg EW et al (2003) External validation is necessary in prediction research: a clinical example. J Clin Epidemiol 56:826–832PubMedCrossRefGoogle Scholar
  51. 51.
    Toll DB, Janssen KJ, Vergouwe Y, Moons KG (2008) Validation, updating and impact of clinical prediction rules: a review. J Clin Epidemiol 61:1085–1094PubMedCrossRefGoogle Scholar
  52. 52.
    Peek N, Arts DG, Bosman RJ, van der Voort PH, de Keizer NF (2007) External validation of prognostic models for critically ill patients required substantial sample sizes. J Clin Epidemiol 60:491–501PubMedCrossRefGoogle Scholar
  53. 53.
    Janssen KJ, Moons KG, Kalkman CJ, Grobbee DE, Vergouwe Y (2008) Updating methods improved the performance of a clinical prediction model in new patients. J Clin Epidemiol 61:76–86PubMedCrossRefGoogle Scholar
  54. 54.
    Collins GS, Moons KG (2012) Comparing risk prediction models. BMJ 344:e3186PubMedCrossRefGoogle Scholar
  55. 55.
    Campbell MK, Piaggio G, Elbourne DR, Altman DG (2012) Consort 2010 statement: extension to cluster randomised trials. BMJ 345:e5661PubMedCrossRefGoogle Scholar
  56. 56.
    Donner A, Klar N (2000) Design and analysis of cluster randomized trials in health research. Arnold, LondonGoogle Scholar
  57. 57.
    Hussey MA, Hughes JP (2007) Design and analysis of stepped wedge cluster randomized trials. Contemp Clin Trials 28:182–191PubMedCrossRefGoogle Scholar
  58. 58.
    Ramsay CR, Matowe L, Grilli R, Grimshaw JM, Thomas RE (2003) Interrupted time series designs in health technology assessment: lessons from two systematic reviews of behavior change strategies. Int J Technol Assess Health Care 19:613–623PubMedCrossRefGoogle Scholar
  59. 59.
    von Elm E, Altman DG, Egger M et al (2007) The Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) statement: guidelines for reporting observational studies. Ann Intern Med 147:573–577CrossRefGoogle Scholar
  60. 60.
    Renaud B, Santin A, Coma E et al (2009) Association between timing of intensive care unit admission and outcomes for emergency department patients with community-acquired pneumonia. Crit Care Med 37:2867–2874PubMedCrossRefGoogle Scholar
  61. 61.
    Chalmers JD, Mandal P, Singanayagam A et al (2011) Severity assessment tools to guide ICU admission in community-acquired pneumonia: systematic review and meta-analysis. Intensive Care Med 37:1409–1420PubMedCrossRefGoogle Scholar
  62. 62.
    Marti C, Garin N, Grosgurin O et al (2012) Prediction of severe community-acquired pneumonia: a systematic review and meta-analysis. Crit Care 16:R141PubMedCentralPubMedCrossRefGoogle Scholar
  63. 63.
    Ewig S, Woodhead M, Torres A (2011) Towards a sensible comprehension of severe community-acquired pneumonia. Intensive Care Med 37:214–223PubMedCrossRefGoogle Scholar
  64. 64.
    Yealy DM, Auble TE, Stone RA et al (2005) Effect of increasing the intensity of implementing pneumonia guidelines: a randomized, controlled trial. Ann Intern Med 143:881–894PubMedCrossRefGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg and ESICM 2014

Authors and Affiliations

  • José Labarère
    • 1
    • 2
    • 7
    Email author
  • Renaud Bertrand
    • 3
    • 4
  • Michael J. Fine
    • 5
    • 6
  1. 1.Quality of Care UnitUniversity HospitalGrenobleFrance
  2. 2.TIMC UMR 5525 CNRSUniversité Joseph Fourier–Grenoble 1GrenobleFrance
  3. 3.Emergency DepartmentCochin and Hôtel Dieu Hospitals, Assistance Publique-Hôpitaux de Paris (AP-HP)ParisFrance
  4. 4.Faculté de Médecine Paris DescartesParisFrance
  5. 5.Veterans Affairs Center for Health Equity and Research PromotionVA Pittsburgh Healthcare SystemPittsburghUSA
  6. 6.Division of General Internal MedicineUniversity of Pittsburgh Medical CenterPittsburghUSA
  7. 7.UQEMGrenobleFrance

Personalised recommendations