Advertisement

Current Anesthesiology Reports

, Volume 6, Issue 3, pp 267–275 | Cite as

Risk Prediction Models in Perioperative Medicine: Methodological Considerations

  • Gary S. CollinsEmail author
  • Jie Ma
  • Stephen Gerry
  • Eric Ohuma
  • Lang’O Odondi
  • Marialena Trivella
  • Jennifer De Beyer
  • Maria D. L. A. Vazquez-Montes
Research Methods and Statistical Analyses (Y Le Manach, Section Editor)
Part of the following topical collections:
  1. Research Methods and Statistical Analyses

Abstract

Purpose of Review

Risk prediction models hold enormous potential for assessing surgical risk in a standardized, objective manner. Despite the vast number of risk prediction models developed, they have not lived up to their potential. The aim of this paper is to provide an overview of the methodological issues that should be considered when developing and validating a risk prediction model to ensure a useful, accurate model.

Recent Findings

Systematic reviews examining the methodological and reporting quality of these models have found widespread deficiencies that limit their usefulness.

Summary

Risk prediction modelling is a growing field that is gaining huge interest in the era of personalized medicine. Although there are no shortcuts and many challenges are faced when developing and validating accurate, useful prediction models, these challenges are surmountable, if the abundant methodological and practical guidance available is used correctly and efficiently.

Keywords

Risk prediction Discrimination Calibration Multivariable Statistical methods 

Notes

Funding

Jennifer De Beyer has received research funding through a Grant from Cancer Research UK.

Compliance with Ethical Guidelines

Conflict of Interest

Authors declares that they have no conflict of intrerst.

Human and Animal Rights and Informed Consent

This article does not contain any studies with human or animal subjects performed by any of the authors.

References

Papers of particular interest, published recently, have been highlighted as • Of importance •• Of major importance

  1. 1.
    Nashef SA, Roques F, Sharples LD, et al. EuroSCORE II. Eur J Cardiothorac Surg. 2012;41(4):734–44 discussion 744–735.CrossRefPubMedGoogle Scholar
  2. 2.
    Moons KGM, Royston P, Vergouwe Y, Grobbee DE, Altman DG. Prognosis and prognostic research: what, why, and how? BMJ. 2009;338:b375.CrossRefPubMedGoogle Scholar
  3. 3.
    Collins GS, Jibawi A, McCulloch P. Control chart methods for monitoring surgical performance: a case study from gastro-oesophageal surgery. Eur J Surg Oncol. 2011;37:473–80.CrossRefPubMedGoogle Scholar
  4. 4.
    Altman DG, Vergouwe Y, Royston P, Moons KGM. Prognosis and prognostic research: validating a prognostic model. BMJ. 2009;338:b605.CrossRefPubMedGoogle Scholar
  5. 5.
    Royston P, Moons KGM, Altman DG, Vergouwe Y. Prognosis and prognostic research: developing a prognostic model. BMJ. 2009;338:b604.CrossRefPubMedGoogle Scholar
  6. 6.
    Le Manach Y, Collins G, Rodseth R, et al. Preoperative score to predict postoperative mortality (POSPOM): derivation and validation. Anesthesiology. 2016;124(3):570–9.CrossRefPubMedGoogle Scholar
  7. 7.
    Gurm HS, Seth M, Kooiman J, Share D. A novel tool for reliable and accurate prediction of renal complications in patients undergoing percutaneous coronary intervention. J Am Coll Cardiol. 2013;61:2242–8.CrossRefPubMedGoogle Scholar
  8. 8.
    Copeland GP, Jones D, Walters M. POSSUM: a scoring system for surgical audit. Br J Surg. 1991;78:355–60.CrossRefPubMedGoogle Scholar
  9. 9.
    Vuylsteke A, Pagel C, Gerrard C, et al. The papworth bleeding risk score: a stratification scheme for identifying cardiac surgery patients at risk of excessive early postoperative bleeding. Eur J Cardiothorac Surg. 2011;39:924–30.CrossRefPubMedGoogle Scholar
  10. 10.
    Wessler BS, Lai YhL, Kramer W, et al. Clinical prediction models for cardiovascular disease: tufts predictive analytics and comparative effectiveness clinical prediction model database. Circ Cardiovasc Qual Outcomes. 2015;8(4):368–75.CrossRefPubMedPubMedCentralGoogle Scholar
  11. 11.
    Damen JAAG, Hooft L, Schuit E, et al. Prediction models for cardiovascular disease risk in the general population: a systematic review. BMJ. 2016;353:i2416.CrossRefPubMedPubMedCentralGoogle Scholar
  12. 12.
    Kleinrouweler CE, Cheong-See FM, Collins GS, et al. Prognostic models in obstetrics: available, but far from applicable. Am J Obstet Gynecol 2016;214(1):79–90 e36.Google Scholar
  13. 13.
    Shariat SF, Karakiewicz PI, Margulis V, Kattan MW. Inventory of prostate cancer predictive tools. Curr Opin Urol. 2008;18:279–96.CrossRefPubMedGoogle Scholar
  14. 14.
    Ettema RG, Peelen LM, Schuurmans MJ, Nierich AP, Kalkman CJ, Moons KG. Prediction models for prolonged intensive care unit stay after cardiac surgery: systematic review and validation study. Circulation. 2010;122(7):682–9 687 p following p 689.CrossRefPubMedGoogle Scholar
  15. 15.
    Bouwmeester W, Zuithoff NP, Mallett S, et al. Reporting and methods in clinical prediction research: a systematic review. PLoS Med. 2012;9(5):e1001221.CrossRefPubMedCentralGoogle Scholar
  16. 16.
    • Collins GS, de Groot JA, Dutton S, et al. External validation of multivariable prediction models: a systematic review of methodological conduct and reporting. BMC Med Res Methodol. 2014;14:40. Provides an overview of the conduct and reporting of external validation studies. Google Scholar
  17. 17.
    Collins GS, Mallett S, Omar O, Yu LM. Developing risk prediction models for type 2 diabetes: a systematic review of methodology and reporting. BMC Med. 2011;9:103.CrossRefPubMedPubMedCentralGoogle Scholar
  18. 18.
    Collins GS, Omar O, Shanyinde M, Yu LM. A systematic review finds prediction models for chronic kidney were poorly reported and often developed using inappropriate methods. J Clin Epidemiol. 2013;66:268–77.CrossRefPubMedGoogle Scholar
  19. 19.
    Nayak S, Edwards DL, Saleh AA, Greenspan SL. Performance of risk assessment instruments for predicting osteoporotic fracture risk: a systematic review. Osteoporos Int. 2014;25(1):23–49.CrossRefPubMedGoogle Scholar
  20. 20.
    Altman DG. Prognostic models: a methodological framework and review of models for breast cancer. Cancer Investig. 2009;27(3):235–43.CrossRefGoogle Scholar
  21. 21.
    Collins GS, Michaëlsson K. Fracture risk assessment: state of the art, methodologically unsound, or poorly reported? Curr Osteoporos Rep. 2013;10:199–207.CrossRefGoogle Scholar
  22. 22.
    Roques F, Nashef SAM, Michel P, et al. Risk factors and outcome in European cardiac surgery: analysis of the EuroSCORE multinational database of 19030 patients. Eur J Cardiothorac Surg. 1999;15:816–23.CrossRefPubMedGoogle Scholar
  23. 23.
    Nashef SA, Roques F, Michel P, Gauducheau E, Lemeshow S, Salamon R. European system for cardiac operative risk evaluation (EuroSCORE). Eur J Cardiothorac Surg. 1999;16(1):9–13.CrossRefPubMedGoogle Scholar
  24. 24.
    Collins GS, Moons KGM. Comparing risk prediction models. BMJ. 2012;344:e3186.CrossRefPubMedGoogle Scholar
  25. 25.
    Masconi KL, Matsha TE, Erasmus RT, Kengne AP. Recalibration in validation studies of diabetes risk prediction models: a systematic review. Int J Stat Med Res. 2015;4:347–69.CrossRefGoogle Scholar
  26. 26.
    Toll DB, Janssen KJ, Vergouwe Y, Moons KG. Validation, updating and impact of clinical prediction rules: a review. J Clin Epidemiol. 2008;61(11):1085–94.CrossRefPubMedGoogle Scholar
  27. 27.
    • Moons KG, de Groot JA, Bouwmeester W, et al. Critical appraisal and data extraction for systematic reviews of prediction modelling studies: the CHARMS checklist. PLoS Med. 2014;11(10):e1001744. Provides a framework and gives guidance for conducting systematic reviews of prediction model studies. Google Scholar
  28. 28.
    Vickers AJ, Cronin AM. Everything you always wanted to know about evaluating prediction models (but were too afraid to ask). Urology. 2010;76(6):1298–301.CrossRefPubMedPubMedCentralGoogle Scholar
  29. 29.
    • Peat G, Riley RD, Croft P, et al. Improving the transparency of prognosis research: the role of reporting, data sharing, registration, and protocols. PLoS Med. 2014;11:e1001671. An article stressing the importance of planning prediction model studies and if possible to register the study and publish the study protocol. Google Scholar
  30. 30.
    Altman DG, Royston P. What do we mean by validating a prognostic model? Stat Med. 2000;19(4):453–73.CrossRefPubMedGoogle Scholar
  31. 31.
    Moons KG, Kengne AP, Grobbee DE, et al. Risk prediction models: II. External validation, model updating, and impact assessment. Heart. 2012;98:691–8.CrossRefPubMedGoogle Scholar
  32. 32.
    Moons KG, Kengne AP, Woodward M, et al. Risk prediction models: I. Development, internal validation, and assessing the incremental value of a new (bio)marker. Heart. 2012;98:683–90.CrossRefPubMedGoogle Scholar
  33. 33.
    Steyerberg EW, Moons KGM, van der Windt DA, et al. Prognosis research strategy (PROGRESS) 3: prognostic model research. PLoS Med. 2013;10(2):e1001381.CrossRefPubMedPubMedCentralGoogle Scholar
  34. 34.
    Tripepi G, Heinze G, Jager KJ, Stel VS, Dekker FW, Zoccali C. Risk prediction models. Nephrol Dial Transplant. 2013;28(8):1975–80.CrossRefPubMedGoogle Scholar
  35. 35.
    Steyerberg EW, Vergouwe Y. Towards better clinical prediction models: seven steps for development and an ABCD for validation. Eur Heart J. 2014;35:1925–31.CrossRefPubMedPubMedCentralGoogle Scholar
  36. 36.
    Hickey GL, Blackstone EH. External model validation of binary clinical risk prediction models in cardiovascular and thoracic surgery. J Thorac Cardiovasc Surg. 2016. doi: 10.1016/j.jtcvs.2016.04.023.
  37. 37.
    Kattan MW, Hess KR, Amin MB, et al. American Joint Committee on Cancer acceptance criteria for inclusion of risk models for individualized prognosis in the practice of precision medicine. CA Cancer J Clin. 2016. doi: 10.3322/caac.21339.
  38. 38.
    Wynants L, Collins GS, van Calster B. Key steps and common pitfalls in developing and validating risk models: a review. BJOG Int J Obst Gynaecol. 2016 (in press).Google Scholar
  39. 39.
    •• Moons KGM, Altman DG, Reitsma JB, et al. Transparent reporting of a multivariable prediction model for individual prognosis or diagnosis (TRIPOD): explanation and elaboration. Ann Intern Med. 2015;162(1):W1–W73. Provides methodological guidance on information to report when publishing a prediction model study. Google Scholar
  40. 40.
    Harrell FE Jr. Regression modeling strategies: with applications to linear models, logistic and ordinal regression, and survival analysis. 2nd ed. New York: Springer; 2015.CrossRefGoogle Scholar
  41. 41.
    • Steyerberg EW. Clinical prediction models: a practical approach to development, validation, and updating. New York: Springer; 2009. Key text book on various aspects on prediction modelling. Google Scholar
  42. 42.
    Groenwold RH, Moons KG, Pajouheshnia R, et al. Explicit inclusion of treatment in prognostic modelling was recommended in observational and randomised settings. J Clin Epidemiol. 2016. doi: 10.1016/j.jclinepi.2016.03.017.
  43. 43.
    Cheong-See FM, Allotey J, Marlin N, et al. Prediction models in obstetrics: understanding the treatment paradox and potential solutions tothe threat it poses. BJOG Int J Obst Gynaecol. 2016;123:1060–4.CrossRefGoogle Scholar
  44. 44.
    Biesheuvel CJ, Vergouwe Y, Oudega R, Hoes AW, Grobbee DE, Moons KG. Advantages of the nested case-control design in diagnostic research. BMC Med Res Methodol. 2008;8:48.CrossRefPubMedPubMedCentralGoogle Scholar
  45. 45.
    Sanderson J, Thompson SG, White IR, Aspelund T, Pennells L. Derivation and assessment of risk prediction models using case-cohort data. BMC Med Res Methodol. 2013;13:113.CrossRefPubMedPubMedCentralGoogle Scholar
  46. 46.
    Sun GW, Shook TL, Kay GL. Inappropriate use of bivariable analysis to screen risk factors for use in multivariable analysis. J Clin Epidemiol. 1996;49(8):907–16.CrossRefPubMedGoogle Scholar
  47. 47.
    Peduzzi P, Concato J, Feinsten AR, Holford TR. Importance of events per independent variable in proportional hazards regression analysis. 2. Accuracy and precision of regression estimates. J Clin Epidemiol. 1995;48(12):1503–12.CrossRefPubMedGoogle Scholar
  48. 48.
    Peduzzi P, Concato J, Kemper E, Holford TR, Feinstein AR. A simulation study of the number of events per variable in logistic regression analysis. J Clin Epidemiol. 1996;49(12):1373–9.CrossRefPubMedGoogle Scholar
  49. 49.
    Ogundimu EO, Altman D, G., Collins GS. Simulation study finds adequate sample size for developing prediction models is not simply related to events per variable. J Clin Epidemiol. 2016 (in press).Google Scholar
  50. 50.
    Courvoisier DS, Combescure C, Agoritsas T, Gayet-Ageron A, Perneger TV. Performance of logistic regression modeling: beyond the number of events per variable, the role of data structure. J Clin Epidemiol. 2011;64:993–1000.CrossRefPubMedGoogle Scholar
  51. 51.
    Collins GS, Ogundimu EO, Altman DG. Sample size considerations for the external validation of a multivariable prognostic model: a resampling study. Stat Med. 2016;35:214–26.CrossRefPubMedGoogle Scholar
  52. 52.
    Vergouwe Y, Steyerberg EW, Eijkemans MJC, Habbema JDF. Substantial effective sample sizes were required for external validation studies of predictive logistic regression models. J Clin Epidemiol. 2005;58(5):475–83.CrossRefPubMedGoogle Scholar
  53. 53.
    Van Calster B, Nieboer D, Vergouwe Y, De Cock B, Pencina MJ, Steyerberg EW. A calibration hierarchy for risk models was defined: from utopia to empirical data. J Clin Epidemiol. 2016;74:167–76. doi: 10.1016/j.jclinepi.2015.12.005.CrossRefPubMedGoogle Scholar
  54. 54.
    Masconi KL, Matsha TE, Echouffo-Tcheugui JB, Erasmus RT, Kengne AP. Reporting and handling of missing data in predictive research for prevalent undiagnosed type 2 diabetes mellitus: a systematic review. EPMA J. 2015;6(1):7.CrossRefPubMedPubMedCentralGoogle Scholar
  55. 55.
    van der Heijden GJMG, Donders ART, Stijnen T, Moons KGM. Imputation of missing values is superior to complete case analysis and the missing-indicator method in multivariable diagnostic research: a clinical example. J Clin Epidemiol. 2006;59(10):1102–9.CrossRefPubMedGoogle Scholar
  56. 56.
    Vach W. Some issues in estimating the effect of prognostic factors from incomplete covariate data. Stat Med. 1997;16:57–72.CrossRefPubMedGoogle Scholar
  57. 57.
    Little RJA, Rubin DB. Statistical analysis with missing data, vol. 2nd. Hoboken, NJ: Wiley; 2002.Google Scholar
  58. 58.
    White IR, Royston P, Wood AM. Multiple imputation using chained equations: issues and guidance for practice. Stat Med. 2011;30(4):377–99.CrossRefPubMedGoogle Scholar
  59. 59.
    Moons KG, Donders RA, Stijnen T, Harrell FE Jr. Using the outcome for imputation of missing predictor values was preferred. J Clin Epidemiol. 2006;59(10):1092–101.CrossRefPubMedGoogle Scholar
  60. 60.
    Donders AR, van der Heijden GJ, Stijnen T, Moons KG. Review: a gentle introduction to imputation of missing values. J Clin Epidemiol. 2006;59(10):1087–91.CrossRefPubMedGoogle Scholar
  61. 61.
    Sterne JAC, White IR, Carlin JB, et al. Multiple imputation for missing data in epidemiological and clinical research: potential and pitfalls. BMJ. 2009;338:b2393.CrossRefPubMedPubMedCentralGoogle Scholar
  62. 62.
    Vergouwe Y, Royston P, Moons KGM, Altman DG. Development and validation of a prediction model with missing predictor data: a practical approach. J Clin Epidemiol. 2010;63(2):205–14.CrossRefPubMedGoogle Scholar
  63. 63.
    Marshall A, Altman DG, Holder RL. Comparison of imputation methods for handling missing covariate data when fitting a Cox proportional hazards model: a resampling study. BMC Med Res Methodol. 2010;10:112.CrossRefPubMedPubMedCentralGoogle Scholar
  64. 64.
    Marshall A, Altman DG, Holder RL, Royston P. Combining estimates of interest in prognostic modelling studies after multiple imputation: current practice and guidelines. BMC Med Res Methodol. 2009;9:57.CrossRefPubMedPubMedCentralGoogle Scholar
  65. 65.
    Turner EL, Dobson JE, Pocock SJ. Categorisation of continuous risk factors in epidemiological publications: a survey of current practice. Epidemiol Perspect Innov. 2010;7:9.CrossRefPubMedPubMedCentralGoogle Scholar
  66. 66.
    Altman DG. Problems in dichotomizing continuous variables. Am J Epidemiol. 1994;139:442.PubMedGoogle Scholar
  67. 67.
    • Collins GS, Ogundimu EO, Cook JA, Le Manach Y, Altman DG. Quantifying the impact of different approaches for handling continuous predictors on the performance of a prognostic model. Stat Med. 2016. An article illustrating the loss of predictive accuracy when continuous measurements are categorised. Google Scholar
  68. 68.
    Royston P, Altman DG, Sauerbrei W. Dichotomizing continuous predictors in multiple regression: a bad idea. Stat Med. 2006;25:127–41.CrossRefPubMedGoogle Scholar
  69. 69.
    van Walraven C, Hart RG. Leave ‘em alone: why continuous variables should be analyzed as such. Neuroepidemiology. 2008;30:138–9.CrossRefPubMedGoogle Scholar
  70. 70.
    Vickers AJ, Lilja H. Cutpoints in clinical chemistry: time for fundamental reassessment. Clin Chem. 2009;55:15–7.CrossRefPubMedGoogle Scholar
  71. 71.
    Bennette C, Vickers A. Against quantiles: categorization of continuous variables in epidemiologic research, and its discontents. BMC Med Res Methodol. 2012;12:21.CrossRefPubMedPubMedCentralGoogle Scholar
  72. 72.
    Dawson NV, Weiss R. Dichotomizing continuous variables in statistical analysis: a practice to avoid. Med Decis Mak. 2012;32:225–6.CrossRefGoogle Scholar
  73. 73.
    Royston P, Sauerbrei W. Multivariable model building: a pragmatic approach to regression anaylsis based on fractional polynomials for modelling continuous variables. Chichester: Wiley; 2008.CrossRefGoogle Scholar
  74. 74.
    Royston P, Altman DG. Regression using fractional polynomials of continuous covariates: parsimonious parametric modelling. Appl Stat. 1994;43(3):429–67.CrossRefGoogle Scholar
  75. 75.
    Sauerbrei W, Royston P, Binder H. Selection of important variables and determination of functional form for continuous predictors in multivariable model building. Stat Med. 2007;26(30):5512–28.CrossRefPubMedGoogle Scholar
  76. 76.
    Sullivan LM, Massaro JM, D’Agostino RB Sr. Presentation of multivariate data for clinical use: the Framingham study risk score functions. Stat Med. 2004;23:1631–60.CrossRefPubMedGoogle Scholar
  77. 77.
    Seel RT, Steyerberg EW, Malec JF, Sherer M, Macciocchi SN. Developing and evaluating prediction models in rehabilitation populations. Arch Phys Med Rehabil. 2012;93(Suppl 2):S138–53.CrossRefPubMedGoogle Scholar
  78. 78.
    Steyerberg EW, Eijkemans MJ, Habbema JD. Stepwise selection in small data sets: a simulation study of bias in logistic regression analysis. J Clin Epidemiol. 1999;52(10):935–42.CrossRefPubMedGoogle Scholar
  79. 79.
    van Buuren S, Oudshoorn CGM: Multivariate imputation by chained equations: MICE V1.0 User’s Manual, vol. PG/VGZ/00.038. Leiden: TNO Preventie en Gezonheid; 2000.Google Scholar
  80. 80.
    Bouwmeester W, Twisk JW, Kappen TH, van Klei WA, Moons KG, Vergouwe Y. Prediction models for clustered data: comparison of a random intercept and standard regression model. BMC Med Res Methodol. 2013;13:19.CrossRefPubMedPubMedCentralGoogle Scholar
  81. 81.
    Steyerberg EW, Harrell FE Jr, Borsboom GJJM, Eijkemans MJC, Vergouwe Y, Habbema JDF. Internal validation of predictive models: efficiency of some procedures for logistic regression analysis. J Clin Epidemiol. 2001;54:774–81.CrossRefPubMedGoogle Scholar
  82. 82.
    Harrell FE Jr, Lee KL, Mark DB. Multivariable prognostic models: issues in developing models, evaluating assumptions and adequacy, and measuring and reducing errors. Stat Med. 1996;15:361–87.CrossRefPubMedGoogle Scholar
  83. 83.
    • Steyerberg EW, Vickers AJ, Cook NR, et al. Assessing the performance of prediction models: a framework for traditional and novel measures. Epidemiology 2010;21(1):128–138. Paper describing many of the key performance measures to calculate when validating a prediction model. Google Scholar
  84. 84.
    •• Collins GS, Reitsma JB, Altman DG, Moons KGM. Transparent reporting of a multivariable prediction model for individual prognosis or diagnosis (TRIPOD): the TRIPOD statement. Ann Intern Med. 2015;162(1):55–63. Key paper on issues to report when publishing a study developing or validating a prediction model. Google Scholar
  85. 85.
    Cook NR. Statistical evaluation of prognostic versus diagnostic models: beyond the ROC curve. Clin Chem. 2008;54(1):17–23.CrossRefPubMedGoogle Scholar
  86. 86.
    Pencina MJ, D’Agostino RB, Song L. Quantifying discrimination of Framingham risk functions with different survival C statistics. Stat Med. 2012;31:1543–53.CrossRefPubMedGoogle Scholar
  87. 87.
    Austin PC, Steyerberg EW. Graphical assessment of internal and external calibration of logistic regression models by using loess smoothers. Stat Med. 2014;33(3):517–35.CrossRefPubMedGoogle Scholar
  88. 88.
    Miller ME, Langefeld CD, Tierney WM, Hui SL, McDonald CJ. Validation of probabilistic predictions. Med Decis Mak. 1993;13(1):49–58.CrossRefGoogle Scholar
  89. 89.
    Cox DR. Two further applications of a model for binary regression. Biometrika. 1958;45:562–5.CrossRefGoogle Scholar
  90. 90.
    Vickers AJ, Van Calster B, Steyerberg EW. Net benefit approaches to the evaluation of prediction models, molecular markers, and diagnostic tests. BMJ. 2016;352:i6.CrossRefPubMedPubMedCentralGoogle Scholar
  91. 91.
    Baker SG, Cook NR, Vickers A, Kramer BS. Using relative utility curves to evaluate risk prediction. J R Stat Soc A. 2009;172:729–48.CrossRefGoogle Scholar
  92. 92.
    Kanis JA, Oden A, Johnell O, et al. The use of clinical risk factors enhances the performance of BMD in the prediction of hip and osteoporotic fractures in men and women. Osteoporos Int. 2007;18(8):1033–46.CrossRefPubMedGoogle Scholar
  93. 93.
    Kattan MW, Marasco J. What is a real nomogram? Semin Oncol. 2010;37(1):23–6.CrossRefPubMedGoogle Scholar
  94. 94.
    Collins GS. How can I validate a nomogram? Show me the model. Ann Oncol. 2015;26:1034–5.CrossRefPubMedGoogle Scholar
  95. 95.
    Hickey GL, Grant SW, Caiado C, et al. Dynamic prediction modeling approaches for cardiac surgery. Circ Cardiovasc Qual Outcomes. 2013;6:649–58.CrossRefPubMedGoogle Scholar
  96. 96.
    Strobl AN, Vickers AJ, Van Calster B, et al. Improving patient prostate cancer risk assessment: moving from static, globally-applied to dynamic, practice-specific risk calculators. J Biomed Inform. 2015;56:87–93.CrossRefPubMedGoogle Scholar
  97. 97.
    Hickey GL, Grant SW, Murphy GJ, et al. Dynamic trends in cardiac surgery: why the logistic EuroSCORE is no longer suitable for contemporary cardiac surgery and implications for future risk models. Eur J Cardiothorac Surg. 2013;43:1146–52.CrossRefPubMedGoogle Scholar
  98. 98.
    Collins GS, Altman DG. Design flaws in EuroSCORE II. Eur J Cardiothorac Surg. 2012;43:871.CrossRefPubMedGoogle Scholar
  99. 99.
    Collins GS, Altman DG. Calibration of EuroSCORE II. Eur J Cardiothorac Surg. 2013;43(3):654.CrossRefPubMedGoogle Scholar

Copyright information

© Springer Science + Business Media New York 2016

Authors and Affiliations

  • Gary S. Collins
    • 1
    Email author
  • Jie Ma
    • 1
  • Stephen Gerry
    • 1
  • Eric Ohuma
    • 1
  • Lang’O Odondi
    • 1
  • Marialena Trivella
    • 1
  • Jennifer De Beyer
    • 1
  • Maria D. L. A. Vazquez-Montes
    • 1
    • 2
  1. 1.Nuffield Department of Orthopaedics, Centre for Statistics in Medicine, Rheumatology and Musculoskeletal SciencesUniversity of OxfordOxfordUK
  2. 2.Nuffield Department of Primary Care Health SciencesUniversity of OxfordOxfordUK

Personalised recommendations