Development and Validation of Risk Prediction Models

  • Damien Drubay
  • Ben Van Calster
  • Stefan MichielsEmail author
Living reference work entry


There has been increased interest in the use of clinical risk prediction models for decision-making in medicine for patient care. This has been accelerated through the focus on precision medicine, the revolution in omics data, and increasing use of randomized controlled trial and electronic health record databases. These models are expected to assist diagnostic assessment, prognostication, and therapeutic decision-making. Randomized controlled trial data are highly relevant for modeling treatment benefit and treatment effect heterogeneity. The development and validation of prediction models requires careful methodology and reporting, and an evidence-based approach is needed to bring risk prediction models to clinical practice. This chapter provides an overview of the key steps and considerations to develop and validate risk prediction models. We comment on the role of clinical trials throughout the process. A risk prediction model for the occurrence of breast cancer is used as an example.


Prediction models Diagnostic Prognostic Treatment effect Precision medicine Development Predictors Validation Calibration Discrimination Utility 



Data collection and sharing was supported by the National Cancer Institute-funded Breast Cancer Surveillance Consortium (HHSN261201100031C). You can learn more about the BCSC at We thank the BCSC participants, investigators, mammography facilities, and radiologists for the data they have provided for this study.

This project has received funding from the European Union’s Horizon 2020 research and innovation programme under grant agreement No 755394. (MyPeBS study), and from the Research Foundation – Flanders (FWO) grant G0B4716N; Internal Funds KU Leuven grant C24/15/037.


  1. Barlow WE, White E, Ballard-Barbash R et al (2006) Prospective breast Cancer risk prediction model for women undergoing screening mammography. JNCI J Natl Cancer Inst 98:1204–1214. Scholar
  2. Bi WL, Hosny A, Schabath MB et al (2019) Artificial intelligence in cancer imaging: Clinical challenges and applications. CA Cancer J Clin 69:caac.21552. Scholar
  3. Blanche P, Dartigues J-F, Jacqmin-Gadda H (2013) Estimating and comparing time-dependent areas under receiver operating characteristic curves for censored event times with competing risks. Stat Med 32:5381–5397. Scholar
  4. Bossuyt PM, Lijmer JG, Mol BW (2000) Randomised comparisons of medical tests: sometimes invalid, not always efficient. Lancet (London, England) 356:1844–1847. Scholar
  5. Bottomley C, Van Belle V, Kirk E et al (2013) Accurate prediction of pregnancy viability by means of a simple scoring system. Hum Reprod 28:68–76. Scholar
  6. Buyse M, Michiels S, Sargent DJ et al (2011) Integrating biomarkers in clinical trials. Expert Rev Mol Diagn 11:171–182. Scholar
  7. Cardoso F, van’t Veer LJ, Bogaerts J et al (2016) 70-gene signature as an aid to treatment decisions in early-stage breast Cancer. N Engl J Med 375:717–729. Scholar
  8. Christodoulou E, Ma J, Collins GS et al (2019) A systematic review shows no performance benefit of machine learning over logistic regression for clinical prediction models. J Clin Epidemiol 110:12–22. Scholar
  9. Collins GS, Moons KGM (2012) Comparing risk prediction models. BMJ 344:e3186. Scholar
  10. Collins GS, Reitsma JB, Altman DG, Moons KGM (2015) Transparent reporting of a multivariable prediction model for individual prognosis or diagnosis (TRIPOD): the TRIPOD statement. BMJ 350:g7594CrossRefGoogle Scholar
  11. Damen JAAG, Hooft L, Schuit E et al (2016) Prediction models for cardiovascular disease risk in the general population: systematic review. BMJ 353:i2416. Scholar
  12. De Bin R, Sauerbrei W, Boulesteix A-L (2014) Investigating the prediction ability of survival models based on both clinical and omics data: two case studies. Stat Med 33:5310–5329. Scholar
  13. Farooq V, van Klaveren D, Steyerberg EW et al (2013) Anatomical and clinical characteristics to guide decision making between coronary artery bypass surgery and percutaneous coronary intervention for individual patients: development and validation of SYNTAX score II. Lancet 381:639–650. Scholar
  14. Hingorani AD, van der WDA, Riley RD et al (2013) Prognosis research strategy (PROGRESS) 4: stratified medicine research. BMJ 346:e5793. Scholar
  15. Janssen KJM, Vergouwe Y, Donders ART et al (2009) Dealing with missing predictor values when applying clinical prediction models. Clin Chem 55:994–1001. Scholar
  16. Justice AC, Covinsky KE, Berlin JA (1999) Assessing the generalizability of prognostic information. Ann Intern Med 130:515–524CrossRefGoogle Scholar
  17. Kent DM, Steyerberg E, van Klaveren D (2018) Personalized evidence based medicine: predictive approaches to heterogeneous treatment effects. BMJ 363:k4245. Scholar
  18. Loi S, Drubay D, Adams S et al (2019) Tumor-infiltrating lymphocytes and prognosis: a pooled individual patient analysis of early-stage triple-negative breast cancers. J Clin Oncol 37:559. Scholar
  19. Luijken K, Groenwold RHH, van Calster B et al (2019) Impact of predictor measurement heterogeneity across settings on performance of prediction models: a measurement error perspective. Stat Med 38(18):3444–2459. Epub 2019 May 31
  20. Michiels S, Kramar A, Koscielny S (2011) Multidimensionality of microarrays: statistical challenges and (im)possible solutions. Mol Oncol 5:190–196. Scholar
  21. Michiels S, Ternès N, Rotolo F (2016) Statistical controversies in clinical research: prognostic gene signatures are not (yet) useful in clinical practice. Ann Oncol 27:2160–2167. Scholar
  22. Pajouheshnia R, Groenwold RHH, Peelen LM et al (2019) When and how to use data from randomised trials to develop or validate prognostic models. BMJ 365:l2154. Scholar
  23. Pauker SG, Kassirer JP (1980) The threshold approach to clinical decision making. N Engl J Med 302:1109–1117. Scholar
  24. Peduzzi P, Concato J, Kemper E et al (1996) A simulation study of the number of events per variable in logistic regression analysis. J Clin Epidemiol 49:1373–1379CrossRefGoogle Scholar
  25. Pepe MS, Janes H, Longton G et al (2004) Limitations of the odds ratio in gauging the performance of a diagnostic, prognostic, or screening marker. Am J Epidemiol 159:882–890CrossRefGoogle Scholar
  26. Riley RD, Ensor J, Snell KIE et al (2016) External validation of clinical prediction models using big datasets from e-health records or IPD meta-analysis: opportunities and challenges. BMJ 353:i3140. Scholar
  27. Riley RD, Snell KI, Ensor J et al (2019) Minimum sample size for developing a multivariable prediction model: PART II – binary and time-to-event outcomes. Stat Med 38:1276–1296. Scholar
  28. Roberts S, Nowak G (2014) Stabilizing the lasso against cross-validation variability. Comput Stat Data Anal 70:198–211. Scholar
  29. Royston P, Altman DG (2013) External validation of a cox prognostic model: principles and methods. BMC Med Res Methodol 13:33. Scholar
  30. Royston P, Parmar MKB, Sylvester R (2004) Construction and validation of a prognostic model across several studies, with an application in superficial bladder cancer. Stat Med 23:907–926. Scholar
  31. Royston P, Altman DG, Sauerbrei W (2006) Dichotomizing continuous predictors in multiple regression: a bad idea. Stat Med 25:127–141. Scholar
  32. Schemper M (2003) Predictive accuracy and explained variation. Stat Med 22:2299–2308. Scholar
  33. Siontis GCM, Tzoulaki I, Castaldi PJ, Ioannidis JPA (2015) External validation of new risk prediction models is infrequent and reveals worse prognostic discrimination. J Clin Epidemiol 68:25–34. Scholar
  34. Sterne JAC, White IR, Carlin JB et al (2009) Multiple imputation for missing data in epidemiological and clinical research: potential and pitfalls. BMJ 338:b2393. Scholar
  35. Steyerberg EW (2008) Clinical prediction models: a practical approach to development, validation, and updating: Springer Science & Business MediaGoogle Scholar
  36. Steyerberg EW, Eijkemans MJC, Harrell FE, Habbema JDF (2001) Prognostic Modeling with logistic regression analysis. Med Decis Mak 21:45–56. Scholar
  37. Steyerberg EW, Moons KGM, van der Windt DA et al (2013) Prognosis research strategy (PROGRESS) 3: prognostic model research. PLoS Med 10:e1001381. Scholar
  38. Steyerberg EW, Uno H, Ioannidis JPA et al (2018) Poor performance of clinical prediction models: the harm of commonly applied methods. J Clin Epidemiol 98:133–143. Scholar
  39. Ternès N, Rotolo F, Michiels S (2017) Robust estimation of the expected survival probabilities from high-dimensional cox models with biomarker-by-treatment interactions in randomized clinical trials. BMC Med Res Methodol 17:83. Scholar
  40. Tyrer J, Duffy SW, Cuzick J (2004) A breast cancer prediction model incorporating familial and personal risk factors. Stat Med 23:1111–1130. Scholar
  41. Vachon CM, Pankratz VS, Scott CG et al (2015) The contributions of breast density and common genetic variation to breast cancer risk. J Natl Cancer Inst 107.
  42. Van Calster B, Nieboer D, Vergouwe Y et al (2016) A calibration hierarchy for risk models was defined: from utopia to empirical data. J Clin Epidemiol 74:167–176. Scholar
  43. Van Calster B, Wynants L, Verbeek JFM et al (2018) Reporting and interpreting decision curve analysis: a guide for investigators. Eur Urol 74:796–804. Scholar
  44. Van Klaveren D, Steyerberg EW, Serruys PW, Kent DM (2018) The proposed “concordance-statistic for benefit” provided a useful metric when modeling heterogeneous treatment effects. J Clin Epidemiol 94:59–68. Scholar
  45. Van Klaveren D, Balan TA, Steyerberg EW, Kent DM (2019) Models with interactions overestimated heterogeneity of treatment effects and were prone to treatment mistargeting. J Clin Epidemiol. Scholar
  46. Van Smeden M, Moons KG, de Groot JA et al (2018) Sample size for binary logistic prediction models: beyond events per variable criteria. Stat Methods Med Res:96228021878472. Scholar
  47. Vergouwe Y, Royston P, Moons KGM, Altman DG (2010) Development and validation of a prediction model with missing predictor data: a practical approach. J Clin Epidemiol 63:205–214. Scholar
  48. Vickers AJ, Kattan MW, Sargent DJ (2007) Method for evaluating prediction models that apply the results of randomized trials to individual patients. Trials 8:14. Scholar
  49. Vickers AJ, Cronin AM, Elkin EB, Gonen M (2008) Extensions to decision curve analysis, a novel method for evaluating diagnostic tests, prediction models and molecular markers. BMC Med Inform Decis Mak 8:53. Scholar
  50. White IR, Royston P, Wood AM (2011) Multiple imputation using chained equations: issues and guidance for practice. Stat Med 30:377–399. Scholar

Copyright information

© Springer Nature Switzerland AG 2020

Authors and Affiliations

  • Damien Drubay
    • 1
    • 2
  • Ben Van Calster
    • 3
    • 4
  • Stefan Michiels
    • 1
    • 2
    Email author
  1. 1.INSERM U1018, CESP, Paris-Saclay University, UVSQVillejuifFrance
  2. 2.Gustave RoussyService de Biostatistique et d’EpidémiologieVillejuifFrance
  3. 3.Department of Development and RegenerationKU LeuvenLeuvenBelgium
  4. 4.Department of Biomedical Data SciencesLeiden University Medical CenterLeidenThe Netherlands

Section editors and affiliations

  • Stephen George
    • 1
  1. 1.Dept. of Biostatistics and Bioinformatics,Basic Science DivisonDuke University, School of MedicineDurhamUSA

Personalised recommendations