Advertisement

Development and Validation of Risk Prediction Models

  • Damien Drubay
  • Ben Van Calster
  • Stefan MichielsEmail author
Living reference work entry

Abstract

There has been increased interest in the use of clinical risk prediction models for decision-making in medicine for patient care. This has been accelerated through the focus on precision medicine, the revolution in omics data, and increasing use of randomized controlled trial and electronic health record databases. These models are expected to assist diagnostic assessment, prognostication, and therapeutic decision-making. Randomized controlled trial data are highly relevant for modeling treatment benefit and treatment effect heterogeneity. The development and validation of prediction models requires careful methodology and reporting, and an evidence-based approach is needed to bring risk prediction models to clinical practice. This chapter provides an overview of the key steps and considerations to develop and validate risk prediction models. We comment on the role of clinical trials throughout the process. A risk prediction model for the occurrence of breast cancer is used as an example.

Keywords

Prediction models Diagnostic Prognostic Treatment effect Precision medicine Development Predictors Validation Calibration Discrimination Utility 

Notes

Acknowledgments

Data collection and sharing was supported by the National Cancer Institute-funded Breast Cancer Surveillance Consortium (HHSN261201100031C). You can learn more about the BCSC at http://www.bcsc-research.org/. We thank the BCSC participants, investigators, mammography facilities, and radiologists for the data they have provided for this study.

This project has received funding from the European Union’s Horizon 2020 research and innovation programme under grant agreement No 755394. (MyPeBS study), and from the Research Foundation – Flanders (FWO) grant G0B4716N; Internal Funds KU Leuven grant C24/15/037.

References

  1. Barlow WE, White E, Ballard-Barbash R et al (2006) Prospective breast Cancer risk prediction model for women undergoing screening mammography. JNCI J Natl Cancer Inst 98:1204–1214.  https://doi.org/10.1093/jnci/djj331CrossRefGoogle Scholar
  2. Bi WL, Hosny A, Schabath MB et al (2019) Artificial intelligence in cancer imaging: Clinical challenges and applications. CA Cancer J Clin 69:caac.21552.  https://doi.org/10.3322/caac.21552CrossRefGoogle Scholar
  3. Blanche P, Dartigues J-F, Jacqmin-Gadda H (2013) Estimating and comparing time-dependent areas under receiver operating characteristic curves for censored event times with competing risks. Stat Med 32:5381–5397.  https://doi.org/10.1002/sim.5958MathSciNetCrossRefGoogle Scholar
  4. Bossuyt PM, Lijmer JG, Mol BW (2000) Randomised comparisons of medical tests: sometimes invalid, not always efficient. Lancet (London, England) 356:1844–1847.  https://doi.org/10.1016/S0140-6736(00)03246-3CrossRefGoogle Scholar
  5. Bottomley C, Van Belle V, Kirk E et al (2013) Accurate prediction of pregnancy viability by means of a simple scoring system. Hum Reprod 28:68–76.  https://doi.org/10.1093/humrep/des352CrossRefGoogle Scholar
  6. Buyse M, Michiels S, Sargent DJ et al (2011) Integrating biomarkers in clinical trials. Expert Rev Mol Diagn 11:171–182.  https://doi.org/10.1586/erm.10.120CrossRefGoogle Scholar
  7. Cardoso F, van’t Veer LJ, Bogaerts J et al (2016) 70-gene signature as an aid to treatment decisions in early-stage breast Cancer. N Engl J Med 375:717–729.  https://doi.org/10.1056/NEJMoa1602253CrossRefGoogle Scholar
  8. Christodoulou E, Ma J, Collins GS et al (2019) A systematic review shows no performance benefit of machine learning over logistic regression for clinical prediction models. J Clin Epidemiol 110:12–22.  https://doi.org/10.1016/j.jclinepi.2019.02.004CrossRefGoogle Scholar
  9. Collins GS, Moons KGM (2012) Comparing risk prediction models. BMJ 344:e3186.  https://doi.org/10.1136/bmj.e3186CrossRefGoogle Scholar
  10. Collins GS, Reitsma JB, Altman DG, Moons KGM (2015) Transparent reporting of a multivariable prediction model for individual prognosis or diagnosis (TRIPOD): the TRIPOD statement. BMJ 350:g7594CrossRefGoogle Scholar
  11. Damen JAAG, Hooft L, Schuit E et al (2016) Prediction models for cardiovascular disease risk in the general population: systematic review. BMJ 353:i2416.  https://doi.org/10.1136/bmj.i2416CrossRefGoogle Scholar
  12. De Bin R, Sauerbrei W, Boulesteix A-L (2014) Investigating the prediction ability of survival models based on both clinical and omics data: two case studies. Stat Med 33:5310–5329.  https://doi.org/10.1002/sim.6246MathSciNetCrossRefGoogle Scholar
  13. Farooq V, van Klaveren D, Steyerberg EW et al (2013) Anatomical and clinical characteristics to guide decision making between coronary artery bypass surgery and percutaneous coronary intervention for individual patients: development and validation of SYNTAX score II. Lancet 381:639–650.  https://doi.org/10.1016/S0140-6736(13)60108-7CrossRefGoogle Scholar
  14. Hingorani AD, van der WDA, Riley RD et al (2013) Prognosis research strategy (PROGRESS) 4: stratified medicine research. BMJ 346:e5793.  https://doi.org/10.1136/bmj.e5793CrossRefGoogle Scholar
  15. Janssen KJM, Vergouwe Y, Donders ART et al (2009) Dealing with missing predictor values when applying clinical prediction models. Clin Chem 55:994–1001.  https://doi.org/10.1373/clinchem.2008.115345CrossRefGoogle Scholar
  16. Justice AC, Covinsky KE, Berlin JA (1999) Assessing the generalizability of prognostic information. Ann Intern Med 130:515–524CrossRefGoogle Scholar
  17. Kent DM, Steyerberg E, van Klaveren D (2018) Personalized evidence based medicine: predictive approaches to heterogeneous treatment effects. BMJ 363:k4245.  https://doi.org/10.1136/bmj.k4245CrossRefGoogle Scholar
  18. Loi S, Drubay D, Adams S et al (2019) Tumor-infiltrating lymphocytes and prognosis: a pooled individual patient analysis of early-stage triple-negative breast cancers. J Clin Oncol 37:559.  https://doi.org/10.1200/jco.18.01010CrossRefGoogle Scholar
  19. Luijken K, Groenwold RHH, van Calster B et al (2019) Impact of predictor measurement heterogeneity across settings on performance of prediction models: a measurement error perspective. Stat Med 38(18):3444–2459.  https://doi.org/10.1002/sim.8183. Epub 2019 May 31
  20. Michiels S, Kramar A, Koscielny S (2011) Multidimensionality of microarrays: statistical challenges and (im)possible solutions. Mol Oncol 5:190–196.  https://doi.org/10.1016/j.molonc.2011.01.002CrossRefGoogle Scholar
  21. Michiels S, Ternès N, Rotolo F (2016) Statistical controversies in clinical research: prognostic gene signatures are not (yet) useful in clinical practice. Ann Oncol 27:2160–2167.  https://doi.org/10.1093/annonc/mdw307CrossRefGoogle Scholar
  22. Pajouheshnia R, Groenwold RHH, Peelen LM et al (2019) When and how to use data from randomised trials to develop or validate prognostic models. BMJ 365:l2154.  https://doi.org/10.1136/bmj.l2154CrossRefGoogle Scholar
  23. Pauker SG, Kassirer JP (1980) The threshold approach to clinical decision making. N Engl J Med 302:1109–1117.  https://doi.org/10.1056/NEJM198005153022003CrossRefGoogle Scholar
  24. Peduzzi P, Concato J, Kemper E et al (1996) A simulation study of the number of events per variable in logistic regression analysis. J Clin Epidemiol 49:1373–1379CrossRefGoogle Scholar
  25. Pepe MS, Janes H, Longton G et al (2004) Limitations of the odds ratio in gauging the performance of a diagnostic, prognostic, or screening marker. Am J Epidemiol 159:882–890CrossRefGoogle Scholar
  26. Riley RD, Ensor J, Snell KIE et al (2016) External validation of clinical prediction models using big datasets from e-health records or IPD meta-analysis: opportunities and challenges. BMJ 353:i3140.  https://doi.org/10.1136/bmj.i3140CrossRefGoogle Scholar
  27. Riley RD, Snell KI, Ensor J et al (2019) Minimum sample size for developing a multivariable prediction model: PART II – binary and time-to-event outcomes. Stat Med 38:1276–1296.  https://doi.org/10.1002/sim.7992MathSciNetCrossRefGoogle Scholar
  28. Roberts S, Nowak G (2014) Stabilizing the lasso against cross-validation variability. Comput Stat Data Anal 70:198–211.  https://doi.org/10.1016/J.CSDA.2013.09.008MathSciNetCrossRefzbMATHGoogle Scholar
  29. Royston P, Altman DG (2013) External validation of a cox prognostic model: principles and methods. BMC Med Res Methodol 13:33.  https://doi.org/10.1186/1471-2288-13-33CrossRefGoogle Scholar
  30. Royston P, Parmar MKB, Sylvester R (2004) Construction and validation of a prognostic model across several studies, with an application in superficial bladder cancer. Stat Med 23:907–926.  https://doi.org/10.1002/sim.1691CrossRefGoogle Scholar
  31. Royston P, Altman DG, Sauerbrei W (2006) Dichotomizing continuous predictors in multiple regression: a bad idea. Stat Med 25:127–141.  https://doi.org/10.1002/sim.2331MathSciNetCrossRefGoogle Scholar
  32. Schemper M (2003) Predictive accuracy and explained variation. Stat Med 22:2299–2308.  https://doi.org/10.1002/sim.1486CrossRefGoogle Scholar
  33. Siontis GCM, Tzoulaki I, Castaldi PJ, Ioannidis JPA (2015) External validation of new risk prediction models is infrequent and reveals worse prognostic discrimination. J Clin Epidemiol 68:25–34.  https://doi.org/10.1016/j.jclinepi.2014.09.007CrossRefGoogle Scholar
  34. Sterne JAC, White IR, Carlin JB et al (2009) Multiple imputation for missing data in epidemiological and clinical research: potential and pitfalls. BMJ 338:b2393.  https://doi.org/10.1136/bmj.b2393CrossRefGoogle Scholar
  35. Steyerberg EW (2008) Clinical prediction models: a practical approach to development, validation, and updating: Springer Science & Business MediaGoogle Scholar
  36. Steyerberg EW, Eijkemans MJC, Harrell FE, Habbema JDF (2001) Prognostic Modeling with logistic regression analysis. Med Decis Mak 21:45–56.  https://doi.org/10.1177/0272989X0102100106CrossRefGoogle Scholar
  37. Steyerberg EW, Moons KGM, van der Windt DA et al (2013) Prognosis research strategy (PROGRESS) 3: prognostic model research. PLoS Med 10:e1001381.  https://doi.org/10.1371/journal.pmed.1001381CrossRefGoogle Scholar
  38. Steyerberg EW, Uno H, Ioannidis JPA et al (2018) Poor performance of clinical prediction models: the harm of commonly applied methods. J Clin Epidemiol 98:133–143.  https://doi.org/10.1016/j.jclinepi.2017.11.013CrossRefGoogle Scholar
  39. Ternès N, Rotolo F, Michiels S (2017) Robust estimation of the expected survival probabilities from high-dimensional cox models with biomarker-by-treatment interactions in randomized clinical trials. BMC Med Res Methodol 17:83.  https://doi.org/10.1186/s12874-017-0354-0CrossRefzbMATHGoogle Scholar
  40. Tyrer J, Duffy SW, Cuzick J (2004) A breast cancer prediction model incorporating familial and personal risk factors. Stat Med 23:1111–1130.  https://doi.org/10.1002/sim.1668CrossRefGoogle Scholar
  41. Vachon CM, Pankratz VS, Scott CG et al (2015) The contributions of breast density and common genetic variation to breast cancer risk. J Natl Cancer Inst 107.  https://doi.org/10.1093/jnci/dju397
  42. Van Calster B, Nieboer D, Vergouwe Y et al (2016) A calibration hierarchy for risk models was defined: from utopia to empirical data. J Clin Epidemiol 74:167–176.  https://doi.org/10.1016/j.jclinepi.2015.12.005CrossRefGoogle Scholar
  43. Van Calster B, Wynants L, Verbeek JFM et al (2018) Reporting and interpreting decision curve analysis: a guide for investigators. Eur Urol 74:796–804.  https://doi.org/10.1016/j.eururo.2018.08.038CrossRefGoogle Scholar
  44. Van Klaveren D, Steyerberg EW, Serruys PW, Kent DM (2018) The proposed “concordance-statistic for benefit” provided a useful metric when modeling heterogeneous treatment effects. J Clin Epidemiol 94:59–68.  https://doi.org/10.1016/j.jclinepi.2017.10.021CrossRefGoogle Scholar
  45. Van Klaveren D, Balan TA, Steyerberg EW, Kent DM (2019) Models with interactions overestimated heterogeneity of treatment effects and were prone to treatment mistargeting. J Clin Epidemiol.  https://doi.org/10.1016/j.jclinepi.2019.05.029CrossRefGoogle Scholar
  46. Van Smeden M, Moons KG, de Groot JA et al (2018) Sample size for binary logistic prediction models: beyond events per variable criteria. Stat Methods Med Res:96228021878472.  https://doi.org/10.1177/0962280218784726MathSciNetCrossRefGoogle Scholar
  47. Vergouwe Y, Royston P, Moons KGM, Altman DG (2010) Development and validation of a prediction model with missing predictor data: a practical approach. J Clin Epidemiol 63:205–214.  https://doi.org/10.1016/j.jclinepi.2009.03.017CrossRefGoogle Scholar
  48. Vickers AJ, Kattan MW, Sargent DJ (2007) Method for evaluating prediction models that apply the results of randomized trials to individual patients. Trials 8:14.  https://doi.org/10.1186/1745-6215-8-14CrossRefGoogle Scholar
  49. Vickers AJ, Cronin AM, Elkin EB, Gonen M (2008) Extensions to decision curve analysis, a novel method for evaluating diagnostic tests, prediction models and molecular markers. BMC Med Inform Decis Mak 8:53.  https://doi.org/10.1186/1472-6947-8-53CrossRefGoogle Scholar
  50. White IR, Royston P, Wood AM (2011) Multiple imputation using chained equations: issues and guidance for practice. Stat Med 30:377–399.  https://doi.org/10.1002/sim.4067MathSciNetCrossRefGoogle Scholar

Copyright information

© Springer Nature Switzerland AG 2020

Authors and Affiliations

  • Damien Drubay
    • 1
    • 2
  • Ben Van Calster
    • 3
    • 4
  • Stefan Michiels
    • 1
    • 2
    Email author
  1. 1.INSERM U1018, CESP, Paris-Saclay University, UVSQVillejuifFrance
  2. 2.Gustave RoussyService de Biostatistique et d’EpidémiologieVillejuifFrance
  3. 3.Department of Development and RegenerationKU LeuvenLeuvenBelgium
  4. 4.Department of Biomedical Data SciencesLeiden University Medical CenterLeidenThe Netherlands

Section editors and affiliations

  • Stephen George
    • 1
  1. 1.Dept. of Biostatistics and Bioinformatics,Basic Science DivisonDuke University, School of MedicineDurhamUSA

Personalised recommendations