Skip to main content

Advertisement

SpringerLink
Log in

Methodological Challenges for Risk Prediction in Perinatal Epidemiology

  • Reproductive and Perinatal Epidemiology (R Platt, Section Editor)
  • Published:
Current Epidemiology Reports Aims and scope Submit manuscript

Abstract

Purpose of Review

To discuss the methodological challenges in developing risk prediction models in perinatal epidemiology and barriers to their implementation in clinical practice.

Recent Findings

In perinatal epidemiology, risk prediction models have been created to examine the risk of adverse health outcomes in pregnancy, delivery, and post-partum periods. However, only a limited number of prediction models are being used to guide clinical decisions.

Summary

The accuracy and utility of prediction models for clinical decision making are contingent on the use of robust methods to develop risk prediction models and appropriate metrics to assess their performance and clinical impact. In order to increase the transportability (i.e., generalizability) of prediction models, careful consideration of the patient populations represented in the data used to develop and externally validate prediction models and the mechanism for data collection are needed. The era of big data provides researchers the opportunity to leverage existing databases, such as birth and pregnancy registries, through linkage to electronic health records, disease registries, and census data in order to enrich the breadth of clinical and sociodemographic information available for prediction modeling. However, these data sources introduce new challenges that require thorough assessment to evaluate their impact on the accuracy of resulting prediction models and their transportability to the general population.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1

Similar content being viewed by others

References

Papers of particular interest, published recently, have been highlighted as: • Of importance

  1. Kannel WB, McGee D, Gordon T. A general cardiovascular risk profile: the Framingham study. Am J Cardiol. 1976;38:46–51.

    Article  CAS  PubMed  Google Scholar 

  2. Steyerberg EW, Moons KG, van der Windt DA, et al. Prognosis research strategy (PROGRESS) 3: prognostic model research. PLoS Med. 2013;10:e1001381.

    Article  PubMed  PubMed Central  Google Scholar 

  3. Coppus SF, van der Veen F, Opmeer BC, Mol BW, Bossuyt PM. Evaluating prediction models in reproductive medicine. Hum Reprod. 2009;24:1774–8.

    Article  CAS  PubMed  Google Scholar 

  4. Ukah UV, De Silva DA, Payne B, et al. Prediction of adverse maternal outcomes from pre-eclampsia and other hypertensive disorders of pregnancy: a systematic review. Pregnancy Hypertens. 2018;11:115–23.

    Article  PubMed  Google Scholar 

  5. Bakalis S, Peeva G, Gonzalez R, Poon LC, Nicolaides KH. Prediction of small-for-gestational-age neonates: screening by biophysical and biochemical markers at 30-34 weeks. Ultrasound Obstet Gynecol. 2015;46:446–51.

    Article  CAS  PubMed  Google Scholar 

  6. Grobman WA, Lai Y, Landon MB, Spong CY, Leveno KJ, Rouse DJ, et al. Development of a nomogram for prediction of vaginal birth after cesarean delivery. Obstet Gynecol. 2007;109:806–12.

    Article  PubMed  Google Scholar 

  7. Koopmans CM, van der Tuuk K, Groen H, Doornbos JPR, de Graaf IM, van der Salm PCM, et al. Prediction of postpartum hemorrhage in women with gestational hypertension or mild preeclampsia at term. Acta Obstet Gynecol Scand. 2014;93:399–407.

    Article  PubMed  Google Scholar 

  8. Ge WJ, Mirea L, Yang J, Bassil KL, Lee SK, Shah PS, et al. Prediction of neonatal outcomes in extremely preterm neonates. Pediatrics. 2013;132:e876–85.

    Article  PubMed  Google Scholar 

  9. Garite TJ, Casal D, Garcia-Alonso A, Kreaden U, Jimenez G, Ayala JA, et al. Fetal fibronectin: a new tool for the prediction of successful induction of labor. Am J Obstet Gynecol. 1996;175:1516–21.

    Article  CAS  PubMed  Google Scholar 

  10. Heida KY, Bots ML, de Groot CJ, et al. Cardiovascular risk management after reproductive and pregnancy-related disorders: a Dutch multidisciplinary evidence-based guideline. Eur J Prev Cardiol. 2016;23:1863–79.

    Article  PubMed  Google Scholar 

  11. Maher GM, O’Keeffe GW, Kearney PM, Kenny LC, Dinan TG, Mattsson M, et al. Association of hypertensive disorders of pregnancy with risk of neurodevelopmental disorders in offspring: a systematic review and meta-analysis. JAMA Psychiat. 2018;75:809–19.

    Article  Google Scholar 

  12. Shah ND, Steyerberg EW, Kent DM. Big data and predictive analytics: recalibrating expectations. JAMA. 2018;320:27–8.

    Article  PubMed  Google Scholar 

  13. • Riley RD, Ensor J, Snell KI, et al. External validation of clinical prediction models using big datasets from e-health records or IPD meta-analysis: opportunities and challenges. BMJ. 2016;353:i3140 This article highlights the challenges and opportunities for predictive modeling with big data. The authors discuss opportunities for external validation and for the assessment of the incremental value of new predictors. However, the authors also discuss the methodological challenges relating to the quality of data and inconsistencies in reporting of data with the use of large datasets.

    Article  PubMed  PubMed Central  Google Scholar 

  14. Leushuis E, Tromp M, Ravelli AC, van Huis A, Mol BW, Visser GHA, et al. Indicators for intervention during the expulsive second-stage arrest of labour. BJOG. 2009;116:1773–81.

    Article  CAS  PubMed  Google Scholar 

  15. Harrell FEJ. Regression modeling strategies - with applications to linear models, logistic and ordinal regression, and survival analysis. New York: Springer-Verlag; 2001.

    Google Scholar 

  16. Ambler G, Brady AR, Royston P. Simplifying a prognostic model: a simulation study based on clinical data. Stat Med. 2002;21:3803–22.

    Article  PubMed  Google Scholar 

  17. Harrell FE Jr, Lee KL, Mark DB. Multivariable prognostic models: issues in developing models, evaluating assumptions and adequacy, and measuring and reducing errors. Stat Med. 1996;15:361–87.

    Article  PubMed  Google Scholar 

  18. Steyerberg EW. Clinical prediction models. a practical approach to development, validation, and updating. New York: Springer; 2009.

    Google Scholar 

  19. Trudell AS, Tuuli MG, Colditz GA, Macones GA, Odibo AO. A stillbirth calculator: development and internal validation of a clinical prediction model to quantify stillbirth risk. PLoS One. 2017;12:e0173461.

    Article  PubMed  PubMed Central  Google Scholar 

  20. von Dadelszen P, Payne B, Li J, Ansermino JM, Pipkin FB, Côté AM, et al. Prediction of adverse maternal outcomes in pre-eclampsia: development and validation of the fullPIERS model. Lancet. 2011;377:219–27.

    Article  Google Scholar 

  21. Tosteson AN, Begg CB. A general regression methodology for ROC curve estimation. Med Decis Mak. 1988;8:204–15.

    Article  CAS  Google Scholar 

  22. Royston P, Altman DG. External validation of a Cox prognostic model: principles and methods. BMC Med Res Methodol. 2013;13:33.

    Article  PubMed  PubMed Central  Google Scholar 

  23. Steyerberg EW, Vickers AJ, Cook NR, Gerds T, Gonen M, Obuchowski N, et al. Assessing the performance of prediction models: a framework for traditional and novel measures. Epidemiology. 2010;21:128–38.

    Article  PubMed  PubMed Central  Google Scholar 

  24. Royston P, Sauerbrei W. A new measure of prognostic separation in survival data. Stat Med. 2004;23:723–48.

    Article  PubMed  Google Scholar 

  25. Robins JM, Finkelstein DM. Correcting for noncompliance and dependent censoring in an AIDS Clinical Trial with inverse probability of censoring weighted (IPCW) log-rank tests. Biometrics. 2000;56:779–88.

    Article  CAS  PubMed  Google Scholar 

  26. Pencina MJ, D'Agostino RB. Overall C as a measure of discrimination in survival analysis: model specific population value and confidence interval estimation. Stat Med. 2004;23:2109–23.

    Article  PubMed  Google Scholar 

  27. Gerds TA, Schumacher M. Consistent estimation of the expected Brier score in general survival models with right-censored event times. Biom J. 2006;48:1029–40.

    Article  PubMed  Google Scholar 

  28. Cook NR. Use and misuse of the receiver operating characteristic curve in risk prediction. Circulation. 2007;115:928–35.

    Article  PubMed  Google Scholar 

  29. Janes H, Pepe MS, Gu W. Assessing the value of risk predictions by using risk stratification tables. Ann Intern Med. 2008;149:751–60.

    Article  PubMed  PubMed Central  Google Scholar 

  30. Pencina MJ, D'Agostino RB Sr, D'Agostino RB Jr, Vasan RS. Evaluating the added predictive ability of a new marker: from area under the ROC curve to reclassification and beyond. Stat Med. 2008;27:157–72 discussion 207-12.

    Article  PubMed  Google Scholar 

  31. Payne BA, Groen H, Ukah UV, Ansermino JM, Bhutta Z, Grobman W, et al. Development and internal validation of a multivariable model to predict perinatal death in pregnancy hypertension. Pregnancy Hypertens. 2015;5:315–21.

    Article  PubMed  Google Scholar 

  32. Efron B, Tibshirani R. Improvements on cross-validation: the .632+ bootstrap method. JASA. 1997;92:548–60.

  33. Moons KG, Kengne AP, Grobbee DE, et al. Risk prediction models: II. External validation, model updating, and impact assessment. Heart. 2012;98:691–8.

    Article  PubMed  Google Scholar 

  34. Ukah UV, Payne B, Hutcheon JA, et al. Assessment of the fullPIERS risk prediction model in women with early-onset preeclampsia. Hypertension. 2018;71:659–65.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  35. Pauker SG, Kassirer JP. The threshold approach to clinical decision making. N Engl J Med. 1980;302:1109–17.

    Article  CAS  PubMed  Google Scholar 

  36. Vickers AJ, Elkin EB. Decision curve analysis: a novel method for evaluating prediction models. Med Decis Mak. 2006;26:565–74.

    Article  Google Scholar 

  37. Steyerberg EW, Pencina MJ, Lingsma HF, Kattan MW, Vickers AJ, Van Calster B. Assessing the incremental value of diagnostic and prognostic markers: a review and illustration. Eur J Clin Investig. 2012;42:216–28.

    Article  Google Scholar 

  38. Vickers AJ, Van Calster B, Steyerberg EW. Net benefit approaches to the evaluation of prediction models, molecular markers, and diagnostic tests. BMJ. 2016;352:i6.

    Article  PubMed  PubMed Central  Google Scholar 

  39. Moons KG, Royston P, Vergouwe Y, Grobbee DE, Altman DG. Prognosis and prognostic research: what, why, and how? BMJ. 2009;338:b375.

    Article  PubMed  Google Scholar 

  40. Qiu H, Yu HY, Wang LY, Yao Q, Wu SN, Yin C, et al. Electronic health record driven prediction for gestational diabetes mellitus in early pregnancy. Sci Rep. 2017;7:16417.

    Article  PubMed  PubMed Central  Google Scholar 

  41. Li T, Gao C, Yan C, Osmundson S, Malin BA, Chen Y. Predicting neonatal encephalopathy from maternal data in electronic medical records. AMIA Jt Summits Transl Sci Proc. 2017;2018:359–68.

    Google Scholar 

  42. Kuzniewicz MW, Puopolo KM, Fischer A, Walsh EM, Li S, Newman TB, et al. A quantitative, risk-based approach to the management of neonatal early-onset sepsis. JAMA Pediatr. 2017;171:365–71.

    Article  PubMed  Google Scholar 

  43. • Goyal NK, Hall ES, Greenberg JM, Kelly EA. Risk prediction for adverse pregnancy outcomes in a medicaid population. J Women’s Health (Larchmt). 2015;24:681–8 This article identifies the potential limitations of using electronic health records for prediction modeling including missing data, losses-to-follow-up, and the selective availability of clinical information as a result of patient contact with the healthcare system. The article also highlights the limitations of the most commonly used evaluation metrics (e.g., c-statistic) to assess the performance of prediction models and their inability to accurately assess the clinical utility of models for use in clinical practice.

    Article  Google Scholar 

  44. Wessler BS, Paulus J, Lundquist CM, Ajlan M, Natto Z, Janes WA, Jethmalani N, et al. Tufts PACE Clinical Predictive Model Registry: update 1990 through 2015. Diagn Progn Res. 2017;1:20–7.

  45. Goldstein BA, Navar AM, Pencina MJ, Ioannidis JP. Opportunities and challenges in developing risk prediction models with electronic health records data: a systematic review. J Am Med Inform Assoc. 2017;24:198–208.

    Article  PubMed  Google Scholar 

  46. Dalton JE, Perzynski AT, Zidar DA, Rothberg MB, Coulton CJ, Milinovich AT, et al. Accuracy of cardiovascular risk prediction varies by neighborhood socioeconomic position: a retrospective cohort study. Ann Intern Med. 2017;167:456–64.

    Article  PubMed  Google Scholar 

  47. Galea S, Keyes KM. Population health science and the challenges of prediction. Ann Intern Med. 2017;167:511–2.

    Article  PubMed  PubMed Central  Google Scholar 

  48. White IR, Royston P, Wood AM. Multiple imputation using chained equations: issues and guidance for practice. Stat Med. 2011;30:377–99.

    Article  PubMed  Google Scholar 

  49. Flenady V, Koopmans L, Middleton P, Frøen JF, Smith GC, Gibbons K, et al. Major risk factors for stillbirth in high-income countries: a systematic review and meta-analysis. Lancet. 2011;377:1331–40.

    Article  PubMed  Google Scholar 

  50. • Varner MW, Silver RM, Rowland Hogue CJ, Willinger M, Parker CB, Thorsten VR, et al. Association between stillbirth and illicit drug use and smoking during pregnancy. Obstet Gynecol. 2014;123:113–25 This article discusses the Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis (TRIPOD) statement which underscores the importance of transparency in reporting of prediction models. The article also emphasizes the need for external validation and assessment of the clinical impact of previously developed prediction models in order to facilitate their use in clinical decision making.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  51. Pepe MS, Janes H, Longton G, Leisenring W, Newcomb P. Limitations of the odds ratio in gauging the performance of a diagnostic, prognostic, or screening marker. Am J Epidemiol. 2004;159:882–90.

    Article  PubMed  Google Scholar 

  52. van Smeden M, Moons KG, de Groot JA, et al. Sample size for binary logistic prediction models: beyond events per variable criteria. Stat Methods Med Res. 2018:962280218784726. https://doi.org/10.1177/0962280218784726

  53. Collins GS, Reitsma JB, Altman DG, Moons KG. Transparent reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis (TRIPOD). Ann Intern Med. 2015;162:735–6.

    Article  PubMed  Google Scholar 

  54. Zulman DM, Vijan S, Omenn GS, Hayward RA. The relative merits of population-based and targeted prevention strategies. Milbank Q. 2008;86:557–80.

    Article  PubMed  PubMed Central  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Epidemiology, Biostatistics and Occupational Health, McGill University, Montreal, QC, Canada

    Sonia M. Grandi, Kristian B. Filion & Robert W. Platt

  2. Centre for Clinical Epidemiology, Lady Davis Institute, Jewish General Hospital, Montreal, QC, Canada

    Sonia M. Grandi, Kristian B. Filion & Robert W. Platt

  3. Department of Obstetrics and Gynaecology, Faculty of Medicine, University of British Columbia, Vancouver, BC, Canada

    Jennifer A. Hutcheon

  4. Department of Medicine, McGill University, Montreal, QC, Canada

    Kristian B. Filion

  5. McGill University Health Center Research Institute, Montreal, QC, Canada

    Robert W. Platt

  6. Department of Pediatrics, McGill University, Montreal, QC, Canada

    Robert W. Platt

Authors
  1. Sonia M. Grandi
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Jennifer A. Hutcheon
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Kristian B. Filion
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Robert W. Platt
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Robert W. Platt.

Ethics declarations

Conflict of Interest

Sonia Grandi is supported by a Doctoral award from the Fonds de recherche du Québec – Santé (FRQS) and reports grants from the Canadian Institutes of Health Research (CIHR), outside the submitted work. Kristian Filion holds a Junior 2 award from the FRQS and a William Dawson Scholar award from McGill University; he reports grants from CIHR and the Quebec Foundation for Health Research, outside the submitted work. Robert Platt and Jennifer Hutcheon each declares no potential conflicts of interest.

Human and Animal Rights and Informed Consent

This article does not contain any studies with human or animal subjects performed by any of the authors.

Additional information

This article is part of the Topical Collection on Reproductive and Perinatal Epidemiology

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Grandi, S.M., Hutcheon, J.A., Filion, K.B. et al. Methodological Challenges for Risk Prediction in Perinatal Epidemiology. Curr Epidemiol Rep 5, 399–406 (2018). https://doi.org/10.1007/s40471-018-0173-9

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s40471-018-0173-9

Keywords

Search

Navigation