Data Mining and Knowledge Discovery

, Volume 29, Issue 4, pp 1033–1069 | Cite as

Data mining for censored time-to-event data: a Bayesian network model for predicting cardiovascular risk from electronic health record data

  • Sunayan BandyopadhyayEmail author
  • Julian Wolfson
  • David M. Vock
  • Gabriela Vazquez-Benitez
  • Gediminas Adomavicius
  • Mohamed Elidrisi
  • Paul E. Johnson
  • Patrick J. O’Connor


Models for predicting the risk of cardiovascular (CV) events based on individual patient characteristics are important tools for managing patient care. Most current and commonly used risk prediction models have been built from carefully selected epidemiological cohorts. However, the homogeneity and limited size of such cohorts restrict the predictive power and generalizability of these risk models to other populations. Electronic health data (EHD) from large health care systems provide access to data on large, heterogeneous, and contemporaneous patient populations. The unique features and challenges of EHD, including missing risk factor information, non-linear relationships between risk factors and CV event outcomes, and differing effects from different patient subgroups, demand novel machine learning approaches to risk model development. In this paper, we present a machine learning approach based on Bayesian networks trained on EHD to predict the probability of having a CV event within 5 years. In such data, event status may be unknown for some individuals, as the event time is right-censored due to disenrollment and incomplete follow-up. Since many traditional data mining methods are not well-suited for such data, we describe how to modify both modeling and assessment techniques to account for censored observation times. We show that our approach can lead to better predictive performance than the Cox proportional hazards model (i.e., a regression-based approach commonly used for censored, time-to-event data) or a Bayesian network with ad hoc approaches to right-censoring. Our techniques are motivated by and illustrated on data from a large US Midwestern health care system.


Bayesian networks Electronic health data Survival analysis  Mining censored data Inverse probability of censoring weights Risk prediction  Medical decision support 



This work was partially supported by NHLBI Grant R01HL102144-01 and AHRQ Grant R21HS017622-01.


  1. Agaku IT, King BA, Dube SR (2011) Vital signs: current cigarette smoking among adults aged 18 years-united states, 2005–2010. MMWR Morb Mortal Wkly Rep 60(35):1207Google Scholar
  2. Andreassen S, Riekehr C, Kristensen B, Schonheyder H, Leibovici L (1999) Using probabilistic and decision-theoretic methods in treatment and prognosis modeling. Artif Intell Med 15(2):121–134CrossRefGoogle Scholar
  3. Assmann G, Cullen P, Schulte H (2002) Simple scoring scheme for calculating the risk of acute coronary events based on the 10-year follow-up of the prospective cardiovascular Munster (PROCAM) study. Circulation 105(7):900–900Google Scholar
  4. Bang H, Tsiatis AA (2000) Estimating medical costs with censored data. Biometrika 87(2):329–343zbMATHMathSciNetCrossRefGoogle Scholar
  5. Bang H, Tsiatis AA (2002) Median regression with censored cost data. Biometrics 58(3):643–649zbMATHMathSciNetCrossRefGoogle Scholar
  6. Bilmes JA (1998) A gentle tutorial of the em algorithm and its application to parameter estimation for gaussian mixture and hidden markov models. Technical report. University of California, BerkeleyGoogle Scholar
  7. Blanco R, Inza M, Merino M, Quiroga J, Larranaga P (2005) Feature selection in Bayesian classifiers for the prognosis of survival of cirrhotic patients treated with TIPS. J Biomed Inform 38(5):376–388CrossRefGoogle Scholar
  8. Buckley J, James I (1979) Linear-regression with censored data. Biometrika 66(3):429–436zbMATHCrossRefGoogle Scholar
  9. Clarke PM, Gray AM, Briggs A, Farmer AJ, Fenn P, Stevens RJ, Matthews DR, Stratton IM, Holman RR (2004) A model to estimate the lifetime health outcomes of patients with type 2 diabetes: the United Kingdom Prospective Diabetes Study (UKPDS) outcomes model. Diabetologia 47(10):1747–1759CrossRefGoogle Scholar
  10. Collins GS, Altman DG (2009) An independent external validation and evaluation of QRISK cardiovascular risk prediction: a prospective open cohort study. Br Med J 339:b2584CrossRefGoogle Scholar
  11. Colombet I, Ruelland A, Chatellier G, Gueyffier F, Degoulet P, Jaulent MC (2000) Models to predict cardiovascular risk: comparison of CART, multilayer perceptron and logistic regression. In: Proceedings of the AMIA symposium, American Medical Informatics Association, pp 156–160Google Scholar
  12. Conroy RM, Pyorala K, Fitzgerald AP, Sans S, Menotti A, DeBacker G, DeBacquer D, Ducimetiere P, Jousilahti P, Keil U, Njolstad I, Oganov RG, Thomsen T, Tunstall-Pedoe H, Tverdal A, Wedel H, Whincup P, Wilhelmsen L, Graham IM (2003) Estimation of ten-year risk of fatal cardiovascular disease in Europe: the SCORE project. Eur Heart J 24(11):987–1003CrossRefGoogle Scholar
  13. Cook NR, Ridker PM (2009) The use and magnitude of reclassification measures for individual predictors of global cardiovascular risk. Ann Intern Med 150(11):795–802CrossRefGoogle Scholar
  14. Cooney MT, Dudina AL, Graham IM (2009) Value and limitations of existing scores for the assessment of cardiovascular risk a review for clinicians. J Am Coll Cardiol 54(14):1209–1227CrossRefGoogle Scholar
  15. Cooney MT, Dudina A, D’Agostino R, Graham IM (2010) Cardiovascular risk-estimation systems in primary prevention: do they differ? Do they make a difference? Can we see the future? Circulation 122(3):300–310CrossRefGoogle Scholar
  16. Cox DR (1972) Regression models and life-tables. J R Stat Soc A Met 34(2):187–220zbMATHGoogle Scholar
  17. D’Agostino RB, Vasan RS, Pencina MJ, Wolf PA, Cobain M, Massaro JM, Kannel WB (2008) General cardiovascular risk profile for use in primary care: the Framingham Heart Study. Circulation 118(4):E86–E86CrossRefGoogle Scholar
  18. Dempster AP, Laird NM, Rubin DB (1977) Maximum likelihood from incomplete data via the EM algorithm. J R Stat Soc B Met 39(1):1–38zbMATHMathSciNetGoogle Scholar
  19. Fraley C, Raftery AE, Murphy TB, Scrucca L (2012) MCLUST version 4 for R: normal mixture modeling for model-based clustering, classification, and density estimation. Technical report, University of Washington, Department of StatisticsGoogle Scholar
  20. Go AS, Mozaffarian D, Roger VL, Benjamin EJ, Berry JD, Blaha MJ, Dai S, Ford ES, Fox CS, Franco S, Fullerton HJ, Gillespie C, Hailpern SM, Heit JA, Howard VJ, Huffman MD, Judd SE, Kissela BM, Kittner SJ, Lackland DT, Lichtman JH, Lisabeth LD, Mackey RH, Magid DJ, Marcus GM, Marelli A, Matchar DB, McGuire DK, III MER, Moy CS, Mussolino ME, Neumar RW, Nichol G, Pandey DK, Paynter NP, Reeves MJ, Sorlie PD, Stein J, Towfighi A, Turan TN, Virani SS, Wong ND, Woo D, Turner MB (2014) Heart disease and stroke statistics 2014 update: a report from the American Heart Association. Circulation 129(3):E28–E292Google Scholar
  21. Greenwood M et al (1926) A report on the natural duration of cancer. Rep Public Health Med Subj Minist Health 33:1–26Google Scholar
  22. Harrell FE (2001) Regression modeling strategies. Springer, New YorkzbMATHCrossRefGoogle Scholar
  23. Hippisley-Cox J, Coupland C, Vinogradova Y, Robson J, May M, Brindle P (2007) Derivation and validation of QRISK, a new cardiovascular disease risk score for the United Kingdom: prospective open cohort study. Br Med J 335(7611):136–141CrossRefGoogle Scholar
  24. Hippisley-Cox J, Coupland C, Vinogradova Y, Robson J, Brindle P (2008a) Performance of the qrisk cardiovascular risk prediction algorithm in an independent UK sample of patients from general practice: a validation study. Heart 94(1):34–39Google Scholar
  25. Hippisley-Cox J, Coupland C, Vinogradova Y, Robson J, Minhas R, Sheikh A, Brindle P (2008b) Predicting cardiovascular risk in England and Wales: prospective derivation and validation of QRISK2. Br Med J 336(7659):1475–1489Google Scholar
  26. Hosmer DW, Lemeshow S (1980) Goodness of fit tests for the multiple logistic regression-model. Commun Stat Theory Methods 9(10):1043–1069CrossRefGoogle Scholar
  27. Ishwaran H, Kogalur UB, Blackstone EH, Lauer MS (2008) Random survival forests. Ann Appl Stat 2:841–860zbMATHMathSciNetCrossRefGoogle Scholar
  28. Jackson CH, Thompson SG, Sharples LD (2009) Accounting for uncertainty in health economic decision models by using model averaging. J R Stat Soc Ser A Stat Soc 172:383–404MathSciNetCrossRefGoogle Scholar
  29. Kalbfleisch JD, Prentice RL (2002) The statistical analysis of failure time data. Wiley, Hoboken, NJzbMATHCrossRefGoogle Scholar
  30. Kattan MW (2003) Comparison of Cox regression with other methods for determining prediction models and nomograms. J Urol 170(6):S6–S9CrossRefGoogle Scholar
  31. Kattan MW, Hess KR, Beck JR (1998) Experiments to determine whether recursive partitioning (CART) or an artificial neural network overcomes theoretical limitations of Cox proportional hazards regression. Comput Biomed Res 31(5):363–373CrossRefGoogle Scholar
  32. Kazmierska J, Malicki J (2008) Application of the naive Bayesian classifier to optimize treatment decisions. Radiother Oncol 86(2):211–216CrossRefGoogle Scholar
  33. Lamon-Fava S, Wilson PW, Schaefer EJ (1996) Impact of body mass index on coronary heart disease risk factors in men and women the framingham offspring study. Arterioscler Thromb Vasc Biol 16(12):1509–1515CrossRefGoogle Scholar
  34. Lappenschaar M, Hommersom A, Lucas PJF, Lagro J, Visscher S (2013) Multilevel Bayesian networks for the analysis of hierarchical health care data. Artif Intell Med 57(3):171–183CrossRefGoogle Scholar
  35. Larranaga P, Sierra B, Gallego MJ, Michelena MJ, Picaza JM (1997) Learning Bayesian networks by genetic algorithms: a case study in the prediction of survival in malignant skin melanoma. Artif Intell Med 1211:261–272Google Scholar
  36. Law MR, Wald NJ, Rudnicka A (2003) Quantifying effect of statins on low density lipoprotein cholesterol, ischaemic heart disease, and stroke: systematic review and meta-analysis. BMJ 326(7404):1423CrossRefGoogle Scholar
  37. Lemeshow S, Hosmer DW (1982) A review of goodness of fit statistics for use in the development of logistic-regression models. Am J Epidemiol 115(1):92–106Google Scholar
  38. Lipsky AM, Lewis RJ (2005) Placing the Bayesian network approach to patient diagnosis in perspective. Ann Emerg Med 45(3):291–294CrossRefGoogle Scholar
  39. Lloyd-Jones DM (2010) Cardiovascular risk prediction basic concepts, current status, and future directions. Circulation 121(15):1768–1777CrossRefGoogle Scholar
  40. Lucas PJF, Boot H, Taal BG (1998) Computer-based decision support in the management of primary gastric non-Hodgkin lymphoma. Methods Inf Med 37(3):206–219Google Scholar
  41. Lucas PJF, deBruijn NC, Schurink K, Hoepelman A (2000) A probabilistic and decision-theoretic approach to the management of infectious disease at the ICU. Artif Intell Med 19(3):251–279CrossRefGoogle Scholar
  42. Lucas PJF, van der Gaag L, Abu-Hanna A (2004) Bayesian networks in biomedicine and health-care. Artif Intell Med 30(3):201–214CrossRefGoogle Scholar
  43. Matheny M, McPheeters ML, Glasser A, Mercaldo N, Weaver RB, Jerome RN, Walden R, McKoy JN, Pritchett J, Tsai C (2011) Systematic review of cardiovascular disease risk assessment tools. Technical report, Agency for Healthcare Research and Quality (US)Google Scholar
  44. Murphy KP (1998) Inference and learning in hybrid Bayesian networks. Technical report, University of California, Berkeley, Computer Science DivisionGoogle Scholar
  45. Pencina MJ, D’Agostino RB Sr, D’Agostino RB Jr, Vasan RS (2008) Evaluating the added predictive ability of a new marker: from area under the ROC curve to reclassification and beyond. Stat Med 27(2):157–172MathSciNetCrossRefGoogle Scholar
  46. Pencina MJ, D’Agostino RB Sr, Steyerberg EW (2011) Extensions of net reclassification improvement calculations to measure usefulness of new biomarkers. Stat Med 30(1):11–21MathSciNetCrossRefGoogle Scholar
  47. Pepe MS (2011) Problems with risk reclassification methods for evaluating prediction models. Am J Epidemiol 173(11):1327–35CrossRefGoogle Scholar
  48. Ridker PM, Buring JE, Rifai N, Cook NR (2007) Development and validation of improved algorithms for the assessment of global cardiovascular risk in women: The Reynolds risk score. J Am Med Assoc 297(13):1433–1433CrossRefGoogle Scholar
  49. Ridker PM, Paynter NP, Rifai N, Gaziano M, Cook NR (2008) C-reactive protein and parental history improve global cardiovascular risk prediction: the Reynolds risk score for men. Circulation 118(18):S1145–S1145Google Scholar
  50. Robins JM, Finkelstein DM (2000) Correcting for noncompliance and dependent censoring in an AIDS clinical trial with inverse probability of censoring weighted (IPCW) log-rank tests. Biometrics 56(3):779–788zbMATHCrossRefGoogle Scholar
  51. Rockwood MR, Howlett SE (2011) Blood pressure in relation to age and frailty. Can Geriatr J 14(1):2–7Google Scholar
  52. Rotnitzky AG, Robins JM (2004) Inverse probability weighted estimation in survival analysis. In: Armitage P, Colton T (eds) The encyclopedia of biostatistics, 2nd edn. Wiley, Hoboken, NJGoogle Scholar
  53. Sarkar S, Koehler J (2013) A dynamic risk score to identify increased risk for heart failure decompensation. IEEE Trans Biomed Eng 60(1):147–150CrossRefGoogle Scholar
  54. Sesen MB, Nicholson AE, Banares-Alcantara R, Kadir T, Brady M (2013) Bayesian networks for clinical decision support in lung cancer care. PLOS One 8(12):e82,349CrossRefGoogle Scholar
  55. Sierra B, Larranaga P (1998) Predicting survival in malignant skill melanoma using Bayesian networks automatically induced by genetic algorithms: An empirical comparison between different approaches. Artif Intell Med 14(1–2):215–230CrossRefGoogle Scholar
  56. Smith WP, Doctor J, Meyer J, Kalet IJ, Phillips MH (2009) A decision aid for intensity-modulated radiation-therapy plan selection in prostate cancer based on a prognostic Bayesian network and a Markov model. Artif Intell Med 46(2):119–130CrossRefGoogle Scholar
  57. Song X, Mitnitski A, Cox J, Rockwood K (2004) Comparison of machine learning techniques with classical statistical models in predicting health outcomes. Medinfo 11(1):736–40Google Scholar
  58. Stajduhar I, Dalbelo-Basic B (2010) Learning Bayesian networks from survival data using weighting censored instances. J Biomed Inform 43(4):613–622CrossRefGoogle Scholar
  59. Stajduhar I, Dalbelo-Basic B (2012) Uncensoring censored data for machine learning: a likelihood-based approach. Expert Syst Appl 39(8):7226–7234CrossRefGoogle Scholar
  60. Stajduhar I, Dalbelo-Basic B, Bogunovic N (2009) Impact of censoring on learning Bayesian networks in survival modelling. Artif Intell Med 47(3):199–217CrossRefGoogle Scholar
  61. Stamler J, Stamler R, Neaton JD (1993) Blood pressure, systolic and diastolic, and cardiovascular risks: US population data. Arch Intern Med 153(5):598CrossRefGoogle Scholar
  62. Stuart R, Peter N (2003) Artificial intelligence: a modern approach, vol 2. Prentice Hall, Upper Saddle River, NJGoogle Scholar
  63. Therneau TM, Grambsch PM, Fleming TR (1990) Martingale-based residuals for survival models. Biometrika 77(1):147–160zbMATHMathSciNetCrossRefGoogle Scholar
  64. Tian J, He R, Ram L (2010) Bayesian model averaging using the k-best Bayesian network structures. In: Proceedings of the twenty-sixth conference annual conference on uncertainty in artificial intelligence (UAI-10). AUAI Press, Corvallis, Oregon, pp 589–597Google Scholar
  65. Tsiatis AA (2006) Semiparametric theory and missing data. Springer, New YorkzbMATHGoogle Scholar
  66. Velikova M, van Scheltinga JT, Lucas PJF, Spaanderman M (2014) Exploiting causal functional relationships in bayesian network modelling for personalised healthcare. Int J Approx Reason 55(1):59–73CrossRefGoogle Scholar
  67. Verduijn M, Peek N, Rosseel PMJ, de Jonge E, de Mol BAJM (2007) Prognostic Bayesian networks I: rationale, learning procedure, and clinical use. J Biomed Inform 40(6):609–618CrossRefGoogle Scholar
  68. Vila-Frances J, Sanchis J, Soria-Olivas E, Serrano AJ, Martinez-Sober M, Bonanad C, Ventura S (2013) Expert system for predicting unstable angina based on Bayesian networks. Expert Syst Appl 40(12):5004–5010CrossRefGoogle Scholar
  69. Woodward M, Brindle P, Tunstall-Pedoe H (2007) Adding social deprivation and family history to cardiovascular risk assessment: the ASSIGN score from the Scottish Heart Health Extended Cohort (SHHEC). Heart 93(2):172–176CrossRefGoogle Scholar
  70. Yet B, Bastani K, Raharjo H, Lifvergren S, Marsh W, Bergman B (2013) Decision support system for Warfarin therapy management using Bayesian networks. Decis Support Syst 55(2):488–498CrossRefGoogle Scholar
  71. Zupan B, Demsar J, Kattan MW, Beck JR, Bratko I (1999) Machine learning for survival analysis: a case study on recurrence of prostate cancer. Artif Intell Med 1620:346–355Google Scholar

Copyright information

© The Author(s) 2014

Authors and Affiliations

  • Sunayan Bandyopadhyay
    • 1
    Email author
  • Julian Wolfson
    • 2
  • David M. Vock
    • 2
  • Gabriela Vazquez-Benitez
    • 4
  • Gediminas Adomavicius
    • 3
  • Mohamed Elidrisi
    • 1
  • Paul E. Johnson
    • 3
  • Patrick J. O’Connor
    • 4
  1. 1.Department of Computer ScienceUniversity of MinnesotaMinneapolisUSA
  2. 2.Division of Biostatistics, School of Public HealthUniversity of MinnesotaMinneapolisUSA
  3. 3.Department of Information and Decision Sciences, Carlson School of ManagementUniversity of MinnesotaMinneapolisUSA
  4. 4.HealthPartners Institute for Education and ResearchBloomingtonUSA

Personalised recommendations