Journal of Thrombosis and Thrombolysis

, Volume 49, Issue 1, pp 1–9 | Cite as

Machine learning versus traditional risk stratification methods in acute coronary syndrome: a pooled randomized clinical trial analysis

  • William J. Gibson
  • Tarek Nafee
  • Ryan Travis
  • Megan Yee
  • Mathieu Kerneis
  • Magnus Ohman
  • C. Michael GibsonEmail author


Traditional statistical models allow population based inferences and comparisons. Machine learning (ML) explores datasets to develop algorithms that do not assume linear relationships between variables and outcomes and that may account for higher order interactions to make individualized outcome predictions. To evaluate the performance of machine learning models compared to traditional risk stratification methods for the prediction of major adverse cardiovascular events (MACE) and bleeding in patients with acute coronary syndrome (ACS) that are treated with antithrombotic therapy. Data on 24,178 ACS patients were pooled from four randomized controlled trials. The super learner ensemble algorithm selected weights for 23 machine learning models and was compared to traditional models. The efficacy endpoint was a composite of cardiovascular death, myocardial infarction, or stroke. The safety endpoint was a composite of TIMI major and minor bleeding or bleeding requiring medical attention. For the MACE outcome, the super learner model produced a higher c-statistic (0.734) than logistic regression (0.714), the TIMI risk score (0.489), and a new cardiovascular risk score developed in the dataset (0.644). For the bleeding outcome, the super learner demonstrated a similar c-statistic as the logistic regression model (0.670 vs. 0.671). The machine learning risk estimates were highly calibrated with observed efficacy and bleeding outcomes (Hosmer–Lemeshow p value = 0.692 and 0.970, respectively). The super learner algorithm was highly calibrated on both efficacy and safety outcomes and produced the highest c-statistic for prediction of MACE compared to traditional risk stratification methods. This analysis demonstrates a contemporary application of machine learning to guide patient-level antithrombotic therapy treatment decisions.

Clinical Trial Registration ATLAS ACS-2 TIMI 46: Unique Identifier: NCT00402597. ATLAS ACS-2 TIMI 51: Unique Identifier: NCT00809965. GEMINI ACS-1: Unique Identifier: NCT02293395. PIONEER-AF PCI: Unique Identifier: NCT01830543.


Machine learning Personalized medicine Acute coronary syndrome Major adverse cardiovascular events Super learner 



Acute coronary syndrome






Dual antiplatelet therapy


Major adverse cardiovascular events


Myocardial infarction


Percutaneous coronary intervention


Thrombolysis in myocardial infarction


Vitamin K Antagonist


Compliance with ethical standards

Conflict of interest

All authors report research grant support from Janssen Pharmaceuticals Inc.

Supplementary material

11239_2019_1940_MOESM1_ESM.docx (72 kb)
Supplementary material 1 (DOCX 71 kb)


  1. 1.
    Kumar A, Cannon CP (2009) Acute coronary syndromes: diagnosis and management, part I. Mayo Clin Proc 84:917–938CrossRefGoogle Scholar
  2. 2.
    Wachira JK, Stys TP (2013) Cardiovascular disease and bridging the diagnostic gap. SD Med. 66:366–369Google Scholar
  3. 3.
    Writing Group M, Mozaffarian D, Benjamin EJ, Go AS, Arnett DK, Blaha MJ, Cushman M, Das SR, de Ferranti S, Despres JP, Fullerton HJ, Howard VJ, Huffman MD, Isasi CR, Jimenez MC, Judd SE, Kissela BM, Lichtman JH, Lisabeth LD, Liu S, Mackey RH, Magid DJ, McGuire DK, Mohler ER, Moy CS, Muntner P, Mussolino ME, Nasir K, Neumar RW, Nichol G, Palaniappan L, Pandey DK, Reeves MJ, Rodriguez CJ, Rosamond W, Sorlie PD, Stein J, Towfighi A, Turan TN, Virani SS, Woo D, Yeh RW, Turner MB (2016) American Heart Association Statistics C and Stroke Statistics S. Heart Disease and Stroke Statistics-2016 update: a report from the American Heart Association. Circulation 133:e38–e360Google Scholar
  4. 4.
    Levine GN, Bates ER, Bittl JA, Brindis RG, Fihn SD, Fleisher LA, Granger CB, Lange RA, Mack MJ, Mauri L, Mehran R, Mukherjee D, Newby LK, O’Gara PT, Sabatine MS, Smith PK, Smith SC Jr, Halperin JL, Levine GN, Al-Khatib SM, Birtcher KK, Bozkurt B, Brindis RG, Cigarroa JE, Curtis LH, Fleisher LA, Gentile F, Gidding S, Hlatky MA, Ikonomidis JS, Joglar JA, Pressler SJ, Wijeysundera DN (2016) 2016 ACC/AHA guideline focused update on duration of dual antiplatelet therapy in patients with coronary artery disease: a report of the American College of Cardiology/American Heart Association Task Force on Clinical Practice Guidelines. J Thorac Cardiovasc Surg 152:1243–1275CrossRefGoogle Scholar
  5. 5.
    Amsterdam EA, Wenger NK, Brindis RG, Casey DE Jr, Ganiats TG, Holmes DR Jr, Jaffe AS, Jneid H, Kelly RF, Kontos MC, Levine GN, Liebson PR, Mukherjee D, Peterson ED, Sabatine MS, Smalling RW, Zieman SJ (2014) 2014 AHA/ACC guideline for the management of patients with non-ST-elevation acute coronary syndromes: a report of the American College of Cardiology/American Heart Association Task Force on Practice Guidelines. J Am Coll Cardiol 64:e139–e228CrossRefGoogle Scholar
  6. 6.
    O’Gara PT, Kushner FG, Ascheim DD, Casey DE Jr, Chung MK, de Lemos JA, Ettinger SM, Fang JC, Fesmire FM, Franklin BA, Granger CB, Krumholz HM, Linderbaum JA, Morrow DA, Newby LK, Ornato JP, Ou N, Radford MJ, Tamis-Holland JE, Tommaso CL, Tracy CM, Woo YJ, Zhao DX, Anderson JL, Jacobs AK, Halperin JL, Albert NM, Brindis RG, Creager MA, DeMets D, Guyton RA, Hochman JS, Kovacs RJ, Kushner FG, Ohman EM, Stevenson WG, Yancy CW, American College of Cardiology Foundation/American Heart Association Task Force on Practice G (2013) ACCF/AHA guideline for the management of ST-elevation myocardial infarction: a report of the American College of Cardiology Foundation/American Heart Association Task Force on Practice Guidelines. Circulation 2013(127):e362–e425Google Scholar
  7. 7.
    Antman EM, Cohen M, Bernink PJ, McCabe CH, Horacek T, Papuchis G, Mautner B, Corbalan R, Radley D, Braunwald E (2000) The TIMI risk score for unstable angina/non-ST elevation MI: a method for prognostication and therapeutic decision making. JAMA 284:835–842CrossRefGoogle Scholar
  8. 8.
    Eagle KA, Lim MJ, Dabbous OH, Pieper KS, Goldberg RJ, Van de Werf F, Goodman SG, Granger CB, Steg PG, Gore JM, Budaj A, Avezum A, Flather MD, Fox KA, Investigators G (2004) A validated prediction model for all forms of acute coronary syndrome: estimating the risk of 6-month post discharge death in an international registry. JAMA 291:2727–2733CrossRefGoogle Scholar
  9. 9.
    Deo RC, Nallamothu BK (2016) Learning about machine learning: the promise and pitfalls of big data and the electronic health record. Circ Cardiovasc Qual Outcomes 9:618–620CrossRefGoogle Scholar
  10. 10.
    Krittanawong C, Zhang H, Wang Z, Aydar M, Kitai T (2017) Artificial intelligence in precision cardiovascular medicine. J Am Coll Cardiol 69:2657–2664CrossRefGoogle Scholar
  11. 11.
    Mega JL, Braunwald E, Mohanavelu S, Burton P, Poulter R, Misselwitz F, Hricak V, Barnathan ES, Bordes P, Witkowski A, Markov V, Oppenheimer L, Gibson CM, Group AA-Ts (2009) Rivaroxaban versus placebo in patients with acute coronary syndromes (ATLAS ACS-TIMI 46): a randomised, double-blind, phase II trial. Lancet 374:29–38CrossRefGoogle Scholar
  12. 12.
    Mega JL, Braunwald E, Wiviott SD, Bassand JP, Bhatt DL, Bode C, Burton P, Cohen M, Cook-Bruns N, Fox KA, Goto S, Murphy SA, Plotnikov AN, Schneider D, Sun X, Verheugt FW, Gibson CM, Investigators AAT (2012) Rivaroxaban in patients with a recent acute coronary syndrome. N Engl J Med 366:9–19CrossRefGoogle Scholar
  13. 13.
    Gibson CM, Mehran R, Bode C, Halperin J, Verheugt F, Wildgoose P, van Eickels M, Lip GY, Cohen M, Husted S, Peterson E, Fox K (2015) An open-label, randomized, controlled, multicenter study exploring two treatment strategies of rivaroxaban and a dose-adjusted oral vitamin K antagonist treatment strategy in subjects with atrial fibrillation who undergo percutaneous coronary intervention (PIONEER AF-PCI). Am Heart J 169(472–8):e5Google Scholar
  14. 14.
    Ohman EM, Roe MT, Steg PG, James SK, Povsic TJ, White J, Rockhold F, Plotnikov A, Mundl H, Strony J, Sun X, Husted S, Tendera M, Montalescot G, Bahit MC, Ardissino D, Bueno H, Claeys MJ, Nicolau JC, Cornel JH, Goto S, Kiss RG, Guray U, Park DW, Bode C, Welsh RC, Gibson CM (2017) Clinically significant bleeding with low-dose rivaroxaban versus aspirin, in addition to P2Y12 inhibition, in acute coronary syndromes (GEMINI-ACS-1): a double-blind, multicentre, randomised trial. Lancet 389:1799–1808CrossRefGoogle Scholar
  15. 15.
    Blecker S, Katz SD, Horwitz LI, Kuperman G, Park H, Gold A, Sontag D (2016) Comparison of approaches for heart failure case identification from electronic health record data. JAMA Cardiol 1:1014–1020CrossRefGoogle Scholar
  16. 16.
    Frizzell JD, Liang L, Schulte PJ, Yancy CW, Heidenreich PA, Hernandez AF, Bhatt DL, Fonarow GC, Laskey WK (2017) Prediction of 30-day all-cause readmissions in patients hospitalized for heart failure: comparison of machine learning and other statistical approaches. JAMA Cardiol 2:204–209CrossRefGoogle Scholar
  17. 17.
    Weng SF, Reps J, Kai J, Garibaldi JM, Qureshi N (2017) Can machine-learning improve cardiovascular risk prediction using routine clinical data? PLoS ONE 12:e0174944CrossRefGoogle Scholar
  18. 18.
    Mortazavi BJ, Downing NS, Bucholz EM, Dharmarajan K, Manhapra A, Li SX, Negahban SN, Krumholz HM (2016) Analysis of machine learning techniques for heart failure readmissions. Circ Cardiovasc Qual Outcomes 9:629–640CrossRefGoogle Scholar
  19. 19.
    Mansoor H, Elgendy IY, Segal R, Bavry AA, Bian J (2017) Risk prediction model for in-hospital mortality in women with ST-elevation myocardial infarction: a machine learning approach. Heart Lung 46:405–411CrossRefGoogle Scholar
  20. 20.
    VanHouten JP, Starmer JM, Lorenzi NM, Maron DJ, Lasko TA (2014) Machine learning for risk prediction of acute coronary syndrome. AMIA Annu Symp Proc 2014:1940–1949PubMedPubMedCentralGoogle Scholar
  21. 21.
    Hu D, Huang Z, Chan TM, Dong W, Lu X, Duan H (2016) Utilizing Chinese admission records for MACE prediction of acute coronary syndrome. Int J Environ Res Public Health 13:912CrossRefGoogle Scholar
  22. 22.
    Austin PC, Tu JV, Ho JE, Levy D, Lee DS (2013) Using methods from the data-mining and machine-learning literature for disease classification and prediction: a case study examining classification of heart failure subtypes. J Clin Epidemiol 66:398–407CrossRefGoogle Scholar
  23. 23.
    Ambale-Venkatesh B, Yang X, Wu CO, Liu K, Hundley WG, McClelland R, Gomes AS, Folsom AR, Shea S, Guallar E, Bluemke DA, Lima JAC (2017) Cardiovascular event prediction by machine learning: the multi-ethnic study of atherosclerosis. Circ Res 121:1092–1101CrossRefGoogle Scholar
  24. 24.
    Cabitza F, Rasoini R, Gensini GF (2017) Unintended consequences of machine learning in medicine. JAMA 318:517–518CrossRefGoogle Scholar
  25. 25.
    Ng K, Steinhubl SR, deFilippi C, Dey S, Stewart WF (2016) Early detection of heart failure using electronic health records: practical implications for time before diagnosis, data diversity, data quantity, and data density. Circ Cardiovasc Qual Outcomes 9:649–658CrossRefGoogle Scholar
  26. 26.
    Mayer-Schonberger V, Ingelsson E (2017) Big data and medicine—a big deal? J Intern Med 283:418–429CrossRefGoogle Scholar
  27. 27.
    Deo RC (2015) Machine learning in medicine. Circulation 132:1920–1930CrossRefGoogle Scholar
  28. 28.
    Gulshan V, Peng L, Coram M, Stumpe MC, Wu D, Narayanaswamy A, Venugopalan S, Widner K, Madams T, Cuadros J, Kim R, Raman R, Nelson PC, Mega JL, Webster DR (2016) Development and validation of a deep learning algorithm for detection of diabetic retinopathy in retinal fundus photographs. JAMA 316:2402–2410CrossRefGoogle Scholar
  29. 29.
    Esteva A, Kuprel B, Novoa RA, Ko J, Swetter SM, Blau HM, Thrun S (2017) Dermatologist-level classification of skin cancer with deep neural networks. Nature 542:115–118CrossRefGoogle Scholar
  30. 30.
    Pirracchio R, Petersen ML, Carone M, Rigon MR, Chevret S, van der Laan MJ (2015) Mortality prediction in intensive care units with the Super ICU Learner Algorithm (SICULA): a population-based study. Lancet Respir Med 3:42–52CrossRefGoogle Scholar

Copyright information

© Springer Science+Business Media, LLC, part of Springer Nature 2019

Authors and Affiliations

  • William J. Gibson
    • 1
    • 2
  • Tarek Nafee
    • 1
  • Ryan Travis
    • 1
  • Megan Yee
    • 1
  • Mathieu Kerneis
    • 1
  • Magnus Ohman
    • 3
  • C. Michael Gibson
    • 1
    Email author
  1. 1.The Cardiovascular Division, Department of Medicine, Beth Israel Deaconess Medical CenterHarvard Medical SchoolBostonUSA
  2. 2.The Department of MedicineBrigham and Women’s HospitalBostonUSA
  3. 3.The Duke Clinical Research InstituteDuke UniversityDurhamUSA

Personalised recommendations