Skip to main content
SpringerLink
Log in

Machine learning versus traditional risk stratification methods in acute coronary syndrome: a pooled randomized clinical trial analysis

Journal of Thrombosis and Thrombolysis volume 49, pages 1–9 (2020)Cite this article

Abstract

Traditional statistical models allow population based inferences and comparisons. Machine learning (ML) explores datasets to develop algorithms that do not assume linear relationships between variables and outcomes and that may account for higher order interactions to make individualized outcome predictions. To evaluate the performance of machine learning models compared to traditional risk stratification methods for the prediction of major adverse cardiovascular events (MACE) and bleeding in patients with acute coronary syndrome (ACS) that are treated with antithrombotic therapy. Data on 24,178 ACS patients were pooled from four randomized controlled trials. The super learner ensemble algorithm selected weights for 23 machine learning models and was compared to traditional models. The efficacy endpoint was a composite of cardiovascular death, myocardial infarction, or stroke. The safety endpoint was a composite of TIMI major and minor bleeding or bleeding requiring medical attention. For the MACE outcome, the super learner model produced a higher c-statistic (0.734) than logistic regression (0.714), the TIMI risk score (0.489), and a new cardiovascular risk score developed in the dataset (0.644). For the bleeding outcome, the super learner demonstrated a similar c-statistic as the logistic regression model (0.670 vs. 0.671). The machine learning risk estimates were highly calibrated with observed efficacy and bleeding outcomes (Hosmer–Lemeshow p value = 0.692 and 0.970, respectively). The super learner algorithm was highly calibrated on both efficacy and safety outcomes and produced the highest c-statistic for prediction of MACE compared to traditional risk stratification methods. This analysis demonstrates a contemporary application of machine learning to guide patient-level antithrombotic therapy treatment decisions.

Clinical Trial Registration ATLAS ACS-2 TIMI 46: https://clinicaltrials.gov/ct2/show/NCT00402597. Unique Identifier: NCT00402597. ATLAS ACS-2 TIMI 51: https://clinicaltrials.gov/ct2/show/NCT00809965. Unique Identifier: NCT00809965. GEMINI ACS-1: https://clinicaltrials.gov/ct2/show/NCT02293395. Unique Identifier: NCT02293395. PIONEER-AF PCI: https://clinicaltrials.gov/ct2/show/NCT01830543. Unique Identifier: NCT01830543.

This is a preview of subscription content, access via your institution.

Access options

Buy single article

Instant access to the full article PDF.

USD 39.95

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Fig. 1
Fig. 2
Fig. 3
Fig. 4

Abbreviations

ACS:

Acute coronary syndrome

ASA:

Aspirin

CV:

Cardiovascular

DAPT:

Dual antiplatelet therapy

MACE:

Major adverse cardiovascular events

MI:

Myocardial infarction

PCI:

Percutaneous coronary intervention

TIMI:

Thrombolysis in myocardial infarction

VKA:

Vitamin K Antagonist

References

  1. Kumar A, Cannon CP (2009) Acute coronary syndromes: diagnosis and management, part I. Mayo Clin Proc 84:917–938

    Article  Google Scholar 

  2. Wachira JK, Stys TP (2013) Cardiovascular disease and bridging the diagnostic gap. SD Med. 66:366–369

    Google Scholar 

  3. Writing Group M, Mozaffarian D, Benjamin EJ, Go AS, Arnett DK, Blaha MJ, Cushman M, Das SR, de Ferranti S, Despres JP, Fullerton HJ, Howard VJ, Huffman MD, Isasi CR, Jimenez MC, Judd SE, Kissela BM, Lichtman JH, Lisabeth LD, Liu S, Mackey RH, Magid DJ, McGuire DK, Mohler ER, Moy CS, Muntner P, Mussolino ME, Nasir K, Neumar RW, Nichol G, Palaniappan L, Pandey DK, Reeves MJ, Rodriguez CJ, Rosamond W, Sorlie PD, Stein J, Towfighi A, Turan TN, Virani SS, Woo D, Yeh RW, Turner MB (2016) American Heart Association Statistics C and Stroke Statistics S. Heart Disease and Stroke Statistics-2016 update: a report from the American Heart Association. Circulation 133:e38–e360

    Google Scholar 

  4. Levine GN, Bates ER, Bittl JA, Brindis RG, Fihn SD, Fleisher LA, Granger CB, Lange RA, Mack MJ, Mauri L, Mehran R, Mukherjee D, Newby LK, O’Gara PT, Sabatine MS, Smith PK, Smith SC Jr, Halperin JL, Levine GN, Al-Khatib SM, Birtcher KK, Bozkurt B, Brindis RG, Cigarroa JE, Curtis LH, Fleisher LA, Gentile F, Gidding S, Hlatky MA, Ikonomidis JS, Joglar JA, Pressler SJ, Wijeysundera DN (2016) 2016 ACC/AHA guideline focused update on duration of dual antiplatelet therapy in patients with coronary artery disease: a report of the American College of Cardiology/American Heart Association Task Force on Clinical Practice Guidelines. J Thorac Cardiovasc Surg 152:1243–1275

    Article  Google Scholar 

  5. Amsterdam EA, Wenger NK, Brindis RG, Casey DE Jr, Ganiats TG, Holmes DR Jr, Jaffe AS, Jneid H, Kelly RF, Kontos MC, Levine GN, Liebson PR, Mukherjee D, Peterson ED, Sabatine MS, Smalling RW, Zieman SJ (2014) 2014 AHA/ACC guideline for the management of patients with non-ST-elevation acute coronary syndromes: a report of the American College of Cardiology/American Heart Association Task Force on Practice Guidelines. J Am Coll Cardiol 64:e139–e228

    Article  Google Scholar 

  6. O’Gara PT, Kushner FG, Ascheim DD, Casey DE Jr, Chung MK, de Lemos JA, Ettinger SM, Fang JC, Fesmire FM, Franklin BA, Granger CB, Krumholz HM, Linderbaum JA, Morrow DA, Newby LK, Ornato JP, Ou N, Radford MJ, Tamis-Holland JE, Tommaso CL, Tracy CM, Woo YJ, Zhao DX, Anderson JL, Jacobs AK, Halperin JL, Albert NM, Brindis RG, Creager MA, DeMets D, Guyton RA, Hochman JS, Kovacs RJ, Kushner FG, Ohman EM, Stevenson WG, Yancy CW, American College of Cardiology Foundation/American Heart Association Task Force on Practice G (2013) ACCF/AHA guideline for the management of ST-elevation myocardial infarction: a report of the American College of Cardiology Foundation/American Heart Association Task Force on Practice Guidelines. Circulation 2013(127):e362–e425

    Google Scholar 

  7. Antman EM, Cohen M, Bernink PJ, McCabe CH, Horacek T, Papuchis G, Mautner B, Corbalan R, Radley D, Braunwald E (2000) The TIMI risk score for unstable angina/non-ST elevation MI: a method for prognostication and therapeutic decision making. JAMA 284:835–842

    Article  CAS  Google Scholar 

  8. Eagle KA, Lim MJ, Dabbous OH, Pieper KS, Goldberg RJ, Van de Werf F, Goodman SG, Granger CB, Steg PG, Gore JM, Budaj A, Avezum A, Flather MD, Fox KA, Investigators G (2004) A validated prediction model for all forms of acute coronary syndrome: estimating the risk of 6-month post discharge death in an international registry. JAMA 291:2727–2733

    Article  CAS  Google Scholar 

  9. Deo RC, Nallamothu BK (2016) Learning about machine learning: the promise and pitfalls of big data and the electronic health record. Circ Cardiovasc Qual Outcomes 9:618–620

    Article  Google Scholar 

  10. Krittanawong C, Zhang H, Wang Z, Aydar M, Kitai T (2017) Artificial intelligence in precision cardiovascular medicine. J Am Coll Cardiol 69:2657–2664

    Article  Google Scholar 

  11. Mega JL, Braunwald E, Mohanavelu S, Burton P, Poulter R, Misselwitz F, Hricak V, Barnathan ES, Bordes P, Witkowski A, Markov V, Oppenheimer L, Gibson CM, Group AA-Ts (2009) Rivaroxaban versus placebo in patients with acute coronary syndromes (ATLAS ACS-TIMI 46): a randomised, double-blind, phase II trial. Lancet 374:29–38

    Article  CAS  Google Scholar 

  12. Mega JL, Braunwald E, Wiviott SD, Bassand JP, Bhatt DL, Bode C, Burton P, Cohen M, Cook-Bruns N, Fox KA, Goto S, Murphy SA, Plotnikov AN, Schneider D, Sun X, Verheugt FW, Gibson CM, Investigators AAT (2012) Rivaroxaban in patients with a recent acute coronary syndrome. N Engl J Med 366:9–19

    Article  CAS  Google Scholar 

  13. Gibson CM, Mehran R, Bode C, Halperin J, Verheugt F, Wildgoose P, van Eickels M, Lip GY, Cohen M, Husted S, Peterson E, Fox K (2015) An open-label, randomized, controlled, multicenter study exploring two treatment strategies of rivaroxaban and a dose-adjusted oral vitamin K antagonist treatment strategy in subjects with atrial fibrillation who undergo percutaneous coronary intervention (PIONEER AF-PCI). Am Heart J 169(472–8):e5

    Google Scholar 

  14. Ohman EM, Roe MT, Steg PG, James SK, Povsic TJ, White J, Rockhold F, Plotnikov A, Mundl H, Strony J, Sun X, Husted S, Tendera M, Montalescot G, Bahit MC, Ardissino D, Bueno H, Claeys MJ, Nicolau JC, Cornel JH, Goto S, Kiss RG, Guray U, Park DW, Bode C, Welsh RC, Gibson CM (2017) Clinically significant bleeding with low-dose rivaroxaban versus aspirin, in addition to P2Y12 inhibition, in acute coronary syndromes (GEMINI-ACS-1): a double-blind, multicentre, randomised trial. Lancet 389:1799–1808

    Article  CAS  Google Scholar 

  15. Blecker S, Katz SD, Horwitz LI, Kuperman G, Park H, Gold A, Sontag D (2016) Comparison of approaches for heart failure case identification from electronic health record data. JAMA Cardiol 1:1014–1020

    Article  Google Scholar 

  16. Frizzell JD, Liang L, Schulte PJ, Yancy CW, Heidenreich PA, Hernandez AF, Bhatt DL, Fonarow GC, Laskey WK (2017) Prediction of 30-day all-cause readmissions in patients hospitalized for heart failure: comparison of machine learning and other statistical approaches. JAMA Cardiol 2:204–209

    Article  Google Scholar 

  17. Weng SF, Reps J, Kai J, Garibaldi JM, Qureshi N (2017) Can machine-learning improve cardiovascular risk prediction using routine clinical data? PLoS ONE 12:e0174944

    Article  CAS  Google Scholar 

  18. Mortazavi BJ, Downing NS, Bucholz EM, Dharmarajan K, Manhapra A, Li SX, Negahban SN, Krumholz HM (2016) Analysis of machine learning techniques for heart failure readmissions. Circ Cardiovasc Qual Outcomes 9:629–640

    Article  Google Scholar 

  19. Mansoor H, Elgendy IY, Segal R, Bavry AA, Bian J (2017) Risk prediction model for in-hospital mortality in women with ST-elevation myocardial infarction: a machine learning approach. Heart Lung 46:405–411

    Article  Google Scholar 

  20. VanHouten JP, Starmer JM, Lorenzi NM, Maron DJ, Lasko TA (2014) Machine learning for risk prediction of acute coronary syndrome. AMIA Annu Symp Proc 2014:1940–1949

    PubMed  PubMed Central  Google Scholar 

  21. Hu D, Huang Z, Chan TM, Dong W, Lu X, Duan H (2016) Utilizing Chinese admission records for MACE prediction of acute coronary syndrome. Int J Environ Res Public Health 13:912

    Article  Google Scholar 

  22. Austin PC, Tu JV, Ho JE, Levy D, Lee DS (2013) Using methods from the data-mining and machine-learning literature for disease classification and prediction: a case study examining classification of heart failure subtypes. J Clin Epidemiol 66:398–407

    Article  Google Scholar 

  23. Ambale-Venkatesh B, Yang X, Wu CO, Liu K, Hundley WG, McClelland R, Gomes AS, Folsom AR, Shea S, Guallar E, Bluemke DA, Lima JAC (2017) Cardiovascular event prediction by machine learning: the multi-ethnic study of atherosclerosis. Circ Res 121:1092–1101

    Article  CAS  Google Scholar 

  24. Cabitza F, Rasoini R, Gensini GF (2017) Unintended consequences of machine learning in medicine. JAMA 318:517–518

    Article  Google Scholar 

  25. Ng K, Steinhubl SR, deFilippi C, Dey S, Stewart WF (2016) Early detection of heart failure using electronic health records: practical implications for time before diagnosis, data diversity, data quantity, and data density. Circ Cardiovasc Qual Outcomes 9:649–658

    Article  Google Scholar 

  26. Mayer-Schonberger V, Ingelsson E (2017) Big data and medicine—a big deal? J Intern Med 283:418–429

    Article  Google Scholar 

  27. Deo RC (2015) Machine learning in medicine. Circulation 132:1920–1930

    Article  Google Scholar 

  28. Gulshan V, Peng L, Coram M, Stumpe MC, Wu D, Narayanaswamy A, Venugopalan S, Widner K, Madams T, Cuadros J, Kim R, Raman R, Nelson PC, Mega JL, Webster DR (2016) Development and validation of a deep learning algorithm for detection of diabetic retinopathy in retinal fundus photographs. JAMA 316:2402–2410

    Article  Google Scholar 

  29. Esteva A, Kuprel B, Novoa RA, Ko J, Swetter SM, Blau HM, Thrun S (2017) Dermatologist-level classification of skin cancer with deep neural networks. Nature 542:115–118

    Article  CAS  Google Scholar 

  30. Pirracchio R, Petersen ML, Carone M, Rigon MR, Chevret S, van der Laan MJ (2015) Mortality prediction in intensive care units with the Super ICU Learner Algorithm (SICULA): a population-based study. Lancet Respir Med 3:42–52

    Article  Google Scholar 

Download references

Author information

Author notes

  1. William J. Gibson and Tarek Nafee have contributed equally to the manuscript.

Authors and Affiliations

  1. The Cardiovascular Division, Department of Medicine, Beth Israel Deaconess Medical Center, Harvard Medical School, 930 Commonwealth Avenue, Boston, MA, USA

    William J. Gibson, Tarek Nafee, Ryan Travis, Megan Yee, Mathieu Kerneis & C. Michael Gibson

  2. The Department of Medicine, Brigham and Women’s Hospital, Boston, USA

    William J. Gibson

  3. The Duke Clinical Research Institute, Duke University, Durham, USA

    Magnus Ohman

Authors
  1. William J. Gibson
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Tarek Nafee
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Ryan Travis
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Megan Yee
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Mathieu Kerneis
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Magnus Ohman
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. C. Michael Gibson
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to C. Michael Gibson.

Ethics declarations

Conflict of interest

All authors report research grant support from Janssen Pharmaceuticals Inc.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary material 1 (DOCX 71 kb)

Rights and permissions

Reprints and Permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Gibson, W.J., Nafee, T., Travis, R. et al. Machine learning versus traditional risk stratification methods in acute coronary syndrome: a pooled randomized clinical trial analysis. J Thromb Thrombolysis 49, 1–9 (2020). https://doi.org/10.1007/s11239-019-01940-8

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11239-019-01940-8

Keywords

search

Navigation