Background

Cardiovascular disease is the leading cause of death in the United States, making it a common target for interventional research [1, 2]. Due to this, the composite endpoint of “major adverse cardiovascular events” (MACE) is an increasingly common primary outcome of interest. In 2008, the United States Food and Drug Administration (FDA), followed by the European Medicines Agency (EMA) in 2012, provided guidance on utilizing a three-point MACE outcome, which includes acute myocardial infarction (AMI), stroke, and cardiovascular mortality in all trials evaluating the cardiovascular safety of diabetic agents [3]. Some trials have utilized a four-point MACE as well, by including hospitalization for unstable angina or revascularization procedures [3, 4]. Five-point MACE further expands on this with the inclusion of heart failure (HF). While MACE is now a better-defined and more ubiquitous outcome among RCTs, its use in observational studies to assess the safety and real-world effectiveness of therapies remains less clear.

Observational studies using large administrative databases can evaluate population-level use and outcomes of therapies in an efficient and cost-effective way [5]. However, several common issues with observational studies limit their reproducibility and comparability to RCTs, thus limiting the utility of MACE as a composite outcome. First, in comparison to RCTs, observational studies often do not consistently report their protocols, including outcome definitions [6]. To allow for useful comparisons between observational studies and RCTs, improved standardization and transparency is needed [7, 8]. Even when MACE components are well-defined, such as AMI or stroke, another challenge encountered in observational studies is how to define outcomes using diagnosis codes available from administrative data. For example, the International Classification of Diseases (ICD) is a commonly used coding system for medical reimbursement and is one of the most frequent sources of information available in administrative databases [9]. Unfortunately, ICD codes can be prone to errors and diagnosis misclassifications, as they are primarily collected for reimbursement purposes and rely on clinical documentation that can vary across settings and providers [5]. Though several studies have attempted to validate the diagnosis codes for commonly used MACE components (e.g., AMI, stroke, and HF), with positive predictive values of upwards of 80–90%, it is unclear how these codes have been taken up in MACE composite outcome definitions [10,11,12]. For these reasons, the current use of MACE in observational studies warrants further investigation.

Thus, the purpose of this review was to systematically determine the most common definitions of MACE employed in observational studies using administrative data. With that, our objectives were: i) assess each study’s definition of MACE components (e.g., AMI, stroke), ii) assess the diagnostic criteria used for outcome ascertainment such as codes used and position of codes, and iii) assess whether outcomes had been validated. We hypothesized that, across observational studies, there is great variability in the definitions of MACE used and minimal alignment with the classic three, four, or five-point MACE outcomes. Our hope is that this work will promote a standard approach to the definition of MACE in future studies, allowing for the improved transparency and reproducibility of observational studies.

Methods

Search strategy and selection criteria

The protocol for this systematic review is based on the 2015 Preferred Reporting Items for Systematic Reviews and Meta-Analysis Protocols (PRISMA-P) statement [13]. The protocol was not registered in the International Prospective Register of Systematic Reviews (PROSPERO), but the original protocol and modifications are included in Supplementary Text 1 (see Additional file 1). The reporting of this review is based on the 2020 PRISMA statement and the checklist is provided in Supplementary Table 1 (see Additional file 2) [14]. We searched MEDLINE and EMBASE for literature published from January 1, 2010 to October 9, 2020 that defined composite MACE as a primary or secondary outcome in studies utilizing administrative databases. A 10-year lookback was used in order to restrict to the most recently published studies. The search strategy was developed by the authors using the search terms presented in Supplementary Text 2 (see Additional file 3) and was performed on October 9, 2020. A standardized protocol for study abstract screening and full-text review was developed and piloted using the same 20 studies among the reviewers (EB, ES, LH), and is further described in Supplementary Text 1 (see Additional file 1). The abstracts from each record were initially screened for relevance by three independent reviewers (EB, ES, LH) using Abstrackr (http://abstrackr.cebm.brown.edu/), a free semi-automated abstract screening tool [15]. Afterwards, we performed a full-length text review of the abstracts that were retained after screening (EB, ES, LH).

We included studies that identified MACE composite endpoints as the primary or secondary study outcome, used administrative data sources in the study, and defined MACE using International Classification of Diseases, Clinical Modification, Ninth Revision (ICD-9-CM) or Tenth Revision (ICD-10-CM) diagnosis codes. The included studies did not need to overtly name “major adverse cardiovascular events” as the primary or secondary outcome to be included into this study, as many definitions and terms are used across studies. For example, MACE may be referred to as “adverse” or “acute cardiovascular events”, “major adverse cardiovascular and cerebrovascular events” or simply “cardiovascular events”, among other terms. Thus, we included studies if the composite outcome that was studied utilized a combination of multiple MACE endpoints. We excluded reviews, meta-analyses, conference abstracts, editorials, papers whose primary or secondary outcome was not a composite of multiple MACE endpoints, and those that did not use or report ICD-9-CM or ICD-10-CM codes to define MACE endpoints. Any uncertainty with regards to inclusion or exclusion of an individual paper was resolved by consensus of the three reviewers.

Data extraction and synthesis

After each of the independent reviewers determined that a full-text met inclusion criteria for the study, definitions for composite MACE outcomes were extracted into a shared document using a standardized protocol by two reviewers (LH, EB). ICD-9-CM and ICD-10-CM codes were also extracted for each MACE endpoint, and were acquired either through full-length text review or through review of published supplements. The position of the diagnosis code required in the study outcome criteria was recorded as primary position, any position, or not reported. Further, information on whether the study outcomes had been previously validated was assessed in each individual study or review of supplements, and citations were extracted. Each study’s administrative data source and study years were also recorded.

Once data were extracted from included studies, we categorized the specific definitions of MACE based on the individual components referred to by the authors as opposed to the specific diagnosis codes used. We made this decision due to the lack of consistency in diagnosis codes used across studies and chose to categorize based on the MACE components that the authors reported. Component definitions of MACE included: AMI, acute coronary syndrome or ischemic heart disease (ACS/IHD), stroke (either ischemic or hemorrhagic stroke), revascularization procedures, cardiovascular (CV) death, and all-cause death. Of note, due to the variability of outcomes used across studies, the ACS/IHD component reflects the following definitions used by study authors: ACS, IHD, coronary artery disease (CAD), and unstable angina (UA). Overall, we performed a qualitative assessment of the evidence only, and did not perform a meta-analysis or strength of evidence assessment due to the nature of the research questions.

Results

Included studies

Our search of MEDLINE and EMBASE yielded 920 unique articles, 412 of which were retained for full-text review after abstract screening (Fig. 1). After excluding 354 studies during full-text review, 58 studies were included in the final analysis.

Fig. 1
figure 1

PRISMA flow diagram for included studies. Abbreviations: MACE, major adverse cardiovascular events; ICD, International Classification of Diseases

Full size image

Overall studies

The included observational studies utilized a range of outcome components to define MACE (Table 1). The majority of the included studies utilized ICD-9-CM to define outcomes. There was poor concordance of MACE definitions used in the included studies when compared to the three-point, four-point, and five-point definitions of MACE commonly used in randomized controlled trials (RCTs). For instance, 8.6% (5/58) of observational studies used a MACE definition that matched the three-point definition of MACE (AMI, stroke, CV death) while no studies matched the four-point or five-point definitions (Table 2). Across all included studies, the most common MACE component definitions were: AMI, stroke, 15.5% (9/58); AMI, stroke, all-cause death, 13.8% (8/58); and AMI, stroke, CV death 8.6% (5/58) (Table 3). Overall, 67% of studies (39/58) did not perform a validation or provided no citations validating the outcome definitions used. Additionally, 70.7% (41/58) of studies did not report the position of diagnosis codes used in endpoints, while 20.7% (12/58) used the primary diagnosis position and 8.6% (5/58) used any position.

Table 1 MACE definitions based on publication
Full size table
Table 2 Most common components of MACE (N = 58 studies)
Full size table
Table 3 Most commonly used MACE components
Full size table

MACE component definitions

Acute myocardial infarction

There were 45 studies that included AMI as a component of MACE (Table 2). Of these, 64.4% (29/45) defined outcomes using ICD-9-CM and 31.1% (14/45) used ICD-10-CM. Among these studies, 20% (9/45) defined AMI in the primary diagnosis position, 6.7% (3/45) in any position, and 73.3% (33/45) did not report the position used. The most common diagnosis codes were: 410.xx, making up 72% (21/29) of the ICD-9-CM studies; and I21.xx, I22.xx, making up 35.7% (5/14) of the ICD-10-CM studies (Table 4).

Table 4 ICD codes by clinical outcome
Full size table

Acute coronary syndrome / ischemic heart disease

There were 18 studies that defined ACS/IHD as a component of MACE. ICD-9-CM diagnosis codes only were used in 66.7% (12/18) of studies and 27.7% (5/18) used ICD-10-CM only. Studies used the primary diagnosis position 22.2% (4/18), any position 11.1% (2/18), and did not report the position 66.7% (12/18) of the time. The most common diagnosis codes were: 410–414.xx, 25% (3/12) of the ICD-9-CM studies; and I21-I24.xx, 40% (2/5) of the ICD-10-CM studies.

Stroke

There were 50 studies that included stroke, either ischemic or hemorrhagic, as a component of MACE. Of these, 60% (30/50) utilized ICD-9-CM only and 34% (17/50) used ICD-10-CM only. The primary diagnosis position was used 24% (12/50) of the time, any position used 8% (4/50), and no position reported 68% (34/50) of the time. Diagnosis codes used to define stroke were highly variable with no clear most common definition. Codes included: 433.xx, 434.xx, 436.xx, 13.3% (4/30) of ICD-9-CM studies; 430–437.xx, 13.3% (4/30) of ICD-9-CM studies; I63.xx, I65.xx, I66.xx, 17.6% of ICD-10-CM studies (3/17); and I63.xx, 17.6% (3/17) of ICD-10-CM studies.

Heart failure

A total of 15 studies included HF as a component of MACE. Of these studies, 80% (12/15) used ICD-9-CM only, and 13.3% (2/15) used ICD-10-CM only. The primary diagnosis position was used 20% (3/15) of the time but was otherwise not reported in 80% (12/15) of studies. The most common diagnosis codes used were: 428.xx, 50% (6/12) of ICD-9-CM studies; I11.0, I50.xx, I97.1, 50% (1/2) of ICD-10-CM studies; and I50.xx, K76.1, I97.1, I11.0, 50% (1/2) of ICD-10-CM studies.

All-cause death and cardiovascular death

There were nine studies that included cardiovascular death as a component of MACE. Of these, 11.1% (1/9) of studies utilized ICD-9-CM codes only for MACE components and 66.7% (6/9) used ICD-10-CM. The diagnosis position for cardiovascular death was primary 11.1% (1/9), any position 33.3% (3/9), and not reported 55.6% (5/9) of the time. The most common diagnosis codes used were: No codes listed, 100% (1/1) for studies using ICD-9-CM only; and I00.xx-I99.xx, 33.3% (2/6) for those using ICD-10-CM only. There were 15 studies including all-cause death as a component of MACE. Due to the nature of all-cause death, diagnosis codes positions are not necessary and were not reported.

Discussion

In our systematic review, we found substantial heterogeneity for MACE composite endpoints used in the literature. The two most common composite MACE definitions were “AMI and stroke”, and “AMI, stroke, and all-cause death,” but they only made up 15.5 and 13.8% of studies, respectively. Compared to MACE definitions used in RCTs, only 8.6% of included observational studies had definitions aligned with three-point MACE and none match four-point or five-point MACE. A large majority of studies included AMI and stroke but there was a lack of consensus with other components included in composite MACE endpoints. For instance, over half of included publications defined MACE using a definition that only was concordant with up to one other study. This diversity makes it challenging to compare findings across studies or to aggregate multiple study results for meta-analyses or systematic reviews when considering different treatment or research questions. Addressing the heterogeneity of MACE definitions used in practice requires attention due to the advantages of the MACE endpoint. Utilizing MACE as a composite outcome can potentially reduce the number of patients that need to be enrolled or identified in a retrospective cohort study, and reduce the follow-up time necessary to observe differences between different treatment groups [99]. These benefits not only potentially reduce research costs but can more expediently answer clinical questions, leading to improved patient care. Therefore, given that MACE endpoints will likely see increasing use as time goes on, there should be a continued effort to standardize and transparently report MACE definitions used in observational studies.

In 2007, findings similar to our study were observed when comparing prospective trials conducted for percutaneous coronary interventions [100]. However, although the FDA in 2008 and the EMA in 2012 attempted to standardize the MACE endpoint definition for RCTs, we found a continued discordance between how observational studies define MACE. Furthermore, we found that the majority of studies did not include mention of the diagnosis position used to define outcomes. Lack of this information prevents the ability to distinguish between incident and prevalent outcomes. While the primary position has been historically used in observational studies to define incident outcomes, the use of only primary diagnosis positions, compared to using secondary positions, may underestimate the rates of MACE for prevalent conditions [101]. The lack of reporting of diagnosis position prevents the ability to make this distinction and prohibits the full interpretation of findings.

Heterogeneity not only existed in defining MACE components but also in which ICD-9-CM and ICD-10-CM codes were used to define each individual MACE component. In our study, none of the top five components of MACE had a consensus ICD-9-CM or ICD-10-CM code that was used among all of the studies. These discrepancies contribute additional variability to our findings, as outcome classification, assuming a standard definition, inherently relies on the providers and medical billers accurately coding diseases. For example, in a study conducted in 11 Canadian emergency departments, the authors reported about 82–86% of agreement with regards to coding hospital conditions [102]. Additionally, a study looking at stroke data collected through the United States Paul Coverdell National Acute Stroke Program found that discordance existed with how strokes were coded, especially at smaller hospitals (< 200 beds) and when differentiating between ischemic strokes, transient ischemic attacks, and subarachnoid and intracerebral hemorrhages [103].

There have been many attempts to validate diagnosis or procedure codes for various MACE component outcomes [32, 104]. Research has predominantly focused on the older ICD-9-CM codes, particularly studies using U.S. administrative data, but studies utilizing ICD-10-CM have more recently been published. Using data from the Centers for Medicare and Medicaid Services, a 2018 study found that the most accurate ICD-9-CM code for coding AMI was 410.xx, with a positive predictive value of 67% [104]. Additionally, the U.S. FDA commissioned the Mini-Sentinel pilot program to systematically assess the validation of common healthcare outcomes in administrative databases [32]. These studies were important steps to identify standard outcome definitions for use across disparate data sources and populations. As part of the program, systematic literature reviews from the U.S. and Canada found that the most accurate ICD-9-CM code for heart failure was 428.xx, with a positive predictive value of 84–100%, while the most accurate ICD-9-CM codes for stroke were 430.x, 431.x and 434.x, all with separate positive predictive values > 80%, and 436.x, which had a positive predictive value > 70% [32, 33]. The Mini-Sentinel program also assessed the ICD-9-CM codes, 410.× 0 or 410.× 1, for AMI and found a positive predictive value of 76.3–94.3% [105]. Fewer validations exist for ICD-10-CM codes. One study conducted in Japan attempted to validate the ICD-10-CM code I21.x for AMI and found a positive predictive value of 82.5% [106]. Another study in Canada found the positive predictive value for correctly coding stroke was 92% for the ICD-10-CM codes, I60.x-I61.x, I63.x-I64.x, H34.1 and G45.x [69]. Based on our findings, the most common ICD-9-CM and 10-CM codes used for AMI were also the most validated, 410.xx and I21.xx, respectively. The most validated and most common ICD-9-CM code used for heart failure was 428.xx. On the other hand, stroke had some discrepancies when comparing the most validated versus the most commonly used ICD codes. This discordance likely existed because of the differences with which stroke can be defined, particularly whether to include acute ischemic strokes with transient ischemic attacks, intracerebral hemorrhages or subarachnoid hemorrhages [69].

There are several limitations to our study. We excluded papers published before 2010, prior to the first published reviews of the Mini-Sentinel program, in order to present a contemporary review of the subject [32]. However, this cutoff likely skewed our results towards more modern MACE definitions and excluded different definitions that were potentially used for older publications. Older studies likely exhibited greater variety in the outcome definitions used. Additionally, because the words “major adverse cardiovascular events” were rarely explicitly written, we had to include any composite definitions that were thought to represent MACE but were listed using another composite name, such “acute cardiovascular events,” based on the discretion of the reviewer. This method potentially excluded studies that used variations of MACE endpoints, but did not conform to similar naming conventions.

Conclusion

Significant heterogeneity exists in how MACE is defined and which ICD-9-CM and 10-CM codes are used to represent each respective MACE outcome in observational studies using administrative databases. The utility of future studies will improve with the use of validated definitions of AMI, stroke, cardiovascular mortality, unstable angina, and HF for the evaluation of MACE outcomes. Further, investigators should ensure that both the ICD diagnosis codes and the code positions are reported in a transparent way. Given the significant heterogeneity that already exists across administrative databases, we recommend the use of more standardized MACE definitions and corresponding ICD-9-CM and ICD-10-CM codes. This practice will allow researchers to more accurately compare findings across studies improve the reproducibility of observational studies, and decrease the potential for misleading conclusions.