Introduction

Patients with transient ischemic attack (TIA) are at risk of recurrent vascular events [1,2,3]. Several risk stratification tools have been developed to identify TIA patients with the greatest risk of recurrent vascular events [4]. The ABCD2 score is the most commonly used TIA risk stratification tool [4]. The ABCD2 score includes age, blood pressure, clinical features (e.g., weakness or speech impairment), neurologic symptom duration, and diabetes [5].

Although the ABCD2 can be calculated prospectively by clinicians caring for patients, electronic health record systems typically do not include neurological symptom descriptions as data fields; therefore, this score, and others that include results from brain imaging cannot be constructed from electronic health record data [4]. TIA risk stratification tools that effectively distinguish low- from high-risk patients and that can be deployed within electronic health record systems are needed to support the Learning Healthcare System model which involves learning from data to improve practice. For example, Learning Healthcare Systems need robust risk adjustment models to identify facilities that have outlier status (those with either better-than-expected or worse-than-expected outcome rates). By exploring differences in policies, practices, infrastructure, and culture between high-performing and low-performing facilities opportunities for quality improvement may be recognized. The objective of this project was to evaluate two TIA risk stratification approaches based on electronic health record data.

Methods

Cohort

A sample from the cohort that was constructed for the Protocol-Guided Rapid Evaluation of Veterans Experiencing New Transient Neurological Symptoms (PREVENT; clinicaltrials.gov: NCT02769338) study was used for this project [6, 7]. We identified patients with TIA who were cared for in any Department of Veterans Affairs (VA) Emergency Department (ED) or inpatient setting based on primary discharge codes for TIA from October 1, 2015 to September 30, 2018 (International Classification of Disease ICD-10 G45.0, G45.1, G45.8, G45.9, I67.848 ) [8]. The first TIA event during the study period per patient was included. This project received human subjects (institutional review board [IRB]) and VA research and development committee approvals. The institutional review board waived the need for patient consent.

Outcomes

We examined three outcomes at 90-days and 1-year post-index TIA. The all-cause mortality rate (defined as death from any cause within 90-days or 1-year of presentation for the index event) was obtained from the VA Vital Status File [9]. Recurrent ischemic stroke (defined as an ischemic stroke in the ED or inpatient stay within 90-days or 1-year of discharge of the index TIA event) was identified using primary diagnosis codes in the ED or inpatient setting using a combination of both VA and fee-basis data (which describes healthcare in non-VA facilities that was paid for by the VA). Therefore, recurrent strokes which occurred in community hospitals, but which were not paid for by the VA, were not included. The combined endpoint of stroke or death was also evaluated at 90-days and 1-year post-index TIA.

Data sources

Data were obtained from the VA Corporate Data Warehouse (CDW) [10] which includes: inpatient and outpatient data files (e.g., diagnostic and procedure codes) in the five-years pre-event to identify past medical history, [11] healthcare utilization, receipt of procedures (Current Procedural Terminology [CPT], Healthcare Common Procedures Coding System [HCPCS], and ICD-9 and ICD-10 procedure codes), vital signs, laboratory data, orders, medications and clinical consults. Fee-Basis Data were also used to identify inpatient and outpatient healthcare utilization and medical history.

The Clinical Assessment Needs (CAN) version 2.0 score is a validated measure of risk of hospitalization, death, or the combination of hospitalization or death within 90-days or 1-year that is calculated for Veterans in VA healthcare [12]. The CAN score version 2.0 is updated weekly for patients that are assigned to a primary care Patient Aligned Care Team (PACT), are Veterans, not hospitalized on the date the score is run, and are alive as of the date the score is generated. The CAN score is based on 32–36 data elements obtained from the CDW including: sociodemographics, healthcare utilization (e.g., clinic visits, inpatient admissions, ED and urgent care visits), vital signs, medications, laboratory data, and number and type of comorbidities.

Analysis

The original sample consisted of 8270 patients from Oct 1, 2015 to March 31, 2018. The cohort was randomly split into training (n = 5506) and validation samples (n = 2764 ) [13]. Within the training sample, logistic regression models were used to identify the patient characteristics that were associated with the outcomes [14]. Separate risk adjustment models were constructed for each outcome. For the purpose of this analysis, we added all TIA patients from April 1, 2018 to September 30, 2018 to the validation sample and excluded all patients with missing CAN scores resulting in a final sample size of 5250 for the training sample and 4248 for the validation sample. The final models that were constructed on the training set (referred to as the PREVENT scores) were then applied to the validation set. Chi-square tests, t-tests, or Wilcoxon Rank Sum tests were used to compare whether patient characteristics differed between the development and validation samples. C-statistics (area under the receiver operating characteristic [ROC] curve) [4] and observed (unadjusted) outcome rates across risk categories (including identification of patients with extremely low [nadir] or high [zenith] outcome rates) [15] were used to evaluate the performance of the final risk model. All analyses were performed using SAS Enterprise Guide version 7.11.

Data sharing statement

The dataset supporting the conclusions of this article is not available. According to Department of Veterans Affairs (VA) policy, these data are stored behind the VA firewall and cannot be shared even after deidentification. Investigators interested in analyses of the existing data are encouraged to contact the corresponding author.

Results

A total of N = 5250 TIA patients were included in the derivation sample and N = 4248 in the validation sample; their baseline characteristics are provided in Table 1.

Table 1 Comparison of baseline characteristics between derivation and validation samples
Full size table

The two samples were similar with the following exceptions: more validation patients had hemiplegia (16.3% versus 14.6%, p = 0.022); more validation patients were taking statins prior to the index-TIA (82.6% versus 80.9%, p = 0.032); fewer validation patients were taking warfarin prior to the index-TIA (10.3% versus 11.7%, p = 0.038); and validation patients had higher APACHE scores indicating modestly greater physiologic disease severity (mean 10.14 ± standard deviation 6.70 versus 9.83 ± 6.75; p = 0.026). The overall outcome rates in the validation set (Table 2) were: 2.1% 90-day mortality, 7.1% 1-year mortality, 3.2% 90-day stroke, 5.6% 1-year stroke, 5.1% 90-day stroke or death, and 11.8% 1-year stroke or death. The factors that were included in the final PREVENT score are provided in Table 3.

Table 2 Comparison of the CAN Score versus PREVENT Score
Full size table
Table 3 PREVENT risk models for each outcome
Full size table

Both the CAN score and the PREVENT score had higher c-statistics for the mortality outcomes than the recurrent stroke or combined stroke or death outcomes and both scores performed similarly in the development and the validation sets (Table 2). The PREVENT score had higher c-statistics than the CAN score across all outcomes in both the development and validation samples (Table 2). Within the validation sample the c-statistics for the PREVENT score were: 0.847 for 90-day mortality, 0.814 for 1-year mortality, 0.665 for 90-day stroke, and 0.653 for 1-year stroke, 0.699 for 90-day stroke or death, and 0.744 for 1-year stroke or death.

Both scores distinguished low-risk from high-risk patients, however the PREVENT score classified patients into categories with more extreme nadir and zenith outcome rates (Table 4; Fig. 1). For example, as described above, the observed 1-year mortality rate among the patients in the validation sample was 7.1%. The PREVENT score, when split into deciles created a nadir group of patients with 0% 1-year mortality and a zenith group with 30.4% 1-year mortality.

Table 4 Comparison of unadjusted outcomes by risk deciles for the validation sample
Full size table
Fig. 1
figure 1

Observed outcome rates: CAN score and PREVENT score

Full size image

Discussion

This study demonstrates that data available from electronic health record systems can be used to describe the risk of mortality, recurrent stroke, or the combined endpoint of recurrent stroke or death among patients with TIA. The PREVENT score provided robust risk stratification for mortality outcomes (c-statistics of 0.814–0.847) and adequate risk stratification for stroke or death (c-statistics of 0.699–0.744), but modest risk stratification for recurrent stroke (c-statistics of 0.665–0.653). A meta-analysis of studies evaluating the ABCD2 score reported an overall c-statistic of 0.72 (0.63 to 0.80) with better performance in the first 7-days post-TIA versus the 90-days post-TIA [16]. These results suggest that TIA risk stratification tools can be implemented within electronic health record systems.

The Learning Healthcare System model has enjoyed widespread adoption as the potential to leverage health data infrastructure to improve care has been recognized [17, 18]. Several approaches to learning from data are consistent with the Learning Healthcare System paradigm. For example, some systems use electronic health record data to develop and report e-quality metrics, which may or may not be reported with risk adjustment [19, 20]. Other systems examine observed outcomes relative to risk-adjusted expected rates [21, 22]. Validated disease specific (e.g., patients with TIA) or setting specific (e.g., patients admitted to the intensive care unit) risk-adjustment models are needed to identify outlier facilities with either lower-than-expected outcome rates or worse-than-expected outcome rates. The identification of outliers is a commonly used approach in quality management [23]. The results of the current study can be used by health systems seeking to evaluate TIA risk-adjusted patient outcomes across facilities.

Our findings may surprise clinicians who might hypothesize that the characteristics of the TIA event (e.g., symptoms) would contribute to risk of recurrent vascular events and hence that approaches which rely solely upon electronic health record data would be inadequate for risk stratification. However, two potential factors may explain the strong performance of the PREVENT score for TIA risk stratification. First, the VA electronic health record includes a broad range of factors including demographics (e.g., age), diagnosis codes (e.g., history of diabetes), medications, laboratory data, and vital signs (e.g., blood pressure). In this way, three of the five ABCD2 score elements were included in the PREVENT score. In addition, the inclusion of hospice or palliative care services in the PREVENT score likely contributed to the modeling of mortality. Second, it may be that patients with transient neurological symptoms who have an infarct on brain imaging are coded as a stroke (not a TIA) [24]. Therefore, the value of including brain imaging results to TIA risk stratification must be evaluated within contemporary cohorts with increased use of magnetic resonance imaging [25].

A key characteristic of the PREVENT system was the ability to identify low-risk patients with 0% mortality and 1.4% stroke rates at 90-days. Although all TIA patients should receive timely, guideline-concordant care, such low-risk patients may not require in-patient admission. Future studies should examine whether high-risk patients with 10.8% mortality, 6.7% stroke, and 14.4% stroke or death rates at 90-days could benefit from intensive clinical management approaches [24]. Quality improvement activities may well differ when targeting high-risk versus low-risk patients. For example, in settings with a relatively higher prevalence of high-risk patients, quality improvement programs might seek to promote hospital admission to facilitate timely risk factor management [24].

Given the evidence that outcome rates are improving among TIA patients over time, it is relevant to examine how risk stratification systems perform in contemporary cohorts [1]. A meta-analysis of 40 studies from 2008 to 2015 period reported a cumulative risk of stroke of 7.4% (95% CI 0.043–0.124) at 90 days [1]. The observed 90-day stroke rate of 3.2% in our cohort is lower than these prior reported rates. However, given that we only included recurrent events from VA data sources, we expect that our observed combined endpoint underestimated the actual recurrent stroke events.

Although TIA events are by their nature transient and hence unlikely to confer direct mortality risk, TIAs are markers of vascular disease, [26] and patients with TIA are at increased risk of death [27, 28]. For example, Amarenco, et al. reported that among 4789 patients with TIA or minor stroke treated in specialized centers, major cardiovascular events were observed in 6.2% of patients at 1-year post-index event and death from any cause was observed in 1.8 % [28]. Our finding of 1-year mortality of 7.1% was much higher than the reports from other cohorts.

Although this study included a relatively large sample size from a national cohort of patients with TIA and included a methodologically rigorous design with separate development and validation samples, several limitations merit description. First, the index TIA events were based on diagnosis codes and given the clinical uncertainty inherent in the TIA diagnosis, some of the TIA patients may eventually have been diagnosed with other clinical conditions (e.g., migraine). Second, the electronic health record data did not include symptom characteristics, therefore, we could not compare the CAN or PREVENT scores with the ABCD2 score. Future studies should compare the PREVENT score with other clinical TIA risk stratification systems such as the ABCD2 score. Third, the cohort included Veteran patients seeking care for an index TIA in a VA hospital. Given differences between the general US population and the Veteran population, the outcome rates may not be generalizable to community cohorts. Future studies should evaluate the performance of the risk models in non-Veteran cohorts. Fourth, the PREVENT score included hospice and palliative care services which may have improved its ability to model mortality outcomes. Future studies should explore how individual components of the score influence the overall stratification ability (e.g., with versus without including hospice patients). Also, given that the VA has a robust electronic health record system, future studies should also examine the implementation of the risk scores in other health systems.

Conclusions

Although the CAN score and the PREVENT score were similar, the PREVENT score had higher c-statistics for each model, produced greater spread between zenith and nadir risk categories, and contained fewer variables and therefore would be easier to implement. Given that the PREVENT score identified both very low-risk and high-risk TIA populations, Learning Healthcare Systems should consider implementing risk scores using electronic health record data to guide quality management.