Abstract
Background
Risk-stratification tools that have been developed to identify transient ischemic attack (TIA) patients at risk of recurrent vascular events typically include factors which are not readily available in electronic health record systems. Our objective was to evaluate two TIA risk stratification approaches using electronic health record data.
Methods
Patients with TIA who were cared for in Department of Veterans Affairs hospitals (October 2015—September 2018) were included. The six outcomes were mortality, recurrent ischemic stroke, and the combined endpoint of stroke or death at 90-days and 1-year post-index TIA event. The cohort was split into development and validation samples. We examined the risk stratification of two scores constructed using electronic health record data. The Clinical Assessment Needs (CAN) score is a validated measure of risk of hospitalization or death. The PREVENT score was developed specifically for TIA risk stratification.
Results
A total of N = 5250 TIA patients were included in the derivation sample and N = 4248 in the validation sample. The PREVENT score had higher c-statistics than the CAN score across all outcomes in both samples. Within the validation sample the c-statistics for the PREVENT score were: 0.847 for 90-day mortality, 0.814 for 1-year mortality, 0.665 for 90-day stroke, and 0.653 for 1-year stroke, 0.699 for 90-day stroke or death, and 0.744 for 1-year stroke or death. The PREVENT score classified patients into categories with extreme nadir and zenith outcome rates. The observed 1-year mortality rate among validation patients was 7.1%; the PREVENT score lowest decile of patients had 0% mortality and the highest decile group had 30.4% mortality.
Conclusions
The PREVENT score had strong c-statistics for the mortality outcomes and classified patients into distinct risk categories. Learning healthcare systems could implement TIA risk stratification tools within electronic health records to support ongoing quality improvement.
Registration
ClinicalTrials.gov Identifier: NCT02769338.
Similar content being viewed by others
Explore related subjects
Find the latest articles, discoveries, and news in related topics.Introduction
Patients with transient ischemic attack (TIA) are at risk of recurrent vascular events [1,2,3]. Several risk stratification tools have been developed to identify TIA patients with the greatest risk of recurrent vascular events [4]. The ABCD2 score is the most commonly used TIA risk stratification tool [4]. The ABCD2 score includes age, blood pressure, clinical features (e.g., weakness or speech impairment), neurologic symptom duration, and diabetes [5].
Although the ABCD2 can be calculated prospectively by clinicians caring for patients, electronic health record systems typically do not include neurological symptom descriptions as data fields; therefore, this score, and others that include results from brain imaging cannot be constructed from electronic health record data [4]. TIA risk stratification tools that effectively distinguish low- from high-risk patients and that can be deployed within electronic health record systems are needed to support the Learning Healthcare System model which involves learning from data to improve practice. For example, Learning Healthcare Systems need robust risk adjustment models to identify facilities that have outlier status (those with either better-than-expected or worse-than-expected outcome rates). By exploring differences in policies, practices, infrastructure, and culture between high-performing and low-performing facilities opportunities for quality improvement may be recognized. The objective of this project was to evaluate two TIA risk stratification approaches based on electronic health record data.
Methods
Cohort
A sample from the cohort that was constructed for the Protocol-Guided Rapid Evaluation of Veterans Experiencing New Transient Neurological Symptoms (PREVENT; clinicaltrials.gov: NCT02769338) study was used for this project [6, 7]. We identified patients with TIA who were cared for in any Department of Veterans Affairs (VA) Emergency Department (ED) or inpatient setting based on primary discharge codes for TIA from October 1, 2015 to September 30, 2018 (International Classification of Disease ICD-10 G45.0, G45.1, G45.8, G45.9, I67.848 ) [8]. The first TIA event during the study period per patient was included. This project received human subjects (institutional review board [IRB]) and VA research and development committee approvals. The institutional review board waived the need for patient consent.
Outcomes
We examined three outcomes at 90-days and 1-year post-index TIA. The all-cause mortality rate (defined as death from any cause within 90-days or 1-year of presentation for the index event) was obtained from the VA Vital Status File [9]. Recurrent ischemic stroke (defined as an ischemic stroke in the ED or inpatient stay within 90-days or 1-year of discharge of the index TIA event) was identified using primary diagnosis codes in the ED or inpatient setting using a combination of both VA and fee-basis data (which describes healthcare in non-VA facilities that was paid for by the VA). Therefore, recurrent strokes which occurred in community hospitals, but which were not paid for by the VA, were not included. The combined endpoint of stroke or death was also evaluated at 90-days and 1-year post-index TIA.
Data sources
Data were obtained from the VA Corporate Data Warehouse (CDW) [10] which includes: inpatient and outpatient data files (e.g., diagnostic and procedure codes) in the five-years pre-event to identify past medical history, [11] healthcare utilization, receipt of procedures (Current Procedural Terminology [CPT], Healthcare Common Procedures Coding System [HCPCS], and ICD-9 and ICD-10 procedure codes), vital signs, laboratory data, orders, medications and clinical consults. Fee-Basis Data were also used to identify inpatient and outpatient healthcare utilization and medical history.
The Clinical Assessment Needs (CAN) version 2.0 score is a validated measure of risk of hospitalization, death, or the combination of hospitalization or death within 90-days or 1-year that is calculated for Veterans in VA healthcare [12]. The CAN score version 2.0 is updated weekly for patients that are assigned to a primary care Patient Aligned Care Team (PACT), are Veterans, not hospitalized on the date the score is run, and are alive as of the date the score is generated. The CAN score is based on 32–36 data elements obtained from the CDW including: sociodemographics, healthcare utilization (e.g., clinic visits, inpatient admissions, ED and urgent care visits), vital signs, medications, laboratory data, and number and type of comorbidities.
Analysis
The original sample consisted of 8270 patients from Oct 1, 2015 to March 31, 2018. The cohort was randomly split into training (n = 5506) and validation samples (n = 2764 ) [13]. Within the training sample, logistic regression models were used to identify the patient characteristics that were associated with the outcomes [14]. Separate risk adjustment models were constructed for each outcome. For the purpose of this analysis, we added all TIA patients from April 1, 2018 to September 30, 2018 to the validation sample and excluded all patients with missing CAN scores resulting in a final sample size of 5250 for the training sample and 4248 for the validation sample. The final models that were constructed on the training set (referred to as the PREVENT scores) were then applied to the validation set. Chi-square tests, t-tests, or Wilcoxon Rank Sum tests were used to compare whether patient characteristics differed between the development and validation samples. C-statistics (area under the receiver operating characteristic [ROC] curve) [4] and observed (unadjusted) outcome rates across risk categories (including identification of patients with extremely low [nadir] or high [zenith] outcome rates) [15] were used to evaluate the performance of the final risk model. All analyses were performed using SAS Enterprise Guide version 7.11.
Data sharing statement
The dataset supporting the conclusions of this article is not available. According to Department of Veterans Affairs (VA) policy, these data are stored behind the VA firewall and cannot be shared even after deidentification. Investigators interested in analyses of the existing data are encouraged to contact the corresponding author.
Results
A total of N = 5250 TIA patients were included in the derivation sample and N = 4248 in the validation sample; their baseline characteristics are provided in Table 1.
The two samples were similar with the following exceptions: more validation patients had hemiplegia (16.3% versus 14.6%, p = 0.022); more validation patients were taking statins prior to the index-TIA (82.6% versus 80.9%, p = 0.032); fewer validation patients were taking warfarin prior to the index-TIA (10.3% versus 11.7%, p = 0.038); and validation patients had higher APACHE scores indicating modestly greater physiologic disease severity (mean 10.14 ± standard deviation 6.70 versus 9.83 ± 6.75; p = 0.026). The overall outcome rates in the validation set (Table 2) were: 2.1% 90-day mortality, 7.1% 1-year mortality, 3.2% 90-day stroke, 5.6% 1-year stroke, 5.1% 90-day stroke or death, and 11.8% 1-year stroke or death. The factors that were included in the final PREVENT score are provided in Table 3.
Both the CAN score and the PREVENT score had higher c-statistics for the mortality outcomes than the recurrent stroke or combined stroke or death outcomes and both scores performed similarly in the development and the validation sets (Table 2). The PREVENT score had higher c-statistics than the CAN score across all outcomes in both the development and validation samples (Table 2). Within the validation sample the c-statistics for the PREVENT score were: 0.847 for 90-day mortality, 0.814 for 1-year mortality, 0.665 for 90-day stroke, and 0.653 for 1-year stroke, 0.699 for 90-day stroke or death, and 0.744 for 1-year stroke or death.
Both scores distinguished low-risk from high-risk patients, however the PREVENT score classified patients into categories with more extreme nadir and zenith outcome rates (Table 4; Fig. 1). For example, as described above, the observed 1-year mortality rate among the patients in the validation sample was 7.1%. The PREVENT score, when split into deciles created a nadir group of patients with 0% 1-year mortality and a zenith group with 30.4% 1-year mortality.
Discussion
This study demonstrates that data available from electronic health record systems can be used to describe the risk of mortality, recurrent stroke, or the combined endpoint of recurrent stroke or death among patients with TIA. The PREVENT score provided robust risk stratification for mortality outcomes (c-statistics of 0.814–0.847) and adequate risk stratification for stroke or death (c-statistics of 0.699–0.744), but modest risk stratification for recurrent stroke (c-statistics of 0.665–0.653). A meta-analysis of studies evaluating the ABCD2 score reported an overall c-statistic of 0.72 (0.63 to 0.80) with better performance in the first 7-days post-TIA versus the 90-days post-TIA [16]. These results suggest that TIA risk stratification tools can be implemented within electronic health record systems.
The Learning Healthcare System model has enjoyed widespread adoption as the potential to leverage health data infrastructure to improve care has been recognized [17, 18]. Several approaches to learning from data are consistent with the Learning Healthcare System paradigm. For example, some systems use electronic health record data to develop and report e-quality metrics, which may or may not be reported with risk adjustment [19, 20]. Other systems examine observed outcomes relative to risk-adjusted expected rates [21, 22]. Validated disease specific (e.g., patients with TIA) or setting specific (e.g., patients admitted to the intensive care unit) risk-adjustment models are needed to identify outlier facilities with either lower-than-expected outcome rates or worse-than-expected outcome rates. The identification of outliers is a commonly used approach in quality management [23]. The results of the current study can be used by health systems seeking to evaluate TIA risk-adjusted patient outcomes across facilities.
Our findings may surprise clinicians who might hypothesize that the characteristics of the TIA event (e.g., symptoms) would contribute to risk of recurrent vascular events and hence that approaches which rely solely upon electronic health record data would be inadequate for risk stratification. However, two potential factors may explain the strong performance of the PREVENT score for TIA risk stratification. First, the VA electronic health record includes a broad range of factors including demographics (e.g., age), diagnosis codes (e.g., history of diabetes), medications, laboratory data, and vital signs (e.g., blood pressure). In this way, three of the five ABCD2 score elements were included in the PREVENT score. In addition, the inclusion of hospice or palliative care services in the PREVENT score likely contributed to the modeling of mortality. Second, it may be that patients with transient neurological symptoms who have an infarct on brain imaging are coded as a stroke (not a TIA) [24]. Therefore, the value of including brain imaging results to TIA risk stratification must be evaluated within contemporary cohorts with increased use of magnetic resonance imaging [25].
A key characteristic of the PREVENT system was the ability to identify low-risk patients with 0% mortality and 1.4% stroke rates at 90-days. Although all TIA patients should receive timely, guideline-concordant care, such low-risk patients may not require in-patient admission. Future studies should examine whether high-risk patients with 10.8% mortality, 6.7% stroke, and 14.4% stroke or death rates at 90-days could benefit from intensive clinical management approaches [24]. Quality improvement activities may well differ when targeting high-risk versus low-risk patients. For example, in settings with a relatively higher prevalence of high-risk patients, quality improvement programs might seek to promote hospital admission to facilitate timely risk factor management [24].
Given the evidence that outcome rates are improving among TIA patients over time, it is relevant to examine how risk stratification systems perform in contemporary cohorts [1]. A meta-analysis of 40 studies from 2008 to 2015 period reported a cumulative risk of stroke of 7.4% (95% CI 0.043–0.124) at 90 days [1]. The observed 90-day stroke rate of 3.2% in our cohort is lower than these prior reported rates. However, given that we only included recurrent events from VA data sources, we expect that our observed combined endpoint underestimated the actual recurrent stroke events.
Although TIA events are by their nature transient and hence unlikely to confer direct mortality risk, TIAs are markers of vascular disease, [26] and patients with TIA are at increased risk of death [27, 28]. For example, Amarenco, et al. reported that among 4789 patients with TIA or minor stroke treated in specialized centers, major cardiovascular events were observed in 6.2% of patients at 1-year post-index event and death from any cause was observed in 1.8 % [28]. Our finding of 1-year mortality of 7.1% was much higher than the reports from other cohorts.
Although this study included a relatively large sample size from a national cohort of patients with TIA and included a methodologically rigorous design with separate development and validation samples, several limitations merit description. First, the index TIA events were based on diagnosis codes and given the clinical uncertainty inherent in the TIA diagnosis, some of the TIA patients may eventually have been diagnosed with other clinical conditions (e.g., migraine). Second, the electronic health record data did not include symptom characteristics, therefore, we could not compare the CAN or PREVENT scores with the ABCD2 score. Future studies should compare the PREVENT score with other clinical TIA risk stratification systems such as the ABCD2 score. Third, the cohort included Veteran patients seeking care for an index TIA in a VA hospital. Given differences between the general US population and the Veteran population, the outcome rates may not be generalizable to community cohorts. Future studies should evaluate the performance of the risk models in non-Veteran cohorts. Fourth, the PREVENT score included hospice and palliative care services which may have improved its ability to model mortality outcomes. Future studies should explore how individual components of the score influence the overall stratification ability (e.g., with versus without including hospice patients). Also, given that the VA has a robust electronic health record system, future studies should also examine the implementation of the risk scores in other health systems.
Conclusions
Although the CAN score and the PREVENT score were similar, the PREVENT score had higher c-statistics for each model, produced greater spread between zenith and nadir risk categories, and contained fewer variables and therefore would be easier to implement. Given that the PREVENT score identified both very low-risk and high-risk TIA populations, Learning Healthcare Systems should consider implementing risk scores using electronic health record data to guide quality management.
Availability of data and materials
The dataset supporting the conclusions of this article is not available. According to Department of Veterans Affairs (VA) policy, these data are stored behind the VA firewall and cannot be shared even after deidentification. Investigators interested in analyses of the existing data are encouraged to contact the corresponding author.
Abbreviations
- CAN:
-
Clinical Assessment Needs
- ED:
-
Emergency Department
- PREVENT:
-
Protocol-guided Rapid Evaluation of Veterans Experiencing New Transient Neurological Symptoms
- TIA:
-
Transient ischemic attack
- VA:
-
Department of Veterans Affairs
- PACT:
-
Patient Aligned Care Team
References
Najib N, Magin P, Lasserson D, Quain D, Attia J, Oldmeadow C, et al. Contemporary prognosis of transient ischemic attack patients: a systematic review and meta-analysis. Int J Stroke. 2019;14(5):460–7.
Johnston S. Short-term prognosis after a TIA: a simple score predicts risk. Cleve Clin J Med. 2007;74(10):729–36.
Rothwell P, Johnston S. Transient ischemic attacks: stratifying risk. Stroke. 2006;37(2):320–2.
Wolf ME, Held VE, Hennerici MG. Risk scores for transient ischemic attack. Front Neurol Neurosci. 2014;33:41–68.
Johnston SC, Rothwell PM, Nguyen-Huynh MN, Giles MF, Elkins JS, Bernstein AL, et al. Validation and refinement of scores to predict very early stroke risk after transient ischaemic attack. Lancet (London, England). 2007;369(9558):283–92.
Bravata D, Myers L, Homoya B, Miech E, Rattray N, Perkins A, et al. The protocol-guided rapid evaluation of veterans experiencing new transient neurological symptoms (PREVENT) quality improvement program: rationale and methods. BMC Neurol. 2019;19(1):294.
Bravata DM, Myers LJ, Perkins AJ, Zhang Y, Miech EJ, Rattray NA, et al. Assessment of the protocol-guided rapid evaluation of veterans experiencing new transient neurological symptoms (PREVENT) program for improving quality of Care for Transient Ischemic Attack: a nonrandomized cluster trial. JAMA Netw Open. 2020;3(9):e2015920.
Bravata DM, Myers LJ, Cheng E, Reeves M, Baye F, Yu Z, et al. Development and validation of electronic quality measures to assess Care for Patients with Transient Ischemic Attack and Minor Ischemic Stroke. Circ Cardiovasc Qual Outcomes. 2017;10(9):e003157.
Sohn MW, Arnold N, Maynard C, Hynes DM. Accuracy and completeness of mortality data in the Department of Veterans Affairs. Popul Health Metrics. 2006;4:2.
VA Informatics and Computing Infrastructure (VINCI). VA HSR RES 13-457, U.S. Department of Veterans Affairs. 2008. Retrieved July 1, 2021, from: https://vincicentral.vinci.med.va.gov.
Borzecki AM, Wong AT, Hickey EC, Ash AS, Berlowitz DR. Can we use automated data to assess quality of hypertension care? Am J Manag Care. 2004;10(7 Pt 2):473–9.
Wang L, Porter B, Maynard C, Evans G, Bryson C, Sun H, et al. Predicting risk of hospitalization or death among patients receiving primary care in the veterans health administration. Med Care. 2013;51(4):368–73.
Kernan WN, Feinstein AR, Brass LM. A methodological appraisal of research on prognosis after transient ischemic attacks. Stroke. 1991;22(9):1108–16.
Li J, Zhang Y, Myers L, Bravata D. Power calculation in stepped-wedge cluster randomized trial with reduced intervention sustainability effect. J Biopharm Stat. 2019;29(4):663–74 In Press.
Esdaile JM, Horwitz RI, Levinton C, Clemens JD, Amatruda JG, Feinstein AR. Response to initial therapy and new onset as predictors of prognosis in patients hospitalized with congestive heart failure. Clin Invest Med. 1992;15(2):122–31.
Giles MF, Rothwell PM. Systematic review and pooled analysis of published and unpublished validations of the ABCD and ABCD2 transient ischemic attack risk scores. Stroke. 2010;41(4):667–73.
Budrionis A, Bellika JG. The learning healthcare system: where are we now? A systematic review. J Biomed Inform. 2016;64:87–92.
Sarakbi D, Mensah-Abrampah N, Kleine-Bingham M, Syed SB. Aiming for quality: a global compass for national learning systems. Health Res Policy Syst. 2021;19(1):102.
Guarneri V, Pronzato P, Bertetto O, Roila F, Amunni G, Bortolami A, et al. Use of electronic administrative databases to measure quality indicators of breast Cancer care: experience of five regional oncology networks in Italy. JCO Oncol Pract. 2020;16(2):e211–20.
Bravata D, Myers L, Cheng E, Reeves M, Baye F, Yu Z, et al. Development and validation of electronic quality measures to assess Care for Patients with transient ischemic attack and minor stroke. Circ Cardiovasc Qual Outcomes. 2017;10:e003157.
Chen LM, Render M, Sales A, Kennedy EH, Wiitala W, Hofer TP. Intensive care unit admitting patterns in the veterans affairs health care system. Arch Intern Med. 2012;172(16):1220–6.
Render ML, Deddens J, Freyberg R, Almenoff P, Connors AF Jr, Wagner D, et al. Veterans affairs intensive care unit risk adjustment model: validation, updating, recalibration. Crit Care Med. 2008;36(4):1031–42.
Waldo SW, McCabe JM, Kennedy KF, Zigler CM, Pinto DS, Yeh RW. Quality of Care at Hospitals Identified as outliers in publicly reported mortality statistics for percutaneous coronary intervention. Circulation. 2017;135(20):1897–907.
Easton J, Saver J, Albers G, Alberts M, Chaturvedi S, Feldmann E, et al. Definition and evaluation of transient ischemic attack. Stroke. 2009;40(6):2276–93.
Chaturvedi S, Ofner S, Baye F, Myers LJ, Phipps M, Sico JJ, et al. Have clinicians adopted the use of brain MRI for patients with TIA and minor stroke? Neurology. 2017;88(3):237–44.
Poledník I, Sulzenko J, Widimsky P. Risk of a coronary event in patients after ischemic stroke or transient ischemic attack. Anatol J Cardiol. 2021;25(3):152–5.
Bravata DM, Myers LJ, Homoya B, Miech EJ, Rattray NA, Perkins AJ, et al. The protocol-guided rapid evaluation of veterans experiencing new transient neurological symptoms (PREVENT) quality improvement program: rationale and methods. BMC Neurol. 2019;19(1):294.
Amarenco P, Lavallée PC, Labreuche J, Albers GW, Bornstein NM, Canhão P, et al. One-year risk of stroke after transient ischemic attack or minor stroke. N Engl J Med. 2016;374(16):1533–42.
Acknowledgements
Not applicable.
Funding
This work was supported by the Department of Veterans Affairs (VA), Health Services Research & Development Service (HSRD), Precision Monitoring to Transform Care (PRISM) Quality Enhancement Research Initiative (QUERI) (QUE 15–280). Support for VA/Centers for Medicare and Medicaid Service (CMS) data is provided by the VA Information Resource Center (SDR 02–237 and 98–004). The funding agencies had no role in the design of the study, data collection, analysis, interpretation, or in the writing of this manuscript.
Author information
Authors and Affiliations
Contributions
All authors participated in the revision of the manuscript and read and approved the final manuscript. DMB: obtained funding and was responsible for the design and conduct of the study including data collection, data analysis, interpretation of the results, and drafting and revising the manuscript. LJM: instrumental in data collection, data analysis, and interpretation of the results, and drafting and revising the manuscript. AJP, YZ: instrumental in the development of the analysis plan, in the conduct of the data analysis, in the interpretation of the results, and revising the manuscript.
Corresponding author
Ethics declarations
Ethics approval and consent to participate
This study received human subjects approval from the Indiana University School of Medicine Institutional Review Board (IRB) and the Richard L. Roudebush VA Medical Center Research and Development (R&D) committee. The Indiana University School of Medicine IRB approved a waiver of informed consent given the retrospective cohort design. All study activities and research methods were performed in accordance with the relevant guidelines and regulations.
Consent for publication
Not applicable.
Competing interests
The authors have no competing interests to disclose.
Additional information
Publisher’s Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated in a credit line to the data.
About this article
Cite this article
Myers, L.J., Perkins, A.J., Zhang, Y. et al. Identifying transient ischemic attack (TIA) patients at high-risk of adverse outcomes: development and validation of an approach using electronic health record data. BMC Neurol 22, 256 (2022). https://doi.org/10.1186/s12883-022-02776-1
Received:
Accepted:
Published:
DOI: https://doi.org/10.1186/s12883-022-02776-1