INTRODUCTION

Stroke is a leading cause of hospitalization and is associated with high mortality, hospital readmission rates, and long-term disability.1 As part of the Quality Reporting Program (QRP), the Centers for Medicare and Medicaid Services (CMS) has developed stroke-specific risk-standardized 30-day mortality and 30-day readmission rates.2,3 However, current risk adjustment methods using administrative claims data to capture patient-level condition severity remain far from being adequate and precise.4,5 Administrative claims records have inadequate condition-specific severity and no function-related information, limiting predictive accuracy of the current risk-adjustment methods.6,7 This also results in residual confounding, which ultimately compromises the intended objective of risk adjustment and standardization methods to eliminate patient-level case-mix differences for comparing performances on quality measures (QMs) between hospitals.

The National Institutes of Health Stroke Scale (NIHSS) was developed to quantify stroke severity and has the potential to guide clinical decision making, while improving risk standardization methods. The American Heart Association/American Stroke Association have long recommended the use of the NIHSS as a validated risk assessment tool for capturing stroke condition severity.8,9 In October 2016, CMS implemented the International Classification of Diseases, 10th Revision, Clinical Modification (ICD-10-CM) codes for NIHSS, allowing hospitals to record and report NIHSS score. CMS plans to incorporate NIHSS scores into their risk-standardized 30-day mortality rate for stroke beginning in fiscal year 2023.10,11 Prior research has also shown that admission NIHSS score is a valid method for predicting in-hospital mortality for patients with stroke.12,13,14

Inclusion of the NIHSS has been shown to improve the predictive ability of risk-adjustment methods for 30-day mortality after stroke.14 However, validation of the NIHSS was not done previously using administrative claims data. Another methodological issue is that these studies have computed 30-day hospital readmission and 30-day mortality rates as mutually exclusive events, without accounting for the occurrence of the other (e.g., mortality precluding hospital readmission) in that time window.15,16,17,18 Recent work by Saber and Saver has shown the distribution of NIHSS scores in a national sample using Healthcare Cost and Utilization Project (HCUP) data.19 However, this paper did not examine the predictive ability of the NIHSS for patient outcomes such as 30-day readmission and mortality. The objectives of this study were to (1) examine the discriminant performance of Medicare claim–based NIHSS scores for 30-day mortality and 30-day hospital readmission rates and to (2) examine the association between the NIHSS and 30-day hospital readmissions and 30-day mortality rates while controlling for the occurrence of the other (competing) event during that time window.

METHODS

Data

We used 100% patient-level Medicare inpatient claims data which represent administrative claims for all short-term hospitalization of Medicare fee-for-service beneficiaries from October 1, 2016, to December 31, 2017. We extracted information about all diagnostic conditions, surgical procedures, length of stay, and calculated NIHSS. The claims data file was linked with the Master Beneficiary Summary File (MBSF), the Provider of Services file, and the MBSF: Chronic Conditions Segment data to retrieve patient and hospital characteristics. The Master Beneficiary Summary File contains information on beneficiaries’ sociodemographic characteristics and indicators for Medicare enrollment. The chronic condition segment was used to determine the prior history of stroke. The provider of services file was used to retrieve information on hospital-level data such as urban setting and safety-net status of the hospital.

Study Population

The study cohort included Medicare beneficiaries aged 66 and older on fee-for-service(FFS) plans who were admitted to acute or critical access hospitals between October 1, 2016, and November 30, 2017, with an admitting diagnosis of ischemic stroke. Since NIHSS reporting started in October 2016, our cohort includes data from that time. The diagnosis of ischemic stroke was identified using MS-DRG (061, 062, 063) or ICD-10 CM codes (eTable 1). The sample was restricted to patients who had 9 months of continuous Medicare FFS enrollment before and 1 month after the index hospitalization. Patients enrolled in Medicare based on disability or end-stage renal disease (ESRD), who left the hospital against medical advice, were excluded from our sample (Figure 1). Patients (N=1417) admitted from nursing homes, long-term care hospitals, hospices were excluded. Out of this, 43.4% had minor to moderate stroke, 20.2% had a moderate stroke, 20.5% had moderate to severe stroke, and 15.9% had a severe stroke. The final sample comprised 43,241 FFS patients admitted with acute ischemic stroke.

Figure 1.
figure 1

The derivation of the cohort

Full size image

Claim-Based NIHSS

We calculated the NIHSS from ICD-10CM codes associated with acute hospitalization (eTable 2 in the supplement). NIHSS includes 15 items to evaluate the effect of acute cerebral infarction on the levels of consciousness, language, motor strength, ataxia, dysarthria, and sensory loss.20 The NIHSS is administered and documented within 12 h of arrival to an emergency department in patients with ischemic stroke. NIHSS scores range from 0 to 42, with higher values indicating more severe stroke. Our study used the valid four-level NIHSS categories (score 0–7= minor to moderate stroke, 8–13= moderate stroke, 14–21 moderate to severe stroke, and 22–42 = severe stroke), which have demonstrated excellent discriminant properties for 30-day mortality risk.12,13,14 In Figure 2, we have presented the distribution of NIHSS scores among our sample, and for those patients who died and or were readmitted within 30 days.

Figure 2.
figure 2

Distribution of National Institutes of Health Stroke Scale (NIHSS) scores in Medicare beneficiaries with ischemic stroke in the USA

Full size image

Outcomes

The primary outcomes were (1)all-cause 30-day mortality and (2)all-cause 30-day hospital readmission. The date of death was retrieved from the MBSF, with the 30-day period starting from the date of acute hospitalization discharge. We defined all-cause risk-adjusted hospital readmissions (yes/no) as per CMS definition using the 30-day period from the date of index acute hospitalization discharge.

Patient-Level Covariates

We included lengths of stay during acute hospitalization, intensive care unit (ICU) stay (yes/no), receipt of tissue plasminogen activator (tPA), use of a feeding tube, and Elixhauser comorbidity index. The Elixhauser index consists of 30 medical conditions and has shown a high association with acute hospital mortality, length of stay, and hospital charges.21 Comorbidity indices derived from claims data alone do not fully capture prevalence of chronic conditions.22 Therefore, in addition, we determined history of stroke from the chronic conditions segment. We identified tPA and feeding tube use by ICD10-CM procedure codes. We also categorized post-acute discharge destinations after index acute hospitalization into inpatient rehabilitation facilities (IRF), skilled nursing facilities (SNF), home with home healthcare (HH), and home without HH (reference group). Hospital-level variables included urban/rural, safety-net status, and volume of patients with stroke treated annually.

Statistical Analyses

Descriptive statistics were stratified by NIHSS categories for patient demographic, clinical, and hospital characteristics. Chi-square tests were used for categorical variables and t-tests for continuous variables to examine differences by NIHSS categories. Four different logistic regression models were compared (C-statistics) to examine the discriminative property of the NIHSS to predict 30-day mortality and 30-day hospital readmission. The base model included age, gender, race, and Medicare-Medicaid dual eligibility status. The second model included the NIHSS with the base model. The third model included the Elixhauser comorbidity index with the base model. In the fourth model, we added both NIHSS and Elixhauser comorbidity index. Potential C-statistic values range from 0.5, meaning the model does no better than chance, to 1.0, meaning perfect discrimination, with a value higher than 0.70 is considered clinically relevant.23

Past research has computed and estimated hospital readmission risk without accounting for occurrence of a competing event, such aspost-discharge mortality.15,16,17,18 Competing risks arise where individuals are subject to a number of potential failure events and the occurrence of one event might impede the occurrence of other events. In this context, after acute hospitalization, a patient with stroke might experience hospital readmission, or the patient might die. For readmission, death is a competing event, because hospital readmission can never occur; thus, the risk of readmission is zero. To mitigate this issue, we used competing risk analysis, a specialized form of time-to-event analysis, where the cause-specific hazard function generalizes the hazard function to the competing risks setting, describing the rate of failure from one event (30-day hospital readmission) in the presence of a competing event (30-day mortality) and vice versa. The cumulative incidence function using the Fine and Gray method24 was used to quantify the risk of failure from a particular event type when there are competing risks.

Sensitivity Analysis

We presented the distribution of NIHSS in Medicare beneficiaries by alive and 30-day mortality in Figure 2. Secondly, we performed the competing risk analysis after excluding patients discharged to hospice from acute hospitals. Third, to address the concern that prior stroke may influence the NIHSS score, we performed a subgroup analysis after excluding patients with a previous history of stroke. All statistical tests were 2-sided with a P < .05 indicating statistical significance. All analyses were conducted using SAS, version 9.4 (SAS Institute Inc.).

RESULTS

There were 63,147 Medicare fee-for-service beneficiaries admitted to 2,659 hospitals in the USA for ischemic stroke from October 1, 2016, through November 2017, and reported NIHSS scores. Our final analytical sample includes 43,241 patients with NIHSS after applying exclusion criteria described in Figure 1. Among 43,241 patients, 2,449 (5.6%) patients died during the initial hospital stay and 10.1% died within the 30-day period from the index discharge date. The mean (SD) age of the study population was 80 (8.4) years, 44.8% were male, 82.8% White, 9.1% Black, 3.8% Hispanic, and 16.5% were dual-eligible in Medicare and Medicaid. Patient demographic, clinical, and hospital characteristics categorized by NIHSS severity are presented in Table 1. Most patients (64.6%) were in the minor to moderate stroke category; patients diagnosed in a more severe NIHSS category were more likely to be older, female, and non-White and have had a previous stroke history. Patients in the most severe NIHSS category had the highest proportion of being admitted to an intensive care unit (ICU), having a feeding tube placed, and having a high 30-day mortality rate. Patients in the severe stroke category were also more likely to be treated at safety-net hospitals that treat a high volume of stroke patients. The distribution of 30-day post-discharge mortality and readmission by NIHSS score is presented in Figure 3.

Table 1 Descriptive Characteristics of Medicare Patients with Ischemic Stroke in the USA by NIHSS Categories
Full size table
Figure 3.
figure 3

Distribution of National Institutes of Health Stroke Scale (NIHSS) scores by 30-day readmission and mortality in Medicare beneficiaries with ischemic stroke in the USA

Full size image

Discriminant Property of Claim-Based NIHSS for Patient Outcomes

The C-statistics for each model predicting patient outcomes are presented in Table 2. For 30-day mortality, the C-statistic from the base model was 0.71 (95% CI: 0.70–0.72), which increased to 0.83 (95% CI: 0.82–0.84) with the addition of the NIHSS. This was a larger increase in the C-statistic compared to the addition of the Elixhauser comorbidity index to the base model (C-statistic = 0.74, 95% CI: 0.73–0.75). There was no significant change in C-statistics for mortality after adding both the Elixhauser and NIHSS to the base model. There was no corresponding improvement in C-statistic with either NIHSS or Elixhauser for the 30-day readmission outcome.

Table 2 Comparison of C-statistics for Predicting In-hospital Mortality, 30-Day Mortality, and Hospital Readmissions
Full size table

Competing Risk Analysis

For 30-day post-discharge mortality, using 30-day hospital readmission as a competing event, and minor to moderate stroke as a reference category, there was significantly higher mortality risk for moderate stroke (HR=2.92, 95% CI=2.59–3.29), for moderate to severe stroke (HR=5.49, 95% CI=4.90–6.15), and for severe stroke (HR=7.82, 95% CI=6.95–8.80) reported in Table 3. For 30-day hospital readmission, using 30-day post-discharge mortality as a competing event, and using minor to moderate stroke as a reference category, there was significantly higher readmission risk in the moderate stroke (HR=1.11, 95% CI=1.03–1.20), but significantly lower readmission risk in severe stroke (HR=0.84, 95% CI=0.74–0.95) categories. Figure 4 A and B illustrate the cumulative incidence function that summarizes the competing risk analysis for estimating 30-day mortality and 30-day readmission for the four-stroke severity groups. It indicates a monotonic relationship between NIHSS and mortality, but not with 30-day hospital readmission.

Table 3 Hazard Ratio Estimates from Competing Risk Analysis and Cause-Specific Hazard Models
Full size table
Figure 4.
figure 4

A, B Cumulative incidence functions for stroke severity groups for 30-day mortality and 30-day readmission competing risk analysis. NIHSS categories: 1= 0–7: minor to moderate stroke, 2= 8–13: moderate stroke, 3= 14–21: moderate to severe stroke, 4= 22–42: severe stroke

Full size image

Sensitivity Analyses

When limiting the sample to individuals with prior stroke, we found similar results from competing risk analyses for 30-day mortality (eTable 4 in the Supplementary Appendices). After excluding patients discharged to hospice, there was significantly higher readmission risk only in the moderate stroke (HR=1.16, 95% CI=1.08–1.26) and moderate to severe stroke (HR=1.23, 95% CI=1.12–1.34), as compared to minor to moderate stroke. After excluding patients with prior history of stroke, there was significantly higher readmission risk in the moderate stroke (HR=1.14, 95% CI=1.04–1.26), but not in moderate to severe stroke (HR=1.08, 95% CI=0.97–1.21) and in severe stroke categories (HR=0.87, 95% CI=0.74–1.02) compared to minor to moderate stroke. Additionally, to examine the reasons for 30-day hospital readmission, we reported the top 25 diagnostic conditions (eTable 5 in the Supplement).

DISCUSSION

Our study has three notable findings. First, we found a monotonic relationship and significant association between NIHSS and 30-day mortality after accounting for 30-day readmission as a competing risk. After accounting for 30-day mortality as a competing risk, the risk of 30-day hospital readmission was significantly associated with patients only in NIHSS moderate and moderate to severe stroke categories. Finally, our study also validated the higher predictive ability of ICD-10CM-based NIHSS. Our findings highlight the importance of adding the NIHSS in stroke-specific risk-standardized 30-day mortality rates. As a standardized and validated assessment tool, the NIHSS has been widely utilized in treatment planning in comprehensive stroke centers and clinical trials to monitor treatment effectiveness and quantify patient recovery.25 Therefore, the findings from our study demonstrate the improved discriminatory property of the NIHSS compared to the Elixhauser index and current CMS risk adjustment method.26

Our findings corroborate the previous studies using Get With The Guidelines®-Stroke data and the CMS technical report that demonstrated the inclusion of the NIHSS improved C-statistics to predict 30-day mortality in patients with stroke.27,3 We also found that both NIHSS and Elixhauser were poor predictors of 30-day hospital readmission following discharge from an acute hospital after adjusting for patient-level demographic characteristics. Findings from this and other studies indicate that commonly used risk adjustments by CMS do not accurately predict the risk of hospital readmission.5,22,27 One plausible explanation for the lack of sensitivity of the NIHSS to predict readmission is the fact that the NIHSS was developed to capture neurological impairment during the early acute phase.

Our results have important implications for risk adjustment methods. Despite stroke being the fourth most common cause of mortality, with approximately 15% 30-day mortality in the Medicare population, there was no information about patient-level severity in Medicare and other administrative claims data. More importantly, post-stroke severity has been shown to be a strong predictor of mortality, and also influences timeliness of care delivery such as administration of tPA,28 initiation of intensive physical rehabilitation, and discharge to post-acute care settings.29 Both Charlson and Elixhauser comorbidity indexes have been used extensively in health services studies, but both only use medical diagnosis.21,30 The CMS currently uses the hierarchical condition category (HCC) to develop risk-adjustment scores for individual patients based on an algorithm that includes combinations of diagnoses, patient age, and comorbidities.31 However, past studies have shown that most of these risk adjustment methods could not capture disease severity and demonstrated weak discriminatory property to predict outcomes.22,32,33Condition-specific severity information in Medicare data will maximize predictive accuracy for patient outcomes and minimize hospital penalty risk for treating sicker patients. In the absence of a sensitive risk adjustment, hospitals in the high prevalence regions (“stroke belt”)34,35 and other regions that treat a large number of patients with severe stroke could perform worse on public reporting.

Competing risk analysis allowed us to control for the competing risk of hospital readmission when examining the risk of 30-day mortality, and the competing risk of mortality when examining 30-day hospital readmission. The Fine-Gray model is appropriate for risk modeling about patient prognosis and in its absence, our hazard models created an upward bias of the probability estimates of occurrence of both 30-day mortality and readmission within the NIHSS categories (Table 3), reflecting a caution that has been addressed by previous investigations.36 The use of this analysis has been encouraged for clinical research17,37,38 and competing risk analysis has been applied in patients with hip fractures39 and adults with chronic heart failure.40

Our study has some limitations. First, Medicare claims data do not record the timing of administration of the NIHSS during acute hospitalization. However, CMS strongly recommends hospitals assess and document the first NIHSS within 12 h of arrival at the emergency department. Secondly, since October 1, 2016, hospitals have been reporting ICD-10CM-based NIHSS scores. There were missing scores among low-volume hospitals and non-academic medical hospitals in the earlier months of reporting. However, reporting continues to improve with time. Third, Medicare claims data may have inaccurate coding and potential missing data, although high levels of agreement have been reported in administrative data and medical records.41 Fourth, besides dual enrollment in Medicare and Medicaid, we did adjust for social determinants of health, which can be associated with both readmission and mortality. We have excluded a small number of patients admitted from nursing homes, long-term care hospitals, and hospice. While we did not find differences in demographic characteristics and outcomes, we found that among these groups, the rate of ICU admissions during hospitalization was higher. Our results may not be applicable to patients enrolled in Medicare Advantage plans. Finally, we used a nine-month lookback period to ensure continuous enrollment in FFS than a commonly used 12-month lookback period. Despite these limitations, our study has many strengths. This is the first study to validate the utility of NIHSS in Medicare claims data. Second, we adjusted for both patient and hospital characteristics, minimizing the risk of bias. Third, our study addresses the limitation of previous studies by adjusting post-acute discharge destinations using a combination of claims and post-acute assessment data.

CONCLUSION

To our knowledge, this is the first study to demonstrate the predictive utility of NIHSS using Medicare claims data for 30-day mortality and 30-day hospital readmission for patients with ischemic stroke. This study affirmed a significant association between ICD-10-based NIHSS and 30-day mortality. These findings support the inclusion of the claim-based NIHSS in risk-standardized measures (mortality). Understanding the impact of the NIHSS on patient outcomes has implications for quality reporting programs and improving the transition of care in the era of value-based care for patients with stroke.