INTRODUCTION

The Veterans Health Administration in the Department of Veterans Affairs (VA), the nation’s largest integrated healthcare system, offers comprehensive healthcare services to eligible U.S. military veterans who enroll. Congress, veterans’ groups, and the press have expressed concerns that access to care and quality of care in VA settings are inferior to access and quality in non-VA settings (Veterans Access, Choice, and Accountability Act of 2014, Public Law 113–146).1, 2

Previous studies have found that quality of care provided in VA settings compares favorably to non-VA care systems.3,4,5,6,–7 The most recent systematic review of the published literature found that the VA often—but not always—performed better than or similarly to other systems of care with regard to the safety and effectiveness of care.8 Similarly, a recent study comparing commonly reported measures in VA and non-VA hospitals found that VA hospitals performed better or the same as non-VA hospitals with regard to patient safety, better than non-VA hospitals with regard to mortality and readmissions, and showed mixed performance compared to non-VA hospitals on measures of patient care experiences and behavioral health.9

Building upon this prior work, we assessed quality of both outpatient and inpatient care in VA settings using nationally recognized performance measures from 2013 to 2014. We examined VA’s performance at the national level, assessed variation in performance across VA facilities, and compared performance in VA and non-VA settings in the USA.

METHODS

We analyzed quality measures related to inpatient and outpatient care, comparing VA and non-VA settings. For inpatient measures, we used propensity score matching to identify a subset of non-VA facilities with similar characteristics to VA facilities. For outpatient measures, we compared VA facility performance rates to rates for health plans reported by the National Committee for Quality Assurance State of Health Care Quality Report.10

Quality Measures and Data Sources

To select VA quality measures for inpatient and outpatient settings, we prioritized quality measures that reflect national standards, are reported for non-VA settings by national performance measurement programs (allowing for comparisons), and address safety, effectiveness, and patient-centeredness of care, three of the domains of quality of care outlined by the Institute of Medicine.11 VA and non-VA performance measure data were not available to us for comparing the other three quality domains of timeliness, efficiency, or equity.

Inpatient Measures

To assess patient safety in the inpatient setting, we used Patient Safety Indicators (PSIs) developed by the Agency for Healthcare Research and Quality (AHRQ) to monitor adverse events and complications of care that may occur in the hospital,12 and 30-day risk-standardized mortality and readmission measures developed by the Centers for Medicare & Medicaid Services (CMS) in conjunction with the Hospital Quality Alliance,13 both for the fourth quarter of fiscal year 2014. Complete descriptions of the standard risk adjustments for the 2014 mortality and readmission measures are published elsewhere.14, 15

To assess effectiveness of care in the inpatient setting, we used ORYX measures (also known as the National Hospital Quality Measures) developed by the Joint Commission16 for the fourth quarter of fiscal year 2014.

To assess patient-centeredness of health care received in VA inpatient settings, we used measures from the VA Survey of Health Experiences of Patients (SHEP) for fiscal year 2014.17 We compared VA measures from the SHEP surveys with measures for the fourth quarter of 2014 from the Consumer Assessment of Healthcare Providers and Systems (CAHPS) Hospital Survey (HCAHPS); measures on the inpatient SHEP parallel those of HCAHPS. VA inpatient SHEP data were adjusted for case mix using CMS’s recommended HCAHPS case mix adjustment for Hospital Compare.

For non-VA facilities, data for all inpatient quality measures were downloaded from the CMS Hospital Compare website for the fourth quarter of fiscal year 2014. For VA facilities, data for inpatient quality measures were obtained from three sources. Risk-standardized readmission and mortality rates and ORYX measures were downloaded from the CMS Hospital Compare website for the fourth quarter of fiscal year 2014. AHRQ PSIs were provided by the VA Inpatient Evaluation Center for calendar year 2014. SHEP measures were provided by the VA Office of Performance Measurement for fiscal year 2014.

Outpatient Measures

To assess effectiveness of care in the outpatient setting, we analyzed 16 measures in the Healthcare Effectiveness Data and Information Set (HEDIS®) developed by the National Committee for Quality Assurance that are reported both by VA facilities and by NCQA for health plans nationwide.10 Rates of outpatient effectiveness for VA facilities were compared with rates for three non-VA groups: commercial health maintenance organizations (HMOs), Medicare HMOs, and Medicaid HMOs. The category of HMOs included HMOs, HMO/Point of Service (POS) combined, HMO/Preferred Provider Organization (PPO)/POS combined, HMO/PPO combined, and POS. HEDIS measure data were provided by the VA Office of Performance Measurement for 140 VA facilities providing outpatient care in fiscal year 2013. For non-VA care, HEDIS measure rates were extracted from an NCQA published report for 814 HMO plans during calendar year 2013.10 As there is no nationally representative source of non-VA outpatient patient experience data, we did not compare patient experiences of outpatient care in VA and non-VA facilities. Complete specifications for all measures are available from the authors upon request.

Analysis

Identifying Non-VA Hospitals

To ensure comparability between VA and non-VA hospitals for comparisons of inpatient measure performance, we identified a subset of non-VA hospitals with characteristics similar to VA hospitals. To do so, we conducted propensity-score matching based on the predicted likelihood that a non-VA facility could be a VA facility given certain characteristics (covariates). Hospital characteristics were identified from the 2014 American Hospital Association dataset,18 which includes facility-level characteristics for 135 VA facilities and 6332 non-VA facilities. Seven of 135 VA facilities in the American Hospital Association database could not be linked to the CMS Hospital Compare file and were therefore not included in the analysis of CMS Hospital Compare measures. For matching VA hospitals to non-VA hospitals, we selected four facility characteristics shown to be predictive of performance on Hospital Compare measures19: bed size (< 100 beds, 100–199 beds, and 200+ beds), census division (East North Central, East South Central, Mid-Atlantic, Mountain, New England, Other, Pacific, South Atlantic, West North Central, and West South Central), location (urban, rural), and teaching status (teaching facility, nonteaching facility). A facility was categorized as urban if it is located inside a Metropolitan Statistical Areas (MSA) and rural if outside an MSA18 Teaching facilities were defined to include major and minor teaching hospitals, with a major teaching hospital having a Council of Teaching Hospitals designation and a minor teaching hospital having another teaching hospital designation. Facilities without a teaching hospital designation were classified as nonteaching facilities. Next, we ran a logistic regression model to compute a propensity score for each facility and matched non-VA facilities to VA facilities based on these probabilities. Three non-VA facilities were matched to each VA facility with a maximum allowable absolute difference between propensity scores of 0.0009. There were no significant differences between VA and the matched non-VA facilities for any characteristic in the model, indicating that the two sets were well matched. We analyzed measures for which data were available both for VA hospitals and the non-VA comparison hospitals. In the analysis, if a VA hospital had a missing value for a measure, the non-VA hospitals matched to that hospital were excluded from the analysis of that measure. In addition, if one of the matched non-VA hospitals had a missing value for a measure, the remaining two non-VA hospitals were “up-weighted” by a factor of 3/2 or 1.5, and if two of the matched non-VA hospitals had a missing value for a measure, the remaining hospital was up-weighted by a factor of 3. This ensures that every VA hospital has the equivalent of three matched non-VA hospitals for each measure, and all analyses account for these weights. The number of VA and non-VA hospitals in the analysis also varies because of CMS reporting criteria.20 Performance data may be missing for hospitals if the number of cases did not meet the minimum number required for public reporting; the measure had inadequate reliability; protection of personal health information could not be assured; there were no data for the hospital; or the hospital did not have any patients who met the inclusion criteria for a measure.

Comparing VA and Non-VA Performance

For both inpatient and outpatient measures, national measure performance rates were calculated as means of the facility-level rates. We tested the statistical significance of the difference between each pair of VA and non-VA means using a t test with the P value reported. Standard deviations were not available for non-VA rates for HEDIS® measures; therefore, the VA standard deviation for each measure was used in the t test.

RESULTS

Inpatient Care

For inpatient quality measures, we included 135 VA hospitals in the CMS Hospital Compare database and identified 402 non-VA hospitals in a 3-to-1 match using propensity-score matching; three non-VA hospitals could not be matched because of missing values. None of the characteristics (bed size, census division, location, and teaching status) differed between the VA hospitals and the matched set of non-VA hospitals (Table 1).

Table 1 Comparison of Characteristics of VA Hospitals and Matched Hospitals Outside VA, 2014
Full size table

On quality measures related to inpatient care, VA hospitals performed on average the same as or significantly better than non-VA hospitals on all six measures of inpatient safety, all three inpatient mortality measures, and 12 inpatient effectiveness measures, but significantly worse than non-VA hospitals on three readmission measures and two effectiveness measures (Table 2). Compared to patient-reported experiences in non-VA hospitals, veteran-reported experiences of care in VA hospitals were, on average, significantly worse on six and significantly better on three of ten measures (Table 2). The largest difference where VA inpatient performance was significantly lower was the patient experience measure for pain management (6.6%; P < 0.001). The largest difference for which VA inpatient performance was significantly higher was the patient experience measure for care transition (10.4%; P < 0.001). For reference, differences of 1, 3, and 5 percentage points are sometimes referred to as small, medium, and large for CAHPS measures.21 Figure 1 summarizes how VA performance compares to non-VA performance on the inpatient and outpatient measures in Tables 2 and 3, respectively.

Table 2 Inpatient Quality Measures: Comparison of Performance of VA Hospitals and Matched Hospitals Outside VA, Fiscal Year 2014, Quarter 4
Full size table
Figure 1
figure 1

VA compared to non-VA quality of care, by type of measure and setting. Source: RAND summary of results of VA to non-VA comparisons. Notes: Categories are defined on the basis of statistical tests for difference in means with P < 0.05 or less: VA better = VA quality of care shown to be better than non-VA; same = quality of care in VA and non-VA did not differ; VA worse = VA quality of care was shown to be worse than non-VA. Non-VA comparison data were not available for outpatient measures of patient-centeredness.

Full size image
Table 3 Outpatient Quality Measures: Comparison of Performance of Veterans Affairs (VA) Facilities and Commercial, Medicare, and Medicaid HMOs, 2013
Full size table

Outpatient Care

The performance of VA facilities was significantly better than commercial HMOs and Medicaid HMOs for all of the 16 HEDIS® measures and for Medicare HMOs, it was significantly better for 14 measures and did not differ for 2 measures (Table 3). The smallest difference between VA and commercial HMOs was in the rate of antidepressant medication management during the acute phase (5.9%; P < 0.001), and the largest difference was in the rate of eye examinations for patients with diabetes (34.3%; P < 0.001). The smallest difference between VA and Medicare HMOs was in the rate of antidepressant medication management during the continuation phase (0.3%; not significant), and the largest difference was in the rate of eye examinations for patients with diabetes (21.5%; P < 0.001). The smallest difference between VA and Medicaid HMOs was in the rate of persistence of beta-blocker treatment after acute myocardial infarction (7.5%; P < 0.001), and the largest difference was in the rate of eye examinations for patients with diabetes (36.4%; P < 0.001).

Variation

On average, VA care performed better than non-VA care on most measures. There was wide variation in performance across VA facilities for some measures and even wider variation across non-VA facilities. As examples, there was a 50-percentage point difference in performance between the lowest- and highest-performing VA facilities on the FY 2014 rate of beta-blocker treatment for at least 6 months in patients after discharge for an acute myocardial infarction hospitalization in the outpatient setting (excluding one facility with a measure rate of zero).In addition, all of the FY 2014 patient experience measures for the inpatient setting exhibited large differences between the lowest and highest measure rates for VA facilities, ranging from a 17-percentage point difference for “discharge information” to a 42-percentage point difference for “quietness of hospital environment” (Fig. 2). Notably, variation in performance was even higher across the matched non-VA hospitals on the same measures, ranging from a 39-percentage point difference for “communication with nurses” to an 83-percentage point difference for quietness of hospital environment (Fig. 2).

Figure 2
figure 2

National variation in VA and non-VA performance on patient experience measures for inpatient setting, FY 2014. Sources: VA facility-level data for patient experience measures for FY 2014 were obtained from the VA Office of Performance Measurement. Non-VA facility-level data for patient experience measures for quarter 4 of FY 2014 were obtained from the CMS Hospital Compare website. Notes: Minimum and maximum values for the reporting facilities in each subgroup are represented by the line extending from each bar. An asterisk next to the measure name indicates a statistically significant difference at P < 0.05 or less between VA and non-VA hospitals.

Full size image

DISCUSSION

Using a broad set of measures reported in national performance measurement programs, we found that rates of adherence to recommended processes of inpatient care, risk-adjusted 30-day mortality rates, and four case-mix adjusted patient experience measures in VA hospitals were better or the same as rates in a matched set of non-VA hospitals. However, risk-adjusted readmission rates were significantly worse than those in matched non-VA hospitals, as were six of ten patient experience measures for the hospital setting. We also found that rates of adherence to almost all recommended outpatient processes of care in VA facilities nationwide met or exceeded those in commercial, Medicare, and Medicaid health plans for outpatient care.

Our findings differ somewhat from other recent studies comparing VA and non-VA inpatient care.9, 22 For example, in contrast to a study of Hospital Compare performance data from a similar timeframe,9 we found that readmission rates were higher for AMI, heart failure, and pneumonia in VA hospitals than in non-VA hospitals; such discrepancies may be because the non-VA comparison hospitals in our analysis are a subset of all U.S. hospitals that have been matched to more closely mirror the characteristics of VA hospitals, rather than a broader set of non-VA hospitals.

There is substantial variation in quality measure performance across VA facilities, indicating that Veterans in some areas are not receiving the same high-quality care that other VA facilities are able to provide. This variation is lower than that observed in private-sector health plans and hospitals, and some variations in performance across regions and VA facilities may be inevitable because of differences in patient characteristics. However, promoting more uniform quality of care across VA facilities is important for ensuring that veterans can count on high-quality care no matter which facility they access.

Our study assesses quality of care across a range of health conditions in both inpatient and outpatient settings and addresses several quality domains, including safety, effectiveness, and patient experience. We use data from CMS Hospital Compare for all inpatient safety and effectiveness measures for VA and non-VA hospitals; we used data provided by VA for PSIs to minimize concerns about comparing data collected using different methods and, therefore, subject to different biases. Although the outpatient measures were based on different data sources, the same specifications were used to derive the rates for most measures. Nonetheless, our study has several limitations. First, some quality measures are based on administrative data, which includes limited clinical information, and therefore, individuals might be misclassified as receiving or not receiving the recommended care.23, 24

Second, veterans may differ from patients in non-VA settings in terms of demographic characteristics (e.g., age, gender) and clinical characteristics (e.g., severity of disease, comorbidities, including mental health conditions). However, the effectiveness measures in the study focus on care recommended for all eligible patients; therefore, all patients, regardless of characteristics, should receive the recommended care, and differing patient characteristics should not bias those comparisons. The outcome measures (i.e., mortality and readmissions) that require risk adjustment for unbiased comparison between subgroups are risk-standardized using the CMS Hospital Compare methods for both VA and non-VA hospitals, although some argue that reporting of these measures is imprecise and standardized adjustments are insufficient across all hospitals.25 Variables used in the risk adjustment include age, gender, condition-specific clinical information (e.g., cardiovascular risk factors for acute myocardial infarction), and comorbidities (e.g., hypertension, diabetes, dementia).26

Third, the propensity score matching of non-VA to VA hospitals relied on limited number of variables, so there may be differences between the two groups of hospitals and the patients they serve. In addition, a small number of VA hospitals did not appear in the AHA file and, therefore, could not be matched to non-VA hospitals and were excluded from the analyses of inpatient quality measures. To estimate the effect of this exclusion, we compared the measure rates in excluded VA hospitals to those included and found that the two sets did not differ significantly.

Finally, although our study included a large number of quality measures, there are many aspects of VA care, such as care coordination, that are not represented. In addition, quality measures focus on care provided for veterans who have successfully accessed the system but do not examine the degree to which VA may be able to facilitate Veterans enrolling or gaining access to care in the VA system in a timely manner.

In conclusion, in 2013–2014, on most publicly reported measures, on average, the quality of VA inpatient care was the same as or better than the quality of non-VA inpatient care and, on average, the quality of VA outpatient care was better than the quality of non-VA outpatient care. High variation in the performance of some quality measures was observed across VA facilities and indicates a need for targeted quality improvement to ensure that veterans received uniformly high-quality care at all VA facilities.