Teaching Hospital Five-Year Mortality Trends in the Wake of Duty Hour Reforms
- First Online:
- Cite this article as:
- Volpp, K.G., Small, D.S., Romano, P.S. et al. J GEN INTERN MED (2013) 28: 1048. doi:10.1007/s11606-013-2401-9
- 393 Views
The Accreditation Council for Graduate Medical Education (ACGME) implemented duty hour regulations for residents in 2003 and again in 2011. While previous studies showed no systematic impacts in the first 2 years post-reform, the impact on mortality in subsequent years has not been examined.
To determine whether duty hour regulations were associated with changes in mortality among Medicare patients in hospitals of different teaching intensity after the first 2 years post-reform.
Observational study using interrupted time series analysis with data from July 1, 2000 to June 30, 2008. Logistic regression was used to examine the change in mortality for patients in more versus less teaching-intensive hospitals before (2000–2003) and after (2003–2008) duty hour reform, adjusting for patient comorbidities, time trends, and hospital site.
Medicare patients (n = 13,678,956) admitted to short-term acute care non-federal hospitals with principal diagnoses of acute myocardial infarction (AMI), gastrointestinal bleeding, or congestive heart failure (CHF); or a diagnosis-related group (DRG) classification of general, orthopedic, or vascular surgery.
All-location mortality within 30 days of hospital admission.
In medical and surgical patients, there were no consistent changes in the odds of mortality at more vs. less teaching intensive hospitals in post-reform years 1–3. However, there were significant relative improvements in mortality for medical patients in the fourth and fifth years post-reform: Post4 (OR 0.88, 95 % CI [0.93–0.94]); Post5 (OR 0.87, [0.82–0.92]) and for surgical patients in the fifth year post-reform: Post5 (OR 0.91, [0.85–0.96]).
Duty hour reform was associated with no significant change in mortality in the early years after implementation, and with a trend toward improved mortality among medical patients in the fourth and fifth years. It is unclear whether improvements in outcomes long after implementation can be attributed to the reform, but concerns about worsening outcomes seem unfounded.
KEY WORDSpatient outcomesmortalityduty hour reformACGMEadministrative data
Widespread concern about effects of resident fatigue, including deaths from medical errors,1 prompted the Accreditation Council for Graduate Medical Education (ACGME) to implement regulations limiting number of work hours, effective July 1, 2003, for all ACGME-accredited residency programs.2,3
Previous research found that the duty hour regulations had little impact on adult mortality4–6 2 years after implementation, even among high-risk patients,7 and minimal or inconsistent impact on indicators of patient safety,8 readmission rates,9 failure-to-rescue rates (e.g. death given occurrence of a complication), or the probability of prolonged lengths of stay.10 The ACMGE proposed new duty hour standards in July 201111 that maintain the 80-hour cap on weekly duty hours, but limit intern shifts to 16 hours. Part of the rationale for this change was that the earlier standards were not adequate in improving patient safety and reducing mortality, and tighter regulation would lead to better outcomes.
The long-term impact of the 2003 duty hour regulations on patient outcomes has not been analyzed. This knowledge gap is important, because some programs did not comply with the regulations until it became clear that they would be strictly enforced.12,13 Additionally, if the regulations had beneficial or detrimental effects on resident education, by increasing self-study time or reducing procedural experience,14,15 then changes in patient outcomes may not have been realized until the residents who began training under these regulations became supervisors (i.e., 3rd year or later). Finally, if negative effects of the regulations were observed in the first 2 years following implementation, hospitals may have instituted mitigating policies starting in the third year post-reform. In this study, we examine the impact of the 2003 duty hour regulations on mortality for medical and surgical patients enrolled in Medicare and treated at non-federal hospitals over the subsequent 5 years.
Approval for this study was obtained from the Institutional Review Boards of The University of Pennsylvania and The Children’s Hospital of Philadelphia.
The outcome measure was death within 30 days of hospital admission for all patients admitted for acute myocardial infarction (AMI), gastrointestinal bleeding, congestive heart failure (CHF), general surgery, orthopedic surgery, or vascular surgery.16 There is evidence that mortality varies substantially across institutions for these conditions, and that high mortality may be associated with deficiencies in the quality of care.17,18
Medicare patients were selected using an approach described in previous studies,4 including all Medicare patients admitted to short-term general non-federal acute care hospitals from July 1, 2000 to June 30, 2008 with a principal diagnosis of AMI, gastrointestinal bleeding, CHF or with a diagnosis-related group (DRG) classification of general, orthopedic, or vascular surgery. Stroke, included in earlier analyses of duty hour effects, was excluded because stroke mortality increased before the 2003 regulations in teaching hospitals compared to non-teaching hospitals.4 The initial sample included 18,331,795 patients from 5,045 acute care hospitals that contributed data for all 8 years within the 50 states or Washington DC.
We excluded patients from hospitals that had fewer than 350 Medicare admissions in any year (n = 45,947 patients, 933 hospitals) in order to eliminate hospitals that may not have been acute care facilities and those too small to yield stable estimates in a fixed effects analysis. We also excluded patients from hospitals that did not have Medicare Cost Report data (n = 71,947 patients, 115 hospitals); or that were missing more than 3 months of data in the pre-reform period or 5 months of data in the post-reform periods (n = 543,223 patients, 938 hospitals).
We also excluded patients younger than 66 years of age to allow 180-day look back for risk adjustment (n = 2,455,361), or over 90 years of age (n = 1,075,413), because the changes in the proportion of such patients treated aggressively may not be reflected in administrative data; patients whose hospitalizations spanned July 1, 2003 (n = 28,557); patients who were enrolled in an HMO at any time during the study period (n = 326,412); patients who had a reported date of death before an admission date (n = 752); or patients who were transferred in from another hospital for the qualifying admission (n = 23,509) to avoid double counting admissions within 30 days. Among AMI patients, those discharged alive in fewer than 2 days (n = 81,718) were excluded because such cases may not represent actual AMIs.19–21 All admissions for patients who met the above conditions were included, leaving a sample of 13,678,956 patients from 3,059 hospitals.
Risk Adjustment and Data
Data on patient characteristics were obtained from the Medicare Provider Analysis and Treatment File (MEDPAR), which includes information on principal and secondary diagnoses, age, sex, comorbidities, and discharge status, including dates of death. Data on health maintenance organization (HMO) enrollment were obtained from the Center for Medicare and Medicaid Services (CMS) denominator files. The number of residents at each hospital was obtained from CMS Medicare Cost Reports.23
The primary measure of teaching intensity was the resident-to-bed ratio, calculated at a defined point in time as (number of interns + residents)/mean number of staffed beds. The resident-to-bed ratio has been used to differentiate major teaching, minor teaching, and non-teaching hospitals in previous studies,24–26 and we used a similar approach in previous work.4,5,8–10
We used a multiple time series research design,27 also known as “difference-in-differences,” to examine whether the change in duty hour rules was associated with a change in the underlying trend in patient outcomes in teaching hospitals, an approach that reduces potential biases from unmeasured variables.28,29 The multiple time series research design compares each hospital to itself, contrasting the changes in hospitals with more residents to the changes in hospitals with fewer or no residents, and adjusting for observed differences in patient risk factors. The design also adjusts for changes in outcomes over time (trends) common to all hospitals and minimizes bias from three possible sources. First, a difference between hospitals that is stable over time cannot be mistaken for an effect of the reform because hospital fixed effects are used to compare each hospital to itself, before and after reform. Second, universal changes affecting all hospitals similarly, such as technological improvements, cannot be mistaken for an effect of the reform because the logit model includes year indicators. Third, if the mix of patients is changing differently among hospitals, and if these changes are accurately reflected in measured risk factors, this cannot be mistaken for an effect of the reform because the logit model adjusts for these measured risk factors.
Notwithstanding these advantages, the difference-in-differences method has limitations. Any diverging trend in mortality over time for more vs. less teaching-intensive hospitals already in progress or coincident with the initiation of the reform could be mistaken for an effect of the reform, although we adjusted for any observed differences in pre-reform trends. Less teaching-intensive hospitals, including all non-teaching hospitals, served as the primary control group because they were less affected by the duty hour reform but were subject to the same technological quality improvement imperatives, changes in market conditions, and Medicare-specific initiatives, such as ‘pay for performance.’ They are also geographically diverse with large patient populations, and similar patient discharge data are available. Data from July 1, 2000 to June 30, 2003 were designated the pre-reform period, with data from July 1, 2003 to June 30, 2008 designated the post-reform period.
Whereas our previous analyses utilized one observation per patient over a 5-year period,4,5,7–10 we now utilize all admissions since multiple admissions per patient are more common during the 8-year time period of this study.30 To estimate confidence intervals and p values that account for the correlation between patient observations, we utilized the bootstrap procedure.28,29,31
The dependent variable was death within 30 days of hospital admission, using logistic regression to adjust for patient comorbidities, secular trends common to all patients (for example, due to general changes in technology), and hospital site. The effect of the change in duty hour rules was estimated using the coefficients of resident-to-bed ratio interacted with dummy variables indicating post-reform years 1, 2, 3, 4, and 5 (e.g. post-reform year 1 was from July 1, 2003 to June 30, 2004). These coefficients, presented as odds ratios, measure the degree to which mortality changed in more vs. less teaching-intensive hospitals after adjusting for cross-sectional differences in hospital quality and general improvements in care. They were measured for each year separately because of the possibility of either delayed beneficial effects or early harmful effects. Conditions were assessed both individually and together as “combined medical” and “combined surgical” groups.
We used a similar modeling strategy as in our previous work.4,5,7–10 Briefly, risk-adjusted mortality was assumed to have a common time trend until implementation of the duty hour rules, after which the trend at teaching hospitals was allowed to diverge. To assess whether underlying trends in risk-adjusted mortality were similar in higher and lower teaching intensity hospitals during the 3 years prior to the duty hour reform, a “test of controls” was performed. Parameters were added to the model for interactions between the resident-to-bed ratio and indicators for pre-reform year 2 and pre-reform year 1. A Wald test was used to determine whether these interactions significantly improved model fit, suggesting that teaching and non-teaching hospitals had differing trends in mortality in the 3 years pre-reform independent of the reform. When such differing trends were found, post hoc analyses were conducted in which post-reform results were compared to pre-reform year 1 as a baseline instead of the entire 3-year pre-reform period.
Unadjusted Mortality Rates by Condition and Year Relative to the Implementation Date of ACGME Duty Hour Regulations in July 2003
Pre 3 (2000–1)
Pre 2 (2001–2)
Pre 1 (2002–3)
Post 1 (2003–4)
Post 2 (2004–5)
Post 3 (2005–6)
Post 4 (2006–7)
Post 5 (2007–8)
Number of cases
Unadjusted mortality rate (%)
Number of cases
Unadjusted mortality rate (%)
Number of cases
Unadjusted mortality rate (%)
Number of cases
Unadjusted mortality rate (%)
Number of cases
Unadjusted mortality rate (%)
Number of cases
Unadjusted mortality rate (%)
Hospitals and Admissions by Teaching Status
Very minor teaching (> 0–0.049)
Minor teaching (0.050–0.249)
Major teaching (0.250–0.599)
Very major teaching (0.600–1.090)
Number (%) of hospitals*
2070 (67.7 %)
296 (9.7 %)
394 (12.9 %)
193 (6.3 %)
106 (3.5 %)
Number (%) of admissions
9,819,358 (49.9 %)
2,818,337 (14.3 %)
4,153,529 (21.1 %)
1,904,338 (9.7 %)
967,884 (4.9 %)
Excluding patients admitted to hospitals in New York State (a state that had instituted resident hour reform prior to our reference years) or patients admitted from nursing homes (whose care may be less aggressive) did not change the results. We also examined whether changes in the coding of comorbidities could explain any of these effects. There was only a 0.8 % relative decrease in the mean number of comorbidities in more teaching-intensive hospitals relative to non-teaching hospitals from pre-reform year 1 to post-reform year 5, and sensitivity analyses without adjustment for comorbidities produced similar results. As in previous work,7 we did not find any significant differences in the degree to which mortality changed in more vs. less teaching intensive hospitals in any of the five post reform years for high vs. low-risk patients for either the combined medical group or the combined surgical group (results in Appendix).
To illustrate the magnitude of the changes in mortality, we estimated the adjusted risk of mortality in each academic year for a hypothetical patient with mean values of all regression covariates at two hypothetical hospitals: a non-teaching hospital with a resident-to-bed ratio of 0, and a highly teaching intensive hospital with a resident-to-bed ratio of 1. In the combined medical group, mortality in a hospital with a resident-to-bed ratio of 1 decreased from 7.56 % in pre-reform year 1 to 5.48 % in post-reform year 5, while in a hospital with a resident-to-bed ratio of 0 mortality decreased from 9.09 % to 7.47 %, a relative decrease of 0.46 percentage points or 5.06 %. For a patient representing the combined surgical group, mortality in a hospital with a resident-to-bed ratio of 1 decreased from 1.50 % in pre-reform year 1 to 1.15 % in post-reform year 5, while in a hospital with a resident-to-bed ratio of 0, mortality decreased from 2.06 % to 1.76 %, a relative decrease of 0.04 percentage points or 1.94 %.
Implementation of the 2003 duty hour rules was arguably one of the largest efforts undertaken to reduce errors in teaching hospitals. The results of this study confirm that there were no systematic effects on mortality in the first 3 years post-reform, but suggest a relative improvement in mortality at high teaching intensity hospitals by the 5th year post-reform among both medical and surgical patients. The absence of relative changes in mortality in the first 3 years post-reform suggests that the duty hour regulations did not have any direct effect on mortality. The outcomes in high-risk patients were similarly unaffected in the first 3 years, further suggesting that the regulations had no direct impact on mortality. Improvements in mortality in post-reform years 4 and 5 suggest that the reform did not harm patients, although we are hesitant to attribute these delayed effects to the reform since it cannot be determined whether the improvements would have been smaller or greater had no reform been implemented.
Other major concurrent policy changes of differential impact on teaching and non-teaching hospitals could play a role in the observed outcomes. For example, Section 501(b) of the Medicare Prescription Drug, Improvement, and Modernization Act (MMA) of 2003 authorized CMS to penalize hospitals that did not report designated quality measures by reducing their annual market basket update. The Deficit Reduction Act of 2005 increased that reduction from 0.4 % to 2.0 %, leading to a rapid increase in participation.31 During the same period, 225 hospitals joined the Premier Hospital Quality Incentive Demonstration, which was established to reward hospitals demonstrating high quality in several of the clinical domains studied here: AMI, CHF, pneumonia, and surgical care.32 The MMA improved the financial attractiveness of Critical Access Hospital designation, such that 1,327 rural non-teaching hospitals have now pulled out of the Prospective Payment System and its attendant public reporting requirements.33 With respect to surgical care, concurrent interventions include the roll-out of the National Surgical Quality Improvement program (NSQIP) from the VA to the private sector and the implementation of The Joint Commission’s Surgical Care Improvement Project (SCIP). Although nationwide initiatives, it is possible that they were differentially adopted by hospitals at different levels of teaching intensity. These concurrent changes make it impossible to determine the incremental impact of any single intervention, but we can clearly say that mortality did not worsen following implementation of the 2003 duty hour rules.
The absence of any clinically important relative change in mortality in the first 3 years post-reform in more teaching intensive hospitals may reflect many factors. It is possible that mortality increased following the rule change, and that any negative effects of the reform were offset by technological improvements or more efficient management of handoffs. Such factors cannot be disentangled given the nature of observational data; we can only study the net change. Lack of compliance with the duty hour rules has also been described, although such surveys are limited to the self-reports of a small and potentially non-representative sample of about 7 % of all interns,34 or data collected by the ACGME.35 Strong financial incentives to avoid loss of accreditation deter widespread resistance to the regulations. Unless appropriate institutional resources were put in place to mitigate the impact of reducing resident work hours, the effect of the reform at the local level would likely be to increase work intensity, which could offset potential benefits of decreased fatigue.
There are several limitations to our study. We included in part an LOS-based definition of AMI historically used to rule out misclassifications of AMI.19–21 Further, we focused on one outcome, mortality. While the duty hour rules were an attempt to reduce deaths from medical errors, measurement of other outcomes such as complications may elucidate the relative effects of decreased continuity of care compared to decreased resident fatigue. Since we do not have information on hours worked by each resident, our study should be considered an effectiveness, not an efficacy, study because we measured the outcomes associated with the duty hour rules as implemented. Even with the size of the Medicare population, some of the confidence intervals are still broad, and we cannot rule out small but clinically meaningful effects. Any observational study is susceptible to unmeasured confounding, which is a particular challenge given the duration of this study. We used administrative data, so risk adjustment is more limited than with clinical data; however, by comparing outcomes over time within each hospital, in more versus less teaching-intensive hospitals, potential bias from unmeasured confounders is markedly diminished. To be a confounder in our difference-in-differences analysis, an unmeasured risk factor must have changed at different rates in more versus less teaching-intensive hospitals. However, survival bias associated with admissions resulting in death being differentially more likely post-reform than pre-reform in teaching hospitals vs. non-teaching hospitals could still affect this study’s results.4
In summary, implementation of duty hour regulations was associated with a significant improvement in risk-adjusted mortality for medical patients in more teaching intensive hospitals in post-reform years 4 and 5 and among surgical patients in post-reform year 5. These results do not address whether the design of duty hour rules is optimal.36 However, they do suggest that the 2003 regulations had little initial effect on risk-adjusted mortality, but that quality, as measured by 30-day mortality, did begin to improve at greater rates in more teaching intensive hospitals several years thereafter. Further empirical work should examine whether similar patterns are observed in conjunction with the duty hour reform of 2011.
This work was funded by grant R01 HL094593-01 from the NHLBI.
Conflict of Interest
The sponsors/funders have had no role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; and preparation, review, or approval of the manuscript. There are no known financial conflicts of interest among any of the authors, including but not limited to employment/affiliation, grants or funding, honoraria, paid consultancies, expert testimony, stock ownership or options, and patents filed, received or pending. Both Dr. Volpp and Dr. Bellini served as unpaid members of the Committee on Innovations for the ACGME from 2005 to 2009.