Variation in Internal Medicine Residency Clinic Practices: Assessing Practice Environments and Quality of Care
- First Online:
- Cite this article as:
- Mladenovic, J., Shea, J.A., Duffy, F.D. et al. J GEN INTERN MED (2008) 23: 914. doi:10.1007/s11606-008-0511-6
- 95 Views
Few studies have systematically and rigorously examined the quality of care provided in educational practice sites.
The objectives of this study were to (1) describe the patient population cared for by trainees in internal medicine residency clinics; (2) assess the quality of preventive cardiology care provided to these patients; (3) characterize the practice-based systems that currently exist in internal medicine residency clinics; and (4) examine the relationships between quality, practice-based systems, and features of the program: size, type of program, and presence of an electronic medical record.
This is a cross-sectional observational study.
This study was conducted in 15 Internal Medicine residency programs (23 sites) throughout the USA.
The participants included site champions at residency programs and 709 residents.
Abstracted charts provided data about patient demographics, coronary heart disease risk factors, processes of care, and clinical outcomes. Patients completed surveys regarding satisfaction. Site teams completed a practice systems survey.
Chart abstraction of 4,783 patients showed substantial variability across sites. On average, patients had between 3 and 4 of the 9 potential risk factors for coronary heart disease, and approximately 21% had at least 1 important barrier of care. Patients received an average of 57% (range, 30–77%) of the appropriate interventions. Reported satisfaction with care was high. Sites with an electronic medical record showed better overall information management (81% vs 27%) and better modes of communication (79% vs 43%).
This study has provided insight into the current state of practice in residency sites including aspects of the practice environment and quality of preventive cardiology care delivered. Substantial heterogeneity among the training sites exists. Continuous measurement of the quality of care provided and a better understanding of the training environment in which this care is delivered are important goals for delivering high quality patient care.
KEY WORDSpractice-based learningsystems-based practicequality of carepreventive cardiologyInternal Medicine residency
Throughout the course of an academic year, over 21,000 residents in Internal Medicine provide ongoing comprehensive care to a panel of ambulatory patients.1,2 Residents provide most of this care during 1 half-day weekly continuity clinic in sites that include community, hospital-based or Veterans Health Affairs clinics, faculty group practices, and private physician offices. Yet, reports from the last 15 years note how most Internal Medicine residents feel unprepared to provide outpatient care at the completion of training. Recently, 4 important reports call for urgent reform to the ambulatory education of residents, express concern that residents too often train in “dysfunctional” ambulatory clinics, and argue that residents should train in high functioning outpatient settings in order to learn how to deliver care effectively and efficiently.3–6
However, little systematic and methodologically rigorous information has been gathered on the quality of care provided by residents in ambulatory training sites.7 A few studies have examined some aspects of the quality of care delivered in residency clinics but were limited to single institutions and small numbers of patients.7–10 Less is known about the characteristics of the clinical microsystems, i.e., the working front-line units in which residents provide patient care.
To this end, this study uses a web-based tool developed by the American Board of Internal Medicine (ABIM) for its Maintenance of Certification™ (MOC) program and adapted for residency practices. The ABIM, in collaboration with the Alliance for Academic Internal Medicine (AAIM), implemented this pilot study using the Preventive Cardiology Practice Improvement Module (PC-PIM) to learn more about the practice environment and quality of care provided in 23 ambulatory training sites of 15 diverse Internal Medicine training programs.
The goals of this paper are as follows: (1) to describe the patient population cared for by trainees in internal medicine residency clinics; (2) to assess the quality of preventive cardiology care provided to these patients; (3) to characterize the practice systems that currently exist in Internal Medicine residency clinics; and (4) to examine the relationships between quality, practice systems, and features of the program [size, type of program, and presence of an electronic medical record (EMR)].
In February 2004, a request for applications was issued for a joint ABIMF/AAIM project, titled the Resident and Faculty Practicum in Practice-Based Learning and Improvement. From 24 applicants, 15 residency programs (comprising 23 unique ambulatory training sites) were funded for an 18-month feasibility project to implement the PC-PIM in training programs. Each program received approval from their institutional review board. Designated program champions participated in a 2-day orientation and quality improvement training session in June 2004. This paper reports results of clinical, patient survey, and practice system data collected during the initial phase in fall 2004.
The 15 residency programs were selected based on size and type of program, geographic location, qualifications of project champion, strength of support letter, and assessment of potential for completion. Seven programs utilized more than 1 training site; therefore, the unit of analysis is the 23 clinic sites.
The PC-PIM is a web-based tool whose purpose is to help physicians better understand and make routine use of the patient and systems data collected from their practice in an effort to improve the quality of care delivered to patients and is closely linked to the Accreditation Council for Graduate Medical Education’s competency goals of practice-based learning and improvement and systems-based practice.1 The tool identifies relevant process and outcome measures based on evidence-based national guidelines of care with broad acceptance from most constituencies.11–14 Participating residents performed chart reviews of a subset of their patients to provide data for calculating measures of preventive cardiology care; the program obtained surveys from patients to assess the presence of and satisfaction with preventive services in the resident-staff clinics, and residents completed one site-level survey that described the practice systems. Power analysis for estimating required sample size to detect differences among the sites for type and size of site and type of medical record determined that 7 chart reviews and 5 patients’ surveys per resident would be sufficient. Patients included were required to have been in the practice for at least 1 year, seen in the last 12 months, and management decisions about their preventive cardiology care made by providers in the practice. The PC-PIM can be viewed in its entirety at www.abim.org/online/pim/demo.aspx.
Residents, assisted by a research assistant, recruited 5 of their patients to complete a survey.15–16 Questions addressed patients’ perspective on care, self-perceived health status, and two subscales: (1) satisfaction with the practice including overall satisfaction with delivery of preventive cardiology care, specific information about prevention, or side effects of prescription medication; (2) access to practice including ease of obtaining appointments, referrals, and prescription refills.
Residents were expected to abstract charts for 7 of their patients. These were not necessarily the same patients who were surveyed. The abstraction form contained the following: (1) patient demographics; (2) the presence or absence of cardiovascular disease (CVD); (3) the presence or absence of risk factors for coronary heart disease (CHD); 4) whether patient barriers to self-care were present, absent, or not known; (5) the presence or absence of processes of care performed (e.g., lipid testing according to guidelines, blood pressure recording, prescribing aspirin); and (6) clinical outcomes such as the result of most recent lipid profile. Although residents were instructed to abstract information from charts, they were not required to strictly report what was recorded and, therefore, could supply answers from knowledge of the patient or by inferences made from other chart information. For instance, a resident could have answered that he/she advised a patient to stop smoking without it being formally recorded in the chart.
Individual measures such as hypertension were scored dichotomously for each patient; a “1” was equivalent to “yes” and signifies its presence, and a “0” was equivalent to “no” and signifies its absence. For each site, the percent of patients with hypertension was calculated by summing the “1s” and dividing by the total number of patients with recorded data for the variable (0s or 1s) and converting the fraction to a percent [e.g., (5/20) × 100 = 25%].
Summary measures consider the individual measures in a particular category together. The average percent of measures present in a category was calculated and used as an overall assessment of patient health. Summary measures were calculated first at the patient level, then the site level. For example, the summary measure “prevalence of risk factors for CHD” consisted of 9 individual measures. For each patient, it was scored by summing the number of risk factors present (1) and then dividing by the total number of risk factors with recorded data (0s or 1s) and converting the fraction to a percent. For example, 33.3% represents that 3 of the 9 risk factors were present for the patient. For each site, the average overall patient percents represent the site mean, i.e., the average percent of risk factors for the site.
Practice System Survey
The site practice champion and the residents completed a survey assessing key structural elements of the clinic’s practice systems. The Practice System Survey was developed by two of the authors (FDD and LAL) [AU1]based on the principles of the Wagner Chronic Care Model, the Institute for Healthcare Improvement Idealized Office Design project, and Putting Prevention into Practice monograph from the Task Force On Clinical Preventive Services.17–19
Six broad categories of practice system elements were included: care management (26 questions), patient-activation and communication (22 questions), modes of communication (9 questions), practice-based learning and improvement (5 questions), information management (12 questions), and environment (10 questions). The response categories for the scale for each question were “yes” signifying that the particular element was operating well in the practice, “maybe” signifying that the element could be improved, and “no” signifying that the element was not available or operational. Subscale scores were derived as the percent of questions that received “yes” responses out of the total number of non-missing responses.
Descriptive statistics including means, standard error of the mean, and range were used to describe the study variables. The unit of analysis was the site level. Differences in processes, outcomes, and systems of care were compared for program type (community or university), presence of an EMR (yes or no), and size of program (small or large) using multivariate analysis of variance with a p value < .01. Pearson correlations were computed between all site level measures. Cronbach’s alpha reliability coefficient measured the internal consistency reliability of the patient survey, the practice system survey, and its subscales.
Risk adjustments were done using generalized estimating equations (patients nested within site) with logistic regression for binary measures and normal linear regression for continuous measures with the assumption that the pattern of correlations among patients nested within sites is exchangeable among sites. Risk adjustments were done for the dependent variables: clinical outcome measure goals [diastolic and systolic BP, low- (LDL) and high-density lipoprotein (HDL) cholesterol, and triglycerides], patient satisfaction, and patient access to practice. The independent measures were patient age, patient gender, and physician’s assessment of patient’s factors limiting self-care.
A total of 709 (96% of all) residents participated, with an average of 31.3 residents (SD = 19.2) per site. Participants were distributed evenly across the 3 years of training (31%, 33%, and 34%, respectively). Overall, 57% of residents were male. Type of residency program was self-described by program directors; 8 of the programs (with 13 sites) were community, 6 (with 9 sites) were university, and 1 was military. Ten sites reported having an EMR. “Size” was dichotomized at the median of 32 residents with 11 sites above the median (range, 4–73). A total of 4,783 charts were abstracted, an average of 6.7 per resident. A total of 3,092 patients responded to the patient survey, an average of 4.4 per resident. The chart review sample was similar to the sample of patient survey responders in terms of gender (44.6% male vs 42.4% male, NS) and prevalence of current smoking (22.0% vs 23.2%, NS) but was slightly older (average 56.0 years vs 52.7 years, p = .05).
Patient Sociodemographics and Risk Factors
Chart Review: Patient Characteristics (n = 23 Sites Based on Data for 4,783 Patients)
Factors limiting self-care
Problems with adherence
Other medical conditions
Psychiatric illness/cognitive impairment
Prevalence of CVD*
Prior myocardial infarction
Other clinical CHD†
Symptomatic carotid artery disease
Peripheral artery disease
Abdominal aortic aneurysm
Prevalence of risk factors for CHD‡
Age (men ≥45, women ≥55)
Overweight or obesity (BMI ≥ 25)
Elevated LDL cholesterol or on lipid-lowering medication
Low HDL cholesterol (<40 mg/dL)
Current cigarette smoking
Family history of premature CHD
Table 1 also reports 5 CVD categories and 9 possible risk factors for CHD. The prevalence is a measure of the health of the patients and reflects an average (in percent) of how many of the categories or risk factors patients have. Patients had an average of 3 to 4 (41.2%) of the 9 potential risk factors for CHD, whereas 65.9% had hypertension only. The wide range across sites in risk factors is striking, often between 30 and 60 percentage points for a given risk factor. On average, only 4.8% of patients (range, 0.4–16.7%) were very healthy having none of the 9 CHD risk factors, whereas 55.0% (range, 29.2–75.7%) had four or more.
Processes of Care
Chart Review: Processes of Care (n = 23 Sites Based on Data for 4,783 Patients)
Diabetes screen in high-risk patients
Timing of lipid panel complies with guidelines
Interventions—Medications prescribed as indicated
Interventions—Lifestyle modifications recommended as indicated
Smoking cessation (for current smokers)
Weight reduction/calorie restriction
Interventions—Dietary recommendations as indicated
Dietary saturated fat and cholesterol restriction
Dietary sodium restriction
Increased fruits, vegetables and/or soluble fibers
Risk-Adjusted Outcomes of Care
Risk Adjusted Outcomes of Care (n = 23 Sites)
Measure of clinical outcomes*
Diastolic BP <85 mm Hg
HDL cholesterol >40 mg/dL
Triglycerides <150 mg/dL
Systolic BP <130 mm Hg
LDL cholesterol at goal†
Access to practice§
Not a problem
Practice Systems of Care
Practice Systems of Care (n = 23 Sites)
No. of questions
Overall systems score*
Environment (teamwork, efficiency and consultant relationships)
Modes of communication (whether the system tracks scheduling issues, encourages patients to contact the practice, and gives proper guidance in receiving care in urgent situations)
Practice-based learning and improvement (how the practice team evaluates the data to improve the outcomes and processes of care)
Information Management (use of structured data entry through templates, use of clinician reminders, use of connectivity with pharmacies, and electronic health records)
Patient activation and communication (patient education, counseling methods and resources, and means of self-care)
Care management (chronic care management activities designed to keep high-risk patients from subsequent hospitalization, systematic follow-up of patients, and planning preventive services)
For each category, on average, about half were reported as operating well in practice. However, as is apparent from the overall score range (18.0–81.0%), some sites had few, whereas others had many systems elements considered ideal for delivering quality care.21–26 More quality environment elements were working well (54.8%) than care management elements necessary in caring for complex patients (41.3%). Even “low-tech” elements were often reported as functioning poorly; for example, 10 clinics (45%) did not have well-functioning standard problem lists in medical records.
Variation Among Program Sites
A final set of analyses explored if variability in program site performance on appropriate intervention (Table 2), adjusted outcomes of care (Table 3), or presence of systems of care (Table 4) was related to size and presence of an EMR. Smaller clinic sites (n ≤ 32 residents) had higher modes of communication scores than larger sites (69.9% vs 48.6%, P = .001). Clinic sites with an EMR had higher information management (80.7% vs 27.34%, P < .001) and modes of communication scores (79.4% vs 43.3%, P < .001).
The quality of the ambulatory care experience and the characteristics of patients of internal medicine residents are described using a web-based tool that assessed the quality of the clinic’s microsystem and the preventive cardiology care provided in their weekly continuity of care practices. Quality of care was measured by a set of evidence-based process and outcome measures, patient satisfaction, and presence of practice-based systems known to support quality care. The variability in performance on these measures differed for certain program features (size, type, and presence of an electronic record).
Most notable was the variability in clinic practices among the 23 training sites. The good news is that nearly half (10 of 23) of the clinic sites have EMRs, compared to less than 20% of practicing physicians nationally.27 As anticipated, when an EMR was present, there was better overall information management in the clinic (81% vs 27%) as well as better systems for patient access and communication between visits (79% vs 43%). Interestingly, there was also better access and use of multiple modes of communication in smaller as compared to larger sites (70% vs 49%). Surprisingly, neither process nor outcome quality measures differed with the presence or absence of an EMR.
The low frequency of functional basic “technologies” in many of the sites is salient. For example, the mean score for systems to facilitate information management about chronic illnesses (e.g., problem lists, templates, and reminders) was only 48%, with a sobering range of 0% to 92% among sites. Likewise, the mean score for patient activation and communication, a measure of systems for promoting self-care, was only 46%, with a range of 22% to 83%. Literature suggests that a better understanding of the elements that comprise functional systems will help to improve processes of care.17 Residents then who work in an effective system and understand the essential concepts of systems-based practice are more likely to be adequately prepared to engage in or implement more effective systems upon graduation from residency.
Process and outcome measure performance of quality in preventive cardiology care varied widely and was sometimes disturbingly low. Appropriate use of guideline-based medications ranged from 33% to 100%. Compliance with lifestyle interventions showed the greatest variability across sites and processes (7% for dietary sodium restriction to 91% for smoking cessation support). Overall, patients received an average of 57% (range, 30% to 77%) of the appropriate interventions, surprisingly similar to a national analysis of quality at the patient level.28
This study importantly examined quality from the perspective of patients’ care experience and showed relatively little variability in responses between programs (Table 4). Patients rated access relatively high (79%), but results show ample opportunities for improvement in patient communication and satisfaction.
What do these results tell us about the current state of residency experience in preventive cardiology care in the ambulatory setting? Although direct comparisons are not possible,29 many challenges in delivering high quality care in training sites mirror those in the external practice environment.24,25 Variability in practice-based systems and patient populations among ambulatory training sites has heretofore not been systematically documented in a multi-site study. The extent of variability is of concern and, although perhaps expected, has been underappreciated.
Interestingly, this variability did not track with the type of residency program nor were patient satisfaction and practice systems of care related to other quality measures. Factors such as faculty supervision, faculty or trainee knowledge, local micro-systems, institutional competence and attitudes, practice patterns, or some unmeasured aspect of the patient population (i.e., presence or absence of insurance) were not studied and may explain some of the variability.
This study has several limitations. First, the programs were not randomly selected, and these findings may not be able to be generalized to other training programs, but this is one of the first studies to examine care and experience across multiple and diverse residency programs. Furthermore, this selection bias would conceivably favor programs with better performance, but the wide range of performance speaks to the contrary. Second, most data came from patient charts, selected and abstracted by residents, but there was no obvious incentive to over-report performance because resident-level identification was not tracked and data were not verified. However, in a study of ABIM’s diabetes practice improvement module, strong agreement was found between trained abstractors and physician audits of the same medical records.30 Furthermore, low rates of compliance for many measures further suggest that significant gaming did not occur. Third, these programs did receive modest compensation for participation. Fourth, the available risk adjusters are inadequate; socioeconomic and race classifications and a more robust measure of co-morbidity are necessary.
To date, this is one of the largest studies of ambulatory care practices among a diverse group of residency programs, involving over 700 residents and 4,700 patients. The study provides a methodological approach for gaining insight into the current state of practice in the residency sites including aspects of the clinic’s environment and the quality of preventive cardiology care. More research is needed to help faculty and institutions understand the variation in systems and patient care in the training environment with the ultimate goal to deliver effective care to the unique population.
This work was supported by an American Board of Internal Medicine Foundation (ABIMF) grant but does not necessarily reflect the views of the ABIMF. The authors would like to acknowledge the Alliance for Academic Internal Medicine (AAIM) for their collaboration on this project. ABIMF paid AAIM $28,000 to support their work on the project.
Conflict of Interest Statement: