Practice Context Affects Efforts to Improve Diabetes Care for Primary Care Patients: A Pragmatic Cluster Randomized Trial
- First Online:
- 805 Downloads
Efforts to improve primary care diabetes management have assessed strategies across heterogeneous groups of patients and practices. However, there is substantial variability in how well practices implement interventions and achieve desired outcomes.
To examine practice contextual features that moderate intervention effectiveness.
Secondary analysis of data from a cluster randomized trial of three approaches for implementing the Chronic Care Model to improve diabetes care.
Forty small to mid-sized primary care practices participated, with 522 clinician and staff member surveys. Outcomes were assessed for 822 established patients with a diagnosis of type 2 diabetes who had at least one visit to the practice in the 18 months following enrollment.
The primary outcome was a composite measure of diabetes process of care, ascertained by chart audit, regarding nine quality measures from the American Diabetes Association Physician Recognition Program: HgA1c, foot exam, blood pressure, dilated eye exam, cholesterol, nephropathy screen, flu shot, nutrition counseling, and self-management support. Data from practices included structural and demographic characteristics and Practice Culture Assessment survey subscales (Change Culture, Work Culture, Chaos).
Across the three implementation approaches, demographic/structural characteristics (rural vs. urban + .70(p = .006), +2.44(p < .001), −.75(p = .004)); Medicaid: <20 % vs. ≥20 % (−.20(p = .48), +.75 (p = .08), +.60(p = .02)); practice size: <4 clinicians vs. ≥4 clinicians (+.56(p = .02), +1.96( p < .001), +.02(p = .91)); practice Change Culture (high vs. low: −.86(p = .048), +1.71(p = .005), +.34(p = .22)), Work Culture (high vs. low: −.67(p = .18), +2.41(p < .001), +.67(p = .005)) and variability in practice Change Culture (high vs. low: −.24(p = .006), −.20(p = .0771), −.44(p = .0019) and Work Culture (high vs. low: +.56(p = .3160), −1.0(p = .008), −.25 (p = .0216) were associated with trajectories of change in diabetes process of care, either directly or differentially by study arm.
This study supports the need for broader use of methodological approaches to better examine contextual effects on implementation and effectiveness of quality improvement interventions in primary care settings.
KEY WORDSDiabetes Contextual effects Multilevel modeling
For the majority of patients in the U.S. with type 2 diabetes, care is provided by primary care practices.1 Many of these patients have multiple conditions and require coordinated, comprehensive, patient-centered approaches to their healthcare in order to optimize health outcomes and improve or sustain quality of life.2,3 There is a large body of evidence to support the effectiveness of the Chronic Care Model (CCM) as a framework to guide practice redesign efforts to improve care for patients with chronic diseases.4, 5, 6, 7, 8
Although results from quality improvement (QI) efforts and cluster randomized trials are promising, implementation of practice transformations based on the CCM and patient-centered medical home (PCMH) in primary care practices has been challenging.9, 10, 11 The considerable heterogeneity in outcomes of practice-focused interventions suggests that practice context, such as structural and demographic features and practice culture, may influence successful program implementation.4,9,12, 13, 14, 15, 16, 17, 18, 19, 20 However, studies of primary care practices rarely follow up primary analyses with careful examination of how practice context interacts with intervention approaches to affect outcomes.
The Colorado Enhancing Practice, Improving Care (EPIC) cluster randomized trial was designed to compare the effectiveness of three approaches for implementing the Chronic Care Model in 40 small to mid-sized primary care practices in an effort to improve diabetes care: 1) practice facilitation using a Reflective Adaptive Process (RAP) approach; 2) practice facilitation using a Continuous Quality Improvement (CQI) approach; and 3) practice “self-direction” (SD) providing Chronic Care Model information and resources but no facilitation. There was improvement in all three arms; the greatest improvement was in the CQI arm, followed by SD and RAP.4 However, there was substantial variability among practices within the same arm, with some successfully adopting and sustaining change and others unable to do so. Consequently, in this report, we ask if practice context may explain some of this variation. Multilevel modeling, recommended as an important strategy for analyzing contextual effects, is used in this study20 to allow us to disentangle cluster level (e.g., practice, organization, etc.) effects from patient-level effects.12 Specifically, we hypothesized that aspects of inner and outer practice context as described by Damschroder et al.,19 including rural location, practice size, EHR presence, Medicaid percentage, and practice culture, would moderate intervention effectiveness. An additional innovative methodological feature includes analysis of the impact of variation in self-reported perspectives regarding practice culture among practice team members on patient-level process-of-care outcomes.21, 22, 23
Study Design and Practice/Patient Recruitment
The setting for the study was 40 small to mid-sized independent mixed-payer primary care practices (n = 32) and community health centers (n = 8) in Colorado. Practices were recruited through multiple contact methods targeting interested primary care clinicians (family medicine and general internal medicine), especially those in the State Networks of Colorado Ambulatory Practices and Partners (SNOCAP), a collaborative of practice-based research networks. Practices were stratified by urban/rural location, practice size, and type (community health center vs. other), and randomized so that the distribution of practice characteristics would be similar across arms. Here, we report on patient chart audits at baseline 9 and 18 months following practice enrollment and baseline clinician and staff surveys. The study was approved by the University of Colorado Institutional Review Board and registered with ClinicalTrials.gov (Protocol Registration Receipt NCT00414986). Further details of the study are reported elsewhere,4 but a brief description of the three study arms is given below.
Practices in the RAP arm received practice facilitation using the Reflective Adaptive Process change model based on complexity theory.9,24, 25, 26 The RAP approach focused on changing organizational functioning in order to improve diabetes care. The conceptual model underlying this approach assumes that improving organizational capacity to make and sustain change is fundamental in achieving practice improvements and implementing changes. The facilitation intervention for this arm lasted six months, with an average of 7.4 meetings (range 4–11), and facilitators were available for consultation for up to 12 months.4,9
Practices in the Continuous Quality Improvement (CQI) arm received practice facilitation based on the Model for Improvement.27, 28, 29, 30, 31 The CQI facilitators provided a structure and process for quality improvement using CQI tools that focused on sequential Plan-Do-Study-Act (PDSA)27 cycles guided by quality measurement data. Implementation of a system for obtaining reliable quality measures was a time-consuming first step, and the length of the intervention was allowed to vary (up to 18 months; mean, 15 months and 9.7 meetings), depending on practice needs. Although the CQI approach differed considerably from the RAP approach described above, conceptually, the importance of an organizational culture that supports the change process in the practice was emphasized in both. Facilitators in both the CQI and RAP arms were external to the practice.
Practices randomized to the self-directed (SD) arm received limited feedback on their baseline practice culture and level of implementation of the Chronic Care Model based on practice surveys, but no facilitation. Self-directed practices were then given access to a website with information about quality improvement and the Chronic Care Model28 for diabetes.
Process of diabetes care, the primary outcome, consisted of documentation, ascertained by chart audit, of performance based on nine quality measures from the American Diabetes Association Physician Recognition Program: HgA1c, foot exam, blood pressure, dilated eye exam, cholesterol, nephropathy screen, flu shot, nutrition counseling, and self-management support.32 Each practice generated a list of all patients with diabetes who had had at least one visit to the practice in the 18 months prior to practice enrollment and at least one visit during the 18 months following enrollment. A random sample of charts was audited, with a target of at least 20–25 patients per practice, sufficient to provide >80 % power to detect a 0.44 effect size difference between any two arms, with an intraclass correlation as high as 10 %; 822 chart audits were completed. IRB-approved procedures for de-identifying the data were followed. Each item was considered up to date if it occurred within the 12 months before the end of each audit period (baseline, 9 months, 18 months). Although this timing resulted in some overlap in patient assessment periods (e.g., a test one month before baseline could count as up to date at nine months), nine-month time blocks were necessary to meet project timeline requirements. The composite score for diabetes process of care, ranging from 0 to 9, consisted of the total number of up-to-date ADA quality measures at the end of each audit period.
Patient characteristics collected in chart audits included age, gender, and chronic medical and psychiatric conditions. Race and ethnicity were generally not recorded in the medical record, and therefore were not included in the analyses.
Each practice provided basic information at baseline, including practice setting and organization type, patient demographics, patient volume, and presence of an EHR. All clinicians and staff members (i.e., every employee in every practice) were asked to complete the Practice Culture Assessment (PCA) survey at baseline (response rate 64 %). Surveys were anonymous so that individual identity was protected. Twenty-nine percent of surveys were from clinicians (family physician, general internist, nurse practitioner, physician assistant), and the remainder from nursing staff and other practice employees. The PCA is a 22-item tool designed to measure perceptions of practice culture potentially related to practice functioning and successful implementation of practice quality improvement. Individual items and subscale development for the PCA are reported elsewhere; internal consistency is noted below.4 Subscales include: a) Change Culture (CC), 10 items that deal with how the practice does collaborative quality improvement, problem resolution, and change management (alpha = .91); b) Work Culture (WC), eight items assessing how the members of the practice work together to achieve a pleasant and productive practice environment with high-quality care (alpha = 0.91); and c) Chaos (CH), four items that assess the level of instability, disruption, and disorganization in the practice (alpha = .78). Subscale scores could range from 0 to 100, and higher scores indicate greater endorsement of the subscale dimension (i.e., more positive attitude toward practice change, better work culture, greater chaos). Means and standard deviations were computed across respondents within each practice to create practice-level variables.
Descriptive statistics were generated for patient sociodemographic measures, diabetes process of care, and practice characteristics. To understand the potential for confounding due to associations among practice characteristics, bivariate relationships were examined using chi-squared tests and t tests.
Effects of Baseline Practice Characteristics
The outcome for all analyses was the patient-level diabetes process of care score (POC) over time, as described above, with repeated measures on patients at baseline and at 9 and 18 months (2,399 unique data points). Multilevel modeling (general linear mixed models that are both longitudinal and hierarchical, with random effects for patient and practice) was used for analysis (SAS PROC MIXED). Covariates included age, gender, and medical and psychiatric comorbidities. The effect of each practice characteristic on patient-level POC scores over time was examined in separate models. To determine whether baseline practice characteristics (PC) moderated intervention effectiveness, we included a three-way interaction term (PC x arm x time) and all relevant two-way interactions in the model (PC x arm, time x arm, PC x time).21, 22, 23 For PCA subscales, we provide estimates of the effect at the overall mean score and 10 points below and above the mean. We also examined the effects of variability in PCA scores, measured by the within-practice SD for each subscale, adjusting for the effects of practice-level mean subscale scores, to determine if lack of agreement among clinicians and staff affected practice trajectories.22 Analyses were performed using SAS version 9.4.
Practice and Patient Characteristics
Baseline Patient and Practice Characteristics
RAP (n = 15)
CQI (n = 10)
SD (n = 15)
4 (27 %)
2 (20 %)
4 (27 %)
4 (27 %)
5 (50 %)
7 (47 %)
Practice Size (# clinicians) §
8 (53.3 %)
3 (30 %)
10 (66.7 %)
7 (46.7 %)
7 (70 %)
5 (33.3 %)
< 20 %
11 (73 %)
8 (80 %)
10 (67 %)
> = 20 %
4 (27 %)
2 (20 %)
5 (33 %)
PCA Change Culture, mean (sd) ‡
PCA Work Culture, mean (sd) ‡
PCA Chaos, mean (sd) ‡
Patient Chart Audit Sample
RAP (N = 312)
CQI (N = 189)
SD (N = 321)
Mean (sd) or %
Mean (sd) or %
Mean (sd) or %
Diabetes process-of-care score (range 0–9 in all groups)* ‖
Gender, % male
Number of medical comorbidities**
Presence of psychiatric comorbidity†
Baseline HgA1c (n = 636) ‖
Baseline systolic BP (n = 799) ‖
Baseline diastolic BP (n = 799)
Baseline total cholesterol (n = 703) ¶
Demographic and Structural Practice Characteristics (PC)
Change in Diabetes Process of Care by Study Arm and Practice Characteristic
Differential Intervention Effect*
F(2,2352) = 26.06, p < .001
p value within arm
F(2,2349) = 11.36, p < .001
> = 4 clinicians
p value within arm
F(2,2352) = 0.26, p = .7720
p value within arm
F(2,2352) = 3.00, p = .0498
p value within arm
Practice Culture Assessment
F(2,2349) = 6.74, p = .0012
p value within arm
F(2,2349) = 9.07, p = .0001
p value within arm
F(2,2349) = 0.63, p = .5331
p value within arm
Baseline Change Culture and Work Culture significantly moderated intervention effects (Table 2). CQI practices with higher Change Culture scores had greater improvement, but the opposite relationship was observed in RAP practices. In both CQI and SD arms, practices with higher Work Culture scores had greater improvement. Chaos scores were not associated with improvement in POC, either overall or differentially by study arm.
Variability in Practice Culture
Practice context plays an important role in successful implementation of effective interventions in primary care. In this cluster randomized trial of three approaches to implementing the CCM model for diabetes, practice structural/demographic characteristics and practice culture significantly affected change in diabetes process of care, often with variation in effects by study arm. Specifically, smaller and rural CQI and RAP practices displayed greater improvement in diabetes POC, whereas rural and higher-percentage Medicaid SD practices displayed less improvement. CQI practices with higher Change Culture scores and CQI and SD practices with higher Work Culture scores had greater improvement; conversely, RAP practices with higher Change Culture scores showed less improvement in POC. In both the CQI and SD arms, practices with greater variability across respondents in Work Culture scores (i.e., less agreement) had less improvement in POC scores.
Other studies investigating relationships between diabetes/chronic disease care and practice characteristics (including EHR,33 Medicaid percentage,34 community health centers,35, 36, 37 and practice size38,39) have reported mixed results. Interestingly, we find that rural and smaller practices, which often do not have as many connections to resources as urban practices, showed significantly greater improvement in both the CQI- and RAP-facilitated arms, while urban practices, usually with more available resources, saw more improvement in the self-directed arm. This pattern highlights the importance that practice facilitation may have for smaller, rural, and independent practices.
The role of practice culture in successful implementation of practice-based interventions is particularly relevant for practice transformation initiatives in primary care. Previous work shows the importance of internal relationships and effective teams14,40, 41, 42, 43 for providing excellent chronic disease care and achieving the hallmarks of patient-centered medical homes. Positive Change Culture and Work Culture among clinicians and staff members may reflect practice capacities that are essential for successful implementation of practice transformation initiatives such as CQI approaches. Different findings in RAP practices suggest other dynamics at work in this arm. Notably, the CQI intervention focused on implementation of a diabetes registry, used registry quality measure data, when available, to guide their QI efforts, and followed a fairly structured facilitation approach. The RAP intervention did not focus on quality measure data, and was expressly designed to improve the relationship systems within the practice25 as a pathway to better patient care and sustainable improvement. This approach allowed more practice latitude and imposed less structure with regard to the change process. In several situations, this led to the expenditure of time and effort on activities with laudable goals, but that were somewhat tangential to the objectives of the project and, in some cases, were ineffectively approached and poorly executed. This may help explain why RAP practices with higher Change Culture scores had less improvement.
Congruent with our findings, another study using the RAP approach to improve adherence to clinical guidelines in 25 primary care practices reported that successful RAP teams often initially addressed “unacknowledged tensions,” a process that resulted in improved communication in 12 practices,26 although primary outcomes did not improve significantly.44 A more focused and structured approach (with tailoring for individual practice culture, circumstances, and needs)—at least over the duration of our project—produced stronger improvement in diabetes quality measures than a less structured and more open-ended approach.
Importantly, these findings suggest that the relationship between practice culture and successful practice transformation efforts is complex and that further study is needed, with close attention to practice relationships and dynamics in light of the nature of the intervention and desired outcomes.
Variability in perceived practice culture among clinicians and staff within practices was also predictive in this study. Variability may indicate differences between line staff and practice leaders in perceptions of practice culture (as could be seen in hierarchical practices), or it could reflect disagreement among clinicians; this level of detail could not be analyzed in the present study. Methodologically, it is important to consider absolute levels (means) as well as variability (standard deviations) in responses to this type of practice culture assessment as potential contextual factors that may moderate intervention effects in future studies.
Several limitations are worthy of note. Although large for a cluster randomized trial, practice sample size was too small for a full exploration of complex interactions among practice characteristics. Targeting practices that are members of research networks may introduce selection bias and limit generalizability, as these practices are more likely to be early adopters of quality improvement innovations, although selection bias should be similar across study arms. Additionally, practice trajectories in response to the intervention were more homogeneous in the RAP arm, with less improvement overall, thus limiting our ability to detect differences by practice characteristics.4 Practice Culture Assessment data were from self-reports of clinicians and staff members and could have been subject to bias. In addition, analytic approaches are vulnerable to the usual issues of possible spurious findings, as well as the potential for missing important relationships due to low power for interactions. Finally, there are likely other factors that might have impacted intervention effectiveness at the patient or practice level that were not assessed in this study (e.g., literacy, engagement, distress, environmental support).
In conclusion, the results of this study offer some insight into which practices do better with certain practice transformation efforts and under what circumstances. Our results support findings from other studies in which practices with organizational attributes that support greater change capacity and better work relationships were likely to be better prepared to successfully implement practice transformation interventions.14,16,17 We need to more fully understand the role of practice context/culture—including variability in perceptions—to enhance practice transformation efforts. Our findings have major implications for PCMHs and other major primary care practice transformation initiatives and the methods used to evaluate the success of such initiatives.
Funding for this work was supported by the National Institute of Diabetes and Digestive and Kidney Diseases (NIDDK DF067083) and the National Institute of Mental Health (NIMH MH069809-04).
Some of this data was presented at the North American Primary Care Research Group meeting in New Orleans, in November, 2012.
Conflict of Interest
L. Miriam Dickinson has no conflict of interest.
W. Perry Dickinson had no conflict of interest.
Paul A. Nutting is employed by the Center for Research Strategies, a research and evaluation business in Denver.
Lawrence Fisher has no conflict of interest.
Marjie Harbrecht has no conflict of interest.
Benjamin F. Crabtree has no conflict of interest.
Russell E. Glasgow has no conflict of interest.
David R. West has no conflict of interest.
- 1.Services, U.S.D.o.H.a.H. National Ambulatory Medical Care Survey Online: http://www.cdc.gov/nchs/data/ad/ad346.pdf 2004. Accessed 11/13/14.
- 2.Wagner EH, Austin BT, Von Korff M. Improving outcomes in chronic illness. Manage Care Q. 1996;4:12–25.Google Scholar
- 4.Dickinson WP, Dickinson LM, Nutting PA, Emsermann C, Tutt B, Crabtree BF, Fisher L, Harbrecht M, Gottsman A, West DR. Practice facilitation to improve diabetes care in primary care: A report from the EPIC randomized clinical trial. Ann Fam Med. 2014;12(1):8–16.CrossRefPubMedCentralPubMedGoogle Scholar
- 7.Stellefson M, Dipnarine K, Stopka C. The chronic care model and diabetes management in US primary care setting. Prev Chron Dis. 2013;10:E26.Google Scholar
- 11.Danz MS, Hempel S, Lim YW, Shaman R, Motala A, Stockdale S, Shekelle P, Rubenstein L. Incorporating evidence review into quality improvement: meeting the needs of innovators. BMJ Qual Saf. 2013;22:931–939.Google Scholar
- 22.Bryk AS, Raudenbush SW. Hierarchical Linear Models: Applications and Data Analysis Methods. Second ed. Newbury Park: Sage Publications; 2000.Google Scholar
- 23.Hedeker D, Gibbons R, Longitudinal Data Analysis. Hoboken, Hew Jersey: Wiley & Sons; 2006, pp 69–76Google Scholar
- 24.Stroebel CK, McDaniel RR, Crabtree BF, Miller WL, Nutting PA, Stange KC. How complexity science can inform a reflective process for improvement in primary care practices. Joint Comm J Qual Patient Saf. 2005;31:438–446.Google Scholar
- 26.Balasubramanian BA, Chase SM, Nutting PA, Cohen DJ, Strickland PAO, Crosson JC, Miller WL, Crabtree BF. Using learning teams for reflective adaptation (ULTRA): insights from a team-based change management strategy in primary care. Ann Fam Med. 2010;8:425–432.CrossRefPubMedCentralPubMedGoogle Scholar
- 27.Institute for Healthcare Improvement. How to Improve. Available at: http://www.ihi.org/resources/Pages/HowtoImprove/default.aspx. Accessed 11/13/14.
- 28.Improving Chronic Illness Care. The Chronic Care Model. Available at: http://www.improvingchroniccare.org. Accessed 11/13/14.
- 30.Berwick DM, Godfrey AB, Roessner J. Curing health care: new strategies for quality improvement. 1st ed. San Francisco: Jossey-Bass; 1990.Google Scholar
- 32.American Diabetes Association website: http://www.ncqa.org/tabid/139/Default.aspx. Last accessed 11/13/14.
- 37.Damberg CL, Shortell SM, Raube K, Gillies RR, Rittenhouse D, McCurdy RK, Casaline LP, Adams J. Relationship between quality improvement processes and clinical performance. Am J Manage Care. 2010;16(8):601–606.Google Scholar
- 38.Beaulier MD, Haggerty J, Tousignant P, Barnsley J, Hogg W, Geneau R, Hudon E, Duplain R, Denis JL, Bonin L, del Grande C, Dragieva N. Characteristics of primary care practices associated with high quality of care. CMAJ. 2013;185(12):ES90–ES96.Google Scholar
- 41.Lanham HJ, McDaniel RR, Miller WL, Stange KC, Tallia AF, Nutting PA. How improving practice relationships among clinicians and nonclinicians can improve quality in primary care. Joint Comm J Qual Patient Saf. 2009;35(9):457–466.Google Scholar
- 44.Balasubramanian BA, Ohman Strickland PA, Crabtree BF. Using learning teams for reflective adaptation: results from a quality improvement intervention to improve adherence to guidelines for diabetes and hypertension in primary care practices. In: North American Primary Care Research Group; October 23–27, 2007. Vancouver, BC: Family Medicine; 2008;40.Google Scholar