INTRODUCTION

Effective provider-patient communication is critical to the overall patient experience.1,2,3,4,5,6,7,8 Many organizations are invested in communication training for clinicians (such as continuing medical education). Training often includes 4 to 8 h of instruction and takes place during regular office hours,9,10,11,12,13 requiring clinicians to be relieved of their other responsibilities. Yet, there is limited evidence that has tested the effect of such training on improving patient experiences and those studies that have provide little evidence of its effectiveness.10,13,14,15,16 Furthermore, initial improvements from such training may dissipate as providers return to overloaded schedules and variable patient expectations.17

Shadow coaching, a type of collaborative learning,18,19 uses peers as coaches who enter into an equal, noncompetitive voluntary relationship with those coached, observe providers in real-time patient encounters, and provide individualized structured, specific feedback to improve task performance and support positive changes.18,20,21,22,23 Sessions usually occur in dyads24 during a half-or full-day to observe several patient encounters.25,26 Mutual trust between recipients and coaches is essential for successful peer-coaching relationships.27,28,29,30

Shadow coaching has proven to be effective, with some studies finding that coaching helps build and maintain competencies and increase compliance with practice guidelines.11,31,32,33 We previously examined patient experience scores before and after coaching that incorporated many features consistent with the literature on successful behavior change (i.e., learner-centered approach, immediate feedback, written recommendations on what skills to practice/behaviors to engage in).34 We found an immediate improvement in patient experience scores following coaching; however, the gains for coached providers eroded, disappearing after 2.5 years.

Our findings correspond to providers gradually slipping back into previous patient-interaction habits.35 Such erosion related to physician behaviors and coaching36 is a specific instance of a general phenomenon that has been found in other behavioral-change interventions (e.g., smoking cessation, anger management programs). Booster sessions, i.e., occasional periodic sessions after the main training used to reinforce progress or troubleshoot obstacles for the continuance of positive changes, have been shown to be important in sustaining improvements for other interventions.37,38 However, no research has evaluated a “booster” component for shadow coaching to maintain desired behaviors39,40 or investigated the timing of re-coaching.

METHODS

In 2019, we designed a random experiment to evaluate adding booster re-coaching sessions. Such re-coaching could ensure a more consistent and persistent coaching effect on provider behaviors.

Setting

The study was conducted in a large, urban Federally Qualified Health Center (FQHC) in California that had a quality-monitoring system based on the overall provider rating (OPR) and provider communication (PC) composite of the CG-CAHPS Visit Survey 2.0, completed by adult patients or parents of pediatric patients.41

Program

Every 6 months, in January and July since 2015, the FQHC calculated every provider’s average 6-month score on the CAHPS OPR (0–100 possible range, higher is better). Providers with a 6-month average top-box score below 90 were eligible and selected for coaching. Details of the coaching intervention and its evaluation are described elsewhere.34 In January 2019, 40 providers, who had already received coaching, again became eligible for re-coaching (i.e., their 6-month average OPR score was below 90).

EVALUATION

We used stratified randomization based on provider type (physician or other) and OPR score to assign half of the 40 eligible providers to an intervention or control group (20 per group). We sorted the list of eligible providers by OPR (high-to-low) for each provider type and alternately assigned the sorted providers to re-coaching or not (20 each). Providers assigned as controls were aware of their eligibility for re-coaching and told they would gain re-coaching in a future cycle. This reminder of coaching may have sustained the effects of the original coaching.

As with initial coaching, coaches shadowed 4 or more patient encounters during a half-to-full day for the providers assigned to the re-coaching/intervention group. After observation, the coach provided verbal feedback about strengths and areas of improvement with a focus on patient-provider interactions. This initial feedback was followed by a written report from the coach to the provider summarizing the comments and recommendations from the coaching session.25 The goal is to identify and target areas of patient-provider interaction that a provider could improve when caring for their patients, with a focus on PC. Re-coaching occurred from April to August 2019.

Of the 20 providers assigned to the control group, one provider asked for re-coaching and the FQHC agreed (resulting in a sensitivity test) and two did not have CG-CAHPS data. Of the 20 providers assigned to the intervention, two left the FQHC and two refused the intervention (switched to controls). This resulted in 19 control providers and 17 providers who were re-coached.

Analysis

Given some providers switched their random assignment, we conducted both “as-treated” and “intent-to-treat” (using original assignment) analyses. We used 17,486 completed CG-CAHPS surveys from August 2012 through June 2021 about visits with the 36 providers across 19 practices. The dependent variables were CG-CAHPS OPR and PC composite (scaled 0–100). We fit mixed-effects regression models with random effects for provider (level of treatment assignment) and fixed-effects for time (linear spline with knots at initial coaching and re-coaching dates),42,43 provider top-box score prior to eligibility for initial coaching (previous performance), patient characteristics (adult/child, age, gender, race/ethnicity, general health status, education, survey language (English/Spanish)), site indicators and COVID-19 pandemic indicator (i.e., visit occurred after 3/19/2020). This spline model allows the slope to change at the time of initial coaching (first knot) and at the time of re-coaching (second knot), allowing for gradual change in scores over time. The spline knots allow for a possible vertical discontinuity or “jump” in the measured scores instantaneously after initial coaching and after re-coaching.

The spline model allows us to detect two different forms of intervention effects, each of which represents a departure in the intervention group compared to controls. The first effect, referred to as “differential immediate change (i.e., jump) at re-coaching for re-coached,” captures an immediate change in scores in the intervention group (re-coached providers), relative to the controls (only initially coached providers), at the time of intervention (re-coaching for re-coached providers). This jump at the time of re-coaching is at the second spline knot in the model (re-coaching date). The null hypothesis is no differential change in scores between re-coached and control providers at the time of re-coaching, i.e., no instantaneous effect of intervention at the time of intervention. A significant, positive value for the coefficient indicates an instantaneous positive change for re-coached providers relative to any change for control providers at re-coaching. The second intervention effect, labeled as “differential slope change at re-coaching for re-coached,” captures any change in the slope of re-coached providers at the time of re-coaching relative to any change in the slope of control providers. The null hypothesis corresponds to no differential slope change and hence no gradual effect of re-coaching. A significant, positive value for this effect indicates the trajectory of the outcome after re-coaching for re-coached providers increases relative to that of controls. Furthermore, including practice fixed-effects and provider random-effects in the models allowed mean performance to vary by provider and practice.

Study protocols were approved by RAND’s Human Subjects Protection Committee (IRB_Assurance_No: FWA00003425; IRB_Number: IRB00000051).

RESULTS

Provider Characteristics

Table 1 describes providers by their as-treated group: re-coached vs. control. Note both groups had OPR scores below 90 (74.5 for intervention and 75.1 for controls) before re-coaching, by design/eligibility. There were no significant differences between the two groups by the number of patient experience surveys, baseline provider top-box score (eligibility for re-coaching), provider type, specialty, and whether the provider had the same initial coach. Differences in patient characteristics among the two groups were: patients of re-coached providers were slightly older, less likely to be non-Hispanic White, in excellent/very good health, and have lower educational attainment. These differences are controlled for in the models.

Table 1 Provider and Patient Characteristics, Overall and By Coached Status
Full size table

Patient Experience Trends Before and After Re-coaching

Figure 1 shows adjusted OPR (panel A) and adjusted PC results (panel B) before and after initial coaching and before and after re-coaching for both control providers and re-coached providers as well as an estimated trend if the re-coached providers had not been re-coached—that is, we predicted what their patient experience trend would have been without re-coaching. Appendix Supplemental Table S1 shows as-treated analysis results and Supplemental Table S2 shows “intent-to-treat” results.

Figure 1
figure 1

Adjusted over time trend, before and after re-coaching, by measure.

Full size image

Among re-coached providers (n=17), we identified a statistically significant (~3.5 points; 0–100 scale) jump for both CAHPS measures—OPR 3.7, standard error (SE) 1.4 and PC 3.5, SE 1.4—at time of re-coaching (labeled as “Differential immediate change (i.e., jump) at re-coaching for re-coached” in Supplemental Table S1) relative to those not re-coached, and taking into consideration trends prior to re-coaching. Differences of 1, 3, and ≥5-points for CAHPS measures are considered small, medium, and large, respectively.44 The change in scores for control providers at mean time of re-coaching (labeled as “Immediate change at re-coaching for control providers” in Supplemental Table S1) was non-significant for both OPR and PC, −0.9 and −1.4, respectively. Slopes from the spline model after re-coaching for both control and re-coached providers (labeled as “Slope change at re-coaching for control providers” and “Differential change in slope at re-coaching for re-coached”) were non-significant.

Despite randomization of providers to re-coaching, OPR trends for re-coached and control providers differed between initial coaching and re-coaching; re-coached providers had a significant −1.3 (SE 0.6) drop in OPR between initial coaching and re-coaching relative to those not re-coached (“Differential change in slope at initial coaching for re-coached”). Surprisingly, we did not detect a statistically significant decline also in the controls after initial coaching, suggesting this may be a type II error.

After re-coaching, the improvement gains in patient experience faded significantly, by ~32% a year (32% for OPR and 42% for PC), disappearing after 3.1 years. That is, we calculate [(re-coached indicator×years since re-coaching×post-re-coaching period indicator)/(re-coached indicator×post-re-coaching period indicator)×100], which is [(−1.2/3.7)×100] equaling 32%. In tests comparing the predicted slope for re-coached providers with their counterfactual slope, we found no differences in slopes at 12 months for OPR and no differences in slopes at 8 months for PC, indicating the time point when re-coaching gains disappeared. Notably, re-coached providers have similar OPR and PC scores over time despite initial coaching or re-coaching, as both interventions have immediate improvement, and those improvements disappear after 2 to 2.5 years.

In the intent-to-treat models (Supplemental Table S2), a non-significant 1.6-point (SE 1.3) jump in OPR and 2.1-point (SE 1.4) jump in PC was observed; this aligns with the 2-point magnitude of the estimates for initial coaching improvements;34 however, these intent-to-treat models, based on fewer providers and a smaller patient sample, do not have enough power to detect differences of this magnitude.

DISCUSSION

Practices use patient experience scores as a metric for patient-centeredness and to improve provider-patient interactions.45,46,47,48,49 OPR and PC scores can be improved using peer shadow coaching that targets modifiable provider behaviors; however, such improvements typically erode over 2.5 years.34 In this stratified random-assignment study using mixed-effect models, we found improvements for re-coached providers relative to controls that exceeded gains from initial coaching, suggesting the reactivation of knowledge from initial coaching. However, these gains from re-coaching also erode over time, suggesting coaching and re-coaching interventions need to occur frequently to sustain improvements from coaching. Additional gains from re-coaching were evident for 12 months for OPR and 8 months for PC. A booster session may have helped renew and maintain desired provider behaviors against the pull of prior habits. These findings indicate that coaching should not be a one-time intervention, but that re-coaching should occur every 6 to 12 months to keep behaviors (and OPR and PC scores) at desired levels.

Limitations

We studied 1 large FQHC that used CAHPS data as the basis for eligibility for re-coaching and used mixed-effects models to account for several important, but not all unobserved confounders. Second, we could not evaluate the long-term effects of coaching (or re-coaching) versus never being coached, since there are no such providers at the FQHC or in the model. Also, any external changes would have affected those only initially coached and those re-coached in the time period after re-coaching. Although our findings may not generalize to all settings, they are suggestive and informative.

CONCLUSION

Shadow coaching booster sessions (re-coaching) improve patient experience scores at the time of re-coaching and exceed gains from initial coaching. If re-coaching is timed 6 to 12 months after initial coaching, it may ward off erosion of gains from prior coaching; this hypothesis should be evaluated in a large-scale, national evaluation.