INTRODUCTION

An estimated 462 million visits to primary care providers occur annually in the USA, and numbers are expected to climb to 565 million by 2025.1 Patient experience is considered a measure of healthcare quality, and measures of patient experience are becoming increasingly requested by both government and non-government payers, and public reporting and healthcare ranking organizations. Administrators and providers in primary care practices need to consider patient experience in designing new practice models to serve increasing size and complexity of patient populations in the coming years.

Patient satisfaction with healthcare is associated with clinical outcomes, provider satisfaction, and healthcare organizational success. Patient experience has been positively correlated with patient ability to self-manage health conditions and quality of life among patients with diabetes,2 and enhanced adherence to treatment plans and medical advice for patients with chronic conditions.3 Patient experience serves as a measure of provider collaboration, teamwork, and accountability. Patient-reported experience has been correlated with workplace social capital, employee burnout, and job satisfaction among primary care providers.4 Measures of patient experience have also been linked to risk of medical malpractice suits5 and patient loyalty to a provider, practice, or organization.6

Increases in patient number and complexity, challenges with the electronic medical record, and patient expectations threaten positive clinical experiences for both patients and providers. The behavior of clinicians and staff may change as the clinic day progresses. Patient appointments scheduled later in the day have been associated with reduced rates of cancer screening7 and immunizations8, and higher rates of opioid9 and antibiotic10 prescribing. However, the impact of these practice variations on patient experience has yet to be explored.

In the current study, we assessed the extent to which patient experience measures within four primary care practices are associated with appointment time. We deployed multilevel mixed effects logistic regression modeling to explore these associations and created locally weighted smoothing (LOESS) curves to visualize the changes of patient-reported experience over the clinic day.

METHODS

Setting and Study Population

This study was conducted using data from July 1, 2016, to September 30, 2017, in four internal medicine primary care practices in Rochester, MN. The Mayo Clinic Rochester practice is an integrated healthcare delivery system providing longitudinal care for 152,000 patients residing in and around Olmsted County, MN. This study was reviewed and approved by the Mayo Clinic Institutional Review Board. Patient consent was not required as data being collected for Mayo Clinic operational objectives were utilized, and under Minnesota law, patients providing authorization for participation in research utilizing existing medical data serves as consent.

Data Collection

Patient experience surveys are routinely collected for clinical practice improvement and monitoring by Mayo Clinic. The institution submits information for patients who had a visit to one of our outpatient practices to our vendor, who performs a simple random sample for surveying. We are blinded to the population who receives the survey but can link completed surveys to our scheduling and registration information. Surveys are deployed electronically or via paper mail by Press Ganey®.

Patient- and provider-level information was explored as potential confounding factors. Patient information included time of appointment, age, sex, race, ethnicity, marital status, education, language, and tobacco use. As a measure of patient complexity, we calculated age-weighted Charlson Comorbidity Indices (CCI)11. We also collected information on the type of provider affiliated with the returned survey (non-staff physician, physician, or nurse practitioner/physician assistant), and the method of survey collection (electronic versus paper).

The primary outcome of interest was patient overall likelihood to recommend our practice (“Likelihood of your recommending our practice to others”) of the Press Ganey® Outpatient Medical Practice Survey (Online Appendix 1). The survey tool consists of twenty-nine questions divided into six subdomains: access (4 questions), moving through your visit (2 questions), nurse or assistant (4 questions), care provider (10 questions), personal issues (6 questions), and overall assessment of the practice (3 questions). Each question measures responses on a Likert scale ranging from 1 to 5 (1 = very poor to 5 = very good). Responses to our overall measure of patient experience (“Likelihood to recommend our practice to others”) were dichotomized into a “top box” response (5) versus all other responses (1 through 4). The survey also consists of questions capturing patient demographic information and patient-reported time spent in the waiting and exam rooms.

Patient Appointment Time

Our primary exposure was scheduled appointment time. Appointment time information was gathered from our scheduling software and linked to completed survey responses. Appointments were grouped by the hour beginning with the first hour block (8:00 AM–8:59 AM) through subsequent hour blocks until 5:00 PM. The noon hour (12:00 PM–1:00 PM) was excluded as this is the designated lunch hour/meeting hour for providers. The first hour of the day (8:00 AM–8:59 AM) was considered the referent group for all analyses.

Statistical Analyses

Study Population Description

We stratified our population by our dichotomized outcome of interest: top score (“Very Good”) on the overall assessment question (“Likelihood of your recommending our practice to others”) and report descriptive statistics on all patient factors. Number (n) and proportion (%) were calculated for all categorical variables, and median (interquartile range, IQR) was reported for all continuous variables due to the skewed nature of our variables. Differences between groups were calculated by Pearson’s chi-square (χ2) test and Student’s t test, unless parametric assumptions were violated and the Fisher exact test and Wilcoxon rank-sum tests were employed.

Effect of Scheduled Appointment Time on Overall Patient Experience

To understand the effect of scheduled appointment time on overall patient experience, we utilized mixed effects logistic regression models. To determine which factors should be included in our final models, we performed univariate logistic regression analyses on all patient-level factors, survey completion method (paper versus electronic), and provider type against our dichotomized measure of overall patient experience. We also explored the effects of interaction between all significant patient and provider factors with the main exposure of appointment time. All variables significant in the univariate logistic regression analysis were used as covariates in subsequent multivariable models. Unadjusted and multivariable logistic regression models also included provider seen as a random effect. Setting individual provider seen as a random effect within the mixed model adjusts for the potential effects within a subgroup for an observational study such as this. We report unadjusted and adjusted odds ratios (OR), 95% confidence intervals (95% CI), and associated p values for all models. For the multilevel mixed effects modeling, we also include the number of groups observed and intraclass coefficients (ICC).

To ensure that observed effects were not attributable to patient-perceived wait time, we conducted a sensitivity analysis controlling for total wait time reported by the patient. Since both waiting room and exam room wait times exhibited extreme outliers, we truncated the waiting room times at 120 min and exam room waiting times at 60 min, similar to other published studies.12

Variation in Overall Patient Experience Through Clinic Sessions

We constructed locally weighted smoothing (LOESS) curves to understand how patient likelihood to recommend our practice varied over the clinic day in aggregate. Creation of a LOESS curve is a nonparametric method whereby an outcome measure (our overall patient experience measure, “Likelihood of your recommending our practice to others”) and exposure measure (appointment time) are estimated based on the whole curve rather than a point estimate.13 The illustration of these curves helps to visually identify the duration of an optimal clinic block to maximize patient-reported overall likelihood to recommend our practice. Construction of LOESS curves to understand effects over time have been published elsewhere.14 The default settings of the “geom_smooth” function in R’s ggplot2 package were used to build our LOESS curve.

Data management and statistical analyses were performed using Statistical Analysis Software (SAS, Cary, NC). LOESS curves were created using RStudio for R version 3.4.2. Statistical significance was set at p = 0.05.

RESULTS

Study Population Description

Overall response rate during the selected time period was 14.0% (3172 / 22,656), of whom 80.2% reported “Very Good” (top box measure) for their “Likelihood to recommend our practice to others” (Table 1). The majority of our respondents were female (60.5%), were white (94.5%), were married or with a life partner (70.4%), spoke English as their primary language at home (98.4%), completed their survey electronically (56.2%), and were seen by a non-staff physician (71.7%). The characteristics by which patients significantly differed in their top box versus non-top box responses to “Likelihood to recommend our practice to others” question included age, gender, marital status, highest level of completed education, self-reported waiting time, comorbidity burden (Charlson Comorbidity Index), type of provider seen, and time of their appointment.

Table 1 Demographic Characteristics of Primary Care Patients Completing the Press Ganey® Outpatient Medical Practice Survey (N = 3172)
Full size table

Effect of Scheduled Appointment Time on Overall Patient Experience

Both unadjusted and adjusted models demonstrated that appointments scheduled later in the day were associated with lower odds of reporting “Very Good” likelihood to recommend our practice to others (Table 2). Within our multilevel mixed effects models in which we analyzed provider assignment as a random effect, we observed relatively lower ICC estimates (6.67% in our standard model, 7.00% when also controlling for patient-reported total wait time). This lower ICC estimate indicates that surveys collected from the same provider were not highly correlated and that there is a high degree of variance between surveys collected for each provider. When controlling for patient-reported wait time in our sensitivity analysis, we observed the same effects of a decreased odds of reporting “Very Good” likelihood to recommend our practice later in the clinic day as when not controlling for patient-reported wait time. We did not find any significant interactions between our patient- or provider-level factors and scheduled appointment time. Results of our full models can be seen in Online Appendix 2.

Table 2 Unadjusted and Adjusted Mixed Effects Logistic Regression Measuring Impact of Appointment Time on the Likelihood of a Patient Recommending Our Practice to Others (N = 3172)
Full size table

Variation in Overall Patient Experience Through Clinic Day

Our LOESS curves (Fig. 1) show a steady decline in patient likelihood to recommend our practice through the clinic day. We observe the highest rates of “Very Good” likelihood to recommend our practice during the first clinic hour, and a slight improvement just before and after the midday break (12:00 PM–1:00 PM). We observe increase variation at the end of the clinic day, attributable to the fewer number of individual appointments.

Figure 1
figure 1

LOESS plot demonstrating relationship of clinic session time with overall likelihood to recommend a clinical practice.

Full size image

DISCUSSION

We identified 3172 patients in our cohort, 80.2% of whom had a top-rated experience during their primary care visit. Primary care appointments scheduled later in the day had significantly lower odds of patients recommending our practice to others. Compared with the first session hour, we observed declines in patient satisfaction as the day progressed. The precise explanation for this phenomenon remains elusive.

We observed that patient factors associated with top-rated patient experience included patient age, gender, marital status, highest level of completed education, and comorbidity burden (CCI). Previous research has also demonstrated that patients older than 65 years and with higher functional status were more likely to be satisfied with healthcare received.15 Clinical practices could enhance the healthcare experience overall by identifying groups of patients with consistently low experience ratings and working to understand the primary drivers of high-level experience for these populations. For example, patient experience data has been used to target quality improvement initiatives for clinical conditions such as acute myocardial infarction and other high-volume patient diagnosis types as well as for hospital operational areas (infection control, pharmacy).16

We observed declines in patient satisfaction as the day progressed. Several potential explanations exist for why patient experience of care changes as the clinic day progresses, primarily related to behavior changes among care team members. In the surgical literature, an increased risk of death has been observed among patients who have elective surgical procedures conducted later in the working week and on the weekend.17, 18 In primary care, an increased likelihood of antibiotic prescribing for acute respiratory infection has been observed if appointments occur later in a clinic half-day session compared with earlier10, and a decrease in ordering and completion of preventive cancer screenings as the clinic day progresses has also been observed.7 Our own group has observed that patients are more likely to receive opioids later in a clinic half-day session compared with earlier appointments.9 Provider fatigue attributed to cognitive load and function has been proposed to account for these observed behavior changes.7, 9, 10 Fatigue in the workplace is associated with professional guideline deviations19 and high practice volume has been observed to be associated with lower participatory decision-making practice style among providers.20 Optimization of time for both patient and provider may enhance patient experience. When studying the same population of primary care providers, it has been observed that longer appointments are associated with higher rates of preventive screening suggesting that appointment characteristics also impact clinician behavior.17, 18

Suboptimal patient care experiences may be the product of poor communication, scheduling delays, or discoordination across the care team throughout the day. A qualitative study within primary care observed that increased frequency of interruptions within the care episode was associated with less favorable patient visit experiences .21 Primary care providers report ordering diagnostic services for 31.4% of their patient encounters, which could result in appointment delays.22 We hypothesize that as the clinic day progresses, providers receive an accumulating number of electronic health record messages, phone calls, and result alerts from patient visits earlier in the day. These factors may be contributing to an increasing cognitive burden felt by providers and perceived by patients as the clinic day progresses.

Our study has several strengths. First, we were able to capture comprehensive information on our primary care visits using multiple data sources. Second, we utilized a population-based sample of all patients seen within four primary care settings. Third, we were able to evaluate the relationships between provider and patient characteristics and patient-reported likelihood to recommend a primary care practice. We focused on patient experience, a topic central to healthcare leadership and administrators, and used a commonly deployed survey instrument, the Press Ganey Medical Practice Survey. Finally, we deployed a novel visualization approach to further the observation that patient appointment time is a factor in quality of care in primary care.

Our study has several limitations. First, we were unable to assess the impact on type of visit (e.g., new complaint or follow-up) or result of visit (e.g., diagnosis), but did account for a measure of patient complexity (i.e., CCI). Second, we were only able to look at the impact of scheduled visits as a proxy for late-in-clinic fatigue and decreased care coordination and teamwork, although it is understood that there is a high volume of electronic and non-visit care, telephonic patient touches, and walk-in patient visits which have gone unaccounted for within our models. Finally, we are unable to assess the impact of non-response bias on our observations due to the operational nature of our survey assessments and the need to protect the confidentiality of our patients who chose to not respond to our surveys. However, our survey vendor does perform a simple random sample of all of our visiting patients, which allows for all patients to have equal opportunity to receive and complete a survey. The response rate to our surveys (14.0%) is not different than other reports of Press Ganey Medical Practice surveys (16.5%).23 Assessment for non-response by Tyser et al.23 found that patients were less likely to respond if they were male and had Medicaid/self-pay as their primary payer. We also observed a greater number of female respondents in our study than males, but did not have access to primary payer information. Odds of responding increased along with age among the population reported in the study by Tyser et al.,23 which could help us to understand why our population also skewed older than the general population.

Patient experience can be an indicator for systems which may be operating sub-optimally with systematic challenges arising from both quality and efficiency domains of care.24 Novel care delivery models can be developed through system redesign resulting in increased patient, provider, and staff experience and satisfaction.25 Measures of patient experience can be leveraged as a means to drive transformation of clinical practices. Our study suggests that novel approaches to the analysis of patient experience could drive this innovation.