INTRODUCTION

Electronic consultation (eConsult) services are emerging as a promising means of leveraging electronic health records (EHR) to improve inter-clinician communication and increase the timeliness and efficiency of medical care. eConsult enables primary care providers (PCPs) to request a virtual, asynchronous consultation with a specialist about a specific patient via a shared EHR. eConsult programs have been successfully implemented at a range of health care institutions in the USA, Canada, and Europe and are associated with reductions in traditional referral rates, shorter wait times for specialty care, and improved communication between clinicians.1,2,3,4,5,6,7

A growing body of literature documents PCP and specialist experiences with various eConsult models,8,9,10,11,12 but less attention has been paid to patients’ awareness of and experiences with eConsult. Studies to date have either drawn on providers’ impressions of patients’ satisfaction with eConsult or have a small sample size.13, 14 In conjunction with the national dissemination of a multi-specialty eConsult program originally developed at the University of California, San Francisco (UCSF) and supported by a Center for Medicare & Medicaid Health Care Innovation Award, this study aimed to (a) assess patients’ awareness of and satisfaction with eConsult and (b) compare the experiences of patients whose PCP used eConsult with those of patients who were referred for an in-person visit with a specialist.

METHODS

Study Setting

Participating health systems included nine academic medical centers that provide primary care and specialty outpatient medical services: University of California health system campuses at San Francisco, San Diego, Irvine, Davis, and Los Angeles; Dartmouth-Hitchcock, Lebanon, NH; University of Virginia, Charlottesville; University of Iowa, Iowa City; and University of Wisconsin, Madison. The eConsult model was launched at UCSF in 2012 and at the remaining institutions between 2014 and 2015. Patients receiving an eConsult at these participating institutions were not responsible for any costs for the service.

Instrument Development

To better understand and compare patients’ experiences with eConsult and traditional referrals, we developed a survey informed by 15 semi-structured interviews with primary care patients whose PCP had used the eConsult service at UCSF. Through these interviews, we found that many patients were not aware that an eConsult had taken place on their behalf. These findings prompted us to develop survey questions that did not depend on patients’ prior awareness of eConsult. The survey included branching logic to direct respondents depending on recognition of the PCP’s intention to place an eConsult or traditional referral order (see Fig. 1 for survey flow and online Appendix A for survey questions). Survey design was informed by a Kaiser Permanente study comparing patients’ satisfaction with virtual and traditional consultation14 and the Consumer Assessment of Health Plans Survey (CG-CAHPS).15 Question wording and order were adjusted based on pilot testing and cognitive interviews with 18 primary care patients at UCSF. The interviews assessed participants’ thought processes as they responded to survey questions, including how questions and key concepts were interpreted and understood.

Figure 1
figure 1

Survey branch points.

Full size image

Participants and Recruitment Strategy

The study was conducted March–November 2016. Survey administration procedures were standardized across sites and managed by each site’s eConsult implementation team. The sampling process included the following procedures. All orders for an eConsult or standard referral generated during adult (≥ 18 years) primary care visits at the academic medical center and satellite clinics were identified monthly. To reduce potential bias from provider or clinic-level influences on patient experience, orders from PCPs who did not submit at least one eConsult in the previous month were removed. Referral orders to specialty departments not participating in eConsult were also removed in order to enable comparison of patients’ experiences with eConsult and referral in the same specialties. Additionally, all patients with more than one order in a given recruitment month were removed. This approach was deemed necessary after initial cognitive interviews revealed that the survey’s focus on one specific clinical encounter was confusing for patients with multiple recent clinical interactions and referrals and that this confusion could bias survey responses.

A web-based mode of survey administration was selected. Surveying patients by phone was not feasible given budget constraints, and a mailed version was ruled out because recent research shows that web-based surveys yield comparable results with mail more quickly and at lower costs.16, 17 Invitations with a link to the survey were distributed by email once a month, timed approximately 2–6 weeks after the primary care encounter. A reminder was sent to non-respondents 1 week after the initial invitation. The survey was administered at all nine sites using Qualtrics software (Provo, UT). All Qualtrics accounts and survey data were located on secure, HIPAA-compliant servers. The institutional review boards at all sites exempted the study from review.

Statistical Methods

Data were summarized overall, by site, and by eConsult or referral order using means and standard deviations for continuous variables and counts and percents for categorical and ordinal variables. Responses from patients recognizing their order type were compared using chi-squared statistics. To identify factors specific to referrals or eConsults, binary and multivariate logistic models were used to examine (a) whether demographic variables and experience with PCP predicted recognition of eConsult/referral service and satisfaction with service and (b) whether satisfaction with service and experience with PCP predicted preference for referral or eConsult for future health problems. Patients’ experiences with the PCP visit that led to the referral or eConsult were captured in a series of five questions that assessed trust and confidence in the PCP and satisfaction with the visit. These questions were aggregated to form a composite measure that we refer to here as “PCP trust” (see online Appendix A, Questions 4–8). Complete case analysis with listwise deletion was used for modeling, with models controlling for cross-site differences in PCP trust and patient demographics (specifically ethnicity, health status, age group, education group). All other analyses excluded missing responses (pairwise deletion). All analyses used P < 0.05 as the criterion for statistical significance and were analyzed using SPSS Statistics V25.18

RESULTS

A total of 29,291 email invitations were sent (3515 [12%] eConsult; 25,776 [88%] referral) and 8087 responses were received. The response rate was 26.7% (N = 938) for eConsult and 27.7% (N = 7148) for referral patients, with an overall response rate of 27.6%. The mean age of respondents was 57.5 and 64.5% were women. Self-reported education level showed 61.5% reporting a 4-year college degree or higher and 88.4% reporting some college or higher. Nearly half of respondents reported very good or excellent health and 84% of the study population identified as white. Statistically significant differences in demographic characteristics between eConsult and traditional referral patients included age and education. The average age for respondents who received an eConsult (58.5 [SD = 16.4]) was higher than that for those who received a referral (57.3 [SD = 15.7]), P = 0.040, while the proportion of patients with a 4-year college degree or higher was 58.4% for eConsult and 61.9% for referral (P = 0.042 chi-square test). (See Table 1 for full demographic details.)

Table 1 Participant Characteristics
Full size table

Order Recognition

When asked if they had an in-person visit, phone call, or electronic message exchange with the identified primary care provider in the prior 6 weeks, the majority of respondents (86.9%; N = 7030) answered yes. Among those who recalled their primary care interaction, 64.9% (N = 4561) correctly identified that the PCP planned to get an advice from another physician or referred them for an office visit with a specialist, and correct identification for referrals was higher 67.2% (4174) compared with eConsults 47.1% (387). Across sites, rates of correct recognition ranged from 33 to 55% for eConsult and 60 to 78% for referral (online Appendix B).

A multivariable regression model using complete cases (eConsult N = 670; referral N = 5121) showed that respondents reporting very good or excellent health were significantly more likely than those with poorer health to recognize their PCP’s intention to use an eConsult (OR 1.62, 95% CI 1.18–2.23; P = 0.003) or to place a referral (OR 1.16, 95% CI 1.03–1.31; P = 0.019). Educational attainment was similarly associated with consult type recognition, with respondents with a 4-year college degree or higher more likely to recognize their PCP’s eConsult (OR 1.40, 95% CI 1.02–1.94; P = 0.040) or referral (OR 1.39, 95% CI 1.23–1.58; P < 0.001) decision. All results reported below are limited to respondents who identified their correct order type.

Comparing Patients’ Experiences Along eConsult and Referral Trajectories

Nearly all patients who recognized that their PCP had sought advice from a specialist through an eConsult or traditional referral agreed with their PCP’s decision to seek specialist input (eConsult N = 370 [98.9%]; referral N = 4115 [99.1%]). In terms of timeliness of care, 97.1% (N = 303) of eConsult patients reported receiving information about the specialist’s advice by the time of survey completion, whereas only 55.8% (N = 2064) of referral patients had completed an office visit with the specialist.

Among the 87.4% (N = 3625) of traditional referral patients who had scheduled a specialist appointment before responding to the survey, most (77.8%; N = 2797) reported being able to obtain the appointment as soon as needed. A higher percentage of eConsult patients agreed that communication about the specialist’s advice (95.8%; N = 297) was prompt. Figure 2 details how specialists’ eConsult advice was communicated to patients.]-->

Figure 2
figure 2

How eConsult recommendations were conveyed to patients.

Full size image

When asked about overall satisfaction with the specialist’s recommendations, 81.4% (N = 249) of eConsult and 81.9% (N = 1675) of referral patients reported that they were satisfied or very satisfied. The specialist’s recommendations were explained clearly according to 97.1% (N = 301) of eConsult and 95.2% (N = 1952) of referral patients, while 80.4% (N = 250) of eConsult and 93.6% (N = 1913) of referral patients reported having had an opportunity to ask questions about the specialist’s recommendations (Fig. 3).]-->

Figure 3
figure 3

Comparison of eConsult and referral patients’ experience receiving specialist advice.

Full size image

In multivariable regression models using complete cases, and adjusting for sample size and demographic differences across sites, we found that patients with high PCP trust (using the 5-question composite measure) were more likely to be satisfied with the specialist’s recommendations (eConsult OR 10.63, 95% CI 2.95–38.32; P < 0.001; N = 251; referral OR 2.86, 95% CI 1.85–4.42; P < 0.001; N = 1771). No other patient factors were associated with satisfaction, nor did we find significant cross-site differences in patient satisfaction.

Preferences for a Similar Problem in the Future

When asked to choose between eConsult or traditional referral for a similar problem in the future, 78.1% (N = 289) of eConsult and 67.6% (N = 2664) of referral patients chose the same service that they had received. Nearly one-third (32.4%; N = 1278) of referral patients said that they would prefer an eConsult in the future, whereas 21.9%; (N = 81) of eConsult patients would opt for traditional referral in the future (Fig. 4).]-->

Figure 4
figure 4

Preference for management of a similar problem in the future.

Full size image

Multivariable regression analyses with complete cases were used to better understand factors associated with a preference for an eConsult in the future. Patients with a referral (N = 3462) were less likely to prefer eConsult in the future if they self-reported as white and not Hispanic (OR 0.65, 95% CI 0.55–0.78; P < 0.001), in very good or excellent health (OR 0.83, 95% CI 0.71–0.96; P = 0.011), or having completed a college degree or higher (OR 0.64, 95% CI 0.55–0.75; P < 0.001). Demographic variables were not predictive of order preference among eConsult patients. Patients who reported higher trust in and satisfaction with the PCP (PCP trust measure) were more likely to prefer eConsult in the future (OR 1.38, 95% CI 1.02–1.86; P = 0.035; N = 3770). When this same analysis was stratified by order type, this finding held true for those who had an eConsult (OR 6.74, 95% CI 2.49–18.27; P < 0.001; N = 308) but was not significant for those who had been referred (OR 1.22, 95% CI 0.89–1.67; P = 0.228; N = 3462).

DISCUSSION

Our results show that patients whose PCP placed an eConsult order were highly satisfied with the experience, to a similar degree as those who had already completed a specialty visit after a referral. Nearly all eConsult and referral patients agreed with the PCP’s decision to consult with a specialist. A large majority of eConsult patients reported receiving information about the specialist’s response in a timely manner. This contrasts with patients who had a referral, among whom nearly half had not yet completed an office visit with a specialist.

Interestingly, we found that a good primary care experience was predictive of higher satisfaction with the specialist’s recommendations among both groups of respondents. These findings suggest that primary care encounters and relationships can have an impact on patients’ experiences with clinicians and clinical services in other settings.

Nearly one-third of patients who were referred to see a specialist in person, and nearly four out of five patients whose PCP used the eConsult service, expressed a preference for eConsult for a similar problem in the future. Given that the eConsult service had been recently introduced at participating medical centers, we interpret this finding to mean that eConsult is an acceptable means of obtaining specialist input not only for the majority of patients who received the service but also for many patients with no prior knowledge of the service. Patients who had a good experience with their PCP were more likely to want an eConsult in the future, suggesting that confidence in one’s PCP increases the acceptability of this novel service.

Although we found statistical differences in age and education between eConsult and referral groups, the differences were quite small and not deemed to be clinically meaningful; regardless, regression analyses controlled for the impact of these characteristics. We found that referral patients who identified as white and non-Hispanic, had a college degree or higher, or were in better health were less likely to prefer an eConsult for a similar problem in the future. These findings cannot be explained without further research into the reasons for patients’ preferences, but we surmise that race/ethnicity and education may be serving as proxies for socioeconomic status and a bias toward in-person specialty care among those with more privilege.

It is concerning that nearly over half of eConsult patients, and nearly a third of referral patients, were not aware of their PCP’s intention to place an eConsult or referral order. The low rate of recognition among eConsult patients may be explained in several ways. First, PCPs may not always tell patients they are using eConsult and may decide in some cases to discuss the specialist’s advice only in the event that it prompts a change in management of the patient’s condition. Second, patients may not consider a virtual consultation between clinicians to be a distinct or noteworthy service, particularly when the patient is not being billed for the service (as was the case here).

Inaccurate recall among referral patients suggests that communication gaps can occur early in the referral process. Research has demonstrated high rates of incomplete referrals,19 whereby patients do not receive the specialty care intended by the referring provider. For instance, a recent study at a large academic medical center showed that only 35% of referrals from PCPs to specialists resulted in a completed appointment.20 Our study demonstrates that one source of incomplete referrals may be a lack of adequate understanding of the referral from the outset. Moreover, many patients have complex health problems and are being cared for by multiple health care providers. Although our study was designed to reduce confusion by limiting inclusion to patients with only one eConsult or referral order in the previous month, it is possible that survey respondents had difficulty recollecting the specific details of the encounter in question.

Our study has several limitations. Use of a web-based survey prevented us from recruiting patients without an email address, which may have biased our sample toward respondents with higher digital literacy and educational attainment. Our response rate was relatively low, which might also have introduced nonresponse bias. Also, the study population was less ethnically and racially diverse than the overall US population, so we were unable to adequately capture the experiences of underrepresented minorities and underserved communities. Research on underserved patients’ views about telemedicine suggests that there are fears and concerns about remote consultation as well as perceived advantages and that perspectives vary considerably across racial/ethnic groups.21 In studies of eConsult specifically, patients have reported concerns about access to specialty care.22 We hope that the relationship between care preferences, socioeconomic status, and access to health services is explored in future evaluations of eConsult acceptability among patients.

Another limitation was introduced by excluding patients with more than one order in the previous month. This limits the generalizability of our findings, particularly because patients with multiple orders may be more positively inclined toward a service that reduces wait times for specialist advice. Indeed, our focus group study with patients at the same academic medical centers found that medically complex patients with multiple providers were particularly enthusiastic about eConsult.23 Item nonresponse bias may have been introduced by eliminating missing responses from descriptive analyses. Moreover, missing data may have biased multivariate model results, although a comparison of participant characteristics in Table 1 with those of participants used for modeling (complete case analysis) did not demonstrate notable demographic differences. Finally, the survey was offered exclusively in English, which limits our insights into the experiences and preferences of non-English-speaking patients.

CONCLUSIONS

Patients’ experiences with eConsult have not been as closely or widely examined as those of clinicians. Our results help to fill this gap by demonstrating that a high proportion of patients find eConsult to be an acceptable strategy for the management of their medical condition and that trust and confidence in one’s PCPs are crucial ingredients for creating a satisfying eConsult experience.

There are prima facie concerns that patients might perceive services like eConsult to be a form of rationing or a lower quality of clinical care. By comparing eConsult and traditional referral perceptions and experiences, this study provides evidence that appears to counter these concerns. Nonetheless, the lack of awareness of eConsult among many patients who were beneficiaries of the service warrants an increased effort to include them in eConsult decision-making and communication. As more payers elect to cover eConsult services (including CMS, which began coverage in 2019), the question of patient awareness takes on additional importance. A charge for a co-payment, for instance, may cause confusion for patients who do not recognize having received the service and may reduce eConsult acceptability.23