INTRODUCTION

Electronic referrals represent an opportunity to use health information technology (health IT) to improve access to subspecialty care. Health IT can improve the safety and efficiency of health care.1 The potential for improvements apply not only to resource-rich settings, but also to safety-net health systems, which the Institute of Medicine characterizes as those that "…offer care to patients regardless of their ability to pay for services, and [for which] a substantial share of their patients are uninsured, Medicaid, or other vulnerable patients."2 While safety-net health systems’ diversity of reimbursement sources may allow for greater innovation in primary care-subspecialist relationships than traditional fee-for-service models,3 safety nets vary in uptake of health IT.4 In addition to a lack of IT resources,5 health systems may encounter barriers because of concerns about the effects on workload, work roles, or workflow.6

In the safety net, the subspecialist shortage is severe.7,8 The under- and uninsured have fewer choices among subspecialists and longer wait times for appointments;9 longer wait times are associated with delays in diagnosis, greater costs, and worse outcomes.10,11 The adoption of electronic referrals could mitigate the effects of the scarcity of subspecialists. Compared to paper referrals, electronic referrals improve the transfer of administrative and clinical information;12 they may reduce duplicate test-ordering,13 and improve both the referring and subspecialty physician’s ability to make treatment decisions.14 These attributes could lead to rational allocation of subspecialty visits, improving clinical outcomes15 while minimizing wasted resources.16-18

Few studies examine the factors that may improve or impede the adoption of electronic referrals in safety net settings.19-21 The literature supports the examination of health IT’s effects on resources and workflow to inform efforts to implement and sustain newer technology.5,22-24 To better understand primary care providers’ experiences with a new electronic referral system for subspecialty care implemented at a public, university-affiliated, teaching hospital [San Francisco General Hospital and Trauma Center (SFGH)], we administered a cross-sectional, web-based survey of all safety net primary care providers who had the option to refer to SFGH for subspecialty care. We hypothesized that, in the opinion of referring providers, electronic referrals would: (1) improve overall clinical care compared to prior methods, (2) improve referring providers’ access to subspecialists for non-urgent patient issues, and that (3) changes in referring clinic workflow would improve referring providers’ satisfaction.

METHODS

Setting

In July 2005, the SFGH gastroenterology (GI) and liver clinics launched an electronic referral system that allowed a gastroenterologist to triage and allocate limited appointments. Other subspecialty clinics subsequently adopted the electronic referral portal: cardiology and pulmonary (January 2007), endocrinology and rheumatology (May 2007), and neurosurgery and orthopedics (July 2007). With each of these clinics, the hospital alerted referring providers that all non-emergent referrals had to be submitted electronically.

Description of Electronic Referral Program

The key attributes of the SFGH electronic referral system include integration of existing electronic health record demographic and clinical data into electronic referrals, centralized triage of referrals by designated subspecialty, and back-and-forth communication between referring providers and a subspecialist reviewer. Referring providers complete an electronic template, to which existing relevant electronic health record information is automatically appended. Subspecialists review requests within 72 h and choose one of the following options: (1) schedule next available regular appointment, (2) schedule an urgent appointment, or (3) do not schedule. When reviewers do not schedule an appointment, they request additional workup or information, or suggest alternative management in lieu of an appointment. The referring and reviewing providers can communicate in an iterative fashion until the reviewer decides to either schedule an appointment or they both agree that the patient does not need one. When the appointment is granted, this decision is electronically transmitted to the clinic scheduler, who makes the appointment. The hospital electronic health record system then generates a letter to the patient and an e-mail to the referring provider alerting him/her to the appointment. The electronic referral portal keeps a database of all submitted referrals, which serves as a tracking mechanism for both referring and subspecialist providers and clinics.

Study Participants

Referrals originate from primary care providers working in one of three safety net clinic systems: (1) SFGH-based primary care clinics (“hospital-based”), (2) community-based, county-funded health centers that share a common electronic health record with SFGH (Community-Oriented Primary Care clinics or “COPC”), and (3) local non-county-funded community health centers (San Francisco Community Consortium Clinics or “Consortium”) whose access to the electronic health record is through a digital firewall. University of California, San Francisco (UCSF) faculty and trainees staff hospital-based clinics, while county-employed providers staff COPC clinics. Individual clinics employ Consortium clinicians. All subspecialist providers are university-employed and hospital-based. In general, UCSF-employed primary care providers have fewer sessions of clinical time per week and care for a smaller panel than COPC and Consortium providers.

Our study included all primary care providers who have the option of referring adult patients to SFGH. We defined primary care providers as either Family Medicine or Internal Medicine physicians or mid-level providers (nurse practitioners or physician assistants) who see adults and practice in primary care clinics.

SURVEY

Survey Method

We developed an 18-item web-based questionnaire based on prior studies and our interest in the domain of impact on clinical care. We chose a priori to pre-test the questionnaire at four sites in order to represent those that had large and small numbers of providers and which demonstrated higher and lower usage of electronic referral. Based on our pre-test, we clarified wording of items and added the domain of impact on clinical practice.

We mailed a letter introducing the study to all eligible participants prior to initiating the survey, and then sent an e-mail to all participants containing a link to the questionnaire. After sending weekly e-mail reminders for 3 weeks, we telephoned and then mailed a paper version to non-responders. We collected questionnaires from October 2007 through January 2008. We offered a light catered lunch to the two clinics with highest response rates.

The institutional review board at University of California, San Francisco approved the study.

Measures of Participant Characteristics

We asked participants to identify their training level (resident, mid-level provider, or attending physician), practice setting (hospital-based, COPC or Consortium), and volume of care (frequency of seeing patients in clinic each week, frequency of using electronic referrals, length of time using electronic referral in months). Because we anticipated that individual preferences for technology would influence providers’ experiences with electronic referrals, we used Prasad and Agarwal’s validated 4-item scale, which asked participants to rate their willingness to use new information technology on a 5-point Likert scale.25

Participant-Specific Process Measures

We asked providers to note when they submitted electronic referrals: “during,” “between,” “after” patient visits, “never, someone else submits for me,” or “never refer.” We defined time spent referring as a categorical variable with five mutually exclusive levels ranging from “less than 2 min from start to submit” to “greater than 10 min from start to submit.”

Measures of Impact on Clinical Care

We asked participants to compare overall clinical care using electronic referrals to prior methods of referring patients to subspecialists on a 5-point Likert scale (“much worse” to “much better than prior methods”).

Measures of Impact on Clinical Practice

We assessed three practice domains: content, process, and access to subspecialists using electronic referrals compared to prior methods. We used a 5-point Likert scale ranging from “much better” to “much worse.” For content measures, participants rated subspecialty guidance of workup and how well the subspecialist addressed the clinical question. For process measures, we asked participants to rate their ability to track the referral. To gauge access to subspecialists, we asked participants to rate wait time for an available appointment for subspecialty clinics, as well as access to a subspecialist for urgent and non-urgent patient issues.

Statistical Analysis

For the main dependent variable “overall clinical care,” we collapsed 5-level Likert scale responses to two levels, “better” (“much” and “somewhat better”) and “not better” (“no change,” “somewhat” and “much worse”). We chose this dichotomization because of our a priori belief that the success of electronic referrals should be measured by its ability to improve clinical care. We tested for bivariate associations and then used a logistic regression model to determine adjusted odds ratios (AOR). We constructed stepwise multiple regression models, considering as candidates all variables that were associated with the outcome at p < 0.20 in bivariate models, using Spearman’s rho for ordinal independent variables and chi-squared tests for dichotomous variables. After constructing the model with the independent variables of “time spent referring” and “affinity for information technology,” we then added other factors singly and in order. We retained the newly added variable if its effect was statistically significant at p < 0.05.

For other dependent variables (i.e., measures of content, process, and access), we were interested in whether things improved, worsened or were left unchanged. Because the results were statistically similar for five versus three categories, we collapsed 5-level responses into three categories: “better” “same” or “worse.”

In Table 1, we present all candidate variables. In Table 2, we present the AOR for the independent variables for overall clinical care. We transferred all responses from a web-based server (DATSTAT Illume 4.5, Seattle, WA) in Excel format to STATA/S.E. 9.2 (Stata Corporation, College Station, TX).

Table 1 Participant Characteristics (n = 298)
Full size table
Table 2 Adjusted Odds Ratios of Physician Report that Clinical Care is Better as a Result of the Electronic Referral Process, by Physician Characteristics
Full size table

RESULTS

We collected 298 of 368 questionnaires for an 81.0% response rate. Nearly all (96.0 %) of the participants that responded used the electronic web-based survey; 4% of providers returned the paper version. Two providers worked in more than one clinic, but each of these providers completed a single questionnaire, and their clinics were from within the safety net clinic system (either COPC or Consortium). Less than 3% of data from the main dependent and independent variables were missing; missing data for the other dependent variables ranged from 4.0 to 7.4%. For each model, we restricted the sample size to those with complete data (Table 3).

Table 3 Participating Clinics by Safety Net Health System (Number of Participants/Number of Eligible Providers at Each Clinic)
Full size table

Measures of Participant Characteristics

The majority of participants were attending physicians (53.5%), with residents making up 23.6% and mid-level providers 22.9%. The majority of participants were physicians practicing internal (43.3%) or family medicine (33.9%). Over half (55.4%) of the survey participants worked at hospital-based clinics, while 27.9% worked at COPC, and 17.1% at Consortium clinics. Most participants saw patients fewer than five half-day clinic sessions per week (70.5%) and had been using electronic referrals for more than 6 months (72.7%). A summary score of four items measuring willingness to use newer information technology showed that 63.5% of participants somewhat or strongly agreed that they were open to using new information technology. This “technophilia” did not differ significantly by training level, type of primary care, or clinic setting. The standardized reliability coefficient (Cronbach’s alpha = 0.88) suggests high reliability of the scale.

Process-Specific Participant Measures

Sixty-seven percent of providers submitted electronic referrals at the end of the clinic, a quarter (26.6%) submitted between visits, and 6.4% of participants reported that someone else submitted for them. While 46.3% of participants reported spending 5 min or less submitting an electronic referral, 36.2% spent 6 to 10 min, and 13.8% spent more than 10 min. (Table 1)

Measures of Impact on Clinical Care

Overall, 71.9% of participants felt electronic referrals had improved clinical care; 7.3% felt electronic referrals had worsened care, and 20.7% felt care had not changed (chi-square p < 0.02). In a bivariate analysis comparing improved versus same or worsened care, significantly more of the resident physicians (87.1%) compared to 67.9% of attending physicians and 64.6% of mid-level providers reported that electronic referrals had improved care (p < 0.01). Similar proportions of internists (73.8%) and of family physicians (74.5%) felt that care had improved. Hospital-based primary care providers had the highest proportion of participants who felt care had improved (80.9%), whereas two-thirds of COPC participants (67.1%) and half (50.0%) of Consortium participants did so, representing a significant trend (Spearman’s rho, p ≤ 0.01). In a multivariate analysis adjusted for training level, subspecialty, clinic setting, technophilia, and time spent submitting, we found that working in a Consortium clinic, as compared to a hospital-based clinic, was significantly associated with lower odds of reporting that care had improved (AOR 0.40, 0.18-0.91). Working in a COPC clinic compared to working in a hospital-based clinic was associated with non-significantly reduced odds of reporting that care had improved (AOR 0.72, 0.35-1.49). More than 6 min spent per referral also predicted lower odds of agreeing that clinical care had improved (AOR 0.33, 0.18-0.61), even after adjusting for clinic setting.

Measures of Impact on Clinical Practice

The majority (71.9%) of participants felt that electronic referrals improved guidance of the pre-subspecialty visit workup, while 22.3% reported no change, and 5.0% reported guidance was worse. (Fig. 1) Over half (57.3%) reported that subspecialists addressed the clinical question better, 30.4% reported no change, and 12.9% reported it was worse.

Figure 1
figure 1

PCPs’ ratings of attributes of electronic referrals compared to prior referral methods.

Full size image

Participants reported that electronic referrals had improved overall wait times for subspecialty appointments (53.9%) and access to the subspecialist for non-urgent patient care issues (59.4%). In contrast, 35.1% reported that electronic referrals had improved access to subspecialists for urgent issues, while nearly half (49.5%) reported no change, and 16.1% reported worse access. For these measures on clinical practice, we found no significant subgroup differences.

DISCUSSION

Seventy-two percent of primary care providers reported that electronic referrals improved overall clinical care compared to prior methods. Several factors, such as perceptions of better communication as manifested by improved tracking and increased access manifested by decreased wait times, may have contributed to this impression. We found that nearly 60% of participants reported that access for non-urgent patient issues had improved, supporting our hypothesis that electronic referrals would improve access to care. Participants reported decreased wait times; this perception was born out by administrative data (data not shown). Features that may have enhanced communication were the improvements in tracking, standardized referral templates, and the centralized review. These may have improved the quality and timeliness of information available to subspecialty reviewers through standardization of communication and equitable access to reviewers. Improving the quality and availability of clinical information has been shown to improve the likelihood that the subspecialist responds to the clinical question.14,26,27 Providers had better guidance of the workup, which may have improved effectiveness of the consultation. Improvement in the ability to track referrals may have improved accountability and prevented duplication of efforts.

We were concerned that electronic referrals would disrupt workflow, which could have impaired the perception of improved care. Workflow challenges present important barriers to adoption and sustainability of health IT.28 We found that a significant proportion of Consortium clinic providers reported taking longer to submit electronic referrals than other providers; the length of time for submission was independently associated with lower satisfaction with overall clinical care. The lack of available computer terminals, multiple-step procedures to access the secured electronic referral portal, and frequent disconnections likely contributed to this increased time spent submitting referrals. We did not find that Consortium providers reported less affinity for newer information technology, increasing the likelihood that the finding of increased dissatisfaction was related to structural barriers.

We found evidence of the impact of the “digital divide” among safety-net health systems on acceptability.29 We are attempting to address these barriers by garnering additional resources to improve computer access and connectivity in Consortium clinics and by encouraging those who are able to submit quickly despite the barriers to share their expertise. Yet, the findings within our study point to a larger phenomenon of “digital disparity” in access to and usability of health IT. Our findings are consistent with prior studies showing that interventions using computerized decision support, electronic referrals, or electronic health records have had variable uptake in part due to the heterogeneity of patients, practices, and resources especially among safety net health systems.30

Our study had important limitations. First, we used web-based questionnaires to ask participants about electronic referrals, which may have selected participants who had greater willingness to use IT. However, we mailed paper versions of the questionnaire to non-responders. Second, we relied on primary care provider recall of the referral system prior to electronic referrals, which may introduce recall bias. Third, this study focuses solely on the referring providers’ perspective, and relies on their subjective reports of time spent submitting and wait times. We did not weight responses by clinic setting because clinic setting was not the only way to categorize the variability in our participants. We may, however, have underestimated the discontent among non-respondents, as those from the Consortium were both less likely to report that electronic referrals had improved clinical care and were less likely to respond. Because we were interested in the acceptability to individual participants, we did not adjust responses for panel size or number of referrals per participant. Fourth, we analyzed results from providers referring to a single hospital in a safety net health system, including clinician investigators and residents who, while working full time, spend only a few half-days a week of engaged in direct outpatient care. However, because we included primary care providers from three different health systems, our findings may generalize to other community- and university-based providers who work in safety net settings.

Electronic referrals offer the opportunity to improve the effectiveness and efficiency of subspecialty care without inappropriately restricting access. Integrated health-care delivery systems that do not rely on fee-for-service reimbursement may be fertile ground for attempts to improve the efficiency of subspecialty care, thus improving access.31,32 We found that digital disparities among safety net health systems depend less on individual provider preferences than on access to and the usability of health IT. Future interventions should incorporate primary care provider and referring clinic experiences in evaluating the adoption and spread of electronic referrals. In our study, safety net primary care providers reported that electronic referrals improved not only access to specialty care, but also the overall clinical care of patients. Our findings suggest that IT innovations can be instrumental for safety net settings in attaining the “holy grail”33 of ensuring access to timely, high-quality subspecialty care.