Ideally, a referral from a primary care physician (PCP) to a specialist results in a completed specialty appointment with results available to the PCP. This is defined as “closing the referral loop.” As health systems grow more complex, regulatory bodies increase vigilance, and reimbursement shifts towards value, closing the referral loop becomes a patient safety, regulatory, and financial imperative.
To assess the ability of a large health system to close the referral loop, we used electronic medical record (EMR)-generated data to analyze referrals from a large primary care network to 20 high-volume specialties between July 1, 2015 and June 30, 2016.
The primary metric was documented specialist appointment completion rate. Explanatory analyses included documented appointment scheduling rate, individual clinic differences, appointment wait times, and geographic distance to appointments.
Of the 103,737 analyzed referral scheduling attempts, only 36,072 (34.8%) resulted in documented complete appointments. Low documented appointment scheduling rates (38.9% of scheduling attempts lacked appointment dates), individual clinic differences in closing the referral loop, and significant differences in wait times and distances to specialists between complete and incomplete appointments drove this gap. Other notable findings include high variation in wait times among specialties and correlation between high wait times and low documented appointment completion rates.
The rate of closing the referral loop in this health system is low. Low appointment scheduling rates, individual clinic differences, and patient access issues of wait times and geographic proximity explain much of the gap. This problem is likely common among large health systems with complex provider networks and referral scheduling. Strategies that improve scheduling, decrease variation among clinics, and improve patient access will likely improve rates of closing the referral loop. More research is necessary to determine the impact of these changes and other potential driving factors.
In an ideal health system, a primary care referral to a specialist would result in a swift, completed consultation, with documentation available to the PCP for review (Fig. 1). This is defined as “closing the referral loop” and can be assessed by looking at a health system’s documented specialist appointment completion rate through electronic medical record (EMR)-generated data.1 Closing the referral loop is vital for patient safety, regulatory, and financial reasons.
Closing the referral loop is critical to ensure patient safety, as the referral process is a point of patient vulnerability.2 During this transition of care, specialists may receive insufficient information or reporting of findings may be delayed, potentially endangering patients.3,4,5 One study showed just half of specialty referrals resulted in documented complete appointments.6 Finally, failure to close the referral loop can lead to underutilization of necessary specialty care, increasing patient morbidity.
For these reasons, regulatory bodies scrutinize closing the referral loop. The Joint Commission and The National Committee for Quality Assurance (NCQA) examines closing the referral loop for Patient-Centered Medical Home (PCMH) certification.7 Centers for Medicare & Medicaid Services (CMS) has included closing the referral loop as a component of meaningful use, and a potential measure in the Merit-based Incentive Payment Systems (MIPS) in 2017 fee-for-service Medicare reimbursement.1,8,9 This regulatory focus requires health systems to first develop data analysis infrastructure to measure rates of closing the referral loop, then develop processes to improve. As momentum linking closing the referral and patient safety grows, regulatory pressure can be expected to grow as well.10
Failure to close the referral loop can also be a financial liability for health systems, both in fee-for-service and value-based, population health financial models. In fee-for-service models, revenue is driven by volume of referrals to in-network specialists, which is influenced by patient outcomes, patient satisfaction, and referring provider satisfaction.11,12 Poor outcomes, patient dissatisfaction, and referring provider dissatisfaction, all of which can be caused by failure to close the referral loop, can lead PCPs to refer patients to out-of-network specialists, reducing referral volume and revenue.
In value-based reimbursement models, health systems prioritize outcomes, cost-savings, and network management. In these models, it is important to optimize in-network referrals, as referrals to out-of-network providers not part of a value-based or capitated model can impact quality monitoring and total cost. Thus, reducing referral leakage to out-of-network specialists is vital. Consistently closing the referral loop inside a health system may encourage PCPs to refer to in-network specialists and help reduce leakage and costs.
The importance of closing the referral loop in health systems is growing. Health systems continue to expand and become more complex.13,14 In general, referral volumes are increasing across the nation and protracted specialist wait times have become access issues for patients.15,16,17 Increasing volume and complexity of networks of care coordination, changing reimbursement models, and rising pressure from accrediting bodies make closing the referral loop critical for health systems of today, and even more important for health systems of tomorrow.
In the context of these systemic changes in healthcare, our objective was threefold: first, we assessed the current state of closing the referral loop; second, we investigated sources of deficiencies; third, we explored avenues for improvement.
Most previous studies have examined rates of closing the referral loop through chart audits, physician surveys, patient surveys, and billing data, but not through EMR data.6,18,19,20 Because it is difficult to regularly survey providers and patients, and because most health systems do not have ready access to insurance claims data, health systems are likely to use EMR data to demonstrate closing the referral loop moving forward.
To the best of our knowledge, this is the first paper to use EMR data to examine the rate of closing the referral loop from a large health system-based primary care network, to assess reasons for deficiencies in closing the referral loop, and to explore paths for improvement. Given the complexity and relatively new scrutiny placed on referral tracking, we suspect this analysis highlights challenges faced by other health systems, and can provide insight into root causes as well as opportunities for improvement.
We analyzed an Epic Systems EMR-generated global referral dataset of a primary care network associated with a large academic medical center. The network contained 34 clinics across 7 counties with over 700,000 patient encounters annually. This dataset included all referral orders made by PCPs and resulting specialty appointment scheduling attempts in Fiscal Year 2016 (July 1, 2015–June 30, 2016). Associated data fields included referral date, referral specialty, primary care clinic of origin, specialist appointment date, and appointment completion status. In this interval, there were 160,243 referral orders placed to specialists and 195,481 appointment scheduling attempts. 18.0% of referrals required multiple scheduling attempts (e.g., a patient who cancels and reschedules counts for two scheduling attempts, but one referral).
From the 195,481 scheduling attempts, we focused on 20 high-volume (> 1000 referrals) medical and surgical specialties from adult-continuity care providers (e.g., no urgent care or pediatric clinics), removing procedural referrals (e.g., colonoscopy). This filtering resulted in 90,437 referral orders and 103,737 scheduling attempts from 24 primary care sites.
This study was exempted by Duke IRB for the purposes of quality improvement. Datasets analyzed included patient health information (PHI) and are not publicly available to ensure patient privacy.
Main Measures and Process
The primary metric of a health system’s ability to close the referral loop was the proportion of scheduling attempts documented as “complete.” EMR documentation of “complete” occurs when a patient is seen by a specialist and completes the appointment linked to the referral. In the health system analyzed, PCPs and internal specialists share a common EMR, and documentation of “complete” referrals from internal specialists is immediately available to PCPs. In addition, we examined the end statuses of all other scheduling attempts (e.g., no status, canceled, and no-show). We then performed additional analyses to identify sources of deficits in closing the referral loop.
First, we examined documented appointment scheduling rate to assess the first stage a referral could be lost. Documented appointment scheduling rate was defined as proportion of scheduling attempts with an associated appointment date. We then looked at breakdown of in-network and out-of-network referrals for scheduling attempts without appointment date.
Second, we examined individual clinic heterogeneity in closing the referral loop within the health system by stratifying scheduling attempts by primary care site of origin and by referral specialty. We evaluated each clinic and each specialty’s documented appointment completion rate, documented appointment scheduling rate, median specialist wait time, percentage of scheduling attempts with wait times greater than 30 days, and total volume of scheduling attempts. We described variation using box plots with outliers.
Third, we examined patient access factors of wait times and geography by grouping scheduled appointments (those with appointment dates) into documented complete and documented incomplete (i.e., canceled, no show, and others) and compared the two groups’ mean wait times and mean driving distances between site of referral and nearest in-network referral specialist clinic. Wait time was defined as number of days between referral order and specialist appointment date. Google Maps data was used to determine driving distances between all primary care clinics and all in-network specialist clinics, and the minimum driving distance between primary care site of referral and in-network referral specialty clinic was associated with each appointment. Means were described with 95% confidence intervals and compared using two-sample t testing.
Of the 103,737 appointment scheduling attempts, 36,072 (34.8%) resulted in documented complete appointments. Of the remaining scheduling attempts, 18,531 (17.9%) were canceled, 4117 (4.0%) were no-shows, 4640 (4.4%) were other (e.g., scheduled post-analysis date), and 40,377 (38.9%) had no documented status (Fig. 2).
Documented appointment scheduling rate was 61.1%, as appointment dates were missing in 40,377 of referrals. Of these 40,377 attempts, 21,089 (52.2%) were categorized as in-network and 19,288 (47.8%) were out-of-network (Fig. 2). Thus, 25.0% of the 84,474 in-network appointments and 100% of the 19,288 out-of-network scheduling attempts lacked appointment dates.
Stratification by primary care site of referral origin and by referral specialty showed wide variation in documented appointment completion rates, appointment scheduling rates, and volume of scheduling attempts (Fig. 3). Referral specialties showed wide variation in median specialist wait times and percentage of specialist wait times greater than 30 days, but this variation was not seen on the level of primary care site (Fig. 3). Notably, specialty volume was not correlated with wait times or documented appointment completion rates.
After isolating for referrals with documented scheduled appointment dates, mean specialist wait times and mean distances to closest in-network specialist were found to be significantly different between appointments documented as complete and incomplete (Table 1).
The most striking takeaway is that the current documented rate of closing the referral loop is under 35% in the health system analyzed. Identified sources for this gap in closing the referral loop encompass three main categories: low documented appointment scheduling rate, individual clinic differences, and patient access issues.
Low Documented Appointment Scheduling Rate
Low documented scheduling rates in the EMR drives 40% of the gap in closing the referral loop in the health system analyzed (stage 2 of Fig. 1). This gap makes it very difficult to know downstream results of those referrals without individual chart review. All out-of-network referrals lacked documented appointment dates, representing nearly half of the referrals without documented appointment dates. Deficiency of out-of-network documented appointment scheduling is unsurprising given EMR incompatibilities between health systems. For patients or out-of-network providers, there is no mechanism to electronically share information about appointments back to PCPs. Even in the ideal situation where a referral is swiftly completed, the lack of EMR recording precludes the ability to use EMR-generated data to effectively monitor closing the referral loop.
Out-of-network referrals, however, did not explain all the gaps in documented appointment scheduling rates; in-network scheduling attempts accounted for roughly half of the scheduling attempts without documented appointment dates. There were several potential explanations. First, the review of EMR data shows 6% of all referred patients were unable to be contacted to be scheduled. Second, 12% of all referred patients declined scheduling for unclear reasons. Third, conversations with and direct observation of scheduling staff reveal that appointments can be scheduled without linking appointment dates to original referral orders (between stages 2 and 3 in Fig. 1). These barriers represent process challenges, which are unsurprising given increasingly large volumes of referrals ordered in health systems without thorough monitoring systems in place. While many of these appointments may have been completed in a method not recorded by EMR data, documented completion is the primary metric to assess closing the referral loop.
Individual Clinic Differences
Our analysis displays wide differences among individual clinics in rates of closing the referral loop at the level of primary care clinic and referral specialty (stage 3 in Fig. 1). The discrepancies between high- and low-performing primary care clinics and specialties in terms of documented specialist appointment completion rate are significant. Conversations with and direct observations of primary care clinic staff suggest that explanations for this discrepancy include differing referral coordination practices, patient population variation, process variation, and established connections with out-of-network specialists driving out-of-network referral volume (and lowering documented appointment scheduling rates, as discussed above). The low variation in wait time metrics for primary care sites is reasonable, as wait times are specialty driven and primary care sites send a relatively similar proportion of referrals to various specialties. Specialty demand is not a driving factor, as volume of referrals per specialty is not correlated with documented appointment completion rates or wait times.
Patient Access Issues
Patient access barriers are the final identified driver of deficiencies in closing the referral loop. The most notable barrier is wait times. Our analysis showed wait times for completed appointments were significantly lower than wait times for incomplete appointments. There are several potential explanations for this. As patients face extended wait times, their clinical issues may resolve, they could find out-of-network specialists with shorter wait times, or they could forget about the appointment. In any case, the referral loop is less likely to be closed. Notably, the relationship between wait time and completion rate was seen across specialties, suggesting this phenomenon is clinically agnostic.
Geographic access is also important. Our analysis showed shorter distances between primary care clinic and specialty clinics for completed scheduling attempts compared to incomplete scheduling attempts. While the magnitude of the difference is small at roughly half a mile, the significant difference highlights the smaller, but still concrete effect of geographic access on documented appointment completion rates. Patients with transportation barriers can be hindered by distance. Additionally, if in-network care is further away, patients may opt for out-of-network care. Finally, specialty density may be lower in areas with lower SES, exacerbating the effect of distance. Further geospatial analysis is necessary to better understand the impact of geography on closing the referral loop.
Paths for Improvement
Identified impediments—namely low documented scheduling rates, individual clinic variation, wait times, and geographic access barriers—should guide improvements to closing the referral loop.
First, low documented appointment scheduling rates must be addressed. Health systems must optimize current EMR processes and data capture to better track referral flow. For example, EMRs should automatically link appointment dates with referrals to minimize “lost” appointments (those without documented appointment dates). For out-of-network referrals, improved EMR interoperability is needed to track referral outcomes. Several health systems and companies have invested in referral tracking systems, with initial results of increased revenue, reduced referral leakage, and reduced cost.21,22,23 Reducing the data gap in EMR-generated reports is vital to accurately measure closing the referral loop.
Second, health systems should disseminate best practices from high-performing primary care clinics and referral specialties. Health systems could develop newer performance metrics for referral scheduling, such as rates of closing the referral loop, appointment scheduling, and specialty wait times, to better manage this process. Health systems could also consider other innovative solutions system-wide to decrease individual clinic differences and improve rates of closing the referral loop; these include machine learning scheduling algorithms, application of quality improvement methodology to referrals, and more direct patient scheduling.24,25,26,27,28,29,30,31
Finally, health systems must improve specialty wait times and geographic access barriers. Given specialty referral volume is not correlated with wait times, wait times likely reflect a supply-demand mismatch, where even a small volume of referrals can outstrip provider capacity. Thus, specialties with poor patient access could decrease wait times by hiring simply hiring more providers, but this can be expensive and time-consuming. When expanding primary care networks, health system leadership should consider specialist wait time and distance from primary care clinics to evaluate how added primary care volume affects systemic rates of closing the referral loop. Innovative solutions include e-consultations, where PCPs can virtually consult specialists with reduced wait times, and telemedicine visits, which can ease wait time and geographic access barriers.32,33,34,35,36,37 As these factors were analyzed only for appointments with documented dates, they represent true patient access barriers, not process challenges outlined elsewhere in this paper.
First, our analysis was limited to a single large, academic health system using Epic Systems EMR. However, many of the challenges discussed above are generalizable to large health systems with complex referral networks. Additionally, as Epic is one of the largest providers of EMR systems in the USA, other health systems likely encounter similar hurdles. Second, the study lacked specialist-to-specialist referrals, referrals from out-of-network PCPs, and patient self-referrals. Third, this analysis is limited to adult-continuity PCPs and their referrals to non-procedural, high-volume specialists. Fourth, our proxy measurement of wait time as days between referral order and appointment date is limited, as some patients choose to schedule later appointment dates than those available. Because this proxy measurement was applied to both complete and incomplete appointments, this discrepancy was adjusted for. Fifth, our proxy measurement of geographic access is limited, as true geographic access measurement would be distance between each patient’s address and the specialist clinic assigned to the appointment. As we had no data on patient addresses or specific specialty clinic assigned, we applied the proxy measures of clinic address and nearest specialty clinic location only to scheduled appointments (nearly exclusively in-network referrals). Sixth, our analysis lacked patient factors, such as insurance status, income, and age, but the problems we identified were present across all specialties and clinic sites, which have heterogeneous populations. Finally, the primary metric of documented “complete” appointments works because specialist results are immediately available to the PCP upon appointment completion due to the shared EMR in the system analyzed; in other systems, this measure would fall short of determining whether the PCP receives specialist documentation.
An analysis of EMR data of referrals from a large primary care network within an academic health system shows just under 35% of appointments has a documented completed specialty appointment. Scheduling issues, individual clinic differences, and patient access barriers contributed to this low rate. As the importance of closing the referral loop in health systems is increasing, there will be pressure to better measure and improve closing the referral loop. While they have limitations, EMR-generated reports are likely the most efficient method of measuring closing the referral loop and systems will need to optimize their electronic capture of referrals data to minimize these gaps. In addition, health systems will need to consider reimagining the referral scheduling process and addressing patient access factors that are driving lower completion rates.
Additional Information Regarding EH Clinical Quality Measures. In: Services CfMM, ed. Baltimore, MD: United States Federal Government; 2014.
Mehrotra A, Forrest CB, Lin CY. Dropping the baton: specialty referrals in the United States. Milbank Q. 2011;89(1):39–68.
Gandhi TK, Sittig DF, Franklin M, Sussman AJ, Fairchild DG, Bates DW. Communication breakdown in the outpatient referral process. J Gen Intern Med. 2000;15(9):626–631.
Forrest CB, Glade GB, Baker AE, Bocian A, von Schrader S, Starfield B. Coordination of specialty referrals and physician satisfaction with referral care. Arch Pediatr Adolesc Med. 2000;154(5):499–506.
Stille CJ, Jerant A, Bell D, Meltzer D, Elmore JG. Coordinating care across diseases, settings, and clinicians: a key role for the generalist in practice. Ann Intern Med. 2005;142(8):700–708.
Weiner M, Perkins AJ, Callahan CM. Errors in completion of referrals among older urban adults in ambulatory care. Journal of evaluation in clinical practice. 2010;16(1):76–81.
Assurance NCfQ. PCMH eligibility. 2017. http://www.ncqa.org/programs/recognition/practices/patient-centered-medical-home-pcmh/before-learn-it-pcmh/pcmh-eligibility. Accessed 26 January 2018.
Services CfMM. Quality measures. 2017. https://qpp.cms.gov/mips/quality-measures. Accessed January 26, 2018.
2009. National patient safety goals. In: Organizations TJCoAoH, ed. Joint commission resources. 2008.
2017. National Patient Safety Foundation and CRICO to Collaborate on Referral Management Best Practices [press release]. PRWeb, January 26, 2018.
Fahey P, Cruz-Huffmaster D, Blincoe T, Welter C, Welker MJ. Analysis of downstream revenue to an academic medical center from a primary care network. Academic medicine : journal of the Association of American Medical Colleges. 2006;81(8):702–707.
Kinchen KS, Cooper LA, Levine D, Wang NY, Powe NR. Referral of patients to specialists: factors affecting choice of specialist by primary care physicians. Annals of Family Medicine. 2004;2(3):245–252.
Cuellar AE, Gertler PJ. How the expansion of hospital systems has affected consumers. Health affairs (Project Hope). 2005;24(1):213–219.
Cuellar AE, Gertler PJ. Trends in hospital consolidation: the formation of local systems. Health affairs (Project Hope). 2003;22(6):77–87.
Widdifield J, Bernatsky S, Thorne JC, et al. Wait times to rheumatology care for patients with rheumatic diseases: a data linkage study of primary care electronic medical records and administrative data. CMAJ open. 2016;4(2):E205–212.
Jaakkimainen L, Glazier R, Barnsley J, Salkeld E, Lu H, Tu K. Waiting to see the specialist: patient and provider characteristics of wait times from primary to specialty care. BMC family practice. 2014;15:16.
Barnett ML, Song Z, Landon BE. Trends in physician referrals in the United States, 1999-2009. Arch Intern Med. 2012;172(2):163–170.
Hacker KA, Weintraub TA, Fried LE, Ashba J. Role of school-based health centers in referral completion. The Journal of adolescent health : official publication of the Society for Adolescent Medicine. 1997;21(5):328–334.
Byrd JC, Moskowitz MA. Outpatient consultation: interaction between the general internist and the specialist. J Gen Intern Med. 1987;2(2):93–98.
Bourguet C, Gilchrist V, McCord G. The consultation and referral process. A report from NEON. Northeastern Ohio Network Research Group. The Journal of family practice. 1998;46(1):47–53.
Rowe J. 3 benefits of eReferral systems. 2013. http://www.healthcareitnews.com/news/3-benefits-e-referral-systems. Accessed January 26, 2018.
Dydra L. Make patient referrals count: how a new technology platform impacts patient care & provider economics. 2014. http://www.beckershospitalreview.com/healthcare-information-technology/make-patient-referrals-count-how-a-new-technology-platform-impacts-patient-care-provider-economics.html. Accessed January 26, 2018.
Baum S. A Q&A with AristaMD CEO: The shifting landscape of the specialist referral software sector. 2016. http://medcitynews.com/2016/07/physician-referral-software-sector-shifting/. Accessed January 26, 2018.
Deckard GJ, Borkowski N, Diaz D, Sanchez C, Boisette SA. Improving timeliness and efficiency in the referral process for safety net providers: application of the Lean Six Sigma methodology. J Ambul Care Manage. 2010;33(2):124–130.
Esquivel A, Sittig DF, Murphy DR, Singh H. Improving the effectiveness of electronic health record-based referral processes. BMC Med Inform Decis Mak. 2012;12:107.
Kaandorp GC, Koole G. Optimal outpatient appointment scheduling. Health care management science. 2007;10(3):217–229.
Kwong T. Patient access: improving wait times in a specialty clinic. The health care manager. 2016;35(1):72–79.
O'Neill S, Calderon S, Casella J, Wood E, Carvelli-Sheehan J, Zeidel ML. Improving outpatient access and patient experiences in academic ambulatory care. Academic medicine : journal of the Association of American Medical Colleges. 2012;87(2):194–199.
Snow BW, Cartwright PC, Everitt S, Ekins M, Maudsley W, Aloi S. A method to improve patient access in urological practice. The Journal of urology. 2009;182(2):663–667.
Waldron MA, Scott K. Improved scheduling operations in diagnostic imaging. Radiology management. 2013;35(1):30–35; quiz 36-37.
Weiner M, El Hoyek G, Wang L, et al. A web-based generalist-specialist system to improve scheduling of outpatient specialty consultations in an academic center. J Gen Intern Med. 2009;24(6):710–715.
Davis A, Gilchrist V, Grumbach K, James P, Kallenberg R, Shipman SA. Advancing the primary/specialty care interface through eConsults and enhanced referrals. Ann Fam Med. 2015;13(4):387–388.
Chen AH, Murphy EJ, Yee HFJ. eReferral—a new model for integrated care. New England Journal of Medicine. 2013;368(26):2450–2453.
Gleason N, Prasad PA, Ackerman S, et al. Adoption and impact of an eConsult system in a fee-for-service setting. Amsterdam, Netherlands: Healthcare; 2016.
Keely E, Liddy C, Afkham A. Utilization, benefits, and impact of an e-consultation service across diverse specialties and primary care providers. Telemed J E Health. 2013;19(10):733–738.
Kim Y, Chen AH, Keith E, Yee HF Jr, Kushel MB. Not perfect, but better: primary care providers’ experiences with electronic referrals in a safety net health system. J Gen Intern Med. 2009;24(5):614–619.
Kim-Hwang JE, Chen AH, Bell DS, Guzman D, Yee HF Jr, Kushel MB. Evaluating electronic referrals for specialty care at a public hospital. J Gen Intern Med. 2010;25(10):1123–1128.
The authors would like to acknowledge Jonathan Woodall, Duke Institute for Health Innovation, and Duke Primary Care. Results of this work were presented at Society of General Internal Medicine meeting (April 2017), American College of Physicians meeting (April 2017), the North Carolina ACP meeting (February 2017), and Duke Patient Safety Conference (March 2017).
Conflict of Interest
Dr. Bosworth receives research funds through Duke University from NIH, VA, Pharma foundation, Johnson & Johnson, Improved Patient Outcomes, Takeda, and Sanofi. He has received honorarium from Sanofi, Otsuka pharmaceuticals, Genentech, and has been a member of Boehringer Ingelheim Speaking bureau. Other authors report no conflicts of interest.
About this article
Cite this article
Patel, M.P., Schettini, P., O’Leary, C.P. et al. Closing the Referral Loop: an Analysis of Primary Care Referrals to Specialists in a Large Health System. J GEN INTERN MED 33, 715–721 (2018). https://doi.org/10.1007/s11606-018-4392-z
- patient safety
- access to care
- managed care
- population health
- primary care redesign