INTRODUCTION

Diabetes is a major source of morbidity and mortality in the USA with over 1.5 million new cases diagnosed every year.1 Diabetes is monitored using hemoglobin A1c (HbA1c) as it correlates with burden of disease.2,3,4 Clinical practice guidelines recommend HbA1c be repeated every 3 months if a patient’s value is above 7%, because a higher HbA1c correlates with worse outcomes.5,6,7,8,9 Complications are particularly high in patients with an HbA1c above 10% and even a 1% reduction in HbA1c can reduce complications by 21%.5 Thus, while regular monitoring of above-goal HbA1c levels is recommended, ensuring patients with an HbA1c over 10% have timely follow-up testing is especially important.

The adoption of electronic health records (EHRs) in clinical care has led to an abundance of data that can be used to improve care delivery.10 One such application has been using data from EHRs to identify errors and adverse events. From this came the concept of building “triggers” or “trigger tools.”11,12 Murphy et al. defined a trigger as a tool that “scan EHR data for clinical and diagnostic clues to identify patients at risk of harm so that their records can be evaluated”.12 These tools use algorithms to identify values, diagnoses, or other pieces of information within an EHR that signal the potential for error or an adverse event.13,14 Early on, triggers were used to detect medication errors and retroactively review those errors, eventually leading to prompts prior to signing an order.13,15,16,17 Now triggers are used throughout various departments to detect past adverse events or delays in care within hospital systems.13,16,18,19,20,21,22,23,24,25,26,27,28 While much of the existing literature on trigger development has focused on evaluating medical errors, their optimal use in clinical care would be to reduce harm in the moment and proactively improve and guide care.

In particular, there are opportunities to use triggers to enhance the management of chronic disease such as poorly controlled diabetes. Triggers can be utilized to accurately identify patients with poorly controlled diabetes in need of follow-up and inform systematic solutions. The purpose of this study was to design a trigger that would identify patients with an HbA1c over 10% who lacked appropriate follow-up HbA1c testing and compare this to the gold standard of chart review by clinicians. Then, we used the trigger to estimate prevalence of delayed follow-up testing, and determine contributing factors for delayed follow-up testing. If we found the prevalence to be high and contributing factors consistent, we could seek to use the trigger to develop an intervention in hopes of improving monitoring and care of patients with diabetes.

METHODS

Study Design

This was a retrospective cohort study at a large, integrated health system using EHR data. The study consisted of three components. First, we developed, refined, and validated a trigger to identify patients with an elevated HbA1c > 10% and delayed follow-up HbA1c testing. Second, we used this trigger to estimate the prevalence of delayed follow-up HbA1c testing in the health system. Third, we sampled trigger positive charts and conducted a qualitative analysis of these charts to understand reasons for delayed follow-up testing. The study team consisted of a multidisciplinary group of clinicians, researchers, outpatient administrative leadership, and quality improvement experts.

Study Setting and Patient Population

The study took place at the NYU Langone Health System, an academic health system with a unified EHR system (Epic, Epic Systems, Verona, WI). The health system includes over 300 ambulatory care sites that are primarily located in New York City and surrounding areas. The majority of patients come from the greater New York City area but our study did not limit eligibility based on home address. The ambulatory care sites are faculty group practices plus a large Federally Qualified Health Center network. Patients were eligible in our study if they were age 18 years or older and had an HbA1c > 10%. Eligibility was not restricted to those with a diagnostic code for diabetes. We included patients from all clinics in the health system.

Trigger Development and Validation

To develop the trigger, the study team iteratively defined and refined denominator definitions, representing all patients who were eligible for the care measure of appropriate follow-up HbA1c testing, and numerator definitions, defined as those patients in the denominator who did not meet the care measure of appropriate follow-up HbA1c testing. Although guidelines recommend follow-up within 3 months for patients with inadequately controlled diabetes, we iterated our definition of appropriate HbA1c follow-up (defined in the numerator) based on our review.7 To develop the trigger, we first extracted a cohort of patients who met our initial denominator definition during calendar year 2018. One clinician then reviewed a random sample of cases that were trigger positive, i.e., met both the initial numerator and denominator criteria, by performing detailed chart review. The clinician determined whether cases were true positives (follow-up HbA1c testing should have been obtained and was not) versus false positives (a follow-up HbA1c was obtained) and made recommendations on how to adjust the definition to reduce the false positive rate. These included recommendations to reduce missed data (e.g., inclusion of follow-up HbA1c tests from outside labs, when available), more accurately define the patient population (e.g., exclusion of patients who died following the initial test), and refining follow-up time definitions (e.g., extension of time of a follow-up HbA1c testing from 3 to 6 months to provide sufficient time and be more consistent with guidelines).29 Notably, the reviewers could not determine if there was an undocumented false positive, which would occur if a follow-up result was obtained outside of our institution and the test was not noted in the EHR. The full multidisciplinary team met collaboratively to review these recommendations and the denominator and numerator were subsequently refined based on the discussion. This process of extraction, sampling of twenty charts, clinician review, multidisciplinary discussion, and denominator and numerator iteration was repeated until “saturation” was reached. Specifically, we stopped iterating the definition when no new changes to the definition of the numerator or denominator were identified in review. Using the final definitions, we randomly sampled an additional thirty charts to confirm that no other changes were needed. This last sample was reviewed by two clinicians with 100% concurrence between reviewers. We found that fifteen to twenty charts were sufficient to capture a number of false positives and identify areas for improvement.

To define characteristics of the final trigger, we randomly sampled 70 charts from January 1 to November 30, 2018. This sample included charts that were trigger positive, i.e., met the numerator inclusion criteria for delayed follow-up testing, and those that were trigger negative, i.e., met denominator inclusion but had appropriate follow-up testing. At least one of the two clinician reviewers assessed each sampled chart to determine whether appropriate follow-up testing was completed or not, with an overlap of 20 to ensure consistency of review. Reviewers were blinded as to whether the chart was trigger positive or trigger negative. These chart reviews were then used as the gold standard to determine the sensitivity, specificity, PPV, and negative predictive value (NPV) of the final algorithm as well as exact binomial confidence intervals for these estimates. Finally, to estimate prevalence of patients meeting our final trigger definition, we separated the 70 patient charts that were reviewed for trigger validation into those that were trigger positive and those that were trigger negative. We estimated the prevalence of delayed follow-up HbA1c testing in each group and used this as an estimated prevalence in the overall cohort.

Evaluation of Reasons for Delayed Follow-up HbA1c Testing

To assess reasons for delayed follow-up HbA1c testing, we performed an in-depth, qualitative review of patient charts identified as trigger positive and confirmed by chart review. To do this, we used a constant comparative analytic approach.30 Initially, three clinicians reviewed a set of 15 charts and then met to discuss emergent concepts for causes of delayed follow-up HbA1c testing. This led to an initial list of reason codes. In qualitative research, codes are labels that are assigned to text and summarize important concepts.30 The three reviewers then independently reviewed and coded these initial 15 charts plus an additional 15 charts. The team met again to further discuss and refine the reason codes and review discrepancies until consensus was reached. From this discussion, we categorized codes into primary and secondary reason codes for delayed follow-up testing. Primary reason codes reflected specific missed actions or items leading to delayed follow-up testing, while secondary reason codes were contributing causes or sub-codes of the primary reasons. Reviewers then independently reviewed an additional 20 charts and, at that point, determined that thematic saturation had been achieved. The team concurrently identified overarching themes based on related primary reason codes.

To estimate frequency of codes for delayed follow-up HbA1c testing, additional trigger positive charts were randomly sampled. We sampled a total of 100 trigger positive charts for ease of calculation. Primary and secondary reason codes were applied to charts based on the code list from the final codebook. To ensure reliability, there was an overlap of 20% of charts by two reviewers. All differences were reviewed and discussed until consensus was reached. Descriptive statistics were used to determine frequencies of primary and secondary reason codes for delayed follow-up HbA1c testing. Finally, to understand the potential use of the patient portal as a means for future intervention, we determined the number of patients in the trigger positive sample who used the EHR patient portal. We defined patient portal use as direct messaging with providers or refill requests through the application.

RESULTS

Trigger Development

We refined the trigger definition over four iterations, at which point we reached saturation with no new suggested changes upon completion of chart review. The initial denominator was defined as all patients with HbA1c over 10% (Table 1). The initial numerator, representing patients without an appropriate follow-up HbA1c result, was defined as patients without a follow-up clinic visit within 3 months of the index HbA1c test. The denominator for the final definition was all adult patients with an HbA1c > 10% who had not subsequently died at time of trigger pull. Time of trigger pull was chosen rather than within 6 months of index given the plan for eventual intervention that may involve contacting the patient and/ or provider at time of trigger pull. The final numerator was patients without a follow-up HbA1c result within 6 months of the index test. The follow-up test could be either listed in the laboratory results section or outside laboratory results that were scanned into the EHR. Table 1 displays details of each trigger definition iteration.

Table 1 Iterations of trigger definition until final definition was reached
Full size table

Based on clinician chart review, the final trigger had a high sensitivity and specificity and a PPV of 89% and a NPV of 100% for detection of delayed follow-up of HbA1c (Table 2).

Table 2 Characteristics of final trigger to detect delayed follow-up of elevated HbA1c, based on gold standard of clinician review
Full size table

Estimate of Prevalence of Delayed Follow-up Testing

Between January 1 and November 30, 2018, 6228 patients had an HbA1c greater than 10%. Of these patients, 3131 (50.3%) were found to be trigger positive, i.e., had no follow-up HbA1c result within the subsequent 6 months based on the EHR data pull, while 3097 (49.7%) were trigger negative, i.e., were found to have a follow-up HbA1c result in the EHR data pull. Based on PPV and NPV of the trigger, we estimated that 2787 (95% CI 2313–3033) of trigger positive patients had no follow-up HbA1c result and 0 (95% CI 0–319) of trigger negative patients had a follow-up HbA1c result. The prevalence of delayed follow-up testing in the overall cohort was 45% (95% CI 33–57%).

Evaluation of Reasons for Delayed Follow-up HbA1c Testing

In a sample of trigger positive 100 charts, we identified six primary reason codes for delayed follow-up HbA1c testing, which fell into three themes (Table 3): (1) no provider accepting responsibility of diabetes management, (2) lack of in-person follow-up after index HbA1c result, and (3) lack of appropriate follow-up HbA1c testing despite in-person follow-up. Our first theme was no provider accepting responsibility of diabetes management and consisted of two codes: outside diabetes provider and no provider within our health system accepting responsibility for diabetes management. The outside diabetes provider code was assigned when a provider within our health system deferred responsibility for HbA1c follow-up to an outside provider based on comment in charts or patient’s response, although we were often unable to find evidence that such follow-up had occurred. We coded charts as no NYU provider taking responsibility when the initial HbA1c was ordered during hospitalization or by a clinician who did not continue to monitor the patient’s diabetes despite in-person follow-up and there was no specific outside provider mentioned. The second theme, lack of in-person follow-up after index HbA1c result, was the most common and accounted for 70% of the charts reviewed. In other words, at our institution, follow-up HbA1cs testing often only occurs if a patient returns for an in-person clinic visit. Within this theme, delayed in-person follow-up was coded when a patient had a clinic appointment where diabetes was addressed and follow-up testing ordered but beyond 6 months. No follow-up was coded when a patient had no follow-up for diabetes management at our institution. Finally, the third major theme was lack of HbA1c testing despite in-person follow-up. This could occur when a patient had an appointment with a provider who mentioned diabetes management within 6 months but did not order a follow-up HbA1c test. This theme also included a code for when an HbA1c was ordered by the provider but the lab draw never occurred (Table 3).

Table 3 Primary reason themes and codes for incomplete HbA1c follow-up based on sample of 100 charts reviewed
Full size table

During our review, we also identified a set of secondary reason codes for delayed follow-up HbA1c testing. The codes related to the processes involving lack of in-person follow-up were most common and included an appointment given but missed and in-person follow-up advised but no appointment given (Table 4). Other secondary reason codes reflected additional contributing factors to lack of appropriate follow-up HbA1c testing and included the patient being out of the country or state, the index HbA1c being ordered during an inpatient or ED visit, the patient not having a PCP assigned in our health system when the index HbA1c was drawn, and patient loss of health insurance. These data are summarized in Table 4.

Table 4 Among appropriate clinical follow-up, sub-categories for lack of appropriate follow-up testing as a percentage of total 100 charts
Full size table

Finally, we determined the percentage of the 70 people without in-person follow-up who had either refills granted through the EHR patient portal or sent a message to the provider via the patient portal. We found the patient portal use to be 32% of the 70 patients.

DISCUSSION

In our retrospective study at a large, integrated health system, we developed a highly sensitive and specific trigger to identify patients with poorly controlled diabetes who lacked appropriate follow-up testing via an iterative review process. Using this trigger, we found that delayed follow-up HbA1c testing is extremely common among those with poorly controlled diabetes, with 45% of patients with an HbA1c over 10% not receiving follow-up testing within 6 months. Qualitative analysis of trigger positive charts revealed three overarching themes describing potential gaps in care related to appropriate follow-up: no provider accepting responsibility of diabetes management, lack of in-person follow-up after index HbA1c result, and lack of HbA1c testing despite in-person follow-up. We identified a number of factors that may contribute to these themes, including missed appointments, transitions of care from an inpatient encounter, and lack of insurance.

Our final algorithm demonstrated excellent sensitivity and specificity for detection of a care gap through use of large clinical data. Other clinical decision support (CDS) tools and electronic triggers that have previously been developed rely on automated data algorithms like ours, but most do not perform as well as ours.12,25,31,32,33,34 Furthermore, few describe a process of iterative refinement based on detailed data review. Our experience was that a seemingly straightforward algorithm needed four iterations and multidisciplinary input before reaching a high accuracy that was ready for clinical deployment. This work suggests that, in general, algorithms that form the underlying basis for CDS or triggers should undergo such refinement in order to minimize errors.

We found that almost half of patients with poorly controlled diabetes lacked a follow-up HbA1c result within 6 months. This data is consistent with other studies reporting testing adherence that shows rates of appropriate HbA1c testing ranging from 25 to 80%.35,36,37,38,39,40,41 Most of these studies similarly used a 6-month cut-off but included patients with an HbA1c over 7%, not over 10%. Thus, our study supports the need for improvement in testing processes, as even patients at the highest risk for diabetic complications did not get appropriate HbA1c follow-up testing.

We found that the reasons for delayed follow-up testing reflect three main system-related steps involved in usual follow-up of HbA1c testing at our institution. First, a provider assumes responsibility of diabetes management or refers the patient to another provider for in-person follow-up visit. Second, the follow-up visit is scheduled and attended. Third, the test is ordered and completed. We have conceptualized the usual process at our institution in Fig. 1.

Fig. 1
figure 1

Process map showing usual process to obtain a follow-up HbA1c test at our institution.

Full size image

Most of our cases fell into the theme of lack of in-person follow-up after index HbA1c result, either because the in-person follow-up occurred beyond 6 months or the patient never followed up in clinic. This finding may suggest that our institution’s usual process (Fig. 1) is too rigidly dependent on a patient attending an in-person clinic visit in order to have a follow-up HbA1c ordered and drawn. Thus, we are focusing our potential solutions to improve diabetes evaluation and management during this time between physical appointments. A number of technologies are increasingly available to help change this paradigm including telemedicine text-based disease monitoring and patient portals which allow for easy communication with patients.42,43,44,45 Notably, 32% of the 70 of patients with no in-person follow-up had interactions with their physicians via the EHR through patient portal messages or refill requests—even in the no in-person follow-up group. These findings suggest that there are opportunities to improve diabetes management through focusing outside of in-person clinic visits.

Multiple other factors may contribute to missed follow-up appointments and testing. For instance, 12% of cases had the initial HbA1c drawn during an inpatient or ED visit. Thus, one area of improvement relates to improved follow-up planning prior to discharge. We also found a small number of cases in which loss of insurance played a role, although this factor is likely under-reported as it relies on a patient eventually returning to clinic, discussing the issue, and then it being documented in the chart. Therefore, health insurance and other social determinants of health likely play an important role in delayed follow-up testing.

A key study limitation was that our review was limited to the EHR from one health system and missed any patient interactions with other health systems. Given the high density of healthcare services within New York City, it is possible that a patient may have had an HbA1c drawn at our health system and then have a subsequent HbA1c drawn at another institution. As a result, our reported prevalence rate for delayed follow-up may be an over-estimate. Nonetheless, in caring for the complete patient, we believe that ensuring—and documenting—appropriate follow-up for poorly controlled diabetes is warranted. We also did not perform an a priori sample size calculation for chart review and thus may have been underpowered in our estimations.

Other limitations deserve consideration. First, our study was limited to chart review, making it provider-centric and lacking information on patient experience other than when documented by the providers. While physician and patient interviews were beyond the scope of this chart review study, future qualitative assessment would be useful to better understand reasons for the lack of follow-up. Second, we did not collect data on patient demographics or compare the demographics of trigger positive and trigger negative patients. While we included all individuals presenting to our diverse health system, the lack of specific characteristics may limit generalizability of our findings. Third, while some patient-related issues like cost of care were occasionally documented in the EHR, we are presumably missing information on other patient-related issues. Fourth, we did not take into account patient home glucose monitoring or the use of finger-stick measurements as an acceptable alternative to HbA1c. Fifth, our trigger development focused on improving PPV rather than sensitivity or NPV as we were concerned that many false positives could lead to trigger fatigue and limit the effectiveness of any trigger-related intervention. Nonetheless, in data validation, we found both sensitivity and NPV to be 100%. Finally, we hypothesize that diabetes HbA1c monitoring correlates with alterations in care to improve a patient’s diabetes. However, we will not know if improvements in diabetes monitoring will correlate with improvements in outcomes at this time.

Overall, this study has three primary implications. First, building an accurate trigger takes iterative review and multidisciplinary input. Our trigger performed better than many previously reported, and we believe this was due to our strategy for trigger development. Second, while diabetes is a commonly treated disease, there is clearly a great need to improve monitoring of HbA1c in the outpatient setting. Third, the reasons for delayed follow-up HbA1c testing are varied and a solution will require a multipronged approach.