Background

Optimizing primary care clinical management of chronic kidney disease (CKD) remains a critical step to reduce overall disease burden [1]. The widespread adoption of electronic health records (EHR) in the United States has garnered excitement about accomplishing this goal by transforming the clinical environment into a “learning health care system” [2,3,4]. In this context, a learning healthcare system is one that can collect and store accurate information for each patient, and in turn, use these data to inform and support improvements in clinical care [5, 6]. Pragmatic randomized trials using the EHR to identify individuals with CKD and quantify the gaps in their care, followed by interventions to improve outcomes, can rigorously test the promise of these learning healthcare systems. In fact, the National Institutes of Health (NIH) has published several requests for applications explicitly focused on design and evaluation of information technology-based tools and interventions to improve CKD care [7, 8].

Necessary pre-requisites to the successful design, implementation, interpretation and dissemination of the results of such pragmatic trials include data accuracy and adequate quantification of gaps in care [9]. Specifically, prior to the design and implementation of EHR-based pragmatic trials, investigators must be confident in the ability of the existing data to identify patients with CKD who are at risk for complications, deliver an intervention with a high probability of improving care, and ascertain relevant outcomes [10]. Accurate phenotyping (i.e. the ability to accurately classify disease status) is important because misclassification can introduce bias and limit interpretation of results [11, 12]. The few pragmatic studies that have used EHR data designed to improve care in individuals with CKD have not reported detailed methodology regarding phenotyping process or ascertainment of the clinical gaps prior to design and implementation [13,14,15,16,17]. Prior registry studies that have validated CKD diagnostic codes to classify CKD status were primarily conducted in inpatient hospital settings, and even fewer have used clinician chart review as a gold standard [18, 19]. One of the largest validated EHR-based CKD registries did not specifically focus on patients actively seen in primary care practices [20]. Investigators must also be able to quantify the clinical practice gap(s) and the potential for resolution of that gap(s) to improve outcomes (i.e. “actionable gap”) [21]. For example, the lack of PCP awareness of CKD has been cited as a major barrier to optimal CKD care, but the degree and importance of this lack of awareness may vary greatly by setting [22, 23]. Thus we see a gap in knowledge between the specific methods used to identify CKD patients in the EHR and the evaluation of the appropriate guideline-driven care practices among the identified patients.

In this methodologic study, we set out to investigate the use of historical laboratory data to identify a cohort of primary care patients with CKD, and, in a subset of the cohort, have clinical nephrologists confirm a CKD diagnosis and the presence or absence of comorbidities known to influence CKD via manual chart review. Additionally, we evaluated the prevalence and usefulness of CKD documentation in the EHR problem list as a surrogate for primary care provider (PCP) awareness of CKD status by assessing its association with guideline-driven care [24].

Methods

Setting and cohort development

We used data from patients regularly seen in primary care at the University of California San Francisco (UCSF) General Internal Medicine (GIM) practices from 2014 to 2016. This practice provides comprehensive primary care internal medicine services for a diverse population of adults with a mixture of public and private insurance. UCSF medical professionals use Epic (Epic Systems Corporation) for appointment scheduling, laboratory and medication orders, recording progress notes, documenting diagnoses, prescription management, and communication between providers and patients [25]. The problem list contains the current and active diagnoses for each patient. The UCSF Institutional Review Board approved the data and methods for this project.

We’ve outlined the data extraction steps in Fig. 1. Our goal was to identify a cohort of patients with CKD defined by historical laboratory values and who also had regular follow-up in primary care. We define this as the overall CKD cohort, and it was created by identifying patients aged 18–80 with at least two serum creatinine measurements at least 90 days apart. We calculated each patient’s eGFRcreat using the CKD Epi equation when not automatically reported [26]. To enter the overall CKD cohort, we required that patients have two eGFRcreat between 15 and 59.999 ml/min per 1.73m2 at least 90 days apart, with their second “qualifying” eGFRcreat in the period from 1/1/2014 and 12/31/2016. To include patients with regular follow-up, we also limited the overall CKD cohort to patients with at least one primary care encounter in the UCSF GIM practice between 1/1/2014 and 12/31/2016. We selected 50 random charts from the overall CKD cohort for validation as described below (Measures).

Fig. 1
figure 1

Patient cohort electronic health record data extraction

Full size image

We then derived a sub-cohort of patients for whom we had complete data on laboratory measurements, medications, and outpatient encounters (the guideline-concordant care cohort in Fig. 1). To ensure a complete set of data, we further restricted the guideline-concordant care cohort to patients who had their second “qualifying” eGFRcreat between 1/1/2014 and 12/31/2015. We also excluded patients from the guideline-concordant care cohort if they had problem list documentation of 1) receiving a kidney transplant or 2) having end-stage renal disease (ESRD) during the study period.

Measures

CKD diagnosis: Problem list and nephrologist confirmation by chart review

We defined “Problem List CKD” as the presence of an EHR problem list diagnosis of CKD in each cohort using the following International Statistical Classification of Diseases, Ninth Revision, Clinical Modification (ICD9) codes: 585, 585.1, 585.2, 585.3, 585.4, 585.4, 585.5, 585.6, 585.9. Epic© records patient diagnoses with an internal identifier to ensure a physician can accurately capture the patient’s condition according to the current ICD code, while maintaining backwards compatibility with previous diagnosis codes.

We defined ‘nephrologist confirmed CKD’ via chart review of a random sample of 50 charts in the overall CKD cohort. Two nephrologists (LL & AM) were asked to review the 50 charts only using data from January 1, 2013 to December 31, 2016. They were instructed to examine labs, echo, imaging, notes, medications, and vital signs to determine if each patient had the following conditions: hypertension, chronic kidney disease, diabetes, coronary artery disease, peripheral vascular disease, congestive heart failure, or cerebrovascular disease (defined as a history of stroke or TIA). The responses were binary (Yes/No), and in the event of a disagreement, a third nephrologist (CP) served as the tie-breaker.

Since we were also interested in determining the validity of using a CKD diagnosis on the problem list as a proxy for CKD awareness by the primary care provider, the reviewers were also asked to read the primary care physician’s notes to determine if they thought the PCP was, “aware that the patient has CKD.” We refer to this measure as “CKD awareness”. All three reviewing clinicians used a combination of the guideline-concordant definition of CKD with clinical judgment to determine each patient’s CKD status [24].

Demographics and comorbidities

We extracted demographic characteristics from the EHR including age (at the time of the patient’s second ‘qualifying’ eGFRcreat), sex, ethnicity (Hispanic or Latino, Not Hispanic or Latino, Not Specified), and race (African American, Asian, White, and Other/Unknown). Patients who indicated multiple races and included Black or African American, were considered Black or African American due to the impact on eGFRcreat. We also extracted five comorbid conditions from the EHR: cerebrovascular disease, congestive heart failure, coronary artery disease, diabetes mellitus, and hypertension. We defined the presence of CKD and each comorbid condition with an ICD9 documented in the patient’s (Additional file 1: Appendix Table S1) problem list or associated with an outpatient encounter.

Outcomes: Guideline-concordant processes of care

We selected the processes of care outlined in the 2012 Kidney Disease, Improving Global Outcomes (KDIGO) guidelines [24]. These included any laboratory test ordered for albuminuria (albumin-to-creatinine ratio), mineral metabolism disorder (serum phosphate, 25-hydroxyvitamin D, and intact parathyroid hormone), any prescription orders for angiotensin-converting enzyme inhibitors or angiotensin receptor blockers (ACEi/ARB), and any active prescription for statins between 1/1/2014 and 12/31/2015. We also extracted the number of nephrology office visits and nephrology consultations between 1/1/2014 and 12/31/2015.

Statistical analyses

Chart review analyses

We began by comparing characteristics of the 50 patients randomly selected for chart review against the overall cohort. We initially evaluated differences in age, race, ethnicity, eGFRcreat, and CKD on the problem list. We then determined the prevalence CKD on the problem list in the overall sample and the 50 random charts. Next--among the 50 randomly selected patients’ charts--we compared characteristics of patients with CKD confirmed by clinician chart review vs. unconfirmed. We tested differences in patient characteristics by eGFRcreat category using one-way analysis-of-variance (ANOVA) for continuous variables and Chi-Square or Fisher exact tests for categorical variables. If there was any question about the normality of a continuous variable’s distribution, we used a nonparametric equivalent test to confirm the parametric results. We then compared prevalence of the comorbidities (cerebrovascular disease, congestive heart failure, coronary artery disease, diabetes mellitus, and hypertension) by ICD9 codes vs. clinician chart review. We also calculated the sensitivity, specificity, negative and positive predictive values for these five comorbid conditions—again comparing the administrative codes against the clinician chart review. Because this cohort was comprised entirely of patients with eGFRcreat defined CKD, we were only able to capture the prevalence for CKD and CKD awareness.

Guideline-concordant CKD care analyses

In the second part of our analyses, we used data from the guideline-concordant care sub-cohort described above. We first compared clinical and demographic characteristics by eGFRcreat level, and we estimated the proportion of patients who had CKD on the problem list overall and by eGFRcreat category. We then tested for associations of problem list CKD documentation with each process of care measure individually, using logistic regression to model the odds of receipt of guideline-concordant care. We initially adjusted for age, sex, race, and ethnicity in model one. In model two, we additionally adjusted for degree of CKD by adding the patient’s second ‘qualifying’ eGFRcreat (15.0–59.999 ml/min per 1.73m2), and for comorbidities by adding indicator variables for the presence or absence of each problem list comorbidity diagnosis (diabetes mellitus, cerebrovascular disease, congestive heart failure, hypertension, coronary artery disease). We also tested whether the association of problem list CKD documentation and any orders for albuminuria testing, ACEi/ARB prescription, or statin prescription differed between patients seen by a nephrologist vs. those followed only in primary care. Specifically, we repeated models one and two stratified by whether the patient had received a nephrology consultation, and tested for an interaction. Statistical significance of our findings was evaluated using an alpha level of P < 0.05. All analyses were conducted using R (version 3.4.3).

Results

Overall CKD cohort and chart review

For the overall CKD cohort, we identified 2214 UCSF primary care patients who met eGFRcreat criteria for CKD and were eligible for the chart review (Fig. 1). Comparison of characteristics for those randomly sampled for the chart review (n = 50) to the overall CKD cohort demonstrated no significant difference in demographic or clinical characteristics (Table 1). Overall, the mean age was 68 (IQR 61–75), mean eGFRcreat was 52 (IQR 44–57). Most (81%) had hypertension, and a large proportion (41%) had diabetes. A total of 943 (44%) had CKD listed on their problem list.

Table 1 Characteristics of primary care patients with CKD based eGFR†
Full size table

Among the 50 patients randomly sampled for chart review, 34 (68%) had nephrologist confirmed CKD. Those identified as having CKD by historical eGFRcreat but not confirmed by chart review were more likely to be female, White and have higher eGFRcreat levels compared to those with CKD confirmed by the nephrologists’ chart review. Those for whom CKD was not confirmed were less likely to have diagnoses of hypertension or diabetes. All 16 individuals with CKD not confirmed had eGFRcreat greater than 45 ml/min/1.73m2 (Table 2). We also found that only 1 of the 16 individuals with CKD not confirmed had CKD listed on the problem list, while 68% of those with nephrologist confirmed CKD had CKD listed on their problem list.

Table 2 Characteristics of 50 patients categorized by clinician chart review confirmation†
Full size table

By examining notes, the nephrologists determined that 23 (68%) of the 34 patients with confirmed CKD also had primary care providers who were aware of the patient’s CKD. Among the 23 patients for whom the nephrologists believed the PCP was aware of the patient’s CKD, 21 (91%) had CKD documented on their problem list.

When we evaluated phenotyping of comorbid condition, we found good agreement between ICD-9 diagnoses and chart review for most comorbid conditions, except congestive heart failure, which was only in moderate agreement (Table 3). The sensitivity was lowest for congestive heart failure and coronary artery disease, and highest for cerebrovascular disease and diabetes mellitus. Specificity values were high, with the lowest values for congestive heart failure and diabetes mellitus.

Table 3 Agreement between administrative (ICD-9) diagnosis codes and manual chart reviewa
Full size table

Guideline-driven care CKD sub-cohort

Of the overall CKD cohort, we had complete data for 1825 persons, and these patients had an earliest second ‘qualifying’ eGFRcreat on or before 12/31/2015. After we excluded and additional 58 patients due to either end-stage renal disease (ESRD) or a kidney transplant on their problem list, a total of 1767 were included in the guideline-driven care CKD sub-cohort. The characteristics of these patients are presented in Additional file 2: Appendix Table S2, stratified by eGFRcreat category. Among 1767 patients, a total of 822 (47%) had a CKD diagnosis on their problem list. The prevalence of CKD on the problem list increased with increasing severity of CKD (Additional file 2: Appendix Table S2). Specifically, among individuals with eGFRcreat < 30 ml/min/1.73m2, 79% had CKD listed, compared with 68% with eGFRcreat 30–45 ml/min/1.73m2 and 37% of patients with an eGFRcreat 45–60 ml/min/1.73m2 (p < 0001).

In the logistic regression models adjusted for age, sex, race, and ethnicity, we found that patients with CKD documented in their problem list had more than two-fold higher odds of having a test order for albuminuria, phosphorus and vitamin D, and nearly seven-fold higher odds for having a test order for parathyroid hormone compared to patients with no documentation (Table 4). Patients with CKD documented in their problem list had nearly two-fold higher odds of having a prescription for an ACEi/ARB or for a statin compared to patients with no documentation. After adjustment for comorbidities and CKD severity, these associations were somewhat attenuated, but remained statistically significant, except for ACEi/ARB prescriptions. The associations of CKD documentation in the problem list with any order for albuminuria, any prescription for an ACEi/ARB, and any prescription for a statin were not materially different after stratification by the presence or absence of a nephrology consultation (p > 0.05). See Additional file 3: Appendix Table S3 and Additional file 4: Appendix Table S4.

Table 4 Associations between listing of CKD on problem list with guideline-driven processes of carea
Full size table

Discussion

In this methodologic study, we show findings with important implications for design and implementation of future pragmatic studies that leverage the EHR to deploy interventions to improve management of individuals with CKD in primary care. First, we found that using a CKD definition based on historical eGFRcreat values only may be too sensitive, since we found that CKD status was confirmed by nephrologist chart review in only 68% of cases. Female gender, White race and higher eGFRcreat were more common among persons with CKD that was not ultimately confirmed by chart review. We also show that CKD on the problem list is a relatively good surrogate for PCP awareness, as it has high concordance with both CKD status and PCP awareness as ascertained by nephrologist chart review. In the primary care practice included in this study, providers have only moderate rates of CKD awareness (44%), defined as CKD present on the problem list. However, awareness increased significantly with severity of CKD. Importantly, CKD awareness was significantly associated with higher odds of guideline-concordant CKD testing, and the use of ACEi/ARB and statins.

Eight prior studies have reported misclassified CKD diagnoses from administrative data [18,19,20, 23]. However, five of these studies were conducted in an inpatient hospital setting or used patients who were hospitalized [18]. Only three of these studies sampled from outpatient primary care settings and included expert clinician chart review as a gold standard [19, 20]. We identified only one study that focused on patients who were receiving active and ongoing outpatient primary care [23]. Narrowing the focus to these patients is critical because these are the individuals who are most likely to be included in intervention studies. Recently, the eMERGE consortium published an algorithm that maximizes sensitivity and specificity of CKD associated with hypertension and diabetes, compared with chart review [20]. Appropriate implementation of the algorithm requires data handling procedures that natural language processing of text, which are unlikely to be feasible for most primary care practices [20]. Our study adds value to the literature as it shows that CKD ascertained only by two historical eGFRcreat values < 60 ml/min/1.73m2 at least 90 days apart may be too sensitive when the goal is to deploy interventions for those individuals with CKD at highest risk for complications who are most likely to benefit from interventions. Our finding that almost a third of patients identified as having CKD by eGFRcreat did not have CKD confirmed by expert physicians has important implications because it is likely that future pragmatic trials will need to include steps to re-test and further risk stratify patients (e.g., with albuminuria and cystatin C testing) to confirm CKD before deploying interventions.

In our study, combining problem list and encounter diagnoses to identify the comorbidities most likely to be relevant in studies of CKD patients correlated well with expert physician chart review. One important exception was presence of heart failure, a finding similar to prior studies showing that presence of heart failure correlates relatively poorly with administrative codes alone [27]. This suggests that, if researchers aim to include or exclude individuals with certain comorbidities from intervention studies, administrative codes and problem list are suitable for most of the conditions we tested, but not for heart failure where additional variables or chart review will be required.

We also found that CKD listed on the problem list may be a good surrogate for PCP awareness of this diagnosis. Low PCP awareness of CKD has been cited as one of the major barriers to improve kidney care in the U.S. [28]. Yet strategies to ascertain the degree of awareness are limited. We showed that, in this setting, CKD documented on the problem list had high concordance with CKD awareness when ascertained by expert chart review. Importantly, we also confirmed findings from prior studies that CKD awareness by PCP remains limited [22, 23, 28]. While only 44% of patients had CKD listed, which is higher than some previously reported estimates, CKD awareness did significantly increase with severity of CKD [19, 22, 23]. This suggests that CKD documentation may be a useful additional variable to increase the specificity of EHR data identification of CKD status. It is possible that the relatively lower prevalence of CKD listing among those patients with higher eGFRcreat represents PCP discomfort with labeling persons with a disease when the eGFRcreat is close to guideline definition cut-points.

Finally, we found that CKD awareness, defined as CKD on the problem list, was significantly associated with higher rates of testing for albuminuria and CKD complications. It was also associated with higher prevalence of ACEi/ARB and statin prescription. These findings highlight improving CKD awareness as a “low hanging fruit” actionable gap in CKD care. Thus, pragmatic trials that include interventions to improve CKD awareness have the potential to improve some important processes of care.

Strengths and limitations

We were able replicate an EHR CKD registry using previously described methods [19, 22]. A strength of our study was the selection of patients recently seen in primary care and with a recent “qualifying” eGFRcreat measurement. Our study also has several limitations. The observational study design and cross-sectional statistical methods makes it impossible to attribute causality. We reviewed a relatively small number of charts; however, we were about to demonstrate that our chart-review sample was representative of the larger cohort. We also cannot definitively conclude that the findings presented represent a primary care provider’s awareness of a patient’s CKD status. These findings are specific to EHR systems built using a problem list linked to each patient, and might not be generalizable to EHR systems without this architecture. However, it’s we expect discrepancies would still exist between an eGFR-defined CKD and the corresponding diagnosis codes. The lack of a problem-list-documented CKD diagnosis does not necessarily mean a provider was unaware of their patient’s CKD. We did not measure any physician behaviors or professional characteristics (e.g., workflow practices, time spent in clinical practice, medical practice team organization, etc.) that might influence EHR problem list use and guideline concordant processes of care. We considered including lab-defined CKD using a dipstick proteinuria, but decided against it because we felt this would create a bias by indication because the majority of the patients receiving proteinuria testing are diabetic. However, we did find good evidence that listing CKD on the problem list was associated with guideline-concordant care.

Conclusions

Performing secondary analyses of EHR data to explore associations between patient characteristics, clinical measurements, delivery of care, and CKD may be useful and appropriate for hypothesis generation or risk prediction. Given the higher likelihood of CKD problem listing with lower eGFRcreat, it may be particularly useful in studies of patients with more advanced disease. However, if the nature of the investigation is to identify patients with CKD who are most likely to benefit from an intervention, researchers should expect that CKD will need to be confirmed.