Background

In the United States (US), progressive chronic kidney disease (CKD) and particularly end-stage renal disease (ESRD) disproportionately affect traditionally underserved groups including racial-ethnic minorities and persons of low socioeconomic means [1–6]. Despite the disproportionate burden of ESRD observed among racial-ethnic minority and low income groups, effective interventions to slow CKD progression and reduce mortality appear to be underutilized in these populations [7, 8].

A central barrier to applying proven therapies in traditionally underserved settings lies in the absence of mechanisms to efficiently monitor and optimize care provided to the nation’s poor and underinsured [9, 10]. Recently, one large safety-net health system has leveraged prediction analytics and data from the electronic health record (EHR) to accurately identify inpatients with specific conditions who are at high risk for subsequent re-hospitalization [11]. Most patients with moderate or severe CKD suffer from multiple chronic conditions and increasingly receive their care from Chronic Disease Management teams [12, 13]. These clinic-based teams typically target patients with specific conditions (e.g., diabetes mellitus, hypertension, chronic viral diseases, congestive heart failure, severe CKD, etc.) and seek to optimize important risk factors for progressive disease, disability and mortality [13, 14]. However, even within these clinic- or disease-based practices most individuals with CKD will not progress to ESRD. Whether data from the EHR can be “meaningfully” used to identify persons at high risk for progressive CKD within this practice-based construct is unclear.

To address this issue, we examined the performance of EHR-based risk predictive models to discriminate among persons with CKD who would and would not progress to ESRD for time frames up to 7 years. We hypothesized that the discriminatory ability (and usefulness) of these models to accurately predict ESRD would vary substantially within clinical subgroups due primarily to differences in patient composition and in the distribution of ESRD risk.

Methods

Design, setting, and participants

We conducted a retrospective cohort study of 28,779 persons aged 18 years or older with non-dialysis requiring CKD stages 3–5 who received healthcare in the San Francisco Health Network (SFHN) or Harborview Medical Center (HMC) from January 1, 1996 to December 31, 2009. Eligible subjects included prevalent patients of SFHN or HMC with at least 2 or more ambulatory serum creatinine measurements separated by at least 3 months. We defined CKD stage 3–5 based on the presence of 2 or more serum creatinine measurements yielding an estimated glomerular filtration rate (eGFR) <60 mL/min/1.73 m2 as calculated by the re-expressed Modification of Diet in Renal Disease (MDRD) study equation based on calibrated serum creatinine, age, race and sex that were separated by at least 3 months [15, 16]. To reduce the potential for exposure misclassification (i.e., acute kidney injury), we excluded inpatient serum creatinine measurements from consideration. We elected to use the MDRD eGFR (rather than more recently proposed equations) to estimate kidney function because the MDRD eGFR was reported by each health system’s clinical laboratory for most of the study period.

The SFHN (formerly Community Health Network) is the integrated healthcare delivery system of the Department of Public Health of the City and County of San Francisco [7]. During the study period, approximately half of San Francisco’s 130,000 uninsured residents and one-quarter of its Medicaid population made at least one visit annually to the SFHN. Harborview Medical Center in Seattle, Washington, is the Pacific Northwest’s largest provider of care to medically underserved populations, providing over 20 % of all uncompensated care in the state of Washington [17]. Both health systems offer an array of healthcare services including primary care, specialty care and acute care through acute care hospitals (San Francisco General Hospital and Harborview Medical Center) with on-site primary and specialty care clinics, as well as through community-based primary care clinics. Specialty care clinics include diabetes, HIV, hypertension, and nephrology clinics. Services are rendered irrespective of a patient’s ability to pay and further include a wide range of interpreter services to facilitate care for each system’s diverse patient population. Specific details of the health systems have been previously described [18].

Outcome measures and data sources

The primary outcome measure was progression to ESRD, defined as having a first service date for maintenance dialysis or kidney transplantation. We ascertained ESRD by performing linkage with the United States Renal Data System (USRDS) files based on patient name, date of birth and Social Security number [1]. To ascertain death, we performed identifier matching with the Social Security Administration Master Death files using the same patient identifiers described above [19]. We assessed ESRD and death through September 29, 2011, the last date that data were available for both outcomes at the time of identifier linkage. We defined survival time as time from the first qualifying serum creatinine date until ESRD, death or the end of follow-up through September 29, 2011, whichever occurred first.

Independent variables

We extracted sociodemographic and clinical variables that we hypothesized might predict progression of CKD to ESRD based on prior studies. These covariates were defined within the 2-year period closest to the index qualifying serum creatinine measurement (used to calculate initial qualifying eGFR). Individual-level sociodemographic covariates included patient age, sex, race-ethnicity (non-Hispanic white, non-Hispanic black, Hispanic, Asian/Pacific Islander, or other), and health insurance coverage (uninsured, Medicaid, Medicare, or commercial insurance) at the time of initial qualifying eGFR. We ascertained comorbid conditions based on established algorithms using discharge diagnostic codes, ambulatory diagnostic codes and procedural codes for diabetes mellitus, hypertension, cardiovascular disease (defined as coronary artery, cerebrovascular, or peripheral vascular disease), chronic viral disease (hepatitis B virus [HBV], hepatitis C virus [HCV], or HIV/AIDS), and drug or alcohol abuse [20–23]. Laboratory covariates included eGFR, hemoglobin, serum albumin, calcium, cholesterol, and phosphorus concentrations, and the presence and severity of proteinuria according to dipstick urinalysis.

Statistical approach

We summarized the characteristics of the cohort using means (standard deviations) and proportions. We calculated unadjusted incidence rates of ESRD for the full cohort, and for clinical subgroups defined by diabetes mellitus, hypertension, chronic viral diseases (HBV, HCV and/or HIV) and severe CKD (<30 mL/min/1.73 m2). We focused on these four subgroups because they represent common conditions frequently targeted by our Chronic Disease Management programs. To approximate our disease-based clinical practice, we did not require the subgroups to be mutually exclusive. Based on prior studies, we tested three proportional hazards regression models to predict progression to ESRD in each subgroup [24–26]. Model 1 included eGFR, dipstick proteinuria, and the “residual” associations with age, sex, and race-ethnicity. To this model, we added health insurance coverage, comorbidities (diabetes mellitus, CVD, hypertension, substance abuse and chronic viral disease) (model 2) and additional laboratory variables (serum albumin, calcium, cholesterol and hemoglobin) (model 3) [18].

To evaluate the predictive capacity of the models we used 2-fold cross validation, training and validating our models on separate subsets of the data [27]. We divided each cohort into training and validation sets (2/3 and 1/3 of each cohort, respectively). To reduce potential bias caused by excluding patients with missing data, we performed multiple imputation by chained equations with 10 imputations in the training and validation sets separately using the R package ‘mice’ based on observed variables related to the missingness (i.e., missing at random) [28, 29]. We fitted each model to the 10 training sets, and estimated the hazard ratios and the 95 % confidence intervals taking into account the variability associated with the multiple imputation. The baseline hazard function and the estimated coefficients for each model fit to the training set were fixed and applied to the validation set in order to obtain the probability of ESRD-free survival beyond year 1, 3, 5 and 7 for each subject in the validation set. We applied this procedure to each of the 10 imputed training-validation dataset pairs. We then applied the following discrimination and calibration criteria to assess the predictive performance of our model on each subgroup: (1) receiver operating characteristics (ROC) curve and the area under the ROC curve (AUC) [30]; (2) prediction error (PE) [31]; (3) proportion of cases followed (PCF(q)), and (4) proportion of the population needed to be followed (PNF(p)) [32]. PCF and PNF are pragmatic measures of risk concentration that are directly relevant to public health decision-making [32]. The PCF(q) represents the estimated proportion of cases (or events) that would be captured if we followed proportion q of the population at highest risk. PNF(p) represents the estimated proportion of the population at highest risk that would need to be followed in order to capture proportion p of the events. Larger values of PCF and smaller values of PNF indicate better predictive performance [32].

Results

Patient characteristics

Consistent with US safety-net healthcare settings, the CKD cohort was young (mean age 60 years) and had a high proportion (59 %) of individuals from racial-ethnic minority groups (Table 1). Among clinical subgroups, the chronic viral disease subgroup was the youngest and had the highest prevalence of substance abuse and mental health conditions as compared with the other three subgroups (Table 1). In contrast, the hypertension subgroup was the oldest and had the highest prevalence of cardiovascular disease. Men comprised a majority of patients with chronic viral disease and severe CKD (eGFR <30 ml/min/1.73 m2), and women comprised over half of patients in the hypertension and diabetes mellitus subgroups. Biochemically, the severe CKD subgroup had higher (mean serum) concentrations of phosphorus, and lower concentrations of albumin, calcium and hemoglobin as compared with the other subgroups (Table 1).

Table 1 Baseline characteristics of 28,779 patients with moderate or severe chronic kidney disease (stage 3–5) from the healthcare safety net
Full size table

ESRD risk distribution

Figure 1a and b show the estimated risk distribution for ESRD in the hypertension and severe CKD subgroups for time frames up to 7 years using the base model (age, race, sex, eGFR and dipstick proteinuria). In the Figures, 80 and 90 % of the ESRD progressors at highest predicted risk are shown to the right of the vertical solid grey line (q80) and vertical dashed line (q90), respectively. In the hypertension subgroup, the base model discriminated progressors (blue line) from non-progressors (red line) well as reflected by little overlap in their respective risk distribution for q80 and q90. In contrast, substantial overlap in the distribution of ESRD among progressors and non-progressors was observed using the same model in the severe CKD subgroup (Fig. 1b). Similar to hypertension, largely favorable separation in risk distribution was observed in the chronic viral disease and diabetes mellitus subgroups (Additional file 1: Figures S1 and S2).

Fig. 1
figure 1

a The distributions of predicted risk of ESRD among persons with hypertension. The distributions of predicted risk of ESRD among subjects who did not develop ESRD (non-progressors) in a given time frame is shown by the blue line and subjects who progressed to ESRD (progressors) in that time frame is represented by the red line. We considered four time frames - 1 year, and 3, 5 and 7 years. 80 % of the ESRD progressors are to the right of the vertical solid grey line (q80), and 90 % of them are to the right of the vertical dashed grey line (q90). The risk predictions are based on application of a proportional hazards regression model incorporating age, race, sex, eGFR and dipstick proteinuria to the validation dataset. b The distributions of predicted risk of ESRD among persons with severe CKD (eGFR < 30 ml/min/1.73 m2). The distributions of predicted risk of ESRD among subjects who did not develop ESRD (non-progressors) in a given time frame is shown by the blue line and subjects who progressed to ESRD (progressors) in that time frame is represented by the red line. We considered four time frames - 1 year, and 3, 5 and 7 years. 80 % of the ESRD progressors are to the right of the vertical solid grey line (q80), and 90 % of them are to the right of the vertical dashed grey line (q90). The risk predictions are based on application of a proportional hazards regression model incorporating age, race, sex, eGFR and dipstick proteinuria to the validation dataset

Full size image

ESRD model performance

Overall, 1730 persons initiated ESRD treatment and 7628 died during median follow-up of 6.6 years (Table 2). Persons with severe CKD experienced the highest and persons with hypertension the lowest incidence rates of ESRD among all subgroups (Table 2). Nearly 1 in 3 patients in the severe CKD subgroup progressed to ESRD as compared with approximately 1 in 14 in the hypertension and 1 in 9 in the diabetes mellitus subgroups.

Table 2 Incidence rates of end-stage renal disease and death by comorbid subgroup among 28,779 patients with moderate or severe chronic kidney disease from the healthcare safety net
Full size table

In the full cohort, models that included five common variables (age, race, sex, eGFR, dipstick proteinuria) from the EHR performed well in discriminating progressors and non-progressors, whereby surveillance of individuals in the highest quintile of ESRD risk yielded 97, 91, 86 and 81 % of cases at years 1, 3, 5 and 7, respectively as estimated by PCF(0.2) (Table 3). A predictive model which included these five variables performed similarly well in the hypertension and modestly lower in the chronic viral disease and diabetes mellitus subgroups (Fig. 2). Model performance was substantially lower in the subgroup with severe CKD. Using the base model (age, race, sex, eGFR and dipstick proteinuria), surveillance of individuals in the highest quintile of ESRD risk yielded an estimated 71 % of ESRD events occurring within 1 year in persons with severe CKD. The corresponding PCF(0.2) values declined to 57, 48 and 42 % (ESRD events captured) for the 3-, 5-, and 7-year time frames, respectively (Table 3). Notably, discrimination based on traditional criteria was generally favorable in all subgroups including in severe CKD (AUC, 0.80–0.87) using the same model (note: models with AUC values <0.70, with 0.50 being random and 1.00 being perfect, are considered to have only moderate ability to discriminate risk for an individual patient) [30]. Similar performance patterns and covariate estimates were observed for more complex models 2 and 3 (Additional file 1: Tables S1-S3).

Table 3 Comparative performance of the base predictive model* for end-stage renal disease by comorbid subgroup among 28,779 patients with moderate or severe chronic kidney disease from the healthcare safety net
Full size table
Fig. 2
figure 2

Performance of the base model* for predicting progression of CKD to ESRD in different clinical conditions. The estimated proportion of ESRD events captured (PCF) among a given proportion of subjects at highest estimated risk of ESRD (PNF) for a model* incorporating age, race, sex, eGFR and dipstick proteinuria at 1, 3, 5 and 7-year time frames among persons with hypertension, chronic viral disease, diabetes mellitus and severe CKD

Full size image

Discussion

Surveillance of “real-world” care delivery to vulnerable groups is challenging because kidney disease metrics are not routinely measured or reported by federally qualified health centers nor are they part of reporting requirements for health plans [33, 34]. Thus, underserved or vulnerable patients with non-dialysis dependent CKD remain invisible to much of the healthcare system unless and until they reach ESRD (at which time most become eligible for Medicare). In this diverse cohort of persons with CKD, we observed that risk predictive models using common data from the EHR can accurately discriminated between most persons who did and did not progress to ESRD for time frames up to 7 years. However, the performance of these risk models varied when applied to specific conditions frequently targeted for Chronic Disease Management. Model performance was highest in the hypertension subgroup, intermediate in chronic viral disease and diabetes mellitus subgroups, and lowest in the subgroup with severe CKD (eGFR < 30 ml/min/1.73 m2). Our study findings may help health organizations and their clinical practices to optimize care assessment by estimating the scope and potential needs of patients with CKD who are at highest risk for disease progression, disability and ESRD-related costs.

Health researchers frequently assess risk at the individual level using epidemiological studies or by examining patient-level interventions in randomized clinical trials [24–26]. Tangri et al. evaluated the performance of several risk predictive models based on data extracted from the EHR of 2 CKD cohorts in Canada using traditional discriminatory criteria (C-statistic/AUC and integrated discrimination improvement). They observed that most ESRD risk predictive models performed well in patients with moderate or severe CKD who were referred for nephrology evaluation [24]. Complementary studies in large study cohorts have yielded additional risk predictive models for ESRD which are intended for use by clinicians to estimate individual patient- rather than population-level risk [25, 26]. Using criteria (proportion of cases followed and proportion needed to follow) designed to inform population-level disease assessment, we recently observed that a model incorporating age, race, sex, eGFR and dipstick proteinuria adequately predicted progression to ESRD among vulnerable persons with moderate or severe CKD who were identified through systematic review of the EHR [18]. Our current study findings extend and leverage our prior work, by placing the proportion of cases followed (PCF) and proportion of the population needed to be followed (PNF) more squarely in the context of how clinical care is actually delivered for this patient population. Collectively, our observations suggest that readily available data from the EHR might be efficiently used in earlier stages of CKD to further inform care assessment and planning for organizations or practices based on clinical conditions commonly targeted by disease management programs. For example, our methods could be applied to hypertension clinic to estimate the potential feasibility or effectiveness of an intervention or program targeting of patients in the highest decile or quintile of ESRD risk (e.g., for additional interventions or pragmatic studies, etc.). The resources needed to follow a targeted group of high-risk patients from such disease-based program could be markedly lower than required for risk-stratifying an entire health system. Because CKD represents a heterogeneous array of underlying disease states, this disease- or risk factor-based approach could theoretically leverage existing programmatic resources and infrastructure as an alternative to lumping all high-risk CKD patients into a single category.

In terms of discriminatory assessment, our study illustrates how the PCF and PNF can be more informative than traditional discriminatory criteria such as AUC by providing estimates of risk concentration (for the event of interest) in the study population. If risk is concentrated, such as in the hypertension subgroup, values for traditional discriminatory criteria will typically be large, but the converse is not necessarily true [32]. In other words, as observed in the severe CKD subgroup, values for traditional discriminatory criteria may be favorable even when risk is not concentrated [32, 35]. This observation occurs because the AUC evaluates the model’s ability to discriminate between progressors and non-progressors for all risk thresholds, even those that are not as relevant in clinical practice, such as risks close to zero. In contrast, when using PCF and PNF, we can set the risk threshold at a clinically meaningful level. Thus, desirable values of PCF and PNF are achieved only if risk in the target population is concentrated among a relatively small proportion of individuals (at highest risk). The suboptimal performance of our risk models based on PCF/PNF values among persons with severe CKD likely reflected the broader distribution of ESRD risk in this subgroup. Notably, values for AUC were universally favorable in all subgroups, and hence less informative from the perspective of clinical practice. Reduced variance of influential predictors such as eGFR and dipstick proteinuria among patients with severe CKD (relative to patients from the other subgroups) likely further reduced the predictive capacity of our risk models in this subgroup. While the urinalysis dipstick remains an excellent population-level screening tool for proteinuria, its limited utility in predicting progression to ESRD among patients in later stages of CKD has been previously documented [36]. Thus, models which incorporate additional predictors such urinary albumin-to-creatinine ratio (which was measured only in a small fraction of patients in our cohort) and annual decline in eGFR would likely enhance predictive performance in the setting of severe CKD. In addition, patients with severe CKD are also at markedly higher risk of premature death than those with higher levels of kidney function [20]. The elevated risk of death in this severe CKD subgroup poses additional challenges for predicting ESRD since many (if not all) of the covariates examined are also significantly associated with death.

Historically, low individual-level provider and patient awareness of CKD have reinforced the need to optimize multi-level strategies (at the community, organization, practice, and patient levels) to help assess and manage CKD [18, 37–41]. Our study findings demonstrate the potential usefulness of clinical data from the EHR to provide reliable information for CKD care assessment at the level of the organization and at the level of a disease- or clinic-based practice which might readily generalize to other chronic diseases. Accordingly, clinical practices might leverage the EHR, for example, to identify, triage, and monitor the care of patients at highest risk for progressing to ESRD. As evidenced by the high prevalence of psychiatric conditions, drug and alcohol use in our study cohort, such a multi-level health approach would likely require the consideration of a broader array of health determinants than in conventional health settings. Intervention approaches might thus combine EHR-based risk surveillance with facilitators of care engagement such as assistance with transportation, housing, or health insurance applications, drug or alcohol cessation programs, and mental health co-management. When necessary, our methods might also be applied to refer and track patients at highest risk for imminent (<1 year) ESRD to ensure timely placement of dialysis access, transplant referral, and dialysis education. Recent advancements in software now enable predictive analytics to interdigitate with the EHR, further highlighting the potential “real world” and “real time” application of EHR-derived predictive models [11].

Our study is strengthened by the inclusion of adults with moderate or severe CKD from two large safety-net health systems—populations which traditionally bear a disproportionate burden of ESRD and which may benefit from enhanced CKD surveillance. Our study also includes several limitations. First, while we were able to provide detailed demographic and clinical data, and link our cohort to national registries (to obtain complete or nearly complete capture of treated ESRD and vital status), our study was subject to potential bias from under-ascertainment of comorbidities based on diagnostic and procedural codes. Second, misclassification of CKD and its severity using the MDRD GFR estimating equation may also be operative since the MDRD study equation was derived in a population of mostly white and black patients with moderate-to-advanced CKD, very few of whom had diabetes mellitus. Third, while our study included patients from diverse social and demographic groups, this cohort may not be fully representative of patients who utilize the healthcare safety-net in other geographic locations. In addition, our observations require further validation using external data as our predictive models may perform differently in other populations. Lastly, rather than restricting our study to only patients with complete data (and potentially introducing bias from case deletion), we leveraged multiple imputation under the assumption that missing values carried no information about probabilities of missingness. However, our study results may be potentially biased in the unlikely event that this assumption of ‘missing at random’ (MAR) was violated [29].

Conclusions

In conclusion, common variables from the EHR can be leveraged to adequately discriminate among most patients with CKD who will and will not progress to ESRD in this safety-net healthcare setting. Recently developed discriminatory criteria may be applied to evaluate the ability of risk predictive models to discriminate between progressors and non-progressors only within a clinically relevant range, and thus, help to inform CKD surveillance at multiple levels in diverse clinical settings.

Abbreviations

AUC, area under the receiver operating characteristics curve; CKD, chronic kidney disease; ESRD, end-stage renal disease; EHR, electronic health record; eGFR, estimated glomerular filtration rate; HBV, hepatitis B virus; HCV, hepatitis C virus; HIV, human immunodeficiency virus; HMC, Harborview Medical Center; MAR, missing at random; MDRD, Modification of Diet in Renal Disease; PCF, proportion of cases followed; PNF, proportion of the population needed to be followed; ROC, receiver operating characteristics; SFHN, San Francisco Health Network; US, United States; USRDS, United States Renal Data System