What was known before

The Charlson Comorbidity Index (CCI) and End-Stage Renal Disease Comorbidity Index (ESRD-CI) are commonly used in studies of dialysis patients, but assessments of their validity are lacking.

What this adds

Both indices had a limited ability to discriminate mortality risk in this study emphasizing that they may not be the best method of risk adjustment in contemporary dialysis cohorts.

Introduction

In patients with end-stage renal disease (ESRD), the presence of comorbid conditions has been shown to have a negative impact on survival [1–3]. A commonly used approach for summating individual conditions into an overall “score” of comorbidity for risk stratification is calculation of the Charlson Comorbidity Index (CCI) [4]. The CCI was initially derived in a cohort of 559 patients, and tested in a second cohort of 680 patients followed for 10 years [4]. In the original study, 19 comorbid conditions were evaluated in a Cox proportional hazards model. Point scores were assigned to each comorbidities hazard ratio depending on the value. The sum of the points equalled a given individuals’ overall CCI score. A higher CCI score is associated with an increased mortality risk in ESRD patients [5–12], however the CCI does have limitations when applied to ESRD patients. The inclusion of renal disease as one of the component comorbidities is redundant and medical advances since the development of the CCI have changed the prognosis of some of the individual comorbid conditions within the index [13].

There have been a number of additional comorbidity indices that have been created for ESRD patients [1, 14–16], including the End Stage Renal Disease Comorbidity Index (ESRD-CI) which avoids some of the limitations associated with the original CCI [15]. Having been designed as an adaptation of the CCI, the ESRD-CI was developed in a cohort of 237 incident dialysis patients [15]. 15 of the 19 conditions in the CCI were evaluated in a multivariable Cox survival analysis. Similar to the CCI, point scores were assigned to each condition’s hazard ratio and summed for each individual in the derivation cohort. In the model derivation study the ESRD-CI had slightly better performance characteristics compared to the CCI in the tested population (c-statistic of 0.73 versus 0.72) [15].

While both the CCI and ESRD-CI are frequently used for risk adjustment in studies of dialysis patients [17–22], only a few studies have attempted to validate either index [15, 23, 24]. In addition, these validation studies have limitations including incomplete inclusion of all necessary comorbid conditions, and validation techniques that are not specific to time-to-event analyses [15, 23, 24]. Finally, validation in a more recent era-cohort (acknowledging that patient characteristics, disease prevalence and outcome after dialysis initiation may differ from those in older cohorts) has not been conducted in many studies. A lack of validity/limited prognostic ability of either index will emphasize the need to re-examine the usefulness of previously derived comorbidity indices in contemporary dialysis cohorts.

Therefore, the purpose of this study was to assess the validity of the CCI and ESRD-CI in a contemporary cohort of ESRD patients, and to determine if a high degree of comorbidity was independently associated with mortality. We hypothesized that both indices would have a reduced level of discrimination compared to the derivation studies, but that increased comorbidity burden would be associated with a higher risk of death for dialysis patients.

Methods

Population

We conducted a cohort study of incident, adult (≥18 years) chronic dialysis patients in a large tertiary care institute between 1 Jan 2006 and 01 Jul 2013. Follow-up for patients began at the initial start date for their dialysis.

Exposure definition

Comorbidity data was collected at the start of dialysis in all patients in a prospective manner, using documentation in patient charts (dating back to the first nephrology visit) and electronic records by the patients’ primary nephrologist. Comorbidities were subsequently verified in all patients by two nephrologists (K.T. and B.K.) and one nephrology trainee (T.A.). All 19 individual comorbid conditions in the CCI were collected at the time of dialysis initiation and scored according to the CCI derivation study. ESRD-CI scores were retrospectively calculated by re-scoring the comorbidities comprising the ESRD-CI based on the derivation paper. ESRD-CI scores were analyzed as ordinal variables and after categorization into six groups to be consistent with the derivation paper (using scores of 0/1, 2, 3, 4, 5 and ≥6) [15]. The CCI was analyzed both as an ordinal variable and in categories (2, 3, 4, 5 and ≥6). Since all patients in our dataset had ESRD, the lowest possible CCI score was 2.

In addition to the CCI and ESRD-CI scores, demographic data (age, race, gender), dialysis access (central venous catheter or arteriovenous fistula) type of dialysis modality (peritoneal or hemodialysis), cause of ESRD (diabetes, glomerulonephritis, hypertension, other) and baseline laboratory data (hemoglobin, phosphate, estimated glomerular filtration rate and albumin) were collected on all patients at the start of dialysis using a combination of electronic records and paper chart review.

Outcome

The primary outcome was all cause mortality after dialysis initiation. Administrative censoring was imposed on 01 Jan 2014. Patient survival was censored at the date of transplantation.

Statistical analysis

Descriptive statistics for baseline characteristics of the study cohort were reported as counts with proportions, mean with standard deviation and median with interquartile range for categorical, normally distributed continuous and non-normally distributed continuous variables, respectively.

External validation of the CCI and ESRD-CI followed the methods previously described by Royston et al. based on availability of data in the original derivation studies [25]:

  1. 1.

    Regression coefficients were estimated on the CCI and ESRD-CI (defined as the precise CCI and precise ESRD-CI).

  2. 2.

    Regression coefficients were also estimated on the categorical CCI (2, 3, 4, 5 and 6+) and ESRD-CI (0/1, 2, 3, 4, 5 and 6+) as a secondary analysis (defined as the categorical CCI and categorical ESRD-CI).

  3. 3.

    Discrimination (defined as the level of concordance between the risk predicted by a model and the rate of events experienced [25]) was assessed for the precise and categorical CCI as well as the precise and categorical ESRD-CI using Harrell’s c-index. Harrell’s c-index assesses the fraction of all possible pairings of patients in which the predictions and outcomes are concordant [26]. Scores range from 0.5 (no discrimination), to 1.0 (perfect discrimination). As a reference, the CHADS2 score for atrial fibrillation stroke risk has a reported c-index of 0.683 [27].

  4. 4.

    Kaplan-Meier Survival curves were plotted for each category of CCI/ESRD-CI and discrimination was also visually assessed according to the ordering and separation of the curves.

Cox proportional hazard models were used to calculated relative hazards and 95 % confidence intervals for each category of the CCI and ESRD-CI. Proportionality of hazards was assessed using Schoenfeld Residuals. Multivariable models included variables based on clinical judgment and those derived from the literature as being associated with mortality in studies of dialysis patients including age [28], gender [29], Caucasian versus non-Caucasian race [28, 30], dialysis modality [31], cause of ESRD, albumin [32], hemoglobin [33], phosphate [34, 35] and modification of diet in renal disease (MDRD) estimated glomerular filtration rate [36]. A two sided P value of <0.05 was the threshold for statistical significance. Approval to conduct this study was granted by our institutional research ethics board (Nova Scotia Health Authority, CDHA-RS/2014-288). All analyses were conducted using Stata version 12.0, College Station, TX, USA.

Results

Baseline characteristics

The cohort consisted of 771 ESRD patients from 01 Jan 2006 to 01 Jul 2013. Baseline characteristics of the cohort are noted in Table 1. The majority of patients were male (62 %) and Caucasian (91 %). Common comorbidities included diabetes (48 %), previous myocardial infarction (31 %), congestive heart failure (22 %) and peripheral vascular disease (20 %). The median CCI score was 4 (Q1-Q3: 3–6), and the median ESRD-CI Score was 2 (Q1-Q3: 0–4).

Table 1 Baseline characteristics of the cohort
Full size table

Validation

Cox regression on the precise CCI and ESRD-CI revealed coefficients of 0.55 (SE 0.08) and 0.52 (SE 0.07), respectively (Table 2). Cox regression on the categorical CCI and ESRD-CI revealed similar coefficients (0.56 and 0.52, respectively). The c-index was 0.61 (SE 0.02) for both the precise and categorical CCI, and 0.63 (SE 0.02) and 0.62 (SE 0.02) for the precise and categorical ESRD-CI (Table 2). Kaplan-Meier survival curves for each ESRD-CI score cut-off are displayed in Fig. 1 (Log-rank P < 0.001). There was separation of the curves for patients with a high versus low score (6+ versus 0/1 or 2). However, discordance was observed for patients with an intermediate score. A similar finding was noted for the CCI, however, there was slightly more separation between the curves in an incremental fashion based on score (Fig. 2).

Table 2 Cox regression coefficients and c-index for precise and categorical CCI/ESRD-CI
Full size table
Fig. 1
figure 1

Kaplan-Meier survival curves for time to death stratified by ESRD-CI Score groups

Full size image
Fig. 2
figure 2

Kaplan-Meier survival curves for time to death stratified by CCI Score groups

Full size image

Mortality

Over 1796.6 patient years at risk, there were 311 deaths. The distribution of deaths stratified by CCI and ESRD-CI scores are graphically displayed in Fig. 3. There was a rise in the number of deaths and fall in the number of patients that received a kidney transplant with each ESRD-CI score cut-off. In an unadjusted Cox survival analysis, relative to patients with an ESRD-CI score of ≤1, those with scores of ≥6 had a mortality HR of 2.64 (95 % CI 1.91 to 3.65, p < 0.001, Table 3). A similar mortality HR was observed for patients with CCI scores of ≥6 compared to 2 (HR 2.91, 95 % CI 2.04 to 4.15, p < 0.001). After multivariable adjustment, there was attenuation in the HR. Similar HR’s were noted for patients with scores of 4–6 for the ESRD-CI. For the CCI, there was separation in the HR’s for scores of 4–6 however, confidence intervals overlapped (Table 3).

Fig. 3
figure 3

Distribution of outcomes stratified by Comorbidity Index Score groups

Full size image
Table 3 Multivariable Cox survival analysis for the ESRD-CI and CCI
Full size table

Discussion

In this cohort study we evaluated the ability of the CCI and ESRD-CI to predict mortality in a population of incident dialysis patients. The association between comorbidity and mortality in our dataset was not as strong as in the derivation cohorts. In addition although higher comorbidity burden using the CCI and ESRD-CI was associated with mortality after multivariable adjustment, there was not a large separation in mortality risk when evaluating incremental changes in comorbidity scores. We can speculate that the limited utility of the CCI and ESRD-CI is due to several potential underlying reasons; limited generalizability of both indices, variability in comorbidity classification and comorbidity prevalence within either cohort, limitations with respect to derivation (including statistical over-fitting) and limited utility of comorbidity indices in general.

Limited generalizability may explain the observation of only partial validity. The CCI and ESRD-CI used a relatively small population from a single geographical area and validation in this study occurred in a separate single geographical area. Differences in the determinants of health in two communities may contribute to the different outcomes of two dialysis patients with similar comorbidities. Social determinants of health, in particular, have been shown to impact mortality rates in patients with ESRD [37–39]. More recently, frailty has been shown to be an important prognostic factor for incident dialysis patients [40]. Therefore, prediction models that incorporate clinical, demographic and social factors as well as assessments of frailty may be more applicable to ESRD cohorts [41]. An attempt at further validation of this index in a national or international sample of patients from a number of different centers in different geographic regions might help to clarify its generalizability and provide a clearer picture of its clinical utility.

There were notable characteristic differences comparing patients in our dataset and the dataset of the original studies, which may explain the reduced discrimination. Our cohort was in a more recent era (2006–2013), and the relative impact of some comorbid conditions may have changed [42–45]. The mortality risk in our cohort was higher than either derivation cohort [15] however; certain comorbidities such as chronic lung disease (16.7 % vs. 27.4 %) and neoplasia (6.5 % vs. 12.2 %) were less common in our cohort [15]. Furthermore, there was a reduced prevalence of overall comorbidity compared to the derivation cohorts. This reduced burden of disease might have affected the predictive value of the individual comorbidities comprising both indices. Alternatively, under-reporting of individual health conditions in each cohort may have explained the observed differences. For example, if comorbidity in our validation cohort was under-reported, despite a “falsely low” burden of comorbidity patients would have continued to have a high mortality rate. This in turn would have reduced the discrimination in our dataset.

The derivation of the CCI or ESRD-CI may have also impacted its validation. The linear predictor from a Cox model is ideal for developing a prognostic index. The linear predictor is described as the weighted sum of the variables in the model, where the regression coefficients are the weights [25]. In the development of the CCI, (which was replicated in the derivation of the ESRD-CI to maintain consistency) scoring weights were assigned to each HR derived from the Cox model [4, 15]. This may lead to over-weighting of conditions with high hazard ratios but limited precision and marked variability around the estimate.

While there are limitations with the comorbidity indices, it is important to acknowledge that both indices did have some validity. It is not unexpected that a higher level of comorbidity would be associated with a higher risk of death among dialysis patients, however most evaluations of comorbidity scores look at short term mortality or use validation techniques that do not incorporate survival time [15, 19, 23, 24]. Discrimination was lower in our validation dataset, however, the relative hazard for death was proportional across the index and the mortality association using either index persisted despite a relatively long duration of follow-up. Furthermore, it is not uncommon for validation studies to identify some reduction in predictive value [46] In addition there are other features of these indices that make them valuable. Both are intuitive; comorbid conditions that would be expected to confer a higher hazard for mortality are weighted more heavily. The major exception to this would be HIV (a component of the original CCI) a condition with a lower contemporary mortality rate [42]. Another advantage is that both indices draw on clinical data that is often present in patient charts, making them practical tools that do not necessarily require other diagnostic testing or laboratory investigation. Finally, the association with mortality persisted in our study even after multivariable adjustment for known predictors of mortality among dialysis patients.

It is important to note that there are limitations of comorbidity indices in general that warrant consideration. Most indices do not fully take into account the stage of progression, severity or proximity of the comorbid condition in relation to dialysis initiation [1, 14–16]. Furthermore, accumulation of comorbidity that often accompanies the early period after dialysis initiation [47] is not typically included in comorbidity indices. Global scoring systems that incorporate all prior comorbidities are easier to calculate and extract from patient records and facilitate ease of clinical application at point-of-care. More novel indices such as the Adult Comorbidity Evaluation-27, which takes into account the proximity and severity of the comorbidity, have been shown to out perform the CCI at predicting mortality [48, 49] and may be better suited for dialysis cohorts. Additionally, simple prognostic models that include the “surprise question” have also been shown to accurately predict survival for patients receiving hemodialysis [50].

This study utilized a substantial Canadian cohort and a long follow up time while leveraging electronic medical records to ensure a high quality and robust dataset. Additionally validation of the index was performed utilizing stringent methodology [25]. Examining the adjusted association between the CCI and ESRD-CI and mortality was enhanced by the relatively large number of outcomes.

There are, however, limitations to this study. The classification of comorbidities for each patient was determined by clinical impression (based on documentation in paper chart and electronic records). This introduces the possibility for misclassification bias. The derivation cohort(s) would have also been affected by misclassification bias, potentially compounding the observed differences in disease prevalence and prognostic utility. Our ability to completely validate either index was limited by the level of information provided in the original studies. In particular, an assessment of calibration (the ability of the index to assign the correct event probability at any relevant follow-up time and every level of predicted risk [25]) could not be performed.

Conclusion

The CCI and ESRD-CI had a limited ability to discriminate risk of death for incident dialysis patients in a contemporary Canadian cohort. Although a higher comorbidity burden was associated with mortality, incremental increases in index scores did not considerably change the risk of death.