Background

Risk adjustment in clinical and health services research studies has been advocated because the effects of confounding factors in nonrandomized studies can be removed or reduced by adjusting the outcome measures according to risk [1]. The severity and comorbidities can be assessed to remove any potential causative factors for the observed variations in health outcomes between groups in order to differentiate treatment effects [2].

The Charlson comorbidity index (CCI) consists of 19 comorbid conditions weighted according to the degree to which they predict mortality among an inpatient cohort, and then the scores are summed to produce an index score [3, 4]. The Charlson comorbidity index was developed based solely on the assessment of medical records. However, despite the widespread use of medical records, there is an inherent concern that there is a selection bias in the results. The differences in the type and number of comorbidities can result from the number of physician visits and the number of hospitalizations rather than an actual difference in comorbidities. In addition, a retrospective analysis of medical records would require great time investment and expense for a large cohort study. Therefore, an alternative data resource must contain the necessary information regarding comorbidities, and it must enable the performance of data collection and analysis in an efficient manner [5].

Fortunately, the method of calculating scores for CCI is easier than are the methods for other comorbidity indices. Therefore, medical records, patient reports, and claims databases are all potential data sources, though investigators must evaluate the quality, availability, and cost of each [6]. Claims-based measures of comorbidity are of particular importance to cancer care researchers, who increasingly use population-based cancer registry data linked with administrative claims to examine such issues as the associations among treatment, health expenditure, and survival [7].

The situation is similar in Korea. Based on the current privacy laws of Korea, access to medical records has been restricted for researchers. Though Korea has its own cancer registry, due to an insufficient amount of overall clinical information such as the comorbidities, use of the cancer registry data alone is not appropriate. As a result, attempts have been made to develop methods for adjusting the risks using the claims data. Though accuracy of disease coding has been questioned given the purpose of claims data, it could be a useful data source owing to its coverage [8]. Especially in Korea, more than 96% of the total Korean population is covered by national health insurance, and all national health insurance claims are reviewed by the Health Insurance Review & Assessment Service (HIRA), claims data of HIRA provides a fair representation of entire patient groups in Korea [9, 10].

In this study, we compared the CCIs derived from medical chart review versus those based on claims data, and we examined whether the CCI scores obtained from the two data sources could predict the length of stay and the medical expenses, particularly for reimbursement cost by HIRA, in patients who underwent surgery for lung cancer.

Methods

Patients and CCI scoring method

Inpatient and outpatient medical records of patients who were initially diagnosed with lung cancer during a period ranging from January 1, 2000 to December 31, 2004 at the National Cancer Center were examined. We primarily selected patients who were preoperatively diagnosed with stage I to III disease, underwent surgery, and were 18 years of age or older. From 461 patients, 69 patients were excluded from the statistical analysis, because cancer stage or surgical treatment types of 64 patients were ill defined and five patients were diagnosed as state IV after the operation.

The final number of patients included in the study was 392, and the Charlson comorbidity conditions were collected using the Korean version of the CCI [11]. Additional clinical data collected from these patients included age, gender, pathologic stage, histologic type, surgical treatment modalities, frequency of surgery, surgical methods, date of operation, date of initial diagnosis of lung cancer and date of discharge. Comorbidities were considered to be present in cases in which any condition was confirmed more than once in the outpatient or inpatient medical records up to two years prior to the diagnosis of lung cancer. The medical records were analyzed by medical doctors in the Department of Family Medicine at the National Cancer Center from November 2007 to February 2008.

To calculate CCI values from the claims data, we collected the outpatient and hospitalization data for the same subjects using the Electronic Data Interchange (EDI) request data between 1999 and 2005. Information such as the date of treatment initiation, primary and secondary diagnosis codes, and reimbursement cost were gathered from the claims data of 392 subjects.

ICD-10 codes are used in Korea. Therefore, definitions of 17 comorbidities were adopted using an ICD-10 algorithm for CCI, developed by Quan et al. [12]. Based on the CCIs calculated using the claims data, comorbidities were determined according to ICD-10 codes. By assigning the weighted CCI values to the corresponding diseases, the sums of the scores were determined to be the final CCI scores.

We examined the EDI data from the two years preceding the initial date of confirmation of lung cancer codes [C33-C34] at the National Cancer Center, and then we confirmed the presence of comorbidity as defined by Charlson comorbidity conditions claimed to the HIRA for primary or secondary diagnoses. The EDI data were collected from all of the medical institutions in the country including the National Cancer Center. To enhance the diagnostic accuracy of comorbidities in claims data, a rule-out algorithm proposed by Klabunde et al. was applied [6]. This algorithm states that any condition confirmed more than twice in the data that was collected during the same period is considered to be comorbidity [6]. This study was approved by the Institutional Review Board of the National Cancer Center.

Outcome definition

To estimate the reimbursement costs that were requested between the date of operation and the date of discharge, we summed the costs requested within a one-month period after the date of discharge at the National Cancer Center. Cost was transformed into a logarithm for analysis because of the right-skewed distribution of cost. The length of stay was considered to be the date of operation to that of discharge [13]. The length of stay also had a right-skewed distribution and was converted into a normal distribution via logarithmic transformation. In a model assessing the predictive validities of medical outcomes, CCI scores based on the medical records and claim data, which were obtained for each individual patient, were categorized according to three scales: 0, 1 and 2 or greater. These CCI scales were selected as the independent variables for health outcomes. Also we considered age, gender, histologic types of cancer, surgical treatment modalities and pathologic disease stages (stage I, II, III) as adjustable variables, as they have been reported to be prognostic factors of lung cancer in previous studies [14–16].

Statistical analysis

The agreement between medical records and claims data on CCI was assessed based on simple agreement rate and kappa statistics. We performed a multiple linear regression analysis to evaluate whether CCI could predict length of stay and reimbursement cost [17]. All statistical analyses were performed using the Stata/SE 9.0 software package, and statistical significance was tested at a value of p < 0.05.

Results

Agreement between medical records and administrative data

The prevalence of 17 comorbidities and the agreement rate between the two data resources were analyzed from CCIs based on medical records and claims data, and the results are presented in Table 1.

Table 1 Agreements between medical records and claims data (unit = n (%))
Full size table

In cases of myocardial infarction, congestive heart failure, peripheral vascular diseases, chronic pulmonary disease, rheumatoid diseases, peptic ulcer diseases, mild hepatic diseases, complicated diabetes mellitus, and metastatic solid cancer, the comorbidities of claims data were more prevalent than were the medical records data. In cases of primary cancer excluding skin cancer, quadriplegia, and hemiplegia, there were similar frequencies between the two data resources. Moreover, the prevalence of dementia and AIDS were 0% in both datasets.

To assess the agreements for 17 comorbidities between the medical records and claims data, kappa statistics and agreement rate were used. To calculate the kappa value, none of the cell frequencies should be zero, although this was observed in eight comorbidities of our study. Hence, the kappa value was calculated based on nine comorbidities. The agreement rate ranged from 66.07% to 100% for each disease, with chronic pulmonary disease having the lowest agreement rate. The kappa analysis revealed that the kappa value of four diseases, myocardial infarction, cerebrovascular disease, uncomplicated diabetes mellitus, and any malignancy except skin neoplasm, illustrated to fair agreement. In the remaining five diseases, which consisted of chronic pulmonary disease, rheumatoid diseases, peptic ulcer diseases, mild hepatic diseases, and complicated diabetes mellitus, the kappa value showed poor agreement. The concordance rates of the CCI scores between the two data resources are shown in Table 2. Agreement for CCI scores calculated by assigning comorbidities with weighted values had a kappa value of 0.054 (standard error = 0.029). For CCI scores categorized into three scales (i.e., 0, 1, 2+), the kappa statistic was 0.096 (standard error = 0.032). This indicates that the agreement was slightly increased, although it was still poor. According to the distributions of CCI scores between the two data resources, we observed that 196 patients (50.0%) had higher CCI scores as determined using the claims data than for that calculated using the medical records, with 45 patients (11.5%) having lower scores.

Table 2 Agreement in Charlson comorbidity index score between medical records and claims data
Full size table

Subject characteristics

For 392 patients who underwent surgery for lung cancer, the mean age was 60.9 years (standard deviation = 8.8 years). The number of male patients was 291 (74.2%), accounting for the majority of patients (Table 3).

Table 3 Patient characteristics and operative characteristics (n = 392)
Full size table

Prediction for length of stay

For the length of stay prediction adjusted by age, gender, histologic differentiation of the cancer, surgical treatment modalities, and pathologic stage, the CCI scores based on medical records and claims data are shown in Table 4. The CCI scores derived from either database were not prognostic for length of stay after adjusting for age, gender, histologic differentiation of the cancer, surgical treatment modalities, and pathologic stage. However, length of stay significantly increased by 1.15 times (95% CI 1.005-1.314) with pathologic stage 3 lung cancer in the claims data model.

Table 4 Prediction for length of stay using medical records and claims data-based comorbidity index score
Full size table

Prediction for reimbursement cost

Based on the medical records and claims data, a multiple linear regression analysis was performed to examine the predictive power of CCI-based reimbursement cost. After adjusting for age, gender, histologic differentiation of the cancer, surgical treatment modalities, and pathologic stage, CCI scores were not selected as a prognostic factor. The only reimbursement cost was significantly higher at pathologic stage 3 of lung cancer by 1.15 times in claims data based model (95% CI 1.020-1.295) (Table 5).

Table 5 Prediction for reimbursement cost using medical records and claims data-based comorbidity index score
Full size table

Discussion

CCI was originally developed based on data derived from medical records. However, many researchers have proposed that the sole dependence on medical records results in a limitation in cases for which a prompt risk assessment is required by hospitals, insurers and Health Management Organizations (HMOs) [18]. Therefore, CCI tools have been developed for use with claims data coded using ICD-9-CM (International Classification of Diseases, 9th revision, Clinical Modification) [19, 20] and ICD-10 (Switzerland, Australia, Canada versions) [12, 21, 22].

However, the consistency between medical records and claims data is not excellent. For 485 patients who underwent prostatectomy, the comorbidities based on the medical records and claims data were compared. The kappa value was greater than 0.61 for five diseases (uncomplicated diabetes mellitus, primary solid tumor, moderate hepatic disease, connective tissue diseases, leukemia), but the remaining diseases had poor agreement [23]. Also Newschaffer et al. compared CCI scores between the medical records and claims data in 404 patients with breast cancer. This comparison revealed that the kappa value was 0.36, corresponding to 'fair agreement' [24].

In a Korean study that targeted patients who underwent surgery for the treatment of gastric cancer, the kappa value of CCI comorbidities between the data resources was fair for five diseases; however, excluding these comorbidities, the kappa value was less than 0.2. Especially the prevalence of peptic ulcer disease was 41.6% according to the claims data and 3.5% according to the medical records data [25]. Another study, which focused on patients who underwent hip joint arthroplasty, reported that the kappa value of comorbidities for two data resources was 0.8 for metastatic solid tumors and 0.51 for uncomplicated diabetes mellitus. In other comorbidities, the kappa value was smaller than 0.29 [26]. In this study, there were discrepancies between the two data sources, supporting the results from previous studies in Korea and other countries.

One possible explanation for this disagreement is that there is an underestimation of CCI comorbidities based on medical records data. Medical records based on CCI scores were retrospectively obtained from the one medical institution, whereas claims data based on CCI scores were gathered from administrative data, which includes all medical institutions' claims data on the selected patient. In addition, poor agreement between medical record data and claims data may result from the differing motivations for data collection between medical records and claims data.

Inconsistencies have also been observed in previous studies investigating whether CCI scores obtained from claims data could be used to predict the length of hospital stay. According to a study of 20,138 patients who underwent surgery for radical urological cancer, the length of stay was prolonged by approximately 1-2 days in the group in whom CCI scores were at least 1 compared to the groups with 0 points [27]. In another study, 1,216 patients who visited an outpatient clinic with a chief complaint of acute chest pain, compared with the group in which CCI scores were calculated to be 0-1 points from the medical records, the length of stay was delayed by 14.4-fold (95%CI 3.9-25.9) in the group in which CCI scores were calculated to be 2-3 points and by 25.3-fold (95%CI 2.4-25.5) in the group in which CCI scores were calculated to be < 4 points [28]. However, in 1,945 patients who underwent carotid endarterectomy, CCI scores based on claims data did not correlate with the length of stay, though CCI scores based on medical records were associated with an increased length of stay [1]. In the current study, according to a length of stay prediction model, CCI scores calculated using medical records and claims data were not shown to be prognostic factors. This may be due to the fact that CCI was originally developed to predict mortality, and it is simply not an appropriate tool for predicting length of stay [29]. Also, the postoperative length of stay may be dependent on the severity of the procedure rather than comorbidities [27].

Moreover, there were disagreements in the predictability of CCI with regard to reimbursement cost. In a medical record-based study of dialysis patients, the mean medical expense per year was $54,000 in cases in which CCI scores were calculated to be less than 4 points, $108,000 in those where CCI scores were calculated to be 4-5 points, $247,000 in those where CCI scores were calculated to be 6-7 points and $407,000 in those where CCI scores were calculated to be 8 or more points. These differences were found to be statistically significant (p < 0.0001) [30]. However, CCI scores did not correlate with medical expense in head-and-neck cancer patients [31]. In our study, according to a reimbursement cost prediction model that was established based on the medical records and claims data as related by CCI, CCI was not selected as a prognostic factor for either model. This could be attributed to the fact that the prediction model was developed for death hazards [29].

This study has several limitations. First, medical records based on the CCI scores of selected patients were retrospectively obtained from the records of the National Cancer Center. However, claims data based on the CCI scores of selected patients were gathered from HIRA, which includes all medical institutions' claims data on the selected patients. But this could be regarded as strength of claims data in terms of accessibility and efficiency. Second, we considered the reimbursement cost of claims data only with respect to the availability of data. As a result, other medical expenses, such as non-covered services, were excluded. Also, the reimbursement cost was calculated as the sum of the medical costs from the date of surgery to one month after the date of discharge. Therefore, it is possible that outpatient visits except follow-up cancer treatment within a one-month period after discharge may have been included. Also though we considered some prognostic factor such as pathologic staging, there are other prognostic factors such as smoking or adjuvant chemotherapy. More information about such factor could affect the health outcomes and the predictability of CCI.

In this study, there was poor agreement between medical records and claims data. In addition, CCIs based on both data sets were not suitable for predicting length of stay or medical expenses. Given how easy it is to calculate CCIs based on claims data, especially in a social health insurance system, there should be further studies to improve methods for calculating CCI scores to predict health outcome.

Conclusions

Considering the difficulty of accessing medical records in the presence of privacy protection laws, the use of claims data to calculate CCI is a very attractive and potentially useful method. Claims data can also be used to supplement the medical records because claims data can be collected easily by different medical institutions. However, in this study, the agreement of CCIs based on the two datasets was not sufficient, and CCIs from neither data set were able to predict length of stay or medical expenses. Therefore, improvement and modification are needed to successfully use CCI scores based on claims data, at least in the case of lung cancer.