Background

Comparing outcomes between clinicians, institutions or area health services requires consideration of potential confounding factors, such as patient age, health status or disease severity, that could account for any observed differences in outcomes. Risk adjustment is a statistical technique used to account for variations in outcome that arise from differences in the characteristics of patients, so as to provide a fair and more accurate comparison of health services. Patient age, sex, cancer site and cancer stage are commonly used in risk adjustment models as they have high levels of completeness and accuracy in routinely collected datasets such as cancer registries and hospital episode collections.

Adjustment for comorbid conditions is common when studying outcomes following cancer surgery [1]. The Charlson comorbidity index summarises information about the presence of a number of medical conditions, assigns a severity score to each, and sums the severity scores to create a single measure of comorbidity [2]. Algorithms have been derived to calculate the Charlson comorbidity index from hospitalisation records coded using the Ninth [3, 4] and Tenth [5, 6] Revisions of the International Classification of Diseases. The Charlson comorbidity index has been shown to be a good predictor of both short-term (in hospital and 30-days) and longer-term (12-month) mortality [7].

Functional status is an additional patient factor that may explain differences in outcome. One measure of functional status is the Eastern Cooperative Oncology Group (ECOG) Performance Status, [8] a score ranging from 0 (“fully active”) through 3 (”capable of only limited self-care”) to 5 (“dead”). An alternative measure of a patient’s general health is the American Society of Anesthesiologists (ASA) score [9]. This measure of physical status ranges from 1 for a normal healthy patient to 5 for a moribund patient who is not expected to survive surgery and is recorded routinely by anaesthetists prior to any surgical procedure. ECOG and ASA scores are available in some administrative data collections.

While recent studies have assessed the predictive capacity of comorbidity and health status measures in risk adjustment for outcomes of cancer surgery, [10, 11] they did not consider performance status. A study that compared comorbidity index, ASA and ECOG performance status [12] considered each measure separately, and did not assess the additional effect of health and performance status over comorbidity. In a recent study comparing ASA with ECOG, [13] no comparison was made with comorbidity index as it was not possible to derive a comorbidity index from the available data. Additionally, these studies used data extracted from patients’ medical records, not routinely collected data.

Therefore, we conducted this study to assess how frequently the ASA score and ECOG performance status can be obtained from population-based administrative data collections in New South Wales, the most populous state in Australia, and to assess the contributions of ASA score and ECOG performance status additional to the Charlson comorbidity index, in comparing outcomes of colorectal cancer surgery between hospital peer groups based on risk adjustment models.

Method

Data sources

Our study population comprised all people with an incident cancer of the colon or rectum registered in NSW, the most populous state in Australia. We used the first recorded tumour of the colon or recto-sigmoid junction (ICD-O-3 codes: C18, C19) or rectum (C20) registered on the NSW Central Cancer Registry (CCR) between 1 January 2000 and 31 December 2008. The NSW CCR, a population-based register of all cancers in NSW residents, receives notifications from public and private hospitals, departments of radiation oncology, nursing homes, pathology laboratories, outpatient departments and day procedure centres as a statutory requirement.

Cancer notifications were linked to the Admitted Patients’ Data Collection (APDC), the Clinical Cancer Registry (ClinCR) and the NSW Registry of Births, Deaths and Marriages (RBDM). The APDC contains demographic and episode-related information on all hospital separations for public and private hospitals in NSW, [14] while the ClinCR contains clinically-based information, including the ECOG performance status, for patients with cancer treated in public facilities within six area health services [15]. Hospital admissions with procedures defined as being for curative intent were used, [15] with these procedures based on clinical consultation and previously used procedure codes [16, 17]. Where patients had more than one hospital admission with a procedure of interest, the first admission was used in analysis. We used hospital records with separations in 2007 and 2008, as this was the period for which clinical cancer registry data were available. Fact of death was obtained from the RBDM for the period up to 2010.

Probabilistic data linkage was conducted by the NSW Centre for Health Record Linkage, and we received only anonymised data. Ethics approval for the study was given by the NSW Population and Health Services Research Ethics Committee (HREC/11/CIPHS/32).

Outcomes

We considered three binary outcomes. Mortality was assessed at 12 months, measured from date of procedure until death from any cause. Hospital stay following surgery was measured from the date of procedure until separation, accounting for transfers, and categorized based on exceeding the recommended length of stay for procedures based on the Australian Refined Diagnostic Related Group [18]. This allowed for length of stay to be assessed based on procedure complexity. Readmission within 28 days was derived from the date of separation until any subsequent readmission.

Peer group

Peer groups classify similar hospitals based on the number of separations, primary role and location (metropolitan or rural). Admitting hospitals were classified into one of six groups based on NSW Peer Group definitions. [19].

Measures of patient health

We calculated the Charlson comorbidity index using codes for comorbid conditions recorded during the index admission in the APDC. We excluded metastatic cancer from the calculation as in the context of cancer treatment, this is part of the condition of interest rather than a comorbidity. Charlson comorbidity indices were categorised into values of 0, 1 and 2 or more in risk adjustment models as very few patients (8 %) had scores of two or higher.

The ASA score was obtained from the suffix of ICD procedure codes indicating administration of an anaesthetic (see Additional File 1). Possible ASA scores are 1 (normal healthy patient); 2 (mild systemic disease); 3 (severe systemic disease); 4 (severe systemic disease that is a constant threat to life) or 5 (moribund patient who is not expected to survive) [9]. We grouped ASA scores of 4 or 5 in risk adjustment models due to the sparseness of higher scores.

ECOG performance status was obtained from ClinCR records. We used the ECOG performance status recorded closest in time on or before the data of procedure. Possible ECOG scores are 0 (fully active, no performance restriction); 1 (restricted in physically strenuous activity but ambulatory, able to carry out work of a light or sedentary nature); 2 (ambulatory and capable of all self-care but unable to carry out any work activities. Up and about more than 50 % of waking hours); 3 (capable of only limited self-care, confined to bed or chair more than 50 % of waking hours); 4 (Completely disabled, cannot carry out any self-care. Totally confined to bed or chair) or 5 (dead) [8]. We grouped ECOG performance status of 2 or higher in risk adjustment models due to the sparseness of higher scores.

Other patient factors

Extent of tumour spread, obtained from the CCR, summarises the most aggressive extent of the disease based on diagnostic and therapeutic evidence within four months of diagnosis [20]. Patient sex, age and emergency status of admission were obtained from the APDC, with age calculated at date of procedure. Country of birth, also obtained from APDC, was classified as Australian versus other.

Statistical analysis

Descriptive statistics were used to compare patient characteristics for patients with non-missing values of Charlson comorbidity index, ASA score and ECOG performance status. Associations between the three patient health measures were summarised using Spearman’s rank correlation coefficients.

Risk adjustment models were constructed using two-level multilevel logistic regression models to compare the binary outcome measures across the peer groups, with patients (level-1) clustered within hospitals (level-2). Random-intercept models were fitted, with hospital included as a random effect. Only patients with non-missing values of Charlson comorbidity index, ASA score and ECOG performance status were used so that risk adjustment models could be compared on the same group of patients. Hence we used only records from patients treated in public hospitals. As such, only the four peer groups representing public hospitals were considered: Principal Referral A, Principal Referral B, Major Metropolitan and Major Non-Metropolitan. Major Metropolitan and Non-Metropolitan peer groups were grouped together due to the low patient numbers in Major Non-Metropolitan hospitals.

We evaluated the utility of ASA score and ECOG performance status by fitting all combinations of these over and above a base model assessing the association between hospital peer group and each of the three study outcomes (12-month mortality, extended length of stay and readmission within 28 days) adjusted for patient age, sex, extent of disease, emergency status of admission and Charlson comorbidity index. Model performance was assessed using Akaike’s information criterion (AIC), with lower values indicating a better fitting model. We also calculated the C statistic, summarising the concordance between observed and predicted events. Concordance ranges between 0.5 (a model with no predictive ability) and 1.0 (a model with perfect predictive ability) [21]. Finally, we compared estimated logistic regression parameters for hospital type to assess the extent of confounding explained by the inclusion of ASA score and ECOG performance status.

Data processing and descriptive analyses were conducted using Version 9.3 of the SAS System for Windows (Cary, NC, USA), with multilevel logistic modelling conducted using adaptive quadrature via the xtmelogit command of Stata Version 12 for Windows (College Station, TX: StataCorp LP).

Results

We obtained records from 6,964 individuals who were operated in 2007 or 2008, and their characteristics are presented in Table 1. The Charlson comorbidity index could be calculated for the entire sample, as it is based on comorbid conditions recorded during the index admission. ASA score was recorded for 78.0 % of these individuals, and while they were similar to those without ASA score recorded for most characteristics, those with ASA score recorded were more likely to have been operated on in a public hospital.

Table 1 Characteristics of patients, operated on between 2007 and 2008 for cancer of the colon or rectum in NSW, Australia
Full size table

There were 3,848 individuals who had a potentially curative surgical procedure at a public hospital in 2007 or 2008. Of these, 3,079 individuals had a record in the Clinical Cancer Registry and 748 (24.3 %) had an ECOG performance status recorded before their cancer surgery. Patients with a recorded ECOG performance status were more likely to be male, have a non-Australian country of birth, and have a more advanced extent of disease compared to those without ECOG performance status.

The three measures of patient health status were not highly correlated with one another (rASA,Charlson = 0.29, rASA,ECOG = 0.15, rECOG,Charlson = 0.11) (Table 2).

Table 2 Cross tabulations of Charlson, ASA and ECOG scores (n = 575)
Full size table

There were 575 records with non-missing measures of health status and other covariates that could be used in risk adjustment comparisons. The majority had Charlson scores of 0 (n = 502, 87.5 %) with 25 (4.4 %), 38 (6.6 %), 3 (0.5 %) and 6 (1.0 %) scoring 1,2, 3 and 4 respectively. There were no Charlson scores higher than 4. Results of model performance appear in Table 3. For each outcome, the estimated intracluster correlation coefficient was zero for all models.

Table 3 Summaries of multilevel logistic regression model performance for three binary outcomes
Full size table

Including ASA score and ECOG performance status reduced the AIC for each outcome, but the effect was not consistent. The AIC statistic was minimised by: the inclusion of both measures for 365-day mortality; the inclusion of ASA score for extended length of stay and the inclusion of ECOG performance status for 28-day readmission. Concordance was maximised by the inclusion of both measures for all outcomes. However the maximum concordance for each outcome was similar to that of the best performing models based on AIC statistics.

The effect of including ASA score and ECOG performance status on the logistic regression parameter estimates for hospital-type is summarised in Table 4. Changes in parameter estimates due to adjusting for different measures of health status were small, relative to the parameter estimates’ standard errors, and there was no consistent pattern in the confounding effect of ASA score and ECOG performance status.

Table 4 Peer-group parameter estimates from multilevel logistic regression analysis for three binary outcomes
Full size table

Discussion

Our study reports on the utility ASA score and ECOG performance status for risk adjustment over and above Charlson comorbidity index using routinely collected, population-based data for NSW. The correlation between the three measures was not strong, suggesting that the measures represent independent attributes of patient well-being. Our study found that, for patients with non-missing Charlson comorbidity index, ASA score and ECOG performance status, there was no clear optimal measure of patient-health for risk adjustment. However, there was no clearly deficient measure. We also found that, after including Charlson comorbidity index, there was little to be gained by adding ASA score or ECOG performance status when comparing outcomes between hospital types.

Within the NSW hospital episode statistics (the APDC), it is standard coding practice that only comorbid conditions which have an impact on the admission of interest are coded. While this means that the Charlson scores derived from the APDC diagnosis codes are based on conditions likely to impact outcome, it leads to under-estimation of the prevalence of comorbid conditions in the patient cohort [22]. We have previously investigated the impact of including additional comorbidity information from previous hospital admissions in risk adjustment models to compare hospital cancer outcomes [13]. This previous work demonstrated that although Charlson scores were higher when more sources of comorbidity information were included, there was little change in the performance of Charlson score in the hospital risk adjustment models. Thus, for comparison of hospital cancer outcomes, calculating the Charlson score from comorbidity information within the index admission was the most efficient approach, and the approach used in this study. We acknowledge however that additional comorbidity information from other sources could improve risk adjustment models in other contexts, for example to compare individual patient outcomes.

While ASA score is reported more commonly in public hospitals than private, patients with ASA score have similar characteristics to those with no recorded ASA score. The reporting of ECOG performance status however, is limited. Only a quarter of patients whose cancer surgery was conducted in a public hospital had an ECOG performance status recorded before their cancer surgery. While we used the ECOG performance measured closest in time prior to cancer surgery, the ideal would be to record ECOG close as possible to surgery, possibly in a preoperative setting. As the analyses in this study were necessarily restricted to a subset of patients with colorectal cancer in NSW who were treated in a public hospital and had complete data for Charlson, ECOG and ASA, it is possible that these findings do not generalise to other patient groups. This should be further investigated in future studies.

The findings from this study regarding predictive ability agree with those from other published work. While comorbidity (as measured by either ASA or Charlson comorbidity index) has been demonstrated as a risk factor for poor outcomes in studies of colorectal cancer surgery, different comorbidity indices added little to the predictive ability regression models, [10] and yielded strikingly similar results [11]. Similar performance of ASA, Charlson comorbidity index and ECOG performance status was seen in a study that modelled90-day post-surgery mortality for carcinoma of the bladder [12]. It should be noted though, that these studies were based on data extracted from primary medical records. Recently, cancer-specific measures of comorbidity (‘C3 indices’) developed using administrative data have been found to have slightly improved performance compared with the Charlson index for colorectal cancer [23]. However, the developers conclude that the differences are so small as to be insignificant, and that any measure of comorbidity will suffice in most circumstances [23]. Consistent with these findings, our study suggests that alternative measures of patient health derived from administrative data collections provide similar predictive ability.

While our study is based on population-level data from high quality data collections, there are a number of limitations. There were only two years of data with the potential for ECOG performance status to be measured, and only a small proportion of patients had ECOG performance status recorded. We could not include patients who received surgery in private hospitals, as ECOG performance status was recorded in the Clinical Cancer Registry which does not yet include private hospitals. Those with non-missing ECOG performance status differed systematically from the entire cohort, and it is not clear whether the same conclusions would be reached in a more representative population. While multiple imputation methods could be used to include patients with missing data, the high proportion and likely non-random nature of the missing ECOG data in our study were against this approach [24]. We could not assess thirty-day mortality, arguably a more important indicator of surgical quality, in the risk adjustment models due to the low numbers of deaths. Finally, the levels of completeness of data fields likely vary in different settings such as the public and private sector, and within and between countries, so our findings may not generalise to other contexts.

Conclusion

ECOG performance status has some use in risk adjustment, but its routine use in population-based comparisons of hospital outcomes cannot be recommended due to limited coverage. Coverage of ASA scores is greater than for ECOG performance status, but there are still substantial issues of non-reporting, particularly in private hospitals.

While predictive ability of regression models is maximised by inclusion of one or both of ASA score and ECOG performance status, there is little to be gained by adding ASA or ECOG to models containing the Charlson comorbidity index to address confounding. The Charlson comorbidity index is an appropriate measure to use in risk adjustment to compare outcomes between hospitals.