Colorectal cancer ranks as the third cause of cancer deaths in the United States and the fourth cause worldwide [1, 2]. Thanks to improvements in diagnostic and surgical modalities as well as the advancement of chemotherapy and radiotherapy, many colorectal cancer patients do survive more than five years. The five-year relative survival rate rose from 51% two decades ago to 65% in recent years [2]. Still there are concerns about the quality of clinical practice since evidence has shown discernible variations for patients with similar conditions [35]. Evidence suggests that underuse and overuse of care may occur for patients with cancer [4]. Also, compared with the outcomes of patients in clinical trials, the outcomes of cancer patients in daily practice may be less favorable [6, 7]. Research on the quality of care throughout at least the last decade has demonstrated that increases in the knowledge of treatments with proven efficacy do not translate directly to the optimal delivery of such treatments to patients [6, 8].

Promoted by the Institute of Medicine, USA, the field of quality measurement has been growing rapidly along with accumulation of experience and knowledge in the cancer care quality area [9]. Core measures, introduced by the Joint Commission on Accreditation of Healthcare Organizations, use standardized sets of valid, reliable and evidence-based "core" measures that have been incorporated in quality measurement and performance improvement systems as well as the accreditation process for several diseases [10, 11]. There are several groups who have previously developed measures for colorectal cancer [12]. Yet, the studies regarding measurement of cancer quality of care based on a national database are scarce and we know of no published studies related to core measures for cancer. Having a single set of measures for the entire country is the only way to set national benchmarks and compare countries that have national programs [13]. Our research team adopted the concept of core measures to assess the quality of care provided to patients with breast cancer, colorectal cancer, cervical cancer, lung cancer, oral cancer and hepatocellular carcinoma in Taiwan.

Health insurance in Taiwan is provided within a single payer system that has covered over 96% of the population since 1995 [14]. Following the Cancer Prevention and Management Law, a revised cancer registry, the Taiwan Cancer Database (TCDB) was created in 2003 [15]. The Bureau of Health Promotion (BHP) in the Department of Health financially supported 27 hospitals (initially 17 major hospitals) to join the TCDB program with the prerequisite that each hospital is required to provide cancer prevention, cancer screening, diagnosis and treatment of cancer, education and clinical audits, and cancer research as well as a quality assessment system [16]. The TCDB is a nationwide oncology outcomes database that currently covers approximately 60% of six new invasive cancer diagnoses in Taiwan each year. Like its counterpart, the National Cancer Data Base (NCDB) in the USA, the TCDB is a large, powerful database providing multiple opportunities for clinical studies and can be used to benchmark hospitals on performance measures--an impetus for quality improvement initiatives at the hospital level [17]. Each hospital that voluntarily participates must submit data twice a year for six types of cancers that mentioned above. The prevalence of each disease is increasing, motivating public interest in cancer care quality nationwide. The core cancer measures were developed in parallel with the TCDB program through a government grant as a national initiative for cancer care quality improvement [15]. Although performance measurement for assessing care quality is booming, a system for measuring the quality of cancer care at the hospital level has not been well developed. The purpose of this study was to develop organization-based core measures for colorectal cancer patient care and apply these measures to compare hospital performance.


Patients and hospitals

The patients and hospitals registered in the TCDB program in Taiwan during 2004 were eligible for this study. Participation in the program is voluntary for each hospital but the number is increasing each year partly due to financial incentives. Some information is required in order to link each cancer registry record to insurance claims data. Due to problems with the timeliness and quality of data for nine of the hospitals, eighteen hospitals were selected as the main sources of data for this study (see Appendix).

Core measures development

The process of core measures development has gone through a preparation stage, a consensus building stage, and two rounds of stakeholder feedback (see Figure 1), similar with our pilot study of breast cancer [16]. This study was sponsored by a grant (DOH93-HP-1057) and approved by Review Board of Data Release (equivalent to Ethics Committee) from the Bureau of Health Promotion, Department of Health, Taiwan. Seven university faculty members (including Author KPC, SHC, LTC, RT, TWL, MJS and a scholar) voluntarily constituted the research team, which held four meetings in the preparation stage. The research team then collaborated with experts and other stakeholders to develop rigorous, evidence-based, scientifically sound measures which could be used as the standards of cancer care quality nationwide. The search terms used included the Medical Subject Heading terms 'outcome and process assessment health care', 'quality indicator health care', 'quality of health care', 'practice guidelines', 'evidence-based medicine', 'colorectal neoplasms' and the keyword 'performance measures' from January 1985 to January 2002. All possible quality-related cancer indicators were summarized (including RAND (Research and Development Corporation, USA) and National Health Service guidelines) [1830] and reduced to a final measure set via a modified Delphi method and stakeholder feedback. In addition to validity and feasibility, indicator selection was guided by other criteria such as scientific acceptability, importance and necessity, and usability [16].

Figure 1
figure 1

Design and development of core measure for colorectal cancer. *members of research team included KPC, SHC, LTC, RT, TWL, MJS and a scholar, #twenty expert were invited (research team member excluded), see also the expert panel in Appendix, 1 including 37 structure indicators, 78 process indicators, and 7 outcome indicators, 2 including 9 structure indicators, 37 process indicators, and 4 outcome indicators, 3including 4 pre-treatment indicators, 21 treatment indicators, and 5 monitor indicators, 4including 3 pre-treatment indicators, 16 treatment indicators, and 6 monitor indicators, 5including 2 pre-treatment indicators, 16 treatment indicators, and 6 monitor indicators, 6including 1 pre-treatment indicator, 11 treatment indicators, and 5 monitor indicators, 7hospitals and clinical professionals are major stakeholders, 8questionnaires were mailed on August 16, 2004, response rate 95% (19/20), 9 expert meeting was held in Taipei, September 22, 2004, 10 questionnaires were mailed on October 26, 2004, response rate 85% (17/20), 11 1st symposium was held in Taipei, December 27, 2005, 12 2nd symposium was held in Taipei, March 27, 2006.

The Delphi group, nominated by the Taiwan Clinical Oncology Group from the National Health Research Institute as well as related specialty societies, encompassed twenty clinical professionals who represented the organization of their specialty. Included were four gastro-intestinal physicians, three oncological physicians, three radiological oncologists, seven colorectal surgeons and three pathologists (see Appendix). The panel members met face-to-face during 2004. Indicators were anonymously rated on a 9-point scale (with 1 denoting not valid and 9 very valid) and only indicators with a median score of 7 or higher were selected as final candidate indicators. After initial analysis of the panel data, the expert panel and stakeholders from different hospitals met on March 27, 2006 to review and streamline the indicators.

Comparisons across hospitals

Currently, the emphasis on quality measurement in clinical, including cancer, care is on evaluating adherence to "evidence-based" process measures. This quantitative approach to quality improvement moves the field forward by ranking quality measures according to both quantitative assessment of under-performance across institutions and the measures' potential impact on outcomes [31]. Reeves et al. recently summarized five methods for computing a composite quality score including "Indicator Average" and "Patient Average" [32], both of which we used in this study to derive adherence across hospitals and were designated as indicator-based and case-based adherence. Indicator-based adherence was derived from "all the care received in a given hospital for a specific indicator" divided by "all the care recommended in a given hospital for that indicator". Case-based adherence was computed as the percentage of indicators that were successfully met for each patient. These percentages were then averaged across all patients in a given hospital. To compare hospitals, three different scenarios were adopted. In the first scenario, each hospital was awarded one point for each indicator in which adherence to criteria was greater than the 50th percentile. These points were then added to form a hospital score. Similar procedures were followed for Scenarios 2 and 3 except the thresholds were set at the 75th and 90th percentiles respectively. This approach made hospital comparisons an easier task.


Patient and disease characteristics

A total of 5585 patients who had newly diagnosed colorectal cancer and complete information regarding their treatment during 2004 entered into this analysis. The demographic characteristics are listed as Table 1. Mean age of the recruited sample was 64 years old. Males were more prone to have colorectal cancer (male to female: 58.3% to 41.7%), even in each stage. More than sixty percent of the patients were older than sixty years old. Patients younger than 40 years old comprised only 6% of the study group. By disease stage, Stage III was the most common stage at the time colorectal cancer was discovered. It constituted 29.9% of the sample, followed by Stage II (24.9%), Stage IV (20.6%) and Stage I (14.1%). Carcinoma in situ was only found in about 3% of the newly diagnosed patients (Table 1).

Table 1 Characteristics of colorectal cancer patients undergoing surgery in 2004 by cancer staging.

Core measures

The final set of core measures for colorectal cancer encompassed seventeen indicators belonging to three groups: one for "Pre-treatment", eleven for "Treatment", and five for "Monitoring". Table 2 (see Additional file 1 for definition) displays the description, rationale, numerator, denominator, and evidence level (or grade of recommendation) for each indicator. However, not all core measures of colorectal cancer developed in this fashion were included in this study. All five indicators for "Monitoring" were excluded due to the constraint of short study interval. Two indicators (T2 and T8) requiring intensive chart reviews were not also included. Finally, ten indicators (T3 was divided into T3a, T3b sub-indicator) could be constructed from attainable data for colorectal cancer. Nearly all data required were retrieved from the TCDB with the exception of the T1 indicator, in which partial information was obtained from claim data.

Table 2 Core measure indicators for colorectal cancer

Hospital performances and comparisons

Hospital performances were displayed as indicator-based and case-based adherence. To show system performance for colorectal cancer, the box plots in Figure 2 illustrate the range of hospital performance with respect to indicator-based adherence. From this informative plot, we learned how good or how bad our hospitals performed with respect to each indicator--the display helps make explicit which indicators might be improved. Five indicators (45% or 5/11) in this plot had mean adherence greater than 90% (T4: 98.3%, T5:99.3%, T6: 97.6%, T10: 91.8%, T11: 96.8%). Still, there were many indicators showing a wide range of performance. We also note that two indicators had mean adherence less than 70% (T1 66.6%, 36.6% to 85.0%; T7: 62.7%, 27.6% to 92.2%). The best and poorest system performance occurred for T5-negative surgical margin (99.3%, ranged from 97.2% to 100.0%) and T7-lymph nodes harvest more than twelve (62.7%, ranged from 27.6% to 92.2%). Both indicators relate to surgical specimens.

Figure 2
figure 2

Box-plot illustrates hospital's adherence for each core measure indicator (indicator-based) of colorectal cancer. Mean of five indicators (T4, T5, T6, T10, T11) had adherence greater than 90%. PT: pre-treatment, T: treatment.

Table 3 summarizes the details of adherence to the core measures and hospital scores for inter-hospital comparisons. Data were presented as means, 95 percent confidence intervals and ranges as indicated. Then we displayed results in a table of inter-hospital comparisons. Adherence to indicator-based and case-based approaches for each hospital ranged from 79.8% to 91.0% and 77.6% to 88.9%, respectively. Results of indicator-based approach are always larger than case-based approach except for one hospital (H06). On average, patients received 85.8% of recommended care for indicator-based adherence, and 82.8% for case-based adherence. The three different scenarios of hospital scoring methods could only be applied to indicator-based adherence. The average hospital scores were 5.3 (2 to 8), 3.5 (0 to 6), and 2.2 (0 to 4) with respect to the 50th, 75th, and 90th percentiles respectively. Hospital rankings were not consistent among the three methods. Adoption of alternative methods of aggregating scores can lead to different interpretations.

Table 3 Results of organizational performance of colorectal cancer care by core measure indicators


The core measures for colorectal cancer discussed in this study are one of the six cancer-specific sets of indicators developed in a government-sponsored program with scoring technique improvements, larger expert participation. Refinement of core measures development from our prior study of breast cancer included: 1. expand the number of Delphi panelists from seven to twenty; 2. adopt the RAND Appropriateness technique (scale 1 to 9) to select indicators from previously developed technique (scale 1 to 5, median equal or greater than four and 86% agreement threshold) [5]; 3. change threshold for selecting indicators to a score of seven or higher. More stakeholders' participation and discussion during stakeholder feedback symposium suggest enthusiasm and the commitment of healthcare professionals in the development of methods for quality improvement in colorectal cancer care. However, the level of evidence for indicators could not match those of the pilot study we developed previously for breast cancer, which included six level I, four level II and five level III recommendations. Nevertheless, it is our belief that these measures represent a valuable contribution to the growing library of clinical cancer measures which can be used for public awareness, potential improvement and accreditation standards for cancer care centers or specialty groups.

Any new performance measure set, including the current cancer core measure sets, frequently encounter a major hurdle, i.e. data resources. It would be easier if the information needed to construct measures came from ICD-9 codes of a billing system. But actually for cancer care research, requirements include the stage, tumor size, node numbers, and status of section margins, none of which are captured in claim-based systems. Although these data are available in medical records, such detailed review of charts on a national scale would be cumbersome and expensive [33]. The TCDB program solved this dilemma; its establishment was the result of the increasing demands for research that would help inform reporting and pay for performance systems. Breast cancer was the first disease entity undertaken by the TCDB program. The number of newly diagnosed breast cancers represented 81.4% of all new cases in 2004, partly due to greater efforts in reporting. Reports of colorectal cancer didn't catch up this pace, but finally we identified 5585 patients newly diagnosed in 2004 within the TCDB. This sample accounted for 58.6% (5585/9535) of all newly diagnosed patients with cancer that occurred in Taiwan that year. We consider this sample representative of the population who has been and is being treated for colorectal malignancy in this country. The scope of the TCDB program will hopefully extend to all types of cancer in the future and will continue to provide feedback to participating hospitals.

Our results suggest that patients with colorectal cancer receive 86% (indicator-based adherence) and 83% (case-based adherence) of recommended care. Adherence to four indicators approached 100%, indicating that excellent quality of care is accomplishable. Yet, we observed four indicators with less than 85% adherence, and for these indicators, there was substantial variability in quality across hospitals. The adherence to twelve core measures for breast cancer in our prior report was inferior (65.8%, indicator-based).

Our results can be compared with a growing literature that describes the quality of cancer care, as well as studies that describe care quality of chronic conditions. For example, results of the National Initiative for Cancer Care Quality (NICCQ) showed that patients with breast cancer receive 86% of recommended care, whereas patients with colorectal cancer receive 78% of recommended care [3]. Using a methodology similar to that of NICCQ, two national studies suggest that the quality of care for cancer may be similar or better than that observed for other chronic medical conditions. Jencks et al. reported 73% of Medicare beneficiaries received health care services specified by one of 24 quality measures addressing heart disease, stroke, and pneumonia [34]. Another study by McGlynn et al. concluded that participants in 12 metropolitan areas of the USA received approximately 55% of recommended care for thirty acute and chronic conditions according to the results from a nation-wide telephone survey [5]. But in that study, they reported 53.9% (95% CI, 45.7-60.4) adherence to twelve quality indicators for colorectal cancer.

Indicator-based adherence is another issue that must be discussed. Since the quality measurement of cancer care is emerging, few studies have reported results of hospital adherence for each indicator. Adherence measured by four indicators in our results (T1-diagnostic evaluation, T3b-surgery, T7-pathological reporting, T9-adjuvant therapy) were less than 85%. In contrast, all diagnostic evaluation, adjuvant therapy (but not surgery), and pathological reporting in the NICCQ showed adherence less than 85% [3]. Another unexpected finding is the adherence to indicator T9, the only indicator with Level I evidence support, was less than 85% (77.4%). Of note, we explored the best (T5) and poorest (T7) adherence for system performance, which might suggest that surgeons did well to achieve safety margins for cancer specimens but failed to remove more than twelve lymph nodes as recommended. But actually there are three major influences on the total number of nodes harvested: the surgeon, with regards to technique and philosophy; the pathologist, with regard to specimen evaluation technique (fat clearing or cherry-picking) and philosophy; and patient characteristics, with regard to tumor site, T stage, immune response, and age [35]. A study by Bilimoria et al. lamented that more than 60% of US institutions failed to achieve a compliance benchmark for the 12-node measure and called for efforts aimed at improvement in colon cancer nodal evaluation [36]. According to their benchmark (examination of ≥12 nodes in ≥75% of patients), we had as much as 72% (13/18) of institutions that failed such a benchmark and actually this percentage might be under-estimated for the daily practice for all cancer care in Taiwan. Improvement should be encouraged for these controllable factors.

Two different types (indicator-based and case-based) of adherence have been presented in this organizational level study. Results of case-based adherence and indicator-based adherence were similar (usually slightly smaller for case-based). But it seems there may be more practical applications for indicator-based adherence since as shown in Table 3, indicator-based adherence can provide more details in terms of potential improvements to be considered than case-based adherence. The box-plot in Figure 2 was also created according to the indicator-based adherence methodology. For quality comparisons at the organizational level, we would suggest indicator-based adherence measures, which provide more needed information on quality improvement than case-based adherence.

Individual quality measures have significant limitations for assessing performance. Despite growing interest in composite measures, methods for combining multiple domains of surgical quality are not well established [37]. Composite scores are an aggregation of underlying performance indicators into a single index and have been used widely in the public sector to create one more easily understood number or rating [38, 39]. But in combining quality indicators to rank hospitals, how one aggregates quality measures for easy understanding is still disputable. Following the criterion-referenced method, we arbitrarily used three different thresholds to rank performance of participating hospitals. Composites are bedeviled by questions of weighting [30]. All weighting systems are arbitrary, and equal weighting is transparent even if it is indefensible at the margins [40]. In the current study, every opportunity to deliver recommended care is weighted equally. However, with equal weighting, quality measured by a composite could seem high even if rare but critical processes were lethally unsafe. Given the increasing use of composites, research is needed to define which indicators can legitimately be made into composites, to develop and test clear rationales for weighting schemes, and to build understanding of how changing weights alters conclusions about performance [41].

As quality measures are gaining momentum in the healthcare industry, our findings have important implications for health policy. The core measures can be used for internal improvement of cancer care in the near future. Also, core measures may provide accountability for the basis of pay-for-performance systems and public reporting, as well as for accreditation.

Several limitations of this study should be discussed. First, the database of this study was secondary data. Although the personnel of the cancer registry for each organization have undergone accreditation and continuous education, we cannot overlook the possibilities of mistyping or underreporting. Second, only hospitals participating in the project reported to the TCDB. These hospitals may exhibit a higher level of specialization than hospitals that are not in the project. If we combined reporting and non-reporting hospitals, the differences in adherence would likely be increased. Third, we focused more on the effectiveness aspect of quality, while other aims for improvement such as safety and patient-centered care were not considered simultaneously [42]. Finally, as we mentioned earlier, the evidence level for colorectal cancer doesn't parallel that available for breast cancer which limits the study's generalizability. Research studies such as randomized clinical trials of cancer care should be encouraged in the future to enhance the evidence level of research on colorectal cancer care.


Standardized performance measurement is an emerging methodology for comparing health care quality among different facilities. To the best of our knowledge, this is the first study on colorectal cancer care quality based on core measures with a nationwide database. Developing core measures for cancer care is the first step to achieving standardized measures for external monitoring, as well as for providing feedback and serving as benchmarks for cancer care quality improvement. By applying a core measures approach more comprehensively in this fashion, hopefully we can expect to standardize clinicians' daily practice as well as reduce care disparities and improve outcomes for the general population. Future research may capture opportunities to improve cancer care quality by including perspectives from patients, physicians and systems as well as the relationships between core measures and outcomes.


  1. 1.

    Expert panel members and affiliations:

  • MD, PhD, Professor, Department of Medicine, National Taiwan University Hospital;

  • MD, PhD, Associate Professor, Department of Surgery, National Taiwan University Hospital;

  • MD, PhD, Department of Radiation Oncology, National Taiwan University Hospital;

  • MD, PhD, Professor, Department of Colorectal Surgery, Taipei Veteran General Hospital;

  • MD, Department of Radiation Oncology, Taipei Veteran General Hospital;

  • MD, Director, Department of Colorectal Surgery, Taichung Veteran General Hospital;

  • MD, Professor, Director, Department of General Surgery, Tri- Service General Hospital;

  • MD, Associate Professor, Director, Department of Radiology, Tri- Service General Hospital;

  • MD, Director, Department of Oncology, Taipei Mackay Memorial Hospital;

  • MD, PhD, Associate Professor, Director, Department of Oncology, Far Eastern Memorial Hospital;

  • MD, Director, Department of Internal Medicine, Taipei Cathy General Hospital;

  • MD, Director, Department of Pathology, Taipei Cathy General Hospital;

  • MD, PhD, Department of oncology, Chung-Gung Memorial Hospital;

  • MD, PhD, Professor, Director, Department of Colorectal Surgery, Chung-Gung Memorial Hospital;

  • MD, PhD, Professor, Director, Department of Medicine, Chung-Gung Memorial Hospital;

  • MD, PhD, Associate Professor, Director, Department of Oncology, Chung-Gung Memorial Hospital;

  • MD, PhD, Professor, Director, Department of Surgery, National Chen-Gung University Hospital;

  • MD, PhD, Professor, Director, Department of Pathology, National Chen-Gung University Hospital;

  • MD, Director, Department of Internal Medicine, Kaohsiung Medical University Hospital;

  • MD, PhD, Director, Department of Oncology, Chung-Gung Memorial Hospital.

  1. 2.

    Hospital enrolled in the context including:

  • National Taiwan University Hospital;

  • National Chen-Gung University Hospital;

  • Taipei Veteran General Hospital;

  • Taichung Veteran General Hospital;

  • Kaohsiung Veteran General Hospital;

  • Taipei Mackay Memorial Hospital;

  • Tri- Service General Hospital;

  • Koo Foundation Sun Yet-Sen Cancer Center;

  • Buddhist Tzu Chi General Hospital, Da-Lin Branch;

  • Taipei Cathy General Hospital;

  • Shin Kong Wu Ho-Su Memorial Hospital;

  • Chung-Gung Memorial Hospital;

  • Chung-Gung Memorial Hospital, Kaohsiung Branch;

  • China Medical University Hospital, Taiwan;

  • Changhua Christian Hospital;

  • Chia-Yi Christian Hospital;

  • Chi Mei Medical Center;

  • Kaohsiung Medical University Hospital.