Introduction

“If you can not measure it, you can not improve it” (Lord Kelvin)

A clinical audit is a process which assesses whether healthcare is meeting standards, with the capacity to reduce care disparities as well as being a method to cut unnecessary costs. Where indicated, changes are implemented at an individual, team, or service level, and further monitoring is used to confirm improvement in healthcare delivery [1]. Reliable data, meticulous measurement, and systematic feedback to participants about their performances are the three pillars at the base of an effective clinical audit. Only good data can bring understanding of what does and does not improve the quality of healthcare, and a continuous assessment process allows for concrete advancement (Fig. 1).

Fig. 1
figure 1

General and ESCA clinical auditing process

Full size image

The so-called “Hawthorne effect” (the awareness of being monitored positively influences the behavior of monitored individuals) also helps improve the process, as already widely demonstrated [2,3,4,5,6].

Medical literature offers a vast number of international clinical audits focused on colorectal cancer surgery [7,8,9,10]. The Dutch Colorectal Audit (DCRA) probably represents one of the most effective initiatives aiming at improving surgical quality outcomes [7]. DCRA started in 2009 and after just 8 years of activity more than 70,000 patients were included, showing a dramatic decrease in postoperative mortality from 3.4 to 1.8% for colon cancer and from 2.3 to 1% for rectal cancer [10]. Moreover, a significant reduction in costs for the entire healthcare system was observed along with a reduction in complications [11] and an optimization of resources such as the reduction in preoperative radiation therapy for rectal cancer without any impact on oncological outcomes[12].

In 2020, more than 43,000 people in Italy were diagnosed with colorectal cancer (CRC). Of these, approximately 800 of them lived in proximity of the centers taking part in this study [13]. Looking at the administrative data of the Emilia-Romagna region, huge variability was also observed in terms of volume and surgical outcomes, ranging respectively from 36 to 290 procedures per hospital and 2.26% up to 9.45% for 30-day mortality [14].

Following the path of the DCRA experience and given the heterogeneity of surgical outcomes among the hospitals in our region, we decided for the first time in Italy to promote a systematic clinical auditing pilot study focused on CRC surgical care and its possible benefit on outcomes.

Methods

Data governance and ethics

The protocol has been described according to the Standard Protocol Items: Recommendation for the Investigational Trials (SPIRIT) checklist [15] (Appendix). The ethical committee approved the project for each of the centers taking part according to local regulations, and it has been registered on the clinicaltrials.gov platform (Identifier: NCT03982641). The Romagna Ethical Committee (CEROM) approval number is 2278.

Study organization, administration, and governance

The Emilia-Romagna Surgical Colorectal Audit (ESCA) is a multicenter, retro-prospective, observational non-profit study promoted by the IRCCS Istituto Romagnolo per lo Studio dei Tumori (IRST) “Dino Amadori” in Meldola and the Local Health Agency of Romagna (AUSL Romagna).

Members of the surgical units participating in the study contributed to its design gave feedback and reviewed the study protocol. ESCA is overseen by a study steering committee composed of surgeons, epidemiologists, statisticians, IT specialists, and healthcare administrators.

ESCA received a financial support from Johnson & Johnson Medical S.p.a Ethicon Division for data management and monitoring.

The objectives of the study are as follows:

  1. 1.

    To systematically collect data of all patients undergoing surgical treatment for primary CRC from diagnosis up to 6 months after surgery;

  2. 2.

    To evaluate risk adjusted post-operative complications and mortality;

  3. 3.

    To evaluate adherence to the clinical audit activity of each participating center;

  4. 4.

    To evaluate the possible positive effects on outcomes, both clinical and economic, of systematic measurement and monitoring (Hawthorne effect).

Setting and study population

Patients will be enrolled from the 7 hospitals located in the Emilia-Romagna region in Northern Italy (Table 1), during a 54-month study period. The recruitment target is around 800 patients per year on the basis of demographic data reported in previous years.

Table 1 Participating centers
Full size table

Inclusion criteria

All consecutive patients undergoing surgery for primary CRC between 15 April 2019 and 31 December 2023 will be enrolled in the study. All types of surgical procedures are included, irrespective of setting (urgent/emergent or elective), intent (curative or palliative), and approach (open or minimally invasive). Eligible patients are asked to sign an informed consent form. Cognitive impairment is not considered an exclusion criterion if informed consent is obtained by an appropriate healthcare proxy.

Exclusion criteria

Patients with multiple synchronous primary tumors are excluded from the analysis. Patients who are unwilling to sign an informed consent form are also excluded.

Patient’s withdrawal

Participating subjects have the right to withdraw at any time for any reason; data will be collected until the patient’s withdrawal point.

Study outcomes

The primary objective of this analysis is to assess the frequency of post-operative complications, unplanned re-interventions, re-admissions, and mortality rates (at 30, 90, and 180 days after surgery).

Secondary outcomes include adherence to the clinical audit by each center (intended as the percentage of enrolled eligible patients) along with the study timeframe and possible positive effects on outcomes, both clinical and economic, of systematic measurement and monitoring.

Key performance indicators, case-mix, benchmarking, and feed-back activity

A core set of key performance indicators (KPIs) will be assessed to measure performance and evaluate the quality of colorectal cancer surgery across the participating centers (Table 2). Each KPI will be estimated both as unadjusted and risk adjusted for differences in patients’ characteristics for a fair comparison among the participating hospitals.

Table 2 Overview of the selected key performance indicators
Full size table

Every 12 months, an anonymized report on volume and KPIs will be delivered to each hospital comparing data of all the participating centers. A set of relevant case-mix variables (patients’ frailty, tumor burden, type of surgery) will be also identified in order to produce a reliable comparison of outcomes between hospitals and give each center case-mix adjusted outcomes.

Data collected during the first 18 months (from April 2019 to October 2020) will be analyzed in order to evaluate preliminary results on primary and secondary outcomes. The first preliminary analysis will be used as benchmark to further implement the auditing activity.

Case-mix adjusted data will be presented using funnel plots with a 95% confidence interval (Fig. 2).

Fig. 2
figure 2

Case-mix adjusted data will be presented using funnel plots with a 95% confidence interval

Full size image

Reports of the enrolment rate, obtained by comparing the number of enrolled patients to the hospital discharge cards (HDC), will be sent to each center every 3 months. The aim of this feedback activity is to progressively obtain a 100% enrolment of eligible patients and thus provide “real-world data” that avoid bias related to patient selection.

Data collection and quality control

The data collection and management for this paper are performed using the OpenClinica open-source software for Electronic Data Capture (EDC), version 3.1 (Copyright OpenClinica LLC and collaborators, Waltham, MA, USA, www.OpenClinica.com). Case report forms are filled in for each patient by trained healthcare providers (attendings, residents) or properly trained data managers. Operative and postoperative data are retrieved both from electronic medical charts and administrative databases. Surgical data analysis, including detection of postoperative complications, will be done under the supervision of a staff surgeon.

Quality control and data authenticity will be performed by data managers and clinical research coordinators. Clinical data will be compared to routinely collect administrative information retrieved from regional registries to ensure reliability and completeness and to avoid selection bias. Registration of each patient is automatically linked to the regional administration database, which by law receives notification on all patients deceased in Emilia-Romagna.

Study variables

The full data set is composed of up to 172 possible variables, and it is organized in sections as follows.

Preoperative assessment

Preoperative functional assessment is conducted using the Eastern Collaborative Oncology Group Performance Status (ECOG PS) [16] and the American Society of Anaesthesiology (ASA) score [17, 18]. Patients aged ≥ 70 years are screened for frailty with the Katz Activities of Daily Living (ADL) [19] and the Flemish version of the Triage Risk Screening Tool (fTRST) [20, 21]. Comorbidities are assessed using the age adjusted Charlson Comorbidity Index (CCI) [22, 23] and the presence of malnourishment is evaluated using the nutritional risk score (NRS) [24]. All screening tests are reported in detail in Table 3. Baseline evaluation further includes information on the living conditions before surgery and polypharmacy.

Table 3 Preoperative functional status assessment tools
Full size table

The diagnostic work-up is then assessed including the following items:

  1. 1.

    Number and location of primary cancer;

  2. 2.

    Presence/absence of distant metastases;

  3. 3.

    Date of preoperative endoscopy and pathology report of biopsy;

  4. 4.

    Presence of cancer-related preoperative complications (anemia, colonic obstruction, perforation);

  5. 5.

    Date and type of preoperative imaging studies (CT scan, MRI, PET-CT) as appropriate.

For rectal cancer patients undergoing pelvic MRI and/or neoadjuvant treatments, specific items will be also collected as reported in Table 4.

Table 4 Pelvic MRI features for rectal cancer and type of neoadjuvant treatment
Full size table

Surgery

Type of surgical procedure, regimen, length of surgery, and operative technique is collected. Possible intraoperative complications are identified and specified as well as the need for intraoperative blood transfusions. The radicality of surgery is assessed as well as type of anastomosis including characteristics and technique. If stoma is created, details are reported as appropriate. All surgical items are reported in detail in Table 5.

Table 5 Surgical variables
Full size table

Postoperative course

Need of the post-operative intensive care unit (ICU), length of ICU stay, postoperative LOS, and discharge settings are all registered. Thirty-day postoperative morbidity is collected and classified according to Clavien–Dindo classification [25, 26]. Cumulative burden of postoperative complications is calculated for each patient according to the comprehensive complication index [27]. If re-intervention is needed, the reason for reintervention, operative procedure, approach, and need for postoperative ICU stay are all collected (Table 6).

Table 6 Postoperative course
Full size table

Pathology

The pathology report includes cancer type and grade of differentiation according to the WHO classification [28], the number of retrieved and positive lymph-nodes, and lympho-vascular and perineural invasion. For patients with rectal cancer, additional information includes extra mural vascular invasion (EMVI), evaluation of distal and circumferential margins, TME quality according to the Quirke classification [29, 30], and grade of regression following neoadjuvant treatment according to Ryan/CAP [31]. Final tumor stage is reported according to the 7th edition of the TNM cancer staging system [32] (Table 7).

Table 7 Pathology and postoperative oncological treatment
Full size table

Follow-up

After hospital discharge, any postoperative oncological treatment will be reported (adjuvant chemotherapy and/or radiation therapy). Possible changes in living conditions, considered as a proxy for functional recovery, will be reported. Any emergency department access until postoperative day 30 and the reason for it will be collected as well as the need for re-hospitalization. Mortality at 30, 90, and 180 days postoperatively will be collected, together with the cause of death (Table 8).

Table 8 Follow up
Full size table

Statistical considerations

All the analyses will be performed considering tumor location (colon or rectum) as a stratification factor. For continuous variables, the arithmetic mean and standard deviation (SD), as well as the median value and minimum–maximum, will be presented. Absolute frequencies together with the percentage relative frequencies will be reported while summarizing qualitative variables.

To evaluate the performance of surgical activity, absolute numbers and relative percentages of each performance indicator reported will be calculated. The mortality rate will be defined as the number of patients who died within 30, 90, and 180 days after surgery. Graphical representation will also be used: funnel plots will be displayed for each KPI, both in unadjusted and adjusted versions.

Appropriate statistical models (i.e., mixed models, logistical regression, multilevel models) will be developed to evaluate the relationship among analyzed indicators and potential explanatory factors as well as for standardization/adjustment purposes. Main covariates will include age, gender, ASA score, CCI, surgery setting (urgent vs. elective), and ECOG.

Hospitals will be used as random effects to account for the presence of possible variability among hospitals. Other exploratory subgroup analyses will be performed.

Furthermore, to assess the effects of continuous monitoring and benchmarking on clinical outcomes, a before-and-after approach will be used. Specifically, assuming the time from the first subject surgery until the release of the first benchmarking report as the reference period (baseline, i.e., “before” period), the analysis will evaluate any significant changes occurred afterwards. Any changes on the outcomes will be, therefore, attributed to the benchmarking effect.

Missing values will be replaced and estimated using multiple imputations. A two-sided 95% confidence interval (95% CI) will be reported as appropriate.

Statistical analysis will be performed using R statistical software (v. 4.0.6) — www.r-project.org.

Discussion

Surgery is the main treatment modality for stages I–III colorectal cancer and frequently represents the most effective choice even in a palliative setting. Despite massive improvements in perioperative care and techniques, colorectal cancer surgery is still associated with a significant burden of postoperative complications which result in greater healthcare costs and severe functional sequalae for patients. If the value of healthcare is maximizing quality care at minimal cost [33], the large-scale participation in an audit, which constantly measure the care quality and the resources used associated for a benchmark feedback, represents a unique opportunity to significantly improve healthcare and limit expenditure.

Administrative data have already revealed their limitations when used to evaluate quality of care and can lead to misinterpretation when used to measure composite postoperative outcomes of complex and/or frail patients [34, 35]. However, together with mortality registries, they are essential for quality check control (enrollment rate) and completeness of data entry for specific items (re-admission rate, emergency department admission after discharge): for this reason, they were integrated into the entire ESCA auditing process.

Clinical data are more difficult and expensive to collect than administrative data, but these challenges are far outweighed by the opportunity clinical data can create in obtaining reliable information on the entire clinical process, ultimately improving quality and reducing hospital costs, as has been demonstrated by previously validated large national clinical audits such as NSQuIP [36] and DCRA [10]. The Dutch experience, thanks to the inclusion of the entire colorectal surgery population, represents to date, the most meaningful one given its “real-world” nature without selection bias and with risk-adjusted outcome data.

ESCA is an initiative which follows one of the main recommendations of the European Cancer Care Organization (ECCO) — Essential Requirements for Quality Cancer Care (ERQCC). The ERQCC recommended that clinical and process outcome data should be systematically measured and collected to give high quality care to patients [37].

With the paramount of a concrete enhancement of postoperative outcomes and a reduction of costs, ESCA aims to investigate, for the first time in Italy, the impact of systematic clinical auditing and feedback in the field of colorectal cancer surgery among a large population representative of a real-world population. Key performance indicators based on evidence-based guidelines, web-based registration of clinical data made by physicians integrated with administrative data and continuous feedback on the enrolment, and risk-adjusted outcomes are the critical elements of the study, which will provide strong and reliable data to measure and improve quality of colorectal cancer surgical care.

Future challenges will be to enhance this project at regional or national level and will use our experience to set regional and national quality standards.