Introduction

Lung cancer, with 2.2 million new diagnoses and 1.8 million deaths per year worldwide, is the leading cause of cancer mortality, responsible for 18% of all cancer deaths [1]. In recent decades, advances in molecular biology, targeted therapies, and new treatments such as immunotherapy have increased lung cancer survival [2, 3]. To ensure quality care, we need to know how we treat our patients and measure how advances in research translate into better outcomes for the general population [4].

In recent years, healthcare systems have undergone rapid digital transformation, resulting in vast amounts of data emerging from the patient's medical care journey [5]. As a result, there is a growing interest in using real-world data to address relevant clinical and health policy questions that cannot be answered by clinical trials, including the assessment of quality of care [6].

The advancement of oncology has outpaced the slow development of process and outcome assessment [7] and the challenge of integrating quality assessment into health systems lies not so much in defining a set of indicators as in developing timely measurement tools and articulating them towards a quality improvement system. One solution to reduce administrative burden and ensure feasible assessment is automated extraction of data from existing sources, such as building measurement systems through feedback from the electronic patient record (EPR) [8, 9].

We aim to measure the quality of care provided to patients with non-small cell lung cancer (NSCLC) using the data contained in the hospital’s information systems, while establishing a quality improvement monitoring system. Applying data science methods, we set up an analysis algorithm, based on recommendations from clinical guidelines, to integrate activity and administrative data extracted from the EPR together with clinical data from a lung cancer registry in order to reconstruct the therapeutic trajectory and calculate quality indicators (QIs) that cover the overall care.

Methods

Design

This retrospective observational cohort study that included all patients with NSCLC treated by the Medical Oncology department at Puerta de Hierro University Hospital (HUPH), with a reference population of 500,000 inhabitants, between January 1, 2016 and December 31, 2020. The final follow-up date was June 30, 2021. The study protocol was approved by the HUPH Institutional Ethics Board.

Selection of quality indicators

A literature review of “quality indicators in lung cáncer” was conducted based on a PubMed search, from 2005 to 2023. An evaluation of clinical guidelines was performed at NSCLC. Previously published recommended indicator characteristics were considered, including scientific evidence, development method, feasibility of calculation, and ability to discriminate differences [4, 10]. We selected QIs that correlated with recommendations from clinical practice guidelines, as previously described in NSCLC [8, 11,12,13] and calculable with the available data. A total of 34 QIs were selected to assess quality of care: 17 process indicators, 6 process indicators assessing the timeliness of care, and 11 outcome indicators.

Data sources and calculation of quality indicators

Patients were identified from the HUPH tumor registry. Data obtained from the tumor registry were cross-referenced, using the hospital identification number, with a cancer repository created by the Medical Oncology department reviewing individual patient records, together with administrative data extracted from the EPR (Supplementary Material), linking all the available information of each patient and resulting in a new database that included demographic variables (age, gender), clinical (smoking history, CHARLSON comorbidity score, Eastern Cooperative Oncology Group score (ECOG)), diagnostic and pathological variables (date of diagnosis, histology, stage, molecular studies), as well as therapeutic information (surgical treatments, radiotherapy and medical treatments received with their related dates) and health status at the last follow-up. Healthcare data was obtained through Oracle Business Intelligence Discoverer 10 g Version 2 (10.1.2.1). Stata v.16 was used for data preparation, cross-referencing and depuration.

The Cross-Industry Standard Process for Data Mining (CRISP-DM) methodology was applied to achieve our goal [14], adapting this technique to analyze integrated clinical and healthcare usage data. Guideline-based decision trees were used to model and describe the patient's journey in healthcare based on the available data, while performing the calculation of the indicators.

Statistical analysis

The categorical variables are described by absolute and relative frequencies, and for the numerical variables, we used the mean (standard deviation) or median (percentiles 25 and 75). Comparisons between categorical variables were performed the Pearson chi-squared test. A p value < 0.05 was considered statistically significant.

In the survival analyses, the Kaplan–Meier method and the log-rank test were used to compare the survival curves, with a p-value of < 0.05 considered significant. For multivariable analysis, a Cox proportional hazards regression model was used to evaluate the association of potential prognostic variables. All potential prognostic variables were included in the model, and then a backward selection strategy was carried out using the likelihood ratio test and a significance level of 0.05. Statistical analysis was performed using Stata v.16.

Results

Patient characteristics

Between January 1, 2016 and December 31, 2020, 821 lung cancer patients were identified in the HUPH tumor registry; 650 patients with NSCLC were consecutively included; the median follow-up was 37.6 months (95% CI: 33.4–41). The median age was 67 years (interquartile range (IQR): 60–74), 66.5% of patients were men; 89.6% were smokers or former smokers. The distribution of histological subtypes was as follows: 61.1% adenocarcinoma, 28% squamous cell carcinoma, 2.8% large cell carcinoma, and 8.1% other histology or carcinoma not otherwise specified. The characteristics of the population are described in Table 1. The treatments administered during the study period for each stage, also considering the presence of targeted mutations in advanced disease, are described in Table 2.

Table 1 Population characteristics
Full size table
Table 2 Therapeutic strategies used in nsclc by stage
Full size table

Quality of care indicators

Analysis algorithm configuration

An algorithm was constructed according to the recommendations of clinical guidelines to reconstruct the trajectory of care for the population of patients with NSCLC from EPR dataset. Illustrated as a decision tree in Fig. 1, it defines therapeutic pathways according to the value of key variables: stage, presence of targeted mutations, and treatments performed. The applied algorithm establishes the relationship between the data that allows the automated calculation of QIs from the hospital's information systems, Table 3, identifying process, timelines of care and result indicators to fully cover the care provided.

  • Calculated process indicators assess adherence to clinical guideline recommendations, Fig. 2. 91.7% of patients in stages I and II undergone surgery, 97.6% received treatment with radical intent. In addition, 71.4% of resected stage II patients received adjuvant chemotherapy. In stage III, 90% received treatment with curative intent, 87.5% received systemic treatment, and 81.3% completed a multimodal strategy. Regarding stage IIIA, 63.8% were treated with surgery. In advanced disease, 73.4% of non-mutated patients and 96.9% of mutated patients received systemic therapy. 24.5% participated in a clinical trial. The incorporation of immunotherapy into clinical practice in stage IV without targeted mutation was also evaluated, with a progressive increase in its use from 40.3% in 2016 to 71.2% in 2020. As part of process evaluation, it was possible to evaluate the timeliness of the correct care previously described, Table 3.

  • Relapse-free survival (RFS), time to treatment failure (TTF) and overall survival (OS) in patients treated according to stage were analyzed as outcome indicators, Table 3 and Fig. 3, making it possible to know the outcome in the general population of the application of the treatments corresponding to each indication. In stage III patients treated with curative intent, the median OS was 52.8 months (95% CI: 30.1—not reached (NR)); In the case of stage IV systemic therapy, median OS was 34.9 months (95% CI: 26–50.1) for mutated patients and 13.6 months (95% CI: 10.5–17.9) for patients without mutation. In addition, the subgroup of 148 stage IV patients without targeted mutation treated with immunotherapy had a median OS of 25.7 months (95% CI: 18.9–32.7). Median OS was not reached in stages I-II or IIIA. An exploratory multivariate analysis was performed in the survival analysis including the variables: stage, histology, ECOG, sex and smoking history, observing an association between the risk of death and ECOG in the different indicators, as well as an increased risk of death for stage IVB compared to stage IVA and the histological subtype squamous cell carcinoma compared to adenocarcinoma in patients without targeted mutation.

  • Diagnostic process indicators. To ensure the correct classification of each patient to their corresponding therapeutic route by the algorithm used, process indicators related to diagnosis were included, observing that 100% of the patients obtained a histological diagnosis, in 91.9% the histological subtype was identified and 100% of the patients had a documented tumor stage. In stage IV, molecular profiling (gene testing for EGFR, ALK, ROS1 and BRAF) was performed in 94.3% of patients with adenocarcinoma, non-specific carcinoma, or nonsmoking squamous cell carcinoma. In patients treated in stage IV without mutation, PDL-1 was determined in 67.7%, showing a progressive increase from 25.8% in 2016 to 96.2% in 2020.

Fig. 1
figure 1

Therapeutic journey of the NSCLC cohort reconstructed using decision tres. BSC: best supportive care; Ch-I-Th: chemotherapy + immunotherapy; Ch-T: chemotherapy; I-Th: immunotherapy; RT/ChRT: radiotherapy/chemotherapy + radiotherapy; Syst Th: systemic therapy; TR: treatment,

Full size image
Table 3 Quality indicators for NSCLC
Full size table
Fig. 2
figure 2

Selected process indicators for patient care in NSCLC. NSCLC: non small cell llung cancer; mut: mutated

Full size image
Fig. 3
figure 3

Outcome indicators for NSCLC. Kaplan–Meier curves of overall survival for patients with resected stage I and II (A), resected stage II receiving chemotherapy (B), stage III treated with curative intention, by stage (C), stage III who received multimodal treatment, by stage (D), mutated stage IV NSCLC treated with targeted therapy (E), stage IV NSCLC patients without driver mutation receiving systemic treatment, by stage (F) and stage IV NSCLC patients without driver mutation receiving immunotherapy, by stage (G)

Full size image

Discussion

Our study demonstrates that it is possible to extract data from the EPR and calculate QIs automatically using an algorithm built based on the recommendations of clinical practice guidelines in NSCLC. The decision tree programmed for the application of data mining in the EPR allows us to identify the therapeutic journey and perform a quality of care assessment in our NSCLC cohort, resulting in a tool to monitor quality improvement.

The process indicators allow us to evaluate adherence to clinical practice guidelines, observing a higher proportion of patients with stages I–II and stage IIIA treated with surgery than reported in other series such as that of Andreano et al., with 55.4%, 48.3% and 35.8% of patients who undergone surgery on stages I, II and III, respectively [11], or 75.2% in stage I and 56.9% in stage II reported by Guerreiro et al [15]. The utilization of systemic treatment in stage IV patients in our study was 73.4% in non-mutated patients and 96.9% in mutated patients, exceeding the 60% target reported for this indicator [15]. It is also possible to evaluate the implementation over time of new therapies such as checkpoint inhibitors for non-mutated patients in stage IV, confirming a progressive increase from 40.3% in 2016 to 71.2% in 2020.

With the outcome indicators, we try to measure the effectiveness of care, the most relevant being survival [12], however, real-life data on this indicator show great variability for NSCLC [13]. In our study, the 3-year survival rate was 86.4% in stages I and II, 54.9% in stage III, and 44.4% and 26.8% in mutated and non-mutated stage IV patients, respectively, higher proportions than those described in The Surveillance, Epidemiology, and End Results (SEER) program at the same follow-up point and for patients diagnosed in the year 2017: 83.3%, 59.5%, 19.9%, for early stages, locally advanced and advanced disease, respectively among adenocarcinoma patients, and 61%, 35.5%, and 11.1% for the same stage groups among squamous cell carcinoma patients [16]. However, our survival assessment does not seek to compare populations, but rather to contribute to the interpretation of global care for NSCLC, allowing us to explain that the differences observed are due to the selection of the population by the treatment received in each subgroup. Our outcome indicator better reflects the effectiveness of the treatments recommended in the clinical guidelines, having previously identified their correct application in the evaluation of processes.

Regarding the diagnostic process, pathological confirmation was performed in 100% of the patients, the histological subtype was identified in 91.9%, the TNM stage was documented in 100% and the relevant molecular study was available in 94.7% of the patients in stage IV. The completeness of these variables, together with the availability of information on health status such as the ECOG and the Charlson comorbidity index, seeks to resolve the lack of information on patient prognosis, a common limitation of real-world studies to evaluate care and outcomes [6]. We observed a higher proportion of compliance with diagnosis-related QIs compared to other studies, such as 81.8% histological confirmation reported by Beck et al [8], or 83.8% in a large series evaluated in Italy [11]. Also in the documentation of the TNM stage as 88.57% of the work of Mazzone et al. [17], or 89% of patients with molecular study observed in the Dutch Lung Cancer Audit [13].

Among QIs, outcome indicators have always attracted greater interest for both patients and health policy makers [7, 18, 19], However, its interpretation is difficult without understanding the processes that lead to its achievement, the latter being more sensitive to differences in the quality of care as they do not depend on individual factors of the patient or on the unfavorable evolution of advanced cancer even if the correct treatment is applied [19]. Improving the OS of patients with NSCLC requires improving all processes related to their diagnosis and treatment, therefore, we propose a set of quality metrics that encompass all NSCLC care, diagnostic processes, therapeutic processes, and outcomes.

Although having references may be useful to propose quality policies, obtaining QIs for comparison with other studies is not the objective of our study. Comparability of treatments and outcomes between studies or hospitals should take into account differences between populations, potential time bias according to the follow-up period of each cohort, or lack of data that may dictate appropriate treatment. In addition to these limitations, we decided not to use target values for the indicators due to the lack of consensus on them and because the recommendations of the guidelines are only applicable to subgroups of patients, while the QIs have often been calculated in heterogeneous populations. Assessing timely care is even more controversial, with studies associating increased survival with shorter time to treatment [12] and others describing the so-called “wait time paradox”, in which shorter time to care was associated with worse outcomes [15]. This phenomenon has been partly justified by the urgency of treatment in more symptomatic patients, or by the delay in completing molecular studies that could be beneficial for patients. Despite the controversy, timely care is one of the dimensions of quality recognized by the Institute of Medicine and considered valuable to patients [20]. Cross-hospital benchmarking could be one of the potential applications of QIs, but should be used with caution when the reasons for the correct exception to each recommendation cannot be assessed [11].

The quality measures described are considered effective in modifying hospital practices and policies, are useful for tracking improvements, and can increase the value of health care [21], but its implementation has historically been hampered by the administrative workload and time required for data collection [9]. In our study, we overcame these limitations using the CRISP-DM methodology, previously applied to the study of healthcare consumption for lung cancer [14], taking advantage of data mining in the use of available data and saving time in their extraction. CRISP-DM includes six phases: business understanding, data understanding, data preparation, modeling, evaluation, and implementation [22], which are cyclically repeated to improve the process. The use of decision trees in data exploration is especially useful for identifying the care received and its sequence, as it allows logical diagrams to be generated to categorize conditions. A path to be followed is defined depending on the value of each variable considered, until it ends up belonging or not to a group [23, 24], with the added value of offering interpretable visual representation. In our study, we configured an analysis algorithm based on a decision tree that defined the therapeutic routes according to the clinical guidelines, which, by interrogating the set of data generated in routine patient care, proves to be a feasible solution for obtaining indicators.

The methodology used not only solves the question of how to measure quality, but also determines which indicators to measure. A variety of techniques have been used in health care quality assessment studies to develop or select indicators: literature review, expert consensus, or Delphi review among them [4, 18, 25,26,27]. In our study, the QIs were selected by matching recommendations from clinical practice guidelines with indicators identified in the literature review, a strategy previously used in NSCLC [8, 11,12,13]. With the use of clinical guidelines, we guarantee the relevance of the selected QIs in the current management of lung cancer. Finally, we assumed as a selection criterion its feasibility of calculation [11, 28], constructing a measurement feedback system using the EPR, previously suggested to solve the slowness and burden of manual data abstraction [8, 9, 29].

The complexity of cancer care requires planning and performance analysis to ensure its alignment with the current state of knowledge and achieve the highest level of quality in the health system [19]. The main implication of our work is to have a tool to respond to this unmet need. Our work proposes a quality assessment procedure, not only for comparative evaluation, but also to monitor continuous quality improvement. As far as we know, this is the first study to evaluate the care provided by a Medical Oncology service to all patients with NSCLC of any stage in Spain. We believe that the use of data automatically collected from the EPR and generated in routine practice allows our methodology to be scalable at the national level.

Our work has several limitations: first, the initial review was carried out by a single oncologist, compared to most studies in which a group of experts participates. We understand that there is enough number of indicators recognized as valid to shorten their discovery and prioritize their applicability. Second, the selection of indicators based on available data may miss important indicators. This should encourage the collection and coding of a minimum dataset needed to monitor care processes, as well as facilitate their automatic acquisition. Finally, real-world data are acquired in an uncontrolled environment, with incorrectly recorded data having been included or resulting in no data at all. Considering this issue, data entry was performed by personnel trained in documentation, consistent with the very low proportion of missing values observed.

Conclusion

To achieve the highest quality in healthcare, it is necessary to have feasible metrics and measurement tools that monitor actions and continuous changes. Our study demonstrates that it is possible, through the application of an algorithm created based on the recommendations of clinical practice guidelines in NSCLC, to use data generated in routine practice to reconstruct the patient trajectory and evaluate the quality of care provided by a Medical Oncology service through 34 QIs.

The digitalization of healthcare has made it possible to access more data than ever before in the history of medicine. Our work proposes a methodology to take advantage of the data contained in hospital information sources, allowing feedback and repeated measurement over time, thus offering a tool to understand compliance with evidence-based recommendations, monitor interventions and, ultimately, establish a system of continuous improvement of the quality of health care.