Introduction

Non-Hodgkin’s lymphoma (NHL) affects over 300,000 people worldwide each year [1]. NHL is listed in the top 10 of most common cancers, with an estimated number of 69,000 new cases in the USA in 2013 [2] and responsible for 3 % of all cancer deaths in the USA [3]. In the Netherlands, the incidence of NHL is as high as 4000 newly diagnosed patients per year. With an expanding and ageing population, these figures are expected to increase. Well-organized and well-carried-out diagnostics and treatments are essential to help patients in the best possible way. Treatment of NHL is highly dependent on the subtype, stage, and aggressiveness of the tumor. For the most common subtypes, diffuse large B-cell lymphoma (DLBCL) and follicular lymphoma (FL), chemotherapy combined with immunotherapy (i.e., monoclonal antibody rituximab) has improved patient outcomes remarkably [4, 5]. Despite these improvements, the 5-year relative survival rate for these patients is only 50–75 % [6].

To deliver high-quality care to patients diagnosed with NHL, multidisciplinary evidence-based guidelines are developed to help professionals in their choices about diagnostics and therapy [7–10]. Quality of care can be measured using quality indicators, which are “measurable elements of practice performance for which there is evidence or consensus that they can be used to assess the quality of care” [11]. From previous research, it is known that discrepancies exist between NHL care delivered in daily practice and the (quality of) care recommended in guidelines [12–14]. Gaining insight into quality of delivered care is an upcoming phenomenon in healthcare; hospitals and other healthcare facilities are exposed to (external) auditing and benchmarking, often using quality indicators [15–17]. Especially healthcare insurances, policy makers (government) and patient organizations are important actors. The USA and UK have both introduced public reporting as a tool to improve the quality of care. National institutes as the National Quality Forum and the Dr Foster Health focus on delivering healthcare information to the public, for example, by providing a hospital guide to the public including information on waiting times, length of stay, and mortality rates for several surgical procedures [18].

In the Netherlands, a national initiative of transparency in hospital care started in 2008, raised and funded by the government (Visible Care program). The aim of this initiative was to increase transparency by providing insight into the quality of healthcare and enabling healthcare providers to report on actually applied diagnostics and delivered therapy and follow-up. Quality indicators were developed and measured for over 100 disease entities, including malignant lymphoma.

This study outlines the differences in quality of delivered NHL care in the Netherlands between 2007 and 2011 and to what extend this was influenced by the Visible Care program.

Materials and methods

Study design and population

Data from two observational studies were used to investigate the natural course of delivered quality of NHL care and the influence of the Visible Care program. First, Wennekes et al. developed quality indicators and measured their performance in 22 Dutch hospitals by including 348 patients diagnosed with NHL between 2006 and 2007 [14]. The quality indicators were derived from evidence-based guidelines and developed by an expert panel using the systematic RAND-modified Delphi method [19]. The main goal was to provide insight into guideline adherence for the most important processes and structures of NHL care. Second, baseline measurements of the Perineal Assessment and Repair Longitudinal (PEARL) study [20], a cluster randomized controlled trial (cRCT), also assessed indicator performance in 19 Dutch hospitals and randomly included 423 patients diagnosed with NHL between 2010 and 2011. Patients eligible for both studies were defined as patients diagnosed with a mature B-, T-, or NK-cell neoplasm and older than 18 years at diagnosis. Patients with multiple myeloma or cutaneous lymphoma or presenting with chronic leukemia were excluded.

Data collection

For the PEARL study, trained registration employees collected data from medical records for the quality indicators. A digital registration form was accomplished in collaboration with the Comprehensive Cancer Centre the Netherlands (CCC). Selection of patients took place using the cancer registry, which is part of the CCC. The CCC used the cancer registry to make a list of potentially eligible patients in the participating hospitals. Patients were randomly listed, after which the first 30 patients were selected and data was collected for 20–25 patients per hospital. This cancer registry is based on the pathology coding system of the World Health Organization (WHO), and patients with mature B-, T-, and NK-cell neoplasms were selected for inclusion.

Patient characteristics (gender, age, hospital region, patient preferences, comorbidities) and tumor-specific information (morphology, stage, extra nodal disease, previous malignancies, and lactate dehydrogenase (LD) values) were collected. Furthermore, dates and data about diagnosis (pathology, imaging techniques, blood counts), treatment of NHL (type of therapy, response), and multidisciplinary consultation were assessed. This information was used to determine the indicator performance scores: the percentage of patients who received care as recommended in the guidelines. If the indicator scores where <90 %, improvement potential was considered to be present [21, 22]. Quality indicators for the domain diagnosis and staging, treatment and follow-up, and organization and coordination were included in the indicator set (see Table 2). The data collection method for the PEARL study (2011 dataset) is in line with the method described by Wennekes et al. [14] (2007 dataset).

The national transparency initiative

In the context of the Visible Care program, a national transparency initiative to provide insight into the quality of hospital care, quality indicators for malignant lymphoma were developed in 2010 [23]. The purpose was to develop quality indicators which represent the most important parts of NHL care and could be easily measured by the hospitals themselves. All disciplines involved in NHL care were asked to provide delegates to participate in the development of the indicator set. The final indicator set included indicators about Ann Arbor staging and International Prognostic Index (IPI) scoring, performing all required staging techniques such as bone marrow biopsy and CT scans, assessing therapy response, arranging multidisciplinary consultations, and providing all diagnostic results within 4 weeks. The indicator set as developed by Wennekes et al. [14] was used as one of the references during the development of this Visible Care indicator set. The implementation of the quality indicators was mandatory, since the program was part of new national policy. Each year, hospitals were asked to provide the malignant lymphoma indicator results about the previous year. Yearly feedback was given to the hospitals by providing anonymous results of all hospitals outlined by the hospitals’ own results. All anonymous results were made publically available through the website www.zichtbarezorg.nl, with the intention that these feedback methods would lead to higher quality of overall NHL care. It was the hospitals’ own responsibility in which way the yearly audit was performed and the feedback was processed.

Wennekes et al. [14] collected data of hospital care before the introduction of the Visible Care program, and we repeated this assessment after the first 2 years of the program. In this way, it was possible to provide insight into the proposed positive effect of the national transparency initiative on the quality of delivered medical care.

Statistical analyses

Descriptive statistics were performed to get insight into the patient characteristics of the two study populations. Quality indicator scores were assessed for both datasets (2007 and 2011) and expressed in percentages to describe current practice. Univariate analyses (Χ 2 tests) were used to test whether differences in patient characteristics and quality indicator scores were present between 2007 and 2011.

The relation between patient and tumor characteristics (e.g., gender, previous malignancies, and LD value) and differences in quality indicator scores between 2007 and 2011 was studied using univariate logistic regression analyses (P < 0.05). Factors often mentioned as reasons for non-adherence were also included: Charlson index, performance status, Ann Arbor stage, patient preferences, and type of lymphoma. Clinical relevant determinants that influenced quality indicator scores in a univariate setting were tested as a group using multivariate, multilevel logistic regression analyses (odds ratios). In case of intercorrelations between characteristics (>0.4), only one of the two was included in multivariate analyses. A multilevel model was used to account for the nested structure of the data, with individual patients nested within hospitals. The intraclass correlation coefficient (ICC) was calculated for each quality indicator to get insight into the cluster effect of the hospitals. ICC values between 0 and 0.40 are found in other secondary care research [24].

Reliability of the data collection was assessed with a duplicate registration of 30 records by two registration employees in three hospitals. The reliability of the indicator scores was calculated with a Kappa value, a statistical measure for interobserver agreement corrected for chance [25]. A missing values analysis (t test) was performed to explore if more than 5 % of the data was missing and if missing values were missing completely at random (Little’s test).

All statistical analyses, except multilevel analyses, were performed using IBM SPSS Statistics for Windows, version 20.0 (IMB Corp, Armonk, NY, USA). Multilevel analyses were performed using SAS software system for Windows, version 9.2 (SAS Institute Inc, Cary, NC, USA).

Results

Study population

Table 1 shows the characteristics of 348 patients of the 2007 dataset and 423 patients of the 2011 dataset. There were no significant differences in gender, age, prevalence of NHL subtypes, aggressiveness of the tumor, Charlson index, patient preferences, and hospital region between the two study populations. In 2011, more patients were included with previous malignancies, Ann Arbor stages III–IV, and normal LD values, compared to 2007.

Table 1 Patient characteristics in 2007 and 2011
Full size table

Quality indicator scores

Table 2 describes the 20 quality indicator scores, corrected for case-mix (i.e., gender, Ann Arbor stage, aggressiveness, previous malignancies, LD value, Charlson index, and hospital region) and the nested structure of the data (multilevel analysis). The quality indicators were divided into three domains: diagnosis and staging, treatment and follow-up, and organization and coordination of care.

Table 2 Indicator scores for quality of delivered NHL care in 2007 and 2011
Full size table

Diagnosis and staging

In this domain, improvement is seen for two of the seven quality indicators. Indicator scores for execution of all recommended staging techniques (QI4) improved from 26 to 43 % (OR = 1.902 (1.268–2.852)), and assessment of the International Prognostic Index (QI5) improved from 21 to 43 % (OR = 2.883 (1.675–4.961)). Two indicators scored significantly lower in 2011 compared to 2007: diagnosis based on morphology and immune phenotype decreased from 99 to 96 % (QI3: OR = 0.190 (0.049–0.744)) and examination of blood counts decreased from 90 to 82 % (QI7: OR = 0.576 (0.340–0.976)).

Treatment and follow-up

No significant differences are seen for the six quality indicators in this domain.

Organization and coordination of care

In this domain, only one of the seven quality indicator scores significantly improved (QI17): more patients (41 %) were discussed in multidisciplinary consultations in 2011 compared to 2007 (23 %) (OR = 3.360 (2.007–5.624)).

Influence of the national transparency initiative

The six quality indicators used as initial concept for the Visible Care quality indicators are underlined in Table 2. Significantly increased quality indicator scores were found for three of the six indicators incorporated in the transparency initiative: the use of all staging techniques increased from 26 to 43 %, the assessment of the IPI from 21 to 43 %, and patients discussed in multidisciplinary consultations from 23 to 41 % (QI4, QI5, and QI17, respectively).

The other three quality indicators included in the transparency initiative did not change significantly: 80 % of the patients was staged according to the Ann Arbor classification (QI2), 60 % had a correct evaluation after chemotherapy (QI11), and the maximum diagnostic period of 4 weeks was realized in about half of the patients (QI19). Two of the 14 quality indicators that were not included in the Visible Care program showed significantly decreased scores (QI3 and QI7).

Statistics

The mean ICC was 0.14 (range 0.005–0.33), which was calculated for all quality indicators to account for clustering. The reproducibility of the data collected for the indicator scores in 2011 was good: the average kappa value was 0.8 (range 0.4–1.0; 87 % ≥0.6). This indicates an overall good agreement between the registration employees. For the 2007 dataset, reproducibility was also good [14]. Missing value analysis showed overall less than 5 % missing data in the overall dataset (2007 and 2011). The factor “performance status” was excluded from analysis because it was missing for most of the cases (80 %). Quality indicator 13 had 10 % missing values; however, these were missing completely at random (MCAR, P = 0.771).

Discussion

In this study, we provided insight into the differences in quality of delivered NHL care in the Netherlands between 2007 and 2011, taking into account the Visible Care program. The data indicated that quality of delivered NHL care improved at a few points, but also showed a greater need for improvement for several quality indicators. Indicator scores that decreased significantly included assessment of immune phenotype for diagnosis and full examination of blood counts. However, significant improvement was shown for execution of all recommended staging techniques, assessment of the IPI, and multidisciplinary discussion of patients.

Quality indicators included in the Visible Care program concerning malignant lymphoma were developed and published in 2010. The main goal of this program was to increase transparency of the Dutch hospital care, for which publically available data were required. This could have been a major incentive for hospitals to meet the quality indicators and thus improve their NHL care (if necessary). It is noticeable that the quality indicators with increased scores (QI4, QI5, and QI17) between 2007 and 2011 were all included in the national initiative, which implies a positive effect of the program. However, other (local) initiatives or international study results were not taken into account in our analysis. For example, the added value of discussing all oncology patients during multidisciplinary consultations has been an increasing point of interest the past years, which could have influenced our results.

Quality indicator scores that significantly decreased between the two measurements included items about pathology diagnosis based on immune phenotype (QI3) and examination of leukocyte differentiation during blood analyses (QI7). Despite these decreases, the scores were still between 80 and 100 %, which indicates good performance and only a small improvement potential. Improvement potential is often assigned to indicator scores below 90 % [21, 22], since higher scores might not be feasible due to case-mix factors as high age, many comorbidities, and patient preferences. It can therefore be argued if the significantly decreased scores seen for the two quality indicators are clinically relevant.

Between 2007 and 2011, there were no large changes in guideline recommendations for NHL care in the Netherlands, besides the addition of a recommendation to use fluorodeoxyglucose-positron emission tomography (FDG-PET) scans for determining therapy response [26, 27]. Using CT scanning, location and size in millimeters should be documented, which is not necessary when a FDG-PET scan is done, since it is based on color intensity instead of size. This could possibly explain the decreasing trend observed in the quality indicator concerning lesions documented in radiology reports after therapy (QI10 67 to 58 %, not significant). Based on our datasets, we determined that FDG-PET (CT-)scans were used more often for evaluation after chemotherapy in 2011 (63 %) compared to 2007 (29 %, P < 0.001).

Another decreasing trend was seen providing all diagnostic results within 4 weeks after the first visit to the hospital (QI19 56 to 49 %, not significant). A factor that may play a role is referral of patients between and within hospitals, which increased from 51 % in 2007 to 64 % in 2011 (P < 0.001). A possible explanation for this might be the increasing burden on the healthcare system, caused by the growing needs of patients. Although this decreasing trend is not significant, and therefore subject to chance, providing all diagnostic results within 4 weeks might be worth arguing because of its clinical importance concerning prognosis and start of treatment for patients diagnosed with NHL. We believe that longer waiting times could result into poorer prognosis for patients, which is already known for patients with head and neck cancer [28]. Furthermore, short waiting times are valuable for the level of patient centeredness of hospitals, of which the importance has grown considerably.

To our knowledge, this is the first study that focused on providing insight into the differences in time in quality of delivered NHL care and evaluated the effect of a national transparency initiative. Besides the present study and Wennekes et al.’s [14], only a few other studies conducted research on quality of NHL care, focusing on parts of the care process such as staging and treatment [12, 13, 29], or follow-up care [30]. In line with our study, they showed that the quality of NHL care is not yet optimal, compared to the recommended care as described in evidence-based guidelines.

Much research has been conducted concerning public reporting, with widely varying results. However, the effect of performance data on improvement of quality of care is less often studied, and if so, mostly focuses on mortality rates [31]. A study by Lamb et al. [32] explored the 5-year impact of voluntary public reporting on physician groups and showed improvement for ambulatory care measures. Our results are in accordance with these studies; however, we were able to provide insight into the quality of care for several measures not included in the public reporting as well. This provided the unique opportunity to examine the effect of public reporting, as done within the Visible Care program, on total quality of NHL care.

Data collection for observational studies included was performed by independent, trained registration employees. In daily practice or public reporting programs, professionals themselves are often responsible for the data collection, which might introduce bias as they are involved as stakeholders. Our data collection showed also high reproducibility between the registration employees, indicating good competency and reliable data collection.

There are also some limitations to this study. Firstly, the hospitals included for data collection in 2011 are not completely identical to those included in 2007: 15 hospitals participated in both measurements, 11 hospitals either in 2007 or 2011. However, no distinct differences were found in indicator scores when the non-overlapping hospitals were left out of the analyses. Together with a relatively large study population (N = 771), we believe that both datasets are representative for hospitals in the Netherlands.

Secondly, we did not have insight into possible local improvement initiatives of the participating hospitals between 2007 and 2011, which makes it difficult to link the Visible Care program as direct and only attribute to the changes seen in some quality indicators of NHL care. One influencing factor could be that high room for improvement rates were seen for four of the six quality indicators included in this program (<30 % scores, 2007 dataset). It can be argued that, in general, quality indicators with low scores are easier to improve than those with higher scores. Hospitals, participated in the 2007 study, received general feedback on their performance as a group and as hospital region, which might have triggered some local improvement initiatives. However, improving NHL care was not the aim of the 2007 study, and no concrete improvement initiatives are known based on the provided feedback. This result is supported by a Cochrane review, which showed that the effect of audit and feedback on improvement is usually small [33].

In summary, during the 4-year period between 2007 and 2011, we found little improvement in overall quality of NHL care. Improvement potential as found in 2007 remained for all quality indicators in 2011, unless some significant increased indicator scores. Based on our results, we can conclude that a national initiative as the Visible Care program might not have enough power to improve hospital care for NHL in general, but might provide a first step towards a more improvement-oriented hospital care. Improvement strategies tailored to the suboptimal quality of NHL care might be a rational step to develop and test in a randomized setting.