FormalPara Key Points for Decision Makers

The cost-effectiveness of clinical interventions is often assessed using current care as the comparator, with national guidelines as a proxy.

The use of national guidelines as comparator is inadequate when clinical practice differs from guidelines, or when clinical practice differs between hospitals.

Consideration of clinical practice variation and deviation from the clinical guidelines should be one of the first steps in any CEA.

If important practice variation or deviation from the guidelines exists, hospital-level CEAs should be performed which compare the care that is actually provided in hospitals.

1 Introduction

The cost-effectiveness of clinical interventions (e.g., diagnostic tests, therapies or medicines) is normally assessed using current clinical care as a comparator, with national guidelines as a proxy for current care [1, 2]. However, this comparison with guidelines is inadequate when clinical practice differs significantly from guidelines and is particularly problematic when clinical practice differs between hospitals.

The National Institute for Health and Care Excellence (NICE) in the UK provides guidance through the ‘Guide to the Methods of Technology Appraisal’ in evaluating clinical care strategies in terms of their cost-effectiveness [3]. In addition, the NICE Diagnostic Assessment Programme Manual specifically provides guidance in the evaluation of cost-effectiveness of diagnostic tests and technologies [4]. However, both documents pay little attention to clinical practice variation and its consequences when performing relevant cost-effectiveness analyses (CEAs). In practice, most cost-effectiveness studies do not take into account possible causes and consequences of clinical practice variation [1, 2].

There are many reasons why clinical practice guidelines are not used in daily practice [5]. In some cases, hospitals may wilfully deviate from the guidelines if those guidelines are not be viewed by hospitals as valid. Other hospitals may have no choice but to deviate from guidelines if they are simply impossible to implement in their hospital. In other cases, the guidelines might not be followed due to solvable problems with logistics or financing. Ultimately, various local hospital conditions may cause variation in clinical practice between hospitals, which may result in varying costs and health effects (i.e., quality-adjusted life years [QALYs]) between hospitals and consequently important differences in estimated cost-effectiveness.

A few studies have used large databases to investigate practice variation and the impact it has on costs and effects [1, 2]. However, their approach is different from ours since we aim to illustrate the importance of investigating possible clinical practice variation and deviation from national guidelines, and the need to perform hospital-level CEAs, which incorporate local hospital conditions when important clinical practice variation exists. We specifically focused on diagnostic imaging tests for the assessment of carotid stenosis and criteria for treatment of patients with a recent transient ischemic attack (TIA) or minor ischemic stroke in the Netherlands.

2 Methods

2.1 National Stroke Guidelines

After diagnostic evaluation and treatment in the acute phase, patients with a recent TIA or minor ischemic stroke undergo an assessment of carotid stenosis and subsequent treatment as part of the secondary prevention (i.e., to prevent a future stroke). For the assessment of carotid stenosis, Dutch guidelines recommend duplex ultrasonography (DUS) as the initial diagnostic test and computed tomography angiography (CTA) or magnetic resonance angiography (MRA) as a confirmatory test [6]. The criterion for performing a confirmatory test is moderate (50–69 %) carotid stenosis for men (based on the initial diagnostic test) or severe (70–99 %) carotid stenosis for women [6]. No distinction is made in the guidelines between the hospital’s emergency unit and outpatient clinic. Furthermore, the Dutch guidelines recommend surgery (i.e., carotid endarterectomy) for patients (men and women) with a severe (70–99 %) carotid stenosis and a TIA or minor ischemic stroke in the past 6 months. In addition, a carotid endarterectomy is advised for men with moderate (50–69 %) carotid stenosis and a TIA or minor ischemic stroke in the past 3 months [6].

2.2 Interviews and Questionnaire

Semi-structured interviews were conducted with 16 vascular neurologists in 6 academic and 10 non-academic hospitals throughout the Netherlands. Only one hospital refused to participate (reason: not interested) resulting in a response rate of 94 %. We included the hospitals that are participating in the Plaque at Risk (PARISk) cohort study [7]. In addition, we do not claim generalizability of the sample of hospitals that we used in our study, despite including 18 % (16/89) of all Dutch hospitals. The interviews were conducted either face-to-face or by telephone, by four different interviewers from universities and various university medical centers. In total, eleven out of sixteen (69 %) interviews were conducted face-to-face. All interviews were conducted between May 2012 and January 2013.

During the interviews, we queried vascular neurologists about the type and sequence of diagnostic imaging tests for the assessment of carotid stenosis as used in their hospitals, and criteria for subsequent treatment. In particular, questions were asked about the use of initial and confirmatory diagnostic imaging tests, and criteria for performing confirmatory diagnostic tests. For each hospital, we examined diagnostic practice in both the emergency unit and outpatient clinic because we expected differences in the use of diagnostic tests between these units. In addition, we asked which criteria were used to perform confirmatory diagnostic tests, and which criteria were used to decide for either surgery (i.e., carotid endarterectomy) in combination with medicines and lifestyle modification, or medicines only (e.g., platelet aggregation inhibitors) and lifestyle modification. We also queried the reasons to deviate from the Dutch guidelines.

A questionnaire, including both open- and multiple-choice questions, was used during the interviews (see Electronic Supplementary Material 1 for the questionnaire). The Dutch stroke guidelines served as the basis for the questionnaire [6]. The questionnaire was designed in collaboration with a radiologist and vascular neurologist to guarantee the clinical relevance of the questions. Subsequently, the questionnaire was reviewed by two experts (two vascular neurologists from two different hospitals), and adjusted where necessary. Additional questions were included in the questionnaire and several options were added to the multiple-choice questions on the use of initial and confirmatory tests.

2.3 Analysis

We calculated the percentage of hospitals using the recommended test combination (both initial and confirmatory test), the percentage of hospitals complying with the guidelines regarding criteria for use of a confirmatory test, and the percentage of hospitals complying with recommended criteria for treatment. Hospitals were categorized as compliant with the recommended test combination when an initial DUS and confirmatory CTA or MRA [e.g., time-of-flight-MRA (TOF-MRA) or contrast-enhanced-MRA (CE-MRA)] is used. Hospitals were categorized as non-compliant to the guidelines if they use extra criteria (besides degree of carotid stenosis and gender) regarding the use of a confirmatory test or extra criteria for treatment (besides degree of carotid stenosis, gender, and time since TIA or minor ischemic stroke onset).

2.4 Case Study of Hospital-Level CEAs

A case study was used to illustrate the value of performing hospital-level CEAs when important clinical practice variation exists or when clinical practice differs from guidelines. A 5-year decision analytic model for men with a recent TIA or minor ischemic stroke was used, which incorporated four diagnostic strategies (see Fig. 1). We assumed that patients who tested positive (i.e., patients with a high risk of a recurrent stroke) underwent a carotid endarterectomy while others received medicines only. Based on the performance (i.e., sensitivity and specificity) of the tests, patients were classified into four groups: true positive (TP), false positive (FP), false negative (FN), and true negative (TN). Final health outcomes were dependent on how patients were classified by the tests and the treatment that followed. The final health outcomes consisted of a minor, major, fatal (i.e., death), or no ischemic stroke event. Death from other causes was incorporated in the model by using the life expectancy from the Dutch population [8].

Fig. 1
figure 1

Cost-effectiveness model. An indeterminate test for men means a moderate (50–69 %) carotid stenosis found in men with an initial DUS and a TIA or minor ischemic stroke in the past 3 months. DUS duplex ultrasonography, CTA computed tomography angiography, CE-MRA contrast-enhanced-magnetic resonance angiography, TP true positive, FP false positive, FN false negative, TN true negative

Full size image

First, a base-case CEA was performed in which the unit costs of diagnostic tests were based on the national unit costs from the Dutch Healthcare Authority for 2012 [9] and the average performance of the tests (see Table 1). The national unit costs represent a national average based on all Dutch hospitals. Second, two hospital-level CEAs were performed, which incorporated hospital-specific unit costs from 2012 and performance of diagnostic tests from two hospitals (see Table 1). Since the case study in this paper is an illustration of the association between practice variation and the cost-effectiveness of different test strategies, we chose to include two (out of the 16) hospitals that use two very different test strategies: one hospital that uses the test strategy recommended in the guidelines (initial DUS and confirmatory CTA) and one hospital that does not (initial CTA and no confirmatory test). The actual diagnostic strategies from each of the two hospitals, the guideline-based strategy and other strategies found in clinical practice were compared with each other in the hospital-level CEAs to investigate the most cost-effective strategy for each hospital.

Table 1 Input parameters of the model
Full size table

The sensitivity and specificity of each test in the hospital-level CEAs were adjusted based on self-reported clinician expertise in performing certain tests using the limits of 95 % confidence intervals as reported in the literature [10, 11]. Since clinicians from hospital 1 reported having great expertise in performing a CTA and an average expertise in performing a DUS and CE-MRA, we assumed that the sensitivity and specificity of a CTA performed in that hospital would be higher than average (and set their values at the upper limit of the 95 % confidence interval) and an average performance of DUS and CE-MRA. In contrast, clinicians from hospital 2 reported having great expertise in performing DUS and CE-MRA, but low expertise in performing CTA. We therefore assumed higher values of sensitivity and specificity of DUS and CE-MRA (and used the upper limit of the 95 % confidence intervals for DUS and CE-MRA) and a lower performance of CTA than average in hospital 2 (and used the lower limit of the 95 % confidence interval for CTA).

The total costs per patient consisted of the costs of diagnosis, treatment (i.e., carotid endarterectomy and medicines), and stroke-related societal costs (which comprise both healthcare and non-healthcare costs). The average diagnostic test costs per patient were calculated by combining the frequency of tests used in the assessment with their unit costs (see Fig. 1). Figure 1 (upper right-hand corner) shows the 5-year average treatment and stroke-related societal costs, and total QALYs for each category of patients (i.e., TP, FP, FN, and TN). The treatment costs of TP and FP patients included the costs of a carotid endarterectomy, which were based on a recent cost analysis [12], and the costs of medicines for each category of patients were based on expert opinion. The utility values and cost input parameters used in the model can be found in the Electronic Supplementary Material 2 along with their sources. The average treatment and stroke-related healthcare costs, and QALYs per patient were dependent on the sensitivity and specificity of the diagnostic tests. For example, a higher sensitivity results in a higher TP rate with more patients correctly identified for carotid endarterectomy. Likewise, a higher specificity results in a higher TN rate, which means that the correctly specified patients were prevented from unnecessary carotid endarterectomies resulting in lower costs and higher QALYs. All costs were calculated in 2012 Euros using a societal perspective. Differential discounting was applied in accordance with the Dutch guidelines, with an annual discount rate of 4.0 % for all costs and 1.5 % for health effects [13].

3 Results

Tables 2 and 3 show the use of initial and confirmatory imaging tests at the emergency units and outpatient clinics, respectively. These tables also show the degree of compliance to the guidelines regarding use of initial and confirmatory tests and criteria for use of a confirmatory test.

Table 2 Clinical practice variation in use of diagnostic tests in the emergency unit
Full size table
Table 3 Clinical practice variation in use of diagnostic tests in the outpatient clinic
Full size table

Table 2 shows that 56 % (9/16) of the hospitals’ emergency units use the test combinations in accordance with the Dutch guidelines; the other seven hospitals use various other test combinations, with an initial CTA and confirmatory DUS as the most common combination.

Table 3 shows that 63 % (10/16) of the hospitals’ outpatient clinics use the test combinations as advised in the Dutch guidelines; the other six hospitals use various other test combinations. In addition, Tables 2 and 3 show that only one hospital uses the criteria for use of a confirmatory test according to the guidelines. In contrast, the other hospitals use broader criteria regarding the degree of carotid stenosis and/or other criteria (e.g., age and plaque characteristics). For example, some hospitals use a confirmatory test for men with a >70 % carotid stenosis or a confirmatory test for women with a 50–69 % carotid stenosis.

3.1 Criteria for Treatment

We found that 38 % (6/16) of vascular neurologists strictly use the criteria for treatment as advised in the guidelines (i.e., degree of carotid stenosis, gender, and time since TIA or minor ischemic stroke onset) in their decision for either surgery or medicines only. The other vascular neurologists use additional criteria in their decision making about surgery. Patient age and life expectancy play a role in 44 % (7/16) of the hospitals. Other factors that influence decision making are co-morbidity, risk of surgery, patient preferences, duration of complaints, severity of the stroke, and plaque characteristics.

3.2 Reasons to Deviate from Guidelines Regarding Use of Initial and Confirmatory Test

According to vascular neurologists, clinical practice variation in the choice of initial and confirmatory tests arises for different reasons, including varying degrees of expertise in performing diagnostic tests, patient case-mix, clinical reasons, financial incentives, logistics, availability of imaging technology, and preferences of radiologists, vascular surgeons and vascular neurologists. For example, one hospital uses a CTA as the initial test due to high expertise in CTA, even though the guidelines recommend CTA-only as a confirmatory test. Another hospital uses an initial CTA instead of the DUS recommended in the guidelines because CTA is more accurate than DUS. In addition, 25 % (4/16) of hospitals use a more costly confirmatory test (i.e., CTA or MRA), even though the guidelines indicate that a DUS is sufficient. Examples of clinical reasons for deviating from the guidelines were that MRA could not be used with patients with a pacemaker and that CE-MRA could not be used for patients with kidney problems or allergy due to the contrast liquid.

3.3 Case Study of Hospital-Level CEAs

Figure 2 shows the results of the base-case analysis and two hospital-level CEAs. For each test strategy, this figure shows the average costs and health effects in QALYs per patient. It also shows that the most cost-effective strategy differs between the base-case analysis and the two hospital-level CEAs. In the base-case analysis, the CTA-only and CE-MRA-only strategies were the most cost-effective strategies (see Fig. 2a). When comparing these two strategies, the CE-MRA-only strategy leads to slightly more QALYs (0.001) and higher costs (€287) versus the CTA-only strategy, resulting in an incremental cost-effectiveness ratio (ICER) of €294,785 per QALY gained. The high ICER means that the CE-MRA-only strategy is not cost-effective versus the CTA-only strategy; therefore, CTA-only is the preferred strategy in the base-case analysis. However, the results changed when hospital-level values for unit costs and test performance were used. To start with, the CTA-only strategy was the dominant strategy in the first hospital, since it had the lowest costs and highest QALYs of all strategies (see Fig. 2b). In contrast, the CE-MRA-only strategy was the dominant one in the second hospital (see Fig. 2c).

Fig. 2
figure 2

Hospital-level cost-effectiveness results. Guideline-based strategy = DUS + CTA (confirmatory), Hospital 1 currently uses CTA-only strategy, Hospital 2 currently uses DUS + CTA (confirmatory) strategy. DUS duplex ultrasonography, CTA computed tomography angiography, CE-MRA contrast-enhanced-magnetic resonance angiography, QALY quality-adjusted life year, ICER incremental cost-effectiveness ratio

Full size image

4 Conclusions and Discussion

CEAs using guidelines as a comparator are inadequate if the guidelines are not used in clinical practice. The existence of important clinical practice variation and the resulting cost differences support the need to perform hospital-level CEAs which incorporate local hospital conditions (e.g., patient case-mix, costs, availability of facilities, and expertise). A hospital-level CEA could examine the cost-effectiveness of the hospital’s current care strategy versus the strategies used in other hospitals as well as strategies incorporating new tests or treatments. This will result in multiple hospital-level ICERs that will help individual hospitals to explore the potential to improve effectiveness and cost-effectiveness by implementing a different strategy. One possible rule to decide if hospital-level CEAs should be performed is to compare which tests or treatments are performed, while another would be to see if the different strategies used in current care have different short-term costs and effectiveness (i.e., when the costs and effectiveness of the different current care strategies are similar, then it would be irrelevant to use multiple comparators). One extreme solution would be to model the long-term impact on costs and effectiveness before concluding whether the observed practice variation is actually important. Lastly, hospital-level CEAs may be of interest to other parties than just individual hospitals. For example, health insurers might want to use hospital-level CEAs to determine how costly and cost-effective the care currently provided in hospitals is compared to the most cost-effective strategy available.

The observed variation in the use of diagnostic tests for patients with a recent TIA or minor ischemic stroke means that the most cost-effective diagnostic strategy may differ between hospitals, as illustrated in our case study. In the first hospital-level CEA, the average 5-year costs per patient range from €15,862 to €17,145 between strategies. While this range may seem small, its effect on budget may be important depending on the annual volume of patients [14]. For example, if this hospital were to assess 500 patients per year (i.e., the average number of patients with a TIA or ischemic stroke per hospital in the Netherlands in 2012 [8, 15]), the total 5-year costs would range from €7,930,779 to €8,572,496, meaning a difference of €641,717.

There are several ways in which practice variation may result in differences in overall costs and health effects [16, 17]. First of all, a cost difference arises from using different diagnostic tests. The use of different strategies may also result in short-term differences in health effects simply due to differences in complication risks or patient discomfort (e.g., a more invasive test leads to more discomfort for the patient). Moreover, the long-term differences in costs and health effects between the diagnostic strategies are caused by differences in sensitivity and specificity of the diagnostic tests used. For example, if patients are more often misclassified (i.e., FPs or FNs) by one test than by another test, this may lead to greater long-term costs and less health. Even if hospitals use the very same diagnostic test, this too may result in different long-term costs and health effects if they use the test results differently when making treatment decisions or when the test’s diagnostic accuracy differs between hospitals (as was illustrated in our case study).

If hospitals perform their own hospital-level CEAs, they should consider the feasibility of the different strategies being considered as well as incorporate the costs of switching to a different strategy in the analysis. This holds true for strategies involving new interventions (such as new diagnostic tests, medicines or therapies) as well as existing ones. Switching to a different strategy is only worth considering when local hospital conditions can readily be modified (e.g., training clinicians and other healthcare personnel to use a more advanced imaging test) [18, 19].

If clinicians lack the necessary expertise to perform the most cost-effective strategy, the hospital-level CEA must include the extra costs of training. In addition, capacity and logistical problems may arise because of limited availability of tests. For example, some small hospitals have DUS available, but not more expensive scanners like CTA or MRA. These hospitals currently refer patients to larger hospitals, resulting in higher costs (e.g., repetition of tests and travel costs) and a delay in decision making. A CEA for such small hospitals should include the costs of purchasing an imaging test and possible training costs of personnel. Viewed in that way, the current strategy may be more cost-effective due to the relatively high implementation costs.

Hospitals should have a sufficient understanding of clinician attitudes when designing an implementation strategy, since attitudes influence behavior and the choice of strategy used in clinical practice [18]. Innovation managers or other professionals may assist in the implementation process, for example by showing clinicians convincing evidence about the improved effectiveness and/or cost-effectiveness of a new strategy versus the existing one [19].

The aim of our paper was to present a major methodological issue that seems to be underestimated by many. We have illustrated the importance of investigating possible practice variation and deviation from the clinical guidelines, and have demonstrated the value of performing hospital-level CEAs based on local hospital conditions (e.g., unit costs) when important practice variation exists. The general principles of performing hospital-level CEAs are valid and should be applied when clinical practice differs between hospitals or when clinical practice differs significantly from guidelines, irrespective of the disease area or countries under study. Further research is recommended that applies the presented general principles of performing hospital-level CEAs to other disease areas, types of care, and countries. We do not claim generalizability of the results from our case study based on two hospitals, to all Dutch hospitals. The aim of the case study was merely to illustrate the importance of performing hospital-level CEAs.

One limitation of performing hospital-level CEAs may be the feasibility. However, the additional data needed to perform hospital-level CEAs (when practice variation is found) are the hospital-specific costs of the tests and the hospital-specific sensitivity and specificity of tests. We recommend hospital-level CEAs if data for potentially influential variables (like the hospital-level costs and performance of tests in this study) can be retrieved or sufficiently estimated. Moreover, there is a good chance that hospitals unable to retrieve data on potentially influential parameters are not functioning very efficiently compared to those that are. The results of a CEA based on a hospital’s own data can help a hospital to perform more efficiently; in the example described in this paper, this would involve comparing various test strategies and selecting the one that is most cost-effective for that particular hospital. This approach may demonstrate that one strategy is the most cost-effective in one hospital, while another strategy is the most cost-effective in another hospital. In this sense, the ultimate choice of a hospital may differ from national guidelines for justifiable reasons.

In conclusion, consideration of clinical practice variation and deviation from the clinical guidelines should be one of the first steps in any CEA. If important practice variation or deviation from the guidelines exists, hospital-level CEAs should be performed which compare the care that is actually provided in hospitals. Moreover, a hospital-level CEA should consider the causes of variation, since they will affect the feasibility and costs of implementing a new strategy.