FormalPara Key Points for Decision Makers

Differing methodological requirements and decision-making criteria are recognised as barriers to transferability of cost-effectiveness analysis (CEA) across jurisdictions.

Even when parameter inputs and methodological assumptions are not exactly aligned, this case study demonstrates that conclusions of CEA may be comparable across jurisdictions.

For international joint procurement initiatives, determining and implementing joint decision rules may be more important than trying to align rules regarding CEA parameter input.

1 Introduction

There is increasing emphasis on collaboration between health technology assessment (HTA) agencies across Europe to manage increasing expenditure in the health sector. EUnetHTA, a network of more than 80 European HTA agencies across 30 countries, was established to create sustainable collaboration in HTA production and use in Europe [1]. In January 2018, the European Commission published a proposal for mandatory joint relative effectiveness assessments of new drug technologies in Europe, envisaging implementation within a 6-year timeframe from the passing of the regulations [2]. Separately, the Beneluxa initiative envisages more ambitious cooperation, proposing collaboration around joint HTA and pricing and reimbursement [3]. Ireland has recently joined the Beneluxa collaboration [4].

Cost-effectiveness analysis (CEA) is a component of HTA. In Ireland, the National Centre for Pharmacoeconomics (NCPE) performs CEA of new drugs on behalf of the national health payer, the Health Service Executive (HSE) [5]. These are conducted in line with the national reference case as outlined by the Health Information and Quality Authority (HIQA) in their Guidelines for Economic Evaluation of Health Technologies in Ireland [6]. There are two stated cost-effectiveness thresholds for pharmaceuticals in Ireland, €20,000 per quality-adjusted life-year (QALY) and €45,000 per QALY, as specified in the current agreement between the HSE and related government bodies, and the industry representative body [7]. There is no explicit use of higher thresholds for cancer or rare disease treatments, although it has been shown that factors other than the incremental cost-effectiveness ratio (ICER), such as quality of the clinical evidence, can influence NCPE recommendations for reimbursement decisions [8]. NCPE recommendations on cost effectiveness and relative clinical effectiveness are considered by the HSE when making their reimbursement decisions, alongside the additional criteria outlined in Schedule 3 Part 3 of the Health (Pricing and Supply of Medical Goods) Act 2013 [9].

The reimbursement process for pharmaceuticals in Ireland is different to that of the other Beneluxa countries, in terms of the specifics of the national reference case for CEA, but also in terms of cost-effectiveness thresholds for reimbursement [10,11,12]. Of interest is how these differences in the national reference case and cost-effectiveness thresholds may post barriers to joint CEA and decision making [13].

Cost-effectiveness estimates for pharmaceuticals vary between countries in western Europe [14]. Cost inputs for CEA vary greatly between jurisdictions, due to differences in both unit costs and resource use [15]. Additionally, it is recognised that outcome measurements for CEA, such as treatment effects and preferences for health, cannot be assumed to be constant across jurisdictions, but are shaped by factors such as baseline population risk, clinical management and cultural influences [16, 17]. The requirement for localised parameter inputs has long been recognised as a barrier to transferring CEA between regions or countries. Moreover, decision rules for drug reimbursement following CEA vary, with many countries having no explicit cost-effectiveness threshold for reimbursement decisions.

Transferability of CEA has been defined as “the ability to extrapolate results obtained from one setting or context to another” [18]. A distinction has been drawn between the ‘generic transferability’, a property of the study itself reflecting chosen methods and reporting, and ‘specific transferability’, a function of the decision context in a given jurisdiction [18]. Critical factors affecting the generic transferability of CEA include quality, transparency, accuracy of reporting and methodology [19]. Methodological requirements such as discount rates, time horizons and assessment of preferences, differ between jurisdictions and have been identified as key factors to consider when assessing generic transferability [18]. There is limited empirical information on how relevant these methodological choices are to reimbursement decisions in the context of cost-effectiveness thresholds; that is, the specific transferability.

We sought to assess the generic and specific transferability of published CEAs of systemic treatments for advanced melanoma skin cancer to the Irish setting. The aim was to determine if the outcomes and conclusions of these CEAs are transferable to the Irish setting. The disease area of melanoma was chosen as recent advances in treatment have increased survival for patients with advanced melanoma, but there is large geographical disparity in reimbursement decisions and patient access across Europe [20].

2 Methods

2.1 Data Identification and Extraction

CEAs were identified through systematic review [21]. De novo decision analytic models investigating the cost effectiveness of treatments for advanced melanoma were eligible for inclusion, regardless of line of treatment. Summary documents of models submitted to HTA agencies for reimbursement purposes were excluded as it was expected they would not contain sufficient information to assess transferability, and would potentially draw conclusions based on mark-down prices (drug prices to which a price reduction has been applied), which would not be considered transferable. Only pharmacological treatments were considered, including comparisons with best supportive care and no treatment. All studies with an outcome of cost per QALY, cost per life-year gained (LYG) or alternative measure of health outcome were included. The search was conducted in the Cochrane database, MEDLINE, Embase and EconLit in September 2018 [Appendix Table A1, see electronic supplementary material (ESM)] [21]. Data on model parameters, predicted total costs, incremental QALYs, incremental costs, ICERs, cost-effectiveness threshold and study conclusions were extracted by two authors independently (CG, LMcC). Total costs were inflated to 2017 values using the consumer price index for health and converted to US dollars using the purchasing power parity (PPP) method [22]. Comprehensiveness of reporting was assessed using the CHEERS checklist [23].

2.2 Assessment of Generic Transferability of Study Outcomes

Transferability to the Irish setting was examined using the Economic Evaluation domain of the EUnetHTA transferability tool [24]. This tool was developed as part of the EUnetHTA HTA Adaptation Toolkit, which focuses on three distinct elements for adaptation: relevance, reliability and transferability. The transferability domain consists of three questions. (i) How generalisable and relevant are the results and validity of the data and model to the relevant jurisdictions and populations? (ii) Are there any differences in the following parameters: perspective, preferences, relative costs, indirect costs, discount rate, technological context, personnel characteristics, epidemiological context, factors that influence incidence and prevalence, demographic context, life expectancy, reproduction, pre- and post-intervention care, integration of technology into the healthcare system and incentives? (iii) Does the evaluation violate the national guidelines for CEA? The tool is a qualitative instrument and no quantitative score for transferability is produced. We chose to focus on Sects. 2 and 3 of the tool to inform our consideration of transferability, as these sections posed defined questions that can be related back to the national reference case for CEA, and allowed the flexibility to record an overall impression of the transferability of the study also.

Two authors (CG, LMcC) completed the assessment independently, and differences were resolved by arbitration. Considerations on generic transferability were made with reference to the publicly funded healthcare system in Ireland, the current clinical management of advanced melanoma in Ireland [25] and the Irish national reference case for CEA, published in 2018 (Table 1) [6]. The extracted data is tabulated, and the authors’ overall conclusions regarding transferability of each study presented. Where differences were documented from the Irish reference case, the published deterministic sensitivity analyses were reviewed to determine the likely impact of these differences on the outcome of the CEA.

Table 1 HIQA reference case for economic evaluation in Ireland [6]
Full size table

Studies were considered to have a relevant technological context if the interventions or comparators are part of current routine care for advanced melanoma in Ireland or were representative of routine care at the time of publication. Differences in pre- and post-intervention care were considered in the same manner. Where insufficient information to assess differences from the national reference case or Irish clinical practice was provided, the assessment was marked ‘unclear’. The template includes ‘Incentives’ as a relevant topic for consideration. This was interpreted to refer to financial incentive structures that may impact on clinical decisions. The national reference case specifies the perspective of the publicly funded health system in Ireland, and thus it was considered that studies in countries with predominately privately funded health insurance markets could have a difference incentive structure to the Irish setting. The template also specified ‘Relative costs’ for consideration, which was interpreted as likely differences in the unit costs of resource use and for the interventions. Studies that were clearly not generically transferable to the Irish setting were excluded from the assessment of specific transferability.

2.3 Assessment of Specific Transferability of Study Conclusions

For studies considered to demonstrate generic transferability to the Irish setting, the specific transferability to the Irish context was considered. The cost-effectiveness thresholds specified in the included CEAs were extracted, and converted to US dollars (2017) using the PPP method. In a comparable manner, the Irish thresholds of €20,000 and €45,000 per QALY were converted to US dollars. The extracted and inflated incremental costs and QALYs from the included CEAs for each comparison (pairwise ICER) were used to calculate the net monetary benefit (NMB) at (i) the cost-effectiveness threshold specified in the published CEA and (ii) the Irish cost-effectiveness thresholds of US$26,667 and US$60,000 per QALY. NMB was calculated according to the formula: NMB = (∆QALYs × λ) – Δ costs, where λ is the cost-effectiveness threshold. A positive value for the NMB implies that the product is cost effective at the chosen threshold. NMB was used in preference to ICERs to allow dominant strategies to be explored.

The pairwise comparisons where the NMB was positive or negative at the cost-effectiveness threshold specified in the CEA was determined, and the proportion considered to be cost effective (positive NMB) calculated. The NMB for each pairwise comparison was then recalculated using the Irish cost-effectiveness threshold (US$26,666 and US$60,000 per QALY). The proportion of pairwise comparisons where the NMB changed from positive to negative or vice versa, indicating a change in the decision when interpreted at the Irish cost-effectiveness threshold, was calculated.

The outcome from the NCPE HTA assessment for each pairwise comparison was extracted from the summary HTA reports published on the NCPE website [26,27,28,29,30,31,32,33,34]. Concordance between recommendations at the study CEA threshold, and when reanalysed at the Irish threshold, with the published NCPE recommendations was investigated.

3 Results

The systematic review identified 660 records, and 131 underwent full-text review [21]. Fifteen academic studies, published between 2000 and 2018, were eligible for inclusion (Appendix 1, Table A2, see ESM) [35,36,37,38,39,40,41,42,43,44,45,46,47,49]. Eight studies were from the US, five from European countries, one from Canada and one from Australia. No study was adjudicated to have perfect compliance with the reporting checklists. From the 15 included studies, 35 distinct pairwise comparisons were identified.

3.1 EUnetHTA Transferability Assessment

The outcomes of the transferability assessment are shown in Table 2. Most of the identified CEAs had the required perspective (payer, n = 10), and therefore the required approach to indirect costs [35, 37, 38, 41, 42, 44,45,46,47, 49]. Two CEAs adopted the societal perspective [36, 43], while three stated a societal perspective was taken but did not document the inclusion of indirect costs and were categorised as payer perspective [39, 40, 48]. Utility values were derived using either health state vignettes and the standard gamble method (n = 5), or from a generic utility instrument (EQ-5D, n = 7), while one study used a mixture of both [44]. Three studies did not apply or did not specify a discount rate [36, 39, 43]. One study applied a different rate to costs and QALYs [44]. Rates applied varied from 3 to 6%; only three studies applied the same discount rate as the Irish reference case of 5% [35, 42, 47]. The technological context for the management of advanced melanoma has moved swiftly in recent times. However, studies were considered transferable if they considered comparators that were relevant at the time of publication, and if the pre- and post-intervention care modelled in the CEA were considered a likely representation of care at that time in Ireland. Thus, no studies were excluded due to differences in the technological context or pre- and post-intervention care.

Table 2 Conclusions of EUnetHTA transferability assessment
Full size table

Treatment acquisition costs were generally based on the publicly available price, with three exceptions where mark-down prices were used, which limits transferability to the Irish context [38, 46, 48]. Three studies did not provide a cost per QALY outcome [36, 43, 48], and were not considered transferable to the Irish context, as there is no specified cost-effectiveness threshold for alternative outcomes.

The EUnetHTA tool includes several health system and national factors when considering transferability: personnel characteristics, the integration of technology into the healthcare system, factors affecting disease incidence and prevalence and the epidemiological and demographic context. The reference case for HTA in Ireland does not provide specific directions relating to these topics, except to highlight that the inputs should be reflective of the relevant target population. The epidemiological context and factors that influence incidence and prevalence were considered likely to be different in the identified cost-effectiveness models, as Ireland has a relatively high incidence of advanced melanoma and a lower incidence of melanoma with the BRAF mutation compared with international norms [50]. Ireland also has a younger population than many other OECD countries, and a relatively high life expectancy [51]. However, in the absence of explicit modelling of baseline risk or national population demographics, these differences were considered unlikely to impact on the outcomes of the included cost-effectiveness models. Insufficient information was provided in the reviewed studies to determine if personnel characteristics or integration of technology into the health service were similar to Ireland, but it was considered unlikely that there would be significant differences. These factors were all marked as unlikely to affect transferability in this assessment.

Incentives were considered likely to be the same in five CEAs with publicly funded health systems, as they provide health care in a similar model to Ireland [38, 42, 46, 47, 49]. Studies conducted within the US health system were considered to have different incentive structures, due to differences in the model of health care provision [36, 37, 39,40,41, 43, 45, 48]. The incentive structure was considered unclear in two studies, set in Australia and Switzerland, due to the interplay of private and public provision and compulsory private health insurance, respectively [35, 44]. However, the incentive structure was considered unlikely to have a significant impact overall on the costs and QALY gain associated with each treatment, and so did not preclude any studies from the specific transferability assessment.

Where deterministic sensitivity analyses were published, the outcomes were reviewed in light of the specified parameters of the Irish reference case. Discount rates, when included, were not found to be important drivers of the ICER in any of the CEAs [35, 38, 47]. Time horizon was an important driver of the ICER in three studies [35, 37, 42]. In two of these studies, the time horizons used were significantly shorter than the recommended lifetime horizon of the Irish reference case, which could potentially limit the transferability of the outcomes to the Irish context [35, 42]. In both cases, the decision was unlikely to change with an increase in the time horizon. Health state costs were important in four studies in determining the ICER [35, 37, 38, 41], but would only have changed the decision in one study when varied to the upper limit in the deterministic sensitivity analysis [41]. Utility was an important determinant of the ICER in seven of the included studies, but only in two of those would it be likely to change the decision based on the tornado plots provided in the published manuscript [45, 48].

None of the identified CEAs were entirely generically transferable to the Irish setting, as no model was fully aligned with the Irish reference case. Alignment with the reference case was most commonly seen for perspective, treatment of indirect costs and the technological context. Disagreement was most commonly seen for the discount rate (n = 12). It was considered highly likely that the results would change or be irrelevant if transferred to the Irish setting for five of the CEAs, two of which did not provide a cost per QALY outcome, and three of which used mark-down prices to estimate the ICER. The remaining studies (n = 10) were included in the consideration of specific transferability.

3.2 Transferability of Conclusions in Accordance with Cost-Effectiveness Threshold

For the assessment of specific transferability, incremental costs, QALYs and cost-effectiveness thresholds were extracted from the included CEAs (n = 10) (Table 3). Twenty-five pairwise comparisons were considered from these studies. The NMB was calculated at the cost-effectiveness thresholds specified in the published studies, which varied from US$35,000 to US$200,000 per QALY (Table 3). Of the 25 comparisons, nine had a positive NMB at the cost-effectiveness threshold specified in the published study (36%) (Table 3), indicating that they were cost effective at the study threshold, and 16 (64%) had a negative NMB, indicating that they were not considered to be cost effective.

Table 3 Net monetary benefit at study cost-effectiveness threshold and Irish cost-effectiveness threshold
Full size table

Secondly, the NMB for these pairwise comparisons was recalculated using the Irish cost-effectiveness thresholds. When the higher Irish threshold (US$60,000 per QALY) was applied to the calculation of NMB, only four comparisons estimated a positive NMB (16%), a reduction of 20% compared with using the CEA-specific threshold (Table 3). Using the lower Irish threshold (US$26,667 per QALY), only three comparisons (12%) retained a positive NMB.

Data on incremental costs and QALYs for the pairwise comparisons were extracted from the published NCPE summary HTA reports and converted to US dollars (Table 4). The NCPE had not evaluated six of the pairwise comparisons for the Irish drug reimbursement process (mainly comparisons of the newest agents versus chemotherapy, and unlicensed regimens). Of the 19 pairwise comparisons that could be compared with the identified CEAs, in 14 (73.7%) instances the NCPE conclusion based on the ICER was in agreement with the study conclusions, and five were in disagreement (26.3%).

Table 4 Extracted outcomes from NCPE summary HTA reports
Full size table

When the study NMB for each pairwise comparison was re-estimated at the higher Irish threshold, 17 of 19 (89.5%) pairwise comparisons were in agreement with the NCPE conclusions based on the ICER. In the five comparisons where the conclusions changed from positive at the CEA-specific threshold to negative at the Irish threshold, three changed to be in agreement with the NCPE conclusions, whilst two that had previously agreed with the NCPE were now different [41, 47].

When the study NMB for each pairwise comparison was re-estimated using the lower Irish threshold, only three comparisons (12%) retained a positive NMB. The single additional study that changed conclusion at this threshold compared with the higher Irish threshold shifted to be in agreement with the NCPE conclusion, resulting in a total of 18 out of 19 comparisons (95%) being in agreement with the NCPE conclusions.

4 Discussion

Decision analytic models evaluating the cost effectiveness of treatments for advanced melanoma have been reviewed for their transferability to the Irish setting. The models varied in the structural assumptions underpinning the model outputs and conclusions, in terms of the perspective, model type, time horizon adopted, preferences and discount rate applied.

Assessment of transferability was difficult due to poor quality reporting [21]. Unsurprisingly, none of the identified CEAs demonstrated complete generic transferability to the Irish setting as none were aligned with the national reference case for CEA. Alignment with the reference case was most commonly seen for perspective, treatment of indirect costs and the technological context. Disagreement was most commonly seen for the discount rate (n = 12). The potential impact on study conclusion of differences from the Irish reference case was not always easy to predict. The parameters considered most likely to cause variation include the discount rate, time horizon and relative costs. However, close examination of the deterministic sensitivity analyses in the published CEAs suggested that variations in most of these parameters would rarely have changed the decision at the cost-effectiveness threshold employed in the CEA. Of the 15 identified CEAs, five were excluded from the assessment of specific transferability as they were deemed inapplicable to the Irish context; two did not contain a cost per QALY outcome and three used mark-down prices.

The EUnetHTA transferability tool does not explicitly consider the cost-effectiveness threshold. We considered the transferability of the study outcomes to the Irish context by estimating the NMB at the study threshold and at the Irish cost-effectiveness thresholds. Using the Irish cost-effectiveness thresholds of US$26,667 and US$60,000 per QALY to estimate the NMB of each intervention, there was a reduction of 24% and 20%, respectively, in the proportion of positive conclusions compared with when the study-specific threshold was used.

The study conclusions were also compared with the conclusions of the NCPE for the same comparisons. Without any adjustment for differences in the cost-effectiveness threshold, there was good agreement between the published CEAs and the conclusions of the NCPE summary HTA reports (74% aligned). Adjusting for the lower Irish threshold led to alignment in conclusions in 95% of cases, and using the higher Irish threshold, in 89.5% cases. Thus, despite differences in model inputs and methodological assumptions, the conclusions of the studies were consistent with the NCPE conclusions, and even more so when updated to consider the Irish cost-effectiveness threshold.

Guidelines for economic evaluation undergo periodic review. In Ireland, the HIQA Guidelines for Economic Evaluation were published in 2014, updated in 2018 [6] and again in 2019 [52]. The NCPE guidelines are updated frequently, in response to methodological advances. If joint HTA is a realistic expectation of the Beneluxa collaboration, then future guideline reviews should consider the guidelines from the Beneluxa countries, to determine if there are areas where closer alignment could be achieved.

Commentary regarding the potential for success of the Beneluxa collaboration has highlighted the challenges posed by the differing HTA assessment processes [13]. We have shown in this study that while parameter inputs may not be exactly aligned with the requirements for the national reference case, the conclusions may be comparable across jurisdictions. Accounting for differing thresholds across jurisdictions produces even greater convergence between study conclusions; up to 95% concordance. Thus, while these studies are not generically transferable to the Irish setting, the specific transferability was high, and increased when the Irish threshold was considered in the analysis. This suggests that for joint HTA and procurement as envisaged under the Beneluxa process, determining and implementing joint decision rules may be more important than trying to align rules regarding parameter inputs.

The consistency of outcomes demonstrated here using advanced melanoma as a case study may not be generalisable to other interventions. When price negotiations begin to lower the ICERs closer to the specified cost-effectiveness threshold, the differences due to parameter inputs and generic methodological assumptions will become more relevant, and are more likely to impact on the reimbursement decision. Additionally, for potentially ‘curative’ gene therapies or vaccine programmes, differences in approaches to discounting are likely to be highly relevant. However, for many high-cost cancer drugs and drugs for rare diseases, the ICERs can be many multiples of the national threshold, and changes in generic parameter inputs or the cost-effectiveness threshold are unlikely to change the decision outcome. In many instances, reimbursement is achieved, indicating that the willingness-to-pay of the payer may be higher that the stated cost-effectiveness threshold. Of note, the objectives of the Beneluxa collaboration are to ensure “timely access and affordability of medicines” [3], without reference to the strict efficiency maximising objective of the standard CEA framework. Thus, there may be a pragmatic approach taken that could allow collaborating partners to overcome these difficulties as decisions approach the chosen cost-effectiveness threshold.

There are limitations to our study. As models used for reimbursement decision making are rarely publicly available, a case study of published CEAs was used. Without access to the cost-effectiveness models, it was not possible to quantitatively assess the impact of generic model inputs such as the discount rate on the ICER or the NMB. The outcomes of the transferability assessment are limited in applicability to other jurisdictions with different reference case requirements to Ireland. Assessment of transferability is subjective, although it was easy to reach agreement between the two assessors. Assessment is constrained by the thoroughness of reporting of the CEA. The assessment of transferability does not explicitly review the quality of the included studies, which is an important metric for consideration.

5 Conclusions

Assessing compliance with the national reference case is essential when considering transferability of CEAs to a local setting. Additionally, consideration must be given to the local decision-making context including cost-effectiveness thresholds. Regional alignments of smaller countries such as the Beneluxa collaboration will encounter issues regarding generic transferability. However, the core issue of specific transferability with regards to the decision thresholds may be a more relevant consideration.