The evidence provided by the company comprised an initial submission, a cost-effectiveness model (which is commercial in confidence) and the company’s response to the ERG’s clarification requests. The ERG report is a summary and critical review of the evidence for the clinical and cost effectiveness of the technology provided by the company. The aims of the report were to:
-
assess whether the evidence submitted by the company conforms to the methodological guidelines issued by NICE;
-
assess whether the company’s interpretation and analysis of the evidence are appropriate;
-
indicate the presence of other sources of evidence or alternative interpretations of the evidence that could help inform the development of NICE guidance.
In addition to providing this detailed critique, the ERG modified several key assumptions and parameters within the company’s economic model in order to explore the robustness of the company’s results.
Clinical Evidence
The comparators specified in the final scope issued by NICE were Gem, Gem + Cap and FOLFIRINOX.
The company presented evidence for the clinical effectiveness of Nab-Pac + Gem from the CA046 trial (also known as mPACT) [11]. The CA046 trial was an open-label, multicentre, phase III, randomized controlled trial that was designed to investigate the efficacy and safety of Nab-Pac + Gem versus Gem in patients with untreated metastatic adenocarcinoma of the pancreas. A total of 831 patients were randomized to receive either Nab-Pac + Gem (n = 431) or Gem (n = 430).
The final overall survival (OS) analysis from the CA046 trial was based on 692 deaths (80% of patients; data cut-off: 17 September 2012) [11]. Median follow-up was 9.1 months in the Nab-Pac + Gem arm and 7.4 months in the Gem arm. An updated analysis of OS from the CA046 trial with an extended data cut-off was also reported (data cut-off: 9 May 2013) [12]. At the time of the updated analysis, 774 (90%) patients in the ITT population had died and median follow-up was 13.9 months.
Treatment with Nab-Pac + Gem was shown to improve median OS significantly compared with treatment with Gem (8.5 months vs. 6.7 months; hazard ratio [HR] 0.72, 95% confidence interval [CI] 0.62–0.83) [11]. The incremental OS benefit of treatment with Nab-Pac + Gem was 1.8 months in the final analysis and 2.1 months in the updated analysis [11, 12]. The effect of Nab-Pac + Gem was consistent over time as survival rates were statistically significantly higher in the Nab-Pac + Gem arm than in the Gem arm at both 1 year and 2 years (p < 0.001 and p = 0.02, respectively) [11]. All sensitivity analyses carried out by the company showed a statistically significant OS treatment effect in favour of patients treated with Nab-Pac + Gem. The estimate of treatment effect favoured treatment with Nab-Pac + Gem rather than Gem in all subgroups, except patients with normal CA19-9 levels for whom no conclusions could be drawn. Key results from the final OS analysis are shown in Table 1 and from the updated OS analysis in Table 2.
Table 1 CA046 trial primary and secondary efficacy endpoints (17 September 2012)
Table 2 Updated survival estimates in the CA046 trial (9 May 2013)
Treatment with Nab-Pac + Gem was shown to improve median progression-free survival (PFS) significantly compared with treatment with Gem in the CA046 trial [11]. Table 1 shows an incremental PFS benefit of 1.8 months for both PFS by independent review (HR 0.69, 95% CI 0.58–0.82) and PFS by investigator assessment (HR 0.61, 95% CI 0.52–0.71). At 1 year, PFS rates were greater in the Nab-Pac + Gem group compared with the Gem group (16 vs. 9%, independent review; 12 vs. 4%, investigator assessment).
The most common grade 3 or 4 adverse events (AEs) associated with treatment with Nab-Pac + Gem were neutropenia, fatigue, metabolism and nutritional disorders, peripheral neuropathy, thrombocytopenia and anaemia. Although these AEs were also associated with treatment with Gem and Nab-Pac monotherapies, they occurred more frequently when patients were treated with Nab-Pac + Gem.
No HRQoL data were collected in the CA046 trial [11]. Instead, the company presented early HRQoL results from the SIEGE trial [13], a phase II randomized trial designed to compare two different treatment schedules of Nab-Pac + Gem; the trial does not provide a comparison of Nab-Pac + Gem with Gem. These data were collected using the European Organisation for Research and Treatment Cancer (EORTC) Quality-of-Life Questionnaire (QLQ-C30). The company reported that Global Health Scores were generally stable throughout treatment; however, towards the end of the six-treatment-cycle period, data were difficult to interpret due to small patient numbers (n = 22 in the appropriate arm at week 24).
To allow a comparison of the effectiveness of treatment with Nab-Pac + Gem versus Gem + Cap and versus FOLFIRINOX, the company performed a network meta-analysis (NMA). Although a connected network could be formed by including only trials that compared treatments relevant to the decision problem, the company base-case network of ten trials [11, 14,15,16,17,18,19,20,21,22] included only three trials [11, 15, 16] that provided evidence for comparators listed in the final scope issued by NICE (i.e. Gem, Gem + Cap and FOLFIRINOX). The company also performed a sensitivity analysis using a reduced network that included only the comparators listed in the final scope issued by NICE. In terms of OS, the results from this sensitivity analysis mirrored the results from the base-case analysis and did not suggest a statistically significant treatment effect for Gem + Cap versus Nab-Pac + Gem (HR 1.10, 95% credible interval [CrI] 0.67–1.84) or for FOLFIRINOX versus Nab-Pac + Gem (HR 0.77, 95% CrI 0.58–1.01). For PFS, the results of the sensitivity analysis also mirrored the results from the base-case analysis which did not suggest a statistically significant treatment effect for Gem + Cap versus Nab-Pac + Gem (HR 1.17, 95% CrI 0.75–1.86); however, the results of the sensitivity analysis did suggest a statistically significant treatment effect for FOLFIRINOX versus Nab-Pac + Gem (HR 0.68, 95% CrI 0.51–0.91), unlike in the company’s base-case analysis. The results of the company’s base-case NMA and reduced network sensitivity analysis are shown in Table 3. The results from the company’s base-case NMA were used in the company’s cost-effectiveness model.
Table 3 Results of company network meta-analysis
Throughout the CS, the company maintained the position that the only comparator to treatment with Nab-Pac + Gem was Gem. The company claimed that Gem + Cap was used only rarely within the NHS, therefore did not represent standard of care and was not a relevant comparator. The company contended that patients who are suitable for treatment with FOLFIRINOX are easily identified in clinical practice and are clinically distinct from patients who would be treated with Gem but who could be treated with Nab-Pac + Gem. The company contended that the use of Nab-Pac + Gem in the NHS would only displace the use of Gem and would not affect the current NHS usage of either Gem + Cap or FOLFIRINOX.
Critique of the Clinical Evidence and Interpretation
The ERG considered that the company’s argument that Gem was the only relevant comparator was not compelling. The ERG noted that the company’s own market research data suggested that although many patients in the NHS receive Gem monotherapy, a proportion of patients receive Gem doublet therapy (such as Gem + Cap). The ERG therefore considered the argument that Gem + Cap was not a relevant comparator due to its limited use to be invalid. The ERG also considered that the company had failed to define the patients who would be suited to treatment with Nab-Pac + Gem but not FOLFIRINOX. Clinical advice to the ERG was that it would be difficult to clearly establish which patients in the NHS would be better suited to treatment with Nab-Pac + Gem rather than with FOLFIRINOX. The ERG considered that the issue of identifying which patients are suitable for treatment with Nab-Pac + Gem, but not with FOLFIRINOX, remained unresolved from TA360, and the ERG was unconvinced by the company’s argument that FOLFIRINOX was not a relevant comparator to Nab-Pac + Gem.
The ERG considered that the CA046 trial [11] was of good quality and well conducted. The trial data were mature and, with no patient crossover, the results allowed for reasonable conclusions to be drawn regarding the clinical effectiveness of Nab-Pac + Gem versus Gem in the trial population. Substantial numbers of patients were recruited and patient baseline characteristics were balanced across both trial arms. The statistical methods used to analyse trial data were generally appropriate. Clinical advice to the ERG was that patients recruited to the CA046 trial [11] were younger and fitter than the population of patients with metastatic disease treated in the NHS. Most notably, only 10% of patients recruited to the trial were aged ≥ 75 years, whereas Cancer Research UK (CRUK) statistics suggest that almost half (47%) of all patients diagnosed with pancreatic cancer are in this age band [23]. None of the participating treatment centres were based in the UK. The ERG considered the absence of HRQoL data from patients in the CA046 trial to be disappointing. The ERG also considered the HRQoL data from the phase II, dose-scheduling SIEGE trial to have the greatest relevance to the appraisal as it is a UK-based randomized trial that recruited patients with metastatic pancreatic cancer. However, it noted that only the ‘concomitant arm’ (i.e. treatment with Gem immediately after treatment with Nab-Pac) of the trial was relevant to this appraisal, which did not provide comparative data, and that only early results were available.
The ERG conducted assessments to determine the validity of the company’s assumption that survival hazards were proportional over time, and thus that the HRs presented in the CS were appropriate. The ERG’s analyses showed that over time the OS and PFS hazards from the two arms of the CA046 trial [11] were not proportional. Consequently, all HR results derived from the CA046 trial [11] should be interpreted with caution. Furthermore, the ERG highlighted that all of the company’s NMA results (base-case and sensitivity analyses) were affected by the lack of proportional hazards (PH) in the CA046 trial [11] and these results should also be interpreted with caution. Additionally, the ERG considered the results from the company’s reduced NMA to be more appropriate than the company’s base-case NMA results.
Cost-Effectiveness Evidence
The company adapted the model submitted within the original submission to NICE for appraisal TA360 [2] rather than constructing a de novo economic model. The company used a Markov structure and employed an area under the curve approach to estimating the proportion of patients who transition between health states over time from the start of treatment until death. There were three primary health states in the model: pre-progression, post-progression and death. The pre-progression state was divided into two secondary health states (pre-progression: on first-line treatment; and pre-progression: off first-line treatment) to more accurately estimate drug costs in cases where treatment was discontinued before progression. The company also included a tunnel state at 4 weeks to death to account for a period of intensive palliative care in the final stages of life.
Kaplan–Meier (K–M) data from the CA046 trial [11] were used as the basis for estimating patient survival for the comparison of treatment with Nab-Pac + Gem versus Gem. Stratified gamma curves were used to model OS, PFS and time on treatment (TOT). Resource use and costs were estimated based on information from the CA046 trial [11], published sources and advice from clinical experts. A confidential Department of Health PAS discount was applied to the cost of Nab-Pac. Full list prices, accessed via the drugs and pharmaceutical electronic market information tool [24], the MIMS database [25] and the British National Formulary [26] in January 2017, were used to calculate the cost of all other drugs. No vial sharing was assumed. Overall drug costs in the first-line setting were subject to the assumption that 50% of all first-time dose reductions and all subsequent dose reductions could be anticipated, meaning that there would be no drug wastage from these reductions. The company also assumed that 50% of all missed doses could be anticipated. Chemotherapy administration costs, monitoring costs, AE costs, and the cost of palliative and end-of-life care were sourced from NHS reference costs 2015/2016 [27] and the Personal Social Services Research Unit 2016 [28].
The company’s base-case analysis prediction was a mean of 0.927 life-years (LYs) gained for patients receiving Nab-Pac + Gem, 0.725 LYs gained for patients receiving Gem, 0.950 LYs gained for patients receiving Gem + Cap and 1.154 LYs gained for patients receiving FOLFIRINOX.
As HRQoL data were not collected as part of the CA046 trial, the company instead adjusted published health state utility values [29] for use in a UK population. These adjusted values were used in the base-case analysis for pre-progression (0.74) and progressive disease (0.67). The company used EQ-5D-5L data from the ‘concomitant’ arm of the SIEGE trial in separate scenario analyses.
The company submitted an updated model during the clarification process to correct an error. The company’s base-case incremental cost-effectiveness ratio (ICER) for the comparison of treatment with Nab-Pac + Gem versus Gem from the updated model was £46,932 per quality-adjusted life-year (QALY) gained. Treatment with Nab-Pac + Gem was dominated (more costly and generated fewer QALYs) by treatment with both Gem + Cap and with FOLFIRINOX.
The company carried out a wide range of deterministic sensitivity analyses for the comparison of treatment with Nab-Pac + Gem versus Gem. The results showed that the most influential parameter was the treatment variable used to parameterise OS.
The results of the company’s probabilistic sensitivity analysis showed that Nab-Pac + Gem had a 64% probability of being cost effective compared with Gem at a willingness-to-pay threshold of £50,000 per QALY gained.
Critique of the Cost-Effectiveness Evidence and Interpretation
The ERG considered the company’s model to be generally well-structured and correctly implemented. The ERG amended one structural feature in the calculation of total LYs and QALYs. The three key issues that required exploration by the ERG in the company’s model were HRs used for treatment with Gem + Cap and with FOLFIRINOX, costing of drugs, and modelling of TOT.
The company used HRs from its base-case NMA to estimate time-to-event outcomes for treatment with Gem + Cap and with FOLFIRINOX, which relied on the PH assumption holding for PFS and OS within the CA046 trial [11]. Since PH had been shown not to hold for PFS or OS in the CA046 trial [11], using the results of the NMA in the model produced unreliable estimates for OS, PFS and TOT for treatment with Gem + Cap and with FOLFIRINOX. The ERG also had concerns about the company’s use of HRs with a stratified Gamma model as the Gamma model is an accelerated failure time model rather than a PH model. The ERG applied published HRs for treatment with Gem + Cap versus Gem [15] and with FOLFIRINOX versus Gem [16] in the model to overcome the need for PH to hold in the CA046 trial [11]; however, PH did not hold for FOLFIRINOX versus Gem for either PFS or OS. The ERG considered that results for the comparison of treatment with Nab-Pac + Gem versus Gem + Cap and versus FOLFIRINOX should be treated with caution.
The company estimated average treatment costs for the intervention and comparators using only a limited range of the vial sizes available to the NHS for each drug. By incorporating all available vial sizes in the calculation of drug costs, the ERG estimated lower average weekly costs for each first-line treatment in the company model.
The ERG prefers the use of K–M data directly as far as possible when time-to-event evidence comes from a single trial, especially when the trial data are mature. The TOT data from the CA046 trial (supplied by the company during the clarification process) were complete and therefore represented the best possible evidence of time spent on treatment for the patients in that trial. However, the company used a fully parametric model to estimate TOT, which introduced unnecessary uncertainties into the analysis and resulted in an overestimation of TOT for both treatments. The ERG re-estimated TOT for treatment with Nab-Pac + Gem and with Gem using K–M data directly from the CA046 trial.
The company also used parametric models to estimate PFS and OS for treatment with Nab-Pac + Gem and with Gem using mature data from the CA046 trial. The ERG investigated remodelling PFS and OS for treatment with Nab-Pac + Gem and with Gem using K–M data as far as possible, then appending a parametric tail to extrapolate beyond the trial data. The ERG found that its remodelling of PFS and OS for treatment with Nab-Pac + Gem and with Gem had only a small impact on the size of the ICERs per QALY gained.
Other issues identified by the ERG included the double counting of AE disutilities. The ERG provided two scenario analyses that investigate the impact of using different costs for some AEs and using a different source of utility values. The impact of the ERG’s various amendments on the company’s base-case ICER per QALY gained are shown in Table 4.
Table 4 Cost-effectiveness results: ERG revisions to company base case
Conclusions of the ERG Report
The ERG considered that the evidence submitted by the company largely reflected the decision problem defined in the final scope issued by NICE, although direct clinical effectiveness evidence was only available for the comparison of the efficacy of Nab-Pac + Gem versus Gem.
The ERG noted that since the PH assumption for OS and PFS in the CA046 trial was violated, any HRs resulting from that trial should be treated with caution. This was true for the CA046 trial and the company’s NMA. The true clinical effectiveness of Nab-Pac + Gem compared with Gem, Gem + Cap or FOLFIRINOX remains to be established.
The ERG considered that the company had failed to clearly define the patient population for whom treatment with Nab-Pac + Gem is appropriate. The ERG remained unconvinced by the company’s case for Gem as the only comparator to Nab-Pac + Gem.
The various changes implemented by the ERG for the comparison of treatment with Nab-Pac + Gem versus Gem, treatment with Nab-Pac + Gem versus Gem + Cap and treatment with Nab-Pac + Gem versus FOLFIRINOX yielded a mixture of effects. Incremental costs and incremental benefits both increased and decreased depending on the individual revision. However, none of the ERG’s individual revisions or revised base-case scenarios yielded ICERs under £30,000 per QALY gained for treatment with Nab-Pac + Gem against any of the comparators. Only the comparison of Nab-Pac + Gem versus Gem yielded ICERs under £50,000 per QALY gained once all the ERG’s revisions and scenarios were applied.