In May 2019, the company (GW Research Ltd.) provided a submission to NICE on the clinical and cost-effectiveness of CBD within its anticipated licensed indication for LGS and DS [1, 11]. Following the usual process for STAs, the company provided additional information in response to clarification questions raised by the ERG and NICE [15,16,17]. Additionally, the ERG adjusted the decision analytic model received from the company to assess the impact of alternative parameter values and assumptions on the model results and to produce an ERG base case.
Sections 3.1–3.6 below summarise the evidence presented in the CS, as well as the ERG’s review of that evidence. Given that the two STAs were similar in terms of the evidence provided (e.g. design of clinical trials), health economic model structure and treatment mix, but differed in smaller but important aspects, some sections will discuss general issues applicable to both submissions and issues specific for the CS of LGS and DS separately.
Clinical Effectiveness Evidence Submitted by the Company
For both DS and LGS, the CSs identified four international randomised controlled trials (RCTs) of CBD (GWPCARE1 and GWPCARE2 for DS, and GWPCARE3 and GWPCARE4 for LGS) and an ongoing open-label extension study (GWPCARE5) as relevant to both submissions [18,19,20]. Both LGS RCTs were conducted in patients aged 2–55 years whose seizures were incompletely controlled with previous AEDs and who had suffered at least two drop seizures per week in the baseline period. Both DS RCTs included patients from 2 to 18 who had to have suffered at least four convulsive seizures in the past 28 days. The intervention was CBD in addition to current clinical management (CCM), and the comparator was CCM without CBD (i.e. CCM plus placebo). GWPCARE2 and GWPCARE3 were three-arm studies and compared two dosages of CBD (10 mg/kg/day and 20 mg/kg/day) in addition to CCM and CCM plus placebo. GWPCARE1 and GWPCARE4 compared CBD (20 mg/kg/day) in addition to CCM and CCM plus placebo. All four RCTs had a dose-escalation phase (14 days in GWPCARE1 and GWPCARE3, 7 or 11 days in GWPCARE 2 and 11 days in GWPCARE4) followed by a 12-week treatment period. GWPCARE 1 and GWPCARE3 included patients from the UK (three centres and 11 and 16 patients, respectively, overall). However, GWPCARE2 and GWPCARE4 did not include patients from the UK. A total of 120 and 198 patients were included in GWPCARE1 and GWPCARE2, respectively. GWPCARE3 had a total of 225 patients, and GWPCARE4 included 171 in total. In the DS RCTs, patients had used four or five prior AEDs on average, and in the LGS RCTs, they had used six or seven on average.
Both DS patients in GWPCARE2 and LGS patients in GWPCARE3 who received 10 mg/kg/day CBD in addition to CCM achieved better seizure frequency outcomes than those who received CCM plus placebo. Specifically, patients in the 10 mg/kg/day CBD groups experienced fewer convulsive seizures (DS patients), drop seizures (LSG patients) and seizures overall during the 14-week treatment period than those in the placebo group. A small number of patients in the CBD group of GWPCARE2 and in the placebo group achieved freedom from convulsive seizures for the whole 14-week treatment period. However, no patient in GWPCARE3 achieved freedom from drop seizures for the whole 14-week treatment period; three patients in the 10 mg/kg/day CBD group and one patient in the placebo group were drop-seizure–free for the whole maintenance phase (day 15 onwards). Safety data in both DS and LGS appeared to indicate a pattern of gastrointestinal and ‘tiredness’-related adverse events in patients taking CBD, as well as a detrimental effect on markers of liver function. The rates of individual, treatment-related adverse events were generally higher in the 20 mg/kg CBD groups than in the 10 mg/kg CBD groups.
Critique of Clinical Effectiveness Evidence and Interpretation
The submission and response to the request for clarification provided sufficient details for the ERG to appraise most of the literature searches. A range of databases were searched, and additional searches of conference proceedings and trial registers were conducted. Searches were carried out in accordance with the NICE guide to the methods of technology appraisal [21]. Errors and inconsistencies in the original search strategies impaired the performance of the company’s searching. As corrected strategies were not provided in the clarification response, the ERG remained concerned about potentially relevant missed evidence.
Although the CS included two international RCTs and an open-label extension study, there are some limitations in applying this evidence to UK practice. In both DS and LGS, one of the RCTs included very few UK patients and the other had none. This is most likely to be relevant when considering the nature of CCM, which may differ between countries and which is the comparator in the trials.
A major limitation of the evidence is the small size of the data set relating to the recommended 10 mg/kg/day CBD dosage. Just a small number of patients in GWPCARE 2 and GWPCARE3 and none in GWPCARE1 and GWPCARE4 received the 10 mg/kg/day dosage.
A further important limitation is the short-term nature of the RCTs (14 weeks) resulting in a lack of long-term efficacy and safety data, particularly for the 10 mg/kg/day dosage. Data from GWPCARE5 are for patients taking 20 mg/kg/day CBD or higher (up to 30 mg/kg/day). Any observations of reduction in seizures in the short-term trials may not be sustained in the long-term, and the effects on outcomes relating to mortality (especially SUDEP) are unknown.
CCM is considered to be a ‘basket’ of choices of AED. Although the company conducted a number of subgroup analyses based on the presence or absence of various AEDs, they assumed that the effectiveness of CBD does not vary with the combinations of AEDs to which it is added. This assumption is crucial to the validity of the ‘mixed’ CCM comparator. The ERG considers that there is currently a lack of evidence to support this assumption.
The innovation section of the CS emphasised the value to patients and carers of periods of seizure-free time. The ERG notes that neither the CS nor the clincal study reports (CSRs) provided any data on the number of days, if any, on which study participants were seizure-free (no seizures of any type). Also, no patient in any of the included LGS studies and very few patients in the DS studies achieved complete freedom from seizures.
Cost-Effectiveness Evidence Submitted by the Company
The company developed two similar cohort state transition models using Microsoft Excel®. The models consisted of five health states that were based on the drop seizure frequency and the number of drop-seizure–free days (for LGS) and convulsive seizure frequency and the number of convulsive-seizure–free days (for DS). The company assumed that improvements in patients’ quality of life would relate to the total number of convulsive or drop seizures and number of convulsive- or drop-seizure–free days. Therefore, each of the convulsive or drop seizure frequency health states was categorised into three subcategories based on the number of convulsive- or drop-seizure–free days experienced in the corresponding health state. Patients receiving CCM plus CBD could transit between the four convulsive seizure frequency health states for the first nine cycles (i.e. 27 months), after which patients stayed in the same health state for the remaining duration of the 15-year time horizon (with ‘death’ being the fifth health state). Patients receiving CCM without CBD could transit between the convulsive seizure frequency health states during the first cycle only and returned to their baseline convulsive seizure frequency state afterwards (i.e. after 3 months). The model cycle length was 3 months with a 15-year time horizon. In the CS, the base-case analysis utilised the maintenance dosage of 10 mg/kg/day, as the company assumed that the majority of patients would receive this dosage in clinical practice.
Short-term treatment effectiveness was estimated separately for patient subgroups < 12 years and ≥ 12 years based on the pivotal clinical trials specific to DS or LGS (see below). Long-term treatment effectiveness was extrapolated using a constant treatment effect by assuming that CBD patients remain in the same health state until CBD discontinuation or death. Adverse events were based on a pooled analysis considering both the DS and LGS phase III trials (GWPCARE1, GWPCARE2, GWPCARE3 and GWPCARE4).
Health state utilities were estimated based on patient vignettes using a visual analogue scale. Health state utilities were assumed to be treatment dependent due to differences in number of days without drop seizures between CBD and CCM. The impact of adverse events on health-related quality of life was not incorporated in the model. In the revised base case, the company included quality-adjusted life year (QALY) decrements for caregivers and incorporated these gains in the total QALY gain of both CBD and CCM.
The cost categories included in the model were costs associated with treatment (drug acquisition costs included concomitant therapies and costs associated with treatment-related adverse events), health state costs and mortality costs. Unit prices were based on the NHS reference prices, British National Formulary (BNF), Personal Social Services Research Unit (PSSRU) and clinical opinion (NHS reference cost schedule 2016–2017) [22, 23].
The analysis took an NHS and Personal Social Services (PSS) perspective. Discount rates of 3.5% were applied to both costs and benefits. Costs were indexed to the year 2018.
The company performed face validity, internal validity and external validity checks.
Following the EMA licence of CBD, the company provided evidence from a subgroup of the trial population (patients treated also with clobazam). The company provided an additional submission following the first AC meeting (see below).
Main Sources of Evidence on Treatment Effectiveness for LGS
The main sources of evidence on treatment effectiveness were the pivotal clinical trials (GWPCARE3 and GWPCARE4) and the open-label extension study (GWPCARE5). It should be noted that GWPCARE4 was not used in the base-case analysis, only in the scenario analyses that used CBD 20 mg/kg/day. These studies were used to obtain evidence for the frequency of drop seizures, number of days without drop seizures, discontinuation rates and adverse events for both CCM plus CBD and CCM. GWPCARE3 was mainly used to inform treatment effectiveness during cycle 1, while GWPCARE5 (in combination with assumptions) was used for subsequent cycles.
The company’s revised analysis resulted in an incremental cost-effectiveness ratio (ICER) of £31,107 per QALY gained for CBD plus CCM versus CCM.
Main Sources of Evidence on Treatment Effectiveness for DS
The main sources of evidence on treatment effectiveness were the pivotal clinical trials (GWPCARE1 and GWPCARE2) and the open-label extension study (GWPCARE5). It should be noted that GWPCARE1 was not used in the base-case analysis, only in the scenario analyses that used CBD 20 mg/kg/day. These studies were used to obtain evidence for the frequency of convulsive seizures, number of days without convulsive seizures, discontinuation rates and adverse events for both CCM plus CBD and CCM. GWPCARE2 was mainly used to inform treatment effectiveness during cycle 1, while GWPCARE5 (in combination with assumptions) was used for subsequent cycles.
The company’s revised analysis resulted in an ICER of £36,046 per QALY gained for CBD plus CCM versus CCM.
Critique of Cost-Effectiveness Evidence and Interpretation
Errors and inconsistencies in the original search strategies impaired the performance of the company’s searching. However, amended strategies were not provided in response to the ERG’s clarification request, nor reported in enough detail for the ERG to appraise them. As a result, the ERG remained concerned about the quality of the searches used for this submission.
The ERG considered that the economic models and base-case analyses described in both CSs only partly met the NICE reference case. Deviations from the NICE reference case included the restricted time horizon of 15 years and the method used to estimate utilities.
The main concern of the ERG related to the structure of the models was the assumption that patients receiving CCM (not CBD) moved back to their baseline drop seizure frequency after the first cycle. The company clarified that this was done as a placebo effect was observed in both the GWPCARE1 and GWPCARE2 (for DS) and GWPCARE3 and GWPCARE4 (for LGS) studies and argued it was not reasonable to assume that these effects would be sustained in clinical practice. The ERG disagreed with the approach, as a placebo effect may also be present in the CBD group, which the placebo group is designed to control for. The inconsistent handling of this in the model resulted in removing the placebo effect for CCM, while not removing it for CBD, which most likely induces a bias and thus might result in a (substantially) overestimated treatment effect for CBD. Furthermore, on discontinuation from CBD, it was assumed that patients would transit to the seizure frequency distribution as assumed for placebo. This assumption was viewed as particularly problematic because patients discontinue from all health states, but with higher probabilities in the severe health states, and hence, patients’ health states could improve upon CBD discontinuation.
The ERG had multiple concerns related to the extrapolation of treatment effectiveness in both CSs. These include, firstly, extrapolating evidence from GWPCARE5, using CBD 20 mg/kg/day or higher as the maintenance dosage (mean modal dosage during treatment was 23 mg/kg/day for LGS and 21.6 mg/kg/day for DS) to model the effectiveness of CBD 10 mg/kg/day beyond 3 months. It is debatable whether this evidence is representative for a CBD maintenance dosage of 10 mg/kg/day. Secondly, the extrapolation after 27 months is uncertain due to the lack of evidence beyond this time period. After 27 months, the company assumed constant treatment effectiveness, i.e. CBD patients remained in the same health state until CBD discontinuation or death, while a constant CBD discontinuation probability was applied. The uncertainty related to extrapolation is, in part, reflected in the ERG base-case ICER range by using two base cases, one assuming a constant treatment effect after 27 months and the other assuming no treatment effect after 27 months.
Another source of uncertainty was the estimated health state utility values. Utilities were estimated using patient vignettes that were based on the health states included in the model. In total, 39 patient vignettes were developed. Patients and/or caregivers of patients with LGS, DS or other forms of epilepsy were asked to complete a quality-of-life questionnaire and to score patient vignettes using a visual analogue scale. The ERG judged that this approach is condition oriented and does not appropriately capture other aspects known to influence quality of life and generally incorporated into utility estimates (e.g. mobility, ability to self-care, ability to undertake usual activities, pain and discomfort, and anxiety and depression), leaving these aspects to the conceptualisation of the respondents. In addition to the use of methodology (vignette study with suboptimal methodology) that is not in line with the NICE reference case, the (implicit) use of treatment-dependent health state utility values is not considered appropriate by the ERG. Particularly for patients who, after CBD discontinuation, reverted back to their baseline frequency of drop seizures (for LGS) or convulsive seizures (for DS), the treatment benefit (compared with CCM) potentially induced by the difference in number of days without drop/convulsive seizures between the treatments is questionable. Moreover, the inclusion of caregivers’ QALYs seems inconsistent with the NICE reference case, which states that the measurement of changes in health-related quality of life should be reported directly from patients.
The validity and transparency of the models can be regarded as a major limitation of the assessment. Despite the company attempting to resolve validity issues (e.g. estimated QALYs that are larger than the time horizon) during the clarification phase, the ERG still considered the model validity as problematic. This is particularly because the models failed to provide the expected results to internal validity tests performed by the ERG. Of most concern, changing the clinical effectiveness input parameters for CBD 10 mg/kg/day to the clinical effectiveness input parameters for CCM still resulted in a QALY benefit of 0.43 for CBD in the LGS model and a QALY benefit of 0.36 for CBD in the DS model (while 0.00 would be expected). Accordingly, the ERG believed that there were fundamental problems with the economic models (i.e. symmetry of model structure [24]) that potentially induced a QALY gain for CBD. Consequently, the cost-effectiveness results, calculated using the economic models submitted by the company, lacked credibility. Due to the complexity and limited transparency of the model, the ERG was unable to satisfactorily resolve these validation issues within the available timeframe. This was compounded by the company making unrequested changes to complex visual basic code. Due to the validity issues described above, the ERG considered the original CS ICER for LGS or DS (in confidence) as well as the revised base-case ICER submitted by the company for LGS (£31,107 per QALY gained, including QALYs gained by caregivers) or DS (£36,046 per QALY gained, including QALYs gained by caregivers) not credible. The inclusion of caregiver disutilities resulted in a QALY decrement that was greater than that for the patient (i.e. negative total QALYs); which lacked face validity.
Additional Work Undertaken by the ERG
In the company’s probabilistic base case, the ICERs of CBD compared with CCM were based on technically implausible QALY estimates (i.e. the number of QALYs gained was bigger than the time horizon of the model) and were, according to the ERG, not credible. Similarly, the revised base-case ICERs submitted by the company (£31,107 per QALY gained for LGS and £36,046 per QALY gained for DS) were regarded with extreme caution given the highlighted validity issues and adjustments (model structure and input) made by the company. The ERG, therefore, incorporated various adjustments to the original CS base case (using the revised economic model with input parameters from the original CS as starting point). The ERG base case consisted of an ICER range, reflecting the uncertainty surrounding the long-term extrapolation of treatment effectiveness. The probabilistic ERG base case indicated that the ICER for CBD compared with CCM would range between £80,205 per QALY gained (assuming a constant treatment effect after 27 months) and £176,638 per QALY gained (assuming no treatment effect after 27 months) for LGS, and for DS, it would range between £76,013 per QALY gained (assuming a constant treatment effect after 27 months) and £477,476 per QALY gained (assuming no treatment effect after 27 months). However, it should be reiterated that some of the abovementioned potential biases (model structure, validity) could not be explored by the ERG. Consequently, the ICERs reported were considered by the ERG to probably be underestimations of the true ICERs. The main driver of the increased ERG ICER (compared with the company ICER) was the exclusion of carer disutilities in the ERG base case. The ERG excluded these given it seemed not consistent with the NICE reference case and the validity of the vignette study (i.e. to estimate carer disutilities) was questionable.
Conclusion of the ERG report
DS patients in GWPCARE2 who received 10 mg/kg/day CBD in addition to CCM experienced fewer convulsive seizures and fewer seizures overall during the 14-week treatment period than those in the placebo group. Similarly, LGS patients in GWPCARE3 who received 10 mg/kg/day CBD in addition to CCM experienced fewer drop seizures and fewer seizures overall. However, safety data from both DS and LGS RCTs appear to indicate a pattern of gastrointestinal and ‘tiredness’-related adverse events in patients taking CBD, as well as a detrimental effect on markers of liver function. Also, a major limitation of the evidence is the small sample size of patients receiving the recommended 10 mg/kg/day CBD dosage, which is specified as the starting dosage for all patients in the company’s clarification response [25]. A further important limitation is the short-term nature of the RCTs (14 weeks). There is a lack of long-term efficacy and safety data, particularly for the 10 mg/kg/day dosage. Data from the GWPCARE5 extension study [20] are for patients taking 20 mg/kg/day CBD or higher (up to 30 mg/kg/day). Any observations of reduction in seizures in the short-term trials may not be sustained in the long-term, and the effects on outcomes relating to mortality (especially SUDEP) are unknown. The ERG is also concerned that the apparently high rate of withdrawals from GWPCARE5 [20], which were not attributable to adverse events, together with the dose escalation in some patients (up to a maximum of 30 mg/kg), may indicate a loss of efficacy over time. No evidence has been provided to support the long-term efficacy (beyond 14 weeks) of the recommended CBD dosage (10 mg/kg/day).
CCM is considered to be a ‘basket’ of choices of AED. Although the company conducted a number of subgroup analyses based on the presence or absence of various AEDs, they assumed that the effectiveness of CBD does not vary with the combinations of AEDs to which it is added (i.e. that there are no interaction effects between CBD and any of the other AEDs that may be included in CCM). This assumption is crucial to the validity of the ‘mixed’ CCM comparator. The ERG considers that there is currently a lack of evidence to support this assumption.
The innovation section of the CS emphasised the value of periods of seizure-free time to patients and caregivers. The ERG notes that neither the CSs nor the CSRs provided any data on the number of days, if any, on which study participants were seizure-free (no seizures of any type) and that no patient, in any of the included studies, achieved complete freedom from all types of seizures.
Conclusion of the Cost-Effectiveness Section on (Revised) Submission
The structure of the models proposed by the company did not fully capture the natural progression of LGS and DS. The model structures were focused on drop/convulsive seizures and did not explicitly capture non-drop/convulsive seizures. Also, assuming that patients treated with CCM revert to their baseline health states after 3 months (with no possibility to become seizure-free) and remain in this state for the remainder of the time horizon is considered restrictive and potentially biases the cost-effectiveness in favour of CBD. Additionally, the ERG considered that the economic models and base-case analyses described in the CS only partly meet the NICE reference case (i.e. deviations from the NICE reference case included the restricted time horizon of 15 years, the method used to estimate utilities, and the inclusion of carer utilities).
The ERG considered that key uncertainties in this cost-effectiveness assessment were the extrapolation of treatment effectiveness, the estimated health state utility values and the model validity. By extrapolating CBD 20 mg/kg/day evidence to CBD 10 mg/kg/day, one may wonder whether this evidence is representative for a CBD maintenance dosage of 10 mg/kg/day. Moreover, the extrapolation after 27 months is uncertain due to the lack of evidence beyond this time period. After 27 months, the company assumed a constant treatment effectiveness, i.e. assuming that CBD patients remain in the same health state until CBD discontinuation (at a constant rate) or death. This uncertainty is, in part, reflected in the ERG base-case ICER range. Another source of uncertainty was the estimated health state utility values. The ERG considered the methodology to not be in line with the NICE reference case (i.e. both the use of a vignette study and the use of patients to value these vignettes/health states) and the resulting utility values to be questionable (particularly given the high seizure-free utility values relative to the general population utility values in the LGS submission). Finally, the validity of the models (as well as transparency) can be regarded as a major limitation. Although the company attempted to resolve validity issues during the clarification phase, the ERG also considered the model validity of the revised model to be problematic. The ERG considered that there were fundamental problems with the economic model that potentially induced a QALY gain for CBD 10 mg/kg/day. Consequently, the cost-effectiveness results, calculated using the economic models submitted by the company, lacked credibility. Due to the complexity and limited transparency of the model, the ERG was unable to satisfactorily resolve these validation issues within the available timeframe.
Conclusions Based on the Updated Models Following the EMA Licence of CBD
Following the EMA licence of CBD, the company provided evidence from a subgroup of the trial population (patients treated also with clobazam). The company provided an additional submission following the first AC meeting. An updated model was submitted that incorporated several changes in response to the ACD. Baseline data for the new subgroup (with clobazam) seemed slightly unbalanced between the CBD 10 mg/kg/day and placebo arms. However, the ERG considered that the difference was unlikely to have a substantial effect on the treatment effect.
In the updated model, the ERG verified that the company’s model produced a null QALY gain (model symmetry) under the conditions set out by the company. The company, together with a third party, undertook a significant number of internal validity tests, which the model passed. However, there were some remaining concerns about the face validity of model assumptions surrounding the health states that patients return to upon treatment discontinuation. On discontinuation from CBD, it was assumed that patients would transit to the seizure frequency distribution as assumed for placebo (i.e. cycle 2 of the comparator arm). This assumption is viewed as particularly problematic because patients discontinue from all health states but with higher probabilities in the severe health states, and hence, patients’ health states might improve upon CBD discontinuation. Transparency issues included many hidden worksheets, hidden cells, and coding most of the model in visual basic (VBA) code, which may not have been necessary and which had changed substantially in producing the latest version of the model. This hampered the ERG’s ability to thoroughly validate the model and explore some assumptions within the model.