FormalPara Key Points for Decision Makers

The National Institute for Health and Care Excellence (NICE) Appraisal Committee recommended cannabidiol (CBD) 10 mg/kg/day with clobazam as an option for treating seizures associated with Lennox-Gastaut syndrome (LGS) and Dravet syndrome (DS) in people aged 2 years and older.

Internal validity checks require transparency and need to include all possible moves between health states and occurrence of events. A common example of an internal validity problem is patients moving to a ‘better’ health state after treatment discontinuation, possibly caused by a ‘back to baseline’ assumption.

Guidance on when to include caregivers’ quality-adjusted life years and methods for how best to incorporate them in a health economic model are still lacking.

There is a need for greater integration of the NICE single technology appraisal process with the marketing authorisation process, which may provide faster access to innovative treatments for patients.

1 Introduction

In order to be recommended by the National Institute for Health and Care Excellence (NICE) for use within the National Health Service (NHS) in England and Wales, health technologies must be demonstrated to be a clinically effective and cost-effective use of NHS resources. NICE is an independent organisation responsible for providing national guidance on promoting good health by assessing the clinical and cost-effectiveness of health technologies. The NICE single technology appraisal (STA) process typically considers new technologies within a single indication. Following this process, companies provide a written submission including a health economic model regarding the company’s estimates of the clinical effectiveness and cost-effectiveness of the technology. The company submission (CS) is then reviewed by an Evidence Review Group (ERG), which is an independent group of experts that reviews the submission and drafts an ERG report in which the ERG expresses their view on the methodological quality of the submission. Next, based on the CS, the ERG report and testimony from experts and other stakeholders, the NICE Appraisal Committee (AC) formulates and Appraisal Consultation Document (ACD), which contains preliminary recommendations on the new technology. Subsequently, stakeholders are invited to provide feedback on the evidence and ACD, which can result in a new ACD or a Final Appraisal Determination (FAD) that is open for appeal.

As part of this process, two CSs were submitted by GW Research Ltd. in support of cannabidiol (CBD), trade name Epidyolex®: one for the adjunctive treatment of seizures associated with Lennox–Gastaut syndrome (LGS) and one for Dravet syndrome (DS), both rare and severe types of epilepsy.

GW Research Ltd. provided NICE with a written submission, including an executable health economic model, detailing the company’s estimates of the clinical effectiveness and cost-effectiveness of CBD. Kleijnen Systematic Reviews Ltd., in collaboration with Maastricht University Medical Center, was commissioned as the independent ERG.

This paper highlights important methodological issues related to the CSs [1, 2], the ERG reports [3, 4] and the FADs for these two STAs and the subsequent development of the NICE guidance for the use of CBD in England and Wales. Full details of all relevant appraisal documents can be found on the NICE website [5, 6].

2 The Decision Problem

The patient populations described in the final scope specified by NICE are ‘Patients with Lennox-Gastaut syndrome and Dravet syndrome whose seizures are inadequately controlled by established clinical management’ [7, 8].

LGS is a severely debilitating, lifelong and treatment-resistant form of epilepsy affecting two in 10,000 children from 2 years of age [2]. Onset of LGS usually occurs before the age of 8 years, peaking between 3 and 5 years of age [9]. LGS is characterised by the presence of multiple seizure types and frequent seizures including atonic, tonic, atypical absence seizures and myoclonic jerks, an abnormal electroencephalogram (EEG) pattern of slow spike-wave (SSW) complexes, and moderate to severe cognitive impairment [2, 10]. Atonic and tonic seizures result in a temporary loss of muscle tone or stiffening of the muscles, respectively. These sudden drop seizures (defined as an attack or spell involving the entire body, trunk or head that led or could have led to a fall, injury, slumping in a chair or the patient hitting their head on a surface) often result in severe injuries, so patients need to wear helmets with full face masks or use wheelchairs to minimise injuries [9].

DS is a rare, severe form of epilepsy affecting children and adults. DS typically starts in the first year of life with prolonged, repeated clonic or unilateral seizures in developmentally normal children, associated in many instances with a fever. Patients with DS present with different seizure patterns, but most include combinations of severe convulsive seizures, including generalised tonic–clonic and clonic seizures, as well as myoclonic, atypical absence and focal seizures [2, 11].

In patients with LGS and DS, moderate to severe cognitive impairments are common, often accompanied by behavioural problems such as hyperactivity, aggression and autistic traits [12, 13]. Furthermore, the risk of death is significantly elevated in patients with drug-resistant forms of epilepsy, and patients with LGS and DS are at increased risk of sudden unexplained death in epilepsy (SUDEP). Furthermore, despite the availability of a broad range of anti-epileptic drugs (AEDs) and non-pharmacological interventions, seizure control in LGS and DS remains inadequate [12, 13].

NICE issued its final scope in October 2018 for both indications to appraise the clinical and cost-effectiveness of CBD in LGS and DS [7, 8]. At the time of submission, the scope for DS stated that the current relevant treatment options within the NHS included sodium valproate, topiramate, clobazam, stiripentol, levetiracetam, ketogenic diet and vagus nerve stimulation. For LGS, the current relevant treatment options within the NHS included sodium valproate, lamotrigine, rufinamide, topiramate, felbamate, clobazam, levetiracetam, ketogenic diet and vagus nerve stimulation. However, on 26 July 2019, the Committee for Medicinal Products for Human Use adopted a positive opinion and recommended the granting of a marketing authorisation for CBD (Epidyolex, GW Pharma) for use as adjunctive therapy for seizures associated with LGS or DS only in conjunction with clobazam for patients 2 years of age or older [14]. Hence, the population implied only patients eligible for clobazam.

Both the scope and CS were published/submitted before the European Medicines Agency (EMA) licence and, hence, did not specify the necessity of clobazam in conjunction with CBD.

3 Independent ERG Review

In May 2019, the company (GW Research Ltd.) provided a submission to NICE on the clinical and cost-effectiveness of CBD within its anticipated licensed indication for LGS and DS [1, 11]. Following the usual process for STAs, the company provided additional information in response to clarification questions raised by the ERG and NICE [15,16,17]. Additionally, the ERG adjusted the decision analytic model received from the company to assess the impact of alternative parameter values and assumptions on the model results and to produce an ERG base case.

Sections 3.1–3.6 below summarise the evidence presented in the CS, as well as the ERG’s review of that evidence. Given that the two STAs were similar in terms of the evidence provided (e.g. design of clinical trials), health economic model structure and treatment mix, but differed in smaller but important aspects, some sections will discuss general issues applicable to both submissions and issues specific for the CS of LGS and DS separately.

3.1 Clinical Effectiveness Evidence Submitted by the Company

For both DS and LGS, the CSs identified four international randomised controlled trials (RCTs) of CBD (GWPCARE1 and GWPCARE2 for DS, and GWPCARE3 and GWPCARE4 for LGS) and an ongoing open-label extension study (GWPCARE5) as relevant to both submissions [18,19,20]. Both LGS RCTs were conducted in patients aged 2–55 years whose seizures were incompletely controlled with previous AEDs and who had suffered at least two drop seizures per week in the baseline period. Both DS RCTs included patients from 2 to 18 who had to have suffered at least four convulsive seizures in the past 28 days. The intervention was CBD in addition to current clinical management (CCM), and the comparator was CCM without CBD (i.e. CCM plus placebo). GWPCARE2 and GWPCARE3 were three-arm studies and compared two dosages of CBD (10 mg/kg/day and 20 mg/kg/day) in addition to CCM and CCM plus placebo. GWPCARE1 and GWPCARE4 compared CBD (20 mg/kg/day) in addition to CCM and CCM plus placebo. All four RCTs had a dose-escalation phase (14 days in GWPCARE1 and GWPCARE3, 7 or 11 days in GWPCARE 2 and 11 days in GWPCARE4) followed by a 12-week treatment period. GWPCARE 1 and GWPCARE3 included patients from the UK (three centres and 11 and 16 patients, respectively, overall). However, GWPCARE2 and GWPCARE4 did not include patients from the UK. A total of 120 and 198 patients were included in GWPCARE1 and GWPCARE2, respectively. GWPCARE3 had a total of 225 patients, and GWPCARE4 included 171 in total. In the DS RCTs, patients had used four or five prior AEDs on average, and in the LGS RCTs, they had used six or seven on average.

Both DS patients in GWPCARE2 and LGS patients in GWPCARE3 who received 10 mg/kg/day CBD in addition to CCM achieved better seizure frequency outcomes than those who received CCM plus placebo. Specifically, patients in the 10 mg/kg/day CBD groups experienced fewer convulsive seizures (DS patients), drop seizures (LSG patients) and seizures overall during the 14-week treatment period than those in the placebo group. A small number of patients in the CBD group of GWPCARE2 and in the placebo group achieved freedom from convulsive seizures for the whole 14-week treatment period. However, no patient in GWPCARE3 achieved freedom from drop seizures for the whole 14-week treatment period; three patients in the 10 mg/kg/day CBD group and one patient in the placebo group were drop-seizure–free for the whole maintenance phase (day 15 onwards). Safety data in both DS and LGS appeared to indicate a pattern of gastrointestinal and ‘tiredness’-related adverse events in patients taking CBD, as well as a detrimental effect on markers of liver function. The rates of individual, treatment-related adverse events were generally higher in the 20 mg/kg CBD groups than in the 10 mg/kg CBD groups.

3.2 Critique of Clinical Effectiveness Evidence and Interpretation

The submission and response to the request for clarification provided sufficient details for the ERG to appraise most of the literature searches. A range of databases were searched, and additional searches of conference proceedings and trial registers were conducted. Searches were carried out in accordance with the NICE guide to the methods of technology appraisal [21]. Errors and inconsistencies in the original search strategies impaired the performance of the company’s searching. As corrected strategies were not provided in the clarification response, the ERG remained concerned about potentially relevant missed evidence.

Although the CS included two international RCTs and an open-label extension study, there are some limitations in applying this evidence to UK practice. In both DS and LGS, one of the RCTs included very few UK patients and the other had none. This is most likely to be relevant when considering the nature of CCM, which may differ between countries and which is the comparator in the trials.

A major limitation of the evidence is the small size of the data set relating to the recommended 10 mg/kg/day CBD dosage. Just a small number of patients in GWPCARE 2 and GWPCARE3 and none in GWPCARE1 and GWPCARE4 received the 10 mg/kg/day dosage.

A further important limitation is the short-term nature of the RCTs (14 weeks) resulting in a lack of long-term efficacy and safety data, particularly for the 10 mg/kg/day dosage. Data from GWPCARE5 are for patients taking 20 mg/kg/day CBD or higher (up to 30 mg/kg/day). Any observations of reduction in seizures in the short-term trials may not be sustained in the long-term, and the effects on outcomes relating to mortality (especially SUDEP) are unknown.

CCM is considered to be a ‘basket’ of choices of AED. Although the company conducted a number of subgroup analyses based on the presence or absence of various AEDs, they assumed that the effectiveness of CBD does not vary with the combinations of AEDs to which it is added. This assumption is crucial to the validity of the ‘mixed’ CCM comparator. The ERG considers that there is currently a lack of evidence to support this assumption.

The innovation section of the CS emphasised the value to patients and carers of periods of seizure-free time. The ERG notes that neither the CS nor the clincal study reports (CSRs) provided any data on the number of days, if any, on which study participants were seizure-free (no seizures of any type). Also, no patient in any of the included LGS studies and very few patients in the DS studies achieved complete freedom from seizures.

3.3 Cost-Effectiveness Evidence Submitted by the Company

The company developed two similar cohort state transition models using Microsoft Excel®. The models consisted of five health states that were based on the drop seizure frequency and the number of drop-seizure–free days (for LGS) and convulsive seizure frequency and the number of convulsive-seizure–free days (for DS). The company assumed that improvements in patients’ quality of life would relate to the total number of convulsive or drop seizures and number of convulsive- or drop-seizure–free days. Therefore, each of the convulsive or drop seizure frequency health states was categorised into three subcategories based on the number of convulsive- or drop-seizure–free days experienced in the corresponding health state. Patients receiving CCM plus CBD could transit between the four convulsive seizure frequency health states for the first nine cycles (i.e. 27 months), after which patients stayed in the same health state for the remaining duration of the 15-year time horizon (with ‘death’ being the fifth health state). Patients receiving CCM without CBD could transit between the convulsive seizure frequency health states during the first cycle only and returned to their baseline convulsive seizure frequency state afterwards (i.e. after 3 months). The model cycle length was 3 months with a 15-year time horizon. In the CS, the base-case analysis utilised the maintenance dosage of 10 mg/kg/day, as the company assumed that the majority of patients would receive this dosage in clinical practice.

Short-term treatment effectiveness was estimated separately for patient subgroups < 12 years and ≥ 12 years based on the pivotal clinical trials specific to DS or LGS (see below). Long-term treatment effectiveness was extrapolated using a constant treatment effect by assuming that CBD patients remain in the same health state until CBD discontinuation or death. Adverse events were based on a pooled analysis considering both the DS and LGS phase III trials (GWPCARE1, GWPCARE2, GWPCARE3 and GWPCARE4).

Health state utilities were estimated based on patient vignettes using a visual analogue scale. Health state utilities were assumed to be treatment dependent due to differences in number of days without drop seizures between CBD and CCM. The impact of adverse events on health-related quality of life was not incorporated in the model. In the revised base case, the company included quality-adjusted life year (QALY) decrements for caregivers and incorporated these gains in the total QALY gain of both CBD and CCM.

The cost categories included in the model were costs associated with treatment (drug acquisition costs included concomitant therapies and costs associated with treatment-related adverse events), health state costs and mortality costs. Unit prices were based on the NHS reference prices, British National Formulary (BNF), Personal Social Services Research Unit (PSSRU) and clinical opinion (NHS reference cost schedule 2016–2017) [22, 23].

The analysis took an NHS and Personal Social Services (PSS) perspective. Discount rates of 3.5% were applied to both costs and benefits. Costs were indexed to the year 2018.

The company performed face validity, internal validity and external validity checks.

Following the EMA licence of CBD, the company provided evidence from a subgroup of the trial population (patients treated also with clobazam). The company provided an additional submission following the first AC meeting (see below).

3.3.1 Main Sources of Evidence on Treatment Effectiveness for LGS

The main sources of evidence on treatment effectiveness were the pivotal clinical trials (GWPCARE3 and GWPCARE4) and the open-label extension study (GWPCARE5). It should be noted that GWPCARE4 was not used in the base-case analysis, only in the scenario analyses that used CBD 20 mg/kg/day. These studies were used to obtain evidence for the frequency of drop seizures, number of days without drop seizures, discontinuation rates and adverse events for both CCM plus CBD and CCM. GWPCARE3 was mainly used to inform treatment effectiveness during cycle 1, while GWPCARE5 (in combination with assumptions) was used for subsequent cycles.

The company’s revised analysis resulted in an incremental cost-effectiveness ratio (ICER) of £31,107 per QALY gained for CBD plus CCM versus CCM.

3.3.2 Main Sources of Evidence on Treatment Effectiveness for DS

The main sources of evidence on treatment effectiveness were the pivotal clinical trials (GWPCARE1 and GWPCARE2) and the open-label extension study (GWPCARE5). It should be noted that GWPCARE1 was not used in the base-case analysis, only in the scenario analyses that used CBD 20 mg/kg/day. These studies were used to obtain evidence for the frequency of convulsive seizures, number of days without convulsive seizures, discontinuation rates and adverse events for both CCM plus CBD and CCM. GWPCARE2 was mainly used to inform treatment effectiveness during cycle 1, while GWPCARE5 (in combination with assumptions) was used for subsequent cycles.

The company’s revised analysis resulted in an ICER of £36,046 per QALY gained for CBD plus CCM versus CCM.

3.4 Critique of Cost-Effectiveness Evidence and Interpretation

Errors and inconsistencies in the original search strategies impaired the performance of the company’s searching. However, amended strategies were not provided in response to the ERG’s clarification request, nor reported in enough detail for the ERG to appraise them. As a result, the ERG remained concerned about the quality of the searches used for this submission.

The ERG considered that the economic models and base-case analyses described in both CSs only partly met the NICE reference case. Deviations from the NICE reference case included the restricted time horizon of 15 years and the method used to estimate utilities.

The main concern of the ERG related to the structure of the models was the assumption that patients receiving CCM (not CBD) moved back to their baseline drop seizure frequency after the first cycle. The company clarified that this was done as a placebo effect was observed in both the GWPCARE1 and GWPCARE2 (for DS) and GWPCARE3 and GWPCARE4 (for LGS) studies and argued it was not reasonable to assume that these effects would be sustained in clinical practice. The ERG disagreed with the approach, as a placebo effect may also be present in the CBD group, which the placebo group is designed to control for. The inconsistent handling of this in the model resulted in removing the placebo effect for CCM, while not removing it for CBD, which most likely induces a bias and thus might result in a (substantially) overestimated treatment effect for CBD. Furthermore, on discontinuation from CBD, it was assumed that patients would transit to the seizure frequency distribution as assumed for placebo. This assumption was viewed as particularly problematic because patients discontinue from all health states, but with higher probabilities in the severe health states, and hence, patients’ health states could improve upon CBD discontinuation.

The ERG had multiple concerns related to the extrapolation of treatment effectiveness in both CSs. These include, firstly, extrapolating evidence from GWPCARE5, using CBD 20 mg/kg/day or higher as the maintenance dosage (mean modal dosage during treatment was 23 mg/kg/day for LGS and 21.6 mg/kg/day for DS) to model the effectiveness of CBD 10 mg/kg/day beyond 3 months. It is debatable whether this evidence is representative for a CBD maintenance dosage of 10 mg/kg/day. Secondly, the extrapolation after 27 months is uncertain due to the lack of evidence beyond this time period. After 27 months, the company assumed constant treatment effectiveness, i.e. CBD patients remained in the same health state until CBD discontinuation or death, while a constant CBD discontinuation probability was applied. The uncertainty related to extrapolation is, in part, reflected in the ERG base-case ICER range by using two base cases, one assuming a constant treatment effect after 27 months and the other assuming no treatment effect after 27 months.

Another source of uncertainty was the estimated health state utility values. Utilities were estimated using patient vignettes that were based on the health states included in the model. In total, 39 patient vignettes were developed. Patients and/or caregivers of patients with LGS, DS or other forms of epilepsy were asked to complete a quality-of-life questionnaire and to score patient vignettes using a visual analogue scale. The ERG judged that this approach is condition oriented and does not appropriately capture other aspects known to influence quality of life and generally incorporated into utility estimates (e.g. mobility, ability to self-care, ability to undertake usual activities, pain and discomfort, and anxiety and depression), leaving these aspects to the conceptualisation of the respondents. In addition to the use of methodology (vignette study with suboptimal methodology) that is not in line with the NICE reference case, the (implicit) use of treatment-dependent health state utility values is not considered appropriate by the ERG. Particularly for patients who, after CBD discontinuation, reverted back to their baseline frequency of drop seizures (for LGS) or convulsive seizures (for DS), the treatment benefit (compared with CCM) potentially induced by the difference in number of days without drop/convulsive seizures between the treatments is questionable. Moreover, the inclusion of caregivers’ QALYs seems inconsistent with the NICE reference case, which states that the measurement of changes in health-related quality of life should be reported directly from patients.

The validity and transparency of the models can be regarded as a major limitation of the assessment. Despite the company attempting to resolve validity issues (e.g. estimated QALYs that are larger than the time horizon) during the clarification phase, the ERG still considered the model validity as problematic. This is particularly because the models failed to provide the expected results to internal validity tests performed by the ERG. Of most concern, changing the clinical effectiveness input parameters for CBD 10 mg/kg/day to the clinical effectiveness input parameters for CCM still resulted in a QALY benefit of 0.43 for CBD in the LGS model and a QALY benefit of 0.36 for CBD in the DS model (while 0.00 would be expected). Accordingly, the ERG believed that there were fundamental problems with the economic models (i.e. symmetry of model structure [24]) that potentially induced a QALY gain for CBD. Consequently, the cost-effectiveness results, calculated using the economic models submitted by the company, lacked credibility. Due to the complexity and limited transparency of the model, the ERG was unable to satisfactorily resolve these validation issues within the available timeframe. This was compounded by the company making unrequested changes to complex visual basic code. Due to the validity issues described above, the ERG considered the original CS ICER for LGS or DS (in confidence) as well as the revised base-case ICER submitted by the company for LGS (£31,107 per QALY gained, including QALYs gained by caregivers) or DS (£36,046 per QALY gained, including QALYs gained by caregivers) not credible. The inclusion of caregiver disutilities resulted in a QALY decrement that was greater than that for the patient (i.e. negative total QALYs); which lacked face validity.

3.5 Additional Work Undertaken by the ERG

In the company’s probabilistic base case, the ICERs of CBD compared with CCM were based on technically implausible QALY estimates (i.e. the number of QALYs gained was bigger than the time horizon of the model) and were, according to the ERG, not credible. Similarly, the revised base-case ICERs submitted by the company (£31,107 per QALY gained for LGS and £36,046 per QALY gained for DS) were regarded with extreme caution given the highlighted validity issues and adjustments (model structure and input) made by the company. The ERG, therefore, incorporated various adjustments to the original CS base case (using the revised economic model with input parameters from the original CS as starting point). The ERG base case consisted of an ICER range, reflecting the uncertainty surrounding the long-term extrapolation of treatment effectiveness. The probabilistic ERG base case indicated that the ICER for CBD compared with CCM would range between £80,205 per QALY gained (assuming a constant treatment effect after 27 months) and £176,638 per QALY gained (assuming no treatment effect after 27 months) for LGS, and for DS, it would range between £76,013 per QALY gained (assuming a constant treatment effect after 27 months) and £477,476 per QALY gained (assuming no treatment effect after 27 months). However, it should be reiterated that some of the abovementioned potential biases (model structure, validity) could not be explored by the ERG. Consequently, the ICERs reported were considered by the ERG to probably be underestimations of the true ICERs. The main driver of the increased ERG ICER (compared with the company ICER) was the exclusion of carer disutilities in the ERG base case. The ERG excluded these given it seemed not consistent with the NICE reference case and the validity of the vignette study (i.e. to estimate carer disutilities) was questionable.

3.6 Conclusion of the ERG report

DS patients in GWPCARE2 who received 10 mg/kg/day CBD in addition to CCM experienced fewer convulsive seizures and fewer seizures overall during the 14-week treatment period than those in the placebo group. Similarly, LGS patients in GWPCARE3 who received 10 mg/kg/day CBD in addition to CCM experienced fewer drop seizures and fewer seizures overall. However, safety data from both DS and LGS RCTs appear to indicate a pattern of gastrointestinal and ‘tiredness’-related adverse events in patients taking CBD, as well as a detrimental effect on markers of liver function. Also, a major limitation of the evidence is the small sample size of patients receiving the recommended 10 mg/kg/day CBD dosage, which is specified as the starting dosage for all patients in the company’s clarification response [25]. A further important limitation is the short-term nature of the RCTs (14 weeks). There is a lack of long-term efficacy and safety data, particularly for the 10 mg/kg/day dosage. Data from the GWPCARE5 extension study [20] are for patients taking 20 mg/kg/day CBD or higher (up to 30 mg/kg/day). Any observations of reduction in seizures in the short-term trials may not be sustained in the long-term, and the effects on outcomes relating to mortality (especially SUDEP) are unknown. The ERG is also concerned that the apparently high rate of withdrawals from GWPCARE5 [20], which were not attributable to adverse events, together with the dose escalation in some patients (up to a maximum of 30 mg/kg), may indicate a loss of efficacy over time. No evidence has been provided to support the long-term efficacy (beyond 14 weeks) of the recommended CBD dosage (10 mg/kg/day).

CCM is considered to be a ‘basket’ of choices of AED. Although the company conducted a number of subgroup analyses based on the presence or absence of various AEDs, they assumed that the effectiveness of CBD does not vary with the combinations of AEDs to which it is added (i.e. that there are no interaction effects between CBD and any of the other AEDs that may be included in CCM). This assumption is crucial to the validity of the ‘mixed’ CCM comparator. The ERG considers that there is currently a lack of evidence to support this assumption.

The innovation section of the CS emphasised the value of periods of seizure-free time to patients and caregivers. The ERG notes that neither the CSs nor the CSRs provided any data on the number of days, if any, on which study participants were seizure-free (no seizures of any type) and that no patient, in any of the included studies, achieved complete freedom from all types of seizures.

3.6.1 Conclusion of the Cost-Effectiveness Section on (Revised) Submission

The structure of the models proposed by the company did not fully capture the natural progression of LGS and DS. The model structures were focused on drop/convulsive seizures and did not explicitly capture non-drop/convulsive seizures. Also, assuming that patients treated with CCM revert to their baseline health states after 3 months (with no possibility to become seizure-free) and remain in this state for the remainder of the time horizon is considered restrictive and potentially biases the cost-effectiveness in favour of CBD. Additionally, the ERG considered that the economic models and base-case analyses described in the CS only partly meet the NICE reference case (i.e. deviations from the NICE reference case included the restricted time horizon of 15 years, the method used to estimate utilities, and the inclusion of carer utilities).

The ERG considered that key uncertainties in this cost-effectiveness assessment were the extrapolation of treatment effectiveness, the estimated health state utility values and the model validity. By extrapolating CBD 20 mg/kg/day evidence to CBD 10 mg/kg/day, one may wonder whether this evidence is representative for a CBD maintenance dosage of 10 mg/kg/day. Moreover, the extrapolation after 27 months is uncertain due to the lack of evidence beyond this time period. After 27 months, the company assumed a constant treatment effectiveness, i.e. assuming that CBD patients remain in the same health state until CBD discontinuation (at a constant rate) or death. This uncertainty is, in part, reflected in the ERG base-case ICER range. Another source of uncertainty was the estimated health state utility values. The ERG considered the methodology to not be in line with the NICE reference case (i.e. both the use of a vignette study and the use of patients to value these vignettes/health states) and the resulting utility values to be questionable (particularly given the high seizure-free utility values relative to the general population utility values in the LGS submission). Finally, the validity of the models (as well as transparency) can be regarded as a major limitation. Although the company attempted to resolve validity issues during the clarification phase, the ERG also considered the model validity of the revised model to be problematic. The ERG considered that there were fundamental problems with the economic model that potentially induced a QALY gain for CBD 10 mg/kg/day. Consequently, the cost-effectiveness results, calculated using the economic models submitted by the company, lacked credibility. Due to the complexity and limited transparency of the model, the ERG was unable to satisfactorily resolve these validation issues within the available timeframe.

3.6.2 Conclusions Based on the Updated Models Following the EMA Licence of CBD

Following the EMA licence of CBD, the company provided evidence from a subgroup of the trial population (patients treated also with clobazam). The company provided an additional submission following the first AC meeting. An updated model was submitted that incorporated several changes in response to the ACD. Baseline data for the new subgroup (with clobazam) seemed slightly unbalanced between the CBD 10 mg/kg/day and placebo arms. However, the ERG considered that the difference was unlikely to have a substantial effect on the treatment effect.

In the updated model, the ERG verified that the company’s model produced a null QALY gain (model symmetry) under the conditions set out by the company. The company, together with a third party, undertook a significant number of internal validity tests, which the model passed. However, there were some remaining concerns about the face validity of model assumptions surrounding the health states that patients return to upon treatment discontinuation. On discontinuation from CBD, it was assumed that patients would transit to the seizure frequency distribution as assumed for placebo (i.e. cycle 2 of the comparator arm). This assumption is viewed as particularly problematic because patients discontinue from all health states but with higher probabilities in the severe health states, and hence, patients’ health states might improve upon CBD discontinuation. Transparency issues included many hidden worksheets, hidden cells, and coding most of the model in visual basic (VBA) code, which may not have been necessary and which had changed substantially in producing the latest version of the model. This hampered the ERG’s ability to thoroughly validate the model and explore some assumptions within the model.

4 National Institute for Health and Care Excellence Guidance

As CBD is first in class in both populations (patients with LGS or DS), both STAs were submitted to NICE by assuming a proposed licensed indication for CBD while still awaiting marketing authorisation. As a consequence, the modelled intervention was not in line with the final licensed indication proposed by the Committee for Medicinal Products for Human Use who recommended the granting of a marketing authorisation for CBD for use as ‘adjunctive therapy for seizures associated with LGS or DS in conjunction with clobazam’. In both STAs, clobazam was not specifically included in the intervention arm. This resulted in an initial negative recommendation for standard use of CBD in the NHS, as the cost-effectiveness estimates for CBD with clobazam compared with usual care were very uncertain. However, after a second AC meeting following the EMA licence of CBD, in November 2019, NICE recommended CBD with clobazam as an option for treating seizures associated with LGS and DS in patients aged 2 years and older only if (1) the frequency of drop seizures is checked every 6 months and CBD is stopped if the frequency is not reduced by at least 30% compared with the 6 months before starting treatment and (2) the company provides CBD according to the commercial arrangement (simple discount patient access scheme) [5, 6].

4.1 Consideration of Clinical Effectiveness

The AC concluded that clinical trials showed that CBD reduced the number of convulsive seizures for DS, the number of drop seizures for LGS and the total number of seizures for both conditions, when compared with usual care. The AC noted that the patient, caregiver, and clinical experts agreed that current treatments often do not control seizures associated with DS and LGS. Furthermore, it was noted that there is an unmet need for an intervention that effectively reduces seizures without markedly increasing adverse events. The AC was aware that GWPCARE2 and GWPCARE3 also included a 20 mg/kg/day arm, and that the EMA concluded that there was no consistent difference in dose response between 10 mg/kg/day and 20 mg/kg/day. In the end, the AC concluded that CBD with clobazam reduces seizure frequency compared with usual care, but that the long-term efficacy after 3 years is uncertain. As the trial results demonstrated that a large proportion of patients having CBD with clobazam had adverse events, the AC concluded that while CBD’s adverse effects are mostly manageable, they are an important consideration when making decisions about whether to start or continue CBD. In the absence of a stopping rule in the marketing authorisation for CBD, NHS England proposed during the technical engagement stage of the appraisal that CBD should be stopped if the frequency of drop seizures has not reduced by at least 30% from baseline. The company also included a stopping rule in their model at 12 or 24 months; however, the AC concluded a stopping rule as proposed by NHS England is appropriate, and that response to treatment, defined by reduction in drop seizures compared with the 6 months before starting CBD, should be assessed every 6 months.

4.2 Considerations of Cost-Effectiveness

The AC concluded that the cost-effectiveness estimates for CBD were uncertain due to some of the assumptions in the company’s model. Furthermore, the AC stated that the cost-effectiveness estimates did not include the benefits of (1) reducing the number of non-drop/non-convulsive seizures; (2) improving the quality of life of the siblings of patients with LGS and DS; and, only for DS, reducing the duration of convulsive seizures. The AC stated that, when taking both the uncertainties and the uncaptured benefits into account, CBD was considered an appropriate use of NHS resources.

The company’s updated cost-effectiveness analyses included most of the committee’s preferred assumptions: (1) using narrower seizure frequency ranges for the health states; (2) removing the effect of non-drop/non-convulsive seizures as calculated; (3) using the mean weight instead of the median to calculate dose; (4) accounting for waning of CBD effects; (5) not assuming that CBD lengthens life; (6) using an average dosage of 12 mg/kg/day; and (7) including health-related quality-of-life effects for 1.8 carers instead of two caregivers (partly given that the committee considered that the company’s method of linearly multiplying the disutility values was inappropriate and that not all patients will have two caregivers). This resulted in an incremental ICER of £33,721 per QALY gained for LGS and an ICER of £32,471 per QALY gained for DS. These analyses did not consider the committee’s preference for stopping rules to be applied at 18 months rather than 24 months. It also recalled that there was additional uncertainty in the cost-effectiveness results because of the company’s assumptions around patients who stop treatment with CBD and because the way the company modelled a waning of treatment effect did not capture all the effects on quality of life of efficacy diminishing over time. The committee concluded, however, that the cumulative effect of addressing these uncertainties was likely to have increased the ICER.

5 Key Methodological Issues

Some methodological issues were highlighted during the appraisal/process of both STAs. As a consequence of assuming a proposed licensed indication for CBD while still awaiting marketing authorisation, an initial negative recommendation for standard use of CBD in the NHS was given by the AC, as the cost-effectiveness estimates for CBD with clobazam compared with usual care were very uncertain. This led to a suboptimal appraisal process and may stress the need for greater integration with the process of obtaining marketing authorisation. This may provide faster access to innovative treatments for patients.

Next, the company included QALY decrements by caregivers dependent on seizure frequency and days without seizures and incorporated these gains in the total QALY calculation for both treatment arms. However, the ERG was not certain that caregivers’ QALYs should have been considered given the NICE’s Reference Case for the Methods of Technology Appraisal. Moreover, the inclusion of caregivers’ QALYs seemed not in accordance with the NICE reference case, as the NICE reference case states that ‘the measurement of changes in health-related quality of life should be reported directly from patients and the utility of these changes should be based on public preferences using a choice-based method’ [21]. Given the impact of caregivers’ QALYs in this submission, we believe it is important to gain more insight into the relevance of caregivers’ QALYs and when and how they should be included in STAs. Moreover, a recent report by the decision support unit commissioned by NICE also stated that it is unclear when and how carer health effects should be included in economic evaluations [26]. In this case, once QALY decrements were estimated for a single caregiver, the total QALY decrement for multiple carers was assumed to be additive (as opposed to multiplicative). This assumption was later also questioned by the committee.

Another issue that was highlighted in these STAs was the importance of model symmetry. The assumption of model symmetry, i.e. modelling costs and effects in the same way for both treatment arms {recommended as an aspect of good modelling practice in International Society for Pharmacoeconomics and Outcomes Research (ISPOR) Taskforce papers [24]}, may seem straightforward, but in complex models (i.e. in terms of structure or the way they are coded) or models which lack transparency, it can be hard to check whether this assumption holds true. Both models demonstrated CBD benefits even if it was assumed that all patients would remain in their baseline seizure frequency health state and/or if all input parameters were assumed to be equal for both the intervention and the comparator. This suggested that there were fundamental problems with the model implementation (i.e. VBA code) that induced model asymmetry. Related to this point is the way the company dealt with the placebo effect. The effectiveness of both arms was assessed separately, but the company adjusted down the risk in the comparator (‘placebo’) only arm by assuming that patients would revert back to baseline seizure frequency to account for a possible placebo effect. However, whenever two treatments are being compared, a placebo effect is likely to occur in both treatments. Adjusting for this effect in one arm but not in the other means that the observed effect in the non-adjusted arm is artificially amplified.

6 Conclusions

This paper described the STAs considering CBD for the treatment of LGS and DS. The ERG had multiple reservations regarding the approach to estimating cost-effectiveness. This was acknowledged by the AC, who stated that the cost-effectiveness estimates for CBD were uncertain due to some of the assumptions in the company’s model. In conclusion, the AC stated that, when taking both the uncertainties and the uncaptured benefits into account, CBD was considered an appropriate use of NHS resources. Hence, the AC recommended the use of CBD with clobazam as an option for treating seizures associated with LGS and DS in patients aged 2 years and older. This appraisal highlights the need for transparency and basic internal validity checks of health economic models. Moreover, it raised the issue of when it is important to consider caregivers’ QALYs and how best to incorporate these outcomes in a health economic model. Lastly, this submission emphasised the need for greater integration with the process of obtaining marketing authorisation and the NICE STA, as this may further optimise the process and thereby provide faster access to innovative treatments for patients.