FormalPara Key Points for Decision Makers

When uncertainties regarding the clinical evidence have been too great for National Institute for Health and Care Excellence (NICE) to recommend routine commissioning, managed access agreements have allowed patients to be treated while additional data are collected.

Immature survival data are an important source of clinical uncertainty, which has largely been addressed by later follow-up of patients in clinical trials rather than by additional real-world data.

Systemic Anti-Cancer Therapy data (an important English source of real-world data) have been used to address a limited number of clinical uncertainties.

1 Introduction

New oncology drugs receive special treatment in England. Since January 2009, differential valuation of the health benefits of many cancer drugs has been implemented by adopting a higher cost-effectiveness threshold for life-extending, end-of-life treatments within the National Health Service (NHS) [1]. In 2010–2011, the Cancer Drugs Fund (CDF) was introduced to provide cancer patients in England with access to drugs that either had not been appraised by the National Institute for Health and Care Excellence (NICE) or had not been recommended for routine commissioning [2]. In the original model of the CDF, there was an absence of clear entry and exit criteria for drugs. This created unsustainable financial pressure without evidence of patient benefit [3,4,5]. In 2016, the CDF was revised to provide a more sustainable approach to funding promising new drugs and to collecting additional clinical data [6].

Since the reform of the CDF (from here 2016 CDF), all new oncology drugs are appraised by NICE. The 2016 CDF offers a mechanism for conditional approval. Figure 1 shows possible NICE recommendation options. If uncertainties regarding a drug are too great for it to be recommended for routine commissioning, a recommendation for use within the CDF can be considered [7]. The appraisal committee uses the criteria in Fig. 2 to decide which drugs are eligible to be used within the 2016 CDF [5]. One of these is whether the clinical uncertainty can be addressed with additional data collected while the drug is provided through the CDF. If the appraisal committee recommends use within the CDF, a data collection arrangement (DCA) working group is formed, with representation from NICE and NHS England. The DCA working group reviews the data collection proposal to translate the committee’s uncertainties related to clinical outcome into defined data collection questions [8]. Additional data are collected in line with the DCA and form the basis for the review appraisal of the case for routine commissioning, which is expected to happen normally within 2 years [9].

Fig. 1
figure 1

Managed access scheme for new cancer drugs in 2016 CDF. CDF Cancer Drugs Fund, NHS National Health Service, NICE National Institute for Health and Care Excellence

Full size image
Fig. 2
figure 2

2016 CDF entry criteria

Full size image

During this period, more evidence would be collected on the clinical effectiveness of the drug to resolve the key areas of uncertainty. The CDF review appraisal considers the data that have become available since the original appraisal, together with any change to the patient access scheme or commercial access arrangement proposed by the company. However, changes to the scope of the appraisal such as the population and comparators are not considered during CDF reviews [10]. There are two main options for data collection, ongoing and new clinical trials and real-world data (RWD) from the Systemic Anti-Cancer Therapy (SACT) dataset [8]. Other established cancer registries are also potential data sources for further review.

When introducing the 2016 CDF, a role for RWD, particularly SACT data, was highlighted [8]. SACT data are routinely collected for patients receiving anti-cancer therapies from NHS England providers, as a mandatory collection under the responsibility of Public Health England (PHE) [11]. The SACT dataset is preferred for any data collection of routine chemotherapy practice in England because the infrastructure (including data protection and information governance) is already established, data are already being collected and progress can easily be monitored [8]. PHE, through a cancer data partnership with the NHS, initially reported SACT data to support the re-appraisal of treatments provided by the 2016 CDF. NHS Digital took over this responsibility from PHE, on 1 October 2021, when the latter was replaced by the United Kingdom (UK) Health Security Agency and the Office for Health Improvement and Disparities.

The additional data collection is expected to address areas of uncertainty highlighted by the appraisal committee. In 2020, PHE indicated that, “Real-world data reported by PHE is the primary information used to answer NICE uncertainty for 25% of CDF treatments” [12]. This report did not say how RWD had been used as primary evidence to address the uncertainty issues. Understanding how such data are used is more important than simply counting appraisals reporting SACT data. Moreover, given the increasing interest in RWD, 25% of CDF treatments is a relatively low proportion, and the reasons for this low utilisation of RWD need to be reviewed.

Managed access agreements (MAAs) give opportunities to gather additional evidence that could help to reduce uncertainty when making a final decision. A review of the 24 CDF review appraisals completed to date can document the extent to which this objective has been met by collecting RWD. Moreover, it can identify the challenges and opportunities for use of RWD by NICE. It is timely to review experience with the 2016 CDF, because 24 drugs have now completed their re-appraisal, and a broadly similar fund entitled the Innovative Medicines Fund (IMF) has recently been introduced. This paper reviews the committee’s recommendations following re-appraisal in order to obtain insight into the performance of the 2016 CDF. It focuses particularly on the uncertainties that led to drugs being provided through the CDF and on how clinical and cost-effectiveness evidence considered at the re-appraisal differed from that in the original appraisal.

We find that re-appraisals have largely resulted in recommendations for the routine commissioning of these drugs, which might suggest “Don’t think twice, it’s all right,” as a maxim for these decision-makers. However, a detailed review of each re-appraisal indicates quite limited success in reducing the uncertainties which led to these drugs not being recommended for routine commissioning in the original appraisal. It also highlights the relative importance of longer follow-up of trial participants, compared to the contribution of RWD, in addressing some of the original uncertainties. Among different types of additionally collected data, a particular focus of this study is on the use of SACT data, which was highlighted when the 2016 CDF was introduced.

2 Methods

NICE technology appraisals (TAs) (https://www.nice.org.uk/guidance) were determined to be eligible if they met the following criteria: (1) the drug was provided for the specific indication through the 2016 CDF following an MAA made between NHS England and the manufacturer and (2) the NICE CDF review appraisal had been completed before 16 August 2022. As a result, 24 appraisals were identified for this review. The terminated appraisal (TA674 pembrolizumab) was included in this review. In this appraisal, the company decided not to make a case after the CDF review started, and consequently, there was sufficient data available to include it in the review.

Data were extracted following a protocol developed to extract information about how RWD have been used in NICE appraisals of oncology medicines [13]. This protocol enables a more comprehensive understanding of the use of RWD in CDF review appraisals by identifying non-parametric and parametric use of RWD in both the base-case and sensitivity analyses. Parametric use of RWD is where such data provide the numerical value of a specific variable in the economic model, whereas non-parametric use is where the data are used to develop the model structure or to support, corroborate or validate assumptions and/or choice of data used to parameterise the model. The distinction is made to facilitate more consistent and comprehensive data extraction and to provide a means of measuring the intensity of the use of RWD in an appraisal.

As this data extraction tool was developed for a more general purpose, a few additional variables were required for the specific purposes of this study (Appendix 1, see the electronic supplementary material). These variables capture references to additional data, especially the SACT data, in the CDF review appraisals and the uncertainties identified in the original appraisals and in the review appraisals. Information about uncertainties was extracted from the Final Appraisal Determinations (FADs) in both the original and CDF review appraisals. Uncertainties were classified as either a “key uncertainty” or “other uncertainty”, following Morrell et al., who reviewed the common types of uncertainty addressed in appraisals of drugs that entered the original CDF and discussed the potential for RWD to resolve these uncertainties [14]. If an uncertainty was described in a section heading or highlighted in the conclusion or in the CDF consideration, this uncertainty was considered as a “key uncertainty”. Any other uncertainty addressed across the appraisal was recorded as “other uncertainty”. The uncertainty in CDF review appraisals was reviewed to assess how much additional data helped to reduce uncertainty. Three categories were used (still uncertain, uncertainty resolved, newly added uncertainty) by comparing the FADs from the original and the subsequent appraisal. Any comments about uncertainty made by the committee were recorded. Given that CDF review appraisals highlighted resolving uncertainty identified in the original appraisals, remaining uncertainties were usually addressed in review appraisals. If an uncertainty was not mentioned in the FAD of the review appraisal, it was classified as “resolved”.

The original and CDF review appraisals were compared in terms of the data used, with particular emphasis on where the additional data came from to address the originally identified uncertainties. RWD were of particular interest because one of the arguments for having MAAs was that they provided opportunities to collect additional data, particularly from routine clinical use of the drug. Data were extracted from the main appraisal documents (final scope, company submission, evidence review group [ERG] report, and FAD). Although most evidence used in decision-making was available in these documents, some parts of the evidence in CDF review appraisals were not fully described. When the assumptions made in original appraisals were followed in CDF reviews, the evidence for these assumptions were not fully described in review appraisals. This was often the case with resource use. In this research, evidence not mentioned in any of the four main documents of the CDF review was assumed to be the same as that in the original appraisal. While this is a reasonable assumption, without access to the underlying economic evaluation models, it cannot be guaranteed that the evidence has not changed. Since this research was restricted to the analysis of data in the public domain, this was a potential limitation.

Another research question was whether the pattern of use of RWD changes or not. While drugs were provided through the CDF, companies could collect their own RWD. Additionally collected data could be used not only to reduce uncertainty but also to support their models with more recent evidence. While it might be anticipated that provision through the CDF would increase the opportunities to use RWD in assessing cost-effectiveness, it was possible that the availability of additional trial data reduced reliance on RWD. Hence, the pattern and intensity of use of RWD were reviewed to see whether these changed over the CDF process. Following the data extraction protocol, patterns were identified from both original and review appraisals. Use of RWD in three specific components of an economic evaluation was defined as major use of RWD (use of RWD in estimating overall survival [OS] for either intervention and comparators, volume of treatment for either intervention and comparators and the choice of comparators). These components are likely to have a major impact on the estimated incremental cost-effectiveness ratio. Along with reviewing use of additional data in addressing identified uncertainties, this pattern and intensity review can give a more comprehensive picture of how NICE has used newly collected RWD in CDF reviews.

3 Results

3.1 CDF Review Recommendations

The recommendations made by the committee, reported in Table 1, were unchanged following re-appraisal in 18 cases. In three further re-appraisals, changes were minor. In the case of atezolizumab (TA739), the change was as a result of a changed marketing authorisation, and in two nivolumab appraisals (TA655 and TA713), guidance was further optimised by requiring no prior programmed cell death protein 1 (PD-1) or programmed death-ligand 1 (PD-L1) inhibitor treatment (reflecting changes in clinical practice). There were three cases where the treatments were not recommended for routine commissioning (TA674, TA692 and TA795). Thus, 87.5% of re-appraisals resulted in a recommendation that the treatment be routinely commissioned.

Table 1 Summary of characteristics of CDF review appraisals in this review
Full size table

3.2 Key Uncertainties Addressed in the Original Appraisals

The uncertainties reported in the FADs of the original appraisals are shown in Table 1. Immature survival data in the clinical trials were identified as a common source of uncertainty in most appraisals (67% of appraisals as key uncertainty; 10% of appraisals as other uncertainty). These data increased uncertainty around the size of clinical benefits or long-term benefits. In ten appraisals (42%), an indirect treatment comparison was a source of uncertainty when assessing the clinical benefits. Among them, six appraisals identified this as a key uncertainty. Indirect treatment comparisons were made because of an absence of randomised clinical trials (RCTs) or because relevant comparators were not included in a single RCT. Another source of uncertainty was how clinical effectiveness varied across subgroups defined by the expression of PD-L1. Duration of treatment effect, time on treatment and health-related quality of life (HRQoL) values were frequently noted as sources of uncertainty, but were not identified as key uncertainties in many appraisals.

3.3 Use of Additional Data in Economic Evaluation in CDF Reviews

3.3.1 Additional Data from Clinical Trials

The average time gap between the publication of the FADs for the original appraisal and the CDF review was 35.6 months (median 36 months). The main evidence for economic evaluation in CDF review appraisals was from clinical trials. The additional data in 17 CDF review appraisals came from further follow up of patients in the trials featured in the original appraisal. Two CDF review appraisals (TA674, TA739) used data from clinical trials that were not presented in the original appraisals. Another appraisal (TA653) used both later follow-up of a trial and new clinical trial data. Three appraisals (TA524, TA795, TA796) used SACT data along with previously used data as the main evidence for the economic evaluation model.

Since the information about median follow-up was redacted in a few appraisals, five appraisals were excluded to estimate the increase in the duration of follow-up. The average increase in median follow-up between the original appraisal and the CDF review appraisal was 22.2 months (median 22.2 months). The longest increase in median follow-up was 50 months in the appraisal of niraparib (TA784). Two review appraisals (TA629, TA770) reported increases in median follow-up of 6 months.

3.3.2 Additional Real-World Data

SACT data were the predominant type of RWD used in CDF review appraisals. The use of SACT is reviewed in a separate section below. Here, RWD, other than SACT data are reviewed. There were three cases where RWD other than SACT data were used in the CDF review, but not in the original appraisal. In the CDF review appraisal of pembrolizumab (TA766), the company used the registry data (the Surveillance, Epidemiology and End Results [SEER] and American Joint Committee on Cancer [AJCC]) as well as SACT data to validate the survival distribution. This appraisal also used the market share data for subsequent treatment lines in a scenario analysis. The appraisal of niraparib (TA784) used a chart review study for clinical outcomes with the comparator, routine surveillance data, in a scenario analysis. One of the uncertainties in the original appraisal derived from an indirect treatment comparison. The company used RWD in a scenario analysis to investigate the uncertainty around the indirect comparison, whereas they used data from another clinical trial for the base-case analysis. The appraisal of cemiplimab (TA802) used a new retrospective chart review study for comparative evidence as the lack of comparative evidence was highlighted during the original appraisal. However, the comparative effectiveness of cemiplimab remained highly uncertain due to the chart review lacking validity.

The CDF review appraisal of pembrolizumab (TA770) stopped using RWD when extrapolating OS. The company’s original model was criticised due to missing information about the second-line treatments. In the review appraisal, the company dropped these data and used more recent clinical trial data. The appraisal of ibrutinib (TA795) substituted the RWD used in the original appraisal with UK-based registry data, to help address the data gap (progression-free survival [PFS]), which SACT could not provide. Also, these data were used to estimate a rate of pre-progression mortality in a scenario analysis, which was one of the key uncertainties in the original appraisal.

Patterns of use of RWD in the original appraisals and in the CDF review appraisals were compared (Appendix 2, see the electronic supplementary material). Although there were changes in use of RWD, limited use was made of RWD collected during the CDF period. Substantial changes in patterns of RWD use were not found. Consequently, the intensity of the use of RWD has not changed. In the CDF review of pembrolizumab (TA766), RWD were used more broadly for supporting diverse assumptions in the model such as validating survival extrapolation, informing subsequent treatment line and baseline age of population in the model. However, the intensity of use of RWD has not changed much as only one additional component (volume of subsequent treatment) was informed by RWD (in this case SACT data).

3.4 Use of SACT Data in CDF Review Economic Evaluations

This study focused on the use of SACT data in CDF review appraisals. SACT data were the most commonly used form of RWD in CDF reviews. Since data collection via SACT was a part of the MAAs, the primary source of additional RWD was substantially the SACT database. Although the SACT dataset was the major vehicle to collect RWD, its overall use was limited. SACT data were not used to update the economic evaluation model in nine out of 24 CDF review appraisals (Table 2). The remaining 15 appraisals made limited use of SACT data. SACT data, newly collected from CDF patients, were used more for non-parametric purposes (11 appraisals), such as validation or corroboration of the model, than for parametric purposes (five appraisals). SACT data featured in both non-parametric and parametric uses in four appraisals (TA766, TA783, TA795, TA796).

Table 2 Summary of use of SACT data in CDF review appraisals
Full size table

3.4.1 Parametric Use

Five cases of parametric use were identified. In the CDF review of brentuximab vedotin (TA524), the company used CDF data to inform the rate of subsequent stem cell transplant following treatment with brentuximab vedotin. This was one of the key clinical uncertainties, which was expected to be resolved during the CDF period. A questionnaire sent to consultants identified the rates of stem cell transplant in patients who had brentuximab vedotin as part of the original CDF between April 2013 and March 2016. Another example was the CDF review appraisal of pembrolizumab (TA766). The company used SACT data for the distribution of subsequent treatments administered in the advanced setting for patients in the adjuvant pembrolizumab arm as clinical evidence was incomplete and SACT data were the best available RWD to reflect the clinical practice observed in the CDF. Similar to TA766, in the CDF review appraisal of daratumumab (TA783), SACT data were used to inform subsequent therapies for all comparators. The appraisal of ibrutinib (TA795) has used SACT data as primary clinical evidence in an economic evaluation model. In the original appraisal, the company used a single-arm trial, Study 1118E, for the clinical outcome. Longer-term clinical effects were highly uncertain due to the limited long-term data. In the CDF review, the company revised their base-case analysis using SACT data to calibrate OS for the transition probability (post-progression mortality). Since SACT data did not record disease progression data, the company used other sources of RWD to estimate PFS. Here, treatment duration from SACT data was used to adjust the hazard compared with PFS. The appraisal committee concluded that there was considerable uncertainty around the most appropriate approach to estimating PFS of ibrutinib although an indirect approach to estimate PFS was reasonable. The CDF review appraisal of venetoclax (TA796) also used SACT data as primary clinical evidence in the economic model. Parametric models for OS were explored using SACT data. In this review appraisal, the company assumed that PFS was equivalent to the duration of venetoclax treatment. During the appraisals, the committee concluded that the assumption regarding PFS was plausible and that SACT data were the best available and were acceptable to represent venetoclax efficacy.

3.4.2 Non-parametric Use

Non-parametric use of the SACT dataset has been made in 11 CDF review appraisals. Five forms of non-parametric use, informing characteristics of the study population, updating the subsequent treatment line, validation of survival outcome, treatment duration and corroboration of survival data, were identified. Two CDF reviews used SACT data to validate the choice of survival curves in the model (TA739, TA766). In both original appraisals, extrapolation of the survival data was highly uncertain. Updated clinical trial data directly informed the estimates of OS in the economic evaluation model. The clinical plausibility of the survival distribution selected in the model in the CDF review appraisal was checked with SACT data. The duration of treatment was reviewed in four appraisals (TA655, TA691, TA725, TA783) by seeing to what extent SACT data were aligned with the trial data. This informed the discussion of the generalisability of the trial data to routine clinical practice in NHS England but did not inform the estimates of time-on-treatment directly. In six appraisals (TA655, TA713, TA725, TA736, TA783, TA802), SACT data were used to corroborate the clinical trial evidence. Median OS in SACT data and the overlaid survival curves were usually presented to support the trial data. There was one appraisal where SACT data were used to update the subsequent treatment line in the base-case analysis (TA766) and two appraisals (TA795, TA796) where SACT data were used to inform the characteristics of the study population.

3.4.3 Use of SACT Data in Sensitivity/Scenario Analyses

SACT data were used in six CDF reviews (TA684, TA766, TA780, TA783, TA784, TA802) to explore the impact of alternative assumptions in sensitivity or scenario analyses. In one appraisal (TA784), the company used the time to discontinuation in SACT data at the request of NHS England. The company used the SACT data in a scenario analysis but not in the base-case economic model, due to limited availability of baseline characteristics in the SACT database.

3.4.4 SACT Not Used

Evidence from SACT was not used to either update the economic model or support the evidence in nine appraisals. Three patterns of non-use of SACT were identified. In pattern 1, no information on SACT data was reported in the appraisal documentation nor was the PHE report uploaded (TA531, TA683, TA770). In pattern 2, SACT data were attached, but were not reported in the company submission (TA674, TA692). In pattern 3, the company submission reported SACT data and the PHE reports were attached (TA629, TA653, TA687, TA798), but the SACT data were neither used as corroboration nor used directly in the economic evaluation model. The small number of patients and the limited follow-up periods were given as reasons for not using the SACT data.

3.5 Assessment of the Extent to Which Additional Data Reduced the Original Uncertainties

In the CDF review, the technical engagement process was important to discuss the methods with which to deal with uncertainties. Technical engagement is a step where companies get a technical report from the NICE technical team and have a chance to mitigate the remaining uncertainties in the evidence base before appraisal committee meetings [15]. In this process, discussion between ERGs and companies is also allowed. Companies have an opportunity to improve their evidence through this engagement.

Although the technical engagement could help to reduce the methodological challenges, some uncertainties remained. Data from new trials and later follow-up of existing trials were important when it comes to resolving these uncertainties. Uncertainty around immaturity was addressed by clinical trials that had further follow-up. However, later analysis of clinical trials could not solve all immaturity issues. Committees in three review appraisals (TA531, TA684, TA766) still had concerns about the immaturity of survival data. Although the clinical trial captured survival events over a longer period, choice of parametric model to predict OS was highly uncertain in five appraisals (TA655, TA683, TA687, TA692, TA713).

Uncertainty around survival benefit due to indirect treatment comparison was resolved by clinical trials when new RCTs were available. When the original appraisal was based on a single-arm trial while RCTs were ongoing, the review appraisal updated the model based on new phase 3 trials (TA492, TA519). However, if the RCTs didn’t include all relevant comparators, clinical trials had limited scope to reduce uncertainty coming from indirect treatment comparisons. Commonly unresolved uncertainties in CDF review appraisals were the duration of continued treatment effects and the best utility values to use. It was common to use the assumptions previously preferred by committee. Also, clinical experts’ opinions were often used to discuss these issues.

The SACT dataset has rarely been actively used to deal with uncertainties because SACT data were not regarded as robust enough for use in the economic evaluation. A few review appraisals (TA629, TA691, TA725, TA766, TA784) directly indicated that the SACT data were too immature. As later clinical trial data were available, SACT data were less relevant to address the uncertainty around immature data. However, SACT data have provided useful information such as time to treatment discontinuation and subsequent treatment. For example, in the CDF review appraisal of TA581 (TA780), one of the uncertainties was answered by SACT data. The committee preferred to use the proportions based on SACT data to weight the effectiveness estimates by risk group in the clinical trials as the SACT data were expected to inform the true proportion.

4 Discussion

The central findings of this study of experience to date with CDF review appraisals are the limited role played by SACT data and the importance of longer follow-up of the patients in the clinical trials upon which the original appraisals were based. Reasons for these key features of the review appraisals are not hard to find. The additional data available from SACT are limited in several respects—SACT data are not randomised, and survival data are generally immature given the period during which the CDF provided the treatment. The value of the SACT data may be further limited by the number of patients included and the information recorded. The former is also a direct consequence of the timetable chosen for the CDF review.

The use of clinical trial data in preference to SACT data is partly because the latter are not randomised. Comparisons of SACT data with other groups of patients in terms of PFS and OS potentially introduce bias because of differences between patient groups in the distribution of effect modifiers [16, 17]. However, not all the trials used in the original appraisals were randomised trials. In such cases, this limitation of SACT data is less important. For example, in the recent re-appraisal of ibrutinib for treating Waldenstrom’s macroglobulinaemia (TA795), the committee concluded that the SACT data (n = 823) were more relevant than updated trial data from the single-arm Study 1118E (n = 63) and iNNOVATE arm C (n = 31).

The number of patients available for analysis is often smaller and the length of patient follow-up is shorter in the SACT database than in the original clinical trial. For example, in the re-appraisal of avelumab (TA691), the number of trial participants exceeded that in the SACT data (n = 116 vs n = 52), also median follow-up in JAVELIN was 16 months versus 6 months in the SACT database. Also, the data required for the economic model is more often available from the trial rather than from the SACT database. Potentially important model inputs such as PFS, HRQoL and response rate are not available in the SACT database [18].

Latimer suggests that the problem lies not just with the SACT database itself but is in part a failure to exploit the analytical opportunities these data offer [19]. In reviewing the early entrants to the 2016 CDF, he notes that little information was given as to how the SACT dataset would be analysed. In recent CDF review appraisals, TA795 and TA796, SACT data have been used to a greater extent for OS and PFS estimation through active technical engagements and exploring the plausible ways of using the data. A more coherent analytical plan for assessing comparative effectiveness could facilitate better use of SACT data to support the reduction of uncertainties [19, 20].

It is important to stress that this paper reviews experience with the first 24 drugs to exit the 2016 CDF. It accurately documents this recent experience. It is not claiming that SACT data (or other RWD) cannot play a major role in resolving the clinical uncertainties, which have in turn contributed to uncertainty regarding the cost-effectiveness of many new oncologic drugs. The claim is simply that to date the contribution to resolving clinical uncertainty has been modest. More detailed planning for future analysis and longer periods of data collection might both increase the potential contribution of SACT data. It is noteworthy that the consultation over the IMF, a recently introduced sister fund to the CDF for non-oncologic medicines, made reference to provision for a period not exceeding 5 years [22], as does the recent NICE process and methods manual.

A review of the operation of the 2016 CDF is particularly relevant since NHS England is expanding the use of managed access schemes with the introduction of the IMF. While it is likely that the IMF will operate in a similar fashion to the CDF, it will support “patients with any condition, including those with rare and genetic diseases, to get early access to the most clinically promising treatments where further data are needed to support NICE make recommendations with respect to routine commissioning by the NHS” [21]. Consideration of experience with the CDF can aid understanding of the opportunities and challenges of using additional data to address uncertainties.

Although the use of RWD in CDF review appraisals is an institution-specific issue, the use of RWD in drug appraisals is of more general interest. The Italian Medicines Agency (AIFA) monitoring platform of registries tracks eligible patients and a complete flow of treatment to evaluate the appropriate use of drugs following their approval in the Italian national health system [22]. The data collected are useful sources for verifying the real impact of the initial reimbursement criteria [23]. In Dutch health technology assessment (HTA) reports, RWD have been used for initial decision-making. In conditional financing, a type of MAA, use of RWD to reduce uncertainty has attracted attention [24]. However, a detailed analysis of the utilisation of different forms of RWD in different HTA systems is beyond the scope of this paper.

This paper has focused on the uncertainties in the original appraisal and the additional data considered at the re-appraisal. It has not sought to assess the success or otherwise of the CDF. Patients have had access to these 24 therapies through the CDF following an initial decision not to recommend routine commissioning. Moreover, following re-appraisal, 21 have moved to routine commissioning. In addition, while in the CDF, the drugs have had a price that is deemed cost-effective given the available evidence. An alternative perspective might be that the original clinical uncertainties do not appear to have been markedly reduced and still the re-appraisals have been overwhelmingly positive. This possibly suggests that CDF review appraisals should be regarded as a “review to ensure that the original decision is consistent with the latest evidence”, rather than as a “final chance to make the case”. However, before accepting Bob Dylan’s rejection of re-appraisal and the re-assurance that NICE committees can generally make the correct decision at the first attempt, it is important to recognise that any assessment of the value of the CDF needs to make a judgment regarding the counter-factual, including how the existence of the CDF might be influencing committees’ decision-making and manufacturers’ research activities and pricing decisions.

5 Conclusions

While additionally collected RWD attracted attention when the 2016 CDF was introduced, RWD were not widely used in CDF review appraisals and (to date) do little to reduce uncertainty. Experience with these appraisals has highlighted the importance of longer follow-up of clinical trials and the relatively limited role of RWD, in general, and SACT data in particular. Although the 2016 CDF, with its MAAs, is a clear improvement on the original CDF, the extent to which the clinical uncertainties have been resolved by additional data is unclear.