FormalPara Key Points for Decision Makers

As part of their critique of the evidence submitted by companies in the National Institute for Health and Care Excellence Single Technology Appraisal (STA) process, the Evidence Review Groups (ERGs) undertake exploratory analyses to explore uncertainties.

Of the 100 STAs included in this analysis, 93 had exploratory analyses undertaken by the ERG.

There is no clear pattern to the presence or frequency of exploratory analyses; these cannot be obviously explained by the disease area covered by the STA, the time the STA took place, or the company’s base-case incremental cost-effectiveness ratio.

There may be a pattern in the mean number of analyses conducted by individual ERGs.

1 Introduction

The UK National Institute for Health and Care Excellence (NICE) Single Technology Appraisal (STA) process is usually undertaken for a technology for a single indication and includes the production of a submission by the manufacturer or sponsor of the technology. NICE commissions independent Evidence Review Groups (ERGs) to critically appraise company submissions as part of the STA process. In their critique of the evidence submitted by companies, the ERGs often undertake exploratory analyses to explore uncertainties around the company’s model and their implications for decision making. The number and type of exploratory analyses undertaken varies between appraisals. The ERG reports are a central component of the evidence considered by the NICE Technology Appraisal Committees (AC) in forming their recommendations. The findings of the committee are used to produce the Appraisal Consultation Document (ACD) and, after further considerations and a consultation period, a Final Appraisal Determination (FAD) is produced, which results in NICE guidance. The STA process is outlined in detail in NICE’s Guide to the Process of Technology Appraisal [1]. The company is expected to follow the decision-analytic approaches as described in the Guide to the Methods of Technology Appraisal [2], and the submission is expected to contain an evaluation of the clinical effectiveness and cost effectiveness of the technology. It is the responsibility of the ERG to determine what additional analyses are required and to undertake them, although they may be constrained by the availability of data, the model structure, or time. The ERG may also identify and correct technical or programming errors. This critical appraisal of the company submission, and any additional work as a consequence of this critical appraisal, forms the basis of the ERG’s report. NICE’s remit to the ERGs is not overly prescriptive, allowing ERGs to use their acknowledged expertise and judgement in the methods used to critically appraise company submissions. This is appropriate due to the wide variation in complexity and quality of the company submissions received.

There are currently nine ERGs:

  • BMJ Technology Assessment Group (BMJ-TAG), BMJ Group.

  • Centre for Reviews and Dissemination (CRD)/Centre for Health Economics (CHE), University of York.

  • Health Economics Research Unit and Health Services Research Unit, University of Aberdeen.

  • Kleijnen Systematic Reviews Ltd.

  • Liverpool Reviews and Implementation Group (LRiG), University of Liverpool.

  • Peninsula Technology Assessment Group (PenTAG), Peninsula Medical School, University of Exeter.

  • School of Health and Related Research (ScHARR), University of Sheffield.

  • Southampton Health Technology Assessment Centre (SHTAC), University of Southampton.

  • Warwick Evidence, University of Warwick.

An additional ERG, the West Midlands Health Technology Assessment Collaboration, undertook STAs during the period 2005–2010. ERGs are assigned to work on STAs based on availability and lack of conflicts for specific topics.

Some assessment of ERG report production process has been undertaken. For example, Wong et al. [3] assessed approaches used by ERGs to critically appraise search strategies within company submissions. Previous research has highlighted issues with company submissions that are particularly challenging to the ERGs [4]. Carroll et al. [5] suggested that company STA submissions could be improved if attention were paid to transparency in the reporting, conduct and justification of the review, and modelling processes and analyses, as well as greater robustness in the choice of data in the model and closer adherence to the scope or decision problem. Kaltenthaler et al. [6] also recommended the need for clear and transparent reporting of company submissions, and for a clear and concise rationale for the synthesis of clinical data, the development of economic models and the assumptions used to develop models.

The aim of this study was to note the number of exploratory analyses undertaken by the ERGs within the NICE STA process and to assess whether their frequency might be explained by variables such as disease area or a company’s base-case incremental cost-effectiveness ratio (ICER). For the purpose of this research, an exploratory analysis was defined as any additional analysis that generated an ICER and was included in the ERG report section titled ‘Exploratory and sensitivity analyses undertaken by the ERG’ (most commonly reported in Sect. 6 of the suggested ERG report template). This is an underresearched area and this study was commissioned by NICE to develop understanding of this aspect of the STA process. This research is of interest to all key stakeholders in the STA process, including the ERGs, pharmaceutical companies, AC members and NICE. The full research report forms part of the National Institute for Health Research Health Technology Assessment (NIHR HTA) monograph series [7]. The objectives of the study presented here were twofold.

  1. 1.

    To identify the extent of ERG exploratory analyses, as defined above.

  2. 2.

    To identify factors that influence or predict the extent of ERG exploratory analyses.

2 Methods

This was a joint project undertaken by researchers from the University of Sheffield and the University of York, all with considerable experience of the NICE STA process. The 100 most recently completed STAs (2009–2014) for which guidance has been published were selected for inclusion in the analysis. A list of the STAs included in this analysis is provided in Appendix 1. Data were extracted from over 400 separate documents associated with these STAs, including ERG reports (unredacted versions, including confidential information used by the ACs), the first ACD issued and the final FAD issued.

A data extraction tool to extract relevant data to address the project objectives was developed and piloted to ensure usability and to standardise extraction [7]. The following items were included in the data extraction form:

  • basic characteristics including company, disease area and ERG;

  • company’s base-case ICER(s);

  • number and type of exploratory analyses conducted by the ERG.

All data extractions were double-checked by a second researcher, and a narrative synthesis [8] of the extracted data was performed, summarising key data through text and tables to highlight any potentially important patterns or relationships in the data. The mean and median numbers of exploratory analyses per ERG report were calculated. The mean number of analyses was used to provide a simple binary variable with which to test some assumptions about relationships between the number of exploratory analyses and variables such as disease area and ICER. It was considered a priori that the disease area and an estimated cost-effectiveness ratio of £20,000 per quality-adjusted life-year (QALY) were the key variables potentially most likely to predict the incidence and frequency of exploratory analyses. This was due to the known impact of disease area on other elements of STAs [9] and the perceived importance of the £20,000 per QALY ICER for NICE decision making [10]. An assessment was also made to identify any changes in the number of exploratory analyses undertaken over time, and whether the number and type of exploratory analyses differed by the ERG undertaking the critical appraisal. A more in-depth investigation of the types of exploratory analyses undertaken, and their role and impact on NICE decision making, is provided elsewhere [7].

The key data used in the synthesis were then reduced to whether an STA conducted more or less than the overall mean number of exploratory analyses, and whether just ‘one or more’ exploratory analyses were explicitly cited as having an influence on a recommendation (defined as being mentioned in the ACD or FAD). These arbitrary selections were made as a means of making the most of the data to address the objectives of the project and are explained more fully elsewhere [7, 11].

3 Results

3.1 Overview of the 100 Single Technology Appraisals

Forty different companies made submissions as part of the NICE STA process. The companies with the largest number of submissions were Roche (n = 16), Novartis (n = 9), Glaxo Smith Kline (n = 7), Bristol-Meyers Squibb (n = 7) and Bayer (n = 6). Other companies involved had five or fewer submissions and the majority of companies made only one or two submissions: Alimera Sciences, Alimta, Allergan, Amgen, Astellas Pharma, Astra Zeneca, Biogen, Boehringer-Ingelheim, Celgene, Cell Therapeutics Inc., Eli Lilly, Eliquis, Eisai, Genzyme, InterMune, Janssen, Laboratoires Servier, Movetis, MSD, Napp, Novo Nordisk, Otsuka Pharmaceuticals, Pfizer, PharmaMar, Pharmaxis, Pierre Fabre, Sanofi Aventis, Savient Pharmaceuticals, Schering-Plough, Stelara, Sucampo Pharma Europe, Takeda UK Ltd, The Medicines Company, Thrombogenics, and UCB.

The principal disease areas covered by the STAs were cancer (44%), blood and the immune system conditions (11%), cardiovascular conditions (10%) and musculoskeletal conditions (8%). Figure 1 shows a breakdown of disease area by company. Cancer is clearly covered by many companies, although Roche, GlaxoSmithKline and Novartis have contributed the most STAs of this nature. Roche and GlaxoSmithKline have both also contributed the most blood and immune system STAs.

Fig. 1
figure 1

Disease area by company. GSK GlaxoSmithKline, STA Single Technology Appraisal

Full size image

Most of the ERG reports contained at least one exploratory analysis (93%), and a total of 40 (43%) included eight or more such analyses. The mean number of exploratory analyses per report containing exploratory analyses was 8.5, with a median of 7 and a range of 1–29. For this reason, a cut-off point of 8 exploratory analyses was chosen for these analyses. Figure 2 shows the distribution of exploratory analyses. With regard to the seven ERG reports that did not include any exploratory analyses, the majority (five ERG reports, 71%) stated that no exploratory analyses were undertaken by the ERG as the company models were considered to have serious flaws. One ERG report stated that no exploratory analyses were undertaken due to no ICERs being presented by the company (TA 191) [12], and one (TA 267) [13] because the ERG was satisfied with the company model and the sensitivity analyses presented by the company.

Fig. 2
figure 2

Distribution of the exploratory analyses

Full size image

3.2 Factors That May Have Influenced the Number of Exploratory Analyses Undertaken by the Evidence Review Group

Four variables that may influence the incidence and frequency of exploratory analyses were considered:

  • disease area;

  • ICER;

  • changes over time;

  • the ERG undertaking the critical appraisal.

The incidence and frequency of ERG exploratory analyses do not appear to be related to the disease area covered by the STA (see Table 1). STAs in the blood/immune system category did have a slightly higher proportion of ERG reports, with more than eight exploratory analyses (7/11, 64%) than the other disease areas. However, as most disease area categories had small numbers of STAs, it is difficult to draw any firm conclusions from this.

Table 1 Number of reports by disease area with one or more, and eight or more, exploratory analyses
Full size table

With regard to the estimated cost-effectiveness ratio, there does not appear to be a relationship between the company’s base-case ICER and the incidence or number of exploratory analyses (see Table 2). Of the 93 ERG reports presenting one or more exploratory analyses, the proportion of ERG reports in which the company’s base-case ICER was below £20,000 per QALY gained was similar to the proportion at or above this ICER (47 and 53%), and the proportion of ERG reports below or above £30,000 per QALY gained shifts slightly but was also similar (54 and 46%). The likelihood of an ERG performing eight or more exploratory analyses did not appear to be affected by the company’s base-case ICER. The proportion of ERG reports with eight or more exploratory analyses was very similar in the STAs in which the company base-case ICER was below £20,000 per QALY gained (41%) or above this ICER (45%).

Table 2 Number of reports by company ICER with one or more, or eight or more, exploratory analyses
Full size table

Developments in the STA process between 2009 and 2014 do not appear to have influenced the frequency of exploratory analyses undertaken, as shown in Fig. 3. In this sample, the total number of STAs undertaken has varied from three in 2009 to 23 in 2011; however, the number of ERG reports with eight or more exploratory analyses appears to be largely consistent over time. For example, between 2011 and 2014, the percentage was consistent in each year and was always between 38 and 45%.

Fig. 3
figure 3

Number of exploratory analyses by year. STAs Single Technology Appraisals

Full size image

However, there does appear to be a possible pattern in the mean number of analyses conducted by particular ERGs, as shown in Table 3.

Table 3 ERGs responsible for producing reports within STAs included in the analysis
Full size table

The University of York CRD/CHE produced the highest number of reports (n = 18), while Warwick Evidence produced the fewest (n = 5). The number of reports varies between the ERGs depending on how long they have been undertaking STAs and the size of their agreed contract. The ScHARR had the highest mean number of exploratory analyses per report (11.4), while West Midlands had the fewest (2.3). Of the ERGs currently involved in the STA process, the teams with the lowest mean number of exploratory analyses per report were Kleijnen Systematic Reviews (5.3) and Liverpool Reviews (5.6). As stated above, the number of exploratory analyses appears to be largely consistent over time for the whole sample. We also looked at the number of exploratory analyses by ERGs over time and present data here for the ScHARR as an example. The other ERGs showed similar results. The number of exploratory analyses undertaken by the ScHARR appears to be the highest and is quite stable over time. In 2011, the ScHARR produced five ERG reports, and the number of exploratory analyses ranged from 3 to 19; in 2012, the ScHARR produced three ERG reports, with the number of analyses ranging from 2 to 22; and in 2013 two ERG reports were produced, with 5 and 19 exploratory analyses. It therefore appears that the number of exploratory analyses is likely to be more dependent on the individual requirements of each STA or other possible factors, rather than due to the ERG becoming more rigorous in its critique (such that the number of exploratory analyses might increase year-on-year) or simply more focused (such that numbers of exploratory analyses might decrease year-on-year).

4 Discussion

In this analysis, the vast majority (93%) of ERG reports reported one or more exploratory analyses, with a mean of 8.5 exploratory analyses per report for the 93 reports where exploratory analyses were undertaken. Of the 93 ERG reports with at least one exploratory analysis, a total of 40 (43%) included eight or more such analyses. The likelihood of an ERG performing eight or more exploratory analyses does not appear to be affected by the disease area covered by the STA or the company’s base-case ICER. In previous research, Barham [9] analysed data from the first 18 STAs undertaken by NICE and found 56% of these to cover cancer topics. In an analysis of the first 4 years of the NICE STA process, 48% of STAs were undertaken in cancer topics [14], slightly higher than reported in this study. This shows that the percentage of STAs that are cancer topics may be decreasing over time.

The proportion of ERG reports with eight or more analyses appears to be relatively stable over time and not related to any developments in the process between 2009 and 2014. Although there have been changes to the NICE technology appraisal process and methods guides during this time period, these appear to have had little effect on the number of exploratory analyses undertaken by the ERGs. As shown in the example of the analysis of ScHARR data, there does not appear to be a time-dependent trend as the number of analyses undertaken by the ERG appears to be stable with respect to time. Additional analyses of the data, both from other ERGs and from pharmaceutical companies, show similar results (data not shown).

The number of exploratory analyses varied by ERG. For the 10 ERGs undertaking STAs, the mean number of exploratory analyses per report ranged from 2.3 (West Midlands) to 11.4 (ScHARR). The reasons for this variation are unclear and potentially complex, and will include such factors as how thoroughly the company has explored the uncertainties and plausible alternative scenarios. It should be noted that no regression analyses were performed to explore the relationship between the mean number of analyses per report and variables such as ERG, disease area or year due to the limitations of the data because there was only one independent variable in any category (cancer in disease area) that exceeded 20 in number, with the majority being very small numbers, and with the result being that any such analysis would be underpowered. Other potentially influential factors, such as the complexity and perceived quality of company submissions, were also not explored or analysed. Therefore, this finding should be interpreted with caution and more research is needed to determine the relationship between variables such as ERG and number of exploratory analyses undertaken, which may also vary according to the skills, experience and judgements of the ERGs and that of individuals within the ERGs. This issue was not explored within this study.

Only seven of the STAs included in this analysis had no exploratory analyses undertaken by the ERGs, principally due to serious flaws identified by the ERG in the company model, which would have rendered analyses irrelevant. In only one of the STAs were no exploratory analyses undertaken by the ERG because the ERG was satisfied with the model and analyses presented by the company. The exploratory analyses undertaken by the ERGs serve various functions: to correct errors and violations within the company’s model; to address uncertainties in the evidence base; and explore a range of plausible scenarios, and therefore support NICE AC decision making. The exploratory analyses undertaken by ERGs frequently influence both ACD and FAD recommendations [11].

This analysis is a good reflection of current practice as the most recent 100 STAs were included. The data extraction tool was extensively piloted and the double-checking of all key data across the 100 STAs by at least two experienced cost-effectiveness modellers reduced the likelihood of inconsistency and inaccuracy in the data. The use of narrative synthesis was principally descriptive, and reduced the likelihood of overstating relationships in the data. A reductive approach was taken to managing data that might be affected by interpretation or by poor reporting in the original documents. However, there are some limitations to this research. The descriptions of analyses undertaken were often highly specific to a particular STA and could be inconsistent across ERG reports, and thus difficult to interpret and categorise. In addition, small inconsistencies might have affected the data extraction, which may be due to several people being involved. A number of other factors not considered in this study may have influenced the number of exploratory analyses undertaken by the ERGs but this was beyond the remit of this research.

This analysis of over 400 documents provides an overview of some of the principal factors potentially affecting the number of exploratory analyses undertaken by the ERGs. The wider study reported that four types of exploratory analysis were conducted in relation to companies’ models: fixing errors; addressing violations; addressing matters of judgement; and the provision of a new, ERG-preferred base-case. Ninety-three of the 100 ERG reports contained at least one of these analyses, and the most frequently reported type of analysis related to the category ‘matters of judgement’. The results of the wider study also suggest that these additional analyses undertaken by ERGs were highly influential in the policy and decision-making process [7, 11]. More in-depth analysis is needed to understand how ERGs make decisions regarding the exploratory analyses to be undertaken. More research is also needed to fully understand the types of exploratory analyses most useful to ACs in their decision making.

5 Conclusions

There is no clear pattern to the presence or frequency of exploratory analyses; they do not appear to be predicted by the disease area covered by the STA, the time the STA took place, or the company’s base-case ICER. In addition, there does appear to be a pattern in the mean number of analyses conducted by individual ERGs, but more research is needed to understand this relationship.