Characteristics of Appraisals with OIR and AWR Recommendations
Of the 184 appraisals conducted up to January 2010, 40 included OIR/AWR recommendations in the draft and/or final guidance. A list of all appraisals including OIR and AWR recommendations is provided in the Supplementary Table (Online Resource). Most guidance with OIR was not specific about the research to be conducted and often referred to the technology being ‘not recommended except within clinical trials’ or being ‘recommended only within clinical trials’. For example, guidance on mycophenolate mofetil (MMF) for use in paediatric renal transplantation stated: “The use of MMF in corticosteroid reduction or withdrawal strategies for child and adolescent renal transplant recipients is recommended only within the context of randomised clinical trials” (TA99 ). There were some exceptions to this. For example, the OIR guidance on spinal cord stimulation provided more information on the type of information the research should provide: “Spinal cord stimulation is not recommended as a treatment option for adults with chronic pain of ischaemic origin except in the context of research as part of a clinical trial. Such research should be designed to generate robust evidence about the benefits of spinal cord stimulation (including pain relief, functional outcomes and quality of life) compared with standard care” (TA159 ).
AWR guidance was routinely more specific about the type of evidence collected. A detailed AWR recommendation was made for the use of inhaled insulin in a subgroup of people: “Data on the use of inhaled insulin according to this guidance should be collected as part of a coordinated prospective observational study. The data collected should include individual patient outcomes, adverse events and measurements of lung function” (TA113 ). Similarly guidance on etanercept and infliximab for the treatment of arthritis was prescriptive regarding the data to be collected: “All clinicians prescribing etanercept or infliximab should (with the patient’s consent) register the patient with the Biologics Registry established by the BSR [British Society for Rheumatology] and forward information on dosage, outcome and toxicity on a 6-monthly basis” (TA36 ).
Multiple ACDs were issued for some appraisals and the 31 ACDs containing OIR/AWR recommendations relate to 25 appraisals. Of the 31 ACDs, 26 (84 %) included OIR recommendations and five (16 %) AWR recommendations. All of the 29 FADs included in the review relate to a unique appraisal and were all published as final guidance for the appraised technology: 25 (86 %) were OIR recommendations and four (14 %) were AWR recommendations. OIR recommendations were much more common than AWR recommendations. Changes to the inclusion of OIR/AWR recommendations between draft and final guidance were more common than suggested by the summary numbers; only 14 appraisals included OIR/AWR recommendations in both draft and final guidance (ACDs were unavailable for a further 12 appraisals). (See the Supplementary Table [Online Resource] for further details.) In most cases, where an OIR/AWR recommendation was removed after consultation, the final guidance usually recommended the technology for all or a specific subgroup of patients.
A single piece of NICE guidance can include several recommendations related to multiple technologies, multiple indications or different settings for the use of the technology. Over half of the OIR/AWR recommendations specified the need for further research in particular subgroups of patients (52 % of OIR/AWR recommendations in final guidance documents). In approximately a quarter of cases, the OIR/AWR recommendations targeted a subset of the technologies included in the appraisal.
Overall, 16 % of all appraisals included an OIR/AWR recommendation in the final guidance. Table 2 shows a recent decline in the frequency of guidance including OIR/AWR recommendations. There were no apparent differences in the decline between OIR and AWR recommendations. No final guidance included OIR/AWR recommendations in 2007, which is the year following the introduction of the STA process. Differences in the frequency of OIR/AWR recommendations were observed between the two NICE appraisal processes. Of appraisals issued through the multiple technology appraisal (MTA) process, OIR or AWR recommendations were included in draft guidance of 23 appraisals and final guidance of 28 appraisals. These 28 TAs account for 19 % of all final guidance issued within the MTA process. In the STA process, only two ACDs and one piece of final guidance contained OIR/AWR recommendations. This accounts for just 2 % of all final guidance issued through the STA process up to the time the review was conducted.
The data were examined for differences in the use of OIR/AWR recommendations according to general disease area and the type of technology under appraisal. In absolute terms, OIR/AWR recommendations were more common for cancer treatments, accounting for over a third of all the OIR/AWR recommendations in the final guidance, followed by musculoskeletal conditions (n = 7), which accounted for almost a quarter of cases identified. However, NICE has appraised a large number of treatments for cancer: 28 % of all published appraisals over the review period. Only 7 % of all NICE TA guidance has related to musculoskeletal conditions and so it appears that a disproportionate amount of these have included OIR/AWR recommendations compared with appraisals for other conditions.
Just over half of the appraisals with OIR/AWR recommendations related to drugs (n = 16; 55 %). However, taking into account the total number of TAs published relating to drugs, the use of OIR/AWR recommendations appears to be on average less common for these appraisals: 11 % of all drug appraisals within the period contained OIR/AWR recommendations, compared with 47 % of all guidance on therapeutic or surgical procedures and 27 % of all guidance on devices.
Cost Effectiveness of Technologies with OIR/AWR Recommendations
NICE requires all appraisals to include an assessment of cost effectiveness usually framed as an incremental cost per QALY. All appraisals that included an OIR/AWR recommendation considered the cost effectiveness of the technologies. Most of the guidance documents reported several different estimates of incremental cost effectiveness based on analyses submitted by different stakeholders, relating to different uses of the technology or based on different sets of assumptions or evidence. However, a formal assessment of cost effectiveness was not always conducted or reported in the ACD or FAD for the use of the technology specified in the OIR/AWR recommendation. Table 3 shows the base-case incremental cost-effectiveness ratios (ICERs) for the overall population and for the specific OIR/AWR indication where this differs. Ideally, the ICER considered most plausible by the Appraisal Committee after reviewing the evidence would be taken to reflect the NICE view of cost effectiveness. As this was not always reported, the base-case estimate from the independent Assessment Group is also reported in the table.
Most of the reported ICERs were higher than the £20,000–30,000 threshold range employed by NICE. Only guidance phrased as AWR reported ICERs within the £20,000–30,000 threshold range for the AWR indication as preferred by the Committee. The two ICERs reported as above this range for AWR guidance and considered plausible by the Committee related to two technologies reviewed within one appraisal and were only marginally above £30,000. In some cases, ICERs were reported but were based on analyses that did not use the QALY as the outcome measure. For example, TA5 on the use of liquid-based cytology reported ICERs of £1,100 and £2,500 per life-year gained depending upon the length of the screening interval . Where ICERs were not directly reported, there was often an indication of whether the technology was considered to be cost effective. For example, TA8 on hearing aid technology stated that: “Whilst it is impossible, on the basis of the present evidence, to estimate meaningful cost-utility ratios … additional spending on this service, if appropriately targeted, has the potential to be highly cost effective.” 
Considerations Leading to OIR/AWR Guidance
The frequency of technologies considered cost effective by the Appraisal Committee when used in the context of the OIR/AWR recommendation is presented in Table 4. In most cases (79 % of FADs with OIR/AWR recommendations), the technology was not cost effective and an OIR recommendation was issued. There were only a couple of cases where OIR recommendations were made for technologies considered likely to be cost effective based on the accepted analyses. These appraisals (TA5  and TA51 ) requested that pilot implementation programmes be undertaken prior to the large-scale and routine introduction of the technologies into the NHS. In the small number of cases where AWR recommendations were issued, the technologies were usually considered to be cost effective. In the single exception, the ICER for the technology was higher than, but close to the upper bound of, the usual threshold range and reported to be in the range of £27,000–35,000 per additional QALY gained (TA36 ).
Table 5 shows the stated rationale for issuing the OIR/AWR recommendations. Of the pieces of final guidance that did not explain the rationale for the OIR/AWR recommendation, three were issued prior to a section on the Committee’s considerations being routinely included in the documents (TA5, TA6, TA17). The OIR in the other appraisal related to three specific subgroups of patients: two were not referred to in the Committee’s considerations at all and it was stated that there was “no clinical or modelling evidence, or expert opinion” to support the use of the technology in the third subgroup (TA75 ).
A need for further evidence on the relative effectiveness of the intervention in the overall population or the OIR/AWR subgroup was the most commonly cited reason for issuing the OIR/AWR recommendation (Table 5). Several reasons were cited in support of most recommendations and the need for further evidence on relative effectiveness, either in the overall population or the OIR/AWR subgroup, was cited in 19 (66 %) FADs identified in the review, and most of these included OIR recommendations. Of those citing a lack of sufficient evidence on relative effectiveness, only one included an AWR recommendation (TA113 ). This appraisal noted a gap in the evidence on clinical effectiveness for the highly selective subgroup of patients targeted in the recommendations and that the cost-effectiveness estimates were sensitive to these estimates. It recommended that the data would be most appropriately collected through a registry.
There was a greater range of considerations cited for draft AWR recommendations than for the final guidance. All final AWR guidance documents referred to the need for long-term data, and most also referred to a need for additional data on adverse effects. These two considerations were also referred to in a small number of OIR final guidance documents. A need for longer-term data was also frequently cited as a consideration leading to the OIR/AWR guidance. Uncertainty in the cost-effectiveness estimates was a common consideration; however, in all cases this was coupled with a need for further clinical evidence. No guidance (draft or final) cited concern about investment and reversal costs as a rationale for OIR/AWR recommendations. However, TA51 on computerized cognitive behavioural therapy (CCBT) did suggest concerns regarding the levels of training required for the implementation of a recommendation to routinely introduce CCBT into the NHS: “Further information is required about the extent of training needed and circumstances under which different staff could provide support for users of CCBT” (TA51 ). Concern about the budget impact of introducing the technology or the potential impact on ongoing research did not lead to the OIR/AWR recommendation in any of the appraisals.
NICE routinely considers the list price of technologies (e.g. as reported in the British National Formulary for drugs) and possible changes in price over time are not usually taken into account. However, a system for considering reductions in the costs of treatment through PASs has been established. None of the OIR/AWR guidance identified within the review included a PAS; however, one appraisal that included an OIR recommendation at an earlier stage of development later included a PAS (TA129). This appraisal included an OIR recommendation in the draft guidance and stated that the technology was not recommended except for use in well-designed clinical studies and that the Committee was not persuaded of its cost effectiveness; however, this was subsequently amended to a ‘reject’ recommendation after concerns were raised about whether the research would be conducted. The final guidance approved the technology following the offer of a PAS, which would reduce the cost of providing treatment. The PAS was designed to offer a rebate to the NHS when patients’ disease responds less than partially to treatment; however, there was no formal requirement for data analysis and reporting beyond the level of rebate and it is therefore not categorized as an OIR recommendation here. In another appraisal, the OIR recommendation was revised to an approval after the Committee revised their estimates of cost effectiveness based on discounted prices of the technology along with further information on quality-of-life improvements (TA166).
Considerations around ethical implications and whether uncertainties in the evidence base would resolve over time were not explicitly stated as reasons for issuing OIR or AWR recommendations. In addition, the relative costs and benefits of conducting research were not reported as considerations of the Committee when formulating its research recommendations.
Most of the appraisals that required relative effectiveness data recommended experimental research designs for evidence collection (Table 6). Two appraisals that cited a need for further evidence on relative effectiveness in the final guidance recommended observational studies due to anticipated difficulties in conducting randomized controlled trials (RCTs) in the specific OIR patient population (TA37) or indication (TA167). There were changes in the recommended type of research between draft and final guidance, which were mainly due to changes in the target OIR/AWR population (e.g. TA68) or the recommendation of a broader type of research (e.g. TA89).
Review of Updated Appraisals
Among the OIR/AWR recommendations in the final guidance, ten were updated following a review, including two that were incorporated into clinical guidelines (CGs). Table 7 provides details of the appraisals, whether additional evidence was provided and the change to the OIR/AWR recommendation (new evidence for other recommendations included within the guidance is not noted in the table).
In the majority of reviewed appraisals, new evidence informing the OIR/AWR recommendation was available for the review. In three cases, no new evidence was provided that was specific to the OIR/AWR indication. For the review of TA6, no new RCT data were available for the OIR recommendation, which was made more restrictive in the review guidance. New evidence on clinical effectiveness was not available for the review of TA33, and although new data on adverse effects were provided, they were considered inadequate and no change was made to the OIR recommendation. The OIR recommendation was removed from the review of TA37 despite a lack of new evidence presented. In this case, there had been a reduction in demand for the drug in this setting (it had since become licensed and NICE approved for treatment of an earlier stage of disease) and there were concerns about the feasibility of future data collection.