FormalPara Key Points for Decision Makers

From January 2015 to December 2017 inclusive, the National Centre for Pharmacoeconomics (NCPE) appraised 158 rapid reviews (RRs) and 49 health technology assessments (HTAs). Just under half of the submissions (49%) were recommended for HTA following RR.

This is the first study in Ireland to quantify the individual stages of the HTA process and how they contribute to the overall timeline of the assessment process.

We found that the time from marketing authorisation (MA) date to submission of RR varied widely across submissions. Time to appraise RRs was found to be in line with the agreed 4-week timeframe. Time taken by the NCPE to appraise HTAs was outside the 90-day timeframe.

1 Background

Applicant companies applying for reimbursement of new drugs in Ireland are required to submit a preliminary rapid review (RR) for appraisal by the National Centre for Pharmacoeconomics (NCPE). The NCPE is commissioned by the Corporate Pharmaceutical Unit of the Health Service Executive (HSE-CPU) to appraise RRs and health technology assessments (HTAs). High-cost drugs or those predicted to have a significant budget impact are required to undergo a full HTA appraisal. The appraisal of an HTA involves a systematic, evidence-based assessment of the comparative effectiveness, cost effectiveness and potential budget impact of drugs prior to reimbursement [1].

1.1 The Framework for Health Technology Assessment (HTA) in Ireland

The requirement for HTA prior to reimbursement of drugs in Ireland is outlined in an agreement between the Irish Pharmaceutical Healthcare Association (IPHA) and the HSE (HSE-IPHA agreement) [2]. The 2006 HSE-IPHA agreement provided a new framework for the pricing and reimbursement of drugs in Ireland. Under the new agreement, high-cost technologies or those with significant budget impact were considered for HTA prior to reimbursement. Since 2006, there have been several updates to this agreement [3]. In 2009, the updated agreement stated that all new drugs would require an HTA prior to a reimbursement decision. The 2012 agreement stated that all technologies must be assessed for cost effectiveness relative to an explicit payer threshold of €45,000 per quality-adjusted life-year and that no drug should be approved at a price higher than the European reference price [4].

On 1 August 2016, a new 4-year framework agreement provided detailed principles and processes for the assessment of pricing and reimbursement of new drugs in Ireland, with increased clarity regarding the decision-making structure and timelines. The timelines outlined in the updated agreement included a 4-week timeframe for the NCPE to complete RR assessments [2].

1.2 The HTA Process

A summary of the HTA process is outlined in Fig. 1. The formal HTA process may begin once the applicant company receives a positive recommendation from the European Medicines Agency (EMA) Expert Advisory Group (Committee for Medicinal Products for Human Use [CHMP]) and they have certainty around the reimbursement price for which they intend to apply. Following release of the CHMP opinion, the European Commission (EC) usually makes its decision, a legally binding authorisation, after 67 days [5]. The HTA process consists of several steps:

Fig. 1
figure 1

Summary of the health technology assessment process in Ireland (during the study period) [8]. CHMP the European Medicines Agency Expert Advisory Group, HSE-CPU Health Service Executive Corporate Pharmaceutical Unit, HTA health technology assessment, NCPE National Centre for Pharmacoeconomics, RR rapid review. aFollowing this recommendation, the applicant company may submit an HTA; however, most choose to engage in price negotiations with the HSE-CPU instead and avoid an HTA. bDuring the study period, the time allocated to the applicant company to complete factual accuracy check was 10 working days; this was reduced to 5 working days in 2019

Full size image

  • The HSE-CPU formally notifies the applicant company of the requirement for RR submission.

  • The applicant company makes an RR submission to the NCPE in accordance with NCPE requirements. Each submission pertains to one drug for one indication.

  • The NCPE review group appraises the RR submission along with any relevant safety and efficacy evidence arising from independent literature reviews. The NCPE endeavours to appraise the RR within a 4-week timeframe [2].

  • The decision criteria considered by the NCPE during the RR appraisal includes eligible patient population, robustness of clinical evidence, cost of the drug relative to current standard of care and budget impact [4].

  • The NCPE informs the HSE-CPU as to whether an HTA is required and notifies the applicant company when the RR is completed and that the outcome will be posted on the NCPE website.

  • Where an HTA has been deemed necessary, the HSE-CPU formally commissions one. A pre-submission consultation is arranged between the NCPE and the applicant company. The pre-submission meeting provides an opportunity for the applicant company and the NCPE to discuss what will be included in the HTA submission. However, no agreements are made during this meeting prior to the NCPE having the opportunity to review all evidence in the HTA submission.

  • The applicant company produces an HTA submission in accordance with NCPE requirements.

  • The NCPE welcomes patient interest group submissions. A template is available via the NCPE website, which is submitted by the patient interest group directly to the NCPE within 90 days of the HTA commencing.

  • An NCPE review group appraises all submitted documents and economic models.

  • The NCPE considers evidence surrounding the cost effectiveness of the drug compared with standard of care, uncertainty around the cost-effectiveness estimates, the validity of the clinical and cost-effectiveness data used to derive these estimates and budget impact.

  • The review group may seek clarification from the applicant company on any aspect of the submission. Preliminary questions are sent to the applicant company once the NCPE have conducted a thorough review of the submission. The applicant company is requested to respond to the questions within 1 month.

  • The appraisal of the HTA should take a maximum of 90 days, excluding the time taken by the applicant company to prepare the submission and to make any required amendments. A draft appraisal report is sent back to the applicant company for a factual accuracy check, which should be carried out within 10 working days during the study period. The period allocated for the applicant company to complete the factual accuracy check was reduced to 5 working days in 2019.

  • The final appraisal report is produced. Where cost effectiveness has been demonstrated, a positive reimbursement recommendation is made. If the drug is not considered cost effective at the submitted price, this is specified. If it is not considered cost effective because of unresolved concerns about the validity of any aspect of the submission, this is detailed. The wording of NCPE recommendations has been updated since the timeframe of this analysis. Further information on the updated NCPE recommendations can be found on the NCPE website [6].

  • The final appraisal report is sent to the HSE-CPU. The report addresses three of the nine criteria outlined in the Health Act 2013: comparative clinical effectiveness, cost effectiveness and budget impact.

  • The HSE considers the appraisal report in conjunction with the additional criteria set out in the Health Act 2013: health needs of the public; clinical need; availability of the appropriate level of clinical supervision to ensure patient safety; availability and suitability of the drug for supply and reimbursement under the current schemes; proposed costs; benefits and risks of the drug relative to therapeutically similar items; level of certainty associated with these costs, benefits and risks; and the resources available to the HSE [7].

  • For certain drugs, a final decision can be reached and the applicant company notified. In most cases, the drug will be considered by the HSE Drugs Group, who will make a reimbursement recommendation to the HSE Leadership Team, who in turn make the reimbursement decision [2, 4, 8].

Other studies analysing the time to reimbursement of drugs have been carried out in various other jurisdictions. Some of these studies have focused on time to reimbursement of cancer drugs specifically, whereas others analysed the timelines of all submissions submitted over a certain time period [9,10,11,12,13,14]. However, the HTA framework and timelines vary greatly between countries. The objective of this study was to analyse the timelines of the HTA process in Ireland for all RR and HTA submissions to the NCPE between 2015 and 2017. We were interested to see how the RR and HTA appraisal timelines compared with those outlined in the HSE-IPHA 2016 agreement. Furthermore, the time to submit RR and HTA were measured to identify how much these processes contributed to the overall HTA timeline. To do this, we focused on quantifying each stage of the HTA process from marketing authorisation (MA) to completion of HTA and also examined the features of submissions and the associations between these variables and assessment timelines.

2 Methods

2.1 Creation of the Dataset

All RRs and HTAs submitted to the NCPE between 1 January 2015 and 31 December 2017 were reviewed and included in the dataset. These dates were selected to minimise the amount of missing data and because assessments completed before 2015 may not reflect current timelines. A total of 12 HTAs that were commenced in 2015 had their corresponding RR submitted before 2015. To determine the impact of including these RRs, we created two datasets and ran analyses on both. Dataset 1 excluded the 12 RRs submitted before 2015; dataset 2 included them. Excluding the 12 RRs submitted before 2015 did not have a significant effect (p > 0.05) on the RR or HTA timelines (see Table 6 for the results of the statistical analysis). Therefore, we excluded the 12 RRs submitted before 2015.

Several variables were documented for each submission, including whether the drug was indicated to treat cancer or was granted orphan designation by the EMA. These variables were included as we suspected they may contribute to assessment time. Since orphan and cancer drugs represent an area of high unmet need, they are often approved by regulatory agencies based on single-arm trials, interim analysis results and non-validated surrogate endpoints [15]. Therefore, we anticipated that the considerable uncertainty associated with the clinical evidence for these submissions could have an impact on the assessment timelines. Submissions were categorised according to funding stream: community drugs scheme, high-tech drug scheme and hospital. The outcome of each RR and HTA was also recorded.

For the timelines analysis, several dates were extracted for each submission. The date of MA was obtained from the EMA website. Where the submission was for a licence extension, we used the date on which the licence extension was granted by the EC. The dates on which the RR was submitted and appraisal was completed by the NCPE was obtained from the NCPE website. If an HTA was recommended, several dates were recorded: date on which the HTA was commissioned by the HSE-CPU, date of the pre-submission meeting with the NCPE and date the HTA dossier was submitted by the applicant company and the appraisal was completed by the NCPE. These dates were sourced from the NCPE website and in-house records. Using these dates, the timeline from MA to completion of HTA appraisal was divided into four stages for the purpose of analyses:

  1. 1.

    Number of calendar days from the date the MA was granted by the EC to submission of RR by the applicant company

  2. 2.

    Number of calendar days to appraise RR

  3. 3.

    Number of calendar days from RR decision to submission of HTA

  4. 4.

    Number of calendar days to appraise HTA.

The number of calendar days to appraise the HTA was further divided into ‘applicant company days’ and ‘NCPE days’ so that the time taken by the applicant company to provide clarification and make amendments to the HTA submission could be accounted for in the total time to appraise an HTA.

2.2 Statistical Analyses

Statistical analyses were performed using the Statistical Package for Social Sciences (SPSS), version 24.0 (IBM Corp.; Armonk, NY, USA). All continuous variables were assessed for normality of distribution using the Kolmogorov–Smirnov (K–S) test. The histograms and normal Q–Q plots of each continuous variable were also assessed to confirm the results of the K–S test of normality. Analysis of variance (ANOVA) tests were performed to assess whether an association existed between certain variables and time to appraise RR and HTA. The variables assessed included drug category, funding stream, outcome of RR and outcome of HTA. The minimum level of statistical significance was 5% (p < 0.05). If ANOVA produced a statistically significant result (p < 0.05), further post hoc comparisons were carried out to quantify the differences. We used the Tukey Honestly Significant Difference (HSD) test because it compares all possible pairs of means and indicates which groups have statistically different means [16].

3 Results

3.1 Descriptive Statistics

From 1 January 2015 to 31 December 2017, a total of 161 RR submissions were received by the NCPE and 158 RR appraisals were completed. Figure 2 shows the movement of all RR and HTA submissions to the NCPE included in the dataset.

Fig. 2
figure 2

Movement of all rapid reviews and health technology assessments included in the dataset. HTA health technology assessment, MA marketing authorisation, NCPE National Centre for Pharmacoeconomics, RR rapid review. aNine RR submissions were excluded from analysis of MA to RR submission; three drugs were granted MA in 2018, five RRs were re-submissions, and one RR was a medical device and therefore did not have a date of MA. bIn one case, the applicant company chose to submit an HTA following a recommendation of ‘HTA at submitted price’. cOne submission was recommended for HTA following submission of new evidence. dTwelve HTAs that were appraised between 2015 and 2017 had their corresponding RR submitted before 2015

Full size image

Of the 158 RR appraisals completed between 2015 and 2017, a total of 78 (49%) were recommended for HTA. Further analysis was carried out on the 78 submissions for which an HTA was recommended. In total, 49 HTAs were appraised by 31 December 2017, two HTAs were resubmissions and one HTA was submitted without a preceding RR. At the time of analysis (31 December 2017), 14 submissions were going through the HTA appraisal process; 12 HTAs were not submitted to the NCPE within 6 months of the RR recommendation so were classified as ‘not submitted’ for the purpose of analysis. A total of 20 submissions were classified as ‘awaiting submission’, meaning the RR recommendation was made within the last 6 months and the NCPE had not yet received an HTA submission.

For 80 of the 158 (51%) RR submissions, an HTA was deemed unnecessary. Of these, 37 (46%) were recommended for reimbursement. Four submissions were not recommended for reimbursement and three were not recommended for reimbursement based on the proposed price. Three submissions were not recommended for HTA until further efficacy or safety data became available. Following this recommendation, one applicant company submitted new evidence and the NCPE recommended an HTA. In total, 33 (41%) submissions were recommended to reduce the price to avoid the need for an HTA (HTA at submitted price). Following this recommendation, only one applicant company chose to submit an HTA; the others agreed to reduce their price in order to be reimbursed.

For the 49 HTA appraisals completed by 31 December 2017, five drugs were recommended for reimbursement and 18 were not; 26 HTAs were not recommended for reimbursement unless cost effectiveness was improved relative to existing treatments (reimbursement not recommended at submitted price).

As described, the timeline from MA to completion of HTA was divided into several time periods for the purpose of analysis. Table 1 provides the mean and median number of days to complete each stage of the HTA process.

Table 1 Number of days to complete each stage of the HTA process
Full size table

The mean number of days from MA to RR submission was 201 days. The data are not normally distributed as there are a number of outliers, evidenced by the large standard deviation (SD) (477 days). The median time of 59 days indicates that, in many cases, the applicant company submitted an RR within a few months of the MA being granted. Only 21 (13.5%) of the 155 RR submissions (RR resubmissions and medical devices excluded) were submitted prior to the MA being granted. Nine RR submissions overall were excluded from analysis of MA to RR submission; three drugs were granted MA in 2018, five RRs were resubmissions and one RR was a medical device and therefore did not have a date of MA as it has a European Conformity mark. The NCPE do not usually appraise medical devices but on occasion are commissioned to carry out RRs on technologies that would be classified as medical devices.

The mean time to appraise an RR was 32 days; half of the RRs were appraised in ≤ 31.5 days. The timeframe outlined for the NCPE to appraise an RR in the HSE-IPHA agreement is 4 weeks (calendar days).

If a submission was recommended for an HTA, then we calculated the time taken by the applicant company to submit an HTA to the NCPE following this recommendation. A total of 63 (14 still undergoing appraisal at time of analysis) HTAs were submitted to the NCPE between 2015 and 2017. The median time from RR decision to submission of HTA was 115 days; the data were not normally distributed. The two HTA resubmissions and the HTA submission without prior RR were excluded from this analysis as there was no date of RR decision. The 12 HTAs that had their RR appraisal before 2015 were also excluded from this analysis. This time period was further divided into three stages: (1) time from RR decision to date HTA was commissioned by the HSE-CPU, (2) time from date HTA was commissioned until the pre-submission meeting with NCPE and (3) time taken by the applicant company to submit their HTA dossier following the pre-submission meeting. The median time from RR decision to the date the HTA was commissioned was 10 days (mean 18.54 ± SD 49.34; n = 39). Eight HTA commissioned dates and three pre-submission meeting dates were unavailable and therefore excluded from analysis. A further seven HTA submissions had their pre-submission meeting take place before the HTA was commissioned; these were exceptional cases and do not reflect regular practice, so these dates were also excluded from the analysis. The median time from date HTA commissioned to pre-submission meeting was 40 days (mean 72.43 ± 105.50; n = 40). The median time from pre-submission meeting to submission of HTA was 76 days (mean 101.71 ± 86.95; n = 49). On average, approximately 58% of the entire time period from RR decision to submission of HTA was taken up by the applicant company preparing the HTA dossier for submission to the NCPE following the pre-submission meeting.

In total, 49 HTAs were fully appraised between 2015 and 2017. The mean ± SD number of days taken by the NCPE to appraise the dossier was 133.33 ± 57.33 (median 131). The mean number of days taken by the applicant company to answer preliminary questions and carry out the factual accuracy check was 57.78 ± 41.27 (median 42). This corresponds to each HTA submission spending an average of 70% of the total time period with the NCPE and 30% of the time with the applicant company.

3.2 Comparison of Assessment Times

The results of the association between drug category and time to appraise RR and HTA are presented in Table 2. The number of days to appraise the RR and HTA yielded a non-significant K–S statistic (p > 0.05), indicating a normal distribution of scores. ANOVA indicated there was insufficient evidence to conclude whether there was a difference between drug categories for number of days to appraise RR [F(3, 154) = 0.042; p = 0.988] or the number of days to appraise HTA [F(3, 45) = 2.143; p = 0.108].

Table 2 Mean and median time to complete rapid review and health technology assessment appraisal stratified according to drug category
Full size table

The results of the association between funding stream and time to appraise RR or HTA are presented in Table 3. Evidence was insufficient to conclude that there was a difference between the groups at a 5% level for RR [F(2, 155) = 0.180; p = 0.835] and HTA [F(2, 46) = 0.078; p = 0.925].

Table 3 Mean and median time to complete rapid review and health technology assessment appraisal stratified according to funding stream
Full size table

The results of the association between outcome of RR and time to appraise RR are presented in Table 4. There was a statistically significant difference in number of days to appraise RR between groups at a 5% level [F(5, 152) = 3.061; p = 0.012]. Post-hoc comparisons using the Tukey HSD test indicated that the mean number of days for RRs that were recommended for HTA was significantly different (p = 0.018) from those with a recommendation of HTA at submitted price [17].

Table 4 Mean and median number of days to complete rapid review appraisal, stratified according to outcome
Full size table

The results of the association between outcome of HTA and time to appraise HTA are presented in Table 5. However, evidence was insufficient to conclude that there was a difference between the groups at a 5% level [F(2, 46) = 2.399; p = 0.102].

Table 5 Mean and median number of days to complete health technology assessment appraisal stratified according to outcome
Full size table

4 Discussion

The time from MA to submission of RR by the applicant company was a major contributor to the overall timeline from MA to completion of HTA. In 21 cases, the applicant company chose to submit an RR before an MA was granted; the rest chose to submit after an MA was granted. The median result suggests that most companies chose to submit an RR within a few months of the MA. However, there were several outliers in the sample where the applicant company waited for more than a year before submitting to the NCPE. The reason for this is unclear but may be because the applicant company chose to submit to other European countries first. Therefore, the time period between MA and RR submission varied greatly across submissions.

The mean time to appraise RR suggests that, on average, RR appraisals are completed in just over 4 weeks. This finding highlights the efficiency of the two-step process. The requirement for all submissions to initially undergo an RR resulted in just under half (49%) of the submissions being recommended to undergo further assessment. This system allows the NCPE to spend more time carrying out extensive HTAs on the submissions that require a more in-depth analysis. Murphy and Redmond [18] recently suggested that compulsory RRs delay initiation of HTAs. They proposed that an opt-in or opt-out approach to RRs may be a better way to optimise the NCPE’s resources; however, they highlighted that formal decision criteria would be required for this to be possible [18]. We acknowledge that, for certain submissions, the requirement for HTA may be obvious; however, our study highlights that, for several submissions, the decision to recommend an HTA was not clear cut. In total, 33 (41%) RR submissions in this study were recommended ‘HTA at submitted price’; this recommendation represents a borderline decision and allows the applicant company the opportunity to reduce their price to avoid an HTA or to submit an HTA if they wish. Nearly all applicant companies chose to reduce their price rather than submit an HTA following this recommendation (at the time of analysis, only one applicant company chose to submit an HTA following this recommendation). Therefore, an opt-in or opt-out RR process could result in several HTAs being carried out where an RR would have been sufficient to make a reimbursement decision. Furthermore, for the submissions recommended for HTA, the RR appraisal provides the opportunity for the NCPE to identify any important issues prior to the submission of the HTA. These issues can then be highlighted during the pre-submission meeting and addressed by the applicant company prior to dossier submission.

The mean time to appraise the HTA was outside the 90-day timeframe. However, this 90-day timeframe refers specifically to the time spent by the NCPE appraising the HTA. Therefore, when the applicant company days were accounted for, the mean number of days for the NCPE to appraise an HTA was 133, which is considerably closer to the 90-day timeframe. A limitation of this study is that the date the HTA was submitted does not reflect the date the HTA appraisal actually commenced, as the NCPE does not operate timed submissions. If the NCPE receives several submissions simultaneously, there may not be sufficient resource to commence the assessment process for each submission immediately. This is one of the factors that contribute to the number of days to appraise HTA falling outside the 90-day timeframe. A possible solution to this could be the introduction of timed submissions, as this would prevent several submissions being submitted simultaneously.

We explored several variables to determine whether an association existed between the type of submission and time to complete assessment. However, evidence was insufficient to conclude that the differences observed were statistically significant at a 5% level. Casson et al. [10] examined the differences in timelines for cancer-related and non-cancer-related HTAs but found that the difference in timelines was no longer statistically significant when adjustments were made for other variables.

There was an association between outcome of assessment and time to complete assessment. The results indicated that RRs that were recommended for HTA took significantly less time than those that were recommended to reduce their price to avoid an HTA. A potential explanation for this is that, during the initial stages of the RR process, it is apparent when an HTA will be required as a further, more detailed, assessment is needed before a decision can be made regarding reimbursement. When HTA outcomes were examined, evidence was insufficient to conclude whether the differences observed were statistically significant at a 5% level.

The limitations of this study include that the date the HTA was submitted does not reflect the date the HTA appraisal commenced, so the results presented here are not a true reflection of the actual time to appraise an HTA. Also, the date the HTA was commenced was not always recorded, so it was not possible to compare it with the HTA submitted date. Furthermore, prior to 2015, several HTA commissioned dates and pre-submission meeting dates were missing; therefore, the timeline analysis had to be restricted to 2015 onwards.

5 Conclusions

To the best of our knowledge, this is the first paper to analyse the HTA timeline in Ireland by quantifying each stage of the HTA process individually. Time from MA to submission of RR was found to contribute substantially to the overall timeline for several submissions. While the applicant companies can submit an RR immediately following a positive CHMP opinion, only a few companies chose to do so (13.5%). This paper has demonstrated the efficiency of the two-step process, negating the need for unnecessary HTAs when an RR is sufficient to make reimbursement recommendations. Time to submit an HTA following RR decision varied between submissions, so this time period was a considerable contributor to the overall HTA timeline for some submissions. The mean time to appraise an HTA was outside the 90-day timeframe. However, this was most likely as a result of a delay between the HTA being submitted and the HTA being commenced. Several factors may have played a part in this, including lack of control over the timing of submissions and the impact of this on resources. Since this analysis was conducted, resourcing of the NCPE has been increased. Further analysis will be required to determine the impact of this on appraisal timelines.