Background

Coverage decision-making on new drugs for rare diseases (DRDs) challenges public and private payers, as well as the pharmaceutical industry in Canada and abroad. In addition to high per patient treatment costs, DRDs typically come with a paucity of outcomes data due in part to the small number of patients available for studies. This creates significant uncertainty around their value proposition. At the same time, DRDs often target diseases with high clinical need (i.e., they are life-threatening and/or chronically debilitating and lack active treatment/disease-modifying alternatives) [1, 2].

In Canada (unlike other countries), participating public payers rely on one of two centralised review processes (depending on disease type) to provide coverage recommendations on new DRDs. Prior to 2003, recommendations were made by individual jurisdictions (e.g., provinces and territories) through separate provincial/territorial processes or, in the case of cancer drugs, jurisdictional cancer agencies. In 2003, the centralized Common Drug Review (CDR) was launched as a means of improving the efficiency of such processes and providing high quality evidence-informed recommendations on non-cancer drugs to guide coverage decisions in all participating jurisdictional drug plans. In 2011, a parallel process, the pan-Canadian Oncology Drug Review (pCODR), was established for new cancer drugs. Both of these centralized processes are overseen by the Canadian Agency for Drugs and Technologies in Health (CADTH) [3, 4]. Although CADTH broadly specifies factors considered when formulating recommendations (clinical benefit, cost, cost-effectiveness, and patient preferences), how they are weighed relative to one another and whether it varies under certain conditions remain unclear [5].

In recent years, several analyses of centralized drug review processes have been published [6,7,8,9,10,11,12]. While most have focused on drugs for common conditions, three have assessed trends and factors associated with different types of coverage recommendations for DRDs. However, these studies excluded cancer drugs and examined only a small number of decision factors [6, 7, 9].

This study aimed to address the following questions:

  1. 1.

    Are certain factors associated with positive or negative reimbursement recommendations for drugs submitted to CDR and pCODR?

  2. 2.

    Does whether or not the DRD is for a cancer indication affect recommendation type?

Methods

A quantitative analysis was conducted to address these questions. The identification and extraction of data from CDR and pCODR submissions followed guidelines for conducting systematic reviews [13].

Identification of DRDs

All DRD recommendations from CDR and pCODR from January 2012 until April 2018 were included in the study. Before 2011, submissions for cancer drugs were reviewed by the interim Joint Oncology Drug Review (iJODR) and information on recommendations was not made publicly available. After 2011, the iJODR was formalized to a permanent body known as pCODR managed by CADTH and the first recommendation was published in 2012. Also, previous studies have shown differences in factors associated with CDR recommendations before and after the establishment of pCODR (i.e. 2012, [7, 9]. Therefore, to make recommendations comparable, CDR submissions prior to 2012 were excluded from the analyses. A rare disease was defined as a condition affecting less than 1 in 2000 people in Canada (the definition proposed in the draft pan-Canadian framework for rare diseases which had been developed by Health Canada, the national regulating body [14]). Ultra-rare diseases were defined as those with a prevalence of less than 1 in 50,000 people [15]. To identify DRDs, prevalence information was obtained from two sources: 1) the Orphanet website, an internationally governed portal for rare disease information containing a comprehensive list of rare diseases [16], and 2) a comprehensive search of published and grey literature for Canadian prevalence estimates (details of the search strategy are available from the authors).

Collection of data on included DRDs

For each included DRD, the CDR and pCODR “Final Recommendation” documents were obtained from their respective websites [3, 4]. These documents are issued by CADTH once a reimbursement recommendation is made by an independent review committee and provide the reasons for the recommendation, considering: current available evidence on safety and clinical effectiveness, cost-effectiveness, patient input, characteristics of the drug and disease, and feasibility of adoption of the current technology (e.g. budget impact analysis and organization feasibility) [3, 4].

Data extraction

For each included DRD, two researchers independently reviewed the relevant “Final Recommendation” document and extracted the following information using a standardized form [13]: submission type, drug name, drug type, condition/indication, final recommendation, year of final recommendation, number of randomized clinical trials (RCTs), number of patients from studies, clinical safety and efficacy/effectiveness, quality of evidence (e.g. presence of bias in outcome measures, availability of comparative data), treatment cost, and cost-effectiveness.

Data analysis

Creation of variables

For each included DRD, the final recommendation was converted into a binary outcome variable coded as positive if the recommendation was to ‘list’ the drug (i.e., include it in a participating publicly funded drug benefit plan) with or without conditions and negative if the recommendation was to not ‘list’ the drug. Factors were converted to categorical variables characterizing the submissions, including the type of submission (new or resubmission), prevalence of the condition (orphan or ultra-orphan) and type of drug (alimentary tract/metabolism product, antineoplastic/immunomodulating agent or other) were created. Four binary variables (‘yes’ or ‘no/ not measured’) were created to describe the presence or absence of meaningful improvements across efficacy and effectiveness outcomes: 1) differences in clinical outcomes, 2) differences in biomarker/surrogate outcomes, and 3) differences in patient reported outcomes (PROs). Classification of the outcomes was based on the definitions described in the “Final recommendation” documents. The following binary (‘yes’ or ‘no’) variables were also created: safety issues, bias in outcome measures, consistency between the patient population in trials and indication(s) for which a reimbursement/listing recommendation was sought, availability of direct comparative data, availability of long-term data, and presence of other methodological or study design issues. A detailed description of these variables is provided in Table 1.

Table 1 Description of variables included in the analyses
Full size table

Statistical analysis

First, a series of two-by-two or three-by-two tables were constructed to examine the percentage of positive recommendations for each variable extracted from the “Final recommendation” document. Data were tabulated for all included recommendations and stratified by type of condition (i.e. cancer and non-cancer) to examine whether the frequency of positive and negative recommendations for each factor (i.e., independent variable) varied with type of condition. Pearson’s chi-square or Fisher’s exact test were used to test the statistical significance of differences in such percentages. This step was also used to check for any errors and spot complete and quasi-complete separation of data (i.e. recommendations were almost perfectly predicted by the independent variables).

Next, factors potentially associated with recommendation type were further explored through multiple logistic regression- a statistical analysis that allows for the assessment of the association between multiple factors and a dichotomous outcome (in this case, positive or negative recommendation) [18]. Two methods for building regression models were used and the results compared: 1) purposeful selection and 2) stepwise selection.

In purposeful selection, covariates whose univariate test had a p-value < 0.21 were first identified [18, 19]. A multivariable model containing these covariates was constructed, and variables with p-values > 0.21 were excluded. Each variable not selected initially for inclusion in the multivariable model was then added one at a time. If its p-value was > 0.05 and none of the coefficients in the model changed by > 20%, the variable was excluded. The resulting model comprised the main effects model. Finally, two-way interactions among the variables were added to the main effects model one at a time and checked for statistical significance. Those with p-values > 0.05 were excluded. To assess the fit of final model, the Hosmer-Lemeshow goodness-of-fit test was used [20, 21].

In stepwise selection, each variable was entered into the model step by step (SAS® Stepwise Logistic Regression). The significance level for entry and stay were set at 0.2. The results were identical with purposeful method.

Results

Initially, 104 submissions (42 CDR and 62 pCODR) with final recommendations on DRDs were identified. Fifteen were excluded (11 resubmissions and 4 with no daily treatment cost information), leaving a total of 88 submissions comprising 92 final recommendations for inclusion in the analysis (Fig. 1). Resubmissions (n = 11) were excluded since they all received a positive recommendation. Of the 103 recommendations, 82 (80%) were positive (Table 2). Most recommendations were for antineoplastic & immunomodulating therapies, but the proportion of positive recommendations among different ‘therapeutic class of drugs’ were similar.

Fig. 1
figure 1

Flow diagram of the search and selection of submissions of DRDs

Full size image
Table 2 Overall description of included recommendations
Full size table

Figure 2 shows the number of recommendations on new submissions made each year since 2012. From 2012 to 2014, the average was around 9 per year, whereas after 2015, the average increased to 19 (2018 was excluded since the data were only available for the first quarter of the year). Overall, the proportion of positive recommendations on new submissions remained high (ranging from 63 to 100%). While the proportion of positive recommendations on cancer DRDs remained similar over the years, that for non-cancer DRDs increased after 2015 (Fig. 3). However, the number of non-cancer DRD submissions was also small prior to 2015.

Fig. 2
figure 2

Distribution of recommendations of new submissions by year of final recommendation

Full size image
Fig. 3
figure 3

Percentage of positive recommendations by year of recommendations for: (a) all new submissions and (b) all new submissions stratified by type of condition

Full size image

Table 3 provides information on the distribution of positive recommendations according to each potential decision factor and stratified by cancer and non-cancer drugs. Twenty-five (96%) out of 26 recommendations showing ‘improvement in clinical outcomes’ were positive. Only one in cancer reporting ‘improvements in clinical outcomes’ was negative. Likewise, 24 of 25 (96%) recommendations reporting ‘improvements in PROs’ were positive and the proportion of positive recommendations was similar for cancer and non-cancer DRDs.

Table 3 Distribution of positive recommendations for all new submissions of DRDs
Full size table

The proportion of positive recommendations for those with no ‘safety issues’ was similar between non-cancer (83%) and cancer (86%) DRDs. However, only 40% of cancer drugs showing no evidence of ‘improvements in biomarker/ surrogate outcomes’ received a positive recommendation, versus 88% of non-cancer drugs.

Neither ‘daily treatment cost’ nor ‘ICERs’ was associated with recommendation type, but the distribution of recommendations related to these factors appeared to differ between cancer and non-cancer DRDs (Table 3).

After controlling for potential confounders through multiple logistic regressions, only three factors were identified as statistically significantly associated with recommendation type: 1) ‘Safety issues’, 2) ‘Improvement in clinical outcomes’, and 3) ‘Improvement in patient reported outcomes’ (Table 4). In all three, DRDs that did not reported ‘safety issues’ or showed significantly improved clinical outcomes or patient reported outcomes were more likely to receive a positive listing recommendation. No factors were found to be statistically significantly associated with negative recommendations.

Table 4 Results of multiple regression analysis of DRDs
Full size table

Discussion

This study examined the potential relationship between factors considered during deliberations by centralized drug review committees in Canada and final recommendation. Cancer drugs were no more or less likely to receive a positive recommendation than those for non-cancer. Similarly, no correlation between per patient treatment cost or size of ICER and type of recommendation was found, suggesting that the economic implications of a DRD are not driving deliberations. In contrast, DRDs that offered improvements in clinical outcomes, or PROs were more likely to receive positive recommendations. Similar to our results, two previously published studies of CDR recommendations determined that clinical effectiveness was a strong predictor of recommendation type and there was no association between the size of the ICER and recommendation type [9, 10]. It may be argued that decision makers consider economic models useful when supported by strong clinical evidence [22], which is not usually available for DRDs [23]. In the case of DRDs, other societal considerations such as the principle of social solidarity and the right to health may play a more important role [23, 24].

In recent years, CADTH has made efforts to better align the CDR and pCODR processes [5, 25]. The lack of differences in recommendations based on type of indication suggests that their efforts have been successful. This now raises questions around the need for two review processes. To our knowledge, no other countries with centralized drug reviews have created separate processes for cancer drugs. While previously published studies have found no rationale for the establishment of pCODR [26], it has been argued that while unclear, there may be good reasons [27].

Overall, the proportions of positive recommendations on new submissions were high, although they fluctuated between 2012 and 2015. From 2012 to 2018, two changes in the deliberative framework may have contributed to the increase in positive recommendations. In November 2012, CADTH published a framework for CDR in which price reduction was added as a condition for listing the drug. The framework also included a category of “do not list at submitted price”, which before 2012, was a subcategory of the “do not list” category [28]. In March 2016, the wording of recommendations was modified once again and the categories of recommendations were reduced to three: “reimburse”, “reimburse with clinical criteria and/ or conditions” and “do not reimburse”. A negative recommendation around price no longer appears to exist, and costly drugs or those with unfavourable ICERs can receive a recommendation under the category of “reimburse conditional of reduced price”. Further, this framework provides the option of issuing a positive recommendation in “exceptional cases” in which there are uncertainties around the effectiveness of a drug. “Exceptional cases” may be relevant to rare diseases (i.e., the drug is for life-threatening conditions and/ or affects a small population) [5]. While our data show an increase in positive recommendations since 2016, further research with long-term data is required in order to investigate the impact of these changes in a more robust way.

Limitations

This study has five main limitations. First, for some recommendations, a judgement call was required in order to classify them as positive or negative. For example, “list if … substantial reduction in price” may be considered a negative recommendation because it could ultimately yield a negative reimbursement decision. However, similar to previously published work, we considered a positive recommendation to be one in which manufacturers were able to proceed to the next stage, which was price negotiation and reimbursement [29]. Second, the sample size was small and information on some of the variables was not available. Third, the association between feasibility, one of the factors described in documents emerging from pCODR deliberations, and recommendation type was not evaluated, since similar information was not available for drugs reviewed by the CDR. Nonetheless, adoption feasibility takes into account budget impact, which may be an important consideration during pricing and reimbursement decision-making [30]. Fourth, biomarker and surrogate outcomes were included in the same category due to small sample size. According to the FDA, surrogate outcomes are biomarkers that can predict clinical outcomes [17, 31]. In the case of DRDs, particularly those for non-cancer indications, long-term studies designed to gather information on the natural progression of the disease are lacking. Consequently, there is a reliance on biomarkers as surrogates for clinically meaningful outcomes. Finally, with the exception of obtaining disease prevalence rates from external sources to determine eligibility of the DRD for inclusion in the study, analyses were solely based on information reported in the recommendation documents available on the CADTH website. It was not possible to determine the extent to which these documents provided an accurate reflection of the full deliberative process that took place when formulating these recommendations.

Conclusion

Whether a new submission is for a cancer drug or for a non-cancer drug does not appear to affect its likelihood of receiving a positive reimbursement recommendation. Safety and clinical effectiveness, not costs, appear to be the key drivers of the type of reimbursement recommendation.