PharmacoEconomics

, Volume 34, Issue 3, pp 285–301 | Cite as

Can the EVIDEM Framework Tackle Issues Raised by Evaluating Treatments for Rare Diseases: Analysis of Issues and Policies, and Context-Specific Adaptation

  • Monika Wagner
  • Hanane Khoury
  • Jacob Willet
  • Donna Rindress
  • Mireille Goetghebeur
Open Access
Original Research Article

Abstract

Background

The multiplicity of issues, including uncertainty and ethical dilemmas, and policies involved in appraising interventions for rare diseases suggests that multicriteria decision analysis (MCDA) based on a holistic definition of value is uniquely suited for this purpose. The objective of this study was to analyze and further develop a comprehensive MCDA framework (EVIDEM) to address rare disease issues and policies, while maintaining its applicability across disease areas.

Methods

Specific issues and policies for rare diseases were identified through literature review. Ethical and methodological foundations of the EVIDEM framework v3.0 were systematically analyzed from the perspective of these issues, and policies and modifications of the framework were performed accordingly to ensure their integration.

Results

Analysis showed that the framework integrates ethical dilemmas and issues inherent to appraising interventions for rare diseases but required further integration of specific aspects. Modification thus included the addition of subcriteria to further differentiate disease severity, disease-specific treatment outcomes, and economic consequences of interventions for rare diseases. Scoring scales were further developed to include negative scales for all comparative criteria. A methodology was established to incorporate context-specific population priorities and policies, such as those for rare diseases, into the quantitative part of the framework. This design allows making more explicit trade-offs between competing ethical positions of fairness (prioritization of those who are worst off), the goal of benefiting as many people as possible, the imperative to help, and wise use of knowledge and resources. It also allows addressing variability in institutional policies regarding prioritization of specific disease areas, in addition to existing uncertainty analysis available from EVIDEM.

Conclusion

The adapted framework measures value in its widest sense, while being responsive to rare disease issues and policies. It provides an operationalizable platform to integrate values, competing ethical dilemmas, and uncertainty in appraising healthcare interventions.

Key Points for Decision Makers

The adapted EVIDEM framework provides an operationalizable platform to integrate individual and social values, competing ethical dilemmas, and uncertainty, which are particularly challenging in appraising interventions for rare diseases.

The addition of subcriteria to further differentiate disease severity, disease-specific treatment outcomes and economic consequences of interventions, inclusion of negative scoring scales for all comparative criteria, and integration of a methodology to further incorporate context-specific population priorities and policies makes the EVIDEM framework particularly responsive to rare diseases, while maintaining its applicability across disease areas.

Additionally, a method was developed to address variability in country or institution policies with respect to prioritization of specific disease areas.

1 Introduction

Appraising the value of interventions for rare diseases raises a number of difficult issues. Many of these conditions are severe, chronic, progressive, and life-threatening, with multiple medical, psychological, and social consequences [1, 2]. Their rareness is a major barrier to developing targeted therapeutic interventions due to both economic and scientific constraints. As a consequence, unmet needs remain significant and widespread [3]. Uncertainty is also a critical issue in rare diseases, and includes limitations in our understanding of natural history [4] as well as challenges in obtaining high-quality epidemiological [5, 6, 7, 8], clinical [9, 10], and economic data. In addition to uncertainty on evidence, there is variability in country or institution policies and across stakeholders on the values at stake. Regulatory policies to support the development of interventions for rare diseases are in place [11, 12] but, once approved, appraising these often highly-priced products for reimbursement is a challenge for health technology assessment (HTA) [13, 14, 15]. Some agencies acknowledge that their standard appraisal approaches need to be modified for orphan products [13], and a few have developed explicit approaches, including the National Institute for Health and Care Excellence (NICE) Interim Process for Highly Specialized Technologies (HST) [16] and the Ontario Public Drugs Program framework [17].

The multiplicity of issues involved in appraising interventions for rare diseases, as highlighted in the NICE interim HST process [16], suggests that multicriteria decision analysis (MCDA)-based approaches that apply a holistic definition of value are uniquely suited for this purpose. MCDA makes explicit which criteria are used and how they are balanced in decision making. Several multicriteria approaches designed for appraising interventions for rare diseases have been proposed, suggesting different sets of decision criteria that were selected based on various approaches [18, 19, 20]. To be justifiable, decisions regarding healthcare interventions, and thus the criteria included in a multicriteria framework, must be based on reasons that reflect society’s ethical principles and substantive values, concepts such as fairness and efficiency [21, 22], while the decision-making process must reflect procedural values such as transparency, accountability, and participation [21] to help legitimize decisions by upholding procedural justice [21, 23, 24]. Thus, there is a need for a comprehensive appraisal framework that features decision criteria that are selected based on ethical principles (i.e. social substantive values) and reflects social procedural values.

EVIDEM is an open-source, collaboratively-developed [25, 26, 27, 28, 29, 30] MCDA framework designed to appraise the holistic value of healthcare interventions [31]. It has been tested [26, 27, 32, 33] and implemented [34, 35] in various real-world decision-making settings. The EVIDEM framework v3.0 is structured around the key objectives (domains) that govern healthcare decision making and define the holistic value of interventions, i.e. to optimize the health of patients, populations, and healthcare systems. Attainment of these objectives is assessed using operationalizable criteria, each of which is rooted in established ethical positions [36, 37], including deontology (imperative to help, beneficence, nonmaleficence) [38, 39, 40, 41], utilitarianism (greatest good for the greatest number) [42, 43], distributive justice and fairness (prioritize those who are worst off) [43, 44], and virtue ethics and practical wisdom [45]. The framework was designed to meet MCDA design principles, i.e. each criterion must make a unique contribution to the value of an intervention, while the criteria set needs to be complete, operationalizable, and free of redundancies and mutual dependencies [46]. Objectives are classified into normative and feasibility domains. In the quantitative part of the framework, criteria are weighted and scored to obtain a numerical measure of the value of an intervention (Value Estimate). The qualitative part captures the impact on value of those criteria that are difficult to quantify. Key procedural values underpinning EVIDEM are transparency, pragmatism, and participatory decision making [25, 26, 27].

Because of its holistic definition of value, explicit rootedness in ethical principles and real-world application, the EVIDEM framework was selected for this study to be analyzed and further developed into a comprehensive MCDA approach that addresses rare disease issues and policies, while maintaining its applicability across disease areas.

2 Methodology

2.1 Identification of Specific Issues Raised by Rare Diseases

A systematic literature review was carried out to identify issues in rare diseases that potentially impact on value appraisal as well as relevant current or proposed regulatory and reimbursement policies. Full-text publications related to disease impact (including size of population), context of interventions, clinical outcomes, costs, ethical issues, and licensing and reimbursement policies were included. Case reports and articles of clinical focus describing the natural history, clinical presentation, or treatment of specific rare diseases were excluded. The principal sources of information were peer-reviewed publications identified from the PubMed/MEDLINE databases (published from January 2003 to September 2013). Using keywords such as ‘orphan disease’ and ‘rare disorder, the PubMed/MEDLINE searches yielded 6525 citations, 115 of which were reviewed in full text, and 41 [2, 3, 4, 5, 6, 7, 8, 9, 10, 17, 18, 19, 20, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74] included in the analysis, including five publications describing current or proposed appraisal frameworks for rare diseases [17, 18, 19, 20, 68]. Additionally, websites of major regulatory and HTA agencies, rare disease organizations and networks (e.g. ORPHANET, EUROPLAN, Office for Rare Disease Research), and bibliographies of key publications were searched for relevant information.

The selected publications were analyzed to identify issues pertaining to value appraisal. The relevant issues were classified by the decision criterion to which they pertain, using the criteria definitions and the structure of the EVIDEM framework as an analytical tool.

2.2 Analysis and Adaptation of the EVIDEM Framework from the Perspective of Issues Raised by Rare Diseases

Ethical and methodological foundations of the EVIDEM framework v3.0 [36, 37] were analyzed on a criterion-by-criterion basis to assess their relevance and implications from the perspective of issues raised by rare diseases. This analysis was used to determine whether a modification of the framework was required to enhance its usefulness for appraising interventions for rare diseases, while preserving its applicability across disease areas. Potential modifications included:
  • definition of subcriteria to allow a more differentiated assessment of specific criteria;

  • further development of scoring scales;

  • structural transformation of the framework by moving criteria that are assessed qualitatively into the quantitative part of the framework to integrate context-specific policies and priorities.

3 Results

The adapted EVIDEM framework is shown in Fig. 1 (see Online Resource 1 for the full framework). For the quantitative part of the framework (a), hierarchical point allocation was selected as the primary weight elicitation technique, a method that allows direct expression of personal values and makes explicit the trade-offs that need to be made across the criteria [75]. As in the original framework, weighting is performed independently of the intervention appraised (generic weighting), with the exception of the subcriteria defining intervention outcomes, for which disease-specific weights are elicited to define the relative weight of each outcome as part of the Comparative Effectiveness criterion. Evaluators score the intervention with respect to each criterion using constructed, cardinal scoring scales, designed to measure their judgments on the evidence presented. These scales include scores of zero, corresponding to situations where the intervention has no value with respect to a criterion. The framework was adapted to include negative scores for all comparative criteria to reflect worse outcomes or economic consequences than comparators, which are relevant to rare diseases, but also in general for fair appraisal of interventions. The impacts of contextual criteria on the value of the intervention are considered qualitatively using a separate tool (b).
Fig. 1

Adapted EVIDEM MCDA framework: (a) quantitative MCDA model (with hierarchical structure and weighting method); and (b) contextual criteria for qualitative appraisal. MCDA multicriteria decision analysis; PRO patient-reported outcome

Criteria-specific analyses and adaptations are summarized in Table 1 and are reported below.
Table 1

Adapted MCDA framework structured by objective with ethical foundations, value contributions, rare disease issues, and adaptations made for each criterion

Objectives/criteria

Ethical foundation

Value contribution

Rare disease issues

Adaptation/comments

Normative universal objectives: quantitative appraisal

 Objective: addressing areas of high therapeutic need

  Disease severity

Alleviate suffering in those who are worst off (theory of justice)

Interventions targeting (i.e. preventing, curing, or alleviating) severe diseases have higher value than those targeting less severe diseases

Severe, chronic, progressive, and life-threatening diseases, with multiple medical and nonmedical (i.e. psychological, social) consequences [1, 2, 3, 57, 72]

Significant impact on caregivers’ QoL [1, 2, 3, 69]

Lack of knowledge/data on natural history [4]

Addition of subcriteria defining all key aspects of disease severity:

Effect on life expectancy

Effect on morbidity (includes disability and function)

Effect on patient QoL

Effect on caregiver QoL

  Size of population affected

Alleviate suffering in as many individuals as possible (utility theory)

Interventions benefiting large numbers of individuals have greater value than those benefiting few individuals

Definition of rare disease

Rarity in itself not a prioritization criterion [51, 52, 63, 66]

Challenge in obtaining high-quality epidemiological data [5, 6, 7, 8]

Specification of scoring scale options with standard definitions of rare and ultra-rare diseases

  Unmet needs

Alleviate suffering in individuals with limited alternative interventions (theory of justice)

Interventions for which there are no alternatives or where alternative interventions have major limitations have greater value than those for which there are alternatives that have only minor limitations

Limited number of targeted treatments [3]

Limited number of clinicians familiar with a rare condition [3]

Limited access to appropriate health services [3]

No adaptation required

 Objective: providing large improvements in health outcomes

  Comparative effectiveness

Alleviate suffering to the greatest extent (beneficence, deontology)

Interventions that provide major improvements in efficacy/effectiveness have greater value than those with lower efficacy/effectiveness than comparators

Need to define most relevant efficacy/effectiveness outcomes for each disease [9, 59, 61, 73]:

Outcomes that reflect natural course of disease

Outcomes that detect clinically meaningful improvement

Value of short-term surrogate outcomes vs. long-term outcomes

Well-powered RCTs may sometimes not be feasible for rare diseases [9, 10]. Alternative trial designs, modeling, observational studies and registries may provide clinical data [9, 10, 49, 58]

Disease-specific outcomes modeled as subcriteria

  Comparative safety/tolerability

Hippocratic Oath “do no harm” (nonmaleficence, deontology)

Interventions that provide major improvements in safety/tolerability have greater value than those with lower safety/tolerability than comparators

Reduced statistical power of results due to small sample size and small number of trials for a given intervention [59]

Risk of rare serious AEs difficult to estimate in a small patient population

Long-term safety issues may not be captured due to short follow-up duration [59]:

Need for postmarketing safety data/outcomes registries

Need to define most relevant safety/tolerability outcomes for each intervention

Disease-specific outcomes modeled as subcriteria

  Comparative patient perceived health/PROs

Alleviate suffering as perceived by the patient (Hippocratic Oath « for the good of my patient ») (deontology)

Respect for autonomy

Interventions that provide major improvements in PROs have greater value than those with worse PROs than comparators

Limited availability of disease-specific PRO instruments [69]:

Not systematically captured in trials; small sample size; short trial duration; absence of relevant comparator

Selection of HRQoL instruments may not be appropriate [69]:

Do generic instruments (vs. disease-specific) capture the relevant QoL impact of each disease?

Is HRQoL measure validated? Can it detect minimally important differences [69]?

Disease-specific outcomes modeled as subcriteria

 Objective: delivering important types of health benefit

  Type of preventive benefit

Protect health and prevent suffering (deontology)

Interventions that provide major reductions in disease risks have greater value than those providing no reduction in disease risks

Most rare diseases are genetic [2]; thus, interventions are generally not preventive

No adaptation required

  Type of therapeutic benefit

Aim to eliminate rather than merely alleviate suffering (deontology)

Interventions that provide major therapeutic services to the patient (e.g. cure) have greater value than those providing minor services (e.g. relief from a minor disease symptom)

Limited data on type of therapeutic benefit due to lack of long-term studies [59]

No adaptation required

 Objective: producing favorable economic consequences

  Comparative cost consequences—cost of intervention

Use scarce resources wisely to maximize health from a specific budget perspective (practical wisdom, utilitarianism)

Interventions that reduce treatment costs have greater value than those that increase treatment costs

High cost per patient but budget impact of single product relatively low [18, 50, 67, 71]

Introduction of negative scoring scale

  Comparative cost consequences—other medical costs

Use scarce resources wisely from a broad perspective (practical wisdom, utilitarianism)

Interventions that free-up other medical resources have greater value than those that require the use of additional medical resources

Patients with rare disease can use a broad range of medical resources and services, including specialist care, medical examinations, and hospitalization [3]

Addition of subcriteria defining the payer:

Other medical costs to healthcare system

Medical cost to patient

Introduction of negative scoring scale

  Comparative cost consequences—nonmedical costs

Preserve societal and individual resources wisely from a broad perspective (practical wisdom, utilitarianism)

Interventions that preserve and free-up nonmedical resources have greater value than those that require the use of additional nonmedical resources

Particular need in rare disease to take a broad (i.e. societal) perspective on economic consequences:

Economic impact on social services [3]

Economic impact of disease and its treatment on patients and their families [2, 70]

Addition of subcriteria defining type of cost and payer:

Patient/caregiver productivity

Cost to wider social care system

Nonmedical costs to patient

Introduction of negative scoring scale

 Objective: reducing uncertainty through solid knowledge

  Quality of evidence

Consider strength of claims about the intervention based on formal evidence (imperative of evidence-based decision making, practical wisdom)

Interventions for which evidence reporting is complete and consistent, relevant to the decision to be made, and valid with respect to international scientific standards have greater value than those for which evidence reporting is incomplete and inconsistent and the evidence has low relevance and validity

Rareness of a condition affects data quality in all areas affecting drug appraisal, including epidemiology [5, 6, 7, 8], clinical efficacy and safety [9, 53, 59, 73], and economic consequences

Addition of subcriteria by field of research:

Clinical evidence

Epidemiology evidence

Economic evidence

  Expert consensus/clinical practice guidelines

Consider strength of claims about the intervention based on expert knowledge and consensus (practical wisdom)

Interventions strongly recommended on the basis of current expert consensus have greater value than those not recommended by clinical experts

Limited number of clinical experts [1, 64]

Availability and quality of clinical guidelines [55]

No adaptation required

Normative contextual objectives: qualitative appraisal

 Objective: aligning with the mandate and scope of healthcare system

  Mandate and scope of healthcare

Promote and protect health of the population served (utilitarianism, beneficence)

Interventions falling within the scope and mandate of the healthcare system have greater value than those not aligned with these

Improving health of patients with severe diseases is aligned with the mandate of healthcare [68]

No adaptation required

 Objective: addressing priorities to increase fairness/justice

  Population priority and access

Principle of fairness (theory of justice)

Interventions targeting established priority populations/disease areas have greater value than those not aligned with these established priorities

Rare diseases represent priorities in several healthcare systems at the regulatory policy and societal levels [12, 56, 80, 82, 95, 96]

Development of the transformation process of this objective into two quantitative criteria—required conceptualization of two criteria and design of appropriate scales. This design can be applied when clearly established priorities (e.g. rare diseases) are available in a given context

 Objective: aligning with the common goal

  Common goal and specific interests

Awareness of stakeholder pressures/barriers helps ensure that decisions are fair-minded and driven by the common goal and not unduly influenced by specific interests (practical wisdom)

Interventions aligned with the common goal have greater value than those aligned with special interests

Highly engaged rare disease patient organizations [47, 48, 54, 62, 65]

No adaptation required

 Objective: ensuring environmental sustainability

  Environmental impact

Principle of utility and beneficence

Interventions that are produced, used, or implemented without causing environmental damage have greater value than those causing environmental damage

Environmental impact of rare disease treatments expected to be low

No adaptation required

Feasibility contextual objectives: qualitative criteria

 Objective: being affordable with low opportunity costs

  Opportunity costs and affordability

Principle of efficiency (practical wisdom, utilitarianism)

Interventions are feasible if they are affordable and associated with a low opportunity cost

Need to consider opportunity cost of benefits forgone in other disease areas [56]

No adaptation required

 Objective: ensuring and preserving system capacity and appropriate use

  System capacity and appropriate use of intervention

Ensure appropriate use of intervention to realize its potential benefit and avoid unintended consequences (practical wisdom, consequentialism)

Interventions are feasible if they can be used appropriately and preserve the healthcare system’s capacity

Lack of local expertise with rare diseases [3]

Difficulty of reaching entire target population [3]

Monitoring and surveillance requirements [58, 60]

No adaptation required

 Objective: having favorable political, historical and cultural context

  Political/historical/cultural context

Awareness of political/historical/cultural aspects ensures that decisions are based on a broad understanding of the context (practical wisdom)

Interventions are feasible if their implementation is supported by the political, historical, and cultural context

Development of therapies for rare diseases may broadly advance innovation beyond a single disease area [56]

Precedence of decisions

No adaptation required

MCDA multicriteria decision analysis, QoL quality of life, RCTs randomized controlled trials, AEs adverse events, PROs patient-reported outcomes, HRQoL health-related quality of life

3.1 Normative Universal Objectives and Quantitative Criteria

These criteria are assessed quantitatively because they are universally quantifiable in the sense that their achievement can be assessed across healthcare systems on scales for which the high and low ends are a priori agreed upon [37].

The objective of addressing areas of high therapeutic need covers three criteria:
  • Disease Severity: Grounded in the ethical imperative to alleviate suffering in those who are worst off (theory of justice), interventions for more severe diseases have greater value than those for less severe diseases [36]. Rare diseases can have multiple impacts on patients, causing morbidity, disability, reduced quality of life (QoL), and shorter life expectancy [1, 2, 69]. Approximately half of these conditions begin in childhood and many cause disabilities [1], thus posing a high burden on caregivers, usually family members [69], which may have a detrimental impact on their QoL [1, 2, 3]. To allow a more differentiated assessment of the multiple domains of disease severity potentially impacted by various conditions, four subcriteria were introduced, including Effect of Disease on Caregivers’ QoL, which is consistent with other assessment frameworks, such as the NICE Interim Process for HST [16].

  • Size of Population: Based on utility theory, this criterion aims at alleviating suffering in as many individuals as possible [36, 37]. Accordingly, a large population of patients affected represents a high therapeutic need, which is in agreement with surveys of the general public revealing that rareness in itself, apart from other considerations such as disease severity, is not considered a healthcare priority [51, 52, 63, 66]. Capturing an important aspect of social value, this criterion needs to be retained in an appraisal framework that is to be applied across different disease areas. Review of epidemiological data on rare diseases [76] suggested an adaptation of the scoring scale to better reflect the wide range of possible disease incidence/prevalence rates, extending from very common (prevalence >5 in 100) to very rare diseases (<5 in 100,000).

  • Unmet Needs, whereby a condition for which current interventions have many and serious limitations regarding their effectiveness, safety, tolerability, and QoL impact represents high need [37]. This criterion is highly relevant for rare diseases, where major therapeutic limitations persist and few interventions targeted for a specific condition are available [3].

The objective of providing large improvements in health outcomes is rooted in deontology (i.e. duty- or rule-based ethics) and outlined in the Hippocratic Oath as well in the principles of beneficence, nonmaleficence, and respect of autonomy (principlism) [21, 36, 41]. This is reflected in three criteria:
  • Comparative effectiveness;

  • Comparative safety/tolerability;

  • Comparative patient-reported outcomes (PROs)

One of the issues in rare diseases pertaining to these criteria is uncertainty or lack of consensus on the outcomes that need to be assessed in clinical studies. This issue can be partially addressed by dividing these criteria into subcriteria to represent outcomes specific to the disease, which will allow appraisers to discuss and differentially weight the importance (relevance) of each outcome.

The objective of delivering important types of health benefit is rooted in the ethical imperative to prevent and eliminate suffering (deontology), and covers two criteria [36]:
  • Type of preventive benefit, whereby preventing a disease entirely (eradication) is the most important type of preventive benefit (risk reduction).

  • Type of therapeutic benefit, whereby alleviating suffering entirely (cure) is the most important type of therapeutic benefit.

Although the vast majority of rare diseases are of genetic origin [2], and interventions are therefore generally therapeutic rather than preventive, this design acknowledges the value of both prophylactic and therapeutic interventions to ensure compatibility of the framework across therapeutic areas.

Both maximization of health benefits and minimization of costs are legitimate objectives of healthcare decision making, rooted in the ethical imperative of making wise use of scarce resources (practical wisdom) to maximize health under resource constraints (utilitarianism) [36, 37]. Thus, an intervention’s economic consequences represent value attributes and are therefore incorporated into a framework intended to measure value from a holistic perspective, as recommended by the UK Government manual conducting multicriteria decision making [46]. The objective of producing favorable economic consequences is addressed with three criteria; for each of these, cost reduction (or release of resources) is seen as a favorable economic consequence:
  • Comparative cost consequences—cost of intervention: Net cost of an intervention to the healthcare system in terms of its acquisition, implementation, maintenance, and replacement of existing treatments or interventions.

  • Comparative cost consequences—other medical costs: All medical cost consequences of an intervention, apart from the cost of the intervention itself (e.g. hospitalization, specialized care, primary care, long-term care, adverse event costs).

  • Comparative cost consequences—nonmedical costs: All cost consequences outside the scope of medical care.

Interventions can affect patients’ medical or nonmedical expenses through co-payments, travel, and paid caregivers, as well as impact patients’ and caregivers’ ability to work, all of which are highly relevant to rare diseases [2, 3, 70], as well as to many other conditions. Therefore, the adapted framework includes subcriteria, defined by payer (i.e. healthcare system, patients), allowing differentiation of diverse economic consequences and adjustments for appraisals from different perspectives.

Cost effectiveness combines several criteria already covered and is therefore not included in the framework, in line with basic design principles of MCDA to avoid double-counting and with other rare disease frameworks [16, 17, 19].

The objective of reducing uncertainty by grounding decisions in solid knowledge is covered under two criteria, both related to practical wisdom [36]:
  • Quality of Evidence: This criterion ensures that higher value is placed on interventions that have a statistically solid and unbiased scientific evidence base [36, 37, 77]. Responding to the imperative of evidence-based decision making, this criterion explicitly rewards high-quality research and development. The rareness of a condition affects data quality in multiple areas, including the burden and natural history of the disease, and the health and economic consequences of a proposed therapy [4, 5, 6, 7, 8]. For example, designing well-powered, double-blind, randomized controlled trials (RCTs) is challenging for rare diseases because patients are few [9] and effective comparators are often absent [10]. Therefore, real-life studies and modeling can play an important role for assessing rare disease therapies [17]. Thus, the adapted framework includes subcriteria representing different fields of research, allowing a differential assessment of data quality.

  • Expert Consensus/Clinical Practice Guidelines (CPGs): This criterion captures the strength of recommendations for optimal clinical practice based on expert interpretation of current scientific knowledge as well as bedside expertise [37]. A strong recommendation in a CPG developed following AGREE principles [78] raises confidence in the value of an intervention. For rare diseases, the availability and quality of CPGs is often limited [55], thus it is particularly important to capture expert knowledge, bearing in mind the scarcity of clinical experts and clinical data [1, 64].

3.2 Transformation of a Generic Qualitative Criterion into a Quantitative Context-Specific Criterion: Population Priority and Access

The objective of Addressing Priorities to Increase Fairness/Justice reflects a society’s sense of justice, which may entail giving priority to certain populations, as defined in specific societies and healthcare systems [36, 37]. Although there are various models of distributive justice (e.g. libertarian, communitarian, egalitarian, and utilitarian) [79], solidarity with those most in need is a key concept [56], as expressed in surveys of the general European population [51, 63]. Patients with rare diseases may be seen as a particularly disadvantaged population as they are often affected by social isolation [1, 57], delayed diagnosis [1, 3, 72], and inequalities in access to adequate treatment and care [1]. The disadvantaged status of these patients is acknowledged in regulatory policies that support research in orphan diseases [11, 12]. In addition, a number of public initiatives were launched to improve delivery of healthcare and social services to patients with rare diseases [80, 81, 82, 83].

Although most decision makers consider population priorities [28], specific priorities may or may not be explicitly established in the context of a given healthcare system. Thus, one can distinguish two types of contexts:
  • If no population priorities are explicitly established, this objective cannot be quantitatively operationalized and must remain in the qualitative part for the framework for qualitative, general (case-by-case) consideration.

  • If specific population priorities are clearly established, the framework is adapted to integrate the objective of addressing priorities into the quantitative part of the framework, bearing in mind that multiple priorities may exist in a given context. To operationalize this integration, each population priority is defined as a criterion:

  • priority 1 (e.g. rare diseases)

  • priority 2 (e.g. HIV)

  • priority 3 (e.g. diabetes)

  • etc.

This design allows relative weighting of each priority versus other, potentially competing priorities that may have been established. Scoring is performed based on how well the intervention to be appraised is aligned with the respective priority. The output of this design can thus quantitatively capture the contribution of priorities (in contexts where such have been established) to the value of an intervention. In addition, this design allows addressing variability in country or institution policies with respect to prioritization of specific disease areas.

3.3 Normative Contextual Objectives and Qualitative Criteria

These criteria cannot be operationalized on a universal basis and are therefore appraised qualitatively in the generic framework [37]. As illustrated in the section above, when adapting the framework to a given context, objectives can be more closely defined and made operationalizable into quantitative criteria, thereby allowing their integration into the quantitative appraisal.

The objective of Aligning with the Mandate and Scope of the Healthcare System rests upon the principle of beneficence and utility, and its inclusion in the framework ensures that this is explicitly considered in each appraisal [36, 37]. The principal mandate of healthcare is to restore and maintain normal functioning [68], and interventions for rare diseases with serious health effects are generally aligned with this mandate.

This objective of Aligning with the Common Goal (i.e. the health of patients, populations, and healthcare systems) rather than with special interests corresponds to the reasonableness condition of the Accountability for Reasonableness framework [23, 24], which demands that decisions be based on relevant and mutually acceptable reasons [36]. Awareness of stakeholder pressures and barriers helps ensure that decisions are fair-minded and not unduly influenced by special interests [37]. For rare diseases, several organizations exist that inform, support, and advocate for patients [47, 48, 54, 62, 65]. While these are important to bring real-life patient perspectives to decision making, a balanced approach needs to be exercised for fair allocation of resources across populations in need.

Although reducing Environmental Impact is a universal normative objective, it currently does not play a significant role in healthcare decision making (although many decision makers think that it should [28]), and is thus appraised qualitatively, but could be incorporated into the quantitative (universal) part of the framework in a given context [37]. Due to their rarity, the environmental impact of rare disease treatments is expected to be low.

3.4 Contextual Feasibility Objectives and Qualitative Criteria

Some interventions may be desirable from a normative point of view, but not feasible in a given context.

Consideration of Affordability and Opportunity Costs, i.e. resources or existing interventions that may be forgone if a new intervention is adopted, is aligned with the principle of efficiency and utilitarianism (maximize health resources) [43], which comes into play at both the patient and societal levels [36, 37]. While the normative aspect of economic considerations (i.e. favorable economic consequences) is included in the quantitative (universal) part of the framework, affordability and opportunity cost considerations require a financial/budgeting exercise to determine feasibility in a given context [37]. For rare diseases, this reflects on whether resources foregone in other disease areas (e.g. more common diseases) are significant in the context of system-wide healthcare economics [56].

The ability of a healthcare system to ensure appropriate use of a new intervention and realize its potential benefit within the system’s capacity, depends on infrastructure, organization, skills, legislation, barriers, and risk of inappropriate use [37]. Relevant for rare diseases is to ensure that specific skills, infrastructure, and surveillance requirements (e.g. patient registries, monitoring) and the ability to reach remote populations are systematically considered [3, 58, 60].

The political, historical and cultural context is important to consider in appraisals. This includes the legal framework, such as, for example, the Equality Act, under which the NICE is required to avoid discrimination based on protected characteristics, and the Human Rights Act, under which it must consider implications for human rights [84]. This objective also covers the impact of the intervention on innovation and research, an important aspect for rare diseases, as research in this area was shown to advance understanding of pathophysiology, broadly contributing to the development of knowledge [56]. Precedence is another important consideration in coverage decision making in the rare disease field, as in other therapeutic areas.

3.5 Hypothetical Example for Application of the Adapted Framework

Table 2 shows an example of the application of the adapted framework in two appraisal contexts: if population priorities have not been established (context A), they are not part of the quantitative model and are considered qualitatively. In contrast, in a context in which population priorities have been explicitly defined (context B), these are incorporated into the quantitative model. In this case, the appraising committee must decide how much relative weight they will trade off from the other domains to ‘population priorities’ and how they will weigh the priority ‘rare diseases’ against other priorities established in their context (Table 2a).
Table 2

Example of the application of the adapted EVIDEM MCDA framework to appraising a hypothetical intervention in two contexts: ‘population priorities’ not explicitly established and thus not part of the quantitative model, and ‘population priorities’ established and thus part of the quantitative model

Domains

Domain weights

Criteria

Criteria weights

Normalized criteria weights

No explicit priorities

Priorities established

No explicit priorities

Priorities established

(a) Relative hierarchical weighting of the domains and criteria of the quantitative MCDA model (independent of the hypothetical intervention), representing the value system and trade-offs of the committee

Normative universal criteria

Need for intervention

30

30

Disease severity

30

0.09

0.09

Size of affected population

20

0.06

0.06

Unmet needs

50

0.15

0.15

Comparative outcomes of intervention

30

28

Comparative effectiveness

50

0.15

0.14

Comparative safety/tolerability

30

0.09

0.08

Comparative patient-perceived health/PROs

20

0.06

0.06

Type of benefit of intervention

10

10

Type of preventive benefit

30

0.03

0.03

Type of therapeutic benefit

70

0.07

0.07

Economic consequences of intervention

20

17

Comparative cost consequences—cost of intervention

50

0.10

0.09

Comparative cost consequences—other medical costs

25

0.05

0.04

Comparative cost consequences—nonmedical costs

25

0.05

0.04

Knowledge on intervention

10

5

Quality of evidence

50

0.05

0.03

Expert consensus/CPGs

50

0.05

0.03

Contextual criteria

Country policy

Population priorities

10a

Rare diseases

Second priority of national health

40a

Assessed qualitatively

0.04

Other priorities

Cancer and mental disorders first and third priority

60a

0.06

Sum

100

100

 

1.00

1.00

Domains

Criteria

Evidence on intervention X for condition Y

Performance scoresb

Value contributionsc

No explicit priorities

Priorities established

(b) Assessment of hypothetical intervention X for condition Y using above weights

Normative Universal Criteria (Quantitative Assessment)

Need for intervention

Disease severity

Y causes severe disability, reduces life expectancy and patients’ and caregivers’ QoL

4

0.07

0.07

Size of affected population

Prevalence of Y: 4/10,000 (rare)

1

0.01

0.01

Unmet needs

Current interventions have low efficacy and affect patients’ QoL

4

0.12

0.12

Comparative outcomes of intervention

Comparative effectiveness

Disability delayed by 2 years; no data on impact on survival

3b

0.09

0.08

Comparative safety/tolerability

Doubles the risk of heart disease; triples the incidence of nausea and skin changes

−3b

−0.05

−0.05

Comparative patient-perceived health/PROs

Inconclusive evidence

0a

0.00

0.00

Type of benefit of intervention

Type of preventive benefit

X does not reduce risk of condition Y

0

0.00

0.00

Type of therapeutic benefit

X delays disability but does not cure Y or prolong life

3

0.04

0.04

Economic consequences of intervention

Comparative cost consequences—cost of intervention

X costs substantially more than current therapy

−4b

−0.08

−0.07

Comparative cost consequences—other medical costs

Model: potential savings due to reduced disability slightly outweigh additional costs for monitoring and AE treatments

1b

0.01

0.01

Comparative cost consequences—nonmedical costs

Model predicts moderately reduced patient and caregiver productivity losses

2b

0.02

0.02

Knowledge on intervention

Quality of evidence

One small RCT; economic model uncertain

2

0.02

0.01

Expert consensus/CPGs

Generally recommended by experts but with caveats

3

0.03

0.02

Contextual criteria

Population priorities

Rare diseases

X fully aligned with rare disease priority

5

Positive impactd

0.04

Other priorities

X not aligned with other priorities

0

0.00

Value estimate (quantitative output)e

0.28

0.30

Contextual Criteria (Qualitative Assessment)

Impact on value estimate ofX

  Contextual normative criteria

Alignment with the mandate and scope of the healthcare system

Severe disease requiring healthcare intervention

Positive impact

Alignment with the common goal rather than special interests

Many conflicting interests

No impact

Environmental sustainability

Intervention X has no environmental consequences

Positive impact

  Contextual feasibility criteria

Affordability and opportunity costs

Low budget impact because of small population

Positive impact

System capacity and appropriate use

Some risk of inappropriate use

Negative impact

Political, historical, and cultural context

Innovative treatment approach

Positive impact

For simplicity, subcriteria are omitted in this example

MCDA multicriteria decision analysis, AE adverse event, CPGs clinical practice guidelines, PROs patient-reported outcomes, QoL quality of life, RCT randomized controlled trial

aThe weights reflect the established priorities in the country

bScale: −5 to 5 for comparative criteria; all other criteria (noncomparative) are scored on a scale of 0–5

cValue contribution = normalized weight (Wx) × standardized score (Sx, assigned score divided by maximum 5)

dAssessed qualitatively in this context

eValue estimate = ∑(Wx × Sx). The maximum (1) corresponds to a hypothetical intervention that prevents and cures severe endemic diseases with significant unmet needs and which, compared with existing approaches, has demonstrated large improvements in efficacy, safety, and PROs, as well as positive economic consequences. (When specific healthcare system priorities are included in the quantitative model, this definition also includes full alignment of the intervention with these priorities.)

Now let us consider assessment of a hypothetical intervention X for a rare condition Y by the same committee (Table 2b). Committee members assign a performance score for each criterion that expresses their judgment of the available evidence, as shown in the example. Scores are standardized and multiplied by the normalized weights (in Table 2a) to calculate the contribution of each criterion to the value of intervention X (Table 2b). The overall value estimate can then be used to rank interventions across disease areas. In this hypothetical example, inclusion of ‘population priorities’ in the quantitative part of the model increases the value estimate of X due to its full alignment with the rare disease priority and the relatively high weight the committee has assigned to this criterion. The committee also qualitatively considers what impact contextual criteria may have on the value of intervention X, which may impact its ranking of intervention X. If priorities have not been included in the quantitative value estimate, their potential impact can be captured qualitatively and may affect ranking.

4 Discussion

In this study, we analyzed the EVIDEM framework from the perspective of issues raised by rare diseases and developed it further in response to these issues. The adapted EVIDEM framework provides an operationalizable platform to integrate individual and social values, competing ethical dilemmas, and uncertainty, which are particularly challenging in appraising interventions for rare diseases. The addition of subcriteria to further differentiate disease severity, disease-specific treatment outcomes and economic consequences of interventions, inclusion of negative scoring scales for all comparative criteria, and integration of a methodology to further incorporate context-specific population priorities and policies makes the EVIDEM framework particularly responsive to rare diseases, while maintaining its applicability across disease areas.

Development of the EVIDEM framework involves continuous research and development on handling of uncertainty, which is particularly pertinent to the context of rare diseases where ethical dilemmas and variations in judgments and perspectives are challenging for the healthcare community [25, 26, 27]. Uncertainty in judgments on evidence can be explored using score ranges [85]; uncertainty in weights can be gauged through application of different weighting techniques (e.g. pair-wise comparison) [30, 46, 75]; uncertainty due to variability in individual perspectives (weights) and judgments (scores) can be assessed through standard measures of statistical dispersion [25, 26, 27, 32, 33]; uncertainty due to model structure can be explored by modifying the structure, e.g. eliminating the lowest weighted criteria [86]; and reproducibility can be examined by repeating the appraisal exercise (test–retest) [33, 87]. In this study, we additionally developed a method to address variability in country or institution policies with respect to prioritization of specific disease areas. This method allows adapting the framework according to whether or not explicit prioritization policies exist and to examine the relative importance of specific priorities.

In a survey of the Norwegian population, the notion that “Patients with rare diseases should have the same right to treatment as others even if more expensive” found strong support [51]. Similarly, in a UK survey, concern for fairness exceeded concern for population health maximization (i.e. utility theory), the guiding principle of the cost-effectiveness-based approach to prioritization [63]. Cost effectiveness is not considered in current [16, 17] or proposed rare disease frameworks [18, 19, 20, 68], and several HTA agencies waive their requirement for cost-effectiveness analysis for orphan drugs [13, 88]. Some propose modifying the current cost-effectiveness paradigm by, for example, assigning quality-adjusted life-year (QALY) weights depending on disease prevalence [9] or different willingness-to-pay thresholds for diseases with high ‘social value’ [89]. However, people do not prioritize rareness but severe, life-threatening diseases, with high unmet needs and interventions that reduce reliance on informal caregivers [63, 66]. These attributes, while characteristic of many rare diseases [1], also apply to other disease areas. Implicit in these proposals is the realization that there is a need for a holistic definition of value, beyond the cost per QALY, to guide appraisal of healthcare interventions.

In comparison to other proposed MCDA frameworks for rare diseases, where selection of decision criteria was based on a literature review, consultation with stakeholders, or rationales [18, 19, 20], selection of decision criteria for the EVIDEM MCDA framework is additionally explicitly rooted in a comprehensive set of ethical principles as well as MCDA design principles, which promotes thorough exploration of the values underpinning appraisal. In addition, due to alignment with the ultimate common goal of all healthcare stakeholders, i.e. identifying interventions that are beneficial to the health of patients as well as to populations and healthcare systems (definition of most valuable interventions), the design proposed here can be applied system-wide across interventions and diseases, in distinction to other MCDA rare disease frameworks. System-wide operationalization is further supported through fully developed implementation methods, particularly the generic design of the scoring scales, which measure evaluators’ judgment of the available evidence across types of interventions and outcomes.

This approach is also in contrast to MCDA models that are designed adhoc to address a specific decision problem, with a discrete set of known options (i.e. interventions). Such ad hoc models can be designed around these options and their attributes, in terms of criteria selection and scoring scales [46, 90, 91, 92]. Ad hoc models, while useful for a circumscribed decision problem, are not adapted to the resource allocation context in which decision makers need to decide, in a consistent manner, whether any proposed intervention provides sufficient value to be funded and adopted into practice. In such contexts, decision makers need a tool to measure value grounded in fundamental principles that reflect the goals of the healthcare system [93]. For example, such an MCDA tool, including decision criteria such as incremental benefits, incremental total cost, and quality of evidence, was proposed for Israel’s Public National Advisory Committee [77]. In addition, EVIDEM has been adapted and implemented in several jurisdictions as a system-wide approach that encompasses all the aspects of decisions for resource allocation [34, 35].

Decision making at its core involves the balancing of ethical dilemmas. Aiming to maximize the overall utility of society (utilitarianism), few resources should be allocated to therapies benefiting few individuals [56]. From a rights-based perspective, access to appropriate healthcare is a right, which is constitutionally established in some jurisdictions, although its scope is open to interpretation [56]. Obviously, a rights-based approach needs to be counterbalanced with considerations of healthcare efficiency and sustainability. The multicriteria framework proposed here makes this balancing act explicit by defining decision criteria, each providing a distinct contribution towards the goal of healthcare. Prioritization of rare diseases is made explicit and weighted against other priorities and the goal of benefiting as many people as possible, while taking into account disease severity, unmet medical needs and economic consequences. This MCDA design incorporates the moral principles of beneficence, nonmaleficence, respect for autonomy, and distributive justice, set forth by Beauchamp and Childress [39, 40] and explicitly subscribed to by the NICE [94], and ensures that these principles are actively considered in decision making. Such approaches facilitate communication to achieve workable resolution of ethical dilemmas across stakeholders, which is necessary to promote the most valuable healthcare interventions to optimize health of patients, populations, and healthcare systems.

5 Conclusion

The adapted framework measures value in its widest sense and is responsive to rare diseases issues and policies. It provides an operationalizable platform to explore values, competing ethical dilemmas, and uncertainty in appraising healthcare interventions.

Notes

Acknowledgments

The authors acknowledge the contribution of the EVIDEM collaboration for the open-source development of the mcda framework on which this study was based.

Compliance with Ethical Standards

This study was made possible by a research grant from Genzyme, Cambridge, MA, USA.

Monika Wagner, Hanane Khoury, Jacob Willet, Donna Rindress, and Mireille Goetghebeur are employees of LASER Analytica, which has received funding from Genzyme (a subsidiary of Sanofi) for the conduct of this study and preparation of the manuscript.

Monika Wagner and Mireille Goetghebeur designed the study and drafted the manuscript. All authors contributed to the collection, analysis and interpretation of data, revised the manuscript, and approved the final version.

Supplementary material

40273_2015_340_MOESM1_ESM.pdf (173 kb)
Supplementary material 1 (PDF 173 kb)

References

  1. 1.
    EURORDIS. Rare diseases: understanding this public health priority. EURORDIS. 2005. Available at: http://www.eurordis.org/publication/rare-diseases-understanding-public-health-priority. Accessed 21 Oct 2013.
  2. 2.
    Barrera LA, Galindo GC. Ethical aspects on rare diseases. Adv Exp Med Biol. 2010;686:493–511.CrossRefPubMedGoogle Scholar
  3. 3.
    Kole A, Faurisson F. Rare diseases social epidemiology: analysis of inequalities. Adv Exp Med Biol. 2010;686:223–50.CrossRefPubMedGoogle Scholar
  4. 4.
    Beutler E. Lysosomal storage diseases: natural history and ethical and economic aspects. Mol Genet Metab. 2006;88(3):208–15.CrossRefPubMedGoogle Scholar
  5. 5.
    Gomez-Rubio V, Lopez-Quilez A. Statistical methods for the geographical analysis of rare diseases. Adv Exp Med Biol. 2010;686:151–71.CrossRefPubMedGoogle Scholar
  6. 6.
    de la Paz MP, Villaverde-Hueso A, Alonso V, Janos S, Zurriaga O, Pollan M, et al. Rare diseases epidemiology research. Adv Exp Med Biol. 2010;686:17–39.CrossRefPubMedGoogle Scholar
  7. 7.
    Groft SC, de la Paz MP. Rare diseases—avoiding misperceptions and establishing realities: the need for reliable epidemiological data. Adv Exp Med Biol. 2010;686:3–14.CrossRefPubMedGoogle Scholar
  8. 8.
    Fuller M, Meikle PJ, Hopwood JJ. Epidemiology of lysosomal storage diseases: an overview. In: Mehta A, Beck M, Sunder-Plassmann G, editors. Fabry disease: perspectives from 5 years of FOS. Oxford: Oxford PharmaGenesis; 2006.Google Scholar
  9. 9.
    Hughes DA, Tunnage B, Yeo ST. Drugs for exceptionally rare diseases: do they deserve special status for funding? QJM. 2005;98(11):829–36.CrossRefPubMedGoogle Scholar
  10. 10.
    Kanters TA, de Sonneville-Koedoot C, Redekop WK, Hakkaart L. Systematic review of available evidence on 11 high-priced inpatient orphan drugs. Orphanet J Rare Dis. 2013;8(1):124.PubMedCentralCrossRefPubMedGoogle Scholar
  11. 11.
    Liu BC, He L, He G, He Y. A cross-national comparative study of orphan drug policies in the United States, the European Union, and Japan: towards a made-in-China orphan drug policy. J Public Health Policy. 2010;31(4):407–20.CrossRefPubMedGoogle Scholar
  12. 12.
    Loorand-Stiver L. Drugs for rare diseases: evolving trends in regulatory and health technology assessment perspectives. Environ Scan. 2013;2013(42):1–40.Google Scholar
  13. 13.
    Roll K, Stargardt T, Schreyogg J. Authorization and reimbursement of orphan drugs in an international comparison [in German]. Gesundheitswesen. 2011;73(8–9):504–14.CrossRefPubMedGoogle Scholar
  14. 14.
    Tambuyzer E. Rare diseases, orphan drugs and their regulation: questions and misconceptions. Nat Rev Drug Discov. 2010;9(12):921–9.CrossRefPubMedGoogle Scholar
  15. 15.
    Herder M. When everyone is an orphan: against adopting a U.S.-styled orphan drug policy in Canada. Acc Res. 2013;20(4):227–69.CrossRefGoogle Scholar
  16. 16.
    National Institute for Health and Care Excellence. Interim process and methods of the Highly Specialised Technologies programme. 2013. Available at: http://www.nice.org.uk/aboutnice/howwework/devhst/DevelopingNICEHighlySpecialisedTechnologies.jsp. Accessed 30 Oct 2013.
  17. 17.
    Winquist E, Bell CM, Clarke JT, Evans G, Martin J, Sabharwal M, et al. An evaluation framework for funding drugs for rare diseases. Value Health. 2012;15(6):982–6.CrossRefPubMedGoogle Scholar
  18. 18.
    Hughes-Wilson W, Palma A, Schuurman A, Simoens S. Paying for the Orphan Drug System: break or bend? Is it time for a new evaluation system for payers in Europe to take account of new rare disease treatments? Orphanet J Rare Dis. 2012;7:74.PubMedCentralCrossRefPubMedGoogle Scholar
  19. 19.
    Paulden M, Stafinski T, Menon D, McCabe C. Value-based reimbursement decisions for orphan drugs: a scoping review and decision framework. Pharmacoeconomics. 2014;33(3):255–69.PubMedCentralCrossRefGoogle Scholar
  20. 20.
    Sussex J, Rollet P, Garau M, Schmitt C, Kent A, Hutchings A. A pilot study of multicriteria decision analysis for valuing orphan medicines. Value Health. 2013;16(8):1163–9.CrossRefPubMedGoogle Scholar
  21. 21.
    Clark S, Weale A. Social values in health priority setting: a conceptual framework. J Health Organ Manag. 2012;26(3):293–316.CrossRefPubMedGoogle Scholar
  22. 22.
    Biron L, Rumbold B, Faden R. Social value judgments in healthcare: a philosophical critique. J Health Organ Manag. 2012;26(3):317–30.CrossRefPubMedGoogle Scholar
  23. 23.
    Daniels N, Sabin J. Limits to health care: fair procedures, democratic deliberation, and the legitimacy problem for insurers. Philos Public Aff. 1997;26(4):303–50.CrossRefPubMedGoogle Scholar
  24. 24.
    Daniels N. Decisions about access to health care and accountability for reasonableness. J Urban Health. 1999;76(2):176–91.PubMedCentralCrossRefPubMedGoogle Scholar
  25. 25.
    Goetghebeur M, Wagner M, Khoury H, Levitt RJ, Erickson LJ, Rindress D. Evidence and value: impact on DEcisionMaking: the EVIDEM framework and potential applications. BMC Health Serv Res. 2008;8(1):270.PubMedCentralCrossRefPubMedGoogle Scholar
  26. 26.
    Goetghebeur M, Wagner M, Khoury H, Rindress D, Gregoire JP, Deal C. Combining multicriteria decision analysis, ethics and health technology assessment: applying the EVIDEM decision making framework to growth hormone for Turner syndrome patients. Cost Eff Resour Alloc. 2010;8(1):4.PubMedCentralCrossRefPubMedGoogle Scholar
  27. 27.
    Goetghebeur MM, Wagner M, Khoury H, Levitt RJ, Erickson LJ, Rindress D. Bridging health technology assessment (HTA) and efficient health care decision making with multicriteria decision analysis (MCDA): applying the EVIDEM framework to medicines appraisal. Med Decis Mak. 2012;32(2):376–88.CrossRefGoogle Scholar
  28. 28.
    Tanios N, Wagner M, Tony M, Baltussen R, van Til J, Rindress D, et al. Which criteria are considered in healthcare decisions? Insights from an international survey of policy and clinical decision makers. Int J Technol Assess Health Care. 2013;29(4):456–65.CrossRefPubMedGoogle Scholar
  29. 29.
    Guindo LA, Wagner M, Baltussen R, Rindress D, van Til J, Kind P, et al. From efficacy to equity: review of decision criteria used in resource allocation and healthcare decisionmaking. Cost Eff Resour Alloc. 2012;10(1):9.PubMedCentralCrossRefPubMedGoogle Scholar
  30. 30.
    van Til J, Groothuis-Oudshoorn C, Lieferink M, Dolan J, Goetghebeur M. Does technique matter: a pilot study exploring weighting techniques for a multi-criteria decision support framework. Cost Eff Resour Alloc. 2014;12:22.PubMedCentralCrossRefPubMedGoogle Scholar
  31. 31.
    EVIDEM Collaboration website . EVIDEM Collaboration. 2015. Available at: http://www.evidem.org. Accessed 12 June 2015.
  32. 32.
    Miot J, Wagner M, Khoury H, Anderson AN, Rindress D, Goetghebeur MM. Field testing of a framework for coverage of a diagnostic test for cervical cancer in South Africa. Cost Eff Resour Alloc. 2011;10(1):2.CrossRefGoogle Scholar
  33. 33.
    Tony M, Wagner M, Khoury H, Rindress D, Papastavros T, Oh P, et al. Bridging health technology assessment (HTA) with multicriteria decision analyses (MCDA): field testing of the EVIDEM framework for coverage decisions by a public payer in Canada. BMC Health Serv Res. 2011;11:329.PubMedCentralCrossRefPubMedGoogle Scholar
  34. 34.
    Radaelli G, Lettieri E, Masella C, Merlino L, Strada A, Tringali M. Implementation of EUnetHTA Core Model(R) in Lombardia: the VTS framework. Int J Technol Assess Health Care. 2014;30(1):105–12.CrossRefPubMedGoogle Scholar
  35. 35.
    MCDA for HTA in action: leveraging experience from Europe and Latin America. Panel presented at the Health Technology Assessment International (HTAi) 12th Annual Conference, 15–17 June 2015: Oslo; 2015.Google Scholar
  36. 36.
    Goetghebeur M, Wagner M. Defining value: the ethical foundations of MCDA in healthcare decision making. Manuscript in preparation. In: Marsh K, Thokala P, Goetghebeur M, Baltussen R, editors. Healthcare decisions supported by multi-criteria decision analysis. Cham: Springer International Publishing AG; 2015.Google Scholar
  37. 37.
    EVIDEM Collaboration. Decision criteria of the framework: conceptual background, definitions, design & instructions. EVIDEM Collaboration. 2015. Available at: https://www.evidem.org/components-decision.php. Accessed 29 July 2015.
  38. 38.
    Alexander L, Moore M. Deontological ethics. In: Zalta EN, editor. The Stanford Encyclopedia of Philosophy (Spring 2015 edition). Available at: http://plato.stanford.edu/archives/spr2015/entries/ethics-deontological/.
  39. 39.
    Beauchamp TL, Childress JF. Principles of biomedical ethics. Oxford; New York: Oxford University Press; 2001.Google Scholar
  40. 40.
    Beauchamp TL. The ‘four principles’ approach. Principles of health care ethics. Chichester: Wiley; 1994. p. 3–12.Google Scholar
  41. 41.
    Lawrence DJ. The four principles of biomedical ethics: a foundation for current bioethical debate. J Chiropr Humanit. 2007;14:34–40.CrossRefGoogle Scholar
  42. 42.
    Driver J. The history of utilitarianism. In: Zalta EN, editor. The Stanford Encyclopedia of Philosophy (Winter 2014 edition). Available at: http://plato.stanford.edu/archives/win2014/entries/utilitarianism-history/.
  43. 43.
    World Health Organization. Guidance on ethics and equitable access to HIV treatment and care. World Health Organization. 2004. Available at: http://www.who.int/ethics/Guidance%20on%20Ethics%20and%20HIV.pdf. Accessed 6 Feb 2009.
  44. 44.
    Hoedemaekers R, Dekkers W. Justice and solidarity in priority setting in health care. Health Care Anal. 2003;11(4):325–43.CrossRefPubMedGoogle Scholar
  45. 45.
    Hursthouse R. Virtue ethics. In: Zalta EN, editor. The Stanford Encyclopedia of Philosophy (Fall 2013 edition). Available at: http://plato.stanford.edu/archives/fall2013/entries/ethics-virtue.
  46. 46.
    Department for Communities and Local Government. Multi-criteria analysis: a manual. London: Department for Communities and Local Government; 2009.Google Scholar
  47. 47.
    Black AP, Baker M. The impact of parent advocacy groups, the Internet, and social networking on rare diseases: the IDEA League and IDEA League United Kingdom example. Epilepsia. 2011;52(Suppl 2):102–4.CrossRefPubMedGoogle Scholar
  48. 48.
    Boon W, Broekgaarden R. The role of patient advocacy organisations in neuromuscular disease R&D: the case of the Dutch neuromuscular disease association VSN. Neuromuscul Disord. 2010;20(2):148–51.CrossRefPubMedGoogle Scholar
  49. 49.
    Cornu C, Kassai B, Fisch R, Chiron C, Alberti C, Guerrini R, et al. Experimental designs for small randomised clinical trials: an algorithm for choice. Orphanet J Rare Dis. 2013;8:48.PubMedCentralCrossRefPubMedGoogle Scholar
  50. 50.
    Denis A, Mergaert L, Fostier C, Cleemput I, Simoens S. Budget impact analysis of orphan drugs in Belgium: estimates from 2008 to 2013. J Med Econ. 2010;13(2):295–301.CrossRefPubMedGoogle Scholar
  51. 51.
    Desser AS, Gyrd-Hansen D, Olsen JA, Grepperud S, Kristiansen IS. Societal views on orphan drugs: cross sectional survey of Norwegians aged 40 to 67. BMJ. 2010;341:c4715.PubMedCentralCrossRefPubMedGoogle Scholar
  52. 52.
    Desser AS. Prioritizing treatment of rare diseases: a survey of preferences of Norwegian doctors. Soc Sci Med. 2013;94:56–62.CrossRefPubMedGoogle Scholar
  53. 53.
    Dimichele DM, Blanchette V, Berntorp E. Clinical trial design in haemophilia. Haemophilia. 2012;18(Suppl. 4):18–23.CrossRefPubMedGoogle Scholar
  54. 54.
    Dunkle M, Pines W, Saltonstall PL. Advocacy groups and their role in rare diseases research. Adv Exp Med Biol. 2010;686:515–25.CrossRefPubMedGoogle Scholar
  55. 55.
    Eidt D, Frank M, Reimann A, Wagner TOF, Mittendorf T. Maánahmen zur Verbesserung der gesundheitlichen Situation von Menschen mit Seltenen Erkrankungen in Deutschland. Bundesgesundheitsblatt Gesundheitsforschung Gesundheitsschutz. 2009. Available at: http://www.bmg.bund.de/uploads/publications/BMG-G-09050-Bericht-Massnahmen-seltene-Krankheiten_200908.pdf. Accessed 28 Nov 2013.
  56. 56.
    Gericke CA, Riesberg A, Busse R. Ethical issues in funding orphan drug research and development. J Med Ethics. 2005;31(3):164–8.PubMedCentralCrossRefPubMedGoogle Scholar
  57. 57.
    Henderson SL, Packman W, Packman S. Psychosocial aspects of patients with Niemann-Pick disease, type B. Am J Med Genet A. 2009;149A(11):2430–6.CrossRefPubMedGoogle Scholar
  58. 58.
    Hollak CE, Aerts JM, Ayme S, Manuel J. Limitations of drug registries to evaluate orphan medicinal products for the treatment of lysosomal storage disorders. Orphanet J Rare Dis. 2011;6:16.PubMedCentralCrossRefPubMedGoogle Scholar
  59. 59.
    Joppi R, Bertele’ V, Garattini S. Orphan drugs, orphan diseases. The first decade of orphan drug legislation in the EU. Eur J Clin Pharmacol. 2013;69(4):1009–24.CrossRefPubMedGoogle Scholar
  60. 60.
    Kenny TD, Jessop EG, Gutteridge WH. Monitoring clinical quality in rare disease services: experience in England. Orphanet J Rare Dis. 2008;3:23.PubMedCentralCrossRefPubMedGoogle Scholar
  61. 61.
    Kesselheim AS, Myers JA, Avorn J. Characteristics of clinical trials to support approval of orphan vs nonorphan drugs for cancer. JAMA. 2011;305(22):2320–6.CrossRefPubMedGoogle Scholar
  62. 62.
    Lavery C. Role of patient support groups in lysosomal storage diseases. In: Mehta A, Beck M, Sunder-Plassmann G, editors. Fabry disease: perspectives from 5 years of FOS. Oxford: Oxford PharmaGenesis; 2006.Google Scholar
  63. 63.
    Linley WG, Hughes DA. Societal views on NICE, cancer drugs fund and value-based pricing criteria for prioritising medicines: a cross-sectional survey of 4118 adults in Great Britain. Health Econ. 2013;22(8):948–64.CrossRefPubMedGoogle Scholar
  64. 64.
    MacLeod S. Optimal therapy for rare disorders and genetic diseases: ethical and political challenges. Proc West Pharmacol Soc. 2007;50:21–3.PubMedGoogle Scholar
  65. 65.
    Mavris M, Le CY. Involvement of patient organisations in research and development of orphan drugs for rare diseases in europe. Mol Syndromol. 2012;3(5):237–43.PubMedCentralPubMedGoogle Scholar
  66. 66.
    Mentzakis E, Stefanowska P, Hurley J. A discrete choice experiment investigating preferences for funding drugs used to treat orphan diseases: an exploratory study. Health Econ Policy Law. 2011;6(3):405–33.CrossRefPubMedGoogle Scholar
  67. 67.
    Orofino J, Soto J, Casado MA, Oyaguez I. Global spending on orphan drugs in France, Germany, the UK, Italy and Spain during 2007. Appl Health Econ Health Policy. 2010;8(5):301–15.CrossRefPubMedGoogle Scholar
  68. 68.
    Pinxten W, Denier Y, Dooms M, Cassiman JJ, Dierickx K. A fair share for the orphans: ethical guidelines for a fair distribution of resources within the bounds of the 10-year-old European Orphan Drug Regulation. J Med Ethics. 2012;38(3):148–53.CrossRefPubMedGoogle Scholar
  69. 69.
    Rajmil L, Perestelo-Perez L, Herdman M. Quality of life and rare diseases. Adv Exp Med Biol. 2010;686:251–72.CrossRefPubMedGoogle Scholar
  70. 70.
    Rosselli D, Rueda JD, Solano M. Ethical and economic considerations of rare diseases in ethnic minorities: the case of mucopolysaccharidosis VI in Colombia. J Med Ethics. 2012;38(11):699–700.PubMedCentralCrossRefPubMedGoogle Scholar
  71. 71.
    Schey C, Milanova T, Hutchings A. Estimating the budget impact of orphan medicines in Europe: 2. Orphanet J Rare Dis. 2011;6:62.PubMedCentralCrossRefPubMedGoogle Scholar
  72. 72.
    van der Kloot WA, Hamdy NA, Hafkemeijer LC, den Dulk FM, Chotkan SA, van Emmerik AA, et al. The psychological burden of an initially unexplained illness: patients with sternocostoclavicular hyperostosis before and after delayed diagnosis. Health Qual Life Outcomes. 2010;8:97.PubMedCentralCrossRefPubMedGoogle Scholar
  73. 73.
    Wyatt K, Henley W, Anderson L, Anderson R, Nikolaou V, Stein K, et al. The effectiveness and cost-effectiveness of enzyme and substrate replacement therapies: a longitudinal cohort study of people with lysosomal storage disorders. Health Technol Assess. 2012;16(39):1–543.CrossRefPubMedGoogle Scholar
  74. 74.
    Drummond M, Towse A. Orphan drugs policies: a suitable case for treatment. Eur J Health Econ. 2014;15(4):335–40.CrossRefPubMedGoogle Scholar
  75. 75.
    Dolan JG. Multi-criteria clinical decision support. A primer on the use of multiple-criteria decision-making methods to promote evidence-based, patient-centered healthcare. Patient. 2010;3(4):229–248.Google Scholar
  76. 76.
    Orphanet Report Series. Prevalence of rare diseases: bibliographic data. 2014. Available at: http://www.orpha.net/consor/cgi-bin/Education_Home.php?lng=EN#REPORT_RARE_DISEASES.
  77. 77.
    Golan OG, Hansen P. Which health technologies should be funded? A prioritization framework based explicitly on value for money. Isr J Health Policy Res. 2012;1(1):44.PubMedCentralCrossRefPubMedGoogle Scholar
  78. 78.
    Brouwers MC, Kho ME, Browman GP, Burgers JS, Cluzeau F, Feder G, et al. AGREE II: advancing guideline development, reporting and evaluation in health care. CMAJ. 2010;182(18):E839–42.PubMedCentralCrossRefPubMedGoogle Scholar
  79. 79.
    Leget C, Hoedemaekers R. Teaching medical students about fair distribution of healthcare resources. J Med Ethics. 2007;33(12):737–41.PubMedCentralCrossRefPubMedGoogle Scholar
  80. 80.
    EUROPLAN Project. EUROPLAN 2012-2015. EUROPLAN Project. 2015. http://www.europlanproject.eu/Content?folder=1&content=1. Accessed 23 Oct 2015.
  81. 81.
    Taruscio D, Vittozzi L, Stefanov R. National plans and strategies on rare diseases in Europe. Adv Exp Med Biol. 2010;686:475–91.CrossRefPubMedGoogle Scholar
  82. 82.
    EUCERD. EUCERD Joint Action: working for rare diseases. EUCERD. 2014. http://www.eucerd.eu/?page_id=54. Accessed 23 Oct 2015.
  83. 83.
    Molster C, Youngs L, Hammond E, Dawkins H. Key outcomes from stakeholder workshops at a symposium to inform the development of an Australian national plan for rare diseases. Orphanet J Rare Dis. 2012;7:50.PubMedCentralCrossRefPubMedGoogle Scholar
  84. 84.
    National Institute for Health and Care Excellence. NICE’s equality objectives and equality programme 2013–2016. 2013. Available at: http://www.nice.org.uk/about/nice-communities/public-involvement/equality-scheme#. Accessed 28 Jan 2015.
  85. 85.
    Goetghebeur M, Wagner M, Bec M, Nikodem M, Zyla A, Micaleff A et al. IMI work package 6: report. Benefit-risk balance re-evaluation case study Raptiva® (efalizumab). European Medicines Agency. 2015. v.2.6:1–99.http://www.imi-protect.eu/. Accessed 23 Oct 2015.
  86. 86.
    Cox R, Sanchez J, Revie CW. Multi-criteria decision analysis tools for prioritising emerging or re-emerging infectious diseases associated with climate change in Canada. PLoS One. 2013;8(8):e68338.PubMedCentralCrossRefPubMedGoogle Scholar
  87. 87.
    Shrout PE, Fleiss JL. Intraclass correlations: uses in assessing rater reliability. Psychol Bull. 1979;86(2):420–8.CrossRefPubMedGoogle Scholar
  88. 88.
    Denis A, Mergaert L, Fostier C, Cleemput I, Hulstaert F, Simoens S. Critical assessment of Belgian reimbursement dossiers of orphan drugs. Pharmacoeconomics. 2011;29(10):883–93.CrossRefPubMedGoogle Scholar
  89. 89.
    Drummond M, Evans B, LeLorier J, Karakiewicz P, Martin D, Tugwell P, et al. Evidence and values: requirements for public reimbursement of drugs for rare diseases: a case study in oncology. Can J Clin Pharmacol. 2009;16(2):e273–81.PubMedGoogle Scholar
  90. 90.
    Bettinger TL, Shuler G, Jones DR, Wilson JP. Schizophrenia: multi-attribute utility theory approach to selection of atypical antipsychotics. Ann Pharmacother. 2007;41(2):201–7.CrossRefPubMedGoogle Scholar
  91. 91.
    Singh S, Dolan JG, Centor RM. Optimal management of adults with pharyngitis: a multi-criteria decision analysis. BMC Med Inform Decis Mak. 2006;6:14.PubMedCentralCrossRefPubMedGoogle Scholar
  92. 92.
    Aenishaenslin C, Hongoh V, Cisse HD, Hoen AG, Samoura K, Michel P, et al. Multi-criteria decision analysis as an innovative approach to managing zoonoses: results from a study on Lyme disease in Canada. BMC Public Health. 2013;13(1):897.PubMedCentralCrossRefPubMedGoogle Scholar
  93. 93.
    Tromp N, Baltussen R. Mapping of multiple criteria for priority setting of health interventions: an aid for decision makers. BMC Health Serv Res. 2012;12:454.PubMedCentralCrossRefPubMedGoogle Scholar
  94. 94.
    National Institute for Health and Care Excellence. Social value judgements. Principles for the development of NICE guidance. 2nd ed. 2013. Available at: http://www.nice.org.uk/proxy/?sourceurl=http://www.nice.org.uk/aboutnice/howwework/socialvaluejudgements/socialvaluejudgements.jsp. Accessed 28 Jan 2015.
  95. 95.
    Australian Government. Therapeutic goods regulations 1990: statutory rules No. 394, 1990 as amended. Australian Government. 2013. https://www.comlaw.gov.au/Details/F2013C00670. Accessed 23 Oct 2015.
  96. 96.
    Office of Legislative and Regulatory Modernization. Initial draft discussion document for a Canadian orphan drug regulatory framework. Orphanet. 2012. 1-25. http://www.orpha.net/national/data/CA-EN/www/uploads/Initial-Draft-Discussion-Document-for-A-Canadian-Orphan-Drug--Regulatory-Framework.doc. Accessed 30 Oct 2013.

Copyright information

© The Author(s) 2015

Open AccessThis article is distributed under the terms of the Creative Commons Attribution-NonCommercial 4.0 International License (http://creativecommons.org/licenses/by-nc/4.0/), which permits any noncommercial use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made.

Authors and Affiliations

  • Monika Wagner
    • 1
  • Hanane Khoury
    • 1
  • Jacob Willet
    • 2
  • Donna Rindress
    • 1
  • Mireille Goetghebeur
    • 1
    • 3
  1. 1.LASER AnalyticaMontrealCanada
  2. 2.LASER AnalyticaNew YorkUSA
  3. 3.University of Montreal, School of Public HealthMontrealCanada

Personalised recommendations