Skip to main content

Criteria and Scoring Functions Used in Multi-criteria Decision Analysis and Value Frameworks for the Assessment of Rare Disease Therapies: A Systematic Literature Review



Traditionally, the economic value of health technologies is assessed with cost-effectiveness (CE) and budget impact (BI) analyses. However, the evaluation of rare disease therapies often considers novel value criteria. Multi-criteria decision analysis (MCDA) is a promising tool in the assessment of value criteria that typically cannot be captured with traditional approaches.


The objective of this research was to investigate the criteria and scoring functions applied in value frameworks and MCDA tools relevant to the evaluation of rare disease therapies. The aim was to gain a better understanding of the domains and measurement of commonly referenced novel value criteria.


A systematic literature review was performed covering the period from 2013 to 2019. MCDA or value framework articles and structured review papers on orphan-drug-specific MCDA articles were reviewed. Information sources included MEDLINE, Embase, Scopus, and 26 other gray literature sources. A descriptive review of identified criteria and scoring functions was performed, with special focus on “novel” value criteria that are traditionally not considered in CE or BI analyses.


In total, 15 relevant value frameworks and MCDA tools were identified. These studies included a large number (n = 56) of individual value criteria. The most commonly included novel criteria were unmet medical need, severity of disease, and reduction in uncertainty. The identified scoring functions (measurement methods) for novel criteria were highly heterogeneous and tailored. Standardized scoring functions were not observed. Additionally, the studies did not provide their rationale for choosing a specific scoring function for a criterion.


MCDA is a promising tool to include novel value criteria into the health technology assessment of therapies for rare diseases. To support the development of a transparent and justified evaluation process, scoring functions should be further investigated.


Comprehensive measurement of health economic value is a considerable challenge. However, appropriate economic value estimation is crucial for scarce resources to be allocated efficiently, and it is necessary to judge whether the healthcare technology at a certain price is good value for a specified society. The value of health technologies can be determined through a multidisciplinary process called health technology assessment (HTA) [1]. This is a formal, systematic, and transparent approach that traditionally assesses the economic implications of a new intervention by two structured components: cost-effectiveness analysis and budget impact analysis. Cost-effectiveness analysis is an approach to estimate the long-term value compared with additional cost, and budget impact analysis is an approach to assess the short-term affordability of new therapies [2,3,4]. In addition to these components, other factors, such as unmet medical need or equity, are taken into account in several jurisdictions during the deliberative process, but no generally used quantitative method exists to measure and incorporate these additional factors [5].

With scientific advances in innovative therapies for targeted populations, including those with rare diseases, the traditional approaches to estimating economic value may be considered less relevant (i.e., not sufficient) [6]. An important reason, besides many others, is the unique regulatory pathway, which results in scarcity of good clinical and health economic evidence [7]. On the other hand, the prices of treatments for rare diseases can be exceptionally high [8]. These factors in combination mean the traditional quantitative HTA methods are less applicable [9, 10]. As a result, in the case of several therapies for rare diseases, decisions and corresponding behaviors suggested discordance with traditional HTA results [11]. Therefore, the methods should be updated, and additional value-bearing criteria can be taken into account during the value assessment. Different value frameworks have been developed by recognized HTA organizations to define criteria that should be considered in a more comprehensive evaluation process [4, 12, 13]. Although these endeavors were successful in highlighting certain “novel” value criteria (e.g., scientific spillover, equity, real option value, etc.), challenges remain in terms of how these fit into the decision-making process in a measurable, structured, and transparent way. Despite these challenges, we can find examples of how to incorporate certain novel value criteria into the HTA process [14].

Using value frameworks, multi-criteria decision analysis (MCDA) tools can be developed [15]. MCDA is a structured decision-making process that offers the flexibility of incorporating multiple objectives and criteria into a single evaluation. Research has explored the application of MCDA in coverage and reimbursement decisions of innovative healthcare technologies [16]. Despite early examples [17] of MCDA applications, the uptake of MCDA by health technology decision makers remained at the level of pilot tests [18]. In most published cases, MCDA was developed as a tool that substituted the existing practices entirely with incorporating traditional criteria such as clinical effectiveness, safety, quality-adjusted life-year (QALY) gains and, in some cases, economic aspects (i.e., cost effectiveness and budget impact) [19]. By following such a holistic approach, ambiguity may be introduced in the assessment, as individuals are asked to trade-off traditional and novel value criteria with overlap across value domains. On the other hand, MCDA remains a promising tool in the assessment of novel value criteria that are difficult to measure and typically do not flow into traditional cost-effectiveness and budget impact approaches.

One of the most researched areas of MCDA application is in the coverage and reimbursement of innovative therapies for rare diseases, especially the assessment of orphan drugs [20,21,22]. Therefore, we selected this therapeutic field for further research.


The objective of this research was to perform a systematic literature review (SLR) of the criteria and scoring functions applied in value frameworks and MCDA tools relevant to the evaluation of therapies for rare diseases. The research aimed to reveal the criteria that directly mapped into the traditional value criteria, and special attention was given to those not commonly used in cost-effectiveness and budget impact analyses. The aim was to gain a better understanding of the measurement of novel value criteria with the help of the scales and scoring functions applied to them in the published studies.

To our knowledge, this is the first SLR to systematically collect the criteria and applied scoring functions to support the development of a tool that focuses on value criteria that are not incorporated into traditional HTA.


We performed a systematic review of scientific and gray literature of value frameworks and MCDA tools relevant for pricing and reimbursement decision making for pharmaceuticals. Value frameworks were defined as comprehensive frameworks used to assess the health economic value of health technologies, mainly focusing on value criteria and their independent evaluation without any aggregation methods. MCDA tools overcame these limitations by explicit scoring and mathematical aggregation of different value criteria. Structured review papers on rare-disease-specific MCDA articles were also investigated. Information sources included PubMed, Embase, Scopus, and 26 other gray literature sources covering the period from January 2013 until October 2019. Gray literature sources included databases of universities, HTA agencies, and other relevant HTA organizations (see electronic supplementary material [ESM] 1 for the complete list of sources). Our search was limited to papers that were written in the English language. The publication date was restricted to 2013 or later to summarize the most recent and relevant evidence published in the literature. The search strategy was built up as a combination of search strings, allowing the capture of all relevant keywords and synonyms that may appear in the papers (see ESM 1 for the detailed search strategies).

Initial title and abstract screening of relevant papers was conducted by NDM and BE, with disputes in sorting arbitrated by TZ. Screening was conducted alongside hierarchical exclusion criteria. Publications not excluded by one of these criteria proceeded to full-text screening. Full-text screening of relevant articles was conducted by NDM and BE separately, with disputes arbitrated by TZ as a third senior investigator (see ESM 1 for the detailed literature selection methods). Abstracts identified as posters or podium presentations were screened separately. Studies were included in the final selection if they used explicit scoring functions for the included evaluation criteria and were either orphan drug specific or were considered “referenced” general frameworks. Value frameworks were considered to be “referenced,” if they were cited more than once in the relevant orphan-specific papers and/or previous literature reviews focusing on orphan-drug-specific MCDA methods. Data extraction from the relevant value frameworks included both general article descriptions (authors, affiliation, year, and country of publication) and evaluation criteria descriptions (name of criterion, definition, scoring function details, weighting).

Criteria were grouped according to whether they mapped directly into the value criteria assessed traditionally from the payer perspective. Three highly referenced value frameworks were selected (i.e., the International Society for Pharmacoeconomics and Outcomes Research [ISPOR] value “flower” [12], the ICER value framework [23], and the Second Panel on Cost-Effectiveness impact inventory [13]) to categorize criteria into three groups according to their use in traditional HTA. The terminology of the ISPOR value flower was used: (1) core criteria (i.e., criteria using traditional HTA), (2) common but inconsistently included criteria (i.e., criteria assessed in certain settings/jurisdictions), and (3) novel criteria (value criteria typically not assessed in traditional HTA) [12].


The systematic search yielded a total of 2913 independent records after deduplicating hits from the different scientific databases. The title and abstract screening further narrowed the number of relevant publications to 434. After full-text screening, 62 peer-reviewed articles including different value criteria remained. With our final set of inclusion criteria requiring explicit scoring functions, as well as the value framework needing to be either orphan drug specific or referenced, 11 relevant studies were included in the final synthesis. In addition to the peer-reviewed publications, 74 conference abstracts were also found, resulting in 12 relevant posters being reviewed and one being included in the final synthesis (Fig. 1). The three highly referenced value frameworks were also included in narrative synthesis as guiding value frameworks. These guiding value frameworks included the ISPOR value flower [12], the ICER value framework [23], and the Second Panel on Cost-Effectiveness Impact Inventory [13]. Altogether, 15 studies were analyzed in this review (11 peer-reviewed publications [24,25,26,27,28,29,30,31,32,33,34], one poster [35], and three guiding frameworks [12, 13, 23]).

Fig. 1
figure 1

Flow of information diagram

Within the 15 included publications, eight publications were orphan drug specific [28,29,30,31,32,33,34,35] and seven were general frameworks [12, 13, 23,24,25,26,27]. Further differentiating the 15 publications, eight were considered value frameworks [12, 13, 23, 25, 27, 28, 32, 33], and the other seven were considered developed MCDA tools [24, 26, 29,30,31, 34, 35]. The criteria included in the 15 relevant studies were extracted and duplications eliminated. There were also significant overlaps between several criteria that was revealed by the thorough evaluation of the definitions and scoring functions of each criterion. Finally, 56 individual value criteria were identified. These were categorized into “core,” “common,” and “novel” value criteria (see ESM 2).

The most common criteria considered to be part of the “core” value criteria and for which a scoring function was published were comparative clinical effectiveness, comparative safety, and health-related quality of life. Some of these criteria substantially overlapped each other. As an example, the four most commonly used individual criteria all contribute to the incremental cost-effectiveness ratio. Table 1 presents the most frequently included core value criteria in orphan-specific, general, and all frameworks (either direct or indirect inclusion).

Table 1 Summary of the most frequent “core” value criteria (type and frequency of inclusion)

Among criteria considered “common,” the most frequently included were adherence-improving factors, labor market earnings lost, and cost of unpaid lost productivity due to illness (Table 2). Most of these criteria can be quantified and therefore taken into account within traditional cost-effectiveness analyses. However, their structured assessment is limited, especially in terms of the incremental therapy impact. In practice, these value criteria are typically folded into cost-effectiveness evaluations when the analysis uses a societal perspective, but their inclusion is inconsistent. Orphan-drug-specific articles less frequently incorporated these common value criteria into the evaluation process.

Table 2 Summary of the most frequent “common” value criteria (type and frequency of inclusion)

For value criteria categorized as “novel,” the most frequent were unmet medical need, severity of disease, and reduction in uncertainty (Table 3). Basically, orphan-specific and general frameworks and tools were similar in terms of the frequency of inclusion of the different criteria in the core, common, and novel categories. However, “size of the population” and “type of therapeutic benefit” seemed to be exemptions, as these criteria were more important in orphan-drug-specific frameworks. On the other hand, “level of innovation” and “burden of illness” appeared more frequently in general frameworks.

Table 3 Summary of the most frequent “novel” value criteria (type and frequency of inclusion)

The scoring functions for these novel criteria were investigated to increase our understanding of these value criteria. “Unmet medical need” and “severity of disease” were two novel criteria found in every orphan-drug-specific value framework and MCDA tool, as these criteria capture crucial aspects of the potential added value of rare disease therapies. The scoring functions provided for these criteria were subjective. For unmet need, the concept was to give a higher score to interventions intended to treat diseases with no available therapeutic alternatives, but the methods differed. Some MCDA tools used a quantitative scale from 0 to 5, where no points were given when there were “no unmet needs,” and 5 points were given when there were “many unmet needs” [34]. Other tools used 3-point scales ranging from “no unmet need” or “available alternatives” to “high unmet need” or “no available alternatives” [31]. For “disease severity,” Iskrov et al. [29] proposed a scoring function in which chronic life-threatening disorders gained a higher score than acute diseases and chronic non-life-threatening diseases. However, the article stated that there was debate about choosing the chronic life-threatening disease over the acute disease to gain a higher score in the severity criterion. To score disease severity, Hughes-Wilson et al. [28] defined a three-level scale where diseases with increased mortality or severe invalidism in infancy had the highest score, increased mortality or severe invalidism in adulthood were at the second level, and low scores were dedicated to diseases that influence patient morbidity rather than mortality [28]. Simple scales from not severe to very severe (on a scale from 0 to 5) also appeared in the frameworks [26]. These examples highlighted the heterogeneity of the current measurement of these criteria.

The “size of the population” criterion is especially interesting, because, unlike general value frameworks, which usually provide higher scores for more common diseases, orphan-specific value frameworks give higher scores for diseases with a lower prevalence, so an ultra-rare disease would be provided the highest score from this aspect [29, 31, 34]. However, exact disease incidence thresholds were not provided in any of the frameworks investigated. Methods of evaluating the criterion “level of innovation” also differed and included items such as the number of new indications, stereochemical properties, or being an advanced therapy medicinal product.

For other novel criteria (i.e., reduction in uncertainty, type of therapeutic benefit, burden of illness, equity/support vulnerable groups, manufacturing complexity), the identified scoring functions were highly heterogeneous and specific for the individual framework or study. Unified scales were not used in any of these criteria (see ESM 2 for the detailed scoring functions included in the studies). Moreover, none of the included MCDA tools or value frameworks described the detailed justification behind the selection of any of the scoring functions.


In this SLR, a large number (n = 62) of scientific publications relevant to the evaluation of therapies for rare diseases were identified. However, only a small portion of these studies (12 of 62) published any recommendations on the scoring functions of the included evaluation criteria. Review of these studies and the three selected guiding frameworks identified a large number of individual criteria. The use of these criteria substantially overlapped each other across the investigated frameworks and MCDA tools. These observations are in line with findings of previous studies [22, 36, 37]. In an SLR, Baran-Kooiker et al. [19] also emphasized the importance of simplifying the MCDA methodology and having fewer and well-defined criteria with less overlap [19].

It was a complex exercise to decide whether or not a criterion is commonly considered in traditional cost-effectiveness or budget impact evaluations. Although such a distinction is inherently arbitrary, and countries have different practices to conduct these evaluations, it is still beneficial to categorize the identified value criteria on a ISPOR value flower spectrum from “core” and “common” to “novel” based on their application as recommended in generally accepted guidelines. Novel criteria carry more uncertainties during the evaluation as less evidence is available on their assessment methods. For these novel criteria, a well-designed MCDA tool could have substantial added benefit in establishing a transparent and reproducible methodology for their evaluation.

An important observation was that criteria that related more to the costs side, and were therefore easier to monetize, were considered more frequently within the traditional evaluations (especially when the broader societal perspective was used), whereas value criteria that related more to the outcomes side but could not easily be translated into QALYs were not included. Such outcomes criteria are usually considered “novel criteria.” In the case of these value criteria, the identified scoring functions may support their more transparent use within MCDA frameworks, especially as the assessment of the novel criteria of value remains less standardized and therefore yet to be part of a structured evidence-generation process.

There was substantial variation in the assessment methods and scoring functions of the included value criteria. No unified scoring function was used for any of the criteria presented in the studies. The exact application may influence the selection of the most applicable, quantifiable, and objective scoring function. Most value frameworks and MCDA tools did not describe the rationale behind choosing a specific scoring function for a criterion; however, in some cases (e.g., “size of population”), key differences were rationalized for scoring functions used in orphan-specific value frameworks compared with a general value frameworks. This missing justification is a significant limitation of the included studies, decreases their level of transparency, and underlines the uncertainties around the evaluation of these novel value criteria.

The investigation of scoring functions supports better understanding of novel value criteria and justifies their application in MCDA tools. In addition, weighting of criteria also depends on the applied scales, so it is necessary to develop credible and transparent scoring functions that are made explicit before the weighting exercise [38]. The ultimate goal of this research was to facilitate the development of well-designed MCDA tools by systematically collecting the existing criteria and their scoring functions, with special focus on novel value criteria. Tables 1, 2 and 3 highlight the most frequently used criteria, and ESM 2 provides the pool of existing scoring functions. These results may provide substantial support for the development of MCDA tools in terms of the selection and measurement of different value criteria.

An important limitation of the current review is that, despite the broad literature search strategy, relevant papers might have been missed, e.g., if a tool was developed for the evaluation of a specific compound or therapies for a specific rare disease, and the general search terms did not capture the MCDA tool. Moreover, the time frame investigated was relatively short to maintain the feasibility of the research. A further limitation is that no formal analysis was performed to judge the appropriateness of the scoring functions identified for the different value criteria.


Our results confirmed that using only traditional criteria in HTA found in cost-effectiveness and budget impact analyses could miss components of health economic value. By reviewing the current literature, more evidence is published in terms of the scope of relevant value criteria. We identified several novel value criteria, but the methods of evaluating these criteria were not sophisticated, and rationales for the applied scoring functions were missing. Therefore, questions remain around what novel components of value are most helpful to measure and how.

MCDA is a promising tool in the implementation of novel value criteria into the evaluation process for orphan drugs. To support the development of a transparent and justified evaluation process, the measurement methods and scoring functions for novel value criteria should be further investigated. Future lines of this research will support the development of an MCDA framework for the evaluation of novel value criteria to be used as a supplement to traditional cost-effectiveness evaluations in HTA.


  1. O’Rourke B, Oortwijn W, Schuller T. The new definition of health technology assessment: a milestone in international collaboration. Int J Technol Assess Health Care. 2020;36(3):187–90.

    Article  PubMed  Google Scholar 

  2. Drummond MF, Sculpher MJ, Claxton K, Stoddart GL, Torrance GW. Methods for the economic evaluation of health care programmes. Oxford University Press; 2015.

    Google Scholar 

  3. Garattini L, van de Vooren K. Budget impact analysis in economic evaluation: a proposal for a clearer definition. Eur J Health Econ. 2011;12(6):499–502.

    Article  PubMed  Google Scholar 

  4. ICER. 2020–2023 Value Assessment Framework. Institute for Clinical and Economic Review 2020. Accessed 20 Nov 2020.

  5. NICE. Guide to the methods of technology appraisal 2013. National Institute for Health and Care Excellence. 2013. Accessed 21 Nov 2020.

  6. Simoens S. Health technologies for rare diseases: does conventional HTA still apply? Expert Rev Pharmacoecon Outcomes Res. 2014;14(3):315–7.

    Article  PubMed  Google Scholar 

  7. Sasinowski FJ, Panico EB, Valentine JE. Quantum of effectiveness evidence in FDA’s approval of orphan drugs: update, July 2010 to June 2014. Ther Innov Regul Sci. 2015;49(5):680–97.

    Article  Google Scholar 

  8. Meekings KN, Williams CSM, Arrowsmith JE. Orphan drug development: an economically viable strategy for biopharma R&D. Drug Discov Today. 2012;17(13):660–4.

    Article  PubMed  Google Scholar 

  9. Drummond MF, Wilson DA, Kanavos P, Ubel P, Rovira J. Assessing the economic challenges posed by orphan drugs. Int J Technol Assess Health Care. 2007;23(1):36–42.

    Article  PubMed  Google Scholar 

  10. Zelei T, Molnár MJ, Szegedi M, Kaló Z. Systematic review on the evaluation criteria of orphan medicines in Central and Eastern European countries. Orphanet J Rare Dis. 2016;11(1):72.

    Article  PubMed  PubMed Central  Google Scholar 

  11. Kawalec P, Sagan A, Pilc A. The correlation between HTA recommendations and reimbursement status of orphan drugs in Europe. Orphanet J Rare Dis. 2016;11(1):122.

    Article  PubMed  PubMed Central  Google Scholar 

  12. Lakdawalla DN, Doshi JA, Garrison LP Jr, Phelps CE, Basu A, Danzon PM. Defining elements of value in health care-a health economics approach: an ISPOR Special Task Force Report [3]. Value Health. 2018;21(2):131–9.

    Article  PubMed  Google Scholar 

  13. Sanders GD, Neumann PJ, Basu A, Brock DW, Feeny D, Krahn M, et al. Recommendations for conduct, methodological practices, and reporting of cost-effectiveness analyses: second panel on cost-effectiveness in health and medicine. JAMA. 2016;316(10):1093–103.

    Article  PubMed  Google Scholar 

  14. Lakdawalla DN, Doshi JA, Garrison LP Jr, Phelps CE, Basu A, Danzon PM. A simple model of value: an economic perspective technical appendix to: defining elements of value in health care—a health economics approach: an ISPOR Special Task Force report [3]. Value Health. 2018;21(2):131–9.

    Article  Google Scholar 

  15. Willke RJ, Neumann PJ, Garrison LP Jr, Ramsey SD. Review of recent US value frameworks—a health economics approach: an ISPOR Special Task Force report [6]. Value Health. 2018;21(2):155–60.

    Article  Google Scholar 

  16. Thokala P, Devlin N, Marsh K, Baltussen R, Boysen M, Kalo Z, et al. Multiple criteria decision analysis for health care decision making–an introduction: report 1 of the ISPOR MCDA Emerging Good Practices Task Force. Value Health. 2016;19(1):1–13.

    Article  PubMed  Google Scholar 

  17. Endrei D, Molics B, Agoston I. Multicriteria decision analysis in the reimbursement of new medical technologies: real-world experiences from Hungary. Value Health. 2014;17(4):487–9.

    Article  PubMed  Google Scholar 

  18. Angelis A, Linch M, Montibeller G, Molina-Lopez T, Zawada A, Orzel K, et al. Multiple criteria decision analysis for HTA across four EU Member States: piloting the advance value framework. Soc Sci Med. 2020;246:112595.

    CAS  Article  PubMed  Google Scholar 

  19. Baran-Kooiker A, Czech M, Kooiker C. Multi-criteria decision analysis (MCDA) models in health technology assessment of orphan drugs—a systematic literature review. Next steps in methodology development? Front Public Health. 2018;6:287.

    Article  PubMed  PubMed Central  Google Scholar 

  20. EMA. Orphan designation: Overview. European Medicines Agency. 2020. Accessed 20 Dec 2020.

  21. FDA. Rare Diseases at FDA. US Food and Drug Administration. 2020. Accessed 20 Dec 2020.

  22. Friedmann C, Levy P, Hensel P, Hiligsmann M. Using multi-criteria decision analysis to appraise orphan drugs: a systematic review. Expert Rev Pharmacoecon Outcomes Res. 2018;18(2):135–46.

    Article  PubMed  Google Scholar 

  23. ICER. Overview of the ICER value assessment framework and update for 2017–2019. Institute for Clinical and Economic Review. 2017. Accessed 10 Dec 2020.

  24. Angelis A, Kanavos P. Multiple criteria decision analysis (MCDA) for evaluating new medicines in health technology assessment and beyond: the advance value framework. Soc Sci Med. 2017;188:137–56.

    Article  PubMed  Google Scholar 

  25. EUnetHTA. HTA Core Model Version 3.0. EUnetHTA Joint Action 2. 2016. Accessed 3 Mar 2020.

  26. EVIDEM. EVIDEM 10th Edition. 2017. Accessed 3 Mar 2020.

  27. Henshall C, Schuller T. Health technology assessment, value-based decision making, and innovation. Int J Technol Assess Health Care. 2013;29(4):353–9.

    Article  PubMed  Google Scholar 

  28. Hughes-Wilson W, Palma A, Schuurman A, Simoens S. Paying for the orphan drug system: break or bend? Is it time for a new evaluation system for payers in Europe to take account of new rare disease treatments? Orphanet J Rare Dis. 2012;7:74.

    Article  PubMed  PubMed Central  Google Scholar 

  29. Iskrov G, Miteva-Katrandzhieva T, Stefanov R. Multi-criteria decision analysis for assessment and appraisal of orphan drugs. Front Public Health. 2016;4:214.

    Article  PubMed  PubMed Central  Google Scholar 

  30. Iskrov GG, Raycheva RD, Stefanov RS. Insight into reimbursement decision-making criteria in Bulgaria: implications for orphan drugs. Folia Med (Plovdiv). 2013;55(3–4):80–6.

    Article  Google Scholar 

  31. Kolasa K, Zwolinski KM, Kalo Z, Hermanowski T. Potential impact of the implementation of multiple-criteria decision analysis (MCDA) on the Polish pricing and reimbursement process of orphan drugs. Orphanet J Rare Dis. 2016;11:23.

    Article  PubMed  PubMed Central  Google Scholar 

  32. MoCA-OMP. Transparent Value Framework. Working Group on Mechanism of Coordinated Access to Orphan Medicinal Products. 2014. Accessed 3 Mar 2020.

  33. Sussex J, Rollet P, Garau M, Schmitt C, Kent A, Hutchings A. A pilot study of multicriteria decision analysis for valuing orphan medicines. Value Health. 2013;16(8):1163–9.

    Article  PubMed  Google Scholar 

  34. Wagner M, Khoury H, Willet J, Rindress D, Goetghebeur M. Can the EVIDEM framework tackle issues raised by evaluating treatments for rare diseases: analysis of issues and policies, and context-specific adaptation. Pharmacoeconomics. 2016;34(3):285–301.

    Article  Google Scholar 

  35. Carvalho D, Etto H, Kim H, Nakada C, de Miranda ÉP, Tannus G. PCV107-multi-criteria decision analysis (MCDA)—discussion about the incorporation of Riociguat in the treatment of chronic thromboembolic pulmonary hypertension (CTEPH) under the perspective of Brazilian public healthcare system (SUS). Value Health. 2018;21:S110.

    Article  Google Scholar 

  36. Schey C, Postma M, Krabbe P, Medic G, Connolly M. The application of multi-criteria decision analysis to inform in resource allocation. F1000Research. 2020;9(445):445.

    Article  Google Scholar 

  37. Wahlster P, Goetghebeur M, Kriza C, Niederländer C, Kolominsky-Rabas P. Balancing costs and benefits at different stages of medical innovation: a systematic review of multi-criteria decision analysis (MCDA). BMC Health Serv Res. 2015;15:262.

    Article  PubMed  PubMed Central  Google Scholar 

  38. Németh B, Molnár A, Bozóki S, Wijaya K, Inotai A, Campbell JD, et al. Comparison of weighting methods used in multicriteria decision analysis frameworks in healthcare with focus on low- and middle-income countries. J Comp Eff Res. 2019;8(4):195–204.

    Article  PubMed  Google Scholar 

Download references


The authors thank Professor Zoltán Kaló for his guidance in the development of the research protocol, and we thank the other pValue investigators, R. Brett McQueen and Melanie D. Whittington, for their support and expert insight during the entire research process.

Author information



Corresponding author

Correspondence to Tamás Zelei.

Ethics declarations


This research was supported by a PhRMA Foundation Center of Excellence Grant, for Pharmaceutical Value (pValue), alongside matching financial support from Data Science to Patient Value from the University of Colorado School of Medicine.

Conflict of interest

Tamás Zelei and Bertalan Németh are employees of the Syreon Research Institute. Baher Elezbawy is an employee of Syreon Middle East. Johnathan D. Campbell worked on this manuscript during his time on faculty as an Associate Professor, and Nicholas D. Mendola is a PhD student and graduate research assistant, at the University of Colorado Anschutz Medical Campus. The content of this paper, as well as the views and opinions expressed therein are those of the authors and not the organizations that employ them.

Availability of data and material

The datasets supporting the conclusions of this article are included in the main text and in its supplementary materials, including the detailed literature search strategy and the flow diagram with predefined exclusion criteria for title-abstract and full-text screening of the records. All other datasets used and analyzed in the current study are available from the corresponding author upon request.

Code availability

Not applicable

Authors’ contributions

TZ (guarantor of the research), initiated the research and drafted the protocol of the SLR. TZ, NDM, BE, and JDC finalized the protocols, conducted the SLR, and interpreted the results. TZ, NDM, and BE drafted the manuscript. JDC and BN reviewed and commented on the draft manuscript. TZ finalized the manuscript. All authors read and approved the current version of the manuscript.

Ethics approval

Not applicable.


Not applicable.

Supplementary Information

Below is the link to the electronic supplementary material.

Supplementary file1 (DOCX 31 kb)

Supplementary file2 (XLSX 24 kb)

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License, which permits any non-commercial use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Zelei, T., Mendola, N.D., Elezbawy, B. et al. Criteria and Scoring Functions Used in Multi-criteria Decision Analysis and Value Frameworks for the Assessment of Rare Disease Therapies: A Systematic Literature Review. PharmacoEconomics Open 5, 605–612 (2021).

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: