The objectives of this systematic review were to identify studies using Multi-Criteria Decision Analysis (MCDA) software tools to support health prioritisation processes and describe the technical capabilities of the MCDA software tools identified.
First, a systematic literature review was conducted in the MEDLINE, EMBASE, Web of Science, EconLit and Cochrane databases in July 2019 to identify studies that have used MCDA software for priority setting in health-related problems. Second, the MCDA software tools found in the review were downloaded (full versions, where freely available, and trial versions otherwise) and tested to extract their key technical characteristics.
Nine studies were included, from which seven different software tools, 1000minds®, M-MACBETH, Socio Technical Allocation of Resources (STAR), Strategic Multi-Attribute Ranking Tool (SMART), Visual PROMETHEE, EVIDEM and the Prioritisation Framework, were identified. These software tools differed in terms of the operating systems (including web interface), MCDA technique(s) available for use, visualisation features, and the capability to perform Value for Money (VfM) and sensitivity analyses.
The use of MCDA software in prioritisation processes has a number of advantages such as inclusion of several types of stakeholders and the ability to analyse a greater number of alternatives and criteria and perform real-time sensitivity analyses. Proprietary software (i.e. software with licensing fees) seemed to have more features than freely available software. However, this field is still developing, with only a few studies where MCDA software was used to support health priority setting and opportunity costs not explicitly captured in many software tools.
This is a preview of subscription content, log in to check access.
Buy single article
Instant access to the full article PDF.
Price includes VAT for USA
Availability of Data and Materials
All data analysed or generated during this study are included in this article.
Defechereux T, Paolucci F, Mirelman A, Youngkong S, Botten G, Hagen TP, et al. Health care priority setting in Norway a multicriteria decision analysis. BMC Health Serv Res. 2012;12:39.
Sabik LM, Lie RK. Priority setting in health care: Lessons from the experiences of eight countries. Int J Equity Health. 2008;7:4.
Nunes R, Rego G. Priority setting in health care: a complementary approach. Health Care Anal. 2014;22(3):292–303.
Baltussen R, Jansen MPM, Bijlmakers L, Grutters J, Kluytmans A, Reuzel RP, et al. Value assessment frameworks for HTA agencies: the organization of evidence-informed deliberative processes. Value Health. 2017;20(2):256–60.
Giedion Ú, Villar M, Ávila A. Los sistemas de salud en Latinoamérica y el papel del seguro privado. 2010. https://www.mapfre.com/ccm/content/documentos/fundacion/cs-seguro/libros/los-sistemas-de-salud-en-latinoamerica-y-el-papel-del-seguro-privado.pdf. Accessed 21 Nov 2019.
Hall W, Williams I, Smith N, Gold M, Coast J, Kapiriri L, et al. Past, present and future challenges in health care priority setting. J Health Organ Manag. 2018;32(3):444–62.
Cox R, Sanchez J, Revie CW. Multi-criteria decision analysis tools for prioritising emerging or re-emerging infectious diseases associated with climate change in Canada. PLoS One. 2013;8(8):e68338.
Baltussen R. Priority setting of public spending in developing countries: do not try to do everything for everybody. Health Policy. 2006;78(2–3):149–56.
Mullen PM. Quantifying priorities in healthcare: transparency or illusion? Health Serv Manag Res. 2004;17(1):47–58.
Hongoh V, Michel P, Gosselin P, Samoura K, Ravel A, Campagna C, et al. Multi-stakeholder decision aid for improved prioritization of the public health impact of climate sensitive infectious diseases. Int J Environ Res Public Health. 2016;13(4):419.
Angelis A, Kanavos P. Value-based assessment of new medical technologies: towards a robust methodological framework for the application of multiple criteria decision analysis in the context of health technology assessment. Pharmacoeconomics. 2016;34(5):435–46.
Confederation NHS. Priority setting: an overview. London: Primary Care Trust Network; 2007.
Belton V, Stewart T. Multiple criteria decision analysis: an integrated approach. Dordrecht: Springer Science + Business Media; 2002.
Marsh K, Lanitis T, Neasham D, Orfanos P, Caro J. Assessing the value of healthcare interventions using multi-criteria decision analysis: a review of the literature. Pharmacoeconomics. 2014;32(4):345–65.
Adunlin G, Diaby V, Xiao H. Application of multicriteria decision analysis in health care: a systematic review and bibliometric analysis. Health Expect. 2015;18(6):1894–905.
Ghandour R, Shoaibi A, Khatib R, Abu Rmeileh N, Unal B, Sözmen K, et al. Priority setting for the prevention and control of cardiovascular diseases: multi-criteria decision analysis in four eastern Mediterranean countries. Int J Public Health. 2015;60(Suppl 1 S1):S73–81.
Kosherbayeva L, Hailey D, Kurakbaev K, Tabarov A, Kumar A, Gutzskaya G, et al. A process of prioritizing topics for health technology assessment in kazakhstan. Int J Technol Assess Health Care. 2016;32(3):147–51.
Ishizaka A, Nemery P. Multi-criteria decision analysis: methods and software. Chichester: Wiley; 2013.
Claxton K. Three questions to ask when examining MCDA. Value Outcomes Spotlight. 2015;1:18–20.
Mustajoki J, Marttunen M. Comparison of multi-criteria decision analytical software-searching for ideas for developing a new EIA-specific multi-criteria software. Environ Modell Softw. 2013;93(C):78–91.
Sengupta R, Gupta A, Dutta J. Decision sciences: theory and practice. 1st ed. Boca Raton: CRC Press; 2016.
Airoldi M, Morton A, Smith JAE, Bevan G. STAR–people-powered prioritization: a 21st-century solution to allocation headaches. Med Decis Mak. 2014;34(8):965–75.
Thokala P, Ochalek J, Leech AA, Tong T. Cost-effectiveness thresholds: the past, the present and the future. Pharmacoeconomics. 2018;36(5):509–22.
Sculpher M, Claxton K, Pearson SD. Developing a value framework: the need to reflect the opportunity costs of funding decisions. Value Health. 2017;20(2):234–9.
Madhavan G, Sangha K, Phelps C, Fryback D, Lieu T, Martinez RM, Committee on Identifying and Prioritizing New Preventive Vaccines for Development; Board on Population Health and Public Health Practice; Board on Global Health; Institute of Medicine, et al., editors. Ranking vaccines: a prioritization framework: phase I: demonstration of concept and a software blueprint. Washington, DC: The National Academies Press; 2012.
Madhavan G, Sangha K, Phelps C, Fryback D, Rappuoli R, Martinez R, Committee on Identifying and Prioritizing New Preventive Vaccines for Development, Phase II; Board on Population Health and Public Health Practice; Board on Global Health; Institute of Medicine, et al., editors. Ranking vaccines: a prioritization software tool: phase II: prototype of a decision-support system. Washington, DC: The National Academies Press; 2013.
Madhavan G, Phelps C, Rappuoli R, Martinez RM, King L, Committee on Identifying and Prioritizing New Preventive Vaccines for Development, Phase III; Board on Population Health and Public Health Practice; Board on Global Health; Institute of Medicine; National Academy of Engineering, editors. Ranking vaccines: applications of a prioritization software tool: phase III: use case studies and data framework. Washington, DC: The National Academies Press; 2015.
Golan O, Hansen P. Which health technologies should be funded? A prioritization framework based explicitly on value for money. Isr J Health Policy Res. 2012;1(1):44.
Sullivan T, Hansen P. Determining criteria and weights for prioritizing health technologies based on the preferences of the general population: a New Zealand pilot study. Value Health. 2017;20(4):679–86.
Castro Jaramillo HE, Goetghebeur M, Moreno-Mattar O. Testing multi-criteria decision analysis for more transparent resource-allocation decision making in Colombia. Int J Technol Assess Health Care. 2016;32(4):307–14.
Public Health England. The prioritisation framework: making the most of your budget. https://www.gov.uk/government/publications/the-prioritisation-framework-making-the-most-of-your-budget. 2018. Accessed 19 Mar 2019.
Bana e Costa CA, de Corte J-M, Vansnick J-C. MACBETH (Measuring Attractiveness by a Categorical Based Evaluation Technique). In: Wiley encyclopedia of operations research and management science. Hoboken: Wiley; 2011.
Hansen P, Ombler F. A new method for scoring additive multi-attribute value models using pairwise rankings of alternatives. J Multi Criteria Decis Anal. 2008;15(3–4):87–107.
Brans JP, Vinke P. Preference ranking organization method for enrichment evaluations. Manag Sci. 1985;31:6. https://difusion.ulb.ac.be/vufind/Record/ULB-DIPOT:oai:dipot.ulb.ac.be:2013/15711/Details. Accessed 17 May 2019
Keeney RL, Raiffa H. Decisions with multiple objectives : preferences and value tradeoffs. Cambridge: Cambridge University Press; 1993.
Pole JD. Programmes, priorities, and budgets. Br J Prev Soc Med. 1974;28(3):191–5.
Abdullah L, Adawiyah CW. Simple additive weighting methods of multi criteria decision making and applications: a decade review. Int J Inf Process Manag. 2014;5(1):39–49.
Weernink MGM, Janus SIM, van Til JA, Raisch DW, van Manen JG, IJzerman MJ. A systematic review to identify the use of preference elicitation methods in healthcare decision making. Pharm Med. 2014;28(4):175–85.
Weistroffer HR, Li Y. Multiple criteria decision analysis software. New York: Springer; 2016. p. 1301–41.
Thokala P, Madhavan G. Stakeholder involvement in multi-criteria decision analysis. Cost Eff Resour Alloc. 2018. https://doi.org/10.1186/s12962-018-0120-0.
Sacha D, Stoffel A, Stoffel F, Kwon BC, Ellis G, Keim DA. Knowledge generation model for visual analytics. IEEE Trans Vis Comput Graph. 2014;20(12):1604–13.
Marsh K, IJzerman M, Thokala P, Baltussen R, Boysen M, Kaló Z, et al. Multiple criteria decision analysis for health care decision making—emerging good practices: report 2 of the ISPOR MCDA Emerging Good Practices Task Force. Value Health. 2016;19(2):125–37.
Broekhuizen H, Groothuis-Oudshoorn CGM, van Til JA, Hummel JM, IJzerman MJ. A review and classification of approaches for dealing with uncertainty in multi-criteria decision analysis for healthcare decisions. Pharmacoeconomics. 2015;33(5):445–55.
Smith PC. Measuring value for money in healthcare: concepts and tools. Quest for quality and improved performance. London: The Health Foundation; 2009. p. 1–54.
Marsh K, Thokala P, Youngkong S, Chalkidou K. Incorporating MCDA into HTA: challenges and potential solutions, with a focus on lower income settings. Cost Eff Resour Alloc. 2018;16(Suppl 1):43.
Li Y, Thomas MA. A multiple criteria decision analysis (MCDA) software selection framework. In: Proceedings of the 2014 47th Annual Hawaii international conference on system sciences. Washington, DC: IEEE; 2014. p. 1084–94.
Weistroffer HR, Li Y. Multiple criteria decision analysis software. In: Greco S, Ehrgott M, Figueira JR, editors. Multiple criteria decision analysis State of the art surveys, vol. 233., International series in operations research & management scienceNew York: Springer; 2016. p. 1301–41.
Mustajoki J, Marttunen M. Comparison of multi-criteria decision analytical software for supporting environmental planning processes. Environ Model Softw. 2017;93:78–91.
Baizyldayeva U, Vlasov O, Kuandykov AA, Akhmetov TB. Multi-criteria decision support systems. Comparative analysis. Middle East J Sci Res. 2013;16(12):1725–30.
The authors would like to acknowledge all of those who contributed to this work.
The manuscript was developed in a doctoral internship funded by the Administrative Department of Science, Technology and Innovation of Colombia (COLCIENCIAS, Bogotá, Colombia), grant number 617.
The original idea and structure of the manuscript were developed by PT and AM. The development of the systematic review was done by AM and TT. AM drafted the manuscript and it was reviewed by PT and TT. All authors contributed to the multiple iterations of the manuscript.
Conflict of interest
Alexander Moreno is a PhD candidate at the Universidad Nacional de Colombia, Bogotá, Colombia. He received a doctoral grant; however, he has no conflicts of interest. Thaison Tong is a Research Associate in Health Economics and Decision Sciences at the School of Health and Related Research, University of Sheffield, Sheffield, UK. He has no financial or non-financial conflicts. Praveen Thokala is a Senior Research Fellow in Health Economics and Decision Sciences at the School of Health and Related Research, University of Sheffield. He has no financial or non-financial conflicts.
See Table 3.
About this article
Cite this article
Moreno-Calderón, A., Tong, T.S. & Thokala, P. Multi-criteria Decision Analysis Software in Healthcare Priority Setting: A Systematic Review. PharmacoEconomics 38, 269–283 (2020). https://doi.org/10.1007/s40273-019-00863-9