Multi-criteria Decision Analysis Software in Healthcare Priority Setting: A Systematic Review

  • Alexander Moreno-CalderónEmail author
  • Thai S. Tong
  • Praveen Thokala
Systematic Review



The objectives of this systematic review were to identify studies using Multi-Criteria Decision Analysis (MCDA) software tools to support health prioritisation processes and describe the technical capabilities of the MCDA software tools identified.


First, a systematic literature review was conducted in the MEDLINE, EMBASE, Web of Science, EconLit and Cochrane databases in July 2019 to identify studies that have used MCDA software for priority setting in health-related problems. Second, the MCDA software tools found in the review were downloaded (full versions, where freely available, and trial versions otherwise) and tested to extract their key technical characteristics.


Nine studies were included, from which seven different software tools, 1000minds®, M-MACBETH, Socio Technical Allocation of Resources (STAR), Strategic Multi-Attribute Ranking Tool (SMART), Visual PROMETHEE, EVIDEM and the Prioritisation Framework, were identified. These software tools differed in terms of the operating systems (including web interface), MCDA technique(s) available for use, visualisation features, and the capability to perform Value for Money (VfM) and sensitivity analyses.


The use of MCDA software in prioritisation processes has a number of advantages such as inclusion of several types of stakeholders and the ability to analyse a greater number of alternatives and criteria and perform real-time sensitivity analyses. Proprietary software (i.e. software with licensing fees) seemed to have more features than freely available software. However, this field is still developing, with only a few studies where MCDA software was used to support health priority setting and opportunity costs not explicitly captured in many software tools.



The authors would like to acknowledge all of those who contributed to this work.


The manuscript was developed in a doctoral internship funded by the Administrative Department of Science, Technology and Innovation of Colombia (COLCIENCIAS, Bogotá, Colombia), grant number 617.

Author contributions

The original idea and structure of the manuscript were developed by PT and AM. The development of the systematic review was done by AM and TT. AM drafted the manuscript and it was reviewed by PT and TT. All authors contributed to the multiple iterations of the manuscript.

Conflict of interest

Alexander Moreno is a PhD candidate at the Universidad Nacional de Colombia, Bogotá, Colombia. He received a doctoral grant; however, he has no conflicts of interest. Thaison Tong is a Research Associate in Health Economics and Decision Sciences at the School of Health and Related Research, University of Sheffield, Sheffield, UK. He has no financial or non-financial conflicts. Praveen Thokala is a Senior Research Fellow in Health Economics and Decision Sciences at the School of Health and Related Research, University of Sheffield. He has no financial or non-financial conflicts.


  1. 1.
    Defechereux T, Paolucci F, Mirelman A, Youngkong S, Botten G, Hagen TP, et al. Health care priority setting in Norway a multicriteria decision analysis. BMC Health Serv Res. 2012;12:39.PubMedPubMedCentralCrossRefGoogle Scholar
  2. 2.
    Sabik LM, Lie RK. Priority setting in health care: Lessons from the experiences of eight countries. Int J Equity Health. 2008;7:4.PubMedPubMedCentralCrossRefGoogle Scholar
  3. 3.
    Nunes R, Rego G. Priority setting in health care: a complementary approach. Health Care Anal. 2014;22(3):292–303.PubMedCrossRefGoogle Scholar
  4. 4.
    Baltussen R, Jansen MPM, Bijlmakers L, Grutters J, Kluytmans A, Reuzel RP, et al. Value assessment frameworks for HTA agencies: the organization of evidence-informed deliberative processes. Value Health. 2017;20(2):256–60.PubMedCrossRefGoogle Scholar
  5. 5.
    Giedion Ú, Villar M, Ávila A. Los sistemas de salud en Latinoamérica y el papel del seguro privado. 2010. Accessed 21 Nov 2019.
  6. 6.
    Hall W, Williams I, Smith N, Gold M, Coast J, Kapiriri L, et al. Past, present and future challenges in health care priority setting. J Health Organ Manag. 2018;32(3):444–62.PubMedCrossRefGoogle Scholar
  7. 7.
    Cox R, Sanchez J, Revie CW. Multi-criteria decision analysis tools for prioritising emerging or re-emerging infectious diseases associated with climate change in Canada. PLoS One. 2013;8(8):e68338.PubMedPubMedCentralCrossRefGoogle Scholar
  8. 8.
    Baltussen R. Priority setting of public spending in developing countries: do not try to do everything for everybody. Health Policy. 2006;78(2–3):149–56.PubMedCrossRefGoogle Scholar
  9. 9.
    Mullen PM. Quantifying priorities in healthcare: transparency or illusion? Health Serv Manag Res. 2004;17(1):47–58.CrossRefGoogle Scholar
  10. 10.
    Hongoh V, Michel P, Gosselin P, Samoura K, Ravel A, Campagna C, et al. Multi-stakeholder decision aid for improved prioritization of the public health impact of climate sensitive infectious diseases. Int J Environ Res Public Health. 2016;13(4):419.PubMedPubMedCentralCrossRefGoogle Scholar
  11. 11.
    Angelis A, Kanavos P. Value-based assessment of new medical technologies: towards a robust methodological framework for the application of multiple criteria decision analysis in the context of health technology assessment. Pharmacoeconomics. 2016;34(5):435–46.PubMedPubMedCentralCrossRefGoogle Scholar
  12. 12.
    Confederation NHS. Priority setting: an overview. London: Primary Care Trust Network; 2007.Google Scholar
  13. 13.
    Belton V, Stewart T. Multiple criteria decision analysis: an integrated approach. Dordrecht: Springer Science + Business Media; 2002.CrossRefGoogle Scholar
  14. 14.
    Marsh K, Lanitis T, Neasham D, Orfanos P, Caro J. Assessing the value of healthcare interventions using multi-criteria decision analysis: a review of the literature. Pharmacoeconomics. 2014;32(4):345–65.PubMedCrossRefPubMedCentralGoogle Scholar
  15. 15.
    Adunlin G, Diaby V, Xiao H. Application of multicriteria decision analysis in health care: a systematic review and bibliometric analysis. Health Expect. 2015;18(6):1894–905.PubMedCrossRefPubMedCentralGoogle Scholar
  16. 16.
    Ghandour R, Shoaibi A, Khatib R, Abu Rmeileh N, Unal B, Sözmen K, et al. Priority setting for the prevention and control of cardiovascular diseases: multi-criteria decision analysis in four eastern Mediterranean countries. Int J Public Health. 2015;60(Suppl 1 S1):S73–81.PubMedCrossRefPubMedCentralGoogle Scholar
  17. 17.
    Kosherbayeva L, Hailey D, Kurakbaev K, Tabarov A, Kumar A, Gutzskaya G, et al. A process of prioritizing topics for health technology assessment in kazakhstan. Int J Technol Assess Health Care. 2016;32(3):147–51.PubMedCrossRefPubMedCentralGoogle Scholar
  18. 18.
    Ishizaka A, Nemery P. Multi-criteria decision analysis: methods and software. Chichester: Wiley; 2013.CrossRefGoogle Scholar
  19. 19.
    Claxton K. Three questions to ask when examining MCDA. Value Outcomes Spotlight. 2015;1:18–20.Google Scholar
  20. 20.
    Mustajoki J, Marttunen M. Comparison of multi-criteria decision analytical software-searching for ideas for developing a new EIA-specific multi-criteria software. Environ Modell Softw. 2013;93(C):78–91.Google Scholar
  21. 21.
    Sengupta R, Gupta A, Dutta J. Decision sciences: theory and practice. 1st ed. Boca Raton: CRC Press; 2016.CrossRefGoogle Scholar
  22. 22.
    Airoldi M, Morton A, Smith JAE, Bevan G. STAR–people-powered prioritization: a 21st-century solution to allocation headaches. Med Decis Mak. 2014;34(8):965–75.CrossRefGoogle Scholar
  23. 23.
    Thokala P, Ochalek J, Leech AA, Tong T. Cost-effectiveness thresholds: the past, the present and the future. Pharmacoeconomics. 2018;36(5):509–22.PubMedCrossRefGoogle Scholar
  24. 24.
    Sculpher M, Claxton K, Pearson SD. Developing a value framework: the need to reflect the opportunity costs of funding decisions. Value Health. 2017;20(2):234–9.PubMedCrossRefGoogle Scholar
  25. 25.
    Madhavan G, Sangha K, Phelps C, Fryback D, Lieu T, Martinez RM, Committee on Identifying and Prioritizing New Preventive Vaccines for Development; Board on Population Health and Public Health Practice; Board on Global Health; Institute of Medicine, et al., editors. Ranking vaccines: a prioritization framework: phase I: demonstration of concept and a software blueprint. Washington, DC: The National Academies Press; 2012.Google Scholar
  26. 26.
    Madhavan G, Sangha K, Phelps C, Fryback D, Rappuoli R, Martinez R, Committee on Identifying and Prioritizing New Preventive Vaccines for Development, Phase II; Board on Population Health and Public Health Practice; Board on Global Health; Institute of Medicine, et al., editors. Ranking vaccines: a prioritization software tool: phase II: prototype of a decision-support system. Washington, DC: The National Academies Press; 2013.Google Scholar
  27. 27.
    Madhavan G, Phelps C, Rappuoli R, Martinez RM, King L, Committee on Identifying and Prioritizing New Preventive Vaccines for Development, Phase III; Board on Population Health and Public Health Practice; Board on Global Health; Institute of Medicine; National Academy of Engineering, editors. Ranking vaccines: applications of a prioritization software tool: phase III: use case studies and data framework. Washington, DC: The National Academies Press; 2015.Google Scholar
  28. 28.
    Golan O, Hansen P. Which health technologies should be funded? A prioritization framework based explicitly on value for money. Isr J Health Policy Res. 2012;1(1):44.PubMedPubMedCentralCrossRefGoogle Scholar
  29. 29.
    Sullivan T, Hansen P. Determining criteria and weights for prioritizing health technologies based on the preferences of the general population: a New Zealand pilot study. Value Health. 2017;20(4):679–86.PubMedCrossRefGoogle Scholar
  30. 30.
    Castro Jaramillo HE, Goetghebeur M, Moreno-Mattar O. Testing multi-criteria decision analysis for more transparent resource-allocation decision making in Colombia. Int J Technol Assess Health Care. 2016;32(4):307–14.PubMedCrossRefGoogle Scholar
  31. 31.
    Public Health England. The prioritisation framework: making the most of your budget. 2018. Accessed 19 Mar 2019.
  32. 32.
    Bana e Costa CA, de Corte J-M, Vansnick J-C. MACBETH (Measuring Attractiveness by a Categorical Based Evaluation Technique). In: Wiley encyclopedia of operations research and management science. Hoboken: Wiley; 2011.Google Scholar
  33. 33.
    Hansen P, Ombler F. A new method for scoring additive multi-attribute value models using pairwise rankings of alternatives. J Multi Criteria Decis Anal. 2008;15(3–4):87–107.CrossRefGoogle Scholar
  34. 34.
    Brans JP, Vinke P. Preference ranking organization method for enrichment evaluations. Manag Sci. 1985;31:6. Accessed 17 May 2019
  35. 35.
    Keeney RL, Raiffa H. Decisions with multiple objectives : preferences and value tradeoffs. Cambridge: Cambridge University Press; 1993.CrossRefGoogle Scholar
  36. 36.
    Pole JD. Programmes, priorities, and budgets. Br J Prev Soc Med. 1974;28(3):191–5.PubMedPubMedCentralGoogle Scholar
  37. 37.
    Abdullah L, Adawiyah CW. Simple additive weighting methods of multi criteria decision making and applications: a decade review. Int J Inf Process Manag. 2014;5(1):39–49.Google Scholar
  38. 38.
    Weernink MGM, Janus SIM, van Til JA, Raisch DW, van Manen JG, IJzerman MJ. A systematic review to identify the use of preference elicitation methods in healthcare decision making. Pharm Med. 2014;28(4):175–85.CrossRefGoogle Scholar
  39. 39.
    Weistroffer HR, Li Y. Multiple criteria decision analysis software. New York: Springer; 2016. p. 1301–41.CrossRefGoogle Scholar
  40. 40.
    Thokala P, Madhavan G. Stakeholder involvement in multi-criteria decision analysis. Cost Eff Resour Alloc. 2018. Scholar
  41. 41.
    Sacha D, Stoffel A, Stoffel F, Kwon BC, Ellis G, Keim DA. Knowledge generation model for visual analytics. IEEE Trans Vis Comput Graph. 2014;20(12):1604–13.PubMedCrossRefGoogle Scholar
  42. 42.
    Marsh K, IJzerman M, Thokala P, Baltussen R, Boysen M, Kaló Z, et al. Multiple criteria decision analysis for health care decision making—emerging good practices: report 2 of the ISPOR MCDA Emerging Good Practices Task Force. Value Health. 2016;19(2):125–37.PubMedCrossRefGoogle Scholar
  43. 43.
    Broekhuizen H, Groothuis-Oudshoorn CGM, van Til JA, Hummel JM, IJzerman MJ. A review and classification of approaches for dealing with uncertainty in multi-criteria decision analysis for healthcare decisions. Pharmacoeconomics. 2015;33(5):445–55.PubMedPubMedCentralCrossRefGoogle Scholar
  44. 44.
    Smith PC. Measuring value for money in healthcare: concepts and tools. Quest for quality and improved performance. London: The Health Foundation; 2009. p. 1–54.Google Scholar
  45. 45.
    Marsh K, Thokala P, Youngkong S, Chalkidou K. Incorporating MCDA into HTA: challenges and potential solutions, with a focus on lower income settings. Cost Eff Resour Alloc. 2018;16(Suppl 1):43.PubMedPubMedCentralCrossRefGoogle Scholar
  46. 46.
    Li Y, Thomas MA. A multiple criteria decision analysis (MCDA) software selection framework. In: Proceedings of the 2014 47th Annual Hawaii international conference on system sciences. Washington, DC: IEEE; 2014. p. 1084–94.Google Scholar
  47. 47.
    Weistroffer HR, Li Y. Multiple criteria decision analysis software. In: Greco S, Ehrgott M, Figueira JR, editors. Multiple criteria decision analysis State of the art surveys, vol. 233., International series in operations research & management scienceNew York: Springer; 2016. p. 1301–41.CrossRefGoogle Scholar
  48. 48.
    Mustajoki J, Marttunen M. Comparison of multi-criteria decision analytical software for supporting environmental planning processes. Environ Model Softw. 2017;93:78–91.CrossRefGoogle Scholar
  49. 49.
    Baizyldayeva U, Vlasov O, Kuandykov AA, Akhmetov TB. Multi-criteria decision support systems. Comparative analysis. Middle East J Sci Res. 2013;16(12):1725–30.Google Scholar

Copyright information

© Springer Nature Switzerland AG 2019

Authors and Affiliations

  1. 1.Universidad Nacional de ColombiaBogotáColombia
  2. 2.School of Health and Related ResearchUniversity of SheffieldSheffieldUK

Personalised recommendations