Skip to main content
SpringerLink
Log in

Multi-criteria Decision Analysis Software in Healthcare Priority Setting: A Systematic Review

  • Systematic Review
  • Published:
PharmacoEconomics Aims and scope Submit manuscript

Abstract

Objective

The objectives of this systematic review were to identify studies using Multi-Criteria Decision Analysis (MCDA) software tools to support health prioritisation processes and describe the technical capabilities of the MCDA software tools identified.

Methods

First, a systematic literature review was conducted in the MEDLINE, EMBASE, Web of Science, EconLit and Cochrane databases in July 2019 to identify studies that have used MCDA software for priority setting in health-related problems. Second, the MCDA software tools found in the review were downloaded (full versions, where freely available, and trial versions otherwise) and tested to extract their key technical characteristics.

Results

Nine studies were included, from which seven different software tools, 1000minds®, M-MACBETH, Socio Technical Allocation of Resources (STAR), Strategic Multi-Attribute Ranking Tool (SMART), Visual PROMETHEE, EVIDEM and the Prioritisation Framework, were identified. These software tools differed in terms of the operating systems (including web interface), MCDA technique(s) available for use, visualisation features, and the capability to perform Value for Money (VfM) and sensitivity analyses.

Conclusions

The use of MCDA software in prioritisation processes has a number of advantages such as inclusion of several types of stakeholders and the ability to analyse a greater number of alternatives and criteria and perform real-time sensitivity analyses. Proprietary software (i.e. software with licensing fees) seemed to have more features than freely available software. However, this field is still developing, with only a few studies where MCDA software was used to support health priority setting and opportunity costs not explicitly captured in many software tools.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1

Similar content being viewed by others

Availability of Data and Materials

All data analysed or generated during this study are included in this article.

References

  1. Defechereux T, Paolucci F, Mirelman A, Youngkong S, Botten G, Hagen TP, et al. Health care priority setting in Norway a multicriteria decision analysis. BMC Health Serv Res. 2012;12:39.

    PubMed  PubMed Central  Google Scholar 

  2. Sabik LM, Lie RK. Priority setting in health care: Lessons from the experiences of eight countries. Int J Equity Health. 2008;7:4.

    PubMed  PubMed Central  Google Scholar 

  3. Nunes R, Rego G. Priority setting in health care: a complementary approach. Health Care Anal. 2014;22(3):292–303.

    PubMed  Google Scholar 

  4. Baltussen R, Jansen MPM, Bijlmakers L, Grutters J, Kluytmans A, Reuzel RP, et al. Value assessment frameworks for HTA agencies: the organization of evidence-informed deliberative processes. Value Health. 2017;20(2):256–60.

    PubMed  Google Scholar 

  5. Giedion Ú, Villar M, Ávila A. Los sistemas de salud en Latinoamérica y el papel del seguro privado. 2010. https://www.mapfre.com/ccm/content/documentos/fundacion/cs-seguro/libros/los-sistemas-de-salud-en-latinoamerica-y-el-papel-del-seguro-privado.pdf. Accessed 21 Nov 2019.

  6. Hall W, Williams I, Smith N, Gold M, Coast J, Kapiriri L, et al. Past, present and future challenges in health care priority setting. J Health Organ Manag. 2018;32(3):444–62.

    PubMed  Google Scholar 

  7. Cox R, Sanchez J, Revie CW. Multi-criteria decision analysis tools for prioritising emerging or re-emerging infectious diseases associated with climate change in Canada. PLoS One. 2013;8(8):e68338.

    CAS  PubMed  PubMed Central  Google Scholar 

  8. Baltussen R. Priority setting of public spending in developing countries: do not try to do everything for everybody. Health Policy. 2006;78(2–3):149–56.

    PubMed  Google Scholar 

  9. Mullen PM. Quantifying priorities in healthcare: transparency or illusion? Health Serv Manag Res. 2004;17(1):47–58.

    Google Scholar 

  10. Hongoh V, Michel P, Gosselin P, Samoura K, Ravel A, Campagna C, et al. Multi-stakeholder decision aid for improved prioritization of the public health impact of climate sensitive infectious diseases. Int J Environ Res Public Health. 2016;13(4):419.

    PubMed  PubMed Central  Google Scholar 

  11. Angelis A, Kanavos P. Value-based assessment of new medical technologies: towards a robust methodological framework for the application of multiple criteria decision analysis in the context of health technology assessment. Pharmacoeconomics. 2016;34(5):435–46.

    PubMed  PubMed Central  Google Scholar 

  12. Confederation NHS. Priority setting: an overview. London: Primary Care Trust Network; 2007.

    Google Scholar 

  13. Belton V, Stewart T. Multiple criteria decision analysis: an integrated approach. Dordrecht: Springer Science + Business Media; 2002.

    Google Scholar 

  14. Marsh K, Lanitis T, Neasham D, Orfanos P, Caro J. Assessing the value of healthcare interventions using multi-criteria decision analysis: a review of the literature. Pharmacoeconomics. 2014;32(4):345–65.

    PubMed  Google Scholar 

  15. Adunlin G, Diaby V, Xiao H. Application of multicriteria decision analysis in health care: a systematic review and bibliometric analysis. Health Expect. 2015;18(6):1894–905.

    PubMed  Google Scholar 

  16. Ghandour R, Shoaibi A, Khatib R, Abu Rmeileh N, Unal B, Sözmen K, et al. Priority setting for the prevention and control of cardiovascular diseases: multi-criteria decision analysis in four eastern Mediterranean countries. Int J Public Health. 2015;60(Suppl 1 S1):S73–81.

    PubMed  Google Scholar 

  17. Kosherbayeva L, Hailey D, Kurakbaev K, Tabarov A, Kumar A, Gutzskaya G, et al. A process of prioritizing topics for health technology assessment in kazakhstan. Int J Technol Assess Health Care. 2016;32(3):147–51.

    PubMed  Google Scholar 

  18. Ishizaka A, Nemery P. Multi-criteria decision analysis: methods and software. Chichester: Wiley; 2013.

    Google Scholar 

  19. Claxton K. Three questions to ask when examining MCDA. Value Outcomes Spotlight. 2015;1:18–20.

    Google Scholar 

  20. Mustajoki J, Marttunen M. Comparison of multi-criteria decision analytical software-searching for ideas for developing a new EIA-specific multi-criteria software. Environ Modell Softw. 2013;93(C):78–91.

    Google Scholar 

  21. Sengupta R, Gupta A, Dutta J. Decision sciences: theory and practice. 1st ed. Boca Raton: CRC Press; 2016.

    Google Scholar 

  22. Airoldi M, Morton A, Smith JAE, Bevan G. STAR–people-powered prioritization: a 21st-century solution to allocation headaches. Med Decis Mak. 2014;34(8):965–75.

    Google Scholar 

  23. Thokala P, Ochalek J, Leech AA, Tong T. Cost-effectiveness thresholds: the past, the present and the future. Pharmacoeconomics. 2018;36(5):509–22.

    PubMed  Google Scholar 

  24. Sculpher M, Claxton K, Pearson SD. Developing a value framework: the need to reflect the opportunity costs of funding decisions. Value Health. 2017;20(2):234–9.

    PubMed  Google Scholar 

  25. Madhavan G, Sangha K, Phelps C, Fryback D, Lieu T, Martinez RM, Committee on Identifying and Prioritizing New Preventive Vaccines for Development; Board on Population Health and Public Health Practice; Board on Global Health; Institute of Medicine, et al., editors. Ranking vaccines: a prioritization framework: phase I: demonstration of concept and a software blueprint. Washington, DC: The National Academies Press; 2012.

    Google Scholar 

  26. Madhavan G, Sangha K, Phelps C, Fryback D, Rappuoli R, Martinez R, Committee on Identifying and Prioritizing New Preventive Vaccines for Development, Phase II; Board on Population Health and Public Health Practice; Board on Global Health; Institute of Medicine, et al., editors. Ranking vaccines: a prioritization software tool: phase II: prototype of a decision-support system. Washington, DC: The National Academies Press; 2013.

    Google Scholar 

  27. Madhavan G, Phelps C, Rappuoli R, Martinez RM, King L, Committee on Identifying and Prioritizing New Preventive Vaccines for Development, Phase III; Board on Population Health and Public Health Practice; Board on Global Health; Institute of Medicine; National Academy of Engineering, editors. Ranking vaccines: applications of a prioritization software tool: phase III: use case studies and data framework. Washington, DC: The National Academies Press; 2015.

    Google Scholar 

  28. Golan O, Hansen P. Which health technologies should be funded? A prioritization framework based explicitly on value for money. Isr J Health Policy Res. 2012;1(1):44.

    PubMed  PubMed Central  Google Scholar 

  29. Sullivan T, Hansen P. Determining criteria and weights for prioritizing health technologies based on the preferences of the general population: a New Zealand pilot study. Value Health. 2017;20(4):679–86.

    PubMed  Google Scholar 

  30. Castro Jaramillo HE, Goetghebeur M, Moreno-Mattar O. Testing multi-criteria decision analysis for more transparent resource-allocation decision making in Colombia. Int J Technol Assess Health Care. 2016;32(4):307–14.

    PubMed  Google Scholar 

  31. Public Health England. The prioritisation framework: making the most of your budget. https://www.gov.uk/government/publications/the-prioritisation-framework-making-the-most-of-your-budget. 2018. Accessed 19 Mar 2019.

  32. Bana e Costa CA, de Corte J-M, Vansnick J-C. MACBETH (Measuring Attractiveness by a Categorical Based Evaluation Technique). In: Wiley encyclopedia of operations research and management science. Hoboken: Wiley; 2011.

  33. Hansen P, Ombler F. A new method for scoring additive multi-attribute value models using pairwise rankings of alternatives. J Multi Criteria Decis Anal. 2008;15(3–4):87–107.

    Google Scholar 

  34. Brans JP, Vinke P. Preference ranking organization method for enrichment evaluations. Manag Sci. 1985;31:6. https://difusion.ulb.ac.be/vufind/Record/ULB-DIPOT:oai:dipot.ulb.ac.be:2013/15711/Details. Accessed 17 May 2019

  35. Keeney RL, Raiffa H. Decisions with multiple objectives : preferences and value tradeoffs. Cambridge: Cambridge University Press; 1993.

    Google Scholar 

  36. Pole JD. Programmes, priorities, and budgets. Br J Prev Soc Med. 1974;28(3):191–5.

    CAS  PubMed  PubMed Central  Google Scholar 

  37. Abdullah L, Adawiyah CW. Simple additive weighting methods of multi criteria decision making and applications: a decade review. Int J Inf Process Manag. 2014;5(1):39–49.

    Google Scholar 

  38. Weernink MGM, Janus SIM, van Til JA, Raisch DW, van Manen JG, IJzerman MJ. A systematic review to identify the use of preference elicitation methods in healthcare decision making. Pharm Med. 2014;28(4):175–85.

    Google Scholar 

  39. Weistroffer HR, Li Y. Multiple criteria decision analysis software. New York: Springer; 2016. p. 1301–41.

    Google Scholar 

  40. Thokala P, Madhavan G. Stakeholder involvement in multi-criteria decision analysis. Cost Eff Resour Alloc. 2018. https://doi.org/10.1186/s12962-018-0120-0.

    Article  PubMed  PubMed Central  Google Scholar 

  41. Sacha D, Stoffel A, Stoffel F, Kwon BC, Ellis G, Keim DA. Knowledge generation model for visual analytics. IEEE Trans Vis Comput Graph. 2014;20(12):1604–13.

    PubMed  Google Scholar 

  42. Marsh K, IJzerman M, Thokala P, Baltussen R, Boysen M, Kaló Z, et al. Multiple criteria decision analysis for health care decision making—emerging good practices: report 2 of the ISPOR MCDA Emerging Good Practices Task Force. Value Health. 2016;19(2):125–37.

    PubMed  Google Scholar 

  43. Broekhuizen H, Groothuis-Oudshoorn CGM, van Til JA, Hummel JM, IJzerman MJ. A review and classification of approaches for dealing with uncertainty in multi-criteria decision analysis for healthcare decisions. Pharmacoeconomics. 2015;33(5):445–55.

    PubMed  PubMed Central  Google Scholar 

  44. Smith PC. Measuring value for money in healthcare: concepts and tools. Quest for quality and improved performance. London: The Health Foundation; 2009. p. 1–54.

    Google Scholar 

  45. Marsh K, Thokala P, Youngkong S, Chalkidou K. Incorporating MCDA into HTA: challenges and potential solutions, with a focus on lower income settings. Cost Eff Resour Alloc. 2018;16(Suppl 1):43.

    PubMed  PubMed Central  Google Scholar 

  46. Li Y, Thomas MA. A multiple criteria decision analysis (MCDA) software selection framework. In: Proceedings of the 2014 47th Annual Hawaii international conference on system sciences. Washington, DC: IEEE; 2014. p. 1084–94.

  47. Weistroffer HR, Li Y. Multiple criteria decision analysis software. In: Greco S, Ehrgott M, Figueira JR, editors. Multiple criteria decision analysis State of the art surveys, vol. 233., International series in operations research & management scienceNew York: Springer; 2016. p. 1301–41.

    Google Scholar 

  48. Mustajoki J, Marttunen M. Comparison of multi-criteria decision analytical software for supporting environmental planning processes. Environ Model Softw. 2017;93:78–91.

    Google Scholar 

  49. Baizyldayeva U, Vlasov O, Kuandykov AA, Akhmetov TB. Multi-criteria decision support systems. Comparative analysis. Middle East J Sci Res. 2013;16(12):1725–30.

    Google Scholar 

Download references

Acknowledgements

The authors would like to acknowledge all of those who contributed to this work.

Funding

The manuscript was developed in a doctoral internship funded by the Administrative Department of Science, Technology and Innovation of Colombia (COLCIENCIAS, Bogotá, Colombia), grant number 617.

Author contributions

The original idea and structure of the manuscript were developed by PT and AM. The development of the systematic review was done by AM and TT. AM drafted the manuscript and it was reviewed by PT and TT. All authors contributed to the multiple iterations of the manuscript.

Conflict of interest

Alexander Moreno is a PhD candidate at the Universidad Nacional de Colombia, Bogotá, Colombia. He received a doctoral grant; however, he has no conflicts of interest. Thaison Tong is a Research Associate in Health Economics and Decision Sciences at the School of Health and Related Research, University of Sheffield, Sheffield, UK. He has no financial or non-financial conflicts. Praveen Thokala is a Senior Research Fellow in Health Economics and Decision Sciences at the School of Health and Related Research, University of Sheffield. He has no financial or non-financial conflicts.

Author information

Authors and Affiliations

  1. Universidad Nacional de Colombia, Carrera 45 # 26-85, Bogotá, Colombia

    Alexander Moreno-Calderón

  2. School of Health and Related Research, University of Sheffield, Regent Court, 30 Regent Street, Sheffield, S1 4DA, UK

    Thai S. Tong & Praveen Thokala

Authors
  1. Alexander Moreno-Calderón
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Thai S. Tong
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Praveen Thokala
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Alexander Moreno-Calderón.

Appendix

Appendix

See Table 3.

Table 3 Search strategies
Full size table

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Moreno-Calderón, A., Tong, T.S. & Thokala, P. Multi-criteria Decision Analysis Software in Healthcare Priority Setting: A Systematic Review. PharmacoEconomics 38, 269–283 (2020). https://doi.org/10.1007/s40273-019-00863-9

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s40273-019-00863-9

Search

Navigation