Skip to main content
Log in

Four Aspects Affecting Health Economic Decision Models and Their Validation

  • Current Opinion
  • Published:
PharmacoEconomics Aims and scope Submit manuscript

Abstract

Health care decision makers in many jurisdictions use cost-effectiveness analysis based on health economic decision models for policy decisions regarding coverage and price negotiation for medicines and medical devices. While validation of health economic decision models has always been considered important, many reviews of model-based cost-effectiveness studies report limitations regarding their validation. The current opinion paper discusses four aspects of current health economic decision modeling with relevance for future directions in model validation: increased use of complex models, international cooperation, open-source modeling, and stakeholder involvement. First, new, more complex clinical study designs and treatment strategies may require relatively complex model structures and/or input data analyses. Simultaneously, more widespread technical knowledge along with wider data availability have led to a broader range of model types. This puts extra requirements on model validation and transparency. Second, increased international cooperation of policy makers and, in particular, health technology assessment (HTA) authorities in performing model assessments is discussed in relation to the repeated use of health economic models (multi-use disease models). We argue such coordinated efforts may benefit model validity. Third, open-source modeling is discussed as one possible answer to increased transparency requirements. Finally, involvement of all relevant stakeholders throughout the whole decision process is an ongoing development that necessarily also includes health economic modeling. We argue this implies that model validity should be considered in a broader perspective, with more focus on conceptual modeling, model transparency, accuracy requirements, and choice of relevant model outcomes than previously.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Similar content being viewed by others

References

  1. Barendregt JJ, Bonneux L. The trouble with health economics. Eur J Public Health. 1999;9(Part 4):309–12.

    Article  Google Scholar 

  2. Miller JD, Foley KA, Russell MW. Current challenges in health economic modeling of cancer therapies: a research inquiry. Am Health Drug Benefits. 2014;7(3):153–62.

    PubMed  PubMed Central  Google Scholar 

  3. Langley PC. Another imaginary world: the ICER claims for the long-term cost-effectiveness and pricing of vesicular monoamine transporter 2 (VMAT2) inhibitors in tardive dyskinesia. Innov Pharm. 2017;8(4):12.

    Article  Google Scholar 

  4. Asche CV, Hippler SE, Eurich DT. Review of models used in economic analyses of new oral treatments for type 2 diabetes mellitus. Pharmacoeconomics. 2014;32(1):15–27.

    Article  PubMed  Google Scholar 

  5. Ghabri S, Binard A, Pers YM, Maunoury F, Caro JJ. Economic evaluation of sequences of biological treatments for patients with moderate-to-severe rheumatoid arthritis and inadequate response or intolerance to methotrexate in France. Value Health. 2020;23(4):461–70.

    Article  PubMed  Google Scholar 

  6. Pennington B, Filby A, Owen L, Taylor M. Smoking cessation: a comparison of two model structures. Pharmacoeconomics. 2018;36(9):1101–12.

    Article  PubMed  Google Scholar 

  7. Mauskopf J, Standaert B, Connolly MP, Culyer AJ, Garrison LP, Hutubessy R, et al. Economic analysis of vaccination programs: an ISPOR good practices for outcomes research task force report. Value Health. 2018;21(10):1133–49.

    Article  PubMed  Google Scholar 

  8. Karnon J. Model validation: has it’s time come? Pharmacoeconomics. 2016;34(9):829–31.

    Article  PubMed  Google Scholar 

  9. Caro JJ, Briggs AH, Siebert U, et al. Modeling good research practices—overview: a report of the ISPOR-SMDM Modeling Good Research Practices Task Force-1. Value Health. 2012;15(5):796–803.

    Article  PubMed  Google Scholar 

  10. Vemer P, Van Voorn GAK, Ramos C, Krabbe PFM, Al MJ, Feenstra TL. Improving model validation in health technology assessment: comments on guidelines of the ISPOR-SMDM modeling good research practices task force. Value Health. 2013;16(6):1106–7.

    Article  PubMed  Google Scholar 

  11. Naylor TH, Finger JM, McKenney JL, Schrank WE, Holt CC. Verification of computer simulation models. Manag Sci. 1967;14(2):B92–106.

    Article  Google Scholar 

  12. Tappenden P, Chilcott JB. Avoiding and identifying errors and other threats to the credibility of health economic models. Pharmacoeconomics. 2014;32:967–79.

    Article  PubMed  Google Scholar 

  13. Radeva D, Hopkin G, Mossialos E, Borrill J, Osipenko L, Naci H. Assessment of technical errors and validation processes in economic models submitted by the company for NICE technology appraisals. Int J Technol Assess Health Care. 2020;3:1–6.

    Google Scholar 

  14. Ghabri S, Stevenson M, Moller J, Caro JJ. Trusting the results of model-based economic analyses: is there a pragmatic validation solution? Pharmacoeconomics. 2019;37(1):1–6.

    Article  PubMed  Google Scholar 

  15. Ciani O, Buyse M, Drummond M, Rasi G, Saad ED, Taylor RS. Time to review the role of surrogate end points in health policy: state of the art and the way forward. Value Health. 2017;20(3):487–95.

    Article  PubMed  Google Scholar 

  16. Vemer P, Corro Ramos I, van Voorn GA, Al MJ, Feenstra TL. AdViSHE: a validation-assessment tool of health-economic models for decision makers and model users. Pharmacoeconomics. 2016;34(4):349–61.

    Article  CAS  PubMed  Google Scholar 

  17. Sargent RG. Verification, validation and accreditation of simulation models. In: Joines JA, Barton RR, Kank K, Fishwick PA (eds) 2000 Winter Simulation Conference Proceedings. Cat no. 00CH37165 ed. IEEE; 2000.

  18. Kasztura M, Richard A, Bempong NE, Loncar D, Flahault A. Cost-effectiveness of precision medicine: a scoping review. Int J Public Health. 2019;64(9):1261–71.

    Article  PubMed  Google Scholar 

  19. Phillips KA, Deverka PA, Marshall DA, Wordsworth S, Regier DA, Christensen KD, et al. Methodological issues in assessing the economic value of next-generation sequencing tests: many challenges and not enough solutions. Value Health. 2018;21(9):1033–42.

    Article  PubMed  PubMed Central  Google Scholar 

  20. Ultsch B, Damm O, Beutels P, Bilcke J, Bruggenjurgen B, Gerber-Grote A, et al. Methods for health economic evaluation of vaccines and immunization decision frameworks: a consensus framework from a european vaccine economics community. Pharmacoeconomics. 2016;34(3):227–44.

    Article  PubMed  Google Scholar 

  21. Liu S, Li Y, Triantis KP, Xue H, Wang Y. The diffusion of discrete event simulation approaches in health care management in the past four decades: a comprehensive review. MDM Policy Pract. 2020;5(1):238146832091524.

    Article  Google Scholar 

  22. Caro JJ, Möller J. Advantages and disadvantages of discrete-event simulation for health economic analyses. Expert Rev Pharmacoecon Outcomes Res. 2016;16(3):327–9.

    Article  PubMed  Google Scholar 

  23. Tappenden P, et al. Whole disease modelling to inform resource allocation decisions in cancer: a methodological framework. Value Health. 2012;15(8):1127–36.

    Article  PubMed  Google Scholar 

  24. Afzali HHA, Karnon J, Merlin T. Improving the accuracy and comparability of model-based economic evaluations of health technologies for reimbursement decisions: a methodological framework for the development of reference models. Med Decis Making. 2013;33(3):325–32.

    Article  PubMed  Google Scholar 

  25. Proposal for a regulation of the European Parliament and of the Council on health technology assessment and amending Directive 2011/24/EU. COM/2018/051 final—2018/018 (COD). https://eur-lex.europa.eu/legal-content/EN/TXT/?uri=celex%3A52018PC0051. Accessed Oct 2021

  26. O’Mahony JF. Beneluxa: What are the prospects for collective bargaining on pharmaceutical prices given diverse health technology assessment processes? Pharmacoeconomics. 2019;37:627–30.

    Article  PubMed  Google Scholar 

  27. Haute Autorité de Santé. Évaluation médico-économique des traitements de fond biologiques dans la prise en charge de la polyarthrite rhumatoïde. 2019. https://www.has-sante.fr/jcms/c_2580906/fr/evaluation-medico-economique-des-traitements-de-fond-biologiques-dans-la-prise-en-charge-de-la-polyarthrite-rhumatoide. Accessed Jul 2021.

  28. Dunlop WCN, Mason N, Kenworthy J, Akehurst RL. Benefits, challenges and potential strategies of open source health economic models. Pharmacoeconomics. 2017;35(1):125–8.

    Article  PubMed  Google Scholar 

  29. Gold M, Siegel J, Russell L, Weinstein M, editors. Cost-Effectiveness in Health and Medicine: Report of the Panel on Cost Effectiveness in Health and Medicine. New York: Oxford University Pres; new edition; 1996.

    Google Scholar 

  30. Incerti D, Jansen JP. An open-source toolkit for developing flexible evidence-based decision and simulation models for value assessment in oncology with R. Value Health. 2018;21(Suppl 1):S223.

    Article  Google Scholar 

  31. Pechlivanoglou P, Enns E, Alarid-Escudero F, Krijkamp E, Jalal H, Yang A, et al. Decision Analysis in R for Technologies in Health (DARTH). 2020. http://darthworkgroup.com/ and https://github.com/DARTH-git. Accessed Nov 2020.

  32. Jalal H, Pechlivanoglou P, Krijkamp E, Alarid-Escudero F, Enns E, Hunink MGM. An overview of R in health decision sciences. Med Decis Making. 2017;37(7):735–46.

    Article  PubMed  Google Scholar 

  33. Incerti D, Hernandez EJM, Tkacz J, Jansen JP, Collier D, Gharaibeh M, et al. The effect of dose escalation on the cost-effectiveness of etanercept and adalimumab with methotrexate among patients with moderate to severe rheumatoid arthritis. J Manag Care Spec Pharm. 2020;26(10):1236–42.

    PubMed  Google Scholar 

  34. Degeling K, Wong HL, Koffijberg H, Jalali A, Shapiro J, Kosmider S, et al. Simulating progression-free and overall survival for first-line doublet chemotherapy with or without bevacizumab in metastatic colorectal cancer patients based on real-world registry data. Pharmacoeconomics. 2020;38(11):1263–75.

    Article  PubMed  Google Scholar 

  35. Sadatsafavi M, Ghanbarian S, Adibi A, Johnson K, FitzGerald JM, Flanagan W, et al. Development and validation of the evaluation platform in COPD (EPIC): a population-based outcomes model of COPD for Canada. Med Decis Making. 2019;39(2):152–67.

    Article  PubMed  Google Scholar 

  36. Corro Ramos I, Hoogendoorn M, Rutten-van Mölken MPMH. How to address uncertainty in health economic discrete-event simulation models: an illustration for chronic obstructive pulmonary disease. Med Decis Making. 2020;40(5):619–32.

    Article  PubMed  PubMed Central  Google Scholar 

  37. Philips Z, Bojke L, Sculpher M, Claxton K, Golder S. Good practice guidelines for decision-analytic modelling in health technology assessment: a review and consolidation of quality assessment. Pharmacoeconomics. 2006;24(4):355–71.

    Article  PubMed  Google Scholar 

  38. Husereau D, Drummond M, Petrou S, Carswell C, Moher D, Greenberg D, et al. Consolidated health economic evaluation reporting standards (CHEERS) statement. Value Health. 2013;16(2):e1-5.

    Article  PubMed  Google Scholar 

  39. CISNET Consortium. CISNET Publication Support and Modeling Resources. 2013. https://resources.cisnet.cancer.gov/projects/. Accessed Nov 2020.

  40. Diabetes Trials Unit, the Oxford Centre for Diabetes, Endocrinology and Metabolism. UKPDS Risk Engine. 2017. https://www.dtu.ox.ac.uk/riskengine/. Accessed Nov 2020.

  41. Jakeman AJ, Letcher RA, Norton JP. Ten iterative steps in development and evaluation of environmental models. Environ Model Softw. 2006;21(5):602–14.

    Article  Google Scholar 

  42. Voinov A, Jenni K, Gray S, et al. Tools and methods in participatory modeling: selecting the right tool for the job. Environ Model Softw. 2018;109:232–55.

    Article  Google Scholar 

  43. Sampson CJ, Arnold R, Bryan S, Clarke P, Ekins S, Hatswell A, et al. Transparency in decision modelling: what, why, who and how? Pharmacoeconomics. 2019;37(11):1355–69.

    Article  PubMed  PubMed Central  Google Scholar 

  44. Harvard S, Werker GR, Silva DS. Social, ethical, and other value judgments in health economics modelling. Soc Sci Med. 2020;253:112975.

    Article  PubMed  Google Scholar 

  45. van Voorn GA, Vemer P, Hamerlijnck D, Ramos IC, Teunissen GJ, Al M, et al. The missing stakeholder group: why patients should be involved in health economic modelling. Appl Health Econ Health Policy. 2016;14(2):129–33.

    Article  PubMed  Google Scholar 

  46. Hannigan A. Public and patient involvement in quantitative health research: a statistical perspective. Health Expect. 2018;21(6):939–43.

    Article  PubMed  PubMed Central  Google Scholar 

  47. Harrington RL, Hanna ML, Oehrlein EM, Camp R, Wheeler R, Cooblall C, et al. Defining patient engagement in research: results of a systematic review and analysis: report of the ISPOR patient-centered special interest group. Value Health. 2020;23(6):677–88.

    Article  PubMed  Google Scholar 

  48. Harvard S, Werker GR. Health economists on involving patients in modeling: potential benefits, harms, and variables of interest. Pharmacoeconomics. 2021;39(7):823–33.

    Article  PubMed  PubMed Central  Google Scholar 

  49. Squires H, Chilcott J, Akehurst R, et al. A framework for developing the structure of public health economic models. Value Health. 2016;19(5):588–601.

    Article  PubMed  Google Scholar 

  50. Lloyd-Williams F, Hyseni L, Guzman-Castillo M, et al. Evaluating stakeholder involvement in building a decision support tool for NHS health checks: co-producing the WorkHORSE study. BMC Med Inform Decis Making. 2020;20(1):1–12.

    Article  Google Scholar 

  51. Xie RZ, Malik ED, Linthicum MT, Bright JL. Putting stakeholder engagement at the center of health economic modeling for health technology assessment in the United States. Pharmacoeconomics. 2021;39(6):631–8.

    Article  PubMed  PubMed Central  Google Scholar 

  52. O’Donnell E, Atkinson JA, Freebairn L, Rychetnik L. Participatory simulation modelling to inform public health policy and practice: rethinking the evidence hierarchies. J Public Health Policy. 2017;38(2):203–15.

    Article  PubMed  Google Scholar 

  53. Büyükkaramikli NC, Rutten-van Mölken MPMH, Severens JL, Al M. TECH-VER: a verification checklist to reduce errors in models and improve their credibility. Pharmacoeconomics. 2019;37(11):1391–408.

    Article  PubMed  PubMed Central  Google Scholar 

  54. Dasbach EJ, Elbasha EH. Verification of decision-analytic models for health economic evaluations: an overview. Pharmacoeconomics. 2017;35(7):673–83.

    Article  PubMed  Google Scholar 

  55. Wang J, Pouwels X, Ramaekers B, et al. Multi-use disease models: a blueprint for application in support of health care insurance coverage policy and a case study in Diabetes Mellitus. RIVM letter report 2020-0145. Bilthoven: Rijksinstituut voor Volksgezondheid en Milieu RIVM; 2020.

Download references

Acknowledgements

The authors thank the audience at the 2019 IHEA workshop ‘Organized Session: From Art to Science—the Future of Model Validation. Four Trends in Health Economic Decision Modeling and Their Implications for Model Validation’ for their discussion. They also express gratitude to Stefan Lhachimi for chairing this workshop. Three anonymous reviewers have substantially commented on the initial version and helped us to better clarify our line of reasoning, which is gratefully acknowledged.

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Talitha Feenstra.

Ethics declarations

Funding

This opinion paper originates from an International Health Economic Association (IHEA) workshop presented in Basel 2019. The authors did not receive any funding for writing this paper, however the initial work as presented in the workshop was funded by an unrestricted grant from ZONMW (The Netherlands Organisation for Health Research and Development), grant no. 152002050(1), for (among others) TF, ICR, DH, and GvV.

Conflicts of interest/competing interests

Talitha Feenstra, Isaac Corro-Ramos, Dominique Hamerlijnck, Salah Ghabri, and George van Voorn declare they have no conflicts of interest to report.

Ethics approval

Not applicable.

Consent to participate

Not applicable.

Consent for publication

Not applicable.

Availability of data and material

Not applicable.

Code availability

Not applicable.

Author contributions

Discussions among all authors served to feed the ideas as presented in this paper. TF drafted the outline of the paper, and TF, ICR, SG, and GvV had important contributions in revising the manuscript versions. All authors commented on the final draft and approved the current version. The opinions expressed in this article are those of the authors and do not necessarily represent the views of their institutions.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Feenstra, T., Corro-Ramos, I., Hamerlijnck, D. et al. Four Aspects Affecting Health Economic Decision Models and Their Validation. PharmacoEconomics 40, 241–248 (2022). https://doi.org/10.1007/s40273-021-01110-w

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s40273-021-01110-w

Navigation