Abstract
Health care decision makers in many jurisdictions use cost-effectiveness analysis based on health economic decision models for policy decisions regarding coverage and price negotiation for medicines and medical devices. While validation of health economic decision models has always been considered important, many reviews of model-based cost-effectiveness studies report limitations regarding their validation. The current opinion paper discusses four aspects of current health economic decision modeling with relevance for future directions in model validation: increased use of complex models, international cooperation, open-source modeling, and stakeholder involvement. First, new, more complex clinical study designs and treatment strategies may require relatively complex model structures and/or input data analyses. Simultaneously, more widespread technical knowledge along with wider data availability have led to a broader range of model types. This puts extra requirements on model validation and transparency. Second, increased international cooperation of policy makers and, in particular, health technology assessment (HTA) authorities in performing model assessments is discussed in relation to the repeated use of health economic models (multi-use disease models). We argue such coordinated efforts may benefit model validity. Third, open-source modeling is discussed as one possible answer to increased transparency requirements. Finally, involvement of all relevant stakeholders throughout the whole decision process is an ongoing development that necessarily also includes health economic modeling. We argue this implies that model validity should be considered in a broader perspective, with more focus on conceptual modeling, model transparency, accuracy requirements, and choice of relevant model outcomes than previously.
Similar content being viewed by others
References
Barendregt JJ, Bonneux L. The trouble with health economics. Eur J Public Health. 1999;9(Part 4):309–12.
Miller JD, Foley KA, Russell MW. Current challenges in health economic modeling of cancer therapies: a research inquiry. Am Health Drug Benefits. 2014;7(3):153–62.
Langley PC. Another imaginary world: the ICER claims for the long-term cost-effectiveness and pricing of vesicular monoamine transporter 2 (VMAT2) inhibitors in tardive dyskinesia. Innov Pharm. 2017;8(4):12.
Asche CV, Hippler SE, Eurich DT. Review of models used in economic analyses of new oral treatments for type 2 diabetes mellitus. Pharmacoeconomics. 2014;32(1):15–27.
Ghabri S, Binard A, Pers YM, Maunoury F, Caro JJ. Economic evaluation of sequences of biological treatments for patients with moderate-to-severe rheumatoid arthritis and inadequate response or intolerance to methotrexate in France. Value Health. 2020;23(4):461–70.
Pennington B, Filby A, Owen L, Taylor M. Smoking cessation: a comparison of two model structures. Pharmacoeconomics. 2018;36(9):1101–12.
Mauskopf J, Standaert B, Connolly MP, Culyer AJ, Garrison LP, Hutubessy R, et al. Economic analysis of vaccination programs: an ISPOR good practices for outcomes research task force report. Value Health. 2018;21(10):1133–49.
Karnon J. Model validation: has it’s time come? Pharmacoeconomics. 2016;34(9):829–31.
Caro JJ, Briggs AH, Siebert U, et al. Modeling good research practices—overview: a report of the ISPOR-SMDM Modeling Good Research Practices Task Force-1. Value Health. 2012;15(5):796–803.
Vemer P, Van Voorn GAK, Ramos C, Krabbe PFM, Al MJ, Feenstra TL. Improving model validation in health technology assessment: comments on guidelines of the ISPOR-SMDM modeling good research practices task force. Value Health. 2013;16(6):1106–7.
Naylor TH, Finger JM, McKenney JL, Schrank WE, Holt CC. Verification of computer simulation models. Manag Sci. 1967;14(2):B92–106.
Tappenden P, Chilcott JB. Avoiding and identifying errors and other threats to the credibility of health economic models. Pharmacoeconomics. 2014;32:967–79.
Radeva D, Hopkin G, Mossialos E, Borrill J, Osipenko L, Naci H. Assessment of technical errors and validation processes in economic models submitted by the company for NICE technology appraisals. Int J Technol Assess Health Care. 2020;3:1–6.
Ghabri S, Stevenson M, Moller J, Caro JJ. Trusting the results of model-based economic analyses: is there a pragmatic validation solution? Pharmacoeconomics. 2019;37(1):1–6.
Ciani O, Buyse M, Drummond M, Rasi G, Saad ED, Taylor RS. Time to review the role of surrogate end points in health policy: state of the art and the way forward. Value Health. 2017;20(3):487–95.
Vemer P, Corro Ramos I, van Voorn GA, Al MJ, Feenstra TL. AdViSHE: a validation-assessment tool of health-economic models for decision makers and model users. Pharmacoeconomics. 2016;34(4):349–61.
Sargent RG. Verification, validation and accreditation of simulation models. In: Joines JA, Barton RR, Kank K, Fishwick PA (eds) 2000 Winter Simulation Conference Proceedings. Cat no. 00CH37165 ed. IEEE; 2000.
Kasztura M, Richard A, Bempong NE, Loncar D, Flahault A. Cost-effectiveness of precision medicine: a scoping review. Int J Public Health. 2019;64(9):1261–71.
Phillips KA, Deverka PA, Marshall DA, Wordsworth S, Regier DA, Christensen KD, et al. Methodological issues in assessing the economic value of next-generation sequencing tests: many challenges and not enough solutions. Value Health. 2018;21(9):1033–42.
Ultsch B, Damm O, Beutels P, Bilcke J, Bruggenjurgen B, Gerber-Grote A, et al. Methods for health economic evaluation of vaccines and immunization decision frameworks: a consensus framework from a european vaccine economics community. Pharmacoeconomics. 2016;34(3):227–44.
Liu S, Li Y, Triantis KP, Xue H, Wang Y. The diffusion of discrete event simulation approaches in health care management in the past four decades: a comprehensive review. MDM Policy Pract. 2020;5(1):238146832091524.
Caro JJ, Möller J. Advantages and disadvantages of discrete-event simulation for health economic analyses. Expert Rev Pharmacoecon Outcomes Res. 2016;16(3):327–9.
Tappenden P, et al. Whole disease modelling to inform resource allocation decisions in cancer: a methodological framework. Value Health. 2012;15(8):1127–36.
Afzali HHA, Karnon J, Merlin T. Improving the accuracy and comparability of model-based economic evaluations of health technologies for reimbursement decisions: a methodological framework for the development of reference models. Med Decis Making. 2013;33(3):325–32.
Proposal for a regulation of the European Parliament and of the Council on health technology assessment and amending Directive 2011/24/EU. COM/2018/051 final—2018/018 (COD). https://eur-lex.europa.eu/legal-content/EN/TXT/?uri=celex%3A52018PC0051. Accessed Oct 2021
O’Mahony JF. Beneluxa: What are the prospects for collective bargaining on pharmaceutical prices given diverse health technology assessment processes? Pharmacoeconomics. 2019;37:627–30.
Haute Autorité de Santé. Évaluation médico-économique des traitements de fond biologiques dans la prise en charge de la polyarthrite rhumatoïde. 2019. https://www.has-sante.fr/jcms/c_2580906/fr/evaluation-medico-economique-des-traitements-de-fond-biologiques-dans-la-prise-en-charge-de-la-polyarthrite-rhumatoide. Accessed Jul 2021.
Dunlop WCN, Mason N, Kenworthy J, Akehurst RL. Benefits, challenges and potential strategies of open source health economic models. Pharmacoeconomics. 2017;35(1):125–8.
Gold M, Siegel J, Russell L, Weinstein M, editors. Cost-Effectiveness in Health and Medicine: Report of the Panel on Cost Effectiveness in Health and Medicine. New York: Oxford University Pres; new edition; 1996.
Incerti D, Jansen JP. An open-source toolkit for developing flexible evidence-based decision and simulation models for value assessment in oncology with R. Value Health. 2018;21(Suppl 1):S223.
Pechlivanoglou P, Enns E, Alarid-Escudero F, Krijkamp E, Jalal H, Yang A, et al. Decision Analysis in R for Technologies in Health (DARTH). 2020. http://darthworkgroup.com/ and https://github.com/DARTH-git. Accessed Nov 2020.
Jalal H, Pechlivanoglou P, Krijkamp E, Alarid-Escudero F, Enns E, Hunink MGM. An overview of R in health decision sciences. Med Decis Making. 2017;37(7):735–46.
Incerti D, Hernandez EJM, Tkacz J, Jansen JP, Collier D, Gharaibeh M, et al. The effect of dose escalation on the cost-effectiveness of etanercept and adalimumab with methotrexate among patients with moderate to severe rheumatoid arthritis. J Manag Care Spec Pharm. 2020;26(10):1236–42.
Degeling K, Wong HL, Koffijberg H, Jalali A, Shapiro J, Kosmider S, et al. Simulating progression-free and overall survival for first-line doublet chemotherapy with or without bevacizumab in metastatic colorectal cancer patients based on real-world registry data. Pharmacoeconomics. 2020;38(11):1263–75.
Sadatsafavi M, Ghanbarian S, Adibi A, Johnson K, FitzGerald JM, Flanagan W, et al. Development and validation of the evaluation platform in COPD (EPIC): a population-based outcomes model of COPD for Canada. Med Decis Making. 2019;39(2):152–67.
Corro Ramos I, Hoogendoorn M, Rutten-van Mölken MPMH. How to address uncertainty in health economic discrete-event simulation models: an illustration for chronic obstructive pulmonary disease. Med Decis Making. 2020;40(5):619–32.
Philips Z, Bojke L, Sculpher M, Claxton K, Golder S. Good practice guidelines for decision-analytic modelling in health technology assessment: a review and consolidation of quality assessment. Pharmacoeconomics. 2006;24(4):355–71.
Husereau D, Drummond M, Petrou S, Carswell C, Moher D, Greenberg D, et al. Consolidated health economic evaluation reporting standards (CHEERS) statement. Value Health. 2013;16(2):e1-5.
CISNET Consortium. CISNET Publication Support and Modeling Resources. 2013. https://resources.cisnet.cancer.gov/projects/. Accessed Nov 2020.
Diabetes Trials Unit, the Oxford Centre for Diabetes, Endocrinology and Metabolism. UKPDS Risk Engine. 2017. https://www.dtu.ox.ac.uk/riskengine/. Accessed Nov 2020.
Jakeman AJ, Letcher RA, Norton JP. Ten iterative steps in development and evaluation of environmental models. Environ Model Softw. 2006;21(5):602–14.
Voinov A, Jenni K, Gray S, et al. Tools and methods in participatory modeling: selecting the right tool for the job. Environ Model Softw. 2018;109:232–55.
Sampson CJ, Arnold R, Bryan S, Clarke P, Ekins S, Hatswell A, et al. Transparency in decision modelling: what, why, who and how? Pharmacoeconomics. 2019;37(11):1355–69.
Harvard S, Werker GR, Silva DS. Social, ethical, and other value judgments in health economics modelling. Soc Sci Med. 2020;253:112975.
van Voorn GA, Vemer P, Hamerlijnck D, Ramos IC, Teunissen GJ, Al M, et al. The missing stakeholder group: why patients should be involved in health economic modelling. Appl Health Econ Health Policy. 2016;14(2):129–33.
Hannigan A. Public and patient involvement in quantitative health research: a statistical perspective. Health Expect. 2018;21(6):939–43.
Harrington RL, Hanna ML, Oehrlein EM, Camp R, Wheeler R, Cooblall C, et al. Defining patient engagement in research: results of a systematic review and analysis: report of the ISPOR patient-centered special interest group. Value Health. 2020;23(6):677–88.
Harvard S, Werker GR. Health economists on involving patients in modeling: potential benefits, harms, and variables of interest. Pharmacoeconomics. 2021;39(7):823–33.
Squires H, Chilcott J, Akehurst R, et al. A framework for developing the structure of public health economic models. Value Health. 2016;19(5):588–601.
Lloyd-Williams F, Hyseni L, Guzman-Castillo M, et al. Evaluating stakeholder involvement in building a decision support tool for NHS health checks: co-producing the WorkHORSE study. BMC Med Inform Decis Making. 2020;20(1):1–12.
Xie RZ, Malik ED, Linthicum MT, Bright JL. Putting stakeholder engagement at the center of health economic modeling for health technology assessment in the United States. Pharmacoeconomics. 2021;39(6):631–8.
O’Donnell E, Atkinson JA, Freebairn L, Rychetnik L. Participatory simulation modelling to inform public health policy and practice: rethinking the evidence hierarchies. J Public Health Policy. 2017;38(2):203–15.
Büyükkaramikli NC, Rutten-van Mölken MPMH, Severens JL, Al M. TECH-VER: a verification checklist to reduce errors in models and improve their credibility. Pharmacoeconomics. 2019;37(11):1391–408.
Dasbach EJ, Elbasha EH. Verification of decision-analytic models for health economic evaluations: an overview. Pharmacoeconomics. 2017;35(7):673–83.
Wang J, Pouwels X, Ramaekers B, et al. Multi-use disease models: a blueprint for application in support of health care insurance coverage policy and a case study in Diabetes Mellitus. RIVM letter report 2020-0145. Bilthoven: Rijksinstituut voor Volksgezondheid en Milieu RIVM; 2020.
Acknowledgements
The authors thank the audience at the 2019 IHEA workshop ‘Organized Session: From Art to Science—the Future of Model Validation. Four Trends in Health Economic Decision Modeling and Their Implications for Model Validation’ for their discussion. They also express gratitude to Stefan Lhachimi for chairing this workshop. Three anonymous reviewers have substantially commented on the initial version and helped us to better clarify our line of reasoning, which is gratefully acknowledged.
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Funding
This opinion paper originates from an International Health Economic Association (IHEA) workshop presented in Basel 2019. The authors did not receive any funding for writing this paper, however the initial work as presented in the workshop was funded by an unrestricted grant from ZONMW (The Netherlands Organisation for Health Research and Development), grant no. 152002050(1), for (among others) TF, ICR, DH, and GvV.
Conflicts of interest/competing interests
Talitha Feenstra, Isaac Corro-Ramos, Dominique Hamerlijnck, Salah Ghabri, and George van Voorn declare they have no conflicts of interest to report.
Ethics approval
Not applicable.
Consent to participate
Not applicable.
Consent for publication
Not applicable.
Availability of data and material
Not applicable.
Code availability
Not applicable.
Author contributions
Discussions among all authors served to feed the ideas as presented in this paper. TF drafted the outline of the paper, and TF, ICR, SG, and GvV had important contributions in revising the manuscript versions. All authors commented on the final draft and approved the current version. The opinions expressed in this article are those of the authors and do not necessarily represent the views of their institutions.
Rights and permissions
About this article
Cite this article
Feenstra, T., Corro-Ramos, I., Hamerlijnck, D. et al. Four Aspects Affecting Health Economic Decision Models and Their Validation. PharmacoEconomics 40, 241–248 (2022). https://doi.org/10.1007/s40273-021-01110-w
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s40273-021-01110-w