Abstract
Modelers seeking to help inform decisions about insurance (public or private) coverage of the cost of pharmaceuticals or other health care interventions face various methodological challenges. In this review, which is not meant to be comprehensive, we cover those that in our experience are most vexing. The biggest challenge is getting decision makers to trust the model. This is a major problem because most models undergo only cursory validation; our field has lacked the motivation, time, and data to properly validate models intended to inform health care decisions. Without documented, adequate validation, there is little basis for decision makers to have confidence that the model’s results are credible and should be used in a health technology appraisal. A fundamental problem for validation is that the models are very artificial and lack sufficient depth to adequately represent the reality they are simulating. Typically, modelers assume that all resources have infinite capacity so any patient needing care receives it immediately; there are no waiting times or queues, contrary to the common experience in actual practice. Moreover, all the patients enter the model simultaneously at time zero rather than over time as happens in actuality; differences between patients are ignored or minimized and structural modeling choices that make little sense (e.g., using states to represent events) are forced by commitment to a technique (and even to specific spreadsheet software!). The resulting structural uncertainty is rarely addressed, because methods are lacking and even probabilistic analysis of parameter uncertainty suffers from weak consideration of correlation and arbitrary distribution choices. Stakeholders must see to it that models are fit for the stated purpose and provide the best possible estimates given available data—the decisions at stake deserve nothing less.
Similar content being viewed by others
References
Caro JJ, Briggs AH, Siebert U, et al. Modeling good research practices—overview: a report of the ISPOR-SMDM modeling good research practices task force-1. Value Health. 2012;15:796–803.
Koplan JP, Schoenbaum SC, Weinstein MC, Fraser DW. Pertussis vaccine—an analysis of benefits, risks and costs. N Engl J Med. 1979;301:906–11.
Hull R, Hirsh J, Jay R, et al. Different intensities of oral anticoagulant therapy in the treatment of proximal-vein thrombosis. N Engl J Med. 1982;307:1676–81.
Lindley DV. Making decisions. London: Wiley; 1985.
Beck JR, Pauker SG. The Markov process in medical prognosis. Med Decis Making. 1983;3:419–58.
Zaric GS. The impact of ignoring population heterogeneity when Markov models are used in cost-effectiveness analysis. Med Decis Making. 2003;23:379–96.
Siebert U, Alagoz O, Bayoumi AM, et al. State-transition modeling: a report of the ISPOR-SMDM Modeling Good Research Practices Task Force-3. Value Health. 2012;15:812–20.
Caro JJ. Pharmacoeconomic analyses using discrete event simulation. Pharmacoeconomics. 2005;23:323–32.
Karnon J, Stahl JE, Brennan A, et al. Modeling using discrete event simulation: a report of the ISPOR-SMDM Modeling Good Research Practices Task Force-4. Value Health. 2012;15:821–7.
Pitman RJ, Fisman D, Zaric GS, et al. Dynamic transmission modeling: a report of the ISPOR-SMDM modeling good research practices task force-5. Value Health. 2012;15:828–34.
Husereau D, Drummond M, Petrou S, et al. Consolidated Health Economic Evaluation Reporting Standards (CHEERS)—explanation and elaboration: a report of the ISPOR health economic evaluations publication guidelines good reporting practices task force. Value Health. 2013;16:231–50.
Eddy DM, Hollingworth W, Caro JJ, et al. Model transparency and validation: a report of the ISPOR-SMDM modeling good research practices task force-4. Value Health. 2012;15:843–50.
Snee RD. Validation of regression models: methods and examples. Technometrics. 1977;19:415–28.
Picard RR, Cook RD. Cross-validation of regression models. JASA. 1984;79:575–83.
Shemilt I, Mugford M, Vale L, Marsh K, Donaldson C. Evidence-based decisions and economics: health care, social welfare, education and criminal justice. Oxford: Wiley-Blackwell; 2010.
Sculpher MJ, Claxton K, Drummond M, McCabe C. Whither trial-based economic evaluation for health care decision making? Health Econ. 2006;15:677–87.
National Institute for Clinical Excellence. Guide to the multiple technology appraisal process; 2009. http://www.nice.org.uk/media/42D/8C/MTAGuideLRFINAL.pdf. Accessed 28 May 2014.
Afzali HHA, Karnon J. Addressing the challenge for well informed and consistent reimbursement decisions: the case for reference models. Pharmacoeconomics. 2011;29:823–5.
Afzali HHA, Karnon J, Merlin T. Improving the accuracy and comparability of model-based economic evaluations of health technologies for reimbursement decisions: a methodological framework for the development of reference models. Med Dec Making. 2013;33:325–32.
Elliott H, Popay J. How are policy makers using evidence? Models of research utilisation and local NHS policy making. J Epidemiol Community Health. 2000;54:461–8.
Eddy DM. Accuracy versus transparency in pharmacoeconomic modelling. Pharmacoeconomics. 2006;24:837–44.
Caro JJ, Eddy DM, Kan H, Kaltz C, Patel B, Eldessouki R, Briggs AH. A modeling study questionnaire to assess study relevance and credibility to inform healthcare decision-making: an ISPOR-AMCP-NPC Good Practice Task Force report. Value Health. 2014;17:174–82.
O’Mahony JF, van Rosmalen J, Zauber AG, van Ballegooijen M. Multicohort models in cost-effectiveness analysis why aggregating estimates over multiple cohorts can hide useful information. Med Dec Making. 2013;33:407–14.
Gooley TA, Leisenring W, Crowley J, Storer BE. Estimation of failure probabilities in the presence of competing risks: new representations of old estimators. Stat Med. 1999;18:695–706.
Eddy DM, Schlessinger L, Kahn R. Clinical outcomes and cost-effectiveness of strategies for managing people at high risk for diabetes. Ann Int Med. 2005;143:251–64.
Caro JJ, Getsios D, Möller J. Regarding probabilistic analysis and computationally expensive models: necessary and required? Value Health. 2007;10:317–8.
Kelton WD, Law AM. Simulation modeling and analysis. Boston: McGraw Hill; 2000.
Briggs AH, Weinstein MC, Fenwick E, et al. Model parameter estimation and uncertainty analysis: a report of the ISPOR-SMDM modeling good research practices task force-6. Value Health. 2012;15:835–42.
Frederix GWJ, van Hasselt JGC, Schellens JHM, Hövels AM, Raaijmakers JAM, Huitema ADR, Severens JL. The impact of structural uncertainty on cost-effectiveness models for adjuvant endocrine breast cancer treatments: the need for disease-specific model standardization and improved guidance. PharmacoEconomics. 2014;32:47–61.
Möller J, Nicklasson L, Murthy A. Cost-effectiveness of novel relapsed-refractory multiple myeloma therapies in Norway: lenalidomide plus dexamethasone vs bortezomib. J Med Econ. 2011;14:690–7.
Habbema JDF, Schechter CB, Cronin KA, et al. Modeling cancer natural history, epidemiology, and control: reflections on the CISNET breast group experience. JNCI Monogr. 2006;36:122–6.
Price MJ, Welton NJ, Briggs AH, Ades AE. Model averaging in the presence of structural uncertainty about treatment effects: influence on treatment decision and expected value of information. Value Health. 2011;14:205–18.
Getsios D, Migliaccio-Walle K, Caro JJ. NICE cost-effectiveness appraisal of cholinesterase inhibitors. Pharmacoeconomics. 2007;25:997–1006.
Law AM. Statistical analysis of simulation output data. Oper Res. 1983;31:983–1029.
Roberts M, Russel L, Paltiel AD, et al. Conceptualizing a model: a report of the ISPOR-SMDM modeling good research practices task force-2. Value Health. 2012;15:804–11.
Acknowledgments
Jaime Caro and Jörgen Möller are both employees of Evidera, a consultancy company that receives contracts to build decision-analytic models, but no outside funding was provided for this work. Both are also authors of an upcoming textbook on discrete event simulation. Together they conceptualized the manuscript, contributed modeling experiences and wrote the paper. Consequently, they share full responsibility for its content. Jaime Caro is the overall guarantor.
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Caro, J.J., Möller, J. Decision-Analytic Models: Current Methodological Challenges. PharmacoEconomics 32, 943–950 (2014). https://doi.org/10.1007/s40273-014-0183-5
Published:
Issue Date:
DOI: https://doi.org/10.1007/s40273-014-0183-5