Decision analytic models play an increasingly important role in the economic evaluation of health technologies. Given uncertainties around the assumptions used to develop such models, several guidelines have been published to identify and assess ‘best practice’ in the model development process, including general modelling approach (e.g., time horizon), model structure, input data and model performance evaluation. This paper focuses on model performance evaluation. In the absence of a sufficient level of detail around model performance evaluation, concerns regarding the accuracy of model outputs, and hence the credibility of such models, are frequently raised. Following presentation of its components, a review of the application and reporting of model performance evaluation is presented. Taking cardiovascular disease as an illustrative example, the review investigates the use of face validity, internal validity, external validity, and cross model validity. As a part of the performance evaluation process, model calibration is also discussed and its use in applied studies investigated. The review found that the application and reporting of model performance evaluation across 81 studies of treatment for cardiovascular disease was variable. Cross-model validation was reported in 55 % of the reviewed studies, though the level of detail provided varied considerably. We found that very few studies documented other types of validity, and only 6 % of the reviewed articles reported a calibration process. Considering the above findings, we propose a comprehensive model performance evaluation framework (checklist), informed by a review of best-practice guidelines. This framework provides a basis for more accurate and consistent documentation of model performance evaluation. This will improve the peer review process and the comparability of modelling studies. Recognising the fundamental role of decision analytic models in informing public funding decisions, the proposed framework should usefully inform guidelines for preparing submissions to reimbursement bodies.
Calibration Process Decision Analytic Model Calibration Target Pharmaceutical Benefit Advisory Committee Model Performance Evaluation
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.
This is a preview of subscription content, log in to check access.
No sources of funding were used in the preparation of this manuscript. The authors have no conflicts of interest that are relevant to the content of this article.
HH abstracted data, undertook the analysis, and led the drafting of the manuscript. JG conducted the literature search, developed the database, and contributed to revising the manuscript. JK abstracted data and contributed to drafting and revising the manuscript. HH is the guarantor for the overall content of the manuscript.
Sculpher M, Fenwick E, Claxton K. Assessing quality in decision analytic cost-effectiveness models: a suggested framework and example of application. Pharmacoeconomics. 2000;17:461–77.PubMedCrossRefGoogle Scholar
Philips Z, Ginnelly L, Sculpher M, et al. Review of guidelines for good practice in decision-analytic modelling in health technology assessment. Health Technol Assess. 2004;8:1–158.PubMedGoogle Scholar
Haji Ali Afzali H, Karnon J, Gray J. A critical review of model-based economic studies of depression: modelling techniques, model structure and data sources. Pharmacoeconomics. 2012;30:461–82.PubMedCrossRefGoogle Scholar
Tarride J, Hopkins R, Blackhouse G, et al. A review of methods used in long-term cost-effectiveness models of diabetes mellitus treatment. Pharmacoeconomics. 2010;28:255–77.PubMedCrossRefGoogle Scholar
Sendi PP, Craig BA, Pfluger D, et al. Systematic validation of disease models for pharmacoeconomic evaluations. J Eval Clin Pract. 1999;5:283–95.PubMedCrossRefGoogle Scholar
Dams J, Bornschein B, Reese JP, et al. Modelling the cost effectiveness of treatments for Parkinson’s disease: a methodological review. Pharmacoeconomics. 2011;29:1025–49.PubMedCrossRefGoogle Scholar
Bolin K. Economic evaluation of smoking-cessation therapies: a critical and systematic review of simulation models. Pharmacoeconomics. 2012;30:551–64.PubMedCrossRefGoogle Scholar
Eddy DM, Hollingworth W, Caro JJ, et al. Model transparency and validation: a report of the ISPOR-SMDM modelling good research practices Task Force. Med Decis Making. 2012;32:733–43.PubMedCrossRefGoogle Scholar
Weinstein MC, O’Brien B, Hornberger J, et al. Principles of good practice for decision analytic modelling in health-care evaluation: report of the ISPOR Task Force on Good Research Practices—modelling studies. Value Health. 2003;6:9–17.PubMedCrossRefGoogle Scholar
Kim LG, Thompson SG. Uncertainty and validation of health economic decision models. Health Econ. 2010;19:43–55.PubMedGoogle Scholar
The Mount Hood 4 Modelling Group. Computer modelling of diabetes and its complications: a report on the fourth Mount Hood Challenge Meeting. Diabetes Care. 2007;30:1638–46.CrossRefGoogle Scholar
Gray AM, Clarke PM, Wolstenholme JL, et al. Applied methods of cost-effectiveness analysis in health care. New York: Oxford University Press; 2011.Google Scholar
Weinstein MC. Recent developments in decision-analytic modelling for economic evaluation. Pharmacoeconomics. 2006;24:1043–53.PubMedCrossRefGoogle Scholar
Karnon J, Vanni T. Calibrating models in economic evaluation: a comparison of alternative measures of goodness of fit, parameter search strategies and convergence criteria. Pharmacoeconomics. 2011;29:51–62.PubMedCrossRefGoogle Scholar
Vanni T, Karnon J, Madan J, et al. Calibrating models in economic evaluation: a seven-step approach. Pharamacoeconomics. 2011;29:35–49.CrossRefGoogle Scholar
Grover SA, Coupal L, Lowensteyn I. Estimating the cost effectiveness of ramipril used for specific clinical indications: comparing the outcomes in four clinical trials with a common economic model. Am J Cardiovasc Drugs. 2007;7:441–8.PubMedCrossRefGoogle Scholar
Logman JF, Heeg BM, Herlitz J, van Hout BA. Costs and consequences of clopidogrel versus aspirin for secondary prevention of ischaemic events in (high-risk) atherosclerotic patients in Sweden: a lifetime model based on the CAPRIE trial and high-risk CAPRIE subpopulations. Appl Health Econ Health Policy. 2010;8:251–65.PubMedCrossRefGoogle Scholar
Lowensteyn I, Coupal L, Zowall H, Grover SA. The cost-effectiveness of exercise training for the primary and secondary prevention of cardiovascular disease. J Cardiopulm Rehabil. 2000;20:147–55.PubMedCrossRefGoogle Scholar
Gaspoz JM, Coxson PG, Goldman PA, et al. Cost effectiveness of aspirin, clopidogrel, or both for secondary prevention of coronary heart disease. N Engl J Med. 2002;346:1800–6.PubMedCrossRefGoogle Scholar
Latour-Perez J, de-Miguel-Balsa E. Cost effectiveness of fondaparinux in non-ST-elevation acute coronary syndrome. Pharmacoeconomics. 2009;27:585–95.PubMedCrossRefGoogle Scholar
Berger K, Hessel F, Kreuzer J, et al. Clopidogrel versus aspirin in patients with atherothrombosis: CAPRIE-based calculation of cost-effectiveness for Germany. Curr Med Res Opin. 2008;24:267–74.PubMedCrossRefGoogle Scholar
Maniadakis N, Kaitelidou D, Siskou O, et al. Economic evaluation of treatment strategies for patients suffering acute myocardial infarction in Greece. Hellenic J Cardiol. 2005;46:212–21.PubMedGoogle Scholar
Latour-Pérez J, Navarro-Ruiz A, Ridao-López M, et al. Using clopidogrel in non-ST segment elevation acute coronary syndrome patients: a cost–utility analysis in Spain. Value Health. 2004;7:52–60.PubMedCrossRefGoogle Scholar
Yock CA, Boothroyd DB, Owens DK, et al. Cost-effectiveness of bypass surgery versus stenting in patients with multivessel coronary artery disease. Am J Med. 2003;115:382–9.PubMedCrossRefGoogle Scholar
Heeg B, Peters R, Botteman M, et al. Long-term clopidogrel therapy in patients receiving percutaneous coronary intervention. Pharmacoeconomics. 2007;25:769–82.PubMedCrossRefGoogle Scholar
Latour-Pérez J, Balsa EM, Betegob L, et al. Using triple antiplatelet therapy in patients with non-ST elevation acute coronary syndrome managed invasively: a cost-effectiveness analysis. Value Health. 2008;11:853–61.PubMedCrossRefGoogle Scholar
Phillips KA, Shlipak MG, Coxson P, et al. Health and economic benefits of increased beta-blocker use following myocardial infarction. J Am Med Rec Assoc. 2000;6:2748–54.CrossRefGoogle Scholar
Ward MJ, Eckman MH, Schauer DP, et al. Cost-effectiveness of telemetry for hospitalized patients with low-risk chest pain. Acad Emerg Med. 2011;18:279–86.PubMedCrossRefGoogle Scholar
Menown I, Montalescot G, Pal N, et al. Enoxaparin is a cost-effective adjunct to fibrinolytic therapy for ST-elevation myocardial infarction in contemporary practice. Adv Ther. 2010;27:181–91.PubMedCrossRefGoogle Scholar
Schwenkglenks M, Brazier JE, Szucs TD, Fox KA. Cost-effectiveness of bivalirudin versus heparin plus glycoprotein IIb/IIIa inhibitor in the treatment of non-ST-segment elevation acute coronary syndromes. Value Health. 2011;14:24–33.PubMedCrossRefGoogle Scholar
Priest VL, Scuffham PA, Hachamovitch R, et al. Cost-effectiveness of coronary computed tomography and cardiac stress imaging in the emergency department: a decision analytic model comparing diagnostic strategies for chest pain in patients at low risk of acute coronary syndromes. JACC Cardiovasc Imaging. 2011;4:549–56.PubMedCrossRefGoogle Scholar
Ringborg A, Lindgren P, Jonsson B. The cost-effectiveness of dual oral antiplatelet therapy following percutaneous coronary intervention: a Swedish analysis of the CREDO trial. Eur J Health Econ. 2005;6:354–62.PubMedCrossRefGoogle Scholar
Ganz DA, Kuntz KM, Jacobson GA, Avorn J. Cost-effectiveness of 3-hydroxy-3-methylglutaryl coenzyme A reductase inhibitor therapy in older patients with myocardial infarction. Ann Intern Med. 2000;132:780–7.PubMedCrossRefGoogle Scholar
Bravo Vergel Y, Palmer S, Asseburg C, et al. Is primary angioplasty cost effective in the UK? Results of a comprehensive decision analysis. Heart. 2007;93:1238–43.PubMedCrossRefGoogle Scholar
Scottish Medicines Consortium. Guidance to manufacturers for completion of new product assessment form (NPAF) [online]. Available from URL: www.scottishmedicines.org.uk [Accessed 2012 Jul 27].
Haji Ali Afzali H, Karnon J. Addressing the challenge for well informed and consistent reimbursement decisions: the case for reference models. Pharmacoeconomics. 2011;29:823–5.PubMedCrossRefGoogle Scholar
Haji Ali Afzali H, Karnon J, Merlin T. Improving the accuracy and comparability of model-based economic evaluations of health technologies for reimbursement decisions: a methodological framework for the development of reference models. Med Decis Making. doi:10.1177/0272989X12458160.