Applied Health Economics and Health Policy

, Volume 11, Issue 2, pp 85–93 | Cite as

Model Performance Evaluation (Validation and Calibration) in Model-based Studies of Therapeutic Interventions for Cardiovascular Diseases

A Review and Suggested Reporting Framework
  • Hossein Haji Ali AfzaliEmail author
  • Jodi Gray
  • Jonathan Karnon
Review Article


Decision analytic models play an increasingly important role in the economic evaluation of health technologies. Given uncertainties around the assumptions used to develop such models, several guidelines have been published to identify and assess ‘best practice’ in the model development process, including general modelling approach (e.g., time horizon), model structure, input data and model performance evaluation. This paper focuses on model performance evaluation. In the absence of a sufficient level of detail around model performance evaluation, concerns regarding the accuracy of model outputs, and hence the credibility of such models, are frequently raised. Following presentation of its components, a review of the application and reporting of model performance evaluation is presented. Taking cardiovascular disease as an illustrative example, the review investigates the use of face validity, internal validity, external validity, and cross model validity. As a part of the performance evaluation process, model calibration is also discussed and its use in applied studies investigated. The review found that the application and reporting of model performance evaluation across 81 studies of treatment for cardiovascular disease was variable. Cross-model validation was reported in 55 % of the reviewed studies, though the level of detail provided varied considerably. We found that very few studies documented other types of validity, and only 6 % of the reviewed articles reported a calibration process. Considering the above findings, we propose a comprehensive model performance evaluation framework (checklist), informed by a review of best-practice guidelines. This framework provides a basis for more accurate and consistent documentation of model performance evaluation. This will improve the peer review process and the comparability of modelling studies. Recognising the fundamental role of decision analytic models in informing public funding decisions, the proposed framework should usefully inform guidelines for preparing submissions to reimbursement bodies.


Calibration Process Decision Analytic Model Calibration Target Pharmaceutical Benefit Advisory Committee Model Performance Evaluation 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.



No sources of funding were used in the preparation of this manuscript. The authors have no conflicts of interest that are relevant to the content of this article.

Author contributions

HH abstracted data, undertook the analysis, and led the drafting of the manuscript. JG conducted the literature search, developed the database, and contributed to revising the manuscript. JK abstracted data and contributed to drafting and revising the manuscript. HH is the guarantor for the overall content of the manuscript.

Supplementary material

40258_2013_12_MOESM1_ESM.pdf (16 kb)
Supplementary material 1 (PDF 16 kb)


  1. 1.
    Hall PS, McCabe C, Brown JM, et al. Health economics in drug development: efficient research to inform health care funding decisions. Eur J Cancer Care. 2010;46:2674–80.CrossRefGoogle Scholar
  2. 2.
    Karnon J, Brown J. Selecting a decision model for economic evaluation: a case study and review. Health Care Manag Sci. 1998;1:133–40.PubMedCrossRefGoogle Scholar
  3. 3.
    Buxton MJ, Drummond MF, Van Hout BA, et al. Modeling in economic evaluation: an unavoidable fact of life. Health Econ. 1997;6:217–27.PubMedCrossRefGoogle Scholar
  4. 4.
    Karnon JE, Goyder E, Tappenden P, et al. A review and critique of modelling in prioritising and designing screening programs. Health Technol Assess. 2007;11:1–145.Google Scholar
  5. 5.
    The ISPOR-SMDM Joint Modeling Good Research Practices Task Force [online]. Available from URL: [Accessed 2012 March 26].
  6. 6.
    Sculpher M, Fenwick E, Claxton K. Assessing quality in decision analytic cost-effectiveness models: a suggested framework and example of application. Pharmacoeconomics. 2000;17:461–77.PubMedCrossRefGoogle Scholar
  7. 7.
    Philips Z, Ginnelly L, Sculpher M, et al. Review of guidelines for good practice in decision-analytic modelling in health technology assessment. Health Technol Assess. 2004;8:1–158.PubMedGoogle Scholar
  8. 8.
    McCabe C, Dixon S. Testing the validity of cost-effectiveness model. Pharmacoeconomics. 2000;17:501–13.PubMedCrossRefGoogle Scholar
  9. 9.
    Haji Ali Afzali H, Karnon J, Gray J. A critical review of model-based economic studies of depression: modelling techniques, model structure and data sources. Pharmacoeconomics. 2012;30:461–82.PubMedCrossRefGoogle Scholar
  10. 10.
    Tarride J, Hopkins R, Blackhouse G, et al. A review of methods used in long-term cost-effectiveness models of diabetes mellitus treatment. Pharmacoeconomics. 2010;28:255–77.PubMedCrossRefGoogle Scholar
  11. 11.
    Sendi PP, Craig BA, Pfluger D, et al. Systematic validation of disease models for pharmacoeconomic evaluations. J Eval Clin Pract. 1999;5:283–95.PubMedCrossRefGoogle Scholar
  12. 12.
    Dams J, Bornschein B, Reese JP, et al. Modelling the cost effectiveness of treatments for Parkinson’s disease: a methodological review. Pharmacoeconomics. 2011;29:1025–49.PubMedCrossRefGoogle Scholar
  13. 13.
    Bolin K. Economic evaluation of smoking-cessation therapies: a critical and systematic review of simulation models. Pharmacoeconomics. 2012;30:551–64.PubMedCrossRefGoogle Scholar
  14. 14.
    Ferdinands JM, Mannino DM. Obstructive lung disease models: what is valid? COPD. 2008;5:382–93.PubMedCrossRefGoogle Scholar
  15. 15.
    Stout NK, Knudsen AB, Kong CY, et al. Calibration methods used in cancer simulation models and suggested reporting guidelines. Pharmacoeconomics. 2009;27:533–45.PubMedCentralPubMedCrossRefGoogle Scholar
  16. 16.
    Eddy DM, Hollingworth W, Caro JJ, et al. Model transparency and validation: a report of the ISPOR-SMDM modelling good research practices Task Force. Med Decis Making. 2012;32:733–43.PubMedCrossRefGoogle Scholar
  17. 17.
    Weinstein MC, O’Brien B, Hornberger J, et al. Principles of good practice for decision analytic modelling in health-care evaluation: report of the ISPOR Task Force on Good Research Practices—modelling studies. Value Health. 2003;6:9–17.PubMedCrossRefGoogle Scholar
  18. 18.
    Kim LG, Thompson SG. Uncertainty and validation of health economic decision models. Health Econ. 2010;19:43–55.PubMedGoogle Scholar
  19. 19.
    The Mount Hood 4 Modelling Group. Computer modelling of diabetes and its complications: a report on the fourth Mount Hood Challenge Meeting. Diabetes Care. 2007;30:1638–46.CrossRefGoogle Scholar
  20. 20.
    Gray AM, Clarke PM, Wolstenholme JL, et al. Applied methods of cost-effectiveness analysis in health care. New York: Oxford University Press; 2011.Google Scholar
  21. 21.
    Weinstein MC. Recent developments in decision-analytic modelling for economic evaluation. Pharmacoeconomics. 2006;24:1043–53.PubMedCrossRefGoogle Scholar
  22. 22.
    Karnon J, Vanni T. Calibrating models in economic evaluation: a comparison of alternative measures of goodness of fit, parameter search strategies and convergence criteria. Pharmacoeconomics. 2011;29:51–62.PubMedCrossRefGoogle Scholar
  23. 23.
    Vanni T, Karnon J, Madan J, et al. Calibrating models in economic evaluation: a seven-step approach. Pharamacoeconomics. 2011;29:35–49.CrossRefGoogle Scholar
  24. 24.
    Grover SA, Coupal L, Lowensteyn I. Estimating the cost effectiveness of ramipril used for specific clinical indications: comparing the outcomes in four clinical trials with a common economic model. Am J Cardiovasc Drugs. 2007;7:441–8.PubMedCrossRefGoogle Scholar
  25. 25.
    Logman JF, Heeg BM, Herlitz J, van Hout BA. Costs and consequences of clopidogrel versus aspirin for secondary prevention of ischaemic events in (high-risk) atherosclerotic patients in Sweden: a lifetime model based on the CAPRIE trial and high-risk CAPRIE subpopulations. Appl Health Econ Health Policy. 2010;8:251–65.PubMedCrossRefGoogle Scholar
  26. 26.
    Lowensteyn I, Coupal L, Zowall H, Grover SA. The cost-effectiveness of exercise training for the primary and secondary prevention of cardiovascular disease. J Cardiopulm Rehabil. 2000;20:147–55.PubMedCrossRefGoogle Scholar
  27. 27.
    Gaspoz JM, Coxson PG, Goldman PA, et al. Cost effectiveness of aspirin, clopidogrel, or both for secondary prevention of coronary heart disease. N Engl J Med. 2002;346:1800–6.PubMedCrossRefGoogle Scholar
  28. 28.
    Latour-Perez J, de-Miguel-Balsa E. Cost effectiveness of fondaparinux in non-ST-elevation acute coronary syndrome. Pharmacoeconomics. 2009;27:585–95.PubMedCrossRefGoogle Scholar
  29. 29.
    Berger K, Hessel F, Kreuzer J, et al. Clopidogrel versus aspirin in patients with atherothrombosis: CAPRIE-based calculation of cost-effectiveness for Germany. Curr Med Res Opin. 2008;24:267–74.PubMedCrossRefGoogle Scholar
  30. 30.
    Maniadakis N, Kaitelidou D, Siskou O, et al. Economic evaluation of treatment strategies for patients suffering acute myocardial infarction in Greece. Hellenic J Cardiol. 2005;46:212–21.PubMedGoogle Scholar
  31. 31.
    Latour-Pérez J, Navarro-Ruiz A, Ridao-López M, et al. Using clopidogrel in non-ST segment elevation acute coronary syndrome patients: a cost–utility analysis in Spain. Value Health. 2004;7:52–60.PubMedCrossRefGoogle Scholar
  32. 32.
    Yock CA, Boothroyd DB, Owens DK, et al. Cost-effectiveness of bypass surgery versus stenting in patients with multivessel coronary artery disease. Am J Med. 2003;115:382–9.PubMedCrossRefGoogle Scholar
  33. 33.
    Heeg B, Peters R, Botteman M, et al. Long-term clopidogrel therapy in patients receiving percutaneous coronary intervention. Pharmacoeconomics. 2007;25:769–82.PubMedCrossRefGoogle Scholar
  34. 34.
    Latour-Pérez J, Balsa EM, Betegob L, et al. Using triple antiplatelet therapy in patients with non-ST elevation acute coronary syndrome managed invasively: a cost-effectiveness analysis. Value Health. 2008;11:853–61.PubMedCrossRefGoogle Scholar
  35. 35.
    Phillips KA, Shlipak MG, Coxson P, et al. Health and economic benefits of increased beta-blocker use following myocardial infarction. J Am Med Rec Assoc. 2000;6:2748–54.CrossRefGoogle Scholar
  36. 36.
    Ward MJ, Eckman MH, Schauer DP, et al. Cost-effectiveness of telemetry for hospitalized patients with low-risk chest pain. Acad Emerg Med. 2011;18:279–86.PubMedCrossRefGoogle Scholar
  37. 37.
    Menown I, Montalescot G, Pal N, et al. Enoxaparin is a cost-effective adjunct to fibrinolytic therapy for ST-elevation myocardial infarction in contemporary practice. Adv Ther. 2010;27:181–91.PubMedCrossRefGoogle Scholar
  38. 38.
    Schwenkglenks M, Brazier JE, Szucs TD, Fox KA. Cost-effectiveness of bivalirudin versus heparin plus glycoprotein IIb/IIIa inhibitor in the treatment of non-ST-segment elevation acute coronary syndromes. Value Health. 2011;14:24–33.PubMedCrossRefGoogle Scholar
  39. 39.
    Priest VL, Scuffham PA, Hachamovitch R, et al. Cost-effectiveness of coronary computed tomography and cardiac stress imaging in the emergency department: a decision analytic model comparing diagnostic strategies for chest pain in patients at low risk of acute coronary syndromes. JACC Cardiovasc Imaging. 2011;4:549–56.PubMedCrossRefGoogle Scholar
  40. 40.
    Ringborg A, Lindgren P, Jonsson B. The cost-effectiveness of dual oral antiplatelet therapy following percutaneous coronary intervention: a Swedish analysis of the CREDO trial. Eur J Health Econ. 2005;6:354–62.PubMedCrossRefGoogle Scholar
  41. 41.
    Hirsch G, Homer J, Evans E, Zielinski A. A system dynamics model for planning cardiovascular disease interventions. Am J Public Health. 2010;100:616–22.PubMedCentralPubMedCrossRefGoogle Scholar
  42. 42.
    Ganz DA, Kuntz KM, Jacobson GA, Avorn J. Cost-effectiveness of 3-hydroxy-3-methylglutaryl coenzyme A reductase inhibitor therapy in older patients with myocardial infarction. Ann Intern Med. 2000;132:780–7.PubMedCrossRefGoogle Scholar
  43. 43.
    Bravo Vergel Y, Palmer S, Asseburg C, et al. Is primary angioplasty cost effective in the UK? Results of a comprehensive decision analysis. Heart. 2007;93:1238–43.PubMedCrossRefGoogle Scholar
  44. 44.
    Canadian Agency for Drugs and Technologies in Health. Guidelines for the economic evaluation of health technologies: Canada [online]. Available from URL: [Accessed 2012 Sep 26].
  45. 45.
    Checklist for authors of modelling studies submitted to Pharmacoeconomics [online]. Available from URL: [Accessed 2012 Sep 26].
  46. 46.
    Scottish Medicines Consortium. Guidance to manufacturers for completion of new product assessment form (NPAF) [online]. Available from URL: [Accessed 2012 Jul 27].
  47. 47.
    Haji Ali Afzali H, Karnon J. Addressing the challenge for well informed and consistent reimbursement decisions: the case for reference models. Pharmacoeconomics. 2011;29:823–5.PubMedCrossRefGoogle Scholar
  48. 48.
    Haji Ali Afzali H, Karnon J, Merlin T. Improving the accuracy and comparability of model-based economic evaluations of health technologies for reimbursement decisions: a methodological framework for the development of reference models. Med Decis Making. doi: 10.1177/0272989X12458160.
  49. 49.
    Kopec JA, Finès P, Manuel DG, et al. Validation of population-based disease simulation models: a review of concepts and methods. BMC Public Health. 2010;10:710.PubMedCentralPubMedCrossRefGoogle Scholar
  50. 50.
    McCabe C, Chilcott J, Claxton C, et al. Continuing the multiple sclerosis risk sharing scheme is unjustified. BMJ. 2010;340:1285–8.CrossRefGoogle Scholar

Copyright information

© Springer International Publishing Switzerland 2013

Authors and Affiliations

  • Hossein Haji Ali Afzali
    • 1
    Email author
  • Jodi Gray
    • 1
  • Jonathan Karnon
    • 1
  1. 1.Discipline of Public Health, School of Population HealthUniversity of AdelaideAdelaideAustralia

Personalised recommendations