, Volume 35, Issue 7, pp 673–683 | Cite as

Verification of Decision-Analytic Models for Health Economic Evaluations: An Overview

Leading Article


Decision-analytic models for cost-effectiveness analysis are developed in a variety of software packages where the accuracy of the computer code is seldom verified. Although modeling guidelines recommend using state-of-the-art quality assurance and control methods for software engineering to verify models, the fields of pharmacoeconomics and health technology assessment (HTA) have yet to establish and adopt guidance on how to verify health and economic models. The objective of this paper is to introduce to our field the variety of methods the software engineering field uses to verify that software performs as expected. We identify how many of these methods can be incorporated in the development process of decision-analytic models in order to reduce errors and increase transparency. Given the breadth of methods used in software engineering, we recommend a more in-depth initiative to be undertaken (e.g., by an ISPOR-SMDM Task Force) to define the best practices for model verification in our field and to accelerate adoption. Establishing a general guidance for verifying models will benefit the pharmacoeconomics and HTA communities by increasing accuracy of computer programming, transparency, accessibility, sharing, understandability, and trust of models.

Supplementary material

40273_2017_508_MOESM1_ESM.xlsm (41 kb)
Supplementary material 1 (XLSM 40 kb)


  1. 1.
    Fazio L, Rosner A, Drummond M. How do U.S. payers use economic models submitted by life sciences organizations? Value & Outcomes Spotlight MARCH/APRIL 2016.Google Scholar
  2. 2.
    Briggs A, Claxton K, Sculpher M. Decision modeling for health economic evaluation. Oxford: Oxford University Press; 2006.Google Scholar
  3. 3.
    National Institute for Health and Clinical Excellence. Guide to the methods of technology appraisal. London: NICE; 2013. Accessed 15 Mar 2017.
  4. 4.
    Haute Autorite´ de Sante´. E´ laboration de recommandations de bonne pratique. Me´thode ‘‘Recommandations pour la pratique clinique ‘‘.Saint-Denis La Plaine: HAS; 2010.
  5. 5.
    Canadian Agency for Drugs and Technologies in Health. Guidelines for the economic evaluation of health technologies. 3rd ed. Canada: Ottawa; 2006.Google Scholar
  6. 6.
    Pharmaceutical Benefits Advisory Committee (PBAC). Guidelines for preparing submissions to the Pharmaceutical Benefits Advisory Committee, version 4.4. 2013. Accessed 25 Feb 2015.
  7. 7.
    Tappenden P, Chilcott JB. Avoiding and identifying errors and other threats to the credibility of health economic models. Pharmacoeconomics. 2014;32(10):967–79.CrossRefPubMedGoogle Scholar
  8. 8.
    Chilcott J, Tappenden P, Rawdin A, Johnson M, Kaltenthaler E, Paisley S, Papaioannou D, Shippam A. Avoiding and identifying errors in health technology assessment models: qualitative study and methodological review. Health Technol Assess. 14(25):iii–iv, ix–xii,1–107. doi:10.3310/hta14250.
  9. 9.
    Eddy DM, Hollingworth W, Caro JJ, et al. Model transparency and validation: a report of the ISPOR-SMDM modeling good research practices task force-4. Value Health. 2012;15:843–50.CrossRefPubMedGoogle Scholar
  10. 10.
    Roberts M, Russell LB, Paltiel AD, Chambers M, McEwan P, Krahn M. Conceptualizing a model: a report of the ISPOR-SMDM modeling good research practices task force–2. Med Decis Mak. 2012;32(5):678–89.CrossRefGoogle Scholar
  11. 11.
    Neumann PJ, Sanders GD, Russell LB, Siegel JE, Ganiats TG, editors. Cost-effectiveness in health and medicine. 2nd ed. New York: Oxford University Press; 2016.Google Scholar
  12. 12.
    Vemer P, Corro Ramos I, van Voorn GA, Al MJ, Feenstra TL. AdViSHE: a validation-assessment tool of health-economic models for decision makers and model users. Pharmacoeconomics. 2016;34(4):349–61. doi:10.1007/s40273-015-0327-2.CrossRefPubMedGoogle Scholar
  13. 13.
    Krishnamurthy S. The role of model verification in model risk management: a QuantUniversity Whitepaper. Accessed 16 Dec 2016.
  14. 14.
    American Institute of Aeronautics and Astronautics. Guide for the verification and validation of computational fluid dynamics simulations, AIAA-G-077-1998, Reston, VA, 1998.Google Scholar
  15. 15.
    Thacker BH, Doebling SW, Hemez FM, Anderson MC, Pepin JE, Rodriguez EA. Concepts of model verification and validation. Los Alamos National Laboratory. Accessed 12 Dec 2016.
  16. 16.
    Beck RJ, Pauker SG. The Markov process in medical prognosis. Med Decis Mak. 1983;3:419–58.CrossRefGoogle Scholar
  17. 17.
    Butcher JC. Numerical methods for ordinary differential equations. New York: Wiley; 2003.CrossRefGoogle Scholar
  18. 18.
    Fenwick E, Claxton K, Sculpher M. Representing uncertainty: the role of cost-effectiveness acceptability curves. Health Econ. 2001;10:779–87.CrossRefPubMedGoogle Scholar
  19. 19.
    Systems Development Life Cycle – Wikipedia. Accessed 23 Nov 2016.
  20. 20.
    Food and Drug Administration (FDA). General principles of software validation; final guidance for industry and FDA Staff, 2002. Accessed 12 Dec 2016.
  21. 21.
    Menn P, Holle R. Comparing three software tools for implementing Markov models for health economic evaluations. PharmacoEconomics. 2009;27(9):745–53.CrossRefPubMedGoogle Scholar
  22. 22.
    Chhatwal J, Kanwal F, Roberts MS, Dunn MA. Cost-effectiveness and budget impact of hepatitis C virus treatment with sofosbuvir and ledipasvir in the United States. Ann Int Med. 2015;162(6):397–406.CrossRefPubMedPubMedCentralGoogle Scholar
  23. 23.
    R Core Team. R: a language and environment for statistical computing. Vienna, Austria: R Foundation for Statistical Computing; 2015.
  24. 24.
    Frederix GWJ. Development of a framework for cohort simulation in cost-effectiveness analyses using a multistep ordinary differential equation solver algorithm in R. Med Decis Mak. 2013;33(6):780–92.CrossRefGoogle Scholar
  25. 25.
    Hawkins N, Sculpher M, Epstein D. Cost-effectiveness of treatments for chronic disease: using R to incorporate time dependency of treatment response. Med Decis Mak. 2005;25:511–9.CrossRefGoogle Scholar
  26. 26.
    Williams C, Lewsey JD, Briggs AH, Mackay DF. Cost-effectiveness analysis in R using a multi-state modeling survival analysis framework: a tutorial. Med Decis Mak. 2016. doi:10.1177/0272989X16651869.Google Scholar
  27. 27.
    Zarca K, Filipović-Pierucci A. Heemod—Health Economic Evaluation MODelling. Accessed 23 Nov 2016.
  28. 28.
    Briggs A, Claxton K, Sculpher M. Decision modeling for health economic evaluation. Oxford: Oxford University Press; 2006.Google Scholar
  29. 29.
    Chhatwal J, He T, Lopez-Olivo MA. Systematic review of modelling approaches for the cost effectiveness of hepatitis c treatment with direct-acting antivirals. PharmacoEconomics. 2016;34(6):551–67.CrossRefPubMedGoogle Scholar
  30. 30.
    Panko RR. What we don’t know about spreadsheet errors today: The facts, why we don’t believe them, and what we need to do. Presented at EuSpRIG 2015, London, UK, July 9, 2015.’t%20Know%20About%20Spreadsheet%20Errors.pdf. Accessed 12 Dec 2016.
  31. 31.
    Rajalingham K, Chadwick D, Knight B, Edwards D. Quality control in spreadsheets: a software engineering-based approach to spreadsheet development. In: Proceedings of the 33rd Hawaii International Conference on System Sciences—2000. Accessed 12 Dec 2016.
  32. 32.
    Panko RR, Halverson RP Jr. Spreadsheets on trial: a framework for research on spreadsheet risks. In: Proceedings of the twenty-ninth Hawaii international conference on system sciences, Maui, Hawaii, January 1996. Accessed 17 Apr 2017.
  33. 33.
    Rajalingham K, Chadwick D. Integrity control of spreadsheets: organization and tools. In: Jajodia S, List W, McGregor GW, Strous LAM, editors. Integrity and internal control in information systems. Dordrecht: Springer/Kluwer Academic Publishers; 1998. pp. 147–68.CrossRefGoogle Scholar
  34. 34.
    Panko RR. Two corpuses of spreadsheet errors. In: Proceedings of the 33rd Hawaii International Conference on System Sciences—2000. volume 4–volume 4, p 4008.Google Scholar
  35. 35.
    Panko R. Revising the Panko–Halverson taxonomy of spreadsheet errors. J Decis Support Syst. 2010;49(2):235–44.CrossRefGoogle Scholar
  36. 36.
    Siebert U, Alagoz O, Bayoumi AM, et al. State-transition modeling: a report of the ISPOR-SMDM modeling good research practices task force–3. Med Decis Mak. 2012;32:690–700.CrossRefGoogle Scholar
  37. 37.
    Riley T, Goucher A, eds. Beautiful testing—leading professionals reveal how they improve software. O’Reilly Media; 2009.Google Scholar
  38. 38.
    Balci O. Validation, verification, and testing techniques throughout the life cycle of a simulation study. Ann Oper Res. 1995;53:215–20.Google Scholar
  39. 39.
    Pidd M. Computer simulation in management science. 5th ed. New York: Wiley; 2006.Google Scholar
  40. 40.
    Sargent RG. Verification and validation of simulation models. In: Henderson SG, Biller B, Hsieh M-H, Tew JD, Barton RR (eds) Proc. 2007 winter simulation Conf., pp 124–137. Piscataway: Institute of Electrical and Electronic Engineers Inc.Google Scholar
  41. 41.
    Kolawa A, Huizinga D. Automated defect prevention: best practices in software management. New York: Wiley-IEEE Computer Society Press; 2007.Google Scholar
  42. 42.
    Chhatwal J, Jayasuriya S, Elbasha E. Changing cycle lengths in state-transition models challenges and solutions. Med Decis Mak. 2016;36(8):952–64.CrossRefGoogle Scholar
  43. 43.
    Karr JR, Sanghvi JC, Macklin DN, et al. A whole-cell computational model predicts phenotype from genotype. Cell. 2012;150:389–401.CrossRefPubMedPubMedCentralGoogle Scholar
  44. 44.
    Wickham H. testthat: Get started with testing. R J. 2011;3:5–10.Google Scholar
  45. 45.
    Heemod—Health Economic Evaluation MODelling. Accessed 15 Dec 2016.
  46. 46.
    Agruss C, Johnson R. Ad hoc software testing: a perspective on exploration and improvisation. Accessed 19 Dec 2016.
  47. 47.
    Tinkham A, Kaner C. Exploring exploratory testing. Paper presented atthe STAR East 2003 Conference, Orlando, FL, USA, 2003. Accessed 19 Dec 2016.
  48. 48.
    Elbasha EH, Robertson MN, Nwankwo C. The cost-effectiveness of testing for NS5a resistance-associated polymorphisms at baseline in genotype 1a-infected (treatment-naïve and treatment-experienced) subjects treated with all-oral elbasvir/grazoprevir (EBR/GZR) regimens in the United States. Aliment Pharmacol Ther. 2016;. doi:10.1111/apt.13882 [Epub ahead of print].PubMedGoogle Scholar
  49. 49.
    Martin RC. Clean code: a handbook of agile software craftsmanship. Upper Saddle River: Pearson Education Inc.; 2009.Google Scholar
  50. 50.
    Fowler M. TestCoverage. Accessed 25 Nov 2016.
  51. 51.
    Dunlop WC, Mason N, Kenworthy J, Akehurst RL. Benefits, challenges and potential strategies of open source health economic models. Pharmacoeconomics. 2017;35(1):125–8. doi:10.1007/s40273-016-0479-8.CrossRefPubMedGoogle Scholar
  52. 52.
    Sullivan W, Hirst M, Beard S, et al. Economic evaluation in chronic pain: a systematic review and de novo flexible economic model. Eur J Health Econ. 2016;17(6):755–70.CrossRefPubMedGoogle Scholar
  53. 53.
    O’Beirne P. Excel 2013 Spreadsheet Inquire. Systems Modelling Ltd. Accessed 15 Mar 2017.
  54. 54.
    Systems Biology Markup Language (SBML). Accessed 23 Nov 2016.
  55. 55.
    Guazzelli A, Zeller M, Lin W, Williams G. PMML: An open standard for sharing models. R J. 2009;1:60–5.Google Scholar
  56. 56.
    Google’s R Style Guide. Accessed 27 Nov 2016.
  57. 57.
    CADTH ECONOMIC GUIDELINES. Guidelines for the economic evaluation of health technologies: Canada. Accessed 16 Dec 2016.
  58. 58.
    Petros P, Fernando A-E. Decision modelling using R. 38th Annual North American Meeting of the Society for Medical Decision Making. Vancouver, BC, Canada, October 23–October 26, 2016.Google Scholar
  59. 59.
    McConnell S. Code complete—a practical handbook of software construction. Redmond: Microsoft Press; 2004.Google Scholar
  60. 60.
    Beck K, Andres C. Extreme programming explained. Boston: Addison-Wesley; 2005.Google Scholar
  61. 61.
    Beck K. Test driven development: by example. Boston: Addison-Wesley; 2003.Google Scholar
  62. 62.
    Chelimsky D, Astels D, Zachs D, Hellesoy A, Helmkamp B, North D. The RSpec Book. Pragmatic Bookshelf 2010.Google Scholar
  63. 63.
    Hippisley-Cox J, Coupland C. Development and validation of risk prediction equations to estimate future risk of blindness and lower limb amputation in patients with diabetes: cohort study. BMJ. 2015;351:1–11.CrossRefGoogle Scholar
  64. 64.
    Suen S, Goldhaber-Fiebert JD. An efficient, noniterative method of identifying the cost-effectiveness frontier. Med Decis Mak. 2016;36:132–6.CrossRefGoogle Scholar
  65. 65.
    Frederix GWJ, Hossein Haji Ali Afzali HHA, Dasbach EJ, Ward RL. Development and use of disease-specific (reference) models for economic evaluations of health technologies: an overview of key issues and potential solutions. PharmacoEconomics. 2015;33:777–81.CrossRefPubMedGoogle Scholar
  66. 66.
    Filipovic-Pierucci A, Zarca K, Durand-Zaleski I. Markov models for health economic evaluation modelling In R With The Heemod Package. Value Health. 2016;19:A369.CrossRefGoogle Scholar
  67. 67.
    Crispin L, Gregory J. Agile testing: a practical guide for testers and agile teams. Addison-Wesley Signature Series (Cohn) Pearson Publishing; 2008.Google Scholar
  68. 68.
    Khan K. Just peachy: making office software more reliable with fuzz testing in Beautiful Testing. Beautiful testing—leading professionals reveal how they improve software. 2009, O’Reilly Media.Google Scholar
  69. 69.
    Mount Hood 4 Modeling Group. Computer modeling of diabetes and its complications: a report on the Fourth Mount Hood Challenge Meeting. Diabetes Care 2007; 30(6):1638–1646.Google Scholar
  70. 70.
    Johnson KN. Software in use. In: Riley T, Goucher A, editors. Beautiful testing—leading professionals reveal how they improve software. Sebastopol, CA: O’Reilly Media; 2009.Google Scholar
  71. 71.
    Cohn M. User stories applied for agile software development. Boston MA: Addison-Wesley; 2004.Google Scholar
  72. 72.
    QuintilesIMS CORE Diabetes Model. Accessed 26 Mar 2017.

Copyright information

© Springer International Publishing Switzerland 2017

Authors and Affiliations

  1. 1.Merck & Co. Inc.KenilworthUSA
  2. 2.Merck Center for Observational and Real-World Evidence, Merck Research LaboratoriesMerck & Co., Inc., UG1C-60North WalesUSA

Personalised recommendations