Skip to main content
SpringerLink
Log in

Verification of Decision-Analytic Models for Health Economic Evaluations: An Overview

  • Leading Article
  • Published:
PharmacoEconomics Aims and scope Submit manuscript

Abstract

Decision-analytic models for cost-effectiveness analysis are developed in a variety of software packages where the accuracy of the computer code is seldom verified. Although modeling guidelines recommend using state-of-the-art quality assurance and control methods for software engineering to verify models, the fields of pharmacoeconomics and health technology assessment (HTA) have yet to establish and adopt guidance on how to verify health and economic models. The objective of this paper is to introduce to our field the variety of methods the software engineering field uses to verify that software performs as expected. We identify how many of these methods can be incorporated in the development process of decision-analytic models in order to reduce errors and increase transparency. Given the breadth of methods used in software engineering, we recommend a more in-depth initiative to be undertaken (e.g., by an ISPOR-SMDM Task Force) to define the best practices for model verification in our field and to accelerate adoption. Establishing a general guidance for verifying models will benefit the pharmacoeconomics and HTA communities by increasing accuracy of computer programming, transparency, accessibility, sharing, understandability, and trust of models.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1

Similar content being viewed by others

References

  1. Fazio L, Rosner A, Drummond M. How do U.S. payers use economic models submitted by life sciences organizations? Value & Outcomes Spotlight MARCH/APRIL 2016.

  2. Briggs A, Claxton K, Sculpher M. Decision modeling for health economic evaluation. Oxford: Oxford University Press; 2006.

    Google Scholar 

  3. National Institute for Health and Clinical Excellence. Guide to the methods of technology appraisal. London: NICE; 2013. https://www.nice.org.uk/process/pmg9/chapter/foreword. Accessed 15 Mar 2017.

  4. Haute Autorite´ de Sante´. E´ laboration de recommandations de bonne pratique. Me´thode ‘‘Recommandations pour la pratique clinique ‘‘.Saint-Denis La Plaine: HAS; 2010. http://www.has-sante.fr/portail/upload/docs/application/pdf/2011-01/guide_methodologique_recommandations_pour_la_pratique_clinique.pdf.

  5. Canadian Agency for Drugs and Technologies in Health. Guidelines for the economic evaluation of health technologies. 3rd ed. Canada: Ottawa; 2006.

    Google Scholar 

  6. Pharmaceutical Benefits Advisory Committee (PBAC). Guidelines for preparing submissions to the Pharmaceutical Benefits Advisory Committee, version 4.4. 2013. http://www.pbac.pbs.gov.au/information/printable-version-of-guidelines.html. Accessed 25 Feb 2015.

  7. Tappenden P, Chilcott JB. Avoiding and identifying errors and other threats to the credibility of health economic models. Pharmacoeconomics. 2014;32(10):967–79.

    Article  PubMed  Google Scholar 

  8. Chilcott J, Tappenden P, Rawdin A, Johnson M, Kaltenthaler E, Paisley S, Papaioannou D, Shippam A. Avoiding and identifying errors in health technology assessment models: qualitative study and methodological review. Health Technol Assess. 14(25):iii–iv, ix–xii,1–107. doi:10.3310/hta14250.

  9. Eddy DM, Hollingworth W, Caro JJ, et al. Model transparency and validation: a report of the ISPOR-SMDM modeling good research practices task force-4. Value Health. 2012;15:843–50.

    Article  PubMed  Google Scholar 

  10. Roberts M, Russell LB, Paltiel AD, Chambers M, McEwan P, Krahn M. Conceptualizing a model: a report of the ISPOR-SMDM modeling good research practices task force–2. Med Decis Mak. 2012;32(5):678–89.

    Article  Google Scholar 

  11. Neumann PJ, Sanders GD, Russell LB, Siegel JE, Ganiats TG, editors. Cost-effectiveness in health and medicine. 2nd ed. New York: Oxford University Press; 2016.

    Google Scholar 

  12. Vemer P, Corro Ramos I, van Voorn GA, Al MJ, Feenstra TL. AdViSHE: a validation-assessment tool of health-economic models for decision makers and model users. Pharmacoeconomics. 2016;34(4):349–61. doi:10.1007/s40273-015-0327-2.

    Article  CAS  PubMed  Google Scholar 

  13. Krishnamurthy S. The role of model verification in model risk management: a QuantUniversity Whitepaper. http://quantuniversity.com/ModelVerificationForMRM.pdf. Accessed 16 Dec 2016.

  14. American Institute of Aeronautics and Astronautics. Guide for the verification and validation of computational fluid dynamics simulations, AIAA-G-077-1998, Reston, VA, 1998.

  15. Thacker BH, Doebling SW, Hemez FM, Anderson MC, Pepin JE, Rodriguez EA. Concepts of model verification and validation. Los Alamos National Laboratory. http://www.ltas-vis.ulg.ac.be/cmsms/uploads/File/LosAlamos_VerificationValidation.pdf. Accessed 12 Dec 2016.

  16. Beck RJ, Pauker SG. The Markov process in medical prognosis. Med Decis Mak. 1983;3:419–58.

    Article  CAS  Google Scholar 

  17. Butcher JC. Numerical methods for ordinary differential equations. New York: Wiley; 2003.

    Book  Google Scholar 

  18. Fenwick E, Claxton K, Sculpher M. Representing uncertainty: the role of cost-effectiveness acceptability curves. Health Econ. 2001;10:779–87.

    Article  CAS  PubMed  Google Scholar 

  19. Systems Development Life Cycle – Wikipedia. https://en.wikipedia.org/wiki/Systems_development_life_cycle. Accessed 23 Nov 2016.

  20. Food and Drug Administration (FDA). General principles of software validation; final guidance for industry and FDA Staff, 2002. http://www.fda.gov/MedicalDevices/DeviceRegulationandGuidance/GuidanceDocuments/ucm085281.htm. Accessed 12 Dec 2016.

  21. Menn P, Holle R. Comparing three software tools for implementing Markov models for health economic evaluations. PharmacoEconomics. 2009;27(9):745–53.

    Article  PubMed  Google Scholar 

  22. Chhatwal J, Kanwal F, Roberts MS, Dunn MA. Cost-effectiveness and budget impact of hepatitis C virus treatment with sofosbuvir and ledipasvir in the United States. Ann Int Med. 2015;162(6):397–406.

    Article  PubMed  PubMed Central  Google Scholar 

  23. R Core Team. R: a language and environment for statistical computing. Vienna, Austria: R Foundation for Statistical Computing; 2015. http://www.R-project.org/.

  24. Frederix GWJ. Development of a framework for cohort simulation in cost-effectiveness analyses using a multistep ordinary differential equation solver algorithm in R. Med Decis Mak. 2013;33(6):780–92.

    Article  Google Scholar 

  25. Hawkins N, Sculpher M, Epstein D. Cost-effectiveness of treatments for chronic disease: using R to incorporate time dependency of treatment response. Med Decis Mak. 2005;25:511–9.

    Article  Google Scholar 

  26. Williams C, Lewsey JD, Briggs AH, Mackay DF. Cost-effectiveness analysis in R using a multi-state modeling survival analysis framework: a tutorial. Med Decis Mak. 2016. doi:10.1177/0272989X16651869.

    Google Scholar 

  27. Zarca K, Filipović-Pierucci A. Heemod—Health Economic Evaluation MODelling. https://pierucci.org/heemod/. Accessed 23 Nov 2016.

  28. Briggs A, Claxton K, Sculpher M. Decision modeling for health economic evaluation. Oxford: Oxford University Press; 2006.

    Google Scholar 

  29. Chhatwal J, He T, Lopez-Olivo MA. Systematic review of modelling approaches for the cost effectiveness of hepatitis c treatment with direct-acting antivirals. PharmacoEconomics. 2016;34(6):551–67.

    Article  PubMed  Google Scholar 

  30. Panko RR. What we don’t know about spreadsheet errors today: The facts, why we don’t believe them, and what we need to do. Presented at EuSpRIG 2015, London, UK, July 9, 2015. http://www.eusprig.org/presentations/Presented%20EuSpRIG%202015%20What%20We%20Don’t%20Know%20About%20Spreadsheet%20Errors.pdf. Accessed 12 Dec 2016.

  31. Rajalingham K, Chadwick D, Knight B, Edwards D. Quality control in spreadsheets: a software engineering-based approach to spreadsheet development. In: Proceedings of the 33rd Hawaii International Conference on System Sciences—2000. http://panko.shidler.hawaii.edu/SSR/HICSS33/HICSS33-Rajal-SoftEngr.pdf. Accessed 12 Dec 2016.

  32. Panko RR, Halverson RP Jr. Spreadsheets on trial: a framework for research on spreadsheet risks. In: Proceedings of the twenty-ninth Hawaii international conference on system sciences, Maui, Hawaii, January 1996. https://pdfs.semanticscholar.org/3f0e/ca06b6af5c0705e22861267bf0dca02e88ff.pdf. Accessed 17 Apr 2017.

  33. Rajalingham K, Chadwick D. Integrity control of spreadsheets: organization and tools. In: Jajodia S, List W, McGregor GW, Strous LAM, editors. Integrity and internal control in information systems. Dordrecht: Springer/Kluwer Academic Publishers; 1998. pp. 147–68.

    Chapter  Google Scholar 

  34. Panko RR. Two corpuses of spreadsheet errors. In: Proceedings of the 33rd Hawaii International Conference on System Sciences—2000. volume 4–volume 4, p 4008.

  35. Panko R. Revising the Panko–Halverson taxonomy of spreadsheet errors. J Decis Support Syst. 2010;49(2):235–44.

    Article  Google Scholar 

  36. Siebert U, Alagoz O, Bayoumi AM, et al. State-transition modeling: a report of the ISPOR-SMDM modeling good research practices task force–3. Med Decis Mak. 2012;32:690–700.

    Article  Google Scholar 

  37. Riley T, Goucher A, eds. Beautiful testing—leading professionals reveal how they improve software. O’Reilly Media; 2009.

  38. Balci O. Validation, verification, and testing techniques throughout the life cycle of a simulation study. Ann Oper Res. 1995;53:215–20.

    Google Scholar 

  39. Pidd M. Computer simulation in management science. 5th ed. New York: Wiley; 2006.

    Google Scholar 

  40. Sargent RG. Verification and validation of simulation models. In: Henderson SG, Biller B, Hsieh M-H, Tew JD, Barton RR (eds) Proc. 2007 winter simulation Conf., pp 124–137. Piscataway: Institute of Electrical and Electronic Engineers Inc.

  41. Kolawa A, Huizinga D. Automated defect prevention: best practices in software management. New York: Wiley-IEEE Computer Society Press; 2007.

    Google Scholar 

  42. Chhatwal J, Jayasuriya S, Elbasha E. Changing cycle lengths in state-transition models challenges and solutions. Med Decis Mak. 2016;36(8):952–64.

    Article  Google Scholar 

  43. Karr JR, Sanghvi JC, Macklin DN, et al. A whole-cell computational model predicts phenotype from genotype. Cell. 2012;150:389–401.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  44. Wickham H. testthat: Get started with testing. R J. 2011;3:5–10.

    Google Scholar 

  45. Heemod—Health Economic Evaluation MODelling. https://github.com/pierucci/heemod/. Accessed 15 Dec 2016.

  46. Agruss C, Johnson R. Ad hoc software testing: a perspective on exploration and improvisation. http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.93.2070&rep=rep1&type=pdf. Accessed 19 Dec 2016.

  47. Tinkham A, Kaner C. Exploring exploratory testing. Paper presented atthe STAR East 2003 Conference, Orlando, FL, USA, 2003. http://www.testingeducation.org/articles/exploring_exploratory_testing_star_east_2003_paper.pdf). Accessed 19 Dec 2016.

  48. Elbasha EH, Robertson MN, Nwankwo C. The cost-effectiveness of testing for NS5a resistance-associated polymorphisms at baseline in genotype 1a-infected (treatment-naïve and treatment-experienced) subjects treated with all-oral elbasvir/grazoprevir (EBR/GZR) regimens in the United States. Aliment Pharmacol Ther. 2016;. doi:10.1111/apt.13882 [Epub ahead of print].

    PubMed  Google Scholar 

  49. Martin RC. Clean code: a handbook of agile software craftsmanship. Upper Saddle River: Pearson Education Inc.; 2009.

    Google Scholar 

  50. Fowler M. TestCoverage. http://martinfowler.com/bliki/TestCoverage.html. Accessed 25 Nov 2016.

  51. Dunlop WC, Mason N, Kenworthy J, Akehurst RL. Benefits, challenges and potential strategies of open source health economic models. Pharmacoeconomics. 2017;35(1):125–8. doi:10.1007/s40273-016-0479-8.

    Article  PubMed  Google Scholar 

  52. Sullivan W, Hirst M, Beard S, et al. Economic evaluation in chronic pain: a systematic review and de novo flexible economic model. Eur J Health Econ. 2016;17(6):755–70.

    Article  CAS  PubMed  Google Scholar 

  53. O’Beirne P. Excel 2013 Spreadsheet Inquire. Systems Modelling Ltd. http://www.sysmod.com/excel-2013-inquire-add-in-review.pdf. Accessed 15 Mar 2017.

  54. Systems Biology Markup Language (SBML). http://sbml.org/Main_Page. Accessed 23 Nov 2016.

  55. Guazzelli A, Zeller M, Lin W, Williams G. PMML: An open standard for sharing models. R J. 2009;1:60–5.

    Google Scholar 

  56. Google’s R Style Guide. https://google.github.io/styleguide/Rguide.xml. Accessed 27 Nov 2016.

  57. CADTH ECONOMIC GUIDELINES. Guidelines for the economic evaluation of health technologies: Canada. https://www.cadth.ca/sites/default/files/pdf/draft_CADTH_Economic_Guidelines.pdf. Accessed 16 Dec 2016.

  58. Petros P, Fernando A-E. Decision modelling using R. 38th Annual North American Meeting of the Society for Medical Decision Making. Vancouver, BC, Canada, October 23–October 26, 2016.

  59. McConnell S. Code complete—a practical handbook of software construction. Redmond: Microsoft Press; 2004.

    Google Scholar 

  60. Beck K, Andres C. Extreme programming explained. Boston: Addison-Wesley; 2005.

    Google Scholar 

  61. Beck K. Test driven development: by example. Boston: Addison-Wesley; 2003.

    Google Scholar 

  62. Chelimsky D, Astels D, Zachs D, Hellesoy A, Helmkamp B, North D. The RSpec Book. Pragmatic Bookshelf 2010.

  63. Hippisley-Cox J, Coupland C. Development and validation of risk prediction equations to estimate future risk of blindness and lower limb amputation in patients with diabetes: cohort study. BMJ. 2015;351:1–11.

    Article  Google Scholar 

  64. Suen S, Goldhaber-Fiebert JD. An efficient, noniterative method of identifying the cost-effectiveness frontier. Med Decis Mak. 2016;36:132–6.

    Article  Google Scholar 

  65. Frederix GWJ, Hossein Haji Ali Afzali HHA, Dasbach EJ, Ward RL. Development and use of disease-specific (reference) models for economic evaluations of health technologies: an overview of key issues and potential solutions. PharmacoEconomics. 2015;33:777–81.

    Article  PubMed  Google Scholar 

  66. Filipovic-Pierucci A, Zarca K, Durand-Zaleski I. Markov models for health economic evaluation modelling In R With The Heemod Package. Value Health. 2016;19:A369.

    Article  Google Scholar 

  67. Crispin L, Gregory J. Agile testing: a practical guide for testers and agile teams. Addison-Wesley Signature Series (Cohn) Pearson Publishing; 2008.

  68. Khan K. Just peachy: making office software more reliable with fuzz testing in Beautiful Testing. Beautiful testing—leading professionals reveal how they improve software. 2009, O’Reilly Media.

  69. Mount Hood 4 Modeling Group. Computer modeling of diabetes and its complications: a report on the Fourth Mount Hood Challenge Meeting. Diabetes Care 2007; 30(6):1638–1646.

  70. Johnson KN. Software in use. In: Riley T, Goucher A, editors. Beautiful testing—leading professionals reveal how they improve software. Sebastopol, CA: O’Reilly Media; 2009.

    Google Scholar 

  71. Cohn M. User stories applied for agile software development. Boston MA: Addison-Wesley; 2004.

    Google Scholar 

  72. QuintilesIMS CORE Diabetes Model. http://www.core-diabetes.com/. Accessed 26 Mar 2017.

Download references

Author information

Authors and Affiliations

  1. Merck & Co. Inc., Kenilworth, NJ, USA

    Erik J. Dasbach & Elamin H. Elbasha

  2. Merck Center for Observational and Real-World Evidence, Merck Research Laboratories, Merck & Co., Inc., UG1C-60, PO Box 1000, North Wales, PA, 19454-1099, USA

    Erik J. Dasbach & Elamin H. Elbasha

Authors
  1. Erik J. Dasbach
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Elamin H. Elbasha
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

ED and EE were responsible for writing the manuscript. Both authors read, edited, and approved the final manuscript. EE is the overall guarantor for the content.

Corresponding author

Correspondence to Elamin H. Elbasha.

Ethics declarations

Conflicts of interest

This work was not supported by any research funding. Elamin Elbasha has no conflicts of interest related to the scope of this work to declare. Erik Dasbach has no conflicts of interest related to the scope of this work to declare.

Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary material 1 (XLSM 40 kb)

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Dasbach, E.J., Elbasha, E.H. Verification of Decision-Analytic Models for Health Economic Evaluations: An Overview. PharmacoEconomics 35, 673–683 (2017). https://doi.org/10.1007/s40273-017-0508-2

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s40273-017-0508-2

Keywords

Search

Navigation