, Volume 33, Issue 5, pp 435–443 | Cite as

Exploring Structural Uncertainty in Model-Based Economic Evaluations

  • Hossein Haji Ali AfzaliEmail author
  • Jonathan Karnon
Practical Application


Given the inherent uncertainty in estimates produced by decision analytic models, the assessment of uncertainty in model-based evaluations is an essential part of the decision-making process. Although the impact of uncertainty around the choice of model structure and making incorrect structural assumptions on model predictions is noted, relatively little attention has been paid to characterising this type of uncertainty in guidelines developed by national funding bodies such as the Australian Pharmaceutical Benefits Advisory Committee (PBAC). The absence of a detailed description and evaluation of structural uncertainty can add further uncertainty to the decision-making process, with potential impact on the quality of funding decisions. This paper provides a summary of key elements of structural uncertainty describing why it matters and how it could be characterised. Five alternative approaches to characterising structural uncertainty are discussed, including scenario analysis, model selection, model averaging, parameterization and discrepancy. We argue that the potential effect of structural uncertainty on model predictions should be considered in submissions to national funding bodies; however, the characterisation of structural uncertainty is not well defined within the guidelines of these bodies. There has been little consideration of the forms of structural sensitivity analysis that might best inform applied decision-making processes, and empirical research in this area is required.


Health Technology Assessment Deviance Information Criterion Funding Decision Structural Uncertainty Bayesian Information Criterion 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.



Hossein Haji Ali Afzali and Jonathan Karnon conceptualized the manuscript and prepared the final draft. They share full responsibility for its content. Hossein Haji Ali Afzali is the overall guarantor.


No sources of funding were used to prepare this manuscript.

Conflicts of interest

Hossein Haji Ali Afzali has served as a member of the Protocol Advisory Subcommittee (PASC) of the Medical Services Advisory Committee (MSAC) since 2012. Jonathan Karnon has served as a member of the Economics Subcommittee (ESC) of the PBAC since 2009.


  1. 1.
    Caro JJ, Moller J. Decision-analytic models: current methodological challenges. Pharmacoeconomics. 2014;32:943–50.CrossRefPubMedGoogle Scholar
  2. 2.
    Kim L, Thompson SG. Uncertainty and validation of health economic decision models. Health Econ. 2010;19:43–55.PubMedGoogle Scholar
  3. 3.
    Frederix GW, Severens JL, Hovels AM, Raaijmakers JA, Schellens JH. The cloudy crystal ball of cost-effectiveness studies. Value Health. 2013;16:1100–2.CrossRefPubMedGoogle Scholar
  4. 4.
    Briggs AH, Weinstein MC, Fenwick EA, Karnon J, Sculpher MJ, Paltiel AD. Model parameter estimation and uncertainty: a report of the ISPOR-SMDM Modeling Good Research Practices Task Force-6. Value Health. 2012;15:835–42.CrossRefPubMedGoogle Scholar
  5. 5.
    Briggs A. Handling uncertainty in cost-effectiveness models. Pharmacoeconomics. 2000;17:479–500.CrossRefPubMedGoogle Scholar
  6. 6.
    Committee Pharmaceutical Benefits Advisory. Guidelines for preparing submissions to the Pharmaceutical Benefits Advisory Committee. Canberra: Australian Government; 2013.Google Scholar
  7. 7.
    National Institute for Health and Clinical Excellence (NICE). Guide to the methods of technology appraisal 2013. London: NICE; 2013.Google Scholar
  8. 8.
    Canadian Agency for Drugs and Technologies in Health. Guidelines for the economic evaluation of health technologies. Ottawa: Canadian Agency for Drugs and Technologies in Health; 2006.Google Scholar
  9. 9.
    Bojke L, Claxton K, Sculpher M, Palmer S. Characterizing structural uncertainty in decision analytic models: a review and application of methods. Value Health. 2009;12:739–49.CrossRefPubMedGoogle Scholar
  10. 10.
    Brisson M, Edmunds WJ. Impact of model, methodological, and parameter uncertainty in the economic analysis of vaccination programs. Med Decis Mak. 2006;26:434–46.CrossRefGoogle Scholar
  11. 11.
    Kim SY, Goldie SJ, Salomon JA. Exploring model uncertainty in economic evaluation of health interventions: the example of rotavirus vaccination in Vietnam. Med Decis Mak. 2010;30:E1–28.CrossRefGoogle Scholar
  12. 12.
    Jackson CH, Bojke L, Thompson SG, Claxton K, Sharples LD. A framework for addressing structural uncertainty in decision models. Med Decis Mak. 2011;31:662–74.CrossRefGoogle Scholar
  13. 13.
    Frederix GW, van Hasselt JG, Schellens JH, Hovels AM, Raaijmakers JA, Huitema AD, et al. The impact of structural uncertainty on cost-effectiveness models for adjuvant endocrine breast cancer treatments: the need for disease-specific model standardization and improved guidance. Pharmacoeconomics. 2014;32:47–61.CrossRefPubMedGoogle Scholar
  14. 14.
    Bojke L, Claxton K, Palmer S, Sculpher M. Defining and characterising structural uncertainty in decision analytic models. York: Centre for Health Economics, University of York; 2006.Google Scholar
  15. 15.
    Strong M, Oakley JE, Chilcott J. Managing structural uncertainty in health economic decision models: a discrepancy approach. J R Stat Soc. 2012;61:25–45.CrossRefGoogle Scholar
  16. 16.
    Epstein DM, Sculpher MJ, Manca A, Michaels J, Thompson SG, Brown LC, et al. Modelling the long-term cost-effectiveness of endovascular or open repair for abdominal aortic aneurysm. Br J Surg. 2008;95:183–90.CrossRefPubMedGoogle Scholar
  17. 17.
    Jones L, Griffin S, Palmer S, Main C, Orton V, Sculpher M, et al. Clinical effectiveness and cost-effectiveness of clopidogrel and modified-release dipyridamole in the secondary prevention of occlusive vascular events: a systematic review and economic evaluation. Health Technol Assess. 2004;8:1–196.Google Scholar
  18. 18.
    Loveman E, Jones J, Hartwell D, Bird A, Harris P, Welch K, et al. The clinical effectiveness and cost-effectiveness of topotecan for small cell lung cancer: a systematic review and economic evaluation. Health Technol Assess. 2010;14:1–204.CrossRefGoogle Scholar
  19. 19.
    Haji Ali Afzali H, Karnon J. Specification and implementation of decision analytic model structures for the economic evaluation of health care technologies. In: Culyer AJ, editor. Encyclopaedia of health economics. 1st ed. London: Elsevier; 2014: p. 340–7.Google Scholar
  20. 20.
    Burnham KP, Anderson DR. Model selection and multi-model inference: a practical information-theoretic approach. New York: Springer; 2002.Google Scholar
  21. 21.
    Jackson CH, Thompson SG, Sharples LD. Accounting for uncertainty in health economic decision models by using model averaging. J R Stat Soc. 2009;172:383–404.CrossRefGoogle Scholar
  22. 22.
    Draper D. Assessment and propagation of model uncertainty. J R Stat Soc. 1995;57:45–97.Google Scholar
  23. 23.
    Kadane JB, Lazar NA. Methods and criteria for model selection. J Am Stat Assoc. 2004;99:279–90.CrossRefGoogle Scholar
  24. 24.
    Spiegelhalter DJ, Best NG, Carlin BR, van der Linde A. Bayesian measures of model complexity and fit. J R Stat Soc. 2002;64:583–616.CrossRefGoogle Scholar
  25. 25.
    Laud PW, Ibrahim JG. Predictive model slection. J R Stat Soc. 1995;57:247–62.Google Scholar
  26. 26.
    Jackson CH, Sharples LD, Thompson SG. Structural and parameter uncertainty in Bayesian cost-effectiveness models. J R Stat Soc. 2010;59:233–53.CrossRefGoogle Scholar
  27. 27.
    Claxton K. Exploring uncertainty in cost-effectiveness analysis. Pharmacoeconomics. 2008;26:781–98.CrossRefPubMedGoogle Scholar
  28. 28.
    Price MJ, Welton NJ, Briggs AH, Ades AE. Model averaging in the presence of structural uncertainty about treatment effects: influence on treatment decision and expected value of information. Value Health. 2011;14:205–18.CrossRefPubMedGoogle Scholar
  29. 29.
    Singh A, Mishra S, Ruskauff G. Model averaging techniques for quantifying conceptual model uncertainty. Ground Water. 2010;48:701–15.CrossRefPubMedGoogle Scholar
  30. 30.
    Garthwaite PH, Kadane JB, O’Hagan A. Statistical methods for eliciting probability distribution. J Am Stat Assoc. 2005;100:680–700.CrossRefGoogle Scholar
  31. 31.
    Speight PM, Palmer S, Moles DR, Downer MC, Smith DH, Henriksson M, et al. The cost-effectiveness of screening for oral cancer in primary care. Health Technol Assess. 2006;10:1–144 (iii-iv).CrossRefPubMedGoogle Scholar
  32. 32.
    Bernardo M, Smith M. Bayesian theory. Chichester: Wiley; 1994.CrossRefGoogle Scholar
  33. 33.
    Kennedy MC, O’Hagan A. Bayesian calibration of computer models (with discussion). J R Stat Soc. 2001;63:425–64.CrossRefGoogle Scholar
  34. 34.
    Haji Ali Afzali H, Karnon J, Gray J. A proposed model for economic evaluations of major depressive disorder. Eur J Health Econ. 2012;13:501–10.CrossRefPubMedGoogle Scholar
  35. 35.
    Karnon J, Vanni T. Calibratingmodels in economic evaluation: a comparison of alternative measures of goodness-of-fit, parameter search strategies, and convergence criteria. Pharmacoeconomics. 2011;29:51–62.CrossRefPubMedGoogle Scholar
  36. 36.
    Haji Ali Afzali H, Karnon J, Merlin T. Improving the accuracy and comparability of model-based economic evaluations of health technologies for reimbursement decisions: a methodological framework for the development of reference models. Med Decis Mak. 2013;33:325–32.CrossRefGoogle Scholar

Copyright information

© Springer International Publishing Switzerland 2015

Authors and Affiliations

  1. 1.School of Population HealthThe University of AdelaideAdelaideAustralia

Personalised recommendations