Skip to main content
SpringerLink
Log in

Handling Uncertainty in Cost-Effectiveness Models

  • Consensus Conference Papers
  • Published:
PharmacoEconomics Aims and scope Submit manuscript

Abstract

The use of modelling in economic evaluation is widespread, and it most often involves synthesising data from a number of sources. However, even when economic evaluations are conducted alongside clinical trials, some form of modelling is usually essential. The aim of this article is to review the handling of uncertainty in the cost-effectiveness results that are generated by the use of decision-analytic-type modelling. The modelling process is split into a number of stages: (i) a set of methods to be employed in a study are defined, which should include a ‘reference case’ of agreed methods to enhance the comparability of results; (ii) the clinical and demographic characteristics of the patients the model relates to should be specified as carefully as in any experimental study; and (iii) the data requirements of the model should be estimated using the principles of Bayesian statistics, such that prior distributions are specified for unknown model parameters. Monte Carlo simulation can then be employed to sample from these prior distributions to obtain a distribution of the cost effectiveness of the intervention. Such probabilistic analyses are related to parameter uncertainty. In addition, modelling uncertainty is likely to add a further layer of uncertainty to the results of particular analyses.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Table I
Table II
Table III
Table IV
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10

Similar content being viewed by others

Notes

  1. 1 Note that in this article, cost effectiveness is used generically to describe results of economic evaluations that are commonly compared in league tables (i.e. those reporting results in terms of cost per life-year and cost per qualityadjusted life-year. Studies reporting results in so-called ‘natural’ units, such as cases detected or ‘symptom-free days avoided’, are also known as cost-effectiveness studies. However, since these natural units cannot be compared between different disease areas, the usefulness of these studies for allocating resources in the health sector is limited.

  2. For UK analysts, this issue has become all the more important recently because of the updated HM Treasury guidelines,[16]which the Department of Health has interpreted as recommending differential discounting for health outcomes in cost-effectiveness analysis (by removing the component of discounting assumed by the Treasury to relate to the combination of annual growth of income and the marginal utility of income).

  3. These costs are often referred to as indirect costs; however, this term is avoided in this article since Drummond and colleagues24 have argued that this terminology can cause confusion through the use of the same term in accountancy to mean overhead costs.

  4. Another margin that might be identified relates to ‘returns to scale’. Standard economic textbooks emphasise falling costs in the presence of returns to scale. However, in terms of the exposition given here, the issue of returns to scale is considered to be a technical efficiency problem. Similarly, where capital equipment results in a discontinuous cost function, it is assumed that the location of such equipment is organised to maximise throughput and therefore minimise overall cost.

  5. In any case, these rates are taken from large national studies and their variation can be assumed to be trivial compared with the other model parameters.

  6. Note that the same individual simulation approach could be used for decision tree-type models, since a given individual can only pass down 1 branch of the tree at a given chance node.

  7. One might be tempted to calculate the individual-level cost-effectiveness ratios and average across the patients. Stinnett and Paltiel36 examined the implications of calculating ‘mean ratios’ versus the ‘ratio of means’ and demonstrated that the analysis of ‘mean ratios’ is inappropriate.

  8. Of course, at the design stage of a clinical trial the sample size for an evaluation is under the control of the design team, and is subject to cost and other logistical constraints. However, it is clear that in a Monte Carlo simulation situation there are no such constraints and the effective sample size (the number of simulations) can be set at the analysis stage.

  9. Note that this causes problems for the dedicated decision analysis software used to generate the individual simulation results.49 Although it is possible to run only a first-order analysis, the second-order uncertainty can only be included as an adjunct to the first-order uncertainty. This problem should be rectified in future releases of the software.

  10. Uncertainty intervals are employed here as a generic term rather than employing the frequentist ‘confidence interval’ or the Bayesian equivalent ‘credible interval’.

  11. The intervals presented in the illustrative example are valid as all the simulated resultswere in the positive quadrant of the cost-effectiveness plane.

  12. It is worth noting that this interpretation of cost-effectiveness acceptability curves as showing the probability that an intervention is cost effective (probability of the hypothesis given the data) necessarily requires a Bayesian interpretation.57

References

  1. Posnett J, Jan S. Indirect cost in economic evaluation: the opportunity cost of unpaid inputs. Health Econ 1996; 5 (1): 13–23

    Article  PubMed  CAS  Google Scholar 

  2. Buxton MJ, Drummond MF, van Hout BA, et al. Modelling in economic evaluation: an unavoidable fact of life. Health Econ 1997; 6: 217–27

    Article  PubMed  CAS  Google Scholar 

  3. Association of British Pharmaceutical Industries. Pharmaceutical industry and Department of Health agree guidelines for the economic analysis of medicines [press release]. London: Association of British Pharmaceutical Industries, 1994

    Google Scholar 

  4. Australian Commonwealth Department of Human Services and Health. Guidelines for the pharmaceutical industry on preparation of submissions to the Pharmaceutical Benefits Advisory Committee: including major submissions involving economic analysis. Canberra: Australian Government Publishing Service, 1995

    Google Scholar 

  5. Drummond MF, Jefferson TO. Guidelines for authors and peer reviewers of economic submissions to the BMJ. BMJ 1996; 313: 275–83

    Article  PubMed  CAS  Google Scholar 

  6. Canadian Coordinating Office for Health Technology Assessment. Guidelines for the economic evaluation of pharmaceuticals: Canada. 2nd ed. Ottawa: Canadian Coordinating Office for Health Technology Assessment (CCOHTA), 1997

  7. Manning WG, Fryback DG, Weinstein MC. Reflecting uncertainty in cost-effectiveness analysis. In: Gold MR, Siegel JE, Russell LB, et al., editors. Cost-effectiveness in health and medicine. New York (NY): Oxford University Press, 1996: 247–75

  8. Drummond M, Brandt A, Luce B, et al. Standardizing methodologies for economic evaluation in health care: practice, problems, and potential. Int J Technol Assess Health Care 1993; 9 (1): 26–36

    Article  PubMed  CAS  Google Scholar 

  9. Lipscomb J. Time preference for health in cost-effectiveness analysis. Med Care 1989; 27 (3 Suppl.): S233–53

    Article  PubMed  CAS  Google Scholar 

  10. Parsonage M, Neuburger H. Discounting and health benefits. Health Econ 1992; 1 (1): 71–6

    Article  PubMed  CAS  Google Scholar 

  11. Cairns J. Discounting and health benefits: another perspective [comment]. Health Econ 1992; 1 (1): 76–9

    Article  PubMed  CAS  Google Scholar 

  12. Coyle D, Tolley K. Discounting of health benefits in the pharmacoeconomic analysis of drug therapies. Pharmacoeconomics 1992; 2 (2): 153–62

    Article  PubMed  CAS  Google Scholar 

  13. Katz DA, Welch HG. Discounting in cost-effectiveness analysis of healthcare programmes. Pharmacoeconomics 1993; 3 (4): 276–85

    Article  PubMed  CAS  Google Scholar 

  14. Lipscomb J. The proper role for discounting: search in progress. Med Care 1996; 34 (12 Suppl.): DS119-23

    Google Scholar 

  15. Lipscomb J, Weinstein M, Torrance, et al. Time preference. In: Gold MR, Siegel JE, Russell LB, et al., editors. Cost-effectiveness in health and medicine. New York (NY): Oxford University Press, 1996: 214–46

  16. HM Treasury. Appraisal and evaluation in central government. London: H.M. Stationery Office, 1997

  17. Donaldson C. Willingness to pay for publicly-provided goods: a possible measure of benefit. J Health Econ 1990; 9: 103–18

    Article  PubMed  CAS  Google Scholar 

  18. Llewellyn TH, Sutherland HJ, Tibshirani R, et al. The measurement of patients’ values in medicine. Med Decis Making 1982; 2 (4): 449–62

    Article  Google Scholar 

  19. Mehrez A, Gafni A. Quality-adjusted life years, utility theory, and healthy-years equivalents. Med Decis Making 1989; 9: 142–9

    Article  PubMed  CAS  Google Scholar 

  20. Torrance GW. Measurement of health state utilities for economic appraisal: a review. J Health Econ 1986; 5: 1–30

    Article  PubMed  CAS  Google Scholar 

  21. Conroy RM, O’Brien E, O’Malley K, et al. Measurement error in the Hawksley random zero sphygmomanometer: what damage has been done and what can we learn? BMJ 1993; 306 (6888): 1319–22

    Article  PubMed  CAS  Google Scholar 

  22. Gold MR, Siegel JE, Russell LB, et al. Cost-effectiveness in health and medicine. New York (NY): Oxford University Press, 1996

    Google Scholar 

  23. Meltzer David, Accounting for future costs in medical cost-effectiveness analysis, J. Health Econ 1997; 16 (1): 33–64

    Article  Google Scholar 

  24. Drummond MF, O’Brien B, Stoddart GL, et al. Methods for the economic evaluation of health care programmes. 2nd ed. Oxford: Oxford University Press, 1997

    Google Scholar 

  25. Koopmanschap MA, Rutten FFH. Indirect costs: the consequence of production loss or increased costs of production. Med Care 1996; 34 (12 Suppl.): DS59–68

    Google Scholar 

  26. Koopmanschap MA, Rutten FFH, Van Ineveld BM, et al. The friction cost method for measuring indirect costs of disease. J Health Econ 1995; 14: 171–89

    Article  PubMed  CAS  Google Scholar 

  27. Russell LB. Is prevention better than cure? Washington, DC: The Brookings Institution, 1986

    Google Scholar 

  28. Briggs AH, Gray AM. Handling uncertainty when performing economic evaluation of health care interventions. Health Technol Assess 1999; 3 (2): 1–134

    PubMed  CAS  Google Scholar 

  29. Anderson MH, Camm AJ. Implications for present and future applications of the implantable cardioverter-defibrillator resulting from the use of a simple model of cost efficacy. Br Heart J 1993; 69 (1): 83–92

    Article  PubMed  CAS  Google Scholar 

  30. Pharoah PD, Hollingworth W. Cost effectiveness of lowering cholesterol concentration with statins in patients with and without pre-existing coronary heart disease: life table method applied to health authority population. BMJ 1996; 312 (7044): 1443–8

    Article  PubMed  CAS  Google Scholar 

  31. Eddy DM. Screening for cervical cancer. Ann Intern Med 1990; 113 (3): 214–26

    PubMed  CAS  Google Scholar 

  32. Johannesson M, Weinstein MC. On the decision rules of cost-effectiveness analysis. J Health Econ 1993; 12: 459–67

    Article  PubMed  CAS  Google Scholar 

  33. Weinstein M, Fineberg HV. Clinical decision analysis. Philadelphia (PA): WB Saunders Company, 1980

    Google Scholar 

  34. Williams A. The economics of coronary artery bypass grafting. BMJ 1985; 291: 326–9

    Article  PubMed  CAS  Google Scholar 

  35. Pauker SG, Kassirer J. Decision analysis. N Engl J Med 1987; 316 (5): 250–8

    Article  PubMed  CAS  Google Scholar 

  36. Stinnett AA, Paltiel AD. Estimating CE ratios under second order uncertainty: the mean ratio versus the ratio of means. Med Decis Making 1997; 17 (4): 483–9

    Article  PubMed  CAS  Google Scholar 

  37. Eddy DM, Hasselblad V, Shachter R. A Bayesian method for synthesizing evidence: the Confidence Profile Method. Int J Technol Assess Health Care 1990; 6 (1): 31–55

    Article  PubMed  CAS  Google Scholar 

  38. Eddy DM, Hasselblad V, Shachter R. An introduction to a Bayesian method for meta-analysis: the confidence profile method. Med Decis Making 1990; 10 (1): 15–23

    Article  PubMed  CAS  Google Scholar 

  39. Zhou XH, Melfi CA, Hui SL. Methods for comparison of cost data. Ann Intern Med 1997; 127 (8 Pt 2): 752–6

    PubMed  CAS  Google Scholar 

  40. Briggs A, Gray A. The distribution of health care costs and their statistical analysis for economic evaluation. J Health Serv Res Policy 1998; 3: 233–45

    PubMed  CAS  Google Scholar 

  41. Gelman A, Carlin J, Stern H, et al. Bayesian data analysis. London: Chapman & Hall, 1995

    Google Scholar 

  42. Wennberg J, Gittelsohn A. Variations in medical care in small areas. Sci Am 1982; 4: 120–4

    Article  Google Scholar 

  43. Andersen TF, Mooney G. The challenge of medical practice variations. Basingstoke: Macmillan, 1990

    Google Scholar 

  44. Cleary PD, Greenfield S, Mulley AG, et al. Variations in length of stay and outcomes for six medical and surgical conditions in Massachusetts and California. JAMA 1991; 266 (1): 73–9

    Article  PubMed  CAS  Google Scholar 

  45. Draper D. Assessment and propagation of model uncertainty. J R Stat Soc Br 1995; 57 (1): 45–97

    Google Scholar 

  46. Briggs A, Sculpher M. An introduction to Markov models for economic evaluation. Pharmacoeconomics 1998; 13 (4): 397–409

    Article  PubMed  CAS  Google Scholar 

  47. Sonnenberg FA, Beck JR. Markov models in medical decision making: a practical guide. Med Decis Making 1993; 13: 322–38

    Article  PubMed  CAS  Google Scholar 

  48. SMLTREE [computer program]. 2.9. Hollenberg J, 1989

  49. Decision Analysis by TreeAge (DATA) [computer program]. Tree Age Software Inc. v3.5. Williamstown (MA): Tree Age, 1998

  50. Briggs AH, Fenn P. Confidence intervals or surfaces? Uncertainty on the cost-effectiveness plane. Health Econ 1998: 7 (8): 723–40

    Article  PubMed  CAS  Google Scholar 

  51. Sharples LD, Briggs A, Caine N, et al. A model for analyzing the cost of main clinical events after cardiac transplantation. Transplantation 1996; 62 (5): 615–21

    Article  PubMed  CAS  Google Scholar 

  52. Hunink MG, Bult JR, de Vries J, et al. Uncertainty in decision models analyzing cost-effectiveness: the joint distribution of incremental costs and effectiveness evaluated with a nonparametric bootstrap method. Med Decis Making 1998; 18 (3): 337– 46

    Article  PubMed  CAS  Google Scholar 

  53. Pratt JW, Raiffa H, Schlaifer R. Introduction to statistical decision theory. Cambridge (MA): MIT Press, 1995

    Google Scholar 

  54. Stinnett AA, Mullahy J. The negative side of cost-effectiveness analysis [letter]. JAMA 1997; 277 (24): 1931–2

    PubMed  CAS  Google Scholar 

  55. Stinnett AA, Mullahy J. Net health benefits: a new framework for the analysis of uncertainty in cost-effectiveness analysis. Med Decis Making 1998; 18 (2 Suppl.): S65–80

    Google Scholar 

  56. van Hout BA, Al MJ, Gordon GS, et al. Costs, effects and C/Eratios alongside a clinical trial. Health Econ 1994; 3 (5): 309–19

    Article  PubMed  Google Scholar 

  57. Briggs AH. A Bayesian approach to stochastic cost-effectiveness analysis. Health Econ 1999; 8 (3): 257–61

    Article  PubMed  CAS  Google Scholar 

  58. Doubilet P, Begg CB, Weinstein MC, et al. Probabilistic sensitivity analysis using Monte Carlo simulation: a practical approach. Med Decis Making 1985; 5: 157–77

    Article  PubMed  CAS  Google Scholar 

  59. Critchfield GC, Willard KE, Connelly DP. Probabilistic sensitivity analysis methods for general decision models. Comput Biomed Res 1986; 19: 254–65

    Article  PubMed  CAS  Google Scholar 

  60. Berwick DM, Cretin S, Keeler E. Cholesterol, children, and heart disease: an analysis of alternatives. Pediatrics 1981; 68 (5): 721–30

    PubMed  CAS  Google Scholar 

  61. Hornberger JC. The hemodialysis prescription and cost effectiveness: Renal Physicians Association Working Committee on Clinical Guidelines. J Am Soc Nephrol 1993; 4 (4): 1021–7

    PubMed  CAS  Google Scholar 

  62. Gabriel SE, Campion ME, O’Fallon WM. A cost-utility analysis of misoprostol prophylaxis for rheumatoid arthritis patients receiving nonsteroidal antiinflammatory drugs. Arthritis Rheum 1994; 37 (3): 333–41

    Article  PubMed  CAS  Google Scholar 

  63. Fiscella K, Franks P. Cost-effectiveness of the transdermal nicotine patch as an adjunct to physicians’ smoking cessation counseling. JAMA 1996; 275 (16): 1247–51

    Article  PubMed  CAS  Google Scholar 

  64. Oh PI, Maerov P, Pritchard D, et al. A cost-utility analysis of second-line antibiotics in the treatment of acute otitis media in children. Clin Ther 1996; 18 (1): 160–82

    Article  PubMed  CAS  Google Scholar 

  65. O’Brien BJ, Drummond MF, Labelle RJ, et al. In search of power and significance: issues in the design and analysis of stochastic cost-effectiveness studies in health care. Med Care 1994; 32 (2): 150–63

    Article  PubMed  Google Scholar 

  66. Sheldon TA. Problems of using modelling in the economic evaluation of health care [editorial]. Health Econ 1996; 5 (1): 1–11

    Article  PubMed  CAS  Google Scholar 

  67. Sculpher MJ, Drummond MF, Buxton MJ. The iterative use of economic evaluation as part of the process of health technology assessment. J Health Services Res Policy 1997; 2: 26–30

    CAS  Google Scholar 

  68. Sonnenberg FA, Roberts MS, Tsevat J, et al. Toward a peer review process for medical decision analysis models. Med Care 1994; 32 (7 Suppl.): JS52–64

    Google Scholar 

  69. Claxton K, Posnett J. An economic approach to clinical trial design and research priority-setting. Health Econ 1996; 5 (6): 513–24

    Article  PubMed  CAS  Google Scholar 

  70. Felli JC, Hazen GB. Sensitivity analysis and the expected value of perfect information. Med Decis Making 1998; 18 (1): 95–109

    Article  PubMed  CAS  Google Scholar 

  71. Fenwick E, Claxton K, Sculpher M, et al. Improving the efficiency and relevance of health technology assessment: the role of decision analytic modelling. Health Economists’ Study Group Conference; 1999 Jan 6–8; Birmingham

Download references

Acknowledgements

I am grateful to Dr Alastair Gray for comments and suggestions on earlier versions of this work, the participants of the Expert Workshop on ‘Validating Cost-Effectiveness Models’, 22 to 23 April 1999, held in Sheffield, England, and particularly to Dr Dennis Fryback for his insightful comments on a previous draft of this article, and finally to 2 anonymous referees for their comments. Of course, the responsibility for errors and inaccuracies in this article is all my own.

Author information

Authors and Affiliations

  1. Health Economics Research Centre, University of Oxford, Institute of Health Sciences, Headington, Oxford, OX3 7LF, England

    Andrew H. Briggs

Authors
  1. Andrew H. Briggs
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Andrew H. Briggs.

Appendix

Appendix

The illustrative model used in this paper will be made available for download from http://www.ihs.ox.ac.uk/herc/.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Briggs, A.H. Handling Uncertainty in Cost-Effectiveness Models. Pharmacoeconomics 17, 479–500 (2000). https://doi.org/10.2165/00019053-200017050-00006

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.2165/00019053-200017050-00006

Keywords

Search

Navigation