Probabilistic Sensitivity Analysis in Cost-Effectiveness Models: Determining Model Convergence in Cohort Models

  • Anthony J. Hatswell
  • Ash Bullement
  • Andrew Briggs
  • Mike Paulden
  • Matthew D. Stevenson
Practical Application


Probabilistic sensitivity analysis (PSA) demonstrates the parameter uncertainty in a decision problem. The technique involves sampling parameters from their respective distributions (rather than simply using mean/median parameter values). Guidance in the literature, and from health technology assessment bodies, on the number of simulations that should be performed suggests a ‘sufficient number’, or until ‘convergence’, which is seldom defined. The objective of this tutorial is to describe possible outcomes from PSA, discuss appropriate levels of accuracy, and present guidance by which an analyst can determine if a sufficient number of simulations have been conducted, such that results are considered to have converged. The proposed approach considers the variance of the outcomes of interest in cost-effectiveness analysis as a function of the number of simulations. A worked example of the technique is presented using results from a published model, with recommendations made on best practice. While the technique presented remains essentially arbitrary, it does give a mechanism for assessing the level of simulation error, and thus represents an advance over current practice of a round number of simulations with no assessment of model convergence.



The content of this manuscript was agreed by AJH, ABu, ABr, MP and MDS. The first draft was prepared by AJH and ABu, and the manuscript was revised by AJH, ABu, ABr, MP and MDS. The method proposed was derived by MDS, ABr, ABu, AJH and MP, and the downloadable workbook was prepared by ABu.

Compliance with Ethical Standards

Conflict of interest

Anthony James Hatswell, Ash Bullement, and Mike Paulden report no conflicts of interest. Andrew Briggs and Matthew D. Stevenson have previously published on probabilistic analysis but have no financial conflicts of interest.


No funding was received for this manuscript.

Supplementary material

40273_2018_697_MOESM1_ESM.xlsm (1.8 mb)
Supplementary material 1 (XLSM 1843 kb)


  1. 1.
    Briggs AH, Gray A. Handling uncertainty when performing economic evaluation of healthcare interventions. Health Technol Assess. 1999;3(2):1–134.PubMedGoogle Scholar
  2. 2.
    Baio G, Dawid AP. Probabilistic sensitivity analysis in health economics. Stat Methods Med Res. 2015;24:615–34.CrossRefPubMedGoogle Scholar
  3. 3.
    Canadian Agency for Drugs and Technologies in Health. Guidelines for the economic evaluation of health technologies: Canada. 4th edn. 2017.
  4. 4.
    National Institute for Health and Care Excellence (NICE). Guide to the methods of technology appraisal. NICE; 2013.
  5. 5.
    Briggs AH, Weinstein MC, Fenwick EAL, Karnon J, Sculpher MJ, Paltiel AD. Model parameter estimation and uncertainty: a report of the ISPOR-SMDM modeling good research practices task force-6. Value Health. 2012;15:835–42.CrossRefPubMedGoogle Scholar
  6. 6.
    Elvidge J, Bullement A, Hatswell AJ. Cost effectiveness of characterised chondrocyte implantation for treatment of cartilage defects of the knee in the UK. PharmacoEconomics. 2016;34:1145–59.CrossRefPubMedGoogle Scholar
  7. 7.
    Van Hout BA, Al MJ, Gordon GS, Rutten FFH. Costs, effects and C/E-ratios alongside a clinical trial. Health Econ. 1994;3:309–19.CrossRefPubMedGoogle Scholar
  8. 8.
    Barton GR, Briggs AH, Fenwick EAL. Optimal cost-effectiveness decisions: the role of the cost-effectiveness acceptability curve (CEAC), the cost-effectiveness acceptability frontier (CEAF), and the expected value of perfection information (EVPI). Value Health. 2008;11:886–97.CrossRefPubMedGoogle Scholar
  9. 9.
    Claxton K, Sculpher M, McCabe C, Briggs A, Akehurst R, Buxton M, et al. Probabilistic sensitivity analysis for NICE technology assessment: not an optional extra. Health Econ. 2005;14:339–47.CrossRefPubMedGoogle Scholar
  10. 10.
    Zorginstituut Nederland. Guideline for economic evaluations in healthcare. Guideline—National Health Care Institute [cited 21 May 2017]. 2016.
  11. 11.
    Briggs AH, Wonderling DE, Mooney CZ. Pulling cost-effectiveness analysis up by its bootstraps: a non-parametric approach to confidence interval estimation. Health Econ. 1997;6:327–40.CrossRefPubMedGoogle Scholar
  12. 12.
    O’Brien BJ, Briggs AH. Analysis of uncertainty in health care cost-effectiveness studies: an introduction to statistical issues and methods. Stat Methods Med Res. 2002;11:455–68.CrossRefPubMedGoogle Scholar
  13. 13.
    Stinnett AA, Mullahy J. Net health benefits. Med Decis Mak. 1998;18(2 Suppl):S68–80.CrossRefGoogle Scholar
  14. 14.
    Zethraeus N, Johannesson M, Jönsson B, Löthgren M, Tambour M. Advantages of using the net-benefit approach for analysing uncertainty in economic evaluation studies. PharmacoEconomics. 2003;21:39–48.CrossRefPubMedGoogle Scholar
  15. 15.
    Fenwick E, Briggs A. Cost-effectiveness acceptability curves in the dock: case not proven? Med Decis Mak. 2007;27:93–5.CrossRefGoogle Scholar
  16. 16.
    Ara R, Blake L, Gray L, Hernández M, Crowther M, Dunkley A, et al. What is the clinical effectiveness and cost-effectiveness of using drugs in treating obese patients in primary care? A systematic review. Health Technol Assess. 2012;16(5):iii–xiv (1–195).CrossRefPubMedPubMedCentralGoogle Scholar
  17. 17.
    National Institute for Health and Care Excellence. Naltrexone-bupropion (prolonged release) for managing overweight and obesity [ID757]. Company evidence submission [cited 16 Sep 2017]. 2017.
  18. 18.
    National Institute for Health and Care Excellence. Colorectal cancer (metastatic): trifluridine with tipiracil hydrochloride, after standard therapy [ID876]. Company evidence submission [cited 16 Sep 2017]. 2016.
  19. 19.
    Scottish Medicines Consortium. Working with SMC—a guide for manufacturers. Glasgow: Scottish Medicines Consortium; 2017. p. 11.Google Scholar
  20. 20.
    Iglehart D. Simulating stable stochastic systems, V: comparison of ratio estimators. Nav Res Logist Q. 1975;22:553–65.CrossRefGoogle Scholar
  21. 21.
    Jackknife-After-Bootstrap Efron B, Errors Standard, Functions Influence. J R Stat Soc Ser B Methodol. 1992;54:83–127.Google Scholar
  22. 22.
    Strong M, Oakley JE, Brennan A. Estimating multiparameter partial expected value of perfect information from a probabilistic sensitivity analysis sample: a nonparametric regression approach. Med Decis Mak. 2014;34:311–26.CrossRefGoogle Scholar
  23. 23.
    Heath A, Manolopoulou I, Baio G. A review of methods for analysis of the expected value of Information. Med Decis Mak. 2017;37:747–58.CrossRefGoogle Scholar

Copyright information

© Springer Nature Switzerland AG 2018

Authors and Affiliations

  • Anthony J. Hatswell
    • 1
    • 2
  • Ash Bullement
    • 2
    • 3
  • Andrew Briggs
    • 4
    • 5
  • Mike Paulden
    • 6
  • Matthew D. Stevenson
    • 7
  1. 1.University College LondonLondonUK
  2. 2.Delta Hat LimitedNottinghamUK
  3. 3.BresMed Health SolutionsSheffieldUK
  4. 4.University of GlasgowGlasgowUK
  5. 5.Memorial Sloan-Kettering Cancer CenterNew YorkUSA
  6. 6.University of AlbertaEdmontonCanada
  7. 7.University of SheffieldSheffieldUK

Personalised recommendations