Abstract
Probabilistic sensitivity analysis (PSA) demonstrates the parameter uncertainty in a decision problem. The technique involves sampling parameters from their respective distributions (rather than simply using mean/median parameter values). Guidance in the literature, and from health technology assessment bodies, on the number of simulations that should be performed suggests a ‘sufficient number’, or until ‘convergence’, which is seldom defined. The objective of this tutorial is to describe possible outcomes from PSA, discuss appropriate levels of accuracy, and present guidance by which an analyst can determine if a sufficient number of simulations have been conducted, such that results are considered to have converged. The proposed approach considers the variance of the outcomes of interest in cost-effectiveness analysis as a function of the number of simulations. A worked example of the technique is presented using results from a published model, with recommendations made on best practice. While the technique presented remains essentially arbitrary, it does give a mechanism for assessing the level of simulation error, and thus represents an advance over current practice of a round number of simulations with no assessment of model convergence.
Similar content being viewed by others
References
Briggs AH, Gray A. Handling uncertainty when performing economic evaluation of healthcare interventions. Health Technol Assess. 1999;3(2):1–134.
Baio G, Dawid AP. Probabilistic sensitivity analysis in health economics. Stat Methods Med Res. 2015;24:615–34.
Canadian Agency for Drugs and Technologies in Health. Guidelines for the economic evaluation of health technologies: Canada. 4th edn. 2017. https://www.cadth.ca/sites/default/files/pdf/guidelines_for_the_economic_evaluation_of_health_technologies_canada_4th_ed.pdf.
National Institute for Health and Care Excellence (NICE). Guide to the methods of technology appraisal. NICE; 2013. https://www.nice.org.uk/process/pmg9/chapter/foreword.
Briggs AH, Weinstein MC, Fenwick EAL, Karnon J, Sculpher MJ, Paltiel AD. Model parameter estimation and uncertainty: a report of the ISPOR-SMDM modeling good research practices task force-6. Value Health. 2012;15:835–42.
Elvidge J, Bullement A, Hatswell AJ. Cost effectiveness of characterised chondrocyte implantation for treatment of cartilage defects of the knee in the UK. PharmacoEconomics. 2016;34:1145–59.
Van Hout BA, Al MJ, Gordon GS, Rutten FFH. Costs, effects and C/E-ratios alongside a clinical trial. Health Econ. 1994;3:309–19.
Barton GR, Briggs AH, Fenwick EAL. Optimal cost-effectiveness decisions: the role of the cost-effectiveness acceptability curve (CEAC), the cost-effectiveness acceptability frontier (CEAF), and the expected value of perfection information (EVPI). Value Health. 2008;11:886–97.
Claxton K, Sculpher M, McCabe C, Briggs A, Akehurst R, Buxton M, et al. Probabilistic sensitivity analysis for NICE technology assessment: not an optional extra. Health Econ. 2005;14:339–47.
Zorginstituut Nederland. Guideline for economic evaluations in healthcare. Guideline—National Health Care Institute [cited 21 May 2017]. 2016. https://english.zorginstituutnederland.nl/publications/reports/2016/06/16/guideline-for-economic-evaluations-in-healthcare.
Briggs AH, Wonderling DE, Mooney CZ. Pulling cost-effectiveness analysis up by its bootstraps: a non-parametric approach to confidence interval estimation. Health Econ. 1997;6:327–40.
O’Brien BJ, Briggs AH. Analysis of uncertainty in health care cost-effectiveness studies: an introduction to statistical issues and methods. Stat Methods Med Res. 2002;11:455–68.
Stinnett AA, Mullahy J. Net health benefits. Med Decis Mak. 1998;18(2 Suppl):S68–80.
Zethraeus N, Johannesson M, Jönsson B, Löthgren M, Tambour M. Advantages of using the net-benefit approach for analysing uncertainty in economic evaluation studies. PharmacoEconomics. 2003;21:39–48.
Fenwick E, Briggs A. Cost-effectiveness acceptability curves in the dock: case not proven? Med Decis Mak. 2007;27:93–5.
Ara R, Blake L, Gray L, Hernández M, Crowther M, Dunkley A, et al. What is the clinical effectiveness and cost-effectiveness of using drugs in treating obese patients in primary care? A systematic review. Health Technol Assess. 2012;16(5):iii–xiv (1–195).
National Institute for Health and Care Excellence. Naltrexone-bupropion (prolonged release) for managing overweight and obesity [ID757]. Company evidence submission [cited 16 Sep 2017]. 2017. https://www.nice.org.uk/guidance/gid-tag486/documents/appraisal-consultation-document-2.
National Institute for Health and Care Excellence. Colorectal cancer (metastatic): trifluridine with tipiracil hydrochloride, after standard therapy [ID876]. Company evidence submission [cited 16 Sep 2017]. 2016. https://www.nice.org.uk/guidance/ta405/documents/committee-papers.
Scottish Medicines Consortium. Working with SMC—a guide for manufacturers. Glasgow: Scottish Medicines Consortium; 2017. p. 11.
Iglehart D. Simulating stable stochastic systems, V: comparison of ratio estimators. Nav Res Logist Q. 1975;22:553–65.
Jackknife-After-Bootstrap Efron B, Errors Standard, Functions Influence. J R Stat Soc Ser B Methodol. 1992;54:83–127.
Strong M, Oakley JE, Brennan A. Estimating multiparameter partial expected value of perfect information from a probabilistic sensitivity analysis sample: a nonparametric regression approach. Med Decis Mak. 2014;34:311–26.
Heath A, Manolopoulou I, Baio G. A review of methods for analysis of the expected value of Information. Med Decis Mak. 2017;37:747–58.
Acknowledgements
The content of this manuscript was agreed by AJH, ABu, ABr, MP and MDS. The first draft was prepared by AJH and ABu, and the manuscript was revised by AJH, ABu, ABr, MP and MDS. The method proposed was derived by MDS, ABr, ABu, AJH and MP, and the downloadable workbook was prepared by ABu.
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Conflict of interest
Anthony James Hatswell, Ash Bullement, and Mike Paulden report no conflicts of interest. Andrew Briggs and Matthew D. Stevenson have previously published on probabilistic analysis but have no financial conflicts of interest.
Funding
No funding was received for this manuscript.
Electronic supplementary material
Below is the link to the electronic supplementary material.
Rights and permissions
About this article
Cite this article
Hatswell, A.J., Bullement, A., Briggs, A. et al. Probabilistic Sensitivity Analysis in Cost-Effectiveness Models: Determining Model Convergence in Cohort Models. PharmacoEconomics 36, 1421–1426 (2018). https://doi.org/10.1007/s40273-018-0697-3
Published:
Issue Date:
DOI: https://doi.org/10.1007/s40273-018-0697-3