Skip to main content
SpringerLink
Log in

Exploring Uncertainty in Economic Evaluations of Drugs and Medical Devices: Lessons from the First Review of Manufacturers’ Submissions to the French National Authority for Health

  • Original Research Article
  • Published:
PharmacoEconomics Aims and scope Submit manuscript

Abstract

Objectives

The objective of this paper was to evaluate how uncertainty has been accounted for in the cost-effectiveness analyses (CEAs) submitted by manufacturers to the French National Authority for Health (HAS) and to identify recurring concerns in these submissions.

Methods

We used a cross-sectional design to evaluate manufacturers’ submissions from the beginning of the evaluation process in October 2013 to the end of May 2015 (n = 28). The sources of uncertainty attached to these CEAs were categorized and assessed. Relevant data were extracted independently by two assessors.

Results

Adherence to the HAS reference case was generally considered to be acceptable. Methodological uncertainty and parameter uncertainty were the sources of uncertainty that were most frequently explored by manufacturers. The quality of reporting of deterministic sensitivity analysis and probabilistic sensitivity analysis varied substantially across submissions, with a frequent lack of justification of the plausible range of parameter point estimates in 12 submissions (43 %). Structural uncertainty was explored much less frequently. Concerns related to omission of either important clinical events or relevant health states or extrapolation of the effects of the technology beyond the time horizon of the clinical trials were identified in 16 submissions (57 %).

Conclusions

This study presented a characterization of the treatment of uncertainty for the first 28 manufacturers’ submissions to the HAS. This work identified important concerns regarding the exploration of sources of uncertainty. The findings may help manufacturers to improve the quality of their submissions and may provide useful insights for extending guidelines on uncertainty analysis in CEAs submitted to the HAS.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1

Similar content being viewed by others

References

  1. Caro JJ, Moller J. Decision-analysis models: current methodological challenges. Pharmacoeconomics. 2014;32:943–50.

    Article  PubMed  Google Scholar 

  2. Frederix GWJ, Van Hasselt JGC, Schellens JHM, Hovels AM, Raaijmakers JAM, Huitema ADR, Severens JL. The impact of structural uncertainty on cost effectiveness models for adjuvant endocrine breast cancer treatments: the need for disease-specific model standardization and improved guidance. Pharmacoeconomics. 2014;32:47–61.

    Article  PubMed  Google Scholar 

  3. Afzali HHA, Karnon J. Exploring structural uncertainty in model-based economic evaluations. PharmacoEconomics. 2015;33(5):435–43.

    Article  PubMed  Google Scholar 

  4. Décret no 2007-1786 du 19 décembre 2007 relatif au financement de la sécurité sociale. JORF no 0296 du 21 décembre 2007; page 20603 texte no 1.

  5. Décret no 2012-1116 du 2 octobre 2012 relatif aux missions médico-économiques de la Haute Autorité de Santé. JORF no 0231 du 4 octobre 2012; page 15522 texte no 8.

  6. Haute Autorité de Santé. Décision du Collège no 2013.0111/DC/SEESP, 18 Septembre 2013. Saint-Denis La Plaine: HAS. http://www.hassante.fr/portail/jcms/c_1647592/fr/decision-n2013-0111/dc/seesp-du-18-septembre-2013-du-college-de-la-has-relatif-a-l-impact-significatif-sur-les-depenses-de-l-assurance-maladie-declenchant-l-evaluation-medico-economique-des-produits-revendiquant-une-asmr-ou-une-asa-de-niveaux-i-ii-ou-iii.

  7. Format de l’avis d’efficience. Juillet 2013. http://www.has-sante.fr/portail/upload/docs/application/pdf/2013-08/format_de_lavis.pdf.

  8. Haute Autorité de Santé Valeurs de références pour l’évaluation économique en santé. Document de travail. Décembre 2014. Saint-Denis La Plaine: HAS. http://www.has-sante.fr/portail/upload/docs/application/pdf/2014-12/valeurs_de_reference_vf.pdf.

  9. Briggs A. Handling uncertainty in cost-effectiveness models. Pharmacoeconomics. 2000;17:479–500.

    Article  CAS  PubMed  Google Scholar 

  10. Briggs A, Claxton K, Sculpher M. Decision modelling for health economic evaluation. New York: Oxford University Press; 2006.

    Google Scholar 

  11. Claxton K. Exploring uncertainty in cost-effectiveness analysis. Pharmacoeconomics. 2008;26:781–98.

    Article  PubMed  Google Scholar 

  12. Bilcke J, Beutels P, Brisson M, Jit M. Accounting for methodological, structural and parameter uncertainty in decision-analytic models: a practical guide. Med Decis Making. 2011;31:675–92.

    Article  PubMed  Google Scholar 

  13. O’Hagen A, Stevenson M, Madan J. Monte Carlo probabilistic sensitivity analysis for patients level simulations models: efficient estimation of mean and variance using ANCOVA. Health Econ. 2007;16:1009–14.

    Article  Google Scholar 

  14. Al MJ. Cost-effectiveness acceptability curves revisited. Pharmacoeconomics. 2013;31:93–100.

    Article  PubMed  Google Scholar 

  15. Fenwick E, O’Brien BJ, Briggs A. Cost-effectiveness acceptability curves: facts, fallacies and frequently asked questions. Health Econ. 2004;13:405–15.

    Article  PubMed  Google Scholar 

  16. Oakley J, O’Hagan A. Probabilistic sensitivity analysis of complex models: a Bayesian approach. J R Stat Soc B. 2004;66:751–69.

    Article  Google Scholar 

  17. Negrin MA, Vasquez-Polo FJ. Incorporating model uncertainty in cost-effectiveness analysis: a Bayesian model averaging approach. J Health Econ. 2008;27:1250–9.

    Article  PubMed  Google Scholar 

  18. Strong M, Pilgrim H, Oakley J, Chilcott J. Structural uncertainty in health economic decision models. ScHARR Occasional Paper. 2009.

  19. Jackson CH, Thompson SG, Sarples LD. Accounting for uncertainty in health economic decision models by using model averaging. JR Stat Soc. 2009;172:383–404.

    Article  Google Scholar 

  20. Jackson CH, Bojke L, Thompson SG, Claxton K, Sharples LD. A framework for addressing structural uncertainty in decisions models. Med Decis Making. 2011;31:662–74.

    Article  PubMed  Google Scholar 

  21. Bojke L, Claxton K, Sculpher M, Palmer S. Characterizing structural uncertainty in decision analytic models: a review and application of methods. Value Health. 2009;12:739–48.

    Article  PubMed  Google Scholar 

  22. Price MJ, Welton NJ, Briggs AH, Ades AE. Model averaging in the presence of structural uncertainty about treatment effects: influence on treatment decision and expected value of information. Value Health. 2011;14:205–18.

    Article  PubMed  Google Scholar 

  23. Bojke L, Soares M. Decision analysis: eliciting experts’ beliefs to characterize uncertainties. In: Culyer AJ, editor. Encyclopedia of health economics. Amesterdam: Elsevier; 2014.

    Google Scholar 

  24. Ramos IC, Maureen PMH, Mölken RV, Al MJ. Determining the impact of modeling additional sources of uncertainty in value-of-information analysis. Value Health. 2015;18:100–9 (Issue 1).

    Article  PubMed  Google Scholar 

  25. Espinoza MA, Manaca A, Claxton K, Sculpher M. The value of heterogeneity for cost-effectiveness subgroup analysis: conceptual framework and application. Med Decis Making. 2014;34:951–64.

    Article  PubMed  PubMed Central  Google Scholar 

  26. Korekamp BG, Weinstein MC, Stijen T, et al. Uncertainty and patient heterogeneity in medical decision models. Med Decis Making. 2004;30:194–205.

    Article  Google Scholar 

  27. Grutters JP, Sculpher M, Briggs AH, et al. Acknowledging patient heterogeneity in economic evaluation. Pharmacoeconomics. 2013;31:111–23.

    Article  PubMed  Google Scholar 

  28. Haute Autorité de Santé. Choices in methods for economic evaluation. 2012. http://www.has-sante.fr/portail/upload/docs/application/pdf/2012-10/choices_in_methods_for_economic_evaluation.pdf.

  29. Briggs AH, Weinstein MC, Fenwick E, et al. Model parameter estimation and uncertainty: a report of the ISPOR-SMDM Modeling Good Research Practices Task Force–6. Value Health. 2012;15:835–42.

    Article  PubMed  Google Scholar 

  30. National Institute for Health and Clinical Excellence. Guide to the single technology appraisal process; 2009. http://www.nice.org.uk/media/42D/8C/MTAGuideLRFINAL.pdf. Assessed 28 May 2014.

  31. Drummond M, Sculpher M, Torrance G, O’Brien B, Stoddart G. Methods for the economic evaluation of health care programmes. 4th ed. New York: Oxford University Press; 2015.

    Google Scholar 

  32. EUnetHTA Methods for health economic evaluations, final version. May 2015. http://www.eunethta.eu/sites/5026.fedimbo.belgium.be/files/2015-04-29-ECO-GL_Finalversion_0.pdf.

  33. EUnetHTA Joint Action 2, Work Package, Subgroup, Heintz E, Gerber-Grote A3, Ghabri S, Hamers FF, Rupel VP, Slabe-Erker R, Davidson T. Is there a European view on health economic evaluations? Results from a synopsis of methodological guidelines used in the EUnetHTA partner countries. Pharmacoeconomics. 2015. (Epub ahead of print).

  34. Latimer N. NICE DSU technical support document 14: survival analysis for economic evaluations alongside clinical trials: extrapolation with patient level data. NICE DSU Technical Support Document 14. Sheffield: Decision Support Unit, ScHARR, University of Sheffield. 2013.

  35. Jackson CH, Sharpes LD, Thompson SG. Survival models in health economic evaluations: balancing fit and parsimony to improve prediction. Int J Biostat. 2010;6:Article 34.

  36. Tremblay G, Haines P, Briggs A. A criterion-based approach for the systematic and transparent extrapolation of clinical trial survival data. JHEOR. 2015;2:147–60.

    Google Scholar 

  37. Kaltenthaler EC, Dickson R, Boland A, et al. A qualitative study of manufacturers’ submissions to the UK NICE single technology appraisal process. BMJ Open. 2012;2:e000562.

    Article  PubMed  PubMed Central  Google Scholar 

  38. Andronis L, Barton P, Bryan S. Sensitivity analysis in economic evaluation: an audit of NICE current practice and a review of its use and value in decision making. Health Technol Assess. 2009. doi:10.3310/hta13290.

  39. Hill SR, Michel AS, Henry DA. Problems with interpretation of pharmacoeconomic analyses. JAMA. 2000;283:2116–21.

    Article  CAS  PubMed  Google Scholar 

  40. Raimond V, Josselin J-M, Rochaix L. HTA agencies facing model biases: the case of type 2 diabetes. Pharmacoeconomics. 2014;32:825–39.

    Article  PubMed  Google Scholar 

  41. Grutters JP, van Asselt MB, Chalkidou K, Joore MA. Healthy decisions: towards uncertainty tolerance in healthcare policy. Pharmacoeconomics. 2015;33:1–4.

    Article  PubMed  Google Scholar 

  42. Walker WE, Harremoes P, Rotmans J, Van der Sluijs JP, Van Asselt MBA, Janssen P, et al. Defining uncertainty: a conceptual basis for uncertainty management in model-based decision support. Integr Assess. 2003;4:5–17.

    Article  Google Scholar 

  43. Kelton WD, Law AM. Simulation modelling and analysis. Boston: Mc Graw Hill; 2000.

    Google Scholar 

Download references

Acknowledgments

SG conceived the study and prepared the first draft of the manuscript. SG and FFH extracted and analyzed the data. All the authors discussed the results and implications and commented on the manuscript at all stages. The authors are grateful to the members of the Department of Economic and Public Health Evaluation (SEESP) and the Economic and Public Health Evaluation Committee (CEESP), who performed the critical assessments of the economic evaluations submitted by the manufacturers. The authors wish to thank Isabelle Hirtzlin for her comments on an early version of the manuscript and Nathalie Merle for proofreading the article. They are grateful to the three anonymous reviewers for their comments that contributed to enhancing the quality of the paper. The findings and conclusions of this study are those of the authors and do not necessarily represent the views of HAS.

Author information

Authors and Affiliations

  1. Department of Economic and Public Health Evaluation, Haute Autorité de Santé (HAS), 5 Avenue Stade de France, 93218, Saint-Denis La Plaine cedex, France

    Salah Ghabri & Françoise F. Hamers

  2. Faculty of Economics, University of Rennes 1 and CREM-CNRS, Place Hoche 7, 35065, Rennes cedex, France

    Jean Michel Josselin

Authors
  1. Salah Ghabri
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Françoise F. Hamers
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Jean Michel Josselin
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Salah Ghabri.

Ethics declarations

No funding was received for performing the study. SG, FFH and JMJ have no conflict of interest. SG and FFH are employed by HAS, and JMJ is employed by the University of Rennes 1.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Ghabri, S., Hamers, F.F. & Josselin, J.M. Exploring Uncertainty in Economic Evaluations of Drugs and Medical Devices: Lessons from the First Review of Manufacturers’ Submissions to the French National Authority for Health. PharmacoEconomics 34, 617–624 (2016). https://doi.org/10.1007/s40273-016-0381-4

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s40273-016-0381-4

Keywords

Search

Navigation