Skip to main content
SpringerLink
Log in

Out of Date or Best Before? A Commentary on the Relevance of Economic Evaluations Over Time

  • Practical Application
  • Published:
PharmacoEconomics Aims and scope Submit manuscript

Abstract

The impact of time on the applicability and relevance of historical economic evaluations can be considerable. Ignoring this may lead to the use of weak or invalid evidence to inform important research questions or resource allocation decisions, as historical economic evaluations may have reached different conclusions compared to if a similar study had been conducted more recently. There are multiple factors that contribute towards evidence becoming outdated including changes to the relevant decision problem (e.g. comparators), changes to parameters (such as costs, utilities and resource use) and methodological updates (e.g. recommendations on uncertainty analysis). Researchers reviewing economic evaluations need to consider whether changes over time would influence the study design and results if the evaluation were repeated, to the extent that it is no longer helpful or informative. In this paper, we summarise these key issues and make recommendations about how and whether researchers can future proof their economic evaluations.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1

Similar content being viewed by others

References

  1. Husereau D, Drummond M, Petrou S, et al. Consolidated Health Economic Evaluation Reporting Standards (CHEERS) statement. BMJ. 2013;346: f1049. https://doi.org/10.1136/bmj.f1049.

    Article  PubMed  Google Scholar 

  2. Drummond MF, Jefferson TO. Guidelines for authors and peer reviewers of economic submissions to the BMJ. BMJ. 1996;313:275–83. https://doi.org/10.1136/bmj.313.7052.275.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  3. Philips Z, Ginnelly L, Sculpher M, et al. Review of guidelines for good practice in decision-analytic modelling in health technology assessment. Health Technol Assess. 2004;8:iii–iv. https://doi.org/10.3310/hta8360 (ix–xi, 1–158).

    Article  CAS  PubMed  Google Scholar 

  4. Evers S, Goossens M, de Vet H, et al. Criteria list for assessment of methodological quality of economic evaluations: consensus on health economic criteria. Int J Technol Assess Health Care. 2005;21:240–5. https://doi.org/10.1017/s0266462305050324.

    Article  PubMed  Google Scholar 

  5. Chiou CF, Hay JW, Wallace JF, et al. Development and validation of a grading system for the quality of cost-effectiveness studies. Med Care. 2003;41:32–44. https://doi.org/10.1097/00005650-200301000-00007.

    Article  Google Scholar 

  6. Watts RD, Li IW. Use of checklists in reviews of health economic evaluations, 2010 to 2018. Value Health. 2019;22:377–82. https://doi.org/10.1016/j.jval.2018.10.006.

    Article  PubMed  Google Scholar 

  7. Luhnen M, Prediger B, Neugebauer EAM, et al. Systematic reviews of health economic evaluations: a structured analysis of characteristics and methods applied. Res Synth Methods. 2019;10:195–206. https://doi.org/10.1002/jrsm.1342.

    Article  PubMed  Google Scholar 

  8. Mandrik OL, Hans Severens JL, Bardach A, et al. Critical appraisal of systematic reviews with costs and cost-effectiveness outcomes: an ISPOR Good Practices Task Force Report. Value Health. 2021;24:463–72. https://doi.org/10.1016/j.jval.2021.01.002.

    Article  PubMed  Google Scholar 

  9. Drummond M. Methods for the economic evaluation of health care programmes, 4th edn. Oxford: Oxford University Press. 2015.

  10. Gould J. Towards understanding the under-recognition of girls and women on the autism spectrum. Autism. 2017;21:703–5. https://doi.org/10.1177/1362361317706174.

    Article  PubMed  Google Scholar 

  11. Zener D. Journey to diagnosis for women with autism. Adv Autism. 2019;5:2–13. https://doi.org/10.1108/AIA-10-2018-0041.

    Article  Google Scholar 

  12. Ziouani S, Granados D, Borget I. How to select the best comparator? An international economic evaluation guidelines comparison. Value Health. 2016;19:A471–2. https://doi.org/10.1016/j.jval.2016.09.726.

    Article  Google Scholar 

  13. Tseng A, Seet J, Phillips EJ. The evolution of three decades of antiretroviral therapy: challenges, triumphs and the promise of the future. Br J Clin Pharmacol. 2015;79:182–94. https://doi.org/10.1111/bcp.12403.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  14. Kasztura M, Richard A, Bempong NE, et al. Cost-effectiveness of precision medicine: a scoping review. Int J Public Health. 2019;64:1261–71. https://doi.org/10.1007/s00038-019-01298-x.

    Article  PubMed  Google Scholar 

  15. Lee DH, Tsao M-S, Kambartel K-O, et al. Molecular testing and treatment patterns for patients with advanced non-small cell lung cancer: PIvOTAL observational study. PLoS ONE. 2018;13(8): e0202865. https://doi.org/10.1371/journal.pone.0202865.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  16. National Institute for Health and Care Excellence. Crizotinib for treating ROS1-positive advanced non-small-cell lung cancer. TA529. 2018. https://www.nice.org.uk/guidance/ta529. Accessed 27 Nov 2021.

  17. National Institute for Health and Care Excellence. Pembrolizumab for untreated PD-L1 positive metastatic non- small-cell lung cancer. TA531. 2016. https://www.nice.org.uk/guidance/ta531. Accessed 27 Nov 2021.

  18. Personal Social Services Research Unit. Unit costs of health and social care. https://www.pssru.ac.uk/project-pages/unit-costs/. Accessed 2 Apr 2021.

  19. NHS England. National cost collection for the NHS. https://www.england.nhs.uk/national-cost-collection/. Accessed 2 Apr 2021.

  20. Capri S, Porta C, Condorelli C, et al. An updated cost-effectiveness analysis of pazopanib versus sunitinib as first-line treatment for locally advanced or metastatic renal cell carcinoma in Italy. J Med Econ. 2020;23:1579–87. https://doi.org/10.1080/13696998.2020.1839240.

    Article  PubMed  Google Scholar 

  21. Fonarow GC, Van Hout B, Villa G, et al. Updated cost-effectiveness analysis of evolocumab in patients with very high-risk atherosclerotic cardiovascular disease. JAMA Cardiol. 2019. https://doi.org/10.1001/jamacardio.2019.1647.

    Article  PubMed  PubMed Central  Google Scholar 

  22. Kazi DS, Penko J, Coxson PG, et al. Updated cost-effectiveness analysis of PCSK9 inhibitors based on the results of the FOURIER trial. JAMA. 2017;318:748–50. https://doi.org/10.1001/jama.2017.9924.

    Article  PubMed  PubMed Central  Google Scholar 

  23. Dilokthornsakul P, Kengkla K, Saokaew S, et al. An updated cost-effectiveness analysis of pneumococcal conjugate vaccine among children in Thailand. Vaccine. 2019;37:4551–60. https://doi.org/10.1016/j.vaccine.2019.06.015.

    Article  PubMed  Google Scholar 

  24. Tai TA, Latimer NR, Benedict A, et al. Prevalence of immature survival data for anti-cancer drugs presented to the National Institute for Health and Care Excellence and impact on decision making. Value Health. 2020;24(4):505–12. https://doi.org/10.1016/j.jval.2020.10.016.

    Article  PubMed  Google Scholar 

  25. Michaels JA, Drury D, Thomas SM. Cost-effectiveness of endovascular abdominal aortic aneurysm repair. Br J Surg. 2005;92:960–7. https://doi.org/10.1002/bjs.5119.

    Article  CAS  PubMed  Google Scholar 

  26. Brown LC, Powell JT, Thompson SG, et al. The UK EndoVascular Aneurysm Repair (EVAR) trials: randomised trials of EVAR versus standard therapy. Health Technol Assess. 2012;16(9):1–218. https://doi.org/10.3310/hta16090.

    Article  CAS  PubMed  Google Scholar 

  27. Epstein D, Sculpher MJ, Powell JT, et al. Long-term cost-effectiveness analysis of endovascular versus open repair for abdominal aortic aneurysm based on four randomized clinical trials. Br J Surg. 2014;101:623–31. https://doi.org/10.1002/bjs.9464.

    Article  CAS  PubMed  Google Scholar 

  28. National Institute for Health and Care Excellence. Abdominal aortic aneurysm: diagnosis and management. NICE guideline [NG156]. 2020. https://www.nice.org.uk/guidance/ng156. Accessed 1 Apr 2021.

  29. National Institute for Health and Care Excellence. Guide to the methods of technology appraisal. 2013. https://www.nice.org.uk/process/pmg9/chapter/foreword. Accessed 10 Aug 2018.

  30. National Institute for Health and Care Excellence. Position statement on use of the EQ-5D-5L value set for England (updated October 2019). https://www.nice.org.uk/about/what-we-do/our-programmes/nice-guidance/technology-appraisal-guidance/eq-5d-5l. Accessed 15 Apr 2021.

  31. National Institute for Health and Care Excellence. Reviewing our methods for health technology evaluation: consultation. 2020. https://www.nice.org.uk/about/what-we-do/our-programmes/nice-guidance/chte-methods-consultation. Accessed 2 Feb 2021.

  32. Camacho EM, Shields G, Lovell K, et al. A (five-)level playing field for mental health conditions? Exploratory analysis of EQ-5D-5L-derived utility values. Qual Life Res. 2018;27:717–24. https://doi.org/10.1007/s11136-017-1768-1.

    Article  CAS  PubMed  Google Scholar 

  33. Briggs AH, O’Brien BJ. The death of cost-minimization analysis? Health Econ. 2001;10:179–84. https://doi.org/10.1002/hec.584.

    Article  CAS  PubMed  Google Scholar 

  34. National Institute for Health and Care Exellence. Methods for the development of NICE public health guidance (third edition). 2012. https://www.nice.org.uk/process/pmg4/chapter/introduction/. Accessed 15 Apr 2021.

  35. National Institute for Health and Care Excellence. Developing NICE guidelines: the manual. 2014. https://www.nice.org.uk/process/pmg20/chapter/incorporating-economic-evaluation. Accessed 15 Apr 2021.

  36. Weinstein MC, Siegel JE, Gold MR, et al. Recommendations of the panel on cost-effectiveness in health and medicine. JAMA. 1996;276:1253–8.

    Article  CAS  Google Scholar 

  37. Sanders GD, Neumann PJ, Basu A, et al. Recommendations for conduct, methodological practices, and reporting of cost-effectiveness analyses: second panel on cost-effectiveness in health and medicine. JAMA. 2016;316:1093–103. https://doi.org/10.1001/jama.2016.12195.

    Article  PubMed  Google Scholar 

  38. Garrison LP, Pauly MV, Willke RJ, et al. An overview of value, perspective, and decision context: a health economics approach: an ISPOR Special Task Force Report [2]. Value Health. 2018;21:124–30. https://doi.org/10.1016/j.jval.2017.12.006.

    Article  PubMed  Google Scholar 

  39. Duevel JA, Hasemann L, Peña-Longobardo LM, et al. Considering the societal perspective in economic evaluations: a systematic review in the case of depression. Health Econ Rev. 2020;10:32. https://doi.org/10.1186/s13561-020-00288-7.

    Article  PubMed  PubMed Central  Google Scholar 

  40. Hatswell AJ, Bullement A, Briggs A, et al. Probabilistic sensitivity analysis in cost-effectiveness models: determining model convergence in cohort models. Pharmacoeconomics. 2018;36:1421–6. https://doi.org/10.1007/s40273-018-0697-3.

    Article  PubMed  Google Scholar 

  41. Canadian Agency for Drugs and Technologies in Health. Guidelines for the economic evaluation of health technologies: Canada. 3rd edition. 2006. https://www.cadth.ca/media/pdf/186_EconomicGuidelines_e.pdf. Accessed 15 Mar 2021.

  42. Canadian Agency for Drugs and Technologies in Health. Guidelines for the economic evaluation of health technologies: Canada 4th edition. Methods and guidelines. 2017. https://www.cadth.ca/about-cadth/how-we-do-it/methods-and-guidelines/guidelines-for-the-economic-evaluation-of-health-technologies-canada. Accessed 13 Aug 2020.

  43. Statens Legemiddelverk. Guidelines for the submission of documentation for single technology assessment (STA) of pharmaceuticals: Legemiddelverket. 2020. https://legemiddelverket.no/english/public-funding-and-pricing/documentation-for-sta/guidelines-for-the-submission-of-documentation-for-single-technology-assessment-sta-of-pharmaceuticals. Accessed 22 Oct 2021.

  44. Zorginstituut Nederland. Guideline for economic evaluations in healthcare. 2016. https://english.zorginstituutnederland.nl/publications/reports/2016/06/16/guideline-for-economic-evaluations-in-healthcare. Accessed 22 Oct 2021.

  45. Sculpher MJ, Claxton K, Drummond M, et al. Whither trial-based economic evaluation for health care decision making? Health Econ. 2006;15:677–87. https://doi.org/10.1002/hec.1093.

    Article  PubMed  Google Scholar 

  46. Committee TPBA. Guidelines for preparing submissions to the Pharmaceutical Benefits Advisory Committee (PBAC): version 4.0. 2006. https://pbac.pbs.gov.au/content/information/archived-versions/pbac-guidelines-v4-2006.pdf. Accessed 15 Mar 2021.

  47. The Pharmaceutical Benefits Advisory Committee. Guidelines for preparing submissions to the Pharmaceutical Benefits Advisory Committee (PBAC): version 5.0. 2016. https://pbac.pbs.gov.au/information/about-the-guidelines.html. Accessed 15 Mar 2021.

  48. Ghabri S, Lam L, Bocquet F, et al. Systematic literature review of economic evaluations of biological treatment sequences for patients with moderate to severe rheumatoid arthritis previously treated with disease-modifying anti-rheumatic drugs. Pharmacoeconomics. 2020;38:459–71. https://doi.org/10.1007/s40273-020-00887-6.

    Article  PubMed  Google Scholar 

  49. Woods BS, Sideris E, Palmer S, et al. Partitioned survival and state transition models for healthcare decision making in oncology: where are we now? Value Health. 2020;23:1613–21. https://doi.org/10.1016/j.jval.2020.08.2094.

    Article  PubMed  Google Scholar 

  50. Cranmer H, Shields GE, Bullement A. A comparison of partitioned survival analysis and state transition multi-state modelling approaches using a case study in oncology. J Med Econ. 2020;23:1176–85. https://doi.org/10.1080/13696998.2020.1796360.

    Article  PubMed  Google Scholar 

  51. Leurent B, Gomes M, Carpenter JR. Missing data in trial-based cost-effectiveness analysis: an incomplete journey. Health Econ. 2018;27:1024–40. https://doi.org/10.1002/hec.3654.

    Article  PubMed  PubMed Central  Google Scholar 

  52. Versteegh M. Impact on the incremental cost-effectiveness ratio of using alternatives to EQ-5D in a Markov model for multiple sclerosis. Pharmacoeconomics. 2016;34:1133–44. https://doi.org/10.1007/s40273-016-0421-0.

    Article  PubMed  PubMed Central  Google Scholar 

  53. Thompson AJ, Turner AJ. A comparison of the EQ-5D-3L and EQ-5D-5L. Pharmacoeconomics. 2020;38:575–91. https://doi.org/10.1007/s40273-020-00893-8.

    Article  PubMed  Google Scholar 

  54. Hernandez Alava M, Wailoo A, Grimm S, et al. EQ-5D-5L versus EQ-5D-3L: the impact on cost effectiveness in the United Kingdom. Value Health. 2018;21:49–56. https://doi.org/10.1016/j.jval.2017.09.004.

    Article  PubMed  Google Scholar 

  55. Pennington B, Hernandez-Alava M, Pudney S, et al. The impact of moving from EQ-5D-3L to -5L in NICE technology appraisals. Pharmacoeconomics. 2019;37:75–84. https://doi.org/10.1007/s40273-018-0701-y.

    Article  PubMed  Google Scholar 

  56. Keetharuth AD, Rowen D, Bjorner JB, et al. Estimating a preference-based index for mental health from the recovering quality of life measure: valuation of recovering Quality of Life Utility Index. Value Health. 2021;24:281–90. https://doi.org/10.1016/j.jval.2020.10.012.

    Article  PubMed  PubMed Central  Google Scholar 

  57. Helter TM, Coast J, Łaszewska A, et al. Capability instruments in economic evaluations of health-related interventions: a comparative review of the literature. Qual Life Res. 2020;29:1433–64. https://doi.org/10.1007/s11136-019-02393-5.

    Article  PubMed  Google Scholar 

  58. Phillippo DM, Ades AE, Dias S, et al. Methods for population-adjusted indirect comparisons in health technology appraisal. Med Decis Mak. 2018;38:200–11. https://doi.org/10.1177/0272989X17725740.

    Article  Google Scholar 

  59. Jansen JP. Network meta-analysis of survival data with fractional polynomials. BMC Med Res Methodol. 2011;11:61. https://doi.org/10.1186/1471-2288-11-61.

    Article  PubMed  PubMed Central  Google Scholar 

  60. National Institute for Health and Care Excellence. Final appraisal document: ribociclib with fulvestrant for treating hormone receptor-positive, HER2-negative advanced breast cancer after endocrine therapy. Technology appraisal guidance [TA687]. 2021. https://www.nice.org.uk/guidance/ta687. Accessed 15 Apr 2021.

  61. Othus M, Bansal A, Koepl L, et al. Accounting for cured patients in cost-effectiveness analysis. Value Health. 2017;20:705–9. https://doi.org/10.1016/j.jval.2016.04.011.

    Article  PubMed  Google Scholar 

  62. Bullement A, Latimer NR, Bell GH. Survival extrapolation in cancer immunotherapy: a validation-based case study. Value Health. 2019;22:276–83. https://doi.org/10.1016/j.jval.2018.10.007.

    Article  PubMed  Google Scholar 

  63. Decision Support Unit. NICE DSU technical support document 21: flexible methods for survival analysis. 2020. http://nicedsu.org.uk/wp-content/uploads/2020/11/NICE-DSU-Flex-Surv-TSD-21_Final_alt_text.pdf. Accessed 15 Mar 2021.

  64. Sampson CJ, Arnold R, Bryan S, et al. Transparency in decision modelling: what, why, who and how? Pharmacoeconomics. 2019;37:1355–69. https://doi.org/10.1007/s40273-019-00819-z.

    Article  PubMed  PubMed Central  Google Scholar 

Download references

Acknowledgements

We thank Zoe Philips for her input and feedback on an initial draft of this paper.

Author information

Authors and Affiliations

  1. Division of Population Health, Health Services Research, and Primary Care, Manchester Centre for Health Economics, The University of Manchester, Manchester, UK

    Gemma E. Shields & Stuart Wright

  2. School of Health and Related Research, University of Sheffield, Sheffield, UK

    Becky Pennington & Ash Bullement

  3. Delta Hat Ltd, Nottingham, UK

    Ash Bullement

  4. Science, Evidence and Analytics Directorate, National Institute for Health and Care Excellence, Manchester, UK

    Jamie Elvidge

Authors
  1. Gemma E. Shields
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Becky Pennington
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Ash Bullement
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Stuart Wright
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Jamie Elvidge
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Gemma E. Shields.

Ethics declarations

Funding

The authors received no financial support for the research, authorship or publication of this article. The views expressed are those of the authors and not those of their employing organisations.

Conflicts of interest/competing interests

All authors have completed the ICMJE uniform disclosure form at www.icmje.org/coi_disclosure.pdf and declare no support from any organisation for the submitted work (with the exclusion of the stated funder), no financial relationships with any organisations that might have an interest in the submitted work in the previous 3 years and no other relationships or activities that could appear to have influenced the submitted work.

Ethics approval

Not applicable.

Consent to participate

Not applicable.

Consent for publication

Not applicable.

Availability of data and material

Not applicable.

Code availability

Not applicable.

Author contributions

Conceptualization: GES, BP, AB and JE. Writing, original draft preparation: GES, BP, AB, SW and JE. Writing, review and editing: GES, BP, AB, SW and JE.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Shields, G.E., Pennington, B., Bullement, A. et al. Out of Date or Best Before? A Commentary on the Relevance of Economic Evaluations Over Time. PharmacoEconomics 40, 249–256 (2022). https://doi.org/10.1007/s40273-021-01116-4

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s40273-021-01116-4

Search

Navigation