Skip to main content
SpringerLink
Log in

Impact of small study bias on cost-effectiveness acceptability curves and value of information analyses

  • Original Paper
  • Published:
The European Journal of Health Economics Aims and scope Submit manuscript

Abstract

It is well known that small, randomized, controlled trials (RCTs) have limited validity. When comparing the results of meta-analyses with those of later large trials or with those of large trials removed from the meta-analyses, discrepancies were reported. This paper addresses two issues: (1) how measures of the uncertainty in cost-effectiveness, i.e., cost-effectiveness acceptability curves (CEACs), and the expected value of perfect information (EVPI) are affected by the limited validity of small trials and (2) how to deal with this bias. To this end, the paper adopts a Bayesian approach. Using empirical estimates for the validity of small RCTs compared to larger RCTs, the probability of cost-effectiveness drops by almost 10 %, while the EVPI is three times higher. In conclusion, traditional CEACs and EVPI analyses based on (small) RCTs may need careful appraisal. Ignoring prior evidence on the validity of small-size trials leads to an underestimation of uncertainty in cost-effectiveness. For future economic analyses, it is important to incorporate aspects of uncertainty which are caused by flawed data on effectiveness .

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2

Similar content being viewed by others

References

  1. Philips, Z., Ginnelly, L., Sculpher, M., Claxton, K., Golder, S., Riemsma, R., Woolacoot, N., Glanville, J.: Review of guidelines for good practice in decision-analytic modelling in health technology assessment. Health Technol. Assess. 8(36), 1–158 (2004)

    Article  PubMed  Google Scholar 

  2. Thorlund, K., Imberger, G., Walsh, M., Chu, R., Gluud, C., Wetterslev, J., Guyatt, G., Devereaux, P.J., Thabane, L.: The number of patients and events required to limit the risk of overestimation of intervention effects in meta-analysis: a simulation study. PLoS ONE 6(10), e254–e291 (2011)

    Article  Google Scholar 

  3. LeLorier, J., Grégoire, G., Benhaddad, A., Lapierre, J., Derderian, F.: Discrepancies between meta-analyses and subsequent large randomized, controlled trials. N. Engl. J. Med. 337(8), 536–542 (1997)

    Article  CAS  PubMed  Google Scholar 

  4. Nüsch, E., Trelle, S., Reichenbach, S., Rutjes, A.W., Tschannen, B., Altman, D.G., Egger, M., Jüni, P.: Small study effects in meta-analyses of osteoarthritis trials: meta-epidemiological study. Br. Med. J. 341, c3515 (2010)

    Article  Google Scholar 

  5. Sterne, J.A., Gavaghan, D., Egger, M.: Publication and related bias in meta-analysis: power of statistical tests and prevalence in the literature. J. Clin. Epidemiol. 53(11), 1119–1129 (2000)

    Article  CAS  PubMed  Google Scholar 

  6. Rothwell, P.M.: External validity of randomised controlled trials: “to whom do the results of this trial apply?” Lancet 365(9453), 82–93 (2005)

    Article  PubMed  Google Scholar 

  7. Gehr, B.T., Weiss, C., Porzsolt, F.: The fading of reported effectiveness. A meta-analysis of randomised controlled trials. BMC Med. Res. Methodol. 11, 6–25 (2006)

    Google Scholar 

  8. Claxton, K., Neumann, P.J., Araki, S., Weinstein, M.C.: Bayesian value-of-information analysis. An application to a policy model of Alzheimer’s disease. Int. J. Technol. Assess. Health Care 17(1), 38–55 (2001)

    Article  CAS  PubMed  Google Scholar 

  9. Claxton, K., Ginnelly, L., Sculpher, M., Philips, Z., Palmer, S.: A pilot study on the use of decision theory and value of information analysis as part of the NHS Health technology assessment programme. Health Technol. Assess. 8(31), 1–103 (2004)

    Article  CAS  PubMed  Google Scholar 

  10. Groot Koerkamp, B., Myriam Hunink, M.G., Stijnen, T., Weinstein, M.C.: Identifying key parameters in cost-effectiveness analysis using value of information: a comparison of methods. Health Econ. 15(4), 383–392 (2006)

    Article  PubMed  Google Scholar 

  11. Fenwick, E., Claxton, K., Sculpher, M.: The value of implementation and the value of information: combined and uneven development. Med. Decis. Mak. 28(1), 21–32 (2008)

    Article  Google Scholar 

  12. Baio, G., Russo, P.: A decision-theoretic framework for the application of cost-effectiveness analysis in regulatory processes. Pharmacoeconomics 27(8), 645–655 (2009)

    Article  PubMed  Google Scholar 

  13. Briggs, A.H.: A Bayesian approach to stochastic cost-effectiveness analysis. Health Econ. 8(3), 257–261 (1999)

    Article  CAS  PubMed  Google Scholar 

  14. O’Hagan, A., Stevens, J.W., Montmartin, J.: Inference for the cost-effectiveness acceptability curve and cost-effectiveness ratio. Pharmacoeconomics 17(4), 339–349 (2000)

    Article  PubMed  Google Scholar 

  15. Hoch, J.S., Blume, J.D.: Measuring and illustrating statistical evidence in a cost-effectiveness analysis. J. Health Econ. 27(2), 476–495 (2008)

    Article  PubMed  Google Scholar 

  16. Claxton, K.: The irrelevance of inference: a decision-making approach to the stochastic evaluation of health care technologies. J. Health Econ. 18(3), 341–364 (1999)

    Article  CAS  PubMed  Google Scholar 

  17. Bayarri, M.J., Berger, J.: The interplay between Bayesian and frequentist analysis. Stat. Sci. 19, 58–80 (2004)

    Article  Google Scholar 

  18. Ioannidis, J.P., Cappelleri, J.C., Lau, J.: Issues in comparisons between meta-analyses and large trials. J. Am. Med. Assoc. 279(14), 1089–1093 (1998)

    Article  CAS  Google Scholar 

  19. Griffiths, T.L., Phillips, C.J., Davies, S., Burr, M.L., Campbell, I.A.: Cost-effectiveness of an outpatient multidisciplinary pulmonary rehabilitation programme. Thorax 56(10), 779–784 (2001)

    Article  PubMed Central  CAS  PubMed  Google Scholar 

  20. Campbell, H.E., Tait, S., Sharples, L.D., Caine, N., Gray, T.J., Schofield, P.M., Buxton, M.J.: Trial-based cost-utility comparison of percutaneous myocardial laser revascularisation and continued medical therapy for treatment of refractory angina pectoris. Eur. J. Health Econ. 6(4), 288–297 (2005)

    Article  PubMed  Google Scholar 

  21. Keränen, J., Soini, E.J., Ryynänen, O.P., Hietaniemi, K., Keränen, U.: Economic evaluation comparing from home to operation same day admission and preoperative admission one day prior to the surgery process: a randomized, controlled trial of laparoscopic cholecystectomy. Curr. Med. Res. Opin. 23(11), 2775–2784 (2007)

    Article  PubMed  Google Scholar 

  22. Taride, J.E., Blackhouse, G., De Rose, G., Novick, T., Bowen, J.M., Hopkins, R., O’Reilly, D., Goeree, R.: Cost-effectiveness analysis of elective endovascular repair compared with open surgical repair of abdominal aortic aneurysms for patients at a high surgical risk: a 1-year patient-level analysis conducted in Ontario, Canada. J. Vasc. Surg. 48(4), 779–787 (2008)

    Article  Google Scholar 

  23. Gusi, N., Reyes, M.C., Gonzalez-Guerrero, J.L., Herrera, E., Garcia, J.M.: Cost-utility of a walking programme for moderately depressed, obese, or overweight elderly women in primary care: a randomised controlled trial. BMC Public Health 8, 8–231 (2008)

    Article  Google Scholar 

  24. Knapp, M., Romeo, R., Mogg, A., Eranti, S., Pluck, G., Purvis, R., Brown, R.G., Howard, R., Philpot, M., Rothwell, J., Edwards, D., McLoughlin, D.M.: Cost-effectiveness of transcranial magnetic stimulation vs. electroconvulsive therapy for severe depression: a multi-centre randomised controlled trial. J. Affect. Disord. 109(3), 273–285 (2008)

    Article  PubMed  Google Scholar 

  25. Smeets, R.J., Severens, J.L., Beelen, S., Vlaeyen, J.W., Knottnerus, J.A.: More is not always better: cost-effectiveness analysis of combined, single behavioral and single physical rehabilitation programs for chronic low back pain. Eur. J. Pain 13(1), 71–81 (2009)

    Article  PubMed  Google Scholar 

  26. Gerli, S., Bini, V., Di Renzo, G.C.: Cost-effectiveness of recombinant follicle-stimulating hormone (FSH) versus human FSH in intrauterine insemination cycles: a statistical model-derived analysis. Gynecol. Endocrinol. 24(1), n18–n23 (2008)

    Article  Google Scholar 

  27. Bravo Vergel, Y., Hawkins, N.S., Claxton, K., Asseburg, C., Palmer, S., Woolacott, N., Bruce, I.N., Scalpher, N.J.: The cost-effectiveness of etanercept and infliximab for the treatment of patients with psoriatic arthritis. Rheumatology (Oxford) 46(11), 1729–1735 (2007)

    Article  CAS  Google Scholar 

  28. Ford, A.C., Delaney, B.C., Forman, D., Moayyedi, P.: Eradication therapy in Helicobacter pylori-positive peptic ulcer disease: systematic review and economic analysis. Am. J. Gastroenterol. 99(9), 1833–1855 (2004)

    Article  PubMed  Google Scholar 

  29. Connock, M., Stevens, C., Fry-Smith, A., Jowett, S., Fitzmaurice, D., Moore, D., Song, F.: Clinical effectiveness and cost-effectiveness of different models of managing long-term oral anticoagulation therapy: a systematic review and economic modelling. Health Technol. Assess. 11(38), 66 (2007)

    Article  Google Scholar 

  30. Piñol, C., Roze, S., Valentine, W., Evers, T.: Cost-effectiveness of the addition of acarbose to the treatment of patients with type-2 diabetes in Spain. Gac. Sanit. 21(2), 97–104 (2007)

    Article  PubMed  Google Scholar 

  31. Garside, R., Pitt, M., Anderson, R., Rogers, G., Dyer, M., Mealing, S., Somerville, M., Price, A., Stein, K.: The effectiveness and cost-effectiveness of carmustine implants and temozolomide for the treatment of newly diagnosed high-grade glioma: a systematic review and economic evaluation. Health Technol. Assess. 11(45), 221 (2007)

    Article  Google Scholar 

  32. Wang, D., Connock, M., Barton, P., Fry-Smith, A., Aveyard, P., Moore, D.: ‘Cut down to quit’ with nicotine replacement therapies in smoking cessation: a systematic review of effectiveness and economic analysis. Health Technology Assessment 12(2), 1–135 (2008)

    Article  Google Scholar 

  33. Wailoo, A.J., Sutton, A.J., Cooper, N.J., Turner, D.A., Abrams, K.R., Brennan, A., Nicholson, K.G.: Cost-effectiveness and value of information analyses of neuraminidase inhibitors for the treatment of influenza. Value Health 11(2), 160–171 (2008)

    Article  PubMed  Google Scholar 

  34. Villar, J., Carroli, G., Belizán, J.M.: Predictive ability of meta-analyses of randomised controlled trials. Lancet 345(8952), 772–776 (1995)

    Article  CAS  PubMed  Google Scholar 

  35. Cappelleri, J.C., Ioannidis, J.P., Schmid, C.H., de Ferranti, S.D., Aubert, M., Chalmers, T.C., Lau, J.: Large trials vs meta-analysis of smaller trials: how do their results compare? J. Am. Med. Assoc. 276(16), 1332–1338 (1996)

    Article  CAS  Google Scholar 

  36. Flather, M.D., Farkouh, M.E., Pogue, J.M., Yusuf, S.: Strengths and limitations of meta-analysis: larger studies may be more reliable. Control. Clin. Trials 18(6), 568–579 (1997)

    Article  CAS  PubMed  Google Scholar 

  37. Sterne, J.A., Egger, M., Smith, G.D.: Systematic reviews in health care: investigating and dealing with publication and other biases in meta-analysis. BMJ 323(7304), 101–105 (2001)

    Article  PubMed Central  CAS  PubMed  Google Scholar 

  38. Löthgren, M., Zethraeus, N.: Definition, interpretation and calculation of cost-effectiveness acceptability curves. Health Econ. 9(7), 623–630 (2000)

    Article  PubMed  Google Scholar 

  39. Groot Koerkamp, B., Hunink, M.G., Stijnen, T., Hammitt, J.K., Kuntz, K.M., Weinstein, M.C.: Limitations of acceptability curves for presenting uncertainty in cost-effectiveness analysis. Med. Decis. Mak. 27(2), 101–111 (2007)

    Article  Google Scholar 

  40. Jakubczyk, M., Kamiński, B.: Cost-effectiveness acceptability curves: caveats quantified. Health Econ. 19(8), 955–963 (2010)

    Article  PubMed  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Institute of Health Economics and Clinical Epidemiology, University Hospital of Cologne, Gleueler Str. 176-178, 50935, Cologne, Germany

    Dirk Müller

  2. Department of Business Administration and Health Care Management, University of Cologne, Cologne, Germany

    Dirk Müller

  3. Department of Clinical Epidemiology and Biostatistics, McMaster University, Hamilton, Canada

    Eleanor Pullenayegum

  4. Frankfurt School of Finance and Management, Frankfurt, Germany

    Afschin Gandjour

Authors
  1. Dirk Müller
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Eleanor Pullenayegum
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Afschin Gandjour
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Dirk Müller.

Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary material 1 (DOCX 26 kb)

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Müller, D., Pullenayegum, E. & Gandjour, A. Impact of small study bias on cost-effectiveness acceptability curves and value of information analyses. Eur J Health Econ 16, 219–223 (2015). https://doi.org/10.1007/s10198-014-0607-3

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10198-014-0607-3

Keywords

JEL Classification

Search

Navigation