Skip to main content
SpringerLink
Log in

Summarising the Evidence for Drug Safety: A Methodological Discussion of Different Meta-Analysis Approaches

  • Leading Article
  • Published:
Drug Safety Aims and scope Submit manuscript

Abstract

Evidence on drug safety obtained from randomised clinical trials is very limited due to, among other reasons, their relatively small sample size. Hence, combining the results of available studies can prove particularly useful. This paper reviews the different data sources for summarising drug safety outcomes, according to study design, publication of data, and origin of the information. It then discusses the various types of overviews that can be used in the study of treatment harms, focusing on meta-analyses of aggregate data and meta-analyses of individual patient data, with their advantages and drawbacks, such as publication bias and heterogeneity. Although the different approaches available for combining the results are of great utility in assessing treatment harms, none of them is free from limitations. Therefore, it might be appropriate to perform an analysis of sensitivity to assess whether the results are sensitive to the technique that has been used.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. Bouvy JC, De Bruin ML, Koopmanschap MA. Epidemiology of adverse drug reactions in Europe: a review of recent observational studies. Drug Saf. 2015;38:437–53.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  2. Clark DW, Coulter DM, Besag FM. Randomized controlled trials and assessment of drug safety. Drug Saf. 2008;31:1057–61.

    Article  PubMed  Google Scholar 

  3. Reynolds RF, Lem JA, Gatto NM, et al. Is the large simple trial design used for comparative, post-approval safety research? A review of a clinical trials registry and the published literature. Drug Saf. 2011;34:799–820.

    Article  CAS  PubMed  Google Scholar 

  4. Ray WA. Improving automated database studies. Epidemiology. 2011;22:302–4.

    Article  PubMed  Google Scholar 

  5. CIOMS Working Group X. Evidence synthesis and meta-analysis: report of CIOMS Working Group X. Geneva: Council for International Organizations of Medical Sciences (CIOMS); 2016.

    Google Scholar 

  6. Ioannidis JP, Lau J. Completeness of safety reporting in randomized trials: an evaluation of 7 medical areas. JAMA. 2001;285:437–43.

    Article  CAS  PubMed  Google Scholar 

  7. Chou R, Helfand M. Challenges in systematic reviews that assess treatment harms. Ann Intern Med. 2005;142:1090–9.

    Article  PubMed  Google Scholar 

  8. Nuesch E, Trelle S, Reichenbach S, et al. The effects of excluding patients from the analysis in randomized controlled trials: meta-epidemiological study. BMJ. 2009;339:b3244.

    Article  PubMed  PubMed Central  Google Scholar 

  9. Hammad TA, Pinheiro SP, Neyarapally GA. Secondary use of randomized controlled trials to evaluate drug safety: a review of methodological considerations. Clin Trials. 2011;8:559–70.

    Article  PubMed  Google Scholar 

  10. Moride Y, Abenhaim L. Evidence of the depletion of susceptibles effect in non-experimental pharmacoepidemiologic research. J Clin Epidemiol. 1994;47:731–7.

    Article  CAS  PubMed  Google Scholar 

  11. Singh S, Loke YK. Drug safety assessment in clinical trials: methodological challenges and opportunities. Trials. 2012;13:138.

    Article  PubMed  PubMed Central  Google Scholar 

  12. Vandenbroucke JP. When are observational studies as credible as randomised trials? Lancet. 2004;363:1728–31.

    Article  PubMed  Google Scholar 

  13. Vandenbroucke JP. The HRT controversy: observational studies and RCTs fall in line. Lancet. 2009;373:1233–5.

    Article  PubMed  Google Scholar 

  14. Papanikolaou PN, Christidi GD, Ioannidis JP. Comparison of evidence on harms of medical interventions in randomized and nonrandomized studies. CMAJ. 2006;174:635–41.

    Article  PubMed  PubMed Central  Google Scholar 

  15. Vandenbroucke JP. What is the best evidence for determining harms of medical treatment? CMAJ. 2006;174:645–6.

    Article  PubMed  PubMed Central  Google Scholar 

  16. Reeves BC, Deeks JJ, Higgins JPT, et el. Chapter 13: Including non-randomized studies. In: Higgins JPT, Green S, editors. Cochrane handbook for systematic reviews of interventions version 5.1.0 (updated March 2011). The Cochrane Collaboration; 2011. Available from http://www.cochrane-handbook.org.

  17. Golder S, Loke YK, Bland M. Meta-analyses of adverse effects data derived from randomised controlled trials as compared to observational studies: methodological overview. PLoS Med. 2011;8:e1001026.

    Article  PubMed  PubMed Central  Google Scholar 

  18. Bailey C, Peddie D, Wickham ME, et al. Adverse drug event reporting systems-a systematic review. Br J Clin Pharmacol. 2016;82:17–29.

    Article  PubMed  Google Scholar 

  19. Saini P, Loke YK, Gamble C, et al. Selective reporting bias of harm outcomes within studies: findings from a cohort of systematic reviews. BMJ. 2014;349:g6501.

    Article  PubMed  PubMed Central  Google Scholar 

  20. Rücker G, Schwarzer G, Carpenter J, et al. Why add anything to nothing? The arcsine difference as a measure of treatment effect in meta-analysis with zero cells. Stat Med. 2009;28:721–38.

    Article  PubMed  Google Scholar 

  21. Song F, Eastwood AJ, Gilbody S, et al. Publication and related biases. Health Technol Assess. 2000;4:1–115.

    Google Scholar 

  22. Morrison A, Moulton K, Clark M, Polisena, et al. English-language restriction when conducting systematic review-based meta-analyses: systematic review of published studies. Ottawa: Canadian Agency for Drugs and Technologies in Health; 2009.

    Google Scholar 

  23. Sedgwick P. What is publication bias in a meta-analysis? BMJ. 2015;351:h4419.

    Article  PubMed  Google Scholar 

  24. Senn SJ. Overstating the evidence: double counting in meta-analysis and related problems. BMC Med Res Methodol. 2009;9:10.

    Article  PubMed  PubMed Central  Google Scholar 

  25. Scherer RW, Langenberg P, von Elm E. Full publication of results initially presented in abstracts. Cochrane Database Syst Rev. 2007:MR000005.

  26. Kotecha D, Manzano L, Krum H, et al. Beta-Blockers in Heart Failure Collaborative Group. Effect of age and sex on efficacy and tolerability of β blockers in patients with heart failure with reduced ejection fraction: individual patient data meta-analysis. BMJ. 2016;353:i1855.

    Article  PubMed  PubMed Central  Google Scholar 

  27. Reichenpfader U, Gartlehner G, Morgan LC, et al. Sexual dysfunction associated with second-generation antidepressants in patients with major depressive disorder: results from a systematic review with network meta-analysis. Drug Saf. 2014;37:19–31.

    Article  CAS  PubMed  Google Scholar 

  28. Veroniki AA, Straus SE, Ashoor HM, et al. Comparative safety and effectiveness of cognitive enhancers for Alzheimer’s dementia: protocol for a systematic review and individual patient data network meta-analysis. BMJ Open. 2016;6:e010251.

    Article  PubMed  PubMed Central  Google Scholar 

  29. Golder S, Loke YK, Wright K, Norman G. Reporting of adverse events in published and unpublished studies of health care interventions: a systematic review. PLoS Med. 2016;13:e1002127.

    Article  PubMed  PubMed Central  Google Scholar 

  30. FDA Amendments Act. Public Law 110-85. 2007. Ref Type: Statute.

  31. Law MR, Kawasumi Y, Morgan SG. Despite law, fewer than one in eight completed studies of drugs and biologics are reported on time on ClinicalTrials.gov. Health Aff (Millwood). 2011;30:2338–45.

    Article  Google Scholar 

  32. Zarin DA, Tse T, Williams RJ, Carr S. Trial reporting in ClinicalTrials.gov—the final rule. N Engl J Med. 2016;375:1998–2004.

    Article  PubMed  PubMed Central  Google Scholar 

  33. Aalaei-Andabili SH, Alavian SM. Important steps for a reliable meta-analysis. Lancet Infect Dis. 2012;12:663.

    Article  PubMed  Google Scholar 

  34. Sterne JAC, Egger M, Moher D. Chapter 10: Addressing reporting biases. In: Higgins JPT, Green S, editors. Cochrane handbook for systematic reviews of interventions version 5.1.0 (updated March 2011). The Cochrane Collaboration; 2011. Available from http://www.cochrane-handbook.org.

  35. Terrin N, Schmid CH, Lau J. In an empirical evaluation of the funnel plot, researchers could not visually identify publication bias. J Clin Epidemiol. 2005;58:894–901.

    Article  PubMed  Google Scholar 

  36. Egger M, Davey Smith G, Schneider M, et al. Bias in metaanalysis detected by a simple, graphical test. BMJ. 1997;315:629–34.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  37. Loke YK, Mattishent K. If nothing happens, is everything all right? Distinguishing genuine reassurance from a false sense of security. CMAJ. 2015;187:15–6.

    Article  PubMed  PubMed Central  Google Scholar 

  38. Chou R, Aronson N, Atkins D, et al. AHRQ series paper 4: assessing harms when comparing medical interventions: AHRQ and the effective health-care program. J Clin Epidemiol. 2010;63:502–12.

    Article  PubMed  Google Scholar 

  39. Prada-Ramallal G, Takkouche B, Figueiras A. Diverging conclusions from the same meta-analysis in drug safety: source of data (primary versus secondary) takes a toll. Drug Saf. 2016. http://rd.springer.com/article/10.1007%2Fs40264-016-0492-z [Epub ahead of print].

  40. Schneeweiss S, Avorn J. A review of uses of health care utilization databases for epidemiologic research on therapeutics. J Clin Epidemiol. 2005;58:323–37.

    Article  PubMed  Google Scholar 

  41. Figueiras A, Ferreira MT, Gestal JJ. Farmacovigilancia. In: Fernández-Crehuet J, Gestal JJ, Delgado M, et al., editors. Piédrola Gil. Medicina Preventiva y Salud Pública. 12ª Edición. Barcelona: Elsevier Masson; 2015. pp. 1093–104.

  42. Van Walraven C, Austin P. Administrative database research has unique characteristics that can risk biased results. J Clin Epidemiol. 2012;65:126–31.

    Article  PubMed  Google Scholar 

  43. Takahashi Y, Nishida Y, Asai S. Utilization of health care databases for pharmacoepidemiology. Eur J Clin Pharmacol. 2012;68:123–9.

    Article  CAS  PubMed  Google Scholar 

  44. Hennessy S. Use of health care databases in pharmacoepidemiology. Basic Clin Pharmacol Toxicol. 2006;98:311–3.

    Article  CAS  PubMed  Google Scholar 

  45. Blettner M, Sauerbrei W, Schlehofer B, et al. Traditional reviews, meta-analyses and pooled analyses in epidemiology. Int J Epidemiol. 1999;28:1–9.

    Article  CAS  PubMed  Google Scholar 

  46. Blettner M, Schlattmann P. Meta-analysis in epidemiology. In: Ahrens W, Pigeot I, editors. Handbook of epidemiology. Berlin: Springer; 2005. pp. 829–59.

    Chapter  Google Scholar 

  47. Nordmann AJ, Kasenda B, Briel M. Meta-analyses: what they can and cannot do. Swiss Med Wkly. 2012;142:w13518.

    PubMed  Google Scholar 

  48. Colquhoun HL, Levac D, O’Brien KK, et al. Scoping reviews: time for clarity in definition, methods, and reporting. J Clin Epidemiol. 2014;67:1291–4.

    Article  PubMed  Google Scholar 

  49. Riley RD, Simmonds MC, Look MP. Evidence synthesis combining individual patient data and aggregate data: a systematic review identified current practice and possible methods. J Clin Epidemiol. 2007;60:431–9.

    PubMed  Google Scholar 

  50. Friedenreich CM. Methods for pooled analyses of epidemiologic studies. Epidemiology. 1993;4:295–302.

    Article  CAS  PubMed  Google Scholar 

  51. Caldwell DM, Ades AE, Higgins JP. Simultaneous comparison of multiple treatments: combining direct and indirect evidence. BMJ. 2005;331:897–900.

    Article  PubMed  PubMed Central  Google Scholar 

  52. Song F, Loke YK, Walsh T, et al. Methodological problems in the use of indirect comparisons for evaluating healthcare interventions: survey of published systematic reviews. BMJ. 2009;338:b1147.

    Article  PubMed  PubMed Central  Google Scholar 

  53. Cameron C, Fireman B, Hutton B, et al. Network meta-analysis incorporating randomized controlled trials and non-randomized comparative cohort studies for assessing the safety and effectiveness of medical treatments: challenges and opportunities. Syst Rev. 2015;4:147.

    Article  PubMed  PubMed Central  Google Scholar 

  54. Duval S, Tweedie R. Trim and fill: a simple funnel-plot-based method of testing and adjusting for publication bias in meta-analysis. Biometrics. 2000;56:455–63.

    Article  CAS  PubMed  Google Scholar 

  55. Lyman GH, Kuderer NM. The strengths and limitations of meta-analyses based on aggregate data. BMC Med Res Methodol. 2005;5:14.

    Article  PubMed  PubMed Central  Google Scholar 

  56. Tobías A, Saez M, Kogevinas M. Meta-analysis of results and individual patient data in epidemiologal studies. J Mod Appl Stat Methods. 2004;1:176–85.

    Article  Google Scholar 

  57. Oakes M. On meta-analysis. In: Statistical inference. Chestnut Hill: Epidemiology Resources Inc; 1990. pp. 157–63.

  58. Thompson SG, Pocock SJ. Can meta-analyses be trusted? Lancet. 1991;338:1127–30.

    Article  CAS  PubMed  Google Scholar 

  59. Feinstein AR. Meta-analysis: statistical alchemy for the 21st century. J Clin Epidemiol. 1995;48:71–9.

    Article  CAS  PubMed  Google Scholar 

  60. Egger M, Smith GD. Meta-analysis. Potentials and promise. BMJ. 1997;315:1371–4.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  61. Ioannidis JP, Lau J. Pooling research results: benefits and limitations of meta-analysis. Jt Comm J Qual Improv. 1999;25:462–9.

    CAS  PubMed  Google Scholar 

  62. Charlton BG. The uses and abuses of meta-analysis. Fam Pract. 1996;13:397–401.

    Article  CAS  PubMed  Google Scholar 

  63. Nissen SE, Wolski K. Effect of rosiglitazone on the risk of myocardial infarction and death from cardiovascular causes. N Engl J Med. 2007;356:2457–71.

    Article  CAS  PubMed  Google Scholar 

  64. Hu M, Cappelleri JC, Lan KK. Applying the law of iterated logarithm to control type I error in cumulative meta-analysis of binary outcomes. Clin Trials. 2007;4:329–40.

    Article  PubMed  Google Scholar 

  65. Berlin JA, Golub RM. Meta-analysis as evidence: building a better pyramid. JAMA. 2014;312:603–5.

    Article  CAS  PubMed  Google Scholar 

  66. Higgins JPT. Heterogeneity in meta-analysis should be expected and appropriately quantified. Int J Epidemiol. 2008;37:1158–60.

    Article  PubMed  Google Scholar 

  67. Berlin JA. Benefits of heterogeneity in meta-analysis of data from epidemiologic studies. Am J Epidemiol. 1995;142:383–7.

    Article  CAS  PubMed  Google Scholar 

  68. Takkouche B, Khudyakov P, Costa-Bouzas J, et al. Confidence intervals for heterogeneity measures in meta-analysis. Am J Epidemiol. 2013;178:993–1004.

    Article  PubMed  PubMed Central  Google Scholar 

  69. Higgins JP, Thompson SG. Quantifying heterogeneity in metaanalysis. Stat Med. 2002;21:1539–58.

    Article  PubMed  Google Scholar 

  70. Berlin JA, Santanna J, Schmid CH, et al. Individual patient- versus group-level data meta-regressions for the investigation of treatment effect modifiers: ecological bias rears its ugly head. Stat Med. 2002;21:371–87.

    Article  PubMed  Google Scholar 

  71. Riley RD, Lambert PC, Staessen JA, et al. Meta-analysis of continuous outcomes combining individual patient data and aggregate data. Stat Med. 2008;27:1870–93.

    Article  PubMed  Google Scholar 

  72. Riley RD, Lambert PC, Abo-Zaid G. Meta-analysis of individual participant data: rationale, conduct, and reporting. BMJ. 2010;340:c221.

    Article  PubMed  Google Scholar 

  73. Simmonds MC, Higgins JP, Stewart LA, et al. Meta-analysis of individual patient data from randomized trials: a review of methods used in practice. Clin Trials. 2005;2:209–17.

    Article  PubMed  Google Scholar 

  74. Stewart LA, Parmar MK. Meta-analysis of the literature or of individual patient data: is there a difference? Lancet. 1993;341:418–22.

    Article  CAS  PubMed  Google Scholar 

  75. Stewart LA, Clarke MJ. Practical methodology of meta-analyses (overviews) using updated individual patient data. Cochrane Working Group. Stat Med. 1995;14:2057–79.

    Article  CAS  PubMed  Google Scholar 

  76. Stewart LA, Clarke M, Rovers M, et al. PRISMA-IPD Development Group. Preferred Reporting Items for Systematic Review and Meta-Analyses of individual participant data: the PRISMA-IPD Statement. JAMA. 2015;313:1657–65.

    Article  PubMed  Google Scholar 

  77. Stewart LA, Tierney JF. To IPD or not to IPD? Advantages and disadvantages of systematic reviews using individual patient data. Eval Health Prof. 2002;25:76–97.

    Article  PubMed  Google Scholar 

  78. Burgess S, White IR, Resche-Rigon M, et al. Combining multiple imputation and meta-analysis with individual participant data. Stat Med. 2013;32:4499–514.

    Article  PubMed  PubMed Central  Google Scholar 

  79. Lambert PC, Sutton AJ, Abrams KR, et al. A comparison of summary patient-level covariates in meta-regression with individual patient data meta-analysis. J Clin Epidemiol. 2002;55:86–94.

    Article  CAS  PubMed  Google Scholar 

  80. Ahmed I, Sutton AJ, Riley RD. Assessment of publication bias, selection bias, and unavailable data in meta-analyses using individual participant data: a database survey. BMJ. 2012;344:d7762.

    Article  PubMed  Google Scholar 

  81. Abo-Zaid G, Guo B, Deeks JJ, et al. Individual participant data meta-analyses should not ignore clustering. J Clin Epidemiol. 2013;66(865–873):e4.

    Google Scholar 

  82. Debray TP, Moons KG, van Valkenhoef G, et al. Get real in individual participant data (IPD) meta-analysis: a review of the methodology. Res Synth Methods. 2015;6:293–309.

    Article  PubMed  PubMed Central  Google Scholar 

  83. Wolfson M, Wallace SE, Masca N, et al. DataSHIELD: resolving a conflict in contemporary bioscience–performing a pooled analysis of individual-level data without sharing the data. Int J Epidemiol. 2010;39:1372–82.

    Article  PubMed  PubMed Central  Google Scholar 

  84. Zorzela L, Loke YK, Ioannidis JP, et al. PRISMA harms checklist: improving harms reporting in systematic reviews. BMJ. 2016;352:i157.

    Article  PubMed  Google Scholar 

  85. Patsopoulos NA, Analatos AA, Ioannidis JP. Relative citation impact of various study designs in the health sciences. JAMA. 2005;293:2362–6.

    Article  CAS  PubMed  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Preventive Medicine and Public Health, Facultad de Medicina, University of Santiago de Compostela, c/ San Francisco s/n, 15786, Santiago de Compostela, A Coruna, Spain

    Guillermo Prada-Ramallal, Bahi Takkouche & Adolfo Figueiras

  2. Consortium for Biomedical Research in Epidemiology and Public Health (CIBER en Epidemiología y Salud Pública–CIBERESP), Santiago de Compostela, Spain

    Bahi Takkouche & Adolfo Figueiras

Authors
  1. Guillermo Prada-Ramallal
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Bahi Takkouche
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Adolfo Figueiras
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Adolfo Figueiras.

Ethics declarations

Conflict of interest

Guillermo Prada-Ramallal, Bahi Takkouche and Adolfo Figueiras have no conflicts of interest that are directly relevant to the content of this study.

Funding

No sources of funding were used to assist in the preparation of this study.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Prada-Ramallal, G., Takkouche, B. & Figueiras, A. Summarising the Evidence for Drug Safety: A Methodological Discussion of Different Meta-Analysis Approaches. Drug Saf 40, 547–558 (2017). https://doi.org/10.1007/s40264-017-0518-1

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s40264-017-0518-1

Keywords

Search

Navigation