Empirical Comparison of Publication Bias Tests in Meta-Analysis

  • Lifeng Lin
  • Haitao Chu
  • Mohammad Hassan Murad
  • Chuan Hong
  • Zhiyong Qu
  • Stephen R. Cole
  • Yong Chen
Original Research

ABSTRACT

Background

Decision makers rely on meta-analytic estimates to trade off benefits and harms. Publication bias impairs the validity and generalizability of such estimates. The performance of various statistical tests for publication bias has been largely compared using simulation studies and has not been systematically evaluated in empirical data.

Methods

This study compares seven commonly used publication bias tests (i.e., Begg’s rank test, trim-and-fill, Egger’s, Tang’s, Macaskill’s, Deeks’, and Peters’ regression tests) based on 28,655 meta-analyses available in the Cochrane Library.

Results

Egger’s regression test detected publication bias more frequently than other tests (15.7% in meta-analyses of binary outcomes and 13.5% in meta-analyses of non-binary outcomes). The proportion of statistically significant publication bias tests was greater for larger meta-analyses, especially for Begg’s rank test and the trim-and-fill method. The agreement among Tang’s, Macaskill’s, Deeks’, and Peters’ regression tests for binary outcomes was moderately strong (most κ’s were around 0.6). Tang’s and Deeks’ tests had fairly similar performance (κ > 0.9). The agreement among Begg’s rank test, the trim-and-fill method, and Egger’s regression test was weak or moderate (κ < 0.5).

Conclusions

Given the relatively low agreement between many publication bias tests, meta-analysts should not rely on a single test and may apply multiple tests with various assumptions. Non-statistical approaches to evaluating publication bias (e.g., searching clinical trials registries, records of drug approving agencies, and scientific conference proceedings) remain essential.

KEY WORDS

Cochrane Library funnel plot meta-analysis publication bias statistical test 

Notes

Compliance with Ethical Standards

Conflict of Interest

The authors declare that they do not have a conflict of interest.

Supplementary material

11606_2018_4425_MOESM1_ESM.pdf (1.1 mb)
ESM 1 (PDF 1104 kb).

REFERENCES

  1. 1.
    Begg CB, Berlin JA. Publication bias: a problem in interpreting medical data. J R Stat Soc Ser A (Stat Soc). 1988;151(3):419–63.CrossRefGoogle Scholar
  2. 2.
    Sutton AJ, Duval SJ, Tweedie RL, et al. Empirical assessment of effect of publication bias on meta-analyses. BMJ. 2000;320(7249):1574–77.CrossRefPubMedPubMedCentralGoogle Scholar
  3. 3.
    Thornton A, Lee P. Publication bias in meta-analysis: its causes and consequences. J Clin Epidemiol. 2000;53(2):207–16.CrossRefPubMedGoogle Scholar
  4. 4.
    Kicinski M. Publication bias in recent meta-analyses. PLoS ONE. 2013;8(11):e81823.CrossRefPubMedPubMedCentralGoogle Scholar
  5. 5.
    Lin L, Chu H. Quantifying publication bias in meta-analysis. Biometrics. 2017.  https://doi.org/10.1111/biom.12817.
  6. 6.
    Eyding D, Lelgemann M, Grouven U, et al. Reboxetine for acute treatment of major depression: systematic review and meta-analysis of published and unpublished placebo and selective serotonin reuptake inhibitor controlled trials. BMJ. 2010;341:c4737.CrossRefPubMedPubMedCentralGoogle Scholar
  7. 7.
    Murad MH, Montori VM, Ioannidis JPA, et al. How to read a systematic review and meta-analysis and apply the results to patient care: users’ guides to the medical literature. JAMA. 2014;312(2):171–79.CrossRefPubMedGoogle Scholar
  8. 8.
    Light RJ, Pillemer DB. Summing Up: The Science of Reviewing Research. Cambridge: Harvard University Press; 1984.Google Scholar
  9. 9.
    Sterne JAC, Egger M. Funnel plots for detecting bias in meta-analysis: guidelines on choice of axis. J Clin Epidemiol. 2001;54(10):1046–55.CrossRefPubMedGoogle Scholar
  10. 10.
    Peters JL, Sutton AJ, Jones DR, et al. Performance of the trim and fill method in the presence of publication bias and between-study heterogeneity. Stat Med. 2007;26(25):4544–62.CrossRefPubMedGoogle Scholar
  11. 11.
    Egger M, Davey Smith G, Schneider M, et al. Bias in meta-analysis detected by a simple, graphical test. BMJ. 1997;315(7109):629–34.CrossRefPubMedPubMedCentralGoogle Scholar
  12. 12.
    Rothstein HR, Sutton AJ, Borenstein M. Publication Bias in Meta-Analysis: Prevention, Assessment and Adjustments. Chichester: Wiley; 2005.CrossRefGoogle Scholar
  13. 13.
    Tang J-L, Liu JLY. Misleading funnel plot for detection of bias in meta-analysis. J Clin Epidemiol. 2000;53(5):477–84.CrossRefPubMedGoogle Scholar
  14. 14.
    Sterne JAC, Gavaghan D, Egger M. Publication and related bias in meta-analysis: power of statistical tests and prevalence in the literature. J Clin Epidemiol. 2000;53(11):1119–29.CrossRefPubMedGoogle Scholar
  15. 15.
    Deeks JJ, Macaskill P, Irwig L. The performance of tests of publication bias and other sample size effects in systematic reviews of diagnostic test accuracy was assessed. J Clin Epidemiol. 2005;58(9):882–93.CrossRefPubMedGoogle Scholar
  16. 16.
    Macaskill P, Walter SD, Irwig L. A comparison of methods to detect publication bias in meta-analysis. Stat Med. 2001;20(4):641–54.CrossRefPubMedGoogle Scholar
  17. 17.
    Peters JL, Sutton AJ, Jones DR, et al. Comparison of two methods to detect publication bias in meta-analysis. JAMA. 2006;295(6):676–80.CrossRefPubMedGoogle Scholar
  18. 18.
    Moreno SG, Sutton AJ, Ades AE, et al. Assessment of regression-based methods to adjust for publication bias through a comprehensive simulation study. BMC Med Res Methodol. 2009;9:2.CrossRefPubMedPubMedCentralGoogle Scholar
  19. 19.
    Bürkner PC, Doebler P. Testing for publication bias in diagnostic meta-analysis: a simulation study. Stat Med. 2014;33(18):3061–77.CrossRefPubMedGoogle Scholar
  20. 20.
    Begg CB, Mazumdar M. Operating characteristics of a rank correlation test for publication bias. Biometrics. 1994;50(4):1088–101.CrossRefPubMedGoogle Scholar
  21. 21.
    Duval S, Tweedie R. A nonparametric “trim and fill” method of accounting for publication bias in meta-analysis. J Am Stat Assoc. 2000;95(449):89–98.Google Scholar
  22. 22.
    Hedges LV. Modeling publication selection effects in meta-analysis. Stat Sci. 1992;7(2):246–55.CrossRefGoogle Scholar
  23. 23.
    Temple LD. RCurl: General Network (HTTP/FTP/...) Client Interface for R. R package version 1.95-4.8, 2016.Google Scholar
  24. 24.
    Walter SD, Cook RJ. A comparison of several point estimators of the odds ratio in a single 2 × 2 contingency table. Biometrics. 1991;47(3):795–811.CrossRefPubMedGoogle Scholar
  25. 25.
    Higgins JPT, Green S. Cochrane Handbook for Systematic Reviews of Interventions. Chichester: Wiley; 2008.CrossRefGoogle Scholar
  26. 26.
    Sweeting MJ, Sutton AJ, Paul LC. What to add to nothing? Use and avoidance of continuity corrections in meta-analysis of sparse data. Stat Med. 2004;23(9):1351–75.CrossRefPubMedGoogle Scholar
  27. 27.
    Bradburn MJ, Deeks JJ, Berlin JA, et al. Much ado about nothing: a comparison of the performance of meta-analytical methods with rare events. Stat Med. 2007;26(1):53–77.CrossRefPubMedGoogle Scholar
  28. 28.
    Cohen J. A coefficient of agreement for nominal scales. Educ Psychosoc Meas. 1960;20(1):37–46.CrossRefGoogle Scholar
  29. 29.
    Landis JR, Koch GG. The measurement of observer agreement for categorical data. Biometrics. 1977;33(1):159–74.CrossRefPubMedGoogle Scholar
  30. 30.
    Ioannidis JPA, Trikalinos TA. The appropriateness of asymmetry tests for publication bias in meta-analyses: a large survey. Can Med Assoc J. 2007;176(8):1091–96.CrossRefGoogle Scholar
  31. 31.
    Sterne JAC, Egger M, Davey Smith G. Investigating and dealing with publication and other biases in meta-analysis. BMJ. 2001;323(7304):101–05.CrossRefPubMedPubMedCentralGoogle Scholar
  32. 32.
    Harbord RM, Egger M, Sterne JAC. A modified test for small-study effects in meta-analyses of controlled trials with binary endpoints. Stat Med. 2006;25(20):3443–57.CrossRefPubMedGoogle Scholar
  33. 33.
    Lin L, Chu H, Hodges JS. Alternative measures of between-study heterogeneity in meta-analysis: reducing the impact of outlying studies. Biometrics. 2017;73(1):156–66.CrossRefPubMedGoogle Scholar
  34. 34.
    Normand S-LT. Meta-analysis: formulating, evaluating, combining, and reporting. Stat Med. 1999;18(3):321–59.CrossRefGoogle Scholar
  35. 35.
    Borenstein M, Hedges LV, Higgins JPT, et al. A basic introduction to fixed-effect and random-effects models for meta-analysis. Res Synth Methods. 2010;1(2):97–111.CrossRefPubMedGoogle Scholar
  36. 36.
    Sterne JAC, Sutton AJ, Ioannidis JPA, et al. Recommendations for examining and interpreting funnel plot asymmetry in meta-analyses of randomised controlled trials. BMJ. 2011;343:d4002.CrossRefPubMedGoogle Scholar
  37. 37.
    Peters JL, Sutton AJ, Jones DR, et al. Assessing publication bias in meta-analyses in the presence of between-study heterogeneity. J R Stat Soc Ser A (Stat Soc). 2010;173(3):575–91.CrossRefGoogle Scholar
  38. 38.
    Higgins JPT, Thompson SG, Deeks JJ, et al. Measuring inconsistency in meta-analyses. BMJ. 2003;327(7414):557–60.CrossRefPubMedPubMedCentralGoogle Scholar
  39. 39.
    Higgins JPT. Commentary: heterogeneity in meta-analysis should be expected and appropriately quantified. Int J Epidemiol. 2008;37(5):1158–60.CrossRefPubMedGoogle Scholar
  40. 40.
    Ioannidis JPA, Patsopoulos NA, Rothstein HR. Reasons or excuses for avoiding meta-analysis in forest plots. BMJ. 2008;336(7658):1413–15.CrossRefPubMedPubMedCentralGoogle Scholar
  41. 41.
    Higgins JPT, Thompson SG. Quantifying heterogeneity in a meta-analysis. Stat Med. 2002;21(11):1539–58.CrossRefPubMedGoogle Scholar
  42. 42.
    Cochran WG. The combination of estimates from different experiments. Biometrics. 1954;10(1):101–29.CrossRefGoogle Scholar
  43. 43.
    Whitehead A, Whitehead J. A general parametric approach to the meta-analysis of randomized clinical trials. Stat Med. 1991;10(11):1665–77.CrossRefPubMedGoogle Scholar
  44. 44.
    Alonso-Coello P, Schünemann HJ, Moberg J, et al. GRADE Evidence to Decision (EtD) frameworks: a systematic and transparent approach to making well informed healthcare choices. 1: Introduction. BMJ. 2016;353:i2016.CrossRefPubMedGoogle Scholar

Copyright information

© Society of General Internal Medicine 2018

Authors and Affiliations

  • Lifeng Lin
    • 1
  • Haitao Chu
    • 2
  • Mohammad Hassan Murad
    • 3
  • Chuan Hong
    • 4
  • Zhiyong Qu
    • 5
  • Stephen R. Cole
    • 6
  • Yong Chen
    • 7
  1. 1.Department of StatisticsFlorida State UniversityTallahasseeUSA
  2. 2.Division of Biostatistics, School of Public HealthUniversity of MinnesotaMinneapolisUSA
  3. 3.Evidence-Based Practice CenterMayo ClinicRochesterUSA
  4. 4.Department of BiostatisticsHarvard School of Public HealthBostonUSA
  5. 5.School of Social Development and Public PolicyBeijing Normal UniversityBeijingChina
  6. 6.Department of EpidemiologyUNC Gillings School of Global Public HealthChapel HillUSA
  7. 7.Department of Biostatistics and EpidemiologyUniversity of PennsylvaniaPhiladelphiaUSA

Personalised recommendations