Skip to main content

Approaches to Assessing and Adjusting for Selective Outcome Reporting in Meta-analysis

Abstract

Background

Selective or non-reporting of study outcomes results in outcome reporting bias.

Objective

We sought to develop and assess tools for detecting and adjusting for outcome reporting bias.

Design

Using data from a previously published systematic review, we abstracted whether outcomes were reported as collected, whether outcomes were statistically significant, and whether statistically significant outcomes were more likely to be reported. We proposed and tested a model to adjust for unreported outcomes and compared our model to three other methods (Copas, Frosi, trim and fill). Our approach assumes that unreported outcomes had a null intervention effect with variance imputed based on the published outcomes. We further compared our approach to these models using simulation, and by varying levels of missing data and study sizes.

Results

There were 286 outcomes reported as collected from 47 included trials: 142 (48%) had the data provided and 144 (52%) did not. Reported outcomes were more likely to be statistically significant than those collected but for which data were unreported and for which non-significance was reported (RR, 2.4; 95% CI, 1.9 to 3.0). Our model and the Copas model provided similar decreases in the pooled effect sizes in both the meta-analytic data and simulation studies. The Frosi and trim and fill methods performed poorly.

Limitations

Single intervention of a single disease with only randomized controlled trials; approach may overestimate outcome reporting bias impact.

Conclusion

There was evidence of selective outcome reporting. Statistically significant outcomes were more likely to be published than non-significant ones. Our simple approach provided a quick estimate of the impact of unreported outcomes on the estimated effect. This approach could be used as a quick assessment of the potential impact of unreported outcomes.

This is a preview of subscription content, access via your institution.

Data Availability

All the data for this study is available via Figshare:7870019.

References

  1. 1.

    Jackson JL, Kuriyama A. From the Editors’ Desk: Bias in Systematic Reviews-Let the Reader Beware. J Gen Intern Med. 2018;33(2):133-135. Not in File. doi:https://doi.org/10.1007/s11606-017-4236-2

    Article  PubMed  Google Scholar 

  2. 2.

    Dwan K, Gamble C, Williamson PR, Kirkham JJ. Systematic review of the empirical evidence of study publication bias and outcome reporting bias - an updated review. PLoS One. 2013;8(7):e66844. Not in File. doi:https://doi.org/10.1371/journal.pone.0066844

    CAS  Article  PubMed  PubMed Central  Google Scholar 

  3. 3.

    Harbord RM, Egger M, Sterne JA. A modified test for small-study effects in meta-analyses of controlled trials with binary endpoints. Stat Med. 2006;25(20):3443-3457. Not in File. https://doi.org/10.1002/sim.2380

    Article  PubMed  Google Scholar 

  4. 4.

    Peters JL, Sutton AJ, Jones DR, Abrams KR, Rushton L. Comparison of two methods to detect publication bias in meta-analysis. JAMA. 2006;295(6):676-680. Not in File. https://doi.org/10.1001/jama.295.6.676

    CAS  Article  PubMed  Google Scholar 

  5. 5.

    Rothstein HR, Sutton AJ, Borenstein M. Publication bias in meta-analysis. Publication bias in meta-analysis: Prevention, assessment and adjustments. John Wiley & Sons; 2005.

  6. 6.

    Chan AW, Hrobjartsson A, Haahr MT, Gotzsche PC, Altman DG. Empirical evidence for selective reporting of outcomes in randomized trials: comparison of protocols to published articles. JAMA. 2004;291(20):2457-2465. Not in File. doi:https://doi.org/10.1001/jama.291.20.2457

    CAS  Article  PubMed  Google Scholar 

  7. 7.

    Dwan K, Gamble C, Kolamunnage-Dona R, Mohammed S, Powell C, Williamson PR. Assessing the potential for outcome reporting bias in a review: a tutorial. Trials. ;11:52. https://doi.org/10.1186/1745-6215-11-52

  8. 8.

    Mathieu S, Boutron I, Moher D, Altman DG, Ravaud P. Comparison of registered and published primary outcomes in randomized controlled trials. JAMA. 302(9):977-984. Not in File. https://doi.org/10.1001/jama.2009.1242

  9. 9.

    Melander H, Ahlqvist-Rastad J, Meijer G, Beermann B. Evidence b(i)ased medicine--selective reporting from studies sponsored by pharmaceutical industry: review of studies in new drug applications. BMJ. 2003;326(7400):1171-1173. Not in File. doi:https://doi.org/10.1136/bmj.326.7400.1171

    Article  PubMed  PubMed Central  Google Scholar 

  10. 10.

    Peters JL, Sutton AJ, Jones DR, Abrams KR, Rushton L. Performance of the trim and fill method in the presence of publication bias and between-study heterogeneity. Stat Med. 26(25):4544-4562. Not in File. https://doi.org/10.1002/sim.2889

  11. 11.

    Rosenthal R. The file drawer problem and tolerance for null results. Psychological Bulletin 1979, 86(3):638-641. Not in File.

    Article  Google Scholar 

  12. 12.

    Egger M, Davey SG, Schneider M, Minder C. Bias in meta-analysis detected by a simple, graphical test. BMJ 1997; 315(7109):629-634. Not in File.

    CAS  Article  Google Scholar 

  13. 13.

    Page MJ, Higgins JP. Rethinking the assessment of risk of bias due to selective reporting: a cross-sectional study. Syst Rev. 2016;5(1):108. Not in File. doi:https://doi.org/10.1186/s13643-016-0289-2

    Article  PubMed  PubMed Central  Google Scholar 

  14. 14.

    Kirkham JJ, Dwan KM, Altman DG, et al. The impact of outcome reporting bias in randomised controlled trials on a cohort of systematic reviews. BMJ. 2010;340:c365. Not in File.

    Article  Google Scholar 

  15. 15.

    Rubin D. Multiple Imputation for Nonresponse in Surveys. Wiley.; 1987.

  16. 16.

    Savage CJ, Vickers AJ. Empirical study of data sharing by authors publishing in PLoS journals. PLoS One. 2009;4(9):e7078. https://doi.org/10.1371/journal.pone.0007078

    CAS  Article  PubMed  PubMed Central  Google Scholar 

  17. 17.

    Copas J, Dwan K, Kirkham J, Williamson P. A model-based correction for outcome reporting bias in meta-analysis. Biostatistics. Apr 2014;15(2):370-83. doi:https://doi.org/10.1093/biostatistics/kxt046

    Article  PubMed  Google Scholar 

  18. 18.

    Frosi G, Riley RD, Williamson PR, Kirkham JJ. Multivariate meta-analysis helps examine the impact of outcome reporting bias in Cochrane rheumatoid arthritis reviews. J Clin Epidemiol. 2015;68(5):542-550. Not in File. https://doi.org/10.1016/j.jclinepi.2014.11.017

    Article  PubMed  Google Scholar 

  19. 19.

    Kirkham JJ, Riley RD, Williamson PR. A multivariate meta-analysis approach for reducing the impact of outcome reporting bias in systematic reviews. Stat Med. 2012;31(20):2179-2195. Not in File. https://doi.org/10.1002/sim.5356

    Article  PubMed  Google Scholar 

  20. 20.

    Kirkham JJ, Altman DG, Chan AW, Gamble C, Dwan KM, Williamson PR. Outcome reporting bias in trials: a methodological approach for assessment and adjustment in systematic reviews. Bmj. 2018;362:k3802. https://doi.org/10.1136/bmj.k3802

  21. 21.

    Copas Adjustment Method. Accessed Feb 16 2020, http://outcome-reporting-bias.ort/HOME/Copas

  22. 22.

    Duval S, Tweedie R. Trim and fill: A simple funnel-plot-based method of testing and adjusting for publication bias in meta-analysis. Biometrics. Jun 2000;56(2):455-63. doi:https://doi.org/10.1111/j.0006-341x.2000.00455.x

    CAS  Article  PubMed  Google Scholar 

  23. 23.

    Jackson JL, Kuriyama A, Kuwatsuka Y, et al. Beta-blockers for the prevention of headache in adults, a systematic review and meta-analysis. PLoS One. 2019;14(3):e0212785. doi:https://doi.org/10.1371/journal.pone.0212785

    CAS  Article  PubMed  PubMed Central  Google Scholar 

  24. 24.

    Silberstein SD, Holland S, Freitag F, Dodick DW, Argoff C, Ashman E. Evidence-based guideline update: pharmacologic treatment for episodic migraine prevention in adults: report of the Quality Standards Subcommittee of the American Academy of Neurology and the American Headache Society. Neurology. 2012;78(17):1337-1345. Not in File. https://doi.org/10.1212/WNL.0b013e3182535d20

    CAS  Article  PubMed  PubMed Central  Google Scholar 

  25. 25.

    Hardy RJ, Thompson SG. A likelihood approach to meta-analysis with random effects. Stat Med. 1996;15(6):619-29. https://doi.org/10.1002/(sici)1097-0258(19960330)15:6<619::Aid-sim188>3.0.Co;2-a

    CAS  Article  PubMed  Google Scholar 

  26. 26.

    Veroniki AA, Jackson D, Viechtbauer W, et al. Methods to estimate the between-study variance and its uncertainty in meta-analysis. Res Synth Methods. 2016;7(1):55-79. doi:https://doi.org/10.1002/jrsm.1164

    Article  PubMed  Google Scholar 

  27. 27.

    Little R. A test of missing completely at random for multivariate data with missing values. Journal of the American Statistical Association. 1988;1988(83):1198-1202.

    Article  Google Scholar 

  28. 28.

    Follmann D, Elliott P, Suh I, Cutler J. Variance imputation for overviews of clinical trials with continuous response. J Clin Epidemiol. 1992;45(7):769-773. Not in File.

    CAS  Article  Google Scholar 

  29. 29.

    Kazis LE, Anderson JJ, Meenan RF. Effect sizes for interpreting changes in health status. Med Care. 1989;27(3 Suppl):S178-S189. Not in File.

    CAS  Article  Google Scholar 

  30. 30.

    Chalmers I. Underreporting research is scientific misconduct. JAMA. 1990;263(10):1405-1408. Not in File.

  31. 31.

    Hahn S, Williamson PR, Hutton JL. Investigation of within-study selective reporting in clinical research: follow-up of applications submitted to a local research ethics committee. J Eval Clin Pract. 2002;8(3):353-359. Not in File.

    CAS  Article  Google Scholar 

  32. 32.

    Norris SL, Holmer HK, Fu R, Ogden LA, Viswanathan MS, Abou-Setta AM. Clinical trial registries are of minimal use for identifying selective outcome and analysis reporting. Res Synth Methods. 2014;5(3):273-284. Not in File. https://doi.org/10.1002/jrsm.1113

    Article  PubMed  Google Scholar 

Download references

Acknowledgements

To Katherine Sherman for discussion about possible statistical approaches for this manuscript.

Author information

Affiliations

Authors

Corresponding author

Correspondence to Jeffrey L Jackson MD MPH.

Ethics declarations

Conflict of Interest

The authors declare that they do not have a conflict of interest.

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

The views in this article should not be construed, in any way, to represent those of the Department of Veterans Affairs.

Supplementary Information

ESM 1

(DOCX 21 kb)

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Jackson, J.L., Balk, E.M., Hyun, N. et al. Approaches to Assessing and Adjusting for Selective Outcome Reporting in Meta-analysis. J GEN INTERN MED (2021). https://doi.org/10.1007/s11606-021-07135-3

Download citation

KEY WORDS

  • Meta-analysis
  • Outcome reporting bias
  • Statistical adjustment