Selective or non-reporting of study outcomes results in outcome reporting bias.
We sought to develop and assess tools for detecting and adjusting for outcome reporting bias.
Using data from a previously published systematic review, we abstracted whether outcomes were reported as collected, whether outcomes were statistically significant, and whether statistically significant outcomes were more likely to be reported. We proposed and tested a model to adjust for unreported outcomes and compared our model to three other methods (Copas, Frosi, trim and fill). Our approach assumes that unreported outcomes had a null intervention effect with variance imputed based on the published outcomes. We further compared our approach to these models using simulation, and by varying levels of missing data and study sizes.
There were 286 outcomes reported as collected from 47 included trials: 142 (48%) had the data provided and 144 (52%) did not. Reported outcomes were more likely to be statistically significant than those collected but for which data were unreported and for which non-significance was reported (RR, 2.4; 95% CI, 1.9 to 3.0). Our model and the Copas model provided similar decreases in the pooled effect sizes in both the meta-analytic data and simulation studies. The Frosi and trim and fill methods performed poorly.
Single intervention of a single disease with only randomized controlled trials; approach may overestimate outcome reporting bias impact.
There was evidence of selective outcome reporting. Statistically significant outcomes were more likely to be published than non-significant ones. Our simple approach provided a quick estimate of the impact of unreported outcomes on the estimated effect. This approach could be used as a quick assessment of the potential impact of unreported outcomes.
This is a preview of subscription content, access via your institution.
Buy single article
Instant access to the full article PDF.
Price includes VAT (USA)
Tax calculation will be finalised during checkout.
All the data for this study is available via Figshare:7870019.
Jackson JL, Kuriyama A. From the Editors’ Desk: Bias in Systematic Reviews-Let the Reader Beware. J Gen Intern Med. 2018;33(2):133-135. Not in File. doi:https://doi.org/10.1007/s11606-017-4236-2
Dwan K, Gamble C, Williamson PR, Kirkham JJ. Systematic review of the empirical evidence of study publication bias and outcome reporting bias - an updated review. PLoS One. 2013;8(7):e66844. Not in File. doi:https://doi.org/10.1371/journal.pone.0066844
Harbord RM, Egger M, Sterne JA. A modified test for small-study effects in meta-analyses of controlled trials with binary endpoints. Stat Med. 2006;25(20):3443-3457. Not in File. https://doi.org/10.1002/sim.2380
Peters JL, Sutton AJ, Jones DR, Abrams KR, Rushton L. Comparison of two methods to detect publication bias in meta-analysis. JAMA. 2006;295(6):676-680. Not in File. https://doi.org/10.1001/jama.295.6.676
Rothstein HR, Sutton AJ, Borenstein M. Publication bias in meta-analysis. Publication bias in meta-analysis: Prevention, assessment and adjustments. John Wiley & Sons; 2005.
Chan AW, Hrobjartsson A, Haahr MT, Gotzsche PC, Altman DG. Empirical evidence for selective reporting of outcomes in randomized trials: comparison of protocols to published articles. JAMA. 2004;291(20):2457-2465. Not in File. doi:https://doi.org/10.1001/jama.291.20.2457
Dwan K, Gamble C, Kolamunnage-Dona R, Mohammed S, Powell C, Williamson PR. Assessing the potential for outcome reporting bias in a review: a tutorial. Trials. ;11:52. https://doi.org/10.1186/1745-6215-11-52
Mathieu S, Boutron I, Moher D, Altman DG, Ravaud P. Comparison of registered and published primary outcomes in randomized controlled trials. JAMA. 302(9):977-984. Not in File. https://doi.org/10.1001/jama.2009.1242
Melander H, Ahlqvist-Rastad J, Meijer G, Beermann B. Evidence b(i)ased medicine--selective reporting from studies sponsored by pharmaceutical industry: review of studies in new drug applications. BMJ. 2003;326(7400):1171-1173. Not in File. doi:https://doi.org/10.1136/bmj.326.7400.1171
Peters JL, Sutton AJ, Jones DR, Abrams KR, Rushton L. Performance of the trim and fill method in the presence of publication bias and between-study heterogeneity. Stat Med. 26(25):4544-4562. Not in File. https://doi.org/10.1002/sim.2889
Rosenthal R. The file drawer problem and tolerance for null results. Psychological Bulletin 1979, 86(3):638-641. Not in File.
Egger M, Davey SG, Schneider M, Minder C. Bias in meta-analysis detected by a simple, graphical test. BMJ 1997; 315(7109):629-634. Not in File.
Page MJ, Higgins JP. Rethinking the assessment of risk of bias due to selective reporting: a cross-sectional study. Syst Rev. 2016;5(1):108. Not in File. doi:https://doi.org/10.1186/s13643-016-0289-2
Kirkham JJ, Dwan KM, Altman DG, et al. The impact of outcome reporting bias in randomised controlled trials on a cohort of systematic reviews. BMJ. 2010;340:c365. Not in File.
Rubin D. Multiple Imputation for Nonresponse in Surveys. Wiley.; 1987.
Savage CJ, Vickers AJ. Empirical study of data sharing by authors publishing in PLoS journals. PLoS One. 2009;4(9):e7078. https://doi.org/10.1371/journal.pone.0007078
Copas J, Dwan K, Kirkham J, Williamson P. A model-based correction for outcome reporting bias in meta-analysis. Biostatistics. Apr 2014;15(2):370-83. doi:https://doi.org/10.1093/biostatistics/kxt046
Frosi G, Riley RD, Williamson PR, Kirkham JJ. Multivariate meta-analysis helps examine the impact of outcome reporting bias in Cochrane rheumatoid arthritis reviews. J Clin Epidemiol. 2015;68(5):542-550. Not in File. https://doi.org/10.1016/j.jclinepi.2014.11.017
Kirkham JJ, Riley RD, Williamson PR. A multivariate meta-analysis approach for reducing the impact of outcome reporting bias in systematic reviews. Stat Med. 2012;31(20):2179-2195. Not in File. https://doi.org/10.1002/sim.5356
Kirkham JJ, Altman DG, Chan AW, Gamble C, Dwan KM, Williamson PR. Outcome reporting bias in trials: a methodological approach for assessment and adjustment in systematic reviews. Bmj. 2018;362:k3802. https://doi.org/10.1136/bmj.k3802
Copas Adjustment Method. Accessed Feb 16 2020, http://outcome-reporting-bias.ort/HOME/Copas
Duval S, Tweedie R. Trim and fill: A simple funnel-plot-based method of testing and adjusting for publication bias in meta-analysis. Biometrics. Jun 2000;56(2):455-63. doi:https://doi.org/10.1111/j.0006-341x.2000.00455.x
Jackson JL, Kuriyama A, Kuwatsuka Y, et al. Beta-blockers for the prevention of headache in adults, a systematic review and meta-analysis. PLoS One. 2019;14(3):e0212785. doi:https://doi.org/10.1371/journal.pone.0212785
Silberstein SD, Holland S, Freitag F, Dodick DW, Argoff C, Ashman E. Evidence-based guideline update: pharmacologic treatment for episodic migraine prevention in adults: report of the Quality Standards Subcommittee of the American Academy of Neurology and the American Headache Society. Neurology. 2012;78(17):1337-1345. Not in File. https://doi.org/10.1212/WNL.0b013e3182535d20
Hardy RJ, Thompson SG. A likelihood approach to meta-analysis with random effects. Stat Med. 1996;15(6):619-29. https://doi.org/10.1002/(sici)1097-0258(19960330)15:6<619::Aid-sim188>3.0.Co;2-a
Veroniki AA, Jackson D, Viechtbauer W, et al. Methods to estimate the between-study variance and its uncertainty in meta-analysis. Res Synth Methods. 2016;7(1):55-79. doi:https://doi.org/10.1002/jrsm.1164
Little R. A test of missing completely at random for multivariate data with missing values. Journal of the American Statistical Association. 1988;1988(83):1198-1202.
Follmann D, Elliott P, Suh I, Cutler J. Variance imputation for overviews of clinical trials with continuous response. J Clin Epidemiol. 1992;45(7):769-773. Not in File.
Kazis LE, Anderson JJ, Meenan RF. Effect sizes for interpreting changes in health status. Med Care. 1989;27(3 Suppl):S178-S189. Not in File.
Chalmers I. Underreporting research is scientific misconduct. JAMA. 1990;263(10):1405-1408. Not in File.
Hahn S, Williamson PR, Hutton JL. Investigation of within-study selective reporting in clinical research: follow-up of applications submitted to a local research ethics committee. J Eval Clin Pract. 2002;8(3):353-359. Not in File.
Norris SL, Holmer HK, Fu R, Ogden LA, Viswanathan MS, Abou-Setta AM. Clinical trial registries are of minimal use for identifying selective outcome and analysis reporting. Res Synth Methods. 2014;5(3):273-284. Not in File. https://doi.org/10.1002/jrsm.1113
To Katherine Sherman for discussion about possible statistical approaches for this manuscript.
Conflict of Interest
The authors declare that they do not have a conflict of interest.
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
The views in this article should not be construed, in any way, to represent those of the Department of Veterans Affairs.
About this article
Cite this article
Jackson, J.L., Balk, E.M., Hyun, N. et al. Approaches to Assessing and Adjusting for Selective Outcome Reporting in Meta-analysis. J GEN INTERN MED (2021). https://doi.org/10.1007/s11606-021-07135-3
- Outcome reporting bias
- Statistical adjustment