Many areas of medicine have shown bias towards the publication of studies with positive results. To estimate publication bias in the anesthesia literature, we reviewed all abstracts presented at the American Society of Anesthesiologists (ASA) annual meetings over a four-year period and compared study results (positive vs negative) with publication outcomes.
This review included all abstracts from the 2001-2004 ASA annual meetings performed as randomized-controlled trials in humans. We scored their outcome results as positive or negative and assessed abstract quality using a 13-point scoring system. We then performed a systematic literature search to identify any subsequent publication of the studies and calculated the relative risk (RR) for journal publication by comparing positive vs negative studies.
Of 5,918 abstracts reviewed, 1,052 met inclusion criteria, and 564 (53.6%) of the abstracts proceeded to publication. The RR for abstracts with positive results proceeding to journal publication was 1.42 (95% confidence interval, 1.22 to 1.66; P < 0.001). This result did not change significantly after adjusting for study size and abstract quality score during logistic regression modelling. There was no significant difference in the abstract quality score between positive vs negative studies or between abstracts proceeding vs not proceeding to publication.
Approximately half of the ASA annual meeting abstracts proceed to publication. After adjustment for study quality and size, abstracts with positive results were more likely to proceed to journal publication than those with negative results, suggesting publication bias in the anesthesia literature.
De nombreux domaines de la médecine font preuve d’un biais en faveur de la publication d’études rapportant des résultats positifs. Afin d’estimer le biais de publication dans la littérature anesthésique, nous avons passé en revue tous les résumés présentés lors des congrès annuels de l’American Society of Anesthesiologists (ASA) sur une période de quatre ans et comparé les résultats des études (positifs vs négatifs) avec la publication.
Ce compte rendu a inclus tous les résumés d’études randomisées contrôlées chez l’humain présentés lors des congrès annuels de l’ASA de 2001 à 2004. Nous avons noté les résultats des études en positif ou négatif et évalué la qualité des résumés selon un système de notation à 13 points. Nous avons ensuite effectué une recherche méthodique de la littérature afin d’identifier toute publication subséquente des études et calculé le risque relatif (RR) de publication dans une revue en comparant les études positives vs négatives.
Parmi les 5918 résumés passés en revue, 1052 respectaient nos critères d’inclusion, et 564 (53,6 %) résumés ont été publiés. Le RR des résumés présentant des résultats positifs d’être publiés dans une revue était de 1,42 (intervalle de confiance 95 %, 1,22 à 1,66; P < 0,001). Ce résultat n’a pas changé de façon significative après ajustement pour tenir compte de la taille de l’étude et de la note de qualité des résumés dans un modèle à régression logistique. Aucune différence significative dans la note de qualité des résumés entre les études positives vs négatives ou entre les résumés étant publiés ou non n’a été observée.
Environ la moitié des résumés présentés lors des congrès annuels de l’ASA sont publiés. Après ajustements pour tenir compte de la qualité et de la taille des études, il était plus probable que les résumés rapportant des résultats positifs soient publiés dans une revue que les résumés présentant des résultats négatifs, ce qui laisse à penser qu’il existe un biais de publication dans la littérature d’anesthésie.
The relationship between study findings and publication rates in the scientific literature has been a longstanding topic of concern in medicine. The influence of publication bias –i.e., the selective submission and acceptance for publication of studies with positive over those with negative results–1 on this relationship has attracted particular attention. Studies have shown that the preponderance of studies published widely in medicine are those with positive findings.1-5 The presence of publication bias can potentially lead to clinical decision making on the basis of an incomplete and unreliable evidence base and has been shown to influence the findings in meta-analysis.6 There is also the potential for unnecessary financial cost and risk of patient harm from conducting clinical trials that are valid but do not progress to publication because of their negative results.
In the field of anesthesia, there has been limited investigation into the relationship between study findings and publication rates or the potential influence of publication bias on this relationship.7 To test our hypothesis that significant publication bias exists in the anesthesia literature, we first conducted a review of study findings in all randomized-controlled trials (RCTs) presented and published as conference abstracts at the American Society of Anesthesiologists (ASA) annual meeting over a four-year period. We then compared these findings with subsequent publication outcome in peer-reviewed journals. The ASA annual meeting is the largest worldwide educational event in anesthesiology, currently attracting over 15,000 attendees from 90 countries, and therefore, we would expect the abstracts presented at these meetings to be held to a high standard and reflect the wider community of anesthesia research. The purpose of our study was to compare the results of abstracts presented at the 2001-2004 ASA annual meetings and determine if those reporting positive findings were published more frequently as full manuscripts in journals indexed in the MEDLINE® database than those that reported negative findings.
Ethics board approval was not required for this retrospective observational study. All abstracts from the 2001-2004 ASA annual meeting were identified from the ASA Abstract Archive (Fig. 1). We chose this time span to allow at least a ten-year lapse pending subsequent publication in the peer-reviewed literature, particularly as it has been shown that studies with negative findings can take an average of eight years to be published.8
All abstracts performed as RCTs in humans were included in a database developed using Microsoft Excel 2010 (Microsoft Corporation, Redmond, WA, USA). A trial was considered an RCT if the participants were prospectively assigned to one of at least two alternative forms of treatment using a random method of allocation.9 We first assessed the study findings as either “positive” or “negative”. For the purposes of our study, we defined positive studies as those showing a statistically significant difference in the direction of the experimental treatment for the primary outcome compared with the control treatment in the abstract. Studies showing no difference in outcome between treatment groups were counted as positive if the stated objective was to show treatment equivalence or non-inferiority. We defined negative studies as those that failed to show either a statistically significant difference in the direction of the experimental treatment compared with the control treatment or equivalence or non-inferiority if that was the stated aim. Some abstracts comparing interventions had no clear “experimental” or “control” group, but their findings were considered positive if a difference toward the investigational treatment was shown between the groups. Four co-authors (S.C., N.C., C.W., G.L.) reviewed and entered information about the abstracts into the database. In cases of initial uncertainty over whether the abstract findings were positive or negative, two authors (S.C. and N.C.) discussed the findings and reached a consensus decision guided by the above definitions.
A systematic literature search was then performed via PubMed and MEDLINE to identify any subsequent publication of the study. The search strategy involved conducting seven separate searches in the order of the first author’s name, the second author’s name, the last author’s name, the first author’s name AND keywords, the second author’s name AND keywords, the last author’s name AND keywords, and keywords alone.
The following baseline characteristics were recorded: the month and year of abstract presentation at the ASA meeting, the month and year of journal publication (if published), author details, and reported study size. We compared the sample sizes in the meeting abstract with those in the subsequent publication to determine whether the study had completed full recruitment at the time of conference presentation. If the number of participants in the final journal publication differed from that in the conference abstract and this discrepancy was not explicitly stated in the final study, we classified the abstract as incomplete at the time of conference presentation.
We assessed the quality of the conference abstract using a scoring system of 13 variables (see Appendix) which we adapted from a checklist that Hopewell et al. 10 had previously devised according to existing reporting standards.11 For those studies proceeding to journal publication, we also recorded the period of time from the date of conference abstract presentation to the date of publication.
Statistical analyses were performed using Microsoft Excel 2010 (Microsoft Corporation, Redmond, WA, USA) and Stata® 12 (StataCorp LP, College Station, TX, USA). Abstract scores, being ordinal data, were compared between published and unpublished studies as well as between positive and negative studies using ordered logistic regression. This was also performed as a multivariable analysis incorporating publication status and positivity/negativity as well their interaction term. A similar comparison was made for study size using Mood’s Median Test, with difference and bias-corrected 95% confidence intervals (CI) for medians calculated using a bootstrap procedure.
The primary endpoint was the relative risk (RR) between positive and negative studies for journal publication. This was calculated as a raw RR and also as a RR adjusted for study size and abstract score. We used a multivariable logistic regression model for the calculation using the ratio of predicted proportions for study findings with the bias-corrected 95% CI obtained from bootstrapping. As a secondary analysis, we compared times to publication between positive and negative studies using Cox regression modelling, including adjustments for the same covariates and use of Kaplan-Meier survival curves.
We calculated our sample size on conservative assumptions about the effect being measured and a baseline publication rate of 50%.2 If the publication rate for studies with positive findings is 50%, a sample size of 1,076 studies would provide 90% power to show a 20% relative lower rate of publication for studies with negative findings with an alpha of 5%.
Of the 5,918 conference abstracts reviewed, 1,052 met inclusion criteria as RCTs in humans. There were 564 (53.6%) abstracts that subsequently proceeded to publication (Table 1). The RR for abstracts with positive findings proceeding to journal publication was 1.42 (95% CI, 1.22 to 1.66; P < 0.001). These data are shown in Table 2.
Study sizes and abstract scores were not normally distributed on the Shapiro-Wilk test. The median (95% CI) study size for conference abstracts was larger for those proceeding vs not proceeding to journal publication [50 (45 to 50) vs 40 (37 to 42), respectively; P = 0.001]. There was no difference in median (95% CI) study size between positive and negative studies [44 (40 to 48) vs 44 (40 to 50), respectively; P = 0.992].
The median [interquartile range (IQR)] abstract quality scores for studies proceeding vs not proceeding to journal publication were 8 (8 to 9) vs 8 (7 to 9), respectively; P = 0.084 (Table 1). Similarly, there was no difference in median [IQR] abstract score between positive and negative studies (8 [8 - 9] vs 9 [8 - 9], respectively; P = 0.226). Using the 13-point abstract quality scoring system, scores of ≤ 7 and ≥ 10 were each collapsed into a single category due to low numbers or empty cells for low and high scores. The resultant four-level ordinal abstract score categories were used for the purposes of all subsequent statistical analyses.
Logistic regression analysis of the primary endpoint was performed with journal publication outcome as the binary dependent variable and with the binary study findings, study size (divided by 10) and abstract score category, as the independent variables. On univariable analysis, study size (divided by 10) was a statistically significant but weak predictor of publication outcome [odds ratio (OR), 1.02; 95% CI, 1.00 to 1.03; P = 0.009]. The highest ranking abstract quality score (10 or more out of 13) associated with journal publication was compared with the lowest ranking score (7 or less out of 13) (OR, 1.60; 95% CI, 1.05 to 2.48; P = 0.029). Nevertheless, using a likelihood ratio test, the overall abstract score was not associated with the overall publication outcome (P = 0.146). In multivariable logistic regression modelling, the RR for publication outcome related to a positive study finding was largely unchanged (RR, 1.42; 95% CI, 1.24 to 1.65; P < 0.001) after adjustment for these covariates, as was their significance as predictors of publication outcome (Table 2).
On multivariable ordered logistic regression analysis using publication status, study conclusions, and their interaction term as covariates, neither interaction (OR, 0.85; 95% CI, 0.52 to 1.42; P = 0.533) nor study conclusions (OR, 0.85; 95% CI, 0.62 to 1.18; P = 0.329) were significant predictors of abstract score category. Publication outcome was a marginally non-significant predictor of abstract score category (OR, 1.47; 95% CI, 0.96 to 2.26; P = 0.076).
The time to publication ranged from one month to a maximum of 131 months (Fig. 2). The median (95% CI) time from the date of the conference abstract presentation to publication for studies with positive findings was 16 (15 to 17) mth compared with 17 (13 to 20) mth for studies with negative findings (P = 1.00 on Mood’s Median Test) (Table 3). On Cox regression modelling, for each increase in study size of ten patients, study size was a statistically significant predictor of time to publication [hazard ratio (HR), 1.003; 95% CI, 1.001 to 1.005; P = 0.011]. Nevertheless, study finding (HR, 1.10; 95% CI, 0.90 to 1.36; P = 0.351) and abstract score category (as determined using a likelihood ratio test, P = 0.862) were not statistically significant predictors of time to publication.
Based on differences in sample size between the conference abstract and the subsequent journal publication, we classified 243 of 564 (43%) studies as incomplete at the conference abstract phase (Table 1). This figure was 39% for studies with negative findings and 44% for studies with positive findings (RR, 0.85; 95% CI, 0.61 to 1.18; P = 0.337). The differences in median (95% CI) study size between conference abstract and published study were similar [19 (12 to 32) for studies published with negative findings and 13 (10 to 15) for studies published with positive findings; P = 0.253].
From a review of 1,052 abstracts of RCTs presented at the largest major international scientific meeting in anesthesia, we found that approximately half eventually proceed to publication (in the subsequent ten years) in peer-reviewed journals. Studies with positive findings were significantly more likely to proceed to journal publication than those with negative findings. This result was unchanged by adjustment for the relatively modest differences in study size between published and unpublished studies and in study quality as estimated by abstract quality score.
This result suggests the likelihood of significant positive publication bias in the anesthesia literature during the period we examined. Publication bias has been shown to exist in many areas of medicine.1,2,4,5 A Cochrane review in 200812 examined 79 reports on the publication of results that were initially presented as abstracts at scientific meetings in various disciplines.12 Abstracts that presented positive results had a significantly higher publication rate, with a risk ratio of 1.28 (95% CI, 1.15 to 1.42), compared with an adjusted RR of 1.42 (95% CI, 1.24 to 1.65) in our study. The authors concluded that there was clear evidence of publication bias in the step between abstract presentation at scientific meetings and subsequent full publication of results, which is consistent with our findings.
De Oliveira et al. specifically investigated bias in the publication of RCTs in anesthesia.7 The authors identified 1,163 RCTs in humans published in 14 anesthesia journals during the period 2008-2009. They identified positive findings in 72% of the articles in the four journals with the highest impact factors compared with 53% in journals with lower impact. If the article had positive results, the odds ratio for publication in a high impact anesthesia journal was 2.28 (95% CI, 1.76 to 3.01), which is similar to the odds ratio calculated from our data (OR, 2.01; 95% CI, 1.53 to 2.66). Nevertheless, it is difficult to compare our findings directly with theirs, as the discriminator used in their methodology (journal impact factor) differed from ours. In addition, the findings of De Oliveira et al. would more narrowly reflect bias in the submission and review process.
Among studies in other disciplines looking for evidence of bias earlier in the research process, Stern and Simes found that, of 130 clinical trials submitted to the ethics committee at a major hospital, those with positive results were much more likely to be published than those with negative results (HR, 3.13), with median times to publication of 4.7 yr and 8.0 yr, respectively, from the commencement date of the study.13 In contrast to the findings of De Oliveira et al., van Lent et al. reviewed over 15,000 RCTs that were submitted to eight major medical journals to determine if trials with positive results were more likely to be published than trials with negative results.14 The authors found no difference, and they concluded that, in the case of the journals studied, publication bias occurred mainly prior to submission.
It is interesting, therefore, that we found no significant difference in time to journal publication between studies with positive vs negative findings, which suggests that any bias occurred mainly prior to this stage. Indeed, positive publication bias may occur at many points in the conduct of a study, and previous authors have shown that failure by researchers to pursue a study to completion and full publication is common and heavily influenced by study findings. Dickersin et al. surveyed 318 authors of published trials and asked whether they had participated in any unpublished RCTs.1 From the responses gathered, non-publication was primarily due to failure to submit the trial for publication as opposed to rejection of submitted papers. The 156 respondents reported 278 unpublished RCTs; 204 of these studies were completed and 74 were stopped early. In 102 of these studies, a report was not submitted. The new therapy was favoured in 14% of the completed unpublished RCTs compared with 55% of the published RCTs. The findings of our study may reflect this broader aspect of publication failure related to study findings; however, our analysis cannot identify publication bias prior to the conference abstract phase.
Our study has other limitations. While we found no clinically significant difference in study quality between the conference abstract and publication score, we could not entirely rule out systematic differences in study power or design quality between negative and positive study findings that influenced publication success. It is likely, for instance, that some research presented at conferences may not have been intended for subsequent publication. Nevertheless, there are no a priori reasons why this should be reflected in a different distribution of positive and negative findings, other than those contributing to publication bias which we have discussed. Because of the heterogeneous nature of the studies in our sample, we were able to use study size only as an indicator of the relative adequacy of study power. The median study size was almost 20% smaller in unpublished conference abstracts compared with those that proceeded to journal publication. Nevertheless, this modest difference is unlikely to explain our results, and the relationship of study findings to publication rate remained strong after adjustment for study size and abstract quality using logistic regression. In addition, this difference in study size could also be due to a lower study completion rate among the unpublished studies. We classified 43% of studies that went on to publication as incomplete at the conference abstract phase, and it is unlikely that this proportion would be higher among the unpublished studies. While we did not attempt follow-up of studies that were never subsequently published after conference presentation, it is possible that this factor also influenced the size of these unpublished studies, but there is no evidence that it explains the preponderance of positive studies reaching publication.
We chose an arbitrary time span of 2001-2004 for review of conference abstracts to allow an adequate time period for studies to be published in the literature. For example, some abstracts we reviewed proceeded to publication as late as 2012 and 2013.15,16 Scherer et al. showed that approximately 8% of abstracts of RCTs in medicine that proceeded to publication were published five to nine years after abstract presentation.12
The suggestion from our findings that positive publication bias occurs prior to the journal submission and peer review process indicates a degree of self-censorship in research, which gives cause for concern. Chalmers has stated that failure to report the findings of valid trials in the literature constitutes a form of scientific misconduct.17 Over the past few decades, many potential solutions have been put forward to address the issue of publication bias, including the development of compulsory clinical trial registries1 and greater regulatory input from research and ethics boards and journals.4,18 Early registration of clinical trials as a prerequisite for acceptance by medical journals may have become more prevalent since 2004. The International Committee of Medical Journal Editors released a statement in 2004 encouraging this practice.19 Nevertheless, in a recent study, Wager et al. 20 revealed that a large number of journals, especially smaller journals, still do not require prospective registration of RCTs as a prerequisite for publication. We await evidence of the effectiveness of these proposed solutions in reducing publication bias. Furthermore, literature reviews such as this one are warranted in all disciplines to monitor this practice in the future.
In conclusion, data on publication outcomes of studies presented at a major anesthesia meeting suggest that positive publication bias may be a significant problem in anesthesia. Regardless of trial outcome, investigators should strive to complete and publish the results of valid trials. Despite evidence that the quality of research published in anesthesia journals has improved over the last 25-30 years,21,22 effective strategies for minimizing publication bias are important to protect the quality of evidence in our field.
Dickersin K, Chan S, Chalmers TC, Sacks HS, Smith H Jr. Publication bias and clinical trials. Control Clin Trials 1987; 8: 343-53.
Scherer RW, Dickersin K, Langenberg P. Full publication of results initially presented in abstracts. A meta-analysis. JAMA 1994; 272: 158-62.
Ioannidis JP. Effect of the statistical significance of results on the time to completion and publication of randomized efficacy trials. JAMA 1998; 279: 281-6.
Hall R, de Antueno C, Webber A; Canadian Research Ethics Board. Publication bias in the medical literature: a review by a Canadian Research Ethics Board. Can. J Anesth 2007; 54: 380-8.
Turner EH, Matthews AM, Linardatos E, Tell RA, Rosenthal R. Selective publication of antidepressant trials and its influence on apparent efficacy. N Engl J Med 2008; 358: 252-60.
Sutton AJ, Duval SJ, Tweedie RL, Abrams KR, Jones DR. Empirical assessment of effect of publication bias on meta-analyses. BMJ 2000; 320: 1574-7.
De Oliveira GS, Jr Chang R, Kendall MC, Fitzgerald PC, McCarthy RJ. Publication bias in the anesthesiology literature. Anesth Analg 2012; 114: 1042-8.
Hopewell S, Clarke M, Stewart L, Tierney J. Time to publication for results of clinical trials. Cochrane Database Syst Rev 2007; 18: Mr000011.
Lefebvre C, Manheimer E, Glanville J; Cochrane Information Retrieval Methods Group. Chapter 6: Searching for studies. In: Higgins JP, Green S (Eds). Cochrane Handbook for Systematic Reviews of Interventions; 2011.
Hopewell S, Clarke M, Askie L. Reporting of trials presented in conference abstracts needs to be improved. J Clin Epidemiol 2006; 59: 681-4.
Moher D, Schulz KF, Altman DG. The CONSORT statement: revised recommendations for improving the quality of reports of parallel-group randomised trials. Lancet 2001; 357: 1191-4.
Scherer RW, Langenberg P, von Elm E. Full publication of results initially presented in abstracts. Cochrane Database Syst Rev 2007; 18: MR000005.
Stern JM, Simes RJ. Publication bias: evidence of delayed publication in a cohort study of clinical research projects. BMJ 1997; 315: 640-5.
van Lent M, Overbeke J, Out HJ. Role of editorial and peer review processes in publication bias: analysis of drug trials submitted to eight medical journals. PLoS One 2014; 9: e104846.
Carvalho B, Drover DR, Ginosar Y, Cohen SE, Riley ET. Intrathecal fentanyl added to bupivacaine and morphine for cesarean delivery may induce a subtle acute opioid tolerance. Int J Obstet Anesth 2012; 21: 29-34.
Pang W, Liu YC, Maboudou E, et al. Metoclopramide improves the quality of tramadol PCA indistinguishable to morphine PCA: a prospective, randomized, double blind clinical comparison. Pain Med 2013; 14: 1426-34.
Chalmers I. Underreporting research is scientific misconduct. JAMA 1990; 263: 1405-8.
Weibel S, Elia N, Kranke P. The transparent clinical trial: why we need complete and informative prospective trial registration. Eur J Anaesthesiol 2016; 33: 72-4.
DeAngelis CD, Drazen JM, Frizelle FA, et al. Clinical trial registration: a statement from the International Committee of Medical Journal Editors. JAMA 2004; 292: 1363-4.
Wager E, Williams P, OPEN Project (Overcome failure to Publish nEgative fiNdings) Consortium. “Hardly worth the effort”? Medical journals’ policies and their editors’ and publishers’ views on trial registration and publication bias: quantitative and qualitative study. BMJ 2013; 347: f5248.
Teoh DC, Schramm B. Changes in clinical research in anaesthesia and intensive care from 1974-2004. Anaesth Intensive Care 2006; 34: 753-7.
Langford RA, Huang GH, Leslie K. Quality and reporting of trial design in scientific papers in Anaesthesia over 25 years. Anaesthesia 2009; 64: 60-4.
The authors sincerely thank Rachel Sore MSc GStat, Statistical Consultant, Statistical Consulting Centre, The University of Melbourne, Melbourne, Australia, for her assistance with statistical analysis.
No funding received.
Conflicts of interest
Simon W. Chong, Neil F. Collins, and Philip J. Peyton helped with the design and conduct of the study, data analysis, and manuscript preparation. Simon W. Chong, Neil F. Collins, Philip J. Peyton, Christine Y. Wu, and Grace M. Liskaser helped with data collection.
This submission was handled by Dr. Hilary P. Grocott, Editor-in-Chief, Canadian Journal of Anesthesia.
This article is accompanied by an editorial. Please see Can J Anesth 2016; 63: this issue.
Appendix: Abstract quality scoring system
Appendix: Abstract quality scoring system
|Scoring of Abstract Quality (13 Criteria)|
|1.||Study objectives described|
|2.||Date of trial given|
|4.||Inclusion / exclusion criteria described|
|5.||Experimental intervention described|
|6.||Comparator intervention described|
|7.||Baseline characteristics of the patient groups described|
|8.||Primary outcome measure described|
|9.||Number of patients analyzed stated|
|10.||Intention-to-treat principle described|
|11.||Important adverse effects described|
|13.||Primary conclusions described|
About this article
Cite this article
Chong, S.W., Collins, N.F., Wu, C.Y. et al. The relationship between study findings and publication outcome in anesthesia research: a retrospective observational study examining publication bias. Can J Anesth/J Can Anesth 63, 682–690 (2016). https://doi.org/10.1007/s12630-016-0631-0
- Publication Bias
- Negative Finding
- Study Size
- Journal Publication
- Publication Rate