Skip to main content
SpringerLink
Log in

Worsening file-drawer problem in the abstracts of natural, medical and social science databases

  • Published:
Scientometrics Aims and scope Submit manuscript

Abstract

The file-drawer problem is the tendency of journals to preferentially publish studies with statistically significant results. The problem is an old one and has been documented in various fields, but to my best knowledge there has not been attention to how the issue is developing in a quantitative way through time. In the abstracts of various major scholarly databases (Science and Social Science Citation Index (1991–2008), CAB Abstracts and Medline (1970s–2008), the file drawer problem is gradually getting worse, in spite of an increase in (1) the total number of publications and (2) the proportion of publications reporting both the presence and the absence of significant differences. The trend is confirmed for particular natural science topics such as biology, energy and environment but not for papers retrieved with the keywords biodiversity, chemistry, computer, engineering, genetics, psychology and quantum (physics). A worsening file-drawer problem can be detected in various medical fields (infection, immunology, malaria, obesity, oncology and pharmacology), but not for papers indexed with strings such as AIDS/HIV, epidemiology, health and neurology. An increase in the selective publication of some results against some others is worrying because it can lead to enhanced bias in meta-analysis and hence to a distorted picture of the evidence for or against a certain hypothesis. Long-term monitoring of the file-drawer problem is needed to ensure a sustainable and reliable production of (peer-reviewed) scientific knowledge.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3

Similar content being viewed by others

References

  • Abt, H. A. (1992). Publication practices in various sciences. Scientometrics, 24, 441–447.

    Article  Google Scholar 

  • Begg, C. B., & Berlin, J. A. (1988). Publication bias: A problem in interpreting medical data. Journal of the Royal Statistical Society A, 151, 419–463.

    Article  Google Scholar 

  • Bensman, S. J. (2007). Garfield and the impact factor. Annual Review of Information Science and Technology, 41, 93–155.

    Article  Google Scholar 

  • Bourne, P. E., & Korngreen, A. (2006). Ten simple rules for reviewers. PLoS Computational Biology, 2, e110.

    Article  Google Scholar 

  • Cicchetti, D. V. (1991). The reliability of peer-review for manuscript and grant submissions: A cross-disciplinary investigation. Behavioral and Brain Sciences, 14, 119–134.

    Google Scholar 

  • Csada, R. D., James, P. C., & Espie, R. H. M. (1996). The “file drawer problem” of non-significant results: Does it apply to biological research? Oikos, 76, 591–593.

    Article  Google Scholar 

  • de Mesnard, L. (2010). On Hochberg et al.’s “The tragedy of the reviewer commons”. Scientometrics, in press doi:10.1007/s11192-009-0141-8.

  • Garfield, E. (1997). A statistically valid definition of bias is needed to determine whether the Science Citation Index(R) discriminates against third world journals. Current Science, 73, 639–641.

    Google Scholar 

  • Gerber, A. S., & Malhotra, N. (2008). Publication bias in empirical sociological research: Do arbitrary significance levels distort published results? Sociological Methods & Research, 37, 3–30.

    Article  MathSciNet  Google Scholar 

  • Gilbody, S. M., Song, F., Eastwood, A. J., & Sutton, A. (2000). The causes, consequences and detection of publication bias in psychiatry. Acta Psychiatrica Scandinavica, 102, 241–249.

    Article  Google Scholar 

  • Greenberg, S. A. (2009). How citation distortions create unfounded authority: Analysis of a citation network. British Medical Journal, 339, b2680.

    Article  Google Scholar 

  • Guetzkow, J., Lamont, M., & Mallard, G. (2004). What is originality in the humanities and the social sciences? American Sociological Review, 69, 190–212.

    Article  Google Scholar 

  • Hauser, M., & Fehr, E. (2007). An incentive solution to the peer review problem. PLoS Biology, 5, e107.

    Article  Google Scholar 

  • Hochberg, M. E., Chase, J. M., Gotelli, N. J., Hastings, A., & Naeem, S. (2009). The tragedy of the reviewer commons. Ecology Letters, 12, 2–4.

    Article  Google Scholar 

  • Kennedy, D. (2004). The old file-drawer problem. Science, 305, 451.

    Article  Google Scholar 

  • Khoury, M. J., Bertram, L., Boffetta, P., Butterworth, A. S., Chanock, S. J., Dolan, S. M., et al. (2009). Genome-wide association studies, field synopses, and the development of the knowledge base on genetic variation and human diseases. American Journal of Epidemiology, 170, 269–279.

    Article  Google Scholar 

  • Killeen, P. R. (2005). An alternative to null-hypothesis significance tests. Psychological Science, 16, 345–353.

    Article  Google Scholar 

  • Klein, J. T. (2006). Afterword: The emergent literature on interdisciplinary and transdisciplinary research evaluation. Research Evaluation, 15, 75–80.

    Article  Google Scholar 

  • Koletsi, D., Karagianni, A., Pandis, N., Makou, M., Polychronopolou, A., & Eliades, T. (2009). Are studies reporting significant results more likely to be published? American Journal of Orthodontics and Dentofacial Orthopedics, 136, 632e1.

    Google Scholar 

  • Krzyzanowska, M. K., Pintilie, M., & Tannock, I. F. (2003). Factors associated with failure to publish large randomized trials presented at an oncology meeting. Journal of the American Medical Association, 290, 495–501.

    Article  Google Scholar 

  • Lawrence, P. A. (2003). The politics of publication. Nature, 422, 259–261.

    Article  Google Scholar 

  • Levine, T., Asada, K. J., & Carpenter, C. (2009). Sample sizes and effect sizes are negatively correlated in meta-analyses: Evidence and implications of a publication bias against non-significant findings. Communication Monographs, 76, 286–302.

    Article  Google Scholar 

  • Marsh, H. W., Bornmann, L., Mutz, R., Daniel, H. D., & O’Mara, A. (2009). Gender effects in the peer reviews of grant proposals: A comprehensive meta-analysis comparing traditional and multilevel approaches. Review of Educational Research, 79, 1290–1326.

    Article  Google Scholar 

  • Nakagawa, S., & Cuthill, I. C. (2007). Effect size, confidence interval and statistical significance: A practical guide for biologists. Biological Reviews, 82, 591–605.

    Article  Google Scholar 

  • Nieminen, P., Rucker, G., Miettunen, J., Carpenter, J., & Schumacher, M. (2007). Statistically significant papers in psychiatry were cited more often than others. Journal of Clinical Epidemiology, 60, 939–946.

    Article  Google Scholar 

  • Paris, G., De Leo, G., Menozzi, P., & Gatto, M. (1998). Region-based citation bias in science. Nature, 396, 6708.

    Article  Google Scholar 

  • Pautasso, M., & Pautasso, C. (2010). Peer reviewing interdisciplinary papers. European Review, 18, 227–237.

    Article  Google Scholar 

  • Pautasso, M., & Schäfer, H. (2010). Peer review delay and selectivity in ecology journals. Scientometrics, in press. doi:10.1007/s11192-009-0105-z.

  • Primack, R. B., & Marrs, R. (2008). Bias in the review process. Biological Conservation, 141, 2919–2920.

    Article  Google Scholar 

  • Provenzale, J. M., & Stanley, R. J. (2005). A systematic guide to reviewing a manuscript. American Journal of Radiology, 185, 848–854.

    Google Scholar 

  • Reinhart, M. (2009). Peer review of grant applications in biology and medicine. Reliability, fairness, and validity. Scientometrics, 81, 789–809.

    Article  Google Scholar 

  • Rosenthal, R. (1979). The file drawer problem and tolerance for null results. Psychological Bulletin, 86, 638–641.

    Article  Google Scholar 

  • Schwartz, S. J., & Zamboanga, B. L. (2009). The peer-review and editorial system: Ways to fix something that might be broken. Perspectives in Psychological Science, 4, 54–61.

    Article  Google Scholar 

  • Smith, A. J. (1990). The task of the referee. IEEE Computer, 23, 46–51.

    Google Scholar 

  • Song, F. J., Parekh-Bhurke, S., Hooper, L., Loke, Y. K., Ryder, J. J., Sutton, A. J., et al. (2009). Extent of publication bias in different categories of research cohorts: A meta-analysis of empirical studies. BMC Medical Research Methodology, 9, 79.

    Article  Google Scholar 

  • Sterling, T. D. (1959). Publication decisions and their possible effects on inferences drawn from tests of significance—Or vice versa. Journal of the American Statistical Association, 54, 30–34.

    Article  Google Scholar 

  • Taborsky, M. (2009). Biased citation practice and taxonomic parochialism. Ethology, 115, 105–111.

    Article  Google Scholar 

  • Tricco, A. C., Tetzaff, J., Pham, B., Brehaut, J., & Moher, D. (2009). Non-Cochrane vs. Cochrane reviews were twice as likely to have positive conclusion statements: Cross-sectional study. Journal of Clinical Epidemiology, 62, 380–386.

    Article  Google Scholar 

  • Vecchi, S., Belleudi, V., Amato, L., Davoli, M., & Peducci, C. A. (2009). Does direction of results of abstracts submitted to scientific conferences on drug addiction predict full publication? BMC Medical Research Methodology, 9, 23.

    Article  Google Scholar 

Download references

Acknowledgements

Many thanks to L. Ambrosino, R. Brown, T. Hirsch, O. Holdenrieder, M. Jeger, C. Pautasso, R. Russo and H. Schäfer for insight, discussion or support and to I. Cuthill, O. Holdenrieder, T. Matoni, P. Vineis, K. West and anonymous reviewers for helpful comments on a previous draft.

Author information

Authors and Affiliations

  1. Division of Biology, Imperial College London, Silwood Campus, Ascot, SL5 7PY, UK

    Marco Pautasso

Authors
  1. Marco Pautasso
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Marco Pautasso.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Pautasso, M. Worsening file-drawer problem in the abstracts of natural, medical and social science databases. Scientometrics 85, 193–202 (2010). https://doi.org/10.1007/s11192-010-0233-5

Download citation

  • Received:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11192-010-0233-5

Keywords

Search

Navigation