Abstract
The file-drawer problem is the tendency of journals to preferentially publish studies with statistically significant results. The problem is an old one and has been documented in various fields, but to my best knowledge there has not been attention to how the issue is developing in a quantitative way through time. In the abstracts of various major scholarly databases (Science and Social Science Citation Index (1991–2008), CAB Abstracts and Medline (1970s–2008), the file drawer problem is gradually getting worse, in spite of an increase in (1) the total number of publications and (2) the proportion of publications reporting both the presence and the absence of significant differences. The trend is confirmed for particular natural science topics such as biology, energy and environment but not for papers retrieved with the keywords biodiversity, chemistry, computer, engineering, genetics, psychology and quantum (physics). A worsening file-drawer problem can be detected in various medical fields (infection, immunology, malaria, obesity, oncology and pharmacology), but not for papers indexed with strings such as AIDS/HIV, epidemiology, health and neurology. An increase in the selective publication of some results against some others is worrying because it can lead to enhanced bias in meta-analysis and hence to a distorted picture of the evidence for or against a certain hypothesis. Long-term monitoring of the file-drawer problem is needed to ensure a sustainable and reliable production of (peer-reviewed) scientific knowledge.
Similar content being viewed by others
References
Abt, H. A. (1992). Publication practices in various sciences. Scientometrics, 24, 441–447.
Begg, C. B., & Berlin, J. A. (1988). Publication bias: A problem in interpreting medical data. Journal of the Royal Statistical Society A, 151, 419–463.
Bensman, S. J. (2007). Garfield and the impact factor. Annual Review of Information Science and Technology, 41, 93–155.
Bourne, P. E., & Korngreen, A. (2006). Ten simple rules for reviewers. PLoS Computational Biology, 2, e110.
Cicchetti, D. V. (1991). The reliability of peer-review for manuscript and grant submissions: A cross-disciplinary investigation. Behavioral and Brain Sciences, 14, 119–134.
Csada, R. D., James, P. C., & Espie, R. H. M. (1996). The “file drawer problem” of non-significant results: Does it apply to biological research? Oikos, 76, 591–593.
de Mesnard, L. (2010). On Hochberg et al.’s “The tragedy of the reviewer commons”. Scientometrics, in press doi:10.1007/s11192-009-0141-8.
Garfield, E. (1997). A statistically valid definition of bias is needed to determine whether the Science Citation Index(R) discriminates against third world journals. Current Science, 73, 639–641.
Gerber, A. S., & Malhotra, N. (2008). Publication bias in empirical sociological research: Do arbitrary significance levels distort published results? Sociological Methods & Research, 37, 3–30.
Gilbody, S. M., Song, F., Eastwood, A. J., & Sutton, A. (2000). The causes, consequences and detection of publication bias in psychiatry. Acta Psychiatrica Scandinavica, 102, 241–249.
Greenberg, S. A. (2009). How citation distortions create unfounded authority: Analysis of a citation network. British Medical Journal, 339, b2680.
Guetzkow, J., Lamont, M., & Mallard, G. (2004). What is originality in the humanities and the social sciences? American Sociological Review, 69, 190–212.
Hauser, M., & Fehr, E. (2007). An incentive solution to the peer review problem. PLoS Biology, 5, e107.
Hochberg, M. E., Chase, J. M., Gotelli, N. J., Hastings, A., & Naeem, S. (2009). The tragedy of the reviewer commons. Ecology Letters, 12, 2–4.
Kennedy, D. (2004). The old file-drawer problem. Science, 305, 451.
Khoury, M. J., Bertram, L., Boffetta, P., Butterworth, A. S., Chanock, S. J., Dolan, S. M., et al. (2009). Genome-wide association studies, field synopses, and the development of the knowledge base on genetic variation and human diseases. American Journal of Epidemiology, 170, 269–279.
Killeen, P. R. (2005). An alternative to null-hypothesis significance tests. Psychological Science, 16, 345–353.
Klein, J. T. (2006). Afterword: The emergent literature on interdisciplinary and transdisciplinary research evaluation. Research Evaluation, 15, 75–80.
Koletsi, D., Karagianni, A., Pandis, N., Makou, M., Polychronopolou, A., & Eliades, T. (2009). Are studies reporting significant results more likely to be published? American Journal of Orthodontics and Dentofacial Orthopedics, 136, 632e1.
Krzyzanowska, M. K., Pintilie, M., & Tannock, I. F. (2003). Factors associated with failure to publish large randomized trials presented at an oncology meeting. Journal of the American Medical Association, 290, 495–501.
Lawrence, P. A. (2003). The politics of publication. Nature, 422, 259–261.
Levine, T., Asada, K. J., & Carpenter, C. (2009). Sample sizes and effect sizes are negatively correlated in meta-analyses: Evidence and implications of a publication bias against non-significant findings. Communication Monographs, 76, 286–302.
Marsh, H. W., Bornmann, L., Mutz, R., Daniel, H. D., & O’Mara, A. (2009). Gender effects in the peer reviews of grant proposals: A comprehensive meta-analysis comparing traditional and multilevel approaches. Review of Educational Research, 79, 1290–1326.
Nakagawa, S., & Cuthill, I. C. (2007). Effect size, confidence interval and statistical significance: A practical guide for biologists. Biological Reviews, 82, 591–605.
Nieminen, P., Rucker, G., Miettunen, J., Carpenter, J., & Schumacher, M. (2007). Statistically significant papers in psychiatry were cited more often than others. Journal of Clinical Epidemiology, 60, 939–946.
Paris, G., De Leo, G., Menozzi, P., & Gatto, M. (1998). Region-based citation bias in science. Nature, 396, 6708.
Pautasso, M., & Pautasso, C. (2010). Peer reviewing interdisciplinary papers. European Review, 18, 227–237.
Pautasso, M., & Schäfer, H. (2010). Peer review delay and selectivity in ecology journals. Scientometrics, in press. doi:10.1007/s11192-009-0105-z.
Primack, R. B., & Marrs, R. (2008). Bias in the review process. Biological Conservation, 141, 2919–2920.
Provenzale, J. M., & Stanley, R. J. (2005). A systematic guide to reviewing a manuscript. American Journal of Radiology, 185, 848–854.
Reinhart, M. (2009). Peer review of grant applications in biology and medicine. Reliability, fairness, and validity. Scientometrics, 81, 789–809.
Rosenthal, R. (1979). The file drawer problem and tolerance for null results. Psychological Bulletin, 86, 638–641.
Schwartz, S. J., & Zamboanga, B. L. (2009). The peer-review and editorial system: Ways to fix something that might be broken. Perspectives in Psychological Science, 4, 54–61.
Smith, A. J. (1990). The task of the referee. IEEE Computer, 23, 46–51.
Song, F. J., Parekh-Bhurke, S., Hooper, L., Loke, Y. K., Ryder, J. J., Sutton, A. J., et al. (2009). Extent of publication bias in different categories of research cohorts: A meta-analysis of empirical studies. BMC Medical Research Methodology, 9, 79.
Sterling, T. D. (1959). Publication decisions and their possible effects on inferences drawn from tests of significance—Or vice versa. Journal of the American Statistical Association, 54, 30–34.
Taborsky, M. (2009). Biased citation practice and taxonomic parochialism. Ethology, 115, 105–111.
Tricco, A. C., Tetzaff, J., Pham, B., Brehaut, J., & Moher, D. (2009). Non-Cochrane vs. Cochrane reviews were twice as likely to have positive conclusion statements: Cross-sectional study. Journal of Clinical Epidemiology, 62, 380–386.
Vecchi, S., Belleudi, V., Amato, L., Davoli, M., & Peducci, C. A. (2009). Does direction of results of abstracts submitted to scientific conferences on drug addiction predict full publication? BMC Medical Research Methodology, 9, 23.
Acknowledgements
Many thanks to L. Ambrosino, R. Brown, T. Hirsch, O. Holdenrieder, M. Jeger, C. Pautasso, R. Russo and H. Schäfer for insight, discussion or support and to I. Cuthill, O. Holdenrieder, T. Matoni, P. Vineis, K. West and anonymous reviewers for helpful comments on a previous draft.
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Pautasso, M. Worsening file-drawer problem in the abstracts of natural, medical and social science databases. Scientometrics 85, 193–202 (2010). https://doi.org/10.1007/s11192-010-0233-5
Received:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11192-010-0233-5