Scientometrics

, Volume 85, Issue 1, pp 193–202 | Cite as

Worsening file-drawer problem in the abstracts of natural, medical and social science databases

Article

Abstract

The file-drawer problem is the tendency of journals to preferentially publish studies with statistically significant results. The problem is an old one and has been documented in various fields, but to my best knowledge there has not been attention to how the issue is developing in a quantitative way through time. In the abstracts of various major scholarly databases (Science and Social Science Citation Index (1991–2008), CAB Abstracts and Medline (1970s–2008), the file drawer problem is gradually getting worse, in spite of an increase in (1) the total number of publications and (2) the proportion of publications reporting both the presence and the absence of significant differences. The trend is confirmed for particular natural science topics such as biology, energy and environment but not for papers retrieved with the keywords biodiversity, chemistry, computer, engineering, genetics, psychology and quantum (physics). A worsening file-drawer problem can be detected in various medical fields (infection, immunology, malaria, obesity, oncology and pharmacology), but not for papers indexed with strings such as AIDS/HIV, epidemiology, health and neurology. An increase in the selective publication of some results against some others is worrying because it can lead to enhanced bias in meta-analysis and hence to a distorted picture of the evidence for or against a certain hypothesis. Long-term monitoring of the file-drawer problem is needed to ensure a sustainable and reliable production of (peer-reviewed) scientific knowledge.

Keywords

History of science Meta-analysis Publication explosion Scientific knowledge Significant differences STM publishing 

References

  1. Abt, H. A. (1992). Publication practices in various sciences. Scientometrics, 24, 441–447.CrossRefGoogle Scholar
  2. Begg, C. B., & Berlin, J. A. (1988). Publication bias: A problem in interpreting medical data. Journal of the Royal Statistical Society A, 151, 419–463.CrossRefGoogle Scholar
  3. Bensman, S. J. (2007). Garfield and the impact factor. Annual Review of Information Science and Technology, 41, 93–155.CrossRefGoogle Scholar
  4. Bourne, P. E., & Korngreen, A. (2006). Ten simple rules for reviewers. PLoS Computational Biology, 2, e110.CrossRefGoogle Scholar
  5. Cicchetti, D. V. (1991). The reliability of peer-review for manuscript and grant submissions: A cross-disciplinary investigation. Behavioral and Brain Sciences, 14, 119–134.Google Scholar
  6. Csada, R. D., James, P. C., & Espie, R. H. M. (1996). The “file drawer problem” of non-significant results: Does it apply to biological research? Oikos, 76, 591–593.CrossRefGoogle Scholar
  7. de Mesnard, L. (2010). On Hochberg et al.’s “The tragedy of the reviewer commons”. Scientometrics, in press doi:10.1007/s11192-009-0141-8.
  8. Garfield, E. (1997). A statistically valid definition of bias is needed to determine whether the Science Citation Index(R) discriminates against third world journals. Current Science, 73, 639–641.Google Scholar
  9. Gerber, A. S., & Malhotra, N. (2008). Publication bias in empirical sociological research: Do arbitrary significance levels distort published results? Sociological Methods & Research, 37, 3–30.CrossRefMathSciNetGoogle Scholar
  10. Gilbody, S. M., Song, F., Eastwood, A. J., & Sutton, A. (2000). The causes, consequences and detection of publication bias in psychiatry. Acta Psychiatrica Scandinavica, 102, 241–249.CrossRefGoogle Scholar
  11. Greenberg, S. A. (2009). How citation distortions create unfounded authority: Analysis of a citation network. British Medical Journal, 339, b2680.CrossRefGoogle Scholar
  12. Guetzkow, J., Lamont, M., & Mallard, G. (2004). What is originality in the humanities and the social sciences? American Sociological Review, 69, 190–212.CrossRefGoogle Scholar
  13. Hauser, M., & Fehr, E. (2007). An incentive solution to the peer review problem. PLoS Biology, 5, e107.CrossRefGoogle Scholar
  14. Hochberg, M. E., Chase, J. M., Gotelli, N. J., Hastings, A., & Naeem, S. (2009). The tragedy of the reviewer commons. Ecology Letters, 12, 2–4.CrossRefGoogle Scholar
  15. Kennedy, D. (2004). The old file-drawer problem. Science, 305, 451.CrossRefGoogle Scholar
  16. Khoury, M. J., Bertram, L., Boffetta, P., Butterworth, A. S., Chanock, S. J., Dolan, S. M., et al. (2009). Genome-wide association studies, field synopses, and the development of the knowledge base on genetic variation and human diseases. American Journal of Epidemiology, 170, 269–279.CrossRefGoogle Scholar
  17. Killeen, P. R. (2005). An alternative to null-hypothesis significance tests. Psychological Science, 16, 345–353.CrossRefGoogle Scholar
  18. Klein, J. T. (2006). Afterword: The emergent literature on interdisciplinary and transdisciplinary research evaluation. Research Evaluation, 15, 75–80.CrossRefGoogle Scholar
  19. Koletsi, D., Karagianni, A., Pandis, N., Makou, M., Polychronopolou, A., & Eliades, T. (2009). Are studies reporting significant results more likely to be published? American Journal of Orthodontics and Dentofacial Orthopedics, 136, 632e1.Google Scholar
  20. Krzyzanowska, M. K., Pintilie, M., & Tannock, I. F. (2003). Factors associated with failure to publish large randomized trials presented at an oncology meeting. Journal of the American Medical Association, 290, 495–501.CrossRefGoogle Scholar
  21. Lawrence, P. A. (2003). The politics of publication. Nature, 422, 259–261.CrossRefGoogle Scholar
  22. Levine, T., Asada, K. J., & Carpenter, C. (2009). Sample sizes and effect sizes are negatively correlated in meta-analyses: Evidence and implications of a publication bias against non-significant findings. Communication Monographs, 76, 286–302.CrossRefGoogle Scholar
  23. Marsh, H. W., Bornmann, L., Mutz, R., Daniel, H. D., & O’Mara, A. (2009). Gender effects in the peer reviews of grant proposals: A comprehensive meta-analysis comparing traditional and multilevel approaches. Review of Educational Research, 79, 1290–1326.CrossRefGoogle Scholar
  24. Nakagawa, S., & Cuthill, I. C. (2007). Effect size, confidence interval and statistical significance: A practical guide for biologists. Biological Reviews, 82, 591–605.CrossRefGoogle Scholar
  25. Nieminen, P., Rucker, G., Miettunen, J., Carpenter, J., & Schumacher, M. (2007). Statistically significant papers in psychiatry were cited more often than others. Journal of Clinical Epidemiology, 60, 939–946.CrossRefGoogle Scholar
  26. Paris, G., De Leo, G., Menozzi, P., & Gatto, M. (1998). Region-based citation bias in science. Nature, 396, 6708.CrossRefGoogle Scholar
  27. Pautasso, M., & Pautasso, C. (2010). Peer reviewing interdisciplinary papers. European Review, 18, 227–237.CrossRefGoogle Scholar
  28. Pautasso, M., & Schäfer, H. (2010). Peer review delay and selectivity in ecology journals. Scientometrics, in press. doi:10.1007/s11192-009-0105-z.
  29. Primack, R. B., & Marrs, R. (2008). Bias in the review process. Biological Conservation, 141, 2919–2920.CrossRefGoogle Scholar
  30. Provenzale, J. M., & Stanley, R. J. (2005). A systematic guide to reviewing a manuscript. American Journal of Radiology, 185, 848–854.Google Scholar
  31. Reinhart, M. (2009). Peer review of grant applications in biology and medicine. Reliability, fairness, and validity. Scientometrics, 81, 789–809.CrossRefGoogle Scholar
  32. Rosenthal, R. (1979). The file drawer problem and tolerance for null results. Psychological Bulletin, 86, 638–641.CrossRefGoogle Scholar
  33. Schwartz, S. J., & Zamboanga, B. L. (2009). The peer-review and editorial system: Ways to fix something that might be broken. Perspectives in Psychological Science, 4, 54–61.CrossRefGoogle Scholar
  34. Smith, A. J. (1990). The task of the referee. IEEE Computer, 23, 46–51.Google Scholar
  35. Song, F. J., Parekh-Bhurke, S., Hooper, L., Loke, Y. K., Ryder, J. J., Sutton, A. J., et al. (2009). Extent of publication bias in different categories of research cohorts: A meta-analysis of empirical studies. BMC Medical Research Methodology, 9, 79.CrossRefGoogle Scholar
  36. Sterling, T. D. (1959). Publication decisions and their possible effects on inferences drawn from tests of significance—Or vice versa. Journal of the American Statistical Association, 54, 30–34.CrossRefGoogle Scholar
  37. Taborsky, M. (2009). Biased citation practice and taxonomic parochialism. Ethology, 115, 105–111.CrossRefGoogle Scholar
  38. Tricco, A. C., Tetzaff, J., Pham, B., Brehaut, J., & Moher, D. (2009). Non-Cochrane vs. Cochrane reviews were twice as likely to have positive conclusion statements: Cross-sectional study. Journal of Clinical Epidemiology, 62, 380–386.CrossRefGoogle Scholar
  39. Vecchi, S., Belleudi, V., Amato, L., Davoli, M., & Peducci, C. A. (2009). Does direction of results of abstracts submitted to scientific conferences on drug addiction predict full publication? BMC Medical Research Methodology, 9, 23.CrossRefGoogle Scholar

Copyright information

© Akadémiai Kiadó, Budapest, Hungary 2010

Authors and Affiliations

  1. 1.Division of BiologyImperial College LondonAscotUK

Personalised recommendations