, Volume 95, Issue 1, pp 277–297 | Cite as

Exploring the bibliometric and semantic nature of negative results

  • Christian Gumpenberger
  • Juan GorraizEmail author
  • Martin Wieland
  • Ivana Roche
  • Edgar Schiebel
  • Dominique Besagni
  • Claire François


Negative results are not popular to disseminate. However, their publication would help to save resources and foster scientific communication. This study analysed the bibliometric and semantic nature of negative results publications. The Journal of Negative Results in Biomedicine (JNRBM) was used as a role model. Its complete articles from 2002–2009 were extracted from SCOPUS and supplemented by related records. Complementary negative results records were retrieved from Web of Science in “Biochemistry” and “Telecommunications”. Applied bibliometrics comprised of co-author and co-affiliation analysis and a citation impact profile. Bibliometrics showed that authorship is widely spread. A specific community for the publication of negative results in devoted literature is non-existent. Neither co-author nor co-affiliation analysis indicated strong interconnectivities. JNRBM articles are cited by a broad spectrum of journals rather than by specific titles. Devoted negative results journals like JNRBM have a rather low impact measured by the number of received citations. On the other hand, only one-third of the publications remain uncited, corroborating their importance for the scientific community. The semantic analysis relies on negative expressions manually identified in JNRBM article titles and abstracts and extracted to syntactic patterns. By using a Natural Language Processing tool these patterns are then employed to detect their occurrences in the multidisciplinary bibliographical database PASCAL. The translation of manually identified negation patterns to syntactic patterns and their application to multidisciplinary bibliographic databases (PASCAL, Web of Science) proved to be a successful method to retrieve even hidden negative results. There is proof that negative results are not only restricted to the biomedical domain. Interestingly a high percentage of the so far identified negative results papers were funded and therefore needed to be published. Thus policies that explicitly encourage or even mandate the publication of negative results could probably bring about a shift in the current scientific communication behaviour.


Bibliometrics Scientometrics Negative result publication S&T information Semantic analysis Publication bias 


  1. Agarwal, S., & Yu, H. (2010). Biomedical negation scope detection with conditional random fields. Journal of the American Medical Informatics Association, 17(6), 696–701.CrossRefGoogle Scholar
  2. Agarwal, S., Yu, H., & Kohane, I. (2011). BioNOT: A searchable database of biomedical negated sentences. BMC Bioinformatics, 12, 420. doi: 10.1186/1471-2105-12-420.CrossRefGoogle Scholar
  3. Aucouturier, J. J., & Pachet, F. (2004). Improving timbre similarity: How high is the sky? Journal of Negative Results in Speech and Audio Sciences, 1, 1.Google Scholar
  4. Browman, H. I. (1999). Negative results. Marine Ecology Progress Series, 191, 301–309.CrossRefGoogle Scholar
  5. Chan, A. W., Hrobjartsson, A., Haahr, M. T., Gotzsche, P. C., & Altman, D. G. (2004). Empirical evidence for selective reporting of outcomes in randomized trials—Comparison of protocols to published articles. JAMA, 291, 2457–2465.CrossRefGoogle Scholar
  6. De Vries, R., Anderson, M. S., & Martinson, B. C. (2006). Normal misbehavior: Scientists talk about the ethics of research. Journal of Empirical Research on Human Research Ethics, 1, 43–50.CrossRefGoogle Scholar
  7. Dickersin, K. (1990). The existence of publication bias and risk factors for its occurrence. JAMA, 263(10), 1385–1389.CrossRefGoogle Scholar
  8. Dickersin, K., Min, Y. I., & Meinert, C. L. (1992). Factors influencing publication of research results—follow-up of applications submitted to 2 institutional review boards. JAMA, 267, 374–378.CrossRefGoogle Scholar
  9. Easterbrook, P. J., Berlin, J. A., Gopalan, R., & Matthews, D. R. (1991). Publications bias in clinical research. Lancet, 337(8746), 867–872. doi: 10.1016/0140-6736(91)90201-Y.CrossRefGoogle Scholar
  10. Fanelli, D. (2009). How many scientists fabricate and falsify research? A systematic review and meta-analysis of survey data. PLoS ONE, 4, e5738.CrossRefGoogle Scholar
  11. Fanelli, D. (2010). Do pressures to publish increase scientists’ bias? An empirical support from US States data. PLoS ONE, 5(4), e10271.CrossRefGoogle Scholar
  12. Fanelli, D. (2011). Positive results increase down the hierarchy of the sciences. PLoS ONE,. doi: 10.1371/journal.pone.0010068.Google Scholar
  13. González-Pereira, B., Guerrero-Bote, V. P., Moya-Anegón, F. de (2010). A new approach to the metric of journals’ scientific prestige: The SJR indicator. Journal of Informetrics 4(3), 379–391. Accessed 23 April 2012.Google Scholar
  14. Gupta, N., & Stopfer, M. (2011). Negative results need airing too. Nature, 470, 39. doi: 10.1038/470039a.CrossRefGoogle Scholar
  15. Higgins, J. P. T., Thompson, S. G., Deeks, J. J., & Altman, D. G. (2003). Measuring inconsistency in meta-analyses. British Medical Journal, 327, 560.CrossRefGoogle Scholar
  16. Jerrells, T. R. (2003). Publication of negative research findings: pros and cons. Alcohol, 31, 123–124.CrossRefGoogle Scholar
  17. Kerr, N. L. (1998). HARKing: Hypothesizing after the results are known. Personality and Social Psychology Review, 2, 196–217.CrossRefGoogle Scholar
  18. Kopcsa, A., Schiebel, E. (2001). Content Visualisation by BibTechMon™. In Informatics 2001/Network Economy-Visions and Reality, OCG-Austrian Computer Society.Google Scholar
  19. Lexchin, J., Bero, L. A., Djulbegovic, B., & Clark, O. (2003). Pharmaceutical industry sponsorship and research outcome and quality: Systematic review. British Medical Journal, 326, 1167.CrossRefGoogle Scholar
  20. Miller, S. C., & Moulder, J. E. (1998). Publication of negative results is an essential part of the scientific process. Radiation Research, 150(1), 3–16.CrossRefGoogle Scholar
  21. Moed, H. F. (2010). The source normalized impact per paper is a valid and sophisticated indicator of journal citation impact. Journal of the American Society for Information Science and Technology, 62(1), 211–213.CrossRefGoogle Scholar
  22. Moed, H. F. (2011). Measuring contextual citation impact of scientific journals. Journal of Informetrics, 4(3), 265–277.CrossRefGoogle Scholar
  23. Morante, R., Daelemans, W. (2009). A metalearning approach to processing the scope of negation. In Proceedings of the Thirteenth Conference on Computational Natural Language Learning, pages 21–29.Google Scholar
  24. Newman, M. E. J. (2008). The mathematics of networks. The New Palgrave Encyclopedia of Economics. (Accessed 2012-07-12).
  25. Prechelt, L. (1997). Why we need an explicit forum for negative results. Journal of Universal Computer Science, 3, 1074–1083.MathSciNetGoogle Scholar
  26. Scargle, J. D. (2000). Publication bias: the “file-drawer” problem in scientific inference. Journal of Scientific Exploration, 14(1), 91–106.Google Scholar
  27. Smith, M. L. (1980). Publication bias and meta-analysis. Evaluation Educ, 4, 22–24.CrossRefGoogle Scholar
  28. Sterne, J. A. C., Egger, M., & Smith, G. D. (2001). Systematic reviews in health care—Investigating and dealing with publication and other biases in meta-analysis. British Medical Journal, 323, 101.CrossRefGoogle Scholar

Copyright information

© Akadémiai Kiadó, Budapest, Hungary 2012

Authors and Affiliations

  • Christian Gumpenberger
    • 1
  • Juan Gorraiz
    • 1
    Email author
  • Martin Wieland
    • 1
  • Ivana Roche
    • 2
  • Edgar Schiebel
    • 3
  • Dominique Besagni
    • 2
  • Claire François
    • 2
  1. 1.Library and Archive Services, Bibliometrics DepartmentUniversity of ViennaViennaAustria
  2. 2.INIST–CNRS Institut de l’Information Scientifique et TechniqueVandoeuvre-les-Nancy CedexFrance
  3. 3.AIT Austrian Institute of Technology GmbH, Tech Gate ViennaViennaAustria

Personalised recommendations