Scientometrics

, Volume 83, Issue 2, pp 493–506

A content analysis of referees’ comments: how do comments on manuscripts rejected by a high-impact journal and later published in either a low- or high-impact journal differ?

  • Lutz Bornmann
  • Christophe Weymuth
  • Hans-Dieter Daniel
Article

Abstract

Using the data of a comprehensive evaluation study on the peer review process of Angewandte Chemie International Edition (AC-IE), we examined in this study the way in which referees’ comments differ on manuscripts rejected at AC-IE and later published in either a low-impact journal (Tetrahedron Letters, n = 54) or a high-impact journal (Journal of the American Chemical Society, n = 42). For this purpose, a content analysis was performed of comments which led to the rejection of the manuscripts at AC-IE. For the content analysis, a classification scheme with thematic areas developed by Bornmann et al. (2008) was used. As the results of the analysis demonstrate, a large number of negative comments from referees in the areas “Relevance of contribution” and “Design/Conception” are clear signs that a manuscript rejected at AC-IE will not be published later in a high-impact journal. The number of negative statements in the areas “Writing/Presentation,” “Discussion of results,” “Method/Statistics,” and “Reference to the literature and documentation,” on the other hand, had no statistically significant influence on the probability that a rejected manuscript would later be published in a low- or high-impact journal. The results of this study have various implications for authors, journal editors and referees.

Keywords

Journal peer review Content analysis Thematic areas for manuscript review Fate of rejected manuscripts 

References

  1. Abelson, P. H. (1980). Scientific communication. Science, 209(4452), 60–62.CrossRefGoogle Scholar
  2. Abelson, P. (1990). Mechanisms for evaluating scientific information and the role of peer review. Journal of the American Society for Information Science, 41(3), 216–222.CrossRefGoogle Scholar
  3. Adam, D., & Knight, J. (2002). Journals under pressure: publish, and be damned. Nature, 419(6909), 772–776.CrossRefGoogle Scholar
  4. Alberts, B., Hanson, B., & Kelner, K. L. (2008). Reviewing peer review. Science, 321(5885), 15.CrossRefGoogle Scholar
  5. Bakanic, V., McPhail, C., & Simon, R. J. (1989). Mixed messages: referees’ comments on the manuscripts they review. Sociological Quarterly, 30(4), 639–654.CrossRefGoogle Scholar
  6. Bornmann, L., & Daniel, H.-D. (2008a). The effectiveness of the peer review process: inter-referee agreement and predictive validity of manuscript refereeing at Angewandte Chemie. Angewandte Chemie International Edition, 47(38), 7173–7178.CrossRefGoogle Scholar
  7. Bornmann, L., & Daniel, H.-D. (2008b). Selecting manuscripts for a high impact journal through peer review: a citation analysis of Communications that were accepted by Angewandte Chemie International Edition, or rejected but published elsewhere. Journal of the American Society for Information Science and Technology, 59(11), 1841–1852.CrossRefGoogle Scholar
  8. Bornmann, L., & Daniel, H.-D. (2009). The luck of the referee draw: the effect of exchanging reviews. Learned Publishing, 22(2), 117–125.CrossRefGoogle Scholar
  9. Bornmann, L., Nast, I., & Daniel, H.-D. (2008). Do editors and referees look for signs of scientific misconduct when reviewing manuscripts? A quantitative content analysis of studies that examined review criteria and reasons for accepting and rejecting manuscripts for publication. Scientometrics, 77(3), 415–432.CrossRefGoogle Scholar
  10. Campanario, J. M. (1998). Peer review for journals as it stands today—Part 1. Science Communication, 19(3), 181–211.CrossRefGoogle Scholar
  11. Daniel, H.-D. (1993/2004), Guardians of science. Fairness and reliability of peer review. Weinheim, Germany: Wiley-VCH. Published online 16 July 2004, Wiley Interscience, doi: 10.1002/3527602208.
  12. Dickersin, K., Ssemanda, E., Mansell, C., Rennie, D. (2007), What do the JAMA editors say when they discuss manuscripts that they are considering for publication? Developing a schema for classifying the content of editorial discussion. BMC Medical Research Methodology, 7, 44.Google Scholar
  13. Gosden, H. (2003). ‘Why not give us the full story?’ functions of referees’ comments in peer reviews of scientific research papers. Journal of English for Academic Purposes, 2(2), 87–101.CrossRefGoogle Scholar
  14. Hames, I. (2007). Peer review and manuscript management of scientific journals: Guidelines for good practice. Oxford, UK: Blackwell.CrossRefGoogle Scholar
  15. Hemlin, S. (1996). Research on research evaluations. Social Epistemology, 10(2), 209–250.CrossRefGoogle Scholar
  16. Hosmer, D. W., & Lemeshow, S. (2000). Applied logistic regression. Chichester, UK: Wiley.MATHCrossRefGoogle Scholar
  17. Judge, T., Cable, D., Colbert, A., & Rynes, S. (2007). What causes a management article to be cited - article, author, or journal? The Academy of Management Journal (AMJ), 50(3), 491–506.Google Scholar
  18. Kupfersmid, J. (1988). Improving what is published—A model in search of an editor. American Psychologist, 43(8), 635–642.CrossRefGoogle Scholar
  19. LaFollette, M. C. (1992). Stealing into print: Fraud, plagiarism and misconduct in scientific publishing. Berkeley, CA, USA: University of California Press.Google Scholar
  20. Long, J. S., & Freese, J. (2006). Regression models for categorical dependent variables using Stata. College Station, TX, USA: Stata Press, Stata Corporation.MATHGoogle Scholar
  21. Marchionini, G. (2008). Rating reviewers. Science, 319(5868), 1335–1336.CrossRefGoogle Scholar
  22. Rabe-Hesketh, S., & Everitt, B. (2004). A handbook of statistical analyses using Stata. Boca Raton, FL, USA: Chapman & Hall/CRC.MATHGoogle Scholar
  23. Shashok, K. (2008). Content and communication: how can peer review provide helpful feedback about the writing? BMC Medical Research Methods, 8(3).Google Scholar
  24. Silberzweig, J. E., & Khorsandi, A. S. (2008). Outcomes of rejected Journal of Vascular and Interventional Radiology manuscripts. Journal of Vascular and Interventional Radiology, 19(11), 1620–1623.CrossRefGoogle Scholar
  25. StataCorp. (2007). Stata statistical software: release 10. College Station, TX, USA: Stata Corporation.Google Scholar
  26. Sternberg, R. J., Hojjat, M., Brigockas, M. G., & Grigorenko, E. L. (1997). Getting in: Criteria for acceptance of manuscripts in Psychological Bulletin, 1993–1996. Psychological Bulletin, 121(2), 321–323.CrossRefGoogle Scholar
  27. Turcotte, C., Drolet, P., & Girard, M. (2004). Study design, originality and overall consistency influence acceptance or rejection of manuscripts submitted to the Journal. Canadian Journal of Anaesthesia [Journal Canadien D Anesthesie], 51(6), 549–556.CrossRefGoogle Scholar
  28. Weller, A. C. (2002). Editorial peer review: Its strengths and weaknesses. Medford, NJ, USA: Information Today, Inc.Google Scholar
  29. Ziman, J. (1968). Public knowledge: An essay concerning the social dimension of science. Cambridge, UK: Cambridge University Press.Google Scholar

Copyright information

© Akadémiai Kiadó, Budapest, Hungary 2009

Authors and Affiliations

  • Lutz Bornmann
    • 1
  • Christophe Weymuth
    • 2
  • Hans-Dieter Daniel
    • 1
    • 3
  1. 1.Professorship for Social Psychology and Research on Higher EducationETH ZurichZurichSwitzerland
  2. 2.Biosynth AGStaadSwitzerland
  3. 3.Evaluation OfficeUniversity of ZurichZurichSwitzerland

Personalised recommendations