Abstract
Using the data of a comprehensive evaluation study on the peer review process of Angewandte Chemie International Edition (AC-IE), we examined in this study the way in which referees’ comments differ on manuscripts rejected at AC-IE and later published in either a low-impact journal (Tetrahedron Letters, n = 54) or a high-impact journal (Journal of the American Chemical Society, n = 42). For this purpose, a content analysis was performed of comments which led to the rejection of the manuscripts at AC-IE. For the content analysis, a classification scheme with thematic areas developed by Bornmann et al. (2008) was used. As the results of the analysis demonstrate, a large number of negative comments from referees in the areas “Relevance of contribution” and “Design/Conception” are clear signs that a manuscript rejected at AC-IE will not be published later in a high-impact journal. The number of negative statements in the areas “Writing/Presentation,” “Discussion of results,” “Method/Statistics,” and “Reference to the literature and documentation,” on the other hand, had no statistically significant influence on the probability that a rejected manuscript would later be published in a low- or high-impact journal. The results of this study have various implications for authors, journal editors and referees.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Abelson, P. H. (1980). Scientific communication. Science, 209(4452), 60–62.
Abelson, P. (1990). Mechanisms for evaluating scientific information and the role of peer review. Journal of the American Society for Information Science, 41(3), 216–222.
Adam, D., & Knight, J. (2002). Journals under pressure: publish, and be damned. Nature, 419(6909), 772–776.
Alberts, B., Hanson, B., & Kelner, K. L. (2008). Reviewing peer review. Science, 321(5885), 15.
Bakanic, V., McPhail, C., & Simon, R. J. (1989). Mixed messages: referees’ comments on the manuscripts they review. Sociological Quarterly, 30(4), 639–654.
Bornmann, L., & Daniel, H.-D. (2008a). The effectiveness of the peer review process: inter-referee agreement and predictive validity of manuscript refereeing at Angewandte Chemie. Angewandte Chemie International Edition, 47(38), 7173–7178.
Bornmann, L., & Daniel, H.-D. (2008b). Selecting manuscripts for a high impact journal through peer review: a citation analysis of Communications that were accepted by Angewandte Chemie International Edition, or rejected but published elsewhere. Journal of the American Society for Information Science and Technology, 59(11), 1841–1852.
Bornmann, L., & Daniel, H.-D. (2009). The luck of the referee draw: the effect of exchanging reviews. Learned Publishing, 22(2), 117–125.
Bornmann, L., Nast, I., & Daniel, H.-D. (2008). Do editors and referees look for signs of scientific misconduct when reviewing manuscripts? A quantitative content analysis of studies that examined review criteria and reasons for accepting and rejecting manuscripts for publication. Scientometrics, 77(3), 415–432.
Campanario, J. M. (1998). Peer review for journals as it stands today—Part 1. Science Communication, 19(3), 181–211.
Daniel, H.-D. (1993/2004), Guardians of science. Fairness and reliability of peer review. Weinheim, Germany: Wiley-VCH. Published online 16 July 2004, Wiley Interscience, doi: 10.1002/3527602208.
Dickersin, K., Ssemanda, E., Mansell, C., Rennie, D. (2007), What do the JAMA editors say when they discuss manuscripts that they are considering for publication? Developing a schema for classifying the content of editorial discussion. BMC Medical Research Methodology, 7, 44.
Gosden, H. (2003). ‘Why not give us the full story?’ functions of referees’ comments in peer reviews of scientific research papers. Journal of English for Academic Purposes, 2(2), 87–101.
Hames, I. (2007). Peer review and manuscript management of scientific journals: Guidelines for good practice. Oxford, UK: Blackwell.
Hemlin, S. (1996). Research on research evaluations. Social Epistemology, 10(2), 209–250.
Hosmer, D. W., & Lemeshow, S. (2000). Applied logistic regression. Chichester, UK: Wiley.
Judge, T., Cable, D., Colbert, A., & Rynes, S. (2007). What causes a management article to be cited - article, author, or journal? The Academy of Management Journal (AMJ), 50(3), 491–506.
Kupfersmid, J. (1988). Improving what is published—A model in search of an editor. American Psychologist, 43(8), 635–642.
LaFollette, M. C. (1992). Stealing into print: Fraud, plagiarism and misconduct in scientific publishing. Berkeley, CA, USA: University of California Press.
Long, J. S., & Freese, J. (2006). Regression models for categorical dependent variables using Stata. College Station, TX, USA: Stata Press, Stata Corporation.
Marchionini, G. (2008). Rating reviewers. Science, 319(5868), 1335–1336.
Rabe-Hesketh, S., & Everitt, B. (2004). A handbook of statistical analyses using Stata. Boca Raton, FL, USA: Chapman & Hall/CRC.
Shashok, K. (2008). Content and communication: how can peer review provide helpful feedback about the writing? BMC Medical Research Methods, 8(3).
Silberzweig, J. E., & Khorsandi, A. S. (2008). Outcomes of rejected Journal of Vascular and Interventional Radiology manuscripts. Journal of Vascular and Interventional Radiology, 19(11), 1620–1623.
StataCorp. (2007). Stata statistical software: release 10. College Station, TX, USA: Stata Corporation.
Sternberg, R. J., Hojjat, M., Brigockas, M. G., & Grigorenko, E. L. (1997). Getting in: Criteria for acceptance of manuscripts in Psychological Bulletin, 1993–1996. Psychological Bulletin, 121(2), 321–323.
Turcotte, C., Drolet, P., & Girard, M. (2004). Study design, originality and overall consistency influence acceptance or rejection of manuscripts submitted to the Journal. Canadian Journal of Anaesthesia [Journal Canadien D Anesthesie], 51(6), 549–556.
Weller, A. C. (2002). Editorial peer review: Its strengths and weaknesses. Medford, NJ, USA: Information Today, Inc.
Ziman, J. (1968). Public knowledge: An essay concerning the social dimension of science. Cambridge, UK: Cambridge University Press.
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Bornmann, L., Weymuth, C. & Daniel, HD. A content analysis of referees’ comments: how do comments on manuscripts rejected by a high-impact journal and later published in either a low- or high-impact journal differ?. Scientometrics 83, 493–506 (2010). https://doi.org/10.1007/s11192-009-0011-4
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11192-009-0011-4