Skip to main content
SpringerLink
Log in

Closed versus open reviewing of journal manuscripts: how far do comments differ in language use?

  • Published:
Scientometrics Aims and scope Submit manuscript

Abstract

Whereas in traditional, closed peer review (CPR) a few, selected scientists (peers) are included in the process of manuscript review, public peer review (PPR) includes, in addition to invited reviewers, a wider circle of scientists who are interested in a manuscript and wish to write a comment on it. In this study, using the data of two comprehensive evaluation studies on the CPR process at Angewandte Chemie—International Edition and the PPR process at Atmospheric Chemistry and Physics, we examined the language characteristics in comments that were written by invited reviewers in CPR and by invited reviewers and interested members of the scientific community in PPR. We used Linguistic Inquiry and Word Count (LIWC), a text analysis software program that counts words in meaningful categories (e.g., positive or negative emotions) using a standardized dictionary. We examined 599 comments from the reviews of 229 manuscripts. The results show that the comments in PPR are much longer than the comments in CPR. This is an indication that PPR reviewing has more of an improvement function and CPR reviewing has more of a selection function. The results also show that CPR is not, as might be expected, more susceptible to the expression of negative emotions than PPR is. On the contrary, positive emotion words are used statistically significantly more frequently in CPR than in PPR.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  • Anon. (2006). Peer review on trial. Nature, 441(7094), 668.

    Google Scholar 

  • Bakanic, V., McPhail, C., & Simon, R. J. (1989). Mixed messages: Referees’ comments on the manuscripts they review. Sociological Quarterly, 30(4), 639–654.

    Article  Google Scholar 

  • Bingham, C. M., Higgins, G., Coleman, R., & Van Der Weyden, M. B. (1998). The Medical Journal of Australia Internet peer-review study. Lancet, 352(9126), 441–445.

    Article  Google Scholar 

  • Bornmann, L., & Daniel, H.-D. (2008a). The effectiveness of the peer review process: Inter-referee agreement and predictive validity of manuscript refereeing at Angewandte Chemie. Angewandte Chemie International Edition, 47(38), 7173–7178. doi:10.1002/anie.200800513.

    Article  Google Scholar 

  • Bornmann, L., & Daniel, H.-D. (2008b). Selecting manuscripts for a high impact journal through peer review: A citation analysis of communications that were accepted by Angewandte Chemie International Edition, or rejected but published elsewhere. Journal of the American Society for Information Science and Technology, 59(11), 1841–1852. doi:10.1002/asi.20901.

    Article  Google Scholar 

  • Bornmann, L., & Daniel, H.-D. (2009a). Extent of type I and type II errors in editorial decisions: A case study on Angewandte Chemie International Edition. Journal of Informetrics, 3(4), 348–352.

    Article  Google Scholar 

  • Bornmann, L., & Daniel, H.-D. (2009b). Reviewer and editor biases in journal peer review: An investigation of manuscript refereeing at Angewandte Chemie International Edition. Research Evaluation, 18(4), 262–272.

    Article  Google Scholar 

  • Bornmann, L., & Daniel, H.-D. (2010a). How long is the peer review process for journal manuscripts? A case study on Angewandte Chemie International Edition. Chimia, 64(1–2), 72–77.

    Article  Google Scholar 

  • Bornmann, L., & Daniel, H.-D. (2010b). Reliability of reviewers’ ratings at an interactive open access journal using public peer review: A case study on Atmospheric Chemistry and Physics. Learned Publishing, 23(2), 124–131.

    Article  Google Scholar 

  • Bornmann, L., & Daniel, H.-D. (2010c). The manuscript reviewing process—empirical research on review requests, review sequences and decision rules in peer review. Library & Information Science Research, 32(1), 5–12.

    Google Scholar 

  • Bornmann, L., & Daniel, H.-D. (2010d). The validity of staff editors’ initial evaluations of manuscripts: A case study of Angewandte Chemie International Edition. Scientometrics, 85(3), 681–687.

    Article  Google Scholar 

  • Bornmann, L., Marx, W., Schier, H., Rahm, E., Thor, A., & Daniel, H. D. (2009). Convergent validity of bibliometric Google Scholar data in the field of chemistry. Citation counts for papers that were accepted by Angewandte Chemie International Edition or rejected but published elsewhere, using Google Scholar, Science Citation Index, Scopus, and Chemical Abstracts. Journal of Informetrics, 3(1), 27–35. doi:10.1016/j.joi.2008.11.001.

    Article  Google Scholar 

  • Bornmann, L., Marx, W., Schier, H., Thor, A., & Daniel, H.-D. (2010a). From black box to white box at open access journals: Predictive validity of manuscript reviewing and editorial decisions at Atmospheric Chemistry and Physics. Research Evaluation, 19(2), 105–118.

    Article  Google Scholar 

  • Bornmann, L., Nast, I., & Daniel, H.-D. (2008). Do editors and referees look for signs of scientific misconduct when reviewing manuscripts? A quantitative content analysis of studies that examined review criteria and reasons for accepting and rejecting manuscripts for publication. Scientometrics, 77(3), 415–432.

    Article  Google Scholar 

  • Bornmann, L., Neuhaus, C., & Daniel, H.-D. (2011a). The effect of a two-stage publication process on the Journal Impact Factor: A case study on the interactive open access journal Atmospheric Chemistry and Physics. Scientometrics, 86, 93–97. doi:10.1007/s11192-010-0250-4.

    Article  Google Scholar 

  • Bornmann, L., Schier, H., Marx, W., & Daniel, H.-D. (2011b). Is interactive open access publishing able to identify high-impact submissions? A study on the predictive validity of Atmospheric Chemistry and Physics by using percentile rank classes. Journal of the American Society for Information Science and Technology, 62(1), 61–71.

    Article  Google Scholar 

  • Bornmann, L., Weymuth, C., & Daniel, H.-D. (2010b). A content analysis of referees’ comments: How do comments on manuscripts rejected by a high-impact journal and later published in either a low- or high-impact journal differ? Scientometrics, 83(2), 493–506.

    Article  Google Scholar 

  • Chung, C. K., & Pennebaker, J. W. (2007). The psychological function of function words. In K. Fiedler (Ed.), Social communication (pp. 343–359). New York: Psychology Press.

    Google Scholar 

  • Fast, L. A., & Funder, D. C. (2008). Personality as manifest in word use: Correlations with self-report, acquaintance report, and behavior. Journal of Personality and Social Psychology, 94(2), 334–346.

    Article  Google Scholar 

  • Fraser, V. J., & Martin, J. G. (2009). Marketing data: Has the rise of impact factor led to the fall of objective language in the scientific article? Respiratory Research, 10. doi:10.1186/1465-9921-10-35.

  • Harnad, S. (2000). The invisible hand of peer review. Exploit Interactive (5). http://www.exploit-lib.org/issue5/peer-review/.

  • Hartley, J., & Betts, L. (2009). Common weaknesses in traditional abstracts in the social sciences. Journal of the American Society for Information Science and Technology, 60(10), 2010–2018.

    Article  Google Scholar 

  • Hartley, J., Pennebaker, J. W., & Fox, C. (2003). Abstracts, introductions and discussions: How far do they differ in style? Scientometrics, 57(3), 389–398.

    Article  Google Scholar 

  • Kahn, J. H., Tobin, R. M., Massey, A. E., & Anderson, J. A. (2007). Measuring emotional expression with the Linguistic Inquiry and Word Count. American Journal of Psychology, 120(2), 263–286.

    Google Scholar 

  • Koonin, E., & Lipman, D. (2006). Systems: Reviving a culture of scientific debate. Can ‘open peer review’ work for biologists? Biology Direct is hopeful. Retrieved 21 June 2006, from http://www.nature.com/nature/peerreview/debate/nature05005.html.

  • Mungra, P., & Webber, P. (2010). Peer review process in medical research publications: Language and content comments. English for Specific Purposes, 29(1), 43–53. doi:10.1016/j.esp.2009.07.002.

    Article  Google Scholar 

  • Oberlander, J., & Gill, A. J. (2006). Language with character: A stratified corpus comparison of individual differences in e-mail communication. Discourse Processes, 42(3), 239–270.

    Article  Google Scholar 

  • Pennebaker, J. W., Chung, C. K., Ireland, M., Gonzales, A., & Booth, R. J. (2007). The development and psychometric properties of LIWC 2007. Austin, TX/Auckland: University of Texas/University of Auckland.

    Google Scholar 

  • Pennebaker, J. W., & King, L. A. (1999). Linguistic styles: Language use as an individual difference. Journal of Personality and Social Psychology, 77(6), 1296–1312.

    Article  Google Scholar 

  • Pennebaker, J. W., Mehl, M. R., & Niederhoffer, K. G. (2003). Psychological aspects of natural language use: Our words, our selves. Annual Review of Psychology, 54, 547–577. doi:10.1146/annurev.psych.54.101601.145041.

    Article  Google Scholar 

  • Popping, R. (2000). Computer-assisted text analysis. London: Sage.

    Google Scholar 

  • Pöschl, U. (2004). Interactive journal concept for improved scientific publishing and quality assurance. Learned Publishing, 17(2), 105–113. doi:10.1087/095315104322958481.

    Article  Google Scholar 

  • Pöschl, U. (2010). Interactive open access publishing and peer review: The effectiveness and perspectives of transparency and self-regulation in scientific communication and evaluation. Liber Quarterly, 19(3/4), 293–314.

    Google Scholar 

  • StataCorp. (2011). Stata statistical software: Release 12. College Station, TX: Stata Corporation.

    Google Scholar 

  • Tausczik, Y. R., & Pennebaker, J. W. (2010). The psychological meaning of words: LIWC and computerized text analysis methods. Journal of Language and Social Psychology, 29(1), 24–54. doi:10.1177/0261927x09351676.

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Max Planck Society, Administrative Headquarters, Hofgartenstr. 8, 80539, Munich, Germany

    Lutz Bornmann

  2. Center for Psychotherapy Research, University Hospital Heidelberg, Bergheimer Str. 54, 69115, Heidelberg, Germany

    Markus Wolf

  3. Evaluation Office, University of Zurich, Zurich, Switzerland

    Hans-Dieter Daniel

  4. ETH Zurich, Mühlegasse 21, MUG, 8001, Zurich, Switzerland

    Hans-Dieter Daniel

Authors
  1. Lutz Bornmann
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Markus Wolf
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Hans-Dieter Daniel
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Lutz Bornmann.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Bornmann, L., Wolf, M. & Daniel, HD. Closed versus open reviewing of journal manuscripts: how far do comments differ in language use?. Scientometrics 91, 843–856 (2012). https://doi.org/10.1007/s11192-011-0569-5

Download citation

  • Received:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11192-011-0569-5

Keywords

Search

Navigation