Scientometrics

, Volume 91, Issue 3, pp 843–856 | Cite as

Closed versus open reviewing of journal manuscripts: how far do comments differ in language use?

Article

Abstract

Whereas in traditional, closed peer review (CPR) a few, selected scientists (peers) are included in the process of manuscript review, public peer review (PPR) includes, in addition to invited reviewers, a wider circle of scientists who are interested in a manuscript and wish to write a comment on it. In this study, using the data of two comprehensive evaluation studies on the CPR process at Angewandte Chemie—International Edition and the PPR process at Atmospheric Chemistry and Physics, we examined the language characteristics in comments that were written by invited reviewers in CPR and by invited reviewers and interested members of the scientific community in PPR. We used Linguistic Inquiry and Word Count (LIWC), a text analysis software program that counts words in meaningful categories (e.g., positive or negative emotions) using a standardized dictionary. We examined 599 comments from the reviews of 229 manuscripts. The results show that the comments in PPR are much longer than the comments in CPR. This is an indication that PPR reviewing has more of an improvement function and CPR reviewing has more of a selection function. The results also show that CPR is not, as might be expected, more susceptible to the expression of negative emotions than PPR is. On the contrary, positive emotion words are used statistically significantly more frequently in CPR than in PPR.

Keywords

Journal peer review Text analysis Linguistic Inquiry and Word Count (LIWC) Angewandte Chemie—International Edition Atmospheric Chemistry and Physics 

References

  1. Anon. (2006). Peer review on trial. Nature, 441(7094), 668.Google Scholar
  2. Bakanic, V., McPhail, C., & Simon, R. J. (1989). Mixed messages: Referees’ comments on the manuscripts they review. Sociological Quarterly, 30(4), 639–654.CrossRefGoogle Scholar
  3. Bingham, C. M., Higgins, G., Coleman, R., & Van Der Weyden, M. B. (1998). The Medical Journal of Australia Internet peer-review study. Lancet, 352(9126), 441–445.CrossRefGoogle Scholar
  4. Bornmann, L., & Daniel, H.-D. (2008a). The effectiveness of the peer review process: Inter-referee agreement and predictive validity of manuscript refereeing at Angewandte Chemie. Angewandte Chemie International Edition, 47(38), 7173–7178. doi:10.1002/anie.200800513.CrossRefGoogle Scholar
  5. Bornmann, L., & Daniel, H.-D. (2008b). Selecting manuscripts for a high impact journal through peer review: A citation analysis of communications that were accepted by Angewandte Chemie International Edition, or rejected but published elsewhere. Journal of the American Society for Information Science and Technology, 59(11), 1841–1852. doi:10.1002/asi.20901.CrossRefGoogle Scholar
  6. Bornmann, L., & Daniel, H.-D. (2009a). Extent of type I and type II errors in editorial decisions: A case study on Angewandte Chemie International Edition. Journal of Informetrics, 3(4), 348–352.CrossRefGoogle Scholar
  7. Bornmann, L., & Daniel, H.-D. (2009b). Reviewer and editor biases in journal peer review: An investigation of manuscript refereeing at Angewandte Chemie International Edition. Research Evaluation, 18(4), 262–272.CrossRefGoogle Scholar
  8. Bornmann, L., & Daniel, H.-D. (2010a). How long is the peer review process for journal manuscripts? A case study on Angewandte Chemie International Edition. Chimia, 64(1–2), 72–77.CrossRefGoogle Scholar
  9. Bornmann, L., & Daniel, H.-D. (2010b). Reliability of reviewers’ ratings at an interactive open access journal using public peer review: A case study on Atmospheric Chemistry and Physics. Learned Publishing, 23(2), 124–131.CrossRefGoogle Scholar
  10. Bornmann, L., & Daniel, H.-D. (2010c). The manuscript reviewing process—empirical research on review requests, review sequences and decision rules in peer review. Library & Information Science Research, 32(1), 5–12.Google Scholar
  11. Bornmann, L., & Daniel, H.-D. (2010d). The validity of staff editors’ initial evaluations of manuscripts: A case study of Angewandte Chemie International Edition. Scientometrics, 85(3), 681–687.CrossRefGoogle Scholar
  12. Bornmann, L., Marx, W., Schier, H., Rahm, E., Thor, A., & Daniel, H. D. (2009). Convergent validity of bibliometric Google Scholar data in the field of chemistry. Citation counts for papers that were accepted by Angewandte Chemie International Edition or rejected but published elsewhere, using Google Scholar, Science Citation Index, Scopus, and Chemical Abstracts. Journal of Informetrics, 3(1), 27–35. doi:10.1016/j.joi.2008.11.001.CrossRefGoogle Scholar
  13. Bornmann, L., Marx, W., Schier, H., Thor, A., & Daniel, H.-D. (2010a). From black box to white box at open access journals: Predictive validity of manuscript reviewing and editorial decisions at Atmospheric Chemistry and Physics. Research Evaluation, 19(2), 105–118.CrossRefGoogle Scholar
  14. Bornmann, L., Nast, I., & Daniel, H.-D. (2008). Do editors and referees look for signs of scientific misconduct when reviewing manuscripts? A quantitative content analysis of studies that examined review criteria and reasons for accepting and rejecting manuscripts for publication. Scientometrics, 77(3), 415–432.CrossRefGoogle Scholar
  15. Bornmann, L., Neuhaus, C., & Daniel, H.-D. (2011a). The effect of a two-stage publication process on the Journal Impact Factor: A case study on the interactive open access journal Atmospheric Chemistry and Physics. Scientometrics, 86, 93–97. doi:10.1007/s11192-010-0250-4.CrossRefGoogle Scholar
  16. Bornmann, L., Schier, H., Marx, W., & Daniel, H.-D. (2011b). Is interactive open access publishing able to identify high-impact submissions? A study on the predictive validity of Atmospheric Chemistry and Physics by using percentile rank classes. Journal of the American Society for Information Science and Technology, 62(1), 61–71.CrossRefGoogle Scholar
  17. Bornmann, L., Weymuth, C., & Daniel, H.-D. (2010b). A content analysis of referees’ comments: How do comments on manuscripts rejected by a high-impact journal and later published in either a low- or high-impact journal differ? Scientometrics, 83(2), 493–506.CrossRefGoogle Scholar
  18. Chung, C. K., & Pennebaker, J. W. (2007). The psychological function of function words. In K. Fiedler (Ed.), Social communication (pp. 343–359). New York: Psychology Press.Google Scholar
  19. Fast, L. A., & Funder, D. C. (2008). Personality as manifest in word use: Correlations with self-report, acquaintance report, and behavior. Journal of Personality and Social Psychology, 94(2), 334–346.CrossRefGoogle Scholar
  20. Fraser, V. J., & Martin, J. G. (2009). Marketing data: Has the rise of impact factor led to the fall of objective language in the scientific article? Respiratory Research, 10. doi:10.1186/1465-9921-10-35.
  21. Harnad, S. (2000). The invisible hand of peer review. Exploit Interactive (5). http://www.exploit-lib.org/issue5/peer-review/.
  22. Hartley, J., & Betts, L. (2009). Common weaknesses in traditional abstracts in the social sciences. Journal of the American Society for Information Science and Technology, 60(10), 2010–2018.CrossRefGoogle Scholar
  23. Hartley, J., Pennebaker, J. W., & Fox, C. (2003). Abstracts, introductions and discussions: How far do they differ in style? Scientometrics, 57(3), 389–398.CrossRefGoogle Scholar
  24. Kahn, J. H., Tobin, R. M., Massey, A. E., & Anderson, J. A. (2007). Measuring emotional expression with the Linguistic Inquiry and Word Count. American Journal of Psychology, 120(2), 263–286.Google Scholar
  25. Koonin, E., & Lipman, D. (2006). Systems: Reviving a culture of scientific debate. Can ‘open peer review’ work for biologists? Biology Direct is hopeful. Retrieved 21 June 2006, from http://www.nature.com/nature/peerreview/debate/nature05005.html.
  26. Mungra, P., & Webber, P. (2010). Peer review process in medical research publications: Language and content comments. English for Specific Purposes, 29(1), 43–53. doi:10.1016/j.esp.2009.07.002.CrossRefGoogle Scholar
  27. Oberlander, J., & Gill, A. J. (2006). Language with character: A stratified corpus comparison of individual differences in e-mail communication. Discourse Processes, 42(3), 239–270.CrossRefGoogle Scholar
  28. Pennebaker, J. W., Chung, C. K., Ireland, M., Gonzales, A., & Booth, R. J. (2007). The development and psychometric properties of LIWC 2007. Austin, TX/Auckland: University of Texas/University of Auckland.Google Scholar
  29. Pennebaker, J. W., & King, L. A. (1999). Linguistic styles: Language use as an individual difference. Journal of Personality and Social Psychology, 77(6), 1296–1312.CrossRefGoogle Scholar
  30. Pennebaker, J. W., Mehl, M. R., & Niederhoffer, K. G. (2003). Psychological aspects of natural language use: Our words, our selves. Annual Review of Psychology, 54, 547–577. doi:10.1146/annurev.psych.54.101601.145041.CrossRefGoogle Scholar
  31. Popping, R. (2000). Computer-assisted text analysis. London: Sage.Google Scholar
  32. Pöschl, U. (2004). Interactive journal concept for improved scientific publishing and quality assurance. Learned Publishing, 17(2), 105–113. doi:10.1087/095315104322958481.CrossRefGoogle Scholar
  33. Pöschl, U. (2010). Interactive open access publishing and peer review: The effectiveness and perspectives of transparency and self-regulation in scientific communication and evaluation. Liber Quarterly, 19(3/4), 293–314.Google Scholar
  34. StataCorp. (2011). Stata statistical software: Release 12. College Station, TX: Stata Corporation.Google Scholar
  35. Tausczik, Y. R., & Pennebaker, J. W. (2010). The psychological meaning of words: LIWC and computerized text analysis methods. Journal of Language and Social Psychology, 29(1), 24–54. doi:10.1177/0261927x09351676.CrossRefGoogle Scholar

Copyright information

© Akadémiai Kiadó, Budapest, Hungary 2011

Authors and Affiliations

  • Lutz Bornmann
    • 1
  • Markus Wolf
    • 2
  • Hans-Dieter Daniel
    • 3
    • 4
  1. 1.Max Planck Society, Administrative HeadquartersMunichGermany
  2. 2.Center for Psychotherapy ResearchUniversity Hospital HeidelbergHeidelbergGermany
  3. 3.Evaluation OfficeUniversity of ZurichZurichSwitzerland
  4. 4.ETH ZurichZurichSwitzerland

Personalised recommendations