Closed versus open reviewing of journal manuscripts: how far do comments differ in language use?
Whereas in traditional, closed peer review (CPR) a few, selected scientists (peers) are included in the process of manuscript review, public peer review (PPR) includes, in addition to invited reviewers, a wider circle of scientists who are interested in a manuscript and wish to write a comment on it. In this study, using the data of two comprehensive evaluation studies on the CPR process at Angewandte Chemie—International Edition and the PPR process at Atmospheric Chemistry and Physics, we examined the language characteristics in comments that were written by invited reviewers in CPR and by invited reviewers and interested members of the scientific community in PPR. We used Linguistic Inquiry and Word Count (LIWC), a text analysis software program that counts words in meaningful categories (e.g., positive or negative emotions) using a standardized dictionary. We examined 599 comments from the reviews of 229 manuscripts. The results show that the comments in PPR are much longer than the comments in CPR. This is an indication that PPR reviewing has more of an improvement function and CPR reviewing has more of a selection function. The results also show that CPR is not, as might be expected, more susceptible to the expression of negative emotions than PPR is. On the contrary, positive emotion words are used statistically significantly more frequently in CPR than in PPR.
KeywordsJournal peer review Text analysis Linguistic Inquiry and Word Count (LIWC) Angewandte Chemie—International Edition Atmospheric Chemistry and Physics
- Anon. (2006). Peer review on trial. Nature, 441(7094), 668.Google Scholar
- Bornmann, L., & Daniel, H.-D. (2008b). Selecting manuscripts for a high impact journal through peer review: A citation analysis of communications that were accepted by Angewandte Chemie International Edition, or rejected but published elsewhere. Journal of the American Society for Information Science and Technology, 59(11), 1841–1852. doi:10.1002/asi.20901.CrossRefGoogle Scholar
- Bornmann, L., & Daniel, H.-D. (2010c). The manuscript reviewing process—empirical research on review requests, review sequences and decision rules in peer review. Library & Information Science Research, 32(1), 5–12.Google Scholar
- Bornmann, L., Marx, W., Schier, H., Rahm, E., Thor, A., & Daniel, H. D. (2009). Convergent validity of bibliometric Google Scholar data in the field of chemistry. Citation counts for papers that were accepted by Angewandte Chemie International Edition or rejected but published elsewhere, using Google Scholar, Science Citation Index, Scopus, and Chemical Abstracts. Journal of Informetrics, 3(1), 27–35. doi:10.1016/j.joi.2008.11.001.CrossRefGoogle Scholar
- Bornmann, L., Nast, I., & Daniel, H.-D. (2008). Do editors and referees look for signs of scientific misconduct when reviewing manuscripts? A quantitative content analysis of studies that examined review criteria and reasons for accepting and rejecting manuscripts for publication. Scientometrics, 77(3), 415–432.CrossRefGoogle Scholar
- Bornmann, L., Schier, H., Marx, W., & Daniel, H.-D. (2011b). Is interactive open access publishing able to identify high-impact submissions? A study on the predictive validity of Atmospheric Chemistry and Physics by using percentile rank classes. Journal of the American Society for Information Science and Technology, 62(1), 61–71.CrossRefGoogle Scholar
- Chung, C. K., & Pennebaker, J. W. (2007). The psychological function of function words. In K. Fiedler (Ed.), Social communication (pp. 343–359). New York: Psychology Press.Google Scholar
- Fraser, V. J., & Martin, J. G. (2009). Marketing data: Has the rise of impact factor led to the fall of objective language in the scientific article? Respiratory Research, 10. doi:10.1186/1465-9921-10-35.
- Harnad, S. (2000). The invisible hand of peer review. Exploit Interactive (5). http://www.exploit-lib.org/issue5/peer-review/.
- Kahn, J. H., Tobin, R. M., Massey, A. E., & Anderson, J. A. (2007). Measuring emotional expression with the Linguistic Inquiry and Word Count. American Journal of Psychology, 120(2), 263–286.Google Scholar
- Koonin, E., & Lipman, D. (2006). Systems: Reviving a culture of scientific debate. Can ‘open peer review’ work for biologists? Biology Direct is hopeful. Retrieved 21 June 2006, from http://www.nature.com/nature/peerreview/debate/nature05005.html.
- Pennebaker, J. W., Chung, C. K., Ireland, M., Gonzales, A., & Booth, R. J. (2007). The development and psychometric properties of LIWC 2007. Austin, TX/Auckland: University of Texas/University of Auckland.Google Scholar
- Popping, R. (2000). Computer-assisted text analysis. London: Sage.Google Scholar
- Pöschl, U. (2010). Interactive open access publishing and peer review: The effectiveness and perspectives of transparency and self-regulation in scientific communication and evaluation. Liber Quarterly, 19(3/4), 293–314.Google Scholar
- StataCorp. (2011). Stata statistical software: Release 12. College Station, TX: Stata Corporation.Google Scholar