Skip to main content
SpringerLink
Log in

In public peer review of submitted manuscripts, how do reviewer comments differ from comments written by interested members of the scientific community? A content analysis of comments written for Atmospheric Chemistry and Physics

  • Published:
Scientometrics Aims and scope Submit manuscript

Abstract

Whereas in traditional peer review a few selected researchers (peers) are included in the manuscript review process, public peer review includes both invited reviewers (who write ‘reviewer comments’) and interested members of the scientific community who write comments (‘short comments’). Available to us for this investigation are 390 reviewer comments and short comments assessing 119 manuscripts submitted to the journal Atmospheric Chemistry and Physics (ACP). We conducted a content analysis of these comments to determine differences in the main thematic areas considered by the scientists in their assessment comments. The results of the analysis show that in contrast to interested members of the scientific community, reviewers focus mainly on (1) the formal qualities of a manuscript, such as writing style, (2) the conclusions drawn in a manuscript, and (3) the future “gain” that could result from publication of a manuscript. All in all, it appears that ‘reviewer comments’ better than ‘short comments’ by interested members of the scientific community support the two main functions of peer review: selection and improvement of what is published.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Similar content being viewed by others

References

  • American Association for the Advancement of Science. (1998). Report in AAAS/UNESCO/ICSU Workshop on Developing Practices and Standards for Electronic Publishing in Science. Paris, October 12–14, 1998. Washington, DC: The American Association for the Advancement of Science.

  • Anon. (2006). Peer review on trial. Nature, 441(7094), 668.

    Google Scholar 

  • Bakanic, V., McPhail, C., & Simon, R. J. (1989). Mixed messages: Referees’ comments on the manuscripts they review. Sociological Quarterly, 30(4), 639–654.

    Article  Google Scholar 

  • Bedeian, A. G. (2004). Peer review and the social construction of knowledge in the management discipline. Academy of Management Learning and Education, 3(2), 198–216.

    Article  Google Scholar 

  • Bornmann, L. (2011). Scientific peer review. Annual Review of Information Science and Technology, 45, 199–245.

    Google Scholar 

  • Bornmann, L., & Daniel, H.-D. (2008). The effectiveness of the peer review process: Inter-referee agreement and predictive validity of manuscript refereeing at Angewandte Chemie. Angewandte Chemie International Edition, 47(38), 7173–7178. doi:10.1002/anie.200800513.

    Article  Google Scholar 

  • Bornmann, L., Marx, W., Schier, H., Thor, A., & Daniel, H.-D. (2010a). From black box to white box at open access journals: Predictive validity of manuscript reviewing and editorial decisions at Atmospheric Chemistry and Physics. Research Evaluation, 19(2), 105–118.

    Article  Google Scholar 

  • Bornmann, L., Mutz, R., & Daniel, H.-D. (2008a). Latent Markov modeling applied to grant peer review. Journal of Informetrics, 2(3), 217–228.

    Article  Google Scholar 

  • Bornmann, L., Mutz, R., & Daniel, H.-D. (2009). The influence of the applicants’ gender on the modeling of a peer review process by using latent Markov models. Scientometrics, 81(2), 407–411.

    Article  Google Scholar 

  • Bornmann, L., Nast, I., & Daniel, H.-D. (2008b). Do editors and referees look for signs of scientific misconduct when reviewing manuscripts? A quantitative content analysis of studies that examined review criteria and reasons for accepting and rejecting manuscripts for publication. Scientometrics, 77(3), 415–432.

    Article  Google Scholar 

  • Bornmann, L., Weymuth, C., & Daniel, H.-D. (2010b). A content analysis of referees’ comments: How do comments on manuscripts rejected by a high-impact journal and later published in either a low- or high-impact journal differ? Scientometrics, 83(2), 493–506.

    Article  Google Scholar 

  • Cohen, J. (1960). A coefficient of agreement for nominal scales. Educational and Psychological Measurement, 20(1), 37–46.

    Article  Google Scholar 

  • Daniel, H.-D. (1993). Guardians of science. Fairness and reliability of peer review. Weinheim: Wiley–VCH.

    Google Scholar 

  • Fleiss, J. (1981). Statistical methods for rates and proportions. New York, NY: Wiley–VCH.

    MATH  Google Scholar 

  • Gosden, H. (2003). ‘Why not give us the full story?’: Functions of referees’ comments in peer reviews of scientific research papers. Journal of English for Academic Purposes, 2(2), 87–101.

    Article  Google Scholar 

  • Harnad, S. (1978). Inaugural editorial. Behavioral and Brain Sciences, 1(1).

  • Harnad, S. (2000). The invisible hand of peer review. Exploit Interactive (5).

  • Hosmer, D. W., & Lemeshow, S. (2000). Applied logistic regression (2nd ed.). Chichester: Wiley.

    Book  MATH  Google Scholar 

  • House of Commons; Science and Technology Committee on alternative models for the federal funding of science. (2011). Peer review in scientific publications. Eighth Report of Session 2010-12. Volume I: Report, together with formal minutes, oral and written evidence. London: House of Commons.

  • Janeway, C. A. J. (1990). JMCI: The last issue. Journal of Molecular and Cellular Immunology, 4, 293.

    Google Scholar 

  • Krampen, G., Becker, R., Wahner, U., & Montada, L. (2007). On the validity of citation counting in science evaluation: Content analyses of references and citations in psychological publications. Scientometrics, 71(2), 191–202.

    Article  Google Scholar 

  • Kumar, P., Rafiq, I., & Imam, B. (2011). Negotiation on the assessment of research articles with academic reviewers: Application of peer-review approach of teaching. Higher Education, 62(3), 315–332. doi:10.1007/s10734-010-9390-y.

    Article  Google Scholar 

  • Kupfersmid, J. (1988). Improving what is published: A model in search of an editor. American Psychologist, 43(8), 635–642.

    Article  Google Scholar 

  • Landis, J. R., & Koch, G. G. (1977). Measurement of observer agreement for categorical data. Biometrics, 33(1), 159–174.

    Article  MathSciNet  MATH  Google Scholar 

  • McCormack, N. (2009). Peer review and legal publishing: What law librarians need to know about open, single-blind, and double-blind reviewing. Law Library Journal, 101(1), 59–70.

    Google Scholar 

  • Mizzaro, S. (2003). Quality control in scholarly publishing: Anew proposal. Journal of the American Society for Information Science and Technology, 54(11), 989–1005. doi:10.1002/Asi.10296.

    Article  Google Scholar 

  • Morrison, J. (2006). The case for open peer review. Medical Education, 40(9), 830–831. doi:10.1111/j.1365-2929.2006.02573.x.

    Article  Google Scholar 

  • Pöschl, U. (2004). Interactive journal concept for improved scientific publishing and quality assurance. Learned Publishing, 17(2), 105–113. doi:10.1087/095315104322958481.

    Article  Google Scholar 

  • Pöschl, U. (2010). Interactive open access publishing and peer review: The effectiveness and perspectives of transparency and self-regulation in scientific communication and evaluation. Liber Quarterly, 19(3/4), 293–314.

    Google Scholar 

  • Rabe-Hesketh, S., & Everitt, B. (2004). A handbook of statistical analyses using Stata. Boca Raton, FL: Chapman & Hall/CRC.

    MATH  Google Scholar 

  • Schultz, D. M. (2010). Rejection rates for journals publishing atmospheric science. Bulletin of the American Meteorological Society, 91(2), 231–243. doi:10.1175/2009BAMS2908.1.

    Article  Google Scholar 

  • Shashok, K. (2008). Content and communication: How can peer review provide helpful feedback about the writing? BMC Medical Research Methods, 8(3).

  • Shum, S. B., & Sumner, T. (2001). JIME: An interactive journal for interactive media. First Monday, 6(2–5).

  • Smith, R. (1999). Opening up BMJ peer review. British Medical Journal, 318(7175), 4–5. doi:10.1136/bmj.318.7175.4.

    Article  Google Scholar 

  • StataCorp. (2011). Stata statistical software: Release 12. College Station, TX: Stata Corporation.

    Google Scholar 

  • Sumner, T., & Shum, S. B. (1996). Open peer review & argumentation: Loosening the paper chains on journals. ARIADNE, 5.

  • van Rooyen, S., Godlee, F., Evans, S., Black, N., & Smith, R. (1999). Effect of open peer review on quality of reviews and on reviewers’ recommendations: Arandomised trial. British Medical Journal, 318(7175), 23–27.

    Article  Google Scholar 

  • von Eye, A., & Mun, E. Y. (2005). Analyzing rater agreement. Manifest variable methods. Mahwah, NJ: Lawrence Erlbaum Associates.

    Google Scholar 

  • Walsh, E., Rooney, M., Appleby, L., & Wilkinson, G. (2000). Open peer review: Arandomised controlled trial. British Journal of Psychiatry, 176, 47–51.

    Article  Google Scholar 

  • Wing, J. M., & Chi, E. H. (2011). Reviewing peer review. Communications of the ACM, 54(7), 10–11. doi:10.1145/1965724.1965728.

    Article  Google Scholar 

  • Xin, H. (2006). Online sleuths challenge cell paper. Science, 314(5806), 1669. doi:10.1126/science.314.5806.1669a.

Download references

Acknowledgments

The research project, which is investigating quality assurance of interactive open access journals, is supported by a grant from the Max Planck Society (Munich, Germany). We thank Dr. Ulrich Pöschl, Chief Executive Editor of Atmospheric Chemistry and Physics, the Editorial Board of Atmospheric Chemistry and Physics, and Copernicus Publications (Katlenburg-Lindau, Germany) for permission to conduct the evaluation of the selection process of the journal, and we are grateful to the members of Copernicus Systems + Technology (Berlin, Germany) for their generous technical support during the carrying out of the study.

Author information

Author notes

  1. Hanna Herich

    Present address: Swiss Federal Laboratories for Materials Science and Technology, Empa, Überlandstrasse 129, 8600, Dübendorf, Switzerland

Authors and Affiliations

  1. Max Planck Society, Administrative Headquarters, Hofgartenstr. 8, 80539, Munich, Germany

    Lutz Bornmann

  2. Institute of Atmospheric and Climate Science, ETH Zurich, Universitätstrasse 16, 8092, Zurich, Switzerland

    Hanna Herich & Hanna Joos

  3. Professorship for Social Psychology and Research on Higher Education, ETH Zurich, Mühlegasse 21, 8001, Zurich, Switzerland

    Hans-Dieter Daniel

  4. Evaluation Office, University of Zurich, Mühlegasse 21, 8001, Zurich, Switzerland

    Hans-Dieter Daniel

Authors
  1. Lutz Bornmann
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Hanna Herich
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Hanna Joos
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Hans-Dieter Daniel
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Lutz Bornmann.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Bornmann, L., Herich, H., Joos, H. et al. In public peer review of submitted manuscripts, how do reviewer comments differ from comments written by interested members of the scientific community? A content analysis of comments written for Atmospheric Chemistry and Physics . Scientometrics 93, 915–929 (2012). https://doi.org/10.1007/s11192-012-0731-8

Download citation

  • Received:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11192-012-0731-8

Keywords

Search

Navigation