Scientometrics

, Volume 84, Issue 2, pp 277–292 | Cite as

Are three heads better than two? How the number of reviewers and editor behavior affect the rejection rate

Article

Abstract

Editors of peer-reviewed journals obtain recommendations from peer reviewers as guidance in deciding upon the suitability of a submitted manuscript for publication. To investigate whether the number of reviewers used by an editor affects the rate at which manuscripts are rejected, 500 manuscripts submitted to Monthly Weather Review during 15.5 months in 2007–2008 were examined. Two and three reviewers were used for 306 and 155 manuscripts, respectively (92.2% of all manuscripts). Rejection rates for initial decisions and final decisions were not significantly different whether two or three reviewers were used. Manuscripts with more reviewers did not spend more rounds in review or have different rejection rates at each round. The results varied by editor, however, with some editors rejecting more two-reviewer manuscripts and others rejecting more three-reviewer manuscripts. Editors described using their scientific expertise in the decision-making process, either in determining the number of reviews to be sought or in making decisions once the reviews were received, approaches that differ from that of relying purely upon reviewer agreement as reported previously in the literature. A simple model is constructed for three decision-making strategies for editors: rejection when all reviewers recommend rejection, rejection when any reviewer recommends rejection, and rejection when a majority of reviewers recommend rejection. By plotting the probability of reviewer rejection against the probability of editor rejection, the decision-making process can be graphically illustrated, demonstrating that, for this dataset, editors are likely to reject a manuscript when any reviewer recommends rejection.

Keywords

Journal Editor Monthly Weather Review Peer review Rounds of reviews Reviewer agreement 

PACS

01.20 

Mathematical Subject Classification (2000)

00A99 

JEL Codes

Z00 

References

  1. Bornmann, L., & Daniel, H.-D. (2009). The luck of the referee draw: The effect of exchanging reviews. Learned Publishing, 22, 117–125.CrossRefGoogle Scholar
  2. Cicchetti, D. V. (1991). Reflections from the peer review mirror. Behavioral and Brain Sciences, 14(1), 167–186.Google Scholar
  3. Eberley, S., & Warner, W. K. (1990). Fields or subfields of knowledge: Rejection rates and reviewer agreement in peer review. The American Sociologist, 21(3), 217–231.CrossRefGoogle Scholar
  4. Ernst, E., Saradeth, T., & Resch, K. L. (1993). Drawbacks of peer review. Nature, 363, 296.CrossRefGoogle Scholar
  5. Frey, B. S. (2003). Publishing as prostitution?—Choosing between one’s own ideas and academic success. Public Choice, 116, 205–223.CrossRefGoogle Scholar
  6. Glenn, N. D. (1976). The journal article review process: Some proposals for change. The American Sociologist, 11, 179–185.Google Scholar
  7. Hargens, L. L. (1988). Scholarly consensus and journal rejection rates. American Sociological Review, 53, 139–151.CrossRefGoogle Scholar
  8. Hargens, L. L., & Herting, J. R. (1990). Neglected considerations in the analysis of agreement among journal referees. Scientometrics, 19(1–2), 91–106.CrossRefGoogle Scholar
  9. Jorgensen, D. P., Rauber, R. M., Heideman, K. F., Fernau, M. E., Friedman, M. A., & Schein, A. L. (2007). The evolving publication process of the AMS. Bulletin of the American Meteorological Society, 88, 1122–1134.CrossRefGoogle Scholar
  10. Lindsey, D. (1988). Assessing precision in the manuscript review process: A little better than a dice roll. Scientometrics, 14(1–2), 75–82.CrossRefGoogle Scholar
  11. O’Brien, R. M. (1991). The reliability of composites of referee assessments of manuscripts. Social Science Research, 20, 319–328.CrossRefGoogle Scholar
  12. Schultz, D. M. (2009). Rejection rates for journals publishing atmospheric science. Bulletin of the American Meteorological Society, 90, accepted with revisions.Google Scholar
  13. Seitter, K. L. (2002). Opening access to the AMS Journals Online. Bulletin of the American Meteorological Society, 83, 1361.CrossRefGoogle Scholar
  14. Strayhorn, J., Jr., McDermott, J. F., Jr., & Tanguay, P. (1993). An intervention to improve the reliability of manuscript reviews for the Journal of the American Academy of Child and Adolescent Psychiatry. American Journal of Psychiatry, 150, 947–952.Google Scholar
  15. Weller, A. C. (2001). Editorial peer review: Its strengths and weaknesses (342 pp.). Medford, NJ: ASIST Monograph Series, Information Today, Inc.Google Scholar
  16. Zuckerman, H., & Merton, R. K. (1971). Patterns of evaluation in science: Institutionalisation, structure and functions of the referee system. Minerva, 9, 66–100.CrossRefGoogle Scholar

Copyright information

© Akadémiai Kiadó, Budapest, Hungary 2009

Authors and Affiliations

  1. 1.Division of Atmospheric Sciences and Geophysics, Department of PhysicsUniversity of Helsinki, and Finnish Meteorological InstituteHelsinkiFinland

Personalised recommendations