Are three heads better than two? How the number of reviewers and editor behavior affect the rejection rate
- David M. Schultz
- … show all 1 hide
Rent the article at a discountRent now
* Final gross prices may vary according to local VAT.Get Access
Editors of peer-reviewed journals obtain recommendations from peer reviewers as guidance in deciding upon the suitability of a submitted manuscript for publication. To investigate whether the number of reviewers used by an editor affects the rate at which manuscripts are rejected, 500 manuscripts submitted to Monthly Weather Review during 15.5 months in 2007–2008 were examined. Two and three reviewers were used for 306 and 155 manuscripts, respectively (92.2% of all manuscripts). Rejection rates for initial decisions and final decisions were not significantly different whether two or three reviewers were used. Manuscripts with more reviewers did not spend more rounds in review or have different rejection rates at each round. The results varied by editor, however, with some editors rejecting more two-reviewer manuscripts and others rejecting more three-reviewer manuscripts. Editors described using their scientific expertise in the decision-making process, either in determining the number of reviews to be sought or in making decisions once the reviews were received, approaches that differ from that of relying purely upon reviewer agreement as reported previously in the literature. A simple model is constructed for three decision-making strategies for editors: rejection when all reviewers recommend rejection, rejection when any reviewer recommends rejection, and rejection when a majority of reviewers recommend rejection. By plotting the probability of reviewer rejection against the probability of editor rejection, the decision-making process can be graphically illustrated, demonstrating that, for this dataset, editors are likely to reject a manuscript when any reviewer recommends rejection.
- Bornmann, L., & Daniel, H.-D. (2009). The luck of the referee draw: The effect of exchanging reviews. Learned Publishing, 22, 117–125. CrossRef
- Cicchetti, D. V. (1991). Reflections from the peer review mirror. Behavioral and Brain Sciences, 14(1), 167–186.
- Eberley, S., & Warner, W. K. (1990). Fields or subfields of knowledge: Rejection rates and reviewer agreement in peer review. The American Sociologist, 21(3), 217–231. CrossRef
- Ernst, E., Saradeth, T., & Resch, K. L. (1993). Drawbacks of peer review. Nature, 363, 296. CrossRef
- Frey, B. S. (2003). Publishing as prostitution?—Choosing between one’s own ideas and academic success. Public Choice, 116, 205–223. CrossRef
- Glenn, N. D. (1976). The journal article review process: Some proposals for change. The American Sociologist, 11, 179–185.
- Hargens, L. L. (1988). Scholarly consensus and journal rejection rates. American Sociological Review, 53, 139–151. CrossRef
- Hargens, L. L., & Herting, J. R. (1990). Neglected considerations in the analysis of agreement among journal referees. Scientometrics, 19(1–2), 91–106. CrossRef
- Jorgensen, D. P., Rauber, R. M., Heideman, K. F., Fernau, M. E., Friedman, M. A., & Schein, A. L. (2007). The evolving publication process of the AMS. Bulletin of the American Meteorological Society, 88, 1122–1134. CrossRef
- Lindsey, D. (1988). Assessing precision in the manuscript review process: A little better than a dice roll. Scientometrics, 14(1–2), 75–82. CrossRef
- O’Brien, R. M. (1991). The reliability of composites of referee assessments of manuscripts. Social Science Research, 20, 319–328. CrossRef
- Schultz, D. M. (2009). Rejection rates for journals publishing atmospheric science. Bulletin of the American Meteorological Society, 90, accepted with revisions.
- Seitter, K. L. (2002). Opening access to the AMS Journals Online. Bulletin of the American Meteorological Society, 83, 1361. CrossRef
- Strayhorn, J., Jr., McDermott, J. F., Jr., & Tanguay, P. (1993). An intervention to improve the reliability of manuscript reviews for the Journal of the American Academy of Child and Adolescent Psychiatry. American Journal of Psychiatry, 150, 947–952.
- Weller, A. C. (2001). Editorial peer review: Its strengths and weaknesses (342 pp.). Medford, NJ: ASIST Monograph Series, Information Today, Inc.
- Zuckerman, H., & Merton, R. K. (1971). Patterns of evaluation in science: Institutionalisation, structure and functions of the referee system. Minerva, 9, 66–100. CrossRef
- Are three heads better than two? How the number of reviewers and editor behavior affect the rejection rate
Volume 84, Issue 2 , pp 277-292
- Cover Date
- Print ISSN
- Online ISSN
- Springer Netherlands
- Additional Links
- Monthly Weather Review
- Peer review
- Rounds of reviews
- Reviewer agreement
- Industry Sectors
- David M. Schultz (1)
- Author Affiliations
- 1. Division of Atmospheric Sciences and Geophysics, Department of Physics, University of Helsinki, and Finnish Meteorological Institute, Erik Palménin Aukio 1, P.O. Box 503, 00101, Helsinki, Finland