Journal peer review relies on the willingness of researchers to volunteer their time to review manuscripts. However, editors often have difficulty recruiting reviewers, and this difficulty can vary quite substantially among manuscripts. This study examines whether the difficulty recruiting reviewers influences outcomes of the peer review process at six journals of ecology and evolution. The difficulty editors had recruiting reviewers varied substantially among papers, with editors successfully recruiting the first two people invited just 22% of the time, and being declined by two or more invitees for more than half (56%) of reviewed papers. Papers for which editors had more difficulty recruiting reviewers were more likely to be declined at all six journals, with an increase in the odds of acceptance ranging from a low of 3.5 ± 1.2% to a high of 17.3 ± 2.0% for each 10% increase in the proportion of reviewers agreeing to review. Papers for which editors had more difficulty recruiting reviewers were also reviewed less positively at all six journals, and this influence on review scores explained most but not all of the influence of recruitment difficulty on outcomes. Reviewers invited close together in sequence (without many declined invitations between them) were more consistent in the scores they submit than were reviewers invited more greatly separated in sequence, suggesting that editors recruit different kinds of reviewers early versus late in the reviewer invitation sequence. However, the scores submitted by later-recruited reviewers were not less predictive of the editor’s decision than were scores of early-recruited reviewers. The influence of reviewer recruitment difficulty on decisions, although of small effect, should be considered among the diversity of variables that influence outcomes of the editorial and peer review process at academic journals.
This is a preview of subscription content, access via your institution.
Buy single article
Instant access to the full article PDF.
Tax calculation will be finalised during checkout.
Albert, A. Y., Gow, J. L., Cobra, A., & Vines, T. H. (2016). Is it becoming harder to secure reviewers for peer review? A test with data from five ecology journals. Research Integrity and Peer Review, 1(1), 14.
Baveye, P. C., & Trevors, J. T. (2011). How can we encourage peer-reviewing? Water, Air, and Soil pollution, 214, 1–3.
Bornmann, L., Mutz, R., & Daniel, H. D. (2010a). A reliability-generalization study of journal peer reviews: A multilevel meta-analysis of inter-rater reliability and its determinants. PLoS ONE, 5(12), e14331.
Bornmann, L., Weymuth, C., & Daniel, H. D. (2010b). A content analysis of referees’ comments: How do comments on manuscripts rejected by a high-impact journal and later published in either a low-or high-impact journal differ? Scientometrics, 83(2), 493–506.
Breuning, M., Backstrom, J., Brannon, J., Gross, B. I., & Widmeier, M. (2015). Reviewer fatigue? Why scholars decline to review their peers’ work. PS: Political Science and Politics, 48(04), 595–600.
Campos-Arceiz, A., Primack, R. B., & Koh, L. P. (2015). Reviewer recommendations and editors’ decisions for a conservation journal: Is it just a crapshoot? And do Chinese authors get a fair shot? Biological Conservation, 186, 22–27.
Fiske, D. W., & Fogg, L. F. (1990). But the reviewers are making different criticisms of my paper! Diversity and uniqueness in reviewer comments. American Psychologist, 45(5), 591.
Fox, C. W., Albert, A. Y., & Vines, T. H. (2017a). Recruitment of reviewers is becoming harder at some journals: a test of the influence of reviewer fatigue at six journals in ecology and evolution. Research Integrity and Peer Review, 2(1), 3.
Fox, C. W., & Burns, C. S. (2015). The relationship between manuscript title structure and success: Editorial decisions and citation performance for an ecological journal. Ecology and Evolution, 5(10), 1970–1980.
Fox, C. W., Burns, C. S., & Meyer, J. A. (2016a). Editor and reviewer gender influence the peer review process but not peer review outcomes at an ecology journal. Functional Ecology, 30(1), 140–153.
Fox, C. W., Burns, C. S., Muncy, A. D., & Meyer, J. A. (2016b). Gender differences in patterns of authorship do not affect peer review outcomes at an ecology journal. Functional Ecology, 30(1), 126–139.
Fox, C. W., Burns, C. S., Muncy, A. D., & Meyer, J. A. (2017b). Author-suggested reviewers: Gender differences and influences on the peer review process at an ecology journal. Functional Ecology, 31(1), 270–280.
Goldman, H. V. (2015) The scarce peer reviewer and challenges journal editors face. http://www.editage.com/insights/the-scarce-peer-reviewer-and-challenges-journal-editors-face. Accessed 2 May 2017.
Kallmes, K. M., Brinjikji, W., Ahmed, A. T., & Kallmes, D. F. (2017). Difficulty in finding manuscript reviewers is not associated with manuscript acceptance rates: a study of the peer-review process at the journal Radiology. Scientometrics, 111, 971–978. doi:10.1007/s11192-017-2331-0.
Lajtha, K., & Baveye, P. C. (2010). How should we deal with the growing peer-review problem? Biogeochemistry, 101, 1–3.
Mingers, J., & Xu, F. (2010). The drivers of citations in management science journals. European Journal of Operational Research, 205(2), 422–430.
Moed, H. F., & Halevi, G. (2016). On full text download and citation distributions in scientific-scholarly journals. Journal of the Association for Information Science and Technology, 67(2), 412–431.
Perneger, T. V. (2004). Relation between online “hit counts” and subsequent citations: Prospective study of research papers in the BMJ. BMJ, 329(7465), 546–547.
Sattler, D. N., McKnight, P. E., Naney, L., & Mathis, R. (2015). Grant peer review: improving inter-rater reliability with training. PLoS ONE, 10(6), e0130450.
Sense About Science (2009). Peer Review Survey 2009. http://archive.senseaboutscience.org/pages/peer-review-survey-2009.html. Accessed 2 May 2017.
Stamm, T., Meyer, U., Wiesmann, H. P., Kleinheinz, J., Cehreli, M., & Cehreli, Z. C. (2007). A retrospective analysis of submissions, acceptance rate, open peer review operations, and prepublication bias of the multidisciplinary open access journal Head & Face Medicine. Head and Face Medicine, 3(1), 27.
Stremersch, S., Verniers, I., & Verhoef, P. C. (2007). The quest for citations: Drivers of article impact. Journal of Marketing, 71(3), 171–193.
Thelwall, M., Haustein, S., Larivière, V., & Sugimoto, C. R. (2013). Do altmetrics work? Twitter and ten other social web services. PLoS ONE, 8(5), e64841.
Tite, L., & Schroter, S. (2007). Why do peer reviewers decline to review? A survey. Journal of Epidemiology and Community Health, 61(1), 9–12.
Vaughan, L., Tang, J., & Yang, R. (2017). Investigating disciplinary differences in the relationships between citations and downloads. Scientometrics, 111(3), 1533–1545.
Vines, T., Rieseberg, L., & Smith, H. (2010). No crisis in supply of peer reviewers. Nature, 468(7327), 1041.
Ware, M., & Monkman, M. (2008). Peer Review in scholarly journals: An international study into the perspective of the scholarly community. Bristol: Mark Ware Consulting.
Willis, M. (2016). Why do peer reviewers decline to review manuscripts? A study of reviewer invitation responses. Learned Publishing, 29(1), 5–7.
Zaharie, M. A., & Osoian, C. L. (2016). Peer review motivation frames: A qualitative approach. European Management Journal, 34(1), 69–79.
The British Ecological Society and the Society for the Study of Evolution provided permission to access their databases for this peer review analysis. Katie Simmons assisted with extracting the reviewer database for Evolution, and Emilie Aimé, Christopher Grieves, Kate Harrison, Simon Hoggart, Jennifer Meyer, Erika Newton, Alice Plane, James Ross and Leila Walker extracted the reviewer databases for the British Ecological Society journals. Emilie Aimé, C. Sean Burns, Allyssa Kilanowski, Melise Lecheta, Josiah Ritchey and Boris Sauterey provided helpful comments on earlier drafts of this manuscript. This work was approved by the University of Kentucky’s Institutional Review Board (IRB 15–0890). C. Fox is Executive Editor of one of the journals examined in this study (Functional Ecology).
About this article
Cite this article
Fox, C.W. Difficulty of recruiting reviewers predicts review scores and editorial decisions at six journals of ecology and evolution. Scientometrics 113, 465–477 (2017). https://doi.org/10.1007/s11192-017-2489-5
- Bias in peer review
- Inter-rater reliability
- Peer review