Skip to main content

Difficulty of recruiting reviewers predicts review scores and editorial decisions at six journals of ecology and evolution

Abstract

Journal peer review relies on the willingness of researchers to volunteer their time to review manuscripts. However, editors often have difficulty recruiting reviewers, and this difficulty can vary quite substantially among manuscripts. This study examines whether the difficulty recruiting reviewers influences outcomes of the peer review process at six journals of ecology and evolution. The difficulty editors had recruiting reviewers varied substantially among papers, with editors successfully recruiting the first two people invited just 22% of the time, and being declined by two or more invitees for more than half (56%) of reviewed papers. Papers for which editors had more difficulty recruiting reviewers were more likely to be declined at all six journals, with an increase in the odds of acceptance ranging from a low of 3.5 ± 1.2% to a high of 17.3 ± 2.0% for each 10% increase in the proportion of reviewers agreeing to review. Papers for which editors had more difficulty recruiting reviewers were also reviewed less positively at all six journals, and this influence on review scores explained most but not all of the influence of recruitment difficulty on outcomes. Reviewers invited close together in sequence (without many declined invitations between them) were more consistent in the scores they submit than were reviewers invited more greatly separated in sequence, suggesting that editors recruit different kinds of reviewers early versus late in the reviewer invitation sequence. However, the scores submitted by later-recruited reviewers were not less predictive of the editor’s decision than were scores of early-recruited reviewers. The influence of reviewer recruitment difficulty on decisions, although of small effect, should be considered among the diversity of variables that influence outcomes of the editorial and peer review process at academic journals.

This is a preview of subscription content, access via your institution.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5

References

  1. Albert, A. Y., Gow, J. L., Cobra, A., & Vines, T. H. (2016). Is it becoming harder to secure reviewers for peer review? A test with data from five ecology journals. Research Integrity and Peer Review, 1(1), 14.

    Article  Google Scholar 

  2. Baveye, P. C., & Trevors, J. T. (2011). How can we encourage peer-reviewing? Water, Air, and Soil pollution, 214, 1–3.

    Article  Google Scholar 

  3. Bornmann, L., Mutz, R., & Daniel, H. D. (2010a). A reliability-generalization study of journal peer reviews: A multilevel meta-analysis of inter-rater reliability and its determinants. PLoS ONE, 5(12), e14331.

    Article  Google Scholar 

  4. Bornmann, L., Weymuth, C., & Daniel, H. D. (2010b). A content analysis of referees’ comments: How do comments on manuscripts rejected by a high-impact journal and later published in either a low-or high-impact journal differ? Scientometrics, 83(2), 493–506.

    Article  Google Scholar 

  5. Breuning, M., Backstrom, J., Brannon, J., Gross, B. I., & Widmeier, M. (2015). Reviewer fatigue? Why scholars decline to review their peers’ work. PS: Political Science and Politics, 48(04), 595–600.

    Google Scholar 

  6. Campos-Arceiz, A., Primack, R. B., & Koh, L. P. (2015). Reviewer recommendations and editors’ decisions for a conservation journal: Is it just a crapshoot? And do Chinese authors get a fair shot? Biological Conservation, 186, 22–27.

    Article  Google Scholar 

  7. Fiske, D. W., & Fogg, L. F. (1990). But the reviewers are making different criticisms of my paper! Diversity and uniqueness in reviewer comments. American Psychologist, 45(5), 591.

    Article  Google Scholar 

  8. Fox, C. W., Albert, A. Y., & Vines, T. H. (2017a). Recruitment of reviewers is becoming harder at some journals: a test of the influence of reviewer fatigue at six journals in ecology and evolution. Research Integrity and Peer Review, 2(1), 3.

    Article  Google Scholar 

  9. Fox, C. W., & Burns, C. S. (2015). The relationship between manuscript title structure and success: Editorial decisions and citation performance for an ecological journal. Ecology and Evolution, 5(10), 1970–1980.

    Article  Google Scholar 

  10. Fox, C. W., Burns, C. S., & Meyer, J. A. (2016a). Editor and reviewer gender influence the peer review process but not peer review outcomes at an ecology journal. Functional Ecology, 30(1), 140–153.

    Article  Google Scholar 

  11. Fox, C. W., Burns, C. S., Muncy, A. D., & Meyer, J. A. (2016b). Gender differences in patterns of authorship do not affect peer review outcomes at an ecology journal. Functional Ecology, 30(1), 126–139.

    Article  Google Scholar 

  12. Fox, C. W., Burns, C. S., Muncy, A. D., & Meyer, J. A. (2017b). Author-suggested reviewers: Gender differences and influences on the peer review process at an ecology journal. Functional Ecology, 31(1), 270–280.

    Article  Google Scholar 

  13. Goldman, H. V. (2015) The scarce peer reviewer and challenges journal editors face. http://www.editage.com/insights/the-scarce-peer-reviewer-and-challenges-journal-editors-face. Accessed 2 May 2017.

  14. Kallmes, K. M., Brinjikji, W., Ahmed, A. T., & Kallmes, D. F. (2017). Difficulty in finding manuscript reviewers is not associated with manuscript acceptance rates: a study of the peer-review process at the journal Radiology. Scientometrics, 111, 971–978. doi:10.1007/s11192-017-2331-0.

    Article  Google Scholar 

  15. Lajtha, K., & Baveye, P. C. (2010). How should we deal with the growing peer-review problem? Biogeochemistry, 101, 1–3.

    Article  Google Scholar 

  16. Mingers, J., & Xu, F. (2010). The drivers of citations in management science journals. European Journal of Operational Research, 205(2), 422–430.

    Article  MATH  Google Scholar 

  17. Moed, H. F., & Halevi, G. (2016). On full text download and citation distributions in scientific-scholarly journals. Journal of the Association for Information Science and Technology, 67(2), 412–431.

    Article  Google Scholar 

  18. Perneger, T. V. (2004). Relation between online “hit counts” and subsequent citations: Prospective study of research papers in the BMJ. BMJ, 329(7465), 546–547.

    Article  Google Scholar 

  19. Sattler, D. N., McKnight, P. E., Naney, L., & Mathis, R. (2015). Grant peer review: improving inter-rater reliability with training. PLoS ONE, 10(6), e0130450.

    Article  Google Scholar 

  20. Sense About Science (2009). Peer Review Survey 2009. http://archive.senseaboutscience.org/pages/peer-review-survey-2009.html. Accessed 2 May 2017.

  21. Stamm, T., Meyer, U., Wiesmann, H. P., Kleinheinz, J., Cehreli, M., & Cehreli, Z. C. (2007). A retrospective analysis of submissions, acceptance rate, open peer review operations, and prepublication bias of the multidisciplinary open access journal Head & Face Medicine. Head and Face Medicine, 3(1), 27.

    Article  Google Scholar 

  22. Stremersch, S., Verniers, I., & Verhoef, P. C. (2007). The quest for citations: Drivers of article impact. Journal of Marketing, 71(3), 171–193.

    Article  Google Scholar 

  23. Thelwall, M., Haustein, S., Larivière, V., & Sugimoto, C. R. (2013). Do altmetrics work? Twitter and ten other social web services. PLoS ONE, 8(5), e64841.

    Article  Google Scholar 

  24. Tite, L., & Schroter, S. (2007). Why do peer reviewers decline to review? A survey. Journal of Epidemiology and Community Health, 61(1), 9–12.

    Article  Google Scholar 

  25. Vaughan, L., Tang, J., & Yang, R. (2017). Investigating disciplinary differences in the relationships between citations and downloads. Scientometrics, 111(3), 1533–1545.

    Article  Google Scholar 

  26. Vines, T., Rieseberg, L., & Smith, H. (2010). No crisis in supply of peer reviewers. Nature, 468(7327), 1041.

    Article  Google Scholar 

  27. Ware, M., & Monkman, M. (2008). Peer Review in scholarly journals: An international study into the perspective of the scholarly community. Bristol: Mark Ware Consulting.

    Google Scholar 

  28. Willis, M. (2016). Why do peer reviewers decline to review manuscripts? A study of reviewer invitation responses. Learned Publishing, 29(1), 5–7.

    MathSciNet  Article  Google Scholar 

  29. Zaharie, M. A., & Osoian, C. L. (2016). Peer review motivation frames: A qualitative approach. European Management Journal, 34(1), 69–79.

    Article  Google Scholar 

Download references

Acknowledgements

The British Ecological Society and the Society for the Study of Evolution provided permission to access their databases for this peer review analysis. Katie Simmons assisted with extracting the reviewer database for Evolution, and Emilie Aimé, Christopher Grieves, Kate Harrison, Simon Hoggart, Jennifer Meyer, Erika Newton, Alice Plane, James Ross and Leila Walker extracted the reviewer databases for the British Ecological Society journals. Emilie Aimé, C. Sean Burns, Allyssa Kilanowski, Melise Lecheta, Josiah Ritchey and Boris Sauterey provided helpful comments on earlier drafts of this manuscript. This work was approved by the University of Kentucky’s Institutional Review Board (IRB 15–0890). C. Fox is Executive Editor of one of the journals examined in this study (Functional Ecology).

Author information

Affiliations

Authors

Corresponding author

Correspondence to Charles W. Fox.

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Fox, C.W. Difficulty of recruiting reviewers predicts review scores and editorial decisions at six journals of ecology and evolution. Scientometrics 113, 465–477 (2017). https://doi.org/10.1007/s11192-017-2489-5

Download citation

Keywords

  • Bias in peer review
  • Inter-rater reliability
  • Peer review
  • Referees