Skip to main content
SpringerLink
Log in

Gender differences among active reviewers: an investigation based on publons

  • Published:
Scientometrics Aims and scope Submit manuscript

Abstract

Peer review of scientific manuscripts before publication is essential in scholarly publishing, and most active researchers hold relationships with a number of journals as both an author and a reviewer. There have been several studies focusing on gender balance in academic research and authorship, but fewer studies on our role as reviewers. Publons is a commercial website run by Clarivate Analytics that allows researchers to track and verify their peer review activities and be recognized for it. The platform features over 2 million researchers and 6.9 million reviews for more than 5,000 partnered journals, listing the most active reviewers as “top reviewers”. Our study focuses on gender representation in this ‘top reviewer’ group while also looking at the countries, regions and research fields they represent, as well as the relationship between their roles as authors and reviewers. The results show that male reviewers dominate in almost all countries, regions, and research fields. Male reviewers generally contribute to review work more frequently than females; however, female reviewers write longer reviews. The correlations between reviewing activity and research activity are generally weak overall and within specific research fields. This may reflect that active reviewers are not necessarily the most productive researchers in their fields. What clearly emerges from our results is the need for more concern over gender representation in the quality assurance and gatekeeping functions of scholarly publishing.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15

Similar content being viewed by others

Notes

  1. https://publons.com/about/mission.

  2. See https://publons.com/community/awards/awards-categories-2018/, https://publons.com/awards/peer-review/2019/methodology/.

  3. https://publons.com/benefits/reviewers/how

  4. https://orcid.org/.

  5. There are 317 reviewers’ Researcher IDs with no retrieval records in WoS. 206 out of 317 reviewers have the ORCID. Since the search results for Researcher ID and ORCID are verified to be consistent in WoS, we use the ORCID of 206 reviewers to obtain supplementary publication data.

  6. We calculated the reviewer’s academic experience based on the year of the first publication. Among all reviewers, 26 reviewers’ have more than 60 years of experience (7 of them are even more than 90 years), which is abnormal in practice. Therefore, publications of these 26 reviewers are manually checked and the ones do not belong to the corresponding reviewers are removed.

  7. http://eigenfactor.org/projects/gender/#.

  8. See https://pypi.python.org/pypi/gender-guesser/.

  9. As shown in Fig. 2, we already manually marked 4753 reviewers with male or female based on their images provided by Publons. We further applied gender-guesser to assign gender based on the name of these reviewers. The overlap ratio was 67.52% (3209) and the share of misclassification is 2.38% (113). There are 8.02% (381) names responded with “andy” and 22.09% (1050) names responded with “unknown”.

  10. Namsor (https://www.namsor.com/), This tool was used in Elsevier’s report The Researcher Journey Through a Gender Lens (2020).

References

  • Abramo, G., D’Angelo, C., & Caprasecca, A. (2009). Gender differences in research productivity: A bibliometric analysis of the Italian academic system. Scientometrics, 79(3), 517–539.

    Google Scholar 

  • Addis, E., & Villa, P. (2003). The Editorial boards of Italian economics journals: Women, gender, and social networking. Feminist Economics, 9(1), 75–91.

    Google Scholar 

  • Adler, E., Hobbs, A., Dhaliwal, G., & Babik, J. M. (2020). Gender differences in authorship of clinical problem-solving articles. Journal of Hospital Medicine, 15(8), 475–478.

    Google Scholar 

  • Amrein, K., Langmann, A., Fahrleitner-Pammer, A., Pieber, T. R., & Zollner-Schwetz, I. (2011). Women underrepresented on editorial boards of 60 major medical journals. Gender Medicine, 8(6), 378–387.

    Google Scholar 

  • Arruda, D., Bezerra, F., Neris, V., Rocha De Toro, P., & Wainera, J. (2009). Brazilian computer science research: Gender and regional distributions. Scientometrics, 79(3), 651–665.

    Google Scholar 

  • Baucom, R. S., Geraldes, A. M., & Rieseberg, L. H. (2019). Some perspective on molecular ecology perspectives: Are women being left out? Molecular Ecology, 28(10), 2451–2455.

    Google Scholar 

  • Bendels, M. H., Müller, R., Brueggmann, D., & Groneberg, D. A. (2018). Gender disparities in high-quality research revealed by Nature Index journals. PloS ONE, 13(1), e0189136.

    Google Scholar 

  • Bianchi, F., & Squazzoni, F. (2015). Is three better than one? simulating the effect of reviewer selection and behavior on the quality and efficiency of peer review. In Proceedings of the 2015 Winter Simulation Conference. (pp.4081–4089).

  • Black, N., van Rooyen, S., Godlee, F., Smith, R., & Evans, S. (1998). What makes a good reviewer and a good review for a general medical journal? JAMA, 280(3), 231–233.

    Google Scholar 

  • Borja, Á. (2015). Is there gender bias in the peer-review process in several Elsevier’s marine journals? Marine Pollution Bulletin, 96(1–2), 1–2.

    Google Scholar 

  • Bornmann, L., & Williams, R. (2017). Can the journal impact factor be used as a criterion for the selection of junior researchers? A large-scale empirical study based on ResearcherID data. Journal of Informetrics, 11(3), 788–799.

    Google Scholar 

  • Bradshaw, C. J., & Courchamp, F. (2018). Gender bias when assessing recommended ecology articles. Rethinking Ecology, 3, 1–12.

    Google Scholar 

  • Buckley, H. L., Sciligo, A. R., Adair, K. L., Case, B. S., & Monks, J. M. (2014). Is there gender bias in reviewer selection and publication success rates for the New Zealand Journal of Ecology? New Zealand Journal of Ecology, 38(2), 335–339.

    Google Scholar 

  • Burke, R. J., & Mattis, M. C. (2007). Women and minorities in science, technology, engineering, and mathematics: Upping the numbers. Edward Elgar.

  • Cameron, E. Z., Gray, M. E., & White, A. M. (2013). Is publication rate an equal opportunity metric? Trends in Ecology & Evolution, 28(1), 7–8.

    Google Scholar 

  • Charlin, L., & Zemel, R. (2013). The Toronto paper matching system: an automated paper-reviewer assignment system. In International conference on machine learning (ICML) 2013 workshop on peer reviewing and publishing models.

  • Cho, A. H., Johnson, S. A., Schuman, C. E., Adler, J. M., Gonzalez, O., Graves, S. J., Huebner, J. R., Marchant, D. B., Rifai, S. W., Skinner, I., & Bruna, E. M. (2014). Women are underrepresented on the editorial boards of journals in environmental biology and natural resource management. PeerJ, 2, e542.

    Google Scholar 

  • Clark, J., & Horton, R. (2019). What is the lancet doing about gender and diversity? The Lancet, 393(10171), 508–510.

    Google Scholar 

  • Clarivate Analytics. (2018). 2018 global state of peer review. Retrieved from https://doi.org/10.14322/publons.GSPR2018. Accessed 13 March 2020.

  • Cole, J., & Zuckerman, H. (1984). The productivity puzzle: Persistence and change in patterns of publication among men and women scientists. In M. W. Steinkamp & M. Maehr (Eds.), Advances in Motivation and Achievement. JAI Press.

  • Débarre, F., Rode, N., & Ugelvig, L. (2018). Gender equity at scientific events. Evolution Letters, 2(3), 148–158.

    Google Scholar 

  • Demarest, B., Freeman, G., & Sugimoto, C. R. (2014). The reviewer in the mirror: Examining gendered and ethnicized notions of reciprocity in peer review. Scientometrics, 101(1), 717–735.

    Google Scholar 

  • Doyle, J. M., Quinn, K., Bodenstein, Y. A., Wu, C. O., Danthi, N., & Lauer, M. S. (2015). Association of percentile ranking with citation impact and productivity in a large cohort of de novo NIMH-funded R01 grants. Molecular Psychiatry, 20(9), 1030–1036.

    Google Scholar 

  • Ebenstein, A. (2014). Patrilocality and missing women. Working Paper, Jerusalem: Hebrew University.

  • Elsevier. (2017). Gender in the global research landscape. Amsterdam, Retrieved from https://www.elsevier.com/research-intelligence/resource-library/ty/gender-in-the-global-research-landscape. Accessed 13 March 2020.

  • Elsevier. (2020). The Researcher Journey Through a Gender Lens. Retrieved from: https://www.elsevier.com/research-intelligence/resource-library/gender-report-2020. Accessed 13 March 2020.

  • Feramisco, J. D., Leitenberger, J. J., Redfern, S. I., Bian, A., Xie, X.-J., & Resneck, J. S., Jr. (2009). A gender gap in the dermatology literature? Cross-sectional analysis of manuscript authorship trends in dermatology journals during 3 decades. Journal of the American Academy of Dermatology, 60(1), 63–69.

    Google Scholar 

  • Fox, C. W., Burns, C. S., & Meyer, J. A. (2016). Editor and reviewer gender influence the peer review process but not peer review outcomes at an ecology journal. Functional Ecology, 30(1), 140–153.

    Google Scholar 

  • Fox, C. W., Burns, C. S., Muncy, A. D., & Meyer, J. A. (2017). Author-suggested reviewers: Gender differences and influences on the peer review process at an ecology journal. Functional Ecology, 31(1), 270–280.

    Google Scholar 

  • Fox, C. W., Ritchey, J. P., & Paine, C. T. (2018). Patterns of authorship in ecology and evolution: First, last, and corresponding authorship vary with gender and geography. Ecology and Evolution, 8(23), 11492–11507.

    Google Scholar 

  • Fox, C. W., & Paine, C. T. (2019). Gender differences in peer review outcomes and manuscript impact at six journals of ecology and evolution. Ecology and Evolution, 9(6), 3599–3619.

    Google Scholar 

  • Fox, C. W., Duffy, M. A., Fairbairn, D. J., & Meyer, J. A. (2019). Gender diversity of editorial boards and gender differences in the peer review process at six journals of ecology and evolution. Ecology and Evolution, 9(24), 13636–13649.

    Google Scholar 

  • Ghiasi, G., Larivière, V., & Sugimoto, C. R. (2015). On the compliance of women engineers with a gendered scientific system. PloS ONE, 10(12), e0145931.

    Google Scholar 

  • Grod, O. N., Budden, A. E., Tregenza, T., Koricheva, J., Leimu, R., Aarssen, L. W., & Lortie, C. J. (2008). Systematic variation in reviewer practice according to country and gender in the field of ecology and evolution. PloS ONE, 3(9), e3202.

    Google Scholar 

  • Helmer, M., Schottdorf, M., Neef, A., & Battaglia, D. (2017). Gender bias in scholarly peer review. Elife, 6, e21718.

    Google Scholar 

  • Holman, L., Stuart-Fox, D., & Hauser, C. E. (2018). The gender gap in science: How long until women are equally represented? PLoS biology, 16(4), e2004956.

    Google Scholar 

  • Huang, J., Gates, A. J., Sinatra, R., & Barabási, A.-L. (2020). Historical comparison of gender inequality in scientific careers across countries and disciplines. Proceedings of the National Academy of Sciences, 117(9), 4609–4616.

    Google Scholar 

  • Jayachandran, S. (2015). The roots of gender inequality in developing countries. Economics, 7(1), 63–88.

    Google Scholar 

  • Jubb, M. (2016). Peer review: The current landscape and future trends. Learned Publishing, 29, 13–21.

    Google Scholar 

  • Käfer, J., Betancourt, A., Villain, A. S., Fernandez, M., Vignal, C., Marais, G., & Tenaillon, M. I. (2018). Progress and prospects in gender visibility at SMBE annual meetings. Genome Biology and Evolution, 10(3), 901–908.

    Google Scholar 

  • Kaltman, J. R., Evans, F. J., Danthi, N. S., Wu, C. O., DiMichele, D. M., & Lauer, M. S. (2014). Prior publication productivity, grand percentile rank-ing, and topic-normalized citation impact of NHLBI Cardiovascular R01 Grants. Circulation Research, 115, 617–624.

    Google Scholar 

  • Karimi, F., Wagner, C., Lemmerich, F., Jadidi, M., & Strohmaier, M. (2016). Inferring gender from names on the web: A comparative evaluation of gender detection methods. In Proceedings of the 25th International conference companion on World Wide Web (pp. 53–54).

  • Kliewer, M. A., Freed, K. S., DeLong, D. M., Pickhardt, P. J., & Provenzale, J. M. (2005). Reviewing the reviewers: Comparison of review quality and reviewer characteristics at the American journal of roentgenology. American Journal of Roentgenology, 184(6), 1731–1735.

    Google Scholar 

  • Knobloch-Westerwick, S., Glynn, C. J., & Huge, M. (2013). The Matilda effect in science communication: An experiment on gender bias in publication quality perceptions and collaboration interest. Science Communication, 35(5), 603–625.

    Google Scholar 

  • Knowles, R., Carroll, J., & Dredze, M. (2016). Demographer: Extremely simple name demographics. In Proceedings of the Workshop on NLP and Computational Social Science (pp.108–113).

  • Kovanis, M., Porcher, R., Ravaud, P., & Trinquart, L. (2016). The global burden of journal peer review in the biomedical literature: Strong imbalance in the collective enterprise. PloS ONE, 11(11), e0166387.

    Google Scholar 

  • Krawczyk, M., & Smyk, M. (2016). Author׳ s gender affects rating of academic articles: Evidence from an incentivized, deception-free laboratory experiment. European Economic Review, 90, 326–335.

    Google Scholar 

  • Langfeldt, L., Nedeva, M., Sörlin, S., & Thomas, D. A. (2020). Co-existing notions of research quality: A framework to study context-specific understandings of good research. Minerva, 58, 115–137.

    Google Scholar 

  • Larivière, V., Ni, C., Gingras, Y., Cronin, B., & Sugimoto, C. R. (2013). Bibliometrics: Global gender disparities in science. Nature News, 504(7479), 211–213.

    Google Scholar 

  • Lerback, J., & Hanson, B. (2017). Journals invite too few women to referee. Nature News, 541(7638), 455–457.

    Google Scholar 

  • Liu, W., & Ruths, D. (2013). What’s in a name? using first names as features for gender inference in twitter. In Analyzing Microtext: 2013 AAAI Spring Symposium.

  • Liu, X., Suel, T., & Memon, N. (2014). A robust model for paper reviewer assignment. In Proceedings of the 8th ACM conference on recommender systems (pp.25–32).

  • Liu, L., Wang, Y., Sinatra, R., Giles, C. L., Song, C., & Wang, D. (2018). Hot streaks in artistic, cultural, and scientific careers. Nature, 559(7714), 396–399.

    Google Scholar 

  • Mauleón, E., Hillán, L., Moreno, L., Gómez, I., & Bordons, M. (2013). Assessing gender balance among journal authors and editorial board members. Scientometrics, 95(1), 87–114.

    Google Scholar 

  • Mendoza-Denton, R., Patt, C., Fisher, A., Eppig, A., Young, I., Smith, A., & Richards, M. A. (2017). Differences in STEM doctoral publication by ethnicity, gender and academic field at a large public research university. PLoS ONE, 12(4), e0174296.

    Google Scholar 

  • Merton, R. K. (1968). The Matthew effect in science: The reward and communication systems of science are considered. Science, 159(3810), 56–63.

    Google Scholar 

  • Merton, R. K. (1973). The sociology of science: Theoretical and empirical investigations. University of Chicago press.

  • Mihaljević-Brandt, H., Santamaría, L., & Tullney, M. (2016). The effect of gender in the publication patterns in mathematics. PloS ONE, 11(10), e0165367.

    MATH  Google Scholar 

  • Milojević, S. (2012). How are academic age, productivity and collaboration related to citing behavior of researchers? PloS ONE, 7(11), e49176.

    Google Scholar 

  • Morton, M. J., & Sonnad, S. S. (2007). Women on professional society and journal editorial boards. Journal of the National Medical Association, 99(7), 764–771.

    Google Scholar 

  • Murray, D., Siler, K., Larivière, V., Chan, W. M., Collings, A. M., Raymond, J., & Sugimoto, C. R. (2019). Gender and international diversity improves equity in peer review. BioRxiv, 400515.

  • Narasimhan, S. D. (2019). A commitment to gender diversity in peer review. Cell, 179(1), 1–2.

    Google Scholar 

  • O’Connor, E. E., Chen, P., Weston, B., Anderson, R., Zeffiro, T., Ahmed, A., & Zeffiro, T. A. (2018). Gender trends in academic radiology publication in the United States revisited. Academic Radiology, 25(8), 1062–1069.

    Google Scholar 

  • Ortega, J. L. (2017). Are peer-review activities related to reviewer bibliometric performance? A Scientometric Analysis of Publons. Scientometrics, 112(2), 947–962.

    Google Scholar 

  • Publons. (2018). Report for Publons State of Peer Review 2018 Survey. Retrieved from https://publons.com/community/gspr#open-elq-form-slider-DLGSPR. Accessed 13 March 2020.

  • Reingewertz, Y., & Lutmar, C. (2018). Academic in-group bias: An empirical examination of the link between author and journal affiliation. Journal of Informetrics, 12(1), 74–86.

    Google Scholar 

  • Rørstad, K., & Aksnes, D. W. (2015). Publication rate expressed by age, gender and academic position–A large-scale analysis of Norwegian academic staff. Journal of Informetrics, 9(2), 317–333.

    Google Scholar 

  • Ross, E. (2017). Gender bias distorts peer review across fields. Nature News. Retrieved from. https://doi.org/10.1038/nature.2017.21685.Accessed13March

    Article  Google Scholar 

  • Santamaría, L., & Mihaljević, H. (2018). Comparison and benchmark of name-to-gender inference services. PeerJ Computer Science, 4, e156.

    Google Scholar 

  • Schmaling, K. B., & Blume, A. W. (2017). Gender differences in providing peer review to two behavioural science journals, 2006–2015. Learned Publishing, 30(3), 221–225.

    Google Scholar 

  • Shamsi, A. (2020). Gender of Highly Cited Researchers focused on the cross-filed category. Gaceta Sanitaria.

  • Sivertsen, G. (2018). The Norwegian Model in Norway. Journal of Data and Information Science, 3(4), 3–19.

    Google Scholar 

  • Solans, M., Adam, P., Vizcaino, E., Radó, N., Güell, O. P., E, et al. (2020). Women’s advancement and leadership in the biomedical research centres of Catalonia II: Bringing about change. Agència de Qualitat i Avaluació Sanitàries de Catalunya.

  • Sotudeh, H., Dehdarirad, T., & Freer, J. (2018). Gender differences in scientific productivity and visibility in core neurosurgery journals: Citations and social media metrics. Research Evaluation, 27(3), 262–269.

    Google Scholar 

  • Steinberg, J. J., Skae, C., & Sampson, B. (2018). Gender gap, disparity, and inequality in peer review. The Lancet, 391(10140), 2602–2603.

    Google Scholar 

  • Thurner, S., & Hanel, R. (2011). Peer-review in a world with rational scientists: Toward selection of the average. The European Physical Journal B, 84(4), 707–711.

    Google Scholar 

  • Tite, L., & Schroter, S. (2007). Why do peer reviewers decline to review? A survey. Journal of Epidemiology & Community Health, 61(1), 9–12.

    Google Scholar 

  • United Nations Educational, Scientific and Cultural Organization. (2015). UNESCO science report, towards 2030: executive summary. Retrieved from https://en.unesco.org/unescosciencereport. Accessed 23 March 2021.

  • United Nations Development Program. (2019). Human Development Report 2019. Retrieved from http://hdr.undp.org/sites/default/files/hdr2019.pdf. Accessed 13 March 2020.

  • van Arensbergen, P., van der Weijden, I., & van den Besselaar, P. (2012). Gender differences in scientific productivity, a persisting phenomenon? Scientometrics, 93(3), 857–868.

    Google Scholar 

  • van Den Besselaar, P., & Sandström, U. (2016). Gender differences in research performance and its impact on careers: A longitudinal case study. Scientometrics, 106(1), 143–162.

    Google Scholar 

  • van Den Besselaar, P., & Sandström, U. (2017). Vicious circles of gender bias, lower positions, and lower performance: Gender differences in scholarly productivity and impact. PloS ONE, 12(8), e0183301.

    Google Scholar 

  • Walker, K. A. (2020). Females are first authors, sole authors, and reviewers of entomology publications significantly less often than males. Annals of the Entomological Society of America, 113(3), 193–201.

    Google Scholar 

  • Weisshaar, K. (2017). Publish and perish? An assessment of gender gaps in promotion to tenure in academia. Social Forces, 96(2), 529–560.

    Google Scholar 

  • West, J. D., Jacquet, J., King, M. M., Correll, S. J., & Bergstrom, C. T. (2013). The role of gender in scholarly authorship. PloS ONE, 8(7), e66212.

    Google Scholar 

  • Wickware, P. (1997). Along the leaky pipeline. Nature, 390(6656), 202–203.

    Google Scholar 

  • Willis, M. (2016). Why do peer reviewers decline to review manuscripts? A study of reviewer invitation responses. Learned Publishing, 29, 5–7.

    Google Scholar 

  • Wing, D. A., Benner, R. S., Petersen, R., Newcomb, R., & Scott, J. R. (2010). Differences in editorial board reviewer behavior based on gender. Journal of Women’s Health, 19(10), 1919–1923.

    Google Scholar 

  • World Economic Forum. (2020). The global gender gap report 2020. Retrieved from http://www3.weforum.org/docs/WEF_GGGR_2020.pdf. Accessed 13 March 2020.

  • Zeina, M., Balston, A., Banerjee, A., & Woolf, K. (2020). Gender and ethnic differences in publication of BMJ letters to the editor: an observational study using machine learning. BMJ open, 10(12), e037269.

    Google Scholar 

  • Zeng, X. H. T., Duch, J., Sales-Pardo, M., Moreira, J. A., Radicchi, F., Ribeiro, H. V., Woodruff, T. K., & Amaral, L. A. N. (2016). Differences in collaboration patterns across discipline, career stage, and gender. PLoS biology, 14(11), e1002573.

    Google Scholar 

Download references

Acknowledgements

This work was supported by the National Natural Science Foundation of China (Grant Nos. 71974150; 71573085; 72004169), the Major Project of National Social Science Foundation of China (Grant No. 19ZDA349), the National Laboratory Center for Library and Information Science in Wuhan University, and the Research Council of Norway (Grant No. 256223).

Author information

Authors and Affiliations

  1. Center for Science, Technology & Education Assessment (CSTEA), Wuhan University, Wuhan, China

    Lin Zhang, Yuanyuan Shang & Ying Huang

  2. Center for Studies of Information Resources, School of Information Management, Wuhan University, Wuhan, China

    Lin Zhang, Yuanyuan Shang & Ying Huang

  3. Centre for R&D Monitoring (ECOOM) and Department of MSI, KU Leuven, Leuven, Belgium

    Lin Zhang & Ying Huang

  4. Nordic Institute for Studies in Innovation, Research and Education, Oslo, Norway

    Gunnar Sivertsen

Authors
  1. Lin Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Yuanyuan Shang
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Ying Huang
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Gunnar Sivertsen
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Lin Zhang.

Ethics declarations

Conflict of interest

The first author (Lin Zhang) is an associate-editor of Scientometrics.

Appendix

Appendix

See Tables 5, 6 and 7.

Table 5 Percent of women among authors of 15 countries and the EU28 overall in each subject area
Full size table
Table 6 Spearman’s correlation coefficients between reviewing activity and research activity by research field
Full size table
Table 7 Spearman’s correlation coefficients between reviewing activity and research activity by research field and gender
Full size table

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Zhang, L., Shang, Y., Huang, Y. et al. Gender differences among active reviewers: an investigation based on publons. Scientometrics 127, 145–179 (2022). https://doi.org/10.1007/s11192-021-04209-1

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11192-021-04209-1

Keywords

Search

Navigation