Skip to main content
SpringerLink
Log in

What does open peer review bring to scientific articles? Evidence from PLoS journals

  • Published:
Scientometrics Aims and scope Submit manuscript

Abstract

This study examined the impact of open peer review (OPR) on the usage and citations of scientific articles using a dataset of 6441 articles published in six Public Library of Science (PLoS) journals in 2020–2021. We compared OPR articles with their non-OPR counterparts in the same journal to determine whether OPR increased the visibility and citations of the articles. Our results demonstrated a positive association between OPR and higher article page views, saving, sharing, and a greater HTML to PDF conversion rate. However, we also found that OPR articles had a lower PDF to citations conversion rate compared to non-OPR articles. Furthermore, we investigated the effects of OPR on citations across various citation databases, including Web of Science, Scopus, Google Scholar, Semantic Scholar, and Dimensions. Our analysis indicated that OPR had a heterogeneous impact on citations across these databases. These findings provide compelling evidence for stakeholders, such as policymakers, publishers, and researchers, to participate in OPR and promote its adoption in scientific publishing. Additionally, our study underscores the importance of carefully selecting bibliographic databases when assessing the effect of OPR on article citations.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1

Similar content being viewed by others

Notes

  1. https://publons.com.

  2. https://theplosblog.plos.org/2019/05/plos-journals-now-open-for-published-peer-review/.

References

  • Adie, E. (2014). Attention! A study of open access vs non-open access articles. Figshare. https://doi.org/10.6084/m9.figshare.1213690

  • Baas, J., Schotten, M., Plume, A., Côté, G., & Karimi, R. (2020). Scopus as a curated, high-quality bibliometric data source for academic research in quantitative science studies. Quantitative Science Studies, 1(1), 377–386.

    Article  Google Scholar 

  • Beck, S., Mahdad, M., Beukel, K., & Poetz, M. (2019). The value of scientific knowledge dissemination for scientists—A value capture perspective. Publications, 7(3), 54.

    Article  Google Scholar 

  • Besançon, L., Peiffer-Smadja, N., Segalas, C., Jiang, H., Masuzzo, P., Smout, C., Billy, E., Deforet, M., & Leyrat, C. (2021). Open science saves lives: Lessons from the COVID-19 pandemic. BMC Medical Research Methodology, 21(1), 1–18.

    Article  Google Scholar 

  • Besançon, L., Rönnberg, N., Löwgren, J., Tennant, J. P., & Cooper, M. (2020). Open up: A survey on open and non-anonymized peer reviewing. Research Integrity and Peer Review, 5(1), 1–11.

    Article  Google Scholar 

  • Birkle, C., Pendlebury, D. A., Schnell, J., & Adams, J. (2020). Web of Science as a data source for research on scientific and scholarly activity. Quantitative Science Studies, 1(1), 363–376.

    Article  Google Scholar 

  • Bornmann, L., & Daniel, H. D. (2008). What do citation counts measure? A review of studies on citing behavior. Journal of Documentation, 64(1), 45–80.

    Article  Google Scholar 

  • Bornmann, L., Marx, W., Schier, H., Thor, A., & Daniel, H. D. (2010). From black box to white box at open access journals: Predictive validity of manuscript reviewing and editorial decisions at atmospheric chemistry and physics. Research Evaluation, 19(2), 105–118.

    Article  Google Scholar 

  • Bornmann, L., Schier, H., Marx, W., & Daniel, H. D. (2012). What factors determine citation counts of publications in chemistry besides their quality? Journal of Informetrics, 6(1), 11–18.

    Article  Google Scholar 

  • Bornmann, L., & Williams, R. (2013). How to calculate the practical significance of citation impact differences? An empirical example from evaluative institutional bibliometrics using adjusted predictions and marginal effects. Journal of Informetrics, 7(2), 562–574.

    Article  Google Scholar 

  • Borsuk, R. M., Budden, A. E., Leimu, R., Aarssen, L. W., & Lortie, C. J. (2009). The influence of author gender, national language and number of authors on citation rate in ecology. The Open Ecology Journal, 2(1), 25–28.

    Article  Google Scholar 

  • Bravo, G., Grimaldo, F., López-Iñesta, E., Mehmani, B., & Squazzoni, F. (2019). The effect of publishing peer review reports on referee behavior in five scholarly journals. Nature Communications, 10(1), 1–8.

    Article  Google Scholar 

  • Corbyn, Z. (2010). An easy way to boost a paper’s citations. Nature. https://doi.org/10.1038/news.2010.406

    Article  Google Scholar 

  • Crystal, M. (2019). PLoS journals now open for published peer review. Retrieved May 23, 2022, from https://theplosblog.plos.org/2019/05/plos-journals-now-open-for-published-peer-review/.

  • Davis, P. M., Lewenstein, B. V., Simon, D. H., Booth, J. G., & Connolly, M. J. (2008). Open access publishing, article downloads, and citations: Randomised controlled trial. British Medical Journal. Retrieved May 23, 2022, from https://www.bmj.com/content/337/bmj.a568.short.

  • Delgado López-Cózar, E., Orduna-Malea, E., & Martín-Martín, A. (2019). Google Scholar as a data source for research assessment. In W. Glänzel, H. F. Moed, U. Schmoch, & M. Thelwall (Eds.), Springer Handbook of Science and Technology Indicators (pp. 95–127). Springer.

    Chapter  Google Scholar 

  • Delikoura, E., & Kouis, D. (2021). Open research data and open peer review: Perceptions of a medical and health sciences community in Greece. Publications, 9(2), 14.

    Article  Google Scholar 

  • Ford, E. (2015). Open peer review at four STEM journals: An observational overview. F1000Research. Retrieved May 23, 2022, from https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4350441/

  • Gorraiz, J., Gumpenberger, C., & Schloegl, C. (2014). Usage versus citation behaviours in four subject areas. Scientometrics, 101(2), 1077–1095.

    Article  Google Scholar 

  • Guo, F., Ma, C., Shi, Q., & Zong, Q. (2018). Succinct effect or informative effect: The relationship between title length and the number of citations. Scientometrics, 116(3), 1531–1539.

    Article  Google Scholar 

  • Heesen, R., & Bright, L. K. (2021). Is peer review a good idea? The British Journal for the Philosophy of Science, 13, 82–102.

    Google Scholar 

  • Herzog, C., Hook, D., & Konkiel, S. (2020). Dimensions: Bringing down barriers between scientometricians and data. Quantitative Science Studies, 1(1), 387–395.

    Article  Google Scholar 

  • Ibanez, A., Bielza, C., & Larranaga, P. (2013). Relationship among research collaboration, number of documents and number of citations: A case study in Spanish computer science production in 2000–2009. Scientometrics, 95(2), 689–716.

    Article  Google Scholar 

  • Jabbour, C. J. C., Jabbour, A. B. L. D. S., & de Oliveira, J. H. C. (2013). The perception of brazilian researchers concerning the factors that influence the citation of their articles: A study in the field of sustainability. Serials Review, 39(2), 93–96.

    Article  Google Scholar 

  • Kowalczuk, M. K., Dudbridge, F., Nanda, S., Harriman, S. L., Patel, J., & Moylan, E. C. (2015). Retrospective analysis of the quality of reports by author-suggested and non-author-suggested reviewers in journals operating on open or single-blind peer review models. British Medical Journal Open, 5(9), e008707.

    Google Scholar 

  • Larivière, V., Gingras, Y., Sugimoto, C. R., & Tsou, A. (2015). Team size matters: Collaboration and scientific impact since 1900. Journal of the Association for Information Science and Technology, 66(7), 1323–1332.

    Article  Google Scholar 

  • Lee, C. C., & Yang, J. (2000). Knowledge value chain. Journal of Management Development, 19(9), 783–794.

    Google Scholar 

  • Letchford, A., Moat, H. S., & Preis, T. (2015). The advantage of short paper titles. Royal Society Open Science, 2(8), 150266.

    Article  Google Scholar 

  • Lira, R. P. C., Amorim, F. H. R., Gonçalves, F. A., Arieta, C. E. L., & Kara-Junior, N. (2014). Comparison of the citation characteristics between the case reports/cases series versus the other study designs in the articles published in Brazilian journals of ophthalmology. Revista Brasileira De Oftalmologia, 73, 07–10.

    Article  Google Scholar 

  • Ma, C., Li, Y., Guo, F., & Si, K. (2019). The citation trap: Papers published at year-end receive systematically fewer citations. Journal of Economic Behavior & Organization, 166, 667–687.

    Article  Google Scholar 

  • Mammola, S., Fontaneto, D., Martínez, A., & Chichorro, F. (2021). Impact of the reference list features on the number of citations. Scientometrics, 126(1), 785–799.

    Article  Google Scholar 

  • Martín-Martín, A., Orduna-Malea, E., Thelwall, M., & López-Cózar, E. D. (2018). Google Scholar, Web of Science, and Scopus: A systematic comparison of citations in 252 subject categories. Journal of Informetrics, 12(4), 1160–1177.

    Article  Google Scholar 

  • Ni, J., Zhao, Z., Shao, Y., Liu, S., Li, W., Zhuang, Y., Qu, J., Cao, Y., Lian, N., & Li, J. (2021). The influence of opening up peer review on the citations of journal articles. Scientometrics, 126(12), 9393–9404.

    Article  Google Scholar 

  • Ni, P., & An, X. (2018). Relationship between international collaboration papers and their citations from an economic perspective. Scientometrics, 116(2), 863–877.

    Article  Google Scholar 

  • Pucker, B., Schilbert, H. M., & Schumacher, S. F. (2019). Integrating molecular biology and bioinformatics education. Journal of Integrative Bioinformatics. https://doi.org/10.1515/jib-2019-0005

    Article  Google Scholar 

  • Rath M., & Wang P. (2017). Open peer review in the era of open science: A pilot study of researchers’ perceptions. In Proceedings of the Joint Conference on Digital Libraries (JCDL) (pp. 317–318).

  • Ross-Hellauer, T. (2017). Open peer review: Bringing transparency, accountability, and inclusivity to the peer review process. Impact of Social Sciences Blog. Retrieved April 23, 2022, from http://eprints.lse.ac.uk/85029/1/impactofsocialsciences-2017-09-13-open-peer-review-bringing.pdf.

  • Ross-Hellauer, T., Deppe, A., & Schmidt, B. (2017). Survey on open peer review: Attitudes and experience amongst editors, authors and reviewers. PLoS ONE, 12(12), e0189311.

    Article  Google Scholar 

  • Schloegl, C., & Gorraiz, J. (2009). Global usage vs global citation metrics using Science Direct pharmacology journals. In Proceedings of the International Conference on Scientometrics and Informetrics (Vol. 1, pp. 455–459).

  • Schmidt, B., Ross-Hellauer, T., van Edig, X., & Moylan, E. C. (2018). Ten considerations for open peer review. F1000Research, 7, 969. Retrieved April 23, 2022, from https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6073088/.

  • Shanahan, D. R., & Olsen, B. R. (2014). Opening peer-review: The democracy of science. Journal of Negative Results in Biomedicine, 13(1), 1–2.

    Article  Google Scholar 

  • Singh, V. K., Singh, P., Karmakar, M., Leta, J., & Mayr, P. (2021). The journal coverage of Web of Science, Scopus and Dimensions: A comparative analysis. Scientometrics, 126(6), 5113–5142.

    Article  Google Scholar 

  • Snell, L., & Spencer, J. (2005). Reviewers’ perceptions of the peer review process for a medical education journal. Medical Education, 39(1), 90–97.

    Article  Google Scholar 

  • Stewart, J. A. (1983). Achievement and ascriptive processes in the recognition of scientific articles. Social Forces, 62(1), 166–189.

    Article  Google Scholar 

  • Tahamtan, I., Safipour Afshar, A., & Ahamdzadeh, K. (2016). Factors affecting number of citations: A comprehensive review of the literature. Scientometrics, 107(3), 1195–1225.

    Article  Google Scholar 

  • Tattersall, A. (2015). For what it’s worth–the open peer review landscape. Online Information Review, 39(5), 649–663.

    Article  Google Scholar 

  • Van Dalen, H., & Henkens, K. (2001). What makes a scientific article influential? The case of demographers. Scientometrics, 50(3), 455–482.

    Article  Google Scholar 

  • Vaughan, L., Tang, J., & Yang, R. (2017). Investigating disciplinary differences in the relationships between citations and downloads. Scientometrics, 111(3), 1533–1545.

    Article  Google Scholar 

  • Vrana, R. (2017). Acceptance of open peer review by scientific publishers. In Central European Conference on Information and Intelligent Systems (pp. 221–228). Faculty of Organization and Informatics Varazdin.

  • Walsh, E., Rooney, M., Appleby, L., & Wilkinson, G. (2000). Open peer review: A randomised controlled trial. The British Journal of Psychiatry, 176(1), 47–51.

    Article  Google Scholar 

  • Wan, J. K., Hua, P. H., Rousseau, R., & Sun, X. K. (2010). The journal download immediacy index (DII): Experiences using a Chinese full-text database. Scientometrics, 82(3), 555–566.

    Article  Google Scholar 

  • Wang, X., Liu, C., Mao, W., & Fang, Z. (2015). The open access advantage considering citation, article usage and social media attention. Scientometrics, 103(2), 555–564.

    Article  Google Scholar 

  • Wang, X., Mao, W., Xu, S., & Zhang, C. (2014). Usage history of scientific literature: Nature metrics and metrics of Nature publications. Scientometrics, 98(3), 1923–1933.

    Article  Google Scholar 

  • Warne, V. (2016). Rewarding reviewers–sense or sensibility? A Wiley study explained. Learned Publishing, 29(1), 41–50.

    Article  Google Scholar 

  • Wolfram, D., Wang, P., Hembree, A., & Park, H. (2020). Open peer review: Promoting transparency in open science. Scientometrics, 125(2), 1033–1051.

    Article  Google Scholar 

  • Wolfram, D., Wang, P., & Park, H. (2019). Open peer review: The current landscape and emerging models. In Proceedings of the 17th International Conference on Scientometrics & Informetrics. International Society for Scientometrics and Informetrics

  • Yang, J., Lai, F., & Yu, L. (2006). Harnessing value in knowledge acquisition and dissemination: Strategic sourcing in product development. International Journal of Technology Management, 33(2–3), 299–317.

    Article  Google Scholar 

  • Zong, Q., Xie, Y., & Liang, J. (2020). Does open peer review improve citation count? Evidence from a propensity score matching analysis of PeerJ. Scientometrics, 125(1), 607–623.

    Article  Google Scholar 

Download references

Acknowledgements

The authors appreciate the constructive comments from Dr Xiao Hu, Dr Lianghao Dai and anonymous reviewers.

Author information

Authors and Affiliations

  1. School of Information Management, Nanjing University, Nanjing, 210023, China

    Chunli Wei, Jingyi Zhao, Jue Ni & Jiang Li

Authors
  1. Chunli Wei
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Jingyi Zhao
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Jue Ni
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Jiang Li
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Jiang Li.

Ethics declarations

Conflict of interest

The corresponding author (Jiang Li) is a member of the Distinguished Reviewers Board of Scientometrics.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Wei, C., Zhao, J., Ni, J. et al. What does open peer review bring to scientific articles? Evidence from PLoS journals. Scientometrics 128, 2763–2776 (2023). https://doi.org/10.1007/s11192-023-04683-9

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11192-023-04683-9

Keywords

Search

Navigation