Abstract
As the gatekeeper of science, reviewers play an essential role in academic publishing by improving the quality of papers and maintaining research integrity. In this research, we examined whether reviewers receive gratitude from authors for their reviewed manuscript. We analyzed over 46 million papers from 2001 to 2020 and found that although the proportion of acknowledges to reviewers in different subject areas has increased in recent years, the overall proportion of acknowledges to reviewers is low, with the lowest proportion being the field of Health Sciences and the highest in the field of Social Sciences. We also discovered that the proportion of acknowledged reviewers from different regions varied, with America, Europe and Oceania having a generally higher proportion than Asia. Our results indicate that the scientific community is generally unaware of the contributions made by reviewers, and reviewers have not received the acknowledgments they deserve. Additionally, we found that a small number of journals had more papers acknowledging reviewers. Through regression analysis, we found that although authors acknowledged the reviewer's contribution, their decision to write down acknowledgements were influenced by their language proficiency.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
An, J., Jeon, S., Jones, T., & Song, M. (2017). Data-driven pattern analysis of acknowledgments in the biomedical domain. Data and Information Management, 1(1), 35–48. https://doi.org/10.1515/dim-2017-0002
Balietti, S., Goldstone, R. L., & Helbing, D. (2016). Peer review and competition in the Art Exhibition Game. Proceedings of the National Academy of Sciences of the United States of America, 113(30), 8414–8419. https://doi.org/10.1073/pnas.1603723113
Bianchi, F., Grimaldo, F., & Squazzoni, F. (2019). The F-3-index. Valuing reviewers for scholarly journals. Journal of Informetrics, 13(1), 78–86. https://doi.org/10.1016/j.joi.2018.11.007
Bornmann, L. (2011). Scientific peer review. Annual Review of Information Science and Technology, 45(1), 197–245. https://doi.org/10.1002/aris.2011.1440450112
Bornmann, L., Mutz, R., & Daniel, H. D. (2010). A reliability-generalization study of journal peer reviews: A multilevel meta-analysis of inter-rater reliability and its determinants. Plos One, 5(12), e14331. https://doi.org/10.1371/journal.pone.0014331
Bornmann, L., Nast, I., & Daniel, H.-D. (2008). Do editors and referees look for signs of scientific misconduct when reviewing manuscripts? A quantitative content analysis of studies that examined review criteria and reasons for accepting and rejecting manuscripts for publication. Scientometrics, 77(3), 415–432. https://doi.org/10.1007/s11192-007-1950-2
Bravo, G., Grimaldo, F., Lopez-Inesta, E., Mehmani, B., & Squazzoni, F. (2019). The effect of publishing peer review reports on referee behavior in five scholarly journals. Nature Communications, 10, 322. https://doi.org/10.1038/s41467-018-08250-2
Brezis, E. S., & Birukou, A. (2020). Arbitrariness in the peer review process. Scientometrics, 123(1), 393–411. https://doi.org/10.1007/s11192-020-03348-1
Buljan, I., Garcia-Costa, D., Grimaldo, F., Squazzoni, F., & Marušić, A. (2020). Large-scale language analysis of peer review reports. eLife, 9, e53249. https://doi.org/10.7554/eLife.53249
Burns, C. S., & Fox, C. W. (2017). Language and socioeconomics predict geographic variation in peer review outcomes at an ecology journal. Scientometrics, 113(2), 1113–1127. https://doi.org/10.1007/s11192-017-2517-5
Casnici, N., Grimaldo, F., Gilbert, N., & Squazzoni, F. (2017). Attitudes of referees in a multidisciplinary journal: An empirical analysis. Journal of the Association for Information Science and Technology, 68(7), 1763–1771. https://doi.org/10.1002/asi.23665
Chen, M. K. (2013). The effect of language on economic behavior: Evidence from savings rates, health behaviors, and retirement assets [Article]. American Economic Review, 103(2), 690–731. https://doi.org/10.1257/aer.103.2.690
Cicchetti, D. V. (1991). The reliability of peer-review for manuscript and grant submissions - A cross-disciplinary investigation. Behavioral and Brain Sciences, 14(1), 119–134. https://doi.org/10.1017/s0140525x00065675
Costas, R., van Leeuwen, T. N., & van Raan, A. F. J. (2013). Effects of the durability of scientific literature at the group level: Case study of chemistry research groups in the Netherlands. Research Policy, 42(4), 886–894. https://doi.org/10.1016/j.respol.2012.11.006
Crawford, E. T., & Biderman, A. D. (1970). Paper money - Trends of research sponsorship in American sociology journals. Social Science Information, 9(1), 51–77.
Cronin, B. (1991). Let the credits roll - A preliminary examination of the role played by mentors and trusted assessors in disciplinary formation. Journal of Documentation, 47(3), 227–239. https://doi.org/10.1108/eb026878
Demarest, B., Freeman, G., & Sugimoto, C. R. (2014). The reviewer in the mirror: Examining gendered and ethnicized notions of reciprocity in peer review. Scientometrics, 101(1), 717–735. https://doi.org/10.1007/s11192-014-1354-z
Edge, D. (1979). Quantitative measures of communication in science: A critical review. History of Science, 17(2), 102–134. https://doi.org/10.1177/007327537901700202
Fox, C. W., Burns, C. S., Muncy, A. D., & Meyer, J. A. (2017). Author-suggested reviewers: Gender differences and influences on the peer review process at an ecology journal. Functional Ecology, 31(1), 270–280. https://doi.org/10.1111/1365-2435.12665
Fox, C. W., & Paine, C. E. T. (2019). Gender differences in peer review outcomes and manuscript impact at six journals of ecology and evolution. Ecology and Evolution, 9(6), 3599–3619. https://doi.org/10.1002/ece3.4993
Garcia-Costa, D., Squazzoni, F., Mehmani, B., & Grimaldo, F. (2022). Measuring the developmental function of peer review: a multi-dimensional, cross-disciplinary analysis of peer review reports from 740 academic journals. Peerj, 10, e313539. https://doi.org/10.7717/peerj.13539
Goodman, S. N., Berlin, J., Fletcher, S. W., & Fletcher, R. H. (1994). Manuscript quality before and after peer-review and editing at annals of internal-medicine [Article]. Annals of Internal Medicine, 121(1), 11–21. https://doi.org/10.7326/0003-4819-121-1-199407010-00003
Huisman, J., & Smits, J. (2017). Duration and quality of the peer review process: The author’s perspective. Scientometrics, 113(1), 633–650. https://doi.org/10.1007/s11192-017-2310-5
Hyland, K. (2003). Dissertation acknowledgments - The anatomy of a Cinderella genre. Written Communication, 20(3), 242–268. https://doi.org/10.1177/0741088303257276
Jefferson, T., Wager, E., & Davidoff, F. (2002). Measuring the quality of editorial peer review. Jama-Journal of the American Medical Association, 287(21), 2786–2790. https://doi.org/10.1001/jama.287.21.2786
Keysar, B., Hayakawa, S. L., & An, S. G. (2012). The foreign-language effect: Thinking in a foreign tongue reduces decision biases [Article]. Psychological Science, 23(6), 661–668. https://doi.org/10.1177/0956797611432178
Laband, D. N. (1990). Is there value-added from the review process in economics - Preliminary evidence from authors. Quarterly Journal of Economics, 105(2), 341–352. https://doi.org/10.2307/2937790
Matsui, A., Chen, E., Wang, Y., & Ferrara, E. (2021). The impact of peer review on the contribution potential of scientific papers. Peerj, 9, e11999. https://doi.org/10.7717/peerj.11999
McClellan, J. E. (2003). Specialist control: the publications committee of the Académie Royale des sciences (Paris) 1700–1793. Transactions of the American Philosophical Society, 93(3), i–134. https://doi.org/10.2307/20020343
Mejia, C., & Kajikawa, Y. (2018). Using acknowledgement data to characterize funding organizations by the types of research sponsored: The case of robotics research. Scientometrics, 114(3), 883–904. https://doi.org/10.1007/s11192-017-2617-2
Mulligan, A., Hall, L., & Raphael, E. (2013). Peer review in a changing world: An international study measuring the attitudes of researchers [Article]. Journal of the American Society for Information Science and Technology, 64(1), 132–161. https://doi.org/10.1002/asi.22798
Murray, D., Siler, K., Larivière, V., Chan, W. M., Collings, A. M., Raymond, J., & Sugimoto, C. R. (2018). Gender and international diversity improves equity in peer review. bioRxiv. https://doi.org/10.1101/400515
Paul-Hus, A., Desrochers, N., & Costas, R. (2016). Characterization, description, and considerations for the use of funding acknowledgement data in web of science. Scientometrics, 108(1), 167–182. https://doi.org/10.1007/s11192-016-1953-y
Paul-Hus, A., Diaz-Faes, A. A., Sainte-Marie, M., Desrochers, N., Costas, R., & Lariviere, V. (2017). Beyond funding: Acknowledgement patterns in biomedical, natural and social sciences. Plos One, 12(10), e0185578. https://doi.org/10.1371/journal.pone.0185578
Paul-Hus, A., Mongeon, P., Sainte-Marie, M., & Lariviere, V. (2017b). The sum of it all: Revealing collaboration patterns by combining authorship and acknowledgements. Journal of Informetrics, 11(1), 80–87. https://doi.org/10.1016/j.joi.2016.11.005
Peters, D. P., & Ceci, S. J. (1982). Peer-review practices of psychological journals - The fate of accepted, published articles, submitted again. Behavioral and Brain Sciences, 5(2), 187–195. https://doi.org/10.1017/s0140525x00011183
Pierie, J., Walvoort, H. C., & Overbeke, A. (1996). Readers’ evaluation of effect of peer review and editing on quality of articles in the Nederlands Tijdschrift voor Geneeskunde [Article]. Lancet, 348(9040), 1480–1483. https://doi.org/10.1016/s0140-6736(96)05016-7
Rattan, G. K. (2014). Acknowledgement patterns in DESIDOC journal of Library & information technology. DESIDOC Journal of Library & Information Technology, 34(3), 265–270. https://doi.org/10.14429/djlit.34.5952
Rose, M. E., & Georg, C.-P. (2021). What 5000 acknowledgements tell us about informal collaboration in financial economics [Article]. Research Policy, 50(6), 104236. https://doi.org/10.1016/j.respol.2021.104236
Schmaling, K. B., & Blume, A. W. (2017). Gender differences in providing peer review to two behavioural science journals, 2006–2015. Learned Publishing, 30(3), 221–225. https://doi.org/10.1002/leap.1104
Seeber, M., & Bacchelli, A. (2017). Does single blind peer review hinder newcomers? Scientometrics, 113(1), 567–585. https://doi.org/10.1007/s11192-017-2264-7
Shin, H. I., & Kim, J. (2017). Foreign language effect and psychological distance. Journal of Psycholinguistic Research, 46(6), 1339–1352. https://doi.org/10.1007/s10936-017-9498-7
Shopovski, J., Bolek, C., & Bolek, M. (2020). Characteristics of peer Review Reports: Editor-suggested versus author-suggested reviewers. Science and Engineering Ethics, 26(2), 709–726. https://doi.org/10.1007/s11948-019-00118-y
Smirnova, N., & Mayr, P. (2023). A comprehensive analysis of acknowledgement texts in Web of Science: A case study on four scientific domains. Scientometrics, 128(1), 709–734. https://doi.org/10.1007/s11192-022-04554-9
Song, M., Kang, K. Y., Timakum, T., & Zhang, X. (2020). Examining influential factors for acknowledgements classification using supervised learning. Plos One, 15(2), e0228928. https://doi.org/10.1371/journal.pone.0228928
Squazzoni, F., Bravo, G., Farjam, M., Marusic, A., Mehmani, B., Willis, M., Birukou, A., Dondio, P., & Grimaldo, F. (2021). Peer review and gender bias: A study on 145 scholarly journals [Article]. Science Advances, 7(2), eabd0299. https://doi.org/10.1126/sciadv.abd0299
Stephan, P., Veugelers, R., & Wang, J. (2017). Reviewers are blinkered by bibliometrics. Nature, 544(7651), 411–412. https://doi.org/10.1038/544411a
Tiew, W. S., & Sen, B. K. (2002). Acknowledgement patterns in research articles: A bibliometric study based on journal of natural rubber research 1986–1997. Malaysian Journal of Library & Information Science, 7(1), 43–56.
Tomkins, A., Zhang, M., & Heavlin, W. D. (2017). Reviewer bias in single-versus double-blind peer review. Proceedings of the National Academy of Sciences of the United States of America, 114(48), 12708–12713. https://doi.org/10.1073/pnas.1707323114
Wang, X., Liu, D., Ding, K., & Wang, X. (2012). Science funding and research output: A study on 10 countries. Scientometrics, 91(2), 591–599. https://doi.org/10.1007/s11192-011-0576-6
Wei, Y. Y., & Lei, L. (2018). Institution bias in the New England journal of medicine? A bibliometric analysis of publications (1997–2016). Scientometrics, 117(3), 1771–1775. https://doi.org/10.1007/s11192-018-2948-7
Wing, D. A., Benner, R. S., Petersen, R., Newcomb, R., & Scott, J. R. (2010). Differences in editorial board reviewer behavior based on gender. Journal of Womens Health, 19(10), 1919–1923. https://doi.org/10.1089/jwh.2009.1904
Xu, S., Zhang, G., Sun, Y., & Wang, X. (2019). Understanding the peer review endeavor. Proceedings of the Association for Information Science and Technology, 56(1), 316–325. https://doi.org/10.1002/pra2.26
Acknowledgment
The authors gratefully acknowledge the grant from the National Natural Science Foundation of China (Grant No. 71974029). We thank the anonymous reviewers for their comments and suggestions with this paper.
Funding
Funding was supported by National Natural Science Foundation of China, (Grant No. 71974029).
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
About this article
Cite this article
Jia, P., Xie, W., Zhang, G. et al. Do reviewers get their deserved acknowledgments from the authors of manuscripts?. Scientometrics 128, 5687–5703 (2023). https://doi.org/10.1007/s11192-023-04790-7
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11192-023-04790-7