Abstract
This study tests for evidence of gaming and attention mongering—here termed ego bias—in the scholarly peer review process. We explore the extent to which authors cite the target journal and its editor and also the relationship between targeted references and editorial decisions. We examine referee reports for the presence and type of references and determine the extent to which reviewers cite their own work in their reports. Our results are based on a sample of 442 manuscripts and 927 referee reports submitted to the Journal of the American Society of Information Science and Technology. We find little evidence that editors, authors or reviewers use the peer review process as an opportunity to play citation games.
Similar content being viewed by others
Notes
An F test was conducted to examine the differences in variance and found unequal variances in both cases [revisions and rejections (p = <0.001); major revision and minor revisions (p = <0.05)].
A χ2 of 3.04 with 3 degrees of freedom was found including all decision levels. A subsequent analysis was done by aggregating revisions and excluding the acceptance category from the analysis. This yielded a χ2 of 1.27 with one degree of freedom.
References
Bonjean, C. M., & Hullum, J. (1978). Reasons for journal rejection: an analysis of 600 manuscripts. PS, 11(4), 480–483.
Bornmann, L. (2011). Scientific peer review. In: B. Cronin (Ed.). Annual Review of Information Science and Technology, 45, 199–245.
Bornmann, L., & Daniel, H.-D. (2009). Reviewer and editor biases in journal peer review: an investigation of manuscript refereeing at Angewandte Chemie International Edition. Research Evaluation, 18(4), 262–272.
Bornmann, L., Weymuth, C., & Daniel, H.-D. (2010). A content analysis of referees’ comments: how do comments on manuscripts rejected by a high-impact journal and later published in either a low- or high-impact journal differ? Scientometrics, 83, 493–506.
Bornmann, L., Wofl, M., & Daniel, H.-D. (2012). Closed versus open reviewing of journal manuscripts: how far do comments differ in language use? Scientometrics, 91, 843–856.
Borrego, A., Barrios, M., Villarroya, A., & Olle, C. (2010). Scientific output and impact of postdoctoral scientists: a gender perspective. Scientometrics, 83(1), 93–101.
Borsuk, R. M., Aarssen, L. W., Budden, A. E., Koricheva, J., Leimu, R., Tregenza, T., et al. (2009). To name or not to name: the effect of changing author gender on peer review. BioScience, 59(11), 985–989.
Bourne, P.E., Clark, T., Dale, R., de Waard, A., Herman, I., Hovy, E., Shotton, D. (Eds.). (2012). Force11 Maniesto: Improving future research communication and e-scholarship. White paper. Retrieved online at: http://force11.org/white_paper.
Brogaard, J., Engelberg, J., & Parsons, C. A. (2011). Network position and productivity: evidence from journal editor rotations. Retrieved from: http://www.hbs.edu/units/finance/pdf/BEP_11_09_Final.pdf.
Campanario, J. M. (1996). The competition for journal space among referees, editors, and other authors and its influence on journals’ impact factors. Journal of the American Society for Information Science, 47(3), 184–192.
Campanario, J. M., & Acedo, E. (2007). Rejecting highly cited papers: the views of scientists who encounter resistance to their discoveries from other scientists. Journal of the American Society for Information Science and Technology, 58(5), 734–743.
Ceci, S. J., & Williams, W. M. (2011). Understanding current causes of women’s underrepresentation in science. Proceedings of the National Academy of Science, 108(8), 3157–3162.
Chubin, D. E. & Hackett, E. J. (1990). Peerless science: Peer review and US science policy. Stony Brook: State University of New York Press.
Clarke, M. (2008). Jim Testa interview at the Thomson Citation Impact Center. Citations in Science—Don’t Quote Me on That forum. Retrieved from: http://network.nature.com/groups/citation-science/forum/topics/2091.
Crane, D. (1967). The gatekeepers of science: some factors affecting the selection of articles for scientific journals. The American Sociologist, 2(4), 195–201.
Cronin, B. (2012). Editorial. Do me a favor. Journal of the American Society for Information Science and Technology, 63(7), 1281.
Daniel, H. -D. (1993). Guardians of science: Fairness and reliability of peer review. Germany: Wiley-VCH. Published online 16 July, 2004. doi:10.1002/3527602208.
Davis, P. (2012). Citation cartel journals denied 2011 Impact Factor. The Scholarly Kitchen. Retrieved from: http://scholarlykitchen.sspnet.org/2012/06/29/citation-cartel-journals-denied-2011-impact-factor/.
Ernst, E., & Kienbacher, T. (1991). Chauvinism. Nature, 352, 560.
Franck, G. (1999). Scientific communication—a vanity fair? Science, 286(5437), 53–55.
Frandsen, T. F., & Nicolaisen, J. (2010). A lucrative seat at the table: are editorial board members generally over-cited in their own journals? Proceedings of the Annual Meeting of ASIS&T, 47, 10.
Frandsen, T. F., & Nicolaisen, J. (2011). Praise the bridge that carries you over: testing the flattery citation hypothesis. Journal of the American Society for Information Science and Technology, 62(5), 807–818.
Grant, B. (2009). Citation amnesia: The results. The Scientist. Retrieved from: http://classic.the-scientist.com/blog/display/55801/.
Jayasinghe, U. W., Marsh, H. W., Bond, N. (2003). A multilevel cross-classified modelling approach to peer review of grant proposals: the effects of assessor and researcher attributes on assessor ratings. Journal of the Royal Statistical Society. Series A (Statistics in Society), 166(3), 279–300.
Laband, D. N., & Piette, M. J. (1994). Favoritism versus search for good papers: empirical evidence regarding the behavior of journal editors. Journal of Political Economy, 102(1), 194–203.
Lane, J. A., & Linden, D. J. (2009). Is there gender bias in the peer review process at Journal of Neurophysiology? Journal of Neurophysiology, 101(5), 2195–2196.
Lee, C. J., Sugimoto, C. R., Zhang, G., & Cronin, B. (2012). Bias in peer review. Journal of the American Society for Information Science & Technology.
Link, A. M. (1998). US and non-US submissions. Journal of the American Medical Association, 280(3), 246–247.
MacRoberts, M. H., & MacRoberts, B. R. (1989). Problems of citation analysis: a critical review. Journal of the American Society for Information Science, 40(5), 342–349.
Mahoney, M. J. (1977). Publication prejudices: an experimental study of confirmatory bias in the peer review system. Cognitive Therapy and Research, 1(2), 161–175.
Merton, R. K. (1973). The sociology of science: Theoretical and empirical investigations. Chicago: University of Chicago Press.
Moed, H. F. (2002). The impact-factors debate: the ISI’s uses and limits. Nature, 415(6873), 731–732.
Moed, H. F., & van Leeuwen, T. N. (1995). Improving the accuracy of institute for scientific information’s journal impact factors. Journal of the American Society for Information Science, 46(6), 461–467.
Moed, H. F., & van Leeuwen, T. N. (1996). Impact factors can mislead. Nature, 381(6579), 186.
Monastersky, R. (2005). The number that’s devouring science. The Chronicle of Higher Education. Available online at: http://chronicle.com/article/The-Number-That-s-Devouring/26481.
Peters, D. P., & Ceci, S. J. (1982). Peer-review practices of psychological journals: the fate of accepted, published articles, submitted again. The Behavioral and Brain Sciences, 5(2), 187–195.
Shatz, D. (2004). Peer review: a critical inquiry. Lanham: Rowman & Littlefield Publishers, Inc.
Smith, R. (2006). Commentary: the power of the unrelenting impact factor—is it a force for good or harm? International Journal of Epidemiology, 35, 1129–1130.
Symonds, M. R. E., Gemmell, N. J., Braisher, T. L., Gorringe, K. L., & Elgar, M. A. (2006). Gender differences in publication output: towards an unbiased metric of research performance. PLoS ONE, 1(1), e127.
Vanclay, J. K. (2009). Bias in the journal impact factor. Scientometrics, 78(1), 3–12.
Wilhite, A. W., & Fong, E. A. (2012). Coercive citation in academic publishing. Science, 335, 542–543.
Wing, D. A., Benner, R. S., Petersen, R., Newcomb, R., & Scott, J. R. (2010). Differences in editorial board reviewer behavior based on gender. Journal of Women’s Health, 19(10), 1919–1923.
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Sugimoto, C.R., Cronin, B. Citation gamesmanship: testing for evidence of ego bias in peer review. Scientometrics 95, 851–862 (2013). https://doi.org/10.1007/s11192-012-0845-z
Received:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11192-012-0845-z