Skip to main content
Log in

Citation gamesmanship: testing for evidence of ego bias in peer review

Scientometrics Aims and scope Submit manuscript

Abstract

This study tests for evidence of gaming and attention mongering—here termed ego bias—in the scholarly peer review process. We explore the extent to which authors cite the target journal and its editor and also the relationship between targeted references and editorial decisions. We examine referee reports for the presence and type of references and determine the extent to which reviewers cite their own work in their reports. Our results are based on a sample of 442 manuscripts and 927 referee reports submitted to the Journal of the American Society of Information Science and Technology. We find little evidence that editors, authors or reviewers use the peer review process as an opportunity to play citation games.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

Notes

  1. An F test was conducted to examine the differences in variance and found unequal variances in both cases [revisions and rejections (p = <0.001); major revision and minor revisions (p = <0.05)].

  2. A χ2 of 3.04 with 3 degrees of freedom was found including all decision levels. A subsequent analysis was done by aggregating revisions and excluding the acceptance category from the analysis. This yielded a χ2 of 1.27 with one degree of freedom.

References

  • Bonjean, C. M., & Hullum, J. (1978). Reasons for journal rejection: an analysis of 600 manuscripts. PS, 11(4), 480–483.

    Article  Google Scholar 

  • Bornmann, L. (2011). Scientific peer review. In: B. Cronin (Ed.). Annual Review of Information Science and Technology, 45, 199–245.

  • Bornmann, L., & Daniel, H.-D. (2009). Reviewer and editor biases in journal peer review: an investigation of manuscript refereeing at Angewandte Chemie International Edition. Research Evaluation, 18(4), 262–272.

    Article  Google Scholar 

  • Bornmann, L., Weymuth, C., & Daniel, H.-D. (2010). A content analysis of referees’ comments: how do comments on manuscripts rejected by a high-impact journal and later published in either a low- or high-impact journal differ? Scientometrics, 83, 493–506.

    Article  Google Scholar 

  • Bornmann, L., Wofl, M., & Daniel, H.-D. (2012). Closed versus open reviewing of journal manuscripts: how far do comments differ in language use? Scientometrics, 91, 843–856.

    Article  Google Scholar 

  • Borrego, A., Barrios, M., Villarroya, A., & Olle, C. (2010). Scientific output and impact of postdoctoral scientists: a gender perspective. Scientometrics, 83(1), 93–101.

    Article  Google Scholar 

  • Borsuk, R. M., Aarssen, L. W., Budden, A. E., Koricheva, J., Leimu, R., Tregenza, T., et al. (2009). To name or not to name: the effect of changing author gender on peer review. BioScience, 59(11), 985–989.

    Article  Google Scholar 

  • Bourne, P.E., Clark, T., Dale, R., de Waard, A., Herman, I., Hovy, E., Shotton, D. (Eds.). (2012). Force11 Maniesto: Improving future research communication and e-scholarship. White paper. Retrieved online at: http://force11.org/white_paper.

  • Brogaard, J., Engelberg, J., & Parsons, C. A. (2011). Network position and productivity: evidence from journal editor rotations. Retrieved from: http://www.hbs.edu/units/finance/pdf/BEP_11_09_Final.pdf.

  • Campanario, J. M. (1996). The competition for journal space among referees, editors, and other authors and its influence on journals’ impact factors. Journal of the American Society for Information Science, 47(3), 184–192.

    Article  Google Scholar 

  • Campanario, J. M., & Acedo, E. (2007). Rejecting highly cited papers: the views of scientists who encounter resistance to their discoveries from other scientists. Journal of the American Society for Information Science and Technology, 58(5), 734–743.

    Article  Google Scholar 

  • Ceci, S. J., & Williams, W. M. (2011). Understanding current causes of women’s underrepresentation in science. Proceedings of the National Academy of Science, 108(8), 3157–3162.

    Article  Google Scholar 

  • Chubin, D. E. & Hackett, E. J. (1990). Peerless science: Peer review and US science policy. Stony Brook: State University of New York Press.

  • Clarke, M. (2008). Jim Testa interview at the Thomson Citation Impact Center. Citations in ScienceDon’t Quote Me on That forum. Retrieved from: http://network.nature.com/groups/citation-science/forum/topics/2091.

  • Crane, D. (1967). The gatekeepers of science: some factors affecting the selection of articles for scientific journals. The American Sociologist, 2(4), 195–201.

    Google Scholar 

  • Cronin, B. (2012). Editorial. Do me a favor. Journal of the American Society for Information Science and Technology, 63(7), 1281.

    Article  Google Scholar 

  • Daniel, H. -D. (1993). Guardians of science: Fairness and reliability of peer review. Germany: Wiley-VCH. Published online 16 July, 2004. doi:10.1002/3527602208.

  • Davis, P. (2012). Citation cartel journals denied 2011 Impact Factor. The Scholarly Kitchen. Retrieved from: http://scholarlykitchen.sspnet.org/2012/06/29/citation-cartel-journals-denied-2011-impact-factor/.

  • Ernst, E., & Kienbacher, T. (1991). Chauvinism. Nature, 352, 560.

    Article  Google Scholar 

  • Franck, G. (1999). Scientific communication—a vanity fair? Science, 286(5437), 53–55.

    Article  Google Scholar 

  • Frandsen, T. F., & Nicolaisen, J. (2010). A lucrative seat at the table: are editorial board members generally over-cited in their own journals? Proceedings of the Annual Meeting of ASIS&T, 47, 10.

    Google Scholar 

  • Frandsen, T. F., & Nicolaisen, J. (2011). Praise the bridge that carries you over: testing the flattery citation hypothesis. Journal of the American Society for Information Science and Technology, 62(5), 807–818.

    Article  Google Scholar 

  • Grant, B. (2009). Citation amnesia: The results. The Scientist. Retrieved from: http://classic.the-scientist.com/blog/display/55801/.

  • Jayasinghe, U. W., Marsh, H. W., Bond, N. (2003). A multilevel cross-classified modelling approach to peer review of grant proposals: the effects of assessor and researcher attributes on assessor ratings. Journal of the Royal Statistical Society. Series A (Statistics in Society), 166(3), 279–300.

    Google Scholar 

  • Laband, D. N., & Piette, M. J. (1994). Favoritism versus search for good papers: empirical evidence regarding the behavior of journal editors. Journal of Political Economy, 102(1), 194–203.

    Google Scholar 

  • Lane, J. A., & Linden, D. J. (2009). Is there gender bias in the peer review process at Journal of Neurophysiology? Journal of Neurophysiology, 101(5), 2195–2196.

    Article  Google Scholar 

  • Lee, C. J., Sugimoto, C. R., Zhang, G., & Cronin, B. (2012). Bias in peer review. Journal of the American Society for Information Science & Technology.

  • Link, A. M. (1998). US and non-US submissions. Journal of the American Medical Association, 280(3), 246–247.

    Article  Google Scholar 

  • MacRoberts, M. H., & MacRoberts, B. R. (1989). Problems of citation analysis: a critical review. Journal of the American Society for Information Science, 40(5), 342–349.

    Article  Google Scholar 

  • Mahoney, M. J. (1977). Publication prejudices: an experimental study of confirmatory bias in the peer review system. Cognitive Therapy and Research, 1(2), 161–175.

    Article  Google Scholar 

  • Merton, R. K. (1973). The sociology of science: Theoretical and empirical investigations. Chicago: University of Chicago Press.

    Google Scholar 

  • Moed, H. F. (2002). The impact-factors debate: the ISI’s uses and limits. Nature, 415(6873), 731–732.

    Article  Google Scholar 

  • Moed, H. F., & van Leeuwen, T. N. (1995). Improving the accuracy of institute for scientific information’s journal impact factors. Journal of the American Society for Information Science, 46(6), 461–467.

    Article  Google Scholar 

  • Moed, H. F., & van Leeuwen, T. N. (1996). Impact factors can mislead. Nature, 381(6579), 186.

    Article  Google Scholar 

  • Monastersky, R. (2005). The number that’s devouring science. The Chronicle of Higher Education. Available online at: http://chronicle.com/article/The-Number-That-s-Devouring/26481.

  • Peters, D. P., & Ceci, S. J. (1982). Peer-review practices of psychological journals: the fate of accepted, published articles, submitted again. The Behavioral and Brain Sciences, 5(2), 187–195.

    Article  Google Scholar 

  • Shatz, D. (2004). Peer review: a critical inquiry. Lanham: Rowman & Littlefield Publishers, Inc.

    Google Scholar 

  • Smith, R. (2006). Commentary: the power of the unrelenting impact factor—is it a force for good or harm? International Journal of Epidemiology, 35, 1129–1130.

    Article  Google Scholar 

  • Symonds, M. R. E., Gemmell, N. J., Braisher, T. L., Gorringe, K. L., & Elgar, M. A. (2006). Gender differences in publication output: towards an unbiased metric of research performance. PLoS ONE, 1(1), e127.

    Article  Google Scholar 

  • Vanclay, J. K. (2009). Bias in the journal impact factor. Scientometrics, 78(1), 3–12.

    Article  Google Scholar 

  • Wilhite, A. W., & Fong, E. A. (2012). Coercive citation in academic publishing. Science, 335, 542–543.

    Article  Google Scholar 

  • Wing, D. A., Benner, R. S., Petersen, R., Newcomb, R., & Scott, J. R. (2010). Differences in editorial board reviewer behavior based on gender. Journal of Women’s Health, 19(10), 1919–1923.

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Cassidy R. Sugimoto.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Sugimoto, C.R., Cronin, B. Citation gamesmanship: testing for evidence of ego bias in peer review. Scientometrics 95, 851–862 (2013). https://doi.org/10.1007/s11192-012-0845-z

Download citation

  • Received:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11192-012-0845-z

Keywords

Navigation