Scientometrics

, Volume 95, Issue 3, pp 851–862 | Cite as

Citation gamesmanship: testing for evidence of ego bias in peer review

Article

Abstract

This study tests for evidence of gaming and attention mongering—here termed ego bias—in the scholarly peer review process. We explore the extent to which authors cite the target journal and its editor and also the relationship between targeted references and editorial decisions. We examine referee reports for the presence and type of references and determine the extent to which reviewers cite their own work in their reports. Our results are based on a sample of 442 manuscripts and 927 referee reports submitted to the Journal of the American Society of Information Science and Technology. We find little evidence that editors, authors or reviewers use the peer review process as an opportunity to play citation games.

Keywords

Peer review Bias Scholarly communication Citation analysis Scientometrics 

References

  1. Bonjean, C. M., & Hullum, J. (1978). Reasons for journal rejection: an analysis of 600 manuscripts. PS, 11(4), 480–483.CrossRefGoogle Scholar
  2. Bornmann, L. (2011). Scientific peer review. In: B. Cronin (Ed.). Annual Review of Information Science and Technology, 45, 199–245.Google Scholar
  3. Bornmann, L., & Daniel, H.-D. (2009). Reviewer and editor biases in journal peer review: an investigation of manuscript refereeing at Angewandte Chemie International Edition. Research Evaluation, 18(4), 262–272.CrossRefGoogle Scholar
  4. Bornmann, L., Weymuth, C., & Daniel, H.-D. (2010). A content analysis of referees’ comments: how do comments on manuscripts rejected by a high-impact journal and later published in either a low- or high-impact journal differ? Scientometrics, 83, 493–506.CrossRefGoogle Scholar
  5. Bornmann, L., Wofl, M., & Daniel, H.-D. (2012). Closed versus open reviewing of journal manuscripts: how far do comments differ in language use? Scientometrics, 91, 843–856.CrossRefGoogle Scholar
  6. Borrego, A., Barrios, M., Villarroya, A., & Olle, C. (2010). Scientific output and impact of postdoctoral scientists: a gender perspective. Scientometrics, 83(1), 93–101.CrossRefGoogle Scholar
  7. Borsuk, R. M., Aarssen, L. W., Budden, A. E., Koricheva, J., Leimu, R., Tregenza, T., et al. (2009). To name or not to name: the effect of changing author gender on peer review. BioScience, 59(11), 985–989.CrossRefGoogle Scholar
  8. Bourne, P.E., Clark, T., Dale, R., de Waard, A., Herman, I., Hovy, E., Shotton, D. (Eds.). (2012). Force11 Maniesto: Improving future research communication and e-scholarship. White paper. Retrieved online at: http://force11.org/white_paper.
  9. Brogaard, J., Engelberg, J., & Parsons, C. A. (2011). Network position and productivity: evidence from journal editor rotations. Retrieved from: http://www.hbs.edu/units/finance/pdf/BEP_11_09_Final.pdf.
  10. Campanario, J. M. (1996). The competition for journal space among referees, editors, and other authors and its influence on journals’ impact factors. Journal of the American Society for Information Science, 47(3), 184–192.CrossRefGoogle Scholar
  11. Campanario, J. M., & Acedo, E. (2007). Rejecting highly cited papers: the views of scientists who encounter resistance to their discoveries from other scientists. Journal of the American Society for Information Science and Technology, 58(5), 734–743.CrossRefGoogle Scholar
  12. Ceci, S. J., & Williams, W. M. (2011). Understanding current causes of women’s underrepresentation in science. Proceedings of the National Academy of Science, 108(8), 3157–3162.CrossRefGoogle Scholar
  13. Chubin, D. E. & Hackett, E. J. (1990). Peerless science: Peer review and US science policy. Stony Brook: State University of New York Press.Google Scholar
  14. Clarke, M. (2008). Jim Testa interview at the Thomson Citation Impact Center. Citations in ScienceDon’t Quote Me on That forum. Retrieved from: http://network.nature.com/groups/citation-science/forum/topics/2091.
  15. Crane, D. (1967). The gatekeepers of science: some factors affecting the selection of articles for scientific journals. The American Sociologist, 2(4), 195–201.Google Scholar
  16. Cronin, B. (2012). Editorial. Do me a favor. Journal of the American Society for Information Science and Technology, 63(7), 1281.CrossRefGoogle Scholar
  17. Daniel, H. -D. (1993). Guardians of science: Fairness and reliability of peer review. Germany: Wiley-VCH. Published online 16 July, 2004. doi:10.1002/3527602208.
  18. Davis, P. (2012). Citation cartel journals denied 2011 Impact Factor. The Scholarly Kitchen. Retrieved from: http://scholarlykitchen.sspnet.org/2012/06/29/citation-cartel-journals-denied-2011-impact-factor/.
  19. Ernst, E., & Kienbacher, T. (1991). Chauvinism. Nature, 352, 560.CrossRefGoogle Scholar
  20. Franck, G. (1999). Scientific communication—a vanity fair? Science, 286(5437), 53–55.CrossRefGoogle Scholar
  21. Frandsen, T. F., & Nicolaisen, J. (2010). A lucrative seat at the table: are editorial board members generally over-cited in their own journals? Proceedings of the Annual Meeting of ASIS&T, 47, 10.Google Scholar
  22. Frandsen, T. F., & Nicolaisen, J. (2011). Praise the bridge that carries you over: testing the flattery citation hypothesis. Journal of the American Society for Information Science and Technology, 62(5), 807–818.CrossRefGoogle Scholar
  23. Grant, B. (2009). Citation amnesia: The results. The Scientist. Retrieved from: http://classic.the-scientist.com/blog/display/55801/.
  24. Jayasinghe, U. W., Marsh, H. W., Bond, N. (2003). A multilevel cross-classified modelling approach to peer review of grant proposals: the effects of assessor and researcher attributes on assessor ratings. Journal of the Royal Statistical Society. Series A (Statistics in Society), 166(3), 279–300.Google Scholar
  25. Laband, D. N., & Piette, M. J. (1994). Favoritism versus search for good papers: empirical evidence regarding the behavior of journal editors. Journal of Political Economy, 102(1), 194–203.Google Scholar
  26. Lane, J. A., & Linden, D. J. (2009). Is there gender bias in the peer review process at Journal of Neurophysiology? Journal of Neurophysiology, 101(5), 2195–2196.CrossRefGoogle Scholar
  27. Lee, C. J., Sugimoto, C. R., Zhang, G., & Cronin, B. (2012). Bias in peer review. Journal of the American Society for Information Science & Technology.Google Scholar
  28. Link, A. M. (1998). US and non-US submissions. Journal of the American Medical Association, 280(3), 246–247.CrossRefGoogle Scholar
  29. MacRoberts, M. H., & MacRoberts, B. R. (1989). Problems of citation analysis: a critical review. Journal of the American Society for Information Science, 40(5), 342–349.CrossRefGoogle Scholar
  30. Mahoney, M. J. (1977). Publication prejudices: an experimental study of confirmatory bias in the peer review system. Cognitive Therapy and Research, 1(2), 161–175.CrossRefGoogle Scholar
  31. Merton, R. K. (1973). The sociology of science: Theoretical and empirical investigations. Chicago: University of Chicago Press.Google Scholar
  32. Moed, H. F. (2002). The impact-factors debate: the ISI’s uses and limits. Nature, 415(6873), 731–732.CrossRefGoogle Scholar
  33. Moed, H. F., & van Leeuwen, T. N. (1995). Improving the accuracy of institute for scientific information’s journal impact factors. Journal of the American Society for Information Science, 46(6), 461–467.CrossRefGoogle Scholar
  34. Moed, H. F., & van Leeuwen, T. N. (1996). Impact factors can mislead. Nature, 381(6579), 186.CrossRefGoogle Scholar
  35. Monastersky, R. (2005). The number that’s devouring science. The Chronicle of Higher Education. Available online at: http://chronicle.com/article/The-Number-That-s-Devouring/26481.
  36. Peters, D. P., & Ceci, S. J. (1982). Peer-review practices of psychological journals: the fate of accepted, published articles, submitted again. The Behavioral and Brain Sciences, 5(2), 187–195.CrossRefGoogle Scholar
  37. Shatz, D. (2004). Peer review: a critical inquiry. Lanham: Rowman & Littlefield Publishers, Inc.Google Scholar
  38. Smith, R. (2006). Commentary: the power of the unrelenting impact factor—is it a force for good or harm? International Journal of Epidemiology, 35, 1129–1130.CrossRefGoogle Scholar
  39. Symonds, M. R. E., Gemmell, N. J., Braisher, T. L., Gorringe, K. L., & Elgar, M. A. (2006). Gender differences in publication output: towards an unbiased metric of research performance. PLoS ONE, 1(1), e127.CrossRefGoogle Scholar
  40. Vanclay, J. K. (2009). Bias in the journal impact factor. Scientometrics, 78(1), 3–12.CrossRefGoogle Scholar
  41. Wilhite, A. W., & Fong, E. A. (2012). Coercive citation in academic publishing. Science, 335, 542–543.CrossRefGoogle Scholar
  42. Wing, D. A., Benner, R. S., Petersen, R., Newcomb, R., & Scott, J. R. (2010). Differences in editorial board reviewer behavior based on gender. Journal of Women’s Health, 19(10), 1919–1923.CrossRefGoogle Scholar

Copyright information

© Akadémiai Kiadó, Budapest, Hungary 2012

Authors and Affiliations

  1. 1.School of Library and Information ScienceIndiana UniversityBloomingtonUSA

Personalised recommendations