Scientometrics

, Volume 95, Issue 3, pp 851–862

Citation gamesmanship: testing for evidence of ego bias in peer review

Authors

    • School of Library and Information ScienceIndiana University
  • Blaise Cronin
    • School of Library and Information ScienceIndiana University
Article

DOI: 10.1007/s11192-012-0845-z

Cite this article as:
Sugimoto, C.R. & Cronin, B. Scientometrics (2013) 95: 851. doi:10.1007/s11192-012-0845-z

Abstract

This study tests for evidence of gaming and attention mongering—here termed ego bias—in the scholarly peer review process. We explore the extent to which authors cite the target journal and its editor and also the relationship between targeted references and editorial decisions. We examine referee reports for the presence and type of references and determine the extent to which reviewers cite their own work in their reports. Our results are based on a sample of 442 manuscripts and 927 referee reports submitted to the Journal of the American Society of Information Science and Technology. We find little evidence that editors, authors or reviewers use the peer review process as an opportunity to play citation games.

Keywords

Peer reviewBiasScholarly communicationCitation analysisScientometrics

Introduction and background

It is claimed recurrently that the system of scholarly peer review is prone to bias (e.g., Chubin and Hackett 1990). What exactly does that mean and what evidence exists to support the claim? Simply stated, bias in peer review may be defined as a violation of the norms of impartiality whereby reviewers (and editors) fail to consistently apply evaluative criteria within and across manuscripts, or apply criteria that are unrelated to the “true quality” of the manuscripts (Lee et al. 2012). By way of illustration, articles lacking prestigious institutional affiliations are rejected at higher rates (e.g., Peters and Ceci 1982), journals tend to preferentially publish articles from in-country authors (e.g., Daniel 1993; Link 1998; Ernst and Kienbacher 1991), women tend to apply more stringent review criteria than men (e.g., Wing et al. 2010), and reviewers tend to favor articles with epistemological frameworks similar to their own (e.g., Mahoney 1977). On the other hand, it is not always or altogether clear that the findings of such studies constitute evidence of bias qua bias. For instance, might not affiliation serve as a relatively consistent proxy for quality? Might not authors from the journal’s host country be more likely to submit to the journal than authors from elsewhere? In a large-scale meta-analysis of the literature of the subject, Ceci and Williams (2011) found that evidence of bias in peer review was anything but clear-cut. However, we do acknowledge at the outset the potential for bias of one kind or another from the various actors in the peer review system—authors, editors, and reviewers. We now consider each in turn.

Authors

Concerns about bias have been raised with regard to authors’ behaviors when it comes to citing the literature and fully and dutifully acknowledging what might be termed the “true contributions” of their peers (Bourne et al. 2012 para. 6). The implication is that the norm of impartiality is violated when authors’ motivations for citing (or failing to cite) specific works are extra-scientific in nature. The “flattery hypothesis” captures the idea that authors excessively cite editors and potential reviewers of the target journal (Frandsen and Nicolaisen 2011), although studies have found no strong evidence of such behavior (e.g., Frandsen and Nicolaisen 2010, 2011). Authors may also experience “citation amnesia” while constructing reference lists (e.g., MacRoberts and MacRoberts 1989; Grant 2009; Lee et al. 2012). Aside from being sometimes forgivably unaware of relevant work, authors may chose to ignore related studies that do not support their model, thesis or findings, and may excessively and preferentially cite their own work and/or that of colleagues. The idea of “citation cartels,” raised by Franck (1999), suggests that scholars, in competing for attention, turn into “citation-maximizers,” crafting articles in such a way as to increase “attentive returns.”

Editors

Editors, for their part, may engage in what has been termed “coercive citation” (Wilhite and Fong 2012), pressuring authors—whether of their own volition or in response to actual or perceived pressure from journal publishers—to pad their papers with citations to the target journal in an effort to boost its Impact Factor (e.g., Cronin 2012). Editors may also publish editorials replete with journal self-citations for the same reasons (Moed and van Leeuwen 1995). Abuses and gaming of the Impact Factor have been well documented (Monastersky 2005; Moed and van Leeuwen 1996; Moed 2002; Vanclay 2009) such that Thomson Reuters will suspend journals from the Journal Citation Reports if anomalous citation patterns are detected (Davis 2012; Clarke 2008). Nonetheless, the obsession with Impact Factors has not lessened (Smith 2006). The appearance of cronyism may also be an occupational hazard for journal editors: studies have found that editors accept papers from colleagues and co-authors at a higher rate than from others (Brogaard et al. 2011; Laband and Piette 1994). It should be noted, however, that these same papers were more highly cited than non-collegiate submissions, nicely underscoring the complex interplay of quality, impact, and social networks (Brogaard et al. 2011). It has also been suggested that “editorial favoritism” functions to “enhance efficiency in the market for scientific knowledge” by using editorial networks to preselect high quality papers (Laband and Piette 1994, p. 194–195). The role of the privileged network of the “gatekeepers of science” (Crane 1967) warrants closer scrutiny.

Reviewers

Reviewers may sometimes be guilty of coercion, given their status as potential ‘competitors’ (Campanario and Acedo 2007). They can use the anonymity afforded by peer review to ‘encourage’ authors to cite, or additionally cite, their work. Evidence of such coercion might be found by examining the extent to which reviewers self-cite in their reports (i.e., include a reference to their own work). Such behavior is not inherently undesirable: reviewers are often approached precisely because they have expertise in a particular area and, therefore, their own work may well be directly relevant to the manuscript at hand. While several studies have examined reviews from the perspective of discourse and linguistic content (e.g., Bornmann et al. 2010, 2012), or explored the reasons given for rejection in referee reports (Bonjean and Hullum 1978; Chubin and Hackett 1990; Bornmann and Daniel 2009), few have looked into the relationship between referencing, editors, and reviewers, specifically the references recommended to authors by reviewers of their manuscripts. The reason for that is simple: lack of readily available data sets. There is the added complexity of overlap between actors in the system: Campanario (1996) found that, for the journals studied, approximately one-third of the manuscripts were authored by at least one journal-affiliated individual (reviewer or editor).

Scope and purpose

Highly productive scholars have been characterized as having “marked ego strength” (Merton 1973 p. 454). The extent to which ego manifests itself in strategic gaming, cronyism, and attention mongering—which we here subsume under the term ‘ego bias’—in the editorial review process is unclear. Is there, we wonder, credible empirical evidence of unilateral, bilateral or multilateral abuse of the peer review process, what might be termed citation gamesmanship? The purpose of this study is to address the following research questions using a sample of submitted manuscripts and their associated referee reports. The anonymized data set comes from the Journal of the American Society for Information Science & Technology (JASIS&T), of which, one of us (BC), is the Editor-in-Chief.
  • RQ1. Do manuscripts referencing JASIS&T receive more favorable publication decisions than those that do not?

  • RQ2. Do manuscripts referencing the editor receive more favorable publication decisions than those that do not?

  • RQ3. Do manuscripts referencing reviewers receive more favorable reviews than those that do not?

  • RQ4. How frequently do reviewers reference specific authors, articles, and venues in their reviews?

  • RQ5. To what extent do reviewers self-cite in their reviews?

  • RQ6. Are reviewers who are referenced in a manuscript more likely to self-cite in their review?

  • RQ7. Is there a difference between the degree to which male and female reviewers self-cite in their reviews?

Previous studies of peer review have used both obtrusive and unobtrusive measures (Bornmann 2011). However, studies looking specifically at the extent to which reviewers insert themselves into reviews have primarily relied on self-reported data rather than unobtrusive measures to gather evidence of presumptive bias (e.g., Wilhite and Fong 2012). Here we analyze manuscripts and reviewers’ reports to better understand the relationship among referencing behavior (in both manuscripts and reviews), stakeholders in the review process (editors, authors, reviewers), and editorial decisions (revision, rejection, acceptance).

Methods

Data

Manuscripts (research articles and brief communications), and their associated reviews, submitted to JASIS&T between June 2009 and May 2011 constitute the data set. Only original submissions were included; resubmitted manuscripts were excluded from the sample. Desk rejections (i.e., those manuscripts rejected by the editor without external review), were also excluded. In total, 442 manuscripts and 927 associated reviews were identified for analysis.

Description of data

Each manuscript was associated with one editorial decision (made by the Editor-in-Chief) and, on average, two reviewer recommendations. The distribution of editorial decisions is shown in Table 1.
Table 1

Percent and number of manuscripts by editorial decision level

Decision

# of manuscripts

% of manuscripts

Accept

2

<1

Minor revision

88

20

Major revision

186

42

Reject

166

38

The distribution of reviewer recommendations is shown in Table 2. Editorial stringency is evident when comparing Tables 1 and 2: 20 % of the editor’s decisions were at the accept/minor revision level, compared with nearly 37 % for reviewers; the editor rejected 38 % of manuscripts, reviewers 29 %.
Table 2

Percent and number of reviews by recommendation level

Recommendation

# or reviews

 % of reviews

Accept

78

8

Minor revision

269

29

Major revision

315

34

Reject

265

29

It is possible that the analyses are moderated by the number of words in any given review—an increase in the number of words in a manuscript may increase the chances that a citation (to oneself or otherwise) will appear. Therefore, we analyzed the number of words by recommendation level, to ascertain any differences attributable to length (Table 3). An ANOVA was run to compare means for minor revision, major revision, and rejections; however, Levene’s test for homogeneity showed the ANOVA to be invalid. Therefore, independent sample t tests were conducted; the results showed statistically significant (p < 0.01) differences between minor and major revisions, major revisions and rejections, and minor revisions and rejections. Aggregating revisions (minor and major) together, no statistically significant difference was found (p = 0.713) between revisions and rejections.
Table 3

Mean word count of reviews by recommendation level

Decision

Mean

Maximum

SD

Accept

163.7

843

186.2

Minor revision

441.9

3,153

405.3

Major revision

728.8

3,958

552.5

Reject

580.8

4,834

588.3

ALL

556.2

4,834

528.9

Analysis

The two main units of analysis were manuscripts and reviews. The number of references made to JASIS&T and its precursors (Journal of the American Society for Information Science and American Documentation) was recorded for each submitted manuscript. A t test was conducted to analyze whether there was any relationship between editorial decisions and the number of references to the Journal. References to the editor (whether as author or editor of a work) were recorded at the binary level: There was either a reference or not in any given manuscript; the actual number was not counted. A χ2 test was used to determine whether the editor seemed to favor manuscripts that cited him. References to reviewers were also tabulated. As before, we noted the presence or absence of a reference rather than the number of times a reviewer’s work was referenced.

References in the reviews were tabulated and assigned to one of three exclusive categories: (1) article: a specific article was mentioned; (2) author: an author’s name was mentioned in a general way (e.g., the author should look at the works of Chen); and (3) venue: a general venue was mentioned (e.g., the author should consult earlier relevant work published in JASIS&T). The reviews were also examined to determine whether or not the reviewers referenced themselves. The Student’s t and χ2 tests were used, where appropriate, to investigate the relationships among variables.

Limitations

Our study is based on a single journal, JASIS&T, covering a short time period, which means that we should eschew generalization. However, it does provide a useful baseline and framework for conducting follow-up studies designed to test for the presence of ego bias in scholarly peer review.

Results

References to JASIS&T

Seventy-five percent of all manuscripts in our sample contained at least one reference to JASIS&T. The total number of references in a single paper ranged from two to 148, with a mean of roughly 41. On average, manuscripts referenced JASIS&T thrice (see Table 4). Manuscripts rejected by the editor referenced JASIS&T less than those manuscripts recommended for revision. A calculation of the percentage of references in each manuscript was made and then a mean percentage for each category was compiled, which showed that, overall, 8 % of references in a given manuscript were to JASIS&T. The table also shows the percentage in each category with at least one reference to JASIS&T; this figure decreases from revision (minor and major) to rejection.
Table 4

References to JASIS&T and total number of references by decision level

Editorial decision

Number of manuscripts

Mean number of references

Mean number of references to JASIS&T

Mean percentage of references to JASIS&T (%)

Percentage of manuscripts with at least one reference to JASIS&T (%)

Accept

2

32.5

0.5

1

50

Minor revision

88

41.9

3.5

9

82

Major revision

186

43.6

3.7

9

76

Reject

166

36.9

2.2

7

70

TOTAL

442

40.7

3.1

8

75

Given the low number of acceptances (n = 2), this category was discarded and we focused on differences between (a) revisions (major and minor) and rejections and (b) major and minor revisions. There was no statistically significant difference between major and minor decisions in the degree to which JASIS&T was referenced (p = 0.66; calculated using a t test assuming unequal variances1). However, there was a statistically significant difference between revisions (minor and major, aggregated) and rejections (p = <0.001). Given that the data were not normally distributed, a Kruskal–Wallis H test was run to test for difference between the mean percentages of references to JASIS&T. Excluding accept, differences between the remaining three variables were significant at the 0.05 level. In a pairwise comparison, no significant differences were found between minor and major revisions, but significant differences (at the 0.05 level) were found between rejections and the other two variables. The data suggest that manuscripts that cite JASIS&T more heavily are somewhat more likely to receive an editorial decision requesting revision, rather than rejection.

References to the editor

Thirteen percent of all manuscripts referenced the current JASIS&T editor. Table 5 shows the percentage of items referencing the editor by decision level: a larger proportion of manuscripts with revisions referenced the editor than did rejected manuscripts. However, these differences were found to be statistically insignificant when testing for all levels and looking only at differences between revisions (major and minor, aggregated) and rejections.2 Therefore, there is no statistically significant relationship between the degree to which the editor is referenced and the editor’s final decision on a manuscript.
Table 5

Number and percent of manuscripts referencing the editor at each decision level

Decision

# of manuscripts

# and % referencing the editor

Accept

2

0 (0 %)

Minor revision

88

16 (18 %)

Major revision

186

24 (13 %)

Reject

166

18 (11 %)

TOTAL

442

58 (13 %)

References to reviewers

Nearly half of all manuscripts (48 %) were assigned to at least one reviewer listed in the cited reference list of the manuscript: 14 % were assigned to more than one individual from the list. There was a statistically significant difference (p = <0.01) in editorial decision between revision (major/minor) and rejection when a reviewer was referenced (Table 6) (χ2 = 15.88; df = 1).
Table 6

Number and percent of manuscripts referencing the reviewer by decision level

Decision

# of manuscripts

# and  % referencing the reviewer

Accept

2

1 (50 %)

Minor revision

88

42 (48 %)

Major revision

186

98 (53 %)

Reject

166

73 (44 %)

TOTAL

442

214 (48 %)

There is also a statistically significant difference (p = <0.05) in reviewer recommendation level between accept and reject when the reviewer is referenced in the manuscript. However, when calculating across all decision levels (accept, minor, major, and reject) no statistically significant differences were found (Table 7) (χ2 = 2.26; df = 3).
Table 7

Number and percent of manuscripts referencing the reviewer by recommendation level

Recommendation

# of reviews

# and % referencing the reviewer

Accept

78

25 (32 %)

Minor revision

269

82 (30 %)

Major revision

315

100 (32 %)

Reject

265

70 (26 %)

TOTAL

927

227 (30 %)

These findings suggest that editorial decisions are somewhat more likely to be favorable when manuscripts are assigned to reviewers who have been referenced, but, except at the most extreme levels, reviewer recommendations are independent of whether or not the reviewer has been referenced.

References in reviews

Of the 927 reviews, 43 % (n = 398) contained at least one reference (as defined in the Methods section). Of the 398 reviews containing at least one reference, 43 % contained a singleton. Twenty-two reviews had 10 or more references; the maximum was 19. The majority of references (74 %; n = 933) were to specific articles (Table 8). The next most common reference type was to the work of a named individual. Venues, such as a journal or conference, were mentioned 53 times in 35 reviews; only three reviews suggested adding a global reference to JASIS&T.
Table 8

Number of embedded references in reviews to articles, authors, and venues by recommendation level

Recommendation level

# of reviews

# of reviews with references

# of embedded references by type

Articles

Authors

Venues

Accept

78

16 (21 %)

16 (2 %)

9 (3 %)

0 (0 %)

Minor revision

269

105 (39 %)

235 (25 %)

57 (20 %)

14 (26 %)

Major revision

315

161 (51 %)

445 (48 %)

120 (42 %)

27 (51 %)

Reject

265

116 (44 %)

237 (25 %)

98 (25 %)

12 (23 %)

TOTAL

927

398 (43 %)

933

284

53

Reviews recommending major revision were most likely to include references in the review and, when they did, these were most likely to be to specific journal articles. Major revision held the highest share of any reference type; reviews recommending minor revision and rejection were fairly similar in their use of reference types, although authors were listed more in recommendations for rejection than in minor revisions.

Reviewer self-referencing

Fifteen percent of the reviews included a self-citation (i.e., a reference to the work of the person writing the review) and 17 % of reviewers self-cited at least once. Such behavior could be seen as problematic if the reviewers were using the review as a way to boost inappropriately their citation count. To test for this, we calculated the proportion of reviews that were assigned to reviewers included in the list of references and the relationship between appearing in the reference list and self-citing in the review. Thirty percent of the reviews were written by reviewers who were also listed in the references. Significance tests (independent sample t test for proportions) demonstrated that if a reviewer is referenced, he or she is statistically significantly more likely to self-cite (p = <0.001) when providing feedback.

We then investigated the relationship between reviewer recommendation levels and self-citation. There were statistically significant differences between recommendation levels and self-citations with major revision receiving the highest percentage of reviewer self-cites (χ2 = 14.33926; df = 3; see Table 9). Cramer’s V was used as a post hoc test to determine the strength of the relationship. The result (phi = 0.1247) indicated a weak association strength.
Table 9

Number and percent of reviews containing a reviewer self-citation by recommendation level

Recommendation

# of reviews

# and  % with reviewer self-citation

Accept

78

4 (5 %)

Minor revision

269

41 (15 %)

Major revision

315

63 (20 %)

Reject

265

31 (12 %)

TOTAL

927

139 (15 %)

No statistically significant difference was found between male and female reviewers in terms of self-citing behavior (χ2 = 1.38; df = 1).

Discussion

The first research question (RQ1), whether there is a relationship between referencing the target journal and favorable reviews, can be answered in the affirmative. Manuscripts referencing JASIS&T were more likely to receive a revision decision than a rejection. However, even in the case of minor revisions, nearly 1 in 5 manuscripts contained no reference to JASIS&T. In addition, in no category did the total number of references to JASIS&T exceed 10 % of all references in the paper. Although there is a lack of comparative data, we speculate that the low number of journal self-citations is a function of information science’s interdisciplinarity and JASIS&T roles as a generalist journal within the field. A more specialized journal would, we suspect, receive a higher proportion of submissions referencing it.

There was no statistically significant relationship between editorial decision and whether or not a manuscript referenced the editor (RQ2). Nevertheless, it would be instructive to know whether the editor was cited more or less frequently by manuscripts submitted prior to his assuming the role. There was, however, some relationship between editorial decisions and the degree to which the manuscript referenced the reviewer (RQ3). This may be a function of scope (matching topic with reviewer), as mentioned earlier. However, this could equally well be indicative of latent conservatism (Shatz 2004), in that manuscripts that reference the core literature of the field are more likely to be accepted than those that do not.

As noted by Wilhite and Fong (2012) p. 542, “coercive self-citation does not refer to the normal citation directions, given during a peer-review process, meant to improve a paper.” Rather, it “refers to requests that (1) give no indication that the manuscript was lacking in attribution; (2) make no suggestion as to specific articles, authors, or a body of work requiring review; and (3) only guide authors to add citations for the editor’s journal” (p. 542). The results of our (admittedly modest) study show that coercive self-citation is not as “uncomfortably common” as Wilhite and Fong (2012, p. 542) seem to believe. Only 3 exhortations to cite JASIS&T appeared in the 927 reviews. The remaining references were to articles, authors, or venues not directly associated with JASIS&T. The most common type of reference was to specific articles (RQ4). This may mean that JASIS&T reviewers are less prone to gaming than the reviewers of other journals, but a more plausible explanation is that anecdotal evidence and survey data reporting perceptions of bias may overestimate the extent of citation gamesmanship.

Nearly half of all manuscripts were assigned to reviewers who were referenced in the manuscript. This did not appear to statistically significantly influence reviewer decisions. Fifteen percent of the reviews in the sample contained a reviewer self-citation (RQ5). However, this does not seem to be a case of citation mongering, as reviewers were more likely to self-cite if their work had already been cited in the manuscript (RQ6). Further, this behavior was most closely associated with major revisions, which suggests that reviewer self-citing serves a clarifying purpose, allowing reviewers to highlight inaccuracies in the representation of their work in the manuscript under review. No difference was found between male and female reviewers in regards to reviewer self-citations (RQ7), despite differences in male and female reviewing found in other studies of peer review (e.g., Wing et al. 2010; Borsuk et al. 2009; Lane and Linden 2009; Jayasinghe et al. 2003). Furthermore, this confirms previous studies that have shown no difference in male and female rates of self-citation in journal articles (e.g., Symonds et al. 2006; Borrego et al. 2010).

Conclusion and future research

Chubin and Hackett (1990) referred to studies of referee reports as a “formidable agenda that demands the attention of researchers” (p. 110). Despite the thousands of articles on peer review (Bornmann 2011), there remains scant literature on the “rhetoric of referees” or any empirical evidence that can be used “toward a theory of refereeing” (Chubin and Hackett 1990 p. 111). This paper provides a novel perspective on the peer review process by examining the relationship among the references in the manuscript, references in referee reports, and the players involved in the peer review process.

Ego does not seem to intrude on the peer review process, at least as far as JASIS&T is concerned. Referencing the editor or reviewers does not statistically significantly influence editorial decisions or reviewer recommendations. Moreover, reviewers are not using the review process as a means to boost their citation count. We did find, however, that articles were more likely to be accepted if they contained a reference to JASIS&T. It is questionable whether this constitutes bias. A parsimonious explanation is that manuscripts referencing JASIS&T are more likely to be within scope than those that do not. The same can be said for references to the editor and the reviewers. Ego bias is only constituted in cases in which references are added purely or primarily for competitive gain, rather than for scholarly reasons. Future research should seek to identify the extent to which gratuitous referencing occurs.

Our study found considerably more references embedded in referees’ reports than did Bornmann et al. (2010) in their analysis of reviews of Chemistry manuscripts. Further research should investigate possible differences that may exist across fields and disciplines. Disciplinary differences in the crafting of reviews can be viewed using a variety of lenses, including discourse, sentiment, and (self-)citation analysis. Finally, our results challenge, tentatively to be sure, the prevailing view that peer review is somehow inherently biased. Future research should seek to empirically refute or support such claims, across both academic disciplines and publication genres.

Acknowledgments

We are grateful to Craig Finlay and Brad Demarest for assistance with data collection, Thomas Sugimoto for assistance in data analysis, Vincent Larivière for comments on an earlier draft, Meghann Knowles for data extraction, and an anonymous reviewer for constructive feedback.

Footnotes
1

An F test was conducted to examine the differences in variance and found unequal variances in both cases [revisions and rejections (p = <0.001); major revision and minor revisions (p = <0.05)].

 
2

A χ2 of 3.04 with 3 degrees of freedom was found including all decision levels. A subsequent analysis was done by aggregating revisions and excluding the acceptance category from the analysis. This yielded a χ2 of 1.27 with one degree of freedom.

 

Copyright information

© Akadémiai Kiadó, Budapest, Hungary 2012