Skip to main content

Gaming the impact factor: where who cites what, whom and when

The 2016 impact factor for a journal “J” is published in mid-2017. It is calculated by dividing the number of Cites by the number of Articles. “Cites” is the total number of citations from documents that were published in 2016, to articles published in “J” in the two preceding years (2014 and 2015). “Articles” is the combined total number of citable articles that journal “J” has published in the years 2014 and 2015. The number of Cites (the numerator) depends on how hard you look for citations, in which places you are looking and who is doing the looking. I will return to this later. The number of Articles (the denominator) depends on what is classified as being a citable article. For example a book review probably would not count as an article—but I don’t know that for certain. Thomson Reuters were the first to introduce this algorithm, but others have copied it. It has been used extensively as a blunt measure of a journal’s quality.

I will not discuss the shortcomings of the Thomson Reuters algorithm here, as its limitations and provisos for use are explained in an excellent and exhaustive exposition by the New World Encyclopedia [1].

Rather we ask whether it is legitimate to game the impact factor? If the impact factor was a valid, fair statistic that was produced by the scientific community to be fit for purpose and was supported by academia and backed by the force of the legislative codes of conduct of scientific and professional societies, the answer would be no. If it is a simplistic index produced by a commercial enterprise without consultation with the affected communities of scholars, is not supported by the scientific community and is being used by people outside of those communities for purposes for which it is not intended, then yes. Let me ask another question. Are institutions/people who use the impact factor inappropriately for purposes for which it was not intended, also gaming it? Should such behaviour be subject to sanction?

Gaming the impact factor is not like plagiarising some text and including it in an article that you submit for publication, nor like an athlete taking performance enhancing drugs. Such behaviours are wrong, illegal and exemplify misconduct. Rather it may be likened to “optimising” your website so that it achieves a higher ranking on Google’s search engine results. Or to paying to Boost your post on “facebook” and so make it appear higher in your News Feed and on Instagram. It’s a victimless crime, we all expect it and accept it. We are mature enough to be aware that it does happen yet ignore it anyway. I know that you know that there are many shortcomings to the impact factor but we seem to tacitly agree to forgive them. There is no watchdog to oversee the legitimate operation of the impact factor (but Thomson Reuters does monitor citation behaviour) or indeed to sanction those who use it for unintended purposes. Hence it is possible to arrange the arithmetic to optimise the outcome for a particular journal. That is, the impact factor rises if the “numerator” can be increased while minimising the “denominator”.

Ways to maximise a journal’s impact factor

A journal may publish an overview of their recently published articles while citing lots of them in the overview. This will increase the numerator by the number of articles cited while increasing the denominator by one. In 2007, in a reaction against the administrative staff of some universities who were discouraging their employed scientists from publication in anything but “top ranking” journals, the journal Folia Phoniatrica et Logopaedica, did this quite overtly. They cited 67 articles from 2005 to 2006. The authors stated: “While the primary goal of this article is to increase the impact factor of the journal, it also provides potentially useful information on the distribution of the articles published in the journal” [2]. The article had the desired effect, was legitimate in my view and Thomson Reuters reacted by excluding the journal from their 2008 Journal Citation Report (it has since been restored). However, the journal’s appearance in the SCImago “Citations per document (2 years)” using Scopus citation data, and in ResearchGate’s “Journal Impact” score calculated using ResearchGate data, continued uninterrupted.

In 1997 the BMJ reported on a letter from the journal Leukemia to the authors of a submitted manuscript, as a blatant attempt to increase the journal’s impact factor. The letter stated: “We have noticed that you cite Leukemia [once in 42 references]. Consequently, we kindly ask you to add references of articles published in Leukemia to your present article” [3]. It was a blatant attempt, but as long as no inappropriate references were included, what damage was done?

A journal may include a direction in their guidelines to authors. For example, the Journal of Applied Clinical Medical Physics once included this in their Submission Preparation Checklist:

“8. IMPORTANT—Perform a Google Scholar search (http://scholar.google.com/) on the keywords of your article as well as key terms from your title and abstract followed by “JACMP”. This should locate all JACMP articles that should be cited by your article in order to maintain the continuity of the investigational narrative within the JACMP community. Please be sure to cite all relevant JACMP articles for your submission”.

This (appropriately) has since been removed from the journal’s website. But just to confirm that it was there, you can find it referred to by Richardson on page 8/24 [4].

It is not in the interests of the publishers of a highly ranked journal to consider for publication manuscripts which they suspect will not be cited. Consequently a journal may direct manuscripts adjudged to be unlikely to attract citations, towards a sister journal. For example: “Articles that have been judged, following peer-review, to be editorially unsuitable for publication in another IOP journal may, at the discretion of the journal team and their Editorial Boards, be deemed appropriate for further consideration by BPEX” [5]. In this way it is possible for a journal with a high end “numerator” to defend it against being divided by a “denominator” bloated by those manuscripts that are not adjudged to be among those likely to be highly cited. It is even better if the redirected manuscript actually cites articles from the journal it was initially submitted to.

A journal publisher may choose to calculate their own “impact factor” using the citation data from SCImago or from Google Scholar. Scientific Research Publishing Inc. does this and they state: “SCIRP is calculating its own impact factor based on Google Scholar’s citation counts. Scientists are used to Thomson Reuters’ way of calculating an impact factor. For this reason, SCIRP applies Thomson Reuters‘(TR) algorithm….…”. They call their metric the “2-year Google-based Journal Impact Factor (2-GJIF)” [6]. Google Scholar is able to discover more citations than either the Web of Science or Scopus [7]. Hence the 2-GJIF will be higher than the corresponding metric produced using Web of Science or Scopus data. However Google Scholar citation counts are unreliable as the example of a chapter in a book that I authored and which was published in 2016 illustrates. At the time I looked it had attracted 17 citations according to Google, but on inspection, all 17 “citations” were spurious, being attributed to documents that had been published prior to 2013 (Table 1).

Table 1 One of the “hits” that resulted from “googling”: M Caon Examination Questions and Answers in Basic Anatomy. The result has since changed

Which impact factor is that?

It was stated earlier that the outcome of the “Cites divided by Articles” algorithm depends on how hard you look for citations (how many journals are mined, which conference proceedings are used), in which places you are looking (in journals, conference proceedings, monographs, media, websites) and who is doing the looking (Google Scholar, Scopus, or Web of Science—each of who may have different definitions for a “citable article”). Consequently the Cites/Articles 2 year impact factor is different if it is calculated by Google, by SCImago or by Thomson Reuters. Furthermore there are now some additional players in the field.

ResearchGate publishes a “Journal Impact” value calculated using ResearchGate data and which is based on average citation counts from work published in the journal in question. They caution that the data used in the calculation may not be exhaustive. Nevertheless, Researchgate’s Journal Impact value is typically higher than the values produced by SCImago or Thomson Reuters. Crossref has a Cited-by Linking service which counts citations and is built on top of the DOI (digital object identifier) infrastructure (see for example the APESM website for the citation counts for articles published in this journal). Crossref states that “It is not tied to any particular “metric”, although the metadata provided by the service could be used by others ……. as the foundation for creating new metrics” [8]. In other words a Crossref 2 year citation score could be calculated. As data collected via the DOI is so reliable, I predict that this will happen in the near future.

The University of Washington publishes the metrics “Eigenfactor” and “Article Influence” based on data held in Thomson Reuters’ Journal Citation Reports (see eigenfactor.org). The Eigenfactor excludes self-citations from journal “J”. The Elsevier publishing group describes the Article Influence metric as follows: “Article Influence is calculated by dividing the Eigenfactor by the percentage of all articles recorded in the Journal Citation Reports that were published in J. Article Influence is therefore conceptually similar to the Impact Factor and SCImago Journal Rank” [9].

Then there is the “impact per publication” which is the same as the impact factor but it counts citations and articles going back for the three previous years rather than two. Extending this concept, the “5 year impact factor” includes Cites and Articles from the five previous years. Strangely citations by articles published in 2016 to other articles published in 2016 don’t count for 2, 3 or 5 year impact factors.

Given that the impact factor metric is somewhat like the sound of one hand clapping, it is legitimate and defensible to use the citation data source that produces the highest “impact factor” when spruiking a journal. This is particularly so given that now it is being used by some universities for disparate and inappropriate purposes.

The impact factor is being used in attempts to alter publication behaviour

We expect specialists and experts in their field to recognise good work wherever it is published and to read it. We do not expect the worth of an individual article to be discounted because it has been published in a journal with a statistic that is lower than that of another journal. Discounting an article because of where it is published (without reading it) is not a rational way to assess the quality of a scientific work. But this is just what is happening. Administrators are using the impact factor to alter the publication behaviour of scholars who publish their research. For example the Queensland University of Technology website states: “QUT encourages researchers to publish in Q1 and Q2 journals and conferences as listed in this database” [10] (Q1 refers to journals adjudged to be within the first quartile of journals in their field using the Scimago journal rankings). Furthermore QUT has a policy that PhD-by-publication will only count publications in Q1 or Q2 journals. The University of Southern Queensland has a “quartile 1 challenge”. A monetary reward is granted for publishing in Q1 journals. Specifically: “$10,000 for organisational units raising the proportion of Q1 publications by 10 percentage points (e.g., from 30 to 40%) over the previous year”. And Aus $20,000 for a 20% point rise [11].

The push back

The Excellence in Research for Australia (ERA) exercise conducted by the Australian Research Council for 2010 attempted to rank journals for quality, but have since rejected the procedure and for the right reasons too:

“Ranked journal publishing profiles were used as part of the suite of indicators in the ERA 2010 evaluation. Following feedback from Research Evaluation Committees that they relied on their own expert knowledge of the quality of research outlets relevant to their discipline, ranked journal profiles were removed as an indicator for the ERA 2012 evaluation. The ranked journal list is no longer available from the ARC website. This is because it was intended solely for the purposes of the ERA 2010 evaluation, and because journals may have changed significantly in the number of years since the rankings were developed”.

The Australian NHMRC will no longer request Journal Impact Factors as part of any applications for funding nor use these in peer review of individual applications [12]. The American Society for Microbiology has announced that in order “to avoid contributing further to the inappropriate focus on journal IFs” they will no longer publish impact factor information on their journal websites [13].

You may be forgiven for being confused by the increasing number of impact factors and other publication metrics and for feeling uneasy that they are flexible enough to be open to skilful optimisation. If this is the case for you, then don’t use them. Instead read the articles in the journal to determine their merit and to form an opinion of the editorial quality of the journal.

References

  1. 1.

    New World Encyclopedia (2014) Impact factor http://www.newworldencyclopedia.org/entry/Impact_factor. Accessed on 2 Feb 2017

  2. 2.

    Schuttea HK, Svec JG (2007) Reaction of Folia Phoniatrica et Logopaedica on the current trend of Impact Factor measures. Folia Phoniatrica et Logopaedica 59:281–285 https://www.karger.com/Article/Pdf/108334

  3. 3.

    Smith R (1997) Journal accused of manipulating impact factor. BMJ 314:461 http://www.bmj.com/content/314/7079/461.5

  4. 4.

    Richardson S, Inside the mind of a JACMP reviewer. http://amos3.aapm.org/abstracts/pdf/99-30800-366478-110629-1690800916.pdf (viewed 2 Feb 2017)

  5. 5.

    BPEX IOP Science http://iopscience.iop.org/journal/2057-1976/page/Scope;jsessionid=81BFD674D86471F82AAF485BC225DDF9.ip-10-40-1-87

  6. 6.

    Scientific Research Publishing http://www.scirp.org/journal/Journalcitationdetails.aspx?JournalID=1573#.WDPEQiuRbAA

  7. 7.

    Trapp J (2016) Web of Science, Scopus, and Google Scholar citation rates: a case study of medical physics and biomedical engineering: what gets cited and what doesn’t? Australas Phys Eng Sci Med 39(4):817–824

    Article  PubMed  Google Scholar 

  8. 8.

    Lammey R (2014) Practice of CrossRef extensible markup language coding and more advanced information for CrossRef deposits. Sci Editing 1(2):91–98. http://www.escienceediting.org/upload/se-1-2-91.pdf

  9. 9.

    https://www.elsevier.com/authors/journal-authors/measuring-a-journals-impact. Accessed on 23 Nov 2016

  10. 10.

    QUT http://libguides.library.qut.edu.au/c.php?g=428573&p=2923659. Accessed on 24 Feb 2017)

  11. 11.

    USQ https://www.usq.edu.au/research/support-development/research-training/initiatives/publications-excellence-awards/quartile-1-challenge-q1c. Accessed on 24 Feb 2017

  12. 12.

    NHMRC (2010) NHMRC removes journal impact factors from peer review of individual research grant and fellowship applications. https://www.nhmrc.gov.au/_files_nhmrc/file/grants/peer/impact%20factors%20in%20peer%20review.pdf. Accessed on 2 Feb 2017

  13. 13.

    Casadevall A et al (2016) Editorial: ASM journals eliminate Impact Factor information from journal websites. J Clin Microbiol 54(9):2216–2217. doi:10.1128/JCM.01418-16

    Article  PubMed  PubMed Central  Google Scholar 

Download references

Author information

Affiliations

Authors

Corresponding author

Correspondence to Martin Caon.

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Caon, M. Gaming the impact factor: where who cites what, whom and when. Australas Phys Eng Sci Med 40, 273–276 (2017). https://doi.org/10.1007/s13246-017-0547-1

Download citation