The Disaster of the Impact Factor
- First Online:
- 1.5k Downloads
Journal impact factor (IF) is a value calculated annually based on the number of times articles published in a journal are cited in two, or more, of the preceding years. At the time of its inception in 1955 (Garfield 1955), the inventor of the impact factor did not imagine that 1 day his tool would become a controversial and abusive measure, as he confessed 44 years later (Garfield 1999). The impact factor became a major detrimental factor of quality, creating huge pressures on authors, editors, stakeholders and funders. More tragically, in some countries the number of publications in journals with “high impact factors” condition the allocation of government funding for entire institutions (Plos Medicine Editorial 2006). Based on the assumption that IF reflects scientific quality, the impact factor produces a widespread impression of prestige and reputation, though no experimental data support this hypothesis (Brembs et al. 2013).
The impact factor was originally conceived as a bibliometric assessment tool for publishers and librarians to provide helpful information for subscription and library collection purposes, but over the years, it has been used abusively to assess the quality of not only journals, universities, and institutions, but also individuals and countries to promote or to “denigrate”. Its deficiencies and perverse effects are tangibly harmful that make one wonders why such a biased tool continues to exist in science. Apart from technical and business-oriented goals (Abbasi 2007, Seglen 1997), the journal impact factors suffer from two important flaws, often neglected by researchers, that need to be highlighted due to their intrinsic contradiction with fundamental scholarly ethics.
First, if an article on the subject of impact factors (based on the arithmetic mean used in the calculation of impact factor) is submitted to one of the so-called “high impact factor” journals, the submitted article would be immediately rejected. The editors and/or reviewers would argue that the statistical approach is biased due to great variability and heterogeneity between the subjects being compared. The arithmetic mean used in the calculation of the impact factor is unsuitable for such an analysis. Beyond this contradiction, the data used in the calculation of the impact factor are neither transparent nor publically available (Rossner et al. 2007), opening a large opportunity for speculations and criticisms.
Second, the impact factor violates some basic and ethical rules of scholarly citation. In scientific standards, a citation should refer to the original or primary sources in which results or observations were reported for the first time. Many people, however, disregard this principle and tend to cite review articles rather than original contributors. As a result, Review Journals (journals that publish only literature reviews) harvest the highest impact factors possible compared to primary-research journals. For example, Nature Reviews Genetics has an impact factor higher than Nature itself (eg., in 2012, IF 41 vs. IF 38, respectively) (see the Nature website below*). The same applies to Annual Reviews that have impact factors higher than the most of journals that publish primary research articles. Based on the impact factor’s conclusions, Review Journals have more “impact” than any generalist or specialist journal publishing primary research findings. If there were only this contradiction, it would be more than enough to remove the impact factor from science metrics forever; how may it be made acceptable that Review Journals have more “impact” than primary research journals? Moreover, in most cases only a few articles contribute to the acquisition of the journal impact factor, excluding any correlation between an article and the journal in which the article was published. A high percentage (~90 %) for example of the impact factor of Nature in 2004 was generated by only 25 % of the papers published in that year (Nature Editorial 2005).
These aberrations lead to other aberrations at editorial policy level. Some editors tend to be elitists and unfairly selective in their submission, acceptance or rejection policies. They try to build their editorial strategies on augmenting the impact factors by inciting authors to cite their journal or by reducing the number of items in the denominator (Plos Medicine Editorial 2006). Elitist journals tend also to restrict submission of review articles to “by invitation only”, in such a way that only senior authors with long career experiences are invited to submit “citable” papers to boost the journal impact factor. Junior authors, on the other hand, are often rejected if they are not endorsed by at least one veteran co-author.
Another disastrous effect of the impact factor fashion is the newborn universities ranking system. This system is far more aberrant than the impact factor itself. Similarly to the impact factor aberration, universities ranking systems are based on subjective and biased criteria that compare heterogeneous institutions, which are different in almost everything including staff, number of students and professors, number and rhythm of workday hours, infrastructure, specialization, and equipment, etc. Such comparison is invalid from a scientific viewpoint because it compares incomparable structures. To be valid, any scientific comparison should compare groups or items that are similar in all but only one or two variables (variables being investigated only). Within a country or between countries, institutions differ in a great number of variables that make any comparison a senseless approach.
The impact factor is also a negotiable game (Plos Medicine Editorial 2006) (Rogers 2002) with an increasing number of shortcomings. The desire to be highly evaluated may lead scientists to waste time and energy by running behind journals with a “high impact factor” in a long process of submission/rejection cycles to the detriment of focusing on their research. Researchers might also consider scientific publication as an open business market rather than a medium for the dissemination of knowledge. Consequently, the urgency to achieve a publication may push some scientists and institutions to unscrupulous and corrupt practices. Examples that illustrate these aberrant trends have been recently reported (Hvistendahl 2013) (Van Noorden 2013) (Bhattacharjee 2011). It is worth noting that the obsession with the impact factor is more pronounced in some countries than in others.
Based on a subjective popularity generated by the impact factor, another important destructive effect still cause damages. Scientists prefer to focus on already “hot research topics” that might promptly produce publications in “high impact factor journals” rather than on risky, unexplored research topics that might yield negative results or take months or years before a worthwhile achievement can be obtained. Potentially important research topics are thus abandoned or delayed.
Other side-effects of the impact factor include; (1) some people decorate their publication list with the journal impact factor showing how many times their articles have been cited, (2) celebration of the impact factor in dedicated articles to announce that “Journal X has now its impact factor YY”, (3) some publishers highlight any upgrading of the impact factor in messages sent to all their subscribers; (4) hiring, funding and promotion are mostly based on the number of publications, particularly in “high impact factor journals”; (5) high impact factor journals are more exposed to publish falsified work than lower ranked journals (Fang et al. 2012). A prompt look at job advertisements or grant attribution criteria also demonstrate a strong fixation on publication record in “high impact factor journals” as a prerequisiste feature before application. It is most likely that none, or very few at most, of such shareholders have themselves published in “high impact factor” journals. Would it be a hidden desire to compensate what some people do not have that leads them to stress the “high impact factor”, despite its weaknesses described here and elsewhere? Fair and experienced researchers, on the other hand, are less likely to emphasize such a flawed assessment criterion, because they know the hidden sides of the impact factors; for which purposes it was created and how it abusively evolved.
In an attempt to repair the deficiency of IF-based deviations and putting science on the right road, the American Society of Cell Biology, endorsed by more than 230 scientists and scientific institutions have recently initiated a Declaration on Research Assessment (http://am.ascb.org/dora/). The declaration advocates the suppression of the journal impact factor from the evaluation of individual’s scientific work, and it urges the major science actors and stakeholders to rely on scientific content rather than publication metrics.
In summary, it is unreliable practice to use a non-scientific approach to assess scientific quality. The impact factor is an unscientific approach. Its attribution breaks the basic rule of citation and scientific standards, especially in applied sciences where impact factors overvalue Reviews Journals over and above primary-research journals. The method of calculation of the impact factor is often rejected by scientists for data analysis but paradoxically maintained in the calculation of the impact factor. The distributions of the citation percentage within journals and between fields (e.g., applied science vs. humanities) are strongly skewed. Impact factor effects at individual and institutional levels are cruelly destructive and counter-productive. The impact factor is also artificially manipulable by some editorial strategies. For these reasons, and others, the impact factor should simply and purely be removed from journal assessment or science metrics. It is the scientific content and the journal integrity that should matter, not an abstract number calculated with technical defects and a violation of some basic scientific ethics. If the impact factor, or any other obsolete ranking systems, continues to be used in classifying people and institutions, the pressure and obsession for the top ranks will continue to cause damages, and elitist journals will continue to reject authors to the detriment of quality and objectivity.
The solution may reside in only the peer-review process and taking readers as the ultimate judges, as was the case for centuries, since the dawn of writing and reading. Major inventions in history (e.g., car, phone, TV, plane, train, etc.) were invented before the existence of the destructive influences of the impact factor.
Conflict of interest
- Brembs, B., Button, K., & Munafo, M. (2013). Deep impact: Unintended consequences of journal rank. Frontier Human Neuroscience, 7, 291.Google Scholar
- Editorial, Nature. (2005). Not-so-deep impact. Nature, 435(7045), 1003–1004.Google Scholar
- Garfield, E. (1999). Journal impact factor: A brief review. CMAJ, 161(8), 979–980.Google Scholar
- * Nature website showing the impact factors of NPG journals: http://www.nature.com/npg_/company_info/impact_factors.html. Accessed 07 January 2014.
- San Francisco Declaration of Research Assesment (DORA). http://am.ascb.org/dora/. Accessed 07 January 2014.