Post retraction citations in context: a case study
This study examines the nature of citations to articles that were retracted in 2014. Out of 987 retracted articles found in ScienceDirect, an Elsevier full text database, we selected all articles that received more than 10 citations between January 2015 and March 2016. Since the retraction year was known for only about 83% of the retracted articles, we chose to concentrate on recent citations, that for certain appeared after the cited paper was retracted. Overall, we analyzed 238 citing documents and identified the context of each citation as positive, negative or neutral. Our results show that the vast majority of citations to retracted articles are positive despite of the clear retraction notice on the publisher’s platform and regardless of the reason for retraction. Positive citations can be also seen to articles that were retracted due to ethical misconduct, data fabrication and false reports. In light of these results, we listed some recommendations for publishers that could potentially minimize the referral to retracted studies as valid.
KeywordsRetracted articles Post retraction citations Positive citations Negative citations Neutral citation
Recent studies on retracted articles show that the number of retracted articles is increasing in relative measure to the overall growth in scientific publications (Cokol et al. 2008; Marcus and Oransky 2014). The major reasons for articles to be retracted are: misconduct and error (Fang et al. 2012; Steen 2011a). Peer review is supposed to guard from publishing fraudulent results, however sometimes mistakes or unethical conduct (plagiarism) cannot be identified during the review process. Thus, when misconduct or unethical behavior are noticed, sometimes by the community, the article is retracted at the request of the editor, the author, the employer or the publisher.
Although the act of retracting flawed articles helps purge the scientific literature of erroneous or unethical research, citations to such research after its been retracted, presents a real challenge to the integrity of the scientific endeavor. Continued citations, or post-retraction citations, of articles that were withdrawn especially due to plagiarism, data falsification or any other unethical practices interferes with the process of eliminating such studies from the literature and research overall.
Essentially, there are two major types of post-publication citations of retracted papers; citations that an article received prior to its retraction and the citations that it received post retraction and despite retraction notices (Unger and Couzin 2006; Campanario 2000). Both types of citations put the scientific process in jeopardy, especially when they are cited as legitimate references to previous work and the reason for retraction was manipulation and fraud. Some studies have shown that retracted articles that received a high number of citations pre-retraction are more likely to receive additional citations post-retraction (Campanario 2000; Redman et al. 2008). One of the early studies on post retraction citations (Kochan and Budd 1992) examined post retraction citations to papers of John Darsee, and showed that over 85% of the post retraction citations are positive, not mentioning fraud or retraction. Other early studies include works by Pfeifer and Snodgrass (1990) and by Garfield and Welljams-Dorof (1990).
A more recent example is described in a study by Bornemann-Cimenti et al. (2015) who studied the case of Scott S. Reuben who was convicted of fabricating data in 25 of his studies which resulted in mass retractions of his articles. The authors of the study have shown that the popularity of Reuben’s articles did not diminish post-retraction even five years after the retractions. Another phenomenon identified in the literature is of authors’ self-citing their retracted articles and thus contributing to the perception that their retracted work is valid (Madlock-Brown and Eichmann 2015).
Other studies on retraction concentrated on the reasons for retraction. Fang et al. (2012) studied a large set of more than 2000 retracted articles indexed by PubMed and found that more than 67% of the retractions are due to misconduct, including fraud and suspected fraud. Steen (2011a) also studied a subset of biomedical, retracted articles retrieved from PubMed, and contrary to Fang et al. (2012), he found that error was the most common reason for retraction. Another study by Wager and Williams (2011) was also based on biomedical retracted articles, and like Steen (2011a) found that error was most prevalent. Temporal aspects were also studied, for example by Fanelli (2013) and Steen et al. (2013). A review article on scientific misconduct was recently published (Gross 2016).
The continued positive citations of retracted articles are a serious issue that warrants a closer examination. As can be seen, most of the previous studies concentrated on biomedical research. Our approach was different, as we retrieved in October 2014 retracted papers from a major scientific publisher, Elsevier, thus our sample includes papers from all areas of science and social science. We selected 15 retracted articles, according to the following criteria: retracted between 1995 and 2014 that received the highest number of citations between January 2015 and March 2016 (called recent citations in this paper) that occurred definitely post retraction, even when the retraction date of the article is not specified. By conducting a context analysis of each of the citations they received, we sought out to find whether they are negatively, positively or neutrally mentioned.
ScienceDirect, Elsevier’s full text database was accessed in October 2014. The database was queried for the term “RETRACTED” in the article title and its retraction notice. In ScienceDirect, each retracted article is preceded with the word “RETRACTED”. In addition, each Elsevier journal incorporates a retraction notice which explains who retracted article and the reason for retraction. This allowed us to manually code each article in our dataset with an additional field “retracted by” that represented the person/s requesting the retraction. An alternative search strategy is to search by Document type: Erratum for retract*, which retrieves retraction notices, however less results were retrieved, because not all retractions are accompanied by separate retraction notices. A sample of the results from the alternative strategy was checked against the results of the search strategy applied in this study: all articles in the sample were found in the data set created by our search strategy.
A total of 1203 results retrieved from which 987 were retracted articles. The results excluded were retraction notices, duplicates and papers whose original titles included the word “retracted”.
For the current paper we chose all retracted articles that were cited more than ten times between January 2015 and March 2016. Fifteen such articles were identified. These 15 articles received altogether 267 citations between January 2015 and March 2016. We were unable to access 29 citing papers (mainly book chapters, or articles in Chinese), thus the analysis relies on 238 citing documents.
Positive A positive citation indicates that the retracted article was cited as legitimate prior work and its findings used to corroborate the author/s current study.
Negative A negative citations indicates that the authors mentioned the retracted article as such and its findings inappropriate.
Neutral A neutral citation indicates that the retracted article was mentioned as a publication that appears in the literature and does not include judgement on its validity.
Case study 1
Donmez, G., Wang, D., Cohen, D. E., and Guarente, L. (2010). RETRACTED: SIRT1 Suppresses β-Amyloid Production by Activating the α-Secretase Gene ADAM10. Cell, 142(2), 320–332.
Séralini, G. E., Clair, E., Mesnage, R., Gress, S., Defarge, N., Malatesta, M.,… and De Vendômois, J. S. (2012). RETRACTED: Long term toxicity of a Roundup herbicide and a Roundup-tolerant genetically modified maize. Food and Chemical Toxicology, 50(11), 4221–4231.
This article, published in 2012 was the subject of a debate surrounding the validity of the findings, use of animals and even accusations of fraud. Its publication and retraction process have resulted in the “Séralini affair” which became a big media news item (Séralini affair 2016). The article described a 2-year study of rats which were fed genetically modified (GM) crops and showed increased tumors. The study, which was also scrutinized by government agencies, received major media attention that resulted in the creation of a social movement against GM food. Despite the accusation of fraud and fabrication of results, the editors found no such evidence to that effect. However, the article was retracted in 2013 because of the “low number of animals” used in this study which lead to the conclusion that “no definitive conclusions can be reached with this small sample size”.
The study was republished in 2014 by Environmental Sciences Europe. The republication of the study stirred another controversial discussion in the scientific community with several scientists writing letters expressing their concerns regarding the appearance of the same study in another journal. For further details, consult Oransky (2013).
The republished article received 17 citations in 2015 and 2016. The vast majority of them being positive mentions (87%), however some criticism towards the peer-review practices of the retracting editors were also detected (Loening, 2015). The one negative mention of the re-published article was criticism towards the media frenzy around the topic and the inability of the scientific community to refute invalid results. The authors state that “Although scientists have investigated each GMO crisis and reached scientific and rational conclusions, they have less ability to disseminate information than the media, so the public is not promptly informed of their rational and objective viewpoints as experts (Xia et al. 2015).
Mukherjee, S., Lekli, I., Gurusamy, N., Bertelli, A. A., and Das, D. K. (2009). RETRACTED: Expression of the longevity proteins by both red and white wines and their cardioprotective components, resveratrol, tyrosol, and hydroxytyrosol. Free Radical Biology and Medicine, 46(5), 573–578.
For this paper we were able to access 17 out of the 21 recent citations. All of these quoted the article’s findings as legitimate. For example, “Plants containing resveratrol, a potent antioxidant, has been used widely in the treatment of various ailments” (Pangeni et al. 2014) or “Recent studies have also shown that red wine upregulates the protein expression of sirtuin” (Romain et al. 2014).
Walumbwa, F. O., Wang, P., Wang, H., Schaubroeck, J., and Avolio, B. J. (2010). RETRACTED: Psychological processes linking authentic leadership to follower behaviors. Leadership Quarterly, 21(5), 901–914.
A close examination of the post retraction citations (2015–2016) shows that all 23 citations were positive citations (we unable to access one citing document), meaning that the citing authors used findings from this article to support their findings. The subject of “authentic leadership” is popular in management studies and has seen a surge in publications since 2012. This could explain the overall positive citations of the article.
Li, C., Tao, X. M., and Choy, C. L. (1999). RETRACTED: On the microstructure of three-dimensional braided performs. Composites Science and Technology, 59(3), 391–404.
Hwang, Eun-Sun, and Gun-Hee Kim. (2007). RETRACTED: Biomarkers for oxidative stress status of DNA, lipids, and proteins in vitro and in vivo cancer research. Toxicology 229 (1–2), 1–10.
It should be noted that the plagiarized article (Mayne, 2003) was cited 304 times, out of which 24 occurred in 2015–2016. None of the 14 recent citing papers of the retracted article cited the article authored by Mayne.
Qiang, L., Fujita, R., Yamashita, T., Angulo, S., Rhinn, H., Rhee, D., and Abeliovich, A. (2011). RETRACTED: Directed conversion of Alzheimer’s disease patient skin fibroblasts into functional neurons. Cell, 146(3), 359–371.
Ji, Z. X., Sun, Q. S., and Xia, D. S. (2011). RETRACTED: A framework with modified fast FCM for brain MR images segmentation. Pattern Recognition, 44(5), 999–1013.
Zhao, R., Zhang, Z., Song, Y., Wang, D., Qi, J., and Wen, S. (2011). RETRACTED: Implication of phosphatidylinositol-3 kinase/Akt/glycogen synthase kinase-3β pathway in ginsenoside Rb1’s attenuation of beta-amyloid-induced neurotoxicity and tau phosphorylation. Journal of Ethnopharmacology, 133(3), 1109–1116.
In addition, all the recent citations accessible to us (10 papers out of the 11 recent citing publications) were positive, citing the findings of the retracted paper as valid. These citations appear after a retraction notice has been issued and the reasons made known to the scientific community.
Nanjawade, B. K., Manvi, F. V., and Manjappa, A. S. (2007). RETRACTED: In situ-forming hydrogels for sustained ophthalmic drug delivery. Journal of Controlled Release, 122(2), 119–134.
Yamagata, K., Fujiyama, S., Ito, S., Ueda, T., Murata, T., Naitou, M., and Kato, S. (2009). RETRACTED: Maturation of MicroRNA is hormonally regulated by a nuclear receptor. Molecular Cell, 36(2), 340–347.
Vaidyanathan, R., Kalishwaralal, K., Gopalram, S., and Gurunathan, S. (2009). RETRACTED: Nanosilver—The burgeoning therapeutic molecule and its green synthesis. Biotechnology Advances, 27(6), 924–937.
Liu, X., Liu, H., Wang, S., Zhang, L., and Cheng, H. (2006). RETRACTED: Preparation and thermal properties of form stable paraffin phase change material encapsulation. Energy Conversion and Management, 47(15), 2515–2522.
Nabae, Y., Moriya, S., Matsubayashi, K., Lyth, S. M., Malon, M., Wu, L.,… and Miyata, S. (2010). RETRACTED: The role of Fe species in the pyrolysis of Fe phthalocyanine and phenolic resin for preparation of carbon-based cathode catalysts. Carbon, 48(9), 2613–2624.
Liu, X. F. (2014). RETRACTED: Substitution reactions of diiron dithiolate complexes with phosphine or isocyanide ligands. Journal of Organometallic Chemistry, 750, 117–124.
This article was retracted in 2014, due to plagiarism which not only pertained to the text but also to the methodology presented. The editor in chief states in the retraction notice that the “some of the work reported as new in this paper, was previously conducted by someone else… the…method used and the proposed mechanism… are similar to those previously reported [previously by someone else] and …portions of the manuscript are worded identically to those in manuscripts that have been published” (retraction notice). As can be seen from the statement, this article presents deep and compound case of plagiarism that ranges from text to methods. Yet it is still positively cited, with 12 citations in 2015: 9 positive and 3 neutral. Since the article was retracted in the same year as it was published, it is difficult to decide what portion of the ten citations received in 2014 were pre-retraction citations. It should be noted that 8 of the 12 recent citations are from a single author, Wei Gao.
Discussion and conclusions
As can be seen from the examples above, retracted articles continue to be cited years after retraction and despite retraction notices being posted on publishers’ platforms. There are different reasons for retraction some articles were retracted for ethical reasons (plagiarism, self-plagiarism or publishing multiple versions of the same paper)—8 out of the 15 studied retractions belong to this category. Here the problem is not with the validity of the findings, but in addition to the ethical issue, the authors of the plagiarized papers are deprived of citations that go to plagiarizing paper.
More serious are the cases where the data or the images were manipulated. This happened in 8 out of the 15 cases studied (Case #9: the article was both self-plagiarized and some of the images were manipulated). Manipulated data lead to unreliable conclusions, which might have far reaching implications (e.g. whether genetically modified corn causes cancer) especially when these articles are continued to be cited after retraction, without stating that the article has been retracted.
The full text of retracted articles is freely available to all
Public and/or Media Attention
In some cases, post-retraction citations could be the result of public and/or media attention. For example, the Séralini (Case #2), article evoked a public debate regarding the safety of genetically modified (GM) foods. This debate continued over media channels well after the article was retracted resulted in a call to enforce labelling all GM food items. This type of public attention could explain the continuing interest in the study despite some of the methodological problems found by the editor. In addition, the article was republished and thus continues to be cited despite the fact that the authors did not modify the original version. It seems that public or media attention can cause a rise in the number of negative citations (Fagan et al. 2015; Nau 2013).
In the case of the Mukherjee (Case #3), article, again, public and media interest could explain its continuing citations. Resveratrol was hailed by the media as an important supplement that could ensure longevity and good health. Today, Resveratrol is offered as an off the counter supplement available in vitamin shops. This is an indication that the study’s results were accepted despite of the inconclusiveness of the results and the problematic study design.
Data and image manipulations are ignored
Retraction due to self-plagiarism or duplicate publication
Post-retraction citations of articles that were flagged for self-plagiarism are also common. Liu et al. (Case #13) were accused of re-using their own data and large sections of articles they published before. These practices violate the principle of originality in science whereas each published work must be original and not published anywhere else. However, neither the data nor the findings were challenged by the editors which make the study valid despite of it being a duplicate of previously published articles.
Nonetheless, our analysis shows that there are many instances where post-retraction citations are seen to articles that were retracted due to methodological flaws, data fabrications and other reasons that make the articles and their findings invalid. This phenomenon is the most concerning. When such articles are referred to and their results are listed in the text as valid step stones in science and discovery, the integrity and advancement of the scientific endeavor is jeopardized.
In this study, we not only looked at the citation distribution pre and post-retraction, but examined all recent citations that were definitely post-retraction, i.e. were published after we collected the retracted articles from the publisher. Only a few studies examined post-retraction citations and their sentiment (positive, neutral or negative) and these studies were conducted mainly in the medical field (e.g., Garfield and Welljams-Dorof 1990; Kochan and Budd 1992; Budd et al. 1999; Steen 2011b).
Publishers should conduct thorough reference checks to detect citations of retracted articles and remove them. If an article lists or refers to a retracted publication, a clear notice of retraction should be listed in the reference list and the reference text as well. Editors should question why authors cite retracted publications and unless the editor and the peer reviewers are convinced that the citation is essential, references to retracted articles should be removed.
The current practice of stamped retracted articles freely available should be reconsidered. The full text of the retracted article should not be freely available on platforms such as ScienceDirect or others. Although versions of these articles may appear elsewhere, the journal websites should not carry these versions and make it difficult for authors to download, read and consequently cite retracted articles. It is rather puzzling why retracted articles are freely available, while the huge majority of the commercial publishers’ articles are behind a paywall.
Publishers should closely collaborate with content aggregators and create a workflow where each retraction notice can be seen on all platforms. There were quite a few instances where we observed a retraction notice on the publisher platform with no parallel notice in content aggregators such as PubMed. In these cases, researchers who use only PubMed for example might think that the article that they are referring to is valid.
Although COPE (Committee on Publication Ethics) provides guidelines for editors on how to handle retractions (COPE 2015; Wager et al. 2009), there are no guidelines on what the editors and the publishers do when they notice references to retracted articles. COPE has more than 10,000 members (COPE 2016), and we believe it should provide guidelines also for handling post-retraction citations. Such guidelines might include recommendations such as clearly tagging retracted articles in the reference list or asking to remove such references altogether or ask the authors for clear explanation why the retracted paper is referenced.
In order to ease the identification of references to retracted articles during the peer review process, a database of retracted articles including the reasons for the retraction should be set up. As one of the reviewers of the article pointed out there are tools in some editorial systems that flag notices linked to the cited article on PubMed.
We conducted a case study based on 15 retracted articles. This is obviously not enough; further larger scale studies are needed to support the current findings. Most previous studies on retractions concentrated on the biomedical field and drew their data from PubMed. Here, we looked at retracted articles from all Elsevier journals, in our sample, 6 out of the 15 articles were not indexed by PubMed. Additional aspects should be explored, e.g. retraction notices that appear only on publishers’ platforms but not on content aggregation platforms as opposed to those who appear in both. This could be a factor in the amount of citations these articles receive. In this study, we noticed instances where articles were flagged as retracted on the publisher’s platform but not on content aggregators’ platform. This could contribute to the post-citations phenomenon as authors are not aware of the retraction notice because they used a database that was not updated. A close study into this could assist with creating some clear guidelines for publishers and content aggregators to streamline the process of flagging and removing flawed studies. Another issue to be examined is the comparison of the post retraction citation rates of retracted articles with the citation rates of articles in the same journal issue that were not retracted along the lines of the previous studies (Furman et al. 2012; Neale et al. 2010).
This article is based upon work from COST Action TD1306 “New Frontiers of Peer Review”, supported by COST (European Cooperation in Science and Technology). The first author was also supported by EU COST Action KnowEscape (TD1210). This paper considerably extends the work presented at the BIRNDL2016 workshop (Halevi and Bar-Ilan 2016).
- Budd, J. M., Sievert, M., Schultz, T. R., & Scoville, C. (1999). Effects of article retraction on citation and practice in medicine. Bulletin of the Medical Library Association, 87(4), 437.Google Scholar
- COPE—Committee on Publication Ethics. (2015). COPE flowcharts. http://publicationethics.org/files/Full%20set%20of%20flowcharts_0.pdf.
- COPE—Committee on Publication Ethics. (2016). About COPE. http://publicationethics.org/about.
- Edú-Valsania, S., Moriano, J. A., & Molero, F. (2016). Authentic leadership and employee knowledge sharing behavior: Mediation of the innovation climate and workgroup identification. Leadership and Organization Development Journal, 37(4), 487–506. doi: 10.1108/LODJ-08-2014-0149.CrossRefGoogle Scholar
- Halevi, G., & Bar-Ilan, J. (2016). Post retraction citations in context. In: Proceedings of the joint workshop on bibliometric-enhanced information retrieval and natural language processing for digital libraries (BIRNDL2016) (pp. 23–29). http://ceur-ws.org/Vol-1610/paper3.pdf.
- Li, C., Gatenby, C., Wang, L, & Gore, J. (2009). A robust parametric method for bias field estimation and segmentation of MR images. In: Proceedings of IEEE conference on computer vision and pattern recognition (pp. 218–223).Google Scholar
- Mayne, S. T. (2003). Antioxidant nutrients and chronic disease: Use of biomarkers of exposure and oxidative stress status in epidemiologic research. Journal of Nutrition, 133(3 SUPPL), 933S–940S.Google Scholar
- Nau, J. Y. (2013). Cancers of GMO: What remains today of the Seralini affair? Revue Médicale Suisse, 9(408), 2260–2261.Google Scholar
- Oransky, I. (2013). Controversial Seralini GMO-rats paper to be retracted. Retraction Watch. http://retractionwatch.com/2013/11/28/controversial-seralini-gmo-rats-paper-to-be-retracted/.
- Oransky, I. (2014a). Cell retraction of Alzheimer’s study is second for Tufts neuroscientist. Retraction Watch. http://retractionwatch.com/2014/08/13/cell-retraction-of-alzheimers-study-is-second-for-tufts-neuroscientist/.
- Oransky, I (2014b). Late resveratrol researcher Dipak Das has up to 20 retractions. Retraction Watch. http://retractionwatch.com/2014/03/27/late-resveratrol-researcher-dipak-das-up-to-20-retractions/#more-19437.
- Oransky, I. (2014c). Shigeaki Kato up to 25 retractions. Retraction Watch. http://retractionwatch.com/2014/05/12/shigeaki-kato-up-to-25-retractions/.
- Palus, S. (2016). Retraction count for Shigeaki Kato climbs to 39. Retraction Watch. http://retractionwatch.com/2016/04/25/retraction-count-for-shigeaki-kato-climbs-to-39/#more-38928.
- PubPeer (2013). Unregistered submission. https://pubpeer.com/publications/20655472.
- Retraction Watch. (2013). “False data” forces retraction of Carbon paper co-authored by postdoc who led to PI’s suspension. Retraction Watch. http://retractionwatch.com/2013/02/06/false-data-forces-retraction-of-carbon-paper-co-authored-by-postdoc-who-led-to-pis-suspension/.
- Retraction Watch (2014). Leadership journal to retract five papers from FIU scholar. Retraction Watch. http://retractionwatch.com/2014/02/07/leadership-journal-to-retract-five-papers-from-fiu-scholar/.
- Romain, C., Bresciani, L., Gaillet, S., Feillet-Coudray, C., Calani, L., & Rouanet, J.-M. (2014). Moderate chronic administration of Vineatrol-enriched red wines improves metabolic, oxidative, and inflammatory markers in hamsters fed a high-fat diet. Molecular Nutrition & Food Research, 58(5), 1212–1215.CrossRefGoogle Scholar
- Séralini affair. (2016). In Wikipedia, the free encyclopedia. https://en.wikipedia.org/w/index.php?title=S%C3%A9ralini_affair&oldid=754544943.
- Wager, E., Barbour, V., Yentis, S., & Kleinart, S. (2009). Retractions: Guidance from the committee on publication ethics (COPE). http://publicationethics.org/files/u661/Retractions_COPE_gline_final_3_Sept_09__2_.pdf.
Open AccessThis article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made.