This paper presents a case study of long-term post-retraction citation to falsified clinical trial data (Matsuyama et al. in Chest 128(6):3817–3827, 2005. https://doi.org/10.1378/chest.128.6.3817), demonstrating problems with how the current digital library environment communicates retraction status. Eleven years after its retraction, the paper continues to be cited positively and uncritically to support a medical nutrition intervention, without mention of its 2008 retraction for falsifying data. To date no high quality clinical trials reporting on the efficacy of omega-3 fatty acids on reducing inflammatory markers have been published. Our paper uses network analysis, citation context analysis, and retraction status visibility analysis to illustrate the potential for extended propagation of misinformation over a citation network, updating and extending a case study of the first 6 years of post-retraction citation (Fulton et al. in Publications 3(1):7–26, 2015. https://doi.org/10.3390/publications3010017). The current study covers 148 direct citations from 2006 through 2019 and their 2542 second-generation citations and assesses retraction status visibility of the case study paper and its retraction notice on 12 digital platforms as of 2020. The retraction is not mentioned in 96% (107/112) of direct post-retraction citations for which we were able to conduct citation context analysis. Over 41% (44/107) of direct post-retraction citations that do not mention the retraction describe the case study paper in detail, giving a risk of diffusing misinformation from the case paper. We analyze 152 second-generation citations to the most recent 35 direct citations (2010–2019) that do not mention the retraction but do mention methods or results of the case paper, finding 23 possible diffusions of misinformation from these non-direct citations to the case paper. Link resolving errors from databases show a significant challenge in a reader reaching the retraction notice via a database search. Only 1/8 databases (and 1/9 database records) consistently resolved the retraction notice to its full-text correctly in our tests. Although limited to evaluation of a single case (N = 1), this work demonstrates how retracted research can continue to spread and how the current information environment contributes to this problem.
Retraction formally removes a paper from the scientific literature. But retraction does not end the diffusion of a paper or its findings. The continued diffusion of retracted papers has been documented since the 1990s (e.g., Pfeifer and Snodgrass 1990).
Previous research has documented the extent to which the current information environment contributes to continued diffusion of retracted research (Davis 2012). For example, readers may not know of the retraction because they saved a local copy of the paper before it was retracted (Davis 2012). Further, journal publishers often fail to alert readers that an article was retracted. Many databases and search engines, such as Web of Science, Google Scholar and Scopus provide nearly no warning of the retraction in their search results (Teixeira da Silva and Bornemann-Cimenti 2017; van der Vet and Nijveen 2016). In addition, although papers are retracted for different reasons, including self-retraction by the authors and various types of scientific misconduct such as data fabrication, data falsification, unethical author conduct, and plagiarism (Samp et al. 2012), about 10% of retraction notices do not indicate the reason for the retraction (Bozzo et al. 2017; Moylan and Kowalczuk 2016; Vuong 2020), making it difficult for a reader to determine the scope or severity of the problem (Guengerich 2015; Marcus and Oransky 2011).
The rate of retraction rose dramatically from 1997 to 2012, and since 2012, the number of journal article retractions appears to be roughly level at about 4 retractions per 10,000 publications (Brainard 2018). The highest percentages of retractions are found in medicine, life sciences, and chemistry journals (Grieneisen and Zhang 2012). Authors with multiple retractions (Grieneisen and Zhang 2012) and the rising total number of publications (Brainard 2018; Grieneisen and Zhang 2012) impact this rate. Scrutiny is also a factor: Fanelli (2013, p. 6) argues that “journal editors are getting better at identifying and removing papers that are either fraudulent or plainly wrong”.
Such scrutiny, however, does not appear to fall on the bibliographies of papers that cite retracted works. Large numbers of citations may accumulate even after a paper has been retracted. In one analysis, over 30% of the 74,000 citations to 3000 retracted articles accumulated a year or more after retraction (Cor and Sood 2018). Worryingly, scientific problems with a paper do not reduce its post-retraction diffusion; Bar-Ilan and Halevi (2018) found that papers with scientific distortion (errors and misconduct that impact the results of a paper) continued to accrue citations and readers at a faster rate than papers retracted for other reasons (including ethical misconduct such as plagiarism; and administrative error).
A significant limitation in our understanding of post-retraction citation is due to the fact that older studies and longitudinal studies of post-retraction citation are impacted by the rapid changes in the information environment and scholarly practices that have occurred in the past few decades. For instance, it is known that positive post-retraction citations can persist over decades in individual cases—for instance, 24 years for “an article retracted in 1982, but cited in 2006” in the field of psychiatry (Korpela 2010). However, article publication as of 1982 was in print, and up to the early-2000s there was an era of particularly rapid change in scholarly publishing, including the adoption of the Internet for Web-based digital journal publishing: “In 1993, very few scientific, technical, and medical (STM) journals had an electronic version, and yet by 2003, virtually all of them did.” (Renear and Palmer 2009). Studies of scholarly article reading behavior show significant changes from 1977 through 2005 (Tenopir et al. 2009), such as the increased likelihood for scholars to browse across disciplinary lines, and the reduced amount of reading time per paper. This rapid change in paper access and in reading behavior (and likely in citation behavior) means that we still need longitudinal data about positive post-retraction citation in the digital publishing era. Within the digital publishing era, the longest documented instance of positive post-retraction citation we are aware of is 9 years: Bar-Ilan and Halevi documented an article both published and retracted in 2007 receiving only positive post-retraction citations in 2015 and 2016 (Bar-Ilan and Halevi 2017), but clearly marked as retracted, and found by searching ScienceDirect for articles with retraction notices and the word “Retracted” in the title. In general, it is not known to what extent the visibility of retraction relates to or correlates with post-retraction citation (Balhara and Mishra 2014; Kim et al. 2019).
To fill this gap, the present paper uses a case study of long-term post-retraction citation to demonstrate that the current digital publishing and digital library environment communicates retraction status poorly and unevenly. Our case study focuses on a 2005 publication in the field of respiratory medicine (Matsuyama et al. 2005) which was retracted in 2008 (CHEST 2008) for reporting on falsified clinical trial data. We build on, and incorporate data from a previous study (Fulton et al. 2015a), which analyzed persistent post-retraction citation over 6 years, as of 2014. Compared to that prior work, our paper illustrates the potential for extended propagation of misinformation over a citation network; to do this we adopt network analysis methods not used in the previous case study. We extend the prior work with citation context analysis data for 66 additional publications (144 total) from 2006 through 2019 and an updated assessment of the problems with the display of retraction information on 12 digital platforms. We also add a new analysis of the second-generation citations that cite the retracted paper’s direct citations from both a network and citation context perspective, including a second-generation citation context analysis of 152 recent publications likely to spread misinformation from the retracted paper.
Our case study on a paper retracted for presenting fake data from a human clinical trial demonstrates how retracted research can continue to spread and how the current information environment contributes to this problem. This work contributes the longest longitudinal study of continued citation of a retracted paper within the era of digital scientific publication—nearly exclusive (96%) positive post-retraction citation 11 years from the 2008 retraction through 2019. We also demonstrate the possibility for misinformation to spread to a second-generation of publications that do not directly cite the retracted paper itself. The remainder of our paper is organized as follows. First, we discuss related work. Then, we describe our case study and our methods for analyzing the diffusion. We then present our results, discuss them, and finally conclude the paper.
Citation of retracted papers
Retraction is intended to remove seriously flawed papers from the scientific literature, and papers are not meant to be cited after retraction (Wager et al. 2009, 2019). While intentional post-retraction citation does happen (see e.g., the example on page 13 of Grieneisen and Zhang 2012), citation of research discredited due to misconduct is typically thought to be accidental or careless. Yet continued citation of papers retracted due to misconduct has been repeatedly observed since the 1990s (Bornemann-Cimenti et al. 2016; Fulton et al. 2015a; Garfield and Welljams-Dorof 1990; Kochan and Budd 1992; Korpela 2010; Neale et al. 2007, 2010; Whitely et al. 1994).
A particular challenge in synthesizing research results on post-retraction citation is that different sources of retracted papers and of citations with substantive differences in coverage (Chen et al. 2013) have been used. Post-retraction citation was first studied on the biomedical literature (Garfield and Welljams-Dorof 1990; Pfeifer and Snodgrass 1990; Wright 1991), and in the subsequent decades most citation analyses sourced retracted papers primarily from MEDLINE/PubMed and citations from Web of Science (e.g., Budd et al. 1998; Madlock-Brown and Eichmann 2015; Redman et al. 2008). The most comprehensive identification of retracted papers is probably from Grieneisen and Zhang (2012), which sought to assemble retracted papers from 42 sources, and the Retraction Watch database (Retraction Watch n.d.), which became publicly searchable in 2018. For retrieving citation data, Web of Science has been the most frequent source of citations (e.g., Budd et al. 1998; Garfield and Welljams-Dorof 1990; Madlock-Brown and Eichmann 2015; Pfeifer and Snodgrass 1990; Redman et al. 2008; Wright 1991), with some recent studies using data from Scopus (e.g., Bar-Ilan and Halevi 2018; Budd et al. 2016; Gray et al. 2018; Rubbo et al. 2018; van der Vet and Nijveen 2016), Google Scholar (Jan et al. 2018), or multiple of these sources (e.g., Avenell et al. 2019; Fulton et al. 2015a; Hamilton 2019).
Limited research has been conducted on post-retraction citation beyond biomedicine. Bar-Ilan and Helavi sourced retractions from ScienceDirect (2017, 2018), and in Wray and Andersen’s study of retracted articles published in the journal Science almost all received post-retraction citations (2018). Retracted engineering articles indexed in Web of Science received fewer citations than a set of control articles published in the same journal volume and number (Rubbo et al. 2018). Some arts and humanities articles from Retraction Watch and Scopus have been shown to receive ongoing post-retraction citations as well (Halevi 2019).
The impact of biomedical retractions on clinical research and clinical care has also been analyzed in citation research. In medical retractions, Steen pointed out the particular risks that citation of papers in clinical medicine has for patients and potential study participants (Steen 2011b). Hundreds of secondary research studies with human subjects cited his sample of retracted clinical papers, and 36% of these were post-retraction citations (Steen 2011b). Citation of 12 clinical trials retracted for misconduct has distorted the evidence base in bone health research: conclusions will change for at least 8 of 23 reviews (35%) on nutritional supplements for preventing hip fractures, falls, osteoporosis, and the utility of vitamin D for reducing severity of Parkinson’s disease (Avenell et al. 2019). Conclusions are also affected in five reviews and guidelines from organizations including the American Heart Association, the American College of Physicians, and the U.S. Agency for Healthcare Research and Quality (Avenell et al. 2019). Yet 85% of the post-retraction citations to these same 12 retracted clinical trials “expressed no concern” (Avenell et al. 2019, p. 2). Also potentially propagating clinical error, at least 5 systematic reviews in nursing included and synthesized articles that had already been retracted (Gray et al. 2018). Such citations do not meet consensus medical journal guidelines stating that retracted papers are not to be cited as science (International Committee of Medical Journal Editors 2019).
On average, retracted papers receive fewer post-retraction citations than unretracted papers. Pfeifer and Snodgrass estimated that post-retraction citations were reduced 35%, based on comparing citation profiles of 82 retracted biomedical articles in Index Medicus published from 1973 to 1987 to a cumulative citation curve of all works published in the same years and same journals using ISI SCISEARCH and SCI Journal Citation Reports (Pfeifer and Snodgrass 1990). Furman et al. estimated that retracted papers received 65% fewer citations than the two adjacent papers in the same journal, published immediately before and immediately after the retracted paper, using 677 retracted biomedical articles from 1972 to 2006 from PubMed that could be matched to Web of Science (Furman et al. 2012). Mott et al. found that citations to randomized controlled trials (RCTs) were reduced by 46% 1 year after retraction compared to matched control RCTs published at similar times in the same journal, by using an interrupted time-series analysis of the monthly citations for 218 RCTs with Web of Science citations (Mott et al. 2019). Retraction is also thought to depress research in closely related fields (Azoulay et al. 2015).
Author self-citation of retracted papers
Self-citation has sometimes been associated with post-retraction citation. In an analysis of 740 retracted articles from MEDLINE with citations from Web of Science, Madlock-Brown and Eichmann found that 5% of post-retraction citations (to 135 or 18% of the articles) were self-citations, and only 10% of self-citations also cited the retraction notice (Madlock-Brown and Eichmann 2015). Rubbo et al. analyzed 238 retracted articles in engineering from Web of Science, and compared them to a control group of 236 articles from the same journal volume and number. The retracted articles had more citations on average and more self-citations, and only 14 (5%) of the retraction notices had been cited. Self-citations made up 481/1291 (37%) of the pre-retraction citations and 74/1057 (7%) of the post-retraction citations (Rubbo et al. 2018). Self-citation was more prevalent before retraction (81 or 34%) than post-retraction self-citation (22 or 9%) (Rubbo et al. 2018).
Accessibility and visibility of retracted papers and their retraction notices
In 1991, Wright wrote, “A retraction can…be published or identified in many sources, and still be ignored in all. Librarians, information scientists, and journal editors have found no infallible means of communicating retractions of publication to the users of the literature.” (Wright 1991). Solutions have changed in the transition from printed journals to electronic access: in the early 1990s, libraries’ physical holdings were the focus of retraction alerting (Pfeifer and Snodgrass 1992) while as late as 2000, rubber stamping retractions in print volumes was compelling. Davis advocated a three-pronged approach for reducing citation of retracted papers in 2012: Databases and search engines should alert readers before they read, bibliographic management software should alert authors before writing, and journals should check bibliographies for retracted articles before publishing (Davis 2012).
Davis’ suggestions have still not been completely implemented, 8 years later. The simplest tool for readers, Crossref’s CrossMark, has limited coverage because as of 2020 publishers must pay to participate; CrossMark provides a “Check for updates” button to find retraction status and other corrections directly from PDF or HTML articles (Bornemann-Cimenti et al. 2016; Kim et al. 2019). CrossMark is also only as good as the publisher data, which seems to be lacking. For instance, 3 years after retraction, 22% of MEDLINE-indexed articles retracted in 2008 were not annotated as retracted on publisher websites and PDFs (Decullier et al. 2013). Missing watermarks (Elia et al. 2014) and database errors in Web of Science and PubMed (Schmidt 2018) also limit alerting of readers.
Reference management software packages do not systematically collect or display retraction status, though new tools inside reference management software are emerging, including ReTracker (Cheng et al. 2019) and, more recently, Zotero’s retraction notification based on the Retraction Watch Database (Stillman 2019). As of 2020, not all citation styles indicate how to format a citation to a retracted paper, and manual correction is needed in order to add “[retracted in (citation)]” as required by the American Medical Association Manual of Style when using common reference management software such as EndNote, RefWorks, and Zotero (Suelzer et al. 2019).
Journals do not systematically check bibliographies for retracted articles. When journals do prevent citation of retracted articles it seems noteworthy (Brand et al. 2017). Medical journal guidelines state that “Authors are responsible for checking that none of the references cite retracted articles except in the context of referring to the retraction” (International Committee of Medical Journal Editors 2019) and refer authors to PubMed, which does, however, have some known errors in its retraction indexing (Schmidt 2018).
Correlation between visibility of retraction status and post-retraction citation
An emerging area of research investigates the correlation between how visible the retraction status is, in databases or on retracted articles themselves, and how often articles receive post-retraction citations. As of 2014, 25% of the retraction notices on articles on mental disorders were not available in Web of Science, and the articles without a freely accessible retraction notice were more likely to gain post-retraction citations (Balhara and Mishra 2014). By contrast, indicating retraction in HTML and PDF versions did not significantly impact the number of post-retraction citations published at least 1 year after the retraction notice for 114 retracted articles from KoreaMed, as cited in Scopus and Korea Medical Citation Index (Kim et al. 2019). The reason for the disconcordance of these studies is not immediately obvious. One limitation is that neither of these studies read the text of post-retraction citations to determine whether they were positive. It is particularly important to determine whether positive post-retraction citation and visibility of retraction status are correlated.
Diffusion and persistence of error
Citation analysis is often seen as a means of measuring knowledge flows. Citation contexts, the texts surrounding citations, have been analyzed for a variety of purposes—perhaps the best known applications in the scientometrics community have been to study citation motivation (Small 1982) or a work’s impact and reception (Bornmann et al. 2020). Distortions in information propagation have been studied using similar methods. For example, a study of the “accuracy of quotations” (including paraphrased information) assessed whether the original work was reflected accurately when cited (De Lacey et al. 1985), and subsequent work has suggested that readers may copy reference strings uncritically without reading them (e.g., Stang et al. 2018). Movement of knowledge between languages, including translation error, has been implicated in some cases of misinformation (Wetterer 2006). Persistence in belief has also been studied using citation contexts: a study by Tatsioni et al. (2007) analyzed the persistence of belief in evidence from lower evidence types such as observational studies when these conflict with evidence from randomized controlled trials (RCTs). This relies on the medical hierarchy of evidence (Grimes and Schulz 2002), in which valid RCTs are prioritized above observational studies.Footnote 1 We consider the underlying method, regardless of the motivation or the particular attributes of the text being analyzed, as citation context analysis.
Network science has also been used to study distortions in citation. Greenberg coined the term ‘citation bias’ to refer to selective citation of the literature, and demonstrated how unsubstantiated claims about Alzheimer’s disease were bolstered in papers and in funded NIH grants by citation bias (Greenberg 2009). In such cases, deceptive citation practices may suggest a consensus that does not yet exist. Consensus formation itself has been studied by sociologists of science by examining the extent of segmentation and modularization of the literature, where extreme fragmentation demonstrates “epistemic rivalries” (Shwed and Bearman 2010). Chen and Song (2017) suggest that uncertainty and contradiction need further attention within the study of research fronts and science of science, and propose visualization as a helpful approach.
Combining citation context analysis and network analysis
While some studies primarily analyze citation contexts (e.g., Suelzer et al. 2019) or citation statistics (e.g., Peterson 2013), we are aware of three studies that have drawn on synergies between citation context analysis and network visualization perspectives (Chen et al. 2013; Dinh et al. 2019; van der Vet and Nijveen 2016). The most similar to the present work (van der Vet and Nijveen 2016) presents a case study of citation networks and citation contexts for a single retracted paper that was published in Nature in December 2012 and retracted in February 2014. Van der Vet and Nijveen extracted the closure of the citation network from Scopus twice: shortly after the article was retracted (March 2014, 37 direct citations, 187 total nodes) and again 1 year later (March 2015, 57 direct citations, 1626 total nodes) (van der Vet and Nijveen 2016). Direct citation contexts are categorized by the article sections in which they appear (i.e., Introduction, Methods and Materials, Results, Discussion) and citing articles are classified as review or original contribution; they note that only 2 citations indicated controversy or cited the retraction notice (van der Vet and Nijveen 2016). They also read 10 articles from the indirect citation network, identified using a keyword search for the topic of the retracted paper to look for indirect citations that might spread results from the retracted paper (van der Vet and Nijveen 2016). A particular limitation of van der Vet and Nijveen (2016) is its short timeframe: the 13-month period used is within the washout period for some studies of post-retraction citation. Our study is larger than (van der Vet and Nijveen 2016) in both timeframe (11 years after the retraction notice in our case compared to 13 months in theirs) and in network size (2542 in our second-generation network compared to 1626 in their citation closure network).
By comparison, the present work focuses on the continued direct citation of our case study paper for more than a decade following its 2008 retraction. We analyze both the direct citations and the depth-two citation network, as of 2019, 11 years after retraction. We also systematically assess how visible the retraction is in 12 common digital platforms and document access errors.
Our case study centers around a single paper (Matsuyama et al. 2005) retracted (CHEST 2008) for presenting fake data from a human clinical trial. The paper was published in December 2005 and retracted in October 2008 because “the university that employs the authors determined that one of the authors, Wataru Matsuyama (now deceased), falsified data” (CHEST 2008). From here on, we will refer to the paper, “Effects of Omega-3 Polyunsaturated Fatty Acids on Inflammatory Markers in COPD” (Matsuyama et al. 2005), as the Matsuyama paper. It is one of 17 papers by Wataru Matsuyama retracted from 2007 to 2010 (Grieneisen and Zhang 2012; Steen 2011a); through mid-2018, he was considered one of the most prolifically retracted authors (Grieneisen and Zhang 2012; The Retraction Watch Leaderboard [Internet Archive, 2018-05-30] 2018).
The Matsuyama paper studied a nutritional intervention for a chronic disease: it purported to show that a diet rich in omega-3 polyunsaturated fatty acids reduced inflammation and improved exercise capacity in persons with chronic obstructive pulmonary disease (COPD). Omega-3 polyunsaturated fatty acids are of particular interest as an adjunct to conventional pharmacotherapies in people with inflammatory diseases such as hypertriglyceridemia and rheumatoid arthritis, due to the cost-effectiveness and favorable safety profile (Calder and Zurier 2001; Chang et al. 2018; Samuel et al. 2011). The design used in the Matsuyama study is the strongest established method for showing treatment efficacy: a randomized controlled trial (RCT). A correctly conducted RCT would better inform clinical practice, as of the time of publication there were only four observational studies investigating the relationship between omega-3 polyunsaturated fatty acids and inflammation in people with COPD (Fulton et al. 2015b), despite promising results in other inflammatory conditions (Calder 2006).
The Matsuyama RCT should have been the seminal RCT on which future clinical research would build and although it is unclear the specific effects of retracting seminal works due to falsified data, it is likely to delay the field of research with researcher focus shifting to other areas. Citations to the Matsuyama paper are particularly problematic. No RCT addressing the same specific aims as the Matsuyama study has been published to date. The evidence base has grown slowly over time and there are now four published RCTs in the general area of omega-3 polyunsaturated fatty acids and COPD; two specifically investigated oral nutritional supplements that contained omega-3 polyunsaturated fatty acids with other supplements such as vitamin D in cachectic COPD patients (Calder et al. 2018; van de Bool et al. 2017), the third focused on COPD exacerbations (Ogasawara et al. 2018) and the fourth was a feasibility study (Fulton et al. 2017). It should be noted that the largest RCT (> 25,000 participants) investigating both vitamin D and fish oil alone and in combination on respiratory symptoms and acute exacerbations in COPD, is expected to be completed in November 2020 (Gold et al. 2016; NCT01728571: LungVITamin D and OmegA-3 Trial (LungVITAL), n.d.). The Matsuyama paper cannot be considered “evidence” since the data was faked. However, at present no other paper at or above this level in the medical evidence hierarchy with the same aim could be cited to support a knowledge claim, for or against the purported (faked) results of the trial. Uncritical citation continues to diffuse the faked data, and implies belief in the efficacy of a yet unvalidated treatment.
Methods and results
Our analysis has three parts, focusing on (1) the direct citations to the Matsuyama paper, (2) the citations to those papers (second-generation citations with respect to the Matsuyama paper), and (3) the visibility of the Matsuyama paper’s retraction status in digital platforms. Data (Schneider and Ye 2019; Ye et al. 2020; Ye and Schneider 2020) and code have also been publicly deposited, and to simplify the presentation, additional details are presented in a supplemental appendix.
Direct citations to the Matsuyama paper
We searched Web of Science and Google Scholar for publications citing the Matsuyama paper. Details are available in the methods supplement to this paper. All forms of publications, published before December 31, 2019, in any language were included.
Our network analysis covers 148 direct citations to the Matsuyama paper (Matsuyama et al. 2005). Articles in fourteen languages cited the Matsuyama paper: Chinese, English, French, German, Greek, Italian, Japanese, Norwegian, Polish, Russian, Serbian, Spanish, Thai, Ukrainian.
Figure 1 shows the year-by-year citation pattern. For the purpose of deeming articles as post-retraction, we use a 2-month washout period after the retraction notice was issued in October 2008 (CHEST 2008). This gives us 32 pre-retraction citations (before January 2009, with the first citations found in 2006) and 116 post-retraction citations (from January 2009 through December 31, 2019).
Citation context analysis
Of the 148 direct citations to the Matsuyama paper (Matsuyama et al. 2005), ultimately, we examined and report on citation contexts of 144 items (112 post-retraction).
We examined bibliographies to verify that each publication cited the Matsuyama paper and to determine how it was cited in the text. We manually extracted all citation contexts referring to the Matsuyama paper. Primarily we used Google Translate to translate citation contexts.
Following (Fulton et al. 2015a), we annotated three aspects of the citation contexts: whether they were positive or negative; whether they described methods or results of the Matsuyama paper as opposed to citing a general concept (as shown in Table 1); and, for post-retraction publications, whether they mentioned the retraction by using words such as “retraction” or “retracted” in the citation context or cited the retraction notice (CHEST 2008). The data supplement to the present paper provides citation contexts and annotations for the 66 new citations not previously reported in Fulton et al. (2015a).
There were only 5 negative citations, i.e., that refer to the article as poor research (5/144; 3.5%). These are a survey of retracted drug literature (Samp et al. 2012); three articles we (Fulton/Hill) authored, namely a protocol (Fulton et al. 2013), the previous study of post-citation retraction (Fulton et al. 2015a), and a systematic review (Fulton et al. 2015b); and a Cochrane Review (Abdelhamid et al. 2018). We deemed that not mentioning the article in-text but including it in the bibliography counted as positive citation; this applied to five items: four U.S. patents (Jackowski et al. 2012a, b, 2014, 2015) and one Spanish textbook chapter (Lozano et al. 2011) with no in-text citation that listed the Matsuyama paper and that did not cite its retraction notice (CHEST 2008) or mention its retracted status.
Only 5 publications mentioned the retraction (5/112; 4.5%)—these were exactly the same as the negative citations mentioned above. The remaining 107 post-retraction citations would not meet the International Committee of Medical Journal Editors’ guidelines, to not cite retracted work as science (International Committee of Medical Journal Editors 2019).
Table 2 shows how many of the citations were annotated as positive/negative overall and mentioned/not mentioned out of the 144 citations we examined from 2006 to 2019.
Table 3 reports on how in-depth the description of the Matsuyama paper was in the 144 citations from 2006 to 2019. Overall 44 of the 112 post-retraction citations we examined described the methods or results of the Matsuyama paper without mentioning the retraction.
Diffusion of unsubstantiated content
Diffusion into clinical venues is particularly troublesome. This includes a French nutrition treatise (Pison et al. 2016) and a Japanese translation of the European Society for Clinical Nutrition and Metabolism’s Life-long Learning Programme in Clinical Nutrition and Metabolism (Nagahama et al. 2013). Likewise, a book promoting Omega-3 oils contains two chapters with post-retraction citations of the Matsuyama paper (Fitzpatrick 2011; Monk et al. 2011). Most shocking is the 2017 version of textbook materials for the European Society for Clinical Nutrition and Metabolism’s Life-long Learning Programme in Clinical Nutrition and Metabolism (Schols 2017).
We focus next on the potential for diffusion of misinformation into a second generation of citations.
Analysis of second-generation citations to the Matsuyama paper
We searched Web of Science and Google Scholar for publications citing the 148 direct citations to the Matsuyama paper. All forms of publications, published before December 31, 2019, in any language were included. This resulted in the identification of 2542 articles.
Figure 2 shows the network of 148 direct and 2542 second-generation citing articles of the Matsuyama paper. For the second-generation, Fig. 4 represents the worst case: that the faked data in the Matsuyama paper impacts its direct citations as well as each publication citing one of the direct citations.
Second-generation articles in thirty-two languages cited an article that cited the Matsuyama paper: Bosnian, Chinese, Croatian, Czech, Dutch, English, Finnish, French, German, Greek, Hungarian, Icelandic, Indonesian, Italian, Japanese, Korean, Latvian, Lithuanian, Mongolian, Norwegian, Persian, Polish, Portuguese, Romanian, Russian, Serbian, Slovenian, Spanish, Swedish, Thai, Turkish, and Ukrainian.
Figure 3 shows the historical growth of the direct and second-generation citation network pre-retraction (including a 2-month washout), from the Matsuyama paper’s publication year (2005) through the end of 2008 (2 months after the retraction notice). The second-generation network has continued to grow after retraction, as shown in Fig. 4.
Diffusion of misinformation from publications discussing methods and results of the Matsuyama paper
Our approach to investigating the possible diffusion of misinformation to a second generation of citations is shown in Fig. 5. This is predicated on the idea that misinformation is most likely to be transmitted by citing a direct citation that describes the methods and results of the Matsuyama paper.
We show two network diagrams. Figure 6 covers all 60 articles describe the methods and results of the Matsuyama paper that do not (post-retraction) or could not (pre-retraction) cite the Matsuyama paper’s retraction notice; there are 1481 second-generation citations to these 60 direct citations. Figure 7 shows the smaller post-retraction network that we sought to analyze using a second-generation citation context analysis. To make a second-generation citation context analysis feasible, we limited our attention to the most recent 35 direct (2010–2019) citations that do not mention the retraction but do mention methods or results of the Matsuyama paper, and the 161 second-generation citations associated with them.
Citation context analysis applied to selected second-generation citations to publications discussing methods and results of the Matsuyama paper
There were 44 direct post-retraction (2009–2019) citations citing methods and results of the Matsuyama paper and their 1481 citations. We limited our attention to the most recent 35 direct (2010–2019) citations that do not mention the retraction but do mention methods or results of the Matsuyama paper. Of their 161 citations—which are second-generation citations from the perspective of the Matsuyama paper—we were able to access 152 second-generation citations. Of these we marked 23 as possibly spreading misinformation, i.e., relying on information in one of Matsuyama’s direct citations that seemed to have come at least in part from the Matsuyama paper. They were spread by 4 different review articles, with the bulk spread by Giudetti and Cagnazzo (2012), cited in 18 of the 23 cases of misinformation, with the other reviews being cited in 1, 2, and 2 papers. These 23 examples are given in the data supplement and we next discuss four of them in detail.
A government research bulletin from Nepal (Jha 2016) cites a book chapter “Health benefits of flaxseed” (Fitzpatrick 2011): “Whilst it is true that very little ALA converts to the long chain polyunsaturated omega-3 found in marine oils, it does have beneficial effects itself (Fitzpatrick 2011).” The Matsuyama study provided ALA (alpha linolenic acid)-rich nutritional support and its retracted results are described in Fitzpatrick (2011) as “evidence” of the anti-inflammatory impact of flax.
As another example, an Irish nutritional support shop recommends n-3 fats to athletes (Healthy Fats, Fish Oils and Omega-3 Supplementation 2017): “During periods of illness, this may help promote recovery and faster return to training. Interestingly, n-3 fats are sometimes provided to COPD patients (severe airway damage and breathing difficulties) and prior to surgery in order to support the immune system and speed recovery by helping to control inflammation and infection, and repair damaged cells .” This is on the basis of “Immunologic impact of nutrient depletion in chronic obstructive pulmonary disease” (Herzog and Cunningham-Rundles 2011), which cites the Matsuyama paper as having demonstrated “Improved 6 min walk test, decreased leukotriene B4 level, TNF-alpha, IL-8 ” (i.e., the faked data for which the Matsuyama paper was retracted).
A pre-clinical study on lung repair following dust exposure (Nordgren et al. 2018) draws indirectly on the retracted science, via citation to Giudetti and Cagnazzo (2012), to motivate its work: “Furthermore, studies reveal diets high in omega-3 polyunsaturated fatty acids (n-3 PUFA) may be beneficial in inflammatory lung conditions, including asthma and COPD (17).” Giudetti and Cagnazzo discuss evidence for benefit in specific inflammatory lung diseases and devote two paragraphs to the effects on COPD; one paragraph cites Matsuyama, a review article, and an intervention in patients undergoing physical rehabilitation, while the second paragraph provides detailed results from the Matsuyama study. Later this is summarized as “Nutritional interventions with n-3 PUFA supplementation have been shown to be particularly beneficial in patients with COPD [110,111]…” Yet in effect, the only cited “evidence” for the effect of n-3 PUFAs on inflammation in COPD came from the retracted Matsuyama paper.
Possible misinformation also may have spread by moving broader claims related to the Matsuyama paper across fields. As discussed above, we considered COPD-related statements supported by Giudetti and Cagnazzo’s review as possibly spreading misinformation, due to reliance on the Matsuyama paper. Yet the Giudetti and Cagnazzo review, titled “Beneficial effects of n-3 PUFA on chronic airway inflammatory diseases,” went beyond COPD to discuss clinical applications for three related diseases—asthma, cystic fibrosis, and Acute Respiratory Distress Syndrome. A weaker case of possible spread of misinformation is found in Thesing et al. (2018), which moves the study of Omega-3 fatty acids into the realm of anxiety and depression. Its three mentions of Giudetti and Cagnazzo (2012) relate to inflammatory diseases, for instance: “Some randomized controlled trials have shown that intake of N-3 PUFAs ameliorate or even prevent physical illnesses such as inflammatory (Giudetti and Cagnazzo 2012; Simopoulos 2002) and cardiovascular diseases (La Rovere and Christensen 2015; Simopoulos 1999), while others have not (Hoogeveen et al. 2014; Kromhout et al. 2010).” This statement is not exactly wrong, since Thesing mentions the wider class of inflammatory diseases without specifying COPD. But the misinformation within the Giudetti and Cagnazzo review about COPD adds fragility to this statement, especially since it is the only lung-related paper in Thesing’s bibliography. Moving this across communities, from lung disease researchers to a psychiatric research community, may increase the risk of propagating misinformation.
Visibility of the Matsuyama paper’s retraction status in digital platforms
We ask how readers, including citing authors, could become aware that the paper is retracted, by assessing how the Matsuyama paper (Matsuyama et al. 2005) and its retraction notice (CHEST 2008) are displayed in 12 digital platforms. Additional details are given in a methods supplement.
Expectations from industry guidelines
The key industry guidelines for medical articles are the International Committee of Medical Journal Editors (ICMJE) “Recommendations for the Conduct, Reporting, Editing, and Publication of Scholarly Work in Medical Journals” (International Committee of Medical Journal Editors 2019) to which the journal publishing the Matsuyama paper, Chest, adheres (International Committee of Medical Journal Editors, n.d.). They direct editors to flowcharts of the Council on Publication Ethics (COPE) for detailed procedures. COPE also publishes retraction guidelines (Wager et al. 2009, 2019), which provide similar guidelines to ICMJE.
As of December 2019, the ICMJE guidelines state that retraction notices “should be prominently labelled, appear on an electronic or numbered print page that is included in an electronic or a print Table of Contents to ensure proper indexing, and include in their heading the title of the original article. Online, the retraction [notice] and original article should be linked in both directions and the retracted article should be clearly labelled as retracted in all its forms.” (International Committee of Medical Journal Editors 2019, p. 9). The retraction notice should indicate its authorship, the reason for the retraction, and “a complete citation reference” to the retracted article (International Committee of Medical Journal Editors 2019, p. 9). COPE does add that “Journals are responsible that retractions [i.e., retraction notices] are labelled in such a way that they are identified by bibliographic databases and should also include a link to the retracted article. The retraction should appear on all online searches for the retracted publication.” (Wager et al. 2019, p. 6).
Expectations for our analysis
Analyzing previous research (Decullier et al. 2013; Elia et al. 2014; K. Wright and McDaid 2011) and industry guidelines (International Committee of Medical Journal Editors 2019; Wager et al. 2019) led us to establish the following visibility expectations for our analysis:
All search results for the title of the retracted article should also return the retraction notice.
For retracted articlesFootnote 2 on full-text sites
Each article landing page, full-text HTML article, and full-text PDF article should have a phrase indicating the retraction status (such as “retracted,” “withdrawn,” etc.) or a watermark indicating the retraction status.
Each landing page, full-text HTML article, and full-text PDF article should have a computer-actionable link to the retraction notice.
For retraction notices in full-text sites
The retraction notice should appear in Table of Contents for the issue in which it appears, with a designated page number.
The heading of the retraction notice should include the phrase “retraction notice” and the title of the retracted article.
The textual content of the retraction notice should state authorship, reason for retraction, and formally cite the retracted article.
Each landing page, full-text HTML notice, and full-text PDF notice should have a computer-actionable link to the retracted article.
For database records for retracted articles
A phrase indicating the retraction status, such as “retraction,” “retracted,” or “withdrawn” should appear in the article record.
The article record should have a computer-actionable link to the retraction notice. This could link to the database’s record for the retraction notice, or to the full-text retraction notice.
The article record has sufficient bibliographic information to retrieve the retraction notice.
For database records for retraction notices
The phrase “retraction notice” should appear in the text of the notice record.
The notice record should have a computer-actionable link to the retracted article. This could link to the database’s record for the retracted article, or to the full-text retracted article.
The notice record has sufficient bibliographic information to retrieve the retracted article.
All search results for the title of the retracted article should also return the retraction notice
For sites with a search interface, we asked whether the retraction notice (CHEST 2008) appeared in searches for the title of the retracted publication (Matsuyama et al. 2005) Effects of omega-3 polyunsaturated fatty acids on inflammatory markers in COPD.
Searches failed to meet our expectations in several ways.
The Matsuyama article’s retraction status could easily be missed, because the retraction notice appears before the article in only 4 out of the 12 searches (CINAHL EBSCOhost, MEDLINE—Ovid, PubMed, and Web of Science—All Databases).
In Google Scholar and Web of Science—Core collection the retraction notice did not appear at all in default searches as shown in Table 4. At first we were puzzled that the retraction notice did not appear in the Web of Science—Core collection. We suspect that this is due to use of MEDLINE indexing as a quality measure for inclusion in Web of Science—Core, because the record that did appear in the Web of Science—All Databases has the designation “PubMed-not-MEDLINE”.
For ScienceDirect, the retraction notice did not appear in the first 25 records, the default first page size. The retracted article was first and the retraction notice was listed as result number 82 out of 194 results and not prominently labeled as shown in Fig. 8, making it very unlikely that a user would see the retraction notice.
Non-default search options at ScienceDirect, not shown in the table, were somewhat better. The retraction notice is the fourth result of four on January 20, 2020 when entering the title string in the “Search in this journal” search from the ScienceDirect journal homepage for Chest. However, the retraction notice was not found by entering the title string in ScienceDirect’s Advanced search option “Title, abstract or author-specified keywords”; this search returned just 1 result, the retracted article, on January 20, 2020.
Retracted article on full-text sites
We considered full-text sources to be digital library sites that hosted PDFs. On these sites, we assessed whether each article landing page, full-text HTML article, and full-text PDF article had (1) text such as “retracted” or “retraction” or a watermark indicating the retraction status and (2) a computer-actionable link to the retraction notice. Results are shown in Table 5.
No watermarks were used. Retraction text that does appear is not necessarily in the metadata: On Semantic Scholar the Retraction Watch blog post “More evidence scientists continue to cite retracted papers” about (Fulton et al. 2015a) is in the “paper mentions” section. The most prominent information here, from an Ovid record, could still be missed by a skimming reader.
Retraction notice in full-text sites
We found the retraction notice in only two full-text sites. For these, we assessed four aspects of the retraction notice (CHEST 2008), as shown in Table 6: the front matter, the heading content, the textual content, and its actionable links to the article. Regarding the front matter, we asked whether the retraction notice appeared in the Table of Contents for the issue in which it appears, with a designated page number. In the heading we checked for words such as “retraction”, “retraction notice”, “withdrawn”, “withdrawal”, etc.) and the title of the retracted article. In the textual content of the retraction notices we checked for authorship (here, by Chest), reason for retraction (here, “because the university that employs the authors determined that one of the authors, Wataru Matsuyama (now deceased), falsified data.”), and a formal citation to the retracted article. Each landing page, full-text HTML notice, and full-text PDF notice should have a computer-actionable link to the retracted article.
Multiple areas for improvement are evident. The lack of article title in the heading impairs search results as shown in Fig. 8 above. Citation databases do not index the retraction notice (CHEST 2008) as a citing article since it is not a formal citation. Linking to the retracted article, called for in the guidelines (International Committee of Medical Journal Editors 2019; Wager et al. 2019), is also lacking.
Database records for the retracted article
For each database record referring to the article (Matsuyama et al. 2005), we asked whether a word such as “retraction” or “retracted” appeared in the article record; whether the article record included a computer-actionable link to the retraction notice (CHEST 2008); and whether there was sufficient bibliographic information to retrieve the retraction notice (CHEST 2008) manually by volume and issue number. Table 7 shows the results.
While 5/8 databases included a word like “retracted” in the record, with the notable exception of PubMed, the retraction status is poorly signaled. Confirming information from the retraction notice would be difficult from these records, as only 1/8 (MEDLINE-Ovid) had a link to the retraction notice, and only 2/8 had full bibliographic information for the retraction notice.
The CINAHL database did show some improvements during our study; the retraction notice was not found in our searches for Effects of omega-3 polyunsaturated fatty acids on inflammatory markers in COPD on September 27, 2018, but by January 2020, a second record with the retraction notice had appeared.
Database records for the retraction notice
The most problematic results came in attempting to resolve to the full-text of the retraction notice from databases. For each database notice record referring to the retraction notice (CHEST 2008), we asked whether a word such as “retraction notice” appeared in the notice record, and whether the retraction record included a computer-actionable link to the retracted article (Matsuyama et al. 2005); and whether there was sufficient bibliographic information within the record itself to retrieve the retracted article manually by volume and issue number (Matsuyama et al. 2005). Table 8 shows the results.
Resolving errors from databases show a significant challenge in a reader reaching the retraction notice via a database search. Only 1/8 databases (and 1/9 database records), again MEDLINE Ovid, consistently resolved the retraction notice to its full-text correctly in our tests. Errors are varied, as shown in Fig. 8 above and Figs. 9, 10, 11 and 12 below. Most errors provided dead ends with no further information, though some redirected to the journal homepage or to a search page (Fig. 10).
Delays in updating were also a factor; for instance, the Cochrane record for the retraction notice indicates that it was added October 31, 2014, 6 years after the retraction notice was published in October 2008. There are several inaccuracies: Scopus lists the retraction notice as an erratum, which it is not. Metadata errors such as an incorrect page number (892 in place of the expected 893) or inclusion of an author name hindered resolution in some cases, as is most evident in Fig. 10.
Due to these errors, readers looking for the retraction notice would either need to use search results (challenging as noted above), type in the DOI (if they somehow find it), or navigate through volume, issue, and page number in order to find the retraction notice.
Discussion and conclusions
Our case study showed that a purported nutritional treatment for lung disease has received continued citation for 11 years after its formal retraction from the literature in 2008, when data from the human trial promoting this treatment was deemed fake. We have demonstrated the spread of misinformation from a unique knowledge claim, that could not be supported by alternative data at the same level of evidence. This diffusion of misinformation has the potential to harm patients, through promoting a nutritional treatment that has not yet been established through strong, concordant evidence, and which may be used as an adjunct to pharmacotherapy. It is also conceivable that, based on the publicity of the outcomes from the Matsuyama study, some individuals with COPD may perceive omega-3 supplements as a possible natural alternative to pharmacotherapy.
Information about the purported treatment benefit has flowed widely to researchers, but also to clinical audiences, through educational modules (Nagahama et al. 2013; Schols 2008, 2013, 2017), clinical nutrition reviews (Fitzpatrick 2011; Hall III et al. 2009), and textbooks (Pison et al. 2016), including direct translation of educational modules (Nagahama et al. 2013; Schols 2013) from English into other languages. Information directed to the general public has not been corrected (Arnold 2009a, b, 2010). Researchers specifically citing the results of the Matsuyama paper seem unaware of the fact that the paper has been retracted: only 5 direct citations to the Matsuyama paper mentioned the retraction (Abdelhamid et al. 2018; Fulton et al. 2013, 2015a, b; Samp et al. 2012), and those same 5 alone described it as poor research. The retraction is not mentioned in 96% (107/112) of direct post-retraction citations we were able to examine. As of the current writing, the post-retraction citation continues unchecked, with eight additional new 2019 citations (Al-Haidose 2019; Cai et al. 2019; Collins et al. 2019; Hani 2019; Nguyen 2019; Omar et al. 2019; Ran et al. 2019; Wang and Wu 2019), including two 2019 Ph.D. dissertations (Al-Haidose 2019; Nguyen 2019); and, of the five peer-reviewed articles, one (Wang and Wu 2019) was published by Elsevier, who had published the retraction notice 11 years earlier in their journal Chest. Authors, editors, and publishers still do not have adequate tools to identify and flag retractions.
Unlike the average retracted RCT, the Matsuyama paper’s citations increased substantially after retraction. Other studies (Azoulay et al. 2015; Furman et al. 2012; Mott et al. 2019) have found decreases in citation after retraction. In particular, (Mott et al. 2019), suggested that awareness of retraction status (such as through media attention) decreases citation. Mott analyzed citations to a set of 218 retracted RCTs, using the Web of Science citations received in a 4-year period centered around retraction. The articles were split into two subgroups: 154 RCTs that received significant media attention when retracted due to 2 high-profile misconduct cases (Fujii and Boldt) and 64 non-high profile retracted RCTs. The high-profile cases received fewer citations: Fujii and Boldt’s RCTs (mean 5.2) as compared to non-high profile retracted RCTs (mean 21.7). The articles were less cited after retraction (e.g., citations to the 64 RCTs decreased 1.8% per month, compared to the trend in matched controls). The Matsuyama paper we studied was on par with the 21.7 mean for non-high-profile cases in Mott’s study, in overall number of citations in Web of Science: 21 citations, 7 in the 24 months before its October 2008 retraction, and 14 in the following 24 months. However, the trend of citations actually increased, which seems consistent with a lack of awareness of the Matsuyama paper’s retraction status.
The wider information environment is implicated in the spread of the Matsuyama paper “Effects of omega-3 polyunsaturated fatty acids on inflammatory markers in COPD” and its faked results. Unknowing citation of a retracted paper is never acceptable. Yet a paper’s retraction status is not always evident to readers. Even subscription databases and publisher websites may lack up-to-date information about a retraction status. In the case of the Matsuyama paper, 11 years after its retraction, basic information that readers need, such as the retraction status, is still missing from several providers, and is difficult to understand at other providers. In particular, retraction status was not indicated on any PDF or HTML full-text. Further, resolving errors from databases show a significant challenge in a reader reaching the retraction notice via a database search, in that only 1/8 databases (and 1/9 database records) consistently resolved the retraction notice to its full-text correctly in our tests.
It is striking to compare our case paper to an also-retracted paper which cites our case paper and was written by some of the same authors: “Use of Tiotropium Bromide for Pre-operative Treatment in Chronic Obstructive Pulmonary Disease Patients: Comparison with Oxitropium Bromide”. It had 11 citations from Google Scholar and Web of Science as of December 2019. Of these 11, three citations—its retraction notice (Matsuyama et al. 2008) (self-retracted for data problems) and 2 studies of retracted publications—cite with awareness of the retraction. Of the remaining 8 citations, 3 were in the year of retraction, while the remaining 5 citations in Greek, Chinese, and English from 2011 to 2017 are clearly post-retraction but only available in Google Scholar. This fits the picture of the lower impact factor of the journal (consistently Q3 in its JCR impact factor category, Medicine, General and Internal—SCIE, since 2001), but may also be impacted by the visibility of the retraction. On Web of Science, the retraction notice (Matsuyama et al. 2008) appears in search results for the article’s title, and in the citations. As of early 2020, the article’s abstract on the publisher website, J-STAGE, reads “This article was retracted. See the Notification.”, the HTML links to the retraction notice (Matsuyama et al. 2008), and the first page of the PDF includes a copy of the retraction notice itself. Visibility of the retraction notice is of course not the only difference between this article and our case paper but it is suggestive.
The current information environment facilitates the spread of research papers, but basic facts about these papers, such as their retraction status, do not spread as swiftly as the PDFs or citations to these papers themselves. Our case study suggests that unknowing and likely unintentional citation of retracted papers could be common, and that post-retraction citation may be correlated with visibility of retraction status. However, this work is limited to evaluation of a single case (N = 1). Future research should conduct large-scale examination of the correlation between visibility of retraction status and post-retraction citation, especially positive post-retraction citation.
Improving this situation will take dedicated efforts from multiple stakeholders in the scholarly communication ecosystem. While continued citation of retracted papers may be acceptable in rare circumstances, authors, editors and publishers should ensure that this happens only in full knowledge of the retraction. Citations to retracted papers should clearly mark the paper as retracted, and citation of the retraction notice should be encouraged as an alternative to direct citation of retracted papers. Citation standards should be explicit, and bibliographic management tools should follow existing standards (Suelzer et al. 2019). Publishers must bear responsibility for clearly marking retracted papers, by watermarking their own copies in all formats (e.g., PDF, HTML, EPUB); by prominently linking to the retraction notice wherever the paper appears, including the article landing page and issue table of contents; and by providing accurate metadata, including the retraction status, to partners. Publishers and typesetters should also surveil bibliographies for papers recognized as retracted, for follow up editorial action, whether to acknowledge the retraction or to replace the cited paper. Database providers and aggregators must demand up-to-date metadata from publishers, and should consider partnering with alternative metadata producers (such as the Retraction Watch DatabaseFootnote 3). Parties responsible for the retraction, including authors for self-retraction and investigative committees for misconduct-related retraction, should search citation databases and notify citing authors directly to the retraction notice, for follow-up action (such as corrections, retrenchment in new citations, etc.). This is particularly important for papers whose conclusions fundamentally depend on retracted work (Fu and Schneider 2020). While currently no tools provide notification to the authors of pre-retraction citations, such tools could also be beneficial.
Quality assessment of digital libraries must ensure that known problems in a paper’s validity or reproducibility, as documented by retraction, are as evident to readers as standard information such as a paper’s title, authors, or venue. While the diffusion of information is often studied at a macro-level, for understanding the flow of information across geographic, linguistic, or field-specific barriers, this paper demonstrates how mixed methods can be used to document the diffusion of a single paper and its ideas, through network analysis and citation analysis. For retracted papers, this detailed assessment of diffusion is justified by the desire to check the spread of misinformation.
In conclusion, to date there are no published high quality RCTs reporting on the effect of omega-3 polyunsaturated fatty acids alone on inflammation and exercise capacity in people with COPD (the aims of the Matsuyama paper) (Fulton et al. 2015b; Scoditti et al. 2019). While the effects of the continued citation of the retracted Matsuyama paper on the research field are difficult to quantify, it is conceivable that, believing this research question to be answered, scientists may have chosen to pursue less-well established research directions, leading to significant delays in determining the true effect. The continued citation of this paper without acknowledgement of the retraction seems to support this notion and possibly will not be resolved until this question is answered in a properly conducted RCT.
Citation impact seems to follow the medical evidence hierarchy: for instance, RCTs receive more citations than any other study type except meta-analyses (Patsopoulos et al. 2005).
We add underlining for emphasis since it can be confusing that assessing articles requires checking notices, and vice versa.
Abdelhamid, A. S., Brown, T. J., Brainard, J. S., Biswas, P., Thorpe, G. C., Moore, H. J., et al. (2018). Omega-3 fatty acids for the primary and secondary prevention of cardiovascular disease. Cochrane Database of Systematic Reviews, 11, CD003177. https://doi.org/10.1002/14651858.CD003177.pub4.
Al-Haidose, A. (2019). Effect of omega-3 polyunsaturated fatty acids on inflammatory biomarkers in chronic obstructive pulmonary disease, Ph.D., University of Salford. http://usir.salford.ac.uk/id/eprint/51509/.
Arnold, G. (2009a, September 22). Trace elements may play role in lung health. Complete Chriropractic Healthcare. Retrieved September 23, 2020, from http://www.completechiropractichealthcare.com/fileupload/NOW%20Foods%20Articles/Lung%20Health/Trace%20Elemnts%20May%20Play%20A%20Role%20In%20Lung%20Health%20-%209.22.09.pdf.
Arnold, G. (2009b, January 13). Flaxseed supplementation increases omega-3 fat levels. Natural Health Research Institute. Retrieved September 23, 2020, from http://www.naturalhealthresearch.org/flaxseed-supplementation-increases-omega-3-fat-levels/.
Arnold, G. (2010, February 19). Fiber again found to help lung health. Complete Chriropractic Healthcare. Retrieved September 23, 2020, from https://pitchingdoc.com/fileupload/NOW%20Foods%20Articles/Lung%20Health/Fiber%20Against%20Found%20to%20Benefit%20Lung%20Health%20-%203.17.10.pdf.
Avenell, A., Stewart, F., Grey, A., Gamble, G., & Bolland, M. (2019). An investigation into the impact and implications of published papers from retracted research: Systematic search of affected literature. British Medical Journal Open, 9(10), e031909. https://doi.org/10.1136/bmjopen-2019-031909.
Azoulay, P., Furman, J. L., Krieger, J. L., & Murray, F. (2015). Retractions. Review of Economics and Statistics, 97(5), 1118–1136. https://doi.org/10.1162/REST_a_00469.
Balhara, Y. P. S., & Mishra, A. (2014). Compliance of retraction notices for retracted articles on mental disorders with COPE guidelines on retraction. Current Science, 107(5), 757–760. https://www.currentscience.ac.in/cs/Volumes/107/05/0757.pdf.
Bar-Ilan, J., & Halevi, G. (2017). Post retraction citations in context: A case study. Scientometrics, 113(1), 547–565. https://doi.org/10.1007/s11192-017-2242-0.
Bar-Ilan, J., & Halevi, G. (2018). Temporal characteristics of retracted articles. Scientometrics, 116(3), 1771–1783. https://doi.org/10.1007/s11192-018-2802-y.
Bornemann-Cimenti, H., Szilagyi, I. S., & Sandner-Kiesling, A. (2016). Perpetuation of retracted publications using the example of the Scott S. Reuben case: Incidences, reasons and possible improvements. Science and Engineering Ethics, 22(4), 1063–1072. https://doi.org/10.1007/s11948-015-9680-y.
Bornmann, L., Wray, K. B., & Haunschild, R. (2020). Citation concept analysis (CCA): A new form of citation analysis revealing the usefulness of concepts for other researchers illustrated by exemplary case studies including classic books by Thomas S. Kuhn and Karl R. Popper. Scientometrics, 122(2), 1051–1074. https://doi.org/10.1007/s11192-019-03326-2.
Bozzo, A., Bali, K., Evaniew, N., & Ghert, M. (2017). Retractions in cancer research: A systematic survey. Research Integrity and Peer Review, 2(1), 5. https://doi.org/10.1186/s41073-017-0031-1.
Brainard, J. (2018). Rethinking retractions. Science, 362(6413), 390–393. https://doi.org/10.1126/science.362.6413.390.
Brand, J. C., Rossi, M. J., & Lubowitz, J. H. (2017). Reliable references, accountable authors, and steadfast reviewers mitigate against retracted publications and scientific misconduct. Arthroscopy: The Journal of Arthroscopic & Related Surgery, 33(8), 1435–1436. https://doi.org/10.1016/j.arthro.2017.05.006.
Budd, J. M., Coble, Z., & Abritis, A. (2016). An investigation of retracted articles in the biomedical literature. Proceedings of the Association for Information Science and Technology, 53(1), 1–9. https://doi.org/10.1002/pra2.2016.14505301055.
Budd, J. M., Sievert, M., & Schultz, T. R. (1998). Phenomena of retraction. Reasons for retraction and citations to the publications. Journal of the American Medical Association, 280(3), 296–297. https://doi.org/10.1001/jama.280.3.296.
Cai, A., Liu, M., Tang, P., Li, J., Xin, M., Yan, M., et al. (2019). Preparation of β-CD-LA aggregate and its application as drug carriers. Materials Technology, 34(10), 626–637. https://doi.org/10.1080/10667857.2019.1611054.
Calder, P. C. (2006). Polyunsaturated fatty acids and inflammation. Prostaglandins Leukotrienes and Essential Fatty Acids, 75(3), 197–202. https://doi.org/10.1016/j.plefa.2006.05.012.
Calder, P. C., Laviano, A., Lonnqvist, F., Muscaritoli, M., Öhlander, M., & Schols, A. (2018). Targeted medical nutrition for cachexia in chronic obstructive pulmonary disease: A randomized, controlled trial. Journal of Cachexia, Sarcopenia and Muscle, 9(1), 28–40. https://doi.org/10.1002/jcsm.12228.
Calder, P. C., & Zurier, R. B. (2001). Polyunsaturated fatty acids and rheumatoid arthritis. Current Opinion in Clinical Nutrition and Metabolic Care, 4(2), 115–121. https://doi.org/10.1097/00075197-200103000-00006.
Chang, C.-H., Tseng, P.-T., Chen, N.-Y., Lin, P.-C., Lin, P.-Y., Chang, J. P.-C., et al. (2018). Safety and tolerability of prescription omega-3 fatty acids: A systematic review and meta-analysis of randomized controlled trials. Prostaglandins Leukotrienes and Essential Fatty Acids, 129, 1–12. https://doi.org/10.1016/j.plefa.2018.01.001.
Chen, C., Hu, Z., Milbank, J., & Schultz, T. (2013). A visual analytic study of retracted articles in scientific literature. Journal of the American Society for Information Science and Technology, 64(2), 234–253. https://doi.org/10.1002/asi.22755.
Chen, C., & Song, M. (2017). Visual analytic observatory of scientific knowledge. In C. Chen & M. Song (Eds.), Representing scientific knowledge: The role of uncertainty (pp. 337–375). Berlin: Springer. https://doi.org/10.1007/978-3-319-62543-0_9.
Cheng, Y.-Y., Parulian, N., Hsiao, T.-K., Dinh, L., Sarol, J., & Schneider, J. (2019). ReTracker: Actively and automatically matching retraction metadata in Zotero. Proceedings of the Association for Information Science and Technology, 56, 372–376. https://doi.org/10.1002/pra2.32.
CHEST. (2008). Retraction [of effects of omega-3 polyunsaturated fatty acids on inflammatory markers in COPD]. Chest, 134(4), 893. https://doi.org/10.1016/S0012-3692(08)60339-6.
Collins, P. F., Yang, I. A., Chang, Y.-C., & Vaughan, A. (2019). Nutritional support in chronic obstructive pulmonary disease (COPD): An evidence update. Journal of Thoracic Disease, 11(Suppl 17), 2230–2237. https://doi.org/10.21037/jtd.2019.10.41.
Cor, K., & Sood, G. (2018, October 26). Propagation of error: Approving citations to problematic research. Unpublished paper (34 pages), retrieved 23 September 2020 from http://www.gsood.com/research/papers/error.pdf.
Davis, P. M. (2012). The persistence of error: A study of retracted articles on the Internet and in personal libraries. Journal of the Medical Library Association, 100(3), 184–189. https://doi.org/10.3163/1536-5050.100.3.008.
De Lacey, G. D., Record, C., & Wade, J. (1985). How accurate are quotations and references in medical journals? Nature, 291, 884–886. https://doi.org/10.1136/bmj.291.6499.884.
Decullier, E., Huot, L., Samson, G., & Maisonneuve, H. (2013). Visibility of retractions: A cross-sectional one-year study. BMC Research Notes. https://doi.org/10.1186/1756-0500-6-238.
Dinh, L., Sarol, J., Cheng, Y.-Y., Hsiao, T.-K., Parulian, N., & Schneider, J. (2019). Systematic examination of pre- and post-retraction citations. Proceedings of the Association for Information Science and Technology, 56, 390–394. https://doi.org/10.1002/pra2.35.
Elia, N., Wager, E., & Tramèr, M. R. (2014). Fate of articles that warranted retraction due to ethical concerns: A descriptive cross-sectional study. PLoS ONE, 9(1), e85846. https://doi.org/10.1371/journal.pone.0085846.
Fanelli, D. (2013). Why growing retractions are (mostly) a good sign. PLoS Medicine, 10(12), e1001563. https://doi.org/10.1371/journal.pmed.1001563.
Fitzpatrick, K. C. (2011). Health benefits of flaxseed. In E. Hernandez & M. Hosokawa (Eds.), Omega-3 Oils: Applications in Functional Foods (pp. 213–264). London: Academic Press and AOCS Press. https://doi.org/10.1016/B978-1-893997-82-0.50013-X.
Fu, Y., & Schneider, J. (2020). Towards knowledge maintenance in scientific digital libraries with the keystone framework. In JCDL '20: Proceedings of the ACM/IEEE Joint Conference on Digital Libraries in 2020, Virtual. https://doi.org/10.1145/3383583.3398514.
Fulton, A. S., Coates, A. M., Williams, M. T., Howe, P. R. C., Garg, M. L., Wood, L. G., et al. (2017). Fish oil supplementation in chronic obstructive pulmonary disease: Feasibility of conducting a randomised controlled trial. Pilot and Feasibility Studies, 3, 66. https://doi.org/10.1186/s40814-017-0211-2.
Fulton, A. S., Coates, A. M., Williams, M. T., Howe, P. R. C., & Hill, A. M. (2015a). Persistent citation of the only published randomised controlled trial of Omega-3 supplementation in chronic obstructive pulmonary disease six years after its retraction. Publications, 3(1), 7–26. https://doi.org/10.3390/publications3010017. Corrected published on 27 July 2018, see Publications 2018, 6(3), 35. https://doi.org/10.3390/publications6030035.
Fulton, A. S., Hill, A. M., Williams, M. T., Howe, P. R. C., & Coates, A. M. (2015b). Paucity of evidence for a relationship between long-chain omega-3 fatty acid intake and chronic obstructive pulmonary disease: A systematic review. Nutrition Reviews, 73(9), 612–623. https://doi.org/10.1093/nutrit/nuv017.
Fulton, A. S., Hill, A. M., Williams, M. T., Howe, P. R., Frith, P. A., Wood, L. G., et al. (2013). Feasibility of omega-3 fatty acid supplementation as an adjunct therapy for people with chronic obstructive pulmonary disease: Study protocol for a randomized controlled trial. Trials, 14(1), 107. https://doi.org/10.1186/1745-6215-14-107.
Furman, J. L., Jensen, K., & Murray, F. (2012). Governing knowledge in the scientific community: Exploring the role of retractions in biomedicine. Research Policy, 41(2), 276–290. https://doi.org/10.1016/j.respol.2011.11.001.
Garfield, E., & Welljams-Dorof, A. (1990). The impact of fraudulent research on the scientific literature: The Stephen E. Breuning case. JAMA, the Journal of the American Medical Association, 263(10), 1424–1426. https://doi.org/10.1001/jama.1990.03440100144021.
Giudetti, A. M., & Cagnazzo, R. (2012). Beneficial effects of n-3 PUFA on chronic airway inflammatory diseases. Prostaglandins & Other Lipid Mediators, 99(3–4), 57–67. https://doi.org/10.1016/j.prostaglandins.2012.09.006.
Gold, D. R., Litonjua, A. A., Carey, V. J., Manson, J. E., Buring, J. E., Lee, I.-M., et al. (2016). Lung VITAL: Rationale, design, and baseline characteristics of an ancillary study evaluating the effects of vitamin D and/or marine omega-3 fatty acid supplements on acute exacerbations of chronic respiratory disease, asthma control, pneumonia and lung function in adults. Contemporary Clinical Trials, 47, 185–195. https://doi.org/10.1016/j.cct.2016.01.003.
Gray, R., Al-Ghareeb, A., Davis, J., McKenna, L., & Amichai Hillel, S. (2018). Inclusion of nursing trials in systematic reviews after they have been retracted: Does it happen and what should we do? International Journal of Nursing Studies, 79, 154. https://doi.org/10.1016/j.ijnurstu.2017.12.006.
Greenberg, S. A. (2009). How citation distortions create unfounded authority: Analysis of a citation network. BMJ, 339, b2680. https://doi.org/10.1136/bmj.b2680.
Grieneisen, M. L., & Zhang, M. (2012). A comprehensive survey of retracted articles from the scholarly literature. PLoS ONE, 7(10), e44118. https://doi.org/10.1371/journal.pone.0044118.
Grimes, D. A., & Schulz, K. F. (2002). An overview of clinical research: The lay of the land. The Lancet, 359(9300), 57–61. https://doi.org/10.1016/S0140-6736(02)07283-5.
Guengerich, F. P. (2015). Some changes in submission and handling policies at the JBC. Journal of Biological Chemistry, 290(46), 28018–28019. https://doi.org/10.1074/jbc.E115.000001.
Halevi, G. (2019). Why articles in arts and humanities are being retracted? Publishing Research Quarterly. https://doi.org/10.1007/s12109-019-09699-9.
Hall, C., III, Fitzpatrick, K. C., & Kamal-Eldin, A. (2009). Flax, perilla, and camelina seed oils: α-linolenic acid-rich oils. In R. Moreau & A. Kamal-Eldin (Eds.), Gourmet and health-promoting specialty oils (pp. 151–183). Amsterdam: Elsevier. https://doi.org/10.1016/B978-1-893997-97-4.50010-3.
Hamilton, D. G. (2019). Continued citation of retracted radiation oncology literature—Do we have a problem? International Journal of Radiation Oncology*Biology*Physics, 103(5), 1036–1042. https://doi.org/10.1016/j.ijrobp.2018.11.014.
Hani, S. Z. (2019). Pengaruh pemberian suplemen omega 3 terhadap kadar TNF-α serum, massa otot, kekuatan otot, dan performa fisik pada pasien PPOK dengan sarkopenia [Effect of omega 3 supplementation on serum TNF-α levels, muscle mass, muscle strength, and physical performance in patients with COPD with sarcopenia], Master’s, Universitas Andalas. http://scholar.unand.ac.id/49966/.
Healthy fats, fish oils & omega-3 supplementation. (2017). ROS Nutrition. Retrieved September 23, 2020, from https://www.rosnutrition.com/ie/insight/knowledge-centre/healthy-fats-fish-oils-and-omega-3-supplementation.
Herzog, R., & Cunningham-Rundles, S. (2011). Immunologic impact of nutrient depletion in chronic obstructive pulmonary disease. Current Drug Targets, 12(4), 489–500. https://doi.org/10.2174/138945011794751500.
International Committee of Medical Journal Editors. (2019, December). Recommendations for the conduct, reporting, editing, and publication of scholarly work in medical journals. Retrieved September 23, 2020, from http://www.icmje.org/icmje-recommendations.pdf.
International Committee of Medical Journal Editors. (n. d.). Journals stating that they follow the ICMJE Recommendations. International Committee of Medical Journal Editors. Retrieved January 2, 2020, from http://www.icmje.org/journals-following-the-icmje-recommendations/#C.
Jackowski, G., MacSweeney, R., Shaikh, N., Yantha, J., & Schini-Kerth, V. (2012a). Cholesterol absorption inhibitor (azetidinone) and omega 3 fatty acids (EPA, DHA and DPA) for the reduction of cholesterol and for the reduction of cardiovascular events (World Intellectual Property Organization Patent No. WO2012112511A1). https://patents.google.com/patent/WO2012112511A1/en.
Jackowski, G., MacSweeney, R., Shaikh, N., Yantha, J., & Schini-Kerth, V. (2012b). Statin and omega 3 fatty acids (EPA, DHA and DPA) for use in cardiovascular diseases (Canada Patent No. CA2827585A1). https://patents.google.com/patent/CA2827585A1/en.
Jackowski, G., MacSweeney, R., Shaikh, N., Yantha, J., & Schini-Kerth, V. (2014). Method for treating obesity with anti-obesity formulations and omega 3 fatty acids for the reduction of body weight in cardiovascular disease patients (CVD) and diabetics (United States Patent No. US8715648B2). https://patents.google.com/patent/US8715648B2/en.
Jackowski, G., MacSweeney, R., Shaikh, N., Yantha, J., & Schini-Kerth, V. (2015). Omega 3 fatty acid for use as a prescription medical food and omega 3 fatty acid diagniostic assay for the dietary management of cardiovascular patients with cardiovascular disease (CVD) who are deficient in blood EPA and DHA levels (United States Patent No. US9119826B2). https://patents.google.com/patent/US9119826B2/.
Jan, R., Bano, S., Ikhlaq ur Rehman, S., & Mehraj, M. (2018). Context analysis of top seven retracted articles: Should Retraction Watch revisit the list? Library Philosophy and Practice, December 1, 2018:1–20. https://digitalcommons.unl.edu/libphilprac/2016/.
Jha, R. (2016). Preparation of Tishyauri and optimization of flaxseed and black gram paste ratio. In Food Research Bulletin (pp. 138–144). Government of Nepal, Ministry of Agricultural Development, Department of Food Technology and Quality Control.
Kim, S. Y., Yi, H. J., Cho, H.-M., & Huh, S. (2019). How many retracted articles indexed in KoreaMed were cited 1 year after retraction notification. Science Editing, 6(2), 122–127. https://doi.org/10.6087/kcse.172.
Kochan, C. A., & Budd, J. M. (1992). The persistence of fraud in the literature: The Darsee case. Journal of the American Society for Information Science, 43(7), 488–493. https://doi.org/10.1002/(SICI)1097-4571(199208)43:7%3C488::AID-ASI3%3E3.0.CO;2-7.
Korpela, K. M. (2010). How long does it take for the scientific literature to purge itself of fraudulent material? The Breuning case revisited. Current Medical Research and Opinion, 26(4), 843–847. https://doi.org/10.1185/03007991003603804.
Lin, Y.-C., Lin, Y.-J., Wu, T.-C., Cheng, C.-C., & Yeh, S.-L. (2016). Supplementation of fish oil improves functional capacity and quality of life in patients with chronic obstructive pulmonary disease. Current Topics in Nutraceutical Research, 14(3), 191–198. https://www.nchpjournals.com/Manuscript?id=550.
Lozano, C. C., Valdez, J. G. C., & Medina, J. R. S. (2011). Terapia nutricional en la enfermedad pulmonar obstructiva crónica [Nutritional therapy in chronic obstructive pulmonary disease]. In R. A. Prado, H. A. Márquez, & D. A. Moya (Eds.), Nutrición enteral y parenteral [Enteral and Parenteral Nutrition]. New York: McGraw-Hill Interamericana.
Madlock-Brown, C. R., & Eichmann, D. (2015). The (lack of) impact of retraction on citation networks. Science and Engineering Ethics, 21(1), 127–137. https://doi.org/10.1007/s11948-014-9532-1.
Marcus, A., & Oransky, I. (2011). The paper is not sacred. Nature, 480(7378), 449–450. https://doi.org/10.1038/480449a.
Matsuyama, W., Mitsuyama, H., Koreeda, Y., Higashimoto, I., Osame, M., & Arimura, K. (2008). RETRACTION: Use of tiotropium bromide for pre-operative treatment in chronic obstructive pulmonary disease patients: comparison with oxitropium bromide. Internal Medicine, 47(14), 1371. https://doi.org/10.2169/internalmedicine.47.R001.
Matsuyama, W., Mitsuyama, H., Watanabe, M., Oonakahara, K., Higashimoto, I., Osame, M., et al. (2005). [RETRACTED] Effects of omega-3 polyunsaturated fatty acids on inflammatory markers in COPD. Chest, 128(6), 3817–3827. https://doi.org/10.1378/chest.128.6.3817.
Monk, J. M., McMurnay, D. N., & Chapkin, R. S. (2011). Clinical effects of n-3 PUFA supplementation in human health and inflammatory diseases. In E. Hernandez & M. Hosokawa (Eds.), Omega-3 oils: Applications in functional foods (pp. 213–264). London: Academic Press and AOCS Press. https://doi.org/10.1016/B978-1-893997-82-0.50005-0.
Mott, A., Fairhurst, C., & Torgerson, D. (2019). Assessing the impact of retraction on the citation of randomized controlled trial reports: An interrupted time-series analysis. Journal of Health Services Research & Policy, 24(1), 44–51. https://doi.org/10.1177/1355819618797965.
Moylan, E. C., & Kowalczuk, M. K. (2016). Why articles are retracted: A retrospective cross-sectional study of retraction notices at BioMed Central. British Medical Journal Open, 6(11), e012047. https://doi.org/10.1136/bmjopen-2016-012047.
Nagahama, T., Sato, N., Schols, A., Steiner, M., & Pison, C. (2013). Topic 38 呼吸器疾患における栄養サポート [Topic 38: Prevalence and mechanisms of weight loss and muscle wasting in COPD]. 静脈経腸栄養 [Parenteral Nutrition], 28(3), 795–818. https://doi.org/10.11244/jjspen.28.795.
NCT01728571: LungVITamin D and OmegA-3 Trial (lungVITAL). (n.d.). ClinicalTrials.Gov. Retrieved September 23, 2020, from https://clinicaltrials.gov/ct2/show/NCT01728571. Last updated May 12, 2020.
Neale, A. V., Dailey, R. K., & Abrams, J. (2010). Analysis of citations to biomedical articles affected by scientific misconduct. Science and Engineering Ethics, 16(2), 251–261. https://doi.org/10.1007/s11948-009-9151-4.
Neale, A. V., Northrup, J., Dailey, R., Marks, E., & Abrams, J. (2007). Correction and use of biomedical literature affected by scientific misconduct. Science and Engineering Ethics, 13(1), 5–24. https://doi.org/10.1007/s11948-006-0003-1.
Nguyen, H. T. (2019). The effectiveness of tailored dietary counselling in treating malnutrition in Vietnamese COPD outpatients: A randomised controlled trial, Ph.D., Queensland University of Technology. https://eprints.qut.edu.au/135061/.
Nordgren, T. M., Heires, A. J., Bailey, K. L., Katafiasz, D. M., Toews, M. L., Wichman, C. S., et al. (2018). Docosahexaenoic acid enhances amphiregulin-mediated bronchial epithelial cell repair processes following organic dust exposure. American Journal of Physiology-Lung Cellular and Molecular Physiology, 314(3), L421–L431. https://doi.org/10.1152/ajplung.00273.2017.
Ogasawara, T., Marui, S., Miura, E., Sugiura, M., Matsuyama, W., Aoshima, Y., et al. (2018). Effect of eicosapentaenoic acid on prevention of lean body mass depletion in patients with exacerbation of chronic obstructive pulmonary disease: A prospective randomized controlled trial. Clinical Nutrition ESPEN, 28, 67–73. https://doi.org/10.1016/j.clnesp.2018.09.076.
Omar, Z. A., Montser, B. A., & Farahat, M. A. R. (2019). Effect of high-dose Omega 3 on lipid profile and inflammatory markers in chronic hemodialysis children. Saudi Journal of Kidney Diseases and Transplantation, 30(3), 634. https://doi.org/10.4103/1319-2442.261337.
Patsopoulos, N. A., Analatos, A. A., & Ioannidis, J. P. A. (2005). Relative citation impact of various study designs in the health sciences. JAMA, 293(19), 2362–2366. https://doi.org/10.1001/jama.293.19.2362.
Peterson, G. M. (2013). Characteristics of retracted open access biomedical literature: A bibliographic analysis. Journal of the American Society for Information Science and Technology, 64(12), 2428–2436. https://doi.org/10.1002/asi.22944.
Pfeifer, M. P., & Snodgrass, G. L. (1990). The continued use of retracted, invalid scientific literature. JAMA, 263(10), 1420–1423. https://doi.org/10.1001/jama.1990.03440100140020.
Pfeifer, M. P., & Snodgrass, G. L. (1992). Medical school libraries’ handling of articles that report invalid science. Academic Medicine, 67(2), 109–113. https://doi.org/10.1097/00001888-199202000-00014.
Pison, C., Pichard, C., Moniard, C., Roth, H., & Cano, N. (2016). Nutrition et maladie respiratoire chronique [Nutrition and chronic respiratory disease]. In D. Quilliot, R. Thibault, P. Bachmann, E. Guex, G. Zéanandin, & P. Coti-Bertrand (Eds.), Traité de Nutrition Clinique à Tous les âges de la Vie [Treatise on Clinical Nutrition at All Ages of Life] (2016th ed., pp. 1165–1182). Le Kremlin-Bicetre: Editions K’Noë.
Ran, N., Pang, Z., Gu, Y., Pan, H., Zuo, X., Guan, X., et al. (2019). An updated overview of metabolomic profile changes in chronic obstructive pulmonary disease. Metabolites, 9(6), 111. https://doi.org/10.3390/metabo9060111.
Redman, B. K., Yarandi, H. N., & Merz, J. F. (2008). Empirical developments in retraction. Journal of Medical Ethics, 34(11), 807–809. https://doi.org/10.1136/jme.2007.023069.
Renear, A. H., & Palmer, C. L. (2009). Strategic reading, ontologies, and the future of scientific publishing. Science, 325(5942), 828–832. https://doi.org/10.1126/science.1157784.
Retraction Watch. (n.d.). Retraction Watch Database. Retrieved September 23, 2020, from http://retractiondatabase.org/. Version: 18.104.22.168. ISSN 2692-465X.
Rubbo, P., Pilatti, L. A., & Picinin, C. T. (2018). Citation of retracted articles in engineering: A study of the Web of Science database. Ethics and Behavior, 29(8), 661–679. https://doi.org/10.1080/10508422.2018.1559064.
Samp, J., Schumock, G., & Pickard, A. (2012). Retracted publications in the drug literature. Pharmacotherapy, 32(7), 586–595. https://doi.org/10.1002/j.1875-9114.2012.01100.x.
Samuel, S., Peskin, B., Arondekar, B., Alperin, P., Johnson, S., Blumenfeld, I., et al. (2011). Estimating health and economic benefits from using prescription omega-3 fatty acids in patients with severe hypertriglyceridemia. The American Journal of Cardiology, 108(5), 691–697. https://doi.org/10.1016/j.amjcard.2011.04.019.
Schmidt, M. (2018). An analysis of the validity of retraction annotation in PubMed and the Web of Science. Journal of the Association for Information Science and Technology, 69(2), 318–328. https://doi.org/10.1002/asi.23913.
Schneider, J., Ye, D., & Hill, A. (2019). Second-generation citation context analysis (2010–2019) to retracted paper Matsuyama 2005. Databank, University of Illinois at Urbana-Champaign. https://doi.org/10.13012/B2IDB-3331845_V2.
Schols, A. (2008). Module 38.1: Prevalence and mechanisms of weight loss and muscle wasting in COPD. In European Society for Clinical Nutrition and Metabolism, Nutritional Support in Respiratory Diseases [Topic 38]. http://lllnutrition.com/mod_lll/TOPIC38/old_version/m381.pdf.
Schols, A. (2013). Module 38.1: Mechanisms and consequences of body composition abnormalities in chronic respiratory diseases. In European Society for Clinical Nutrition and Metabolism, Nutritional Support in Pulmonary Disease [Topic 38]. http://lllnutrition.com/mod_lll/TOPIC38/Old_Version_2017/m381.pdf.
Schols, A. (2017). Module 38.1: Mechanisms of wasting in chronic respiratory diseases and the impact of disease exacerbations. In European Society for Clinical Nutrition and Metabolism, Nutritional Support in Pulmonary Diseases [Topic 38]. http://lllnutrition.com/mod_lll/TOPIC38/m381.pdf.
Scoditti, E., Massaro, M., Garbarino, S., & Toraldo, D. M. (2019). Role of diet in chronic obstructive pulmonary disease prevention and treatment. Nutrients. https://doi.org/10.3390/nu11061357.
Shwed, U., & Bearman, P. S. (2010). The temporal structure of scientific consensus formation. American Sociological Review, 75(6), 817–840. https://doi.org/10.1177/0003122410388488.
Small, H. (1982). Citation context analysis. In B. Cervin & M. J. Voigt (Eds.), Progress in communication sciences (Vol. 3, pp. 287–310). Norwood: Ablex Publishing Corporation..
Stang, A., Jonas, S., & Poole, C. (2018). Case study in major quotation errors: A critical commentary on the Newcastle-Ottawa scale. European Journal of Epidemiology, 33(11), 1025–1031. https://doi.org/10.1007/s10654-018-0443-3.
Steen, R. G. (2011a). Retractions in the scientific literature: Do authors deliberately commit research fraud? Journal of Medical Ethics, 37(2), 113–117. https://doi.org/10.1136/jme.2010.038125.
Steen, R. G. (2011b). Retractions in the medical literature: How many patients are put at risk by flawed research? Journal of Medical Ethics, 37(11), 688–692. https://doi.org/10.1136/jme.2011.043133.
Stillman, D. (2019, June 14). Retracted item notifications with Retraction Watch integration. Zotero Blog. https://www.zotero.org/blog/retracted-item-notifications/.
Suelzer, E. M., Deal, J., Hanus, K. L., Ruggeri, B., Sieracki, R., & Witkowski, E. (2019). Assessment of citations of the retracted article by Wakefield et al with fraudulent claims of an association between vaccination and autism. JAMA Network Open, 2(11), e1915552. https://doi.org/10.1001/jamanetworkopen.2019.15552.
Sun, B., Luo, C., Cui, W., Sun, J., & He, Z. (2017). Chemotherapy agent-unsaturated fatty acid prodrugs and prodrug-nanoplatforms for cancer chemotherapy. Journal of Controlled Release, 264, 145–159. https://doi.org/10.1016/j.jconrel.2017.08.034.
Tatsioni, A., Bonitsis, N. G., & Ioannidis, J. P. A. (2007). Persistence of contradicted claims in the literature. JAMA, 298(21), 2517–2526. https://doi.org/10.1001/jama.298.21.2517.
Teixeira da Silva, J. A., & Bornemann-Cimenti, H. (2017). Why do some retracted papers continue to be cited? Scientometrics, 110(1), 365–370. https://doi.org/10.1007/s11192-016-2178-9.
Tenopir, C., King, D. W., Edwards, S., & Wu, L. (2009). Electronic journals and changes in scholarly article seeking and reading patterns. Aslib Proceedings, 61, 5–32. https://doi.org/10.1108/00012530910932267.
Retraction Watch Leaderboard. (2018, May 30). Retraction Watch. Archived copy from Internet Archive Wayback Machine. https://web.archive.org/web/20180530113326/https://retractionwatch.com/the-retraction-watch-leaderboard/.
Thesing, C. S., Bot, M., Milaneschi, Y., Giltay, E. J., & Penninx, B. W. J. H. (2018). Omega-3 and omega-6 fatty acid levels in depressive and anxiety disorders. Psychoneuroendocrinology, 87, 53–62. https://doi.org/10.1016/j.psyneuen.2017.10.005.
van de Bool, C., Rutten, E. P. A., van Helvoort, A., Franssen, F. M. E., Wouters, E. F. M., & Schols, A. M. W. J. (2017). A randomized clinical trial investigating the efficacy of targeted nutrition as adjunct to exercise training in COPD. Journal of Cachexia, Sarcopenia and Muscle, 8(5), 748–758. https://doi.org/10.1002/jcsm.12219.
van der Vet, P. E., & Nijveen, H. (2016). Propagation of errors in citation networks: A study involving the entire citation network of a widely cited paper published in, and later retracted from, the journal Nature. Research Integrity and Peer Review, 1, 3. https://doi.org/10.1186/s41073-016-0008-5.
Vuong, Q.-H. (2020). The limitations of retraction notices and the heroic acts of authors who correct the scholarly record: An analysis of retractions of papers published from 1975 to 2019. Learned Publishing. https://doi.org/10.1002/leap.1282.
Wager, E., Barbour, V., Kleinert, S., & Yentis, S. (2009). COPE guidelines for retracting articles . Committee on Publication Ethics.
Wager, E., Barbour, V., Kleinert, S., & Yentis, S. (2019). COPE Guidelines for retracting articles . Committee on Publication Ethics. https://doi.org/10.24318/cope.2019.1.4.
Wang, X., & Wu, J. (2019). Modulating effect of fatty acids and sterols on skin aging. Journal of Functional Foods, 57, 135–140. https://doi.org/10.1016/j.jff.2019.04.011.
Wetterer, J. K. (2006). Quotation error, citation copying, and ant extinctions in Madeira. Scientometrics, 67(3), 351–372. https://doi.org/10.1556/Scient.67.2006.3.2.
Whitely, W. P., Rennie, D., & Hafner, A. W. (1994). The scientific community’s response to evidence of fraudulent publication: The Robert Slutsky case. JAMA, 272(2), 170–173. https://doi.org/10.1001/jama.1994.03520020096029.
Wray, K. B., & Andersen, L. E. (2018). Retractions in science. Scientometrics, 117(3), 2009–2019. https://doi.org/10.1007/s11192-018-2922-4.
Wright, N. D. (1991). A citation context analysis of retracted scientific articles, Ph.D., University of Maryland, College Park. ProQuest Dissertations Publishing. 9133185.
Wright, K., & McDaid, C. (2011). Reporting of article retractions in bibliographic databases and online journals. Journal of the Medical Library Association: JMLA, 99(2), 164–167. https://doi.org/10.3163/1536-5050.99.2.010.
Ye, D., Hill, A., Whitehorn, A., & Schneider, J. (2020). Citation context annotation for new and newly found citations (2006-2019) to retracted paper Matsuyama 2005. Databank, University of Illinois at Urbana-Champaign. https://doi.org/10.13012/B2IDB-8150563_V1.
Ye, D., & Schneider, J. (2020). Network of first and second-generation citations to Matsuyama 2005 from Google Scholar and Web of Science. Databank, University of Illinois at Urbana-Champaign. https://doi.org/10.13012/B2IDB-1403534_V2.
Conceptualization: Jodi Schneider; Data curation—Network Analysis: Di Ye (lead), Jodi Schneider (supporting); Data curation—First Generation Citation Context Analysis: Jodi Schneider (lead) and Di Ye (supporting); Data curation—Second Generation Citation Context Analysis: Jodi Schneider (lead) and Di Ye (supporting) and Alison Hill (supporting—confirmation of some annotation); Formal analysis: Di Ye (lead) and Jodi Schneider (supporting); Methodology: Jodi Schneider (lead), Di Ye (supporting), Alison Hill (supporting) and Ashley Whitehorn (supporting); Investigation: Di Ye (co-lead) and Jodi Schneider (co-lead) and Alison Hill (supporting—confirmation of annotation) and Ashley Whitehorn (supporting—confirmation of annotation); Project administration—Jodi Schneider; Resources—Jodi Schneider; Software—Di Ye; Visualization—Di Ye (lead) and Jodi Schneider (supporting); Supervision: Jodi Schneider; Writing—original draft preparation: Jodi Schneider (lead) and Di Ye (supporting); Writing—review and editing: Jodi Schneider, Alison Hill and Ashley Whitehorn. https://www.casrai.org/credit.html. Thanks to the anonymous reviewers and to Halle Burns, Kiel Gilleade, Hoa Luong at the Illinois DataBank, Ly Dinh, Nur Aini Rakhmawati, Yohta Shimizu, and Linda Smith. This study could not have been completed without the assistance of the Interlibrary Loan Office at the University of Illinois Urbana-Champaign; thanks also to the library’s Sharon Han for Web of Science assistance. JS also gratefully acknowledges the University of Illinois at Chicago and the University of Pittsburgh for facilitating access to their library holdings, and the assistance of Danny Miller and Helena VonVille at the Health Sciences Library System at the University of Pittsburgh. Alfred P. Sloan Foundation G-2020-12623. NIH R01LM010817.
Conflict of interest
Jodi Schneider has been an invited speaker for the publisher organization CrossRef; no other conflicts of interest. Di Ye, Alison M. Hill, and Ashley S. Whitehorn have no conflicts of interest to declare.
Electronic supplementary material
Below is the link to the electronic supplementary material.
About this article
Cite this article
Schneider, J., Ye, D., Hill, A.M. et al. Continued post-retraction citation of a fraudulent clinical trial report, 11 years after it was retracted for falsifying data. Scientometrics 125, 2877–2913 (2020). https://doi.org/10.1007/s11192-020-03631-1
- Post-retraction citation
- Problems with bibliographic libraries
- Problematic citation
- Diffusion studies
- Citation context analysis
- Link resolver errors
- Case study
Mathematics Subject Classification