Introduction

Systematic reviews are regarded as the best sources of evidence to guide clinical decisions, health care policies, and the development of clinical guidelines (Eden, 2011; Page, 2016; Sackett, 1996;). Cochrane reviews are internationally recognized for providing some of the best quality in evidence-based health care as they follow a very strict methodology compared with other systematic reviews (Bhambhvani et al. 2021; Bunn et al. 2015; Higgins, 2022; Mascarenhas et al. 2020; Masic et al. 2008; Matthias et al. 2020). With an Impact Factor (IF) of 9.3 for the Cochrane Database of Systematic reviews (CDSR), Cochrane has a high impact within the field of medical research (Clarivate Analytics, 2022). Also, Cochrane reviews were used in 90% of World Health Organization (WHO) guidelines from 2016 (Cochrane, 2016) and most NICE guidelines from 2019 (Cochrane, 2016), as well as being the most referenced journal on Wikipedia (Jemielniak, 2019). To ensure this authoritative position within the field of medical research Cochrane reviews should contain the most recent evidence of the highest quality. Therefore, Cochrane has a policy that authors of Cochrane systematic reviews should update their reviews when needed based on the importance of the reviews question to decision-makers and the availability of new data or methods that would have a meaningful impact on the review’s findings (Higgins, 2022).

The median survival time of systematic reviews, before showing a quantitative or qualitative signal for need of updating, was 5.5 years in a previous study of 100 systematic reviews (Shojania et al. 2007). Another study confirmed that more than half of the Cochrane reviews published before 2015 had not been updated for more than 5.5 years and defined these as inactive (Hoffmeyer, 2021). This is not in line with one of Cochrane’s key principles: to provide up-to-date evidence (Higgins, 2022). If these inactive Cochrane reviews are still used to guide treatment and research, it may contribute to the distribution of outdated evidence to other research articles, guidelines and decision-makers (Higgins, 2022). It can therefore affect patients negatively if Cochrane reviews are not kept up to date (Murad, 2017). Quantitative citation analysis can be used to evaluate the extent to which scientific articles are distributed in academic circles and citer motive analysis can provide a more nuanced understanding of how a cited article is used in another scientific paper. In this study, the citer motive analysis provides information on the risk of transmitting outdated evidence when inactive Cochrane reviews are cited. Citing motives can be multiple: The most important motives are augmentation, scientific tradition, social alignment, credit, data, and methodology (Erikson and Erlandson 2014; Nicolaisen, 2017; Sugimoto, 2016).

Citations can be assessed through different sources (Anker et al. 2019; Gusenbauer, 2019) including The Lens (Lens Version 8.2.1, 2022; Jefferson, 2019). The Lens is a free database of scientific papers and their metadata combines data from PubMed, Crossref, Microsoft Academic, and Core (The Lens, 2022; Penfold, 2020). With over 225 million scholarly works and their metadata, it is one of the largest citations indexes and contains most hits after Google Scholar (Gusenbauer, 2019; The Lens, 2022).

The aim of this study was to investigate if inactive Cochrane reviews (> 5.5 years old) were still being used in academic circles and cited in other scientific papers without being cited as outdated.

Methods

To investigate the use of inactive Cochrane reviews in academia, a citation analysis was conducted for all reviews available in the CDSR (Cochrane Library, 2022) and linked with citation data from The Lens (Lens Version 8.2.1). The study was reported in accordance with the REporting of studies Conducted using Observational Routinely-collected health Data (RECORD) Statement (Nicholls et al. 2015). All data were publicly available, and the study did not require any permissions. Data were enabled by The Lens through a scholar Application Programming Interface (API) access.

All Cochrane reviews in the CDSR (Cochrane Library, 2022) were included in the study. They were linked through their Digital Object Identifier (DOI) (International DOI Foundation, 2017) with citation information from The Lens. Articles from The Lens without DOI, duplicates of DOIs, or reviews where two different articles and their citations were counted under the same Lens ID were excluded. We also excluded withdrawn Cochrane reviews and Cochrane reviews without citation data from The Lens. Cochrane reviews were defined as inactive if there was no revision event indicated in “Version History” or no publication of a new version of the review within the last 5.5 years (Hoffmeyer et al. 2021). The study size was determined by the number of reviews available in the CDSR that fulfilled the inclusion criterion and none of the exclusion criteria.

The primary outcome of this study was an analysis of the citation patterns over time since last update of inactive Cochrane reviews based on citation count from The Lens. The secondary outcome was a characterization of the general citation pattern over time of Cochrane reviews per year after publication and an analysis of the citing motives of the most outdated reviews..

In the CDSR, all versions of all Cochrane reviews were identified and their DOI, title, and publication date were extracted in August 2020. We were provided with a scholar API to The Lens that allowed us to access data directly from the database (Lens Version 8.2.1, 2022; The Lens, 2021). In The Lens, all Cochrane reviews were extracted by searching for publications in journals that included “Cochrane” in the title to ensure that also reviews published from abbreviations of “Cochrane Database of Systematic Reviews” were included. We extracted information from all publications in the Lens in January 2021. The extracted information included: Lens ID, title, publication date, DOI from all versions of reviews listed under each Lens ID, number of total citations for each Lens ID, and year of publication of each citation.

When linking information from CDSR and The Lens using DOI, some circumstances had to be considered: The Lens combines several versions of reviews, their DOI, and their citations under the same Lens ID. We therefore gathered citations of all versions of a Cochrane review under a short version of the review’s DOI by leaving out the five last characters of the DOI (e.g..pub2), which gave information about the version of the review. We then had a unique identifier for each review that combined all versions of the review. The short DOI was used when linking citation information from The Lens with the Cochrane reviews in the CDSR. When the number of citations for several versions of the Cochrane review had not been summed in The Lens, we combined them. When there were several versions of the Cochrane review, the citations were grouped under the publication date of the first version of the review. The age of a citation was defined as the difference between the year of publication of the citing reference and the publication date of the first version of the Cochrane review. The age since last update was defined as the difference between the year of last update in “Version history” in the Cochrane library or publication of a new version of the review. We focused on citations from first to 12th year after last update as very few reviews were cited after this, and thus were not representative for the general trend in citation of Cochrane reviews. Some citations were counted in The Lens before the publication data of the first version of the review. These citations were not included in the citation analysis.

To adjust for inflation in citation over time, we summarized the citations for reviews published the same year and calculated the mean number of citations of reviews. Furthermore, based on earlier studies’ attempt to prevent the results from being affected by citation inflation, we stratified reviews in five-year periods, according to their first publication date (Galiani & Gàlvez 2017).

A subgroup analysis was made of the citing motives for the 20 most outdated reviews, i.e. the reviews with the most time passed since their last update. These were all last updated in February 2008. We identified all their most recent citations the past three years (according to the date of data extraction, i.e. 2018–2020) and analyzed their motives for citing the reviews. We chose this subgroup as we hypothesized that these ciations were the most likely to mark the reviews as outdated. Two authors independently screened all citing documents for the precise mention of the review within the text. Each citation was categorized as follows:

  • Positive: indicated that the review was cited as legitimate prior work and its findings were used to corroborate the citing document.

  • Negative: indicated that the review was cited either directly or indirectly as being inactive, no longer up-to-date or that the finding no longer pertains to current knowledge.

  • Not cited: citation of the reviews was not found in the citing document, or the document cites another version of the review.

This categorization was based on a previous study of citing motives (Bar-Ilan & Halevi 2017). We chose this categorization as it enabled us to approach the risk of transmitting outdated information when citing an inactive review. After independent screening of the citations by two authors, the two authors went through conflicts and came to an agreement for each citation.

The statistical analyses were performed using IBM SPSS Statistics 2 version 25 (IBM Corp., 2019, Armonk, New York) and Microsoft Excel version 16.46 (2021, Microsoft). Graphs were made in.

RStudio version 09.0 (2021, Boston). Categorical data were expressed as percentages with their numerators and denominators. We examined the inactive reviews separately to compare them with the rest of the Cochrane reviews.

Results

We included 7,729 Cochrane reviews published in CDSR from 1995 to 2020. A flow diagram depicting the number of eligible and the number of included Cochrane reviews is presented in Fig. 1. In total, 3,735 inactive and 3,994 active Cochrane reviews were included for analysis. For the subgroup analysis of citing motives, 245 citing documents were included.

Fig. 1
figure 1

Flowchart showing the Cochrane reviews included in the study. CDSR: Cochrane Database of Systematic Review. DOI: Digital object identifier. Short DOI: DOI without the five last characters of the DOI (e.g..pub2) that gives information about the version of the Cochrane review. The short DOI was used as a unique identifier that combined all versions of a review

Full size image

The distribution of citations according to years since last update of Cochrane reviews are shown in Table 1. In 2020, 41% of all citations of Cochrane reviews were citations of reviews that had not been updated for more than 5 years. The group of Cochrane reviews that had had no update the past 5.5 years, thus defined inactive, represented 48% of all Cochrane reviews (Fig. 2).

Table 1 Summary of the total number of citations in 2020 of the 7,729 Cochrane reviews in the Cochrane Database of Systematic Reviews (CDSR) based on citation count in The Lens [18].
Full size table
Fig. 2
figure 2

Citation of inactive Cochrane reviews per year after last update of either a new version of the reviews or update in “Version history”. Reviews are stratified in 5-year intervals according to their publication date of first version of the reviews. Inactive reviews definition: reviews without update in “Version history” in the Cochrane Library or no publication of a new version of review the past 5.5 years. Source of citation data: The Lens [18].

Full size image

The development of the mean number of citations per year after last update of the inactive Cochrane reviews is depicted in Fig. 3. On average, the inactive reviews were cited 8.6 times five years after last activity. Ten years after last update, 1,899 reviews were still cited with mean number of 4.3 citations in the 10th year.

Fig. 3
figure 3

Citations of all Cochrane reviews per year after publication of first version of reviews, stratified in 5-year intervals according to publication date of first version of the reviews. Source of citation data: The Lens [18]

Full size image

Figure 3 visualizes the citation pattern for all Cochrane reviews. It shows that newer Cochrane reviews were cited faster than the reviews first published before 2005. Cochrane reviews from 2011 to 2015 reached a plateau in the number of citation three years after publication whereas reviews from 2001 to 2005 reached a plateau in the number of citations 12 years after publication (Table 2). Likewise, the mean number of citations 2 years after publication was 7.2 for Cochrane reviews published from 2006 to 2018 and 1.5 for Cochrane reviews published from 1995 to 2005.

Table 2 Summary of citations of Cochrane reviews based on citation count in The Lens [18].
Full size table

In the citing motive analysis, we found that the 20 most inactive reviews (all last updated in February 2008) had 245 citations in total between 2018 and 2020. Among these, 225 (92%) citations were positive, meaning that they cited the review as legitimate prior work, and used its findings to corroborate the citing document. Only eight documents were negative as six mentioned the year of publication of the review in the text, one cited the findings of the reviews as being “endorsed previously” and another cited the reviews finding as from “the past decade”. Ten citing documents did not actually cite the review or cited another version of the review.

Discussion

We analyzed all available Cochrane reviews and found that 41% of all citations of Cochrane reviews in 2020 cited Cochrane reviews that had not been updated for more than 5.5. years and were thus defined as inactive. Furthermore, a quarter of all Cochrane reviews were still cited 10 years after last update. Our citation analysis study also showed that newer Cochrane reviews were cited more often the first three years after publication compared with older reviews, thus confirming a general trend of inflation in citation.

Furthermore, our citation analysis showed that the Cochrane reviews that have not been updated for more than 5.5 years, so-called inactive reviews, were still used in academia and that a considerable amount of these continued to be cited in other academic papers up to 10 years or even longer after last update. Our analysis of the citing motives of the 20 most inactive reviews showed that even 10–12 years after last update, none of the citing documents marked the review as being out of date, and most of them cited the review as legitimate prior work and used its findings to corroborate the citing document. This could be problematic if they contain outdated evidence, i.e. conclusions that would be altered in an updated version of the Cochrane review as it could contribute to distribution of outdated evidence. Even though the inactive Cochrane reviews might still contain the most recent evidence without being updated for 5.5 years (Shojania et al. 2007), hence still be timely even though they are old (Woolwine & Lloyd 2010). If authors find that a Cochrane review is still timely when they are considering the need of update, they should state this in “Version history” in the CDSR. Nevertheless, for the inactive reviews, we could not know if they were timely, as the authors had not made an update in” Version history” in CDSR. Also, inactive Cochrane reviews would not impose a problem or any risk of dissemination of out-dated evidenve if they were not cited in other research papers or if, when cited, they were marked as inactive (negative citation). Nevertheless, this was neither the case and the risk of dissemination outdated evidence might be considered as present.

We also found that Cochrane reviews followed the presumed life cycle of citations of scientific articles: an initial rise after publication, eventually a peak in total citation after a couple of years and then a decline as newer papers supplant the older article (Galiani & Gàlvez 2017). Our analysis found that for the last 20 years, Cochrane reviews have been cited increasingly faster, and the peak in citations have been reached earlier after publication, thereby following the general trend of inflation in citation of academic literature. This is also reflected in Cochrane’s rising IF through the years, as it has risen from 5.7 in 2009 to 7.9 in 2019 (Clarivate Analytics, 2022). Compared with other high-impact medical journals, CDSR was the 11th most cited in 2020 (Clarivate Analytics, 2022).

A strength of this citation analysis study was that we included citation data from 92% of the Cochrane reviews in the entire CDSR and analyzed all their citations. Thus, we had a large amount of citation data and thereby minimized the risk of selection bias. To increase the transparency of the reporting, the study was reported in accordance with the RECORD guidelines since no specific reporting guidelines exist for citation analyses. To adjust for inflation in citation, we stratified our results into 5-year intervals. To elaborate on the risk of transmitting outdated information, we made an analysis of the citing motives for the newest citations of the oldest group of reviews. For instance, if an updated version of a review cited an older inactive version of the review or if an inactive review was cited with a statement that more recent research was not available, it would not impose any problems with distribution of outdated evidence. Nevertheless, this was not the case, and most inactive reviews were cited as legitimate prior work and used to corroborate the citing document. This increases the risk of dissemination of outdated information.

We chose The Lens as our source for citation count as it is the search engine that contains the second most citations, but only includes scientific material, and is thus a reliable way to clarify the extent to which Cochrane reviews were used in academia (Gusenbauer, 2019; Nicholls et al. 2015). One limitation in the use of The Lens is that few other citation analyses have used this search engine, and our results are thus not directly comparable with these other studies. Another limitation of our study was that we had to pool all versions of reviews and sum their citations. This might affect the curves of Fig. 3, by showing a falsely longer time to peak in citation as there might be a peak in citation about 3 years after publication of each new version of a Cochrane review. However, as 63% of all Cochrane reviews are only published once (Hoffmeyer et al. 2021), most of the Cochrane reviews in our analysis were not affected by this bias. Another limitation of the study was that many factors impact the number of citations of a paper, e.g. institution, author of the publication, the Matthew effect, which is seen when studies that are highly cited become more visible in public and are therefore cited even more (Petersen et al. 2011), and general inflation in citation (Bornmann and Mutz 2015; Petersen et al. 2019; Pan et al. 2018). We adjusted for inflation in citation in our analysis by stratifying reviews into 5-years intervals according to their year of publication. As newer reviews have not yet reached their peak in citation in the public and the oldest reviews have had a longer time to reach a higher citation count, we may see a time bias in our study. Newer articles that refer to older articles must be published before it makes sense to count an article’s citation (AU Library, 2021). This normally takes 3–4 years, and therefore we saw a low number of citations from 2016 to 2020. A last limitation of this study was, that it did not take the publication lag into account. Some authors might cite Cochrane reviews when they are still active, but at the time of the publication, the Cochrane review might be outdated. Cochrane reviews also suffer from publication lag. A study showed that the median time to publication of a Cochrane reviews was 2 years after the protocol, and 11% of Cochrane reviews were published more than 5 years after the protocol was made (Andersen et al. 2019). Some Cochrane reviews thereby risk being outdated already at the time of publication or shortly after. To handle this problem Cochrane is moving towards more living systematic reviews that adopt a continual updating process (Higgins, 2022).

As a previous study showed that a large amount of inactive Cochrane reviews exists in the CDSR (Hoffmeyer et al. 2021), this citation analysis investigated the extent to which these were continuously used in academia. Citations of old Cochrane reviews could potentially be problematic as Cochrane is to some extent regarded as an authoritative base for knowledge due to their rigid methodology. Thus, one might be concerned that the use and citation of inactive reviews could lead to a distribution of outdated information in academia. Furthermore, the number of citations is used as a quality stamp of academic papers, e.g. used to calculate the impact factor of scientific journals (Petersen et al. 2011). Search engines like Google Scholar are highly affected by the number of citations, which contributes to the Matthew effect (Bornmann & Mutz 2015). To assess the consequences of citation of inactive reviews, further studies are needed to evaluate if the inactive reviews contain outdated information or if conclusions would have changed if the reviews had been updated. If this is the case it might lead to the distribution of outdated evidence and this could challenge the validity of Cochrane reviews. In the meantime, authors of Cochrane reviews are strongly advised to update their reviews regularly or at least supply information to the “Version history” tab if a formal update is not needed.

In conclusion, this citation analysis showed that inactive Cochrane reviews continue to be used in other research papers for many years even though the Cochrane reviews have not been updated. This might contribute to distribution of outdated evidence and could challenge the validity of Cochrane reviews. Further studies are needed to assess the consequences of citation of inactive reviews and to clarify obstacles of updating Cochrane reviews and the possible ways of motivating authors to update their reviews.