Background

Systematic reviews can facilitate decision-making by relevant professionals, policy makers and the public and they need to be kept up to date if new studies would change their conclusions. Reviews that are out of date might provide misleading information [1] due to the absence of the latest evidence, and cause cascading damage to the quality of research [2]. Since its inception, the Cochrane Collaboration (now, Cochrane) has been dedicated to the periodic updating of Cochrane Reviews [1] with an update defined as “a new edition of a published Cochrane review with changes that can include new data, new methods, or new analyses to the previous edition” [3]. These updates include an updated search for eligible studies [4] and are marked to indicate whether or not the updating led to a change in the review’s conclusions [4]. An update of a Cochrane Review can provide users with the latest outcomes or conclusions from the evidence and, even without any changes in the results or conclusions, the update can reassure users that no recent evidence is missing [1]. However, the original Cochrane Review might continue to be cited and used after the update is published. Bodil et al. found that twenty-five percent of Cochrane reviews were still cited 10 years after last update and were on average cited 4.3 times in the 10th year [5].

Cochrane allows co-publication of Cochrane Reviews in other journals in certain circumstances [6]. Co-publication is not a duplicate publication or academic misconduct, but rather the co-published version is a secondary publication of the Cochrane Review [7], and should be peer reviewed and edited according to the co-publishing journal’s editorial process. Co-publication of Cochrane Reviews should be carried out with the agreement of Cochrane and the co-publishing journals [8], and might take the form of be an abridged version of the full review, a “Cochrane corner” of a journal (where the summary of the review is accompanied by commentary), or a short version translated into a language other than English [6].

In an earlier study, the annual co-publication rate for Cochrane Reviews fluctuated from 0.96% to 3.94% [9] between 2005 and 2015. It has also been shown to vary across different Cochrane groups. For example, 19.6% of reviews from the Cochrane Eyes and Vision Group were co-published in 2007, and 16.2% in 2014 [10, 11]. Co-publication of Cochrane Reviews in other journals might promote access to the evidence and increase citations [10, 12] and journals that co-publish Cochrane Reviews might also increase their impact factor (IF) [12]. Our previous study [9] found that the total number of citations for co-published Cochrane Reviews (combining the citations for the original Cochrane Review and the co-published version) was significantly higher than that for non-co-published reviews. This showed that co-publication not only improved the dissemination and accessibility of Cochrane evidence but also facilitated its uptake.

In the study reported here, we investigated the effect of updating and of co-publication on the number of citations to Cochrane Methodology Reviews. Most Cochrane Reviews relate to the effects of health and social care, but Cochrane Methodology Reviews are a subset focused on research into the methods used in research into health and social care. We have compared citations to the updated and the original Cochrane Methodology Review in the years after the update was published, the citations for co-published and non-co-published Cochrane Methodology Reviews and the effect of co-publishing the review on the co-publishing journal’s IF.

Methods

Study design

This was a retrospective study of Cochrane Methodology Reviews. In August 2022, we identified all Cochrane Methodology Reviews in the Cochrane Database of Systematic Reviews (CDSR) in the Cochrane Library (https://www.cochranelibrary.com/) which had been published before 31 December 2018. We searched for co-published versions of these reviews in another journal in the Web of Science Core Collection database up to 16 August 2022. The number of citations for each version of the Cochrane Methodology Review and any co-published versions was also obtained from the Web of Science Core Collection database on 16 August 2022.

The included reviews were divided into two cohorts: (1) those that had been updated before 2018 and (2) those that had been co-published in another journal. For the updated cohort, the most recent versions or previous version of the updated Cochrane Methodology Reviews were the updated group and the original review were original group. If a review is updated multiple times, each updated version is included in the updated group. For the co-published cohort, reviews which were co-published in another journal at the same time or later than the original review were the co-publication group, and the Cochrane Methodology Reviews had not been co-published were non-co-publication group.

Inclusion and exclusion criteria

Inclusion criteria were (1) Cochrane Methodology Reviews that had been updated before 2018; or (2) Cochrane Methodology Reviews that had been published before 2018 and co-published in another journal at the same time or after the publication of the Cochrane Methodology Review in the CDSR. Exclusion criteria were (1) Cochrane Methodology Reviews that had not been updated or co-published before 2018; and (2) Cochrane Reviews or updated versions or co-published versions for which the number of citations could not obtained. Although there are examples of co-publication of Cochrane Methodology Reviews after 2018 [13, 14], we restricted this study to reviews published before 2018 in order to be able to investigate the trajectory of citations in the five years after updating or co-publication.

Review selection and data extraction

Two authors (LLZ and ZYY) reviewed and extracted data independently. Disagreements were resolved by consensus, with arbitration by a third author (HYD) if necessary. A standardized extraction form was used to collect the following data for eligible reviews: title, number of included studies, results and conclusions, authorship of CDSR and co-published versions, journal of co-publication, time interval between original and updated or co-published review. The number of citations to the Cochrane Methodology Reviews and to the co-published versions, and the IF of the co-publishing journal were taken from the Web of Science database (Journal Citation Report).

Outcomes

The primary outcomes were the average (mean and median) number of citations to the updated and the original Cochrane Methodology Reviews in the five years after the update had been published and the average number of citations to the co-published and non-co-published Cochrane Methodology Reviews in the first five years after co-publication. The secondary outcome was the ratio of an adjusted IF (excluding the data for the co-published Cochrane Methodology Review) and the actual IF of the co-publishing journal.

We calculated the number of citations for each co-published review as the sum of the citations to the Cochrane Methodology Review and to its co-published version. The journal’s actual IF is calculated by:

$$\mathrm{IF}=\frac{\mathrm{Citations\ in\ year\ {\rm X}}\mathrm{\ to\ items\ published\ in\ year\ }\left(\mathrm{\rm X}-2\right)\ +\ \mathrm{Citations\ in\ year\ {\rm X}}\mathrm{\ to\ items\ published\ in\ year\ }\left(\mathrm{\rm X}-1\right)}{\mathrm{Number\ of\ citable\ items\ in\ year\ }\left(\mathrm{\rm X}-2\right)\ +\ \mathrm{Number\ of\ citable\ items\ in\ year}\ \left(\mathrm{\rm X}-1\right)}$$

Our adjusted IF was calculated by:

$$\mathrm{Adjusted\ IF}=\frac{\mathrm{Citations\ in\ year\ {\rm X}}\mathrm{\ to\ items\ published\ in\ year }\left(\mathrm{\rm X}-2\right)\ +\ \mathrm{A }}{\mathrm{Number\ of\ citable\ items\ in\ year\ }\left(\mathrm{\rm X}-2\right)\ +\ \mathrm{B}}$$
$$\mathrm{\rm A}=\left[\mathrm{Citations\ in\ year\ {\rm X}}\mathrm{\ to\ items\ published\ in\ year\ }\left(\mathrm{\rm X}-1\right)\right]-[\mathrm{Citations\ in\ year\ {\rm X}}\mathrm{\ to\ Cochrane\ review\ copublications\ in\ year\ }\left(\mathrm{\rm X}-1\right)]$$
$$\mathrm{\rm B}=\left[\mathrm{Number\ of\ citable\ items\ in\ year}\left(\mathrm{\rm X}-1\right)\right]-\mathrm{\ Number\ of\ Cochrane\ review\ copublications}$$

We then calculated a ratio by dividing the adjusted IF by the journal’s actual IF. If this ratio was greater than or equal to 1, the co-published reviews did not increase the journal IF; while values less than 1 indicated that the co-published reviews increased the journal IF. When more than one co-publication of a Cochrane Methodology Review was identified in a single journal, we calculated the mean ratio of IF and its 95% confidence interval (CI).

Statistical snalysis

We used the Fisher’s exact test to compare categorical variables, the Student t-test for continuous and normally distributed variables and the Mann–Whitney U-test to compare medians if the data were not normally distributed. We conducted a normality test on the data using a histogram and a normal P-P plot. A p-value of < 0.05 was considered statistically significant. We used SPSS 26.0 software for the statistical analyses.

Results

Baseline characteristics of Cochrane reviews

There were 38 full Cochrane Methodology Reviews in CDSR on 16 August 2022, which had been updated between zero and five times by that date. Six of these reviews [15,16,17,18,19,20] had been updated [21,22,23,24,25,26] before 2018. We also retrieved four versions of Cochrane Methodology Reviews (two original reviews [17, 27] and two updated reviews [23, 28]) that had been co-published in another journal [29,30,31,32] before 2018 from the Web of Science Core Collection database on 16 August 2022. In total, our sample comprises 14 published versions of Cochrane Methodology Reviews and four co-published articles. The flow diagram is shown in Fig. 1.

Fig. 1
figure 1

The flow diagram of included Cochrane Methodology Reviews

Full size image

We categorized the included records into two cohorts as described above. In the updated cohort, the updated group included 8 updated reviews, while the original group included 6 original Cochrane Methodology Reviews. The co-published cohort consisted of 4 co-published and 11 non-co-published Cochrane Methodology Reviews. The included reviews had been published between 2007 and 2018. The main characteristics of the two cohorts are shown in Tables 1 and 2.

Table 1 Characteristics of the updated and original Cochrane Methodology Reviews
Full size table
Table 2 Characteristics of the co-published and non-co-published Cochrane Methodology Reviews
Full size table

Characteristics of updated and co-published Cochrane review

Topics addressed in the eight updated Cochrane Methodology Reviews included issues relating to randomization and recruitment to randomized trials (4 reviews) [25, 26, 28, 33], conflict of interest (1 review) [23], technical editing (1 review) [21], publication (1 review) [22] and questionnaire response (1 review) [24]. Five (62.5%) had been updated only once before 2018 [21,22,23,24,25], while the other had been updated three times [26, 28, 33]. The interval time between versions of the reviews varied from 0.5 year to 11 years. Characteristics are summarized in Table 3.

Table 3 Characteristics of eight updated Cochrane Methodology Reviews
Full size table

The four co-published Cochrane Methodology Reviews were co-published in three journals. Three (75%) were co-published more than two years after the publication of the Cochrane Methodology Review. The types of co-publication were mostly a short version of the full review (3 reviews), with one being a specially prepared summary of the review. One Cochrane Methodology Review had a co-publication for both its original and updated version. The other two co-publications were for the original version of the Cochrane Methodology Review (1 review) and for the updated review (1 review). For all four cases, the results and conclusions were the same between the Cochrane Methodology Review and the co-published version. Characteristics are summarized in Table 4.

Table 4 Characteristics of four co-publications (journal version)
Full size table

Number of citations for updated and original Cochrane Methodology Reviews

As shown in Table 5, the original Cochrane Methodology Review was still being cited after the updated Cochrane review had been published in all six cases and, in one, there were 485 citations to the original version of the review and only 105 to the update in the five years after publication of the update. Combining all the reviews and considering the total number of citations between the publication date of the updated version and 16 August 2022, this was non-significantly higher for the updated Cochrane Reviews than for their original version [161 (IQR 85, 198) versus 113 (IQR 15, 433), p = 0.518]. There were also non-significantly more citations to the updated review than the original review if this analysis is limited to the first five years after publication of the update [78 (IQR 61, 149) versus 91 (IQR 14, 247), p = 0.897]. Table 6 shows that in the first three years after the publication of the update, the median number of citations to the updated review was lower than to the original review but it was higher in each of the next two years.

Table 5 Total number of 5-year citations for each of the updated reviews
Full size table
Table 6 Number of citations for the updated and original Cochrane reviews after the updated year
Full size table

Number of citations for co-published and non-co-published Cochrane Methodology Reviews

The median for the total number of citations (combining citations to the original Cochrane Methodology Review and to its co-publication) up to 16 August 2022 was non-significantly higher in the group of co-published reviews than in the non-co-published group [362 (IQR 179, 840) versus 145 (IQR 75, 445), p = 0.090]. Similarly, the median number of citations in each of the five years (Table 7) and sum of the first five years after publication in the co-published group was higher than in the non-co-published group [177 (IQR 99, 338) versus 75 (IQR 37, 126); p = 0.037].

Table 7 Number of citations for the co-published and non-co-published reviews
Full size table

Number of citations for the Cochrane Methodology Review and its co-publication

As shown in Table 8, in the five years after co-publication, the original Cochrane Methodology Review received more citations than the co-published version for two reviews while the co-published version received more citations for the other two reviews. The median for the number of citations was non-significantly higher for the original Cochrane Methodology Review than for its co-published version for the total number of citations up to 16 August 2022 [157 (IQR 102, 720) vs 85 (IQR 54, 265), p = 0.248], for the number of citations in the first five years after co-publication [92 (IQR 45, 290) versus 49 (IQR 37, 102)] and for each of those five years (Table 9).

Table 8 Total number of 5-year citations for each of the co-published reviews
Full size table
Table 9 Number of citations for the Cochrane Methodology Review and its co-publication
Full size table

Effect of co-publication on journal impact factor

The four co-published Cochrane Methodology Reviews were co-published in three journals (Table 10). Among these three journals, the ratio of the adjusted IF to the actual IF was less than 1 in one journal for the first year after publication, and less than 1 in two of the three journals in the second year after publication. This means that the co-published review increased the IF for these journals in those years.

Table 10 Ratio of the adjusted IF to the actual IF
Full size table

Discussion

This study is the first that we are aware of that investigates the citing of updated and original Cochrane Methodology Reviews after an update has been published, and the impact of co-publication for this type of Cochrane Review. Our results show that the original Cochrane Methodology Review was still being cited after its updated version had been published and, in some cases, there were more citations to the original review than to the update in the five years after the update was published.

We speculate that there are several reasons for the original version of a Cochrane Methodology Review to continue to be cited after an update is published. For instance, the citing authors may have a preference based on the conclusions of the two versions of the review [35], they may be more familiar with the original version or might not have been aware of the updated version at the time they prepared their article, they might be referring to something in the original version that was not included in the update or they might be re-using text that they have written previously. It is also possible that the cited Cochrane Methodology Review was updated between the submission and acceptance of the citing article [36]. Another possible explanation is “second-hand” citation of the Cochrane Methodology Review based on it being cited elsewhere before the update [36], with the citing author using the citation from another paper without checking the original [37]. Actually, this does not comply with the International Committee of Medical Journal standards [38].

It is also worth noting that the number of citations for the updated Cochrane Methodology Reviews were much higher than those for the original version in the fourth and fifth year after the update was published, which might suggest that there is a lag of a few years in the uptake of evidence from Cochrane Methodology Reviews into the wider literature. This is in accordance with academic productivity trends, which have found that there were more citations from 2.5 to 5 years after publication compared with the first 2.5 years [39].

Turning to co-publication, the number of citations for co-published Cochrane Methodology Reviews is higher than that for non-co-published reviews at least for the first five years after co-publication, with the number of citations to the co-published version being higher than the number of citations to the Cochrane Methodology Review in two cases and lower in two cases. This is consistent with a previous study [9, 10] and suggests benefits for dissemination of the findings of Cochrane Reviews. First, if the co-publishing journal has a specialist audience, the Cochrane evidence would be brought to that audience [40,41,42] and some previous studies have shown that meta-analyses have higher rates of citation in specialty journals [39, 43,44,45]. Second, the higher number of citations suggest that the evidence in the Cochrane Review will have received more attention because of the co-publication. Third, the co-published version might make the Cochrane evidence more accessible to potential users than the full Cochrane Review and might be more useful to them as a reference source [46]. Among the three journals that co-published a Cochrane Methodology Review, the IF was raised by this in the first year in one journal and in the second year in two journals. This is also consistent with a previous study [9, 10]. The longer term effects on journals are less clear because of how the IF is calculated but citation rate peaks have been shown to vary across different journals, ranging from 2.5 to 7.2 years after publication [47, 48], which suggests that journals may benefit from co-publication beyond the two years used to calculate their IF [49].

Our study has some limitations. First, our sample may be too small to allow sufficiently powered analyses and conclusions, the results might be influenced by chance [50], and the generalisability to other types of Cochrane Review needs to be investigated in further research. Second, we only included data on citations and co-publications from the Web of Science Core Collection database, which may underestimate the true number of citations and may have missed some co-publications. Third, although some studies have found that meta-analyses have higher rates of citation, which may make it more likely that co-publishing a version of a Cochrane Review may have more impact on a journal’s IF than co-publishing of any other type of article, other studies have argued that study design does not significantly influence citation rate [51, 52]. For example, there may also be important associations with number of authors [53,54,55] or their geographic location [56]. These are issues that should be investigated in further research.

In conclusion, we have shown that the citation of updated Cochrane Methodology Reviews lags a few years after their publication and confirmed that co-publication increases the number of citations for this type of Cochrane Review and may increase the IF of the co-publishing journal. However, this raises issues about whether people wishing to use the evidence from Cochrane Methodology Reviews are always using the most up to date and complete information. This highlights the importance that authors who cite any previous research should confirm the validity of the reference and the need for ways to allow people to more clearly see that a review has been updated.