Background

Literature searching for systematic reviews of interventions in health care has been largely based on finding all randomized controlled trials (RCTs), as this study design is considered the gold standard. However RCTs make up only a very small proportion of all the articles included in bibliographic databases, and so the problem for systematic reviewers has been to devise a search strategy which is sensitive enough to find all the RCTs, but specific enough not to bury them in a large number of other unwanted retrievals needing to be manually excluded.

Two developments have facilitated searching for RCTs. The first is CENTRAL (the Cochrane Central Register of Controlled Trials), the world's most comprehensive database consisting exclusively of controlled clinical trials. It currently contains over 425,000 citations. The majority of the trials in CENTRAL have been identified through systematic searches of MEDLINE and EMBASE. The first two phases of a three phase highly sensitive search strategy (HSSS) have been used to search MEDLINE [1]. CENTRAL also includes citations to reports of controlled trials that are not indexed in MEDLINE or EMBASE, derived through searches of other bibliographic databases, and extensive hand searching.

Because identification has relied solely on the titles and, where available, the abstracts, some relevant trials may not have been identified. Therefore, the Cochrane Handbook says "it may still be worthwhile for reviewers to search MEDLINE using the Cochrane highly sensitive search strategy and to obtain and check the full reports of possibly relevant citations" [2]. However, this strategy is complicated to run, and may require time-consuming screening of abstracts, and perhaps of full articles.

The second development is the CONSORT statement, introduced in 1996 [3], and since revised [4]. CONSORT comprises a 22 item checklist and a flow diagram to help improve the quality of reports of RCTs, and has been endorsed by prominent medical journals There is evidence from a comparative before-and-after evaluation that there has been an increase over time in the number of CONSORT checklist items included in the reports of RCTs [5].

Item 1 on the CONSORT checklist recommends that the method in which participants were allocated to interventions (e.g., "random allocation", "randomized" or "randomly assigned") is described in the title and abstract. This allows instant identification of RCTs, and should help ensure that a study is appropriately indexed as an RCT in bibliographic databases.

As the Cochrane Collaboration has already done exhaustive work to ensure that CENTRAL is as complete as possible, we wanted to examine whether this eliminates the need for individual reviewers to run the HSSS, and the effectiveness of replacing this with a simplified search strategy.

The aims were

  1. 1.

    To determine the effect on the results of Cochrane reviews of using a brief RCT search strategy (BRSS).

  2. 2.

    To examine the change in use of variants of the word random in the title or abstract of journal articles, pre- and post-CONSORT.

Methods

All reviews new to the Cochrane Database of Systematic Reviews in the Cochrane Library 2004 issue 2 were selected. Those that stated that they were considering only RCTs, and which found at least one RCT, were selected.

All trials included in each review were identified from the section 'References to studies included in this review'. Each trial was checked to determine whether it was indexed in CENTRAL (on Cochrane Library 2004, issue 2), and then MEDLINE. If not in either of these databases, EMBASE was checked.

The full bibliographic records of all trials that were in either MEDLINE or EMBASE (using the OVID interface) were examined to determine whether random$ was in any field. (Random$.af. means a search of variants of random in all fields, where $ is truncation symbol).

The full papers of any trials that were: 1) not in CENTRAL, or 2) did not have random$ in any field in the bibliographic record, were obtained and checked to see whether they were actually RCTs. All non-English articles were translated, apart from those in Japanese and Chinese, as resources were not available.

The impact of omitting the RCTs not found with the BRSS was quantified, using Review Manager 4.2.7. The forest plots for the meta-analyses were reproduced, both with and without data from the missing trials, and the results compared to see if omission would make any important difference. The differences could theoretically include;

  1. a)

    There might be less, or no data left, for some outcomes.

  2. b)

    There could be a different result; no benefit over comparator, or vice versa.

  3. c)

    There could be the same result, but with a different effect size.

  4. d)

    There could be the same result and effect size, but with a wider confidence interval, and possibly loss of statistical significance.

In summary, the BRSS would involve: 1) searching CENTRAL, and 2) supplementing that with a search of MEDLINE and EMBASE, using a search of 'random$.af.' to pick up trials not in CENTRAL.

Results

There were 78 new reviews, of which 57 met the inclusion criteria. They cited a total of 920 trials; 79% (725) were journal articles. The remaining 21% were from the grey literature, 80% of these being conference abstracts).

Twenty one reviews were excluded from our study; 14 because they did not find any RCTs that met their inclusion criteria, and seven because they included other study designs in addition to RCTs.

Determination of the proportion of journal articles found using the BRSS

It was found that 93.3% (677) of the 725 journal articles included in the systematic reviews were in CENTRAL. It was assumed that these were all RCTs, but this was not checked due to the large numbers involved (and it was not relevant to the aims of the study). This left 48 journal articles not indexed in CENTRAL.

The full texts of all 48 of these articles (apart from the four in Japanese and five in Chinese) were translated and checked. It was found that 40 were RCTs, and the remaining eight used non-RCT study designs. (It was assumed that the untranslated articles were RCTs). Eleven of the 40 RCTs were found by searching MEDLINE or EMBASE with 'random$.af'. Therefore, 29 (4%) of RCTs would not have been found with the BRSS.

Details of the 29 RCTs not found by the BRSS

The 29 trials were distributed over 12 reviews. Ten were in MEDLINE, 11 in EMBASE only. Twelve were non-English language.

In two reviews [6, 7], each with one missing trial, the data used in the meta-analyses were available in two other papers; both were in CENTRAL, and included in the reviews. Zhang 1990 [8] was confirmed (by authors of the review) to be the same trial as Chang 1996 [9]. The data in the Stensrud trial [10] was also reported in another paper by Stensrud [11].

Therefore, this leaves 27 missing journal articles, in 10 separate reviews that contain at least one trial with data not found by the BRSS. Table 1 shows the detail of these trials and the impact of excluding them from the reviews.

Table 1 Summary of impact of excluding trials not found with brief RCT search.
Full size table

Only one review did not do a meta-analysis[12] The nine remaining reviews did a total of 129 meta-analyses of various outcomes. In five out of the 10 reviews the missing trials made no significant difference, in that there were no clinically significant changes in effect size, nor any change in whether results were statistically significant or not. Hence, there was some difference in only five reviews.

The impact of the 'missing trials' being excluded from the reviews

Two consequences of missing data were found:

  1. 1)

    In three reviews [13–15], the missing trials were the only ones providing data for seven (of a total 35) outcomes, so no data were available for these seven outcomes. In a fourth review [16], omission of a trial would lose 92% of patients for two outcomes out of 28, but this did not change the significance or direction of the result.

  2. 2)

    In three reviews [15–17], the meta-analyses lost statistical significance due to the wider confidence intervals, for three of 74 outcomes, but one of these was not a clinical outcome.

Comparison of retrieval of HSSS and random$.af. in MEDLINE

For the period 1966 to October, 2004, the HSSS search strategy retrieved 2,505,742 records in MEDLINE, compared to 'random$.af.', which retrieved 399,208 records. Therefore, only 16% of the number of records were found using 'random$.af.', compared to HSSS.

CONSORT and the change over time in the proportion of RCTs with random$ in title or abstract

Using the 725 journal articles in our sample, we compared the frequency of random$ in the title or abstract between pre-CONSORT (published up to 1996) and post-CONSORT trials (published from 1997 onwards). Figure 1 shows the change in the proportion of RCTs with 'random$' in the title or abstract (with three year moving average trendline added).

Figure 1
figure 1

Change in the proportion of RCTs with 'random$' in the title or abstract (with three year moving average trendline added).

Full size image

In the post-CONSORT years, 1997–2003, the term appeared in 85% of the titles or abstracts, compared to 68% in the seven years before CONSORT (1990–96); mean difference was 17% (95% CI 8.3 to 25.9%; p = 0.001). If a longer pre-CONSORT period is used, 1980 to 1996, the proportion is similar at 62%.

Discussion

Using the BRSS, rather than the much more exhaustive HSSS, to retrieve RCTs in journal articles for Cochrane reviews, affected only a very small percentage of total outcomes of a few reviews, and made no important difference to the conclusions of these reviews. The most affected review had four of the five included trials published in Chinese [14].

The CONSORT statement appears to be associated with a significant increase in the frequency of random in the titles and abstracts of journal articles. Whether this is directly due to CONSORT, or whether CONSORT simply accelerated a pre-existing trend, and raised the general awareness amongst authors and editors for better reporting of RCTs, is uncertain. This improved description of RCTs by authors should mean that all RCTs are indexed with the appropriate publication type in MEDLINE, and also result in more sensitive retrieval of RCTs using the BRSS.

The BRSS had a sensitivity of 96% for RCTs in journal articles. Compared to the HSSS, the BRSS reduced the MEDLINE retrieval by 84%. This would represent a major time and cost saving in manual screening.

The strengths of this study include firstly that we used Cochrane reviews, as they approximate the 'gold standard' in searching, due to the requirement for exhaustive searches. Hence we can be fairly certain that we were starting with as comprehensive set of included trials as possible. Secondly, we quantified the impact of omitting trials not found with the simplified search.

A weakness of this study was that we could not check whether the RCTs would have been in CENTRAL at the time the searchers did their searching, and hence eliminate the possibility that some RCTs were first identified by the reviewers, and then passed to CENTRAL. However, given the extensive searching routinely done for CENTRAL, it is highly likely most RCTs would be identified sooner or later, and therefore be included in a subsequent update of the review.

A range of subject areas was included, which helps with generalisability, though it may decrease power in any one subject area. However, a recent study on identifying quality RCTs in pain relief gives general support to our findings [18]. It investigated the efficiency of the search strategy DBRCT.af., ("double-blind," "random," or variations of these terms) in MEDLINE and EMBASE, and was found have a sensitivity of 97%.

This study focused only on a simplified search strategy for retrieving RCTs in journal articles, since these made up the vast majority of the references used in Cochrane reviews, and are most important in terms of the quality and quantity of assessable data available. By contrast, most grey literature (the vast majority of which is meeting abstracts) gives very limited data. However, CENTRAL includes many grey literature trials, so these will be identified with the BRSS.

There are currently over 2200 Cochrane reviews, and these will need maintaining in the future. Authors may be encouraged to update their reviews, if they can be confident they can identify RCTs comprehensively with a simple search. The simplified search may also be useful for those doing reviews in a tight timescale, or by clinicians who just want a rapid but reliable answer to a question. There is a case for the 'not quite perfect but rapid, easy and almost complete' search'.

In practice, some of the few trials missed by the BRSS were small or of poor quality, and as Egger and colleagues have reported, the last few studies found by exhaustive searching could introduce bias, by being of poor quality [19].

The issue is whether the marginal benefits of exhaustive searching justify the extra costs. When the inclusion criteria demand only RCTS, this study suggests that exhaustive searching is now, in the era of CENTRAL and CONSORT, no longer cost-effective.