Background

A number of papers have been published on the methodological and reporting quality of reviews. A review of 86 English language meta-analyses published between 1950 and 1986 by Sacks [1] assessed every report on fourteen items covering six content areas believed to be critical in the reporting of meta-analysis. Only 28% of these meta-analyses were found to address all six content areas. An updated survey in 1996 showed little change [2]. Shea [3] compared the methodological quality of paper-based and, electronic systematic reviews and found little difference and a lot of room for improvement. Assendelft [4] reviewed 51 reviews and noted that reviews that favoured a given intervention tended to have higher methodological quality scores. In contrast, Jadad and McQuay [5] reviewed 80 systematic reviews published between 1980 and 1992 and found a disconcerting link between reviews whose results favoured an intervention and poor methodological quality. Jadad found that Cochrane reviews had greater methodological rigor, more frequent updates and higher overall quality scores than those published in peer-reviewed paper journals, though both types were found to contain extensive and serious flaws.

The Cochrane Collaboration [6] is an international not-for-profit organization that conducts and updates systematic reviews of healthcare studies. With the exception of a few studies completed by Jadad [7, 8] and Shea [9], little is known about the quality of Cochrane reviews and whether their quality is improving over time and with updating. To our knowledge, the impact of updating on their quality has not been evaluated nor studied.

Studies of the quality of systematic reviews can focus on methodological or reporting quality. Methodological quality is concerned with how well a systematic review was designed and conducted (e.g. literature searching, pooling of data, etc.). Reporting quality considers how well systematic reviewers have reported their methodology and findings.

The purpose of our study was to compare the methodological and reporting quality of Cochrane systematic reviews with that of their updated versions in order to determine whether updating contributed significantly to the improvement of their quality in these two dimensions.

Methods

We selected all updated Cochrane systematic reviews from the Cochrane Database of Systematic Reviews 2002 [10], and the same reviews prior to their update. Updated reviews were chosen following the definition of 'updating' included in the Cochrane handbook [11].

Based on a previously published study [12], two instruments chosen to assess the quality of Cochrane systematic reviews for this study were the enhanced Overview Quality Assessment Questionnaire (OQAQ) scale [12–14] (Additional File 1) and the Quality of Reporting of Meta-analsyses (QUOROM) checklist [15] (Additional File 2).

The OQAQ was selected because it has strong face validity, provided data on several essential elements of its development, and had a published assessment of its construct validity available [13]. In addition, its validity has been thoroughly tested and established using a number of independent measures [14]. However, we noted difficulty applying the questions so we developed an enhanced version of the OQAQ which incorporated guidelines for its' use [12]. The OQAQ scale measures across a continuum using nine questions (items 1–9) designed to assess various aspects of the methodological quality of systematic reviews and one overall assessment question (item 10). When the scale is applied to a systematic review, the first nine items are scored by selecting either yes, no, partial/can't tell. The tenth item requires assessors to assign an overall quality score on a 7-point scale [13].

The QUOROM statement was chosen for assessing reporting quality. Although, this checklist has not yet been fully validated, extensive work has been conducted and reported [15]. The QUOROM statement is comprised of a checklist and flow diagram and was developed using a consensus process designed to strengthen the reliability of the estimates it yields when applied by different assessors. It estimates the overall reporting quality of systematic reviews. The checklist asks whether authors have provided readers with information on 18 items, including searches, selection, validity assessment, data abstraction, study characteristics, quantitative data syntheses and trial flow. It also asks whether authors have included a flow diagram with information about the number of randomized controlled trials identified, included and excluded, and the reasons for any exclusion. Individual checklist items included in this instrument are answered either yes, no and partial/can't tell [15].

For each included Cochrane systematic review, we calculated the percentage (with a 95% confidence interval) of 'yes' answers to each item of OQAQ and QUOROM. Individual Cochrane systematic reviews were then compared to their updated versions with respect to the percentage of 'yes' ratings received. A percentage difference (with a 95% confidence interval and p-values) of 'yes' answers was calculated for each individual question. In addition, a mean difference (with a 95% confidence interval and p-value) was calculated for the overall quality score (OQAQ) of the updated and pre-updated versions.

Assessments of all individual reviews were conducted independently by at least two assessors. One of these assessors (CH) had been involved with the Cochrane Collaboration for six years. The other assessor (BS) had been involved with the Cochrane Collaboration for ten years, and had carried out a number of systematic reviews. A third assessor (DF) was available to assist in the resolution of assessment discrepancies.

Results

In total, 53 reviews were included. The mean period between the publication of the original and updated versions of the reviews was 2.7 years (range four months to five years). Complete results for both instruments applied to the reviews are provided in Tables 1 and 2, but highlights are summarized below.

Table 1 Comparisons of percent 'yes' and percent 'differences' using the enhanced Overview Quality Assessment Questionnaire (OQAQ).
Full size table
Table 2 Comparisons of percent 'yes' and percent 'differences' using the Quality of Reporting of Meta-analyses (QUOROM).
Full size table

OQAQ

Table 1 presents the methodological quality assessments obtained using this scale. There was no significant difference in the global assessment (item 10 – How would you rate the scientific quality of the overview?) (Mean score original review 4.70, mean score updated review 4.81, difference in means 0.11 (95% CI -0.28; 0.70 p = 0.52)).

There were improvements on seven individual items, although only one item showed a significant improvement (item 9 – Were the conclusions made by the author(s) supported by the data and/or analysis reported in the overview?) (percent original reviews complying with item 76% (95% CI 4.47; 4.94) percentage updated reviews complying with item 94% (95% 88.1; 100.0) difference 18.9% (95% CI 7.2%; 30.6% p < 0.01)).

QUOROM

Table 2 presents the assessments of reporting quality obtained using the Quality of Reviews of Meta-analyses (QUOROM) checklist. Scores (awarded for 'yes' response) on the 18 items for original Cochrane reviews ranged from 0% (item 1) to 100% (items 2 and 18). Four of the 18 items revealed improvements relating to data sources reported in the abstract (item 4) with a significant difference of 17.0% (9.8; 28.7 p = 0.01), review methods reported in the abstract (item 5) with a difference of 35% (24.1; 49.1 p = 0.00), searching methods (item 9) with a difference of 18.9% (9.7; 31.6 p = 0.01), and data abstraction (item 12) with a difference of 18.9% (11.7; 30.9 p = 0.00).

Three questions, item (10) -3.7% (95% CI: -9.0; 3.8 p = 0.15), item (14) -3.7% (-14.8; 6.4 p = 0.62) and item (16), -5.7% (-15.8; 3.0 p = 0.40) had lower mean scores on updated reviews than on original reviews, but these differences were not statistically significant.

Discussion

In assessing the quality of the sample of 53 systematic reviews of the Cochrane Collaboration, two assessment tools were used. The larger improvement on the QUOROM checklist than on the OQAQ suggests that although the quality of reporting has improved slightly, the quality of design and conduct has not changed. For example, the items in reporting of selection criteria and searching are improved on QUOROM, but the equivalent items on OQAQ, which relate to how well these were carried out have not changed.

The significant improvements for the OQAQ item assessing whether the conclusions drawn by the author(s) were supported by the data and/or analysis reported in the overview is also worthy of note. This could suggest that authors are paying more attention to state their conclusion in relation to the data provided. This item has been reported by other methodologists in the context of asessing quality [16, 17].

Improvement in the quality of reporting of the abstract, as assessed by the QUOROM instrument, might also suggest that authors are paying more attention to the details. For example, authors might be more aware of better access to the Cochrane abstracts through electronic systems such as Medline. Also, reporting of literature searches improved. However, the literature searches themselves did not improve. This should be further explored.

Assessors expressed that they found the two instruments used in the study to have associated practical weaknesses. Their combined length (28 items) made their use somewhat cumbersome and inefficient. Another problem encountered was the lack of adequate published guidance on the application of the OQAQ question, assessing the overall methodological quality. The QUOROM checklist had the advantage of coming with fairly detailed user instructions. However, it proved very time-consuming to apply [12]. Another problem noted was that several of the questions asked by these two instruments appeared to cover the same subject matter.

The methodological and reporting quality of Cochrane reviews was fair-to-good, but further improvement is obviously needed in both areas. This was also concluded in a recent study by Moja [16]. In addition, the quality-improvement impact of updating was found to be relatively minor. On some assessed factors, particularly with respect to reporting quality, updated reviews actually scored lower than original reviews. Currently, Cochrane review updates are carried out primarily to incorporate new findings, rather than to improve quality. It would be beneficial for updates to also address reporting, such as the omission of methodological descriptions. And of course, updates should also try to improve methodological weaknesses of the reviews.

The Cochrane Collaboration endeavours to improve the quality of its systematic reviews through the application of a continuous peer review process during their development. The effectiveness of the peer-review process could probably be improved by increased attention to areas of reporting and/or methodological weaknesses. Reviewers should adhere more faithfully to the guidelines provided in the Cochrane handbook [17] in order to improve the design and conduct of reviews and to the QUOROM statement [15] to improve the quality of reporting of systematic reviews.

Conclusion

The overall quality of Cochrane reviews is fair-to-good. Although quality of reporting improved on certain individual items there was no overall improvement seen with updating and methodological quality remained unchanged. Further improvement of quality of reporting is possible. There is room for improvement of methodological quality as well. Authors updating reviews should address identified methodological or reporting weaknesses. We recommend to give full attention to both quality domains when updating SRs.