Background

Cochrane systematic reviews are produced using rigorous and evolving methodological standards and are therefore considered the gold standard when it comes to synthesis of evidence. The Cochrane has been at the forefront of applying the methods of evidence-based medicine (EBM) in the treatment and management of various conditions [1].

An important part of the systematic review methodology is appraisal of the risk of bias (RoB) in included studies. The potential effect of bias is that trialists will reach wrong conclusions about efficacy and safety of studied interventions. Bias can, therefore, negatively affect the estimated intervention effects [2].

In Cochrane systematic reviews RoB is appraised using Cochrane RoB tool, which has seven domains; one of them is called ‘incomplete outcome data (attrition bias)’. Incomplete outcome data can yield attrition bias due to amount, nature or handling of incomplete outcome data [3]. The main strength of RCTs is that study arms should generally be balanced in terms of their baseline characteristics, and any imbalance should be result of chance. Attrition can occur if participants are lost to follow-up, or if they miss one or more measurement time points during a trial. Therefore, attrition can lead to bias if the characteristics of participants with missing data are different between the randomized groups [4]. Akl et al. analyzed potential impact of losses to follow-up on the estimates of the effect of treatment in 235 RCTs, and found that different assumptions about outcomes of participants lost to follow-up could change interpretation of results of up to 58% of RCTs published in top medical journals, and a third of the analyzed trials failed to report whether any loss to follow-up occurred [5].

In the Cochrane RoB tool, the authors need to provide judgment about whether this risk is high, unclear or low for each domain. Furthermore, each judgment needs to be accompanied with a supporting explanation called ‘support for judgment’, which “describes what was reported to have happened in the study, in sufficient detail to support a judgment about the risk of bias”. The aim of the support for judgment is to ensure transparency about how these judgments about the level of risk of bias were reached [3].

The Cochrane Handbook provides vague instructions about assessing attrition bias, which may lead to inconsistent use of supporting explanations for judgments of attrition bias that one can find in Cochrane reviews. Da Costa et al. have published a study in 2017 about training authors for risk of bias assessment, and showed that “Kappa values between the minimal training group and reference across items of the risk of bias tool ranged from 0.10 (poor agreement) for incomplete outcome data (…)” [6]. Therefore, inter-rater agreement in participants with minimal training was worst for the attrition bias domain, compared to other domains of Cochrane RoB. Since Cochrane authors rarely have structured training that was tested in the study of da Costa et al. [6], their data could very well indicate real-world difficulties and discrepancies that authors face when assessing attrition bias.

The aim of this study was to analyze whether Cochrane authors use consistent judgments for different supporting explanations of attrition bias in Cochrane reviews of interventions published in the Cochrane Database of Systematic Reviews (CDSR).

Methods

Study design

This was a cross-sectional, primary methodological study in which we analyzed methods of published Cochrane reviews.

Inclusion and exclusion criteria

Cochrane reviews of interventions published from July 2015 to June 2016 were included by using Advanced search in The Cochrane Library. We excluded diagnostic reviews, empty reviews, overviews of systematic reviews and Cochrane reviews withdrawn in this period and reviews that included only non-randomized studies. If the Cochrane reviews included randomized, quasi-randomized and non-randomized studies, we analyzed attrition bias in the RoB tables for the randomized studies only. Cochrane reviews that had multiple attrition bias judgments assessed for different outcomes in the same study were rare; therefore we reported them separately in order to better describe that methodological approach.

Screening

Two authors (JASC, LP) independently assessed all titles/abstracts to establish eligibility of Cochrane reviews for inclusion. There were no discrepancies in judgment.

Data extraction

Data extraction table was developed and piloted using five Cochrane reviews. Seven authors extracted data manually (RT, JASC, IN, JS, MV, IM, IV) and initially another author (AB) checked 10% of the extractions randomly. Discrepancies in data extraction were planned to be resolved by the third author (LP), but we found only several discrepancies, which did not require adjudication by the third author. In 2018, for the purpose of another project we developed customized software acting as a parsing tool, which can extract clearly delimited information from Cochrane reviews. Using the parsing tool, we extracted again the same data for attrition bias from the Cochrane RoB table, and found only 34 discrepancies that needed to be corrected.

The following data were extracted: number of included trials, judgment of attrition risk of bias for each included trial (low, unclear or high) and accompanying ‘support for judgment’. To avoid terminological confusions, instead of ‘support for judgment’ hereby we use the expression ‘supporting explanation’. We also assessed how many Cochrane reviews had inconsistent judgments for the same supporting explanations (i.e. whether they had different judgments for the same supporting explanations). In the main analysis we reported only analysis of attrition bias for included Cochrane reviews with a single judgment (i.e. Cochrane reviews with only one domain for attrition bias, and one judgment in that one domain), regardless of the number of supporting explanations that were provided for that judgment.

In the secondary analysis we investigated i) attrition bias reporting for Cochrane reviews that reported multiple judgments of attrition bias for the same trial (i.e. Cochrane reviews with multiple assessments of attrition bias for the same RCT, where this RoB domain was split into two or more sub-domains analyzing specific aspects of attrition bias), ii) characteristics of risk of bias reporting in Cochrane reviews that did not have attrition bias domain, and iii) characteristics of risk of bias judgment reporting in Cochrane reviews that did not provide judgment in the form of “low, unclear and high”. Specific Cochrane reviews are marked in the body of this manuscript with the serial number of the downloaded record (for example, Cochrane review #1). A list of included and excluded studies with a serial number of each record is in the Additional file 1: Table S1.

Statistics

Descriptive statistics was performed and data presented as frequencies and percentages. Data were analyzed using Microsoft Excel (Microsoft Inc., Redmond, WA, USA).

Results

Among 955 Cochrane systematic reviews published from July 2015 to June 2016 we included 729 Cochrane reviews in the main analysis. In the 729 included reviews there were 1–105 included studies (median: 8 studies). In those reviews we found 10,292 attrition bias domains with single judgment about whether the Cochrane review authors found this bias to be low, unclear or high. Although there was a single judgment, 3504/10292 (34%) supporting explanations contained more than one type of explanations related to risk of attrition bias. We categorized these different types of supporting explanations into four categories: #1: percent of attrition in the RCT groups with higher attrition, #2: difference in attrition between the groups, #3: reporting of reasons for attrition and #4: statistical comments. Only 27/10292 (0.26%) of supporting explanations had all four categories of explanations.

First category: percent of attrition in the RCT groups with higher attrition

In the first category, called ‘percent of attrition in the RCT groups with higher attrition’ a third of supporting explanations were unclear (32%). While there were too many examples of unclear explanations, we provide some examples of explanations categorized by us as unclear explanations in the Table 1. The next most common type of supporting explanations were mentioning only total attrition (16%), indicating there was no attrition (15%) in the trial, providing only number of patients without a percent (11%), or indicating that attrition was not reported in a trial (8.8%) (Table 2).

Table 1 Examples of unclear supporting explanations
Table 2 Number of explanations in a category for percent of attrition per group

We categorized reported percent of attrition in the group with higher attrition into four categories: attrition under 10%, between 10 and 20%, between 21 and 30% and above 30%. Since some Cochrane reviews had multiple supporting explanations for a single judgment, we analyzed separately only reviews where the only supporting explanation was about percent of attrition in the study groups (Table 3). The purpose of this analysis was to see whether Cochrane authors use consistent judgments for various thresholds of attrition in this category of supporting explanations. In the Table 3 we listed total number of Cochrane reviews that had supporting explanations related to percent of attrition in the RCT groups with higher attrition. However, on the right side of the Table 3 we presented data only for reviews where the only supporting explanation was about percent of attrition in the study group because only for these Cochrane reviews we can be sure that the single judgment applies only to that comment. As Table 3 indicates, Cochrane authors use very heterogeneous judgments for each category of comment.

Table 3 Frequency of different judgments for the same supporting explanation related to percent of attrition in RCT groups and comments about statistics

Second category: difference in attrition between the groups

In the second category of supporting explanations about difference in attrition between the groups, 302/10292 (2.9%) explanations reported this category, and in all of them it was reported if the difference was above 10%.

Third category: reporting of reasons for attrition

There were 2157/10292 (21%) supporting explanations related to reasons for attrition. The majority of these explanations referred to reasons for attrition that were reported in a trial, while the remaining supporting explanations indicated either that reasons for attrition were not reported in a trial, or that they were inadequately reported (Table 4).

Table 4 Number of supporting explanations in a category for reporting reasons for attrition

Fourth category: supporting explanations about statistics

We found 1572/10292 (15.3%) supporting explanations related to statistics; Table 5 lists all of them in a way that they were described by the Cochrane review authors themselves. Most of the explanations about statistics were referring to presence or absence of intention-to treat analysis (ITT), per protocol analysis (PP) or last observation carried forward (LOCF) (Table 5). Detailed analysis of risk of bias judgment categories was shown only for the most commonly used categories that reported only supporting explanation about statistics; for each statistical comment, Cochrane authors had highly heterogeneous judgments regarding their impact on risk of attrition bias (Table 3).

Table 5 Supporting explanations about statistics used that was related to attrition bias

There were 35 Cochrane reviews that indicated that it was unclear whether ITT analysis was used or not, because its usage was not described. We did not analyze this group of CRSs because none of those listed this item as the only supporting explanation for risk of attrition bias judgment.

Inconsistencies in judgments in single Cochrane reviews

We found only 34/729 (4.7%) Cochrane reviews that had inconsistencies in judging risk of attrition bias in the same review. This means that they gave different judgment for the same explanation. For example, “No incomplete outcome data” was judged as either low or unclear risk of bias in the review #210. In the review #255 explanation “No pre-publication protocol identified” was judged either as unclear or high. In the review #277 “No missing data” was judged as low or unclear. In the review #330 “No withdrawals mentioned” was judged as either low or unclear risk of attrition bias. There were 66/729 (9.1%) Cochrane reviews for which this analysis was not applicable because they included only one trial. All the other reviews had consistent judgments for the given supporting explanations.

Secondary analysis: studies with multiple judgments of attrition bias for the same study

We found 27 Cochrane reviews that had multiple assessments of attrition bias for the same RCT. They had 2–7 multiple assessments separately, which we categorized in assessments related to aspects of attrition bias, time, objectivity and clinical outcomes.

Five Cochrane reviews had separate assessments of different aspects of attrition bias were assessments of drop-outs, participants analyzed in the group to which they were allocated and whether ITT analysis was performed. Seven reviews had assessments related to time were multiple assessments for short-term or long-term outcomes, sometimes defined with specific time-frame (i.e. before or after 12 weeks or childhood outcomes), or end-of-intervention and end of follow-up. Five Cochrane reviews had separate assessments for subjective and objective outcomes. One of them specified what was a subjective and what an objective outcome was. Ten reviews had separate assessments for different clinical outcomes (Table 6). The review authors did not analyze all these sub-domains for all studies included in those reviews.

Table 6 Description of domains in Cochrane reviews that had multiple separate domains for assessing attrition bias for different outcomes

Cochrane reviews that did not have a domain for attrition bias in the RoB table

There were 12 Cochrane reviews that did not have a domain for attrition bias at all in the RoB table. They were not included in the main analysis, and hereby we report characteristics of their RoB tables. Five reviews analyzed only 1 RoB domain, and this was ‘Allocation concealment in four cases (reviews #341, #465, #672 and #904) and ‘Method for selecting cases to adjudicate?’ in one case (reviews #269). One review analyzed 3 RoB domains (Random sequence generation, Allocation concealment and Blinding as one domain for all outcomes), but not attrition bias (review #294). Three reviews analyzed 4 RoB domains; one of them analyzed ‘Random sequence generation’, ‘Allocation concealment’, ‘Blinding of outcome assessment’, ‘Selective reporting’ (review #585) and two analyzed domains for ‘Random sequence generation’, ‘Allocation concealment’, ‘Blinding of participants and personnel (performance bias)’, ‘Size’ (review #924, #936). Two Cochrane reviews analyzed five RoB domains (review #174, #947) and one analyzed six RoB domains – but none of the domains were attrition bias (review #309).

Risk of bias assessed with ‘yes’ or ‘no’ judgments

In 4/729 Cochrane reviews (0.5%) there was no standard judgment of risk of bias as high, unclear or low; instead RoB was judged as yes, unclear, no, or yes/no (reviews #212, #292, #830 and #884). In one review risk of bias was graded as “low, unclear or high”, but in the supporting explanation also rated as A – Adequate, B – Unclear, C – Inadequate (review #244).

Other inconsistencies that were encountered

Several Cochrane reviews had different name of the relevant domain. In the review #641 the domain was called “Intention-to-treat analysis performed?”, in the #419 “Losses to follow-up taken into account?” and in the review #873 “Complete follow-up?”

Explanations that should not be used for judging attrition bias

Finally, we decided to report examples of curious explanations for attrition bias judgments in Table 7. It appears to us that such explanations should not be used for explaining risk of attrition bias judgments.

Table 7 Examples of curious supporting explanations for attrition bias judgments that may not appear to be suitable for judging this risk of bias domain

Discussion

We found high inconsistency in the assessment of risk of bias related to incomplete outcome data, i.e. attrition bias in Cochrane systematic reviews. Cochrane authors do not have uniform approach to judging attrition bias. We did not observe clear numerical rules about the percent of attrition in trial groups or clear rules about statistics that was used or not used, that were consistently labeled as low, unclear or high risk of bias. One third of Cochrane review authors had more than one category of explanations; some had up to four different categories. Inconsistencies were found even with the number of judgments, names of risk of bias domains and different judgments for the same explanations in the same review.

Cochrane Handbook indicates that “Missing outcome data, due to attrition (drop-out) during the study or exclusions from the analysis, raise the possibility that the observed effect estimate is biased.” The term attrition bias is used for both exclusions and attrition [3]. Besides numerical indicators of attrition – absolute numbers and frequencies – that provide information about the magnitude of attrition, in the context of this domain of risk of bias different statistical methods for imputing missing data are often mentioned. For example, trial authors can use ITT analysis, or a ‘modified ITT analysis’. However, it has been reported that the term ‘ITT analysis’ does not always have a clear and consistent definition, and that it is not consistently used in trial reports [7]. The same was concluded for the modified ITT analysis and therefore it has been recommended by the Cochrane Handbook that the review authors should always ask information about who exactly was included in such analysis [3]. Simple imputations, such as last observation carried forward (LOCF) remain very popular despite warnings of statisticians against their use [8].

Judgments about different statistical methods varied in our analysis; we found very inconsistent judgments for different statistical methods. If we want to judge by the frequency of statistical comments in reviews where this was the only available explanation, we could not reach any conclusion, because the majority of authors judged presence of ITT analysis with low risk of bias, but also in the group that reported explicitly that there was no ITT analysis, this absence of ITT analysis was also predominantly judged with low risk of bias. Using per protocol analysis was mostly judged as low risk of bias, as well as LOCF analysis.

It has been published previously that attrition under 5% is not likely to introduce bias, while attrition rates above 20% raise concerns about the study validity [9]. While Cochrane handbook does not give clear guidance about the total attrition or attrition per group regarding specific numerical values, there is an example in the Fig. 8.6.a. in that handbook: “17/110 missing from intervention group (9 due to ‘lack of efficacy’); 7/113 missing from control group (2 due to ‘lack of efficacy’)” that is judged as high risk [3]; in this example the first group has attrition of 15%. If a Cochrane author should follow this example, then attrition that is 15% or above per group should be labeled as high risk of bias. In Table 8, we present examples of vague instructions for Cochrane authors regarding judgments of attrition bias, in line with the current instructions for judging attrition bias that are available in the Cochrane Handbook in Table 8.5.d., which gives authors instructions about specific situations where each domain should be judged as low, unclear or high [3].

Table 8 Examples of vague instructions for Cochrane authors regarding judgments of attrition bias

In our study we found that numerical indicators for what represents attrition were widely inconsistent. When we categorized reported percent of attrition in the group with higher attrition and which threshold was predominantly judged in a certain way, attrition in a group that was under 10% was judged as low risk of bias in 83% of the cases, attrition 10–20% was judged as low risk of bias in 64% of cases, attrition 20–30% was judged as low risk of bias in 57% of cases. If we judge from the majority opinion of Cochrane authors, threshold of ‘above 30% is considered predominantly high risk of bias because 61% of judgments indicated so in Cochrane reviews where this was the only judgment so we could isolate the effect of this category for the overall judgment.

As for the risk of bias as a tool, it has been reported that it has low reliability between individual reviewers and across consensus assessments of reviewer pairs [10]. It has been argued that low reliability of the RoB assessment can have negative effects on decision making and quality of health care [11]. It has also been shown by da Costa et al. that standardized intensive training on RoB assessment may significantly improve the reliability of the Cochrane RoB tool [6]. However, our study points out that we would need first to have standardized instructions about what situations really represent risk of attrition bias. Having clear instructions, such as “attrition above 20% represents high risk of attrition bias” it would be much easier to achieve higher reliability of RoB assessment, even without formal training.

Instructions for assessing risk of attrition bias should include specific indications about all categories of assessment that should be appraised. It should be clearly specified which of those categories systematic review authors should assess, such as four that we used in this manuscript, including percent of attrition per group and difference between the groups, whether reasons for attrition were reported or not, and what is the appropriate statistics for dealing with attrition. If the authors do not have clear guidance about assessment of attrition bias, they can behave as we found – they can use one or more of those categories for their attrition RoB assessment as they personally see fit.

Some authors used multiple judgments for different follow-ups or different outcomes. This also introduces inconsistency in the attrition RoB assessment. Just as the option for authors to change the titles of attrition RoB domains in the RoB table in a Cochrane review.

In our previous analyses of other domains of Cochrane RoB tool in Cochrane reviews have shown that judgments and supports for judgments in those other domains were very inconsistent as well [12,13,14], further supporting the idea that more attention needs to be devoted to the way authors use this tool.

New version of Cochrane RoB tool, called RoB tool 2.0 is being developed, and its draft version is available online [15]. The draft version of the RoB tool 2.0 has five domains, the domain comparable to the current “Incomplete outcome data (attrition bias)” is the third out of five domains, called “Bias due to missing outcome data”. The RoB tool 2.0 has signaling questions in each domain, and this particular domain has three signaling questions [15]. Theoretically, having three signaling questions could help authors to produce three categories of responses, but this will not be the case because some of those signaling questions address more than one category of attrition bias, in the context of categories defined in this manuscript. For example, elaboration for the second signaling question includes both discrepancies in missing data across intervention groups, and reporting reasons for missing data [15].

Furthermore, we consider that this specific domain in the RoB tool 2.0 is not even a step forward in terms of specific instructions to Cochrane authors, because the field “elaboration” of the signaling questions is still as vague as in the current RoB tool, and could be interpreted by Cochrane authors in various ways. The first signaling question is “3.1 Were outcome data available for all, or nearly all, participants randomized?”. In the elaboration for the first signaling question there is a phrase “low or modest amount of missing data”, but it is not specified what exactly should Cochrane authors consider as “low” and “modest”. The elaboration further says “availability of data from 95% (or possibly 90%) of the participants would often be sufficient”, but it is unclear what is "often" and when is this not sufficient [15].

The second signaling question is “Are the proportions of missing outcome data and reasons for missing outcome data similar across intervention groups?” Elaboration does not give specific instructions about the magnitude of discrepancies; instead it says “minor degree of discrepancy” [15].

The third signaling question is “Is there evidence that results were robust to the presence of missing outcome data?”, and the elaboration says “Evidence for robustness may come from how missing data were handled in the analysis and whether sensitivity analyses were performed by the trial investigators, or from additional analyses performed by the systematic reviewers”. [15]. Again, to us, this elaboration does not give specific instructions to Cochrane authors, and may result in heterogeneous perception and judgment.

Future studies on this topic should explore how to reduce inconsistency in assessment of attrition RoB, and they should attempt to reach consensus about what exactly should be assessed in this RoB domain.

Conclusion

We found very high inconsistency in methods of appraising risk of attrition bias in recent Cochrane reviews. Systematic review authors need clear guidance about different categories they should assess and judgments for those explanations. Clear instructions about appraising risk of attrition bias will improve reliability of the Cochrane risk of bias tool, help authors in making decisions about risk of bias and help in making reliable decisions in healthcare.