Background

Chronic wounds, of which venous leg ulcers, pressure ulcers and foot ulcers make up the majority [1], are prevalent conditions that have considerable impact on patients’ quality of life and are costly to treat. For example, venous leg ulcers are estimated to affect 15 to 18 in 1,000 people in the United Kingdom (UK) and cost between £300 and £600 million (UK pounds) per year in health expenditure [2]. Despite the clinical and financial significance of chronic wounds, a number of commentators have expressed concerns about both the quantity and quality of RCTs within the broader area of wounds research [3–5].

RCTs within the field of chronic wounds, as within all areas of medicine, represent an essential part of this evidence base due to their ability to provide unbiased estimates of relative treatment effects. However, the generation of such unbiased estimates depends on adequate methodological measures being taken to minimise bias. For example, it has been demonstrated that failure to conceal allocation and failure to blind outcome assessors and participants results in biased, larger estimated effect sizes [6–12]. The failure to specify a primary outcome a priori can also introduce bias as authors are free to cherry pick outcomes for reporting that show a statistically significant difference [13]. As well as methodological features of RCTs that can minimise bias, other elements of study design also impact the overall quality of evidence generated. These design features include sample size, choice of primary outcome and the duration of post-randomisation follow-up. Trial sample size is important because while estimates from small trials are not necessarily biased, they will be under-powered to detect anything but the largest treatment effects, which also leads to small trials being associated with publication bias [14]. The choice of primary outcome is similarly important and needs to be meaningful to both clinicians and patients. Duration of follow-up is linked to issues of study power and outcome selection. Important outcomes such as wound healing can take a long time to achieve (often months for chronic wounds). Studies with a short duration of follow-up will potentially miss outcome events and be underpowered. One approach is the selection of other surrogate outcomes that can be measured over a shorter period but which may not be valid proxies for, or may be harder to interpret than, clinically meaningful outcomes.

The importance of methodological characteristics, such as those outlined above, has led to a number of studies being conducted that sought to quantify the methodological quality on both samples of RCTs from across medicine [15, 16] and within specific areas [17–20]. However, despite the methodological concerns described above, no such overview exists within the area of chronic wounds - an area of medicine where most treatments are medical devices for which RCT data are not necessary for licensing and marketing.

Previous research in other areas of medicine suggests that funding can have an important impact on a number of trial characteristics. For example, studies have observed that industry-funded drug trials are more likely to report in favour of (the sponsored) treatment when compared with research not funded by a commercial organisation [21–24]. Studies have also shown commercial funding to be associated with shorter duration of follow-up, [25] more frequent use of non-active comparators [26, 27] and a tendency to be at a lower risk of bias [24, 28–30]. In the area of chronic wounds, there is little data regarding the prevalence of industry-funded trials or the influence of industry funding on trial design. This study is a methodological review of recent trial evidence in the field of chronic wounds. It aims to assess the prevalence of industry-funded trials in chronic wound care and identify whether funding source is associated with features of methodological quality.

Methods

This study was protocol-driven with all search terms, selection criteria and analysis pre-specified.

Selection

A library of eligible RCTs in chronic wound care was assembled and used. Eligible studies were:

  1. 1.

    Randomised comparisons of treatments for foot, leg or pressure ulcers in any setting and for any category of patient (studies that included other wound types as well were excluded)

  2. 2.

    Published between 2004 and 2011 (inclusive)

  3. 3.

    Reported in English (due to a lack of translation resources)

Studies were excluded if they were:

  1. 1.

    Secondary reports, where the primary report of the main study was referenced

  2. 2.

    Conference abstracts (since these typically do not contain sufficient methodological information in order to assess risk of bias (RoB)

  3. 3.

    A protocol of a trial

  4. 4.

    A cost effectiveness study (unless they also reported the effectiveness of an intervention)

  5. 5.

    A phase 1 trial (since these are not aimed at determining effectiveness)

Trials were considered to be a RCT and included if they were described as ‘randomised’ either in the title or in the text of the paper. If no randomisation process was referred to, the study was excluded.

This library was constructed by searching the Cochrane Wounds Group Specialised Register of Trials. The Register is maintained by the Cochrane Wounds Group and aims to identify all randomised and quasi-randomised controlled trials in the area of wounds research. Reports are identified for inclusion in the register by regular searches of a number of databases including Medline, Embase, CINAHL and Central along with periodic searches of other databases. Studies included in the register have been coded on several criteria including wound type. A search was therefore carried out using the following search terms in the condition field: Pressure* or Venous or Leg* or Ulcer* or Diabet*, and intervention field: Treat*.

The titles and abstracts (where available) of the identified studies were screened by a single author to exclude studies which obviously did not meet the inclusion criteria. The full text was obtained for the all studies that were potentially relevant. Two authors independently checked the full papers for eligibility. Any disagreements were resolved through discussion, and where agreement could not be reached, a third author acted as arbitrator.

Data extraction

For each eligible study relevant data were initially extracted by one of seven authors. A second independent extraction was completed by a single author (blinded to the initial extraction). Any disagreements were resolved through discussion. Where agreement could not be reached, a third author arbitrated. The process of data extraction was extensively piloted with data recorded on a Microsoft Access database (Microsoft, Redmond, WA, USA - http://office.microsoft.com/en-gb/access/), using drop-down menus where possible.

Data extracted included the publishing journal, impact factor of publishing journal, journal specialty, wound type, study design (parallel or other), number of randomised groups, sample size, duration of follow-up, funding source, primary outcome if defined, and information on the interventions and comparators investigated (further information on decision rules for data classification can be found in Additional file 1). Risk of bias was assessed using the Cochrane risk of bias tool [31, 32] for the domains sequence generation, allocation concealment and blinding of outcome assessors. An overall assessment risk of bias was also made following Cochrane guidelines [32]. See Table 1 for definitions used in assessments.

Table 1 Definitions used in assessment
Full size table

Data analysis

Descriptive summary statistics were calculated for each of the general and methodological items specified, and outcomes were stratified by wound type and by funding source. The descriptive summary statistics were then used to compare the methodological quality of commercially funded and non-commercially funded randomised trials for which mean differences or odds ratios (OR) (as appropriate) with 95% confidence intervals (CI) were calculated. To assess difference in study duration between studies with different funding sources, mean differences were logged. Results were initially recorded in Microsoft Access (Microsoft, Redmond, WA, USA - http://office.microsoft.com/en-gb/access/) and Stata 12 (StataCorp, College Station, TX, USA – http://www.stata.com) was used for data analysis.

Results

The results of the study selection process and reasons for exclusion are presented in Figure 1. We identified 647 potentially eligible studies. After an initial assessment of titles and abstracts, 385 studies were excluded and full text copies of the remaining 262 potentially eligible studies were obtained. Ninety-five were excluded for the following reasons: 36 concerned an ineligible wound type; 26 had an ineligible study type/design; three were duplicate publications of existing studies; 23 were secondary reports of included studies; 1 paper was not in English; 2 were trial protocols; and 4 studies were excluded due to exceptionally poor reporting, which made judgement of whether the study met the inclusion criteria impossible. The remaining 167 studies met the inclusion criteria.

Figure 1
figure 1

Overview of review process.

Full size image

Table 2 presents the general characteristics of the 167 included studies stratified by wound type. Leg ulcers were the most frequently investigated of the three wound types (38%; 63/167), closely followed by foot ulcers (34%; 57/167). Only 19% (31/167) of trials focused on pressure ulcers over the eight-year period investigated. A further 9.6% (16/167) of trials investigated more than one wound type.

Table 2 General characteristics of included studies
Full size table

The majority of trials were parallel in design (96%; 160/167), with only 4.2% (7/167) using other trial designs (either crossover or factorial). Most (parallel) trials had just two intervention arms (90%; 144/160); 8.8% (14/160) had three arms, and 1.3% (2/160) had four arms. The most frequent intervention types investigated were dressings and topical agents (25%; 42/167), drugs (20%; 33/167), growth factors (9.6%; 16/167), bandages and stockings (8.4%; 14/167), and tissue grafts (6.6%; 11/167). There were a further 23 different technologies investigated.

The majority of trials were reported in specialty journals (93%; 156/167), with 34% (57/167) published in wounds journals. The remaining studies (7.0%, 11/167) were published in general medical journals. Of the included studies, 77% (128/167) were published in a journal with an ISI impact factor. The median impact factor for the 128 studies was 1.93, (interquartile range (IQR 1.21 to 3.01)).

In total, 35% (58/167) of studies were reported as having been commercially funded, 33% (55/167) were not commercially funded, and 6.0% (10/167) had funding from both commercial and noncommercial sources. The remaining 26% (44/167) of studies either did not report the source of funding, or the status of the funding source was unclear.

Table 3 presents information on a number of methodological characteristics of the included trials. The median number of participants was 60 (IQR 35 to 99), with a median of 28 (IQR 16 to 48) participants per treatment arm.

Table 3 Methodological characteristics of included studies
Full size table

The median duration of follow-up of the trials was 12 weeks (IQR 6.7 to 24) and varied significantly depending on the nature of the primary outcome. Studies reporting surrogate measures of healing (for example, change in ulcer size) had a median follow-up of 12 weeks (IQR 7.3 to 15.9) compared with 24 weeks (IQR 12.9 to 26.1) for studies with a primary outcome of either time to complete healing or proportion of wounds completely healed.

The proportion of trials that defined a primary outcome measure was 59% (98/167). Of those that defined a primary outcome, 41% (40/98) reported a measure of complete healing as the primary outcome (either time to complete healing or proportion of wounds completely healed), 48% (47/98) reported an intermediate (surrogate) measure of healing (for example, change in ulcer size/area) and 11% (11/98) of trials reported a primary outcome that was unrelated to healing (for example, pain).

The Cochrane risk of bias assessment [30, 31] revealed that of the 167 included trials, only 40% (67/167) described an appropriate method of random sequence generation; 59% (98/167) were unclear or did not report the method of sequence generation and 1.2% (2/167) reported using inappropriate methods of sequence generation such as alternation or date of birth. Only 25% (41/167) of trials reported adequate methods of allocation concealment, such as remote telephone randomisation or sealed, opaque, sequentially numbered envelopes; 74% of (123/167) trials were unclear or did not report their method of allocation concealment; and 1.8% (3/167) did not adequately conceal allocation. With regards to blinding of outcome assessors, 34% (56/167) of trials were classified as at low risk of bias, 38% (64/167) were at unclear risk and 28% (47/167) of trials were at high risk of bias. Following Cochrane review guidelines [31] to construct an overall risk of bias assessment, 30% (50/167) of trials were judged to be at high risk of bias, 64% (107/167) at unclear risk of bias and just 6.0% (10/167) at low risk of bias.

Table 4 presents the methodological characteristics of the included studies stratified by funding source. The sample sizes of commercially funded trials were not statistically significantly different from those of non-commercially funded trials (difference 8.0 participants, 95% CI −25 to 41). Differences in duration were modelled using a log model due to the data being highly skewed. There was no statistically significant difference in the duration of follow-up between commercially funded and non-commercially funded trials (with commercially funded studies having, on average −1.3 months, less follow-up , 95% CI −1.8 to 1.06 months). Commercially funded trials were not more or less likely to specify a primary outcome than non-commercially funded trials (OR 1.48, 95% CI .74 to 2.94), and were no more likely to identify a surrogate healing measure as their primary outcome (OR 0.68, 95% CI 0.29 to 1.57). There was no statistically significant difference between commercially funded and non-commercially funded trials in the likelihood of being classified as at either high or unclear risk of bias in the domains of sequence generation (OR 0.63, 0.34 to 1.16), allocation concealment trials (OR 1.08, 95% CI 0.62 to 1.88) or blinding of outcome assessors (OR 0.84, 95% CI 0.47 to 1.50). There was no statistically significant difference between industry-funded and nonindustry-funded trials in the odds of being classified as at high or unclear risk of bias overall (OR 0.91, 95% CI 0.54 to 1.52).

Table 4 Methodological characteristics of included studies by funding type
Full size table

Discussion

Key findings

This study aimed to describe a number of methodological characteristics of trials in chronic wounds published during the period from 2004 to 2011 and identify whether funding source was associated with the methodological quality of trials. Trial reports in chronic wounds tend to be published in comparatively low impact journals with a significant proportion of studies being published in journals with no published impact factor.

The average sample size observed in our sample of trials was very small. The statistical power of a hypothetical trial involving 60 participants (the median sample size), where approximately 50% of ulcers heal in 12 weeks [32] (the median duration of follow-up) and where it is assumed there is a modest effect size of 15% [32], would be just 21%. In other words, only one in five of the trials in our sample are sufficiently large to detect a statistically significant treatment effect should one exist. The vast majority of chronic wounds trials are therefore underpowered to detect all but very large effects. While there is some debate regarding the importance of adequately powering a trial to detect a difference, [33–35], there is evidence that published small trials yield larger effect sizes than large trials [14], most likely as a result of publication bias. The fact that many chronic wounds trials are small means that it is very important that systematic reviewers check for publication bias wherever possible. Equally, it is important that authors register trials so that reviewers can identify on-going and potentially unpublished material.

Whilst many trials specified a primary outcome, a significant proportion did not. There were opportunities for authors to cherry pick results (that is, presenting outcomes for which their statistically significant differences). In a related study, Lockyer et al.[36] found that 86% of wound trials that did not define a primary outcome claimed a significant treatment effect. This raises deep suspicion of bias given the inadequate statistical power of trials in this field.

Our results show that the use of surrogate healing measures (for example, change in ulcer size) is widespread and indeed the majority of studies that define a primary outcome use these intermediate measures of healing. The cost of longer term follow-up is likely to be the primary reason for this. Clearly such cost reductions represent an advantage of utilising surrogate outcomes, and there is good evidence that a number of surrogate endpoints such as initial change in ulcer size are good predictors for the clinically meaningful endpoint of complete healing [37–39]. There are, however, a number of reasons to remain cautious about using surrogate endpoints. Firstly, surrogate measure of healing such as change in ulcer size or area introduce the complex issue of how to measure ulcer size and opens the potential for inaccuracies and lack of reliability in systems of measurement. A recent systematic review into the performance of instruments for the measurement of pressure ulcers revealed little evidence to support the reliability of different methods of measurement, particularly with regard to ulcer depth and volume [40]. Secondly, while some studies have shown surrogate healing measures to be good predictors of complete healing, what is less clear is their ability to differentiate between treatment effects. It is not enough that a surrogate predicts the clinical endpoint, it must also be able to predict changes in the clinical endpoint due to different treatment effects [41, 42].

The results of the risk of bias assessments raise a number of concerns. Across all three bias domains, a minority of trials were at low risk of bias. Assessment of bias was, however, hindered by poor reporting with the vast majority of studies at unclear risk for at least one domain. While it is encouraging that only a small minority of studies were at high risk of bias for either sequence generation or allocation concealment, it is impossible to make any judgement about the prevalence of high risk of bias trials in these domains when such a large proportion of trials are poorly reported. Previous research has suggested that reporting is worse than conduct [43, 44] and perhaps the majority of studies are at low risk of bias, however the previous studies [43, 44] were conducted in highly regulated fields and it would be inappropriate to generalise to the area of chronic wounds trials. Even under the most optimistic scenario (that the unclear risk trials reflect poor reporting rather poor trial methodology) there remains a substantial risk of bias in chronic wounds trials due to almost a third of trials not blinding outcome assessors.

The methodological quality of commercially funded trials and non-commercially funded trials was not significantly different across all the included measures of methodological quality. Based on current evidence it is not possible to draw the conclusion regarding the influence of funding source on the methodological quality of chronic wounds trials, although we acknowledge the limited power in these analyses that in turn also limits the conclusions that can be drawn. Furthermore, the overall standard of trial quality was very poor and wound trialists share a culture that international trial design, conduct and reporting standards do not seem to have penetrated. Almost a quarter of the sample had an unclear funding source; more transparency in declaring support for research will allow more studies to be included in future analysis of this type.

Comparison with other studies

Our results are very similar to those obtained in other methodological reviews that used samples of trials from across medicine (for example, Hopewell et al. [16]). Chronic wounds trials are therefore not considerably methodologically weaker than trials conducted in some other areas of medicine. The cohort assessed in Hopewell et al.[16], however, was much older and dated back to 2000 and this study includes trials up to 2011, thus limiting comparability, particularly as the impact of CONSORT may mean that trials are now of higher methodological quality than observed by Hopewell et al. [16]. It should also be noted that the methodological quality of reports assessed by Hopewell et al. [16] was poor and the fact that chronic wounds trials may be of comparable quality is no reason for complacency or celebration.

Limitations

The initial selection of studies was carried out by a single reviewer and lack of translation services means that only English language papers were included. As a result our sample may not be fully representative of all the chronic wounds trials published during the period 2004 to 2011. While it is difficult to predict how inclusion of non-English language studies would have impacted the findings, evidence from other fields suggest that methodological quality is similar or slightly higher in English language publications, compared with those published in other language [45–48]. On this basis, our results would represent a conservative assessment of methodological quality of chronic wounds trials.

A further limitation of this study is that it was based only on information contained within trial reports and does not necessarily mean poor trial conduct. Studies [43, 44] have compared the content of published reports with their protocol and found that the methodological techniques in the actual trial were of better quality than reported in the final publication. However, these focused on specific areas of medicine, and as such, the results may not be generalizable to a chronic wounds context. Assessment of review protocols and contacting trialists for more information may have revealed that the trials were of better quality than reported in the final publication. This was not done in this study.

Conclusions

This methodological review of recent chronic wounds trials presents a profile of the recent trial literature in this area. It identifies that chronic wounds trials tend to suffer from a number of methodological deficiencies, with trials tending to be small and underpowered, with short follow-up periods, widespread use of surrogate healing measures and frequently no primary outcome specified. Further, on the basis of Cochrane review guidelines, a significant proportion of trials would be considered to be at either unclear or at high risk of bias.

The fact that such a large proportion of the trial evidence base is methodologically weak inevitably has a huge impact on the potential for current chronic wounds trials to inform practice. Given the considerable expense of conducting RCTs and the ethical implications of recruiting participants to methodologically weak studies, much greater efforts must be made by researchers to ensure that best practice is followed wherever possible. In particular, much greater efforts must be made to blind outcome assessors [49]. While there are occasions where blinding of assessors is not possible, it is important to emphasise that blinding is possible in wounds trials as has been demonstrated in a number of studies, often by using photographic assessment by masked adjudicators [33, 50, 51].

While funding source was not associated with greater risk of bias, the unfortunate reality is that, with few exceptions, chronic wounds trials are unacceptably poor across a whole range of measures of methodological quality. In the light of these findings the publication of methodological guidelines for comparative effectiveness research in chronic wounds by the Centre for Medical Technology Policy represents a positive step in improving the methodological quality of chronic wounds trials [52]. Moving forward, it is incumbent upon all stakeholders in chronic wounds research, irrespective of funding source, to insist on the improvements and recommendations outlined in these guidelines. If improvements are not made, there will continue to be a deficit in high quality evidence available to inform practitioners’ decisions regarding the best treatment options for their patients.