Background

Outcome reporting bias in a clinical study is defined as the selection of a subset of the original variables recorded on the basis of the significance and direction of the results [1]. It distorts the results of trials and leads to a reduction in the strength of evidence for conducting systematic reviews and developing clinical practice guidelines [2-5].

Prospectively trial registration in international clinical trial registries before participant enrollment is one of the main ways to potentially reduce selective outcome reporting and further outcome reporting bias [6,7]. The International Committee of Medical Journal Editors (ICMJE) announced a policy of requiring investigators to register clinical trials in a clinical trial registry before participant enrollment as a prerequisite for publication in 2004. This policy took effect in 2005 and is applicable to any clinical trial beginning enrollment after 1 July 2005 [8]. The World Health Organization formally established the International Clinical Trials Registry Platform (ICTRP; http://www.who.int/ictrp/en/) in 2004 to push for clinical trial registration. Furthermore, several peer-reviewed journals, such as The Lancet and Trials, publish study protocols to achieve transparency regarding performance and reporting of clinical trials.

Outcome reporting bias has received much attention. Many empirical studies evaluating the consistency concerning outcomes between the protocol or trial registry entry of randomized clinical trials (RCTs) and subsequent publications have demonstrated that outcome reporting bias is prevalent and favors statistically significant results [9-16]. A recent study focusing on RCTs of traditional Chinese medicine (TCM) showed discrepancies were identified in primary outcomes (29%) and safety reporting (28%) when comparing registered records with their subsequent publications [17]. However, the problem of outcome-reporting bias has not been systematically studied among RCTs of acupuncture.

Our objectives were to evaluate the consistency between the registered records and subsequent publications with respect to outcomes and other data as well as to determine whether outcome-reporting bias favors significant primary outcomes.

Methods

Inclusion criteria

We included RCTs on acupuncture alone or in combination with other interventions that were registered in trial registries. The current status was limited to ‘completed’ in the registered records. Acupuncture was defined as a collection of procedures involving penetration of the skin with needles manipulated by the hands or by electrical stimulation to stimulate certain points on the body [18]. We excluded phase 1 trials.

Search and selection of trials

A systematic search of 15 major international trial registries was conducted using the search terms “acupuncture” or “electroacupuncture” or “auriculotherapy” from their inception to January 2014. Trial registries included the Australian New Zealand Clinical Trials Registry (ANZCTR), Brazilian Clinical Trials Registry (ReBec), Chinese Clinical Trial Register (ChiCTR), Clinical Research Information Service (CRiS) public of Korea (KCT), ClinicalTrials.gov, Clinical Trials Registry-India (CTRI), Cuban Public Registry of Clinical Trials (RPCEC), EU Clinical Trials Register (EU-CTR), German Clinical Trials Register (DRKS), International Standard Randomized Controlled Trial Number Register (ISRCTN), Iranian Registry of Clinical Trials (IRCT), Japan Primary Registries Network (JPRN), Pan African Clinical Trial Registry (PACTR), the Netherlands National Trial Register (NTR), and Sri Lanka Clinical Trials Registry (SLCTR). If trials were listed as ‘completed’ in the registered records, the full texts of subsequently published articles were retrieved by searching PubMed without a language limit and three Chinese electronic bibliographic databases including China National Knowledge Infrastructure (http://www.cnki.net), Chinese Biomedical Database (http://sinomed.imicams.ac.cn/index.jsp), and Chinese VIP Information (http://vip.hbdlib.cn/index.asp) using the trial registration number, registered trial title, and investigator name. Identified articles were checked against the target registered records to determine whether they were a correct match. For each trial, all published articles reporting the final results were included. Abstracts and reports of preliminary results were excluded.

Data extraction

Two authors (Su CX, Han M, Ren J, or Li WY) independently extracted data using a standardized, piloted data extraction form. The form was modified based on the previous research [17]. We collected the following information reported in trial registries and published articles: trial design, sample size, participants and diseases/conditions, interventions, controls, primary and secondary outcomes along with the time frame of assessment, inclusion and exclusion criteria, and funding. For each registered trial, we also extracted the date registered, start and end dates of participant enrollment/the trial, sponsors, and country of origin. Publication year and language, journal name, registered number, and P values for all the outcomes were also extracted from each subsequently published report. The 2013 impact factor of journals where registered trials were published was sought. ‘Instructions for authors’ of the included journals was checked for whether the registration of the trial and reporting of the trial’s registration number were required in the published report. The “history of changes” in the registries was tracked to determine whether the registered information had been changed since the initial registration. Conditions were classified according to International Statistical Classification of Diseases and Related Health Problems, 10th Revision (ICD-10) (http://apps.who.int/classifications/icd10/browse/2010/en; last accessed on 27 January 2014).

Data analyses

Methodological quality of registered records and subsequent publications was assessed independently by two authors (Su CX, Ren J, or Li WY) using the Cochrane Collaboration risk of bias tool, which included generation of the allocation sequence, allocation concealment, blinding, incomplete outcome reporting, selective outcome reporting, and other risk of bias [19]. Other risk of bias was assessed based on the estimated sample size, explicit inclusion and exclusion criteria, and the risk of funding bias.

Two authors (Su CX, Ren J, or Li WY) evaluated the consistency and selective outcome reporting regarding the trial design, methodology, and outcomes from the registered records and subsequently published articles. The disagreements were resolved by consensus with a third party (Liu JP). We defined inconsistency as “when the items in the trial registry were not the same as those in the subsequent publications” [17]. If over 20% discrepancy was identified between the sample size in the registered trials and published reports, we considered it inconsistent [19]. Selective outcome reporting was defined as “when the primary outcome specified in the published articles was different, or changed from those defined in the trial registry, or the timing of assessment of the registered and published primary outcomes differed” [20,21].

A discrepancy was considered to favor statistically significant results (P < 0.05) if a new statistically significant primary outcome was described in the published articles or if a nonsignificant primary outcome was defined as a nonprimary outcome in the published articles [10,21].

Data were presented as counts, percentage and frequency for categorical variables. Proportions were compared by the χ2 test. P < 0.05 was considered statistically significant. SAS version 9.1 (SAS Institute Inc, Cary, NC) was applied for statistical analysis.

Results

Of 860 registered acupuncture trials retrieved, 399 indicated their current status as ‘completed’. After excluding 160 registered trials because of duplicates or being non-RCTs, non-acupuncture and phase I, we searched subsequent publications for 88 of the 239 registered trials (36.8%). The 88 trials produced 96 publications reporting study outcomes, among which all publications were in English (Figure 1).

Figure 1
figure 1

Flow diagram of study selection.

Full size image

Of the 88 included trial protocols, 65 (73.9%; 72 articles) were from western countries; 23 (26.1%; 24 articles) were from eastern countries, of which only 7 were performed in mainland China (excluding Hong Kong and Taiwan). The included trial protocols were registered in eight registries and the majority was registered in either clinicaltrials.gov (43/88, 48.9%) or ISRCTN (36/88, 40.9%). The first trial was registered in 2001. Only 19.3% (17/88) were registered before the start of the trial, suggesting prospective registration. The remaining 71 trials were registered after the study began, even after the completion of the study. The proportion of retrospective registration was similar for trials from western countries (54/65, 83.1%) and eastern countries (17/23, 73.9%; P = 0.33). When comparing trials starting before 1 July 2005 (36/40, 90.0%), the proportion of retrospective registration was higher than that in trials starting after 1 July 2005 (35/48, 72.9%; P = 0.04).

Ninety-six articles were published in 49 journals from 1997 to 2014 (Figure 2). Seven (7.3%) were published in alternative and complementary journals. Sixteen (16.7%) were published in journals whose 2013 impact factor was high than 10 (Additional file 1: Table S1). Twenty-seven journals (48 articles; 50.0%) provided guidance on trial registration and reporting of the registration number in their instructions for the author. Of 36 articles (37.5%) without description of the registration number, 50.0% (18/36) were published in journals requiring the reporting of the registration number.

Figure 2
figure 2

Number of included randomized clinical trials on acupuncture in the trial registry and subsequent publication.

Full size image

The characteristics of included articles are presented in Table 1. The majority of the included articles were parallel group (92, 95.8%), single center (55, 57.3%), two armed (59, 61.5%), and sponsored by a university, government or hospital (77, 80.2%). The five most frequent conditions based on ICD-10 were musculoskeletal (26.1%), nervous (19.3%), mental and behavioral disorders (17.0%), respiratory conditions (5.7%) and neoplasms (5.7%; Table 2). Manual acupuncture and electro-acupuncture were reported in 80 (83.3%) and 16 (16.7%) articles, respectively. Interventions in the treatment groups involved acupuncture alone in 70 (72.9%) articles or acupuncture combined with other interventions in 26 (27.1%) articles. There was considerable variation in the control group. Fifty-one articles (53.1%) applied sham/placebo acupuncture; other commonly used controls involved western medicine (26.0%), no intervention (9.4%) and non-pharmaceutical interventions (9.4%). A clear description of the primary outcome and secondary outcome was provided in 71 (74.0%) and 60 articles (62.5%), respectively. However, safety and health-economic outcomes were only reported in 38 (39.6%) and 5 articles (5.2%), respectively.

Table 1 Characteristics of the included articles
Full size table
Table 2 Conditions based on the ICD-10 classification treated by acupuncture
Full size table

Seventy-five subsequent publications (78.1%) provided detailed information on the generation of allocation sequences with the most use of a computer random number generator. Fifty-eight articles (60.4%) provided details on allocation concealment. Of 69 articles (71.9%) providing information on blinding, 37 applied single blinding, 28 double blinding, and 4 triple blinding. However, only 3.4% (3/88), 2.3% (2/88) and 61.4% (54/88) of the registered records provided information on the generation of an allocation sequence, allocation concealment and blinding, respectively. Incomplete outcome reporting occurred in 33 (34.4%) articles. The sample size of the 96 articles ranged from 10 (pilot trials) to 960, with the majority (88.6%) of sample sizes between 20 and 500 per trial (Additional file 2: Table S2). Only 34 articles (35.4%) carried out a sample size calculation. All registered records lacked information on sample estimation. The majority of the registered records and the subsequent publications clearly described the inclusion and exclusion criteria.

A comparison of registered and published primary outcomes was impossible in 25 articles (26.0%) because outcomes were not specified as primary outcome or secondary outcome in the subsequent publications. Of the remaining 71 articles, the primary outcomes of 32 articles (45.1%) were inconsistent with those specified in registered records (Table 3). The discrepancies involved a registered primary outcome omitted in the published article (22/71, 31.0%), an absent primary outcome in the registry defined as a primary outcome in the published article (9/71, 12.7%), a published primary outcome registered as a secondary outcome (1/71, 1.4%), a registered primary outcome defined as a secondary outcome in the published article (15/71, 21.1%) and different timing of assessment of the primary outcome in the published article and the registry (12/71, 16.9%). Fourteen articles had two reasons for the difference in primary outcomes, five articles had three reasons for the difference in primary outcomes, and one article had four reasons for the difference in primary outcomes. This inconsistency was also addressed in 13 articles published in journals requiring trial registration; the proportion of inconsistency was lower than those published in journals not requiring trial registration [13 of 38 (34.2%) vs. 19 of 33 (57.6%); P = 0.048]. Five of 13 articles were published in a high-impact-factor journal (IF > 10). When comparing trials from western countries with trials from eastern countries, the proportion of this discrepancy showed no significant difference [6 of 21 (28.6%) vs. 26 of 50 (52.0%); P = 0.07].

Table 3 Proportion of discrepancies in the specified primary outcomes between registered trials and published articles on acupuncture and discrepancies favoring statistically significant results
Full size table

Furthermore, 81.7% (138/169) registered primary outcomes were clinically relevant outcomes (patient-centered outcomes). Among them, 22.5% (31/138) clinically relevant outcomes from the 15 registered trial records were missing in the final publications, and 9.4% (13/138) clinically relevant outcomes from 11 registered records had been changed from the original primary outcome to the secondary outcome in the publications.

Of the 32 articles with inconsistency in primary outcomes between the published article and registry, the influence of this inconsistency could only be evaluated in 65.6% (21 of 32). Among them, 15 of 21 (71.4%) had discrepancies that favored statistically significant primary outcomes, while 6 of 21 (28.6%) favored nonsignificant primary outcomes.

Comparing the 88 registered records with their 96 subsequent publications, the other inconsistencies were identified in the inclusion criteria (54.7%), exclusion criteria (47.9%) and controls (22.9%) (Table 4).

Table 4 Comparison of methodological components between registered records and subsequent publications of randomized clinical trials on acupuncture
Full size table

The ‘history of changes’ was available in 44 registered trial records (44/88, 50.0%) from six registries. After the initial registration, a change of primary outcomes was identified in seven registered trial records (7/88, 8.0%), and 14 registered trial records (14/88, 15.9%) had the following changes: changes of study arms, sample size, blinding, controls and inclusion/exclusion criteria.

Discussion

To assess outcome reporting bias, we identified a sample of 96 articles of acupuncture trials. Among them, 33.3% (32/96) articles showed clear discrepancies between the registered primary outcomes and the published primary outcomes. The inconsistency was addressed in 13 articles published in those journals requiring trial registration, and 5 of 13 articles were published in high-impact-factor journals. Furthermore, the other inconsistencies between the registry records and subsequent publications involved the inclusion criteria (54.7%), exclusion criteria (47.9%) and control interventions (22.9%). Selective outcome reporting has received widespread recognition. In previous studies, inconsistencies of primary outcomes between the registered records and corresponding publications in Western medicine varied from 18% and 49% [22-24].

Prospective trial registration provides a good opportunity for the editors and peer reviewers to evaluate outcome reporting bias and other deviations between the registered records and subsequent publications. However, we found some evidence of selective outcome reporting in 45.1% of the articles and other discrepancies between the published articles and the registered records in terms of inclusion/exclusion criteria, blinding, sample size, etc. These results highlight that editors and peer reviewers may not cross-check the consistency between the submitted manuscript and the registered records to identify any discrepancies, and to further limit an outcome reporting bias, even though they had a chance to access the registered records. A previous qualitative study of editors’ views on trial registration and publication bias showed that 11 of 15 journals required trial registration as a requisite for publication, but most editors seldom check whether the trial is adequately registered, and some editors did not even enforce trial registration [25].

This study also demonstrated that the trial registration number was unavailable in 36 published reports. Our finding is consistent with previous research that also indicated poor reporting of the registration number in published reports [26]. This may hamper editors and peer reviewers in accessing the registration records and to further assess the selective outcome reporting [23]. Accordingly, an accurate trial registration number should be provided in the publications. Then, the editors and peer reviewers should systematically check the consistency between the submitted manuscript and registration records, especially regarding the adequate reporting of important items such as the primary outcome.

As highlighted in our results, which confirmed the earlier findings [20,21], changes in the primary outcomes, as specified in the publications, from those specified in the registration records favored statistically significant over nonsignificant results. Such changes pose a threat to the reliability of results and further preclude clinicians and policymakers from making correct clinical decisions. Possible reasons for the discrepancies include the following: a preference for primary outcomes with statistical significance to increase the opportunity of the trial for publication and to support the use of the treatment [23,27].

The main purpose of prospective registration of clinical trials is to enhance the transparency of the clinical trial and provide access to the trial protocol [6,8]. If the trial is retrospectively registered, the meaning of the trial registration is lost; if information in registries is insufficiently detailed, with some items not registered or inadequately registered, this information will be useless for comparing the consistency between the registered records and subsequent publications. Therefore, adequate registration is of great significance to safeguard against publication bias [21]. However, our study showed that the proportion of prospective registered trials was still considerably low even with the introduction of the ICJME initiative policy implementation; registered information differed widely among trials; essential information such as randomization and blinding was poorly registered in the registries. Hence, the principal investigators and sponsors have the responsibility to adequately register a clinical trial. Furthermore, greater efforts should be made to improve the quality control processes of trial registries [21].

Although more than half of journals have required trial registration following the ICMJE guidelines, some journals use ambiguous language (i.e., encouragement of the trial registration) in their instruction to authors. We found similar results in previous studies [22,28]. A lack of submission guideline clarity might contribute to explaining inadequate trial registration and poor reporting of registration numbers.

We acknowledge that our study has several potential limitations. First, only trials indicated as having a ‘completed’ status in registries were searched for publications. A recent study indicated that many authors seemed to forget to change the status of their study to being completed, even among published studies [29]. Therefore, those trials in the registries where the status was not listed as ‘completed’ but that had been completed may not have been searched for and included in this study, which may have caused a bias.

Second, only PubMed and three Chinese databases were searched for published reports on acupuncture; therefore, some trials that have been published and included in other databases may have been missed.

Conclusions

Although trial registration is now the rule, we find that prospective registration for RCTs on acupuncture is insufficient, selective outcome reporting is prevalent, and the change of primary outcomes is intended to favor statistical significance. These discrepancies in outcome reporting may lead to biased and misleading results of RCTs on acupuncture. To ensure publication of reliable and unbiased results, further promotion and implementation of trial registration are still needed.