Background

Randomised controlled trials (RCTs) are considered to be the “gold standard” of clinical research [1, 2], by which health care professionals make decisions about the efficacy and effectiveness of interventions [3–6]. However, poorly designed and reported studies continue to be published, leading to a compromised evidence base [7]. This can adversely influence meta-analysis findings and clinical practice recommendations [7–10]. As a result of the poor reporting of RCTs, the CONSORT (Consolidated Standards of Reporting Trials) statement was developed in 1996 [6], and updated in 2010 [11], with the aim of improving the quality of reporting of RCTs through standardization, comprehensiveness and transparency [6, 11].

Reporting research in manual therapies presents obstacles not experienced in medical pharmacological trials. Non-pharmacological trials, such as chiropractic RCTs, test complex therapeutic interventions, which tend to be multi-faceted [12]. As a result, they are more challenging to describe, standardise, reproduce and administer consistently to all participants involved in a clinical trial [12]. These variants, along with others, such as care provider’s expertise may substantially impact estimates of treatment effect [12]. This makes it imperative for such studies to adhere to the CONSORT 2010 [13] and CONSORT for Non-Pharmacologic Treatments statements criteria [12].

Reviews indicate that the quality of reporting of RCTs in the medical literature is less than optimal [14–18]. As a result, many reviewers have drawn conclusions that the overall quality of reporting was poor to moderate [7, 9, 19–21], and require improving [22–26].

To our knowledge there has not been an assessment of the quality of reporting of RCTs in chiropractic. As a result, the aim of this study was to assess the reporting quality of RCTs in chiropractic and to identify factors associated with better reporting quality. The candidate factors that were chosen for this study, have previously been identified in the medical literature as influencing the reporting quality of RCTs [8, 9, 18].

The objectives of this study were to:

  1. 1.

    Assess the overall quality of reporting of RCTs in chiropractic using a customised tool, based on the CONSORT 2010 and CONSORT for Non-Pharmacologic Treatments statements.

  2. 2.

    To report on 4 key methodological items that minimise bias, based on allocation concealment, blinding of participants and assessors, and use of intention-to-treat analysis.

  3. 3.

    To determine factors associated with higher quality of reporting.

We hypothesized that quality of reporting was influenced by industry funding, positive findings, larger sample sizes, latter year of publication and publication in non-chiropractic journals.

Methods

This study has ethics approval from Murdoch University, Research Ethics and Integrity Office: Ethics #2014/119. The study protocol has been published previously, [27] however an outline is presented below.

Study selection

We searched ten clinical trial registers (refer to Fig. 1) and two electronic databases (PubMed and the Cochrane Database of Systematic Reviews), to identify publications of RCTs involving chiropractic studies, published between January 2005 to July 2014. The search terms used were: “Spine” OR “Lower Extremity” OR “Upper Extremity” AND “Musculoskeletal Manipulations” OR, “Manipulation, Chiropractic” OR “Spinal Manipulative Therapy” AND “Chiropractic”. Full text articles of RCTs in the English language were included if they met inclusion criteria as outlined in Table 1. Article selection and data extraction was conducted by all authors independently, and disagreements were resolved by consensus.

Fig. 1
figure 1

Flow diagram of RCT selection (N = 35)

Full size image
Table 1 Inclusion and exclusion criteria
Full size table

We chose to limit this review to high-velocity, low-amplitude (HVLA) studies only, as manual manipulative procedures are the basis of training for all chiropractors. Furthermore, HVLA procedures are reported to be the most popular chiropractic adjusting techniques, used by 93 % of chiropractic practitioners in the US, with similar numbers internationally [28].

It should be recognised that, for the purpose of this study we have included RCTs where both chiropractors and non-chiropractors were involved in the delivery of the interventions, such as physiotherapists, physical therapists and osteopaths [29–32]. However, the HVLA interventions were all delivered by a chiropractor who was part of the study team. We also included studies where the HVLA intervention was the comparator rather than the primary intervention [32–36].

Pilot and feasibility studies were not included as the CONSORT checklist could not be applied to such studies without them being disadvantaged during scoring, in that they typically do not include all items from the CONSORT, such as a power analysis and ITT analysis. Similarly, studies not published as full papers were not included, as it is impossible to properly assess those papers against the CONSORT criteria.

Review strategy

The characteristics of included studies have been reported in Additional file 1. The characteristics of excluded studies have been reported in Additional file 2.

Rating the overall reporting quality

This study was modeled upon previously published medical studies assessing the quality of reporting RCTs [9, 14, 16, 18, 19, 37], which used the CONSORT checklists. Furthermore, the CONSORT was used, as it is considered to have both face and content validity and is a measure of methodological quality [38].

A 39-point customised CONSORT checklist was developed by three authors (FK, RB and BB) in order to ascertain the overall quality of reporting of chiropractic RCTs. The overall quality of reporting checklist was developed by integrating items from the CONSORT 2010 [13], and the CONSORT for Non-Pharmacological Treatments statements [12]. Twenty-two items were included from the CONSORT 2010 statement [13], i.e. items one through to 25, excluding items 21, 22 and 24. Items 21 (generalizability [external validity] of the trial findings) and 22 (interpretation of results), which are included in the discussion section, were excluded from the customised checklist because it is challenging to objectively evaluate them [7, 39]. Item 24 (access to trial protocol), was also excluded, as historically it was not a requirement to publish protocols prior to publication of results. Items from the CONSORT 2010 checklist included in our assessment tool are outlined in Table 2. In addition, nine items from the CONSORT for Non-Pharmacological Treatments statement [12], were included i.e. Extensions 1, 3, 4a, 4b, 4c, 8, 13, 15 and ‘New Item’ and are outlined in Table 3.

Table 2 Frequencies of CONSORT 2010 items from customized overall quality of reporting checklist (N = 35)
Full size table
Table 3 Frequencies of CONSORT for Non-Pharmacological Treatment statement items from customised overall quality of reporting checklist (N = 35)
Full size table

The assessment of the adequacy of reporting, was based on the CONSORT 2010 guidelines and its extensions [12, 13, 40]. Items were defined as ‘yes’ if they were clearly and adequately reported and received a score of 1; or ‘no’ if they were unclear or not reported at all, and received a scored of 0. Items that were not applicable to a specific study were defined as ‘not applicable’ (‘N/A’) and were coded 9. The overall quality of reporting score of the trial was calculated as a percentage of the items rated as ‘yes’ (with a score ranging between 0 and 39 points).

Key methodological items, that safeguard against biases [9, 18, 39], have also been reported in the literature [16], such as: allocation concealment (Item 9), blinding (Item 11), and use of ITT analysis (Item 16). The separate assessment of the key methodological items was deemed necessary because, even within published articles with high overall reporting scores, these are often under reported [38] (Table 4). Blinding of participants was scored separately to blinding of assessors. The question of blinding of care-providers was excluded for pragmatic purposes. It has been established that blinding manual therapy practitioners is virtually impossible [41, 42], with similar constraints to the blinding of surgeons in medical clinical trials [14].

Table 4 Frequencies of key methodological items from the customised CONSORT checklist (N = 35)
Full size table

All authors were involved in the scoring of the RCTs. Each RCT was scored by at least two authors, who were blinded to each other’s results. Results were collated, and any discrepancies were resolved via consensus.

Definition of trial characteristics

A “positive finding” in a trial was defined as a trial in which the chiropractic intervention was deemed by authors to have statistically significant results and hence was considered superior to the comparator (i.e. placebo/sham, usual care, standard care, medical care, other health care modality, no care or other chiropractic intervention). If the trial produced results that stated that the chiropractic manipulative therapy and the comparator both produced positive outcomes in the study, then the RCT was rated as “no” to the question of “positive finding”, as the chiropractic intervention was not deemed superior to comparator (Refer to Additional file 1).

Trials were considered to be industry-funded, if there was at least partial industry funding. Industry funding included chiropractic research organizations, chiropractic governing bodies or other industry organizations with potentially vested interests in the research. Chiropractic departments funding research within private chiropractic colleges were also deemed to be industry funding, whereas chiropractic and non-chiropractic departments within government educational institutions were considered to be non-industry. Trials that did not have any funding, were also classified as non-industry funding (Refer to Additional file 1).

Trials were considered as published in chiropractic journals, if the journal was dedicated predominantly to the advancement of chiropractic research, education and health care (Refer to Additional file 1).

Statistical analysis description

This study used descriptive statistics to characterise the overall quality of reporting of chiropractic RCTs, as well as the key methodological items. The percentage of trials that scored ‘yes’ to each CONSORT 2010 item were tabulated and are presented in Table 2. The percentage of trials that scored ‘yes’ to each item from the CONSORT for Non-Pharmacological Treatments, are presented in Table 3. The key methodological items are presented in Table 4.

Two continuous variables were dichotomised. The sample size variable was divided into a smaller group with n = 1–100 and a larger group where n > 100. The ‘year of publication’ variable was also divided into two time periods (2005–2007 and 2008–2014), which were used in an additional analysis. These time periods were created to distinguish between chiropractic RCTs published before and after the publication of the CONSORT Extension for Non-Pharmacological Treatments statement.

All univariate regression analyses explored associations between the outcome, i.e. the overall quality of reporting score and the exploratory variables (i.e. industry funding, positive findings, sample size group, year of publication and journal type). To test these five exploratory variables, we constructed five univariate models, which included each of the exploratory variables. The exploratory variables that produced results that had a p ≤ 0.1, in the univariate regression analysis, were included in the multivariate model [22, 39]. The intention for building this multivariate regression model, was in order to ascertain which of the exploratory variables were independently associated with higher overall quality of reporting scores for the 35 RCTs included in this study. The method used in the multivariate regression analysis was stepwise approach. In the final multivariate regression analysis, variables were considered statistically significant if p < 0.05.

An additional ‘final’ multivariable model was created. This model differed in that, the year of publication, which was originally used as a continuous variable, was substituted for the dichotomous variable (as described above). By dividing the year of publication variable into two time periods, pre and post introduction of the CONSORT Extension for Non-Pharmacological Treatments statement, we could analyse the data to investigate whether this new CONSORT statement, impacted the overall quality of reporting.

Variation Inflation Factors (VIFs) were used to test collinearity between exploratory variables. None of the VIFS were >10, indicating that there was no collinearity among the variables. All assumptions for normality and linearity were checked using the Mahalanobis’ and Cook’s Distance statistics. Statistical analyses were conducted using SPSS © 22.0.0.0 (IBM Corporation 2013).

Results

Sources yielded a total of 21,331 trials. Of the 85 studies that met the first round of inclusion criteria, only 35 (41 %) involving 4435 participants, were published as full-text articles in English journals (Refer to Fig. 1 and Additional file 1). These 35 articles, were assessed for their overall quality of reporting. The RCTs involved adult populations ranging between 17 and 78 years of age. Twenty-five of the 35 (71 %) RCTs reported positive findings in favour of the chiropractic intervention. Seventeen of the 35 (49 %) RCTs were published in a chiropractic journal. Only 43 % (15/35) of the RCTs were industry funded. The sample sizes of the included RCTs ranged between 20 and 444 participants with a mean of 127 (SD ± 102).

Overall quality of reporting score

The overall quality of reporting score, ranged between 10 and 33 with median score of 26.0 (IQR = 8.00). Individual scores are outlined in the Additional file 1. With regard to reporting frequencies of individual CONSORT items, refer to Tables 2 and 3.

Items that were most poorly reported from the CONSORT 2010 checklist were as follows: item 4b (settings and locations where data were collected), with 17/35 (49 %) of trials reporting this; item 11(a)(i) (whether participants were blinded) with 16/35 (46 %) of trials reporting this; item 13b (the description of each groups losses, exclusions and reasons within the flow diagram), with 20/35 (57 %) reporting on this; item 16 (i) (numbers analysed….in absolute numbers e.g. 10/20, not 50 %) with 16/35 (46 %) of trials reporting this; item 19, (the reporting of adverse events), with 22/35 (63 %) of trials reporting this; and item 23, (the reporting of clinical trial registration) with only 19/35 (54 %) of trials reporting this (Refer to Table 2).

The items that were most underreported, were from the CONSORT Extension for Non-Pharmacological Treatments checklist, with only one of the nine items achieving a high score. Item 4(a) extension requires the reporting of the description of the different components of the interventions and whether they were tailored to individuals, with 29/35 (83 %) of RCTs reporting this item. All other items from the CONSORT for Non-Pharmacological Treatments checklist were very poorly reported with an overall quality of reporting score ranging between 1/35 (3 %) for item 4c extension, (which details how adherence of care providers with the protocol was assessed or enhanced) through to 13/35 (37 %) for item 3 extension, (which describes the eligibility of centers or care providers of the interventions) (Refer to Table 3).

The scoring of the key methodological items also revealed some areas of weakness. Poor reporting of item 11(a)(i) (the blinding of participants) which was reported in 16/35 (46 %) of RCTs, and item 16(ii) (the ITT analysis) which was reported in 21/35 (60 %) of RCTs. The other two items were reported more frequently (Refer to Table 4).

Results of statistical analyses

The univariate regression analysis revealed that year of publication (t33 = 4.99, p = 0.000), journal type (t33 = 3.28, p = 0.002), and sample size group (t33 = 2.75, p = 0.010), were all individually and significantly associated with overall quality of reporting (Refer to Table 5 and Figs. 2, 3, and 4 respectively).

Table 5 Univariate and multivariate regression analysis for overall quality of reporting score vs exploratory variables (N = 35)
Full size table
Fig. 2
figure 2

Scatterplot of the correlation between the Overall Quality of Reporting Score and Year of Publication(N = 35)

Full size image
Fig. 3
figure 3

Boxplot of the distribution of Overall Quality of Reporting Scores for Chiropractic vs Non-Chiropractic Journals (N = 35)

Full size image
Fig. 4
figure 4

Boxplot of the distribution of Overall Quality of Reporting Scores for Sample Size 1–100 vs >100 (N = 35)

Full size image

The multivariate regression analysis subsequently revealed that year of publication (t32 = 5.17, p = 0.000), and sample size group (t32 = 3.01, p = 0.005), were the only two factors associated with the overall quality of reporting. For each additional year between 2005 and 2014, the overall quality of reporting score increased on average, by an estimated 1.26 points (95 % CI: 0.76, 1.76)(Refer to Table 5). Compared to the smaller sample size group (n = 1–100), the larger sample size group (n > 100) scored on average 4.19 points higher (95 % CI: 1.36, 7.02) (Refer to Table 5).

The additional multivariate regression analysis conducted with the two time periods for the year of publication, revealed that, compared to the period 2005–2007, chiropractic RCTs published between 2008–2014, scored on average 8.16 points higher (95 % CI: 4.64, 11.67) (Refer to Table 5 and Fig. 5). The outcome was not affected by this additional analysis, as the multivariate regression analysis revealed that year of publication and sample size were the only two factors associated with the overall quality of reporting.

Fig. 5
figure 5

Boxplot of the distribution of Overall Quality of Reporting Scores for Year of Publication by group, 2005–2007 vs 2008–2014 (N = 35)

Full size image

The final model in the multivariate regression analysis, revealed that 56 % of the variability in the reporting quality of the included RCTs can be explained by later year of publication and larger sample size (Adjusted R 2 = 0.556).

Discussion

This appears to be the first study investigating the quality of reporting of chiropractic RCTs relative to the CONSORT checklist. This study suggests, that there has been a significant improvement in the reporting quality of chiropractic RCTs between 2005 and 2014. This may be explained by an increased uptake of the CONSORT guidelines by journal editors and authors, but also by an increasingly professional cadre of chiropractic researchers. Furthermore, studies with sample sizes with n > 100, also revealed this trend. This is understandable, as studies with larger sample sizes are associated with greater resources. Furthermore, studies with larger sample sizes are also more likely to be adequately powered in order to find a statistically significant result, if in fact one exists.

While recent publications were more likely to adhere to the CONSORT 2010 criteria, the same cannot be said for the CONSORT Extension for Non-Pharmacological Treatments criteria. Some specific areas, such as: describing items related to care providers and centers, details of adherence to protocols, and how interventions were standardised and if training was administered as prescribed, were very poorly reported. Perhaps this is due to a lack of awareness within the chiropractic research community of these extension criteria.

Under-reported items from the CONSORT 2010 statement included: blinding; explanation of losses and exclusions after randomization with reasons on the flow chart; adverse event reporting; and analysis according to ITT principles, despite the fact these criteria have been established since the 2001 CONSORT statement [40].

Factors such as publishing in non-chiropractic journals showed a trend towards improved quality of reporting scores, although this was not statistically significant in the multivariate regression analysis.

Industry funding was not associated with improved quality of reporting of chiropractic RCTs. In contrast with several medical studies [7, 16, 18, 24, 39, 43], and some reviews [44, 45], which reported concerns that industry funding may be associated with publication bias [44–47].

We also found that a positive finding, was also not associated with the overall quality of reporting within the 35 chiropractic RCTs analysed. This was in contrast to several medical studies that reported that, improved quality of reporting was associated with positive findings [7, 39, 48]. One particular review found that there was a positive association between reporting of favorable outcomes among pharmaceutical trials registered in ClinicalTrials.gov and industry funding [48].

Transparency and accuracy of RCT reporting contributes to the evidence-based information for the profession and will make assessing the validity of RCT results easier. This in turn can lead to better decision-making, helping chiropractic professionals improve their clinical decision making and thus providing better outcomes for patients [49]. As the chiropractic profession is the largest non-medical healthcare profession [50, 51], it is important to continue developing the evidence base so as to inform evidence-based practices. This in turn, will enable the profession to maintain and broaden acceptance from the public, mainstream healthcare and policy makers.

The insights gained from this study should be viewed as an opportunity for improved reporting of RCTs and increased awareness as to the importance of using the CONSORT for Non-Pharmacologic Treatments statement amongst chiropractic researchers. To enhance the practice of evidence-based chiropractic care, researchers are encouraged to implement the CONSORT guidelines with greater rigor, especially in reporting of key methodological items, such as allocation concealment, blinding, and the use of ITT analysis. As these key methodological items can safeguard against bias in the execution and the reporting of future RCTs.

It has been known for some time that the quality of reporting has significantly improved in the medical literature with the adoption of the CONSORT guidelines [10]. Similar outcomes have been reported in a physiotherapy review [52], and a chiropractic review investigating low back and neck pain studies [53]. Our present study suggests that the quality of reporting in chiropractic spinal and non-musculoskeletal studies has also followed this trend.

Limitations

One limitation to this study was that it is possible that our search strategy did not capture every available chiropractic RCT. We searched ten Clinical Trial registries and two databases and only included published full-text articles in the English language. Additionally, we could not always verify the trial methodology from authors or check their protocols.

Our assessment does not offer any insight into the external validity of the RCTs analysed, as it was too challenging to rate the reporting of such items [7], as there have not yet been any scales developed that have been validated to accomplish this task [39].

Although the CONSORT Extension for Non-Pharmacological Treatments statement was published in 2008 [12], we decided to use time periods starting in 2005, because that was the year the International Committee of Medical Journal Editors published guidelines that required trials to be registered prior to participant enrolment as a precondition for publishing [54]. Furthermore, the original CONSORT was published in 1996 [6], revised in 2001 [55], and again in 2010 [13], and the original CONSORT items continue to exist in all the versions of the CONSORT statements.

Another limitation to the study is that we cannot generalize the results to all forms of chiropractic. We have included studies, which used adjunctive techniques as long as those RCTs also employed an HVLA procedure.

A potential weakness of this study is that, we created a sum score for the overall quality of reporting and used it for both descriptive purposes and as the dependent (outcome) variable for the regression analysis. A problem may be that the attributes we were adding are multi-dimensional and it may not be appropriate to simply add their scores together, as some items in the CONSORT carry greater importance than others. Furthermore, two RCTs may receive the same score but differ in the areas considered deficient with respect to reporting. This can make the overall quality of reporting score somewhat difficult to interpret between studies.

Conclusion

Reporting quality of RCTs varies widely in chiropractic research. While steady improvement has been observed over the last decade, the chronological improvement observed in this study appears to reflect a more thorough and stringent adoption of the CONSORT criteria. This study suggests that quality reporting was influenced by year of publication and sample size and may also have been be influenced by factors such as journal choice, but not funding source or outcome positivity. This should be regarded as a reassuring finding for the profession and scientific community.

Recommendations

In light of these findings, we have made some simple recommendations for the improvement of reporting of future chiropractic RCTs.

  1. 1:

    Researchers are encouraged to design and fully report studies to meet the requirements of CONSORT 2010 statement, with extra emphasis on key methodological items:- allocation concealment, blinding of participants and assessors and the use of ITT analysis.

  2. 2:

    Researchers are encouraged to incorporate items from the CONSORT Extension for Non-Pharmacological Treatments statement.

  3. 3:

    Researchers must register their clinical trials, which is in alignment with the standards established by the International Committee of Medical Journal Editors in 2005.

  4. 4:

    Chiropractic journals could exhort researchers to publish the protocols for RCTs in their respective journals for assessment of the study and statistical review, with the understanding that their results are more likely to be published if the protocol meets the CONSORT criteria. (The Lancet is just one of several medical journals that encourages this practice)