Skip to main content
Download PDF

Reporting quality of Cochrane systematic reviews with Chinese herbal medicines

Systematic Reviews volume 8, Article number: 302 (2019) Cite this article

Abstract

Background

Chinese herbal medicines (CHMs) are the major interventions of traditional Chinese medicine (TCM), which are typically administered as either single herbs or formulas. The Cochrane systematic reviews (SRs) of CHMs are essential references for evaluating the efficacy and safety of CHMs interventions; they are expected to be accurate and reliable. This study aimed to assess the reporting quality of these SRs, particularly whether necessary information related to CHM was adequately reported.

Methods

The Cochrane Database was systematically searched for all SRs of CHM that were published up to 31 December 2017. The primary analysis was to assess their reporting quality based on 27-item of the Preferred Reporting Items for Systematic reviews and Meta-Analyses (PRISMA) and 9-item of CHM-related information designed according to TCM theory. Descriptive statistics were additionally used to analyze their baseline characteristics.

Results

A total of 109 Cochrane SRs of CHM were identified from 1999 to 2017. For 27-item of PRISMA, 26 had the reporting compliances higher than 50%, of which 11 were fully reporting (100%). However, for CHM-related information, 65 (59.6%) SRs did not report the specific name of the CHM in the title, 42 (38.5%) lacked TCM-related rationales in the introduction, 62 (56.9%) did not include CHM-related characteristics in the additional analyses, and 77 (70.6%) did not analyze CHM results in terms of TCM-related theories in the discussion. Of 97 SRs that included clinical trials, 38 (39.2%) did not provide the details of composition and dosage of CHMs, 85 (87.6%) did not report the CHM sources, 13 (13.4%) did not provide the dosage form, 95 (97.9%) lacked CHM quality control information, and 57 (58.8%) did not describe details of the controls. For 62 (72.9%) of 85 SRs that included meta-analysis, it was impossible to assess whether meta-analysis had been properly conducted due to inadequate reporting of CHM interventions.

Conclusion

Although the Cochrane SRs of CHM showed reporting compliance with PRISMA checklist, their reporting quality needs improvement, especially about full reporting of CHM interventions and of TCM-related rationales. Reporting guideline of “PRISMA extension for CHM interventions” should be developed thus to improve their quality.

Peer Review reports

Background

Systematic reviews (SRs) summarize large bodies of evidence and synthesize all relevant studies that address a specific clinical question [1]. A meta-analysis (MA) is a tool that uses statistical methods to quantitatively combine and summarize the results of several independent studies in an SR [2]. SR/MA can help clinicians keep up to date with their field and policymakers judge the risks and benefits of health care behaviors; they provide a starting point for clinical practice guideline developers and summaries for funders seeking new research to support [3]. As with individual research reports, the value of an SR/MA largely depends on its transparent reporting [4].

Unfortunately, there is considerable evidence that the reporting quality of SR/MA is often poorly, thus diminishing their potential usefulness [5,6,7]. The suboptimal reporting quality of SR/MA led to the development of the QUOROM (Quality Of Reporting Of Meta analyses) Statement and its updated revision named PRISMA (Preferred Reporting Items for Systematic reviews and Meta-Analyses), published in 1999 and 2009, respectively [8, 9]. The PRISMA Statement consists of a 27-item checklist and a four-phase flow diagram, with an explanation and elaboration for each checklist item also published in 2009 [10]. The PRISMA checklists are used to guide authors of SR/MA to improve reporting quality. It is also a universal criterion to assess the reporting quality of available SR/MA publications [11,12,13]. For example, Tian JH et al. used the PRISMA checklist to evaluate the reporting quality of SR/MAs published in 2014, including 100 from China and 100 from the USA, and found that the PRISMA score was 21.2 (China) and 20.6 (USA), respectively. The authors concluded that the quality of SR/MAs from both countries needs to be further improved [14].

The Cochrane Collaboration is an international organization that aims to prepare and maintain rigorous systematic reviews in order to help people make well-informed decisions about health care [15]. Some scholars have found that the Cochrane reviews appear to have greater methodological rigor, are more frequently updated, and are less prone to bias than other reviews published in non-Cochrane journals [16, 17]. Handoll H et al. have indicated that most Cochrane reviews are of a good standard [18]. For example, Fleming PS et al. used the AMSTAR (A MeaSurement Tool to Assess systematic Reviews) checklist to assess and compare the methodological quality of Cochrane SRs (e.g., published in Cochrane Database from January 2000 to July 2011) and non-Cochrane SRs (e.g., published in five leading orthodontic journals from February 2002 to July 2011). They finally identified 109 SRs, including 26 Cochrane SRs, and found that the Cochrane SRs showed higher levels of methodology quality than non-Cochrane SRs (P < 0.01) [19].

Chinese herbal medicine (CHM), an essential part of TCM and the typical representative of TCM interventions, is recognized, more and more, as having profound value because of its demonstrated curative effects [20]. CHM interventions include Chinese medicinal substances (single herbs) and CHM formulas (“Fu-Fang,” or specific combinations of generally more than two Chinese medicinal substances). Chinese medicinal substances mainly originate from natural sources, including plants, animals, minerals, and some chemical or biological products; they can be raw (fresh or dried) or processed, while the CHM formulas are combinations of Chinese medicinal substances that are either individualized or fixed, often traditional, now sometimes patented [21]. Since the first SR of CHM was published in 1997 [22], the number of SR/MA of CHM increased rapidly [23]. For example, Chen M et al. analyzed 218 SR/MA of CHM published in Chinese journals from 1998 to 2008 and found that 82.1% were CHM formulas (including both patent proprietary CHM formulas and individualized CHM formulas), 10.1% were Chinese medicinal substance (including single herbs and herbal extracts), and 7.8% were unspecified [24].

In terms of the reporting quality of SR/MA of CHM, many studies have examined the compliance with PRISMA 27 items and have concluded similarly that the quality of SR/MA published in Chinese journals is poor and needs much improvement [25,26,27]. Also, the poor-quality reviews have been criticized for addressing too broadly defined topics and selecting too many different kinds of herbal medicines and formulas, thus leading to bias in the results [28, 29]. Although the quality of data provided in the primary clinical trials affects the quality of results in the SR/MA, proper synthesis of data and sufficient analysis of potential bias are also essential to determine the overall quality of the final results of SR/MA [30]. Therefore, it is necessary to collate the original information related to CHM, considering their source, dosage, duration, processing method, composition, and form of the CHM intervention, in order to make sure whether the interventions used in different clinical trials are the same [31]. However, no previous study has assessed whether CHM-related information is sufficiently collected and reported in the published SR/MA of CHM; nor has any study identified what key information related to CHM is the basis of the synthesis of results, especially for meta-analysis [32, 33].

Given the importance of reporting CHM-related information in SR/MA, a first step would be a systematic survey of CHM SRs to identify the common problems, if any. As Cochrane reviews are usually noted to have better methodology quality, this study aimed to examine the reporting of Cochrane reviews of CHM. In this study, the assessments of reporting quality were not only based on the standard 27-item of PRISMA but also on the 9-item of CHM-related information which specially designed according to the TCM theories of principles, methods, formulas, and herbs (also called “Li-Fa-Fang-Yao”). In clinical practice, the CHM treatment determination is usually guided with TCM theories of principles, methods, formulas, and herbs [34]. With reference to the reporting guidelines of CHM interventional trials [35], such as “CONSORT Extension for CHM Formulas 2017” [36], we have developed the 9-item checklist which reflecting CHM-related information and TCM rationale.

Therefore, this study had the following objectives: (a) to summarize the general characteristics of all included Cochrane SRs of CHM, (b) to assess the reporting quality of these SRs based on the PRISMA checklist, (c) to evaluate whether necessary information related to CHM is adequately reported, and (d) to assess whether these SRs are properly conducted in terms of synthesis of results (e.g., meta-analysis).

Methods

Inclusion and exclusion criteria

This study included all SR/MAs of CHM published in the Cochrane Library until 31 December 2017. The CHM interventions are typically administered as either Chinese medicinal substances (namely single herbs) or CHM formulas (namely “Fu-Fang”). Various dosage forms of CHM interventions, such as decoction, granule, pill, tablet, capsule, powder, medicated tea, medicated wine, oral liquid, plaster, and injection, were included. CHM interventions may have been administered alone or in combination with other interventions of conventional Western medicine or complementary alternative medicine. There were no limitations in the participants, comparisons, and outcomes. We excluded the following SR/MAs: repeat publications, comprehensive interventions focused on pharmacological treatment rather than herbal medicine, non-herbal TCM interventions (e.g., acupuncture, moxibustion, Taichi), non-TCM herbal medicine (e.g., Tibetan, Japanese), plant extracts (e.g., plant-derived chemicals or synthetic chemicals which contain constituents of plants), protocols, and withdrawal SR/MAs.

Search strategy

The Cochrane Database of Systematic Reviews was searched on 29 May 2018 for all Cochrane SR/MAs of CHM that had been published up to 31 December 2017. The search terms included “Chinese herbs,” “Chinese medicine,” “herb,” “traditional herbal medicine,” “Chinese materia medica,” “Chinese medicine prescription,” “formula,” “Chinese patent,” etc. The detailed search strategy is given in Additional file 1: S1.

Screening

The titles and abstracts of the SRs were independently screened by two researchers (XZ and Q-YA) based on inclusion and exclusion criteria, and the full-texts of potentially suitable articles were retrieved for further assessment. Disagreements were resolved by a third reviewer (Z-XB).

Data extraction

There were three pre-designed forms for data collection: (1) General characteristics form, including publication year, information of the authors, and descriptive information of included SRs. (2) PRISMA assessment form, including 27 items of the checklist. (3) CHM-related information form, which was designed according to (i) the reporting guidelines of “CONSORT Extension for CHM Formulas 2017”; and (ii) the TCM theories of principles, methods, formulas, and herbs. Aiming for easy calculation, the specifics of CHM-related information were categorized into nine items, including title, introduction, information source, eligibility criteria for participants and outcomes, study characteristics (for CHM interventions and control groups), additional analyses, synthesis of results, and discussion. The details are presented in Table 1.

Table 1 Nine items for reporting assessment on CHM-related information
Full size table

Data analysis

The 27-item of PRISMA and the 9-item of CHM-related information were used as the tools for assessing reporting quality. Although some items included several subitems, the scoring criteria is consistent, namely each item or subitem was given a “1” score if fully reported or “0” if incompletely reported or absent. The specific methods for scoring each item/subitem are presented in Additional file 1: S2. In order to increase the accuracy of scoring, the predefined scoring rules were tested on 20 random SRs (approximately 20% of included SRs) first, and then subsequently used to assess all SRs. After the scoring rules were determined, two researchers (XZ and Q-YA) assessed the SRs independently, and the results were double-checked. Any problems or ambiguities were resolved by discussion with third review (Z-XB). All data were collected and recorded in Microsoft Office Excel (Version 2016). Categorical data is presented as a number (n) and percent (%).

Results

Search

Our initial literature search identified 1188 records. Preliminary screening excluded 1018 SRs due to duplication or focus on non-CHM interventions. After examination of the full texts of 170 articles, a total of 109 SRs was eligible for inclusion in this study (Fig. 1).

Fig. 1
figure 1

Flow chart of CDSR in CHM

Full size image

General characteristics of included SRs

The earliest Cochrane SR of CHM was published in 1999. Since 2005, the number of SRs has increased markedly, especially in 2013. More than half (58.7%, 64/109) of the SRs were published between 2012 and 2016 (Fig. 2).

Fig. 2
figure 2

The number of included CDSR in CHM from 1999 to 2017

Full size image

Table 2 presents the general characteristics of included SRs, involving the number of authors, background of the first author, number of included randomized controlled trials (RCTs) and participants, etc. Most of SRs (97.2%) published protocols, and 78.0% SRs conducted meta-analysis. Of 109 SRs, 45.0% have been periodical updated, and 43.1% has been cited more than five times.

Table 2 General characteristics of included SRs
Full size table

Targeted diseases and conclusions of included SRs

As shown in Table 3, the three most frequently examined conditions were diseases of the circulatory system, genitourinary system, and certain infectious and parasitic diseases (14.7%, 13.8%, and 11.0%, respectively). Most (95.4%) SRs concerned the efficacy and safety of CHM interventions, of which 45.9% treatments included both single herbs and CHM formulas. For conclusions, only six SRs draw a certain conclusion (positive or negative); the remaining 103 SRs cannot do that, although 32 of them tended to conclude that CHM interventions were beneficial, the low quality of RCTs is the primary limitation.

Table 3 Descriptive information of included SRs
Full size table

PRISMA checklist score

As presented in Table 4, except the title requirement (item 1) of PRISMA was not applicable for Cochrane SRs, the total reporting rates of the remaining 26 items varied from 54.1 to 100%. Eleven items, including item 2–4, 6, 13, 18–21, 24, and 26 were fully reported (100%). Thirteen items, namely item 5, 7–10, 12, 14–17, 22, 25, and 27 were reported in more than 70% of all SRs. In comparison, only item 11 and 23 were reported in almost 50% of all SRs.

Table 4 Reporting quality of 27 items of PRISMA (n = 109 SRs)
Full size table

Reporting quality of CHM-related information

As presented in Table 5, nine items of CHM-related information were assessed. Of 109 SRs, 40.4% reported the specific names of CHM in the title, such as “Oral Astragalus (Huang qi),” “Danshen (Chinese medicinal herb)” for single herbs, or “Wendan decoction (Traditional Chinese medicine),” “Chinese herbal medicine suxiao jiuxin wan” for CHM formulas. Further, 83.5% SRs had searched Chinese databases or journals, and the common databases were CNKI (National Knowledge Infrastructure), VIP (VIP Chinese Science and Technique Journals Database), CBM (Chinese Biomedical Database), and Wanfang Database.

Table 5 Reporting quality of 9 items of CHM-related information (n = 109 SRs)
Full size table

In CHM SRs, the TCM-related rationale, diagnostic criteria, and outcome(s) were inadequately reported in the Introduction (61.5%), Discussion (29.4%), Eligibility criteria for participants (1.8%), and for outcomes (3.7%). The details of CHM interventions were also reported insufficiently, especially in terms of composition and dosage (60.8%), source (12.4%), and quality control of the CHM (2.1%). Besides, less than half (43.1%) SRs conducted additional analysis (e.g., subgroup analysis) based on specific CHM characteristics. Thus, for 85 SRs with meta-analysis, it is impossible to evaluate whether the synthesis of results had been appropriately conducted in 62 SRs (72.9%) due to their inadequate reporting in CHM treatments.

Discussion

General characteristics of included Cochrane SRs

In this study, we included 109 Cochrane SRs of CHM from 1999 to 2017 and described the baseline characteristics. Some problems have been discovered. Firstly, few authors of SRs have TCM relevant background. Most first-authors were clinicians and methodologists; however, only 25% (27/109) of them had TCM-related experience. For a person doing an SR of CHM, having TCM knowledge is better because they might well identify the clinical heterogeneity of different kinds of CHM interventions. Some scholars have indicated that it could be better to include professionals with TCM-related background in the author group of an SR of CHM [37]. Secondly, less than half of SRs have updated on time. Although it is well known that results from SRs are most useful when they are current, this study found that 55% (60/109) had not been timely updated, especially 28 SRs were published more than 5 years ago (namely before 2012). Moher D et al. have pointed out that an updating usually occurs after a certain period, such as 5 years, has passed since the completion of the original (or already updated) systematic review [38]. Generally, Cochrane SRs have a better record of updating than other SRs published in Chinese journals or other international journals; however, as indicated in this study, it still has room for improvement. Thirdly, nearly half (45.9%) of SRs selected a broad category of CHM single herbs and formulas. Indeed, choosing broad types of CHM interventions for one disease in an SR requires more rigorous methodology techniques in data analysis and combining to ensure reasonable results synthesis. Unfortunately, some scholars have found that as the proportion of broad selection of CHM interventions rises, and the rates of unreasonable synthesis of results more easily appeared [39].

PRISMA score of included Cochrane SRs

For 27 items of the PRISMA, 18 items were well reported (> 90%). Except for the item 1 (title) was not applicable for Cochrane SRs, only two items, namely item 11 (Data items) and 23 (additional analyses), were reported relatively less frequently (nearly 55%). For SRs, whether original data items were extracted entirely and accurately is closely related to the synthesis of results. According to the characteristics of data items, whether the additional analyses (e.g., subgroup analysis) were properly designed is the key for assessing the value of summary results. For CHM SRs, the specific reporting items of CHM interventions are not provided in the standard PRISMA checklist, so there might be a gap between the international reporting guideline and specific reporting of CHM SRs. Thus, we have further assessed the reporting of CHM-related information based on a self-designed checklist.

CHM-related information assessment of included Cochrane SRs

This study is the first attempt to assess the reporting quality of CHM-related information based on a self-designed checklist. As a result, we found that the inadequate reporting of CHM interventions and TCM-related rationales are needed to be improved urgently.

Firstly, for the reporting of CHM interventions, including type, dosage form, administration route and time, composition and dosage, source, and quality control, only nine SRs (8.3%) had a 100% reporting rate of these items. The least frequently reported information was the quality control (2.1%), and the CHM source (12.4%). We understand that such details may be not reported in the primary studies (e.g., RCT) [40], but they are necessary for judging consistency and heterogeneity of CHM interventions in the SRs [41]. If these details had not been collated completely, the combination of data must be interpreted with caution [42].

Secondly, the TCM rationales were inadequately reported in the included SRs. In clinical practice, the determination of CHM treatments should be guided by TCM theory of pattern differentiation [34]. Pattern (also called syndrome) differentiation is a critical component of TCM diagnosis and treatment; it is the main feature distinguishing it from Western medicine. Further, pattern differentiation refers to analyzing and summarizing the clinical symptoms gained by the four diagnostic methods of TCM (inspection, auscultation and smell, inquiry, and pulse taking and palpation) [43]. Similar to our results, previous studies have indicated that the CHM SRs published in Chinese journals had a poor reporting of TCM theories in the part of results discussion [44].

Thirdly, TCM-related diagnostic criteria and outcomes were rarely adopted in the included SRs. In CHM interventional clinical trials, it is common to adopt TCM pattern diagnostic criteria and relevant outcomes together with the Western medicine indicators [45]. The information of TCM pattern(s) is, however, often omitted in the final reports of SRs. If an SR of CHM does not consider TCM pattern diagnosis criteria or utilize TCM-related outcome(s), the summary results on the efficacy of CHM interventions may not be appropriately analyzed [46]. Authors of an SR should report the diagnostic criteria and/or outcomes of TCM-related factors, especially when these factors were used in the included RCTs of the SR.

Finally, due to the inadequate reporting of CHM interventions, more than half (57%, 62/109) SRs did not conduct subgroup analysis based on different features of CHM interventions (e.g., type, form, dosage). Moreover, for 85 SRs with meta-analysis, 73% (62/85) was impossible to assess whether data synthesis had been conducted properly. The synthesis of results includes statistical, methodological, and clinical considerations. Although the former two factors are perhaps more technical and evidence-based, the clinical considerations should be highly valued, especially for CHM treatments. According to different characteristics or categories of the CHM interventions (if any), the proper solution might be set subgroup analysis. For meta-analysis, it should be conducted under the condition of no heterogeneity between CHMs used in different trials [47].

Improvement measures and suggestions

As some deficiencies of reporting were identified in this study, specific improvements are needed, because inadequate reporting of CHM interventions in the primary RCTs can directly compromise data collection and reporting quality of SRs [48]. Firstly, before starting an SR of CHM, it is necessary for the researchers to (a) consider the CHM-related rationale based on the TCM principles, methods, formulas, and herbs; and (b) design the form to extract the data about the study characteristics with sufficient CHM-related information (e.g., CHM intervention details).

Secondly, during the conduction of the SR, the authors need to (a) examine the heterogeneity of the participants, interventions, comparisons, outcomes (PICO) under the consideration of CHM characteristics; and (b) extract adequate information from the included clinical trials or contact their authors for inadequate or missing details. If the information cannot be obtained by contacting authors, then authors of an SR should describe this information as “not reported” [49]. We understand the data from original clinical trials with CHM should be improved too. For the reporting of RCTs, our working group has published the “CONSORT Extension for CHM formula: recommendations, explanation, and elaboration” in 2017, which can help authors standardize and improve the reporting quality of RCTs with CHM formula interventions [36].

Thirdly, the clear requirements about the CHM interventions and relevant rationales can improve the reporting quality of SRs with CHM. Existing PRISMA checklist did provide the requirements about the intervention. Because of the specificity of CHM, the readers may not provide the sufficient information about CHM. The CHM-specific items, such as TCM rationale, pattern (if any), outcomes, and details of CHM interventions, cannot be adequately captured by those items designed or written for typical healthcare SRs [50]. In other words, although achieving the full completeness of the PRISMA checklist, the current reporting quality of CHM SRs is still not optimal. Therefore, a reporting guideline of SRs for CHM interventions is necessary to be developed as an extension of the PRISMA checklist. This may improve the reporting quality of SRs of CHM. The guideline of “PRISMA extension for CHM interventions” should include a series of reporting items related to CHM interventions, and should reflect the characteristics of TCM principles, methods, formulas, and herbs. Our group has initiated the related work, and pre-registered this reporting guideline on the EQUATOR (Enhancing the QUAlity and Transparency Of health Research) in August 2016 [51]. We wish to finish the development soon.

Limitations

This study has some limitations. Firstly, the results of this study were limited to Cochrane SRs and therefore may not be applicable to SRs published in other journals. Because Cochrane SRs are generally of better quality, problems in other SR publications may be even worse. Secondly, this study assessed the reporting quality of SRs of CHM mainly according to the PRISMA checklist and self-designed items of CHM-related information, and the assessment scoring (“1” or “0”) did not allow partial information to be used. All incomplete reporting (e.g., partial and absence) were given as “0”. This might influence the results due to artificial factors.

Conclusion

CHM interventions, as either single herbs or formulas, are the primary type of TCM treatments. SRs of CHM summarize evidence relating to efficacy and safety of CHM interventions—but they are valuable only if done accurately and reliably. Although Cochrane SRs of CHM had compliance with the PRISMA checklist, their reporting quality still needs improvement in full reporting of CHM intervention details and related TCM rationales. This could be achieved by extending the PRISMA checklist to include CHM specific which are based on the unique characteristic of TCM principles, methods, formulas and herbs.

Availability of data and materials

The data used for this study is available from the corresponding author upon receiving a reasonable request.

Abbreviations

CHM:

Chinese herbal medicine;

CONSORT:

Consolidated Standards of Reporting Trials

CSR:

Cochrane systematic review

EQUAOR:

Enhancing the QUAlity and Transparency Of health Research

MA:

Meta-analysis

PICOS:

Participant, Intervention, Control, Outcome, and Study design

PRISMA:

Preferred Reporting Items for Systematic reviews and Meta-Analyses

QUOROM:

Quality Of Reporting Of Meta analyses

RCT:

Randomized controlled trial

SR:

Systematic review

TCM:

Traditional Chinese medicine

References

  1. Cook DJ, Mulrow CD, Haynes RB. Systematic reviews: synthesis of best evidence for clinical decisions. Ann Intern Med. 1997;126(5):376–80. https://doi.org/10.7326/0003-4819-126-5-199703010-00006.

    CAS  Article  PubMed  Google Scholar 

  2. Villar J, Carroli G, Belizán JM. Predictive ability of meta-analyses of randomised controlled trials. Lancet. 1995;345(8952):772–6. https://doi.org/10.1016/S0140-6736(95)90646-0.

    CAS  Article  PubMed  Google Scholar 

  3. Mulrow CD, Cook DJ, Davidoff F. Systematic reviews: critical links in the great chain of evidence. Ann Intern Med. 1997;126(5):389–91. https://doi.org/10.7326/0003-4819-126-5-199703010-00008.

    CAS  Article  PubMed  Google Scholar 

  4. Moher D, Simera L, Schulz KF, Hoey J, Altman DG. Helping editors, peer reviewers and authors improve the clarity, completeness and transparency of reporting health research. BMC Med. 2008;6:13. https://doi.org/10.1186/1741-7015-6-13.

    Article  PubMed  PubMed Central  Google Scholar 

  5. Dixon E, Hameed M, Sutherland F, Cook DJ, Doig C. Evaluating meta-analyses in the general surgical literature: a critical appraisal. Ann Surg. 2005;241(3):450–9. https://doi.org/10.1097/01.sla.0000154258.30305.df.

    Article  PubMed  PubMed Central  Google Scholar 

  6. Moher D, Tetzlaff J, Tricco AC, Sampson M, Altman DG. Epidemiology and reporting characteristics of systematic reviews. PLoS Med. 2007;4(3):e78. https://doi.org/10.1371/journal.pmed.0040078.

    Article  PubMed  PubMed Central  Google Scholar 

  7. Wen J, Ren Y, Wang L, Li YP, Liu Y, Zhou M, et al. The reporting quality of meta-analyses improves: a random sampling study. J Clin Epidemiol. 2008;61(8):770–5. https://doi.org/10.1016/j.jclinepi.2007.10.008.

    Article  PubMed  Google Scholar 

  8. Moher D, Cook DJ, Eastwood S, Olkin I, Rennie D, Stroup DF. Improving the quality of reports of meta-analyses of randomised controlled trials: the QUOROM statement. Quality of Reporting of Meta-analyses. Lancet. 1999;354(9193):1896–900. https://doi.org/10.1016/S0140-6736(99)04149-5.

    CAS  PubMed  Google Scholar 

  9. Moher D, Liberati A, Tetzlaff J, Altman DG. The PRISMA Group. Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. PLoS Med. 2009;6(7):e1000097. https://doi.org/10.1371/journal.pmed.1000097.

    Article  PubMed  PubMed Central  Google Scholar 

  10. Liberati A, Altman DG, Tetzlaff J, Mulrow C, Gøtzsche PC, Ioannidis JP, et al. The PRISMA statement for reporting systematic reviews and meta-analyses of studies that evaluate healthcare interventions: explanation and elaboration. PLoS Med. 2009;6(7):e1000100. https://doi.org/10.1371/journal.pmed.1000100.

    Article  PubMed  PubMed Central  Google Scholar 

  11. Fleming PS, Seehra J, Polychronopoulou A, Fedorowicz Z, Pandis N. A PRISMA assessment of the reporting quality of systematic reviews in orthodontics. Angle Orthod. 2013;83(1):158–63. https://doi.org/10.2319/032612-251.1.

    Article  PubMed  Google Scholar 

  12. Tan WK, Wigley J, Shantikumar S. The reporting quality of systematic reviews and meta-analyses in vascular surgery needs improvement: a systematic review. Int J Surg. 2014;12(12):1262–5. https://doi.org/10.1016/j.ijsu.2014.10.015.

    Article  PubMed  Google Scholar 

  13. Li JL, Ge L, Ma JC, Zeng QL, Yao L, An N, et al. Quality of reporting of systematic reviews published in “evidence-based” Chinese journals. Syst Rev. 2014;3:58. https://doi.org/10.1186/2046-4053-3-58.

    Article  PubMed  PubMed Central  Google Scholar 

  14. Tian JH, Zhang J, Ge L, Yang K, Song F. The methodological and reporting quality of systematic reviews from China and the USA are similar. J Clin Epidemiol. 2017;85:50–8. https://doi.org/10.1016/j.jclinepi.2016.12.004.

    Article  PubMed  Google Scholar 

  15. Green S, Higgins JP, Alderson P, Clarke M, Mulrow CD, Oxman AD. Cochrane Handbook Version 5.0.2: The Cochrane Collaboration; 2008.

  16. Jadad AR, Cook DJ, Jones A, Klassen TP, Tugwell P, Moher M, et al. Methodology and reports of systematic reviews and meta-analyses: a comparison of Cochrane reviews with articles published in paper-based journals. JAMA. 1998;280(3):278–80. https://doi.org/10.1001/jama.280.3.278.

    CAS  Article  PubMed  Google Scholar 

  17. Petticrew M, Wilson P, Wright K, Song FJ. Quality of Cochrane reviews. Quality of Cochrane reviews is better than that of non-Cochrane reviews. BMJ. 2002;324(7336):545.

    Article  PubMed  PubMed Central  Google Scholar 

  18. Handoll H, Madhok R. Another study found that most Cochrane reviews are of a good standard. BMJ. 2002;324(7336):545.

    Article  PubMed  PubMed Central  Google Scholar 

  19. Fleming PS, Seehra J, Polychronopoulou A, Fedorowicz Z, Pandis N. Cochrane and non-Cochrane systematic reviews in leading orthodontic journals: a quality paradigm? Eur J Orthod. 2013;35(2):244–8. https://doi.org/10.1093/ejo/cjs016.

    Article  PubMed  Google Scholar 

  20. Xiong XJ, Yang XC, Liu YM, Zhang Y, Wang PQ, Wang J. Chinese herbal formulas for treating hypertension in traditional Chinese medicine: perspective of modern science. Hypertens Res. 2013;36(7):570–9. https://doi.org/10.1038/hr.2013.18.

    Article  PubMed  PubMed Central  Google Scholar 

  21. Bensky D, Gamble A. Chinese Herbal Medicine: Materia Medica. Revised edition. Seattle: Eastland Press, 1993.

  22. He YT, Hou J. Using meta-analysis on secondary prevention of esophageal. Chin J Prevention Control of Chron Dis. 1997;5(3):33–4. [Article in Chinese]. https://doi.org/10.16386/j.cjpccd.issn.1004-6194.1997.03.017.

    Article  Google Scholar 

  23. Wang JY, Wang JF, Cheng SQ, Tian GH, Wang YP, Shang HC, et al. Analysis of 19 years development and present situation of Chinese medicine systematic review/meta-analysis. J Tradit Chin Med. 2017;58(11):919–25. [Article in Chinese]. https://doi.org/10.13288/j.11-2166/r.2017.11.006.

    Article  Google Scholar 

  24. Chen M, He J, Xiao Y, Huang R, Zhou ZF, Chen CY, et al. Status quo analysis on TCM systematic reviews/meta-analyses published in Chinese journals. Chin J Evid-based Med. 2012;12(12):1526–30 [Article in Chinese].

    Google Scholar 

  25. Ma B, Guo JW, Qi GQ, Li HM, Peng JY, Zhang YL, et al. Epidemiology, quality and reporting characteristics of systematic reviews of traditional Chinese medicine interventions published in Chinese journals. PLoS One. 2011;6(5):e20185. https://doi.org/10.1371/journal.pone.0020185.

    CAS  Article  PubMed  PubMed Central  Google Scholar 

  26. Zhang JH, Shang HC, Gao XM, Zhang BL, Xiang YZ, Cao HB, et al. Methodology and reporting quality of systematic review/meta-analysis of traditional Chinese medicine. J Altern Complement Med. 2007;13(8):797–805. https://doi.org/10.1089/acm.2007.7195.

    Article  Google Scholar 

  27. Li Q, Xia Y, Mou YJ, Wang YY, Liu JP. Reevaluation on quality of TCM systematic reviews and Meta-analyses documents published in domestic Chinese journals. Mod Chin Clin Med. 2012;19(3):28–33 [Article in Chinese].

    Google Scholar 

  28. Li TQ, Liu XM, Zhang MM, Ma JX, Du L, Zhou YD, et al. Assessment of systematic reviews and meta-analyses on traditional Chinese medicine published in Chinese journals. Chin J Evid-based Med. 2007;7(3):180–8 [Article in Chinese].

    Google Scholar 

  29. Hu D, Kang DY, Wu YX, Wang JL. Methodological quality assessment of systematic reviews correlated to traditional Chinese medicine published in China. Zhongguo Zhong Xi Yi Jie He Za Zhi. 2011;31(3):402–6 [Article in Chinese].

    CAS  PubMed  Google Scholar 

  30. Hu J, Zhang JH, Zhao W, Zhang YL, Zhang L, Shang HC. Cochrane systematic reviews of Chinese herbal medicines: an overview. PLOS One. 2011;6(12):e28696. https://doi.org/10.1371/journal.pone.0028696.

    CAS  Article  PubMed  PubMed Central  Google Scholar 

  31. Leung KS, Bian ZX, Moher D, Dagenais S, Li YP, Liu L, et al. Improving the quality of randomized controlled trials in Chinese herbal medicine, part III: quality control of Chinese herbal medicine used in randomized controlled trials. Chin J Integr Med. 2006;4(3):225–32.

    Article  Google Scholar 

  32. Ma B, Qi GQ, Lin XT, Wang T, Chen ZM, Yang KH. Epidemiology, quality, and reporting characteristics of systematic reviews of acupuncture interventions published in Chinese journals. J Altern Complement Med. 2012;18(9):813–7. https://doi.org/10.1089/acm.2011.0274.

    Article  PubMed  Google Scholar 

  33. Wang X, Chen Y, Liu Y, Yao L, Estill J, Bian Z, et al. Reporting items for systematic reviews and meta-analyses of acupuncture: the PRISMA for acupuncture checklist. BMC Complement Altern Med. 2019;19(1):208. https://doi.org/10.1186/s12906-019-2624-3.

    Article  PubMed  PubMed Central  Google Scholar 

  34. Wu TT, Zhang SQ, Guo SW, Gu YZ, Dou LP, Wang YY, et al. Correspondence analysis between traditional Chinese medicine (TCM) syndrome differentiation and histopathology in colorectal cancer. Eur J Integr Med. 2015;7(4):342–7. https://doi.org/10.1016/j.eujim.2015.07.003.

    Article  Google Scholar 

  35. Wu TX, Li YP, Bian ZX, Li TQ, Li J, Dagenais S, et al. Consolidated standards for reporting trials of traditional Chinese medicine (CONSORT for TCM) (For solicitation of comments). Chin J Evid-based Med 2007;7(8): 601-605. [Article in Chinese]

  36. Cheng CW, Wu TX, Shang HC, Li YP, Altman DG, Moher D, et al. CONSORT extension for Chinese herbal medicine formulas 2017: recommendations, explanation and elaboration. Ann Intern Med. 2017;167(2):112–21. https://doi.org/10.7326/M16-2977.

    Article  PubMed  Google Scholar 

  37. Guo XF, Lai SL. Key methodological and institutional issues on how to promote the evidence based traditional Chinese medicine. Chin J Evid-based Med 2008;8(1):10-12. [Article in Chinese]

  38. Moher D, Tsertsvadze A. Systematic reviews: when is an update an update? Lancet, DOI. 2006;367(9514):881–3 https://doi.org/10.1016/S0140-6736(06)68358-X.

    Article  Google Scholar 

  39. Hu D, Kang DY, Hong Q. Heterogeneity analysis of systematic reviews on traditional Chinese medicine. Chin J Evid-based Med 2010;10(4):488-491. [Article in Chinese]

  40. Wang G, Mao B, Xiong ZY, Fan T, Chen XD, Wang L, et al. CONSORT Group for traditional Chinese medicine. The quality of reporting of randomized controlled trials of traditional Chinese medicine: a survey of 13 randomly selected journals from mainland China. Clin Ther. 2007;29(7):1456–67. https://doi.org/10.1016/j.clinthera.2007.07.023.

    Article  PubMed  Google Scholar 

  41. Han M, Wang YY, Mu YJ, Liu ZJ, Li Q, Li XX, et al. Reporting specification of systematic reviews on Chinese medicine and methodological evaluation. Zhongguo Zhong Xi Yi Jie He Za Zhi 2012;32(7):872-874. [Article in Chinese]

  42. Xiong J, Du YH, Li B, Shi L, Xu YY, Liu Q, et al. Assessment of methodology and reporting quality of systematic evaluation and Meta-analysis of acupuncture-moxibustion in China. Zhongguo Zhen Jiu 2009;29(9):763-768. [Article in Chinese]

  43. Ferreira AS, Lopes AJ. Chinese medicine pattern differentiation and its implications for clinical practice. Chin J Integr Med. 2011;17(11):818–23. https://doi.org/10.1007/s11655-011-0892-y.

    Article  PubMed  Google Scholar 

  44. Yang GY, Luo H, Liu JP. Reporting parts of discussion and conclusion in systematic reviews. Modern Chin Clin Med 2012;19(5):30-32. [Article in Chinese]

  45. Cheng CW, Zhang L, Zhao C, Zhong LL, Shi LJ, Dai L, et al. Whether syndrome differentiation affects treatment result: study protocol of MaZiRenWan for functional constipation in a randomized controlled trial. Chin J Integr Med, DOI. 2019;25(3):175–81 https://doi.org/10.1007/s11655-018-2848-y.

    Article  PubMed  Google Scholar 

  46. Liu JP, Han M, Li XX, Mu YJ, Lewith G, Wang YY, et al. Prospective registration, bias risk and outcome-reporting bias in randomised clinical trials of traditional Chinese medicine: an empirical methodological study. BMJ Open. 2013;3(7):e002968. https://doi.org/10.1136/bmjopen-2013-002968.

    Article  PubMed  PubMed Central  Google Scholar 

  47. Wu TX, Liu GJ, Li J. Some risks of affecting the quality of published systematic reviews in China. Chin J Evid-based Med 2005;5(1):51-58. [Article in Chinese]

  48. Ma B, Chen ZM, Xu JK, Wang YN, Chen KY, Ke FY, et al. Do the CONSORT and STRICTA checklists improve the reporting quality of acupuncture and moxibustion randomized controlled trials published in Chinese journals? a systematic review and analysis of trends. PLoS One. 2016;11(1):e0147244. https://doi.org/10.1371/journal.pone.0147244.

    CAS  Article  PubMed  PubMed Central  Google Scholar 

  49. Hoffmann T, Glasziou P, Boutron I, Milne R, Perera R, Moher D, et al. Better reporting of interventions: template for intervention description and replication (TIDieR) checklist and guide. BMJ, DOI. 2014;348:g1687 https://doi.org/10.1136/bmj.g1687.

    Article  PubMed  Google Scholar 

  50. Liu Y, Zhang R, Huang J, Zhao X, Liu D, Sun W, et al. Reporting quality of systematic reviews/meta-analyses of acupuncture. PLoS One. 2014;9(11):e113172. https://doi.org/10.1371/journal.pone.0113172.

    Article  PubMed  PubMed Central  Google Scholar 

  51. Enhancing the QUAlity and Transparency Of health Research (EQUAOR) Network. http://www.equator-network.org/library/reporting-guidelines-under-development/#38 Accessed 26 May 2018.

Download references

Acknowledgments

The authors thank Dr. Martha Dahlen for her critical English editing of the manuscript.

Funding

This work was supported by FunRenHuang Medical Foundation.

Author information

Author notes

  1. Xuan Zhang and Qi-Ying Aixinjueluo have contributed equally to this study.

Authors and Affiliations

  1. Chinese Clinical Trial Registry (Hong Kong), Hong Kong Chinese Medicine Clinical Study Centre, School of Chinese Medicine, Hong Kong Baptist University, Kowloon Tong, Hong Kong

    Xuan Zhang, Qi-Ying Aixinjueluo, Si-Yao Li, Chung-Tai Lau, Ran Tan & Zhao-Xiang Bian

  2. Dr. Stephen Riady Chinese Medicine Library, Hong Kong Baptist University, Kowloon Tong, Hong Kong

    Lisa-L Song

Authors
  1. Xuan Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Qi-Ying Aixinjueluo
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Si-Yao Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Lisa-L Song
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Chung-Tai Lau
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Ran Tan
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Zhao-Xiang Bian
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

Z-XB designed the study; L-LS and XZ conducted the search strategy and searched for Cochrane SRs; Q-YA, C-TL, and RT extracted the data; XZ, S-YL, and Q-YA assessed the reporting quality; XZ and Q-YA analyzed the data; XZ wrote the manuscript; Z-XB revised and finalized the manuscript. All authors read and approved the final manuscript.

Corresponding author

Correspondence to Zhao-Xiang Bian.

Ethics declarations

Ethics approval and consent to participate

Not applicable.

Consent for publication

Not applicable.

Competing interests

The authors declare that they have no competing interests.

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary information

Additional file 1.

S1 Search strategy. S2 Quality assessment rules of included CDSRs.

Rights and permissions

Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated.

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Zhang, X., Aixinjueluo, QY., Li, SY. et al. Reporting quality of Cochrane systematic reviews with Chinese herbal medicines. Syst Rev 8, 302 (2019). https://doi.org/10.1186/s13643-019-1218-y

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1186/s13643-019-1218-y

Keywords

  • Cochrane systematic review (CSR)
  • Chinese herbal medicine (CHM)
  • PRISMA statement
  • Reporting quality
  • Traditional Chinese medicine (TCM)
  • TCM principles, methods, formulas, and herbs