Background

Systematic reviews (SR) can help practitioners to keep abreast of the medical literature by summarizing large bodies of evidence and explaining differences among studies on the same question. SRs can be used to inform medical decisions and to establish clinical policy. A meta-analysis (MA) is a type of SR that uses statistical methods to combine and summarize the results of several primary studies [1]. SR/MA can help clinicians to keep up to date with their field and help policymakers to judge the risks and benefits of health care behaviors. They provide a starting point for clinical practice guideline developers and summaries for funders seeking new research to support [2]. But inadequate reporting occurs in some field, such as surgery and traditional medicine etc. [3,4,5]. The suboptimal reporting quality of SR/MA led to the development of the Quality Of Reporting Of Meta analyses (QUOROM) Statement and its updated revision named Preferred Reporting Items for SRs and Meta-Analyses (PRISMA), published in 1999 and 2009, respectively [6, 7]. The PRISMA Statement consists of a 27-item checklist and a four-phase flow diagram, with an explanation and elaboration for each checklist item also published in 2009 [8]. The PRISMA checklists are used to guide authors of SR/MA to improve reporting quality. It is also a universal standard to assess the reporting quality of available SR/MA publications [9,10,11].

With a history of over 2000 years, the CM practice offers natural, safe and effective therapies and cures for many diseases with much less side effects. CM takes a unique theoretical and practical approach to health. For treatments, it includes the use of various interventions, such as moxibustion and acupuncture. The moxibustion technique is commonly used in clinical practice, including direct moxibustion with a traditional moxa stick, and indirect moxibustion. Mugwort leaves (Artemisia vulgaris, moxa) are the most commonly material used for moxibustion. Other Chinese herbal medicines could also be used in combination with mugwort. Mugwort is considered to be warm, acidic and bitter. It can promote better circulation of qi and blood by warming meridians. According to CM theory, the meridians are the channels inside the human body that circulate vital energy (in Chinese called qi and blood). Although it includes adverse effects, such as allergic reactions, burns and infections, sometimes, the moxibustion intervention is always believed as a safety therapy [12, 13].

As an increasing number of randomized controlled trials (RCTs) and SRs of moxibustion published in recent years, the importance to improve reporting quality of moxibustion studies has been highlighted by both researchers and users of moxibustion evidence [14,15,16]. In 2013, our working group has published The STandards for Reporting Interventions in Clinical Trials Of Moxibustion (STRICTOM), which aims to improve the reporting quality of RCTs with moxibustion [17]. However, no previous study has assessed the reporting quality of moxibustion SRs. Thus, this study was designed to: (i) summarize the general characteristics of all included SRs of moxibustion, (ii) assess the reporting quality of these SRs based on the PRISMA checklist, (iii) evaluate whether necessary information related to moxibustion is adequately reported, and (iv) assess whether these SRs are properly conducted in terms of synthesis of results (e.g., meta-analysis).

Methods

Inclusion and exclusion criteria

This study included all SR/MAs of moxibustion published in seven databases, including Embase, Cochrane Library, MEDLINE, Allied and Complementary Medicine (AMED), CNKI, VIP and Wanfang, up to 31 December 2019. Moxibustion interventions may have been given alone or in combination with other interventions or complementary alternative medicine. There were no limitations in the participants, comparisons, and outcomes. We excluded the following SR/MAs: repeat publications, research on acupuncture or other treatments as the main intervention, protocols, and withdrawal SR/MAs.

Search strategy

The databases were searched on 13 Feb 2020 for all SR/MAs of Moxibustion that had been published up to 31 December 2019. The search terms included “meta-analysis,” “systematic review,” “moxibustion,” “mora,” “Jiu,” “vesiculation,” “Sanfu or dog days,” “Fapao or blister,” “Suanni or Garlic”, “Baijiezi or Bai Jie Zi or White mustard seed”, “Maogen or Japan Buttercup”, “acupuncture and moxibustion”, “Hanlian or Eclipta”, “Tiannanxing or Araceae or Arisaema”, etc. were included. The detailed search strategy is given in Additional file 1: S1.

Screening

The titles and abstracts of the SRs were independently screened by two researchers (SY-L and RT) based on inclusion and exclusion criteria, and the full-texts of potentially suitable articles were retrieved for further assessment. Disagreements were resolved by a third reviewer (XZ).

Data extraction

There were three pre-designed forms for data collection: (1) General characteristics form, including publication year, information of the authors, and descriptive information of included SRs. (2) PRISMA assessment form, including 27 items of the checklist. (3) Moxibustion-related information form, which was designed according to (a) CM theory and (b) STRICTOM guideline. The details are presented in Table 1.

Table 1 Fourteen items for reporting assessment on Moxibustion-related information
Full size table

Data analysis

Two reviewers (RT and SY-L) used the PRISMA and special-designed moxibustion-related list to independently assess the reporting quality of included SRs. Each item was given a “1” score if fully reported or “0” if incompletely reported or absent. Disagreements were resolved by discussion or consultation with a third researcher (XZ). All data were collected and recorded in Microsoft Office Excel 2016. Categorical data is presented as a number (n) and percent (%).

Results

Search

Our initial literature search identified 3182 records. Preliminary screening excluded 3070 SRs due to duplication or focus on non-moxibustion interventions. After examination of the full texts of 112 articles, a total of 97 SRs was eligible for inclusion in this study (Fig. 1).

Fig. 1
figure 1

Flow chart of the performed search and selection process

Full size image

General characteristics of included SRs

The first included SR of moxibustion was published in 2011, and the number of publications has been continued increasing (Fig. 2). The general features of these SRs are presented in Tables 2 and 3. The three most commonly examined conditions were diseases of musculoskeletal system or connective tissue (30.9%), diseases of the digestive system (12.4%), and diseases of the respiratory system (10.3%), respectively.

Fig. 2
figure 2

The number of included moxibustion SRs from 2011 to 2019

Full size image
Table 2 General characteristics of included SRs
Full size table
Table 3 Descriptive information of included SRs
Full size table

PRISMA score of included SRs

As presented in Table 4, the total reporting rates of 27 items varied from 12.4% to 100%. Two items, including item 20 and 26, were fully reported (100%). Thirteen items, namely item 1, 2, 4, 6, 7, 9, 10, 12, 13, 14, 17, 18, and 24 were reported in more than 80% of all SRs. Eight Items, including item 3, 11, 15, 19, 21, 22, 25, and 27, were reported in more than 55% SRs. The least reported items including 5, 8, 16, and 23 were lower than 55%.

Table 4 Reporting quality of 27 items of PRISMA (n = 97)
Full size table

Reporting score of moxibustion-related information

As presented in Table 5, fourteen items of Moxibustion-related information were assessed. Of 97 SRs, 25.8% reported the specific name of studied moxibustion in the title, but no title included studied CM syndrome. In the included articles, few included CM syndrome-related content, such as diagnostic criteria (6.2%) and outcomes (3.1%). Relevant CM theory and rationale was provided in the Introduction among only 25.8% of included SRs. For the details of moxibustion, except one item of the materials and techniques used for moxibustion was reported in 60.8% of included SRs, the other items are seriously inadequate reporting, such as treatment environment (5.2%), the number and frequency of treatment (27.8), the safety assessment of moxibustion (30.9%), the selection of acupoints and meridians (33%), and the duration of treatment (43.3%). Besides, few SRs (27.8%) considered subgroup analysis. Based on the inadequate reporting, it is impossible to evaluate whether the synthesis of results had been appropriately conducted in more than half (55.4%) of 92 SRs with meta-analysis.

Table 5 Reporting quality of 14 items of Moxibustion-related information (n = 97)
Full size table

Discussion

General characteristics of included Moxibustion SRs

In this study, we included 97 Moxibustion SRs from 2011 to 2019 and described the baseline characteristics. Some problems have been identified. Firstly, most SRs (94.8%, 92/97) included the meta-analysis but more than half (69.1%, 67/97) did not report the specific type of studied moxibustion(s). Generally, choosing a wide range of moxibustion interventions for one disease in an SR requires rigorous methodology techniques in data analysis. Without transparent reporting, it is hard to judge the reasonable rationale for results synthesis. Secondly, very few SRs (2.1%, 2/97) have updated on time. Although it is well known that results from SRs are most useful when they are current, this study found that all included SRs published in Chinese journals did not mention any update. Based on the features of moxibustion SRs, this review is the first to systematically assess the reporting quality of moxibustion SRs with two instruments of the PRISMA checklist and moxibustion-related items. The additional moxibustion-related checklist was designed to identify the important moxibustion details in the procedure of SRs’ design, conduct, and analysis.

PRISMA assessment of included moxibustion SRs

In this review, most SRs (84.5%, 82/97) were published in Chinese medical journals. As Chinese journals usually did not require the registration and protocol for SRs, thus the reporting of these items was very low. Consistent with previous studies, the reporting compliance of these information in Chinese SRs, particularly in the interventions of Chinese herbal medicines, acupuncture, and massage (Tuina) was seriously inadequate [18,19,20]. In addition, the reporting of “Search” (item 8) was unsatisfactory because more than half of the included SRs only provided some keywords for search instead of a comprehensive search strategy according to the PRISMA requirement. In comparison, the moxibustion SRs published in English journals usually have a better reporting for this item. Honestly, the inadequate reporting of “Search” could be related to the word limitation of Chinese journals, and most such journals did not provide an online appendix choice. Although most Chinese medical journal includes the endorsement of the PRISMA statement, strictly compliance and application is the key to improve the reporting of moxibustion SRs [21]. Besides, the items of “additional analysis”, either in the section of Methods or Results, were reported relatively less frequently. According to the characteristics of moxibustion, whether the additional analyses (e.g., subgroup analysis) were properly designed is the key for assessing the value of summary results [22]. Similar to previous reports, however, SRs with CM interventions (e.g., Chinese herbal medicines) usually prefer to include many different types of studied interventions [23, 24]. As the proportion of wide intervention ranges rises, and the rates of unreasonable synthesis of results more easily appeared. Thus, an appropriately pre-design of subgroup analysis is much important.

Moxibustion-related information assessment of included SRs

For moxibustion SRs, the specific reporting items of interventions are not provided in the PRISMA checklist, so there might be a gap between the international reporting guideline and specific reporting of moxibustion SRs. Thus, we have further assessed the reporting of moxibustion-related information based on a self-designed checklist. Obviously, some deficiencies have been discovered. Firstly, we found that the inadequate and absent reporting of moxibustion details and CM related rationales are needed to be improved urgently. For SRs, whether original data items were extracted entirely and accurately is closely related to the synthesis of results. In terms of moxibustion details, the following elements are critical to be extracted: type of moxibustion, materials and technique used for moxibustion, selection of acupoints and location for moxibustion, number, frequency and duration of treatment sessions, treatment environment, and safety assessment [17]. As a result, in this study, the least frequently reported information was the number and frequency (27.8%), safety assessment or adverse effect (30.9%), the types of moxibustion (30.9%), names of acupoints and meridians (33.0%), and the duration of the treatment (43.3%). Relatively, the materials and techniques were reported in 60.8% of the included SRs. Secondly, due to the inadequate reporting of intervention details, more than half (72.2%, 70/97) SRs did not conduct subgroup analysis based on different features of moxibustion interventions (e.g., type, material, dosage). Moreover, for 92 SRs with meta-analysis, 55.4% (51/92) was impossible to assess whether data synthesis had been conducted properly. The synthesis of results includes statistical, methodological, and clinical considerations. Although the former two factors are perhaps more technical and evidence-based, the clinical considerations should be highly valued, especially for the moxibustion therapy used in clinical practice [25]. Relevant CM rationale, such as syndrome differentiation, should be proposed appropriately [26]. This content was also rarely mentioned in the included SRs, although most of them studied CM-based moxibustion.

Improvement measures and suggestions

As some deficiencies of reporting were identified in this study, specific improvements are urgently needed. In agreement with previous findings, guidelines do help improve the quality of reporting [27, 28]. Therefore, except for continuously improved the completeness reporting of the PRISMA checklist, a series of standard reporting items especially for moxibustion SRs should be developed as an extension for the PRISMA. With the PRISMA extensions for acupuncture and Chinese herbal medicines have been published [29, 30], the specific guideline for moxibustion is helpful to supplement the SRs with CM interventions. Our group has registered this reporting guideline on the Enhancing the QUAlity and Transparency Of health Research (EQUATOR) under the item of “PRISMA for traditional Chinese medicine”, and has initiated the related work [31]. We wish to publish the Guideline soon.

Limitations

This study has some limitations. Firstly, this review identified moxibustion SRs published up to 31 December 2019 in the targeted seven main databases. Any records which had not been included in these databases by that cut-off period have not been included. Secondly, the assessment scoring (“1” or “0”) did not allow partial information to be used. All incomplete reporting (e.g., partial and absence) were given as “0”, which decrease the reporting rate of some items. These limitations mean that the results of the study may not necessarily be comprehensive. We do however believe that the general trends indicated by the analysis of the information we did use, even if incomplete, are valid.

Conclusions

Moxibustion SRs summarize evidence relating to efficacy and safety of moxibustion interventions—but they are valuable only if done accurately and reliably. Due to the inadequate reporting, it is a challenge to assess whether the data synthesis had been properly conducted in moxibustion SRs. Such situation seriously affects the readers’ judgments about the conclusion of these SRs and further compromises the values of moxibustion therapy. Therefore, the development of a reporting guideline for moxibustion SRs, as an extension of the PRISMA, could be an effective measure to improve the current situation.