Background

To get a balanced perspective of an intervention, systematic reviewers need to report both its beneficial and adverse effects [1]. In this cross-sectional study we assessed whether adverse effects were sought, whether the findings on these effects were reported, and what types of adverse effects were identified in systematic reviews published in the Cochrane Database of Systematic reviews [2] and in 5 leading orthodontic journals.

‘Cochrane defines an adverse effect as ‘an adverse event for which the causal relation between the intervention and the event is at least a reasonable possibility’ [3, 4]. This definition and other key terms used in this manuscript are listed in Table 1 [5, 6]. A wide body of epidemiological studies has shown that adverse effects of interventions in primary research studies are often under-assessed, and/or under-reported, and/or distorted [7,8,9,10,11,12,13]. These issues can misinform anyone trying to make valid decisions on a healthcare intervention. An extension of the Consolidated Standards of Reporting Trials (CONSORT) Statement was developed to tackle poor reporting of harms in randomized trials [14]. Since the publication of this statement, the reporting of adverse events in clinical trials has improved, but is still suboptimal [10, 12, 15, 16].

Table 1 Glossary of terms
Full size table

Systematic reviews could provide even more information on adverse effects, because they assess large amounts of data from a wide spectrum of sources (possibly including both published and unpublished data). By assessing the data of multiple single studies, systematic reviewers can make a more balanced assessment of an intervention. This is an important issue, because serious adverse effects may occur rarely and might be missed in single studies. However, epidemiological research showed that the seeking and reporting of adverse effects of interventions and the methods used to identify and synthesize them [17,18,19,20,21] were also poor in systematic reviews. The Preferred Reporting Items for Systematic reviews and Meta-Analyses (PRISMA) harms checklist was published in 2016 [22] to improve harms reporting in systematic reviews, but its consequences are still largely unknown.

We performed 2 cross-sectional studies on assessing and reporting of adverse effects in systematic reviews of orthodontic interventions. In this study (part 1), we assessed whether adverse effects were sought and reported and what findings on these adverse effects were reported in systematic reviews of orthodontic interventions published in the Cochrane Database of Systematic reviews [2] and in 5 leading orthodontic journals. In a second study (part 2) we assessed the reporting on adverse effects and the presence of spin on adverse effects in the abstracts of these reviews [23]. Adverse effects of orthodontic interventions refer to for example, pain associated with orthodontic tooth movement, root resorption, decalcifications, periodontal problems, relapse, and undesired health experiences [24]. Recent (November 22 2021) scoping searches confirmed that our research objectives have not been addressed previously.

Objectives

The objectives of this research study are formulated in the following four research questions:

  1. 1)

    Was seeking of adverse effects of interventions defined as a research objective of the review?

  2. 2)

    Did the review seek any findings related to adverse effects of interventions in the included studies?

  3. 3)

    Did the review report findings related to adverse effects of interventions sought in the included studies?

  4. 4)

    Were potential adverse effects of the intervention considered, discussed (weighed) anywhere in the review?

We also assessed what adverse effects of interventions were defined as research objectives and what adverse effects of interventions were sought and reported in the review.

Methods

The Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) statement [25] and the PRISMA 2020 statement [26, 27] were consulted for reporting this cross-sectional study. The STROBE checklist of items for reporting cross-sectional studies was presented in Additional file 1. The methods for this cross-sectional study were explained in our published protocol [6] and can be consulted through the following link https://systematicreviewsjournal.biomedcentral.com/articles/10.1186/s13643-019-1000-1. We adopted the framework of this protocol to report the methods section of this study and its additional files. Raw data are recorded in Open Science Framework (https://osf.io/ka7mp/). Differences between methods originally planned in the protocol and those implemented in the final research study were given with the rationales for these differences in Additional file 2. No patients were involved in the development of the protocol or in the conduct of this study.

Eligibility criteria

The eligibility criteria have been published previously [6, 28] and are presented again in Table 2 [29].

Table 2 Eligibility criteria
Full size table

Information sources and search strategy

The information sources for this study were the Cochrane Database of Systematic Reviews [2] and the websites of 5 leading orthodontic journals. The selection of these 5 orthodontic journals was based on having been published at least 10 years and the highest impact factor [30]. The impact factor in 2018, i.e., the year when the protocol was developed, was used to select these journals. The 5 selected orthodontic journals are: European Journal of Orthodontics [EJO], American Journal of Orthodontics and Dentofacial Orthopedics [AJODO], Angle Orthodontist (AO), The Korean Journal of Orthodontics (KJO), and Orthodontics and Craniofacial Research (O&CR). The impact factors of these journals are listed in Additional file 2. August 1 2009 was chosen as the inception date for searching the information sources, because it coincides with the publication of the PRISMA statement and guidance document on 21 July 2009 [31, 32]. Eligible systematic reviews were manually searched in these information sources from the inception date until July 31 2021.

Study records

Data management

All study selection and data extraction procedures were conducted independently by 2 authors (PS and RMR). Pilot tests were done a priori to train and calibrate these operators [33]. Disagreements between these reviewers during these study selection and data collection were resolved in the following order: Firstly, through discussions; secondly, through rereading the article in question; and thirdly, through contacting of the authors of the pertinent manuscript by email to obtain additional information that could help with decision-making [27]. Persistent disagreements were resolved through discussions with a methodologist (SB). All eligible systematic reviews with their supplementary files were downloaded as PDFs and merged in binder files [34, 35]. Data were collected in an Excel spreadsheet [36].

Study selection and data collection procedures

Titles and abstracts were screened for eligible reviews in the websites of the 5 selected orthodontic journals. Eligible Cochrane reviews were searched in the ‘Dentistry and Oral health’ section of the Cochrane Database of Systematic Reviews [2]. When Cochrane reviews were updated, we only considered the latest published version. A PRISMA flow diagram was presented to illustrate the selection process of the eligible reviews [26, 27]. All included studies and excluded studies were reported and the rationales for exclusion were given in Additional file 3. Contacting of authors was not necessary to clarify eligibility or data extraction issues. We used our pilot tested data collection forms for the extraction of all pertinent data items. These forms are presented in Additional file 2. The entire eligible review except the abstract and protocol were searched, i.e., the main text, tables, figures, and supplemental files. This strategy was implemented for all eligible reviews. In Cochrane systematic reviews, we also did not search data items in the plain language summary.

Assigning adverse effects of orthodontic interventions

Cochrane defines an adverse effect as ‘an adverse event for which the causal relation between the intervention and the event is at least a reasonable possibility’ [3, 4]. These events can have a permanent or temporary adverse effect on the health condition of the patient. Root resorption, decalcifications of enamel or caries around orthodontic appliances are well known permanent adverse effects of orthodontic interventions, while pain and discomfort during tooth movement are generally temporary adverse effects. Events associated with orthodontic interventions that could have an adverse effect on the health condition were also labeled as adverse events, e.g., breaking of appliances, failure to complete treatment, and tolerability of orthodontic appliances.

According to our protocol we adopted the framework of known orthodontic adverse effects as reported previously by Preoteasa et al. [24] (Additional file 2) and made some changes in labeling the headings of the various categories of adverse effects (Additional file 2). A total of 12 categories of adverse effects were defined. Additional adverse effects identified during our data extraction procedures were also included in this framework and when ambiguous the rationale for including these adverse effects was given. The following types of adverse events were not labeled as adverse effects: (1) effects that do not refer to health conditions and could be ambiguous, e.g., costs, duration of treatment, number of appointments etc. (2) effects that refer to pre-existing health problems that can actually improve as a result of the intervention, e.g., respiratory problems as a result of maxillary expansion or self-esteem as a result of the retraction of protruding maxillary incisors.

Power calculation

Epitools epidemiological software was used to calculate the required sample size of eligible systematic reviews of orthodontic interventions [37]. We calculated the required sample size of 73 reviews based on the following input: estimated proportion 0.25, desired precision 0.1, and confidence level 0.95. The estimated proportion was based on the findings in our pilot tests as reported in our protocol [6]. These pilot test showed that findings related to adverse effects were sought in 3 of 12 systematic reviews on orthodontic interventions representing the estimated proportion of 0.25 (3/12).

Outcomes and statistical analyses

We reported the number of retrieved systematic reviews and eligible reviews and calculated the prevalence proportions that addressed our research questions. All outcomes were calculated as originally planned in our published protocol [6]. Prevalence proportions were calculated for: (1) all journals together (2) each journal separately and (3) the group of 5 leading orthodontic journals together and the Cochrane reviews separately. Univariable logistic regression models were built to determine the association between each one of four outcomes and the journal in which the systematic review was published, using the Cochrane Database of Systematic reviews as reference. The strength of association was quantified using odds ratios (OR), and 95% confidence intervals (95% CI). Analyses were performed with the use of commercial software (IBM SPSS 22.0, SPSS Inc, Chicago, IL). A two-sided P value of 0.05 was considered to be statistically significant.

Results

Results of the search

Through our searches in the databases of the Cochrane Database of Systematic reviews and the 5 leading orthodontic journals we identified 324 reports. One Cochrane review was excluded, because it was later updated leaving 323 reports for screening. A total of 180 papers was excluded during the title and abstract screening and 45 during full text screening. A total of 98 systematic reviews fulfilled the eligibility criteria of this study. The results of the individual selection steps are presented in a PRISMA flow diagram (Fig. 1) [26, 27]. All included studies are listed in Additional file 3 and excluded studies with the rationale for their exclusion are given in Additional file 4.

Fig. 1
figure 1

Flow diagram for the selection of systematic reviews of orthodontic interventions

Full size image

Included studies

Figure 2 presents the number of eligible systematic reviews of orthodontic interventions published during the eligible time span. Table 3 presents the number of eligible reviews for each selected journal and shows that 72.4% (71/98) of the included reviews came from the EJO, AJODO, and AO. Table 3 also gives the types of orthodontic interventions for each of these journals, which are divided in three categories. Category 1 refers to orthodontic interventions to move teeth modify jaws such as fixed orthodontic appliances or palatal expansion appliances. Category 2 refers to orthodontic interventions that also include additional surgical, pharmacological or vibrational interventions such as mini-implants, prostaglandins, piezo surgery, or vibratory stimulation. Category 3 refers to orthodontic interventions with appliances to maintain or stabilize orthodontic treatment results such as retainers. The majority of included reviews, 70.4% (69/98), assessed orthodontic interventions to move teeth or modify jaws and 28.6% (28/98) assessed orthodontic interventions with additional surgical, pharmacological or vibratory interventions.

Fig. 2
figure 2

Line chart of the number of eligible systematic reviews published between August 1 2009 and July 31 2021

Full size image
Table 3 Characteristics of included reviews
Full size table

Outcomes to the research questions

Figure 3 presents the answers to each individual research question and Table 4 gives the proportions. We reported the proportions in answering the four research questions over time in Table 5. The prevalence of reviews that defined seeking of adverse effects of interventions as a research objective was low, i.e., 35.7% in the 98 eligible reviews. Instead, the proportions that addressed the other 3 research question were 85% and higher indicating that seeking and reporting of findings related to adverse effects of interventions in the included studies and considering or discussing potential adverse effects anywhere in the review were implemented in most of the eligible reviews. As compared to the Cochrane Database of Systematic Reviews, the journal of Orthodontics and Craniofacial research had approximately 7 times the odds (OR 7.20, 95%CI 1.08 to 47.96) to report that adverse effects were sought in the research objectives. The other journals were not significantly more likely to report that adverse effects were sought in the research objectives (Table 6). For the other 3 outcomes, no statistical analysis was performed considering the low variability in the response scored (prevalence of ‘no’ ranging from 9.2 to 15.3%) and the overall small sample sizes (Table 4).

Fig. 3
figure 3

Seeking adverse effects in systematic reviews of orthodontic interventions

Full size image
Table 4 Outcomes on seeking adverse effects in systematic reviews of orthodontic interventions
Full size table
Table 5 Outcomes on seeking adverse effects in systematic reviews of orthodontic interventions by year of publication
Full size table
Table 6 Results of univariable logistic regression including journal as a predictor variable of seeking adverse effects as an objective of the systematic review
Full size table

Labeling adverse effects of orthodontic interventions

The type of adverse effects most frequently defined as research objectives were adverse effects related to (1) tooth structures, (2) periodontal tissues, (3) undesired treatment results, (4) relapse and stability, and (5) negative qualitative experiences by the patient or carer(s) (Table 7). These were also the most prevalent types of adverse effects sought in the included studies and reported in the review and accounted for 83.1% (162/195) of all adverse effects sought and reported (Table 8). We were able to categorize all 195 adverse effects except one and labeled it ‘Additional adverse effects’ (Table 8).

Table 7 Type of adverse effects defined as research objectives of the review
Full size table
Table 8 Type of adverse effects sought and reported in the review
Full size table

Discussion

Principal findings of the study

This cross-sectional study showed that in 35.7% (35/98) of reviews of orthodontic interventions seeking of adverse effects was defined as an objective. In 85.7% (84/98) of these reviews, findings related to adverse effects of interventions were sought and in 84.7% (83/98) the reviewers reported on these findings. In more than 90% (89/98) of included systematic reviews, the reviewers discussed (weighed) potential adverse effects of interventions somewhere in the review. Five types of adverse effects accounted for 83.1% (162/195) of adverse effects that were sought and reported in the eligible reviews.

Comparisons with other studies

The proportion of included reviews that defined seeking of adverse effects as a research objective was low, i.e., 35.7% (35/98) in both Cochrane and non-Cochrane systematic reviews (Table 4). Assessing potential adverse effects of interventions is considered a mandatory item when setting the research question for Cochrane intervention reviews [1]. Not defining seeking of adverse effects as a research objective can mislead end-users of systematic reviews. Authors therefore need to include this item in their research objectives and editors and peer reviewers should verify its implementation.

The proportions of reviews that reported findings related to adverse effects of interventions were higher in this sample of orthodontic reviews (84.7% (83/98) compared with gastroenterology reviews (66.7% (52/78) [18], Cochrane reviews of interventions (75.6% (59/78), and Database of Abstracts of Reviews of Effects (DAREs) reviews (48.1% (38/79) [38]. Explanations for these higher proportions could be: (1) the time period of inclusion of reviews (2) the research design and type of interventions of the studies included in the reviews (3) the field of research. Orthodontic research could be more focused on assessing adverse effects of interventions than other fields, because this assessment is an integral part of routine clinical practice. For example, assessing adverse effects such as undesired treatment results and relapse and stability are part of everyday problems in orthodontic practice and accounted for 40.5% (79/195) of adverse effects sought and reported in this sample of systematic reviews of orthodontic interventions (Table 8).

Strengths and limitations

This cross-sectional study has the following strengths: (1) scoping searches were conducted to identify knowledge gaps, (2) pilot studies were conducted to calibrate researchers and fine-tune research questions and methodology, (3) a protocol was developed and published a priori [6], and (4) all raw data were included with this manuscript or recorded in Open Science Framework (https://osf.io/ka7mp/). This study also has limitations. First, the findings of this cross-sectional study are expected to be better than those reported in the entire body of orthodontic literature, because we assessed reviews published in the five leading orthodontic journals and those listed in the Cochrane Database of Systematic Reviews. Second, the risk of selective (non) reporting bias regarding adverse effects in the eligible reviews. Third, only reviews published in a pre-established period (August 1 2009 until July 31 2021) were eligible, instead of having considered a larger sample, e.g., by having included reviews prior to the inception date. However, we chose this inception date, because it coincides with the launch of the PRISMA statement [31, 32], which provides reviewers better guidance on reporting.

Implications and future research

Several of our findings seem promising at a first glance. For example, the proportion of reviews that sought and reported adverse effects was relatively high, i.e., (84.7% (83/98), but a variety of issues has to be considered when interpreting this finding. First, this proportion only refers to whether or not reviewers implemented this item, but not how. For example, the reviewers could have reported on just one or a selection of all adverse effects assessed and reported in the eligible studies for their reviews. Second, this proportion also does not give any information on the magnitude, and duration of adverse effects nor on the time points for assessing them. Third, we do not know whether all adverse effects were indeed sought and reported as originally planned in the registered protocols of the included reviews. For example, Parsons et al. [39] showed that this was not the case in their sample of systematic reviews of health care interventions. In 35% (51/146) of these reviews they found discrepancies between what was planned in the protocol as registered in PROSPERO and what was reported on adverse effects in the final published reviews. Fourth, a wide body of evidence has shown that adverse events were often assessed inconsistently and reported inadequately in clinical trials and that most results on these events were not available in public sources [8, 40,41,42]. If these limitations also apply to the clinical trials that fed the reviews of this study one should further question the validity of the findings on adverse effect of systematic reviews of orthodontic interventions.

Strategies to improve the validity of what is reported on adverse effects of orthodontic interventions in systematic reviews include developing tailored core outcome sets on these effects [43] as well as guidelines for assessing and reporting them in both primary research and systematic reviews. Additional strategies on synthesizing adverse effects in systematic reviews at multiple levels were published in a recent paper by Qureshi et al. [19]. By implementing such strategies progress on the assessing and reporting of adverse effects of orthodontic interventions in both primary studies and systematic reviews can be made.

In conclusion the promising findings of this study should be interpreted with caution by its end users, because they could be jeopardized by numerous uncertainties. Much research is ahead to create valid and usable knowledge on adverse effects of orthodontic interventions involving a wide body of stakeholders.