Introduction

The use of synthetic mesh for pelvic organ prolapse (POP) surgical treatment has recently decreased because of concerns around patient safety. These concerns have been the subject of extensive debates [1], and synthetic meshes for transvaginal POP repair were reclassified by the FDA as high-risk devices [1,2,3,4]. A high level of evidence on efficacy and safety through systematic reviews and meta-analyses is warranted to aid clinicians, policy makers and women in choices of treatment for pelvic organ prolapse. However, published data are frequently conflicting possibly on account of arbitrary outcome selection and reporting. Current research evidence is of variable quality and methodology and hence robust practice recommendations are lacking because of limitations in research evidence. Studies have reported on variable outcomes, and comparisons among several trials are not possible because of this heterogeneity [5].

The process of the development of a Core Outcome Set (COS) includes in-depth evaluation of the selection of outcomes reported in primary trials. Trials often use variable methods, and selection of outcomes varies because of authors’ expertise, research priorities and objectives, ethical issues and other factors. Frequently, trials on the same treatment intervention have been designed with different methods and study objectives and have been published in various journals (subspecialized, specialized, general journals) over the years. Our previous systematic review developed the inventory of reported outcomes and outcomes measures as a first step in the process of developing relevant COS based on established standards and following well-defined study protocols and high-quality methods [6]. Our systematic reviews on outcome reporting in trials evaluating surgical treatments in different prolapse procedures, incontinence and childbirth trauma [5, 7,8,9,10,11,12] included assessments of associations between outcome reporting quality, methodological quality and publication characteristics.

Evaluation of the quality outcome reporting and its associations with methodological quality might provide useful evidence in developing a core outcome set for this area of research and possibly provide valuable guidance and directions for future studies.

The aim of this study was to evaluate methodological quality and outcome quality in trials on surgical treatment of POP using synthetic mesh and assess the associations of methodological quality and outcome quality with publication characteristics of the trials including year of publication and journal impact factor.

Materials and methods

This review is part of CHORUS (An International Collaboration for Harmonizing Outcomes, Research and Standards in Urogynecology and Women’s Health, i-chorus.org). This study was a secondary analysis of data that were part of a recently published systematic review [5]. Ethical approval was not required for this study, as this is a secondary analysis of data included in a previously published systematic review.

The search strategy was described in the original study in accordance with PRISMA guidelines. We used the following keywords and MeSH terms: management, repair, operation and pelvic organ prolapse in the Cochrane, EMBASE, Medline and Scopus databases from inception until September 2017. Randomized controlled trials (RCTs) using synthetic meshes for any type of pelvic prolapse were included (Fig. 1). We excluded retrospective studies, literature reviews, case reports and non-randomized studies. The studies selected are listed in Appendix 1.

Fig. 1
figure 1

Randomized controlled trials (RCTs) using synthetic meshes for any type of pelvic prolapse were included

Full size image

The methodological quality of the included trials was evaluated using the modified Jadad score. This is a 5-point scale that evaluates randomization, adequate method for randomization; blinded trial described; adequate method for blinding and if the trial accounts for the patients selected [13]. The outcome quality was assessed using the MOMENT criteria (Management of otitis media with effusion in cleft palate score system), in a 6-point scale. Areas included in the scoring system are stating a primary outcome; explaining if the primary outcome is defined for reproducible measures; stating a secondary outcome; reporting if the secondary outcome is defined as for reproducible measures; explaining if the choice of outcome and if the methods used are designed to improve appropriately the quality of measures [14]. High-quality studies were considered those that reached score ≥ 4 based on these criteria. All assessments were undertaken independently by two assessors in line with our previous studies and similar research [7,8,9,10,11,12]. In cases of disagreement, a third assessor reviewed the scores and provided additional scores. The final scores were calculated and reviewed by the senior investigator.

Year and journal of publication and journal impact factor (IF) were retrieved and documented according to Thomson Reuters’ (NY, USA) citation reports for obstetrics and gynecology. Statistical analysis was undertaken using SPSS statistical software (IBM Corp., USA). Association among methodological quality, outcome quality, year of publication and journal’s impact factor was calculated by non-parametric correlation (Spearman’s correlation). Statistical significance was defined as p < 0.05.

Results

We reviewed and assessed the 71 RCTs previously included in our systematic review.

Methodological quality, outcome quality and publication parameters are presented in Table 1. Year of publication ranged from 2000 to 2017 and follow-up interval ranged from 1.5 to 74 months. The mean Jadad and MOMENT scores were 3.59 and 4.63. Figure 2 shows the quality of outcome (MOMENT) distribution for the total score, and Fig. 3 shows the number of studies scored according to each statement. Fifty-nine studies (83%) were classified as high-quality outcome reporting, presenting score 4, 5 or 6.

Table 1 Quality of studies
Full size table
Fig. 2
figure 2

Distribution of scores on quality of outcome reporting (MOMENT scores)

Full size image
Fig. 3
figure 3

Number of studies satisfying the specific statements (yes or no) on quality of outcomes’ assessment

Full size image

Primary and secondary outcomes were clearly stated in 69 (97%) and 46 (64%) of the studies. Only 34 studies (47%) provided a rationale for the choice of outcomes but all of them showed methods to improve the outcome measurement described.

A non-parametric correlation revealed that the outcome reporting quality was positively correlated to the methodological quality of the trial (rho = 0.662; p = 0.000) and to the year of publication (rho = 0.262; p = 0.028). Methodological quality does not appear to be influenced by year of publication (rho = 0.092; p = 0.444) or the journal’s impact factor (rho = 0.100; p = 0.417) (Table 2).

Table 2 – Univariate correlation on publication characteristics, JADAD score and MOMENT score
Full size table

Discussion

In our study,outcome reporting quality ( MOMENT criteria) demonstrated a positive correlation to methodological quality (Jadad score) and to year of publication. The quality of outcome reporting has improved in more recently published trials. However, methodological quality did not have an association with year of publication or the journal’s impact factor.

Not surprisingly, methodological quality and outcome reporting quality appear correlated possibly because of the overall study design process and the overall quality of a research protocol. However, such “grading” of the overall quality of a study does not necessarily translate into a publication in a journal with a higher impact factor. This observation highlights the need for harmonization of the quality of methodology and of the reported outcomes and possibly the development of set criteria in research protocols that may assist journal editors in the peer review publication process.

The methodological quality and quality of reported outcomes as well as publication characteristics should be taken into consideration during the process of development of core outcome sets.

One of the strengths of our systematic review is to be the first one to our knowledge evaluating the methodological quality as well as outcome reporting quality in trials using mesh for the treatment of pelvic organ prolapse. We followed a well-established methodology in a standardized manner in order to provide unbiased and objective evaluation of the above-mentioned parameters of the published trials. Another strength of our study was the independent assessments that were undertaken and the process of review and consensus around the final scores of the different domains and items.

However, a number of limitations warrant caution in the interpretation of our results.

We evaluated a highly selective cohort of studies leaving out studies that were non-randomized and with different methodologies. Nevertheless, inclusion of a wider variety of studies would be likely to demonstrate wider variations and accentuate our findings. Furthermore, our findings are based on the instruments used and their inherent limitations. Some studies with higher quality protocols but with suboptimal descriptions of these protocol in the published text may have received lower scores than deserved.

Finally, on reviewing the correlation of the quality of the trial and the journal impact factor, we should take into consideration that the choice of the journal a study was published in can be influenced by many factors not necessarily associated directly with the quality of the study. For example, a presentation of a study at a conference may be rewarded by an offer of a fast track review of the paper in the journal associated to the society organizing the conference. Some authors may also select a journal to submit based on personal preference, influenced by factors such as journal loyalty of the author, previous or current association with the journal’s editorial board or other factors. Hence, this correlation between quality of a trial and journal of publication should always be interpreted with caution for such biases that cannot be weighed.

In the studies included in this evaluation, methodological quality of trials was positively associated with outcome quality but was not strongly associated to year of publication and the journal’s impact factor. The instruments and methodology described, already widely used in many fields of medical research, in gynecology, obstetrics and urogynecology, were applied to analyze in a standardized way important parameters of published research [5]. This assessment is in our opinion a fundamental prerequisite in the process of developing a high quality COS in line with the standards established by the COMET, COSMIN and CROWN initiatives.

Nevertheless, we believe that this study may also provide invaluable guidance for improving better selection of outcomes and measurement tools, outcome reporting, research methods and publication strategies for future research in this area. Harmonized methodology and outcome selection and reporting may improve the comparability of primary research, which in turn may inform robust meta-analyses and eventually improve clinical practice.

figure a