Background

An emerging problem that has rarely been investigated concerns the human factors that undermine clinical randomised controlled trials (RCTs) at various stages [1]. In the 1920s and 1930s two scientists in different research fields [2, 3] helped enormously to reappraise statistical theory and methodology in designing trials, thus clarifying how bias influences research. Previous papers have already compared protocols and registered data for clinical outcomes in published RCTs in various clinical settings [4–7], and three studies have investigated discrepancies in selectively reported outcomes [8, 9]. Even though concealment and blinding tools can control human factors in RCT designs [10] and the International Committee of Medical Journal Editors (ICMJE) [11] and Consolidated Standards of Reporting Trials (CONSORT) [12] receive wide consensus, a recent survey among journal editors endorsing the ICMJE and CONSORT established policies disclosed that only 27 % of the 33 respondents cross-checked the data reported in the submitted manuscript against the data registered prospectively in clinical trial registries (CTRs) [13]. In recent years the Food and Drug Administration (FDA) expanded the regulatory requirements for conducting clinical trials and the truthfulness of the data submitted, and now requires authors to annually update CTRs, thus providing clear informative results [14]. Whether trial design, conduction and outcome data from the various CTRs and published RCTs differ or are incompletely reported remains unknown [4, 15, 16]. One systematic review assessed primary outcome discrepancies [17]. No studies have analysed trial reporting domains more widely, and none have addressed paediatric trials. Apart from the Critical Appraisal Skills Programme (CASP) checklist [18], nor do paediatricians and clinical researchers have tools for assessing discrepancies and risk of bias that compare what clinical researchers record in the registered study hypothesis and protocol, and what they then publish in RCTs [19–23]. Knowing more about trial discrepancies should alert paediatricians, clinical researchers, peer-reviewers, editors, and policymakers to possible dissemination or reporting bias undermining paediatric trials whose results provide the best information for medical care [24].

To help in critically appraising research relevant for clinical practice we sought possible discrepancies between what CTRs record and paediatric RCTs actually publish. For this purpose, after identifying six reporting domains, including funding, design, and outcomes, we collected data from a sample of 20 unselected consecutive RCTs published in a widely read peer-reviewed paediatric journal and cross-checked reported features with those in the corresponding CTRs.

Methods

In a study conducted from November 2012 to January 2016, to seek possible discrepancies between what CTRs record and paediatric RCTs actually publish, two reviewers, an experienced clinical paediatrician and an experienced researcher (PR and RD), identified six major reporting domains: five based on their long experience in critically appraising well-conducted clinical trials (reported funding and conflict of interest incompletely declared; discrepant or unclear sample size; inclusion and exclusion criteria not being respected or selective crossover; primary outcome downgraded and secondary outcomes upgraded and reported as primary outcomes in the publication; early study completion unjustified), and one domain (main outcome selectively reported or unreported) based on the Cochrane risk of bias tool [23, 24]. Over the first year they developed, assessed, and graded CTR-RCT discrepancy scores. Over the ensuing years five investigator pairs, supervised by two tutors (PR and RD), and attending the annual course held at Bambino Gesù Children’s Hospital, and subsequent weekly meetings on critically appraising scientific publications (G.A.L.I.L.E.O.), carefully read the 20 consecutive RCTs published monthly from July to November 2013 in the journal Pediatrics [25–44]. They then searched the CTR web link for each corresponding CTR, assessed details, and ‘history of changes’ after the initial registration, and critically appraised each published RCT with the CASP checklist. The five investigator pairs then independently mapped and coded inconsistencies for each reporting domain in the 20 trials, and repeatedly searched online (last access 20 January 2016) from the corresponding CTR websites: the United States National Institute of Health (NCT) (https://clinicaltrials.gov), the International Standard Randomised Controlled Trial Number (ISRCTN) (currently BioMed Central Open Access publishers) (http://www.isrctn.com/), the Nederlands Trials Register (NTR) (http://www.trialregister.nl/trialreg/index.asp), the Australian and New Zealand Clinical Trial Registry (ACTRN), (http://www.anzctr.org.au) and the Clinical Trial Registry-India (CTRI) (http://ctri.nic.in/Clinicaltrials/login.php). Meanwhile, PR and RD supervised the five investigator pairs for scoring the six individual reporting domains and combined scores. PR and RD then consulted the external tutor (FP), and conflicting interpretations were resolved by consensus. PR and RD with the five investigator pairs then reassessed inconsistencies between CTRs and RCTs for each trial over 2 months and, after several attempts, reached 100 % final agreement on grading discrepancy scores from 1 to 3 points, according to the reporting domain importance, and on grading combined discrepancy scores as low, medium and high (Table 1, Additional file 1). Higher discrepancy scores suggested risk of bias.

Table 1 Clinical trial registry (CTR)-randomised controlled trial (RCT) discrepancies in 20 RCTs scored by assessing and cross-checking inconsistencies in the six reporting domains identified, assessed and scored according to their importance in a well-conducted triala
Full size table

Results

When we compared what the 20 paediatric RCTs published in the journal Pediatrics reported and what the collected five matching cross-checked CTRs recorded, 9 trials had medium (5–9) and 10 high (10–14) combined discrepancy scores (Table 1, Additional file 1). The five investigator pairs who critically appraised the trials with the CASP checklist found that all 20 published RCTs fulfilled the general criteria for assessing trial internal validity. Of the 20 RCTs, 11 were registered in the NCT [25, 26, 30, 32, 34–39, 44], 4 in the ISRCTN [27, 29, 31, 43], 2 in the ACTRN [33, 42], 1 in the NTR [28], 1 was registered both in the NTR and in the ACTRN [40], and 1 in the CTRI [41]. Assessment for the reporting domains disclosed that 9 trials had discrepancies in declaring sponsorship and conflict of interests [25, 33–37, 40, 41, 44], 8 trials had a discrepant or unclear sample size [26, 31–33, 40–43], 9 trials failed to respect inclusion or exclusion criteria [27, 28, 30, 34, 38–41, 44], 11 trials downgraded or modified primary outcome measures or upgraded secondary outcomes [30, 33, 35–39, 41–44], 13 trials completed early [29, 31, 32, 34–40, 42–44], and all 20 paediatric clinical trials selectively misreported outcomes or failed to report main outcomes, thus tending to overstress the positive results (Table 1, Additional file 1). A single-centre trial was retrospectively registered in the ACTRN [42], and one multicentre trial was registered prospectively in the NTR and retrospectively in the ACTRN [40]. For this multicentre trial, although the NTR reported that the trial had stopped, the ACTRN stated ‘still recruiting’, and neither CTR was updated. Two papers failed to respect the intention-to-treat analysis [29, 40]. Two trials were completed early during a planned interim analysis by an external Data Safety Monitoring Committee (DSMC): the first was stopped for efficacy results (more harm than good in the intervention group), and the target sample remained unreached, but the ISRCTN failed to report the cause [31], and the second, a multicentre trial, owing to futility in the results [40], underreported or misreported outcomes in the two registries (NTR and ACTRN). Published RCT abstracts and results both contained inconsistencies in reporting the reasons for stopping the trials [31, 40]. In another three CTRs and corresponding RCTs, the five investigator pairs detected a discrepancy between the primary outcome and efficacy results reported (more harm than good in the intervention group) [29, 35, 41]. Three published RCTs underreported insignificant results [28, 30, 33], one upgrading the secondary outcome [30], and one downgrading the primary outcome [33]. Two trials, one registered in the ACTRN and one in the CTRI, downgraded primary outcome in the published RCT [33, 41], two upgraded secondary outcomes, both registered in the NCT [30, 39], and another eight modified primary outcome or primary outcome measures, six registered in the NCT, one in ISRCTN, and one in ACTRN [35–39, 42–44]. For one clinical trial, the NCT automatically reported the previous RCT paper reference that included the NCT primary outcome, but neglected to report the RCT reference that included secondary outcomes that had been upgraded and yielded insignificant results [30]. For another clinical trial, the authors reported their previous published papers in the NCT record, but neglected to report the second published RCT giving a partially modified primary outcome in the updated NCT data, recorded after RCT publication [37]. Of the 20 clinical trials, seven CTRs (three registered in the NCT, two in the NTR, and two in the ACTRN) failed to index automatically the RCT reference or Uniform Resource Locator (URL) address, and all these trials had discrepancies in outcome data or yielded insignificant results [28, 33, 34, 37, 38, 40, 42]. Twelve CTRs reported the RCT references or URL addresses, but the authors failed to summarise the main results [25–27, 29–32, 35, 36, 39, 43, 44]. For only one clinical trial did the authors report the main results in the CTRI but neglect to report increased side effects in the intervention group [41] (Table 1, Additional file 1).

Discussion

By comparing the six reporting domains, mapping, coding and cross-checking 20 published RCTs with the matched five CTRs, our study, applied to clinical paediatric trials published in a widely read peer-reviewed journal, suggests that many trials have discrepancies in reporting domains. The medium or high combined discrepancy scores we found, when we repeatedly searched each database online until January 2016, underline major widely ranging discrepancies between what CTRs record and what published paediatric RCTs then report. The discrepancies we identified in declaring funding and conflict of interests in nine trials, in the number of eligible and enrolled participants in eight clinical trials and in another nine inclusion and exclusion criteria not being respected, emphasise the generally imperfect reporting. These major discrepancies, especially those involving changes in the original study hypotheses, trial designs, study conduction and reporting outcomes raise concern on trustworthiness in scientific trials, as previous papers have underlined [5–9, 13, 15, 16].

Surprisingly, of the 20 published RCTs 11 modified or downgraded primary outcomes or upgraded secondary outcomes (reported secondary outcomes as primary outcomes in the publication), misreporting or tending to overstress positive results. This discrepancy underlines concerns about trial creditability that the CASP checklist overlooks. By assessing and scoring CTR-RCT discrepancies in six clinical trial reporting domains, our study therefore expands current knowledge, thus emphasising the need for international clinical trial regulators to make publicly available CTR-RCT discrepancies in published RCT findings, as recently underlined by the WHO Statement on public disclosure of clinical trial results (http://www.who.int/ictrp/results/reporting/en/; last accessed 20 January 2016). For example, trial (number 5 in Table 1) [29] addresses as primary outcome cognitive improvement and provides significant statistical difference between the intervention and control groups, but leaves unaddressed clinically important patient-centred outcomes, such as deaths and cerebral palsy. Even though deaths increased and cerebral palsy doubled in the intervention group, in their conclusions the investigators paradoxically report that ‘nonsignificant trends in the data suggested a small adverse effect’. In another trial (number 7 in Table 1) [31] an external DSMC decided to stop RCT completion early owing to hyperthermia in the infants enrolled in the intervention group. Neither the highlights section in the RCT nor the conclusions report this result. Although the published RCT reports when the trial stopped, our findings disclose an important clinically relevant feature, namely more harm than good for the primary outcome in the intervention group.

Another unexpected discrepancy in a multicentre trial (number 16 in the Table 1) [40] was that the authors inappropriately and unclearly reported that an external DSMC stopped the trial for futility. The numerous CTRs updated only by dataset supervisors (URL or RCTs reference cited in 12/20 trials), and failing to report results (7/20 trials) underline discrepancies involving incompletely and selectively reported clinical outcome results [5, 45–47]. This finding, along with underappreciated core patient-centred outcomes, raises ethical concerns and suggests dissemination or reporting bias [45–49]. Even though the journal Pediatrics complies with ICMJE requirements [11], and authors are required to submit a completed flowchart and checklist for the CONSORT statement [12] before publication (http://www.aappublications.org/content/pediatrics-author-guidelines#acceptance_criteria; last accessed 20 January 2016), and all the 20 published RCTs give the trial registration number on the first page, our findings underline that still today few authors endorse these rules [13, 14, 17].

An unexpected finding concerned prospective trial registration [11]. Although our study design did not require us to check clinical trial registration timing, we detected two trials (numbers 16 and 18 in Table 1) [40, 42] that had been registered retrospectively and failed to comply with ICMJE registration requirements. One of these two, a multicentre RCT (number 16 in Table 1) [40], was registered prospectively in the NTR and retrospectively in the ACTRN, and although the NTR reported that the trial had stopped for futility, the ACTRN stated ‘still recruiting’ Because a retrospectively registered trial could be hard to identify, regulators need to find new ways to encourage researchers to update information in a timely manner [50–52].

Most important, the major discrepancies our study highlighted in paediatric clinical trials give new clinically important information that researchers synthesising evidence from published RCTs in scientific literature reviews could fail to identify without cross-checking CTRs. Hence, they could provide less reliable scientific evidence, and misdirect future research priorities [49–53]. Our findings could also alert medical journals on the need to introduce rules that require investigators to submit the original Institutional Review Board-approved protocol (and its subsequent amended versions), and to explain later changes, thus helping reviewers and editors in deciding whether to publish the trial and what the final manuscript should state.

Limitations

We acknowledge that our study has several limitations. Because we applied our study method only in few unselected consecutive paediatric RCTs published in a major paediatric journal, rather than including other authoritative journals with higher impact factors, our findings require further validation. Even though our scoring for reporting domains needs further refinement, our findings should make it easier for physicians to use research results from clinical trials. Because most authors neglected to update CTRs, we were unable to seek discrepancies in what authors recorded in CTRs and reported in published RCTs by cross-checking clinical outcome results. Similarly, because none of the five CTRs recorded or the cross-checked matched 20 RCTs appraised gave the necessary information, nor were we able to assess data for patient nonresponse and refusal. Although we analysed a small RCT sample, we found no differences in combined CTR-RCT discrepancy scores among the various CTRs. A final limitation is that we failed to assess interobserver reliability for the different assessor times for individual items and combined CTR-RCT discrepancy scores.

Conclusions

Our study identifies major discrepancies between what CTRs record and paediatric RCTs publish. Our findings should make clinicians who rely on RCT results for medical decision-making, aware of dissemination or reporting bias. Trialists need to bring CTR data and reported protocols in line with published data. Clinical researchers and reviewers could search for CTR-RCT discrepancies to cross-check inconsistencies in core clinical trial reporting domains. Medical journals need to introduce rules that require investigators to submit the original Institutional Review Board-approved protocol (and its subsequent amended versions), and to explain any discrepancies. Assessing discrepancies would with little effort provide greater transparency, avoid wasting research resources, and encourage those who prepare medical recommendations and guidelines to think more critically. Future studies need to clarify whether the trial discrepancies we report warrant scepticism regarding study validity, or call for trialists to be more diligent about updating CTR data.