Background

Since 1995, the pharmaceutical industry has written structured clinical study reports of randomized clinical trials following international guidelines to document their products’ benefits and harms when applying for marketing approval [1]. Clinical study reports are usually confidential documents, but can be requested or downloaded from the European Medicines Agency (EMA) [2], ClinicalStudyDataRequest.com (CSDR), GlaxoSmithKline’s trial register website and, in the future, possibly from the US Food and Drug Administration (FDA) [3]. Publicly available trial data mainly come from biomedical journal publications and trial register entries such as those on ClinicalTrials.gov. The intention of ClinicalTrials.gov is that all studies publish all results and that those who do not publish results within 12 months of trial completion are fined. According to fdaaa.trialstracker.net, 32% of studies on ClinicalTrials.gov have no results posted and no fines have been issued. Clinical study reports usually have all prespecified data included or include amendments. There can be important differences in results from published [4] and unpublished [5] versions of corresponding study documents. Clinical study reports include highly detailed information on all aspects of a trial [6] and are on average about 2000 pages long [7], but it can be difficult to obtain complete and unredacted clinical study reports [8].

We carried out a systematic review of the human papillomavirus (HPV) vaccines’ clinical study reports [9] based on an index we constructed of 206 HPV vaccine studies [10]. As of July 2017, 62% (92/149) of the completed studies were not published in journal publications and 48% (71/147) of the completed studies on ClinicalTrials.gov had no study results posted [10]. Systematic reviewers often only use journal publications and trial registers for their reviews, which may increase the risk of using a data set influenced by selective outcome reporting.

To our knowledge, no study has looked at differences of pooled estimates—such as meta-analyses—of corresponding study documents of the same intervention. Our primary aim in this study was to compare meta-analyses of HPV vaccine data from clinical study reports with data from corresponding trial register entries and journal publications. Our secondary aim was to compare the reporting of study design aspects of the corresponding study documents.

Methods

We compared corresponding HPV vaccine study documents of clinical study reports, trial register entries and journal publications to investigate the degree of reporting bias for prespecified outcomes and the reporting of trial design aspects; see our protocol on PROSPERO [11] (registered as ‘Protocol amendment no. 3’ for our systematic review of the HPV vaccines [9]).

Clinical study reports were obtained from EMA and GlaxoSmithKline [9]. We identified the clinical study reports’ corresponding trial register entries on ClinicalTrials.gov and corresponding primary journal publications from our published index of the HPV vaccine studies. The search strings used to identify the studies are available in the index publication [10]. We assessed all identified journal publications for a study (including supplementary documents and errata) for eligible information and chose the primary publication that corresponded to the clinical study report for our comparison. We did not check for eligible information in additional trial registers (such as the EU Clinical Trials Register) or letters to the editors.

Data extraction and comparison of the study documents were carried out by two researchers (LJ extracted the data; TJ checked the extractions; and PCG arbitrated). For each study document, the following data were compared: study ID, number of pages, date of document, time from study completion to publication in a journal, result availability, protocol availability (including pre-specification of outcomes and inclusion of a statistical analysis plan), reporting of PICO criteria (participants, interventions, comparisons and outcomes) and reporting of six major design-related biases defined by the Cochrane Handbook (version 2011) for the Cochrane risk of bias tool [12] (random sequence generation, allocation concealment, blinding of outcome assessors, blinding of personnel, blinding of participants and loss to follow-up). We collected these data, as they are important to evaluate a study’s internal and external validity. We did not include the Cochrane risk of bias tool domain ‘selective outcome reporting’, since we compared this domain quantitatively between corresponding documents.

For each study document, we extracted and compared data on the outcomes we assessed in our systematic review [9]. As our review contained 166 meta-analyses, we only compared the 20 most clinically relevant outcomes (or statistically significant outcomes with a p value ≤ 0.05; noted in parentheses). Benefit outcomes included all-cause mortality, HPV-related cancer mortality, HPV-related cancer incidence, HPV-related carcinoma in situ, HPV-related moderate intraepithelial neoplasia, HPV-related moderate intraepithelial neoplasia or worse and HPV-related treatment procedures. Harm outcomes include fatal harms, serious harms (including those judged as ‘definitely associated’ with postural orthostatic tachycardia syndrome [POTS] and complex regional pain syndrome [CRPS; see our systematic review protocol amendment [13] for these two post hoc exploratory analyses] and the nervous system disorders that were Medical Dictionary for Regulatory Activities [MedDRA] classified in this system organ class), new-onset diseases (including back pain, vaginal infection and the vascular disorders that were MedDRA classified in this system organ class) and general harms (including fatigue, headache and myalgia). Histological outcomes were assessed irrespective of involved HPV types. The most aggregated data account (participants with events over the total number of participants) was used for the meta-analyses, and the most detailed harm account of MedDRA preferred terms was used for event comparisons. For example, if harms were registered separately per harm, we would count the separate harms and summarize them as a total number of harms. For all GlaxoSmithKline clinical study reports and for serious harms for Merck clinical study reports, we pooled MedDRA preferred terms in their respective system organ classes. A participant could potentially be included more than once in a separate analysis (e.g. if a participant experienced both serious ‘headache’ and serious ‘dizziness’, the participant would be counted twice in the MedDRA system organ class analysis of serious nervous system disorders); we therefore consider the MedDRA system organ class analyses exploratory.

Merck Sharp & Dohme did not provide a formal definition for its new-onset disease category—new medical history—but described the category as ‘all new reported diagnoses’ in the clinical study report of trial V501-019. Although ‘new medical history’ was not explicitly mentioned in the trial register entries and journal publications, we included eligible new reported diagnoses not reported as serious or general harms in this category.

For our meta-analyses, we used the intention to treat principle. Risk ratios (RRs) were calculated with the random effects inverse variance method. Random effects estimates were compared to fixed effect estimates, as the former method may weigh small trials unduly if there is considerable heterogeneity between trials [12].

Results

We included study documents from 22 randomized clinical trials and 2 follow-up studies and obtained 24 clinical study reports, 24 corresponding trial register entries and 23 corresponding primary journal publications (for the remaining journal publication—HPV-003, of 61 participants—the manufacturer confirmed that no journal publication had been published [10]). See Additional file 1 for our study’s PRISMA statement.

Characteristics of included studies

The 24 included studies investigated four different HPV vaccines: Cervarix™, Gardasil™, Gardasil 9™ and an HPV type 16 vaccine, and included 95,670 healthy participants (79,102 females and 16,568 males) aged 8 to 72. One (4%) study used a saline placebo comparator, but its participants had been HPV vaccinated before randomization. Fourteen (58%) studies used vaccine adjuvants as comparators: amorphous aluminium hydroxyphosphate sulphate (AAHS), aluminium hydroxide (Al[OH]3) or carrier solution. Nine (38%) studies used hepatitis vaccine comparators: Aimmugen™, Engerix™, Havrix™ or Twinrix Paediatric™.

Characteristics of included study documents

Nearly all study documents (70/72) reported data from study start to completion, except for the clinical study report and journal publication of study HPV-040 that described interim analyses. The median number of pages in the clinical study reports was 1351 (range 357 to 11,456) (see Table 1). For four studies (HPV-008, HPV-013, HPV-015 and HPV-040), we obtained clinical study reports from both EMA and GlaxoSmithKline (we did not account for duplicate pages). EMA’s clinical study reports were only 22% of the length of the corresponding GlaxoSmithKline reports (5316 vs. 23,645 pages). After transformation to PDFs, the median number of pages in the trial register entries was 32 (range 11 to 167). Results were posted on ClinicalTrials.gov for 19 studies (79%) but were not posted for 5 studies: HPV-001, HPV-003, HPV-013, HPV-033 and HPV-035. The median number of pages in the journal publications—including supplementary appendices—was 11 (range 7 to 83). Twelve (52%) journal publications contained supplementary appendices. The mean time from study completion to journal publication was 2.3 years (see Table 1).

Table 1 Comparison of HPV vaccine clinical study reports with trial register entries and journal publications: date and availability of clinical study reports, trial registry report results and journal publications
Full size table

Inclusion of protocols

Ten clinical study reports (42%), no trial register entries (0%) and 2 journal publications (9%) included protocols. All 12 protocols listed prespecified outcomes and contained statistical analysis plans (see Table 2). The GlaxoSmithKline trial register entries contained protocol hyperlinks to ClinicalStudyDataRequest.com, but the protocols were not freely available and had to be requested. We did not request the protocols, as this required us to sign a data sharing agreement, which would restrict our ability to publish our results.

Table 2 Comparison of HPV vaccine clinical study reports with trial register entries and journal publications: inclusion of protocol and reporting of trial design aspects including PICO criteria
Full size table

Reporting of major design-related biases

All 24 (100%) clinical study reports, no (0%) trial register entries and 9 (39%) journal publications reported explicitly on all six domains to be assessed for bias according to the Cochrane Handbook version 2011 [12] (see Table 2).

Reporting of PICO criteria

Compared to the trial register entries and journal publications, the clinical study reports reported on average more inclusion criteria (mean 7.0 vs. 5.8 and 4.0, respectively) and exclusion criteria (mean 17.8 vs. 11.7 and 5.0) (see Table 2). As an example, while 20 (83%) clinical study reports reported that participants with immunological disorders were excluded, only 12 (50%) trial register entries and 9 (39%) journal publications reported this criterion. All clinical study reports and journal publications specified the intervention and comparator contents (including antigens, adjuvants and doses), whereas only 18 (75%) and 8 (33%) trial register entries specified these. Active comparators (AAHS, Al[OH]3 and carrier solution) were referred to as ‘placebos’ in 14 (58%) clinical study reports, 13 (54%) trial register entries and 17 (74%) journal publications. The mean number of reported primary outcomes was higher in the trial register entries (3.5) than in the clinical study reports (1.6) and the journal publications (1.2). This was also the case for secondary outcomes (13.0 vs. 8.8 and 3.2) (see Table 2).

Meta-analyses of benefits

Of our seven prespecified benefit outcomes from the clinical study reports, the trial register entries included data for 2 (29%) and the journal publications for 6 (86%) (see Table 3 and Additional file 2). Compared to the clinical study reports, the trial register entries and journal publications contained 3% and 44% of the assessed benefit data points (6879 vs. 230 and 3015). Due to the lack of data in the trial register entries and journal publications, it was only possible to calculate the ratios of relative risk for half (10/21) of the prespecified benefit comparisons (see Table 4). The meta-analysis risk ratio estimates from corresponding study documents did not differ much (see Table 3), and the ratios of relative risk differences that could be calculated was not statistically significant (see Table 4).

Table 3 Comparison of HPV vaccine clinical study reports with trial register entries and journal publications: results of benefit and harm meta-analyses of intention to treat analyses irrespective of HPV type
Full size table
Table 4 Comparison of HPV vaccine clinical study reports with trial register entries and journal publications: ratio of relative risk differences of results of benefits and harms
Full size table

Meta-analyses of harms

Of our 13 prespecified harm outcomes from the clinical study reports, the trial register entries included data for 11 (85%) and the journal publications for 10 (77%) (see Tables 3 and 4 and Additional file 2). Compared to the clinical study reports, the trial register entries and journal publications contained 38% and 31% of the assessed harm data points (167,550 vs. 64,143 and 51,899). It was only possible to calculate the ratios of relative risk for 80% (31/39) of the prespecified harm comparisons (see Table 4). The meta-analysis risk ratio estimates did not differ much (see Table 3), and the ratio of relative risk differences that could be calculated was not statistically significant (see Table 4).

Random effects vs. fixed effect analyses

We found similar results with the fixed effect model but with narrower confidence intervals, as the between-trial variance is not included in this model.

Subgroup analyses

When we excluded the studies that had no results posted on their corresponding trial register entries (HPV-001, HPV-003, HPV-013, HPV-033 and HPV-035) from the clinical study report meta-analyses, the results did not differ significantly.

Study document differences

There were substantial differences between the amount of data in the three study document types (see Figs. 1, 2, 3, 4 and 5). For example, the journal publication for V501-013 included more cases of HPV-related moderate intraepithelial neoplasia or worse compared to its clinical study report (417 vs. 370; see Fig. 1). The trial register entry for HPV-015 reported fewer HPV-related treatment procedures than the clinical study report (160 vs. 198; see Fig. 2). The trial registry entry of HPV-040 reported 10 deaths (five in each group), whereas the clinical study report reported ‘no deaths considered as possibly related to vaccination according to the investigator (up to 30 April 2011)’, and the journal publication reported ‘No deaths had been reported at the time of this interim analysis (up to April 2011)’. Compared to the corresponding clinical study report, the journal publication of HPV-008 only contained an aggregate total number of serious harms (1400), whereas the clinical study report contained all individual serious harms classified with MedDRA preferred terms (2043). Only the trial register entries and journal publications for HPV-023 and HPV-032 included serious harms classified with MedDRA preferred terms (see Fig. 3). No journal publication of Merck Sharp & Dohme studies included their new-onset disease category: ‘new medical history’ (V501-005 to V503-006). Merck Sharp & Dohme did not provide a formal definition for ‘new medical history’ but described the category as ‘all new reported diagnoses’ in the clinical study report of trial V501-019. Although not mentioned as an explicit category, the trial register entries reported fewer events of new diagnoses than the clinical study reports (e.g. for V501-015: 329 vs. 35,546; see Fig. 4). Only the trial registry report of HPV-032 and the journal publication of V501-013 included general harms (see Fig. 5).

Fig. 1
figure 1

Comparison of HPV vaccine study documents: number of reported cases of HPV-related moderate intraepithelial neoplasia or worse

Full size image
Fig. 2
figure 2

Comparison of HPV vaccine study documents: number of reported cases of HPV-related referral procedures

Full size image
Fig. 3
figure 3

Comparison of HPV vaccine study documents: number of reported serious harms

Full size image
Fig. 4
figure 4

Comparison of HPV vaccine study documents: number of reported new-onset diseases

Full size image
Fig. 5
figure 5

Comparison of HPV vaccine study documents: number of reported general harms

Full size image

Discussion

There were on average 50 and 121 times more pages in the clinical study reports than in their corresponding trial register entries and journal publications. This was likely a main reason why the clinical study reports were superior at reporting trial design aspects. If our systematic review of clinical study reports [9] had relied on trial register entries or journal publications, it would have had no data for a quarter of our prespecified outcomes (11/40). Although the inclusion of clinical study reports led to significantly more eligible and available data, no changes in the direction of available results occurred when comparing the risk ratios of corresponding meta-analyses as ratios of relative risks. This may have several explanations. First, GlaxoSmithKline might be more transparent than other pharmaceutical companies [36], so corresponding study documents from GlaxoSmithKline could be more consistent compared to corresponding study documents from other companies [37,38,39,40]. Second, we used the random effects model, but more risk ratios had narrower confidence intervals with a fixed effect model. Third, there were low event numbers for several outcomes; differences in low event numbers may be overestimated when using risk ratios [12]. Finally, the studies were designed with a lack of placebo controls and incomplete reporting of harms [8] and the trial register entries and journal publications only included very few of the assessed data points (from 3% to 44%) compared to the clinical study reports. This may have skewed some of our comparison results towards being false-negative and led to an underestimation of harms caused by the HPV vaccines. Major study design features such as the use of active comparators and the reporting format of harms are not affected by the number of pages in a study document, but the vast increase in the amount of detail in clinical study reports allows for a more complete understanding that might impact conclusions. We have expanded on the issues of the lack of placebo controls and incomplete harms reporting elsewhere [8].

Strengths and limitations

Our comparison included 71 of 72 primary study documents (except for the journal publication of trial HPV-003 with 61 participants, which does not exist). Nearly all corresponding study documents (70/72) reported data from initiation to completion. To our knowledge, our study is the first with the aim of comparing meta-analyses from different study document data. The majority of study document comparison studies have mainly looked at harms [37,38,39,40]; we looked at both benefits and harms.

We did not obtain a single complete and unredacted clinical study report, so the included reports are less useful than complete and unredacted ones. We did not prespecify comparisons of clinical study reports obtained from different sources (i.e. EMA vs. GlaxoSmithKline), and we only prespecified ClinicalTrials.gov register entries for inclusion, as these are intended to have detailed summaries uploaded within 12 months of a study’s completion. We considered it appropriate to only compare a clinical study report with a single corresponding primary register entry and a single corresponding primary journal publication. A comparison that included all published information would become very complex and, in our view, less useful for researchers conducting systematic reviews.

As the clinical study reports were incomplete and often redacted, some eligible data may have been left out. We have described these issues elsewhere [8]. Cervarix™ clinical study reports obtained from EMA were a fifth of the length of the reports that we downloaded from GlaxoSmithKline’s trial register. Merck Sharp & Dohme clinical study reports (of Gardasil™, Gardasil 9™ and the HPV type 16 vaccine) were only obtained from EMA. These consisted of 9588 pages for seven trials. Thus, potentially 40,000 pages remain undisclosed for our comparison of Merck Sharp & Dohme clinical study reports [8].

Only 12 of 71 study documents contained the study protocol. We believe that all study publications should include the study protocol, as readers otherwise are less able to evaluate whether selective outcome reporting, protocol amendments or post hoc analyses were present in the study publication.

It was not possible to compare meta-analyses of per-protocol and intention to treat populations, as we had prespecified [11]. In the trial register entries and journal publications, per-protocol benefit outcomes were not reported irrespective of HPV type and harm results were not reported for per-protocol populations. Differences might have been more marked for these comparisons. For example, in the journal publication for HPV-015, it was stated that ‘Few cases of CIN2+ (moderate cervical intraepithelial neoplasia or worse) were recorded’ for the per-protocol population for CIN2+ related to HPV types 16 and 18 (25 vs. 34), but the corresponding clinical study report reported four times as many CIN2+ cases for the intention to treat population irrespective of HPV type (103 vs. 108).

The lower amount of data points in journal publications might be due to space restrictions, but in many biomedical journals, it is possible to include large electronic appendices. As there is no space restriction on ClinicalTrials.gov [41], the lower amount of data points was likely due to incomplete reporting.

Journal publications for five studies (HPV-031, HPV-035, HPV-040, HPV-058 and HPV-069) only included figures with graphs of general harms without exact numbers. We could calculate the absolute numbers from the percentages of general harms that were provided for four of the five journal publications (HPV-035, HPV-040, HPV-058 and HPV-069).

No journal publication of Merck Sharp & Dohme mentioned ‘new medical history’—a category used in all seven Merck clinical study reports. Merck Sharp & Dohme described ‘new medical history’ as ‘all new reported diagnoses’.

Some data in the trial register entries and journal publications were not comparable for our prespecified outcomes; for example, whereas the clinical study reports had reported an aggregate number of participants experiencing ‘solicited and unsolicited’ harms, the trial register entries and journal publications only reported general harms as ‘solicited’ and ‘unsolicited’ harms and that on a MedDRA preferred term and total level, respectively. We decided to compare such data as number of events but excluded non-aggregated data from the meta-analyses, as the data would constitute a considerable risk of counting participants more than once in an analysis (e.g. for trial register entries for GlaxoSmithKline studies, we only used ‘unsolicited’ events for general harms, as these were reported aggregately). For trial register entries for Merck studies, general harms were reported aggregately with local harms. We had not prespecified local harms as an outcome, so we did not use these data.

Since a journal publication page usually has a higher word and character count than a clinical study report page (that usually has a higher word count than a trial register PDF page), it may have been more appropriate to compare the word count of the study documents instead of the number of pages. As we received clinical study reports both from EMA and GlaxoSmithKline for some clinical study reports, some of the pages were duplicates and the median number of pages was therefore overestimated to some extent.

Similar studies

Our study supplements earlier studies that found reporting bias from clinical study reports to trial register entries and journal publications [38,39,40, 42]. Golder et al. performed a systematic review of 11 comparison studies that compared the number of harms in corresponding published and unpublished study documents [37]. They found that 62% (mean) of the harms and 2–100% of the serious harms would have been missed if the comparison studies had relied on journal publications. Similarly, our systematic review of the HPV vaccines of clinical study reports would have missed 62% of the assessed harm data points if it had relied on trial register entries and 69% of the harms if it had relied on journal publications. Our systematic review would have included 1% more serious harms classified with MedDRA preferred terms if it had relied on trial registers but missed 26% serious harms classified with MedDRA preferred terms if it was based on journal publications. It would also have missed 97% of the benefit data points if it had relied on trial register entries and 56% if it had relied on journal publications.

We found a mean time from trial completion to journal publication of 2.3 years. This is similar to a study by Sreekrishnan et al.—from 2018, of 2000 neurology studies—that found a mean time to publication of 2.2 years [43], but less similar to a study by Ross et al.—from 2013, of 1336 clinical trials—that found a mean time to publication of 1.8 years [44].

Conclusion

There were no significant differences in the meta-analysis estimates of the assessed outcomes from corresponding study documents. The clinical study reports were the superior study documents in terms of the quantity and the quality of the data they contained and should be used as primary data sources in systematic reviews; trial register entries and journal publications should be used concomitantly with clinical study reports, as some data may only be available in trial register entries or journal publications. A systematic review of the HPV vaccines would have had considerably less information and data included if it relied on trial register entries and journal publications instead of clinical study reports. A full data set would be expected to be available from case report forms and individual participant data, but there are regulatory barriers that need to be lifted before independent researchers can access such data [8]. Corresponding study documents ought to use consistent terminology and provide all aggregate and individual benefits and harms data. To test our results’ generalizability, we recommend that other researchers replicate and expand on our method of comparison for other interventions.