INTRODUCTION

Conflicts of interest (COIs) increase potential for biased judgment1 and have historically led to considerable suspicion of study findings. Multiple studies and papers have alluded to the pervasive nature of financial relationships between clinicians and manufacturers as well as reflected the increased scrutiny of such interactions.2 COIs are sources of bias3 defined by the Institute of Medicine (IOM) as “circumstances that create a risk that professional judgment or actions regarding a primary interest will be unduly influenced by a secondary interest.”4 They may be financial or non-financial.4, 5 Case reports and original research support an influence of COIs on study conception, data collection, and writing.6, 7 A study claiming that exposure to asbestos-containing roofing products was within safe limits had to be withdrawn following criticism concerning the approving editor’s associations with the asbestos industry.8 An industry-sponsored drug study provided a deceptively favorable profile of the medication by reporting 12-month outcomes as if they occurred at 6 months.9, 10 In 2001, an industry-sponsored study claimed the antidepressant Paroxetine to be “generally well tolerated and effective” in depressed adolescents. A reanalysis of the data by independent researchers, published in 2015, concluded that Paroxetine was actually neither safe nor effective in this population.11 Independent organizations’ reviews of studies written by a physician–scientist who received funding from organizations critical of genetically modified organisms concluded that the researcher’s statements were actually unsupported by his data.12

Until recently, reporting of COIs has relied almost entirely on author self-disclosure. In an attempt to improve transparency, the Physician Payments Sunshine Act, passed in 2010, now requires manufacturers of drugs, medical devices, and biological products to report all payments to clinicians having a National Provider Identifier (NPI).13 These financial transactions are reported to the public through the Open Payments Database (OPD).14 In 2015, 11.90 million transactions were reported on the OPD, amounting to a total of $7.52 billion dollars.3, 14

While COIs have contributed to the skepticism of published findings and clinical recommendations,15,16,17 few quantitative, scientific evidence exist concerning the global impact of disclosed and undisclosed financial COIs on study conclusions. We hypothesized that (1) articles with COI undisclosed by authors as opposed to articles with no COI are more likely to provide favorable impressions of discussed article product(s); (2) the presence of COIs, regardless of disclosure status, increases favorability; and (3) as the amount of total payments to the authors of an article increases, the favorability portrayed by the article toward a discussed product also increases.

MATERIALS AND METHODS

This was a blinded, observational study of previously published primary research articles. Strengthening the Reporting of Observational Studies in Epidemiology Statement (STROBE) guidelines were followed.18 PubMed was searched for articles in multiple randomly selected specialties as defined by the Accreditation Council for Graduate Medical Education (Supplementary Table 1).19 Articles were retrieved in reverse chronological order. Inclusion criteria included an acceptance date from 2013 to 2016 (corresponding to dates for which OPD data was available), at least 1 product listed in the title or abstract, at least 1 author with an NPI number (to qualify for inclusion in the OPD), and availability of an accompanying abstract. Self-disclosure statements were obtained from the full text of each article. The industry-reported COI for every author was obtained from the OPD. In accordance with the IOM, we defined payments listed in the OPD as COI if they were reported as honoraria, gifts, payments for research or consulting, compensation for serving as faculty or speaker, company ownerships/partnerships, and food/travel expenses greater than or equal to $5000.3 The number of industry-reported COIs, total payments received by all authors in the 1 year preceding a manuscript’s acceptance date, and the total number of interactions with industry (equal to the number of interactions recorded on the OPD) between authors of each manuscript and companies were also recorded. Two reviewers assessed relevance of self-reported and industry-reported COIs to the discussed product(s) in an article. Relevance was determined by performing an Internet search of all products produced by each named company and comparing these products to those described by each article. If a company produced the products directly mentioned by an article or produced competing products, that company was considered to be a relevant COI. Relevant COIs were determined to be either disclosed by manuscript authors (self-disclosed), by the OPD (non-self–disclosed), or both (full disclosure). The reviewers were blinded to abstract authors and their COIs as well as disclosure status of the conflicts.

Manuscripts were categorized into 4 disclosure categories:3 Full Disclosure, Incomplete Self-Disclosure (authors failed to disclose all COIs described on the OPD), Incomplete Industry Disclosure (industry did not disclose all COIs self-disclosed by authors), and No COI. Incomplete Self-Disclosure was also subdivided into Partial and No Disclosure categories. Partial Disclosure included all articles for which authors listed some but not all COIs reported in the OPD. No Disclosure included all articles for which authors listed no COI but for whom the OPD had listed COI. Disclosure categories were determined by comparing all self-disclosed COIs versus all OPD-disclosed COIs for each manuscript. Therefore, if 1 author of a study disclosed Conflict A for whom the OPD also disclosed only Conflict A, and a second author of the same study disclosed nothing but for whom the OPD disclosed Conflict B, this manuscript would have been coded as Incomplete Self-Disclosure. If, on the other hand, the second author was found to also be associated with Conflict A per the OPD, this manuscript would have been categorized as Full Disclosure. We considered disclosure categories on the manuscript rather than on the author level to reflect the collaborative nature of manuscript-writing; all authors’ inputs, as well as COIs, were considered important for each paper’s overall message and COI disclosure status.

Primary outcome was favorability defined as the impression an article provides of its subject matter. A panel of 3 independent, blinded clinicians reviewed and ranked the impression each article conveyed of product(s) within the article title on a 1–5 Likert scale in which “1” and “2” were “Unfavorable,” “3” was neutral, and “4” and “5” were “Favorable.” The reviewers were blinded to the article authors, institution, and COI. Article favorability was oriented based on specific author COI. For example, to address competitive company COI, if an article whose authors examined a product sold by Company B but who were affiliated with a competitor Company A, the ordinal response was flipped to adjust for such affiliations. If an article discussed more than 1 product, each reviewer also answered separate questions, or Likert scales, relating to the article’s specific favorability for each product. Therefore, the vendors of all products, including those favorably, neutrally, or unfavorably portrayed in the papers, were compared with the self-declared sources of support for the authors. Secondary outcomes included inter-reliability between the favorability scores of each manuscript question (3 favorability scores per manuscript), number of COIs, number of interactions, amount of total payments, and amount of research- and non-research–related funding.

Data Availability

The data reported in the paper can be made available by contacting the corresponding author.

STATISTICAL ANALYSIS

Univariate statistical analysis was performed for favorability using chi-square tests. One-way random-effect intraclass correlation was calculated to determine inter-rater reliability between the favorability scores of each manuscript question. Primary analysis compared Incomplete Self-Disclosure to No COI. Secondary analyses compared articles with any COI and all subgroups (n = 4) to those with No COI using chi-square or Kruskall–Wallis test. Due to multiple testing of the secondary analyses, we performed a Bonferroni correction (p < 0.05/number of analyses = 5) and considered p < 0.01 as significant.

A two-level (manuscript, question) multivariable mixed-effects ordered logistic regression model was used to assess factors associated with favorability to account for clustering on the rater- and manuscript- levels. The dependent outcome variable, favorability as an ordinal variable (1 to 5), was modeled as a function of the following independent variables: COI Disclosure Category (Full Disclosure, Incomplete Industry Disclosure, Incomplete Self-Disclosure, and No Disclosure), Relevance (yes/no), Research Payments ($0, $1–5000, $5001–25,000, greater than $25,000), and Non-Research Payments ($0, $1–5000, $5001–25,000, greater than $25,000). Interactions were also assessed between disclosure category, relevance, and payments. Model fit was examined by comparing Akaike Information Criterion (AIC). Two sensitivity analyses including the same covariates and random effects were performed. One excluded articles with incomplete industry reporting (due to missing information in the OPD). The second one used a dichotomized favorable outcome (scores of 4 to 5 versus 1 to 3) in a multilevel logistic model.

Sample size was calculated by assuming a difference of 10% in favorability (assuming 50% of studies with no COI were “favorable”), an alpha of 0.05, and a beta of 0.20. It was determined that 357 responses (119 manuscripts) would be needed per category (no COI and incomplete self-disclosure) for the primary study outcome. Based upon these calculations, we estimated a need to review 476 articles (4 categories × 119 articles).

All statistical analyses were performed with Stata Version 14.1®.

RESULTS

Three clinicians reviewed a total of 591 articles (8 overall reviewers, 3 reviewers per manuscript; total number of reviews n = 1773) (Fig. 1). The mean number of questions per article was 1.2, reflecting the majority of articles describing 1–2 products (1 question, or Likert scale, per product). The estimated intra-class correlation between scores within each manuscript question was 0.621 (95% CI, 0.581–0.659).

Figure 1
figure 1

Flowsheet of manuscripts.

Full size image

Articles with authors who did not disclose all COI as opposed to those with no COI showed increased favorability (73% versus 62%, RR 1.17, 95% CI 1.08–1.28, p < 0.001, Table 1).

Table 1 Number of Favorable and Unfavorable Responses per COI Disclosure category
Full size table

Articles with financial COI, regardless of disclosure status, had statistically significantly higher favorability than articles with no COI (74% versus 62%; RR 1.19, 95% confidence interval 1.10–1.29, p = <0.001). When comparing Full Disclosure, Incomplete Self-Disclosure, and Incomplete Industry Disclosure individually to No COI (Table 1), there was a significant difference in percentage of favorable responses and a significant increase in relative risk of having a favorable response if financial COI were present per disclosure category (Table 1). On subgroup analysis, both Partial Disclosure (n = 321/438, 73%; RR = 1.18, 95% CI 1.08–1.30, p < 0.001) and No Disclosure (322/444, 73%; RR = 1.17, 95% CI 1.07–1.29, p < 0.001) demonstrated increased favorability over no COI. Excluding manuscripts for which COI were determined to be not relevant to the products listed in the article titles did not significantly change the percentage of favorable responses (Table 2). Total payments, research payments, non-research payments, and number of interactions also significantly differed per COI Disclosure Category (p < 0.001 for all, Table 3).

Table 2 Number of Favorable Responses per COI Disclosure Category Sorted by COI Relevance
Full size table
Table 3 Univariate Analysis of Factors Stratified by COI Disclosure Category
Full size table

MIXED-EFFECTS MULTIVARIABLE ORDERED LOGISTIC REGRESSION ANALYSIS

In order to assess the relationship to COI, disclosure, payment amount and type, and favorability, a regression analysis was performed. COI Disclosure Category was found to be significantly associated with favorability (Table 4). Relevance, Research, and Non-Research Payments were not found to be significant. No interaction was detected between disclosure category, relevance, and payments. Subgroup and sensitivity analyses demonstrated similar results.

Table 4 Mixed-Effects Multivariable Ordinal Logistic Regression Analysis
Full size table

DISCUSSION

Financial COI, disclosed or undisclosed, “relevant” or “irrelevant,” may influence published scientific studies compared to studies with no COI. Financial support provided by relevant COI may affect results by providing authors a material incentive to favor the source of monetary support. Authors having COI not considered relevant may potentially be seeking additional or future monetary support from industry. This is the first study to correlate study favorability, data from the OPD, self-reported COI, relevance of COI, and payment amount and type into a single analysis.

There was an extremely high rate of discordance in the reporting of COI. Reasons for discordance between self-disclosure and OPD-disclosed COI are unknown but may include the following: human error, researchers omitting COI considered not relevant to investigated topic(s), lack of awareness of COI, and errors in the OPD. Excluding articles with “irrelevant” COI did not substantially change favorability rates. While analysis of articles in the Full Disclosure category is hampered by their small number (n = 20/511, or just 4% of the articles sampled), the high favorability observed within this category may be explained by the high number of articles perhaps being explicitly funded by 1 particular company. The lower number of median COI supports this for the Full Disclosure Category than for the other disclosure groups (Table 3). It may be that these authors had a clear and strong relationship with industry when designing and writing the study. However, our multivariable analysis suggests that favorability rates are increased regardless of concordance of disclosure between authors and the OPD. Authors who “over-disclose” ties also led to industry-favored publications; this represented only a small percentage, 11%, of manuscripts. We attributed the bulk of this group to industry or human error which is a more realistic estimate as compared to the group of authors who failed to self-disclose (58% of manuscripts). In addition, some authors who “over-disclose” could have relationships with the industry outside of the 1-year preceding article acceptance for publication. Once the OPD has accumulated multiple years of data, future studies should assess the duration of effect of financial transactions on favorable reporting.

While articles with financial COI demonstrate a high favorability rate, articles lacking any financial COI still showed strikingly high favorability toward industry and products, suggesting that factors other than monetary relationships affect publications or that total food/travel expenses less than $5000 may also affect the writing and research styles of physician–scientists. The actual probability of “true” studies has been posited to rely on a combination of power, ratio of true to not-true relationships, and bias; well-powered epidemiological studies, per this logic, may have only a 20% chance of being true.17 In this study, articles with no financial COI were determined to have a favorability of 62%, suggesting that potentially 40% of favorability results from non-financial sources of COI. This could include individual researcher biases driven by career pressures, mentorship, pride, etc., or systems-based biases such as publication bias of positive results. However, these could also be related to non-financial COI such as relationships physicians form with industry apart from financial transactions. Our multivariable analysis begins to quantify these other variables by evaluating, in addition to payments and disclosure status, the actual relevance of the COI to the article topic. Additional variables that we did not evaluate in our hierarchical linear regression model include career position and aspirations of physician–scientists, choice of researchers to perform studies on experiments likely to have positive outcomes, and publication bias at the level of scientific meetings and journals. Future studies should evaluate the influence of these additional non-financial COI on article favorability.

Our finding, that both disclosed and undisclosed financial COI influence study findings, is supported by major historical cases. In what has been dubbed “the most damaging medical hoax of the last 100 years,” Dr. Andrew Wakefield posited a connection between the measles–mumps–rubella vaccine and autism–enterocolitis. Editors of the journal in which the article had been published later discovered that Dr. Wakefield had failed to disclose that he had received approximately $670,000 in compensation as a consultant to lawyers of parents of children allegedly harmed by the vaccine and that he was an inventor on a patent for a new vaccination. Despite 11 of the 12 coauthors subsequently issuing a formal retraction of the manuscript’s claims and the General Medical Council Fitness to Practice Panel recommending revocation of Dr. Wakefield’s license to practice medicine in England, measles outbreaks began to occur in London and rapidly spread to Scotland and Ireland following claims made by the criticized paper.20 These incidents may have been prevented had Dr. Wakefield accurately disclosed his COI.

Limitations of this study include the inability to find disclosure statements for 14% of manuscripts. While high, this number is actually smaller than those cited by other studies.21 The favorability of these excluded manuscripts was only found to significantly differ from manuscripts with no COI (Fig. 1, 181/240, 75%, p = 0.14). The authors of this manuscript were also limited by the constraints of the OPD, which can affect the generalizability of this study. To be listed on the OPD, an article author had to be a practicing clinician in the USA with an NPI number. The applicability of these results to other countries and scientists without an NPI number is unclear. While not every author of every manuscript examined in this study possessed Doctor of Medicine degrees, approximately 85% of the authors possessed NPI numbers, thereby meeting inclusion criteria for the OPD. An additional limitation may be that our clinician raters who provided article favorability ratings are admittedly not experts in each manuscript’s specific field. However, we intentionally did not utilize experts within each specialty in order to better reflect the majority of manuscript readers, who are not experts, and to ensure the integrity of blinding. Experts may already be familiar with each abstract’s authors and their COI due to preexisting knowledge of the field. Future studies could include assessing individual university databases of conflict of interest, interviewing published researchers on their perceptions of COIs and disclosure.