Introduction

Multiple sources of bias are found in health research. Two of the most discussed biases include selection bias and confirmation bias. Selection bias happens when participants or other types of research material such as biobank material, cell lines or mouse strains, are not randomly selected for a study. The best way to select for people or animals in a research study is through randomisation where everyone within the group that are investigated are selected randomly to attend the study, however, this is not always possible (Medical Research Council n.d.). Second, confirmation bias appears when researchers, intentionally or unintentionally, look for information or patterns in their data that confirm the ideas or opinions that they already have (Medical Research Council n.d.). Other types of biases which are especially relevant for clinical trial studies or observational studies include (i) channelling bias, where patients within the study are not randomly selected for a given study subgroup (Lobo et al. 2006), for example if prognostic biomarkers or degree of illness affects which cohort the patients are placed in; (ii) performance bias, for example in clinical trials involving surgery which can have technical variability between surgeons; (iii) interviewer bias, which refers to a systematic difference between how information is gained or interpreted in the different groups, for example if the interviewer ask different questions or formulate them differently when interviewing the different groups; (iv) recall bias, where respondents or patients need to remember what has happened in the (more or less distant) past; (v) observation bias, where participants alter their behaviour in the knowledge that they are studied; (vi) chronology bias, when historic controls are used as a comparison group for patients undergoing a therapeutic intervention; and (vii) transfer bias, where patients drop off from the study, and it has to be considered whether these patients are fundamentally different from those who remained in the study (Pannucci and Wilkins 2010). Bias also happens after a study is completed – we have for instance publication bias, which will be the focus of this chapter, meaning that the published results are not a representative selection of all results within a study, and not all studies are published causing unfavourable results to be less reported.

Meaningful research outcomes have been defined as findings that advance their respective field of research and have a practically useful effect on society (Helmer et al. 2020), but this requires research to be shared with peers, decision makers and citizens. The main format of research dissemination is through publications in peer reviewed scientific journals. However, other formats of information sharing also play an important role, like oral or poster presentations, newspaper articles, books, web pages, research archives, informal discussions, and all kinds of forums where research results and related information are being communicated. Research results that are not shared with anyone will not be of any value to anyone other than the researchers who performed the study. This is why disseminating and communicating research results, and more generally, opening up research to other’s scrutiny, is an integral part of the role of a researcher.

Publication bias occurs in scientific research if the outcome of a study influences whether or not the results will be published in a scientific journal, presented at conferences or otherwise distributed and made available for society as a whole (Song et al. 2010). Publication bias could thus be defined as “the selective publishing of research based on the nature and direction of the findings” (Marks-Anglin and Chen 2020). When studies with significant or favourable results are more likely to be published than those with non-significant, unexpected or unfavourable findings, it skews the balance in the pool of available research results, thus causing a bias in favour of so-called ‘positive’ results (Song et al. 2010; Marks-Anglin and Chen 2020). Factors that determine the selection of results include experimental outcome and how the results sit in light of the original hypothesis and previously published work. For example, an experimental result is often considered ‘positive’ when a difference that is statistically significant is observed. In other cases, ‘representative’ results may be considered in the light of the original hypothesis and selected in a manner that excludes contradicting outcomes, best suits the hypothesis and fits into the logical flow of the paper, in order to maximise the probability that the results are accepted for publication in peer reviewed journals.

This chapter aims to explore publication bias in the context of precision oncology and cancer biomarker research; why it exists, implications it has for researchers, patients, and society, as well as reflecting on the deeper roots of the problem. Section “Evidence of publication bias in medical research” provides evidence of publication bias in medical research in general, and how this applies to cancer biomarker research specifically. Section “The impact of publication bias on the validity of the scientific literature and contribution to the reproducibility crisis” explains the different types of publication bias based on whether or not the statistical hypothesis is true or false and how this has an impact on the validity of the scientific literature, and the contribution of publication bias to the reproducibilitycrisis. Section “Discussion: Publication bias in precision oncology and cancer biomarker research; implications and reflections on the deeper roots of the problem” discusses possible implications of these biases for patients, researchers and the scientific society and the general public, and offers reflections on how to minimise the occurrence of publication bias.

Evidence of Publication Bias in Medical Research

The term ‘publication bias’ started appearing sporadically in the literature in the 1980s but the number of publications have increased remarkably over the years, as illustrated in Fig. 1 (Marks-Anglin and Chen 2020; Simes 1986; Easterbrook 1987; Boisen 1979; Begg 1985). However, although the term publication bias did not appear in the literature until 1979, the concept itself was discussed much earlier (Marks-Anglin and Chen 2020; Editors 1909). In 1959, statistician Theodore Sterling and colleagues presented evidence that published results are not representative of all scientific studies (Sterling 1959; Sterling et al. 1995). Sterling found that as much as 97% of the papers published in some of the major journals in the field of psychology had statistically significant findings for their major scientific hypothesis, highly indicative of publication bias in the field (Sterling 1959; Sterling et al. 1995).

Fig. 1
The line graph illustrates the number of publications from the year 1980. The average trend rises to a maximum limit of up to 3500.

Number of publications available at PubMed (https://pubmed.ncbi.nlm.nih.gov/) for the search query «publication bias» between 1979 and 2020. (The figure was created by the authors in GraphPad Prism v8.3.0)

Full size image

In a retrospective study published by Easterbrook and colleagues in The Lancet in 1991, the authors followed 487 research projects and found that studies with statistically significant results were more likely to be published than studies that were statistically nonsignificant (Easterbrook et al. 1991). In addition, research projects with significant results also led to a greater number of publications and presentations, and the results were published in journals with higher impact factors (Easterbrook et al. 1991). It has also been found a greater tendency towards publication bias in observational or laboratory experimental studies compared to studies of randomised clinical trials (Easterbrook et al. 1991). Easterbrook and colleagues further claimed that “the most serious potential consequences of publication bias would be an overestimate of treatment effects or risk-factor associations in published work, leading to inappropriate decisions about patient management or health policy” (Easterbrook et al. 1991).

Multiple studies have later investigated publication bias in the scientific literature in a systematic way (Dickersin and Min 1993; Franco et al. 2014; Driessen et al. 2015; Vera-Badillo et al. 2016; Scherer et al. 2018). These studies have looked at projects receiving ethical approvals, external funding, reports to authorities or conference abstracts, and studied the correlations between the amount and type of scientific publications and whether or not the study gained positive or significant results (Marks-Anglin and Chen 2020). It was clear from many of these studies that publication bias indeed occurs and positive results are more likely to be published than negative results (Dickersin and Min 1993; Driessen et al. 2015; Vera-Badillo et al. 2016; Scherer et al. 2018). Reports have also shown that as many as 50% of studies may not be published in any given area of research and that it is more than twice as likely that null results will not be published or communicated (Shields 2000). In addition, these studies also demonstrated that other types of publication related biases exist including time-lag bias where favourable results are published within shorter time (Ioannidis 1998; Shields 2000), citation bias meaning that favourable results are more cited (Nieminen et al. 2007; Shields 2000), and sponsorship-bias in the way that studies sponsored by industrial funding are less likely to be published compared to government funded research (Marks-Anglin and Chen 2020; Scherer et al. 2018; Lexchin et al. 2003).

Evidence of publication bias has also been reported for clinical trial publications (Simes 1986; Vera-Badillo et al. 2016; Bardy 1998). As an example, Simes and colleagues reported in 1986 that when only published results from clinical trials were considered, combinational chemotherapeutic regimes were statistically preferable compared to single agent therapy in ovarian cancer (Simes 1986; Sterling et al. 1995). However, when all registered trials were included in their analysis, the statistically significant advantage disappeared. Another important bias observed in clinical trials in oncology is the under-reported toxicity which is essential for the approval of new treatments (Vera-Badillo et al. 2016).

Publication Bias in Precision Oncology and Cancer Biomarker Research

Cancer therapy has greatly developed over the years, and as new and more targeted therapies become available, cancer therapy is moving from standard treatment regimens to a more personalised and tailored therapy, also referred to as precision oncology. However, despite the development of multiple different molecularly targeted therapies, most patients with advanced cancer will not experience durable clinical response from targeted therapies (Marquart et al. 2018). As cancer therapy becomes more personalised, there is therefore a constant need for novel predictive biomarkers to guide tailored therapy based on the patients and the tumours’ unique characteristics, as the treatment is no longer solely based on the tumour type. The BEST (Biomarkers, EndpointS and other Tools) glossary defines a biomarker as “a characteristic that is measured as an indicator of normal biological processes, pathogenic processes, or responses to an exposure or intervention, including therapeutic interventions” (FDA-NIH Biomarker Working Group 2016). In the field of oncology, biomarkers have multiple applications including diagnosis and subtyping of cancer (diagnostic biomarker), and they can also be used to estimate prognosis (prognostic biomarker), predict treatment effect (predictive biomarker), or to monitor the treatment effect or cancer recurrence over time by longitudinal sampling (FDA-NIH Biomarker Working Group 2016; Wu and Qu 2015).

Cancer biomarker research ranges from experimental studies to clinical applications and involves various types of studies including cell culture and animal models, research on humans or human material (including databases and clinical trial studies) or even computational modelling. Publication bias in precision oncology can occur at any stage of the process ranging from the early discovery to the clinical validation of new biomarkers. Within a complex biological system as tumours are, there is a high degree of both intra-tumour and inter-tumour heterogeneity which also changes over time and affects drug responses. Cancerbiomarkers include a wide range of molecules, including DNA, mRNA, enzymes, metabolites, transcription factors, and cell surface receptors (Wu and Qu 2015), and many of these are continuous variables (e.g. protein expression) or exists only in a certain proportion of the cells or tissues analysed (e.g. frequency of DNA mutations). Since biomarker definitions are often based on measurements of a continuous variable, such as protein expression or proportion of biomarker positive cells, there is therefore not a clear cut-off between a biomarker positive and biomarker negative sample. In addition, the analysed material is typically only taken from a small part of the tumour and this subsampling might not be representative for the whole primary tumour and potential metastases that could have a very different biology than the primary tumour it is derived from. Biomarkers are frequently used to guide therapy, and even the therapeutic outcome for the patient, typically measured as responders or non-responders in accordance with a biomarker, does not have a clear cut-off as some patients can have a partial response.

The story of HER2 in breast cancer is often referred to as an example of a successful biomarker story. Multiple copies of the gene encoding for the HER2 receptor causes cancer cells to be more responsive to growth signals making the tumour more aggressive, and HER2 gene amplification is associated with worse prognosis than HER2 negative breast cancer when kept untreated (Lakhtakia and Burney 2015). However, research during the 1990s led to development of the monoclonal antibody trastuzumab (brand name: Herceptin) which specifically targets the HER2 receptor and significantly improves the outcome of HER2 positive breast cancer patients (Slamon et al. 2001; Lakhtakia and Burney 2015). This biomarker could thus be considered as “ideal” in the sense that it has entered clinical use as a biomarker that effectively identifies a subgroup of breast cancer patients that are more likely to benefit from HER2 targeted treatment, thus contributing to a more tailored and effective breast cancer treatment, which also saves the biomarker negative subgroup from potential side effects and toxicity from a treatment that is less likely to have an effect. However, although HER2 is frequently used as a textbook example of the “ideal” cancer biomarker, it is still not perfect in determining treatment response and some patients develop therapy resistance. Thus, it could be argued that it is more accurate to talk about biomarkers as “good enough” rather then “ideal”, leaving more room for accepting the complexity and uncertainty of biological systems that biomarkers are based upon (Blanchard and Wik 2017). It has also been detected HER2 positive metastases in patients with HER2 negative primary breast cancer and vice versa (Xiao et al. 2011; Ulaner et al. 2016), which illustrates the complexity of the tumour biology and highlights some of the challenges with applying a cancer biomarker in clinical practise.

The example of the HER2 biomarker illustrates that even for the most promising biomarkers, the outcome of a biomarker test is not absolute in predicting patient response to a therapy, it can only place the tumour or the patient in a group that has statistically higher or lower chances of some degree of response (Fleck 2017). When biomarkers are included in clinical practice and thus accepted as valid predictors of biological or clinical outcome, this creates a ‘skew’ in the available literature caused by publication bias, and it will have an impact on the set threshold for biomarker positive or negative samples or when deciding which biomarker defined subgroups of patients that will receive a given therapy. It is also likely that the more complex a field of research is, the more it will be influenced by publication bias. Indeed, the outcome from studies of complex, uncertain, and non-linear systems will have more variations, and therefore more room for subjective selection of results prior to publication. Publication bias will then in turn create an illusion from the literature that the biology behind the findings is less complex and more certain than it actually is (Blanchard 2016). In a field such as precision oncology where the researchers are aiming for perfection, this could also increase the risk of publication bias as there might be less room for publication of negative results.

The Impact of Publication Bias on the Validity of the Scientific Literature and Contribution to the Reproducibility Crisis

The fact that many scientific studies are difficult or even impossible to replicate or reproduce has become so evident that the term ‘reproducibility crisis’ is used to describe this phenomenon (Miyakawa 2020; Twa 2019). A major contributor to this crisis is believed to be publication bias caused by the fact that statistically insignificant results are rarely published or discussed in scientific publications (Marks-Anglin and Chen 2020). In an online survey performed by Nature answered by more than 1500 participating scientists, 70% of the researchers answered that they had tried and failed to reproduce others’ experiments (Baker 2016). When the researchers were asked what led to these problems of reproducibility, more than 60% mentioned the strong pressure to publish, or ‘publish or perish’ culture, and selective reporting of results (Baker 2016).

In principle, there are two different scenarios of publication bias based on statistical significance depending on whether the statistical hypothesis is true or not: the false hypothesis bias and the true hypothesis bias (Sterling et al. 1995). Typically, the statistical null hypothesis (H0) is defined as ‘no differences between the groups’. To illustrate these two types of biases, we define the two groups as two different biomarker based subgroups named X and Y. The statistical null hypothesis will then be defined as no differences in treatment effect between the groups, meaning that the treatment effect is similar in group X and Y. A study is then performed to see if there is evidence to disprove H0, and if it is, the study concludes by rejecting H0 and accepting that there are differences between the groups (often referred to as accepting H1). Importantly, the statistical null hypothesis should not be confused with the scientific hypothesis, which typically will be to disprove the statistical null hypothesis. In our example, the statistical null hypothesis is defined as no differences between group X and Y, while the scientific hypothesis will be that there are in fact differences in the treatment effect between the two biomarker defined subgroups.

In all statistical testing, the null hypothesis can either be wrongly rejected (Type I error) or the test can fail to reject a false null hypothesis (Type II error). The experiment or study performed is defined as significant based on a significance level alpha, often set to 0.05 (5%). Given that H0 actually is true and the significance level is set to 0.05, there will then be a 5% chance that the null hypothesis will be (wrongly) rejected, also referred to as a type I error. This will be the case every time this same experiment is performed, and if either the same researchers perform this experiment many times or the same experiment is repeated by different investigators (that might not even be aware of each other’s research), the chances that a type I error will occur by chance in at least one of the performed studies will accumulate over time and number of experiments/studies performed. Then, if only the one or the few studies that showed statistically significant results are published, while the majority of the studies that showed insignificant results are ignored in the sense that they are not published or otherwise made available, this will lead to a situation where the published results are not at all representative for all the experiments that have been performed. Multiple repetitions of the same experiment will thus accumulate the chance of a wrongly rejected true statistical null hypothesis, also referred to as the true hypothesis bias. In addition, simply increasing the number of replicates, applying another type of statistical test, increasing the statistical power by comparing only selected subgroups, or removing so-called ‘outliers’, may be the difference between a significant and non-significant result, and are other examples of how publication bias can skew the statistics in favour of increasing the chances of type I errors.

The second scenario is that the statistical null hypothesis is false. In our example this means that there actually is a difference between group X and Y. In this case, publication bias will cause a bias against elimination of type II errors, meaning a bias in favour of eliminating false negative results, referred to as the false hypothesis bias. Although this will cause a bias in the direction of the correct conclusion, it will still have implications since it will skew the results presented in the literature that will make it look like the differences are bigger, or at least more significantly different, than they actually are. This could have implications for example when the benefit and side-effects or toxicity of a treatment are considered against each other or when evaluating the validity of a biomarker. Therefore, no matter if the statistical null hypothesis is true or false, publication bias will make the probabilities of statistical type I and type II errors different for the reader than for the initial researchers that performed the study (Sterling et al. 1995), and it will “skew” the available literature by increasing the chances of Type I errors and decreasing the chances of Type II errors.

Publication bias also serves as a threat to the validity of meta-analysis (Marks-Anglin and Chen 2020). Meta-analysis is a method that combines the results from multiple similar studies and aims to make it possible to draw conclusions with a higher degree of certainty. Meta-analyses are frequently used in oncology, for example when evaluating how good a new treatment regime is compared to standard treatment, or it can also be used to evaluate the validity of a cancer biomarker. Meta-analyses are based on the assumption that the meta-study summarises all relevant studies, or at least a representative selection (Sterling 1959). However, publication bias will have an impact on the conclusion of a meta-analysis if it only includes published results. It should also be mentioned in this setting that other types of biases including citation bias and time-lag bias could skew the results of meta-analysis (Marks-Anglin and Chen 2020). However, although the meta-analysis could have a wrong conclusion based on a biased ‘selection’ of only published data, meta-analyses still tend to be trustworthy and are especially convincing since they cover multiple studies. However, this discussion on publication bias invites us to handle those meta-analyses with a critical eye.

Discussion: Publication Bias in Precision Oncology and Cancer Biomarker Research; Implications and Reflections on the Deeper Roots of the Problem

Implications for Patients, Decision-Making in Clinical Practice and Socio-economic Aspects

In the case of medical research in general and also precision oncology and cancer biomarker research specifically, publication bias will have many possible implications that eventually affect the patients, for instance through consequences for medical practice and evidence-based medicine (Marks-Anglin and Chen 2020). More specifically, policy and decision-making processes rely on the scientific literature, and publication bias can therefore result in inappropriate decisions about health policy and patient management (Marks-Anglin and Chen 2020; Easterbrook et al. 1991). Worst case scenario, publication bias in the field of oncology can cause inappropriate estimation of the balance between treatment effects and toxicity, resulting in inappropriate treatment of cancer patients. Ideally, a cancer biomarker should be reliable, cost-effective and powerful in detecting and monitoring cancer risk, cancer detection and tumour classification so that improved medical decisions can be made and the patients will receive the most appropriate therapy (Wu and Qu 2015; Blanchard and Wik 2017). Biomarkers are thus important for subtyping patients into groups, for example when a new treatment regime is considered for use in clinical practice. Publication bias in the field has therefore a direct impact on this decision-making.

There is not always an obvious cut-off for biomarkers, both in relation to what is a positive or negative sample and whether the defined subgroup will benefit from a given therapy. Publication bias will in this setting skew the literature which could affect where these cut-offs are set and further which patients that are given the therapy. One such example of a biomarker where there is no obvious cut-off or implementation of the biomarker is the use of the protein expression of programmed cell death ligand 1 (PD-L1) to predict response to immune checkpoint inhibitors (ICB). Although PD-L1 expression is established as a biomarker to predict response to ICB, its clinical utility as a biomarker remains to be further investigated. Clinical trials are not consistent in their conclusions of weather PD-L1 predicts response to ICB, and the biomarker defined cut-offs are varying as much as from >1% to >50% of PD-L1 positive cells necessary to define the tumour as PD-L1 positive (Yi et al. 2018). When the results are less clear like in the example of PD-L1, it is also likely that publication bias (if it exists) could have greater implications than if the results are clearer, as only a small bias in the literature then can make a big difference when decisions about patient treatment are made.

A cut-off can either be selected prior to the study based on previous knowledge or experience or by applying a statistical method to the data to estimate new cut-off values (Woo and Kim 2020). When applying a statistical method there are two different approaches either based on the biomarker distribution itself or a selection can be made based on the association between biomarker and outcome (Woo and Kim 2020). A popular method to predict biomarker based patient outcome is to select a cut-off value that minimises the p-value when comparing the outcome in different groups (Woo and Kim 2020; Polley and Dignam 2021). However, this strategy of minimising the p-value results in highly unstable p-values and increases the chances of significant findings when the biomarker is not associated with outcome (Polley and Dignam 2021). It can thus be argued that this method directly causes publication bias or misinterpretation of results since the cut-off is selected in a way that corresponds to the most statistical significant difference in the data set and the significance of the chosen cut-off value therefore tends to be overestimated causing an increased rate of false positive errors. Although methods have been developed to reduce this effect of false positives, the lack of reproducible biomarker cut-offs is still a challenge that might have hindered the adaptation of biomarkers into clinical practice (Polley and Dignam 2021).

A type of bias that is related to publication bias is the overinterpretation of results, in the sense that the meaningfulness of the result can be embellished with overly optimistic terms (Fong and Wilhite 2017) or the speculation for its application in the clinic may be exaggerated to maximise acceptance for publication. In a systematic review of ovarian cancerbiomarkers, Ghannad and colleagues found that interpretation bias is abundant in evaluation of cancer biomarker studies and that it is a practice of making study findings appear more favourable than what could be justified from the results (Ghannad et al. 2019). The authors further claim that this misinterpretation or overinterpretation may lead to an unbalanced and unjustified optimism in the performance of potential biomarkers, and the published literature might suggest stronger evidence than what is justified. The most frequent misinterpretations found in their study include claiming other purposes of the biomarker that were not investigated, mismatch between the aim and the conclusion and incorrect presentation of the results (Ghannad et al. 2019). In particular, the most frequent mismatch in the results was the selective reporting of the most positive or statistically significant results in the abstract. This illustrates again that the ‘ideal’ of precision oncology, aiming for perfect biomarkers to support perfect clinical decision-making and highly tailored treatments to individual patients, puts a high pressure on researchers to put forward positive results that support this ideal. We can see how difficult it is, then, to totally avoid the practice of publication bias: the ideal of precision oncology demands ‘perfect’ results and biomarkers, but since the biology around cancerbiomarkers is so complex, these results can only be achieved through a biased analysis, interpretation, and presentation of results.

Publication bias in precision oncology and cancer biomarker research also has broader socio-economic aspects. It is known that, in addition to the devastating effects that cancer has on patients and their families, the economic consequences of cancer are enormous (Wu and Qu 2015). Cancer-related economic costs include the direct health care resources and the cost of expensive cancer therapies, and it also includes loss of human capital due to early mortality or inability to work because of the disease (Wu and Qu 2015). When a new drug or treatment regime is evaluated in a subgroup of patients based upon a set of biomarkers, the health benefit are evaluated against the cost, and publication bias could skew this equation in favour of increased benefit of a treatment, which potentially could lead to approval of a treatment that otherwise would not have been approved for the given clinical application, thereby affecting health care resource allocations. A lot of resources are used in biomarker research, and despite that, the harsh reality is that less than 1% of published biomarkers end up entering clinical practice (Kern 2012). There are many possible explanations for this including the complexity of malignant tumours, but publication bias is also one out of many contributors to the fact that only few of the potential biomarkers end up reaching a clinical application. For example, some studies might find that a particular protein X is indicative of response to a treatment in their study while other studies may not find that this trend exists in their cohort. If the former studies are more likely to be accepted for publication while the latter will not even be considered submitted, this could cause a biased availability in the literature of the evidence for using X as a biomarker. This will in turn have an impact on designing new research projects evaluating biomarker X, and publication bias in the follow up studies, will further escalate the problem, and is likely to affect the number of biomarkers that in the end will end up in the clinic. Biomarker research is often funded by the government or funding agencies, and as a general rule the goal of all government funded research should be to benefit the community. If research results are not published nor otherwise made available, the knowledge gained from this research cannot be used for the benefit of society and it could therefore be argued that these resources could be better spent somewhere else. It could also be argued that it is unethical to perform research without publishing it, both in respect to the funding agencies, but also in respect to the participants including patients or volunteers who contribute to the research material, and also with respect to the society in general since tax money is used to fund government funded biomarker research.

Implications for Researchers and the Scientific Community

As seen in section “The impact of publication bias on the validity of the scientific literature and contribution to the reproducibility crisis”, publication bias also has important implications for the researchers and the validity of the scientific literature. Justifying the design of new research projects relies heavily on previously published studies and literature, and when these are not representative, researchers run the risk of basing further studies on false premises. To obtain positive results is especially important for PhD candidates (and other early-stage researchers) that are early in their career. They have a limited time to do their research, but they are faced with the pressure of ‘publish or perish’ and are expected to have publications preferably in high impact journals, so that they can contribute with new and valuable knowledge to their field in order to graduate. Publication bias leading to the reproducibilitycrisis causes a situation where new PhD projects could be based on a skewed or even wrong literature, which is increasing the chance for the candidates to have difficulties publishing their findings – soon enough, they might realise that their project is actually a dead end. In addition, supervisors might encourage PhD candidates to test hypotheses that are considered as dead ends from the beginning, and cases has also been discussed at forums (including https://academia.stackexchange.com) where multiple candidates within a research group are set to do very similar projects, and whoever finishes the task first will get their name on the publication (Lowe 2019). Approaches like this further increase the rate of publication bias in the field, and the pressure to publish is likely to be one of the reasons for the mental health challenges in science. In a study by Levecque and colleagues, an increased prevalence of mental health problems for PhD candidates was observed, compared to the highly educated general population, and a third of PhD candidates in the study was at risk of a psychiatric disorder (Levecque et al. 2017; Pain 2017). Encouraging research into dead ends also causes a waste of time, money and research effort since multiple researchers perform the same studies potentially without being aware of each other’s null results. Statistically insignificant results are therefore significant in their own right because they provide valuable information to scientists designing new studies which will ultimately save researchers time and resources that could be more efficiently spent.

Negative results that are either insignificant or disprove the original scientific hypothesis could be at least as important as the positive results. For example, one of the now world’s best-selling breast cancerdrugs, Tamoxifen, first synthesized in 1962 as a contraceptive pill in the pharmaceutical laboratories of ICI (now part of AstraZeneca) was not patented because it stimulated, rather than suppressed, ovulation. The project was nearly stopped but was reportedly saved partly because team leader, Arthur Walpole, threatened to resign, and pressed on with a project to develop tamoxifen for the treatment of breast cancer. It was initially used as a palliative treatment for advanced breast cancer but later became a best-selling medicine in the 1980s, when clinical trials showed that it was also useful as an adjuvant to surgery and chemotherapy in the early stages of the disease and even later, trials showed that it could prevent occurrence or re-occurrence of disease in at high-risk individuals. Tamoxifen therefore became the first preventive for any cancer, helping to establish the broader principles of chemoprevention, and further extending the market similar drugs (Quirke 2017).

A primary goal of research is to test hypotheses, but the researchers are in no control of whether this process will lead to a ‘positive’ finding. If you are unlucky and end up with only null results in your project, or you are not able to replicate the ‘common knowledge’ in your field (which might be wrongly represented in the literature because of publication bias), then we know that these results are more difficult to publish. Difficulties by publishing contradictive findings could thus result in the literature only supporting a certain hypothesis or established scientific opinion in the field although there are a lot of unpublished data supporting the opposite hypothesis (Prinz et al. 2011). This in turn will have consequences for the researcher’s career as scientists are generally judged and ranked by their number of publications, impact factors and citations, when applying for an academic position or project funding. The lack of control over experimental outcomes generates unbiased results, and so it seems ironic that this core aspect of research that is beyond the researcher’s control plays such an influential role in whether they have a ‘successful’ future. It is also ironic that this dilemma exists in the field of science where logic and fairness are the pillars of its foundation. Hence, it is inevitable that these factors are likely to become major influencers driving motivation, overshadowing consideration of patient/societal benefit. Researchers may also become mentally and physically stressed because their careers and livelihood can be dependent on the attainment of positive results.

Reflecting on the Roots of the Problem of Publication Bias

In order to reflect on ways to avoid, as much as possible, the problem of publication bias, we first need to discuss what is causing the problem. There are at least two possible explanations of why we have publication bias: (i) researchers might decide not to submit ‘negative’ results because they are in a system where negative results might jeopardise their career or their opportunity for future funding; (ii) the journals are more likely to reject a manuscript where the results are ‘negative’ because the ideal of precision medicine does not leave much space for negative results. The reality is probably a combination of the two. Not all studies performed are even prepared as manuscripts to be submitted to a scientific journal, and not all results from a particular study are included in the final manuscript. Further, the submitted manuscript could be rejected in the peer-review process. It is likely that publication bias occurs in all these steps, and for every step, the likelihood of proceeding to the next step of this process is higher for positive results.

The publishing process is highly competitive, and to publish in a high-quality journal you are required to have good quality data, but is it enough to have high quality research or do you also have to have the ‘right’ results? For example, one of the criteria of publishing in the journal Nature is that the papers “are of outstanding scientific importance” [https://www.nature.com/nature/for-authors/editorial-criteria-and-processes]. More proof is also generally needed to go against the established knowledge than to publish something that already has great support in the literature. Therefore, misleading knowledge or false positive results could remain ‘common knowledge’, especially in fields such as cancer research, that are based on a highly complex and heterogeneous biology.

Authorship bias involves misattribution in publications and could indirectly be related to publication bias. It is not unusual to add individuals who contribute nothing to the research effort research papers or grant proposals. In some cases, editors pressurize authors to add citations that are not relevant to their work. Adding highly recognized author names to manuscripts has become a common practice. Junior academics are more likely to add individuals in positions of authority or mentors to papers. A study showed that 60% added an individual because they thought the added scholar’s reputation increased their chances of a positive review (Fong and Wilhite 2017). This type of bias will thus have a lot of the same implications as publication bias if adding a name to the paper increases the chances of making it through the peer-review process and adds to the number of citations after publication, since the paper is not considered solely by the quality of the research but also by the names on the author list.

In some cases, researchers might feel that they are forced to biasedly select their data in order to get the data published. This could for example be selection of results that fits a logical flow of events, or they can select only successful replicates of an experiment, redo the statistics, include only selected subgroups in the analyses to get significant results and so on (Fig. 2). Other types of selection bias introduced by the researcher might be selecting only the findings that are statistically significant or fit the hypothesis. A more crucially dangerous selection is to intentionally exclude replicates without any logical reason other than to present the replicates of the experiment that was expected or ‘successful’. It is not practical to publish all research that has ever been performed, and therefore some degree of selection is required. Many journals also have strict word count limitations, leaving no room for all results in the manuscript, and the scientist is therefore forced to prioritise the most ‘important’ results. It is therefore not evident exactly where to draw the line between what is an acceptable selection of data or if certain types of data selection could be considered as data falsification and thereby fall under the definition of scientific misconduct, and there have even been some historical cases reported when researchers have gone so far as to intentionally fabricate their results (Else 2019; Stebbing and Sanders 2018; Müller et al. 2014; “Beautification and fraud” 2006; Fanelli 2009). Pressure to increase the number of publications coupled with the increased difficulty of publishing, can motivate academics to violate research norms even though majority of academics disapprove of this, others suggest that it is just the way the game is played. Examples include falsifying data, falsifying results, opportunistically interpreting statistics, and fake peer-review. A study reported that 1.97% academics admit to falsifying data, although this is likely understated (Fong and Wilhite 2017).

Fig. 2
The figure represents the mindset of researchers. It depicts the conversation between the two researchers discussion about the experiment results.

Example of how the mindset of researchers can contribute to publication bias. (The figure was created by the authors with BioRender.com)

Full size image

Shields reported in 2000 that one of the most typical factors influencing publication bias is investigators who do not submit their research for publication due to a lack of enthusiasm and the consequential drive to publish only the statistically significant studies, or the educated assumption that null outcomes are given low publication priority. He speculates whether the publication of null studies is more commonly driven by junior investigators who must publish to become known, or busier senior investigators who are less intrigued by null findings. Most importantly, he concludes that also journals contribute to publication bias when they refuse to publish null studies (Shields 2000).

In order to reduce publication bias, some journals like Cancer Epidemiology, Biomarkers and Prevention have begun to publish null results in specified formats where the articles are brief enough to encourage researchers to submit their findings but also sufficiently robust to ensure that the strengths and limitations of the study are discussed in light of other studies in the field (Shields 2000). One attempt to reduce publication bias is to have separate journals that specialise in publishing null results that only base their peer-review process on the quality of the research and have no requirements for the outcome of the study. Another alternative could be to have requirements for scientific journals to report a balance of significant and insignificant findings. A third strategy could be not to make the problem of publication bias in peer-reviewed journals disappear, but rather by minimising its impact by making research results available elsewhere, for example through publicly available databases or archives such as bioarchive (BioRxiv.org) and medarchive (MedRxiv.org). The limitation of such archives is that there is no control over what is published since the manuscripts are uploaded without going through a peer-review process, and it will therefore be up to the reader to evaluate the quality of the research. This has strong limitations as it is possible to cite or refer to such articles and the plausibility of unchecked citations can easily become overlooked.

Registration of clinical trials prior to the results is another attempt to reduce publication bias and ensure that the results become publicly available regardless of the results. Registries of clinical trials have therefore been created to increase transparency and reproducibility, and these registries have also been used to study publication bias and its impact on meta-analysis (Marks-Anglin and Chen 2020). Multiple funding agencies have therefore encouraged or made it mandatory with trial registration including the US Food and Drug Administration (FDA) (FDA n.d.), the International Committee of Medical Journal Editors (ICMJE) (De Angelis et al. 2004), and the National Institutes of Health (NIH) (Zarin et al. 2016).

Clinical trial registration has been implemented to reduce publication bias, but this only partly solves the problem and is not sufficient to eliminate publication bias completely. While these registries have relatively good coverage today, this was not the case previously and only 20 years from now the grey literature was barely available (Marks-Anglin and Chen 2020). Meta-analyses tend to cover studies spanning decades of work and there will still be a bias in these meta-analyses although recent results from unpublished trials are included in the analysis. In addition, despite the registries there can still be publication bias within which results that are published or otherwise reported from the trials. In the case of cancer biomarker studies, these will often not be directly included as a part of the clinical trial design. Biomarker studies could for example be retrospective studies investigating potential biomarkers of clinical trials already performed. Indeed, biomarker studies are often either retrospective studies of clinical trials or even pre-clinical in vitro or animal studies. When a biomarker study is investigating many potential biomarkers, it is likely to think that the candidates that show significant results are much more likely to be included in the reports/publications than those that did not show significant differences in patient outcome.

In this section, we have seen that to address publication bias, we need to go to the root of the problem and question the mindset of both the researchers, journals/editors and the general community perception, and abolish the stigma that null results are less meaningful than positive results. The impression that there is a direct relationship between statistical significance and scientific importance is not always true. It can thus be argued that popularisation of reporting P values starting in the early twentieth century has led to an overuse of statistical testing (Marks-Anglin and Chen 2020). The competition amongst researchers that are valued based on their number of publications, citations, and impact factors, combined with a constant race for funding or extended contracts, is a system which creates a risk of favouring or ‘selecting’ the researchers that are most biased in their presentation of results to continue and even propel their careers as opposed to those who are more open minded and honest about their research results. More focus on the problem of publication bias, increased awareness and methods developed to understand and address the problem, and to study the extent of publication bias is important in this context.

Conclusion

Publication bias within the field of precision oncology and cancer biomarker research, meaning that positive or significant results are more likely to get published than negative results, have many possible implications for researchers, patients and the general society. Over time, publication bias skews the scientific literature in favour of positive results which thus influence the design of new research projects and contributes to the reproducibilitycrisis that questions the validity of the scientific literature. In the field of oncology, this ultimately affects the treatment of cancer patients as clinical decision making rely on the scientific literature. The issue of publication bias seems to be even more evident for precision oncology and biomarker research, as aiming for perfection will leave less space for ‘negative’ results than in medical research in general. In addition, the complexity of precision oncology research that is based on a highly complex and heterogenous tumour biology will also be likely to generate more variations in the research outcomes which makes room for a more biased selection of results. Indeed, although biomarker and precision oncology research has received significant financial support recent years, still only a few biomarkers end up in the clinic, and even for the most successful biomarkers there are still challenges with defining biomarker cut-offs and deciding how different biomarker subgroups should be defined and treated.

Publication bias could be a consequence of either researchers deciding not to submit ‘negative’ results or the journals rejecting manuscripts where the results are ‘negative’, and it is likely that publication bias occurs at both these levels. Multiple actions have been suggested to reduce publication bias including clinical trial registration, forcing journals to report a balance between positive and negative results or make research results available elsewhere than in peer-reviewed journals. However, to address publication bias, we need to go to the root of the problem and convince researchers, journals/editors and the general community that negative results could be at least as important as positive results. Increased awareness about publication bias and methods developed to understand and address the problem, and to study the extent of publication bias is important in this context.