The first and foremost characteristic of pharmaceutical R&D is its highly uncertain nature, manifested directly in the high failure rates of new drugs. Developing new products is challenging and entails a great deal of risk for most industries . Even so, the attrition rates of new pharmaceutical products are at a different order of magnitude compared even with the riskiest industries: for those drugs that successfully reach clinical testing (testing in humans) fewer than one in five (around 16%) will receive regulatory approval . These high attrition rates have been relatively stable across the past few decades, unlike the cost of conducting clinical trials, which has consistently increased .
The high attrition rates mean that pharmaceutical R&D teams are often susceptible to so-called progression-driven behavior where they focus their efforts on advancing a compound to the next stage without first ensuring that this is the right decision. However, in many cases, potential failures could be detected earlier. A recent study by Pfizer examined the decision-making processes behind 44 development programs . A striking finding was that all programs that had successfully completed clinical proof of concept (PoC) could provide clear evidence that proper testing procedures had been followed in previous stages. On the contrary, for almost half (43%) of those that failed at PoC, there was little evidence that all the necessary testing procedures had been followed. Had proper testing taken place, these failures would have been detected much earlier, saving the company millions of dollars. The need for superior clinical outcomes against the standard of care is even greater when a price premium is required to meet the cost of development. The existence of this progression-driven behavior is also highlighted in an extensive longitudinal study of AstraZeneca’s small-molecule pipeline that examined the root causes behind project failures . A possible explanation, suggested by the authors, is that “the use of volume-based metrics encouraged project teams and leadership groups to progress projects to the next phase in order to meet yearly goals”.
A second characteristic of the drug development process that further complicates collaboration and efficient decision-making is its long development cycles. Even if we exclude the discovery phase of a new compound, it still takes 6–7 years from filing of a new molecular entity to getting regulatory approval (Pharmaceutical Research and Manufacturers of America ). During these lengthy cycles, a number of factors can separate the actions of an individual or group from the end goal of the project: market launch. In addition, given the highly specialized nature of the tasks and activities involved, this process is highly fragmented: different groups in the company, or even different companies (e.g., contract research organizations), are involved at different stages. As a result, scientists and managers are often rewarded on the basis of interim goals . While such a reward scheme might seem understandable, given the lengthy development times, a by-product of these interim goals is that managers are often held accountable for reaching only a specific milestone, without bearing any overall responsibility for the subsequent clinical or market access success of the drug. As such, the different organizational units tend to operate in “silos” [17,18,19] that develop a “throw-over-the-wall” mentality at the hand-over points. This can be hugely inefficient if seemingly promising products that meet interim safety and efficacy goals lead to commercial failures if the end customer’s criteria are not taken into account early on and incorporated into the clinical development plan .
The high costs associated with the development process in the pharmaceutical industry can pose a third critical obstacle to collaboration across the different organizational units. Given the high cost of conducting clinical trials, which often run into the hundreds of millions , the different business units (e.g., therapeutic areas) within the company often have to compete for very scarce resources. To secure their budget and, therefore, their future development, business units might convey overly optimistic forecasts about the technical and commercial success of a given drug . Because of the highly specialized nature of those forecasts, it is not a trivial exercise for someone outside the business unit to challenge them or the methods used to reach them. Clearly, this is extremely inefficient, as those forecasts are bound to fail in later stages, after having consumed an enormous amount of resources.
Taken together, the three key characteristics of the pharmaceutical industry (high failure rates, long development times, and huge development costs) can create an environment in which there is misalignment in incentives between specific individuals or groups and the overall organization. When that happens, collaboration suffers and development teams focus on the short-term viability (or competence) of their narrowly defined “silo”, rather than contributing to the end goal: developing customer-focused medicines.
Among the most pervasive managerial behaviors is the practice of “throwing good money after bad”. As noted in Cooper : “projects get a life of their own and become like express trains, slowing down at the stations, but never with the intention of stopping until they reach their ultimate destination, market launch”. Royer  also analyses a number of product failures, highlighting that development teams kept going even though there were clear and consistent signs pointing to a near-certain failure. For example, during the development of a novel lens, both the regulatory authorities and opticians expressed clear concerns about its benefits. However, the development team chose to ignore them .
The seminal work of Staw  demonstrated that subjects were likely to continue making investments in failing projects, despite evidence of negative performance. Importantly, more money was invested in a project when the subjects themselves, rather than an outsider, were also responsible for earlier funding decisions. This self-justification effect has been replicated in a number of subsequent experiments [25, 26].
A second important bias that prevents efficient decision-making is the confirmatory bias. Here, it is not the past involvement of the decision-maker that biases their future actions. Instead, their current beliefs affect how they seek new information about the project. Specifically, cognitive studies have shown that people tend to seek information that confirms their existing beliefs [27, 28], and at the same time heavily discount information that contradicts those beliefs .
As discussed earlier, the R&D process in the pharmaceutical industry is a complex endeavor that requires highly specialized expertise. Moreover, developing ground-breaking science for unmet medical needs requires an extraordinary amount of perseverance and commitment to succeed. However, those very qualities can often lead to overconfidence : scientists become “true-believers” that their compound will be successful, despite all the evidence against it. This creates an emotional attachment that further amplifies self-justification and confirmatory bias. As a result, the scientists (or the specific development team) develop a “groupthink” mentality  where the team isolates itself from the rest of the organization and defends its product by being overly optimistic or by focusing on a narrow set of criteria that might not include the end customer’s needs.
Developing and marketing new products is a truly interdisciplinary process that involves a range of individuals from different organizational functions. Research has shown that employees’ beliefs and values, and hence their behaviors, are largely driven by the specific functions or groups they belong to. This is expressed in Dougherty : “Departmental thought worlds partition the information and insights. Each has a distinct system of meaning which colors its interpretation of the same information, selectively filters technology-market issues, and produces a qualitatively different understanding of product innovation”.
The existence of an insular culture in which people work closely with and learn only from their own group, while excluding those outside the group, is a major barrier to collaboration, and ultimately to product performance. The effect is clearly demonstrated in Hansen et al.  who studied 121 product development teams at Hewlett-Packard: the study identified certain teams that only sought solutions within their own team rather than reaching out to other divisions (even when the problem at hand required an interdisciplinary solution). Interestingly, the data also revealed that when teams did reach out to other divisions, they did not necessarily reach out to the ones that would have the highest expertise for their particular problem. People tend to approach those they know and have good relationships with  or those they feel most comfortable with .
Such organizational barriers to cooperation become even more critical in highly complex development environments such as the pharmaceutical industry. First, the specialized nature of the different processes and tasks creates a direct, and at first sight unavoidable, barrier to collaboration. Specifically, development teams in the pharmaceutical industry might consist of experts from target and drug discovery teams (from synthetic organic chemistry to molecular and cellular biology), clinical teams (from pharmacodynamics and toxicology to pharmacokinetics), and regulatory experts. At the same time, more commercially focused teams consist of experts spanning market access, marketing, medical affairs, finance, and corporate affairs, as well as business development teams that identify the most promising opportunities from the external environment. These individuals are experts in their own area, and possess knowledge and experience that cannot be easily codified or quantified. The organizational literature refers to such tacit knowledge as “sticky”: and transferring it across organizational functions is not a trivial task [35, 36]. Hence, an integrated decision-making process has to rely on the collaboration and effective participation of all teams.