Appeals to evidence for the resolution of wicked problems: the origins and mechanisms of evidentiary bias

Technical bias

There is, of course, a rich literature mapping out a variety of ways that research evidence can inform policy making [c.f. the classic work of Weiss (1979) identifying seven meanings of ‘research utilization’ or more recent discussion on evidence use by Nutley et al. (2007)]. While some argue that modern EBP advocates too-narrowly focus on particular forms of research use (such as Weiss’ ‘knowledge driven’ or ‘problem-solving’ models) in their assumption that evidence can solve policy problems, there are still some valid concerns about evidence manipulation voiced by EBP advocates. Pieces of evidence can indeed be interpreted in ways that are more or less faithful to their methods and results, and bodies of evidence can be reviewed more or less comprehensively. Deliberate manipulation of findings is a particular problem, but also problematic for informing policy are cases of selective cherry-picking of evidence, as well as interpretations of research findings that are not valid with respect to the methodology or data of the study. All of these examples provide cases of technical bias, which can be defined from a perspective of scientific best practice. Yet technical bias can take many forms. Indeed, empirical examples illustrate that such bias can occur in either the creation, selection or the interpretation of evidence.

Technical bias in the creation of evidence

At times evidence is biased simply because research is conducted in ways that undermine good scientific practice. Some of the most obvious examples of biased evidence creation can be seen where corporate industry actors have undertaken their own research studies designed to provide favorable results. Investigations have repeatedly found manipulation by the tobacco industry, for example, to produce research findings showing smoking was less harmful than actually the case (c.f.: Wertz et al. 2011; Bero 2005; Cummings et al. 2007; Tong and Glantz 2007). Goldacre has similarly argued that the pharmaceutical industry is biased in its generation of evidence on product effectiveness, claiming “[d]rugs are tested by the people who manufacture them, in poorly designed trials, on hopelessly small numbers of weird, unrepresentative patients, and analyzed using techniques which are flawed by design, in such a way that they exaggerate the benefits of treatments…” (Goldacre 2014, p. xi). One study by Fries and Krishnan (2004) analyzed randomized trials of rheumatology drug trials, and concluded that ‘design bias’ led to an incredibly high proportion of trials (100 % of trails reviewed in their sample) showing positive results when that study had industry funding. Of course it is not only corporations pursuing monetary profit create evidence in biased ways. The growth in so called ‘creation-research’ (research specifically conducted to illustrate a divinely-created ‘young earth’) provides an example of how a particular religious belief has led to a body of research described as fundamentally flawed (technically biased) due to its starting premise to prove biblical literalism (Pigliucci 2002).

Technical bias in the selection of evidence

Technical bias can similarly occur in the selection of evidence, evidence is cherry picked so as to only highlight the pieces of evidence which support a desired outcome. Schön and Rein’s quote above noted the ability to ‘focus on different facts’, and there are numerous examples of where data can be strategically selected in policy debates. The supposedly ‘super wicked’ example of climate change, for instance, shows no shortage of accusations of strategic selection of data, no doubt contributing to intractability at times. Indeed, accusations of one specific form of cherry picking—strategic use of start and end years to show trends—have been made by climate change acceptors against skeptics (Institute of Climate Studies 2013), by skeptics against acceptors (Whitehouse 2014), or even by climate change acceptors against other acceptors as well [c.f. criticisms of the Obama administration’s use of data to show success at reducing emissions (Kessler 2014; Klare 2014)].

Technical bias in the interpretation of evidence

Finally, Schön and Rein’s quote also noted that facts can be interpreted in different ways—yet it is clear that interpretations of evidence can be technically biased where invalid conclusions are drawn from a particular body of (presumably valid or robust) evidence. There appears to be continual need for reminders that ‘correlation does not equate to causality’, or that an ‘absence of evidence does not mean evidence of absence’, such that these biased interpretations are so common. Misunderstanding risk statistics can provide another common case for biased interpretations—such as in the confusion of relative risk and absolute risk (Baron 1997; Barratt et al. 2004); e.g. a reduction in risk of 10 % does not equate to a 90 % chance of something occurring, yet it might be interpreted as such. One example of this was recently illustrated by Parkhurst et al. (2015) in a study looking at the use of evidence within the political debate over male circumcision for HIV prevention in Malawi. While clinical trials had measured a reduced (relative) risk of HIV infection in circumcised men of around 60 % (Mills et al. 2008), a local politician rejected this on the (inaccurate) grounds that there was still a 40 % (absolute) chance of infection in circumcised men. This interpretation appeared to support a political position, but provides a case of technical bias as the conclusion about absolute risk was simply invalid.

Issue bias

In contrast to these forms of technical bias, which are principally concerned with internal validity issues, issue bias instead reflects cases where the choice of evidence can bias a policy decision. This distinction is particularly important to address those who see evidence as playing a value-neutral role, or who argue that particular forms of evidence (such as RCTs) should be prioritized in political debates or on policy agendas—a logic that conflates intervention certainty with outcome desirability. As such, even when technically valid, issue bias can still occur in both in the creation and selection of bodies of evidence.

Issue bias in the creation of evidence

In terms of the evidence creation, issue bias arises in the decision of which outcomes to measure or evaluate. For years in the field of health, for instance, sociologists and policy analysts have identified how the choice of social categories or concepts to include in research (such as race, class, socio-economic status, etc.) will have implications for whether or how these issues are addressed in public health policies (Schwartz and Kart 1978; Krieger 1992; Navarro 2009). In addition, individual evaluation studies of social programs will also need to actively choose which social outcomes are measured in order to judge success—a choice that can shape the potential policy implications of the evaluation. A choice of relevant outcomes will always need to be done by some agent, yet the ways this choice can bias policy implications can be particularly obscured if evidence is assumed to be value neutral, or only judged based on its methodological rigor.

So, for example, gun control has been a persistently intractable problem in the United States for decades (Bruce-Briggs 1976; Kleck 1986), yet an appeal to evidence to provide a solution can be fraught with difficulty due to the multiple possible ways to evaluate the effects of gun laws. As Kleck (1986) has explained, research can look at a number of possible outcomes, such as assault-instigating effects, crime-facilitating effects, or assault-intensifying effects. Policy evaluations can also choose to evaluate either the effects of restrictions placed on the general public, or on specific ‘high risk’ individuals. There are a host of other outcomes of wider gun ownership that might also be included in an evaluation—from accidental deaths, to children’s tendencies towards violence, to citizens’ use of guns to resist violent crime or to privately apprehend felons—the choices between which will define the supposed ‘impact’ of any gun regulations. If one chooses to evaluate how criminals behave in relation to armed citizens (c.f.: Southwick 2000; Kleck 1986), this can lead to policy conclusions supporting gun liberalization, while evaluating death risks linked to widespread availability of guns (c.f.: Leenaars and Lester 1997; Burger 2002) typically leads to conclusions supporting greater control. As such the choice of what to measure can bias evaluation findings towards one or another political position.

Issue bias in the selection of evidence

A second form of issue bias involves the choices of which bodies of evidence are utilized to inform a policy debate or decision. While EBP advocates believe evidence can serve a rationalizing role, when there are multiple impacts of a policy, and multiple social goals at stake (as is typical for many wicked problems), claims that a policy solution is ‘evidence based’ can be equally made by opposing sides simply by selecting evidence of different social concerns. So for example, with regard to an infrastructure project there may be an evidence base about economic growth from similar projects, as well as evidence base about environmental harms. Supporters or opponents can each point to a body of evidence on which to justify their position. Again, this is not inherently a problem when considering that political debate rests on value differences, but it can lead to intractability in cases where opposed sides continue to appeal to ‘evidence based policy’ in a failure to recognize the underlying value differences.

A particularly relevant form of issue bias therefore can be seen in the contemporary EBP movement’s demands for policies to be guided by RCTs or by the application of so-called hierarchies of evidence (which place RCTs, or meta-analyses of multiple trials, at the top). It has been noted elsewhere that hierarchies of evidence were designed to judge methods used to demonstrate intervention effect, yet policy decisions typically involve a range of factors, the effectiveness of an intervention merely being one consideration (Petticrew and Roberts 2003). The prioritization of particular research methods (e.g. RCTs)—can therefore skew political attention to those issues conducive to experimental evaluation, or those strategies for which evidence has already been created (Barnes and Parkhurst 2014).

In health care, for instance, drug treatments are almost always evaluated through experimental designs. Yet there is an important question of whether treatment—and treatment through drugs—is necessarily the most important policy concern simply because it has the most robust evidence of intervention effect. Indeed, by 2001, the drug Sildenafil (marketed as Viagra) had been subject to no less than 16 randomized trials, every single one of which showed effectiveness over placebo for treating erectile dysfunction (Burls et al. 2001)—yet this overwhelming evidence of intervention effect clearly does not imply that provision of Viagra should be a health policy priority.

Instead, formalizing rules and norms that prioritize RCT evidence or impose hierarchies to guide political decisions can be seen to reflect Bachrach and Baratz’s (1970) idea of political power imposed through the mobilization of bias, described as: “[a] set of predominant values, beliefs, rituals, an institutional procedures (‘rules of the game’) that operate systematically and consistently to the benefit of certain persons and groups at the expense of others.”(p. 43). This is why a distinction between technical bias and issue bias is particularly important. While political manipulation or cherry picking of evidence can indeed be a problem from a technical perspective (one of internal validity), removing political considerations from policy decisions represents a form of depoliticization that many find undemocratic (c.f.: Young 2011). Indeed, as Wesselink et al. (2014) has explained, “[o]vert deference to EBP does not remove the need for political reasoning; rather politics is introduced ‘through the back door’ through debates on what is valid evidence rather than on what values should prevail” (p. 341).

Bias and intractability

Both technical and issue bias can contribute to issue intractability by undermining the ability for evidence to serve as a source of agreement and policy rationalization. Creation of technically biased pieces of evidence give competing advocacy groups different pieces of conflicting evidence to justify their views or positions, undermining scientific consensus or increasing uncertainty. Biased selection and interpretation of evidence can similarly perpetuate disagreements over what the evidence actually says or means. Intractability can also arise from issue bias, however, whereby value differences between competing policy actors lead to creation or selection of evidence related to one set of goals to the exclusion of others, with appeals to ‘evidence based policy’ serving to obscure those essential value differences. This permits multiple actors to take ‘evidence based’ positions without policy resolution. The increasing embrace of policies following hierarchies of evidence or RCTs further risks obscuring the political origins of contestation, leading to frustration amongst policy actors who may not realize their use of, otherwise technically valid, evidence has political implications from a perspective of issue bias.

Given how such biases leads to intractability, addressing these forms of intractability will not, necessarily, come from increased knowledge transfer or greater calls for evidence use. Rather, this paper argues that a deeper understanding of the political origins of these forms of bias is required to understand how their mechanisms of effect and where their impact may be mitigated.

Interests and contestation: overt politics of evidence

It has been argued that current calls for EBP fail to recognize that policy making deals with competing interests and multiple social goals. There appears to be an underlying assumption in much of the EBP movement that fidelity to evidence and scientific honesty will be valued in their own right—seen, for instance, in the continued surprise when evidence is misused, as well as the enormous efforts placed on efforts to ‘translate’ or ‘transfer’ research into policy—typically efforts that do not question whether anyone would not wish to use evidence, and which assumes that non-use of evidence must be because the evidence is either not known, or not understood (Rutter et al. 2013; Oliver et al. 2014; Caplan 1979).

Yet the understanding of policy making as dealing with the choice between contested sets of social values and goals has been a conceptual starting point of the policy sciences for over half a century. Easton 1971 (1953), for example, famously described political science as the study of the “authoritative allocation of values for a society” (p. 129). Similarly Brecht (1959) argued that politics is fundamentally about debates over what a good society looks like, a question that science cannot answer. Conceptualizing political debates as a process of competition to achieve policy results or achieve a desired societal outcome, therefore, leads to a very different perspective on how evidence can be expected to be used. There is no reason to assume that any competing actor or coalition will weaken their position for the sake of scientific accuracy. Indeed, when policy decisions can determine the political or financial survival of involved actors, we would expect groups to sacrifice scientific accuracy if doing so provides a competitive advantage.

From this perspective, many forms of evidentiary bias become increasingly predictable. The biased creation of evidence by industry actors noted above is a case in point. Corporations compete in the private sector to pursue profit. As such, we should expect private corporations to produce biased evidence in their interests if it might increase their competitive advantage. This is because any corporate actor who does not do so risks go out of business, replaced by an otherwise equal competitor willing to manipulate evidence to compete. The tobacco industry can again be looked to for a wealth of examples of evidence strategies aimed at achieving a desired policy outcome. One recent example is in how the industry has argued against policies requiring so called ‘standardised packaging’ (or ‘plain packaging’). Often the opposition to such policies has been highly visible—such as when Japan Tobacco International (JTI) placed full-page advertisements in British newspapers in 2013 (when the UK was considering this policy), claiming that the proposal was not ‘evidence-based policy’ but rather ‘policy-based evidence.’ [these advertisements were subsequently banned by advertising regulators for making false claims (Thomas 2013)].² Yet according to one analysis of the industry’s tactics in this policy debate, tobacco companies pursued an evidence strategy that insisted on unrealistic methodological perfection of evaluations conducted, that misquoted published studies, and that withheld evidence counter to their policy goals (Ulucanlar et al. 2014).

This inherent pressure to bias caused by financial interest is, of course, why conflict of interest statements matter, and why most scientific journals (and, indeed, many political bodies) require them. There can even be situations, however, where entrenched interests are so often challenged by research findings that a ‘rational’ political strategy can be to undermine faith in the whole of scientific practice in the first place—with Oreskes and Conway (2011) exploring how corporations have constructed doubt or undermined the credibility of the research process itself in areas such as tobacco harms or environmental issues.

It is not only corporate actors that may be involved in such strategies, however. Political competition faced by those in positions of authority will similarly provide pressure to use data in biased ways due to the need for a competitive advantage to gain, or remain in, power. In a pluralistic society, groups will compete for public support, and in individual elections, politicians will equally compete for survival through convincing enough of the public to vote for them. Unless scientific accuracy is a particularly important social value required to obtain citizen support (i.e. demanded by the populace), the nature of political competition will lead to scientific rigor taking a back seat to the more important goal of winning or surviving in political debates—which can manifest in technical bias. Such realities can help to explain examples of hiding or obscuring unwanted evidence by those in political power. In 2015, for instance, accusations of such practices were leveled against a number of national governments including: the Indian government for failing to release health survey data results that showed poor results in the Prime Minister’s home state of Gujerat (BBC News 2015); the UK government for failing to release numbers of people who have died within six weeks of having their government benefits stopped (Brown 2015); and the Chinese government for failing to report urban pollution data (Chen 2015). Each of these cases can be seen as a case of technical bias, as it represents a violation of scientific best practices of making all data available in a systematic and rigorous manner, yet presumably the risk of being accused of obfuscation was judged to be less politically harmful than the implications of making the data widely available.

Even in cases where technical accuracy is highly valued, and technical bias would undermine a political position, however, issue bias can still result from competition and contestation, as groups will focus on the relevant evidence (often in technically valid ways) that frames debates around their issues of interest. The competition between interests and ideologies therefore provide the incentives to undertake, accept, or maintain many forms of evidentiary bias detailed in the previous section. Indeed, these political origins of bias can be said to represent an overt politics of evidence as they arise from often deliberate strategies between groups that are openly competing in political spaces.

Cognitive shortcuts and motivated reasoning: subtle politics of evidence

In addition to bias arising from overt political competition between groups, there can also be biased uses of evidence that have less obvious origins. A more subtle politics of evidence can arise in particular when individuals are unaware how their value systems, or their group identities, bias their understandings and interpretations of evidence. Researchers in the field of cognitive psychology have particularly explored how heuristics (simplifying processes) can lead to biases in understanding information—typically defined as departures from ‘rational’ or purely analytical decision processes (often seen as incorrect conclusions from what evidence actually shows, or violations of accurate understandings of probability) (Gilovich and Griffin 2002). Such biases are often seen to manifest from decisions made under uncertainty, or when time and other pressures lead to intuitive judgments instead of more deliberate considerations of evidence (Gilovich and Griffin 2002; Kahneman 2011; Kahneman and Tversky 1974).

Some of the most common and important heuristics affecting affect how scientific evidence include the availability heuristic (drawing conclusions aided by utilizing the memory of similar cases) (Gilovich et al. 2002; Tversky and Kahneman 1973), affect heuristic (judgements influenced by existing affective feelings)(Finucane et al. 2000), and representativeness heuristic (conclusions derived from perception of similarity between a given situation and a prototypical one) (Gilovich and Griffin 2002; Kahneman and Tversky 1974). Common biases resulting from such processes include: Confirmation bias—Defined as: “the seeking or interpreting of evidence in ways that are partial to existing beliefs, expectations, or a hypothesis in hand”(Nickerson 1998 p. 175); illusory correlation (incorrect assumption of correlation) (Sternberg 1996); and attribute substitution, explained by Kahneman and Frederick (2002) as: “when confronted with a difficult question people often answer an easier one instead, usually without being aware of the substitution.”(p. 53). Finally, the concept of motivated reasoning has further been developed to explore cases of self-serving biases in which information, evidence, or arguments are understood in ways conducive to an individual’s desires, needs, or goals (Kunda 1990; Hahn and Harris 2014; Kahan 2011; Richey 2012).

There has been only limited application of these insights to the understanding of evidence in political settings, however, with much of the research in the field consisting of experiments in political psychology conducted in the US context. In one such study, for example, preference for a preferred political candidate was evaluated after providing negative information about the individual. Notably, Redlawsk et al. (2010) found that “voters may become even more positive about a candidate they like after learning something negative about that candidate.”(p. 564—emphasis in original).

Motivated reasoning research has also looked at how existing party affiliation affects interpretation of factual data about polarized issues (c.f.: Taber and Lodge 2006; Kahan et al. 2013), or perceptions of risk and of expertise—e.g. over climate change (Kahan 2014). Such research has even shown cases where greater scientific knowledge or numeracy could be correlated with increased bias in the interpretation of data. Kahan et al. (2013) for example, compared the incidence of invalid conclusions drawn from a set of data when the same information was presented as representing either the effect of a skin rash treatment, or the effect of a gun control policy. While the authors expected to see bias increase when data were framed to be about gun control, they further found that more numerate individuals (who tested higher on mathematical skills) were more likely interpret gun control data in biased ways.

Finally, Kahan (2013) also has reported on an experiment illustrating that individuals with greater cognitive reflection (measured in terms of ability to correctly answer mathematical questions which have an intuitive incorrect answer) had a greater propensity for motivated reasoning in an experimental setting. The author concludes that motivated reasoning may not necessarily be due to an over-reliance on heuristic (or intuitive) reasoning in all cases. In this case, he hypothesizes that “ideologically motivated cognition [is] a form of information processing that promotes individuals’ interest in forming and maintaining beliefs that signify their loyalty to important affinity groups”(p. 407).

These concepts help to understand the more subtle politics of evidence that explains additional elements of technical and issue bias beyond those which arise from explicit policy contestation. Individuals and groups with contrasting belief systems interpret and use evidence differently—not just because they are actively pursuing particular policy results, but because of how existing beliefs, motivations, and values structure the cognitive processes through which evidence is understood and applied. These theories can further help explain the source of the issue bias related to the prioritization of hierarchies of evidence or RCTs, which appears to be a form of attribute substitution—substituting the difficult questions of what to do to make society better with more straightforward questions of what interventions produce an effect.

While most of the political psychological research conducted on motivated reasoning has been done on populations of voters in the United States, we can also see the influence of cognitive in international expert bodies as well. The World Bank, for instance, dedicated a section of its 2015 world development report to exploring how its own expert staff were subject to cognitive biases which could lead to erroneous interpretations of policy-relevant evidence. These biases at times derived from existing ideological views of staff members; for instance finding that that pre-existing views on the importance of income inequality led to errors in the interpretation of hypothetical minimum wage data. Ultimately the World Bank concluded that its staff were subject to a range of biases that could undermine development planning, including confirmation bias, sunk cost, bias, and inaccurate predictions of the beliefs of the poor people they are meant to be serving with their work (World Bank 2015).

Complexity

One of the most common terms used in discussion of wicked problems is that of complexity. The term has been taken to mean many things, but a useful distinction comes from the field of complexity theory which often distinguishes between what are considered complicated as opposed to complex problems (Snyder 2013; Snowden and Boone 2007). In complicated problems there are many interacting or mutually interdependent elements, but each one may be solvable or knowable. A common example presented is sending a rocket to the moon (c.f. Glouberman and Zimmerman 2002): rocket science is difficult for many to understand, and reliant on many interacting pieces of expert information, but essentially predictable in most cases. Complex issues, on the other hand, are defined as having elements of uncertainty in causal relationships (many of which can only be identified after the fact) that prohibit prediction of similar outcomes if the same actions are taken in subsequent iterations. Raising a child is often noted as an example, as would be many experiences of social change (Kurtz and Snowden 2003; Glouberman and Zimmerman 2002).

In this conceptualization, complicated processes are typified by difficulty, while complex problems are typified by uncertainty. For many scholars, complicated problems are not necessarily wicked, as they can be ultimately solvable. Yet features of complicated problems can manifest in evidentiary bias, often in unconscious ways. Understanding the origins of bias for both complicated and for complex problems is, therefore, essential to address problem intractability. The distinction is particularly important given the origins of bias discussed above and the use of these academic fields to respond to bias. Cognitive psychology has identified different heuristics or mechanisms humans utilize in each of these cases. Kahneman (and colleagues) have famously described two forms of thinking—thinking ‘fast’ (intuitively) and ‘slow’ (deliberately and analytically)—noting the human propensity to think fast when tasks are complicated or difficult. Intuitive thinking can lead to biases such as attribute substitution and representative bias, even when, in theory, solutions are knowable (Kahneman 2011; Kahneman and Frederick 2002; Kahneman and Tversky 1974). When faced with uncertainty, thinking ‘fast’ can exacerbate different problems such as affective bias (Finucane et al. 2000), yet thinking ‘slow’ does not eliminate all uncertainty, meaning that other heuristics that deal with situations of partial information can still manifest in bias (Kahneman and Tversky 1974).

Complicated public policy problems furthermore typically involve multiple social outcomes of importance. Regardless of uncertainty in these cases, an increased number of relevant social concerns will increase the possibilities of issue bias, as there will be a greater number of evidence bases to draw on for each relevant concern. Complicated problems can also increase technical bias as multiple affective values will exist (with affective feelings towards each of the relevant concerns potentially leading to bias); and there may more opportunities for cognitive dissonance when values do not align between the multiple social outcomes at stake.

Contestation

A second source of bias arising from features of policy problems can be the level of contestation of an issue—in terms of how important that issue is to the groups in competition. As noted, fidelity to science (or scientific honesty) can be seen as one of the many social values that stakeholders in political arenas may possess. For policy debates with low levels of contestation (involving differences in opinions on values that are of marginal importance to interested parties), it would be unlikely that either side would sacrifice scientific credibility or violate the value for data accuracy to gain a political advantage. One would hardly expect groups to start manipulating research or cherry-picking evidence to influence policy decisions over which color to paint a new school, or to change the weekday that municipalities collect household waste, for instance. Yet when a policy debate revolves around an issue held to be of high importance, there is much a greater incentive for strategic technical bias. Corporate actors may see their company’s survival (or perhaps their profit line) as their most important interest. Hence a willingness to sacrifice scientific accuracy to produce evidence more likely to achieve desired policy outcomes. ‘Creation science’ has similarly no doubt grown from a recognition that mainstream scientific teaching poses an existential threat to the core religious beliefs of ‘young earth’ divine creation.

The level of importance to the individual will also influence cognitive processes, with issue importance part of what has been termed ‘attitude strength’ (Pomerantz et al. 1995). Studies of the affect heuristic, for instance, have shown that stronger attitude strength can increase biased information processing (c.f: Alhakami and Slovic 1994; Brannon et al. 2007). Cognitive dissonance theory similarly has illustrated a range of mechanisms used to avoid, dismiss, or ignore cases where values are inconsistent, and stronger value violations would be anticipated when evidence is brought to bear on highly contested debates. Indeed, Taber and Lodge (2006) found that stronger prior held beliefs on controversial subjects like gun control and affirmative action (in the US) led to increased biased reasoning, including reliance on strategies such as confirmation bias to avoid cognitive dissonance.

Polarization

The third and final component of the framework presented here is that of policy polarization. This can refer to how many viable options or positions there are for individuals (or policy makers) to take on an issue, but it can also reflect the polarization of the political landscape more broadly. In terms of polarized issues, at one extreme would be policy debates where there are a large number of intermediary positions. At the other would be policy debates where there appear to only be two diametrically opposed positions with no possibility for compromise. An example of a less-polarized issue might be market regulation. Extreme views no doubt exist, but the vast majority of people would fall along a spectrum of middle-range positions about which elements of markets should be regulated for public safety, or to address market failures. An example of a highly polarized issue might be that of abortion—which is often presented as morally wrong and unacceptable by its nature, or alternatively not a moral violation and as such a personal right for a mother. Other examples might be debates over the validity of including ‘intelligent design’ in a school science curriculum, or whether gay marriage should be permitted (or not). At times we see compromises made by politicians on these issues (offering ‘civil partnerships’ or abortion in particular circumstances), but in general the issues still are debated over a single binary division.

Policy environments may also be polarized as well, however. Kahan (2013), for instance, blames, in part, the polarization of American politics for what he sees as “ubiquity and ferocity of ideological conflicts over facts that turn on empirical evidence” (p. 407). Sunstein (2002) has explored group polarization and argued that individuals tend to move to more extreme positions when they deliberate with others sharing their own tendencies. This is said to come from “people’s desire to maintain their reputation and their self-conception” (p. 176). Kahan’s experiments in the American political context illustrate just how these pressures can lead to a biasing tendency in the interpretation of factual data, through what he terms ‘identity-protective cognition’—explaining “the goal of protecting one’s identity or standing in an affinity group that shares fundamental values can generate motivated cognition relating to policy-relevant facts” (p. 408). In highly polarized environments, then, the incentive for such identity-protective cognition to maintain group acceptance would be exacerbated. If we imagine individuals sharing political values, ideologies, and social interactions with a particular group, in a polarized society, an individual may risk losing their entire network and sense of community if they accept a valid (technically unbiased) evidence assessment which contradicts their group’s positions. The individual may have few shared values, ideologies, or social interactions with those on the other polarized position, and have a great deal to lose from group exclusion. In a non-polarized context, however, there may be several positions to adhere to, and multiple alternative groups in which to integrate.

Finally, polarization of the overall political landscape may also drive issue bias through the clustering of concerns that tend to split between divided groups. If valued social outcomes or concerns are highly clustered between opposed groups, we might expect that selection of issues over which to review evidence to more likely to come from within a single cluster, due to shared information and common thinking (and biases) within a the group.