Where is the Motivation in Motivated Numeracy?

Abstract

In a series of very influential papers, Dan Kahan argues for “the identity protective cognition thesis”: the claim that politically motivated reasoning is a major factor explaining current levels of polarization over matters of fact, especially in the US. An important part of his case consists of experimental data supporting the claim that ideological polarization is more extreme amongst more numerate individuals. In this paper, we take a close look at how precisely this “numeracy effect” is supposed to come about. Working with Kahan’s own notion of motivated reasoning, we reconstruct the mechanism that according to him produces the effect. Surprisingly, it turns out to involve plenty of motivation to reason, but no motivated reasoning. This undermines the support he takes the numeracy effect to provide for the identity protective cognition hypothesis.

That the more numerate or cognitively sophisticated also are “better” at motivated reasoning has lately become something of a commonplace. Science journalists, editorial writers, and the interested public, maybe unsurprisingly, revel in the seeming irony of such “numeracy effects”.¹ Describing Dan Kahan’s influential findings, New York Times podcaster Ezra Klein for instance tells us: “The smarter the person is, the dumber politics can make them” (Klein 2020; p. 92). Numeracy effects certainly are interesting in their own right, but become even more so when claimed to be a key element in the case for the hypothesis that motivated reasoning is the source of the fact polarization that seems to be “a signature feature of contemporary political life” (Kahan 2016a, 1).

Fact polarization is “intense, persistent partisan contestation over facts that admit of scientific evidence” (Kahan 2016b; p. 1). And it indeed seems to be the case – in the US and maybe the Western World in general – that growing numbers of people either resist or embrace compelling and widely accessible scientific evidence along partisan lines. Such polarization matters when it concerns policy-relevant facts such as vaccine safety, the relation between gun control and violent crime, or that between human activity and climate change. It is therefore paramount to understand how it comes about.

In a series of highly influential papers, Kahan has argued that fact polarization results from politically motivated reasoning. More precisely, he has argued for “the identity-protective cognition thesis” (ICT), the hypothesis that politically motivated reasoning is identity-protective reasoning, and that this is a major driver of fact polarization (cf. e.g. Kahan et al. 2012; Kahan 2013; 2015; 2016a, b; Kahan et al. 2017). Kahan’s case for ICT crucially depends on a numeracy effect – the effect is supposed to provide an essential piece of evidence supporting ICT over rival explanations.

Its being subject to cognitive sophistication effects should be intriguing to anyone interested in motivated reasoning, not least the philosopher. Understanding exactly how such effects are supposed to work is not easy, however, and there are quite different suggestions around. In this paper, we will try to make some progress here by offering a close reading of Kahan et al.’s work on “motivated numeracy”. In Section 1, we’ll spell out Kahan’s notion of politically motivated reasoning, his explanation of fact polarization in terms of it, and the role of the numeracy effect in the case for ICT. In Section 2, we are going to reconstruct Kahan et al.’s understanding of the mechanism generating the numeracy effect. Surprisingly, this mechanism – we’ll call it “Kahan’s mechanism” – turns out to involve motivation to reason, but no motivated reasoning. As far as we can see, this undermines the support Kahan et al. take the effect to provide for ICT. In Section 3, we conclude with some reflections on the wider significance our being right about the nature of Kahan’s mechanism would have – both for epistemology and for debates concerning the nature of fact polarization and “partisan cognition” (Williams 2023; p. 4) in general.²

1 Identity Protective Cognition and Numeracy

Kahan explicates motivated reasoning as “the tendency of individuals to unconsciously conform assessment of factual information to some goal collateral to assessing its truth” (Kahan 2016a, 2).³ The goal in question concerns the formation or maintenance of specific (kinds of) belief. In politically motivated reasoning, the truth-independent goal is identity protection (cf. ibid., 3), and it is achieved by a form of selectively “skewed” information processing: “individuals are adjusting the weight assigned evidence conditional on its identity congruence” (ibid., 10).⁴ Let’s call this the “IP-account” of politically motivated reasoning.

According to Kahan, identity protective reasoning drives fact polarization because certain factual claims have become symbols of membership in what he calls “identity defining affinity groups”. Examples would be climate change denial and covid vaccine skepticism; these seem to have become “identity markers” in certain conservative groups in the US. Relinquishing an identity marker can carry heavy social and even economic costs. Group members thus have powerful incentives to maintain belief in such markers – even in the face of strong evidence against them.

The threat such “identity-incongruent” evidence poses to one’s identity can be averted by engaging in politically motivated reasoning. To repeat, in such reasoning “individuals are adjusting the weight assigned to the evidence conditional on its identity congruence” (Kahan 2016a, 10). A threat from evidence for climate change, for instance, can be averted by (unconsciously) adjusting the evidential value of such information in a way allowing for maintaining climate denial. In general, identity-incongruent information thus can be resisted by processing it as if it had lower evidential value than it would be (epistemically) rational to assign to it.⁵

To test whether politically motivated reasoning on the IP-account explains fact polarization, Kahan has developed what he calls the “politically motivated reasoning paradigm (PMRP)” (cf. Kahan 2016a, b):

The best test of politically motivated reasoning is whether study subjects alter the weight they assign the same piece of evidence in response to an experimental manipulation of the perceived relationship between that evidence and positions that predominate in their cultural group. This experimental setup can be called the PMRP design (Kahan 2016b; p. 2).

In experiments of this design – “outcome switching” – subjects’ evaluation or endorsement of new information is used as outcome variable. Subjects are randomly assigned to receive one of two pieces of information. The substantive detail of the information is held constant across conditions. But its implication for subjects’ political identities is varied between conditions: In one condition, the information is identity-conguent, in the other it is incongruent. The key result is that subjects’ evaluation of the information differs by condition, and, in particular, that this difference is correlated with their political identities or preferences.⁶

In the quite famous “skin cream/gun control” study that will be central to our concerns (Kahan et al. 2017), for instance, the information consists of a table with numerical values. What varies is what these values are described as representing. In the two control conditions, they are supposed to represent the results of a scientific study concerning the effectiveness of a skin cream in treating a certain kind of rash. What is switched is the description of what the numbers represent: people whose rash got better vs. people whose rash got worse. Participants then are asked which of the following two claims are supported by the study: People who used the skin cream were more likely to get better/get worse than those who didn’t. In the experimental conditions, the very same numbers are described as results of a scientific study investigating the effects of banning the carrying of concealed handguns on violent crime. And again, outcomes are switched between conditions and participants are asked which of two claims were supported by the study: Cities that enacted a ban on carrying concealed handguns were more likely to have a decrease/increase in crime than cities without bans. The results were clear: As opposed to the control conditions, subjects’ evaluation of the information differed between experimental conditions in a way correlated with their political identities.

As we have described the results so far, however, Kahan et al. do not seem to find them to be quite enough to support ICT over what they appear to consider its main rival: “the science comprehension thesis (SCT)”:

ICT, in an important sense, stands SCT on its head. Whereas SCT attributes conflicts over decision-relevant science to deficits in science comprehension, ICT sees the public’s otherwise intact capacity to comprehend decision-relevant science as disabled by cultural and political conflict” (Kahan et al. 2017; p. 56).

Deciding between ICT and SCT clearly is of some practical importance; effectively counteracting fact polarization presumably takes quite different forms depending on which is the correct explanation. What is less clear is why Kahan et al. would consider both ICT and SCT to be compatible with the observed tendencies to evaluate information in ways correlated with political identity. Prima facie at least, political identity would not seem to play much of a role in the SCT framework, after all. We need to dig a little deeper here.

SCT attributes polarization over risks and other policy-relevant facts “to deficits in the public’s capacity to comprehend scientific evidence” (Kahan et al. 2017; p. 56). According to SCT, a weak understanding of science makes ordinary citizens liable to misunderstand what scientists say as well as vulnerable to being misled about it by interested parties. The effects of limited knowledge about science are aggravated by the heuristic-driven form of information processing (“system 1” reasoning, cf. Stanovich & West 2000; Kahneman 2003) thought to be typical of human risk perception:

Over-reliance on System 1 heuristics is the root of myriad cognitive biases. By fixing attention on emotionally gripping instances of harm, or by inducing selective attention to evidence that confirms rather than disappoints moral predispositions, System 1 information processing induces members of the public variously to overestimate some risks and underestimate others relative to the best available evidence, the proper evaluation of which requires exercise of more deliberate and reflective ‘System 2’ forms of information processing (Kahan et al. 2017; p. 56).⁷

As characterized by Kahan et al., SCT would indeed seem to predict difficulties with interpreting scientific evidence. But as these would seem to be difficulties with interpreting evidence quite generally, one might still wonder whether the theory has resources to explain difficulties differentially affecting the interpretation of identity-incongruent evidence.⁸

Kahan et al. do not explicitly address this question, but we find it quite plausible to think that identity-(in)congruence does have significance in the framework of SCT, if only indirectly so. What gets challenged in Kahan et al.’s experimental conditions are identity markers, i.e. prior beliefs possessing a certain social significance. According to Kahan et al., a challenge to such beliefs affects the interpretation of the challenging evidence in the specific way characteristic of motivated reasoning. SCT, it seems to us, at least suggests alternative, non-motivational ways in which such challenges might affect the interpretation of the evidence.⁹ After all, the relevant prior beliefs presumably are held quite strongly. When presented with tables of numerical values supposedly representing the results of a scientific study, subjects lacking in the ability to interpret such evidence might for instance just fall back on their strongly held prior beliefs. Moreover, as Kahan et al. point out, a lack in science comprehension is apt to make people more vulnerable to being misled into science-skeptical attitudes, thus becoming prone to disregarding scientific evidence in favor of their prior beliefs or “gut feelings”. And even subjects with some science literacy might fall prey to various biases favoring strongly held prior beliefs, biases like confirmation bias or “belief bias” (Tappin et al. 2020; p. 84). In sum, it does not seem implausible to think that both SCT and ICT are compatible with the results of the skin cream/gun control experiment as described so far.

It is here that the numeracy effect finally comes in: the skin cream/gun control study is designed to test a particular prediction distinguishing ICT from SCT. This prediction concerns the particular shape the data will be taking. According to Kahan et al., ICT predicts a numeracy effect; it predicts that “ideological polarization (…) should be most extreme among those highest in numeracy” (Kahan et al. 2017; p. 66). The idea is this:

[M]ore numerate individuals have a cognitive ability that lower-numeracy ones do not. ICT predicts that more numerate individuals will use that ability opportunistically in a manner geared to promoting their interests in forming and persisting in identity-protective beliefs (Kahan et al. 2017; p. 75).

By contrast, no such prediction comes with SCT. If anything, SCT would predict the opposite (cf. Kahan et al. 2017, p. 65f): If SCT were correct, more numerate subjects would get things right more often, regardless of whether the numbers are identity-congruent or incongruent. This would be because more numerate subjects in general will both be higher in science-literacy and more prone to System 2 reasoning.

The results of Kahan et al.’s study bear out ICT’s prediction; his data do show a numeracy effect.¹⁰ As noted above (fn. 1), the effect does not seem to replicate very well. We’ll get back to that towards the end of this paper. But for now, we shall not be concerned with the numeracy effect’s existence. What we would like to understand is how this effect is supposed to work. While the more numerate of course have cognitive abilities that the less numerate lack, we do not find it obvious that higher numeracy would generate “better” identity protection. What kind of “mechanism” is supposed to produce this effect?

We shall continue to focus on Kahan et al. (2017). In the next section, we shall carefully reconstruct their own interpretation of the skin cream/gun control experiment and its results. We are going to do this in order to gain a precise understanding of what we shall call “Kahan’s mechanism”, i.e. the mechanism that according to Kahan et al. explains the numeracy effect in their data. No doubt many other interesting questions can be raised, for instance about the interpretation of the skin cream/gun control study and its results. While we shall touch upon the significance of such questions towards the end of the paper, this paper is not about the correct interpretation of this study. Our main goal is modest: it is to understand Kahan’s mechanism. As already indicated, we think that this investigation leads to surprising and quite intriguing results.

2 Kahan’s Numeracy Mechanism

An intuitive take on numeracy or cognitive sophistication effects on motivated reasoning goes something like this: Those with higher reasoning capacities are able to selectively bring these capacities to bear on evidence for or against beliefs they cherish. For instance, they are better at finding fault with arguments or evidence against their views. They are also better at finding further arguments or evidence for their views. Their higher reasoning capacities thus will make them “better” at both undervaluing evidence against, and at overvaluing evidence for, their cherished beliefs.

This intuitive take is not uncommon among those researching these matters. Taber and Lodge, for instance, predict (and find) a “sophistication effect” described as follows:

[T]he politically knowledgeable, because they possess greater ammunition with which to counterargue incongruent facts, figures, and arguments, will be more susceptible to motivated bias than will unsophisticates (Taber & Lodge 2006, 757).

And Kahan himself comments on reasoning under identity threat:

Under these conditions, it is a perfectly rational thing for one to attend to information in a manner that promotes beliefs that express one’s identity correctly, regardless whether such beliefs are factually correct (…). And if one is really good at conscious, effortful information processing, then it pays to apply that reasoning proficiency to give information exactly this effect (Kahan 2016b; p. 4).

There is, however, something quite puzzling about this kind of idea. It can hardly be the case that more numerate individuals are better at consciously and deliberately engaging in lowering (or raising) the evidential value of a piece of evidence in order to maintain some cherished belief. Rather, such maintenance would seem to require that at least some of the reasoning remains hidden from the subject’s own view – quite irrespective of their numeracy levels.

In order to understand how this is exactly supposed to work, we are now going to take a careful look both at Kahan et al.’s skin cream/gun control study (Kahan et al. 2017) and at the details of the account of the numeracy effect provided in that paper. The design of the skin cream/gun control study study is quite sophisticated. The table shown to participants in one of the experimental conditions looks as follows (Fig. 1).

Fig. 1

Table from Kahan et. al. 2017

In the other experimental condition, the labels on the columns are switched (Fig. 2).

Fig. 2

Label switching

The numbers in this table are “difficult”, i.e. they are such that the correct answer isn’t obvious (to most of us). To figure it out, we need to calculate the percentage of cities in each category that experienced an increase/decrease in crime, which requires summing up the numbers in each row first. In Fig. 1, for instance, there are 298 cities that banned concealed handguns. Of these, 75 (roughly 25%) experienced a decrease in crime. And there were 128 cities that did not have a ban, of which 21 (roughly 16%) experienced such a decrease. In Fig. 1, the correct answer thus is that the study supports the conclusion that cities that enacted a ban on carrying concealed handguns were more likely to have a decrease in crime than cities without bans.

But the numbers are not only difficult, they are also designed to be “misleading”: They are such that either of the two heuristics that people tend to use when answering such questions will lead to the wrong answer. That is, if we skip the laborious system 2 reasoning just illustrated and go for what system 1 suggests, we will get it wrong. With respect to tables like these, what we shall call “the table heuristics” consist in either just comparing the numbers in the upper row, or in comparing the number in the upper left corner with that in the lower left corner (Fig. 3; cf. Kahan et al. 2017, 61f).

Fig. 3

Heuristics

Because of the table heuristics, these numbers “look” misleading to most people: As illustrated in Fig. 3, either one of them leads to the wrong conclusion that cities that banned handguns were more likely to see an increase in crime. In sum, arriving at the correct answer requires actually doing (or at least approximating) the calculation. The study thus is designed to test for a numeracy effect by using numbers that are both difficult and misleading.

Predictably, people on average don’t do extremely well in this task even in neutral conditions. More numerate subjects – also predictably – perform better in the neutral conditions. And they also perform better in the experimental condition congruent with their political identity. But in Kahan et al.’s study, they perform just as badly as everyone else in identity-incongruent conditions.

Kahan et al.’s explanation of this numeracy effect turns on both the misleadingness and the difficulty of the numbers. As far as we can tell, it is supposed to work as follows (cf. Kahan et al. 2017; p. 75): The initial use of a heuristic will make a condition that in fact is identity-congruent look identity-incongruent. This puts the subject under identity threat. But since the numbers are difficult, it seems possible to the subject that they in fact are not incongruent. They will therefore be motivated to check this by doing a proper calculation, and those higher in numeracy (more often) will get it right. The identity-incongruent condition, by contrast, will, by the heuristic, look congruent. Lacking any incentive to calculate, most subjects – whatever their numeracy score – will be content with the answer delivered by the heuristic. All of these subjects will get it wrong.

Kahan et al.’s explanation of the numeracy effect is quite intriguing, but it does raise the question: Where is the motivated reasoning in this mechanism? And as far as we can tell, there isn’t any.

To begin with, Kahan’s mechanism is rather unlike the intuitive idea we started this section with. According to Kahan et al., what the subjects with higher numeracy scores do when they feel under identity threat is not some sophisticated kind of counterarguing. They just calculate. The motivation to do so being the same for all subjects, the numeracy effect here really is nothing but the effect higher numeracy usually has when calculating: getting it right more often.

Moreover, when under identity threat, the subjects higher in numeracy calculate in precisely the same way they do in the neutral conditions. The motivation deriving from the perceived identity threat does not interfere with or “skew” their information processing. Nor does the motivation result in the signature effect of motivated reasoning as understood by Kahan: the selective (and epistemically irrational) adjusting of the evidential value of the information processed. After all, the numeracy effect comes about precisely because the more numerate subjects when feeling under threat get things right. As far as we can see, there is no motivated reasoning in this. What there is is motivation to reason. But that is a very different thing.

If what generates the numeracy effect indeed is an interaction between motivation to reason and numeracy, it is hard to see how the effect would support ICT over rival explanations such as SCT. This is just because it would not support ICT to begin with. To do so, the effect would have to be due to identity protective motivated reasoning, and so far, it does not look as if it were. If the effect is due to Kahan’s mechanism, what identity threat generates in those higher in numeracy is not motivated reasoning at all, but quite the opposite: correct calculations.¹¹

There is, however, an additional, somewhat overlooked aspect of the mechanism. So far, we have seen how it is supposed to explain why subjects calculate in incongruent looking conditions: they feel under identity threat. But this is only half the story, so to speak: we also need an explanation for why those higher in numeracy do not calculate in the experimental conditions where they don’t feel under identity threat. Kahan et al. themselves appear to think that no further explanation is required here: Since no identity threat is perceived, there is no motivation to calculate.

But this won’t do. Those higher in numeracy are assumed to be inherently more motivated to do the relevant calculations. As Kahan et al. explain, “the numeracy scale measures a disposition to subject intuition to critical interrogation in light of all available information, and thus to avoid mistakes characteristic of overreliance on heuristic, System 1 information processing” (Kahan et al. 2017; p. 64, with reference to Liberali et al. 2012). And as we saw, many of those higher in numeracy do the calculations in the neutral conditions of the skin cream/gun control study. That’s why they get it right more often in those conditions, after all. So, more numerate subjects are typically motivated to calculate not only when under identity threat, but also in conditions without any such threat. Neutral conditions clearly fall in the latter category, but so do experimental conditions in which no identity threat is perceived – prima facie at least. That more numerate subjects do not calculate in experimental conditions where no identity threat is perceived thus requires explanation.

At this point, it might even seem as if the more numerate subjects after all did have a particular ability beyond their numeracy: They appear to be able to “switch off” their numeracy in conditions where using it would not serve their identity protective needs. This is how for instance Neil Levy interprets Kahan et al.:

Kahan suggests that the greater polarization seen among more capable and informed participants is due to their greater capacity. This capacity gives them an ability less capable participants don’t possess: to clearly recognize how threatening the correct response is to their world-view, or identity. They are therefore motivated to selectively inhibit Type 2 cognition (Levy 2022; p. 31).

The problem is that this gets the experiment wrong, at least according to Kahan et al.’s own take on it: in the skin cream/gun control study, the motivation provided by (perceived) identity threat is not motivation to inhibit system 2, on the contrary, it is motivation to switch it on. While we indeed need an explanation for why high numeracy subjects “inhibit Type 2 cognition” in congruent looking conditions, recognized identity threat can’t be it. The conditions are designed precisely to keep the threat hidden. To recognize it, the calculation has to already have been carried out. In other words, to recognize the real, but hidden threat, the subjects would need to do precisely the opposite of what they appear to be doing: use, instead of inhibit, system 2.¹²

As Kahan et al. themselves appear to think that the absence of perceived threat suffices as an explanation here, we can only speculate as to what a better one might be. Presumably, it would have to do with political charge. In the experimental conditions, the topic is one on which the subjects presumably have a desire not to receive identity-incongruent information. They might thus refrain from calculating out of apprehension of receiving counterevidence.

If this were the explanation, it would be quite ironic. For again, the motivation provided here is the same for all subjects. The “ability” the less numerate lack would then be the ability to inhibit higher numeracy skills when apprehending counterevidence. But putting it like this is misleading. If apprehension of counterevidence provides motivation to not calculate, all subjects have that motivation. And all subjects have the ability to not even try to calculate.¹³

More importantly, if calculation is refrained from because of apprehension of counterevidence, the question returns: Where is the motivated reasoning in this mechanism? As far as we can tell, there isn’t any. The motivation provided by apprehension of counterevidence does not interfere with, or “skew”, any reasoning process. Rather, what takes place is the quite normal initial employment of a heuristics. What does not take place is further reasoning. There is motivation, but it does not result in motivated reasoning.¹⁴ It is motivation to refrain from reasoning, and again, that is a very different thing.¹⁵ So, no matter how we turn it, we cannot find any motivated reasoning in Kahan’s mechanism.

As we said above, if the effect is generated by an interaction between motivation to reason and numeracy, it is hard to see how it would support ICT over rival non-motivational explanations of fact polarization (such as SCT). This verdict does not change if the mechanism not only involves motivation to reason, but also motivation to refrain from reasoning. Either way, if it is Kahan’s mechanism that generates it, the effect is not due to motivated reasoning at all, and, consequently, its existence does not provide any (additional) support for ICT.¹⁶

3 Conclusion

According to Kahan, identity-protective reasoning is the main driver of the political fact polarization increasingly affecting political life (at least in the Western world). One crucial piece of evidence for the identity-protective cognition thesis (ICT) consists of experimental data supporting the claim that ideological polarization is more extreme amongst more numerate individuals. In this paper, we have taken a close look at how precisely this numeracy effect is supposed to come about. Identity-protective reasoning supposedly being a (political) kind of motivated reasoning, we have used Kahan’s own notion of motivated reasoning to carefully reconstruct the mechanism that according to him and his colleagues produces the effect. Surprisingly, it has turned out to involve plenty of motivation to reason (and, possibly, to refrain from reasoning), but no motivated reasoning.

In these concluding remarks, we would like to add some reflections on the wider significance of this result, both from the perspective of epistemology and from that of the debate concerning motivational and non-motivational explanations of fact polarization and partisan cognition more generally. Concerning fact polarization, Kahan himself clearly considers the results obtained by means of the PMPR paradigm, and the numeracy effect in particular, as a bit of a silver bullet disentangling the motivational from the non-motivational. But as we saw, if it is Kahan’s mechanism that generates it, the numeracy effect is not due to motivated reasoning at all. Consequently, its existence provides no (additional) support for either ICT or the motivated nature of partisan cognition in general.

We’ll return to this debate in a moment. But first, we would like to take a step back and consider some questions concerning rationality and reasoning, questions we have so far only alluded to. As we observed earlier, identity-protective reasoning might be rational in some practical sense, but it certainly is irrational in the epistemic sense. This does not mean, however, that non-motivational accounts of fact polarization automatically construe the partisan cognition involved as epistemically rational, far from it. The distinction between these two families of accounts rather turns on the question whether the (typically unconscious) reasoning (or belief formation process) in question is purely “cognitive” or not. In motivated reasoning it is not; here, the process is “skewed” – interfered with – by motivational states (desires), and thereby irrational. According to non-motivational accounts, by contrast, the relevant reasoning involves cognitive states only. Nevertheless, it might be subject to various biases or use heuristics making it less than rational.¹⁷ Whether motivational or non-motivational, accounts involving irrationality typically are such that the irrationality results from some kind of systematic tendency or mechanism.

Interestingly, Kahan’s mechanism is quite different. In a sense, it cuts across the motivational/non-motivational distinction: It does not involve motivational interference with belief formation, but it does involve motivational states. The motivation, however, is just your quite normal, practical motivation: it is motivation to act in a certain way (or to refrain from so acting). Moreover, the reasoning subjects are motivated to perform will be (fairly complicated) system 2 reasoning, which tends to be conscious and “explicit”. The mechanism generates a numeracy effect simply because less numerate subjects are more error-prone when engaging in such reasoning. Calculating mistakes do result in epistemically irrational beliefs, but this is not the kind of systematic tendency or mechanism supposed to do the explanatory work in “irrationality accounts” of partisan cognition.

Now, in what we shall call “calculating cases”, i.e. cases in which subjects have motivation to calculate, there is a clear sense in which doing so is practically rational (as long as they don’t have any stronger reasons against calculating). And analogously for subjects in “refraining cases”, i.e. subjects having motivation to refrain from calculating. Keeping the distinction between practical and epistemic irrationality strict, one might then worry that while Kahan’s mechanism can give rise to epistemic irrationality, it can do so only in calculating cases (where it moreover is restricted to calculating mistakes). Intuitively, this might not seem quite right; there seems to be more irrationality than just that. Let’s call this the “more irrationality intuition”.

There is a lively debate in current epistemology that would seem relevant to this intuition.¹⁸ It concerns the epistemic relevance of evidence a subject does not have, but could easily have obtained. A growing number of epistemologists argue that such evidence can determine whether beliefs it is relevant for are epistemically rational (or justified), and that therefore at least certain “zetetic norms” are epistemic despite their practical character (cf. Goldberg 2017; Ichikawa 2022; Flores & Woodward 2023, Simion 2023, 2024). Zetetic norms are norms of inquiry, in this case norms for gathering evidence. Consider a piece of evidence e that a subject S does not have, but could easily obtain. Roughly, the basic idea is that under certain circumstances, S “should” have e. Under such circumstances, refraining from getting e is epistemically irrational and maintaining or forming a belief that is (or would be) defeated by e is unjustified.¹⁹ On such a picture, we might be able to account for the more irrationality intuition: Our refraining cases might be such that the subjects “should” have at least tried to obtain the evidence, i.e. to calculate. Refraining then would itself be epistemically irrational and the beliefs formed, or held on to, despite the availability of evidence against them unjustified.

This is certainly interesting, but we can’t explore it much further here. We would like to note, however, that even though calculating mistakes are the only epistemic irrationalities Kahan’s mechanism can bring about, there is an alternative directly relevant source of irrationality in the subjects’ responses to the numbers: the table heuristics. After all, it is their outputs that make the mechanism kick in. And in refraining cases, the mechanism allows subjects to stick with their heuristics-generated beliefs. Whether heuristics-generated beliefs in general are epistemically irrational is a matter of debate, but the particular heuristics relevant here – the table heuristics – might just be sufficiently unreliable for such a verdict. In that case, we might not need zetetic norms, or the effects of evidence not actually had by the subjects, to account for the more irrationality intuition. The relevant beliefs would be irrational already in virtue of being heuristics-generated, irrespective of any unobtained evidence “out there”.

Thus far, we have considered Kahan’s mechanism not only on the assumption that there is a numeracy effect, but also that it is generated by Kahan’s mechanism. To round things off we would like to comment on the question whether Kahan et al. are right about what generates the numeracy effect – if there is one. The short answer is: We don’t know. Even so, some things can be said: It does, for instance, seem fair to say that Kahan et al. provide a reasonably plausible explanation of such an effect, an explanation, moreover, that cannot be ruled out on the basis of experiments such as the skin cream/gun control experiment alone. That by no means excludes the possibility of other plausible explanations.²⁰ Nor can it be ruled out that the correct explanation might yet involve motivated reasoning. But that none of these possibilities is ruled out by them suffices to undermine the claim that the results of such experiments provide additional support for ICT. So, for undermining the support a numeracy effect could provide for ICT (and, thereby, the motivated nature of partisan cognition) we do not need to assume that Kahan’s mechanism would be the correct explanation – all we need is that this explanation is a “live option”, an option that has not been ruled out.

By itself, the existence of the numeracy effect – far from being a silver bullet – thus is quite powerless to take us out of the impasse many think the debate between motivational and non-motivational accounts of fact polarization, and partisan cognition more generally, has reached (cf. Williams 2023; p. 89).²¹ In a sense, the question whether the numeracy effect actually exists therefore becomes moot. Even if it replicated, which at the time of writing appears rather doubtful (see above, fn. 1), some other means of breaking the impasse would be needed.²² Thinking about how to best go forward is beyond the scope of this paper, however. For now, we conclude that on careful examination, Kahan’s mechanism turns out to involve motivation to reason (and, possibly, to refrain from it), but no motivated reasoning, and that this undermines the support he takes the numeracy effect to provide for the identity protective cognition hypothesis.²³