On the (non)persuasive power of a brain image

Abstract

The persuasive power of brain images has captivated scholars in many disciplines. Like others, we too were intrigued by the finding that a brain image makes accompanying information more credible (McCabe & Castel in Cognition 107:343-352, 2008). But when our attempts to build on this effect failed, we instead ran a series of systematic replications of the original study—comprising 10 experiments and nearly 2,000 subjects. When we combined the original data with ours in a meta-analysis, we arrived at a more precise estimate of the effect, determining that a brain image exerted little to no influence. The persistent meme of the influential brain image should be viewed with a critical eye.

A number of psychological research findings capture our attention. Take the finding that people agree more with the conclusions in a news article when it features an image of the brain, even though that image is nonprobative—providing no information about the accuracy of the conclusions already in the text of the article (McCabe & Castel, 2008). In a time of explosive growth in the field of brain research and the encroaching inevitability of neuroscientific evidence in courtrooms, the persuasive influence of a brain image is both intriguing and worrying.

Perhaps because of its implications, this research has received much attention in both the popular and scholarly press (nearly 40 citations per year, according to Google Scholar, as of November 30, 2012). Although McCabe and Castel (2008) did not overstate their findings, many others have. Sometimes, these overstatements were linguistic exaggerations. One author of a paper in a medical journal reported that “brain images . . . can be extremely misleading” (Smith, 2010). Other authors of a paper in a social issues journal concluded, “clearly people are too easily convinced” (Kang, Inzlicht, & Derks, 2010). Other overstatements made claims beyond what McCabe and Castel themselves reported: In an education journal, authors wrote that “brain images make both educators and scientists more likely to believe the statements” (Hinton & Fischer, 2008), while in a forensic psychiatric journal, others worried about “the potential of neuroscientific data to hold significant prejudicial, and at times, dubious probative, value for addressing questions relevant to criminal responsibility and sentencing mitigation” (Treadway & Buckholtz, 2011).

These and other misrepresentations show that the persuasive power of brain images captivates scholars in many disciplines. We too were captivated by this finding and attempted to build on it—but were surprised when we had difficulty obtaining McCabe and Castel’s (2008) basic finding. Moreover, in searching the published literature, we were likewise surprised to discover that the effect had not been replicated. In one paper, some brain images were more influential than others on subjects’ evaluations of an article’s credibility, but because there was no condition in which subjects evaluated the article without a brain image, we cannot draw conclusions about the power of brain images per se (Keehner, Mayberry, & Fischer, 2011). Other papers show that written neuroscience information makes “bad” explanations of psychological phenomena seem more satisfying (Weisberg, Keil, Goodstein, Rawson, & Gray, 2008) and that written fMRI evidence can even lead to more guilty verdicts in a mock juror trial (McCabe, Castel, & Rhodes, 2011). We even found work in another domain showing that meaningless mathematics boosts the quality of abstracts (Erikkson, 2012). But we did not find any other evidence that brain images themselves wield power.

Given the current discussion in psychological science regarding the importance of replication (see the November 2012 Perspectives on Psychological Science, the February 2012 Observer, and www.psychfiledrawer.org), we therefore turned our attention to a concentrated attempt to more precisely estimate how much more people will agree with an article’s conclusions when it is accompanied by a brain image. Here, we report a meta-analysis including McCabe and Castel’s (2008) original data and 10 of our own experiments that use their materials.¹ We arrive at a more precise estimate of the size of the effect, concluding that a brain image exerts little to no influence on the extent to which people agree with the conclusions of a news article.

Method

Subjects

Across 10 experiments, a total of 1,971 people correctly completed all phases of the experiment (Table 1 shows the experiment number, subject pool, medium of data collection, sample size, and compensation). We collected demographic information from the Mechanical Turk subjects. These subjects ranged in age from 16 to 82 years (M = 29.24, median = 26, SD = 10.78). Eleven percent had completed a PhD or Masters degree; 35 % had completed a Bachelor’s degree; 52 % had completed high school; and the remaining 2 % had not finished high school.

Table 1 Characteristics of our 10 experiments included in the meta-analysis

Design

In each experiment, we manipulated, between subjects, the presence or absence of a brain image.

Procedure

We told subjects that they were taking part in a study examining visual and verbal learning styles. Subjects then read a brief news article, "Brain Scans Can Detect Criminals,” from McCabe and Castel's (2008) third experiment. The article was from the BBC News Web site and summarized a study discussed in Nature (BBC News, 2005; Wild, 2005). All of our attempts to replicate focused on this experiment, because it produced McCabe and Castel's largest effect (d = 0.40). Although there were two other experiments in their paper, the first used different materials and a different dependent measure and was a within-subjects design. Their second experiment, also a within-subjects design, examined the effects of different types of brain images but did not have a baseline condition with no brain image and, therefore, did not permit brain versus no-brain comparisons.

In their third experiment, McCabe and Castel (2008) used a 2 × 2 between-subjects design, manipulating (1) the presence or absence of a brain image depicting activity in the frontal lobes and (2) whether the article featured experts critiquing the article’s claims. Although they did not explain the rationale for the critique manipulation, it stands to reason that criticism would counteract the persuasive influence of a brain image. Indeed, they adopted that reasoning in a later paper showing that the persuasive influence of written neuroscientific evidence on juror verdicts decreases when the validity of that evidence is questioned (McCabe et al., 2011). McCabe and Castel found that the critique manipulation did not influence people’s ratings of the article’s conclusions, nor did it interact with the presence of a brain image, so in Experiments 1–5 we used the article without the critique.

But when we took a closer look at McCabe and Castel’s (2008) raw data, we found that the influence of the brain image was larger when the article’s claims were critiqued, as compared with when they were not, t _critique(52) = 2.07, p = .04, d = 0.56; t _{no_critique}(52) = 1.13, p = .27, d = 0.31. Note that this surprising result runs counter to the explanation that evidence is less influential when its validity is called into question (McCabe et al., 2011). With these findings in mind, in Experiments 6–10, we used the article in which experts criticized the article’s claims in an extra 100 words.

After reading the article (critiqued or not), subjects responded to the statement "Do you agree or disagree with the conclusion that brain imaging can be used as a lie detector?" on a scale from 1 (strongly disagree) to 4 (strongly agree). Subjects were randomly assigned to a condition in which the article appeared alone or a condition in which an image of the brain appeared alongside the article. In all online experiments, subjects then encountered an attention check, which they had to pass to stay in the data set (Oppenheimer, Meyvis, & Davidenko, 2009).²

Results and discussion

How much influence does an image of the brain have on people’s agreement with the conclusions of a news article? To answer this question, we first calculated the raw effect size for each experiment by determining the difference between mean agreement ratings among people in the brain and the no-brain conditions. We report these findings in Table 2.

Table 2 Summary of results of our 10 experiments included in the meta-analysis

To find a more precise estimate of the size of the effect, we used ESCI software (Cumming, 2012) to run a random effects model meta-analysis of our 10 experiments and two findings from McCabe and Castel (2008). We display the results in Fig. 1. The result of this meta-analysis is an estimated raw effect size of 0.07, 95 % CI [0.00, 0.14], z = 1.84, p = .07. On a 4-point scale, this estimate represents movement up the scale by 0.07 points, or 2.4 % (cf. McCabe and Castel’s [2008] original raw effect size of 0.26, or 8.7 %). We also found no evidence of heterogeneity across the experiments. Tau—the estimated standard deviation between experiments—was small (0.07), and the CI included zero as a plausible value (95 % CI [0, 0.13]; note, of course, that tau cannot be less than 0), suggesting that the observed variation across experiments could very plausibly be attributed to sampling variability. This finding is important because, at first glance, it appears as though the brain image might be more persuasive on paper than online, but the statistics simply do not support this idea. We also examined the impact of other potentially important moderators. We ran an analysis of covariance on the dependent measure, using age and education as covariates and condition (brain, no brain) as an independent variable. Neither covariate interacted with the independent variable, suggesting that the persuasive influence of a brain image is not moderated by age or education.

Fig. 1

Forest plot of effect sizes between studies. Each row represents one experiment, starting with the original McCabe and Castel (2008) finding when the article did not feature criticism, then our five replications, then McCabe and Castel’s finding when the article featured criticism, then our five replications. Our experiments are numbered 1–10, as also in Tables 1 and 2. The location of each square on the horizontal axis represents the effect size—the difference between mean agreement ratings in the no-brain and brain conditions (the maximum possible difference between these two means is ±3; positive values indicate a higher mean score for brain). The black lines extending either side of a square represent a 95 % confidence interval. The size of each square indicates the sample size and weighting an experiment is given in the meta-analysis. Finally, the red diamond shows the result of the meta-analysis, with the center of the diamond indicating the estimated effect size and the spread of the diamond representing a 95 % confidence interval

How are we to understand the size of the brain image effect in context? Let us consider subjects’ hypothetical responses, as shown in Fig. 2. The line marked “No Brain” represents the weighted mean agreement of subjects who read the article without a brain image. The line marked “Brain” represents how far subjects’ agreement would shift, in the mean, if they had read the article with a brain image. Taken together, this figure, coupled with the meta-analysis, makes it strikingly clear that the image of the brain exerted little to no influence. The exaggerations of McCabe and Castel’s (2008) work by other researchers seem even more worrisome in light of this more precise estimate of the effect size.

Fig. 2

An illustration of the brain effect. The no-brain bar represents the weighted average for subjects who read the article without a brain image (2.77). The difference between the no-brain bar and the brain bar is the estimated effect size from the meta-analysis (0.07)

It is surprising, however, that an image of the brain exerted little to no influence on people’s judgments. We know that images can exert powerful effects on cognition—in part, because they facilitate connections to prior knowledge. For instance, when pictures clarify complex ideas (such as the workings of a bicycle pump) and bridge the gap between what nonexperts know and do not know, people comprehend and remember that material better (Mayer & Gallini, 1990; see Carney & Levin, 2002, for a review).

Manipulations like these that boost comprehension can also make other concepts related to the material feel more easily available in memory, and we know that people interpret this feeling of ease as diagnostic of familiarity and truth (Newman, Garry, Bernstein, Kantner, & Lindsay, 2012; Tversky & Kahneman, 1973; Whittlesea, 1993; see Alter & Oppenheimer, 2009, for a review). But a brain image depicting activity in the frontal lobes is different. To people who may not understand how fMRI works, or even where the frontal lobes are, seeing an image of the brain may not be any more helpful than seeing an ink blot. It seems reasonable to speculate, therefore, that images of the brain are like other technical images: To people who cannot connect them to prior knowledge, there is no boost of comprehension, nor a feeling of increased cognitive availability. This speculation leads directly to an interesting question: To what extent is the influence of a brain image moderated by prior knowledge?

Another explanation for the trivial effect of brain images is that people have become more skeptical about neuroscience information since McCabe and Castel’s (2008) study. Indeed, the media itself has begun engaging in critical self-reflection. For instance, a recent article in the New York Times railed against the “cultural tendency, in which neuroscientific explanations eclipse historical, political, economic, literary and journalistic interpretations of experience” and “phenomena like neuro law, which, in part, uses the evidence of damaged brains as the basis for legal defense of people accused of heinous crimes” (Quart, 2012). If people have indeed grown skeptical, we might then expect them to also be protected against the influence of other forms of neuroscience information. To test this hypothesis, we ran a series of replications of another well-known 2008 study showing that people rated bad explanations of scientific phenomena as more satisfying when those explanations featured neuroscience language (Weisberg et al., 2008).

Our five replications, which appear in Table 3, produced similar patterns of results. To estimate the size of the effect more precisely, we followed a similar approach as we did earlier, running a random effects model meta-analysis of our five experiments and the original finding from Weisberg et al. (2008). The result was an estimated raw effect size of 0.40, 95 % CI [0.23, 0.57], z = 4.71, p < .01. On a 7-point scale, this estimate represents movement up the scale by 0.40 points, or 6.67 %. The CI does not include zero as a plausible value, providing evidence against the idea that people have become savvy enough about neuroscience to be protected against its influence more generally.

Table 3 Summary of results of experiments replicating Weisberg, Keil, Goodstein, Rawson, and Gray (2008)

Why the disparity, then, between the trivial effects of a brain image and the more marked effects of neuroscience language? A closer inspection reveals that the Weisberg et al. (2008) study is not simply the language analog of the McCabe and Castel (2008) study. For instance, Weisberg et al. found that neuroscience language makes bad explanations seem better but has less (or no) effect on good explanations. By contrast, McCabe and Castel did not vary the quality of their explanations. In addition, Weisberg et al. compared written information that did or did not feature neuroscience language, but McCabe and Castel added a brain image to an article that already featured some neuroscience language. Perhaps, then, the persuasive influence of the brain image is small when people have already been swayed by the neuroscience language in the article. Although such a possibility is outside the scope of this article, it is an important question for future research.

How are we to understand our results, given that other research shows that some brain images are more influential than others (Keehner et al., 2011)? One possibility is that Keehner and colleagues’ within-subjects design—in which subjects considered a series of five different brain images—encouraged people to rely on relative ease of processing when making judgments across the different images (Alter & Oppenheimer, 2008). By contrast, McCabe and Castel’s (2008) between-subjects design does not allow people to adopt this strategy. And recall, of course, that because Keehner and colleagues did not compare the influence of any brain image with that of no brain image, their work still does not address that basic question.

Although our findings do not support popular descriptions of the persuasiveness of brain images, they do fit with very recent research and discussions questioning their allure (Gruber & Dickerson, 2012; Farah & Hook, 2013). Importantly, our estimation approach avoids the dichotomous thinking that dominates media discourse of popular psychological effects and, instead, emphasizes—in accord with APA standards—interpretation of results based on point and interval estimates. Furthermore, research in the domain of jury decision making suggests that brain images have little or no independent influence on juror verdicts—a context in which the persuasive influence of a brain image would have serious consequences (Greene & Cahill, 2012; Schweitzer & Saks, 2011; Schweitzer et al., 2011). Taken together, these findings and ours present compelling evidence that when it comes to brains, the “amazingly persistent meme of the overly influential image”³ has been wildly overstated.