Do Online Advertisements Increase Political Candidates’ Name Recognition or Favorability? Evidence from Randomized Field Experiments

Ad Treatments

The campaigns’ Facebook ads appeared on the right side of computer users’ screens on all pages on the site and were 125 pixels high by 255 pixels wide, as is standard for such ads. When clicked, all ads brought individuals to the candidate’s Facebook ‘page’ (though as is typical for online advertisements, click rates were well below 0.1 %).¹⁰

The candidate’s constituency includes a large number of people connected to the farming industry, and thus the candidate expected this to be a particularly salient issue.

Random Assignment Procedure

We implemented the clustered random assignment procedure described in the previous section. First, we received a copy of the public list of voters from the campaign. At the campaign’s request, we removed individuals under the age of 30 and over the age of 75 from the study, leaving 32,029 voters who were then assigned to 1,220 clusters across 18 age ranges,¹¹ the 34 towns in the candidates’ district, and 2 genders.

We then blocked clusters into 244 groups of five based on cluster size¹² (which ranged from one person to five hundred), then age range, then town, and finally gender. Within these blocks of five clusters, we assigned two clusters to the control condition and one each to our three treatments: the name recognition appeal, the character appeal, and the policy appeal. (For further explanation of the mechanics of this blocked-clustered design, see Appendix 1.)

Treatment Delivery

We uploaded these ads to Facebook on Saturday, October 6, 2012 and the site approved them for delivery to individuals in the treatment clusters shortly thereafter. The ads were served on Facebook beginning on Monday, October 8, 2012 and ending Friday, October 12, 2012.

Online ads are purchased through ‘mini-auctions’ among potential advertisers. The campaign placed an extremely high bid for each ad, $1.51 per thousand impressions.¹³ Although a bid of only 0.151 cents per impression may seem like a pittance, the market for targeted Facebook advertisements actually typically clears at prices well below $0.30 per thousand impressions, or 0.030 cents per impression ($0.0003 per impression). We selected this unusually high bid so as to be sure that we displayed as many ads as possible. According to Facebook’s accounting records, although the campaign contracted with Facebook to spend up to $150 per day delivering impressions at up to this plum price, the platform was only able to deliver about $40 per day in advertising due to the finite supply of Facebook users from the targeted constituency. The campaign therefore ran as many Facebook advertisements to the treatment group as was possible.

During the course of the week-long advertising campaign, the Facebook ad interface reported that essentially every single person who could have seen the ads on Facebook did indeed see them (in Facebook parlance, the number of ‘targeted’ individuals was identical to the number of individuals ‘reached’)—5,012 users in the family treatment, 4,752 users in the character treatment, and 4,970 in the policy treatment, or 14,734 Facebook users in all. 19,377 voters on the voter file were assigned to these clusters. Facebook’s records suggest that over the course of the week the typical targeted person saw the ads about three dozen times.

Outcome Measurement

To assess the impact of the ads, on Saturday, October 13 through Monday, October 15 the polling firm AMM Political Strategies completed live interviews with 2,984 individuals on the voter file (all of whom had associated phone numbers). The firm called the numbers in random order and did not have access to the treatment assignment status of the respondents. At our request, the campaign ceased all advertising on Facebook during this 3 day polling period.

The questionnaire, given in Appendix 2, asked respondents (1) whether they had a positive, negative, or no impression of the collaborating candidate, (2) whether they had a positive, negative, or no impression of the opposing candidate, (3) their vote intention in the upcoming election, (4) whether they recalled seeing any ads for the candidate on the internet, and (5) how often they used Facebook over the last week. As mentioned earlier, this question ordering was crucial to the credibility of the estimates: questions (4) and (5) were asked after the main dependent variables of interest so as to avoid priming respondents with the ads’ content or tipping off treated users to the connection between the poll and the online advertisements.

Results: Descriptive Statistics

Experimental Results

Table 2 presents three specifications in order to demonstrate the robustness of the results. The first specification compares means without covariate adjustment and uses randomization inference to calculate confidence intervals. The second specification compares means after regressing the outcome measures on covariates describing the subjects’ party identification, age, and vote history, again using randomization inference to form confidence intervals and test the sharp null hypothesis of no effect. These covariates were selected prior to the launch of the survey, in accordance with the authors’ ex ante analysis plan, registered at the EGAP website.¹⁶ Finally, we present the results from regressions that control for blocks and form confidence intervals using clustered standard errors and an assumed normal sampling distribution.

Across four dependent variables, the results of the experiment are consistent: the ad treatments appeared to have no politically consequential effect on knowledge of the candidate, favorable evaluation of the candidate, or electoral support. Moreover, these results are precisely estimated, with confidence intervals that rule out politically meaningful impacts. Indeed, the results cast doubt on the proposition that dozens of online ads increased the candidate’s name recognition in his district by more than approximately 1.8 % points, which marks the upper end of the 95 % confidence interval.

Because only Facebook users were exposed to the ads, we also estimate the treatment effect for the subset of the respondents that reported using Facebook over the previous week. These estimates are shown in the fourth, fifth, and six rows of Table 2. Although these effects are less precisely estimated due to the decreased sample size in this subgroup, the estimates are in accordance with the results calculated among the broader sample. The 95 % confidence intervals again rule out effects that would be politically consequential, a point we elaborate further in the “Discussion” section.¹⁷

Consistent with the finding that the ads had little or no effect on name recognition, the other columns of Table 2 show that those in the treatment group did not become significantly more favorable toward the candidate or more likely to vote for him. Indeed, Facebook users in the treatment group were not significantly more likely to recall seeing the online advertisements.

Ad Treatments

The campaigns’ sidebar ads appeared on the right side of Facebook users’ screens on all pages on the site and were 125 pixels high by 255 pixels wide, as in Study 1. When clicked, all ads brought individuals to the candidates’ Facebook page.

As mentioned above, the campaign also purchased sponsored story ads that were shown to users in the treatment group who had friends that “Liked” the candidate’s Facebook page.¹⁸ At least once per day, the candidate posted new updates to his Facebook page, and users eligible to receive these ads saw these updates in their Facebook “news feeds.” These updates included stories about the candidate’s visits to places in the district (e.g., a factory), rallies, television commercials, endorsements, and favorable news articles.

Random Assignment Procedure

We followed the clustered random assignment procedure described in the previous section and from Study 1. First, we received a copy of the public voter registration list from the campaign. We then removed voters age 65 and older and those without known phone numbers. Remaining voters were then assigned to 752 clusters across 47 values of age (each age 18–64), 8 counties, and 2 genders. We then blocked 752 clusters into groups of four based on cluster size (Middleton and Aronow 2011). Within each block of four clusters, we assigned three clusters to the control condition and one to receive the ad treatments. Our procedure thus randomly assigned 25 % of 261,150 identifiable individuals to an online treatment or a control condition.

Treatment Delivery

We uploaded these ads to Facebook on Monday, October 28, and the site approved them for delivery to the treatment clusters shortly thereafter. The ads were served on Facebook beginning early in the morning on Tuesday, October 29, 2012 and ending late in the evening on Sunday, November 4, 2012. As with the previous study, the campaign placed an unusually high bid for each ad, $0.80 per thousand impressions, more than triple the market price it actually paid. Users in the treatment group were thus exposed to the ads as many times as the platform would allow. Facebook’s records indicate that 108,783 individuals were shown the ads an average of 36.6 times, corresponding to 3.98 million impressions overall.¹⁹

Outcome Measurement

To measure subjects’ attitudes, the polling firm Winning Connections attempted automated interviews with 154,024 individuals on the voter file (all of whom had associated phone numbers) on the evening of Monday, November 5. The firm called the numbers in a random order and did not have access to data on the treatment assignment status of the individuals. At our request, the campaign ceased all online advertising on this day.

The poll’s text is given in Appendix 2. It first asked respondents to enter their age, zip code, and gender. Because the poll was conducted automatically, we could not instruct the pollster to verify voters’ names; the initial questions were therefore used to verify that we reached the intended voter on the phone. We then asked respondents (1) whether they had a positive, negative, or no impression of the collaborating candidate, (2) whether they had a positive, negative, or no impression of the opposing candidate, (3) whether they recalled the candidates’ main campaign issue described in the sidebar ads (hydraulic fracking; subjects were also given the choices ‘trade with China’ and ‘abortion’), (4) whether they recalled seeing any ads for the candidate on the internet, and (5) how often they used Facebook over the last week. Questions (4) and (5) were asked after the main dependent variables of interest so as to avoid priming respondents to recall the ads.

A total of 4,359 voters answered at least one question in the automated poll.²⁰ Of these responses, 3,557 were successfully matched back to the voter file based on an exact match on each subject's telephone number, age, gender, and zip code. In other words, 802 responses to the poll were from respondents who did not appear in the voter file under that phone number. When we impute treatment assignment to these individuals (on the basis of self-reported age, gender, and zip code), the results are nearly identical, but we exclude them from the analysis below in keeping with the cautious procedure laid out in our ex ante pre-analysis plan.

Results: Descriptive Statistics

Experimental Results

The number of valid survey responses for each variable is also presented under the estimates and decrease slightly across columns. As typically occurs with automated polls, some subjects abandoned the calls after answering each question. (However, there is no evidence of differential attrition across the treatment groups during the course of the survey.) Low response rates reduce statistical power but do not threaten unbiased estimation of average treatment effects among those for whom outcome measures are available.

As in Study 1, we find no evidence that the ads had consequential effects on knowledge of the candidate or his favorability ratings. The new variable we included in this study, correct recall of the candidate’s main campaign issue (fracking, which was featured in the ads), also generated substantively small treatment effects that are statistically indistinguishable from zero. Although the confidence intervals are larger than in the previous study, the point estimates and confidence intervals again cast doubt on the proposition that the ads have politically meaningful effects.

The results from Study 2 did depart in one important respect from the findings from Study 1. As the last column of Table 4 indicates, subjects randomly assigned to online ads were 5.3 percentage points more likely to recall seeing items about the candidate on the Internet, an estimate that is highly statistically significant (p < 0.001). Among Facebook users this effect is even more pronounced, an 8.1 percentage point increase.²⁵ Again, randomization inference places the p value at less than 0.001. As intuition would suggest, ad recall shows no treatment effect among non-Facebook users.

As a methodological matter, this finding gives us confidence that our clustered randomization methodology worked successfully. Study 1’s null effects caused us to be concerned that Facebook had inadvertently delivered the ads to some members of the control group (although we meticulously checked and verified that the treatments had been delivered as intended in Study 1). Study 2 found that ad recall was significantly higher in the treatment group, putting this concern further to rest; nevertheless, the overall pattern of findings is consistent with the results obtained from Study 1.

Pooled Results and Comments on Assessing Cost Effectiveness

To put this finding in perspective, consider the scope of the intervention and the manner in which it was delivered. Recall that treated Facebook users were typically exposed to the ads about 38 times in these studies. Suppose that these 38 exposures did generate a cumulative effect of 2.2 percentage points, the top of the 95 % confidence interval. If 38 exposures were necessary for this 2.2 percentage point effect, it follows that each marginal exposure to the online political advertisements increased name recognition or candidate favorability by an average of (2.2/38=) 0.058 percentage points. In other words, even if the true average treatment effect were in fact at the top of the 95 % confidence interval, just (1/0.00058=) 1 in 1,700 people learned the name of the candidates or gained a favorable impression of them from each exposure to their Facebook advertisements. Based on this evidence, online advertisements appear unlikely to play a meaningful role in determining a candidate’s success or failure.

Although these results generally run counter to the notion that online ads have substantively large effects on their viewers’ attitudes and behavior, it remains possible that online advertising remains a cost-effective persuasion tactic given the ads’ very low price (Lewis and Rao 2012). Suppose, for example, that the candidates’ online ads attracted votes at a rate of $10 per vote, which would be efficient by comparison to most campaign tactics (Green and Gerber 2008, p. 139). Recall that the candidate in Study 1 was only able to purchase $200 worth of ads per week—Facebook was unable to provide any more advertising given the finite supply of Facebook users and the finite number of pages they each load on the site. If Facebook advertisements won votes at a respectable $10 per vote and the candidate purchased $200 worth of ads, we would expect only 20 votes to change. To change merely 20 voters’ minds out of the roughly 20,000 who were exposed to the ads would have rendered the $200 ad buy fairly cost effective (at $10/vote). However, to reliably detect the implied 0.1 percentage point effect (20/20,000 = 0.001) of such an ad would require an experiment of roughly 5 million voters. Our experiment (and indeed the legislative districts we studied) contained far fewer than 5 million individuals and could not have detected effects of this miniscule size. As with commercial online advertisements, understanding whether very cheap political online advertisements are cost effective will likely remain “nearly impossible” (Lewis and Rao 2012). At the same time, our experiments do cast doubt on the view that online advertising has substantively meaningful impacts on political attitudes or electoral outcomes.

Two experiments, of course, can hardly provide the last word on a phenomenon as complex as online political advertising, and further research is needed to assess whether online ads would prove more potent if delivered via different modes (e.g., video) or in different contexts. For example, although we anticipated that relatively low salience elections would be conducive to strong advertising effects, it is possible that online advertisements could prove more effective in higher salience contests or for candidates who are already relatively well known. Likewise, although we study political candidates’ efforts at winning votes, perhaps other entities such as non-profit or issue advocacy organizations tend to be more persuasive; or, perhaps online ads are successful in altering attitudes beyond candidate evaluations or in affecting behavior. Online video advertisements or large and colorful banner ads could prove more effective than the static display ads we purchased. The Internet can also be deployed for a variety of political purposes other than persuasion; for example, the Obama campaign leveraged Facebook data to allow supporters to identify their friends in swing states and mobilize them to vote, a tactic that may prove more effective than impersonal display advertisements. The experimental methodology developed in this article will allow scholars and practitioners to shed light on important questions such as these in future research.

Theoretical Implications for Mass Communication

Our results also leveraged the unique potential of online advertisements to contribute to longstanding theoretical questions about the degree to which individuals’ perceptions, beliefs, and evaluations can be influenced by impersonal mass appeals (e.g., Hartmann 1936; DellaVigna and Gentzkow 2009). Reigning theories of mass communication stress the potency of repeated exposure (e.g., Atkin and Heald 1976; Lodge et al. 1995; Grimmer et al. 2012)—even to subtle messages (Zajnoc 1968; Bargh et al. 1992; Kam and Zechmeister 2013)—and the power of communication that does not run counter to individuals’ pre-existing values (e.g., Zaller 1992). From such perspectives, online advertisements would seem to represent a propitious way to generate sizeable shifts in the public’s perceptions, beliefs, and evaluations, especially for outcomes such as candidate name recognition and candidate support (e.g., Zaller 1996; Ladd and Lenz 2009): it can be essentially guaranteed that individuals will be exposed to online ads dozens of times, and the ads often contain messages that few individuals should be predisposed to resist (e.g., a graphic displaying the name of a candidate for office for the mere purpose of informing viewers of his candidacy). Our results present a challenge to the sufficiency of these theories: the treatments essentially guaranteed that subjects in our studies crossed these two theoretical hurdles to communication effects, but in neither study do we find evidence of increasing awareness of the candidates. It seems that a final hurdle for effective mass communication—individuals’ interest in processing these messages and retaining their contents (e.g., Petty and Cacioppo 1986)—largely stymied these attempts at mass influence.

Campaigns to change the public’s attitudes, beliefs, and behaviors increasingly rely upon online advertisements, yet evidence remains sparse that impersonal mass communications are able to effect large, enduring changes in individuals’ attitudes and behaviors, online or not. To be sure, the evidence strongly suggests that mass communications can sometimes influence individuals in the short term, even in sufficient numbers to swing very close elections (e.g., Hill et al. 2012). Indeed, our findings may seem surprising in light of other field experiments that have found sizeable short-term effects of mass communication on candidate choice in both low-salience (Panagopoulos and Green 2008; Gerber et al. 2011b) and high-salience elections (Huber and Arceneaux 2007; Gerber et al. 2011a).²⁷ However, a rich research tradition also raises doubts about the ability of mass messages to leave more than a fleeting impression on the public (e.g., Campbell et al. 1960; Klapper 1960), even when sympathetic subjects are directly exposed to persuasive content (e.g., Hovland et al. 1949). Consistent with this “minimal effects” perspective, observational studies and field experiments suggest that individuals often forget televised messages within a matter of days or even hours (e.g., Gerber et al. 2011a; Hill et al. 2012; Sears and Kosterman 1994); lab studies of negative advertising (e.g., Mutz and Reeves 2005) and issue framing (e.g., Chong and Druckman 2010) often find that the effects of experimental stimuli decay rapidly; and impersonal behavioral interventions often exhibit rapid decay or fail altogether (e.g., Allcott and Rogers 2012; Galiani et al. 2012).

Whether it is because people typically do not attend closely to impersonal mass messages concerning subjects they have little interest in or because they quickly forget their content amid life’s distractions, attempts at mass influence quite often have minimal effects. A growing conventional wisdom among internet advertisers and journalists suggests that online advertisements nonetheless have profound effects on the public—for example, Facebook has claimed that its ads have moved vote shares by around 20 percentage points in some cases (e.g., Facebook 2011), Google has suggested that Senator Scott Brown’s online advertisements “seal[ed his] upset victory in 2010” (Google 2013), and numerous journalistic retrospectives have credited online ads with political candidates’ victories (e.g., Edwards 2012a, b). Our experimental design offers a way to rigorously evaluate such claims by assessing the efficacy of online messages. In light of our evidence, it appears that attempts to influence the mass public online warrant the same healthy skepticism as their offline counterparts.