This study utilized a between-subject experimental web-survey design (treatment vs. control), with two treatment conditions. Participants were first presented with an online consent form that masked the purpose of the study as a test on whether people’s online reading habits, attention spans, and information retention differed by demographic characteristics. Following this, they filled in pre-test demographic, political psychographic, and media consumption measures. They were then randomly assigned to either a (1) anti-Clinton DFN treatment condition, (2) anti-Trump DFN treatment condition, or (3) control condition, and exposed to study materials via a questionnaire packet with the outcome measures presented at the end of the survey.
The surveying agency Qualtrics was commissioned to recruit participants during February 2018. Data were collected from 552 American residents aged 25–49 (M = 36.3, SD = 6.9 years). Participants from these ages were selected, because they are likelier to vote than younger cohorts (Castillo 2016), and more likely to use new media than older cohorts. As such, they provide an arguably ideal vantage point from which to gage the potential effects of DFN on electoral decisions. The sample was 48.6% (female) and 51.4% (male), and 78.1% (White), 8% (African-American), 6% (Latino/Hispanic), 6.5% (East Asian), and 1.4% (Other). Moreover, 3.3% reported having received a (high-school education or less with no diploma), 34.4% had a (high-school diploma or GED), 12.3% had an (Associate of Arts degree), 33.5% had a (Bachelor’s degree), and 16.5% had a (Master’s degree or higher). Additionally, 11.1% reported having an annual family income of between (0–$20,000); with 22.8% reporting ($20,001–$40,000), 21% reporting ($40,001–$60,000), 17.2% reporting ($60,001–$80,000), 10.7% reporting ($80,001–$100,000), and 17.2% reporting ($100,001–over).
In all, 184 participants were randomized into the anti-Clinton DFN condition, 167 into the anti-Trump DFN condition, and 201 into the control condition. Following their randomization, participants completed three reading tasks that were disguised as attention-memory tests and presented across the questionnaire in between several filler items. Before each task, a pop-up note informed participants that they would be shown random social media news feed posts or online news articles, and then asked a question on what was displayed. Participants were asked three post-task questions, and those who answered any of these questions incorrectly had their sessions automatically ended and data deleted. This admittedly strict protocol was taken based on three important methodological considerations. First, unlike in a standard laboratory experiment, researchers conducting web-survey experiments cannot control external distractions or be certain that participants are sufficiently engaging with the presented tasks. Second, as Maniaci and Rogge (2014, p. 75) argue, “inattentive participants might not be affected by text-based manipulations, potentially adding error variance to the effects and thereby obscuring meaningful results”. Third, inattentive responding can adversely affect statistical analyses and generate spurious results (Maniaci and Rogge 2014). The aforementioned protocol, therefore, helps to ensure that the data set was comprised of participants who evidenced adequate news stimuli encoding. We deemed meeting this condition critical, so that in the event of finding significant treatment effects, we could be more confident that these stemmed from attention to the experimental stimuli materials.
Accordingly, for the first task, participants were instructed to read two Facebook news feed posts (each displayed separately). For the second and third tasks, participants read a short online news article. Additionally, participants were later in the questionnaire shown an additional Facebook newsfeed post that was displayed for 5 s. This served to further prime participants in the treatment conditions with either negative framings of Clinton or Trump. In all, every participant attended to two short news articles and three Facebook posts. Finally, the stimuli materials were displayed in between 18 distractor questions. This design was implemented to help mask the purpose of the study and minimize demand effects.
The treatment stimuli consisted of DFN materials about Hillary Clinton and Donald Trump, because they are the most recent major US Presidential candidates to date. Trump will also very likely run for re-election in 2020, and Clinton continues to flirt with the possibility of running again. However, in any event, experimental exposure to anti-Clinton and anti-Trump DFN can shed light on how DFN may potentially affect upcoming elections. Moreover, we followed the ecological approach of a related study by Pennycook et al. (2017), which entailed using actual DFN. More specifically, the DFN stories which we used for the three reading tasks in each of the treatment conditions were selected from and have been identified as fake news by the fact-checking website Snopes.com. However, we completely made up the DFN Facebook posts used as the final priming stimuli in the treatment conditions (viz., the ones displayed for 5 s), to introduce new DFN information to participants.
All of the anti-Clinton and anti-Trump DFN Facebook posts featured images of the candidates appearing dour and surly along with sensationalistic headlines. These include for example, (Hillary Caught On Tape Laughing About Irma “Wiping Out All Of Those Florida Hillbillies”), and (Trump To Deport California Democratic State Senator’s Family Because They Are “All Illegals”). The anti-Clinton DFN short online articles featured libelous headlines and detailed reports of Clinton’s fabricated ridicule of Hurricane Irma victims and involvement in a pay-to-play scheme. The anti-Trump DFN short online articles presented defamatory headlines and respectively detailed reports of Trump’s fabricated executive order for mass deportations of undocumented immigrants and ratification of a tax bill that would negatively impact firefighters, first-responders, and unions. Additionally, all DFN materials were made to resemble realistic Facebook news feed posts or website articles to enhance ecological validity. The dates of these materials were also changed to the week of recruitment, and their displayed popularity metrics were bolded and increased to make the posts and articles appear recent and popular, and thus ostensibly more credible.
Furthermore, since the two treatment conditions contained distinct sets of DFN about opposite candidates, their potential main effects are on different target attitudes, and hence cannot be compared with each other. There is also a possibility that differences in perceptions of DFN believability could result from the anti-Clinton DFN materials being more or less convincing than the anti-Trump ones, despite both sets being equally partisan. Therefore, we created a control condition consisting of three Facebook news feed posts and two web-articles collected from legitimate press outlets that included Rolling Stone and NBC News. These materials featured non-political real news headlines and reports about sports, music, and a contaminated lettuce scare (see “Appendix” for examples of the materials used for each condition).
Using a six-point forced-choice response scale (1 = very negative, 6 = very positive), participants were asked to indicate their “opinion/impression” of two major Democrats [sc., Hillary Clinton, Barack Obama) and two major Republicans [sc., Donald Trump, Paul Ryan]. The average was (M = 3.1, SD = 1.7) for Clinton and (M = 2.8, SD = 1.8) for Trump.
Candidate voter choice
Participants were asked: “If you could vote for President today, which one of the following options would you choose?”, and presented with the following response options: (1) Hillary Clinton, (2) Donald Trump, (3) Third Party Candidate (e.g., Green Party or Libertarian Party), and (4) Would Not Vote. Of these options, 34.1% of participants selected Clinton and 29.9% selected Trump. These measures were used to create two dummy variables: Votes for Clinton and Votes from Trump. It should be noted that these variables reflect measures for behavioral intentions rather than actual behaviors. Nonetheless, while certainly limited, voting intents are well established and strong predictors for voting (Gutsche et al. 2014; Vaske and Donnelly 1999).
Mediator and moderator variables
News message believability
Using a six-point forced-choice response scale (1 = strongly disagree, 6 = strongly agree), participants were asked to select the extent which they agreed that all of the Facebook newsfeed headlines and online news articles presented to them were (1) Credible and (2) Trustworthy. These measures were averaged to create a composite variable: [M = 3.4, SD = 1.3, r(551) = 0.96, p < 0.001].
Participants were asked to indicate how much they agreed on a six-point forced-choice response scale (1 = strongly disagree, 6 = strongly agree) with six self-developed statements—three of which were reversed coded. Some of these statements included the following. “People should be responsible for paying for their own healthcare and not expect the government to fund it”. “Welfare makes people lazy and unwilling to work”. These measures were averaged to create a composite variable (M = 3.0, SD = 1.1, a = 0.79), which denotes a left- to right-wing ideological continuum. Specifically, low, middle, and high scores, respectively, represent politically leftist, centrist, and conservative orientations. While party affiliation was measured, the composite variable above was selected as the proxy for partisanship, because it captures a more precise and nuanced measure of participants’ political belief and attitudinal orientation and strength than a simple categorical party variable.
In addition to the demographic variables reported earlier, the political psychographic variables below have been consistently found to modulate the impact of news media on candidate evaluations and voter support (Moy et al. 2016; Newton 2019). They were thus measured as follows and controlled for in the analyses.
This was measured via a question on prior voting, as this is indicative of political interest. Specifically, participants were asked whether they voted in the 2016 US Presidential Election. 88% reported having voted and 12% reported that they did not vote.
Participants were shown a six-item list and asked to select which political party they “most identified with and supported”. 31.6% identified as Republicans, 37.3% as Democrats, 5.8% as Libertarians, 1.2% as Green, 6.7% as Other, and 17.4% as none.
Participants were asked to answer five multiple choice test questions on general politics knowledge unrelated to the study. Some of these included the following: “What body of government is tasked with interpreting the constitutionality of a piece of legislation (i.e., law)?” “Which of the following sectors does the US government spend the most money on?” “What was one of the original reasons given by the Bush Administration for invading Iraq in 2003?” Correct responses were coded 1 and 0 otherwise, and then summed to create an additive index (M = 2.4, SD = 1.3).
The following media selective-consumption measures were taken to partly control for the potential recency and ceiling effects caused by DFN or DFN corrections that relate to the aforementioned treatment conditions, and that participants may have encountered prior to this experiment. Using a five-point response scale (1 = never, 5 = very often), participants were asked to indicate how often they received news from the five major television news organizations (Fox News, ABC News, CBS News, MSNBC, CNN). These measures were averaged to create a composite variable for Television News Consumption (M = 2.8, SD = 0.96, a = 0.81). Participants were also asked to indicate their weekly usage of Facebook (M = 4.1, SD = 1.2) and Twitter (M = 2.6, SD = 1.6). Finally, participants were asked to indicate how often they received news from Breitbart News (M = 4.1, SD = 1.2) and InfoWars (M = 2.6, SD = 1.6), which are possibly the most popular and prominent disseminators of political DFN (Friedersdorf 2017; Hayden 2018).