Introduction

Misinformation about coronavirus disease 2019 (COVID-19) has spread widely, pervasively, and rapidly following the emergence of the disease [1,2,3]. The nature of this misinformation has ranged from clearly conspiratorial and misinformed, such as the idea that 5G cell towers spread COVID-19, to conceptually possible but implausible narratives about the origins of the disease and motivations underlying preventive public health efforts [4]. These narratives can spread very quickly [5] and have been associated, directly and indirectly, with harmful outcomes [6,7,8] as well as reduced personal wellness [9].

Prevention of COVID-19 misinformation uptake, as well as public health misinformation in general, is an important, but complex, area of research. For example, efforts to “fact check” or restrict access to misinformed narratives risk being counterproductive [10]. In addition, ethical concerns reasonably can be raised regarding attempts to restrict access to public speech. An alternative approach, often described as inoculation theory [11], focuses on interventions occurring prior to exposure to new misinformation. Such approaches have been used, for example, in addressing anti-vaccination narratives [12].

Based on recent studies [13, 14], our research team is currently investigating the potential for an intervention designed to improve public trust in science and scientists to serve as a possible approach for easily disseminated misinformation prophylaxis [15]. Specifically, we have proposed a randomized, controlled superiority trial comparing an infographic about the scientific process to a placebo infographic in terms of trust in science and scientists, reported believability of misinformed narratives about COVID-19, and behavioral intentions to engage in Centers for Disease Control and Prevention (CDC)-recommended prevention behaviors [15]. Part of the study protocol involves iterative design and selection of a single infographic from among multiple alternatives to be used in the primary trial. This Research Note describes the preliminary work and pilot test.

Main text

Infographic design

The infographics used in this pilot study were first conceptualized as text-only messaging based on our interpretation of underlying principles of trust in science as described by Nadelson et al. [16]. These included: (a) credibility and consensus, (b) epistemology, (c) trustworthiness, (d) stereotypes of scientists/“scientist-as-person,” and (e) science as methodology, not field. These ideas were workshopped extensively among the study team for clarity, and written descriptions of potential visual components were also recorded alongside each narrative.

As indicated in the protocol [15], these narratives were informally discussed within the authors’ nonscientific social networks. This feedback was discussed among the researchers and was used to make decisions about both the written and visual elements of the infographics. For example, non-scientists uniformly rejected statements beginning with “All scientists…” preferring instead the more guarded “Most scientists…” They also encouraged linking visuals to commonly discussed science, like the Space X program; we also observed that we should avoid politically controversial topics such as climate change in designing our infographics.

Our written descriptions were then presented in a meeting with a subcontracted graphics design team at Indiana University. That team prepared a set of five infographics and our research team reviewed the images, collectively made suggestions, and then the graphics design team modified the infographics accordingly (see Additional files 3, 4, 5, and 6). Though infographics had core themes, there was considerable overlap given the conceptual complexity of trust.

  • Infographic 1: evolution in cigarette smoking recommendations (trustworthiness).

  • Infographic 2: space X engineer putting on pants in the morning (scientist-as-person).

  • Infographic 3: changing recommendations about butter/margarine (credibility/consensus).

  • Infographic 4: John Snow and cholera (science as methodology).

  • Infographic 5: relying on a weather forecast (epistemology).

Pilot test methods

Data collection

The procedure for the pilot test was outlined in the published study protocol [15].

Data were obtained on December 19, 2020 from a sample of 100 US-based Amazon Mechanical Turk (mTurk) users ages 18 and older (individuals must be age 18+ to enroll as a mTurk worker). To ensure data quality, minimum qualifications were specified to initiate the survey (task approval rating > 99%, successful completion of more than 100, but fewer than 10,000 tasks, US-based IP address). Checks were embedded in the first part of the survey to control for dishonest workers, survey response bots or virtual private network users, and inattentive participants [17, 18]. Failing at these checkpoints resulted in the termination of the task and exclusion from the study, and participants were warned of this possibility on the study information page. Participants who successfully completed the study were compensated $0.61 USD. In the process of collecting 100 responses from workers, one additional worker refused consent, and 48 additional workers began the survey but were excluded prior to randomization for failing a quality check.

Procedures and instrument

Eligible workers completed the trust in science inventory, which consists of 21 Likert-type items yielding a mean score from 1 (low trust) to 5 (high trust) [16] and then were randomized with equal allocation to view one of the five infographics (n = 20 per arm, though due to simultaneous survey participation, infographic 4 had 21 workers and infographic 5 had 19). Participants were required to pause for at least one minute while viewing the infographic to loosely replicate uptake from multiple, but much shorter, exposures that would occur through social media. After viewing the infographic, workers were asked a qualitative question about the infographic’s meaning [19] and then were asked to complete a modified version of the narrative believability scale (nbs-12), which consists of 12 Likert-type items that produce a mean score from 1 (low believability) to 7 (high believability) [20]. Finally, workers completed the trust in science inventory a second time.

Analysis plan

Mean changes in trust in science between pretest and posttest were analyzed separately for each infographic using paired sample t-tests with unadjusted alpha set at .05. Differences in narrative believability between the five infographics were assessed using a one-way between-subjects analysis of variance (ANOVA), with Tukey’s HSD selected as a post-hoc test if the main effect was significant. All analyses were completed in SPSS v27.

Qualitative data were interpreted using a general inductive approach [21], with a primary focus on whether the participant described the infographic in such a way that it was clear that they understood the intended meaning.

Pilot test results

Trust in science

The trust in science inventory was reliable at pretest (α = 0.940) and posttest (α = 0.946) for the full sample. The mean level of trust at pretest was 3.79 (SD = 0.67), and at posttest was 3.86 (SD = 0.66).

All study arms reported higher trust in science at posttest than at pretest (ranging from mean differences of 0.03 to 0.12, see Table 1), but only one within-arm difference was statistically significant (infographic 4, t(20)=− 2.11, p = 0.048, 95% CI of Diff: 0.001 to 0.239).

Table 1 Pretest-posttest comparison of trust in science scores
Full size table

Narrative believability

The nbs-12 instrument was reliable for the full sample (α = 0.916). Each of the infographics had reasonably high believability, ranging from a low of 5.27 for infographic 1 to a high of 5.97 for infographic 2 (see Table 2). A one-way between-subjects ANOVA did not indicate any significant differences in mean narrative believability by infographic (F(4, 95) = 1.71, p = 0.154).

Table 2 Narrative believability scores
Full size table

Qualitative results

The infographics were designed to convey specific meanings related to subconstructs of trust in science. Thus, responses where participants described the infographic in a way that reflected the message that we intended to communicate were marked as being ‘consistent,’ and those that shared other messages were marked as ‘inconsistent.’ In total, 25 of the 100 responses were determined to be inconsistent, split among: infographic 1 (n = 10), infographic 2 (n = 5), infographic 3 (n = 3), infographic 4 (n = 6), and infographic 5 (n = 1).

Exemplars of consistent and inconsistent responses, by infographic, are available in Table 3. In some cases, participants focused on the image itself rather than the meaning. For example, the intended emphasis of infographic 1 was on how doctors’ recommendations about cigarette smoking evolved due to new scientific evidence, but multiple people conflated scientists and doctors and focused on the medical recommendation rather than the reason for it. In other cases, it appeared that the narrative message was unclear, and some workers were not sure how to interpret an infographic. “I’m honestly not really sure what it was trying to communicate…” Finally, in a few cases, participants addressed the infographic’s meaning, but derived context or content outside of what we intended to communicate. This was observed most notably for infographic 4, which depicted John Snow and cholera: “The scientist came up with a theory and used a silly observation of what was seen to prove a theory, to which the scientific community agreed because of the scientist’s identity. Basically, it was an appeal to authority.” While the respondent identified the actors as scientists, they interpreted the message as implying that scientists must be trusted because they are scientists, and not because they have provided evidence to support their claim (which was the opposite of the intended message).

Table 3 Exemplars of qualitative data
Full size table

The other 75 responses reflected at least partial understanding of our intended message without any additional unintended content (see Table 3).

Pilot test discussion

The primary purpose of the pilot test was to assist our research team in selecting an infographic to use as part of a larger randomized trial [15]. As prespecified, no single analysis (change in trust, narrative believability) was to be interpreted as a sole means of determining which infographic to select for the upcoming trial. Further, the quantitative results were to be interpreted in tandem with the qualitative data.

Quantitative

The pilot study was not powered to test for significant differences at pre- and post-test for the different infographics (and even then, within-group changes such as those we presented do not allow for inference of causality). Instead, our goal with those items was to check for any signs of potential iatrogenic changes (e.g., trust scores decreasing from pre- to post-test), which we did not observe, and to examine general trends.

The narrative believability score was not significantly different across arms—on a scale ranging from one to seven, the range of means was narrow, from 5.27 to 5.97, indicating generally good believability. Those scores, along with subscale variability (not shown, but available via the data and syntax), were conceptually consistent with other research on narrative believability [22]. Thus, no infographics were inherently eliminated from consideration due to the quantitative data alone.

Qualitative

For multiple infographics (#1, 2, and 4), when respondents were asked to describe the meaning of the infographics in their own words, between 5 and 10 of participants veered away from the messages we were attempting to communicate. As a result, the qualitative evidence was weighted in support of infographics 3 and 5, for which most responses (17 and 18 descriptions, respectively) indicated that we successfully communicated our message (see Additional files 3, 4, 5, and 6).

On closer examination, though, we wondered whether the presence in infographic 5 of additional text, relative to other infographics, may have inflated the prevalence of consistent descriptions for that infographic. Because epistemology (the conceptual target of the infographic 5) is a complex concept, we felt that this extra text was necessary. However, some responses describing infographic 5 that were classified as ‘consistent’ contained direct restatements of provided text. As a result, it was unclear to us whether the high frequency of accurate restatement reflected an understanding of the message in the infographic or rote repetition of written text.

Conclusions

The quantitative data didn’t strongly make the case for any specific infographic, and the infographic with arguably the “best” quantitative case (infographic 4) also appeared to create uncertainty and even oppositional interpretation. Infographics 3 and 5 both performed well qualitatively, but we were somewhat concerned that infographic 5’s qualitative performance may have been artificially high. As a result, we made the difficult decision to adopt infographic 3 for our larger study, though we note that a case could be made for infographics 4 and 5 as well, and encourage research and exploration of those files, which we have released alongside this note.

Limitations

This cross-sectional pilot study was intended to select an infographic to be used as part of a larger randomized trial. As such, it was designed to provide exploratory insight into five infographics, but not to draw widely generalizable conclusions. Further, the breadth of infographics that we designed and tested was limited by our own belief that the messages should be trustworthy—that is, regardless of the goals of the study, our intention was that the messages communicated by the infographics should be things that are true, even if the possibility exists that exaggerated claims could produce larger effects.