The COVID-19 pandemic (and its aftermath) highlights a critical need to communicate health information effectively to the global public. Given that subtle differences in information framing can have meaningful effects on behavior, behavioral science research highlights a pressing question: Is it more effective to frame COVID-19 health messages in terms of potential losses (e.g., “If you do not practice these steps, you can endanger yourself and others”) or potential gains (e.g., “If you practice these steps, you can protect yourself and others”)? Collecting data in 48 languages from 15,929 participants in 84 countries, we experimentally tested the effects of message framing on COVID-19-related judgments, intentions, and feelings. Loss- (vs. gain-) framed messages increased self-reported anxiety among participants cross-nationally with little-to-no impact on policy attitudes, behavioral intentions, or information seeking relevant to pandemic risks. These results were consistent across 84 countries, three variations of the message framing wording, and 560 data processing and analytic choices. Thus, results provide an empirical answer to a global communication question and highlight the emotional toll of loss-framed messages. Critically, this work demonstrates the importance of considering unintended affective consequences when evaluating nudge-style interventions.
Managing the COVID-19 pandemic (and its aftermath) hinges in part on effectively communicating health messages to the global public. One critical question is how to frame such messages, given widespread evidence from psychology and related fields that the way in which information is framed can have meaningful effects on behavior, even when the core information is essentially the same across distinct frames (for reviews, see Gallagher & Updegraff, 2012; Rothman et al., 2020). Indeed, in their widely cited review recommending social and behavioral science applications for reducing the spread of COVID-19, Van Bavel et al. (2020) highlighted this very question: “Research is needed to determine whether a more positive [vs. negative] frame could educate the public and relieve negative emotions while increasing public health behaviors” (p. 462). More generally, Sunstein and Thaler (2003, p. 1182) have long argued that “In order to be effective, any effort to inform people must be rooted in an understanding of how people actually think. Presentation makes a great deal of difference: The behavioral consequences of otherwise identical pieces of information depend on how they are framed.” In their view, framing constitutes a potentially powerful nudge—i.e., a way of altering people’s behavior in a predictable way without changing the underlying incentives (Thaler & Sunstein, 2009; see also de Bruin & Bostrom, 2013; Downs, 2014).
In the case of COVID-19 health messaging, communicators could emphasize either (a) the benefits of compliance (i.e., gain framing) or (b) the costs of non-compliance (i.e., loss framing) with recommended actions. For example, as depicted in Fig. 1, the United States Centers for Disease Control and Prevention (CDC) website (perhaps unintentionally) framed messages in terms of gains, asking the public to: “Wear a mask. Save lives” (CDC, 2021). However, an alternative loss framing might have said: “If you do not wear a mask, lives may be lost.”
Given the ability of news media, national and international health organizations, and political leaders to reach wide audiences, message framing effects could save a substantial number of lives with limited implementation costs. With this possibility in mind, we conducted an experiment to test the effect of loss- versus gain-framing of COVID-19-related public health messages on behavioral intentions, policy attitudes, and information seeking among participants in 84 countries during the pandemic. Moreover, we sought to assess the potential benefit of changes on those outcomes against the potential emotional costs that loss (vs. gain) framing might elicit.Footnote 1 Prior studies suggest that loss frames (versus gain frames) are associated with relatively more global negative than positive affect (Nabi et al., 2020; Gosling et al., 2020). Here, we chose to examine whether loss (versus gain) framing would increase participants’ anxiety, in particular, given that framing effects on anxiety have received little to no empirical attention and that anxiety has the potential to trigger significant health burdens.
Anxiety, “an emotion characterized by feelings of tension, worried thoughts, and physical changes like increased blood pressure” (American Psychological Association, 2021), may take the form of a temporary state, a chronic trait-like tendency, or a clinical disorder.Footnote 2 Anxiety has been linked with leading causes of human morbidity and mortality. For example, heightened anxiety is linked to increased risk of cardiovascular disease mortality and morbidity (e.g., heart disease, stroke, and heart failure; Levine et al., 2021). It has also been linked to increased reactivity to losses (Hartley & Phelps, 2012; Xu et al., 2013) and increased stress hormone secretion (i.e., cortisol), which, when chronic, diminishes immune function and complicates individuals’ ability to cope with stress (for review, Taylor, 2021). Moreover, the effect of anxiety on stress hormone secretion may worsen with age (ó Hartaigh et al., 2012; Otte et al., 2005), potentially putting elderly individuals who already face heightened risks from COVID-19 in an even more vulnerable position. While the anxiety triggered by exposure to public health messages is likely mild compared to the levels associated with a clinical disorder, any potential behavioral benefit from message framing must still be weighed against a potential emotional cost (intended or otherwise).
Given the global nature of the pandemic, it is critical to assess the generalizability of message framing effects on a global scale. Traditionally, psychological research on human behavior includes sample populations in western, educated, industrialized, rich, and democratic societies (i.e., WEIRD societies; Henrich et al., 2010a, 2010b). However, extrapolating from studies conducted in only a single location may miss meaningful cross-regional variation in effects. Consequently, this can lead to incomplete—and even potentially detrimental—policy recommendations. Thus, rather than assume generalization from a single population, research that aims to inform global policy recommendations during COVID-19 should incorporate a global sample (c.f., Bauer, 2019).
We launched a global participant recruitment effort between April and September 2020, collecting data in 48 languages from 15,929 participants in 84 countries.Footnote 3 Participants were recruited by (1) research groups affiliated with the Psychological Science Accelerator (PSA; Moshontz et al., 2018) and (2) semi-representative research panels. The present experiment was bundled with another experiment—also conducted in collaboration with the PSA, but led by an independent research group—that assessed the relative effects of autonomy-supportive messages vs. controlling messages on motivation and behavioral intentions relevant to COVID-19. Participants completed both experiments in a randomized order after completing a pre-study survey that included demographic questions (for full wording of all questions from the pre-study survey and relevant descriptive statistics, see Table 1). The order of the study (first vs. second) did not have a main effect on any of the dependent variables, although there was one higher-order interaction with self-reported anxiety (described below). A third experiment investigated the effect of cognitive reappraisal, an emotion regulation strategy, and was conducted concurrently by the PSA with a different sample of participants (Wang et al., 2021).
In the present experiment, participants were randomly assigned to read COVID-19 health recommendations adapted from World Health Organization (WHO) advisories (e.g., social distancing, mask wearing) that were framed in terms of losses (e.g., “if you do not practice these four steps, you can endanger yourself and others”) or gains (e.g., “if you practice these four steps, you can protect yourself and others”). To ensure that any observed effects arose from meaningful conceptual differences (as opposed to particular wording; see Wells & Windschitl, 1999), we also examined three variations of the framed messages (described below). These variations of the framed messages were designed to assess generalizability of loss vs. gain framing across different wordings. As such, the differences in wording are relatively minor compared to the more central manipulation of loss vs. gain framing. Thus, participants were randomly assigned to one of six between-subjects experimental conditions that varied both the framing and wording/version of the COVID-19 health recommendation.
Following the message framing manipulation, we measured four outcome variables: (1) behavioral intentions to follow guidelines to prevent COVID-19 transmission, (2) attitudes toward COVID-19 prevention policies, (3) whether participants chose to seek more information about COVID-19, and (4) self-reported anxiety. Seeking to create conditions under which one might detect any systematic effect of framing, we selected scale responses concerning behavioral intentions and information seeking as outcome variables. We selected attitudes toward COVID-19 prevention policies because garnering citizen support for public policies is a critical ingredient in successfully combating the COVID-19 pandemic. Finally, we measured self-reported anxiety to assess the extent to which message framing may trigger unintended affective consequences, beyond traditional behavioral or policy outcomes.
Psychological Science Accelerator (PSA) COVID-19 Rapid Project
We conducted the present experiment as part of a larger PSA COVID-19 Rapid Project, which involved one pre-study general survey and three experiments related to COVID-19 (Forscher et al., 2020). The study and the experiments were presented online through the formR survey platform (Arslan et al., 2020). The present experiment was bundled with another experiment, both of which participants completed in random order after completing the pre-study general survey that included questions about beliefs and behaviors related to COVID-19.
Sample size was primarily determined by the availability of resources among members of the PSA. Nevertheless, results from an a-priori power simulation estimating power as a function of number of countries, number of participants per country, intraclass correlations, effect sizes, and between-country variability in effect sizes can be found at https://osf.io/m6q8f/. After excluding data from participants who (a) had corrupted data due to technical difficulties, (b) did not provide responses to our outcomes of interest, or (c) did not indicate their country of origin, we were left with data from 15,929 participants (62% female, 37% male, 1% other or non-response, < 1% other; Mage = 33.70 , SDage = 14.45), who lived in 84 different countries and completed the survey in a total of 48 languages. Participants were recruited either through semi-representative research panels (n = 5,555) or by PSA research groups (n = 10,374; see Forscher et al., 2020, for more details on sampling and translations). The survey was conducted during the Spring and Summer of 2020.
Participants were randomly assigned to view loss- or gain-framed versions of four recommendations related to COVID-19 adapted from the WHO in Spring 2020. These recommendations related to (1) staying home (unless absolutely necessary), (2) avoiding all shops other than necessary ones (such as for food), (3) wearing a mouth and nose covering in public at all times, and (4) completely isolating if exposed to COVID-19. All participants viewed four similarly worded recommendations—but were randomly assigned to view either a loss- or gain-framed message. To examine whether our conclusions generalize across multiple variants of framed messages, we created three different versions of each frame (see Wells & Windschitl, 1999, for more information on the importance of this stimulus sampling approach). Thus, the experiment took the form of a 2 (Framing: gain, loss) × 3 (Version: Version 1, Version 2, Version 3) between-subjects factorial design, featuring the following messages:
Gain/Version 1: “There is so much to gain. If you practice these four steps, you can protect yourself and others.”
Gain/Version 2: “You have so much to gain. You can protect yourself and others if you practice these four steps.”
Gain/Version 3: “There is so much to gain. Practicing these four steps can help you stay healthy and protect the health of others.”
Loss/Version 1: “There is so much to lose. If you do not practice these four steps, you can endanger yourself and others.”
Loss/Version 2: “You have so much to lose. You can endanger yourself and others if you do not practice these four steps.”
Loss/Version 3: “There is so much to lose. You can get sick and endanger the health of others if you do not practice these four steps.”
The four recommendations and dependent variables were displayed for all participants, with the message frame and version type varied by condition. The manipulated message appeared at the top of the pages displaying each recommendation and instructions when completing the outcome variables.
At the end of the survey, participants completed a manipulation check. We asked participants which of the following phrases, if any, they recalled reading during the survey: (a) There is so much to gain. You can stay healthy and protect others by...; (b) There is so much to lose. You can avoid losing your health and avoid endangering others by...”; or (c) neither. Exact wording varied to match the precise wording across the six conditions.
After reading the four recommendations (with message framing varied by condition), participants completed three self-report questionnaires: behavioral intentions to follow guidelines to prevent COVID-19 transmission, attitudes toward COVID-19 prevention policies, and self-reported anxiety (described below). Afterwards, participants completed a behavioral measure, wherein they indicated whether they would be interested in learning more information about safe practices regarding COVID-19 (and were thus directed to the WHO website). Full wording of all items are presented in Table 2. While the questions themselves were identical across conditions, participants received different instructions depending on their randomly-assigned condition. For example, for the behavioral intention questionnaire, participants in the gain/version 1 condition saw: “Stay healthy and protect others. There is so much to gain. We are interested in how you yourself will respond in the coming week in order to stay healthy and protect others.” Participants in the loss/version 1 condition saw: “Avoid losing your health and avoid endangering others. We are interested in how you yourself will respond in the coming week in order to avoid losing your health and avoid endangering others.” The presentation order of the dependent variables was held constant for all participants.
For the outcome variables, we created ad-hoc face-valid measures and relied on exploratory analyses to assess internal consistency and convergent validity (see “Results” section).Footnote 4 Participants first indicated their intentions to engage in a variety of COVID-19 preventative behaviors (adapted from WHO recommendations at the time of survey launch in Spring 2020). Specifically, participants indicated how likely they were to (1) stay at home at all times unless absolutely necessary, (2) avoid all shops other than necessary ones (such as for food), (3) wear a mouth and nose covering (such as a mask) in public at all times, and (4) completely isolate themselves if they think they have been exposed to COVID-19. The four questions were presented in a randomized order and all responses were on a 7-point scale (1 = Extremely unlikely to 7 = Extremely likely).
Of note, we observed an unexpected J-shaped distribution in behavioral intentions—wherein a large majority of participants indicated very strong intentions to engage in protective behaviors (M = 6.47, SD = 0.91 on a 7-point scale). In the SI, we discuss potential explanations for, and additional analyses regarding, the restriction of range. Despite the restriction of range (and thus smaller-than-expected variation in the measure), behavioral intentions were still correlated with other variables in the convergent validity analyses (rs from .04 to .35; described in “Results” section below). Furthermore, we did not observe a restriction of range in the other continuous outcomes: attitudes about policies that empower individuals (M = 3.46, SD = 1.93 on a 7-point scale), attitudes about policies that extend government power (M = 5.67, SD = 1.31 on a 7-point scale), and anxiety (M = 2.44, SD = 1.17 on a 5-point scale). Concerns about restrictions of range also were not applicable to the measure of information seeking (25% no, 75% yes).
After responding to the behavioral intention items, participants reported their attitudes toward five statements regarding COVID-19 prevention policies. The policy attitude items focused on trade-offs between individual rights and collective security. Two statements emphasized individual rights and autonomy (e.g., “Individuals, not governments, should decide how best to act during the COVID-19 pandemic”), whereas the other three statements emphasized collective security (e.g., “Government health officials should do everything in their power to address the spread of COVID-19, even if it severely limits daily activities for citizens”). The five questions were presented in a randomized order and all responses were on a 7-point scale (1 = Strongly disagree to 7 = Strongly agree).
Next, the survey asked participants to indicate the extent to which they felt anxious, afraid, and fearful when considering the COVID-19 health recommendations. The three questions were presented in a randomized order and all responses were on 5-point scales (1 = Not at all to 5 = Extremely).
Last, participants were asked if they would like to learn more information about COVID-19. (All participants, regardless of stated preference, received additional information about COVID-19 at the end of the study.) A one-item question asked participants: “At the end of the study today, would you like to learn the latest reliable information about COVID-19?” The dependent variable was assessed as a binary variable (Yes, No).
All participating research groups either obtained approval from their host institution’s ethics committee, indicated that their institution did not require approval to conduct this type of experiment, or indicated that the experiment was covered by a preexisting ethics approval. All participants provided informed consent.
First, we report a set of preliminary analyses concerning the manipulation check, internal consistency of scales, and convergent validity among variables. Next, we report the results of our inferential analyses. Finally, we report additional exploratory analyses regarding anxiety. Data, code, materials, power simulation details, and the pre-registered analysis plan for this experiment are available at https://osf.io/m6q8f/.
Results revealed that 73% of participants correctly identified their condition from among three different response options (gain message, loss message, or neither). In order to be conservative, and to keep with our pre-registration plan, we reported results with the full (Intent to Treat) sample even though 27% of participants did not correctly identify which treatment they received. Importantly, however, the pattern of results was similar when we restricted the sample to just the portion of the sample that passed the manipulation check (see SI for more information).
Internal Consistency of Outcome Measures
Internal consistency for both the four-item behavioral intention and three-item self-reported anxiety measures was appropriate (α > .78, average inter-item r > .47). The internal consistency of the five-item policy support measure, however, was lower than expected (α = .67; average inter-item r = .29). Thus, per our pre-registration plan, we performed an exploratory factor analysis. This exploratory factor analysis used varimax rotation and a minimal residual factoring method to identify two distinct subgroups of items: support for (1) policies that empower individuals to make decisions about COVID-19 (two items; α = .74; average inter-item r = .59), and (2) policies that extend governments’ ability to stop the spread of COVID-19 (three items; α = .77; average inter-item r = .53). These two scales were weakly and negatively correlated (r = − .15, p < .001), and we analyzed the two subscales separately. Our behavioral measure of information-seeking was a single item and thus internal consistency analyses are not applicable.
Convergent Validity of Outcome Variables
We examined the extent to which our outcome measures were associated with conceptually-related variables. To do so, we (a) post-hoc identified conceptually-related variables from the pre-study general survey, and (b) examined the extent to which they were associated with the outcome variables. Notably, these general survey items were administered before the present study (and thus were not affected by participants’ experience in the study). In all cases, we observed associations in the anticipated direction (ps < .001) that ranged from very small (|r| = .04) to medium (|r| = .35) in size. For example, behavioral intentions were positively associated with the self-reported number of times that participants had recently worn a mask (r = .28, p < .001; see SI for more detail).
We first modeled each outcome variable using linear (for continuous variables) or logistic (for dichotomized variables) mixed-effects regression with message framing entered as an effect-coded factor, country-level random intercepts, and country-level random slopes. For all outcomes besides behavioral intentions, country-level random slopes led to singular fits and were subsequently removed. These convergence issues provided preliminary evidence that the estimated effects of message framing on our outcomes of interest were consistent across countries. To facilitate comparisons across outcomes, we also estimated the overall message framing effects using random-effects meta-analysis. For the meta-analysis, we used Cohen’s d as the effect size index, wherein positive values indicated higher levels of the outcome variables in the loss- (vs. gain-) framed conditions.Footnote 5
Effects on Behavioral Intentions, Policy Support, and Information Seeking
Our first set of analyses tested the effect of message framing on behavioral intentions, attitudes toward two types of policies, and information seeking. Results indicated that framing messages in terms of losses vs. gains had extremely small, non-significant effects on (1) intentions to engage in protective behavior a 0.03 increase on a 7-point scale; F(1, 35.17) = 2.70, p = .110, d = 0.03, 95% CI [− 0.01, 0.07], τ2 = 0.005; (2) support for policies that empower individuals to make decisions about COVID-19 a 0.01 increase on a 7-point scale; F(1, 15871) = 0.05, p = .826, d = 0.004, 95% CI [− 0.03, 0.04], τ2 ≈ 0; (3) support for policies that extend governments’ ability to stop the spread of COVID-19 a 0.04 increase on a 7-point scale; F(1, 15877) = 3.46, p = .063, d = 0.03, 95% CI [0.002, 0.06], τ2 ≈ 0; and (4) the probability that participants sought additional information about COVID-19 (a 1.2% point decrease; z = − 1.80, p = .071, d = − 0.008, 95% CI [− 0.02, 0.004], τ2 ≈ 0). Notably, the low τ2 values suggest that the estimated effects of message framing on our outcomes of interest were consistent across countries (see Fig. 2).
While we found little evidence of between-country heterogeneity in the effects of message framing on behavioral intentions, attitudes, and information seeking, we next examined whether these estimated effects were moderated by methodological features of the study, such as (a) the version of the framed message (versions 1–3), (b) the sampling pool (panel, non-panel), and (c) the order in which participants completed the two bundled studies (present experiment first, present experiment second). To do so, we separately added each moderator-of-interest and its higher-order interaction with message framing as effect-coded factors in the mixed-effects models described above. Results did not indicate that the message framing effects interacted with any of the moderators of interest (ps > .138).
To probe the robustness of the estimated effects of message framing on behavioral intentions, attitudes, and information seeking, we performed exploratory multiverse analyses (also sometimes described as a specification-curve analysis; Simonsohn et al., 2020; Steegen et al., 2016).Footnote 6 The present multiverse analyses examined how 398 justifiable approaches to data processing and modeling affected our conclusions. Most approaches indicated that message framing did not impact intentions to engage in protective behavior (87% of models) or support for COVID-19-related policies (67% of models). In the scenarios where the estimated message framing effects were significant, the magnitudes were extremely small (i.e., less than a 0.06 change on a 7-point behavioral intentions measure; less than a 0.20 change in a 7-point policy support measure). Many justifiable data processing and analysis approaches did indicate that framing messages in terms of losses (vs. gains) decreased information seeking (80% of models). However, in these scenarios, the magnitude was small (i.e., less than a 4% point decrease in the probability of seeking information; see SI for more information).
Effects on Self-Reported Anxiety
The next set of analyses examined whether loss-framed vs. gain-framed messages had a differential impact on self-reported anxiety. Results indicated that participants reported higher levels of anxiety after being exposed to loss- (M = 2.58, SD = 1.18) vs. gain-framed (M = 2.30, SD = 1.14) messages, F(1, 15881) = 253.67, p < .001, d = 0.25, 95% CI [0.21, 0.29], τ2 = 0.007. Once again, the low τ2 value suggests that the estimated effect of message framing on anxiety was consistent across countries (see Fig. 2).
To assess these anxiety results in terms of practical perspective, we estimated the association between (a) self-reported personal exposure to COVID-19 (a presumably anxiety-producing event that was measured as a binary variable in the pre-study survey), and (b) experienced anxiety after the framing manipulation. The estimated effect of message framing on anxiety was nearly 1.5 times the size of the estimated association between actual exposure to COVID-19 and anxiety (which was associated with a 0.19 increase on the 5-point anxiety measure). Thus, in practical terms, the effect of message framing on anxiety appeared substantial. That being said, comparing the size of these relationships could be complicated by the fact that people who were exposed to COVID-19 and avoided negative outcomes could have decreased (rather than increased) anxiety.Footnote 7 Future research is needed to further benchmark the relative size of loss- vs. gain-framing on self-reported anxiety.
Similar to the analyses of the other outcome variables, we next examined whether the estimated effect of framing on anxiety was moderated by methodological features of the study. Results did not indicate that the effect of message framing on anxiety was moderated by the version of the message (p = .368) or the sampling pool (p = .799). This implies that the underlying construct itself (loss vs. gain framing), rather than the particular wording associated with any instantiation of it, drives the effects. Inconsequentially, the message framing effect was moderated by the order in which participants completed the study, F(1, 15880) = 4.35, p = .037. Follow-up contrasts indicated that the effect of framing on anxiety was slightly larger when participants completed our study second (where message framing led to a 0.32 shift on the 5-point anxiety measure) vs. first (where message framing led to a 0.24 shift in the anxiety measure). It could be the case that completing the other study first (which also asked participants to read COVID-19 health messaging) heightened attention to COVID-19, and thus magnified the anxiety effects observed in the present data. Importantly, however, the observed effect of message framing on anxiety was significant regardless of the order of the studies (both ps < .0001) and the moderation by study order was relatively inconsequential in size compared to the overall effect of loss- vs. gain-framing.
Finally, we conducted a multiverse analysis to examine how 162 justifiable approaches to data processing and modeling affected our conclusions about anxiety. Strikingly, all 162 justifiable data processing and modeling approaches examined in the multiverse analysis indicated that framing messages in terms of losses (vs. gains) significantly increased anxiety (all ps < .001; all mean differences > 0.21). These results suggest that the inferences regarding the effects of message framing on anxiety are robust across a wide variety of justifiable analytic decisions.
Additional Analyses Regarding Pre-Study Worry
Our analyses to this point have examined anxiety in response to the framed messages. However, the pre-study survey also included two items assessing anxiety-relevant states: worry regarding one’s physical and emotional health. Both items were moderately correlated (r = .58) and answered on 5-point scales (1 = Not at all worried, 5 = Extremely worried). For simplicity, we averaged the two items and refer to this combined index as pre-study worry. (Statistical significance of results remains unchanged when we analyze the two items separately.)
In order to be maximally comprehensive, we conducted a set of exploratory (post-hoc) analyses concerning whether loss (vs. gain) framing would exert differential effects on any of the four outcome variables for individuals higher (vs. lower) on pre-study worry. That is, we tested whether pre-study worry moderated any of the message framing effects documented above. To test this possibility, we modeled each outcome variable with (a) frame entered as an effect-coded factor, (b) pre-study worry entered mean-centered, (c) their higher-order interaction, and (d) random intercepts for country. For behavioral intentions, policy support, and post-study anxiety, we used linear mixed-effect models; for information seeking, we used a logistic mixed-effect model. For all outcomes, there was not a significant interaction between message framing and pre-study worry (p > .43), suggesting that the effect of message framing did not depend on levels of pre-study worry.
While framing messages in terms of loss (versus gain) conferred little-to-no measured benefits, such loss framing exerted moderately sized and extremely consistent costs in terms of increased state anxiety (see Fig. 2). Moreover, the results for anxiety appeared consistent across countries, message wording, sampling pool, study order, and analytic choices—increasing confidence about generalizability.
The COVID-19 pandemic (and its aftermath) highlights a critical need to effectively communicate health information to the global public. It also highlights the importance of rapidly testing psychological interventions on a global scale. We experimentally tested the differential effects of framing messages in terms of losses vs. gains on COVID-19-related behavioral intentions, policy attitudes, information seeking, and experienced anxiety.
Results indicated that message framing had little-to-no measurable benefit for behavioral intentions, policy attitudes, or information seeking, but did have a significant emotional cost in terms of increased anxiety. These results were consistent across 84 countries, three variations of the message framing wording, across semi-representative and non-representative samples, across survey order, and across 560 data processing and analytic choices. Taken together, these results imply that the conceptual difference between loss- and gain-framing accounts for its effect on anxiety (rather than any particular phrasing of stimuli, culturally specific connotation, methodological feature, or data analytic approach).
The effect of message framing on anxiety when reading loss- vs. gain-framed health recommendations was nearly 1.5 times the size of the association between self-reported personal exposure to COVID-19 and anxiety when reading the health messages, revealing the important practical impact of loss framing. Because heightened anxiety has been associated with major causes of morbidity and mortality, diminished coping abilities, and neuroendocrine dysregulation, the heightened levels of anxiety under loss-framed messages represent an important outcome. Of course, the anxiety triggered in our study was relatively mild compared to acute levels associated with clinically-diagnosable anxiety disorders. Indeed, the average post-treatment anxiety was quite low in both framing conditions (2.58/5 and 2.30/5 for the loss and gain conditions, respectively). Nevertheless, public health communicators should benefit from learning that gain-framed messages COVID-19 messages are at least as effective as loss-framed messages in their impact on behavioral intentions, policy attitudes, and information seeking behavior—but induce significantly less anxiety at a population level.
While some commentators have urged organizations to “scare people” when communicating COVID-19 health information (e.g., in the New York Times; Rosenthal, 2020), the present results cast doubt on the wisdom of reminding people how much they stand to lose during the pandemic. Despite eliciting higher levels of anxiety, loss-framed (vs. gain-framed) messages did not meaningfully change behavioral intentions, information seeking behavior, or policy attitudes in the context of COVID-19. Admittedly, literature on fear appeals is nuanced (e.g., Kok et al., 2018; Peters et al., 2018). But because the present study is the largest and most globally representative study ever conducted on message framing and anxiety, there is compelling evidence that triggering anxiety through COVID-19 messaging does not improve behavioral intentions, attitudes, or actual behavior—at least in this context.
More generally, the present results contribute to a nascent literature broadening the scope of behavioral decision (nudge-style) interventions beyond strictly behavioral outcomes. Fields such as public health and health psychology have long considered affective states to be crucial outcome variables in and of themselves (e.g., Epel et al., 2018; Mikels et al., 2016; Taylor, 2021). The field of communication has also begun to consider affect as both an outcome itself and as a mediator of behavioral outcomes (Hameleers, 2021; Wong et al., 2013; Nabi et al., 2020). In the present work, we build both on these fields, and on emerging literature in behavioral decision research (Allcott & Kessler, 2019; Haushofer et al., 2021; Zlatev & Rogers, 2020), to propose that emotional consequences should be considered when evaluating the costs vs. benefits of nudge-style interventions (c.f., Glaeser, 2005).Footnote 8 In the present case, under an expanded cost-benefit analysis that includes emotional consequences (c.f. Dukes et al., 2021), messages framed in terms of gains appear superior (for related discussion, see Loewenstein & O'Donoghue, 2006).
Limitations and Future Directions
Despite its global scope, the present experiment features some methodological limitations. First, it remains unknown whether sustained framing interventions (rather than single shot) could have stronger effects. Given that the measures rely on self-report and that the anxiety effects are measured immediately (rather than over time), it is unclear to what extent such effects would persist outside of the specific experimental context. Second, the behavioral intentions variable exhibited restriction of range, which may have contributed to diminishing a message framing effect. However, behavioral intentions had sufficient variance to correlate with other expected predictors in the study (e.g., self-reported mask wearing), providing some evidence that the range was not sufficiently limited to preclude the detection of meaningful relationships. Moreover, we did not observe restrictions of range on policy attitudes and information seeking (variables that we similarly did not find affected by message framing).
A few future directions merit note. Most centrally, future research is needed to understand the lack of differential effects of loss vs. gain message framing on behavioral intentions, policy support, and information seeking. Perhaps the strongest explanation for why loss-gain framing shows substantial effects in other contexts—but not here—is because the present set-up differs substantially from classic loss-gain work on risk preferences. In canonical risk preference paradigms (e.g., Dorison & Heller, 2022; Kahneman & Tversky, 1979; Ruggeri et al., 2020; Tversky & Kahneman, 1991), participants are confronted with choices between a sure option and a risky gamble. Importantly, probabilities for each option are provided. Prior research identifies a robust effect that generalizes across contexts: people are typically risk-averse when the choice options are presented as losses but risk-seeking when choice options are presented as gains. Our paradigm intentionally deviated from this large body of research on loss-gain framing effects on risk preferences. In the present paradigm, probabilities were unknown and participants were not presented with a choice between a sure option and a risky gamble because it would have been unrealistic to provide known probabilities about the pandemic. Thus, the present paradigm follows more directly from research in the health psychology literature that compares health actions associated with gains (e.g., wearing sunscreen to clear skin) vs. inaction associated with losses (e.g., not wearing sunscreen to skin cancer). This literature has yielded mixed results (Rothman & Salovey, 1997; Rothman & Sheeran, 2021) for the effects of framing, suggesting that key moderators remain to be identified (for reviews, see Levin et al., 1998; van’t Riet et al., 2016).
There is at least one study, however, that used a reasonably comparable paradigm but which found divergent results: Abhyankar et al. (2008) found a loss-frame advantage on intentions to obtain the MMR vaccine for one’s child. It could be the case that the effects of loss- vs. gain-message framing differ when assessing health intentions for oneself vs. another person, especially when the other person is a child under one’s care. Additional possibilities include that there may be something specific about an unfolding (and highly uncertain) pandemic that blunted such effects or that the gain/loss manipulations were weaker in the present study.
Four additional future directions merit note. First, following from the point above, while we found limited heterogeneity by country, future research could explore heterogeneity in the effect of message framing across other dimensions (e.g., such as the tightness vs. looseness of the culture; Gelfand et al., 2021; Uskul et al., 2009). Indeed, it could be the case that our operationalization of country was limited by the manner in which we sampled participants. Second, while we also found limited heterogeneity in the effect of message framing across the different versions of loss and gain framing, future research could examine additional versions of these messages (e.g., self- vs. other-focused messages). Third, while we conducted an initial set of analyses with the pre-study survey (focused on pre-study worry), future research could test a more comprehensive set of hypotheses using these data. Finally, while the present work expanded the scope of nudge-style outcomes beyond behavior to include anxiety, future research is needed to further integrate emotional outcomes (both immediate and long-term) into cost-benefit calculations for implementing nudge-style interventions (e.g., framing). Not only does the subjective experience of emotion matter in and of itself (anxiety creates suffering) but also the myriad effects of emotion on health (e.g., Emdin et al., 2016; Kubzansky & Kawachi, 2020) and health behavior (e.g., Dorison, Wang et al., 2020; Dorison, Klusowski et al., 2020; Ferrer et al., 2020) matter as well.
In a global experiment spanning 84 countries and nearly 16,000 participants, loss vs. gain message framing had a widespread effect on self-reported anxiety while exerting no notable effects on cognitive and behavioral outcomes related to the COVID-19 pandemic. To the extent that policymakers and health organizations aim to minimize anxiety during a pandemic that has engendered high levels of stress and illness, our results provide evidence that gain framing may be superior to loss framing in communicating COVID-19 prevention messages. The results hold theoretical implications for multiple literatures, including research on health message framing, social influence, affective science, and public policy. More generally, the results underscore the lesson that, for policymakers and health organizations, large-scale collaborations can provide empirical answers to global questions (Coles et al., 2022; Forscher et al., in press), freeing communicators from having to rely on either intuition or speculation about applications of theory in particular contexts (c.f., Haushofer & Metcalf, 2020).
Although behavioral decision researchers studying loss vs. gain framing have traditionally examined emotional states to understand their influence on behaviors and attitudes (for reviews, Dorison, Klusowski et al., 2020; Lerner et al., 2015), they have tended to omit emotion as an outcome in nudge-style interventions (i.e., interventions that encourage desirable behavior without restricting choice or introducing economic incentives; Thaler & Sunstein, 2009). For counter-examples, see Allcott & Kessler, 2019; Loewenstein & O'Donoghue, 2006; Zlatev & Rogers, 2020.
Anxiety disorders are ranked as the sixth largest contributor to non-fatal health loss globally and appear in the top 10 causes of years of healthy life lost in all WHO Regions (World Health Organization, 2021). We chose anxiety not only because it was a focal emotional state heightened by the pandemic (Aknin et al., 2021), but also because of its association with negative downstream consequences for coping and for overall health.
For country classification, we relied on standards promoted by the International Organization for Standardization. Nevertheless, we acknowledge the presence of ongoing territory disputes that are not reflected in these standards.
Unfortunately, due to the time pressure to launch this international data collection effort at the onset of the global pandemic, we did not have time to pretest the stimuli for the study.
For dichotomous outcomes (i.e., information seeking), we converted log odds ratios to Cohen’s ds (Borenstein et al., 2009). Countries without at least one observation in each of the conditions were excluded from the meta-analysis.
Such multiverse analyses acknowledge that (1) there are often many justifiable approaches to processing and modeling data, (2) justifiable differences in the processing and modeling of data can change the inferences one might draw from the data, (3) examining different data processing and modeling approaches helps probe the robustness of a set of results, and (4) reporting how different data processing and modeling approaches impact results can improve the transparency and credibility of research findings (Lebel et al., 2018). In the main text, we describe the results of multiverse analysis models that converged. Nevertheless, we describe the results of additional analytic approaches that yielded model convergence issues in the SI.
We thank the review team for this point.
This idea is not new. Economist Jeremy Bentham’s original (1879) conception of utility emphasized happiness as “the greatest good” (for discussion, see Lerner et al., 2022).
Abhyankar, P., O'connor, D. B., & Lawton, R. (2008). The role of message framing in promoting MMR vaccination: Evidence of a loss-frame advantage. Psychology, Health and Medicine, 13(1), 1–16.
Aknin, L., De Neve, J. E., Dunn, E., Fancourt, D., Goldberg, E., Helliwell, J., ... & Amour, Y. B. (2021). A review and response to the early mental health and neurological consequences of the COVID-19 pandemic. PsyArXiv [Preprint]. 10.31234/osf.io/zw93g. Accessed 7 May 2021.
Allcott, H., & Kessler, J. B. (2019). The welfare effects of nudges: A case study of energy use social comparisons. American Economic Journal: Applied Economics, 11(1), 236–276.
American Psychological Association (2021), Anxiety. https://www.apa.org/topics/anxiety/. Accessed 3 May 2021.
Arslan, R. C., Walther, M. P., & Tata, C. S. (2020). formr: A study framework allowing for automated feedback generation and complex longitudinal experience-sampling studies using R. Behavior Research Methods, 52(1), 376–387.
Bauer, P. J. (2019). Expanding the reach of psychological science. Psychological Science, 31(1), 3–5.
Borenstein, M., Hedges, L. V., Higgins, J. P., & Rothstein, H. R. (2009). A basic introduction to fixed-effect and random-effects models for meta-analysis. Research Synthesis Methods, 1(2), 97–111.
Centers for Disease Control and Prevention (2021). Public Service Announcement. https://www.google.com/search?q=stay+home+save+lives&rlz=1C1GCEB_enUS850US887&oq=stay+home+save+lives. Accessed 4 May 2021.
Coles, N. A., Hamlin, J. K., Sullivan, L. L., Parker, T. H., & Altschul, D. (2022). Build up big-team science. Nature, 601, 505–507.
de Bruin, W. B., & Bostrom, A. (2013). Assessing what to address in science communication. Proceedings of the National Academy of Sciences, 110(Supplement 3), 14062–14068.
Dorison, C. A., & Heller, B. H. (2022). Observers penalize decision makers whose risk preferences are unaffected by loss–gain framing. Journal of Experimental Psychology: General Advance online publication.
Dorison, C. A., Wang, K., Rees, V. W., Kawachi, I., Ericson, K. M. M., & Lerner, J. S., (2020). Sadness, but not all negative emotions, heightens addictive substance use. Proceedings of the National Academy of Sciences, 117(2), 943–949.
Dorison, C. A., Klusowski, J., Han, S., & Lerner, J. S. (2020). Emotion in organizational judgment and decision making. Organizational Dynamics, 49(1), e100702.
Downs, J. S. (2014). Prescriptive scientific narratives for communicating usable science. Proceedings of the National Academy of Sciences, 111(Supplement 4), 13627-13633.
Dukes, D., Abrams, K., Adolphs, R., Ahmed, M. E., Beatty, A., Berridge, K. C., & Sander, D. (2021). The rise of affectivism. Nature Human Behaviour, 5(7), 816–820.
Emdin, C. A., Odutayo, A., Wong, C. X., Tran, J., Hsiao, A. J., & Hunn, B. H. (2016). Meta-analysis of anxiety as a risk factor for cardiovascular disease. American Journal of Cardiology, 118, 511–519. https://doi.org/10.1016/j.amjcard.2016.05.041
Epel, E. S., Crosswell, A. D., Mayer, S. E., Prather, A. A., Slavich, G. M., Puterman, E., & Mendes, W. B. (2018). More than a feeling: A unified view of stress measurement for population science. Frontiers in Neuroendocrinology, 49, 146–169.
Ferrer, R. A., Taber, J. M., Sheeran, P., Bryan, A. D., Cameron, L. D., Peters, E., Lerner, J. S., Grenen, E., & Klein, W. M. P. (2020). The role of incidental affective states in appetitive risk behavior: A meta-analysis. Health Psychology, 39(12), 1109–1124.
Forscher, P. S., Paris, B., Primbs, M., & Coles, N. (2020). PSACR: The Psychological Science Accelerator's COVID-19 Rapid-Response Project. PsyArXiv [Preprint]. https://psyarxiv.com/x976j. Accessed 13 Aug 2021.
Forscher, P. S., Wagenmakers, E., Coles, N. A., Silan, M. A., Dutra, N. B., Basnight-Brown, D., & IJzerman, H. (in press). The benefits, barriers, and risks of big team science. Perspectives on Psychological Science.
Gallagher, K. M., & Updegraff, J. A. (2012). Health message framing effects on attitudes, intentions, and behavior: A meta-analytic review. Annals of Behavioral Medicine, 43(1), 101–116.
Gelfand, M. J., Jackson, J. C., Pan, X., Nau, D., Pieper, D., Denison, E., & Wang, M. (2021). The relationship between cultural tightness–looseness and COVID-19 cases and deaths: A global analysis. The Lancet Planetary Health, 5(3), e135–e144.
Glaeser, E. L. (2005). Paternalism and psychology. NBER working paper. https://www.nber.org/system/files/working_papers/w11789/w11789.pdf. Accessed 7 May 2021.
Gosling, C. J., Caparos, S., & Moutier, S. (2020). The interplay between the importance of a decision and emotion in decision-making. Cognition and Emotion, 34(6), 1260–1270.
Hameleers, M. (2021). Prospect theory in times of a pandemic: The effects of gain versus loss framing on risky choices and emotional responses during the 2020 coronavirus outbreak—Evidence from the US and the Netherlands. Mass Communication and Society, 24(4), 479–499.
Hartley, C. A., & Phelps, E. A. (2012). Anxiety and decision-making. Biological Psychiatry, 72(2), 113–118.
Haushofer, J., & Metcalf, C. J. E. (2020). Which interventions work best in a pandemic? Science, 368(6495), 1063–1065.
Haushofer, J., Mudida, R., & Shapiro, J. (2021). The comparative impact of cash transfers and a psychotherapy program on psychological and economic well-being. SSRN [preprint]. https://doi.org/10.2139/ssrn.3759722. Accessed 7 May 2021
Henrich, J., Heine, S. J., & Norenzayan, A. (2010a). Most people are not WEIRD. Nature, 466(7302), 29.
Henrich, J., Heine, S. J., & Norenzayan, A. (2010b). We agree it’s WEIRD, but is it WEIRD enough? Behavioral and Brain Sciences, 33(2-3), 61–83.
Kahneman, D., & Tversky, A. (1979). Prospect theory: an analysis of decision under risk. Econometrica, 47(2), 263-291.
Kok, G., Peters, G. J. Y., Kessels, L. T., Ten Hoor, G. A., & Ruiter, R. A. (2018). Ignoring theory and misinterpreting evidence: The false belief in fear appeals. Health Psychology Review, 12(2), 111–125.
Kubzansky, L., & Kawachi, I. (2020). Going to the heart of the matter: Do negative emotions cause coronary heart disease. Journal of Psychosomatic Research, 48(4–5), 323–337. https://doi.org/10.1016/S0022-3999(99)00091-4
LeBel, E. P., McCarthy, R. J., Earp, B. D., Elson, M., & Vanpaemel, W. (2018). A unified framework to quantify the credibility of scientific findings. Advances in Methods and Practices in Psychological Science, 1(3), 389–402.
Lerner, J. S., Li, Y., Valdesolo, P., & Kassam, K. S. (2015). Emotion and decision making. Annual Review of Psychology, 66, 799–823.
Lerner, J.S., Dorison, C.A., & Klusowski, J., (2022). How do emotions affect decision making? Handbook of Emotion Theory. Routledge.
Levin, I. P., Schneider, S. L., & Gaeth, G. J. (1998). All frames are not created equal: A typology and critical analysis of framing effects. Organizational Behavior and Human Decision Processes, 76(2), 149–188.
Levine, G. N., Cohen, B. E., Commodore-Mensah, Y., Fleury, J., Huffman, J. C., Khalid, U., et al. (2021). Psychological health, well-being, and the mind-heart-body connection: A scientific statement from the American Heart Association. Circulation, 143(10), e763–e783.
Loewenstein, G., & O'Donoghue, T. (2006). “We can do this the easy way or the hard way”: Negative emotions, self-regulation, and the law. The University of Chicago Law Review, 73(1), 183–206.
Mikels, J. A., Shuster, M. M., Thai, S. T., Smith-Ray, R., Waugh, C. E., Roth, K., & Stine-Morrow, E. A. (2016). Messages that matter: Age differences in affective responses to framed health messages. Psychology and Aging, 31, 409–414.
Moshontz, H., Campbell, L., Ebersole, C. R., IJzerman, H., Urry, H. L., Forscher, P. S., et al. (2018). The Psychological Science Accelerator: Advancing psychology through a distributed collaborative network. Advances in Methods and Practices in Psychological Science, 1(4), 501–515.
Nabi, R. L., Walter, N., Oshidary, N., Endacott, C. G., Love-Nichols, J., Lew, Z. J., & Aune, A. (2020). Can emotions capture the elusive gain-loss framing effect? A meta-analysis. Communication Research, 47(8), 1107–1130.
ó Hartaigh, B., Loerbroks, A., Thomas, G. N., Engeland, C. G., Hollands, M. A., Fischer, J. E., & Bosch, J. A. (2012). Age-dependent and-independent associations between depression, anxiety, DHEAS, and cortisol: From the MIPH Industrial Cohort Studies (MICS). Psychoneuroendocrinology, 37(7), 929–936.
Otte, C., Hart, S., Neylan, T. C., Marmar, C. R., Yaffe, K., & Mohr, D. C. (2005). A meta-analysis of cortisol response to challenge in human aging: Importance of gender. Psychoneuroendocrinology, 30(1), 80–91.
Peters, G. J. Y., Ruiter, R. A., Ten Hoor, G. A., Kessels, L. T., & Kok, G. (2018). Towards consensus on fear appeals: A rejoinder to the commentaries on Kok, Peters, Kessels, ten Hoor, and Ruiter (2018). Health Psychology Review, 12(2), 151–156.
Rosenthal, E. (2020). It’s time to scare people about Covid. New York Times., https://www.nytimes.com/2020/12/07/opinion/covid-public-health-messaging.html. Accessed 7 May 2021.
Rothman, A. J., & Salovey, P. (1997). Shaping perceptions to motivate healthy behavior: The role of message framing. Psychological Bulletin, 121(1), 3.
Rothman, A. J., & Sheeran, P. (2021). The operating conditions framework: Integrating mechanisms and moderators in health behavior interventions. Health Psychology, 40(12), 845.
Rothman, A. J., Desmarais, K. J., & Lenne, R. L. (2020). Moving from research on message framing to principles of message matching: The use of gain-and loss-framed messages to promote healthy behavior. In Advances in motivation science (Vol. 7, pp. 43–73). Elsevier.
Ruggeri, K., Alí, S., Berge, M. L., Bertoldo, G., Bjørndal, L. D., Cortijos-Bernabeu, A., et al. (2020). Replicating patterns of prospect theory for decision under risk. Nature Human Behaviour, 4(6), 622–633.
Simonsohn, U., Simmons, J. P., & Nelson, L. D. (2020). Specification curve analysis. Nature Human Behaviour, 4(11), 1208–1214.
Steegen, S., Tuerlinckx, F., Gelman, A., & Vanpaemel, W. (2016). Increasing transparency through a multiverse analysis. Perspectives on Psychological Science, 11(5), 702–712.
Sunstein, C. R., & Thaler, R. H. (2003). Libertarian paternalism is not an oxymoron. The University of Chicago Law Review, 70(4), 1159-1202.
Taylor, S. E. (2021). Health Psychology. McGraw Hill.
Thaler, R. H., & Sunstein, C. R. (2009). Nudge: Improving decisions about health, wealth, and happiness. Penguin.
Tversky, A., & Kahneman, D. (1991). Loss aversion in riskless choice: A reference-dependent model. The Quarterly Journal of Economics, 106(4), 1039–1061.
Uskul, A. K., Sherman, D. K., & Fitzgibbon, J. (2009). The cultural congruency effect: Culture, regulatory focus, and the effectiveness of gain-vs. loss-framed health messages. Journal of Experimental Social Psychology, 45(3), 535–541.
Van Bavel, J. J., Baicker, K., Boggio, P. S., Capraro, V., Cichocka, A., Cikara, M., et al. (2020). Using social and behavioural science to support COVID-19 pandemic response. Nature Human Behaviour, 4(5), 460–471.
Van’t Riet, J., Cox, A. D., Cox, D., Zimet, G. D., De Bruijn, G. J., Van den Putte, B., et al. (2016). Does perceived risk influence the effects of message framing? Revisiting the link between prospect theory and message framing. Health Psychology Review, 10(4), 447–459.
Wang, K., Goldenberg, A., Dorison, C. A., Miller, J. K., Uusberg, A., Lerner, J. S., et al. (2021). A multi-country test of brief reappraisal interventions on emotions during the COVID-19 pandemic. Nature Human Behaviour, 5(8), 1089–1110.
Wells, G. L., & Windschitl, P. D. (1999). Stimulus sampling and social psychological experimentation. Personality and Social Psychology Bulletin, 25(9), 1115–1125.
Wong, N. C., Harvell, L. A., & Harrison, K. J. (2013). The unintended target: assessing nonsmokers' reactions to gain-and loss-framed antismoking public service announcements. Journal of Health Communication, 18(12), 1402–1421.
World Health Organization (2021). Depression and other common mental disorders global health estimates. Accessed 7 May 2021.
Xu, P., Gu, R., Broster, L. S., Wu, R., Van Dam, N. T., Jiang, Y., et al. (2013). Neural basis of emotional decision making in trait anxiety. Journal of Neuroscience, 33(47), 18641–18653.
Zlatev, J. J., & Rogers, T. (2020). Returnable reciprocity: Returnable gifts are more effective than unreturnable gifts at promoting virtuous behaviors. Organizational Behavior and Human Decision Processes, 161, 74–84.