Data preprocessing
The data set delivered by Qualtrics contained completed records from 403 participants, ten of whom did not meet the criterion of passing at least two out of three comprehension questions. Those records were eliminated, yielding a final sample of 393 participants for analysis (198 male, 195 female). The mean age of the sample was 46 (median = 45, Q1 = 29, Q3 = 60).Footnote 2 The data set is available at https://github.com/StephanLewandowsky/Blog-comments.
We next considered the time participants spent on reading the comments. Reading time for the comments varied considerably across participants, from 2.5 s to 4530 s (1 h 15 min). Jackson and McClelland (1979) reported adult reading speed estimates for college-level text that ranged from 33 words per minute (wpm) to 454 wpm. Those estimates translate into expected reading times ranging from around 130 s to 1800 s for the roughly 1000 words in our comment stream. Given that our comments were written in a colloquial style that would likely be easier to process than the texts used by Jackson and McClelland (1979), we decided to define “careful readers” as anyone who spent between 100 s and 1800 s processing the comments. All further analyses were conducted both on the full set of participants who passed the comprehension questions (N = 393) and a subset of careful readers (N = 183). With two exceptions noted below, the two analyses yielded qualitatively identical results except that all effects were more pronounced with the subset of careful readers. We therefore only report the analyses of the full sample.
Tests of experimental effects
The five items probing climate attitudes were reverse-scored as appropriate (see Table 2 for a description of all items) and then averaged to form a single composite score, called AGW in the figures, that was used for the descriptive statistics. Figure 1 shows summary statistics and distributional information for all dependent variables across the four cells of the experimental design (panels a–e), and pairwise correlations between perceived reader consensus and acceptance of AGW for the four cells separately (panel f). Table 3 shows the cell means for the same set of dependent variables. Skewness and kurtosis for all measures were within ± 1 and there were no particularly outlying observations based on the interquartile range outlier detection rule (Hoaglin & Iglewicz, 1987).
Table 3 Means of principal dependent measures across experimental conditions The variables in Fig. 1 and Table 3 were analyzed by frequentist as well as Bayesian techniques. We report the Bayesian analysis in the Appendix; here we present a series of 2 × 2 ANOVAs that explored the pattern in Fig. 1. Except where noted, the Bayesian analysis supported identical conclusions. For the item querying whether readers felt they were affected by blog comments (item Unaffected in Table 2), no effects reached significance, all F < 1. The overall mean of this item also did not differ from the mid-point of the scale (M = 3.02, t(392) < 1), suggesting that participants were ambivalent about whether or not they were affected by others’ views and this ambivalence was invariant across all conditions.
People’s support for the blog post (item Support) was not affected by the type of post, F(1,389) < 1, or the type of comments, F(1,389) < 1, although there was strong evidence for the interaction of both variables, F(1,389) = 10.75,p < 0.001, partial η2 = .027, Cohen’s F = .166. This reflected the fact that when the post and comments were of the same type (both rejecting or both endorsing AGW), people supported the post more than when the polarities of the post and the comments were in opposition. This result establishes that reading the comments affected people’s views of the post. Supportive comments enhanced endorsement of the post whereas critical comments undermined that support.
The AGW composite score was similarly unaffected by the type of comments, F(1,389) = 1.59,p > 0.1, although there was very strong evidence of a role of the type of post, F(1,389) = 14.28,p < 0.0001, partial η2 = .035, Cohen’s F = .192, indicating that people accepted global warming more after reading the post supporting mainstream science than after reading a contrarian post. The interaction between the two variables was non-significant, F(1,389) = 1.42,p > 0.1. In this instance, however, the subset analysis of careful readers additionally returned a main effect of type of comment, F(1,179) = 6.58,p < 0.02, reflecting the fact that contrarian comments reduced acceptance of the mainstream science among careful readers.
The perceived consensus among blog readers (item ReaderCons) was affected neither by the type of post, F(1,389) < 1, nor the type of comments, F(1,389) < 1, but it was strongly affected by the interaction of both variables, F(1,389) = 50.44,p < 0.0001, partial η2 = .115, Cohen’s F = .360. That is, similar to people’s expressed support for the blog post, their perception of consensus among readers was greatest when the comments were consonant with the post (both rejecting or both endorsing AGW) as opposed to when there was a mismatch in polarity. The subset analysis of careful readers additionally returned a main effect of type of post, F(1,179) = 5.80,p < 0.02, reflecting the fact that more readers were presumed to endorse the science-based post than its contrarian counterpart.
For the presumed consensus among scientists (item SciCons), by contrast, what mattered strongly was the type of post, F(1,388) = 12.96,p < 0.0003, partial η2 = .032, Cohen’s F = .183,Footnote 3 reflecting the fact that the science-based post underscored the scientific consensus whereas the contrarian post undermined it. A weak effect for the type of comments, F(1,388) = 4.56,p < 0.05, partial η2 = .012, Cohen’s F = .108, was not confirmed by the Bayesian analysis (see Appendix). The interaction between both variables was non-significant, F(1,388) < 1.
Taken together, the analyses support two main conclusions: First, the type of post strongly affected people’s attitudes towards climate change, with the science-based post increasing belief in global warming relative to a contrarian post. By contrast, there was no evidence that comments alone affected attitudes towards climate change directly when all participants were considered. Only when we focused on careful readers did a direct effect of comments on climate attitudes emerge, such that comments that endorsed the science were associated with higher acceptance than contrarian comments.
Second, the match or mismatch between post and comments mattered to people’s endorsement of the post and the perceived consensus among readers: critical comments (e.g., contrarian comments following a science-based post or vice versa) undermined support and perceived consensus, whereas favorable comments increased both. In light of the strong effects of comments on perceived consensus among readers, and in light of the direct effect on attitudes among careful readers, we next explored the possibility that reader comments may affect AGW attitudes indirectly, via their effect on consensus.
Inter-variable associations and structural equation modeling
This analysis modeled people’s acceptance of AGW as a function of the two experimental design variables and perceived endorsement among readers. We applied process analysis (Hayes, 2018) to model the relation between those intertwined variables. We first devised a measurement model associated with the five items that queried climate attitudes (items agw1 though agw5). The single-factor model associated with an AGW latent variable was found to be moderately well-fitting, χ2(5) = 66.64,p < 0.001, CFI = .910,TLI = .821, RMSEA= .177, SRMR= .069. However, there was an indication that a non-negligible amount of covariance was shared between the two items with negative polarity, agw1 and agw5 (see Table 2). With the addition of the covariance between these two items (r = .40,p < 0.001), the model fit substantially better, χ2(4) = 6.39,p > 0.1, CFI = .997, TLI = .991, RMSEA= .039, SRMR= .017. We therefore used this single-factor latent variable as our criterion variable for the remaining modeling.
We next examined some of the key bivariate correlations. The association between perceived scientific consensus (SciCons) and the AGW latent variable was large, r = .59,95%CI : .51,.66, p < 0.0001. By contrast, the overall correlation between perceived consensus among readers (ReaderCons) and the AGW latent variable escaped significance, r = .10,95%CI : −.03,.22, p > 0.05. The overall absence of a correlation is unsurprising in light of the opposing directions of the relationship between reader consensus and endorsement of AGW across conditions (shown in panel f in Fig. 1). For the posts that rejected science, the correlations between perceived reader consensus and the AGW latent variables were negative/near zero (r = −.16 and r = .05), whereas they were positive for the posts that endorsed the mainstream scientific position (r = .45 and r = .25). The reversed directionality between types of post points to a coherent role of the presumed prevalence among readers of support for the scientific position. That is, a negative correlation between the presumed share of readers who endorsed a rejectionist post is equivalent to a positive association between the share of readers who disagreed with that post and, by implication, endorsed the mainstream scientific position. Thus, to simplify presentation of the results, the perceived reader consensus scores provided by participants who were exposed to the rejectionist post were therefore reflected to express dissensus with the post (i.e., 100 - ReaderCons score). This reflected score represented the perceived endorsement of mainstream science among readers. This reflected measure behaved consistently across conditions was found to correlate moderately with AGW across all conditions, r = .25,95%CI : .12,.37, p < 0.0001.
Finally, we estimated the correlations between the two experimental variables, type of post and type of comments, and the AGW latent variable. For both experimental variables, endorsement of AGW was coded as 1 and rejection as 0. The correlations were r = .18,95%CI : .07,.27, p < 0.003 for type of post, and r = .04,95%CI : −.08,.15, p = 0.520 for type of comments, respectively. Those correlations mirror the ANOVAs reported in the previous section, and they confirm that there was no direct (or total) effect between one of the key experimental variables, type of comments, and AGW. However, contemporary process analysis does not require the presence of a significant total effect, as the observation of an indirect effect by itself is considered sufficient (Hayes, 2009; Preacher & Hayes, 2004; Mathieu & Taylor, 2006; MacKinnon, Krull, & Lockwood, 2000). We therefore explored several candidate process models to explore whether the type of comment may still affect AGW attitudes via an indirect route.
The first model is shown in Fig. 2. Both experimental variables and their interaction were used to predict AGW directly and indirectly, via the presumed mediator of reflected perceived reader consensus. This model was found to fit acceptably well, χ2(20) = 51.28,p < 0.001, CFI = .975, TLI = .955, RMSEA = .063, SRMR= .049. However, several of the estimated effects were not statistically significant (indicated by dashed lines in the figure). In particular, the interaction between the two experimental variables, type of post and type of comments (TP × TC), was not associated with either a direct (b = −.16,β = −.11,p = 0.22) or indirect (b = .01,β = .01,p = 0.79) effect on AGW. Furthermore, the type of comments (TC) did not have a direct effect on AGW (b = .04,β = .03,p = 0.75).
Consequently, a revised and simplified process model was estimated which excluded the interaction term between the experimental variables, as well as the direct effect of type of comment on AGW. The revised model is shown in Fig. 3 and was found to be associated with acceptable levels of model fit, χ2(17) = 45.93,p < 0.001, CFI = .964, TLI = .941, RMSEA= .066, SRMR= .052. All weights and correlations shown in the figure are significant. It can be seen that the type of post (TP) had a direct effect on AGW (b = .16,β = .13,p = 0.007). Thus, exposure to the scientific post was associated with greater levels of acceptance of the mainstream science, as already indicated by the corresponding main effect in the earlier ANOVA. A corresponding direct effect for type of comments (TC) was absent. However, both types of post, b = .07,β = .05,p = 0.002, and type of comment, b = .09,β = .07,p = 0.001, had indirect effects on AGW via the reflected reader consensus score. Thus, people’s acceptance of mainstream science was in part determined by their perception of how many other readers shared their view, and that perception in turn was influenced by our experimental design variables, namely the type of post and type of comments. The multiple R associated with the model was .280 (p = 0.004). Thus, 7.7% of the AGW true score variance was accounted for by the direct effect of the type of comment variable and the indirect effects of both experimental variables.