Participants and Procedure
998 participants were recruited and compensated using Amazon.com’s Mechanical Turk platform. Sample size was determined by prior power analysis using G*Power. The eligibility criteria were living in the United States and being 18 or older. Ages ranged from the bracket 18–29 to the bracket 74 and up, with a median in the 29 to 39 bracket. 63% of participants were male. Median education completed was a Bachelor’s Degree; 68% had a Bachelors or higher level of education. The median annual household income fell in the bracket of $50,000 to $74,999. 55% of respondents were married; 34% had never married; 7% were divorced; 2% separated; and 2% widowed. 38% of respondents identified as Republican to various degrees; 47% as Democrats; and 15% as Independent. 74% of respondents were White / Caucasian; 12% were Black or African American; 5% Hispanic; 7% Asian or Pacific Islander; and 2% American Indian or Alascan Native. 49% of respondents rated religion as not at all important or not very important; 18% as moderately important, and 33% as important or extremely important. Hence our sample is more male, more White / Caucasian, and more educated than the US as a whole, and probably also slightly less religious and less Republican, although different ways of eliciting this information make comparisons difficult.
The 16 items of the EVS that we identified as promising based on the exploratory factor analysis above were administered in random order using a five-point Likert scale (anchored on “strongly disagree” through “strongly agree”) -- the ten items used in the final exploratory factor analysis, as well as the six items removed last, to see whether they perform better in this study. The items are marked with an asterix in Table 1 above.
Each participant answered demographic questions about age, gender, race, education, household income, and marital status.
To evaluate unique variance, we administered three instruments. First, administered the conspiracy instrument we used in study 1 (α = 0.91, items = 5) Descriptive statistics are in Table 4.
Second, to demonstrate real-world relevance of the EVS, we administered a 12-item measure of Covid-19-related misinformation based on the “myth-busting” page of the World Health OrganisationFootnote 1 (α = 0.94). Items concern essential information about Covid-19 for the public, e.g. “Being able to hold your breath for 10 seconds or more without coughing or feeling discomfort means you are free from the coronavirus disease”, “Spraying and introducing disinfectant into your body will protect you against COVID-19”, and “Regularly rinsing your nose with saline helps prevent infection with the new coronavirus” (see Table 5 for all items and descriptive statistics). Participants were asked to respond to all items in random order on a five-point scale (“Definity false”, “Probably false”, “Don’t know”, “Probably true”, “Definitely true”). The aggregate measure takes the mean of the responses to the first three items. “Don’t know” responses were excluded from the analysis, for the same reason as in study 1 that the response might either indicate epistemic virtue or vice.
Third, we administered a measure of fake-news endorsement related to the Covid-19 pandemic (α = 0.91, items = 4). Participants were presented with screenshots of five online news articles related to Covid-19 in random order, four fake and one legitimate (see Table 6 and Figs. S3 to S7 in the online supplemental material for all items and descriptive statistics). Participants were asked to respond to the statement “The article displayed above is credible” on a five-point agree/disagree scale. The four fake news items were picked from prominent fake news sources. They spread claims about evidence that the new Coronavirus is a bioweapon; that the pandemic is deadlier due to the introduction of 5G technology; and that reporting on Covid-19 is a ploy to distract from the shady dealings of certain politicians. From a content perspective, they are quite similar to the Covid-19 misinformation instrument. But this instrument differs by eliciting how much credence respondents give to fake news articles advancing such claims, rather than the claims themselves.
To evaluate convergence and divergence with related constructs, we administered 10 scales: First, we measured all dimensions of the Big Six personality model using the 24-item QB6, α(Honesty) = .56, α(Agreeableness) = .62, α(Emotionality) = .62, α(Extroversion) = .60, α(Conscientiousness) = .68, α(Intellect) = .43 (Thalmayer & Saucier, 2014). Second, a seven-item version of the Cognitive Reflection Test, α = .80 (Sirota & Juanchich, 2018). Third, Rosenberg’s 10-item self-esteem scale, α = .85 (Rosenberg, 1965). Fourth, a 15-item scale to measure need for closure, α = .89 (Roets & Van Hiel, 2011). Fifth, a 18-item scale to measure need for cognition, α = .93 (Cacioppo et al., 1984). Sixth, a 15-item scale to measure faith in intuition, α = .93 (Alós-Ferrer & Hügelschäfer, 2012). Seventh, the general version of a 6-item scale on open-minded cognition, α = .77 (Price et al., 2015). Eighth, a 20-item dogmatism scale, α = .90 (Altemeyer, 2002). Ninth, a 6-item scale measuring trust in experts, adapted from Imhoff et al. (2018), α = .74 (see Table S3 for items). Tenth, a 25-item scale to measure overclaiming, α = .87 (Bing et al., 2011).
We measure religiosity by asking respondents how important religion is to them, on a five-point scale from “not at all important” to “extremely important.”
We measure political partisanship by asking participants whether they “consider themselves a Republican, a Democrat, an Independent, or what?” Responses are “Strong Democrat”, “Moderate Democrat”, “Lean Democrat”, “Lean Republican”, “Moderate Republican”, “Strong Republican”, “Independent”, “Other”, and “Prefer not to say”.
Before conducting the study, we recorded our hypotheses in the process of pre-registration.Footnote 2 We expected the EVS to be strongly positively correlated with all three outcome measures. We expected that the scales we included for convergent/divergent validity would show the following correlations: positive correlations for the scales measuring faith in intuition, dogmatism, overclaiming (accuracy), and need for closure; negative correlations for all other scales: personality, cognitive reflection, self-esteem, trust in experts, need for cognition, open-minded cognition, and the accuracy measure on the overclaiming scale. We expected religiosity to be positively related to the EVS, and education to be related negatively. We expected that all correlations will be moderate, which would establish epistemic vice as a distinct construct. Our most important hypothesis was this: EVS can explain additional variance with regard to all three outcome metrics, over and above the demographic information, the scales included for divergent validity, and political affiliation and religiosity.
Confirmatory Factor Analysis
In the CFA we included the ten items that were used in the final exploratory factor analysis in Study 1. We also experimented with the inclusion of some or all of the six items cut last in Study 1, but decided to discard them because the models had less good fit. The models were examined according to several indices: the robust versions of the comparative fit index (CFI), the root mean square error of approximation (RMSEA), and the standardized root mean square residual (SRMR). CFI ranges from 0 to 1, with values closer to 1 indicating better fit, and reflects the proportion of improvement in fit relative to the null (independence) model. RMSEA and SRMR are measures of absolute fit, that is, how well on average the correlation matrix has been reproduced by the model. According to Hu and Bentler (Hu & Bentler, 2009), CFI should not be smaller than .95, RMSEA should not be more than .06, and SRMR should not be more than .08. The two-factor solution met Hu and Bentler’s standards, χ2(34) = 150, CFI = .0.98, RMSEA = .06, SRMR = .03 – though note that the RMSEA value is at the cutoff point. See Table 7 for the final 2-factor solution with ten items and standardized estimates.
The scale has very similar reliability to the reliability reported in study 1. Reliability of the Epistemic Vice Scale is high (α = .90, items = 10), as are the reliability of the indifference to truth subscale (α = .90, items = 4), and rigidity subscale (α = .83, items = 6).
We supplemented the confirmatory factor analysis with analyses from the perspective of item response theory (IRT). IRT is used for investigating item and test properties; it assumes a latent trait or ability that is a function of both the participants’ responses and the properties of the items. Thus, IRT allows us to estimate both an individual’s trait level and the relevant item parameters. The goal was to estimate the overall reliability of the measure in a way that is distinct from the classical testing theory approach. We used a graded response model implemented in the ltm package in the R statistical language (Rizopoulos, 2006).
Figures 1 and 2 show the item characteristic curves for the indifference and rigidity subscales, respectively. Table 8 gives threshold parameters and item slopes for items. Item slopes (column a) are discrimination parameters, which are particularly interesting because they describe an item’s ability to differentiate between participants having levels of the latent trait above or below the item’s location. Threshold parameters (b) can be considered cut-off points on the latent trait’s continuum. Respondents with the given level of the latent trait are equally likely to select the respective response category rather than the next higher response category. Consider the indifference items. One thing to note is that threshold parameters are skewed towards the positive end of the latent trait. This means that items provide more information on people who are more indifferent. While it would be desirable for items to differentiate well across the whole spectrum of the latent trait, for the purposes of identifying epistemically vicious individuals, differentiation toward the positive end of the spectrum is most important. The slope parameters are reasonably similar across items, supporting scoring of the scale as an unweighted sum. Considering the rigidity items, we find that threshold parameters are more balanced across the latent trait spectrum, compared to the indifference items, and that like in the case of the indifference items, item slopes are fairly similar.
Figure 3 shows the test information functions for the indifference and rigidity subscales. Consistent with the analysis above, the indifference subscale is particularly informative towards the positive end of the spectrum. The items measuring rigidity are more evenly distributed over the center of the latent trait.
We analyzed the extent to which the scale displayed convergent and discriminant validity. Table 9 shows correlation coefficients between all measures. Table S4 provides a detailed correlation table for key covariates. The first four columns show correlations with the four outcome metrics we are testing: Covid-19 misinformation, conspiracist thinking, susceptibility to fake news, and overclaiming bias. People who are susceptible to conspiracist thinking are also likely to believe Covid-19 misinformation, endorse fake news, and overclaim, as indicated by the high correlations between the four instruments.
The table shows correlations between covariates in percentages (pairwise Pearson correlations). The shade indicates strength of correlation (absolute value). Correlations are significant at p < 0.01, unless crossed out (2-tailed).
Epistemic vice is strongly associated with all four instruments. Measures for competing explanations are less strongly associated with any of the outcome metrics. The measure with the next-highest correlation, dogmatism, shows substantially lower correlations.
We conceptually replicate the findings of Stanley et al. (2020) and Pennycook et al. (2020) that the Cognitive Reflection Test predicts acceptance of Covid-19 misinformation (their outcome variable was measured slightly differently, but the headline result is the same). Yet the absolute value of the correlation coefficient of cognitive reflection with Covid-19 misinformation is only about half of the correlation coefficient of epistemic vice with Covid-19 misinformation. This gives epistemic vice a fairly strong lead over alternative measures.
The associations between epistemic vice and other measures all have the expected sign, with the exception of education. We expected that higher levels of formal education would be associated with lower readiness to endorse fake news. Yet the opposite is the case. The finding that more formal education is not an important determinant of epistemic virtue and vice is however consistent with what Haggard et al. (2018) find in their validation study.
No correlation of epistemic vice or its subscales with other scales is so high as to suggest that the EVS simply replicates an existing scale. However, the EVS may still be tapping into the same latent construct as some of the other scales that it is correlated with. If so, however, there is evidence that the EVS measures that common latent trait better, because it is correlated more strongly with outcome measures. Consider for instance faith in intuition, the scale that the EVS and its subscales is most strongly correlated with (r ranges between .45 and .60). Faith in intuition is correlated to a much smaller extent with the outcome metrics (r ranges between .47 and .56) than epistemic vice (r ranges between .68 and .76).
The correlations between epistemic vice and some established psychological scales such as dogmatism, faith in intuition, and the cognitive reflection test raise a question about whether the epistemic vice predicts outcome variables above and beyond what these scales can predict. Epistemic vice could just be a combination of existing constructs, making a new scale less useful.
Given the large number of existing psychological constructs that appear to be related to epistemic vice, one important question is whether epistemic vice is already sufficiently well measured by one or a combination of these other scales. It is possible that epistemic vice is tapping into the same psychological construct as one of the scales above and merely describes it differently. We test this by conducting hierarchical regressions to investigate how much variance in outcomes such as the propensity to endorse misinformation or conspiracy theories the EVS can explain in addition to these existing measures.
To test this, we performed hierarchical regressions where we tested how much variance in outcome variables epistemic vice predicts above and beyond what other scales and demographic variables predict.
Table 10 shows the summary results of the second and final step of a series of hierarchical regressions. The dependent variables were Covid-19 misinformation, conspiracist thinking, fake news, and overclaiming bias, respectively. Each row of the table represents the summary results of Step 2 of a separate hierarchical regression. For Step 1, the variables listed in the column “Model” were entered as a block. For Step 2, epistemic indifference and epistemic rigidity were entered as a block. All continuous predictors as well as the dependent variables are mean-centered and scaled by one standard deviation. However, adding the epistemic vice scales increases R2 substantially across all models and all dependent variables. Consequently, epistemic vice explains substantial variance over and above each scale for each outcome variable. Moreover, the effect size as measured by the regression coefficients of epistemic indifference and epistemic rigidity is large across models and outcome variables (epistemic indifference: min = .29, max = 53, mean = .37; epistemic rigidity: min = .28, max = .68, mean = .48).
This table shows summary results of the final second step of a series of hierarchical regressions. The dependent variables are susceptibility to Covid-19 misinformation, endorsement of conspiracy theories, susceptibility to fake news, and overclaiming bias, respectively. For the first step, the variables mentioned in the Model column were entered as a block. The R2 column shows variance explained in the first step of the regression (without epistemic vice). Each subsequent column provides results of the second step of a different hierarchical regression. For the second step, epistemic indifference and epistemic rigidity were added to the respective model as a block. ΔR2 shows the difference in R2 between the first and the second stage. ΔF shows the F-statistic for the difference between the two models. ß Indiff. and ß Rigidity show the coefficient of epistemic indifference and epistemic rigidity in the final step of the regression, respectively. All continuous predictors as well as the dependent variables are mean-centered and scaled by 1 standard deviation. T-test Indiff. and t-test Rigidity show the corresponding t-test statistics. *** indicates p < .001. N = 998.
Importantly, as the final hierarchical regression in Table 10 shows, epistemic vice explains additional variance even when all other measures are included in the regression jointly, from ΔR2 = .03 for conspiracist thinking to ΔR2 = .09 for susceptibility to Covid-19 misinformation. Table 11 shows detailed results of this hierarchical regression for Covid-19 misinformation, conspiracist thinking, fake news, and overclaiming bias as dependent variables, respectively. For each model there are two steps. Step 1 includes all demographic variables as well as psychological scales. Step 2 adds epistemic indifference and epistemic rigidity. All continuous predictors, as well as the dependent variables, are mean-centered and scaled by one standard deviation. Despite the large number of independent variables, multicollinearity is not a serious concern. Coefficients are fairly stable across outcome variables and when adding or deleting coefficients, and the highest variance inflation factor across these regressions is 3.68, below the threshold of 5 (Hair et al., 2010). Across outcome variables, the EVS shows moderate to large effect sizes. For Covid-19 misinformation and fake news, the coefficients for epistemic vice are the highest coefficients in the regression. This result strongly supports our hypothesis that the EVS explains additional variance with regard to Covid-19 misinformation, over and above the demographic information and the other psychological scales.
We were able to replicate the factor structure of the epistemic vice scale, with the confirmatory factor analysis showing excellent fit. The epistemic vice scale is correlated to the psychological constructs with the hypothesized sign. However, the epistemic vice scale is not merely redundant with them. The scale predicts a range of relevant outcomes from susceptibility to Covid-19 misinformation and fake news to endorsement of conspiracy theories and overclaiming bias. For each of these outcomes, the epistemic vice scale is a better predictor than any other construct studied. Note that Alfano et al. (2017) investigated associations between their scale measuring intellectual humility and overclaiming bias. The correlations of the epistemic vice scale with overclaiming bias are more than four times higher than what they find. Perhaps the most impressive result is that the epistemic vice scale explains substantial variance over and above all other included scales and demographic variables, for all four outcome measures, and even explains additional variance over all other measures combined.
The EVS captures important components of epistemic vice that have been identified by philosophers and validated by subject matter experts. Using these analyses to inform a broad item pool, two dimensions emerged from our studies: epistemic indifference and epistemic rigidity. Indifference manifests itself in a lack of motivation to find the truth. Rigidity manifests itself in being insensitive to evidence once one’s mind is made up. Further work could test to what extent these two dimensions also underlie individual differences in other epistemic vices that are not covered by the taxonomy we used to generate our item pool, such as excessive malleability.
The validity of these subscales was supported by convergence and divergence with existing scales and the ability of the scale to predict four relevant outcomes. Epistemic vice adds predictive power beyond demographic variables and established psychological scales, suggesting it is not redundant with already-established measures.
A next step for validating the scale would be to reproduce the research outside of North America and in languages other than English. Another next step would be to investigate a broader range of consequences and behaviors. Three of our outcome measures focused on susceptibility to problematic beliefs as the most immediate plausible consequence of epistemic vice: susceptibility to Covid-19 misinformation, to conspiracist thinking, and to fake news. The outcome measure looking at overclaiming bias goes beyond beliefs to behaviors. Future research should investigate a broader range of beliefs as well as behaviors plausibly associated with these beliefs, such as lack of taking important precautions in the case of Covid-19 susceptibility. Yet further work could investigate the development of epistemic vice, as well as interventions that may blunt its effects.