The goal of Experiment 2 was to replicate the findings of Experiment 1 and assess the degree to which the false insight effect in Experiment 1 was driven by either the semantic priming or the misleading visual configuration of the anagrams. We predicted that participants who saw both semantic priming and visually similar anagrams would experience the highest proportion of false insights (as in Experiment 1), followed by participants exposed to semantic priming and given randomly scrambled anagrams, followed by participants who were not semantically primed but were given visually similar anagrams. Thus, we expected that the false insight effect documented in this experiment would be driven more by semantic priming than visual similarity. Finally, we expected that participants would again report lower subjective intensity for false versus correct insights. We did not pursue the relationship between false insights and false memories in this experiment as the aim of this study was to understand the driving factors of the false insight effect.
Open practice statement
This experiment is preregistered on the Open Science Framework. The data, materials, intended study design, exclusion criteria, and analysis scripts are available at https://osf.io/ez4y6/?view_only = 97a1183e9f954b749f41aab3ce0424bf.
Given the mean differences between each anagram type observed in Experiment 1, we simulated and analyzed the results from 2,000 datasets based on 37 participants in each of the four groups. This sensitivity analysis revealed that our design would be sufficiently powered to detect an effect size of η2G = .15 for the main effect of Anagram Type in all 2,000 of these simulations (100%). By decreasing the mean differences between the four anagram types to derive the smallest effect size of interest, which was η2G = .02 (Lakens et al., 2018), we could still detect a significant main effect in 1,600 out of 2,000 simulated datasets (80%). This entire sensitivity analysis is documented at (https://osf.io/ez4y6/files/). We therefore decided to use the same sample size as Experiment 1. A sample of 150 native-English speaking participants (79 female, 66 male, four other) with a mean age of 29.67 years was recruited using the online crowdsourcing platform Prolific Academic, who received $6 for their participation.
Design and materials
This experiment had a 2 (Semantic Priming: present, absent) 𝗑 2 (Visual Similarity: present, absent) 𝗑 4 (Anagram Type: primed lure, presented target, random, primed target) mixed design with Semantic Priming and Visual Similarity as between-subjects factors, and Anagram Type as a within-subjects factor. Since there was no difference in the rates of false insights produced by each counterbalancing condition in Experiment 1, we included both sets of stimuli in Experiment 2 but presented them randomly to participants so they were not included as a factor in our analyses. The Semantic Priming and Visual Similarity materials were the same as the first experiment. For the conditions without semantic priming, we presented lists of randomly generated words (created by https://randomwordgenerator.com) instead of the semantic associates lists. In these conditions, the anagrams were the same visually similar configurations (cosine of .85) used in Experiment 1 but lacked any semantic relation to the priming list. Because we expected the effect of anagram type to depend on the presence of semantic priming, these conditions served as controls to assess whether false insights would occur less frequently for the primed lure anagrams when there was no semantic relationship to the study list, despite being in a misleading configuration. For the conditions without visual similarity, we used the same priming lists as in Experiment 1 but did not compute a cosine similarity for any of the anagrams relative to their intended solution. Rather, for the three control anagrams, we scrambled the words using an online random word scrambling tool (instead of arranging them to resemble the correct solution at a.85 level of cosine similarity). Likewise with the primed lures, instead of arranging the incorrect solution to resemble a specific primed associate, we randomly scrambled it using the same tool. This process thereby removed the effect of visual similarity to investigate the possible interaction between semantic priming and anagram types. We expected this manipulation to demonstrate that regardless of how the anagrams were scrambled, primed lures would elicit false insights more than other anagram types simply due to their semantic association with the study list.
We programmed the experiment to run as closely as possible to Experiment 1, with each word being presented at the same rate, and the answer boxes appearing for the same time and in the same fashion. One difference was that the word lists were not spoken aloud by the computer, but simply appeared on the screen instead.
Measures and procedure
The procedure was nearly identical to Experiment 1 except participants provided their consent electronically. Due to the deviations from the original experiment necessitated by the online format, participants received written instructions instead of the video used in the first experiment. A full transcript of these instructions is available in Appendix 1.
As in Experiment 1, we isolated trials with reported insight moments and computed the proportions of false insights for each condition Table 2.
To test our first preregistered hypothesis, we examined the effects of the experimental manipulations on false insights by running a 2 (Semantic Priming: present, absent) 𝗑 2 (Visual Similarity: present, absent) 𝗑 4 (Anagram Type: primed lure, presented target, random, primed target) mixed ANOVA with Semantic Priming and Visual Similarity as the between-subjects factors and Anagram Type as the within-subjects factor. This analysis revealed the predicted main effect of Anagram Type, F(3,438) = 68.77, p < .001, η2G = .11. To examine the source of this main effect of Anagram Type, we ran a series of post hoc Tukey comparisons. As can be seen in Fig. 3, these comparisons revealed a significant difference between the primed lure (M = 0.13, SD = 0.17) and each of the three other conditions: the presented target (M = 0.03, SD = 0.10), t(438) = 11.87, p<.001, d = 1.09, the primed target (M = 0.03, SD = 0.12), t(438) = 11.06, p < .001, d = 1.01, and the random anagrams (M = 0.03, SD = 0.11), t(438) = 12.15, p < .001, d = 1.06.
Note. The ‘raincloud’ plots in Fig. 3 depict the proportion of all trials with reported false insights across the four anagram types, combining boxplots, raw jittered data, and a split-half violin. Each plot represents one version of the experiment (between-subjects) and each distribution represents one type of anagram (within-subjects)
No main effect emerged for Visual Similarity, F(1,146) = 0.52, p = .474, η2G < .01, or Semantic Priming, F(1,146) = 0.27, p = .606, η2G < .01, and there was no interaction between these two variables, F(1,146) = 1.50, p = .223, η2G < .01. An interaction emerged between Anagram Type and Semantic Priming, F(3,438) = 23.20, p < .001, η2G = .04, and between Visual Similarity and Anagram Type, F(3,438) = 3.38 p = .018, η2G < .01. Finally, a three-way interaction emerged between Anagram Type, Semantic Priming, and Visual Similarity, F(3,438) = 7.39, p < .001, η2G = .01 (see Fig. 3).
To decompose these interactions, we ran a series of Tukey pairwise comparisons between false insight rates for each anagram type across each level of Semantic Priming and Visual Similarity. These comparisons were almost exclusively significant when the lure was involved and almost exclusively not significant when the lure was not a target of comparison (for the results of all these comparisons see Tables 1–3 in the OSM). We therefore decided to run exploratory analyses on the lures to examine the joint effects of Semantic Priming and Visual Similarity. Specifically, we conducted a two-way, between-subjects ANOVA on false insight rates with the lure anagrams across both levels of Semantic Priming and Visual Similarity. This analysis revealed a main effect for Semantic Priming, with lures eliciting more false insights among participants who were exposed to priming (M = 0.21, CI = 0.17, 0.24) than those who were not (M = 0.07, CI = 0.03, 0.10), F(1,146) = 13.57, p < .001, η2G = .09. No main effect emerged for Visual Similarity, F(1,146) = 3.09, p = .081, η2G = .02, but a significant interaction emerged between Semantic Priming and Visual Similarity, F(1,146) = 7.17, p = .008, η2G = .05.
To decompose this simple interaction effect, we first examined the effect of Visual Similarity at both levels of Semantic Priming. In the absence of Semantic Priming, there was no difference in false insight rates between visually similar (M = 0.07, SE = 0.03) and randomly scrambled (M = 0.09, SE = 0.03) anagrams, t(146) = 0.66, p = .510. In the presence of Semantic Priming, participants reported more false insights when the stimuli were visually similar (M = 0.24, SE = 0.02) than when they were randomly scrambled (M = 0.12, SE = 0.03), t(146) = -3.09, p = .002. Next, we examined the effect of priming at both levels of Visual Similarity. These analyses revealed that when anagrams were visually similar, Semantic Priming elicited significantly more false insights (M = 0.24, SE = 0.02) than no priming (M = 0.07, SE = 0.03), t(146) = -4.57, p < .001. When the anagrams were randomly scrambled, participants did not report more false insights for primed lures (M = 0.12, SE = 0.03) than those who received no Semantic Priming (M = 0.09, SE = 0.02), t(146) = -0.70, p = .485.
Next, we examined the false insight rates of the remaining three anagram types. A 2 (Semantic Priming: present, absent) 𝗑 2 (Visual Similarity: present, absent) 𝗑 3 (Anagram Type: presented target, primed target, random) ANOVA revealed no effect of Semantic Priming (Present: M = 0.02, Absent: M = 0.04), F(1,146) = 1.52, p = .220, η2G = .01, Visual Similarity (Present: M = 0.03, Absent: M = 0.03), F(1,146) = 0.01, p = .941, η2G < .01, or Anagram Type, F(2,292) = 1.27, p = .284, η2G < .01, and no interaction between these three variables, F(2,292) = 0.75, p = .473, η2G < .01. This analysis confirmed that the manipulations uniquely affected the primed lures and had virtually no impact on the remaining anagram types.
For our second preregistered analyses, as in Experiment 1, we examined whether participants gave weaker intensity ratings to false insights compared to correct ones. Again, we looked at false insights across all conditions and computed the mean intensity ratings for true and false insights for participants who experienced both (N = 76). A paired t-test revealed that false insights were again given lower intensity ratings (M = 5.43, SD = 2.18) than correct ones (M = 6.06, SD = 1.88), t(75) = 3.03, p = .003, d = .35.