Reading and learning require not only the incorporation of new information but also the outdating and revision of information that is not or no longer accurate. Findings from several studies suggest that this process of knowledge revision is rather challenging, as outdated information continues to be automatically reactivated. The reactivated but outdated information can hinder comprehension processes (Johnson & Seifert, 1994; Lewandowsky et al., 2012; O’Brien et al., 1998, 2010; Rich & Zaragoza, 2016). This effect is also known as the continued influence effect (Ecker et al., 2022). Previous work has shown that explanations that include causal inferences about why the outdated information is no longer accurate are sufficient to reduce its reactivation and therefore facilitate the knowledge revision process (Johnson & Seifert, 1994; Kendeou et al., 2013, 2019).

Existing work has primarily used non-illustrated texts to evaluate revision processes and outcomes. However, most texts in textbooks, on the web, and in social media are multimodal. We posit that one factor that might facilitate knowledge revision processes is the illustration of the updated information via pictures. As pictures often positively affect learning outcomes (e.g., Mayer, 2021), it is important to investigate how pictures possibly influence knowledge revision processes. Understanding the extent to which and how pictures influence knowledge revision processes is essential for at least two reasons. First, knowledge revision is a central component of learning. By understanding which factors influence this process, it is possible to improve learning (i.e., instruction and instructional materials). Second, such understanding can expand and refine existing theories on knowledge revision during reading, such as the knowledge revision framework (Kendeou & O’Brien, 2014), by identifying how processing of pictures possibly contribute to knowledge revision.

In a series of three experiments, we aimed to identify how and to what extent pictures influence the knowledge revision process. On the one hand, we expected that illustrating updated information would strengthen the activation of the updated information and therefore facilitate knowledge revision processes (Experiments 1, 2, and 3). On the other hand, we expected that illustrating outdated information would strengthen the activation of outdated information and therefore hinder knowledge revision processes (Experiment 3). These hypotheses were based on studies showing better memory for illustrated text information than for non-illustrated text information (Levie & Lentz, 1982; Mayer, 2021). This set of experiments can help us understand under what conditions pictures can facilitate or hinder revision processes during reading. As a starting point in building the theoretical basis of this work, we will focus on the automatic activation process during the reading of non-illustrated texts.

Activation and knowledge revision processes when reading non-illustrated texts

According to most text comprehension models, activation of prior knowledge (i.e., information presented in the text or general world knowledge) is necessary for comprehension (McNamara & Magliano, 2009). Many memory-based comprehension models assume an automatic activation process during reading (Albrecht & O’Brien, 1993; Kintsch, 1998; Myers & O’Brien, 1998; Sanford & Garrod, 2005). For example, the resonance-integration-validation (RI-Val; O’Brien & Cook, 2016a) model incorporates activation as the first of three automatic cognitive processes during reading. According to the RI-Val model, the activation process works with the resonance principle. That means that just-read information sends a signal to long-term memory and automatically activates related information (Myers & O’Brien, 1998; O’Brien & Cook, 2016a). Therefore, irrelevant, false, or outdated information may be automatically activated if it overlaps with the currently read content. The activated content influences subsequent comprehension processes if the activation is above a specific threshold. According to the RI-Val model, the activated information is then automatically integrated with the just-read content and validated (i.e., checked whether the integrated information is consistent; O’Brien & Cook, 2016a, 2016b).

Empirical evidence for the automatic activation of irrelevant or outdated information stems, for example, from research using an adapted version of the contradiction paradigm (O’Brien et al., 1998, 2010). In this paradigm, participants read one of three possible text conditions: consistent, inconsistent, or outdated. The focus is on measuring reading times for a target sentence (that is either consistent or inconsistent with preceding text) and the sentence following this target sentence (i.e., spillover sentence). The target and spillover sentences are the same in all conditions, but the information presented beforehand in the texts varies. For example, consider a text about Susan who loves an old oak tree in her front yard. The target sentence refers to the fact that this tree had to be cut down: “All that remained of the tree was a stump.” In the consistent condition, the readers learn that there was a huge thunderstorm, the tree was struck by lightning, and it had to be cut down. This information is consistent with the target sentence. In the inconsistent condition, Susan’s husband wants to cut down the tree, but Susan convinces him not to cut it down. This information is inconsistent with the target sentence. In the outdated condition, the same information as the inconsistent condition is presented but with one qualifying sentence. In this example, the qualifying sentence describes that the tree had to be cut down due to a thunderstorm. Thus, the initial information is not consistent with the target sentence, but with the addition of the qualifying sentence it is. After a background section, which is the same for all conditions, the target sentence is presented.

Reading times for this target sentence are typically the fastest in the consistent condition, followed by the outdated condition, and the slowest in the inconsistent condition. This difference “spills over” to the following sentence, the spillover sentence, which continues the story but does not directly mention consistent or inconsistent information (O’Brien et al., 2010). The difference in reading times between the consistent and the inconsistent conditions suggests that the earlier information about the tree is activated, integrated, and validated when reading the target sentence. More importantly, the reading times for the target and spillover sentences are significantly longer in the outdated condition than in the consistent condition—even though the target sentence is consistent with information presented in the consistent and the outdated condition (in both conditions, the tree must be cut down). This difference in reading times suggests that the outdated information (Susan convincing her husband not to cut down the tree) was activated, resulting in longer reading times than in the consistent condition.

Note that the difference in reading time between the outdated and the consistent condition is not necessarily an indicator that readers failed to update their knowledge. In fact, many participants can correctly answer that the tree was cut down. Thus, the reading time measure does not measure comprehension or knowledge updating directly. Instead, reading times show that outdated information affects comprehension and reasoning processes, even though participants have updated their knowledge (Ecker et al., 2015; Lewandowsky et al., 2012; Rich & Zaragoza, 2020). However, there must be some limits to the reactivation and influence of outdated information. Otherwise, revision of outdated information would never be possible.

The knowledge revision components framework (KReC; Kendeou & O’Brien, 2014) is one theoretical framework identifying the necessary cognitive processes for knowledge revision during reading. Consistent with the empirical findings described above, this framework assumes that once knowledge is encoded, it cannot be erased entirely from long-term memory and may be automatically activated when related information is presented. It further identifies three necessary cognitive processes for knowledge revision: co-activation, integration, and competing activation. First, the outdated and the updated information must be simultaneously activated and then integrated. That means the outdated and the updated information are connected in memory. According to the competing activation principle, revision is facilitated to the extent that the activation is drawn away from the outdated information towards the updated information. Note that there are also other theoretical assumptions about how and when knowledge revision occurs (Ayers & Reder, 1998; Lee et al., 2017; St. Jacques et al., 2013). In addition, knowledge revision or updating does not necessarily mean that only new, correct information is added to memory. For example, studies on the so-called misinformation effect suggest that later given but wrong information can distort event memories (Loftus, 1975; Loftus & Palmer, 1974). However, in the current experiments, we focus on cases in which one piece of information is given and later clearly revealed to be no longer true.

Empirical studies have so far often focused on how textual features influence the competing activation proposed within the KReC framework and, thus, knowledge revision (Cook & Guéraud, 2005; Ecker et al., 2015; Guéraud et al., 2018; Johnson & Seifert, 1994; Kendeou et al., 2013, 2019; Kendeou & O’Brien, 2014). For example, in a series of experiments, Kendeou et al. (2013) used a version of the adapted contradiction paradigm described above. When giving causal explanations as to why the previously provided information was outdated, reading times for the target sentences did not significantly differ between the consistent and the outdated condition. This indicates that the outdated information no longer interfered with the reading processes. Causal explanations seem especially beneficial in drawing away activation from outdated information, as they provide a rich elaboration network. This rich representational network for updated information strengthens the activation of the updated information, while drawing away activation from the outdated information. Above and beyond textual features (such as causal explanations), other factors may also influence whether outdated information is activated (Ecker et al., 2022), including accompanied pictures or illustrations.

How pictures might influence the knowledge revision process

Findings from a wealth of research show that illustrated texts result in higher memory and comprehension than non-illustrated texts (Carney & Levin, 2002; Levie & Lentz, 1982; Mayer, 2021; Schnotz, 2014). Several mechanisms might contribute to this effect. First, pictures might guide attention toward the illustrated content (see Danielson et al., 2016, for a similar argument), increasing later memory for the illustrated information. Second, pictures are more similar to the outcome of comprehension—namely, a mental model—than text (Glenberg & Langston, 1992; Gyselinck & Tardieu, 1999). For example, Gyselinck and Tardieu (1999) argue, that “pictures can be viewed as one possible expression of a mental model, and presenting pictures may facilitate the construction of a mental model” (p. 212). Third, pictures are often easier to process than text. Studies on scene perception show that as little as 300 ms are sufficient to extract gist form a scene (Biederman et al., 1974; Henderson & Hollingworth, 1999). This ease in processing can lead to the subjective impression that illustrated information is more reliable/true than non-illustrated information (Newman et al., 2012). However, the “truthiness effect of pictures” has a rather small effect size and depends on additional conditions (Nadarevic et al., 2020; Newman et al., 2020).

Finally, following the cognitive theory of multimedia learning (CTML), having information presented in two different modalities (i.e., text and picture) can increase learning outcomes (Mayer, 2021). CTML theorizes that text and pictures are processed in two different, capacity-limited channels. When processing illustrated texts, learners first select the relevant text and picture information, construct a verbal and a pictorial mental model, and subsequently integrate both models into an integrated mental model. CTML argues that having information available in two different codes supports memory (Paivio, 1990) and increases the overall available processing capacity as information is processed in both channels. Moreover, it assumes that the integration of the verbal and the pictorial mental model is essential for deeper understanding.

Taken together, findings from this work suggest that illustrating a text with pictures can guide attention toward the depicted content, facilitate mental model construction, create the impression that the presented information is true, lead to more processing capacity available, and result in an integrated mental representation. All these factors might contribute to the construction of a more coherent and stable mental model in illustrated than in non-illustrated text. In turn, the more coherent and stable model can increase memory and comprehension.

When linking this beneficial effect of pictures with the competing activation principle of the KReC framework, one can argue that pictures might strengthen the activation of the content they illustrate. Consider the aforementioned text example about Susan. In the outdated condition, participants first learned that Susan decided to not cut down the oak tree. However, later it was cut down due to a thunderstorm. Illustrating the updated information with a picture of the fallen tree might result in a more coherent and stable mental model about the fallen oak tree. This might draw activation towards the updated information (fallen tree) and away from the outdated information (tree standing). When reading the target sentence about the fallen tree, the illustrated, updated information might be more activated than the non-illustrated, outdated information. Vice versa, illustrating the outdated information with a picture of the oak tree standing in the garden might lead to easier and faster activation of the outdated information than the updated information when reading the target sentence. Taken together, we expect the picture to influence the competing activation of outdated and updated information.

It is important to note, though, that previous studies investigating how pictures specifically influence knowledge revision have produced mixed results (Danielson et al., 2016; Mason et al., 2017). These studies used different combinations of refutational texts + graph (Danielson et al., 2016) or refutational texts + refutational graph (Mason et al., 2017). Refutational texts are texts designed in accordance with the KReC framework: First, the updated information and the incorrect/outdated information are described (co-activation and integration principle). Then, the correct explanation is given (competing activation principle). After participants had read illustrated or non-illustrated texts, knowledge revision was measured at several time points. In both studies, the graph did not influence knowledge revision above and beyond the effects of the refutational texts. However, none of the studies directly compared a standard text condition with a standard text + standard graph condition. Moreover, both studies measured knowledge revision via post-test knowledge scores. It is unclear if, and if so how, pictures directly influence cognitive processes while processing of the illustrated text. Other measures, such as reading times, can more directly assess these ongoing cognitive processes. In turn, understanding these processes can help to determine if and under what conditions a picture might influence the activation of outdated and updated information and therefore knowledge revision processes.

The current experiments

In the current experiments, we investigated how pictures influence the activation of outdated and updated information. All three experiments used an adapted version of the contradiction paradigm described above. Within this paradigm, reading times are measured, which allows for measuring the ongoing cognitive processes more directly. In Experiments 1 and 2, we focused on whether illustrating the updated information would strengthen the activation of the updated information and therefore facilitate knowledge revision processes. In Experiment 3, we additionally focused on whether illustrating the outdated information would strengthen the activation of the outdated information and therefore hinder knowledge revision processes.

Specifically, in Experiment 1, participants read a text containing only consistent, updated information (consistent text condition) or a text containing outdated information that was later updated (outdated text condition). We varied whether the updated information was illustrated by a picture. Experiment 2 replicated Experiment 1 with a refined material set. In Experiment 3, participants read only the outdated texts. We varied whether the updated or the outdated information was illustrated by a picture. All experiments were preregistered at aspredicted.org and approved by the local ethic commission.

Experiment 1

In Experiment 1, our aim was to examine if illustrated updated information would strengthen the activation of the updated information. We used the contradiction paradigm and relied on texts already used by O’Brien et al. (2010).Footnote 1 This made it possible to compare the results with those in the extant literature. We varied the factor text condition (consistent information versus outdated information that is then updated) and picture (text is not illustrated vs. text is illustrated with picture showing the updated information). In line with the competing activation principle during knowledge revision (Kendeou & O’Brien, 2014) and research showing better comprehension and memory for illustrated texts compared with non-illustrated texts (e.g., Mayer, 2021), we hypothesized that in the outdated condition the picture would lead to faster reading times for target sentences compared with texts without pictures. We assumed that when contradictory information is available (i.e., outdated and updated information), illustrating the updated information would strengthen the activation of this information (competing activation principle). In the consistent condition, we did not expect any effect of the picture factor. This was due to two reasons. First, from a theoretical perspective we expected the picture to influence the competing activation of outdated and updated information. However, there is no outdated information in the consistent condition, and therefore no competing activation. Second, in a previous study we compared consistent illustrated and non-illustrated texts and did not find any effect of the picture on target sentences’ reading times (Frick & Schüler, 2023) indicating that with consistent texts, pictures have no add-on effect. To summarize this, we hypothesized a significant interaction effect between the factors text condition (consistent vs. outdated) and picture (non-illustrated vs. illustrated with a picture showing the updated information). For consistent texts, we expected no difference in the reading times for the target sentences as a function of the factor picture. For outdated texts, we expected faster reading times for the target sentences if the updated information was illustrated compared with if the updated information was not illustrated. The preregistration can be found online (https://aspredicted.org/ws89r.pdf).

Method

Participants and design

We varied the factors text condition (consistent vs. outdated) and picture (non-illustrated vs. illustrated with a picture showing the updated information) within participants (i.e., 2 × 2 within-subjects design). We conducted a power simulation based on the data by O’Brien et al. (2010) and previous data (Frick & Schüler, 2023). Both factors were dummy-coded (consistent text = 0, outdated text = 1; non-illustrated = 0, illustrated = 1). The power simulation revealed that 400 participants were necessary to detect an interaction effect between the factors text condition and picture of −120 ms with sufficient power (1 − β >/= 0.8).Footnote 2 Due to possible dropout and in line with our pre-registration, we recruited 450 participants. The participants were recruited via Prolific and received ₤4.25 for their participation. Ten participants had to be excluded due to data-saving issues or self-reported technical problems, three were older than 35 or did not report their age, and 16 answered more than 40% of the attention check tasks incorrectly (see below for details). Hence, 421 participants remained for the analysis (257 = male, 154 = female, 10 = non-binary, Mage = 28.23 years, SD = 4.56).

Materials

Sixteen texts were adapted from O’Brien et al. (2010) and slightly modified for our research purpose. Each text was presented in four conditions: consistent non-illustrated, consistent illustrated, outdated non-illustrated, and outdated illustrated. Each text consisted of six parts: an introduction, an elaboration section, a background section, the target sentence, the spillover sentence, and an ending. In the introduction (M = 10.06 words, SD = 1.82), a protagonist was introduced. The elaboration section contained the experimental manipulation (i.e., consistent information vs. outdated information, which was later updated) and, hence, differed between the consistent and the outdated version. In the original material, the elaboration sections differed in length. We adjusted the length so that the consistent (M = 95.94 words, SD = 3.78) and outdated (M = 95.63 words, SD = 2.69) text conditions were equal in length. The next section, the background section (M = 59.31 words, SD = 2.64), gave additional information about the protagonist but was irrelevant for the target sentence. It served to background the information given in the elaboration section making activation processes necessary when reading the target sentence. The target sentence referred to the information given in the elaboration section. The target sentences were controlled for length at character level (M = 40.19 characters, SD = 1.59). Then, the spillover sentence followed. The spillover sentences were controlled for length at character level as well (M = 38.43 characters, SD = 3.41). The ending concluded the story (M = 11.56 words, SD = 1.46).

In the conditions with pictures, the relevant (updated) information for the target sentence (e.g., the fallen tree) was illustrated by a picture (elaboration picture). This information was always given in the last sentence of the elaboration section. In addition, two other pictures were presented for each text to mask the purpose of the investigation: one picture illustrating a sentence of the introduction (introduction picture), and the other picture illustrating a sentence in the background section (background picture). The three pictures were colored, drawn by the research team, and depicted the scene described in the corresponding sentence. They were the same for the consistent and the outdated text condition. Between the last illustrated sentence and the target sentence were always at least two sentences. A full example text including the picture can be seen in Table 1.

Table 1 Example text and picture material for Experiment 1
Full size table

To counterbalance texts and text conditions across participants, four lists were created that included 16 texts each. Every list contained each text only once and four texts in each of the four possible conditions. Participants were assigned to one of these four lists. The order in which the texts were presented within a list was randomized.

Measures

Dependent measure

We measured the reading time for the target sentences in milliseconds, estimated from their appearance on the screen until the space bar was pressed. For additional analysis, we also measured the reading times for spillover sentences. This and the following experiments were programmed in PsychoPy (Bridges et al., 2020; Peirce et al., 2019).

Attention-check measures

After each text, participants had to answer a short yes–no question about the text’s content. This text attention check measure ensured that participants were paying attention to the presented content. The questions did not refer to information given in the elaboration sections. After reading all 16 texts, participants also completed a picture attention check measure. Participants were asked to indicate whether they had seen the picture presented on the screen before (i.e., old) or whether it was new. We included eight old and eight new pictures in the task. The elaboration pictures were never included in this task. Importantly, participants were not informed beforehand about this attention check to avoid drawing their attention to the pictures and altering naturalistic processing. Therefore, participants only learned that they had to complete this picture attention check measure after reading all experimental texts.

Procedure

Participants were randomly assigned to one of the four lists. After answering demographic questions about age, gender, and educational level, participants learned that they were to read several short texts and answer one yes–no question about each text after its presentation. Each text started with the words “READY? Please press the space bar to start the story.” The texts were then presented sentence by sentence. Every time participants pressed the space bar, the current sentence (and picture, if applicable) disappeared, and the next sentence appeared. If a picture was present, it was printed above the corresponding sentence. After the final sentence of each text, the word “QUESTION” appeared on the screen for 1,000 ms. Then, the corresponding text attention check measure was presented. Participants responded via two different key presses: if the answer to the question was yes, participants pressed the K key; if it was no, they pressed the D key. To practice the whole procedure, participants read two consistent test text passages (one illustrated, one non-illustrated). Then, the 16 texts were presented in randomized order. After participants had read all 16 texts, they completed the picture attention check measure. Finally, participants answered questions about possible technical problems and were debriefed and thanked. The experiment lasted about 25 minutes.

Results

The data were analyzed with R (R Core Team, 2022). Linear mixed-effect models were fitted to the data with the lme4 package (Bates et al., 2015); the lmerTest package (Kuznetsova et al., 2017) was used for the significance values. Single comparisons were computed with the emmeans package (Lenth, 2022).

The results of the text and picture attention-check measure can be seen in Table A1 in Appendix A. Furthermore, Table B1 in Appendix B lists the mean reading times for each illustrated sentence and the non-illustrated counterpart.

We excluded target sentences with exceptional fast (<500 ms) or slow (>7,000 ms) reading times, which resulted in 3.99% of removed data. Table 2 lists descriptive statistics.

Table 2 Mean reading times (in ms), standard deviations, estimated marginal means, and standard errors for the target sentences as a function of the factors text condition and picture
Full size table

Data distributions for this and the following experiments were graphically checked with q-q plots, scatter plots for residuals, and histograms. Because the reading time data were skewed, we applied a log-transformation.Footnote 3

We fitted a linear mixed-effect model with participants and texts (item) as random effects (random intercept) and text condition (dummy coded: consistent = 0 outdated = 1) and picture (dummy coded: non-illustrated = 0, illustrated = 1) as fixed effects to the reading times for the target sentences. Table 3 shows the model-estimated fixed and random effects. We found a significant main effect of the factor text condition, p = .028 with longer reading times for outdated than for consistent texts. Moreover, the factor picture, p = .012, was significant with longer reading times in the non-illustrated than in the illustrated conditions. The interaction between the factors text condition and picture was not significant, p = .662. Figure 1 illustrates this result.

Table 3 Fixed and random effects of the linear mixed-effects model for the target sentences
Full size table
Fig. 1
figure 1

By the linear mixed-effect model estimated and back-transformed reading times as a function of the factors text condition and picture. Error bars show ±1 SE

Full size image

In line with our preregistration, we also analysed the reading times for the spillover sentences (i.e., the sentence following the target sentence). We applied the same exclusion criteria as for the target sentences and additionally excluded reading times of spillover sentences that followed excluded target sentences (5.94% of removed data). The mean reading times for the spillover sentences as a function of the factors text condition and picture can be seen in Table C1 in Appendix C. We then log transformed the data and fitted a linear mixed-effect model to them. Again, participants and texts (item) were entered as random effects (random intercept), and text condition (dummy coded: consistent = 0, outdated = 1) and picture (dummy coded: non-illustrated = 0, illustrated = 1) were entered as fixed effects. The model revealed a significant main effect of the factor picture, estimate = −0.03, SE = 0.01, t(5905.98) = −2.99, p = .003. Reading times were significantly faster in illustrated conditions than in non-illustrated conditions. The effect of the factor text condition, estimate = 0.02, SE = 0.01, t(5904.72) = 1.46, p = .145, and the interaction effect, estimate = 0.02, SE = 0.01, t(5905.13) = 1.04, p = .299, were not significant.

Discussion

In Experiment 1 we investigated whether pictures showing updated information can facilitate knowledge revision. To do so, we designed texts that either did or did not contain outdated information, which was later updated. Additionally, we varied whether the updated information was illustrated or not. We measured reading times for the target sentence referring to the elaboration section and for the spillover sentence. For outdated texts, we expected faster reading times for the target sentences in the illustrated than in the non-illustrated condition. For the consistent texts, we did not expect any effect of the factor picture.

We found a main effect of the factor picture with faster reading times in the illustrated than in the non-illustrated conditions in both text conditions. As hypothesized, reading times were faster in the outdated illustrated condition than in the outdated non-illustrated condition, suggesting the pictures might strengthen the activation of the updated information and thereby reduce the influence of outdated information. As the target sentences always referred to the updated information, the reduction in reading time could suggest that pictures did indeed strengthen the activation of the updated information. However, and contrary to our hypothesis, reading times were also faster in the consistent illustrated condition than in the consistent non-illustrated condition. This was especially surprising as an earlier study using a similar procedure did not find reduced reading times in the illustrated compared with the non-illustrated condition for consistent texts (Frick & Schüler, 2023). This finding suggests that illustrations overall facilitate reading comprehension processes.

Moreover, we also found a main effect of the factor text condition. Reading times on target sentences were longer for texts containing outdated information and therefore required knowledge revision compared with texts that only contained consistent information. This is in line with previous studies showing longer reading times for texts requiring knowledge revision (O’Brien et al., 2010).

As cognitive processes evoked by one sentence might also influence the reading time of the following sentence (i.e., spillover effect), we also analysed spillover sentences’ reading times. The spillover sentences’ reading times were faster for illustrated than for non-illustrated conditions. The text condition did not influence the spillover sentences’ reading times.

Taken together, we found faster reading times for target and spillover sentences for illustrated texts compared with non-illustrated texts for both text conditions. These findings only partially supported our hypothesis since we expected an effect of picture presentation only for the outdated text conditions. One possible explanation for this unexpected finding could be differential text-picture alignment between the two text conditions. More precisely, the same elaboration picture was combined with different sentences depending on the text condition (see, for example, the elaboration picture in Table 1). We conducted a follow-up experiment (N = 128) to explore whether the text-picture alignment differed depending on the text condition. It revealed that the sentence–picture combinations of the outdated text condition fitted significantly better than the sentence-picture combinations of the consistent text condition, MDifference = −0.27, t(1887) = −7.03, 95% CI [−0.35, −0.20], p < .001. A more detailed description of this experiment can be found in the preregistration (https://aspredicted.org/79768.pdf) and on OSF (see link in open practices statement). To address this confounding variable, we revised the material set and conducted a replication in Experiment 2. This material set controlled better for text–picture alignment between the consistent and the outdated text conditions.

Experiment 2

The aim of Experiment 2 was to replicate Experiment 1 with an improved material set. More precisely, the material set in Experiment 2 better controlled for text–picture alignment (see the Materials section for details). As in Experiment 1, we expected significantly faster reading times for target sentences in the outdated text illustrated condition compared with the outdated text non-illustrated condition. We did not expect any difference in reading times as a function of the factor picture for the consistent conditions. The preregistration can be found online (https://aspredicted.org/4fg69.pdf).

Method

Participants and design

Again, we varied the factor text condition (consistent vs. outdated) and picture (non-illustrated vs. illustrated with picture showing the updated information) within participants (i.e., 2 × 2 within-subjects design). Based on the data of Experiment 1, we computed a power simulation. As in Experiment 1, both factors were dummy-coded (consistent text = 0, outdated text = 1; non-illustrated = 0, illustrated = 1). The power simulation revealed that 400 participants were necessary to detect an interaction effect between the factors text condition and picture of -85 ms with sufficient power (1 − β >/= 0.8).Footnote 4 Due to possible drop out and in line with our preregistration, we recruited 450 participants. The participants were recruited via Prolific and received ₤4.25 for their participation. One participant withdrew the consent for data processing, for six participants data saving was not working, 10 participants were excluded due to self-reported technical problems, four were older than 35 or did not report their age, and seven participants made too many errors (error rate > 40%) in the attention check task. Hence, 422 participants remained (243 = male, 165 = female, 14 = non-binary, M = 28.56 SD = 4.53).

Materials

The materials consisted of eight texts adapted from Cook (2014, third Experiment),Footnote 5 and eight newly generated texts (16 texts in total). The basic structure of the material was the same as in Experiment 1. Each text was presented in four conditions (i.e., consistent non-illustrated, consistent illustrated, outdated non-illustrated, outdated illustrated) and consisted of six different parts (i.e., an introduction, an elaboration section, a background section, the target sentence, the spillover sentence, and an ending). Again, in the illustrated conditions, the (updated) information conveyed through the last sentence of the elaboration section was illustrated by a colored picture drawn by the research team. Additionally, the first sentence of the elaboration and the second or third sentence of the background section were illustrated to mask the purpose of the investigation. In contrast to the texts used in Experiment 1, the consistent and the outdated elaboration section differed in only two sentences: The sentence of the elaboration section with the critical information differed in one word and one sentence was added at the end of the outdated elaboration section. For example, one text was about the protagonist Ellen, who buys new clothes. In the consistent condition, she sees a dress which she buys. In the outdated condition, she first sees an overall and shortly before buying it, she decides to buy a dress instead. In the outdated elaboration section, one qualifying sentence was added to describe this fact: “While walking through the store she found a dress which was way more beautiful.” Importantly, the sentence which was combined with the picture was the same for both conditions: “Ellen went to the checkout to buy the dress.” Hence, all pictures were presented with the same sentences in both conditions. This ensured that the text–picture alignment was the same in both conditions. The full example text, including the pictures, can be seen in Table 4. The counterbalancing of the different text versions was done as in Experiment 1.

Table 4 Example text and picture material for Experiment 2
Full size table

Measures

Dependent measures and attention-check measures were the same as in Experiment 1.

Procedure

The procedure was the same as in Experiment 1.

Results

The results of the text and picture attention-check measure can be seen in Table A1 in Appendix A. Furthermore, Table B1 in Appendix B lists the mean reading times for each illustrated sentence and the non-illustrated counterpart.

Again, we excluded target sentences with exceptional fast (<500 ms) or slow (>7,000 ms) reading times, which resulted in 5.20% of removed data. Table 5 lists the mean reading times and standard deviations for the target sentences as a function of the factors text condition and picture.

Table 5 Mean reading times (in ms), standard deviations, estimated marginal means, and standard errors for the target sentences as a function of the factors text condition and picture
Full size table

We log-transformed the reading time data as they were skewed. We fitted a linear mixed-effect model with participants and texts (item) as random effects (random intercept) and text condition (dummy coded: consistent = 0 outdated = 1) and picture (dummy coded: non-illustrated = 0, illustrated = 1) as fixed effects to the reading times for the target sentences. Table 6 shows the model-estimated fixed and random effects.

Table 6 Fixed and random effects of the linear mixed-effects model for the target sentences
Full size table

This time, only the main effect of the factor text condition was significant, p = .007, with longer reading times for the outdated than for the consistent texts. The main effect of picture, p = .076, and the interaction effect of the factors text condition and picture, p = .278, were not significant. Figure 2 illustrates this result.

Fig. 2
figure 2

By the linear mixed-effect model estimated and back-transformed reading times as a function of the factors text condition and picture. Error bars show ±1 SE

Full size image

Again, we analysed the reading times for the spillover sentences to detect any delayed effects. After applying the same exclusion criteria as for the spillover sentences in Experiment 1 (6.87% of removed data), we log transformed the data, and fitted a linear mixed effect model to them. The descriptive reading times for the spillover sentences and the estimated marginal means can be seen in Table C2 in Appendix C. Again, participants and texts (item) were entered as random effects (random intercept), and text condition (dummy coded: consistent = 0 outdated = 1) and picture (dummy coded: non-illustrated = 0, illustrated = 1) were entered as fixed effects. The model revealed a significant main effect of the factor text condition, estimate = −0.04, SE = 0.01, t(5852) = 3.50, p < .001. Reading times were significantly faster in the consistent than the outdated condition. The effect of the factor picture, estimate = 0.004, SE = 0.01, t(5852) = 0.34, p = .731, and the interaction effect, estimate = −0.03, SE = 0.02, t(5852) = −1.77, p = .077, were not significant.

Discussion

In Experiment 2, we replicated Experiment 1 with an improved material set. We found a main effect of the factor text condition for target and spillover sentences: target and spillover sentences’ reading times were significantly longer for texts containing outdated information compared with texts that only contained consistent information. For the target sentences’ reading times, this finding aligns with previous research (O’Brien et al., 2010) and with the results of Experiment 1.

More importantly, and contrary to our hypothesis, the interaction between the factors text condition and picture was not significant. Also, the main effect of picture, which was obtained in Experiment 1, was not significant. In Experiment 1, where a main effect of picture presentation was observed, pictures facilitated reading times independent of text condition. It is important to note that the descriptive data of Experiment 2 pointed towards faster reading times in the illustrated conditions as well. As no knowledge revision processes were necessary in the consistent condition, this might imply that pictures facilitate reading processes in general and not the specific cognitive processes involved in knowledge revision.

To summarize, in Experiments 1 and 2 we investigated whether illustrating the updated information with a picture would strengthen the activation of that information. The findings were relatively mixed and pointing into a potential general facilitative effect of pictures during reading. We reasoned that if pictures strengthen the activation of the depicted content in general, depicting outdated information should conversely strengthen the activation of the outdated information. We conducted Experiment 3 to explore this question.

Experiment 3

In Experiments 1 and 2, we focused on comparing text conditions with pictures showing the updated information to text conditions without pictures. If pictures strengthen the activation of the depicted content, depicting outdated information should conversely strengthen the activation of the outdated information. In other words, if the outdated information is illustrated by a picture, knowledge revision might become even more challenging, as the outdated information is more likely to be reactivated. We conducted Experiment 3 to investigate this hypothesis. Participants were presented with the outdated text conditions used in Experiment 2, and we varied whether the text was illustrated by a picture showing the updated information, illustrated by a picture showing the outdated information, or non-illustrated. We expected longer reading times for the target and spillover sentences when the picture showed the outdated information compared with when the picture showed the updated information. We reasoned that if pictures facilitate activation of information, the difference between the condition with a picture showing the outdated information and the condition with a picture showing the updated information will be larger than the differences we found in Experiments 1 and 2, and ‘spill over’ to the next sentence. Moreover, we also expected longer reading times in the non-illustrated condition compared with the condition with a picture showing the updated information. The preregistration can be found online (https://aspredicted.org/4ka99.pdf).

Method

Participants and design

We varied within participants the factor picture version: Text was illustrated by a picture showing the updated information, illustrated by a picture showing the outdated information, or non-illustrated. Based on the data from Experiments 1 and 2, we conducted a power simulation. The power simulation revealed that 420 participants were necessary to detect a difference of 70 ms between the two illustrated conditions (i.e., picture shows updated vs. outdated information) with sufficient power (1 − β >/= 0.8).Footnote 6 Due to possible dropout and in line with our preregistration, we recruited 470 participants. The participants were recruited via Prolific and received ₤4.25 for their participation. Six participants were excluded due to self-reported technical problems, two reported only poor language skills, three did not report their age, and 18 participants made too many errors (error rate > 40%) in the attention check tasks. Hence, 441 participants remained (232 = male, 185 = female, 23 = non-binary, 1 = none of these options, Mage = 27.95 years, SD = 4.48).

Materials

We selected 12 outdated text conditions used in Experiment 2. We varied whether the texts were illustrated by a picture showing the updated information (same picture as in Experiment 2), illustrated by a picture showing the outdated information, or not illustrated. The pictures for the introduction and the background section were the same as in Experiment 2. Table 7 shows an example elaboration section with text and pictures. The picture showing the outdated information was similar to the picture showing the updated information, only the updated information was replaced by the outdated information (e.g., picture showing the protagonist with an overall vs. a dress). The counterbalancing of the different text versions was done as in Experiment 1. However, only three lists were generated this time, as only three different conditions existed.

Table 7 Example elaboration section for Experiment 3
Full size table

Measures

Dependent measures and attention-check measures were conceptually the same as in Experiment 1.

Procedure

The procedure was the same as in Experiment 1.

Results

The results of the text and picture attention-check measure can be seen in Table A1 in Appendix A. Furthermore, Table B3 in Appendix B lists the mean reading times for each illustrated sentence and the non-illustrated counterpart.

We excluded target and spillover sentences with exceptional fast (<500 ms) or slow (>7,000 ms) reading times. Moreover, we excluded spillover sentences that followed excluded target sentences. This resulted in 2.60% of removed data for the target sentences and 4.34% of removed data for the spillover sentences. Table 8 lists the mean reading times and standard deviations for the target and spillover sentences as a function of the factor picture version and sentence type.

Table 8 Mean reading times (in ms), standard deviations, estimated marginal means, and standard errors as a function of the factors picture version and sentence type
Full size table

We log transformed the reading time data as they were not normally distributed. We fitted a linear mixed-effect model with participants and text (item) as random effects (random intercepts) and picture version (dummy coded with updated picture as reference category) and sentence type (effect coded: 0.5 = target sentence, −0.5 = spillover sentence) as fixed effects to the log-transformed reading times.Footnote 7 Table 9 (upper part) shows the model-estimated fixed and random effects.

Table 9 Model estimated fixed and random effects for the model with updated picture as reference category (upper part) and non-illustrated as reference category (lower part)
Full size table

Consistent with our first hypothesis, we found significantly longer reading times when the outdated compared with when the updated information was illustrated, p = .011. Consistent with our second hypothesis, reading times were significantly longer in the non-illustrated condition than in the condition with a picture showing the updated information, p <. 001. There was no significant interaction between the picture version and sentence type.

To compare the non-illustrated condition with the condition with a picture showing the outdated information, we fitted the same model as in the main analysis but with the non-illustrated condition as the reference category.Footnote 8 Reading times were significantly longer in the non-illustrated condition than in the condition with a picture showing the outdated information, p <. 001. Figure 3 illustrates this result pattern.

Fig. 3
figure 3

By the linear mixed-effect model estimated and back-transformed reading times as a function of the factors text version and sentence type. Error bars show ±1 SE

Full size image

Discussion

In Experiment 3, we focused on examining further the effects of pictures on knowledge revision processes. We reasoned that if pictures strengthen the activation of the depicted content (as suggested by Experiments 1 and 2), depicting outdated information should conversely strengthen the activation of the outdated information. In other words, if the outdated information is illustrated by a picture, knowledge revision might become even more challenging, as the outdated information is more likely to be reactivated. Thus, we expected longer reading times for the target and spillover sentences when the picture showed the outdated information compared with when the picture showed the updated information. Additionally, we expected longer reading times in the non-illustrated condition compared with the condition with a picture showing the updated information. Results were in line with both hypotheses revealing longer reading times when the outdated compared with the updated information was illustrated and longer reading times in the non-illustrated compared with the updated illustrated condition. Surprisingly, reading times were also longer in the non-illustrated condition compared with the condition with a picture showing the outdated information.

Taken together, reading times were the longest in the non-illustrated condition, followed by the illustrated condition with a picture showing the outdated information, and the fastest in the illustrated condition with a picture showing the updated information. This might suggest that even pictures showing the outdated information can facilitate reading processes. Therefore, our data speak against the claim that illustrating outdated information with a picture hinders knowledge revision processes. However, this finding will need to be replicated in future studies to better understand its mechanisms.

Several mechanisms might explain why even outdated pictures seemed to facilitate knowledge revision processes during reading. One possibility is that pictures might draw attention to the illustrated content. This attention allocation is likely not only directed to the specific content (e.g., a dress or an overall) but also to the general concept (e.g., clothes). In the example presented in Table 7, this would mean that attention shifted towards the concept of clothes more generally. When reading the target sentence, all items connected to the concept of clothes might be activated more easily, leading to faster reading times. Nonetheless, the content of the pictures seems to be crucial, as reading times were faster when the updated than when the outdated information was illustrated.

General discussion

Outdated information (i.e., information that is not or no longer accurate) continues to be automatically activated during reading (O’Brien et al., 2010). This can hinder further reading and comprehension processes (Johnson & Seifert, 1994; Lewandowsky et al., 2012; O’Brien et al., 1998, 2010; Rich & Zaragoza, 2016). Existing research mainly focused on non-illustrated texts and on how textual features influence the activation of outdated information (Kendeou et al., 2013; Tippett, 2010). However, as most texts are multimodal, it is important to understand how pictures influence the activation of outdated as well as updated information, and, hence, knowledge revision processes.

We situated this work within the KReC framework (Kendeou & O’Brien, 2014), which posits that knowledge revision takes place if the activation of the updated information is stronger than the activation of the outdated information (via a competing activation mechanism). We argued that pictures possibly strengthen the activation of the information they illustrate (e.g., Gyselinck & Tardieu, 1999; Mayer, 2021). Illustrating the updated information should consequently lead to stronger activation of the updated information compared with the outdated information, supporting knowledge revision. Vice versa, illustrating the outdated information should lead to stronger activation of the outdated information compared with the updated information, hindering knowledge revision.

We conducted a series of three experiments to test these hypotheses. All three experiments used an adapted version of the contradiction paradigm (O’Brien et al., 1998, 2010). In Experiments 1 and 2, participants read texts that contained outdated information that was later updated (i.e., outdated text condition) or texts that only contained the updated information (i.e., consistent text condition). We varied whether a picture illustrated the updated (consistent) information or not. In Experiment 3, participants read only the outdated text condition. We varied whether outdated, updated, or no information was illustrated. In all experiments, reading times for a target sentence referring to the elaboration section and the spillover sentence following the target sentence were measured.

Taken together, we found faster reading times for illustrated than for non-illustrated texts (Experiments 1 and 3; Experiment 2 descriptively). This finding demonstrates that pictures influence both reading and knowledge revision processes. We argued above that text information illustrated by a picture might yield a more coherent and stable mental model than non-illustrated text information (Gyselinck & Tardieu, 1999; Mayer, 2021). This more coherent and stable mental model may lead to a stronger activation of the illustrated information. Even though it would be tempting to conclude that pictures influence competing activation (principle proposed within the KReC framework), such a conclusion is not supported by the findings. The main effect of picture presentation on reading times and, more specifically, the faster reading times in the consistent illustrated conditions than in the consistent non-illustrated conditions (Experiment 1 and descriptively in Experiment 2) rather suggest that pictures might have influenced the activation principle in general, as no knowledge revision was required in the consistent text conditions. That is, pictures might generally facilitate the activation of related information. Furthermore, as the effect of the pictures did not depend on whether knowledge revision was required (i.e., there was no interaction between the text conditions and the picture in Experiments 1 and 2), it is also likely that pictures did not (only) influence the cognitive processes involved in knowledge revision but rather the cognitive processes involved in reading comprehension more broadly. This could also mean that pictures might not have (only) influenced activation processes, but the integration and validation processes proposed within the RI-Val model (O’Brien & Cook, 2016b) as well.

Next to this general effect of picture presentation on reading speed of the target sentence, we would like to note that reading times were faster when the updated than when the outdated information was illustrated (Experiment 3), suggesting that the content of the pictures seems to be crucial as well. If we assume that the picture presentation leads to a more stable and coherent mental model of the presented information (Gyselinck & Tardieu, 1999; Mayer, 2021), the mental model arising from illustrating outdated information should interfere more strongly with the reading of the target sentence than the mental model that arises when the updated information is illustrated. This could explain the difference between illustrating updated and outdated information in Experiment 3.

Finally, we also found longer reading times for outdated than for consistent texts (Experiments 1 and 2), which is in line with the encoding and activation principle proposed within the KReC framework (Kendeou & O’Brien, 2014) and theories of text comprehension (Kintsch, 1998; O’Brien & Cook, 2016a). As described above, once information is encoded in long-term memory, it is not entirely erased, even if it is outdated. Therefore, the outdated information can be activated through the automatic activation process. The longer reading times for the target sentences in the outdated conditions support that argument and suggest that the outdated information was indeed reactivated. This finding is robust and consistent with many other studies showing that outdated information is automatically activated (Guéraud et al., 2018; Kendeou et al., 2013; O’Brien et al., 1998, 2010).

Limitations and future research

When interpreting these findings, it is important to note that we used rather simple texts and pictures about different main characters. The effect of pictures on knowledge revision processes may vary depending on the complexity of the material. It is, for example, possible that pictures may influence knowledge revision processes when the information to be revised is more complex. For example, a lot of research on the effect of illustrated versus non-illustrated texts focused on more complex mechanisms, such as weather phenomena (e.g., lightning formation, tornado formation) or the functioning of a pulley system (Eitel et al., 2013; Mayer, 2021; Schüler, 2019). It might be possible that the beneficial effect of pictures is stronger if the material depicted in the picture is more complex to describe in the text.

Moreover, our main dependent variable was reading times. While this variable can hint at the underlying cognitive processes and show whether the ongoing cognitive processes were facilitated or disrupted, it cannot answer the question of how exactly the picture changed the involved cognitive processes. For example, the reading time measure cannot determine which of the involved cognitive processes (i.e., activation, co-activation, but also integration and validation) were affected by the pictures. More detailed measures, such as eye-tracking or think-alouds, could provide further insights into these questions. Moreover, reading times do not directly measure comprehension, nor do they measure whether participants revised their knowledge. More direct measures, such as knowledge scores or reasoning tasks are necessary to access knowledge revision more directly. Future studies should also take these variables into account. Nevertheless, measuring the underlying cognitive processes via reading times is important in understanding whether pictures can influence knowledge revision processes.

Finally, future studies should also consider memory theories stemming from different but interconnected research areas such as neurobiological theories investigating memory effects. For example, studies using fMRI can provide insights into ongoing neuronal mechanisms (e.g., St. Jacques et al., 2013). Furthermore, studies differentiating between the continued influence effect (investigated in the current experiments) and the misinformation effect (i.e., later given, suggestive information distorts event memories; Loftus, 1975) might offer a deeper understanding for the involved cognitive processes and help us understand better when memories are updated (Ecker et al., 2015). By connecting these different research areas, it might be possible to understand further the exact processes involved in knowledge revision and updating.

Open questions aside, our results provide important new insights into knowledge revision processes. We were able to show that pictures illustrating updated or outdated information influenced reading and knowledge revision processes. As most texts we encounter in education and our current information ecosystem are multimodal, it is essential to understand how pictures influence cognitive processing.