Across all trials, participants selected the correct (named) picture on 79.7 % of trials. Participants had significantly higher accuracy for known trials (99.9 %) than for unknown trials (59.5 %), t(22) = 16.07, p < .0001, d = 3.35. Participants took an average of 1974.8 ms to make their selection across all trials. They were significantly faster to respond on known trials (M = 1,370.4 ms) than on unknown trials (M = 2,579.3 ms), t(18) = 17.76, p < .0001, d = 4.07.
Across all trials, participants made a correct response on auditory/picture congruity on 65.7 % of trials. To compare accuracy on the congruity task, a 2 (congruency: congruent/incongruent) × 2 (knowledge: known/unknown) repeated measures ANOVA was run on the mean accuracy for each participant. There was a significant interaction of congruency and knowledge [F(1, 22) = 72.51, p < .0001, η
2 = .74]. Post-hoc paired-samples t tests showed significantly higher accuracies for unknown incongruent (86.4 %) than unknown congruent trials (33.2 %), t(22) = 8.54, p < .0001, d = 1.78; this pattern may reflect a bias on unknown trials toward responding that the auditory cue did not match the picture. We found no accuracy differences between known congruent (98.2 %) and known incongruent trials (98.8 %), t(22) = 1.53, p = .14, d = 0.32.
To compare RTs on the congruity task, a 2 (congruency: congruent/incongruent) × 2 (knowledge: known/unknown) repeated measures ANOVA was run on the mean RTs. There was a significant interaction of knowledge and congruency [F(1, 22) = 7.21, p < .05, η
2 = .002]. Post-hoc paired-samples t tests showed significantly slower RTs for unknown congruent trials (M = 927.8 ms) than for unknown incongruent trials (M = 878.7 ms), t(22) = 2.69, p < .05, d = 0.56. No differences in RTs was apparent between known congruent trials (M = 759.0 ms) and known incongruent trials (M = 793.0 ms), t(22) = 1.53, p = .14, d = 0.32. That participants responded faster to congruent trials for known words, but to incongruent trials for unknown words, may again reflect a response bias in the incongruent condition.
Word familiarity ratings
Known words were given significantly higher word familiarity ratings (M = 8.99) than were unknown words (M = 2.58), t(22.05) = 31.35, p < .0001, d = 6.54.
Table 1 presents the mean values on all of the dependent measures derived from eye movements for the known and unknown conditions. Sample eye movement data are shown in Fig. 3. We observed a greater total number of fixations for unknown than for known trials, t(22) = 17.71, p < .0001, d = 3.69. On average, mean fixation durations on the named picture were longer in the known than in the unknown condition, t(22) = 2.39, p < .05, d = 0.50. The length of the first fixation on the named picture was longer for known than for unknown trials, t(22) = 3.68, p < .01, d = 0.77. The length of the first dwell on the named picture was also longer for the known than for the unknown condition, t(22) = 3.86, p < .001, d = 0.80. The latencies to first fixation on the named picture, t(22) = 8.56, p < .0001, d = 1.79, and to refixation, t(22) = 8.64, p < .0001, d = 1.80, were both shorter for known than for unknown trials. The proportion of time spent fixating on the stimulus (i.e., proportion of fixation duration on the stimulus) was greater in the known than in the unknown condition, t(22) = 12.99, p < .0001, d = 2.71. The proportion of time spent dwelling on the stimulus (i.e., looking at the named picture, with or without fixation) was also greater in the known than in the unknown condition, t(22) = 11.13, p < .0001, d = 2.32. The stimulus was the first picture to be fixated on a significantly higher percentage of known than of unknown trials, t(22) = 2.46, p < .05, d = 0.51. Finally, the stimulus was also the last picture to be fixated on a significantly higher percentage of known than of unknown trials, t(22) = 19.55, p < .0001, d = 4.08.
The pupillometry data are also summarized in Table 1. Larger peak dilations (relative to baseline) were observed for unknown than for known trials, t(22) = 9.24, p < .0001, d = 1.93. Mean changes in pupil size from baseline were also larger for unknown than for known trials, t(22) = 8.32, p < .0001, d = 1.73. The maximum percent change in pupillary dilation was also larger for unknown than for known trials, t(22) = 10.86, p < .0001, d = 2.26.
ERPs for the four conditions, as well as topographical plots of the incongruent–congruent difference for the known and unknown conditions, are presented in Fig. 4.
A 2 (congruency: congruent/incongruent) × 2 (knowledge status: known/unknown) × 3 (site: frontal/central/parietal) × 2 (hemisphere: left/right) repeated measures ANOVA was performed on the average amplitudes over a window from 450 to 700 ms after sound presentation (shaded regions in Fig. 4). The full results can be found in Table 2. We observed a significant three-way interaction of knowledge, congruency, and site [F(2, 44) = 6.14, p < .01, η
2 = .07].
To explore this interaction, we performed a 2 (congruency) × 2 (knowledge) ANOVA for each site (collapsed over hemisphere). There was a significant interaction of congruency and knowledge at parietal sites [F(1, 22) = 7.34, p < .05, η
2 = .02]. We investigated this interaction at parietal sites by performing a two-way (congruency) ANOVA separately for the known and unknown words. For the known condition, a main effect of congruency emerged [F(1, 22) = 7.69, p < .05, η
2 = .07]: The mean amplitude observed to known congruent items (M = 0.08 μV, SE = 0.07) was more positive than that observed to known incongruent items (M = –0.21 μV, SE = 0.10). No such difference by congruency was observed for unknown items (F < 1, p = .39, η
2 = .002).
To summarize, a significant N400 congruency effect (a reduction in the amplitude of the N400 for congruent trials, relative to incongruent trials) occurred from 450 to 700 ms over bilateral parietal electrode locations—but only for the known items. No such N400 congruency effect was found for the unknown items.
Effects of familiarity observed to the picture
In addition to the expected N400 congruency effect, we also observed an earlier difference in the waveforms recorded to the picture, before the auditory stimulus was presented. Because the auditory word had not yet been presented, any difference observed in this time window would be tied to knowledge differences for the pictures themselves, and could not be linked to congruity (since this was determined by the auditory stimulus). To examine this difference further, we collapsed the ERPs across congruence conditions to look at the differences elicited to the pictures in the known and unknown conditions; these ERPs are shown in Fig. 5.
Running t tests identified a sustained difference between the known and unknown conditions beginning approximately 200 ms after picture onset (i.e., –600 ms, relative to the onset of the auditory stimulus). The length of this significant window differed over sites. To compare these effects statistically, a 2 (knowledge: known/unknown) × 3 (site: frontal/central/parietal) × 2 (hemisphere: left/right) repeated measures ANOVA was run on the mean amplitudes for the known and unknown conditions (collapsed over congruency) over a window from 200 to 500 ms after picture presentation (–500 to –200 ms, relative to onset of the auditory token). This window was chosen as the minimum length at which all sites showed differences in the running t tests (Fig. 5). The ANOVA showed an interaction of knowledge and site [F(2, 44) = 28.41, p < .0001, η
2 = .07 ; see Table 3 for the full results]. To follow up this interaction, we collapsed over hemispheres and performed a two-way (knowledge) ANOVA for each site. Frontal sites showed a main effect of knowledge [F(1, 22) = 30.00, p < .0001, η
2 = .01], such that the mean amplitude to the known condition (M = –0.93 μV, SE = 0.16) was more positive than that to the unknown condition (M = –1.19 μV, SE = 0.18). This effect was also evident over central sites, where there was also a main effect of knowledge [F(1, 22) = 11.91, p < .01, η
2 = .01] due to a greater relative positivity to the known (M = –0.24 μV, SE = 0.10) than to the unknown (M = –0.39 μV, SE = 0.12) condition. At parietal sites, we also found a main effect of knowledge [F(1, 22) = 31.89, p < .0001, η
2 = .02], but here, the polarity of the effect was reversed: There was a greater relative positivity to unknown (M = 1.20 μV, SE = 0.14) than to known (M = 0.97 μV, SE = 0.13) items.
Thus, we observed differences in the response to the visual stimulus between the known and unknown conditions prior to the presentation of the (congruent or incongruent) auditory stimulus. At frontal and central electrode locations, this difference was in the form of a relatively more positive mean amplitude to pictures in the known condition, whereas at parietal sites, there was a greater relative positivity to pictures in the unknown condition. This difference onset early across all scalp locations (beginning approximately 200 ms after presentation of the picture) and extended in time for several hundred milliseconds (especially at parietal sites).
Correlations among measures
To consider the relationship between knowledge status and our various dependent measures, we ran Pearson’s correlations between several variable pairs separately for known and unknown items.
Behavioral measures with implicit measures
First, we examined the correlations between the three behavioral measures and the implicit measures; these results are shown in Table 4. The first behavioral measure was the PPVT score for each participant. For known items, we observed several negative correlations between PPVT score and the EM duration measures (such as mean fixation duration, first fixation duration, and proportion of dwell time on the stimulus), suggesting that larger vocabulary scores were associated with shorter looking times for known items. For unknown words, on the other hand, such correlations were not observed; the only significant correlation for this set was a positive correlation between PPVT score and the percentage of trials on which the named item was the last to be fixated.
The second behavioral measure was the RT on the forced-choice task. We ran correlations between these RTs and the EM and pupillometry measures, which were collected using the same paradigm. These are also shown in Table 4. Of note, for known words, we observed several positive correlations between the RT and EM measures, suggesting that longer times to select the named picture from the display were accompanied by longer looking times. This was not seen for unknown items, for which we saw only one marginally significant correlation between RT and the EM measure of first dwell time. There was, however, a significant negative correlation between RT and the mean change in pupil size for unknown items, suggesting that faster RTs were accompanied by smaller changes in pupil size for unknown items.
The third behavioral measure was RT on the congruity task, which we correlated with the N400 effect size from the concurrent ERP task. For the congruity task, RT effect sizes were calculated by subtracting the mean RT in the congruent condition from the mean RT in the incongruent condition, separately for known and unknown items. From the ERP data, the N400 effect was calculated by first calculating the difference wave (incongruent minus congruent) for each individual word, then finding the most negative peak in the difference wave within a window from 200 to 800 ms after sound presentation. The average difference wave amplitude within a 200-ms window around the most negative difference wave peak was then calculated, yielding an N400 effect measure for each individual word, which was averaged over known and unknown trials. As can be seen in Table 4, no significant correlations emerged between the RT effect size on the congruity task and N400 effect size.
Correlations among implicit measures
We also ran correlations among the various implicit measures themselves. The results of these correlations are shown in Tables 5 (for known items) and 6 (for unknown items). Some patterns are worth highlighting. First, for both known and unknown words, the EM measures are all highly intercorrelated, as are two of the three pupillometry measures (peak dilation and maximum percent change in pupil dilation), suggesting that these measures may be tapping into the same underlying processes. There are also a number of significant correlations between the measures from the different implicit assessment techniques; for example, for known words (but not for unknown words), we observed significant correlations between the N400 effect size and EM measures such as the mean fixation duration and first fixation duration.