Attempts to understand information use through time in high-level cognitive tasks, such as decision making or problem solving, face an empirical challenge in determining how the various inputs and outputs of the task are utilized over time. Traditionally, overt report of a person’s final output has been used as the primary measure for investigating the processes underlying performance in such domains. Although overt report has supported a wealth of insight, it is ill-suited for addressing issues regarding the temporal dynamics unfolding within a task. The primary reason for this insufficiency is that overt report is highly disruptive to ongoing cognitive dynamics.

For instance, consider a cued recall task such as medical diagnosis. The items relevant to this task are the patient’s symptom states and the diagnoses that are retrieved on the basis of these inputs. Understanding the temporal dynamics of this task requires understanding how and when these items are placed into (and become displaced from) working memory (WM). As the task unfolds, each item in memory (i.e., symptoms, and diagnoses once they are recalled) possesses a particular amount of memory activation. In order to understand the temporal dynamics of this task, we would like to measure the evolution of these activations throughout the task.

Methods for examining the dynamic utilization of information over time would benefit from four characteristics. First, such techniques should show minimal interference to the contents of WM and their utilization in the task. Second, it would be possible to deploy the measure within (and potentially throughout) the task as it unfolds. Third, the measure would be item-specific, in that it would allow measurement at the level of individual items, rather than simply the engagement of generalized processing as measured in neuroscience (e.g., fMRI, EEG, MEG). Finally, the measure would be of superior usefulness if it were sensitive not only to items being in or out of WM, but to their current levels of activation in memory, as well. At present, no methodology satisfying these criteria exists.

Interestingly, the literature on top-down capture, demonstrating a tight coupling between attentional processes and the contents of WM, suggests a solution (Downing, 2000; Huang & Pashler, 2007; Moores, Laiti, & Chelazzi, 2003; Olivers, 2009; Olivers, Meijer, & Theeuwes, 2006; Soto, Heinke, Humphreys, & Blanco, 2005; Soto & Humphreys, 2007). Such work by Soto et al. (2005) and by Soto and Humphreys (2007) has suggested that attention is automatically captured by the contents of WM when matching items reappear in search arrays (but see Woodman & Luck, 2007, for a demonstration in which WM content was used strategically). Of particular importance for the present method, Soto et al. (2005) found that eye movements were sensitive to the spatial congruency of the target and WM-matching items. Additionally, Moores et al. (2003) found that first saccades were directed toward the semantic associates of WM content more often than to control items.

Given this coupling between WM content and the deployment of attention and eye movements, it may be possible to measure the contents of WM by observing eye movements executed early on within briefly presented visual arrays containing content related to the task being executed. Relatedly, Makovski and Jiang (2008) demonstrated that congruity effect response times (RTs) can provide insight into WM representational strength when they are inserted into an additional task of interest. We believe that early eye movements can provide insight into the current contents of WM in tasks within and beyond the domain of attention.

As mentioned above, such a measure would be of enhanced usefulness if eye movements were sensitive to item-specific memory activation (ISMA), above and beyond the presence of an item in WM. To address this potential, we developed a novel paradigm through a fusion of standard free recall and visual search tasks. By providing an initial test of eye movement sensitivity to WM activation, this experiment provides a first step toward the methodology presently described.

An initial test of eye movement sensitivity to item-specific memory activation

We hypothesized that eye movements toward WM-matching items are sensitive to the ISMA levels of those items. That is, the extent to which eye movements are modulated by memory-matching content appearing in search arrays should be predicted in part by the memory activations of those items. To test this claim, we borrowed a prediction from the context-activation model (CAM) of memory recall (Davelaar, Goshen-Gottstein, Ashkenazi, Haarmann, & Usher, 2005) and fused standard free recall and visual search tasks into a novel paradigm. While representational strength has been suggested to influence RTs in a top-down capture task (Makovski & Jiang, 2008), the extent to which memory activation (over and above an item’s presence in WM) predicts eye movements has remained unaddressed.

The CAM produces fine-grained predictions concerning the rise and fall of ISMA trajectories throughout memory tasks. Crucial for the present experiment is that the model predicts a recency gradient in the activation levels of the terminal items of a study list. Figure 1 displays the activation trajectories of individual items during the study phase of a 16-item list. At the end of the study period (far right of the display), the activations for the last four items are monotonically ordered according to their activations, with the last item being the most active and each preceding item possessing a lower activation level. By placing a search task directly after a list study phase and selectively drawing items from the last four serial positions (SPs) to reappear in the search arrays, we tested how this recency gradient in memory activation influences the deployment of eye movements in a top-down capture task.

Fig. 1
figure 1

Item-specific memory activation trajectories predicted by the context-activation model. The activations of the terminal list items are predicted to be monotonically ordered by serial position (SP). The dotted line at WM activation 0.2 represents the model’s working memory threshold

Full size image

Method

Participants

A total of 17 participants took part in this experiment for course credit. Prior to the analysis, calibration criteria were applied. The participants’ gazes were required to be within 1.15° of eight points of a circular arrangement for four calibrations (following every other trial block) throughout the task in order to qualify for the analysis. Only data from the 11 participants meeting these criteria were subjected to analysis.

Procedure, design, and stimuli

Eye movements were recorded with an Arrington ViewPoint eyetracker with a sampling rate of 30 Hz. Stimulus presentation was handled by a Dell Optiplex PC running E-Prime 2.0 (Psychology Software Tools Inc., Sharpsburg, PA). Each trial began with the presentation of a study list in the center of the display, consisting of 16 semantically unrelated words (at a rate of 1.25 s/word), to be memorized for later free recall. Directly following the last word, a visual search array appeared containing four icons. The search task was to report either the presence or absence of a consistent target (icon of a jacket). One of the items appearing in the search array was an icon representing a study list item. This reappearing item was randomly drawn from one of the last four SPs (13th, 14th, 15th, or 16th). We refer to this item as the WM probe. The array remained on the screen until the response (button press) was executed. The participant was provided eight opportunities to recall words from the study list by typing them individually and pressing Return between entries. The recall task was encouraged as the primary task, and the participants were instructed to enter their search responses as quickly as possible while maintaining accuracy.

The design of the experiment was a 2 (target present vs. absent) × 4 (WM-probe SP) within-subjects design. Each participant was presented with 128 total trials (16 per condition).Footnote 1 The shape of the search array was displayed in either a square or a diamond configuration selected randomly on each trial. In order to encourage overt search, the target was randomly shown in one of four orientations (0 °, 90 °, 180 °, or 270 °) when present. Each item in the array appeared 10.3 ° of visual angle from the center of the display and subtended approximately 4.8 ° × 4.8 °. Figure 2 displays a schematic of the sequence of events on each trial. In this particular example, the item reappearing in the search array was the last item presented in the word list (soccer).

Fig. 2
figure 2

Example of the trial sequence used in the experiment: study list of 16 words presented sequentially for later recall, visual search task (the jacket target is present, in this case), and free recall of the word list.

Full size image

Results

Two exclusion criteria were applied to each trial: A trial was discarded if the participant’s gaze was not within a central 3.8 ° × 3.8 ° region upon onset of the search array and/or if the search response was inaccurate. As a result, 12 % of the data were excluded from the analyses (primarily due to the gaze criterion, as search was 96 % accurate throughout the experiment across participants). To assess which items were initially visited by the eyes, we took as our primary measure the proportion of trials on which the participant’s gaze moved into each item type’s region of interest (ROI) first. Each ROI covered a 7° × 7° area centered on each array item.

As can be seen in Fig. 3, the standard serial-position effects were obtained for the participants’ free recall, demonstrating primacy over the first five SPs, F (1, 40) = 6.25, p < 0.01, \( \eta_{\mathrm{p}}^{2} = 0.384 \), and substantial recency over the last five SPs, F(1, 40) = 14.33, p < 0.001, \( \eta_{\mathrm{p}}^{2} = 0.589 \). Crucially, the focal prediction of the experiment was borne out, as we found a recency effect for eye movements. Participants were more likely to enter the WM-probe ROI first as the SP of the probe increased, F (3, 30) = 5.177, p < 0.01, \( \eta_{\mathrm{p}}^{2} = 0.341 \). Additionally, a significant within-subjects correlation emerged between item recall and WM probe engagement over the last four SPs, r (10) = 0.55, p < 0.05.

Fig. 3
figure 3

Proportions demonstrating free recall performance in the memory task, along with first region-of-interest engagement of the working memory probe in the visual search task, plotted by the serial positions of items in the study list. Error bars represent standard errors of the means

Full size image

Breaking the data into target-present and target-absent conditions (Fig. 4), we found that the effect of SP on WM-probe engagement was present due to the increased probability of engaging WM-probe items from later SPs in the target-present condition, F (3, 30) = 5.376, p < 0.005, \( \eta_{\mathrm{p}}^{2} = 0.35 \), but not in the target-absent condition, F(3, 30) = 1.353, p = 0.276, \( \eta_{\mathrm{p}}^{2} = 0.119 \). Additionally, we compared the engagement rates for early (collapsed over SP13 and SP14) versus late (collapsed over SP15 and SP16) WM probes, finding that later WM probes were engaged first more often when the target was present, t (10) = 3.06, p < 0.05, but not when the target was absent, t (10) = 1.02, p = 0.332.

Fig. 4
figure 4

Rates of the first region-of-interest engagement for each item type in the visual search task, as a function of probe serial position and target presence/absence. Error bars represent standard errors of the means

Full size image

To address differences between the rates of engagement between distractors and WM probes, we computed a difference score for each participant within each condition by subtracting the proportion of first engagement to the WM probe from the proportion for individual distractors (see Fig. 5).Footnote 2 One-way ANOVAs testing the effect of probe SP within the target-present and target-absent conditions revealed a main effect of WM-probe SP in the target-present condition, F (3, 30) = 4.99, p < 0.01, \( \eta_{\mathrm{p}}^{2} = 0.{333} \), but not in the target-absent condition, F (3, 30) = 1.51, p = 0.231, \( \eta_{\mathrm{p}}^{2} = 0.{131} \). Planned t tests between each difference score and the null difference of zero revealed a highly significant effect for the difference score in the target-present condition at SP13, t (10) = −5.27, p < 0.001, while no other significant departures from zero were observed in the remaining seven conditions. We utilized an additional empirical baseline comparison (defined as 1 minus the target engagement proportion, the result of which was then divided by 3) within each target-present condition. Comparing WM-probe engagement to this baseline resulted in a similar pattern of results, t (10) = −5.992, p < 0.001, and t (10) = −1.77, p = 0.053, for SPs 13 and 14 in the target-present condition, respectively, with no other differences for the remaining conditions.

Fig. 5
figure 5

Difference scores between the rates of engagement for working memory probes and individual distractors, as a function of probe serial position and target presence/absence. Error bars represent standard errors of the means

Full size image

Discussion

The present results implicate the serial position of WM content as a factor influencing overt visual search. We predicted this effect as a consequence of the ISMA gradient across the terminal items in the memory list. Although we expected this effect to occur regardless of the presence of the target, the data demonstrated this effect only when the target was present. The data further suggested that this recency effect resulted from a combination of the ISMA possessed by the WM probes across SPs and a strategy to avoid the WM probes in the arrays. The earliest WM probe was engaged less than neutral distractors, and the earliest two WM probes were engaged less than would be expected relative to an empirical baseline measure, suggesting that the WM probes were strategically avoided in the search array. The present data, as interpreted through our current theoretical framework, suggests that the ability to direct overt attention away from array items corresponding to WM content critically depends on that item’s activation level. Interestingly, an item must be in WM but possess a low level of activation in order for the inhibitory strategy to be observed. If an item possesses too much activation, strategic avoidance of matching content in the array becomes more difficult, leading to greater engagement of the item and corresponding nonobservation of inhibition for items with greater activation.

Although inhibitory effects already figure prominently in the literature on top-down capture (Woodman & Luck, 2007), it is important to note that this is the first demonstration of such effects on eye movements. Relatedly, the present result suggests that the role of ISMA may inform the conditions under which strategic versus automatic effects are observed in top-down capture paradigms. Moreover, consideration of ISMA levels, as predicted by the CAM, may also inform the competitive processes between multiple WM items in the production of the attentional template (cf. Olivers, Peters, Houtcamp, & Roelfsema, 2011).

An alternative explanation of the present results, however, deserves consideration. In tasks capitalizing on SP differences, it could be posited that behavioral differences are the result of the items associated with different SPs being either in or out of WM, and thereby do not implicate sensitivity to graded differences between items cohabitating WM. For the present data, this argument would hold that the effect of SP arises simply because the later items are more often resident in WM at the time of the search task than are earlier items. Although this certainly occurred on some proportion of trials, this explanation cannot account for all of the data. It was observed that the engagement rate of the WM probe in the earliest SP (13th) of the target-present condition was substantially less than that of the distractors. This suggests that these items were maintained in WM, in support of the inhibitory strategy observed in the presence of the target. As for why this strategy was only evidenced in the target-present conditions, this remains an intriguing issue in need of future research. One road forward is suggested by a model related to the CAM that was developed by Usher and Davelaar (2002), in which the contents of the search array modulate the activation of WM items. This model predicts the recency gradient to be modest or absent in the absence of the target, but to be exaggerated in its presence due to increased competitive WM processes. Future research could test this model’s ability to account for the present data.

The fact that early oculomotor behavior is sensitive to ISMA suggests that procedures designed to measure dynamic information use through initial eye movements may be sensitive to the memory activations associated with the initially fixated items (above and beyond their presence in WM). We envision such procedures as providing dynamic “snapshots” of WM’s contents when assessed across time. As a participant is performing a task, arrays containing task-relevant items would briefly appear (~400 ms), and by tracking the initial items visited in the arrays, we would acquire a measure of an item’s inclusion in WM at that point in the task, and perhaps its memory activation level. By presenting these arrays at various points during the trial sequence, differences in item engagement rates would chart the dynamic rise and fall of individual items’ memory strengths over the course of the task.Footnote 3 Importantly, such a methodology would not be tied to any particular domain and could be used to investigate dynamic information use in a wide host of tasks. The development of such procedures would not be trivial, but the benefits of such a methodology could be great.