People freely viewing a scene often return their gaze to previously fixated objects, a behavior known as an oculomotor refixation. Refixations are a ubiquitous property of normal gaze behavior and have been noted in tasks as diverse as reading (Rayner, 1978, 1998), pattern copying or block sorting (Ballard, Hayhoe, & Pelz, 1995; Droll & Hayhoe, 2007; Hayhoe, Bensinger, & Ballard, 1998), portrait painting (Locher, 1996; Nodine, Locher, & Krupinski, 1993), solving arithmetic and geometry problems (Epelboim & Suppes, 1996; Hegarty, Mayer, & Green, 1992), visual search (Gilchrist & Harvey, 2000; Peterson, Kramer, Wang, Irwin, & McCarley, 2001), and undirected picture viewing (Mannan, Ruddock, & Wooding, 1996, 1997). Refixations can also comprise a significant portion of the gaze behavior accompanying a task—by one estimate, up to 25% of the observed fixations (Mannan et al., 1997).

Why do people choose to refixate objects that were previously inspected? The answer to this question likely depends on the task. In the case of reading, gaze is often returned to previously read portions of a text to resolve specific lexical ambiguities (see, e.g., Frazier & Rayner, 1982; Murry & Kennedy, 1988) or to obtain elaborative details needed for narrative comprehension (see, e.g., Blanchard & Iran-Nejad, 1987; Just & Carpenter, 1978; Shebilske & Fisher, 1983). Refixations, however, certainly play very different roles in nonreading tasks. For example, in one study, when observers had to reconstruct a spatially complex multicolored block pattern (the model) from a set of individual colored blocks scattered in a resource area, they tended to look first to the model pattern to determine the color of the next block to select from the resource area, then refixated the model to determine the exact location at which to place the selected block (Ballard et al., 1995; Hayhoe et al., 1998). In this case, refixations were therefore used to acquire a specific piece of information, the spatial information needed to correctly position a block in the model reconstruction. These fundamentally different functions of refixations complicate even the simplest of generalizations across tasks; whereas accuracy in a scene memory task improves with the number of refixations during study (Holm & Mäntyla, 2007), refixations in a search task are generally associated with poor spatial memory and inefficient processing (Beck, Peterson, Boot, Vomela, & Kramer, 2006; Dickinson & Zelinsky, 2005; Gilchrist & Harvey, 2000).

Our study explores the relationship between refixations and memory rehearsal in the context of an explicit working memory task. We propose that refixations in visual working memory (VWM) tasks, unlike the refixations made during reading or block copying, may serve a straightforward rehearsal function (see also Tremblay, Saint-Aubin, & Jalbert, 2006). Many studies have shown that memory for specific stimulus properties (e.g., exact color or surface form) declines soon after gaze shifts away from an object (Carlson-Radvansky, 1999; Carlson-Radvansky & Irwin, 1995; Henderson, 1997; Henderson & Siefert, 2001; Irwin, 1991, 1992, 1996; Irwin & Gordon, 1998). Given this rapid loss of object properties following an eye movement, a person freely viewing a multiple-object scene may frequently find it necessary to offset this information loss by visually reacquiring an object and encoding its properties anew. Our premise is that this need has led to the development of a system for refixating, or rehearsing, objects while viewing a scene for the purpose of representing it in memory, with each refixation of an object being an active attempt to refresh its representation and maintain its availability in VWM. Such a use of refixations is conceptually related to the results of Droll and Hayhoe (2007), who found that as the number of target object features needing to be stored in working memory to accomplish a task increased, viewers switched from a strategy of relying on their working memory to a strategy of refixating the target objects to acquire those features. It is also conceptually related to the phenomenon of refixating the locations of no-longer-present stimuli during a retention interval to rehearse their spatial locations (Brandt & Stark, 1997; Brockmole & Irwin, 2005; Laeng & Teodorescu, 2002; Spivey & Geng, 2001). We extend this idea by suggesting that such rehearsal occurs during the actual viewing of objects while studying a scene, and that this process is used to refresh object information above and beyond spatial location.

To assess the potential existence of this rehearsal system, we reanalyzed the data originally collected by Zelinsky and Loschky (2005). They monitored eye movements as observers studied the objects in a simple scene in anticipation of an immediate memory test. Using a gaze-contingent methodology, they made the duration of this study period contingent on the number of nontarget objects that observers looked at after fixating a predesignated target. When recognition accuracy for a probed object was analyzed conditionally, depending on when it was fixated during study, the results showed a pronounced recency effect. Recognition accuracy was high for the one or two objects fixated last during scene viewing, but this recency effect declined sharply into an above-chance pre-recency level of accuracy for objects fixated further back in the viewing sequence. Even though these objects were presented simultaneously as part of a visual scene, observers’ memory for them was serialized by their order of fixation. Importantly, trials in which the probed object was refixated were excluded from analysis so as to avoid contaminating the fixation-based serial position function with refixations. The focus of the present study is on these refixation trials, making the analyses in this article complementary to, and completely distinct from, the analyses reported by Zelinsky and Loschky (2005). Building on the logic developed in their earlier study, if subsequently fixated objects do indeed interfere with the representation of a target, then the more objects fixated post-target, the greater the demand for a target refixation.

Our specific research objectives can be framed in terms of two questions. First, is the refixation behavior observed during the study of a scene consistent with a rehearsal function? If target refixations do not increase with the number of subsequent object fixations, or if this refixation rate is very low, it would be unlikely that refixations would meaningfully affect memory performance even if they were used as a rehearsal tool. Moreover, rehearsal cannot simply be inferred from the existence of refixations—the refixation of an object must be shown to improve recognition accuracy for that object later during test. If this relationship is not observed, refixations obviously could not be serving a rehearsal function.

Second, does the pattern of refixations require working memory, or can this behavior be explained by random oculomotor inspection? We will address this question by comparing the behavioral refixation patterns with the refixation patterns produced by several random or minimally constrained models, as well as with a model that includes a simple memory process. If refixation behavior cannot be accurately characterized without appeal to a memory process, this would lend support to the conclusion that a relationship exists between refixations and rehearsal in working memory.

Method

The following is a slightly abbreviated description of the methods used in Zelinsky and Loschky (2005); the interested reader should consult this earlier source for additional details. The events comprising a typical trial are shown in Fig. 1. A trial began with the presentation of a nine-object study scene (Fig. 1a). The stimuli were real-world objects (toy, tool, or food items) arranged on an appropriate background surface (a crib, workbench, or dining table). Each scene was presented in color and subtended 18° × 11.6° of visual angle. Individual objects were scaled to fit within a 2.4° bounding box, and their locations in the scene were constrained to 18 positions, creating the appearance of a random arrangement of items on a surface. Multiple trials for a given scene type were created by randomly pairing objects to locations.

Fig. 1
figure 1

Observers freely viewed a nine-object display, in this case a diner table scene, with the intention of remembering the identity and location of each object (a). The display program counted the number of different nontarget objects fixated after gaze left the target (the butter dish in the depicted scene), then terminated the study display when a criterion number of these intervening objects were fixated (b). A spatial probe then appeared at the target’s location in an emptied version of the scene (c), and this was replaced after 1 s by a four-alternative forced choice decision grid used to indicate the study scene object that had appeared at the probed location (d)

Full size image

Six observers studied scenes in preparation for an immediate recognition test at the conclusion of each trial. At test, a spatial probe (a colored noise mask) was presented, and the task was to indicate the object from the study scene that appeared at the probed location. Observers were therefore required to encode from the study scene information about both the identity and location of each object.

Distinguishing this paradigm from many other working memory paradigms is the fact that testing was contingent upon the observer’s eye movements while viewing the study scene. Although the experiment instructions made no reference to oculomotor behavior, observers invariably chose to make eye movements during this challenging memory task. Eye position was sampled at 1000 Hz using a Generation VI dual-Purkinje-image eyetracker (Fourward Technologies, Inc.) and analyzed in real time to determine the object being fixated. Unbeknownst to the observer, one of the objects was predesignated as the memory target for that particular trial (e.g., the butter dish in Fig. 1). As the observer freely viewed the study scene, gaze would eventually be directed to this target object (Fig. 1a). This fixation event was detected by the program controlling the stimulus presentation, which then started counting the number of different nontarget objects fixated after gaze left the target. We refer to these items fixated after the initial target fixation (but before the memory test) as intervening objects (Waugh & Norman, 1965), in recognition of the fact that memory varies as a function of the number of these objects that are fixated (Zelinsky & Loschky, 2005).

A variable criterion placed on the number of intervening objects was used to terminate the study display. If this intervening-object criterion was preset to two, the observer would be allowed to fixate exactly two different nontarget objects following the target (Fig. 1b). As gaze shifted away from the second posttarget object, the study scene would be replaced by a colored noise mask appearing at the target’s location on the emptied background surface (Fig. 1c). This mask, which was visible for 1 s, served as the spatial probe for the memory test. There were seven intervening-object conditions (one to seven objects). In the one-intervening-object condition, the study display was terminated during the saccade away from the first posttarget object (i.e., gaze was not allowed to land on a second nontarget item after leaving the target); in the seven-intervening-object condition, the observer was allowed to fixate exactly seven different nontarget objects following the initial target fixation. The intervening-object conditions were randomly interleaved throughout the experiment. Postexperiment questioning revealed that none of the 6 participants realized that their opportunity for study depended on their own pattern of eye movements in conjunction with an intervening-object criterion; instead, they attributed the variable study scene duration to a random presentation schedule.

Following probe offset, a four-object array was presented, and observers had to indicate which of these objects appeared at the probed location (Fig. 1d). One of these objects was always the target, with the other three randomly selected from the study scene. The observer registered their four-alternative forced choice (4AFC) judgment by looking at the desired object, which caused a white box to be drawn around the item, then pressing a button when satisfied with their selection. Observers were instructed to respond as accurately as possible without regard for time, and each participated in 378 trials, 54 per each of the 7 intervening-object conditions.

Results and discussion

Is refixation behavior consistent with a rehearsal function?

How frequent are target refixations?

An important first step in characterizing the relationship between refixations and memory rehearsal is determining the refixation frequency, since a very low refixation rate would make the broader question moot. Refixations are operationally defined as a shift in gaze back to the target object before the scene-terminating intervening-object criterion was achieved. We isolated and analyzed those trials in which observers made at least one target refixation and found an overall refixation rate of 35%, a rate somewhat higher than is typically found in visual search or picture viewing tasks (Dickinson & Zelinsky, 2005; Mannan et al., 1997). This refixation rate also varied with intervening-object condition. Table 1 (top two rows) shows that the frequency of target refixation trials increased with the number of fixated intervening objects required by the gaze-contingent paradigm. When observers were allowed to fixate only one object following the target, a target refixation could not occur because the study scene would terminate upon gaze leaving the nontarget object. This contingency is indicated in the data by a refixation frequency of 0 in the one-intervening-object condition. However, as soon as refixations were allowed by the paradigm, the refixation rate jumped to 21%, and it continued to increase with the number of fixated intervening objects. This monotonic increase in refixation rate is not surprising. Each additional fixation forced by our intervening-object manipulation created another opportunity for observers to direct their gaze back to the target. What is surprising is that refixations occurred on up to 53% of the trials, making this behavior frequent enough to serve as a potential method of maintaining object information in working memory. Table 1 (bottom two rows) shows that the mean number of target refixations also increased with the number of intervening objects, but this increase was more modest. When a target was refixated on a trial, it was typically refixated only once.

Table 1 Target refixations by intervening-object condition
Full size table

Do refixations improve memory?

A necessary property of any behavior believed to serve a memory rehearsal function is, of course, a demonstrated relationship to an actual memory benefit. Figure 2 compares recognition accuracy for trials in which there was a target refixation with trials in which there were no target refixations; the no-refixation data are replotted from Zelinsky and Loschky (2005). Refixating the target in the two- to five-intervening-object conditions improved observers’ ability to later pick this object out of a 4AFC display, t(5) = –2.91 (one-tailed), p = .017, effect size (η2) = .628. This improvement amounted to a roughly constant 16% benefit over the two- to five-intervening-object range, followed by a decline into the no-refixation level of performance with additional intervening objects. An essential component of the asserted relationship between refixation and a memory rehearsal function is therefore supported. To the extent that refixations serve a rehearsal function, they would be expected to improve memory for the refixated object, which appears to be the case. Returning gaze to an object during the study phase of an immediate memory task increased the probability of that object being correctly identified in a subsequent recognition test.

Fig. 2
figure 2

Target recognition accuracy plotted as a function of intervening nontarget objects fixated after the initial target fixation. The filled markers show data from trials in which the target was not refixated during study; the unfilled markers show data from trials in which there was a target refixation. Note that the net effect of target refixation is to functionally reset the intervening-object function; the two-intervening-object data point in the with-refixation function is therefore functionally equivalent to the one-intervening-object condition in the without-refixation function. Errors bars indicate standard errors of the means

Full size image

When were targets refixated during study?

We now know that refixations were common in this explicit memory task and that they benefited recognition accuracy, but we do not yet know when they actually occurred during the free viewing of the study scene. For example, a target refixation in the three-intervening-object condition might have occurred following fixation on either the first or second intervening nontarget object. Increasing the number of intervening objects of course worsens this temporal ambiguity, with a refixation in the seven-intervening-object condition potentially occurring after any of the six nontarget objects fixated following the target. To better specify this refixation behavior, we analyzed when in the sequence of intervening object fixations observers elected to return their gaze to the target. If our hypothesized role of refixations is correct, knowing when these refixations occurred might shed light on how the system attempts to maintain object representations in working memory.

Figure 3 replots the refixation data from Table 1 (top row), segregated by four intervening-object conditions. Each panel is a relative frequency histogram showing when in the intervening-object sequence observers elected to make their initial gaze shift back to the target, henceforth referred to as the refixation lag. More specifically, refixation lag is defined as the number of different nontarget objects fixated after the initial target fixation and before the initial target refixation. This analysis shows that target refixations within a given condition did not occur equally often after every allowable intervening object fixation. Although observers had multiple opportunities to return gaze to the target in the three- to six-intervening-object conditions, we see that the modal refixation behavior occurred after gaze left the first posttarget object. Observers were in fact almost twice as likely to refixate the target after one intervening object as at any other time during the study scene presentation. If a target was ultimately refixated during study, that refixation was most likely to have occurred soon after the target was initially fixated (see also Dickinson & Zelinsky, 2007, and Gilchrist & Harvey, 2000, for reports of immediate target refixations in the context of search tasks).

Fig. 3
figure 3

Relative frequency histograms showing when in the three- to six-intervening-object conditions (labeled panels) observers made their initial refixation of a target. Data are not shown for the two-intervening-object condition because this is a degenerative case in which the gaze-contingent paradigm would force all of the refixations to occur after gaze left the first intervening object. Data are also not shown for the seven-intervening-object condition due to its similarity in pattern to the six-intervening-object data

Full size image

Might these immediate target refixations simply reflect an incomplete encoding of the target? Rather than indicating a rehearsal function, perhaps observers looked back to the previously fixated object to finish encoding it, due to gaze having left the object before its processing was complete (see, e.g., Hooge & Erkelens, 1999). To test this incomplete-encoding hypothesis, we analyzed the gaze dwell times on targets preceding their initial refixation. This was based on the assumption that shorter dwell times would create a need for additional processing, and thus more immediate refixations. Specifically, if the tendency to immediately refixate the target (Fig. 3) is due to incomplete encoding, then immediate refixations should be preceded by shorter initial target dwell times, whereas nonimmediate (i.e., lagged) refixations should be preceded by longer initial target dwell times. However, contrary to this prediction, we found essentially no difference between these two groups (dwells on targets preceding an immediate refixation, M = 240 ms, SEM = 19 ms; dwells on targets preceding a lagged refixation, M = 242 ms, SEM = 16 ms; paired-group t(5) = –0.25, p = .82). We also analyzed the distributions of these dwell times to see whether there were more short target dwell times in the immediate-refixation group. In fact, we found the opposite pattern: Only 28% of the target dwell times preceding an immediate refixation were <200 ms, whereas 31% of the corresponding dwell times in the lagged group were that short. We can therefore reject the incomplete-encoding hypothesis as an explanation for the reported preponderance of immediate refixations, and move on to consider how other factors known to affect eye movements might have produced the patterns of refixations shown in our task.

Is working memory needed to explain these behavioral patterns of refixations?

The analyses from the previous sections showed that the patterns of refixations in this task have the potential to serve a rehearsal function, but missing from this discussion was evidence that a simpler explanation, one that does not assume a system for rehearsing information in VWM, cannot describe these behavioral patterns equally well. If such a simpler explanation exists, Occam’s razor would require us to reject the more complex account involving purposive refixations to serve a rehearsal function.

To examine this possibility, we implemented and tested four computational models of eye movement behavior in our task. Each model ran on a trial-by-trial basis (16,200 runs per model; 300 per each of the 54 behavioral trials at each of the 7 intervening-object conditions), using the same object locations, target designations, and intervening-object termination criteria as in the behavioral experiment. The models generated as output a sequence of simulated fixations from which we determined the target fixation and refixation rates. Thus, for each intervening-object condition, we were able to model the percentage of trials on which one or more target refixations occurred (with model fits summarized in Table 2) and the percentage of target refixations as a function of the number of intervening object fixations (i.e., refixation lag, with model fits summarized in Table 3). These analyses correspond to the behavioral analyses provided in Table 1 and Fig. 3, respectively. Figure 4 directly compares each model’s refixation rates and refixation lag with the behavioral data.

Table 2 Model fits for percentage of trials with one or more target refixations
Full size table
Table 3 Model fits for proportion of target refixations as a function of the number of preceding fixations on intervening nontargets (Refixation Lag)
Full size table
Fig. 4
figure 4

Behavioral and corresponding simulated data from four computational models: The random model (a, b), the random+IOR model (c, d), the distance+IOR model (e, f), and the distance+IOR+VWM model (g, h). Refixation functions are shown in the left column (behavioral data replotted from Table 1); refixation lag data are shown in the right column for the five-intervening-object condition (behavioral data replotted from Fig. 3). Errors bars indicate standard errors of the means

Full size image

All models were probabilistic, meaning that every object was assigned a base probability of being selected, and each simulated fixation was based on sampling from this probability distribution. These base probabilities assured that each object had a greater than zero probability of being selected on each fixation. All of the models also operated under the common constraint that the same object could not be selected consecutively. The following are details specific to each model.

The random refixation model

The simplest model of refixation behavior would assume a random movement of gaze from one object to the next. With sufficient time and eye movements, observers would inevitably refixate the target, even if their gaze behavior was completely random. It is therefore essential to determine the patterns of refixations that would be expected from a random process, and to rule out this possibility before moving to more complex accounts of these behaviors. The random model used random selection with replacement to generate sequences of simulated fixations. The base probability of an object being selected by this model was therefore 1/n, where n represents the number of objects available to be selected for the next fixation.Footnote 1

If observers refixated the target as a result of a random eye movement between objects, the simulated refixation function should not differ from the behavioral refixation function, which was clearly not the case. As shown in Fig. 4a, observers in the two-intervening-object condition refixated the target somewhat more frequently than would be expected by chance, and the behavioral refixation rate dipped well below chance when there were six or seven intervening objects. Observers therefore preferentially refixated the target as soon as the paradigm allowed them to do so, but then systematically avoided returning to the target when the paradigm forced them to fixate a large number of intervening objects. These observations are quantified in Table 2, which reports the results of χ2 goodness-of-fit tests comparing the simulated and behavioral refixation functions for all of the models tested. Specifically, a highly significant difference between the random model and human behavior was found, suggesting that the proportion of trials on which one or more target refixations occurred cannot be described by a random process.

As shown in Fig. 4b, the random model also failed to describe when observers chose to look back to the target in the sequence of intervening object fixations. Table 3 reports significant differences between the random model and human behavior over the entire two- to six-intervening-object range. These discrepancies were due largely to the fact that observers returned their gaze to the target after fixating only one intervening object more often than predicted by the random model. We can therefore exclude this model as an explanation for the refixation behavior observed in our task. Simply put, observers’ refixation patterns were clearly nonrandom.

The random+IOR refixation model

A purely random model of object fixation can be argued, a priori, to be unrealistic on the basis of the many studies demonstrating a reduced likelihood of refixating a recently fixated object (e.g., Boot, McCarley, Kramer, & Peterson, 2004; Klein & MacInnes, 1999; Li & Lin, 2002; Rayner, Juhasz, Ashby, & Clifton, 2003; Sogo & Takeda, 2006), a phenomenon known as inhibition of return (IOR; Klein, 1988; Posner & Cohen, 1984). We therefore added an IOR component to the random model to evaluate whether this more plausible model might better explain our refixation data.

In this random+IOR model, the base probabilities were adjusted to temporarily reduce the likelihood of an item being refixated. This was done by subtracting an IOR weight from the base probabilities of fixating recently viewed objects. When gaze shifted away from an object, that object’s IOR weight was set to its maximum value. With the selection and fixation of each new object, the IOR weight for the previously fixated object would decay according to a linear function. Upon refixation of the previously viewed object, its IOR weight would be reset to the maximum value, and the process would repeat. The present implementation used a maximum IOR weight of .14 and a slope of –.02 for the IOR decay function. These parameters were determined so as to produce at least partial IOR for each of the intervening-object conditions explored in this study. Objects fixated more than seven objects back in the viewing sequence would have an IOR weight of 0 and would therefore not be inhibited.

The random+IOR model, rather than describing human behavior better than the purely random model, produced a significantly poorer fit to the behavioral data (Table 2). Figure 4c indicates profound underestimations of the behavioral refixation function for each of the intervening-object conditions except seven. Figure 4d and Table 3 show equally profound failures in predicting when these refixations occurred in the sequence of intervening object fixations.Footnote 2 Both of these inadequacies again stem from the model failing to capture the disproportionately high rate of early refixations found in the human behavior, as well as an overall refixation rate that was considerably lower than what we found for our observers. Clearly, a random model with the addition of a simple IOR component can be rejected as an explanation for refixation behavior in our task.

The distance+IOR refixation model

Our third attempt to model refixation behavior added to the random+IOR model a bias to fixate nearby objects, based on the many eye movement studies documenting such a bias in free-viewing tasks (e.g., Dickinson & Zelinsky, 2007; Findlay & Brown, 2006; Loschky & McConkie, 2002; Motter & Belky, 1998). To implement this bias, we adjusted the probability of selecting an object on the basis of the distance of that object to the current fixation. More specifically, these probabilities were defined by the equation

$$ {\text{p}}\left( {{\text{fixated}}} \right) = \left( {\min \_{\text{dist}}/{\text{obj}}\_{\text{dist}}} \right)/\sum\nolimits_{{i - 1}}^{j} {\left( {\min \_{\text{dist}}/{\text{obj}}\_{\text{dis}}{{\text{t}}_{i}}} \right),} $$
(1)

where min_dist is the distance between the current gaze position and the object closest to gaze, obj_dist is the distance between a given object and the current gaze position, and j is the number of objects available to be selected for the next fixation. This distance bias results in the probability of fixating an object decreasing with increasing distance between this object and the current fixation position. Objects nearest the current gaze position would therefore have the highest probability of being selected, and two or more equally distant objects would have the same probability of being selected for fixation, all else being equal.

The distance+IOR model fared only marginally better in predicting human refixation behavior than the random+IOR model. As before, the model tended to underestimate the behavioral refixation rate over much of the intervening-object function (Fig. 4e), again resulting in these functions differing significantly by χ2 test (Table 2). The model was also uniformly unsuccessful in predicting refixation lag (Fig. 4f, Table 3). It is therefore highly unlikely that refixation behavior in our task can be described by a probabilistic model combining only a distance bias with IOR.Footnote 3

The distance+IOR+VWM refixation model

In light of the failures of the previous models to describe refixation behavior in our task, most notably our observers’ preference to refixate the target after looking at only a single intervening object, we next considered the possibility that these early refixations might be serving to rehearse information in VWM. Specifically, when an object’s representation in VWM undergoes rapid decay, gaze is summoned back to that object to refresh its fading representation.

To model this rehearsal function, we added a rudimentary VWM component to our distance+IOR model, with its sole function being to signal gaze to return to a previously fixated object. The distance+IOR+VWM model accomplishes this by biasing the probability of an object refixation, with the strength of this bias decreasing with increasing fixation serial order. More specifically, the currently fixated object (O k ) would be assigned a maximum VWM activation of .94669, indicating a fully intact representation in memory. However, with the fixation of each subsequent object (k + 1 . . . n), the activation of O k would decrease according to the equation

$$ {\text{VW}}{{\text{M}}_k} = {\text{sp}}_k^l - .05331{ }, $$
(2)

where sp is the object’s serial position and l is a negative exponent (set at –.213 in the present implementation). All of these values for the VWM parameters were determined by fitting a decay function to the no-refixation recognition accuracy data.Footnote 4 Barring refixation, the VWM activation of O k would therefore decline sharply; with refixation, however, its serial position would reset to 1 and its VWM activation would return to the maximum value as the cycle began anew. An object’s VWM bias, or weight, in the selection of the next fixation was then computed by subtracting its current VWM activation from its previous activation level, using the equation

$$ {\text{VW}}{{\text{M}}_{\text{weight}}} = \left( {{\text{sp}} - {\text{l}}} \right)_k^l - {\text{sp}}_k^l{, } $$
(3)

where sp – 1 represents the object’s previous serial position and sp represents its current serial position. This weight would then be added to the object’s distance-based probability and IOR weight (if applicable) to determine its final probability of being selected for fixation by the next eye movement. In essence, then, the VWM weight is a difference signal, indicating the loss of VWM activation with increasing numbers of fixated intervening objects; this weight is largest after fixating the first intervening object because this is when the change in VWM activation was greatest, corresponding to the drop in recognition accuracy in the Zelinsky and Loschky (2005) data.

With the addition of a VWM component, the combined Distance+IOR+VWM model was able to describe all facets of our behavioral refixation data. Figure 4g and Table 2 show excellent fits to the refixation function for all of the intervening-object conditions except seven. The model also predicted the refixation lag for each of these conditions with a high degree of accuracy (Fig. 4h, Table 3). Most notably, it finally captured our observers’ bias to immediately refixate the previously viewed object. The failure of this model to perfectly describe the refixation data can be attributed to the linear decay function used by the IOR component leaving too little inhibition remaining after a large number of fixated intervening objects; had we explored nonlinear IOR decay functions, we would quite likely have been able to also fit the refixation data from the seven-intervening-object condition.

This generally good agreement with the behavioral data is made possible by the model’s components interacting with each other and changing dynamically with each intervening object fixation. The IOR and VWM components introduce opposing forces, one pushing gaze away from previously fixated items and the other pulling it back. What biases the model to make immediate refixations is the failure of these two components to completely cancel each other out; because the VWM component decays exponentially while the IOR component decays linearly, a strong bias to refixate is limited largely to the most recently viewed object. Similarly, the relatively slow decay rate of the IOR component biases against refixations in general, thereby discouraging “looping”—the tendency for simulated gaze to be called back to the same object with each sudden drop in its VWM activation. This, combined with the basic probabilistic nature of the selection process, serves to continuously reanchor the refixation cycle to a different part of the display. So, whereas our analysis of the random+IOR model suggested that an inhibitory mechanism was generally counterproductive in describing refixation behavior, in the context of the distance+IOR+VWM model it plays an essential role, preventing looping behavior and an unrealistically high rate of immediate target refixations that would otherwise result from an unconstrained VWM component. Importantly, if either of these components were left out, this push–pull mechanism would become unbalanced and the model would no longer be able to completely describe the behavioral refixation data.

General discussion

Memory rehearsal has long been implicated in tasks involving the sequential presentation of list items (Murdock & Metcalfe, 1978; Rundus & Atkinson, 1970; see Baddeley, 1986; Laming, 2009; Neath, 1998, for reviews). And just as verbal rehearsal may serve to maintain a list of digits long enough for them to be dialed into a telephone, we propose that gaze refixations may serve an analogous rehearsal function for VWM, keeping active the representations of objects presented simultaneously as part of a visual scene.

Motivating this proposal was the observation of serial order effects introduced by the sequence of fixations made to objects during scene viewing (Dickinson & Zelinsky, 2007; Irwin & Zelinsky, 2002; Korner & Gilchrist, 2007; Zelinsky & Loschky, 2005). The availability of an object in memory depends on when it was fixated during study, with more recently fixated objects enjoying a higher probability of memory retrieval. This is clear in the case of the Fig. 2 no-refixation data; accuracy was 87% after fixating only one intervening object, 76% after two, and 65% after three intervening object fixations (Zelinsky & Loschky, 2005). Such a pattern suggests that the target was highly available for retrieval immediately after fixation, but that interference introduced by the following object fixations caused this initially high level of availability to rapidly decline. Eventually, after fixating three intervening objects, the recency effect disappeared entirely, asymptoting into an above-chance pre-recency level of performance.

To attenuate this rapid decline in immediate memory for objects, we propose that gaze refixations are used to actively maintain object information as part of a monitor–refixate rehearsal system. According to this system, the activation levels of object representations in an explicit VWM task are continuously monitored, and a refixation is made to an object in danger of being forgotten when a sudden drop is detected in its activation. In the context of the present study, this would typically occur after the fixation of an intervening object. This event would cause the activation of the previously fixated object representation to drop suddenly, which in turn generates a change signal that the system uses to bias the saccade target selection process to refixate this object. The goal of an object refixation, whether explicit or otherwise, is therefore to offset the interference introduced by fixations of other objects (see, e.g., Zelinsky & Loschky, 2005), thereby reinstating its previously high level of activation and maintaining its place in VWM.

Such a monitor–refixate system can account for many of the patterns observed in the present data. First, it explains the improved accuracy resulting from refixations, reported in Fig. 2, and the disappearance of this advantage at higher numbers of fixated intervening objects. Such benefits derive from the fact that each target refixation essentially resets the intervening-object function. If a target is refixated during study, the nontarget object inspected immediately following this refixation functionally represents only one intervening object, regardless of when the target was initially fixated. The 90% level of accuracy on two-intervening-object refixation trials is therefore comparable to the 86% level of accuracy on one-intervening-object no-refixation trials because the former is functionally equivalent to a one-intervening-object condition without refixation. Similarly, because refixations in the three-intervening-object condition would occur after fixations on either the first or the second posttarget object, accuracy in this condition should fall between the 87% and 76% levels delineated by the no-refixation function, and indeed this was the case (82%). Note, however, that this relationship would be expected to break down with larger intervening-object separations. Consider a six-intervening-object trial. Although a target refixation could have occurred following the fixation of two to five intervening objects in this condition, these refixations were not uniformly distributed over this range (as shown in Fig. 3). On the majority of these trials, the target was refixated after the first two intervening objects in the viewing sequence, meaning that any recency benefit would ordinarily have disappeared into the pre-recency asymptote by the time the six-intervening-object criterion was satisfied. This relationship between recency and intervening objects therefore explains the gradual convergence of the refixation and no-refixation functions after the fixation of six to seven intervening objects.

A monitor–refixate system can also account for our observers’ tendency to refixate a target early in the intervening-object sequence (Fig. 3). As described by the distance+IOR+VWM model, these immediate refixations are a by-product of an exponential decay process affecting object representations in VWM; an object’s activation in VWM decreases the most after fixation of the first intervening object, but the rate of this decrease decelerates with each subsequently fixated object. This change in an object’s VWM activation from one fixation to the next is what is monitored by the monitor–refixate system, with the probability of a refixation increasing with the size of this change signal. The “pull” on gaze exerted by VWM is therefore greatest after the first fixated intervening object, resulting in the skewed probability of immediate refixations.

As for what might produce this negatively accelerating decay of VWM activation, there are at least two possibilities. One explanation is that the interference produced by the fixation of the first intervening object leaves fewer features of the original representation to be interfered with by subsequent object fixations. As the number of original target-object features dwindles with each fixated intervening object, so does the opportunity for interference, resulting in a slowing of the decay rate. This explanation is consistent with the sudden loss of features known to accompany the initial saccade away from an object (Irwin, 1991, 1992, 1996), as well as the slightly more protracted recency effect reported for objects in scenes (Irwin & Zelinsky, 2002; Zelinsky & Loschky, 2005). An alternative explanation is that gaze may leave an object before its features have been fully encoded or consolidated in VWM (Hooge & Erkelens, 1999). These hastily coded features might be particularly vulnerable to interference upon gaze shifting to a new object, thereby again creating a large decrease in activation and a strong signal to refixate the previously viewed object. Although both explanations are consistent with the proposed monitor–refixate system, our analysis offered no evidence for an incomplete encoding of targets preceding a refixation. We therefore tentatively conclude that the signal ultimately responsible for triggering a refixation in our task originated from the rapid loss of feature information as a result of shifting gaze to another object.

What might the operation of a monitor–refixate system for memory rehearsal look like on an individual trial? Discerning a complex cognitive function from any individual’s pattern of eye movements is notoriously difficult (see, e.g., Droll & Hayhoe, 2007; Hayhoe et al., 1998; Hegarty & Just, 1993; Just & Carpenter, 1976; Steinman, Kowler, & Collewijn, 1990; Zelinsky, 2001), and memory rehearsal is no exception. Figure 5 illustrates why this is the case. The eye movements accompanying presentation of this study scene seemingly defy explanation. Driving this oculomotor behavior are probably a host of factors not directly related to working memory, some very low-level (e.g., object salience, proximity to the current fixation, display configuration) and others high-level (e.g., object familiarity, path planning, and preexisting conceptual associations between objects). Although all are interesting topics in their own right, in the present context these factors introduce considerable noise into the task of finding evidence for a refixation-based rehearsal process embedded in free-viewing eye movement behavior.

Fig. 5
figure 5

A representative eye movement scanpath for a four-intervening-object trial. Numerical labels indicate fixations after each eye movement following study scene onset. Note that the panda target was refixated after only one intervening fixation on the doll

Full size image

Upon closer inspection, however, the Fig. 5 scanpath reveals an interesting regularity. A monitor–refixate rehearsal process suggests that observers should periodically refixate objects after shifting gaze to one or two intervening items. With respect to the panda bear target on this trial, the observer fixates this object with their 12th fixation, then looks to the doll (13th fixation), the first intervening object relative to the target. According to the proposed system, interference from the fixated intervening object causes a sudden loss in VWM for features relating to the panda, which in turn creates a signal biasing gaze to return to the panda (14th fixation) after fixating only a single intervening object. Of course, from the perspective of an observer studying this scene, the panda target was just another object in the display; to the extent that observers were systematically rehearsing objects in VWM, one might expect to see a similar pattern of refixations of other objects along the scanpath. This proved to be the case. The same scenario of fixations described for the panda target also appeared for the toy train and doll objects in this representative trial.

Although our proposed monitor–refixate system can successfully account for many aspects of refixation behavior in our task, one might argue that the process described by this system is not proper rehearsal. Rehearsal in working memory is most often conceptualized as a cyclical process consisting of multiple passes through a sort of loop (see, e.g., Baddeley, 1986), with each pass being instrumental in the maintenance of the rehearsed items. The rehearsal process described here is quite different, consisting of typically only one repeating cycle, when a repeating cycle appears at all. Although we mainly attribute this abbreviated rehearsal cycle to the dynamic interplay between the components of the monitor–refixate system (i.e., VWM and eye movement biases), it is also likely due in part to the unique constraints imposed by our free-viewing memory task. In a standard working memory task, a list of items is presented, followed by a retention interval, during which rehearsal is used to maintain these previously presented items in memory. However, rehearsal in our task was more interleaved with the actual presentation of the memory “list,” which was a product of the observer’s viewing behavior. Our observers were forced to trade off the acquisition of a new item in this list with the rehearsal of an old item; acquiring a new list item necessitated making an eye movement to a new object, not refixating an old one (and vice versa). Had our observers adopted a standard rehearsal strategy of cycling through the same small set of old objects, the other new objects in the display would have never been inspected.

Of course, this online characterization of rehearsal begs the question of why our observers managed not to get caught in an endless cycle of refixating the same objects. Strict adherence to the above-described monitor–refixate rehearsal process should have quickly locked the observer in a perpetual loop, with the same subset of objects continuously refixated and forgotten. This problem was avoided in our modeling efforts by assuming that the selection of each object for fixation was probabilistic, a solution that was successful yet somehow unsatisfying—leaving one wondering what specific factors determined these probabilities.

At least two mechanisms might be used to periodically break from the monitor–refixate maintenance loop long enough to include new items in the rehearsal set. One possibility assumes a contribution from long-term memory (LTM). Borrowing from the early modal model conception of memory rehearsal (Broadbent, 1958; Waugh & Norman, 1965), the repeated fixation of an object might strengthen its representation in LTM. If so, as this LTM representation becomes stronger, the need to actively maintain a VWM representation for the same object is lessened. More mechanistically, the growing “push” from LTM, when combined with IOR, would eventually overpower the “pull” back to an object by its fading VWM representation. The maintenance of the rehearsed object would therefore be abandoned, resulting in the monitor–refixation process periodically including new items in the rehearsal set. A second possibility is that observers have more direct control over the objects that they choose to maintain in VWM. We speculate that this might be achieved by monitoring the VWM activations of only a subset of the studied objects. Given that there may be many object representations calling out for rehearsal but only one locus of gaze to serve this function, the representations that observers choose to monitor would ultimately determine their refixation behavior. Using Fig. 5 as an example, the overt decision to look from the ducky (15th fixation) to the doll (17th fixation) rather than back to the panda might reflect an internal decision to monitor only the activation of the doll representation and to abandon monitoring all of the objects fixated earlier in the viewing sequence. Future work will attempt to distinguish between these possibilities, although it is quite possible that both forces may be in play.

Finally, we hope to use gaze refixation patterns to study VWM capacity, as has been done in studies of eye movements in visual search (Dickinson & Zelinsky, 2005, 2007; Gilchrist & Harvey, 2000). To the extent that these patterns reflect a memory rehearsal process, it may be possible to obtain online estimates of VWM capacity by analyzing when refixations occur in extended viewing sequences (see also Droll & Hayhoe, 2007). And, because such estimates would be relatively continuous, one might even be able to study changes in capacity during the course of individual trials and (potentially) to link these changes to properties of the specific objects being held in VWM. As these spatial and temporal processes are better understood, VWM will take on a new and dynamic conceptualization—a sort of moving window of highly available objects, with the size of this window providing an estimate of VWM capacity, and its location in the scene indicating the contents of the memory set.