Do object refixations during scene viewing indicate rehearsal in visual working memory?
Do refixations serve a rehearsal function in visual working memory (VWM)? We analyzed refixations from observers freely viewing multiobject scenes. An eyetracker was used to limit the viewing of a scene to a specified number of objects fixated after the target (intervening objects), followed by a four-alternative forced choice recognition test. Results showed that the probability of target refixation increased with the number of fixated intervening objects, and these refixations produced a 16% accuracy benefit over the first five intervening-object conditions. Additionally, refixations most frequently occurred after fixations on only one to two other objects, regardless of the intervening-object condition. These behaviors could not be explained by random or minimally constrained computational models; a VWM component was required to completely describe these data. We explain these findings in terms of a monitor–refixate rehearsal system: The activations of object representations in VWM are monitored, with refixations occurring when these activations decrease suddenly.
KeywordsEye movements Scene perception Active vision Memory rehearsal
People freely viewing a scene often return their gaze to previously fixated objects, a behavior known as an oculomotor refixation. Refixations are a ubiquitous property of normal gaze behavior and have been noted in tasks as diverse as reading (Rayner, 1978, 1998), pattern copying or block sorting (Ballard, Hayhoe, & Pelz, 1995; Droll & Hayhoe, 2007; Hayhoe, Bensinger, & Ballard, 1998), portrait painting (Locher, 1996; Nodine, Locher, & Krupinski, 1993), solving arithmetic and geometry problems (Epelboim & Suppes, 1996; Hegarty, Mayer, & Green, 1992), visual search (Gilchrist & Harvey, 2000; Peterson, Kramer, Wang, Irwin, & McCarley, 2001), and undirected picture viewing (Mannan, Ruddock, & Wooding, 1996, 1997). Refixations can also comprise a significant portion of the gaze behavior accompanying a task—by one estimate, up to 25% of the observed fixations (Mannan et al., 1997).
Why do people choose to refixate objects that were previously inspected? The answer to this question likely depends on the task. In the case of reading, gaze is often returned to previously read portions of a text to resolve specific lexical ambiguities (see, e.g., Frazier & Rayner, 1982; Murry & Kennedy, 1988) or to obtain elaborative details needed for narrative comprehension (see, e.g., Blanchard & Iran-Nejad, 1987; Just & Carpenter, 1978; Shebilske & Fisher, 1983). Refixations, however, certainly play very different roles in nonreading tasks. For example, in one study, when observers had to reconstruct a spatially complex multicolored block pattern (the model) from a set of individual colored blocks scattered in a resource area, they tended to look first to the model pattern to determine the color of the next block to select from the resource area, then refixated the model to determine the exact location at which to place the selected block (Ballard et al., 1995; Hayhoe et al., 1998). In this case, refixations were therefore used to acquire a specific piece of information, the spatial information needed to correctly position a block in the model reconstruction. These fundamentally different functions of refixations complicate even the simplest of generalizations across tasks; whereas accuracy in a scene memory task improves with the number of refixations during study (Holm & Mäntyla, 2007), refixations in a search task are generally associated with poor spatial memory and inefficient processing (Beck, Peterson, Boot, Vomela, & Kramer, 2006; Dickinson & Zelinsky, 2005; Gilchrist & Harvey, 2000).
Our study explores the relationship between refixations and memory rehearsal in the context of an explicit working memory task. We propose that refixations in visual working memory (VWM) tasks, unlike the refixations made during reading or block copying, may serve a straightforward rehearsal function (see also Tremblay, Saint-Aubin, & Jalbert, 2006). Many studies have shown that memory for specific stimulus properties (e.g., exact color or surface form) declines soon after gaze shifts away from an object (Carlson-Radvansky, 1999; Carlson-Radvansky & Irwin, 1995; Henderson, 1997; Henderson & Siefert, 2001; Irwin, 1991, 1992, 1996; Irwin & Gordon, 1998). Given this rapid loss of object properties following an eye movement, a person freely viewing a multiple-object scene may frequently find it necessary to offset this information loss by visually reacquiring an object and encoding its properties anew. Our premise is that this need has led to the development of a system for refixating, or rehearsing, objects while viewing a scene for the purpose of representing it in memory, with each refixation of an object being an active attempt to refresh its representation and maintain its availability in VWM. Such a use of refixations is conceptually related to the results of Droll and Hayhoe (2007), who found that as the number of target object features needing to be stored in working memory to accomplish a task increased, viewers switched from a strategy of relying on their working memory to a strategy of refixating the target objects to acquire those features. It is also conceptually related to the phenomenon of refixating the locations of no-longer-present stimuli during a retention interval to rehearse their spatial locations (Brandt & Stark, 1997; Brockmole & Irwin, 2005; Laeng & Teodorescu, 2002; Spivey & Geng, 2001). We extend this idea by suggesting that such rehearsal occurs during the actual viewing of objects while studying a scene, and that this process is used to refresh object information above and beyond spatial location.
To assess the potential existence of this rehearsal system, we reanalyzed the data originally collected by Zelinsky and Loschky (2005). They monitored eye movements as observers studied the objects in a simple scene in anticipation of an immediate memory test. Using a gaze-contingent methodology, they made the duration of this study period contingent on the number of nontarget objects that observers looked at after fixating a predesignated target. When recognition accuracy for a probed object was analyzed conditionally, depending on when it was fixated during study, the results showed a pronounced recency effect. Recognition accuracy was high for the one or two objects fixated last during scene viewing, but this recency effect declined sharply into an above-chance pre-recency level of accuracy for objects fixated further back in the viewing sequence. Even though these objects were presented simultaneously as part of a visual scene, observers’ memory for them was serialized by their order of fixation. Importantly, trials in which the probed object was refixated were excluded from analysis so as to avoid contaminating the fixation-based serial position function with refixations. The focus of the present study is on these refixation trials, making the analyses in this article complementary to, and completely distinct from, the analyses reported by Zelinsky and Loschky (2005). Building on the logic developed in their earlier study, if subsequently fixated objects do indeed interfere with the representation of a target, then the more objects fixated post-target, the greater the demand for a target refixation.
Our specific research objectives can be framed in terms of two questions. First, is the refixation behavior observed during the study of a scene consistent with a rehearsal function? If target refixations do not increase with the number of subsequent object fixations, or if this refixation rate is very low, it would be unlikely that refixations would meaningfully affect memory performance even if they were used as a rehearsal tool. Moreover, rehearsal cannot simply be inferred from the existence of refixations—the refixation of an object must be shown to improve recognition accuracy for that object later during test. If this relationship is not observed, refixations obviously could not be serving a rehearsal function.
Second, does the pattern of refixations require working memory, or can this behavior be explained by random oculomotor inspection? We will address this question by comparing the behavioral refixation patterns with the refixation patterns produced by several random or minimally constrained models, as well as with a model that includes a simple memory process. If refixation behavior cannot be accurately characterized without appeal to a memory process, this would lend support to the conclusion that a relationship exists between refixations and rehearsal in working memory.
Six observers studied scenes in preparation for an immediate recognition test at the conclusion of each trial. At test, a spatial probe (a colored noise mask) was presented, and the task was to indicate the object from the study scene that appeared at the probed location. Observers were therefore required to encode from the study scene information about both the identity and location of each object.
Distinguishing this paradigm from many other working memory paradigms is the fact that testing was contingent upon the observer’s eye movements while viewing the study scene. Although the experiment instructions made no reference to oculomotor behavior, observers invariably chose to make eye movements during this challenging memory task. Eye position was sampled at 1000 Hz using a Generation VI dual-Purkinje-image eyetracker (Fourward Technologies, Inc.) and analyzed in real time to determine the object being fixated. Unbeknownst to the observer, one of the objects was predesignated as the memory target for that particular trial (e.g., the butter dish in Fig. 1). As the observer freely viewed the study scene, gaze would eventually be directed to this target object (Fig. 1a). This fixation event was detected by the program controlling the stimulus presentation, which then started counting the number of different nontarget objects fixated after gaze left the target. We refer to these items fixated after the initial target fixation (but before the memory test) as intervening objects (Waugh & Norman, 1965), in recognition of the fact that memory varies as a function of the number of these objects that are fixated (Zelinsky & Loschky, 2005).
A variable criterion placed on the number of intervening objects was used to terminate the study display. If this intervening-object criterion was preset to two, the observer would be allowed to fixate exactly two different nontarget objects following the target (Fig. 1b). As gaze shifted away from the second posttarget object, the study scene would be replaced by a colored noise mask appearing at the target’s location on the emptied background surface (Fig. 1c). This mask, which was visible for 1 s, served as the spatial probe for the memory test. There were seven intervening-object conditions (one to seven objects). In the one-intervening-object condition, the study display was terminated during the saccade away from the first posttarget object (i.e., gaze was not allowed to land on a second nontarget item after leaving the target); in the seven-intervening-object condition, the observer was allowed to fixate exactly seven different nontarget objects following the initial target fixation. The intervening-object conditions were randomly interleaved throughout the experiment. Postexperiment questioning revealed that none of the 6 participants realized that their opportunity for study depended on their own pattern of eye movements in conjunction with an intervening-object criterion; instead, they attributed the variable study scene duration to a random presentation schedule.
Following probe offset, a four-object array was presented, and observers had to indicate which of these objects appeared at the probed location (Fig. 1d). One of these objects was always the target, with the other three randomly selected from the study scene. The observer registered their four-alternative forced choice (4AFC) judgment by looking at the desired object, which caused a white box to be drawn around the item, then pressing a button when satisfied with their selection. Observers were instructed to respond as accurately as possible without regard for time, and each participated in 378 trials, 54 per each of the 7 intervening-object conditions.
Results and discussion
Is refixation behavior consistent with a rehearsal function?
How frequent are target refixations?
Target refixations by intervening-object condition
% of Trials
Do refixations improve memory?
When were targets refixated during study?
We now know that refixations were common in this explicit memory task and that they benefited recognition accuracy, but we do not yet know when they actually occurred during the free viewing of the study scene. For example, a target refixation in the three-intervening-object condition might have occurred following fixation on either the first or second intervening nontarget object. Increasing the number of intervening objects of course worsens this temporal ambiguity, with a refixation in the seven-intervening-object condition potentially occurring after any of the six nontarget objects fixated following the target. To better specify this refixation behavior, we analyzed when in the sequence of intervening object fixations observers elected to return their gaze to the target. If our hypothesized role of refixations is correct, knowing when these refixations occurred might shed light on how the system attempts to maintain object representations in working memory.
Might these immediate target refixations simply reflect an incomplete encoding of the target? Rather than indicating a rehearsal function, perhaps observers looked back to the previously fixated object to finish encoding it, due to gaze having left the object before its processing was complete (see, e.g., Hooge & Erkelens, 1999). To test this incomplete-encoding hypothesis, we analyzed the gaze dwell times on targets preceding their initial refixation. This was based on the assumption that shorter dwell times would create a need for additional processing, and thus more immediate refixations. Specifically, if the tendency to immediately refixate the target (Fig. 3) is due to incomplete encoding, then immediate refixations should be preceded by shorter initial target dwell times, whereas nonimmediate (i.e., lagged) refixations should be preceded by longer initial target dwell times. However, contrary to this prediction, we found essentially no difference between these two groups (dwells on targets preceding an immediate refixation, M = 240 ms, SEM = 19 ms; dwells on targets preceding a lagged refixation, M = 242 ms, SEM = 16 ms; paired-group t(5) = –0.25, p = .82). We also analyzed the distributions of these dwell times to see whether there were more short target dwell times in the immediate-refixation group. In fact, we found the opposite pattern: Only 28% of the target dwell times preceding an immediate refixation were <200 ms, whereas 31% of the corresponding dwell times in the lagged group were that short. We can therefore reject the incomplete-encoding hypothesis as an explanation for the reported preponderance of immediate refixations, and move on to consider how other factors known to affect eye movements might have produced the patterns of refixations shown in our task.
Is working memory needed to explain these behavioral patterns of refixations?
The analyses from the previous sections showed that the patterns of refixations in this task have the potential to serve a rehearsal function, but missing from this discussion was evidence that a simpler explanation, one that does not assume a system for rehearsing information in VWM, cannot describe these behavioral patterns equally well. If such a simpler explanation exists, Occam’s razor would require us to reject the more complex account involving purposive refixations to serve a rehearsal function.
Model fits for percentage of trials with one or more target refixations
Model fits for proportion of target refixations as a function of the number of preceding fixations on intervening nontargets (Refixation Lag)
Intervening Object Condition
Random+IOR Model a
Distance+IOR + VWM Model
All models were probabilistic, meaning that every object was assigned a base probability of being selected, and each simulated fixation was based on sampling from this probability distribution. These base probabilities assured that each object had a greater than zero probability of being selected on each fixation. All of the models also operated under the common constraint that the same object could not be selected consecutively. The following are details specific to each model.
The random refixation model
The simplest model of refixation behavior would assume a random movement of gaze from one object to the next. With sufficient time and eye movements, observers would inevitably refixate the target, even if their gaze behavior was completely random. It is therefore essential to determine the patterns of refixations that would be expected from a random process, and to rule out this possibility before moving to more complex accounts of these behaviors. The random model used random selection with replacement to generate sequences of simulated fixations. The base probability of an object being selected by this model was therefore 1/n, where n represents the number of objects available to be selected for the next fixation.1
If observers refixated the target as a result of a random eye movement between objects, the simulated refixation function should not differ from the behavioral refixation function, which was clearly not the case. As shown in Fig. 4a, observers in the two-intervening-object condition refixated the target somewhat more frequently than would be expected by chance, and the behavioral refixation rate dipped well below chance when there were six or seven intervening objects. Observers therefore preferentially refixated the target as soon as the paradigm allowed them to do so, but then systematically avoided returning to the target when the paradigm forced them to fixate a large number of intervening objects. These observations are quantified in Table 2, which reports the results of χ2 goodness-of-fit tests comparing the simulated and behavioral refixation functions for all of the models tested. Specifically, a highly significant difference between the random model and human behavior was found, suggesting that the proportion of trials on which one or more target refixations occurred cannot be described by a random process.
As shown in Fig. 4b, the random model also failed to describe when observers chose to look back to the target in the sequence of intervening object fixations. Table 3 reports significant differences between the random model and human behavior over the entire two- to six-intervening-object range. These discrepancies were due largely to the fact that observers returned their gaze to the target after fixating only one intervening object more often than predicted by the random model. We can therefore exclude this model as an explanation for the refixation behavior observed in our task. Simply put, observers’ refixation patterns were clearly nonrandom.
The random+IOR refixation model
A purely random model of object fixation can be argued, a priori, to be unrealistic on the basis of the many studies demonstrating a reduced likelihood of refixating a recently fixated object (e.g., Boot, McCarley, Kramer, & Peterson, 2004; Klein & MacInnes, 1999; Li & Lin, 2002; Rayner, Juhasz, Ashby, & Clifton, 2003; Sogo & Takeda, 2006), a phenomenon known as inhibition of return (IOR; Klein, 1988; Posner & Cohen, 1984). We therefore added an IOR component to the random model to evaluate whether this more plausible model might better explain our refixation data.
In this random+IOR model, the base probabilities were adjusted to temporarily reduce the likelihood of an item being refixated. This was done by subtracting an IOR weight from the base probabilities of fixating recently viewed objects. When gaze shifted away from an object, that object’s IOR weight was set to its maximum value. With the selection and fixation of each new object, the IOR weight for the previously fixated object would decay according to a linear function. Upon refixation of the previously viewed object, its IOR weight would be reset to the maximum value, and the process would repeat. The present implementation used a maximum IOR weight of .14 and a slope of –.02 for the IOR decay function. These parameters were determined so as to produce at least partial IOR for each of the intervening-object conditions explored in this study. Objects fixated more than seven objects back in the viewing sequence would have an IOR weight of 0 and would therefore not be inhibited.
The random+IOR model, rather than describing human behavior better than the purely random model, produced a significantly poorer fit to the behavioral data (Table 2). Figure 4c indicates profound underestimations of the behavioral refixation function for each of the intervening-object conditions except seven. Figure 4d and Table 3 show equally profound failures in predicting when these refixations occurred in the sequence of intervening object fixations.2 Both of these inadequacies again stem from the model failing to capture the disproportionately high rate of early refixations found in the human behavior, as well as an overall refixation rate that was considerably lower than what we found for our observers. Clearly, a random model with the addition of a simple IOR component can be rejected as an explanation for refixation behavior in our task.
The distance+IOR refixation model
The distance+IOR model fared only marginally better in predicting human refixation behavior than the random+IOR model. As before, the model tended to underestimate the behavioral refixation rate over much of the intervening-object function (Fig. 4e), again resulting in these functions differing significantly by χ2 test (Table 2). The model was also uniformly unsuccessful in predicting refixation lag (Fig. 4f, Table 3). It is therefore highly unlikely that refixation behavior in our task can be described by a probabilistic model combining only a distance bias with IOR.3
The distance+IOR+VWM refixation model
In light of the failures of the previous models to describe refixation behavior in our task, most notably our observers’ preference to refixate the target after looking at only a single intervening object, we next considered the possibility that these early refixations might be serving to rehearse information in VWM. Specifically, when an object’s representation in VWM undergoes rapid decay, gaze is summoned back to that object to refresh its fading representation.
With the addition of a VWM component, the combined Distance+IOR+VWM model was able to describe all facets of our behavioral refixation data. Figure 4g and Table 2 show excellent fits to the refixation function for all of the intervening-object conditions except seven. The model also predicted the refixation lag for each of these conditions with a high degree of accuracy (Fig. 4h, Table 3). Most notably, it finally captured our observers’ bias to immediately refixate the previously viewed object. The failure of this model to perfectly describe the refixation data can be attributed to the linear decay function used by the IOR component leaving too little inhibition remaining after a large number of fixated intervening objects; had we explored nonlinear IOR decay functions, we would quite likely have been able to also fit the refixation data from the seven-intervening-object condition.
This generally good agreement with the behavioral data is made possible by the model’s components interacting with each other and changing dynamically with each intervening object fixation. The IOR and VWM components introduce opposing forces, one pushing gaze away from previously fixated items and the other pulling it back. What biases the model to make immediate refixations is the failure of these two components to completely cancel each other out; because the VWM component decays exponentially while the IOR component decays linearly, a strong bias to refixate is limited largely to the most recently viewed object. Similarly, the relatively slow decay rate of the IOR component biases against refixations in general, thereby discouraging “looping”—the tendency for simulated gaze to be called back to the same object with each sudden drop in its VWM activation. This, combined with the basic probabilistic nature of the selection process, serves to continuously reanchor the refixation cycle to a different part of the display. So, whereas our analysis of the random+IOR model suggested that an inhibitory mechanism was generally counterproductive in describing refixation behavior, in the context of the distance+IOR+VWM model it plays an essential role, preventing looping behavior and an unrealistically high rate of immediate target refixations that would otherwise result from an unconstrained VWM component. Importantly, if either of these components were left out, this push–pull mechanism would become unbalanced and the model would no longer be able to completely describe the behavioral refixation data.
Memory rehearsal has long been implicated in tasks involving the sequential presentation of list items (Murdock & Metcalfe, 1978; Rundus & Atkinson, 1970; see Baddeley, 1986; Laming, 2009; Neath, 1998, for reviews). And just as verbal rehearsal may serve to maintain a list of digits long enough for them to be dialed into a telephone, we propose that gaze refixations may serve an analogous rehearsal function for VWM, keeping active the representations of objects presented simultaneously as part of a visual scene.
Motivating this proposal was the observation of serial order effects introduced by the sequence of fixations made to objects during scene viewing (Dickinson & Zelinsky, 2007; Irwin & Zelinsky, 2002; Korner & Gilchrist, 2007; Zelinsky & Loschky, 2005). The availability of an object in memory depends on when it was fixated during study, with more recently fixated objects enjoying a higher probability of memory retrieval. This is clear in the case of the Fig. 2 no-refixation data; accuracy was 87% after fixating only one intervening object, 76% after two, and 65% after three intervening object fixations (Zelinsky & Loschky, 2005). Such a pattern suggests that the target was highly available for retrieval immediately after fixation, but that interference introduced by the following object fixations caused this initially high level of availability to rapidly decline. Eventually, after fixating three intervening objects, the recency effect disappeared entirely, asymptoting into an above-chance pre-recency level of performance.
To attenuate this rapid decline in immediate memory for objects, we propose that gaze refixations are used to actively maintain object information as part of a monitor–refixate rehearsal system. According to this system, the activation levels of object representations in an explicit VWM task are continuously monitored, and a refixation is made to an object in danger of being forgotten when a sudden drop is detected in its activation. In the context of the present study, this would typically occur after the fixation of an intervening object. This event would cause the activation of the previously fixated object representation to drop suddenly, which in turn generates a change signal that the system uses to bias the saccade target selection process to refixate this object. The goal of an object refixation, whether explicit or otherwise, is therefore to offset the interference introduced by fixations of other objects (see, e.g., Zelinsky & Loschky, 2005), thereby reinstating its previously high level of activation and maintaining its place in VWM.
Such a monitor–refixate system can account for many of the patterns observed in the present data. First, it explains the improved accuracy resulting from refixations, reported in Fig. 2, and the disappearance of this advantage at higher numbers of fixated intervening objects. Such benefits derive from the fact that each target refixation essentially resets the intervening-object function. If a target is refixated during study, the nontarget object inspected immediately following this refixation functionally represents only one intervening object, regardless of when the target was initially fixated. The 90% level of accuracy on two-intervening-object refixation trials is therefore comparable to the 86% level of accuracy on one-intervening-object no-refixation trials because the former is functionally equivalent to a one-intervening-object condition without refixation. Similarly, because refixations in the three-intervening-object condition would occur after fixations on either the first or the second posttarget object, accuracy in this condition should fall between the 87% and 76% levels delineated by the no-refixation function, and indeed this was the case (82%). Note, however, that this relationship would be expected to break down with larger intervening-object separations. Consider a six-intervening-object trial. Although a target refixation could have occurred following the fixation of two to five intervening objects in this condition, these refixations were not uniformly distributed over this range (as shown in Fig. 3). On the majority of these trials, the target was refixated after the first two intervening objects in the viewing sequence, meaning that any recency benefit would ordinarily have disappeared into the pre-recency asymptote by the time the six-intervening-object criterion was satisfied. This relationship between recency and intervening objects therefore explains the gradual convergence of the refixation and no-refixation functions after the fixation of six to seven intervening objects.
A monitor–refixate system can also account for our observers’ tendency to refixate a target early in the intervening-object sequence (Fig. 3). As described by the distance+IOR+VWM model, these immediate refixations are a by-product of an exponential decay process affecting object representations in VWM; an object’s activation in VWM decreases the most after fixation of the first intervening object, but the rate of this decrease decelerates with each subsequently fixated object. This change in an object’s VWM activation from one fixation to the next is what is monitored by the monitor–refixate system, with the probability of a refixation increasing with the size of this change signal. The “pull” on gaze exerted by VWM is therefore greatest after the first fixated intervening object, resulting in the skewed probability of immediate refixations.
As for what might produce this negatively accelerating decay of VWM activation, there are at least two possibilities. One explanation is that the interference produced by the fixation of the first intervening object leaves fewer features of the original representation to be interfered with by subsequent object fixations. As the number of original target-object features dwindles with each fixated intervening object, so does the opportunity for interference, resulting in a slowing of the decay rate. This explanation is consistent with the sudden loss of features known to accompany the initial saccade away from an object (Irwin, 1991, 1992, 1996), as well as the slightly more protracted recency effect reported for objects in scenes (Irwin & Zelinsky, 2002; Zelinsky & Loschky, 2005). An alternative explanation is that gaze may leave an object before its features have been fully encoded or consolidated in VWM (Hooge & Erkelens, 1999). These hastily coded features might be particularly vulnerable to interference upon gaze shifting to a new object, thereby again creating a large decrease in activation and a strong signal to refixate the previously viewed object. Although both explanations are consistent with the proposed monitor–refixate system, our analysis offered no evidence for an incomplete encoding of targets preceding a refixation. We therefore tentatively conclude that the signal ultimately responsible for triggering a refixation in our task originated from the rapid loss of feature information as a result of shifting gaze to another object.
Upon closer inspection, however, the Fig. 5 scanpath reveals an interesting regularity. A monitor–refixate rehearsal process suggests that observers should periodically refixate objects after shifting gaze to one or two intervening items. With respect to the panda bear target on this trial, the observer fixates this object with their 12th fixation, then looks to the doll (13th fixation), the first intervening object relative to the target. According to the proposed system, interference from the fixated intervening object causes a sudden loss in VWM for features relating to the panda, which in turn creates a signal biasing gaze to return to the panda (14th fixation) after fixating only a single intervening object. Of course, from the perspective of an observer studying this scene, the panda target was just another object in the display; to the extent that observers were systematically rehearsing objects in VWM, one might expect to see a similar pattern of refixations of other objects along the scanpath. This proved to be the case. The same scenario of fixations described for the panda target also appeared for the toy train and doll objects in this representative trial.
Although our proposed monitor–refixate system can successfully account for many aspects of refixation behavior in our task, one might argue that the process described by this system is not proper rehearsal. Rehearsal in working memory is most often conceptualized as a cyclical process consisting of multiple passes through a sort of loop (see, e.g., Baddeley, 1986), with each pass being instrumental in the maintenance of the rehearsed items. The rehearsal process described here is quite different, consisting of typically only one repeating cycle, when a repeating cycle appears at all. Although we mainly attribute this abbreviated rehearsal cycle to the dynamic interplay between the components of the monitor–refixate system (i.e., VWM and eye movement biases), it is also likely due in part to the unique constraints imposed by our free-viewing memory task. In a standard working memory task, a list of items is presented, followed by a retention interval, during which rehearsal is used to maintain these previously presented items in memory. However, rehearsal in our task was more interleaved with the actual presentation of the memory “list,” which was a product of the observer’s viewing behavior. Our observers were forced to trade off the acquisition of a new item in this list with the rehearsal of an old item; acquiring a new list item necessitated making an eye movement to a new object, not refixating an old one (and vice versa). Had our observers adopted a standard rehearsal strategy of cycling through the same small set of old objects, the other new objects in the display would have never been inspected.
Of course, this online characterization of rehearsal begs the question of why our observers managed not to get caught in an endless cycle of refixating the same objects. Strict adherence to the above-described monitor–refixate rehearsal process should have quickly locked the observer in a perpetual loop, with the same subset of objects continuously refixated and forgotten. This problem was avoided in our modeling efforts by assuming that the selection of each object for fixation was probabilistic, a solution that was successful yet somehow unsatisfying—leaving one wondering what specific factors determined these probabilities.
At least two mechanisms might be used to periodically break from the monitor–refixate maintenance loop long enough to include new items in the rehearsal set. One possibility assumes a contribution from long-term memory (LTM). Borrowing from the early modal model conception of memory rehearsal (Broadbent, 1958; Waugh & Norman, 1965), the repeated fixation of an object might strengthen its representation in LTM. If so, as this LTM representation becomes stronger, the need to actively maintain a VWM representation for the same object is lessened. More mechanistically, the growing “push” from LTM, when combined with IOR, would eventually overpower the “pull” back to an object by its fading VWM representation. The maintenance of the rehearsed object would therefore be abandoned, resulting in the monitor–refixation process periodically including new items in the rehearsal set. A second possibility is that observers have more direct control over the objects that they choose to maintain in VWM. We speculate that this might be achieved by monitoring the VWM activations of only a subset of the studied objects. Given that there may be many object representations calling out for rehearsal but only one locus of gaze to serve this function, the representations that observers choose to monitor would ultimately determine their refixation behavior. Using Fig. 5 as an example, the overt decision to look from the ducky (15th fixation) to the doll (17th fixation) rather than back to the panda might reflect an internal decision to monitor only the activation of the doll representation and to abandon monitoring all of the objects fixated earlier in the viewing sequence. Future work will attempt to distinguish between these possibilities, although it is quite possible that both forces may be in play.
Finally, we hope to use gaze refixation patterns to study VWM capacity, as has been done in studies of eye movements in visual search (Dickinson & Zelinsky, 2005, 2007; Gilchrist & Harvey, 2000). To the extent that these patterns reflect a memory rehearsal process, it may be possible to obtain online estimates of VWM capacity by analyzing when refixations occur in extended viewing sequences (see also Droll & Hayhoe, 2007). And, because such estimates would be relatively continuous, one might even be able to study changes in capacity during the course of individual trials and (potentially) to link these changes to properties of the specific objects being held in VWM. As these spatial and temporal processes are better understood, VWM will take on a new and dynamic conceptualization—a sort of moving window of highly available objects, with the size of this window providing an estimate of VWM capacity, and its location in the scene indicating the contents of the memory set.
Note that at the start of the trial n = 9, reflecting the fact that simulated gaze started at the center of the scene and not on an object; thereafter, when a given object was being selected, n = 8.
Because for some conditions this model generated fewer than 5% refixations, not all of the model’s data could be included in the calculation of the χ2 statistic. For this reason, the χ2 values for this model are more conservative than they might be otherwise.
We also implemented a model that included only a distance bias, and not IOR, but found that it too failed to reproduce the refixation and refixation lag data. For brevity, we therefore omit a more detailed discussion of this distance-only model.
We determined the value of the exponent by using the curve-fitting procedure from SPSS (Ver. 16) to find the decay function that best fit the recognition accuracy data from trials on which a target was not refixated. To do this, it was first necessary to extrapolate a no-refixation recognition accuracy value for a zero-intervening-object condition, which we determined to be 94.37%. Because the curve-fitting procedure used to determine the exponent yielded a maximum value for the decay function of .94669, it was necessary to subtract a constant of .05331 from all computed VWM activation levels. The resulting decay function provided a close fit to the no-refixation recognition data (R2 = .856).
G.J.Z. was supported by NIMH Grant 2-R01-MH63748 and NSF Grant IIS-0527585. L.C.L. was supported by funds from the Kansas State University Office of Research and Sponsored Programs and by the NASA Kansas Space Grant Consortium.
- Baddeley, A. (1986). Working memory (vol. 11). Oxford: Oxford University Press, Clarendon Press.Google Scholar
- Epelboim, J., & Suppes, P. (1996). Window on the mind? What eye movements reveal about geometric reasoning. In G. Cottrell (Ed.), Proceedings of the eighteenth annual conference of the cognitive science society (pp. 59–60). Mahwah, NJ: Erlbaum.Google Scholar
- Just, M. A., & Carpenter, P. A. (1978). Inference processing during reading: Reflections from eye fixations. In J. W. Senders, D. F. Fisher, & R. A. Monty (Eds.), Eye movements and the higher psychological functions (pp. 157–174). Hillsdale, NJ: Erlbaum.Google Scholar
- Locher, P. J. (1996). The contribution of eye-movement research to an understanding of the nature of pictorial balance perception: A review of the literature. Journal of the International Association of Empirical Studies, 14, 143–163.Google Scholar
- Murry, W. S., & Kennedy, A. (1988). Spatial coding in the processing of anaphor by good and poor readers: Evidence from eye movement analyses. The Quarterly Journal of Experimental Psychology, 40A, 693–718.Google Scholar
- Neath, I. (1998). Human memory: An introduction to research, data, and theory. New York: Brooks/Cole.Google Scholar
- Posner, M. I., & Cohen, Y. (1984). Components of visual orienting. In H. Bouma & D. G. Bouwhuis (Eds.), Attention and performance X: Control of language processes (pp. 531–556). Hillsdale, NJ: Erlbaum.Google Scholar
- Shebilske, W. L., & Fisher, D. F. (1983). Eye movements and context effects during reading of extended discourse. In K. Rayner (Ed.), Eye movements in reading: Perceptual and language processes (pp. 153–179). New York: Academic Press.Google Scholar