Visual working memory contaminates perception
Indirect evidence suggests that the contents of visual working memory may be maintained within sensory areas early in the visual hierarchy. We tested this possibility using a well-studied motion repulsion phenomenon in which perception of one direction of motion is distorted when another direction of motion is viewed simultaneously. We found that observers misperceived the actual direction of motion of a single motion stimulus if, while viewing that stimulus, they were holding a different motion direction in visual working memory. Control experiments showed that none of a variety of alternative explanations could account for this repulsion effect induced by working memory. Our findings provide compelling evidence that visual working memory representations directly interact with the same neural mechanisms as those involved in processing basic sensory events.
KeywordsVisual working memory Visual perception
People are able to hold and manipulate visual information in a limited-capacity working memory system (Luck & Vogel, 1997; Miller, 1956) that is believed to be related to multiple cognitive abilities, including intelligence (Desimone & Duncan, 1995; Fukuda, Vogel, Mayr, & Awh, 2010) and emotional self-regulation (Schmeichel & Demaree, 2010). Remarkably, recent brain-imaging work suggests that the contents of visual working memory might actually be represented within retinotopically organized visual areas, including the primary visual cortex (Harrison & Tong, 2009; Serences, Ester, Vogel, & Awh, 2009). If neural processes responsible for visual perception and visual working memory are indeed so intertwined, we would expect the contents of memory to be reflected in our perceptual experiences. It is known that the contents of working memory can bias the allocation of attentional resources (Desimone & Duncan, 1995; Soto, Hodsoll, Rotshtein, & Humphreys, 2008; Woodman & Luck, 2007), but whether perception itself is altered by those contents remains an unanswered question. Because sensory neurons are known to multiplex (Panzeri, Brunel, Logothetis, & Kayser, 2010), it is possible that the cognitive processes of perception and the storage of information in working memory do not directly interact even though the same neurons are involved in both processes. In this study, we directly addressed the question of whether working memory representations change the appearance of new sensory inputs, using psychophysical techniques to measure observers’ perception of motion while holding a different motion direction in visual working memory.
When two superimposed sets of random dots move in different directions, the perceived angular separation of the two directions of motion are exaggerated relative to their actual angular separation. This well-studied perceptual illusion is called motion repulsion (Hiris & Blake, 1996; Marshak & Sekuler, 1979). We reasoned that if representations of motion in visual working memory directly influence sensory inputs, an observer’s motion perception should systematically deviate away from a direction of the motion represented in visual working memory. The motion repulsion effect is especially well suited for studying interactions between perception and working memory, for two reasons. First, the neural mechanisms mediating visual working memory retention have been systematically investigated with random-dot motion stimuli in humans and nonhuman primates (Blake, Cepeda, & Hiris, 1997; Pasternak & Greenlee, 2005). Second, it is difficult to explain the repulsion effect on the basis of demand characteristics, an experimental artifact where participants form an interpretation of the experiment's purpose and the experimenter’s expectations (Orne, 1979). The direction of motion repulsion is opposite to the direction specified by assimilation due to proactive interference (Huang & Sekuler, 2010), replacement by the subsequent display (Logie, Brockmole, & Vandenbroucke, 2009), or direction averaging (Treue, Hol, & Rauber, 2000), three more intuitively obvious expectations an observer might derive about superimposed directions of motion.
All observers had normal or corrected-to-normal visual acuity, were 21 to 37 years old, and provided informed consent. Seven observers participated in Experiments 1 and 3, 4 in Experiments 2, 4, and 5, and 6 in Experiment 6. The first author participated in all but Experiment 3, and the second author participated in all the experiments. The other observers were naïve as to the purposes of the experiments and showed effects that were qualitatively similar to those of the authors.
Stimulus presentation on a Sony GDM-FW900 monitor (100-Hz frame rate) and the collection of behavioral responses were controlled by the Psychophysics Toolbox (Brainard, 1997; Pelli, 1997). Stimuli were presented on a dark background (0.2 cd/m2) 90 cm in front of the observers in a dimly illuminated room. The fixation point was a circle subtending 0.3° of visual angle (14 cd/m2) surrounding a dot. The motion stimulus was a field of approximately 157 dots moving at 3.5°/s, displayed within a circular aperture with a 4° diameter. Each dot was 0.05°, with a low luminance of 14 cd/m2 to avoid creating afterimages. The lifetime of each dot was 100 ms, meaning that dots were replotted at random locations every 100 ms, thereby reducing the likelihood that the observer could usefully track a single dot. The directions of the moving dots were normally distributed, with the mean at the designated direction of motion and a standard deviation of 20 angular degrees.
For Experiment 1, the stimulus sequence consisted of two motion stimuli and the two tasks shown in Fig. 1a. Each trial, initiated by the observer’s buttonpress, began with a 600-ms fixation period, followed by the 500-ms presentation of the memory display. A 2-s interval with only the fixation point preceded the 500-msec presentation of the perception display. Presented immediately after the offset of that stimulus was a reference bar (a thin white line, 0.03° wide, 14 cd/m2) subtending 2.7°–3.7° of visual angle from fixation. Observers reported whether the direction of motion was CCW or CW with respect to this reference bar by placing the arrow cursor (approximately 0.3° of visual angle) within a region either CCW or CW of the reference bar and then clicking the mouse button. This response had to be made within 2 s after the onset of the reference bar. One second after the perceptual judgment response, the memory recall task began, in which observers moved a clock hand to indicate the memorized direction of motion from the memory display. The clock hand extended from the outer ring of the fixation point to 2° of visual angle from the center of the monitor. Observers were required to move the mouse to the left or the right to move the clock hand either CCW or CW and then clicked the button to report the remembered direction of motion. Accuracy was stressed, and memory recall was not speeded. The initial position of the clock hand was randomly chosen from the 360° range. Feedback was provided after the memory recall response in the form of another green clock hand (visible for 500 ms; shown in black in Fig. 1a) indicating the actual direction of motion.
In Experiment 1, two independent variables were parametrically manipulated. One was the angular separation (θ) between the motion directions of the memory and perception displays, defined as θ = φ1 – φ2. For convention, angle increased in the CCW direction. Angular separation θ was manipulated at ±30°, ±60°, and ±90°, and we determined significance using a regression analysis, because motion repulsion is characteristically maximal at 30° of separation and decreases linearly with increasing separation. The other independent variable was the angular separation (δ) between the reference (φr) and the motion direction of the perception display (φ2), as defined by δ = φr – φ2. The δ was varied from –18° to 18° with 6° step sizes. For each of the four sessions with each observer, the number of measurements at each δ varied for each angular separation θ such that there was one measurement for δ = ±18°, two measurements for δ = ±12°, four measurements for δ = ±6°, and eight measurements for δ = 0°. The memory recall task produced the dependent measure (ω), which is the difference between the recalled motion direction and actual direction of the first motion stimulus (ω = φm – φ1).
Experiment 2 was identical to Experiment 1, except that the angular separation (θ) covered the entire 360° range of possible motion directions in 30° steps. Experiment 3 differed from Experiment 1 only in that there was no memory recall task. In Experiment 4, two perception displays were presented for 500 ms each, and observers performed a direction judgment task immediately after the offset of each display. The angular separation (δ) between the motion direction and the reference bar of the first motion display was not correlated with that of the second motion display. We also provided auditory feedback for correct responses only for the first perceptual judgment task in Experiment 4. Experiment 5 was identical to Experiment 1, except that the memory display was replaced by a static clock hand pointing in a direction corresponding to what had previously been a direction of motion in Experiment 1 (2° in length from the fixation to the outer ring for 500 ms). Experiment 6 was identical to Experiment 1, except that the reference bar for the perceptual judgment was presented simultaneously with the random-dot motion stimuli and observers rotated a response knob to indicate the motion direction. To compare the findings across experiments, we used regressions with the between-subjects factor of experiment (two levels; e.g., Experiment 1 vs. 6) and with the within-subjects factor of angular separation θ.
Memory contaminates perception
Consider a trial on which the perception display moved CW with respect to the memory display (Fig. 1b). If the motion represented in visual working memory contaminated the appearance of the subsequent perception display, the perceived motion direction in the perception display should be shifted CW from the actual direction, and the memorized motion direction should be shifted CCW. To describe the results more efficiently, we collapsed the results across CW and CCW relative positions, using the metrics shown in Fig. 1b.
We considered whether the repeated measurements within the limited range of angular separations θ used in Experiment 1 might have allowed observers to predict the direction of motion of the perception display and, thereby, induced a response bias that mimicked motion repulsion. We tested this in Experiment 2 by repeating the procedures of Experiment 1 with angular separations spanning the entire 360° range of motion directions. Figure 2c shows that motion repulsion for the perception display was robustly reproduced at small angular separations (30° and 60°) and the magnitude of repulsion varied significantly across angular separations, as shown by a significant effect of θ on the accuracy of the perceptual judgment, F(1, 3) = 34.46, η2 = .92, p < .01. In addition, the regression on memory recall performance also showed a significant repulsion effect, F(1, 3) = 68.85, η2 = .96, p < .01, induced by the perception display on the representation of the memory display. Evidently, the absence of a robust repulsion effect in the memory recall task in Experiment 1 occurred because the motion direction of the perception display provided predictive information that allowed observers to infer the memorized direction, due to the limited angular separation range. Note also that the repulsion effect we observed in Experiment 2, approximately 10°, is comparable to the size of perceptual repulsion reported previously (Rauber & Treue, 1999), demonstrating the robustness of the repulsion effect exerted by the working memory representation.
Might our findings be due to motion adaptation? Note that an aftereffect induced by motion adaptation after the memory display has been viewed could produce a repulsion-like effect. However, the results from Experiments 1 and 2 cannot be explained by motion adaptation, for the following two reasons. First, the 500-msec presentation of the random-dot motion we used was too brief to induce a strong motion aftereffect (Blake & Hiris, 1993). Moreover, adaptation could not build up, because the direction of motion in the memory display varied randomly over trials. Second, if our effects had been due to a motion aftereffect, we would have obtained the opposite pattern of results. Specifically, motion adaptation predicts that the size of the repulsion effect at the 30° angular separation should be smaller than the effect size at 60° and 90°. Figure 2d shows the results of modeling in which repulsion magnitude was predicted by a simple summation of the motion vector of the perception display and a motion vector associated with the motion aftereffect of the memory display (5%, 10%, and 15% of aftereffect magnitude). Our results in Experiments 1 and 2 are clearly opposite to those that would have been produced by the motion aftereffect.
Previous studies have shown that smooth pursuit eye movements can impair performance on a short-term memory span task involving judgments of spatial location (e.g., D. Pearson & Sahraie, 2003; Postle, Idzikowski, Sala, Logie, & Baddeley, 2006). Is it possible that the random-dot motion in our 500-ms displays evoked pursuit eye movements and, therefore, diluted the effect of working memory on subsequent visual perception of motion direction? Even if there were unintended eye movements despite the instruction of strict fixation, we are disinclined to believe that they could have produced the pattern of results we found. First, Postle et al. (2006) showed that it is voluntary eye movement control, not the movements per se, that causes the deleterious effects of eye movements on visuospatial working memory. Second, Postle et al. also found that effects of voluntary eye movements were seen only on tasks that involved remembering spatial locations, a very different kind of judgment from the motion direction task. Third and finally, several studies of perceptual motion repulsion have found that eye movements play essentially no role in mediating motion repulsion (Marshak & Sekuler, 1979; Rauber & Treue, 1999).
Two tasks, no memory demands
Remembering a vector
One might argue that the memory task is accomplished by retaining an abstract vector representation of the motion display, with that vector instigating the subsequently measured motion repulsion and not remembered motion per se. If motion repulsion is being produced by a vector representation in memory, motion repulsion caused by two simultaneously visible random-dot motion displays would likely be due to very different neural mechanisms than the repulsion we observed here (Marshak & Sekuler, 1979).
Eliminating the memory demands of the perceptual judgment
In the literature on working memory, there has been considerable work on the nature of the cognitive representations forming the contents of visual working memory. An effective technique for addressing this question involves assessing the influence of ongoing visual perception on visual working memory (e.g., Logie, 1986; Quinn & McConnell, 1999). In this study, we used a complementary strategy—namely, assessing the effect of visual working memory on visual perception. Our findings demonstrate that a motion representation retained in visual working memory influences the appearance of motion during a retention interval. This influence takes the form of a directional repulsion that mirrors the effects observed when two different sensory inputs are presented simultaneously. Our findings cannot be explained by motion adaptation, eye movements, visual priming, processing of the initial display without memory demands, interactions based on mnemonic representations of the motion direction, or the memory demands of the perceptual judgment task. Thus, the motion repulsion effect we observed between a visual working memory representation and a new moving stimulus implies that genuine visual features represented in working memory can interact with and influence the appearance of newly presented sensory information.1
How is this interaction mediated? Earlier work pointed to a strong influence of visual working memory on attentional selection (Olivers, Peters, Houtkamp, & Roelfsema, 2011). Here, we show that visual working memory’s influence can penetrate deep enough into the stream of events underlying perception that the contents of memory can literally change the appearance of what we are viewing. It seems logical to conclude that this influence arises because working memory co-opts the neural machinery involved in sensory processing. Indeed, we know from recent neurophysiology and neuroimaging studies that visual working memory representations may be maintained for several seconds in early visual areas (e.g., V1, V2, V3, and V4) where the analysis of basic visual information occurs (Harrison & Tong, 2009; Serences et al., 2009; Supèr, Spekreijse, & Lamme, 2001). There is an abundance of evidence indicating that imagining a visual stimulus can influence subsequent visual perception of related visual stimuli (Farah, 1985; Ishai & Sagi, 1995; Kosslyn, Ganis, & Thompson 2001; Pearson, Clifford, & Tong, 2008; Perky, 1910). One could be led to the belief that the neural concomitants of visual imagery may be found within visual working memory, meaning that our effects are similar in their source. However, this view disregards recent evidence that visual perception influences imagery but not visual working memory (e.g., Andrade, Kemps, Werniers, May, & Szmalec, 2002; Baddeley, 2007), suggesting that imagery and visual working memory have different substrates that interact with perception in different ways.
Finally, our results may have important practical implications. There are numerous instances where we engage in visually guided activities (e.g., driving) while rehashing visual events in our mind’s eye. Common sense tells us that this mental replay is harmless in that it does not interfere with our ability to register and react to objects within our visual field. Evidently, however, that is not always true when the contents of our working memories overlap with the contents of our perceptual world.
We thank Robert Sekuler and two anonymous reviewers for valuable comments. This research was supported by grants from the National Institutes of Health (RO1-EY019882; P30-EY008126), from the National Science Foundation (BCS 09–57072), and from the World Class University Initiative funded by the National Research Foundation of Korea and the Ministry of Education, Science and Technology (R32-10142).