Visual objects interact differently during encoding and memory maintenance

Abstract

The storage mechanisms of working memory are the matter of an ongoing debate. The sensory recruitment hypothesis states that memory maintenance and perceptual encoding rely on the same neural substrate. This suggests that the same cortical mechanisms that shape object perception also apply to maintained memory content. We tested this prediction using the Direction Illusion, i.e., the mutual repulsion of two concurrently visible motion directions. Participants memorized the directions of two random dot patterns for later recall. In Experiments 1 and 2, we varied the temporal separation of spatially distinct stimuli to manipulate perceptual concurrency, while keeping concurrency within working memory constant. We observed mutual motion repulsion only under simultaneous stimulus presentation, but proactive repulsion and retroactive attraction under immediate stimulus succession. At inter-stimulus intervals of 0.5 and 2 s, however, proactive repulsion vanished, while the retroactive attraction remained. In Experiment 3, we presented both stimuli at the same spatial position and observed a reappearance of the repulsion effect. Our results indicate that the repulsive mechanisms that shape object perception across space fade during the transition from a perceptual representation to a consolidated memory content. This suggests differences in the underlying structure of perceptual and mnemonic representations. The persistence of local interactions, however, indicates different mechanisms of spatially global and local feature interactions.

Introduction

The mechanisms and cortical regions underlying working memory maintenance are the matter of an ongoing debate (Bloem, Watanabe, Kibbe, & Ling, 2018; Gayet, Paffen, & Van der Stigchel, 2018; Harrison & Bays, 2018; Scimeca, Kiyonaga, & D’Esposito, 2018; Xu, 2017, 2018). One account that has gained some popularity in recent years proposes a crucial role of primary sensory cortices for the short-term maintenance of sensory information. Specifically, it has been suggested that sensory working memory storage recruits the same sensory brain regions that underlie perceptual encoding (“sensory recruitment hypothesis”; e.g., Christophel, Hebart, & Haynes, 2012; Emrich, Riggall, LaRocque, & Postle, 2013; Harrison & Tong, 2009; Riggall & Postle, 2012; Serences, Ester, Vogel, & Awh, 2009). While there are several variants of this hypothesis (e.g., sustained activity of a perception-like code – Serences, et al., 2009; cortically distributed networks of functionally distinct areas – Scimeca et al., 2018; latent “activity-silent” synaptic storage in sensory areas – Wolff, Jochim, Akyürek, & Stokes, 2017), it raises the general question about shared characteristics of mnemonic and perceptual representations. Assuming a common neural basis of perception and working memory implies that memory maintenance should be constrained by the same rules of competition and context interdependencies that characterize perceptual encoding.

On the level of perceptual processing, it has been shown that objects that are concurrently present in a visual scene affect each other’s representation both on the neural and the phenomenal level. Concurrent stimuli mutually suppress the neural responses related to stimulus processing (Allman, Miezin, & McGuinness, 1985; Born & Tootell, 1992; Eifuku & Wurtz, 1998; Li, Lei, & Yao, 1999; Raiguel, Van Hulle, Xiao, Marcar, & Orban, 1995; Störmer & Alvarez, 2014; Wachtler, Sejnowski, & Albright, 2003) and alter each other’s perceptual appearance (Blakemore, Carpenter, & Georgson, 1970; Braddick, Wishart, & Curran, 2002; Ejima & Takahashi, 1985; Kim & Wilson, 1997; Klauke & Wachtler, 2015; Marshak & Sekuler, 1979; Tzvetanov, Womelsdorf, Niebergall, & Treue, 2006; Wiese & Wenderoth, 2007). A color in front of a differently colored background appears shifted away from the background in color space (e.g., Klauke & Wachtler, 2015). Likewise, the perceived luminance contrast of a grating is reduced when it is flanked by higher-contrast gratings, and enhanced when flanked by lower-contrast gratings (e.g., Ejima & Takahashi, 1985). The orientations of two simultaneously presented lines appear repelled from each other, leading to a larger perceived angular separation than physically present (e.g., Blakemore et al., 1970). Similarly, the Direction Illusion describes the phenomenon that two motion directions that are either superimposed (e.g., Braddick et al., 2002; Marshak & Sekuler, 1979; Wiese & Wenderoth, 2007; Xiao & Huang, 2015) or presented next to each other (typically in a center-surround fashion; e.g., Kim & Wilson, 1997; Tzvetanov et al., 2006; Wiese & Wenderoth, 2010) are perceived with a larger angular separation than objectively present. The key mechanism suggested to underlie such repulsive interactions is lateral inhibition between neurons in sensory cortex that code for the respective visual objects (e.g., Benton & Curran, 2003; Blakemore et al., 1970; Blakemore & Tobin, 1972; Gilbert, 1992; Kim & Wilson, 1997; Li et al., 1999; Mather, 1980; Mather & Moulden, 1980; Wachtler et al., 2003; Wenderoth, O’Connor, & Johnson, 1986).

The present study used the Direction Illusion (e.g., Kim & Wilson, 1997; Tzvetanov et al., 2006; Wiese & Wenderoth, 2010) to assess the potential parallelism of perceptual encoding and memory maintenance. If the neural processes that underlie working memory maintenance indeed resemble the processes engaged in perceptual encoding, then the Direction Illusion should not be restricted to concurrently perceived motion directions but equally occur between two concurrently maintained directions, regardless of the temporal relationship between the stimuli during encoding. An absence of the Direction Illusion for concurrently maintained, but separately encoded directions, on the other hand, would indicate that repulsive inter-item mechanisms are not active during memory retention. We tested this hypothesis in a series of three experiments. In each experiment, subjects had to memorize two motion directions for later continuous recall. In Experiment 1 we tested whether motion repulsion requires concurrent stimulation (perceptual interaction) or whether it arises equally under sequential encoding (memory-based interaction). In Experiment 2 we differentiated the impact of perceptual after-effects from memory-induced effects by varying the temporal separation under sequential encoding conditions. The absence of motion repulsion for sequential encoding would challenge a strong parallelism account of the shared neural underpinnings of perception and working memory. Finally, in Experiment 3 we adopted the design of previous studies (i.e., spatially overlapping stimuli) to integrate our findings into the existing literature.

Experiment 1

In Experiment 1 (Fig. 1a) subjects had to encode and maintain either one or two motion directions of random dot patterns (RDPs). We presented the two RDPs either simultaneously or sequentially at different locations. We varied the similarity of their directions on the basis of subjective perceptual thresholds that were independently determined for simultaneous and sequential encoding to increase comparability. After a maintenance phase, subjects were cued to report one of the two presented items by adjusting manually the motion direction of a probe RDP. This design allowed us to investigate distortions of direction representations by analyzing shifts of the error distribution of the target item relative to the non-target item in feature space. Crucially, in both conditions subjects had to maintain the items concurrently in memory for a 1-s period. Thus, both conditions included concurrent maintenance, but only in the simultaneous condition were both items concurrently present on-screen. Assuming a neural parallelism between perceptual and memory representations, we expected mutual repulsion between the presented motion directions, both under simultaneous and sequential encoding.

Fig. 1

Task display and results of Experiment 1. (a) Subjects viewed two random dot patterns (RDPs) and memorized their motion directions (indicated here by arrows for illustration only). The RDPs were presented on different retinal positions either simultaneously (top) or sequentially (bottom). After a short delay, subjects were cued to report one of the memorized directions by adjusting the direction of a probe RDP. (b) Distortion effects measured as a shift of the mean error are shown for each presentation condition (simultaneous, S1 sequential and S2 sequential) and similarity condition (2, 3, and 4 steps of JND), with positive values indicating direction repulsion and negative values indicating attraction. Error bars depict between-subject standard errors of the mean. (c) Sign-adjusted response distributions of the different presentation conditions of Experiment 1 (pooled across JNDs). Positive values indicate responses away from the non-target directions. The gray area represents the location of the non-target directions relative to the target (simultaneous condition: −6 to −49°, sequential conditions: −10° to −49°). The black vertical line is at 0° (i.e., at the expected center of an unbiased distribution)

Methods and materials

Participants

Twenty-five adults (14 females; age 19–30 years; M = 22.27, SD = 2.16) participated in the experiment after providing written informed consent. The study was approved by the ethics committee of the University of Frankfurt Medical Faculty. All participants reported normal or corrected-to-normal visual acuity. They were naive to the purpose of the experiment and were either paid (€10/h) or received course credit for their participation. The experiment comprised two sessions, held on different days, lasting about 90 min each. The data from four subjects were excluded. One subject dropped out between the first and the second sessions. Two subjects were excluded from further data analysis because the standard deviation of their recall errors differed more than 3 SD from the mean SD. One subject was excluded due to an experimenter error (the experimental stimuli were accidentally determined by another subject’s discrimination thresholds).

Discrimination threshold estimation

Prior to the main experiment, we determined each participant’s direction discrimination thresholds for eight different orientations (from 0° to 315° in 45° steps) using a two-alternative forced choice task in an adaptive procedure controlled by the Psi-Marginal algorithm (Prins & Kingdom, 2009). Thus, the presented item pairs were generated with one stimulus always chosen from the eight predefined orientations and the second stimulus was adaptively created by the algorithm to determine the just noticeable difference (JND) set as the 75% discrimination threshold. We chose this procedure since simultaneous and sequential perception show different discrimination thresholds (Lakshminarayanan, Raghuram, & Khanna, 2005). By tuning our similarity steps to psychophysical thresholds we were able to keep similarity constant across the presentation modes. Since direction discrimination is known to be influenced by the proximity to the cardinal directions with discrimination being most precise close to the cardinals and least sensitive at oblique directions (the so-called oblique effect; Appelle, 1972; Gros, Blake, & Hiris, 1998; Matthews & Qian, 1999), we determined discrimination thresholds for different directions (cardinals and oblique) to account for these differences in our similarity manipulation.

Stimuli and apparatus

In each trial two random dot patterns (RDP) were presented at a distance of 10° of visual angle to the left and the right of a fixation square (edge length 0.3° of visual angle) that was located in the center of the screen. RDPs consisted of 400 white dots on a black background, with each dot covering 0.15°. Dots were displayed within an invisible circular aperture of 15° in diameter. Dots reaching the edge of the circular aperture were repositioned randomly on the opposite side of the aperture; therefore, dot density was kept constant throughout the presentation. Motion was 100% coherent and within a trial all dots moved with the same speed. Speed varied randomly between 6°/s and 14°/s between trials, with no speed being repeated in immediate succession. In the sequential presentation condition, spatial position and serial position were counterbalanced, i.e., the first presented stimulus appeared in half of the trials to the left of the fixation square. MATLAB R2010a and the Psychophysics Toolbox (Brainard, 1997) were used to generate and display the stimuli. The participants viewed stimuli on an LCD monitor (refresh rate 60 Hz) from a distance of 50 cm in a dimly lit room.

Procedure

Each trial began with a 1-s fixation period. The two stimuli then appeared either simultaneously for 1 s or sequentially for 0.5 s each, without an inter-stimulus interval (ISI). Simultaneous and sequential trials were randomly interleaved. Subsequent to stimulus presentation, the participants judged which one of the two stimuli was more clockwise oriented by pressing the corresponding mouse button (left button for the left stimulus, right button for the right stimulus). There was no time limit on the response. After the response the participants received feedback. The fixation square turned red if the response was wrong and green if the response was correct. In total the threshold estimation procedure comprised 800 trials. Each of the eight directions was measured separately for simultaneous and sequential presentation, with 50 trials for each direction and presentation mode. Subjects were instructed to fixate the central square throughout a trial.

Analysis

For simultaneous presentation the mean JNDs for the reference directions (0°, 45°, 90°, 135°, 180°, 225°, 270°, and 315°) were 6.36°, 8.90°, 5.94°, 9.42°, 5.56°, 9.56°, 4.93°, and 8.41°, respectively. For sequential presentation the respective mean JNDs were 6.80°, 10.41°, 5.44°, 9.75°, 6.11°, 9.68°, 4.58°, and 10.43°. The resulting individual discrimination thresholds for the eight different reference directions were subsequently used in a curve-fitting procedure to extrapolate the thresholds across the whole 360° range (i.e., to achieve threshold estimates for those directions we did not test directly). We computed separate fits for each quadrant (i.e., 0° - 90°, 90° - 180°, 180° - 270°, 270° - 360°) to account for possible direction-specific asymmetries in motion perception (e.g., motion to the right vs. motion to the left). The function used for fitting was: f(x) = a × sin(2 × x/180 × pi) + b + c × x, with a being the amplitude, b the base level, and c the skew (see Supplementary Fig. 1 for a graphic depiction of the measured and fitted data). Curve fitting was done with the EzyFit toolbox for Matlab (Moisy, 2011). As thresholds tend to differ between simultaneous and sequential presentations (Lakshminarayanan et al., 2005), we calculated threshold functions separately for both data sets of each subject.¹

Working memory experiment

Stimuli and apparatus

The stimuli were the same as used for discrimination threshold estimation except that a set size 1 condition was included in which only a single stimulus was presented. For the set size 1 condition an item’s direction was randomly chosen from 360 different directions, spanning the whole circle in steps of 1°. In the set size 2 conditions the presented item pairs were generated by drawing one item direction randomly from 360 directions and picking another direction to be 2, 3, or 4 JNDs more clockwise. Cumulated JNDs were calculated using the graphic method described by Luce and Edwards (1958). The three inter-item differences (2, 3, and 4 JNDs) appeared equally often within each presentation condition and were randomly interleaved. The average angular separations were 16.65° (inter-subject range 10.22–24.63), 24.90° (range 15.28–37.58), and 32.40° (range 20–48.92) for JND2, 3, and 4 in the sequential condition, and 15.56° (range 5.99–24.62), 23.06 (range 9.24–37.01), and 30.67° (range 12.18–48.97) for JND2, 3, and 4 in the simultaneous condition.

MATLAB R2010a and the Psychophysics Toolbox (Brainard, 1997) were used to generate and display the stimuli. The participants viewed stimuli on an LCD monitor (refresh rate 60 Hz) from a distance of 50 cm in a dimly lit room.

Procedure

Figure 1a depicts the trial structure for the simultaneous and sequential presentation condition, respectively. Each trial began with a 1-s fixation period. The stimuli then appeared either simultaneously for 1 s or sequentially for 0.5 s each without an inter-stimulus interval (ISI) (in the set size 1 condition only one stimulus appeared for 0.5 s; presentation durations were chosen to match the average time-per-item in the different presentation conditions). Subsequent to stimulus presentation there was a 1-s delay. In the set size 1 condition, half of the trials had a 1-s delay and the other half a 1.5-s delay to match the time between stimulus presentation and report with the first and second presented stimulus of the sequential condition, respectively. Following the delay, a cue appeared for 1 s, indicating the item for recall by pointing to the left or right. After the cue-offset a randomly oriented RDP appeared at the position of the cued item. Participants adjusted the direction of the RDP by moving the mouse to the left or right. There was no upper time limit to the response. After entering their response, participants received feedback for 0.3 s. A colored dot, appearing next to the fixation square to the side of the recalled item in a color from green to red drawn from a continuous color scale indicated the precision of the response. Relative direction (clockwise, counter-clockwise), serial position (S1, S2), and spatial position (left, right) of the probed item were counter-balanced. The experiment consisted of 1296 experimental trials, 432 trials per condition (set size 1, simultaneous, sequential). Participants performed 15 practice trials at the beginning that were excluded from data analysis.

Analysis

Response errors were calculated as the difference between the reported and the veridical direction of the cued item with a positive sign indicating responses away from the non-target item and a negative sign indicating responses towards the non-target. For this calculation we inverted the response errors for counter-clockwise directions. In consequence, general motor response biases and item-inherent biases averaged out in the analysis since they received opposite signs for clockwise and counter-clockwise directions (see Huang & Sekuler, 2010). The precision of memory representations is expressed as the circular standard deviation of the error distributions. The set size 1 condition served to identify a possible memory decay over time associated with the different maintenance durations between S1 (1.5 s) and S2 (1 s) (see Huang & Sekuler, 2010, for a similar procedure). For the set size 1 condition, we merely calculated the standard deviation of the error distribution. Note that no second item was present in set size 1 trials to which a response bias could be related. Hence, the set size 1 condition did not qualify for a bias analysis comparable to the analyses for the set size 2 conditions.

Results

Figure 1b shows the mean errors for each presentation and similarity condition, with positive values indicating motion repulsion and negative values indicating attraction. Simultaneously presented items showed a repulsion effect; i.e. they were reproduced as shifted away from the non-target item direction. For sequential presentation, distortion directions differed between items: the first presented item (S1) was attracted towards the second item in the sequence (S2), while S2 was repulsed from S1. To test for systematic distortions in the different conditions, we separately tested the mean response error for each cell of our design (Simultaneous, S1, S2 × JND2, JND3, JND4) against the null hypothesis that there is no distortion in the mean of the error distributions. All cells showed significant deviations from zero (see Table 1).

Table 1 Comparisons of observed mean biases against zero (one-sample t-tests)

To compare the distortions in the different conditions directly, we calculated a two-way analysis of variance (ANOVA) with the two within-subject factors presentation condition (Simultaneous vs. S1 vs. S2) and similarity (2 vs. 3 vs. 4 JNDs). We found that while there was no main effect of similarity, F(2, 40) = 2.40 p = .104, η² = 0.11, there was a significant main effect of presentation condition, F(1.38, 27.67) = 76.59, p < .001, η² = 0.79 (Greenhouse-Geisser corrected), and a significant interaction, F(4, 80) = 2.97, p = .024, η² = 0.13. To disentangle the results, we ran two separate ANOVAs. First, we compared Simultaneous vs. S2 to see if both conditions differed in distortion magnitude and if there was a modulation of the effect strength by similarity. The ANOVA yielded main effects of similarity, F(1.42, 28.37) = 6.03, p = .012, η² = 0.23, and presentation condition, F(1, 20) = 6.52, p = .019, η² = 0.25, but no interaction, F(2, 40) = 0.21, p = .815, η² = 0.01. To test if there was a similarity modulation for S1, we conducted a one-factorial ANOVA with all three JND steps. There was no significant effect of similarity, F(2, 40) = 1.15, p = .328, η² = 0.05.

However, due to the lack of an ISI between S1 and S2 in the sequential condition, the sudden S2 onset might have interfered with S1 consolidation. Furthermore, since both items were presented successively, the time intervals between item presentation and cue differed (1.5 s for S1; 1.0 s for S2), thus potentially introducing time-based differences of memory precision. To rule out the possibility that the observed distortion effects were confounded by these factors, we ran two control analyses. First, to test whether the consolidation of S1 may have been interrupted, resulting in decreased S1 performance relative to S2, we compared the memory precision of S1 and S2 (see Fig. 2a) with a two-way analysis of variance (ANOVA) with the two within-subject factors serial position (S1 vs. S2) and similarity (2 vs. 3 vs. 4 JNDs). Memory precision of S1 did not differ from S2, F(1, 20) = 0.62, p = .441, η² = 0.03. Furthermore, there was no main effect of similarity, F(2, 40) = 0.992, p = .380, η² = 0.05, and no interaction, F(2, 40) = 1.55, p = .224, η² = 0.07. The 500-ms interval of item exposition therefore was most likely sufficient to fully consolidate the S1. Second, to test for a possible impact of different delay durations, we compared memory precision in the set size 1 conditions where a single item was tested after long (1.5-s) versus short (1-s) delays. We observed comparable memory precision for both delay periods (short delay: M = 22.24°, SD = 10.71°; long delay: M = 23.32°, SD = 10.02°), t(20) = −1.44, p = .167, d = −0.31 (paired t-test).

Fig. 2

Recall precision across all three experiments, expressed as the standard deviation (SD) of the recall errors. Recall precision was comparable across serial positions in Experiment 1 (a) and Experiment 2 (b). A recency benefit (higher precision of the second direction of the sequence, S2) emerged in Experiment 3 (c). Sim = simultaneous presentation condition, S1 = first presented item in the sequential presentation condition, S2 = second presented item in the sequential presentation condition. Error bars depict between-subject standard errors of the mean

Discussion

Experiment 1 showed that when two items were presented simultaneously in the periphery, subjects reported the directions as repelled from each other. This is in line with the Direction Illusion literature that reports repulsion between two concurrent motion directions in the high-similarity range (e.g., Kim & Wilson, 1997).

For sequential encoding the distortion effects diverged: subjects reported S1 as attracted by S2, and S2 as repulsed by S1. The proactive S2 repulsion showed a highly similar profile to the simultaneous encoding condition, suggesting a similar interference to between two concurrently viewed stimuli. Yet, the overall magnitude of the repulsion effect in the simultaneous condition was slightly higher, which may result from the differences in the presentation duration between the conditions. This result hence demonstrates that motion repulsion is not restricted to simultaneous perceptual stimulation but also occurs between successive stimuli in a proactive manner. In contrast, S1 was not repulsed but attracted by S2. The opposite distortion directions of S1 and S2 suggest that distortions were based on different mechanisms, or happened between different representational states of the items.

Experiment 1, however, does not allow unequivocally attributing the observed interactions between S1 and S2 to the level of working memory representations. As the two stimuli were presented in direct succession without an ISI, several types of interactions between them are conceivable that are not related to working memory. For example, rapid serial presentation produces a pseudo-simultaneous perception of succeeding stimuli due to temporal integration of previous and subsequent visual input (Di Lollo, 1977, 1980; Shioiri & Cavanagh, 1992; Sugita, Hidaka, & Teramoto, 2018). Thus, instead of relying on the memory representation of S1, S2 repulsion might have been induced by a lingering perceptual activation related to S1 (e.g., visual persistence or iconic memory, Coltheart, 1980) that created a short-term, quasi-simultaneous experience for both stimuli. Visual persistence for dot motion signals has been shown to last for about 130 ms, to rely on cortical rather than retinal post-stimulus-offset activity, and to even survive successive masking with coherent motion (Shioiri & Cavanagh, 1992). Iconic memory for motion direction has been shown to decay even more slowly with a decay up to about 500 ms (Shooner, Tripathy, Bedell, & Ögmen, 2010). Furthermore, the S2 repulsion we observed could reflect S1-induced adaptation. Adaptation to a motion direction induces a repulsed perception of subsequently viewed motion directions (Levinson & Sekuler, 1976; Patterson & Becker, 1996). Even though adaptation is usually associated with prolonged exposition and retinotopic specificity, there is evidence that neural adaptation can also appear for very brief exposition durations (e.g., Glasser, Tsui, Pack, & Tadin, 2011; Kanai & Verstraten, 2005; Patterson, Wissig, & Kohn, 2013; Pavan, Marotti, & Campana, 2012) and that direction adaptation induces slight repulsion effects beyond the retinal position of the adapter stimulus (Wenderoth & Wiese, 2008).

Regarding the S1 attraction effect, the immediate succession of S1 and S2 bears a possible explanation as well. To detect motion, it is necessary to integrate sensory samples across a prolonged time frame, as motion perception implies change detection of an object’s spatial positon. Motion integration has been found to occur across a period of about 100 ms of signal sampling (e.g., Alais, Apthorp, Karmann, & Cass, 2011; Burr, 1980; Snowden & Braddick, 1991). Since motion perception relies on signal integration across time, the S1 attraction effect might stem from an accidental integration of early S2 motion signals into a not-yet-fully consolidated S1 representation. Due to the short presentation time, participants might have used the S1 offset to signal the end of S1 information sampling. Since S1 offset and S2 onset coincided, a small proportion of the initial S2 motion signals might have blended into the S1 motion calculation before the integration window closed and mental processing shifted from S1 processing to S2 processing. Note that this framework suggests that the initial S2 motion perception is shifted towards the S1 direction. However, such a proactive signal merging would probably be corrected in the stream of ongoing S2 signal sampling.

With regard to the JND-modulation, we found an increase of the repulsion effect with increasing inter-item differences. This similarity modulation was comparable for simultaneously presented items and for S2 during sequential presentation. This result might seem counter-intuitive at first, because increased motion repulsion has been found for similar as compared to dissimilar items (e.g., Braddick et al., 2002; Kang & Choi, 2015; Kim & Wilson, 1997), with a peak repulsion effect at about 45° angular separation between two directions. However, those studies presented items across a wide range of similarity steps. In contrast, we used individual discrimination thresholds as the unit of our similarity manipulation with a maximum difference of four JNDs. In fact, the largest angular deviation presented in our experiment was 49° (that is, for the subject with the largest threshold in our sample). Therefore, all presented angular deviations were equal to or below this value. In other words, for most subjects, all direction differences in our experiment were on the increasing side of the repulsion profile observed in other studies (i.e., below 45° angular separation). Hence, finding an increase of effect magnitude with increasing dissimilarity can be attributed to the limited similarity range tested here.

Experiment 2

To distinguish interactions based on transient post-expositional processes from interactions induced by consolidated working memory representations, we ran a second experiment that focused solely on sequential presentation and varied the ISI in three steps (0 s, 0.5 s, 2 s). If the distortions we observed in the sequential condition of Experiment 1 were merely due to interference with consolidation (retroactive attraction effect), or some kind of short-term adaptation or concurrency with residual activity in the sensory processing areas (proactive repulsion effect), then we would assume the distortion effects to disappear or gradually decrease with a prolonged temporal separation of the stimuli. If, on the other hand, the distortions were based on interactions involving stable working memory representations, ISI length should not modulate the effects. Or, at least, interactions should be observed at all ISIs.

Methods and materials

Subjects

Fifteen adults (10 females; age 18–27 years; M = 20.80, SD = 2.76) participated in the experiment after providing written informed consent. Based on the effect sizes of the mean distortion effects in Experiment 1 (d = −0.78, for S1 attraction, d = 1.19, for S2 repulsion), and an alpha level of 0.05, a sample size of 15 was determined to achieve a power of 80% for the smaller effect (i.e., S1 attraction, computed with G*Power, 3.1.9.2; Faul, Erdfelder, Lang, & Buchner, 2007). The study was approved by the ethics committee of the University of Frankfurt Medical Faculty. All participants reported normal or corrected-to-normal visual acuity. They were naive to the purpose of the experiment and were either paid (€10/h) or received course credit for their participation.

Stimuli

The stimuli were identical to Experiment 1. Note that we did not conduct a discrimination threshold estimation in Experiment 2. Instead, the difference between the directions of S1 and S2 was drawn randomly from the JND2–JND4 range that we had observed in Experiment 1 (i.e., 20–50° inter-item difference with a 1° resolution).

Procedure

Figure 3a depicts the trial structure of Experiment 2. Experiment 2 used the same procedure as the sequential condition of Experiment 1. A 0-s ISI condition served as a replication of the sequential condition in Experiment 1. In addition, we introduced a 0.5-s and a 2-s ISI condition. Subjects completed 768 experimental trials (256 trials per ISI condition) and 15 practice trials in one 2-h session. Trials from the three ISI conditions were randomly interleaved and subjects had no knowledge about the upcoming ISI to avoid ISI-specific encoding strategies (Bankó & Vidnyánszky, 2010).

Fig. 3

Task display and results of Experiment 2. Top row. (a) Subjects viewed two random dot patterns (RDPs) and memorized their motion directions (indicated here by arrows for illustration only). The RDPs were presented on different retinal positions, separated by an inter-stimulus interval (ISI) of 0, 0.5, or 2.0 s. After a short delay, subjects were cued to report one of the memorized directions by adjusting the direction of a probe RDP. (b) Distortion effects measured as a shift of the mean error are shown for each ISI condition, with positive values indicating direction repulsion and negative values indicating attraction. Error bars depict between-subject standard errors of the mean. Middle row. Sign-adjusted response distributions of the first presented item (S1) in Experiment 2. The gray area represents the location of the non-target directions (−20° to −50°). Positive values indicate responses away from the non-target directions. Bottom row. Sign-adjusted response distributions of the second presented item (S2) in Experiment 2. In the 0.5 s (middle panel) and 2 s (right panel) ISI condition responses were centered around the true target value (0°, solid vertical line). In the 0-s ISI condition (left panel) the distribution appears slightly shifted to the right, i.e., away from the non-target directions. The black vertical line is at 0° (i.e., at the expected center of an unbiased distribution)

Analysis

Data analysis was identical to Experiment 1. For ease of analysis, we pooled all trials per condition (i.e., we did not differentiate between different degrees of inter-item similarity). In addition to the frequentist analyses, we added Bayes factors (BF₁₀) to the one-sample t-tests to evaluate our data in terms of evidence for the null hypothesis (H0). A BF₁₀ > 1 indicates evidence in favor of the alternative hypothesis (H1) (systematic distortion), while a BF₁₀ < 1 supports the H0 (no systematic distortions). In accordance with Kass and Raftery (1995), we consider BFs between 1/3 and 3 as inconclusive evidence. Consequently, we treat BF₁₀ > 3 as support for the H1, and BF₁₀ < 0.33 as support for the null hypothesis. All data analyses were conducted with the statistics software JASP (version 0.9.1.0, JASP Team, 2018) and the default settings of the Bayes factor package (Morey & Rouder, 2015); that is, Bayesian t-tests (Rouder, Speckman, Sun, Morey, & Iverson, 2009) were computed with a Cauchy prior with a width of r = 0.707. Priors were centered on zero.

To quantify the amount of guessing in our data, we modelled the data according to the mixture model by Zhang and Luck (2008), with the mean as an additional free parameter, to get an estimate of the guessing rate for the different conditions (modeling was done via the MemToolbox for Matlab, Suchow, Brady, Fougnie, & Alvarez, 2013; memtoolbox.org). The mixture model dissects the empirical response distributions into two different components, a uniform guessing distribution under which each direction is equally likely to be reported given the participant does not remember the probed item, and a van Mises distribution that represents the likelihood to report a direction given the participant knew the probed item. The guessing rate can also be estimated using the model by Bays, Catalao, and Husain (2009). This model includes also a third parameter that allows for identification of misreports (“swapping”) in continuous-recall data. As an attractive mean shift, as observed in our data, can in principle result from such misreports, the model proposed by Bays et al. (2009) could be preferable to Zhang and Luck (2008) as it estimates both guessing rate and swapping rate. However, Bays (2016) noted that for highly similar items, as in our study, their model may fail to distinguish attractive target shifts from swapping. Hence, valid estimations of mean shift and swapping are certainly not feasible (for a more detailed description of this problem see Discussion in Experiment 2). In addition, we also found that a formal model comparison between the Zhang and Luck and Bays et al. models preferred the former (for a detailed analysis, see Supplementary Analysis 3). Together, given the possible misattributions of mean shift and swapping in the Bays et al. model and the results from our model comparison in favor of the Zhang and Luck model, we quantified the amount of guessing using the Zhang and Luck model.

Results

First, we tested for significant distortions in each condition (serial position × ISI) separately. Figure 3b shows the mean errors for each serial position and ISI condition, with positive values indicating repulsion and negative values indicating attraction. One-sample t-tests revealed significant attraction effects in all three ISI conditions for S1 (all Bayes factors showed strong evidence in favor of the H1), t(14) = −3.48, p = .004, d = −0.90, BF₁₀ = 12.77, t(14) = −3.41, p = .004, d = −0.88, BF₁₀ = 11.45, t(14) = −4.84, p < .001, d = −1.25, BF₁₀ = 123.73_, for the 0-s, 0.5-s, and 2-s ISI conditions, respectively. A repeated-measures ANOVA that comprised all three conditions found no significant difference in the magnitude of the attractive mean shift, F(2,28) = 0.07, p = .931, η² = 0.01.

For S2 we found a significant repulsion effect in the 0-s condition, t(14) = 3.50, p = .004, d = 0.90, BF₁₀ = 13.23. We observed no significant mean shifts for any of the other two ISIs, t(14) = 0.45, p = .660, d = 0.12, and t(14) = 0.20, p = .842, d = 0.05, for the 0.5-s and the 2-s ISI conditions, respectively. The corresponding Bayes factors showed moderate evidence in favor of a null effect, BF₁₀ = 0.29, and BF₁₀ = 0.27, for the 0.5-s and the 2-s ISI conditions, respectively. A repeated-measures ANOVA revealed a significant difference between conditions, F(2,28) = 8.12, p = .002, η² = 0.37. The mean bias of the 0-s condition differed significantly from the 0.5-s condition, t(14) = 2.78, p = .015, d = 0.72, and from the 2-s condition, t(14) = 3.53, p = .003, d = 0.91. Means did not differ between the 0.5-s and the 2-s conditions, t(14) = 0.29, p = .773, d = 0.08.

Next we compared the precision for serial position × ISI to ensure that our distortion results were not confounded by fidelity differences across the ISI steps (see Fig. 2b). The two-way repeated-measures ANOVA showed a higher precision for S2 than for S1, F(1,14) = 5.11, p = .040, η² = 0.27 (main effect “serial position”), but no effect of ISI, F(2, 28) = 1.16, p = .327, η² = 0.08, and no interaction, F(2,28) = 2.53, p = .097, η² = 0.15.

Finally, we compared the guessing rates for the S1 across all three ISI conditions (Supplementary Fig. 4) to rule out the possibility that the loss of the S2 repulsion effect was due to forgetting the inducer stimulus (i.e., S1) at longer ISIs. The average S1 guessing rate was small, with a guessing probability of 6.20% (SD = 9.70), 5.54% (SD = 9.29), and 7.68% (SD = 13.14) for 0-s, 0.5-s, and 2-s ISI, respectively. A comparison of S1 guessing rates across the three ISI conditions showed no effect of the ISI, F(1.28,17.97) = 1.68, p = .214, η2 = 0.11 (one-factorial repeated-measures ANOVA, Greenhouse-Geisser corrected). We applied mixture modelling specifically to test for the guessing account of our data pattern. A full analysis including all modelled parameters (mean bias, precision) can be found in the Supplementary Material (Analysis 2).

Discussion

Experiment 2 replicated the S2 repulsion in the 0-s ISI condition. However, S2 repulsion disappeared for ISIs of 0.5 s and 2 s. In contrast, S1 attraction was present for the entire range of ISIs.

The finding that repulsion for S2 occurred only when both items followed each other immediately strongly indicates that it was based on a brief, transitional process following S1 presentation. For example, it could be attributable to short-term adaptation or lateral inhibition exerted by residual sensory S1 activity that quickly faded following stimulus offset. The loss of the S2 repulsion effect under prolonged ISIs suggests that once S1 had been consolidated into working memory, it no longer affected the S2 representation, even though both items were concurrently maintained for 1 s. Thus, we found no evidence that motion repulsion occurs between concurrently maintained working memory representations.

However, there are several alternative explanations for the loss of the repulsion effect with increasing ISI. First, it could be possible that in the 0.5- and 2-s ISI conditions, S1 was routinely forgotten at the point where the item interaction otherwise would have occurred. In fact, the ISI manipulation altered the maintenance duration for S1 (but not for S2), and previous research has shown that the likelihood of forgetting an item in working memory increases with the duration of memory maintenance (Zhang & Luck, 2009). To assess this possibility, we compared estimated guessing rates between ISIs. Even though a large proportion of guessing seemed unlikely given the small set size of two items (Pertzov, Manohar, & Husain, 2017; Zokaei, Gorgoraptis, Bahrami, Bays, & Husain, 2011), if the loss of S2 repulsion at longer ISIs was due to forgetting S1, we would expect a marked increase of the S1 guessing rate between the 0-s and the other two ISI conditions. The estimated guessing rates for S1 were, however, comparable across all three ISI steps, indicating that the loss of the S2 repulsion effect at longer ISIs could not be explained by forgetting the S1 stimulus. A second possibility is that a counteractive process covered the S2 repulsion at longer ISIs. For example, a separate attraction mechanism of similar strength to the repulsion effect could have nullified an existent repulsion effect. In fact, proactive attraction has been observed for successive, trial-separated motion directions (Czoschke, Fischer, Beitner, Kaiser, & Bledowski, 2019; Fischer, Czoschke, Peters, Rahm, Kaiser, & Bledowski, 2019). The same positive serial dependence might also operate between concurrently task-relevant representations (but see Czoschke et al., 2019, for strongly opposing distortion directions within and between trials). Alternatively, repulsion could also be masked by an increased number of occasional misreports of the non-target (“swapping”; see the next paragraph for a more detailed description of this error type). However, both accounts would require that the repulsive and attractive forces are well balanced to result in the null effects we observed.

With regard to S1 attraction, Experiment 2 strongly indicated that it did not result from an accidental integration of S2 motion signals into a not-yet fully consolidated S1 representation. Rather it appears that S2 interacted with a stable S1 memory representation in an attractive manner. However, attractive distortions, measured as the mean shift of the error distribution towards another item, have to be interpreted with caution, as they could also be an artifact resulting from occasional misreports of the non-target (“swapping”). If, in a subset of trials, the non-probed item is reported, the mean bias is shifted accordingly. Thus, it is possible that the S1 attraction in our data resulted from accidentally reporting the S2 direction when S1 was probed, rather than from a systematic distortion of the S1 representation. Even though Bays, Catalao, and Husain (2009) introduced a modeling approach that allows for identification of misreports in continuous recall data, Bays (2016) noted that for highly similar items the current models fail to distinguish attractive target shifts from swapping. Hence, we cannot rule out this possibility and are unaware of any way of doing so.

In summary, combining the results from Experiments 1 and 2, we observed motion repulsion under conditions that arguably involve co-active perceptual information about the stimuli in the sensory areas (i.e., during simultaneous presentation in Experiment 1, and during immediate serial presentation of both items in Experiments 1 and 2). In contrast, this effect vanished when the stimuli were sufficiently separated in time, and, consequently, only the consolidated memory representation of the S1 remained as a potential inducer.

However, our finding of the disappearing repulsion effect stands in contrast to two other studies that investigated motion repulsion. In a study by Kang, Hong, Blake, and Woodman (2011), participants had to memorize the direction of a briefly presented (0.5 s) single RDP for later recall, and during the retention interval saw a second RDP for a perceptual two-alternative forced choice decision task. Here the memorized direction repelled the perceived direction of the intervening stimulus that appeared 2 s after offset of the memory item. In another study, Kang and Choi (2015) presented two RDPs for memorization, separated by an ISI of 1 s. Again, they observed a proactive repulsion effect. Importantly, both studies found proactive repulsion within an ISI range where we did not find any proactive distortion. However, in both studies, RDPs were presented to the fovea, and thus to the same retinal position, while we presented our stimuli to different hemifields. Previous studies have suggested that sensory working memory representations are bound to the spatial position where they have been encoded (e.g., Foster, Bsales, Jaffe, & Awh, 2017; Hollingworth, 2006, 2007; Pratte & Tong, 2014; Sneve, Alnaes, Endestad, Greenlee, & Magnussen, 2011, Zaksas, Bisley, & Pasternak, 2001). Furthermore, a recent study by Rademaker, Chunharas, & Serences (2019) presented evidence that a single memory content remains actively represented in sensory visual cortex areas even during concurrent sensory distraction at the same spatial position. While our results so far indicated that concurrent maintenance does not trigger repulsive interactions between spatially distinct working memory representations, a maintained memory representation might collide locally with upcoming stimuli. Hence, we ran a third experiment to test whether memory-based motion repulsion requires spatial overlap of the stimuli or is specific to foveal processing.

Experiment 3

In Experiment 3 we presented the same stimuli as in Experiment 2 with an ISI of 2 s, but this time either foveally or at identical retinal positions in the extra-foveal periphery. If memory-based motion repulsion selectively involves the spatial position of the memoranda, we expected motion repulsion in both conditions. If it is specific to foveal processing, we expected motion repulsion under foveal presentation only.

Methods and materials

Subjects

Sixteen adults (13 females; age 18–27 years; M = 22.75, SD = 2.86) participated in the experiment after providing written informed consent. Sample size was similar to Experiment 2 but with one additional subject due to balancing of the block design. Two previous studies that report proactive, memory-induced motion repulsion for foveally presented RDPs measured seven (Kang et al., 2011, Experiment 1) and 12–14 participants (Kang & Choi, 2015, Experiment 1 and Experiment 3, respectively). Sufficient data for power analysis are, however, not provided in the publications. The study was approved by the ethics committee of the University of Frankfurt Medical Faculty. All participants reported normal or corrected-to-normal visual acuity. They were naive to the purpose of the experiment and were either paid (€10/h) or received course credit for their participation.

Stimuli

The stimuli were identical to Experiment 2. However, in Experiment 3 both RDPs were either presented to the same peripheral position or to the center of the screen.

Procedure

Figure 4a depicts the trial structure of Experiment 3. Experiment 3 used the same procedure as the 2-s ISI condition of Experiment 2 except for the following differences: Both stimuli of a trial appeared at the same spatial position on the screen with a temporal separation of 2 s. The presentation conditions (peripheral and foveal presentation) were blocked with one block per condition, and counterbalanced across subjects. In the peripheral presentation condition in half of the trials the stimuli appeared on the left side of the fixation square. The side of presentation (i.e., left field, right field) was randomly interleaved. The cue that indicated the recall item was presented as a number (“1” or “2” for the first and second presented stimulus, respectively). Subjects completed 624 experimental trials (312 trials per presentation condition) and 15 practice trials in one 2-h session.

Fig. 4

Task display and results of Experiment 3. (a) Subjects viewed two random dot patterns (RDPs) and memorized their motion directions (indicated here by arrows for illustration only). The RDPs were presented on the same retinal position either foveally (top) or peripherally (bottom). After a short delay, subjects were cued to report one of the memorized directions by adjusting the direction of a probe RDP. (b) Distortion effects measured as a shift of the mean error are shown for each encoding condition, with positive values indicating direction repulsion and negative values indicating attraction. Error bars depict between-subject standard errors of the mean. (c) Sign-adjusted response distributions of Experiment 3. Positive values indicate responses away from the non-target directions. The gray area represents the location of the non-target directions (−20° to −50°). The black vertical line is at 0° (i.e., at the expected center of an unbiased distribution)

Analysis

Data analysis was identical to Experiment 2.

Results

First, we tested for significant distortions in each condition (serial position × spatial position) separately. Figure 4b shows the mean errors for each serial position and encoding condition, with positive values indicating motion repulsion and negative values indicating attraction. For S1, one-sample t-tests revealed no significant mean bias for either condition, t(15) = −1.35, p = .197, d = −0.34, t(15) = −1.62, p = .126, d = −0.41, for the foveal and peripheral conditions, respectively. Results of the Bayesian analysis, however, were inconclusive, with BF₁₀ = 0.55 (foveal), and BF₁₀ = 0.75 (peripheral). The mean bias did not differ significantly between conditions, F(1,15) = 0.48, p = .501, η² = 0.03 (repeated-measures ANOVA).

For S2, we found a significant repulsion effect in both conditions, t(15) = 3.12, p = .007, d = 0.78, BF₁₀ = 7.30, t(15) = 2.84, p = .012, d = 0.71, BF₁₀ = 4.60, for the foveal and peripheral conditions, respectively. A repeated-measures ANOVA showed no significant difference between both conditions, F(1,15) = 0.52, p = .483, η² = 0.03.

Next, we compared the precision for serial position × spatial position (see Fig. 2c). The two-way repeated-measures ANOVA showed a higher precision for S2 than for S1, F(1,15) = 39.79, p < .001, η² = 0.73 (main effect “serial position”), but no difference between foveal and peripheral presentation, F(1, 15) = 0.13, p = .720, η² = 0.01, and no interaction, F(1,15) = 1.01, p = .330, η² = 0.06.

Discussion

In Experiment 3 we presented both stimuli at the same retinal position either to the fovea (screen center) or the periphery. We observed proactive motion repulsion in both conditions. This replicates and extends previous work that showed proactive repulsion of motion direction with ISIs of up to 2 s under foveal stimulation (Kang et al., 2011; Kang & Choi, 2015). The proactive repulsion in Experiment 3, however, is in apparent contrast to Experiment 2 where we presented our stimuli to non-overlapping spatial positions and did not observe proactive repulsion at the ISI of 2 s. This discrepancy suggests a retinotopically specific repulsion mechanism that is less sensitive to the temporal separation of sensory input than the mechanism that caused long-range repulsion across hemifields in Experiments 1 and 2.

Experiments 2 and 3 also differed with respect to the cue type used to indicate the recall target. Experiment 2 cued the spatial position while Experiment 3 cued the serial position of the target item. The reappearance of proactive repulsion in Experiment 3 might thus result from the different cueing feature rather than from spatial overlap. Indeed, results from Fischer et al. (2019) suggest that the binding structure between a recall feature and its visuo-temporal context is strongly modulated by the assigned feature dimension for internal item selection. Stressing the spatial position (in contrast to serial position) might therefore aid the separation of memory representations within mental space. This possibility should be addressed in future research.

Note that the proactive repulsion does not necessarily reflect interactions on the memory level. The study by Kang et al. (2011), and especially a study by Scoccia, Cicchini, and Triesch (2013; even though in the orientation domain), strongly suggest that the proactive repulsion already affected the perceptual representation of the S2 during encoding.

With regard to the S1, unlike in Experiment 1s and 2, we observed no statistically meaningful attraction of the S1 towards the S2 direction. Bayesian analysis of the S1 mean bias yielded only inconclusive results (i.e., our data do not present convincing evidence in favor of H1 or H0). Thus, regarding the S1, the data of Experiment 3 should not be interpreted in terms of presence or absence of a retroactive distortion effect.

General discussion

According to the sensory recruitment hypothesis, short-term maintenance and perceptual processing of sensory stimuli rely on the same neural substrate. Therefore, psychophysical phenomena that are known to reflect the activity and organization of sensory cortices during perception should arise equally on the memory level. We tested this prediction by investigating the occurrence of the Direction Illusion under simultaneous and sequential encoding of two motion directions in a working memory task. During concurrent viewing, perceived repulsion between two motion directions constitutes the Direction Illusion, which is attributed to inhibitory interactions between perceptual representations within the sensory cortex (e.g., Benton & Curran, 2003; Blakemore et al., 1970; Blakemore & Tobin, 1972; Gilbert, 1992; Kim & Wilson, 1997; Li et al., 1999; Mather, 1980; Mather & Moulden, 1980; Wachtler et al., 2003; Wenderoth et al., 1986). If working memory maintenance recruits the same neural substrate as visual perception, motion repulsion should occur under concurrent maintenance of two motion directions, regardless of their temporal relationship during encoding. Our results did not support such an account.

We observed both mutual motion repulsion under simultaneous encoding (Experiment 1) and proactive motion repulsion under immediate succession of two RDPs (0-s ISI, Experiments 1 and 2). However, from a temporal separation of 0.5 s onwards, motion repulsion disappeared for two spatially distinct stimuli, even though both were successfully encoded and concurrently maintained in memory (Experiment 2). The disappearance of the repulsion effect clearly suggests that motion repulsion, as we observed it, relied on perceptual, stimulation-specific processes that extended briefly beyond stimulus offset and faded quickly thereafter, possibly reflecting adaptation (e.g., Glasser, Tsui, Pack, & Tadin, 2011; Levinson & Sekuler, 1976; Wenderoth & Wiese, 2008) or residual activity (e.g., Coltheart, 1980; Di Lollo, 1977, 1980; Shioiri & Cavanagh, 1992; Shooner et al., 2010; Sugita et al., 2018). Once this process had faded, motion repulsion no longer occurred. This pattern demonstrates that the transition from a perceptual representation to a consolidated working memory content is accompanied by a change of representational properties. Specifically, while a perceptual representation causes motion repulsion of subsequent input across space, a consolidated memory representation does not.

The lack of a repulsion effect is in contrast to two previous studies that have reported a memory-induced Direction Illusion (Kang et al., 2011; Kang & Choi, 2015). Both studies presented the stimuli to the same foveal position. In line with these studies, Experiment 3 showed that proactive motion repulsion can occur even with long ISIs if both stimuli appear at the same spatial position. The observation of proactive memory-induced motion repulsion between spatially overlapping stimuli in Experiment 3 and the lack of such an effect under spatial separation of the items in Experiment 2 points towards a distinction between spatially local and global mechanisms of motion repulsion. A key characteristic of the Direction Illusion and analogue phenomena with other stimulus materials is that the interactions between concurrently presented stimuli are not bound to spatial overlap of the stimuli (e.g., Ejima & Takahashi, 1985; Kim & Wilson, 1997; Klauke & Wachtler, 2015; Loomis & Nakayama, 1973; Tzvetanov et al., 2006; Westheimer, 1990). However, another distortive phenomenon in the motion domain is the Direction Aftereffect that is caused by adaptation (i.e., after prolonged visual stimulation) to a motion direction (Levinson & Sekuler, 1976). Adaptation to a motion direction causes a shifted perception of subsequently viewed directions. While the effect is similar to the distortion profile of the Direction Illusion, the Direction Aftereffect is retinotopic (Wenderoth & Wiese, 2008). Furthermore, both phenomena have been dissociated on the grounds of behavioral research, suggesting that they take place at different locations in the motion processing stream (Curran, Clifford, & Benton, 2006, 2008; Farrell-Whelan, Wenderoth, & Brooks, 2012; Wiese & Wenderoth, 2007; for an in-depth comparison of both mechanisms in the orientation domain see Schwartz, Hsu, & Dayan, 2007). Saad and Silvanto (2013) have recently demonstrated a functional connection between visual working memory maintenance and adaptation. They showed that a single orientation maintained in working memory strengthens subsequent adaptation if the adapter stimulus has a similar orientation to the working memory content. Based on this finding they proposed that working memory maintenance is accompanied by persistent neural firing that leads to similar neural effects to prolonged visual stimulation, i.e., adaptation. One explanation for the divergent findings of Experiments 2 and 3 would thus be that visual memory maintenance relies on retinotopically specific sustained sensory activation (Pratte & Tong, 2014; Sneve, et al., 2011) that acts selectively on subsequent input that falls onto the same receptive fields (but see Harrison & Bays, 2018, who argue that to-be-memorized representations are transferred from retinotopic sensory areas to other areas briefly after stimulus onset, e.g. within 1s for orientations). In this context it is noteworthy that several other studies have investigated distortions between two sequentially encoded memory items with ISIs longer than 0.5 s, using different stimulus materials (Bae & Luck, 2017; Czoschke et al., 2019; Dubé et al., 2014; Huang & Sekuler, 2010; Rademaker , Bloem, De Weerd, & Sack, 2015; Wildegger, Myers, Humphreys, & Nobre, 2015). While all of these studies observed proactive distortions, they all presented their items to the fovea and thus to the same spatial position. Our results suggest that proactive, memory-based item interactions might be restricted to this special case of spatial overlap between stimuli. This would have implications for the generalizability and functional interpretation of memory-based alterations of stimulus representations.

Recent research presented a highly reliable proactive attraction effect between two successively encoded items. This phenomenon has been termed “Serial Dependence” and has been shown to act upon a variety of different visual features, such as orientation (e.g., Fischer & Whitney, 2014; Fritsche, Mostert, & de Lange, 2017), color (Barbosa & Compte, 2018), motion direction (Czoschke et al., 2019; Fischer et al., 2019), spatial location (e.g., Bliss, Sun, & D’Esposito, 2017), numerosity (Fornaciai & Park, 2018), and faces (Liberman, Fischer, & Whitney, 2014). Crucially, this Serial Dependence effect is typically investigated between consecutive items that belong to subsequent trials (as opposed to the present study in which we investigated distortions between consecutive items within a common memory episode). In a recent publication from our laboratory (Czoschke et al., 2019), we directly contrasted proactive distortion effects of two immediately successive motion directions that were either separated by a demarked “trial border” or belonged to the same trial and had to be maintained concurrently. We found attractive serial dependence across trials and repulsive serial dependence within trials. Both effects differed not only in the direction of the distortion effect but also in the similarity tuning. Attractive serial dependence across trials occurred in a narrow range between highly similar items, with the peak attraction effect occurring at a difference of 17° between the directions. Repulsive serial dependence within trials showed a broader tuning and the strongest distortion effect at a difference of 39° between the directions. These different distortion profiles point towards different underlying mechanisms that have been speculated to be goal-directed (segregating concurrently task relevant material, but integrating previously relevant material, Czoschke et al., 2019) and to rely on different processing stages (Fornaciai & Park, 2019; Pascucci et al., 2019), with attraction requiring an explicit or implicit decision about the inducer stimulus to be subsequently elicited (Pascucci et al., 2019). Our current results add to the evidence for different mechanisms of attractive and repulsive serial dependence, as between-trial attraction has been shown to operate across visual space (Fischer & Whitney, 2014), while our current results suggest that within-trial repulsion of temporally separated stimuli requires spatial overlap.

While the primary focus of our study was on detecting motion repulsion under different encoding conditions, in Experiments 1 and 2 we also observed retroactive attraction. The motion direction of the first encoded stimulus was attracted towards the direction of the second stimulus. S1 attraction is highly interesting for several reasons. First, S1 attraction is certainly surprising under a parallelism account, as the predominant bias seen under concurrent perception of two similar motion directions is repulsive. This further strengthens the notion that the mechanism responsible for motion repulsion was not at work during memory maintenance. Second, the attraction bias did not diminish with increasing ISI. Thus, retroactive attraction is clearly dissociated from proactive repulsion, with the former persisting across long ISIs and the latter disappearing. This asymmetry suggests that maintained and newly encoded items reside in different representational states, i.e., they are characterized by different sets of properties. While the mechanisms behind retroactive attraction are unknown, previous research points towards a retroactive interference effect of ongoing perceptual stimulation. Retroactive attraction of visual working memory stimuli is a common finding in the memory masking literature. In the standard memory-masking paradigm subjects encode only one item (“S1”) into working memory. During the retention interval a mask, i.e., a task-irrelevant distractor from the same feature dimension, is presented. S1 has been found to be attracted by this distractor mask in several studies using a wide range of different stimulus materials including spatial frequency (Nemes, Whitaker, Heron, & McKeefry, 2011), color (Nemes, Parry, Whitaker, & McKeefry, 2012; Schneegans, Spencer, Schöner, Hwang, & Hollingworth, 2014), shape (Ono & Watanabe, 2014), orientation (Lorenc, Sreenivasan, Nee, Vandenbroucke, & D’Esposito, 2018; Rademaker et al., 2015), speed (Nemes, 2013), and motion direction (Seidel Malkinson, Pertzov, & Zohary, 2016). Thus, mere perceptual stimulation during a maintenance period seems sufficient to induce attractive shifts of a memory representation. In this respect, the S2 in our study might have triggered the same mechanisms that are responsible for attraction in memory masking, even though it had to be attended for working memory consolidation.

Our study does not allow conclusions to be drawn about the sites and processes involved in item interactions. However, contrasting the conditions that did and did not show signs of motion repulsion, a parsimonious conclusion would be that different neural codes underlie perceptual representations and working memory storage, at least with respect to the mechanisms responsible for long-range motion repulsion. Our results challenge the view that working memory maintenance relies on sustained recruitment of the same neural populations that are engaged during perceptual encoding (e.g., Serences et al., 2009). The present study thus adds to the literature that points towards differences in the psychophysical characteristics of perceptual and working memory representations (Bloem et al., 2018; Harrison & Bays, 2018). Harrison and Bays (2018) investigated whether the spatial proximity of memory items, which is known to influence fidelity under simultaneous encoding, equally hampers memory under sequential encoding conditions. Bloem et al. (2018) tested whether contrast normalization (i.e., the reduced contrast perception for surrounded vs. isolated stimuli), known to occur between concurrently viewed stimuli, also occurs between concurrently maintained items. Neither study found that concurrent maintenance leads to similar consequences to concurrent viewing.

However, while our results demonstrate that motion repulsion does not occur during concurrent memory maintenance of spatially distinct stimuli, they do not imply that repulsion is specific to perceptual processing (i.e., neural processes during sensory stimulation). There are at least two studies that present strong evidence for repulsive interactions between working memory representations. The previously mentioned study by Kang and Choi (2015) presented two motion directions sequentially to the fovea and, after a brief retention period, subjects had to reproduce both motion directions in a whole-report procedure without knowing the order of report in advance. In their experiment, both items (S1 and S2) repulsed each other, as evident by a repulsion effect for the first reported item of the report sequence. However, the second reported item showed an increased repulsion magnitude. While the distortions measured at the first report position could be attributed to interactions between the maintained S1 and the S2 during S2 presentation, the additional distortion observed for the second reported item seemed to be a result of the prior recall (see also Bae & Luck, 2017, for a similar result with clock-hand directions). In an even more persuasive study, four directions were presented simultaneously for memorization, one of which was subsequently cued (Myers, Chekroud, Stokes, & Nobre, 2018). Crucially, this task required reproducing two of the four presented items, and the cue informed the participants about one of the two recall-targets. While the reproduction of the cued item was not distorted, the remaining uncued items were repulsed from the cued memory representation. Together with our current results, these findings demonstrate that repulsive interactions on the memory level can occur if item representations within working memory are internally activated during retention to prepare for recall, but that they do not occur during mere concurrent maintenance. This pattern of findings supports the view that stimulus-specific sensory activation can be selectively reactivated via top-down signals from higher areas to reinstate sensory representations if needed for the specific task at hand (LaRocque, Riggall, Emrich, & Postle, 2016; Lewis-Peacock, Drysdale, Oberauer, & Postle, 2012; Scimeca et al., 2018; Sprague, Ester, & Serences, 2016; Wheeler, Petersen, & Buckner, 2000; Xu, 2017). The studies by Kang and Choi and Myers et al. show that if two items within working memory are serially activated for report, the first activated item repels the second one. This suggests that activation reinstates a perception-like code that comes with the psychophysical consequences known from perceptual stimulation. Thus, activated memory representations and online perceptual representations indeed might have common neural properties, but, as our results suggest, mere memory maintenance relies on different neural processes that do not share the same characteristics as perceptual code.