Rapid eye movements (saccades) shift objects on the retina 2–3 times a second while we are awake (Rayner, 1998). How does the brain maintain a stable and continuous perception of the external world despite the retinal displacement of objects by saccades? One possibility is that the brain represents objects in world-centered or spatiotopic coordinates (e.g., d’Avossa et al., 2007; Turi & Burr, 2012; Zimmermann et al., 2011). Following each saccade, the shifted retinal image of an object is quickly integrated into this map to maintain a stable visual perception (e.g., Melcher & Morrone, 2003). This theoretical supposition appears intuitive and plausible at first glance, but there is scarce neurophysiological evidence for such a world-centered brain map (e.g., Wurtz, 2008). Moreover, if perception depends on a world-centered brain map that is available at all times, why would people frequently fail to detect significant changes in scenes (i.e., “change blindness”; O’Regan & Noë, 2001; Simons et al., 2000; Simons & Rensink, 2005). An alternative theory is that the brain maintains visual representations in eye-centered or retinotopic coordinates and updates object representations with a neural mechanism known as predictive remapping (e.g., Duhamel et al., 1992).

Predictive remapping refers to the observation that some neurons are predictively activated by a visual stimulus (or the memory trace of a visual stimulus) that would be brought into their receptive fields by an impending saccade. A schematic illustration of predictive remapping is presented in Fig. 1a–b. An apple was initially in the receptive field of Neuron a (Fig. 1a, oculocentric brain map), but it falls into Neuron b’s receptive field following a rightward saccade. To compensate for the retinal displacement, Neuron b becomes active even before the intended saccade occurs (Fig. 1b, blue dashed line). Remapping was first discovered in monkey lateral intraparietal area (LIP; Duhamel et al., 1992), but it has since been observed in the frontal eye fields (FEF; Sommer & Wurtz, 2006; Umeno & Goldberg, 1997), the superior colliculus (Walker et al., 1995), and the early visual cortices (Nakamura & Colby, 2002).

Fig. 1
figure 1

a An illustration showing that saccades shift the retinal locus of an object, and consequently, an object in one neuron’s response field will fall into another neuron’s response field following a saccade. Dashed circles represent the receptive fields of neurons a and b. The currently active neuron and its receptive field are highlighted in black. To reveal the predictive activation of neuron a, one needs to probe its receptive field (i.e., the future retinal locus of the currently attended object) before saccade onset (Rolfs et al., 2011). Earlier studies had mistakenly probed the future receptive field of neuron a, which is irrelevant to predictive remapping (Melcher, 2007; Mathôt & Theeuwes, 2010; cf. Rolfs et al., 2011). b Hypothetical changes in neuronal activity immediately before and following saccades. The corollary discharge of a pending saccade triggers remapping, so the activation of neuron b increases while the activation of neuron a decreases immediately before saccades. The remapping of neuronal activity is not a perfect process. It may not finish by the end of the pending saccade, leaving two spotlights on the attentional map for a brief period (Golomb, 2019). c–d A graphical review of the literature on the reference frame (eye-centered vs. world-centered) of IOR and remapping. IOR is frequently examined with a spatial cueing paradigm. Robust IOR is seen at both the cued spatial location and the cued retinotopic locus (analogous to the receptive field of neuron a in (a), immediately following saccades. IOR is also seen at the future retinal locus of the cue (analogous to the receptive field of neuron b in (a) immediately before saccades (Yan et al., 2016). The data from Abrams and Pratt (2000) were not included in this graphical review because the gaze was shifted by smooth pursuits instead of saccades. The results of Mathôt and Theeuwes (2010) were from the re-analysis by Hilchey et al. (2012). The post-saccade probing time of Sapir et al. (2004; healthy controls) was our best estimate. As is clear from the data summarized in (c–d), following saccades, IOR quickly dissipates at both the cued spatial location and the cued retinal locus. IOR also quickly builds up at the future retinal locus of the cue immediately before saccades, showing that IOR is predictively remapped to maintain at the cued spatial location (Yan et al., 2016)

Full size image

The neural substrates known to support remapping (e.g., SC, FEF, and LIP) are also involved in the control of spatial attention (e.g., Awh et al., 2006; Schall, 2004), and there is neurophysiological evidence that only attended, or salient stimuli are remapped (e.g., Gottlieb et al., 1998). Not surprisingly, the field has seen an increasing effort in examining whether predictive remapping is the neural mechanism underlying spatial attention (e.g., Jonikaitis et al., 2013; Mathôt & Theeuwes, 2010; Rolfs et al., 2011; Yan et al., 2016).Footnote 1 In a behavioral study, Rolfs et al. (2011) probed presaccadic remapping of attention in a double-step saccade task. Given that two successive saccades were executed sequentially, attention was shifted to both saccade targets even before the execution of the first saccade (e.g., Godijn & Theeuwes, 2003). The authors observed reliable attentional benefits at the future retinal locus of the second saccade target (analogous to the receptive field of Neuron b in Fig. 1a) just before the onset of the first saccade (see also Jonikaitis et al., 2013; Szinte et al., 2018), suggesting that attention is predictively remapped to keep track of attended objects across saccades (Cavanagh et al., 2010; Rolfs et al., 2011).Footnote 2

Attention is either voluntarily controlled or driven by external events or past selection history (Awh et al., 2012; Theeuwes, 2018). Rolfs et al. (2011) and follow-up studies (Arkesteijn et al., 2019; Jonikaitis et al., 2013) broadly fall into the category of voluntary attentional control. For stimulus-driven attention, the study of its connection to eye-centered brain maps dates back to the early 1980s, when Posner and colleagues examined attentional facilitation and inhibition with a spatial cueing paradigm. In a cueing task, a peripheral target that requires a speeded response is preceded by an uninformative peripheral onset cue. Responses to targets at the cued location are initially facilitated due to attentional capture by the cue. The facilitatory effect quickly turns into an inhibitory effect and slows down the responses to targets at the cued location. The latter inhibitory effect discourages attention from returning to previously inspected locations and thus was given the name “inhibition of return” (IOR; Posner & Cohen, 1984; Posner et al., 1985). Cohen (1981) discovered that the early attentional facilitation was coded in eye-centered coordinates (for similar findings of voluntary controlled attention, see Golomb et al., 2008). However, the inhibitory aftereffect of attention was found to reside in world-centered coordinates (Posner & Cohen, 1984; see also Abrams & Pratt, 2000; Maylor & Hockey, 1985). This observation contradicts the fact that IOR is closely linked to the oculomotor system (for a recent review of the neural basis of IOR, see Satel et al., 2019). Recent studies have revealed robust IOR effects in both eye-centered and world-centered coordinates immediately following saccades (Hilchey et al., 2014; Krüger & Hunt, 2013; Mathôt & Theeuwes, 2010; Pertzov et al., 2010; Yan et al., 2016; but see Malevich et al., 2020; see Fig. 1c–d for a graphical review), indicating that IOR is natively coded in eye-centered brain maps but (predictively) remaps to maintain at spatially relevant locations. Using the presaccade probing technique proposed by Rolfs et al. (2011), Yan et al. (2016) provided direct evidence of predictive remapping for IOR (see Fig. 1c).

The attentional effects observed at the future retinal locus of an attended location suggest that predictive remapping is likely the neural mechanism underlying spatial attention (e.g., Rolfs et al., 2011; Yan et al., 2016). However, it is unlikely that the brain relies solely on remapping to maintain spatial attention. Attention is also modulated by top-down signals, such as task relevance, selection history, and reward. For remapping to have functional significance in updating spatial attention, remapped activity should persist for a while on eye-centered brain maps, so other slower neural mechanisms have time to take over, if needed. To the best of our knowledge, there remains a lack of empirical evidence on this matter.Footnote 3 The present study was set out to fill this gap by examining the “lifetime” of IOR at the remapped retinal locus of an attended location.

Method

The remapping theory predicts that, around the time of saccades, neurons encoding the future retinal locus of an attended object will be predictively activated. As noted, to maintain attention at the appropriate spatial location, the remapped neuronal activity should persist for at least a few hundred milliseconds on eye-centered brain maps. The Posner cueing paradigm (Posner, 1980) was adopted to verify this prediction in a series of three experimental tasks. In the first task, a saccade was executed following the offset of a peripheral cue, and a probe later appeared at the cued location or a control location. This task helped reveal whether IOR is maintained at the cued spatial location (“nonreturn-saccade task”; see Fig. 2a for an illustration; cf. Hilchey et al., 2012; Maylor & Hockey, 1985). In the second task, the participant also made a saccade following the presentation of a peripheral cue, but they quickly return the gaze to the initial fixation (“return-saccade task”; see Fig. 2b for an illustration). The first saccade will evoke remapping and shift IOR to the future retinal locus of the cue (Yan et al., 2016), which is in the opposite direction of the saccade (see Fig. 1a). Following the return saccade, however, the remapped retinotopic locus is shifted to a new spatial location that has never been stimulated by the cue. By probing this location, one can easily verify whether remapping has left an IOR trace on eye-centered brain maps. In the third task (“no-remapping task”; see Fig. 2c), we aimed to rule out the possibility that the IOR trace left by remapping was an artifact created by the spillover of IOR from the cued spatial location.

Fig. 2
figure 2

Task procedures for the nonreturn-saccade (a), return-saccade (b), and no-remapping (c) tasks. A fixation dot was first displayed at the center of the display for 500 ms, following by a peripheral onset cue (empty green box) for 100 ms. In the nonreturn-saccade task (A), the fixation dot displaced for 7° to the left or the right side of the screen, and the participants moved their eyes to follow the dot. After the eyes had reached the new fixation, a probe was presented at the cued spatial location or a distance-matched control location (see the lower panel). In the return-saccade task (B), the procedure was similar to that of the nonreturn-saccade task, except the fixation dot shifted back to the original fixation (center of the display). In addition to the cued location and its control location, the probe could also appear at the retinal locus activated by predictive remapping (the remapped retinal locus) or a distance-matched control location (see the lower panel). In the no-remapping task (c), the participants maintained fixation all time. The spatial relation between the probe, the cue, and the fixation was the same as the return-saccade task. d Saccade latencies for the cued spatial location and the distance-matched control location in the nonreturnsaccade task. e Saccade latencies at the cued and control location following a return saccade; note that the cued spatial location overlaps with the cued retinal locus (Spatial + Retinal), whereas the Remapped retinal locus was never stimulated by the cue. f Saccade latencies at the cued location and three control locations in the no-remapping task. The light grey dots connected by lines were saccade latencies from individual participants. The dark grey dots connected by lines represent the first author. Error bars represent ±1 SEM. ** p < .01, *** p < .001

Full size image

Participants

Our previous work has revealed robust IOR effects at the cued retinal locus and the cued spatial location following a saccade, with only 12 participants (He et al., 2015; Hilchey et al., 2012). The effect size of IOR in those studies ranged from 0.66 to 2.53 (dz). To reproduce the weakest IOR effect reported in these studies, 16 participants are required to reach a power of 0.8 at a significance level of 0.05 (as suggested by Cohen, 1988; determined with the power analysis tool G*Power; Faul et al., 2007). Eighteen volunteers were recruited from Hangzhou Normal University to participant in both the nonreturn-saccade and return-saccade tasks. Two of them were excluded from the analysis because they failed to fulfill the tasks. These two tasks were counterbalanced across participants (i.e., half of the participants performed the nonreturn-saccade first, whereas the other half performed the return-saccade task first). Twelve volunteers participated in the no-remapping task. Five of them (including the first author, C.Y.) also participated in nonreturn-saccade and return-saccade tasks before performing the no-remapping task. The results presented here involve 23 participants (11 females, Mage = 22.9 years, SD = 2.61).

The research protocol of the present study was approved by the institutional review board of Center for Cognition and Brain Disorders at Hangzhou Normal University. Written informed consent was obtained from all participants. All participants were naive to the purpose of the tasks presented here, except for the first author on this paper (C.Y.). They all reported normal or corrected-to-normal visual acuity.

Apparatus

Participants were tested in a quiet and dimly lit laboratory. Visual stimuli were presented against a black ground (1.95 cd/m2) on a 21-inch NESO FS210A CRT monitor. The monitor’s visible area measured 33° × 24.7°; the viewing distance was maintained at 64 cm with a chin rest. Stimulus presentation was controlled by Python scripts, running on a Windows PC. The left eye were monitored online with a desktop mounted eye-tracker system (EyeLink® 1000; SR Research, Mississauga, Canada), sampling at 500 Hz. The tracking accuracy of this eye tracker was reported to be 0.2° or better. Saccades were detected online, with a velocity threshold of 30°/s and an acceleration threshold of 8000°/s2.

Task procedures and design

Nonreturn-saccade task

The nonreturn-saccade task was carried out to reveal IOR at the cued spatial location immediately following saccades. The sequence of events in a sample trial is illustrated in Fig. 2a. A drift-check was first performed to examine the accuracy of the eye tracker. During the drift-check, a fixation dot was presented at the center of the screen, and the participant pressed the space bar on the keyboard to confirm his/her gaze on the fixation dot. If the tracker reported a large gaze error (>2°), the experimenter could pause the task to recalibrate the tracker. A successful drift-check was followed by the presentation of a gray fixation dot (diameter: 0.5°; Weber contrast: 8.69) for 500 ms. Then, a cue (a green square, 1° × 1°; Weber contrast: 59.51) was flashed on the screen for 100 ms, at one of four possible locations on an imaginary circle (radius = 7°) centered at the fixation dot (see Fig. 2a, bottom panel). Following a 300-ms delay, the fixation dot jumped 7° to a new location, in the same hemifield (left or right) as the cue. The participant was instructed to quickly shift the gaze to follow the fixation dot. One hundred milliseconds after the required saccade had landed on the new fixation, a probing dot (diameter: 0.5°; Weber contrast: 8.69) appeared at either the cued location or a distance-matched control location in the opposite visual field (upper or lower). The participant was instructed to look at the probe as quickly as possible. After a random intertrial interval between 1,000 and 1,500 ms, the next trial began.

Return-saccade task

The return-saccade task was designed to reveal the IOR trace left by remapping on eye-centered brain maps. The setup of this task was the same as the nonreturn-saccade task except that, following the first saccade to the peripheral, the participant was required to make a second saccade to return gaze to the initial fixation (see Fig. 2b, upper panel). The probe was presented 100 ms after the gaze had returned. It is worth noting that, following the return saccade, the cued spatial location coincided with the cued retinal locus again. However, the remapped retinal locus of the cue was shifted to a spatial location that had never been occupied by the cue. Specifically, we expected the first saccade to trigger remapping, which should leave an IOR trace at the future retinal locus of the cue (i.e., the remapped retinal locus). Following the return saccade, the remapped retinal locus was separated from the cued spatial location and occupied a new spatial location (7° left to the cue location if the first saccade was directed rightward, and vice versa) after the return saccade. The probe appeared at the cued location or the remapped retinal locus, or their distance-matched control locations in the opposite hemifield (see Fig. 2b, bottom panel). An IOR effect at the remapped retinal locus would suggest that the IOR trace left by remapping can last for at least a few hundred milliseconds on eye-centered brain maps.

No-remapping task

This task was carried out to examine whether the IOR effect at the remapped retinal locus, if any, was an artifact caused by the spillover of IOR at the cued location (He et al., 2015; Malevich et al., 2020). The display setup was the same as that in the return-saccade task, except that no saccade was required before the onset of the probe. The stimulus-onset asynchrony (SOA) between the cue and the probe was 1,000 ms, close to the mean SOA in the return-saccade task (M = 1,026 ms; SD = 57).

In the nonreturn-saccade task, the participants each completed eight practice trials and 48 testing trials. In the return-saccade and no-remapping tasks, the participants completed 15 practice trials and two blocks of 48 testing trials. Trials were flagged if the gaze deviated more than 2° from the fixation dot when maintaining fixation was required or if the participant failed to respond to the probe within 500 ms. In the nonreturn-saccade and return-saccade tasks, trials were also flagged if the participant missed the displaced fixation dot for more than 2°. All flagged trials were retested in random order until all trials were completed. Thus, there were 24 successfully completed trials for each probe location in all tasks.

Results

In the present study, 38 (0.8%) trials were recycled because the participants failed to maintain fixation, 69 (1.5%) trials were recycled because the participants failed to respond to the probe within 500 ms, 1,009 (21.4%) trials were recycled because participants missed the displaced fixation dot or the probe for more than 2°. Only successfully completed trials entered the analyses. Trials with extremely short (<70 ms) saccade latencies were excluded from the analyses. The excluded trials accounted for 5% of the trials in the return-saccade task and less than 1% in the nonreturn-saccade and the no-remapping tasks. The number of excluded trials did not differ across conditions, with all ps > 0.708.

Nonreturn-saccade task

This task examined whether the present experimental setup was sensitive enough to reveal IOR effects in world-centered coordinates following saccades. IOR was measured with the latency difference between saccades to probes at the cued spatial location and the control location (see Fig. 2a, bottom panel). The analysis revealed an IOR effect of 12 ms (SD = 12) at the cued location, t(15) = 3.960, p = .001, dz = 0.497. In line with Yan et al. (2016; see also Hilchey et al., 2012; Pertzov et al., 2010), remapping must have occurred before or during the saccade that followed the fixation displacement.

Return-saccade task

In this task, the participant made two saccades before the probe. Once the eyes had returned to the initial fixation, the remapped retinal locus was dissociated from the cued spatial location, which coincided with the cued retinal locus (spatial + retinal; see Fig. 2b, bottom panel). Probing the remapped retinal locus would reveal if the remapped IOR persists on eye-centered brain maps.

A 2 × 2 repeated-measures analysis of variance (RM ANOVA) was performed, with variables cueing (cued vs. control) and coordinates (spatial + retinal vs. remapped retinal). A significant main effect of cueing was observed, F(1, 15) = 20.308, p < .001, η2 = 0.575, showing that the saccade latencies were overall longer for probes that appeared on the same side as the cue (relative to the saccade vector). Planned contrasts revealed significant IOR effects at both the cued location (spatial + retinal; M = 8 ms, SD = 7), t(15) = 4.486, p < .001, dz = 0.530, and the remapped retinal locus (M = 5 ms, SD = 6), t(15) = 3.489, p = .003, dz = 0.467. There was also a significant two-way interaction, F(1,15) = 4.665, p = .047, η2 = 0.237, suggesting the IOR effect was weaker at the remapped retinal locus than the cued location. At the remapped retinal locus, the IOR effect was rather robust, observed in 12 out of the 16 participants (see Fig. 2e). Analysis of the eye movement data revealed a 430-ms (SD = 30) delay between the termination of the first saccade and the onset of the probe. These results clearly show that remapping indeed leaves an IOR trace on eye-centered brain maps, and this residual IOR trace can last for at least 430 ms.

No-remapping task

IOR is known to spill over to its adjacent spatial locations (Bennett & Pratt, 2001; Wang et al., 2018). As shown in Fig. 2b, compared with its control location, the remapped retinal locus is closer to the cued location, raising the possibility that the longer saccade latencies at the remapped retinal locus are an artifact created by the spillover of IOR from the cued location. The no-remapping task was carried out to rule out this possibility.

An ANOVA on the saccade latencies revealed a significant effect for probe location, F(3, 30) = 10.126, p < .001, η2 = 0.479. As shown in Fig. 2f, the longest saccade latencies were observed at the cued location, all ts > 3.893, all ps < .003; no saccade latency difference was observed among the three control locations, all ts < 0.325, all ps > .751. These results clearly show that the IOR spillover did not confound the RTs observed at the other three possible probing locations in the present experimental setup. Otherwise, shorter saccade latencies would be observed at Control 3 in Fig. 2c (furthest from the cue location) compared with the other two control locations (Controls 1 and 2 in Fig. 2c).

We have also analyzed our data without the first author (C.Y.). The overall pattern of results did not change, except that two-way interaction between cueing and coordinates in the return-saccade task became marginal (see Supplementary Results). This marginal interaction, nevertheless, did not compromise of finding of IOR at the remapped retinal locus.

Discussion

How does the brain maintain attention at spatially relevant location despite the retinal displacement of objects across saccades? By probing attention at spatially or retinotopically relevant locations immediately before or following saccades, previous studies have suggested predictive remapping as a possible neural mechanism for maintaining spatial attention (e.g., Pertzov et al., 2010; Rolfs et al., 2011; Szinte et al., 2018; Yan et al., 2016). If remapped activity on eye-centered brain maps is recruited to sustain spatial attention, the remapped activity should persist for a while immediately following saccades. The present study examined the “lifetime” of predictively remapped activity on eye-centered brain maps with an attentional aftereffect known as IOR. In the primary experimental task (the “return-saccade” task), a peripheral onset cue was first presented to evoke IOR. The participants made a saccade to the peripheral and then quickly returned to the initial fixation. The first saccade would trigger predictive remapping and remap IOR to the future retinal locus of the cue (Yan et al., 2016); the second saccade brought this remapped retinal locus to a spatial location that had never been stimulated by the cue (see Fig. 2b). At the remapped retinal locus, probes revealed an IOR effect about 430 ms following the saccade that triggered remapping, showing that predictively remapped activity on eye-centered brain maps persists immediately following saccade. This novel finding has significant theoretical implications. Remapping is likely the fastest neural mechanism the brain recruits to update spatial attention across saccades. However, it is unlikely the only mechanism the brain relies on to maintain spatial attention. Top-down signals such as task relevance and expectation are known to modulate sensory processing in a spatially specific fashion through long-range corticocortical projections (e.g., Kastner et al., 1999; Zhang et al., 2014). The relatively long-lasting activity at the remapped retinal locus would help sustain attention immediately following saccades, so the brain has time to recruit slower neural mechanisms to take over if necessary.

Golomb (2019) proposed a “two-spotlight” theory to describe the perisaccadic updating of spatial attention. When a saccade triggers remapping, attention builds up at the future retinal locus and ramps down at the current retinal locus. These two processes do not necessarily occur synchronously, such that the former process is more rapid (see Yan et al., 2016; Rolfs et al., 2011) and the latter is slower (Golomb et al., 2010; Golomb et al., 2008; Golomb & Kanwisher, 2012; He et al., 2015; see also Fig. 1d). Consequently, attention may be maintained in both eye-centered and world-centered coordinates immediately following saccades. The fast buildup of attention at the future retinal locus is likely driven by the corollary discharge of an upcoming saccade (Sommer & Wurtz, 2006). The slow decaying trace at the previously attended retinal locus probably reflects the internal dynamics of eye-centered brain maps (for a discussion of other possibilities, see Golomb, 2019). The postsaccade attentional effect reported by Golomb (2010) and Golomb et al., (2008) was fundamentally different from the attentional effect (IOR) observed in the present study. The former was observed at a retinal locus previously stimulated by an attentional cue (analogous to the response field of Neuron a in Fig. 1a). The latter effect was triggered by predictive remapping, and the relevant retinal locus was never stimulated by external events (analogous to the response field of Neuron b in Fig. 1a). Because both attentional traces can persist for a few hundred milliseconds, saccades may lead to a messy attentional landscape briefly and cause perceptual errors (e.g., Dowd & Golomb, 2019; Szinte et al., 2016). Our perception of the external visual world is not seriously impacted, likely because the lingering retinotopic traces do not win the competition for selection on attention maps.

In line with the theory that IOR evolved to bias orienting toward new locations (Posner & Cohen, 1984), Danziger et al. (1998) showed that IOR is behaviorally measurable at three previously cued locations. Similar findings have been reported in eye-tracking and manual pointing studies, where responses to probes at previously examined locations are impaired (e.g., Klein & MacInnes, 1999; Thomas et al., 2006). In the return-saccade task, IOR was observed at the remapped retinal locus, which no longer coincides with the cued location, leaving people wondering whether the lingering IOR effect left by remapping undermines IOR’s ecological function in biasing attention toward new locations. We believe this is less a concern. Firstly, the IOR effect observed at the remapped retinal locus was only 5-ms, and so it is unlikely to survive additional saccades. Secondly, although previous studies have shown that IOR effects persist in retinotopic coordinates following saccades (see Fig. 1d), they are much weaker compared with coexisting IOR effects in spatiotopic coordinates (e.g., He et al., 2015; Hilchey et al., 2012; Pertzov et al., 2010; see also the two-way interaction in the return-saccade task). So, the retinotopic trace of IOR left by remapping is unlikely to dominate the attentional map and seriously disrupt orienting.