Transsaccadic processing: stability, integration, and the potential role of remapping
While our frequent saccades allow us to sample the complex visual environment in a highly efficient manner, they also raise certain challenges for interpreting and acting upon visual input. In the present, selective review, we discuss key findings from the domains of cognitive psychology, visual perception, and neuroscience concerning two such challenges: (1) maintaining the phenomenal experience of visual stability despite our rapidly shifting gaze, and (2) integrating visual information across discrete fixations. In the first two sections of the article, we focus primarily on behavioral findings. Next, we examine the possibility that a neural phenomenon known as predictive remapping may provide an explanation for aspects of transsaccadic processing. In this section of the article, we delineate and critically evaluate multiple proposals about the potential role of predictive remapping in light of both theoretical principles and empirical findings.
KeywordsEye movements Predictive remapping Saccades Transsaccadic integration Transsaccadic perception Transsaccadic processing Visual stability
During waking hours, our eyes dart from one location to the next about three times every second (Schiller, 1998). These fast, darting movements called saccades take place in alternating sequence with short periods known as fixations when the eyes remain relatively still. It is during fixations that we obtain useful information from the visual world, viewing a sort of ‘snapshot’ of the scene from the current vantage point (Neisser, 1967). Because only the fovea, a central region of the retina spanning about 2 ° of visual angle, is specialized for high acuity processing, we acquire detailed visual information from only a small part of the world during any given fixation (Jones & Higgins, 1947).
Saccades allow the proximal stimulus of vision, the retinal image, to be selected quickly and efficiently based on processing requirements. Both the locations we fixate and the durations of our fixations reflect online selection of critical information from the visual environment (for reviews, see Rayner, 1998, 2009). This seems an enormously beneficial attribute of the system. Our continually shifting gaze, however, also raises potential challenges for interpreting and acting upon the information acquired.
First, the mobility of the eye raises the question of how we plan actions based on visual position information. Because our eyes move frequently about, a given retinal location does not map onto a unique location in space relative to the arms, legs, or other motor effectors. Therefore, some means of tracking the eye’s current position relative to the effectors seems important for calculating the appropriate movement vector at any given moment. While this is a critical issue, it lies somewhat beyond the scope of the present article, which focuses more narrowly on how we perceive and represent the visual world across saccades (for reviews focusing on transsaccadic updating for motor control, see Klier & Angelaki, 2008; Medendorp, 2011.) In particular, in the present article we will review critical findings on (1) phenomenal stability, and (2) memory and integration across eye movements, addressing both behavioral phenomena and potential neural underpinnings.
Research on phenomenal stability stems from the observation that, while visual input is disjointed and discontinuous—comprised of a stream of shifting retinal images interleaved with smears of motion—we do not experience perceptual disruptions each time we move our eyes. In this article, we examine accounts of how such phenomenal stability may be achieved. We then discuss findings on visual memory and integration across saccades, asking what we retain from prior fixations and how this information may be used in subsequent processing. Finally, we address a neural phenomenon called predictive remapping, which has been proposed as an explanation for some of the computations involved in transsaccadic processing. In predictive remapping, cells in some retinotopic areas of the visual system appear to become transiently sensitive, just before a saccade, to a region outside their classic receptive fields. In particular, on the standard account, such cells respond presaccadically to the region of the scene that will lie within their receptive fields after the saccade has been executed. In the final section of this article, we will set forth and evaluate multiple proposals about the potential contributions of remapping to transsaccadic processing.
The term phenomenal stability simply refers to the fact that no sensation of movement or displacement accompanies our eye movements even though they entail (often quite dramatic) shifts of the image on the retina. Scientists have considered how this may be achieved at least since the 11th century, when Persian scholar Alhazen speculated about how shifts of the eye might be distinguished from the external movements of objects (Alhazen, 1083 as cited in Melcher, 2011; for historical reviews of this topic, see Grüsser, 1995; Bridgeman, 2007).
The question of stability may be divided into two parts (Wurtz, 2008). Why do we not (1) perceive the motion of the retinal image during the saccade, and 2) experience a ‘jump’ or displacement of the scene from the presaccadic to the postsaccadic fixation location? These will be addressed, in turn, in the following sections of the article. 1
During saccades, the image of the world projected onto the retina sweeps across its surface at hundreds of degrees of visual angle per second (Rayner, 1998). We might expect that this would trigger strong and distracting percepts of motion each time we moved our eyes. However, this is not the case: our frequent saccades go unperceived. This phenomenon is known as saccadic suppression (for detailed reviews, see Matin, 1974; Volkmann, 1986; for a more recent review focused on neural underpinnings, see Wurtz, 2008).2
Saccadic suppression first emerged in the literature at the turn of the 20th century when Erdmann and Dodge (1898) noted that we cannot see our own eye movements in the mirror and that experimental subjects are unable to perceive letters presented during saccades. It is not simply that the duration of a saccade is too short to allow for visual perception; we can, after all, easily see the eye movements of others.
Early scholars proposed two main possible explanations for the phenomenon. First, the pattern of retinal stimulation that occurs around the time of the saccade may, in itself, tend to prevent us from perceiving the movement of the eyes (Woodworth, 1906; see also Dodge, 1900). Second, visual processing may be actively and fleetingly suppressed in the brain when an extraretinal signal indicates that an eye movement occurs. Holt (1903), a proponent of this latter position, claimed that the visual system is subject to ‘central anesthesia’ during saccades. Extraretinal suppression may be triggered by a signal known as corollary discharge. Corollary discharge (to be discussed in detail below) is a purported copy of the motor command that drives the eye movement. According to this theory, corollary discharge is issued to perceptual centers in the brain where it informs the analysis of incoming visual information.3
Research conducted in the intervening decades demonstrated that aspects of the retinal stimulus before, during, and after the saccade do indeed play a critical role in saccadic suppression. Some further evidence also suggests, however, that a partial and perhaps selective suppression of neural activity may also contribute to this perceptual effect.
In order to distinguish effects of visual input from those of the eye movement itself, several experiments have used a control condition in which the stimulus is artificially shifted, mimicking the retinal effects of performing a saccade, while subjects maintain steady fixation. Findings using this technique, which we will refer to as a ‘simulated saccade’ paradigm, show that retinal input consistent with a saccade is sufficient to produce significant suppression, even in the absence of a true eye movement (see, e.g., Brooks & Fuchs, 1975; Diamond, Ross, & Morrone, 2000; MacKay, 1970; Noda & Adey, 1974; Thiele, Henning, Kubischik, & Hoffmann, 2002; Wurtz, 1969). As will be discussed below, however, a subset of these studies (e.g., Diamond et al., 2000; Thiele et al., 2002) found that some additional suppression appears to result from the eye movement itself.
With regard to visual input, at least two distinct factors may be important: (1) masking due to presaccadic and postsaccadic retinal images (see Breitmeyer & Ganz, 1976 for a functional discussion of saccadic masking), and (2) the blurring of the stimulus during the saccade. Matin, Clymer, and Matin (1972) demonstrated the importance of backward masking by extinguishing the stimulus, a vertical bar of light against a dark background, at variable moments just before or just after postsaccadic refixation. When the light disappeared at the end of the eye movement, subjects clearly perceived the horizontal smear induced by the saccade. When the light remained on for a sufficient period after refixation, however, backward masking eliminated any percept of the saccadic blur, so that only the crisp, postsaccadic image of the vertical bar could be perceived. In subsequent experiments, Campbell and Wurtz (1978) and Brooks, Impelman, and Lum (1981) obtained consistent findings and also found that forward masking from the presaccadic image contributes to suppression.
Several lines of research also suggest that stimulus blurring helps to suppress visual perception during eye movements. First, in a seminal early study, Volkmann (1962) found that when lights were flashed very briefly during a saccade, thus substantially reducing stimulus blurring, detection rates were only about half a log unit reduced when compared with fixation. Notably, the magnitude of this suppression was much lower than would be required to account for our near-perfect, real-world perceptual suppression, given that very bright stimuli are often present in the environment. Furthermore, Uttal and Smith (1968) found that letters presented during a saccade could be perceived and identified with high fidelity provided they were also composed of briefly flashed lights. A number of further studies using different experimental paradigms also provided support for the notion that blurring contributes to suppression (e.g., Mitrani, Mateeff, & Yakimoff, 1970; Mitrani, Mateeff, & Yakimoff ,1971).
It seems plausible that, in real-world conditions, blurring may aid suppression at multiple levels. First, it clearly disrupts the contours of the scene. Thus devoid of recognizable objects, the intrasaccadic stimulus may become less salient to the viewer, potentially accounting for a part of the effect. Second, a blurred intrasaccadic image may make a particularly effective target for masking (Ibbotson & Cloherty, 2009). Third, as Matin (1974) pointed out, if we consider a single light against a dark background, more photons will fall on a given region of the retina within a temporal integration window during a fixation than during a saccade, when the same amount of energy will be spatially diffused across the moving retina. As Matin also noted, the situation becomes more complicated when we consider the capacity for spatial (in addition to temporal) integration of retinal stimulation. Nonetheless, this dimming may contribute to sensitivity differences between fixation and saccade conditions when stimuli are presented for all but the shortest durations. Finally, retinal blurring selectively removes high spatial-frequency edges from the image (Morgan, 1994). As will be discussed below, this may complement an extraretinal mechanism that preferentially suppresses low spatial frequency components of the scene.
Some findings from the neuroscience literature suggest that some neural suppression of visual processing occurs during saccades (for more detailed discussions of this topic, see Ibbotson & Krekelberg, 2011; Wurtz, 2008; Wurtz, McAlonan, Cavanaugh, & Berman, 2011). For example, Thiele et al. (2002) compared neural responses in medial temporal (MT) and medial superior temporal (MST) regions of the brains of monkeys during saccades and simulated saccades. Within these areas, important for motion processing, one population of cells responded vigorously to stimulus motion during a simulated saccade, when the eye remained still and the stimulus moved, but not during actual saccades, perhaps suggesting the operation of an active, suppressive mechanism. In addition, Paus, Marrett, Worsley, and Evans (1995) found that blood flow in humans was somewhat suppressed in the striate cortex (and some other areas) during saccades completed in darkness, thus providing some evidence of active neural suppression during saccades.
One early finding that seemed to implicate an extraretinal contribution to the perceptual phenomenon of suppression was that stimuli appearing somewhere on the order of 50 ms before saccade initiation are suppressed (Duffy & Lombroso, 1968; Latour, 1962; Volkmann, Schick, & Riggs, 1968; Zuber & Stark, 1966). While saccadic retinal changes begin to unfold only once the eye has started to move, a corollary discharge signal of an impending movement may be sent to perceptual centers during the preparation for the movement, thus making this signal seem a natural candidate to explain these early effects (Wurtz, 2008). MacKay (1970), however, provided evidence that corollary discharge may not be necessary for early suppression. He found that, even during simulated saccades, when the eyes remained at a fixed location while the stimulus shifted in a saccadic fashion, presaccadic stimuli were suppressed. Therefore, a saccadic pattern of visual stimulation alone, seemingly via backward masking, may be sufficient to induce this effect.
Some additional evidence, however, has pointed to a role for a saccade-specific, extraretinal mechanism in perceptual suppression. First, the results of Volkmann (1962) discussed above suggest that modest suppression persists when visual sources of inhibition are minimized. In addition, Riggs, Merton, and Morton (1974) eliminated masking and blur by inducing phosphenes with electrical stimulation in a darkened room and found that visual sensitivity during the saccade was nonetheless significantly suppressed. Furthermore, Kennard, Hartmann, Kraft, and Boshes (1970) found that some afterimages were, according to subjects’ self-reports, perceptually suppressed during saccades (but see García-Pérez & Peli, 2011, for an argument that these findings do not provide persuasive support for perceptual suppression due to an active mechanism).
Relatively recently, some researchers have proposed that an active suppressive mechanism may preferentially target a particular visual processing stream—the magnocelluar pathway. The magnocellular pathway specializes in processing luminance (Gegenfurtner, 2003), low spatial frequencies, and motion. Conversely, the parvocelluar pathway specializes in red-green chromaticity-modulated contrast and high spatial frequency processing (Maunsell & Newsome, 1987). Diamond et al. (2000) conducted an experiment in which subjects performed saccades or viewed simulated saccades while low frequency gratings modulated either by luminance or chromaticity appeared briefly before them. During real saccades, contrast sensitivity for luminance-modulated gratings (thought to be the specialization of the magnocellular pathway) was suppressed by a factor of about 10. In contrast, only very minimal suppression obtained during simulated saccades. Notably, subjects did not show any discernible saccadic suppression for the red-green chromaticity-modulated bars (thought to be the specialization of the parvocelluar pathway). Several additional studies have produced similar findings (Burr, Holt, Johnstone, & Ross, 1982; Burr, Morrone, & Ross, 1994; Ross, Burr, & Morrone, 1996; see also Kleiser, Seitz, & Krekelberg, 2004, for consistent findings obtained using fMRI).
Furthermore, Uchikawa and Sato (1995) reported that subjects’ spectral sensitivity (i.e., sensitivity to lights along the color spectrum) during real saccades, unlike during simulated saccades, was consistent with preferential magnocellular suppression. Based on findings of this kind, Morgan (1994), among others, speculated that selective magnocelluar pathway suppression represents an elegant solution to the challenge of visual stimulation during the saccade, selectively inhibiting precisely the sort of low spatial frequency information that is relatively robust to natural suppression via blurring.
However, some have questioned whether such interpretations are justified. In particular, Castet, Jeanjean, and Masson (2001) proposed an alternative hypothesis to findings suggesting magnocellular suppression, suggesting that mechanical aspects of how photons fall upon the photoreceptors during saccades may explain these effects (see also Richards, 1969, for a similar, mechanical account; see Ross, Morrone, Goldberg, and Burr, 2001, for a response to Castet et al., 2001). In addition, García-Pérez and Peli (2011) reported that the psychometric functions for detecting low spatial frequency stimuli presented briefly either only during the saccade or during a simulated saccade were quite similar which, they argue, provides evidence against substantial involvement of an active suppressive mechanism (see also Castet & Masson, 2000).
An additional extraretinal contribution to perceptual suppression may, in fact, operate during the postsaccadic fixation and involve the heightening rather than the reduction of neural activity. Visual sensitivity may actually be enhanced, relative to a mid-fixation baseline, shortly after a fixation begins. This, Ibbotson and colleagues suggested, could serve to ‘strengthen’ the postsaccadic image and, in turn, to facilitate backward masking of the blurred saccadic stimulus (e.g., Ibbotson & Cloherty, 2009; Ibbotson & Krekelberg, 2011; Ibbotson, Price, Crowder, Ono, & Mustari, 2007).
A final point to consider is that evidence from aftereffects suggests that substantial unconscious processing of visual stimuli can occur during saccades. Watson and Krekelberg (2009) discovered that even undetected stimuli presented during saccades could drive shape contrast aftereffects. For example, if a horizontal bar were presented during the eye movement, a circle presented upon refixation tended to appear as a vertically elongated ellipse. Because the effect occurs even when the adaptor and target do not retinally overlap, it is thought to arise relatively late in visual processing, perhaps at the level of shape-tuned cell populations in the inferotemporal cortex (Suzuki & Cavanagh, 1998). This suggests that, even when we are unaware of intrasaccadic stimuli, they may nonetheless receive significant visual processing.
In summary, then, the evidence demonstrates that the visual system does not become fleetingly insensitive to external stimulation each time we move our eyes. This is evident both from the fact that certain kinds of stimuli are consciously perceived, provided retinal conditions are right, and from the finding that even those images that are not subject to conscious perception may nonetheless receive substantial visual processing. Indeed, taken together, the evidence suggests that visual factors are likely the primary drivers of perceptual suppression during saccades, since the magnitude of suppression is relatively small when these factors are minimized or eliminated (Volkmann, 1986; Wurtz, 2008). Nonetheless, there is evidence of suppressed visual processing in the brain during saccades. Furthermore, extraretinal factors may complement retinal suppression by selectively suppressing motion signals and low spatial frequency components during the saccade and, perhaps, by bolstering backward masking upon refixation.
Although the percept of motion itself is suppressed during a saccade, the fact that objects suddenly appear at quite different retinal locations after eye movements nonetheless raises a potential challenge for visual stability. Like saccadic motion, however, these continual jumps go unperceived. This phenomenon is sometimes referred to as space constancy.
Some classic observations and experiments have provided evidence that, at least under some conditions, we may perform a computation of this sort. For example, von Helmholtz (1867/1962) reported that subjects with ocular paresis (partial paralysis of the muscles that move the eyes) experienced a perceptual jump in the direction of each intended—but, of course, not fully executed—saccade. A possible explanation is that corollary discharge signaled a larger saccade than was actually executed. Because the shift in the retinal image was insufficient to ‘cancel’ this signal, the external stimulus was perceived to shift. Subsequent experiments using temporary, artificially induced paralysis have replicated this finding (e.g., Brindley & Merton, 1960; Skavenski, Haddad, & Steinman, 1972). Related effects have also been observed while subjects experience an afterimage in a darkened room (Grüsser, Krizic, & Weiss, 1987). Under some circumstances, then, the command or intention to perform a saccade, in the absence of a corresponding retinal shift, can, it seems, cause our experience of visual stability to dissolve.
In addition, some have proposed that a shift of the scene on the retina without a saccade (or associated command) may have a similar effect. Indeed, when one taps lightly on the corner of the eye, the scene appears to jump in the opposite direction (Descartes, 1664/1972). Bridgeman and colleagues, however, offered an alternative analysis of the phenomenon (Bridgeman, 1979; Bridgeman & Stark, 1991; Ilg, Bridgeman, & Hoffmann 1989). They maintained that the finger tap does not, in fact, cause a translation of the image on the retina because the external force from the finger is annulled by oculomotor tracking mechanisms that hold the eye steady on the fixation point with a compensatory rotary movement. This compensatory movement in turn produces a record of an eye movement that is not reflected in a retinal shift and thus results in the perception of external motion. Therefore, this explanation is almost precisely the converse of the traditional account. Nonetheless, both theories posit that a mismatch of corollary discharge with retinal shift drives the percept of instability. Another possibility, however, is that moving the eye by tapping it may produce quite different visual input than a natural eye movement; any retinal change (whether due to translation, as the traditional account maintains, or compensatory rotation, as proposed by Bridgeman and colleagues) may be much slower and thus more perceptible than during a real saccade. Thus, atypical visual input offers an additional alternative explanation of the loss of stability in this situation.
Given, however, that there appear to be some conditions under which an eye movement command can be used to ‘cancel’ a retinal shift, how is this achieved? Historically, the central concern of the literature has been the origin of the signal specifying the saccadic vector.4 Two main candidates have been proposed. One is corollary discharge (or ‘outflow’ or the ‘efference copy’), a purported copy of the motor command for the eye movement that is routed to perceptual areas (von Helmholtz, 1867/1962; Sperry, 1950; von Holst & Mittelstaedt, 1950/1971). The other is propioceptive feedback (or ‘inflow’) from receptors in the extraocular muscles, small muscles around the eye that actually perform the movement (e.g., Sherrington, 1918). After decades of debate, consensus seems to have emerged that, though inflow may play a role in calibrating the proper interpretation of the corollary discharge signal over time (as discussed below), corollary discharge itself seems a more likely candidate for tracking a single saccade (e.g., Bridgeman & Stark, 1991; Wurtz, 2008). Notice, for instance, that the results of the paresis and temporary paralysis experiments described above seem consistent with corollary discharge but not propioception as the cancellation signal. Since little actual movement of the extraocular muscles took place, it is unclear why a largely stable retinal image should contradict the inflow signal and lead to perceived external motion. In addition, certain localization errors have been found to arise just prior to a saccade, before the eyes actually begin to move (e.g., Jeffries, Kusunoki, Bisley, Cohen, & Goldberg, 2007; see Ross, Morrone, Goldberg, & Burr, 2001 for a relevant review). This may suggest a slight temporal misalignment of corollary discharge with actual eye position (Honda, 1995; Matin, Matin, & Pearce, 1969; Morrone, Ross, & Burr, 1997). Moreover, invasive ablation studies in non-human primates suggest that inhibiting central signals of the eye movement command, but not prioprioceptive feedback, can lead to failures to compensate for eye movements (e.g., Guthrie, Porter, & Sparks 1983; Sparks, Mays, & Porter, 1987). Further research on non-human primates (reviewed in Klier & Angelaki, 2008) indicates that when a saccade is artificially triggered by directly stimulating the nerves that move the eyes, no compensation occurs for that saccade occurs. However, when a saccade is induced by stimulating more central areas (such as the superior colliculus), this movement appears to be tracked by the system and to be compensated for, as demonstrated by subsequent accurate movements to a remembered target. Once again, this seems to favor corollary discharge over proprioception as a mechanism for tracking particular saccades.
However, corollary discharge alone may not be sufficient to explain visual stability. One reason is that the gain of saccade-producing neural ‘bursts’ may change over time, in order to maintain accuracy, if the strength of the extraocular muscles changes with aging (Hopp & Fuchs, 2004). As these muscles weaken, the burst required to produce a movement of a given amplitude may increase, perhaps making it impossible to infer the length of a saccade from corollary discharge alone. If this signal is used to compute the saccadic vector, then some additional calibration signal specifying the gain may be required. Pairings of corollary discharge signals with attendant inflow signals (Steinbach, 1986; Wurtz, 2008) or with visual error may, however, allow proper calibration of the corollary discharge signal.
More fundamentally, there is some reason to doubt that a cancellation procedure is actually critically important for preserving space constancy during our frequent, real-world saccades. First, from a normative perspective, it seems important that the system should avoid false alarms. Frequent false perceptions of external movement during saccades would be quite disruptive. Given that actual movements of the full visual field during saccades would, presumably, be very rare (outside the context of a psychophysics experiment), it would seem reasonable that the system should be conservative about triggering a percept of external movement across saccades. MacKay (1972) captured this idea when he argued that external stability should be understood as the ‘null hypothesis’ of the visual system, overturned only in the face of overwhelming opposing evidence. More recently, Niemeier, Crawford, and Tweed (2003) proposed a Bayesian model implementing this principle. In the model, the prior probability of an external shift is set to be extremely low, making the model ‘distrust’ apparent signals to that effect.
While oculormotor error is certainly an important issue to consider, recent evidence suggests that targeting error does, in fact, seem to be fairly accurately captured or predicted by the corollary discharge signal (Collins et al., 2009). Thus, targeting error may not pose as significant a challenge to corollary discharge-based theories of stability as one might initially suppose.
However, substantial experimental evidence supports the notion that, during real-world saccades, visual stability is the default assumption. Consider again the example of illusory stimulus motion under paralysis. A follow-up study by Matin, Picoult, Stevens, Edwards, Young, and MacArthur (1982) revealed an important caveat to the original findings. They discovered that the illusory motion only occurred when the stimulus was an isolated point of light surrounded by darkness. When the room was illuminated and a complex visual context surrounded the stimulus, the subject experienced the world as stable. When the room was lit, the fact that the relative locations of objects in the scene were preserved across the eye movement may have provided a strong signal of external stability, thus overriding any discharge-based evidence to the contrary. Matin et al. (1982) described this apparent tendency of the system to favor visual evidence as a form of ‘visual capture’.
Further evidence against a strong version of cancellation theory, in which the saccadic vector must ‘annul’ retinal displacement in order for us to experience space constancy, comes from experiments in which the display is covertly shifted during eye movements. This manipulation, of course, artificially induces a discrepancy between retinal shift and (to the extent that it is accurate) the corollary discharge signal upon refixation. According to strong cancellation theory, at least given that this discrepancy exceeds whatever the tolerance of the system may be, subjects should experience a perceptual jump when this occurs, attributing the unequal magnitudes of the retinal shift and the saccade vector to an unstable external world. Consistently, however, these studies have shown that, within a surprisingly large margin, subjects are not, in fact, particularly sensitive to these shifts. This phenomenon is known as saccadic suppression of displacement (Bridgeman, Hendry, & Stark, 1975).
There is wide agreement that saccadic suppression of displacement rises with increasing saccadic amplitude, with many reports suggesting an approximately linear relation between the variables (see Bridgeman et al., 1975; Deubel, Schneider, & Bridgeman, 1996; Li & Matin, 1990). The size of the effect, however, has been reported with somewhat less consistency across studies. For instance, a 50 % detection rate has been linked with shifts of approximately one-tenth (Li & Matin, 1990) to more than one-third (Bridgeman et al., 1975) the length of the saccade. Both of these studies used artificial stimuli (though with differing levels of visual complexity). Employing complex, meaningful, and relatively large photographs as stimuli, however, McConkie and Currie (1996) reported even higher levels of saccadic suppression of displacement, with 50 % detection sometimes occurring for shifts of approximately one-half the length of the saccade.
There are several possible explanations for this range of findings, including the precise timing of the shift relative to the saccade and, notably, characteristics of the stimulus, such as size and complexity. Indeed, given the findings concerning visual capture under paralysis discussed above, it is perhaps not surprising that full scenes, in which spatial relations among objects are maintained, seem to drive stronger assumptions of external stability than do isolated, artificial stimuli. It is also plausible that differing task demands, such as the requirement to provide a yes/no response to each trial (as in Li & Matin, 1990) or to respond only when detecting a shift (as in Bridgeman et al., 1975), could have affected subjects’ response criteria in these experiments and hence their hit rates for detecting shifts. Indeed, consistent with this suggestion, false alarms were ‘negligible’ in the work of Bridgeman et al. (1975) but ranged from 8 % to 16 % for Li and Matin (1990). Despite such quantitative differences, however, the findings are in agreement that even relatively large transsaccadic shifts can escape our notice. Thus, if a cancellation mechanism (or similar process) were engaged to confirm stability during real-world saccades, this mechanism would, it seems, need to allow a very large error tolerance.
Our sensitivity to stimulus shifts is not homogeneous across the visual field. We are, in fact, more likely to notice shifts of a small saccadic target object against a stationary background than shifts of the background behind a stable target (Bridgeman, 1981; Brune & Lucking, 1969; Currie, McConkie, Carlson-Radvansky, & Irwin, 2000; McConkie & Currie, 1996). What does this mean for cancellation theory? The finding precludes an account wherein the saccadic vector is simply compared with average ‘pixel’ displacement across retinal images in order to gauge stability. However, it may be that we do, in fact, compare the saccadic vector with retinal shift as the theory supposes, but that displacements of the target—a highly salient element of the visual field (e.g., Deubel & Schneider, 1996; Hoffman & Subramaniam, 1995; Kowler, Anderson, Dosher, & Blaser, 1995; Rolfs & Carrasco, 2012)—are somehow given more weight than other parts of the scene when determining whether the external world has remained stable.
Currie, McConkie, and colleagues, however, proposed a different hypothesis, one that does not rely on corollary discharge. It is known simply as the saccade target theory (McConkie & Currie, 1996; Currie et al., 2000). Knowing the identity of the target, they pointed out, is sufficient to derive a prediction; given a stable world, the target object should lie near the fovea when the saccade is complete, with some variability expected due to targeting error. On their account, we store a few features of the target object across the saccade in visual short-term memory and then search for this object near the center of the visual field when the new fixation begins. Notably, such a mechanism does not require keeping track of the vector of the saccade. In addition, they maintained, we store some information about the relative locations of certain other important parts of the scene with respect to the target. If the target is not found upon refixation, according to this theory, visual stability is broken. If, however, the target is located, a second stage of processing ensues wherein the system checks whether relative spatial relationships have been preserved (Currie et al., 2000).5
Deubel and colleagues endorsed a similar model. However, on their account, a latent corollary discharge signal is preserved across the eye movement. In normal circumstances, though, they argued, this signal is not used. This idea stems from the discovery of the blanking effect (Deubel, Schneider, & Bridgeman, 1996; Deubel, Bridgeman, & Schneider, 1998). They found that when a brief blank interval was presented immediately upon refixation, subjects became much more sensitive to covert shifts of the scene. The study used an alternative forced choice paradigm, in which subjects had to report the direction of the intrasaccadic shift. Performance improved dramatically when a short blank interval was introduced upon refixation, rising from less than 75 % correct to near perfect results. When the target was blanked, Deubel et al. suggested, the assumption of a stationary world was broken and the system then turned to signals such as corollary discharge (Deubel et al., 1996).6 They suggested that fairly precise information about the saccadic vector may indeed be stored but that under normal circumstances, when the target is indeed present near the fovea, this record simply is not consulted. Only when expectations are grossly violated, as in the blanking condition, is the cancellation procedure enacted. Germeys, De Graef, Van Eccelpoel, and Verfaillie (2010) offered a somewhat different interpretation of this finding, suggesting that the blanking condition may have attenuated backward masking of the presaccadic image by the postsacadic image. In a broad sense, however, these higher and lower level interpretations of the effect seem to be compatible; both propose that information obtained immediately following refixation generally overwrites (whether by masking or higher level cognitive mechanisms) detailed information from the prior fixation.
In summary, the visual system indeed appears to treat external stability across saccades as the ‘null hypothesis’. Indeed, given the low probability of external displacement during a saccade, this seems a rational approach. The null hypothesis may, however, be rejected in the face of sufficient evidence to the contrary. The phenomenology of stability breaks down, for instance, when the scene is briefly blanked upon refixation or covertly shifted by an extremely large vector during the saccade. Space constancy also dissolves when saccades are attempted while under paralysis, but only when the stimulus appears in isolation, without the presence of a full visual context to bias interpretation toward stability. Additionally, shifts of the saccade target object in particular may disrupt stability. According to Deubel and colleagues, however, once the assumption of stability is broken in circumstances such as these, a latent signal specifying the precise vector of the saccade may, in fact, become available for comparison with retinal displacement.
Memory and Integration
Transsaccadic memory and integration, a topic treated only obliquely above with regard to its possible role in visual stability, will be the central focus of this section. Suppose that you have just finished a long walk and arrived at the summit of a hill to find a panoramic view. To your left lies the coastline and, beyond it, the ocean dotted with sailboats. To your right lies the city skyline. If you glanced from a sailboat to a high-rise building, what sorts of visual memories would persist from the prior fixation and how might they contribute to subsequent processing? Would you begin to build up a single, detailed, spatiotopic representation of the scene across different fixations? If you then pulled out a book and began to read, how might you integrate samples of text acquired over distinct fixations? Would integration proceed similarly or quite differently depending on the activity (scene perception or reading) in which you were engaged? Below, we examine research on these and related questions of transsaccadic memory.7
One theoretical possibility is that we can fuse images obtained across multiple fixations to form a single, highly detailed, spatiotopic representation of the entire scene. McConkie and Rayner (1976) proposed that we might possess a special, transsaccadic memory store, known as the integrative visual buffer, to support precisely this function (see also Rayner, 1978; for similar views, see Breitmeyer, Kropfl, & Julesz, 1982; Feldman, 1985; Trehub, 1977), allowing Neisser’s ‘snapshots’ to be combined across fixations. Memory traces, similar to iconic memories, could be properly aligned on this account based either upon visual overlap or a signal, such as corollary discharge, that would specify the vector of the intervening saccades. While this idea seems to capture the phenomenal experience of viewing a scene—it feels as though we are able to create a stable representation of the complete visual environment—the theory is not well supported by evidence.
An early finding reported by Jonides, Irwin, and Yantis (1982) initially seemed to support the hypothesis of the integrative visual buffer. They found that two complex arrays of dots presented successively at the same spatial location, but different retinal locations, could be seemingly be fused across a saccade. When fused, the arrays of 12 dots each would leave a single dot in a 25-dot array unfilled. Subjects were to identify that missing dot. In a true saccade condition, subjects performed substantially better than in a simulated saccade condition, suggesting that the distinct arrays could be perceptually combined across saccades.
Subsequent studies revealed, however, that these findings were almost certainly due to an artifact; the persistence of the phosphor on the cathode ray tube (CRT) monitor likely caused the first array to physically remain upon the screen even after the eye movement was complete and the second set of points was illuminated, thus enabling good performance in the transsaccadic task. Attempts to replicate the finding failed when a light-emitting diode display was used in place of the CRT monitor, when filters were applied to the original monitor to counteract persistence (Irwin, Yantis, & Jonides, 1983; Jonides, Irwin, & Yantis, 1983), when dimmer dots were used, thus speeding phosphor decay (Bridgeman & Mayer, 1983), and when all 25 dots were flashed for an instant during the eye movement, obscuring any lingering illumination from the first display (Rayner & Pollatsek, 1983). In a related study, O'Regan and Lévy-Schoen (1983) presented complementary line segments to subjects across a saccade. If perceptually fused, the segments would form letters spelling a short word which subjects were asked to identify. While the subjects performed well on the task in the condition in which stimuli were retinally aligned during a single fixation, performance in the saccade condition was poor. Early positive results reported by Wolf, Hauske, and Lupp (1978, 1980), suggesting spatiotopic fusion of sinusoidal gratings across a saccade and by Davidson, Fox, and Dick (1973) suggesting phenomenal spatiotopic masking also failed to find support in subsequent studies (see Irwin, Zacks, & Brown, 1990; Irwin, Brown, & Sun, 1988, respectively).
Apparent motion, in which an object viewed in different locations at successive points in time gives rise to an illusion of movement (Anstis, 1978), also does not seem to operate in quite the same way across eye movements as it does within a single fixation. Apparent motion can, in fact, occur across saccades (Rock & Ebenholtz, 1962; Cavanagh, Hunt, Afraz, & Rolfs, 2010; Fracasso, Caramazza, & Melcher 2010; Szinte & Cavanagh, 2011). For example, when the spatial location of a simple stimulus changes across an eye movement while its retinal location remains the same, subjects do report apparent motion (Rock & Ebenholtz, 1962). However, research conducted by Irwin (1991) suggests that transsaccadic displacement, unlike apparent motion within a single fixation, may not be sufficient to cause a ‘moving’ stimulus to perceptually ‘pop-out’ among a complex field of distractors.
Irwin (1991) investigated subjects’ ability to detect the displacement of a single dot within an array of distractors from one stimulus presentation to the next within a single fixation, across a saccade, or in a simulated saccade condition. Several interesting differences in performance emerged between the single fixation condition and the saccade and simulated saccade conditions. First, performance in the single fixation condition at short interstimulus intervals (about 40 ms) was quite good regardless of the visual complexity of the arrays, provided that these arrays occupied precisely the same sptatial location in the two presentations. In the saccade condition and simulated saccade conditions, however, performance was relatively low at the shortest interstimulus intervals, eventually peaking only when several seconds elapsed between array onsets. Performance in these conditions was also highly dependent on the complexity of the array, declining markedly as the patterns grew more complex. Finally, performance in the saccade condition seemed less dependent upon spatial congruity of the first and second array presentations than it did in the fixation condition.8
In the single fixation condition (but not the saccade condition), subjects reported perceiving apparent motion when the dot was displaced, given that the arrays appeared at the same location and were separated by low interstimulus intervals. Irwin suggested that this perception of apparent motion may have caused the critical dot to perceptually ‘pop-out’ from the rest of the display, thus supporting superior displacement detection in the fixation condition relative to the saccade condition. Thus, it seems plausible that the apparent motion experienced across saccades may lie somewhat closer to deduction on a perception-deduction processing continuum than does apparent motion within a single fixation. This form of apparent motion may not be capable of triggering ‘pop-out’ of a critical stimulus among a complex field of distractors.
Transsaccadic memory performance in this task, Irwin argued, was likely attributable to visual short-term memory (as opposed to something like an integrative visual buffer). Visual short-term memory is limited in capacity,9 not highly spatially selective, and relatively robust over the duration of several seconds (see also Hollingworth, Richard, & Luck, 2008; Irwin, 1992; Irwin, 1996; Irwin et al., 1990). Indeed, Hollingworth et al. (2008) speculated that a primary function of visual short-term memory may be to span the saccade so that the target can be located upon refixation and, if necessary, so that a corrective saccade may be programmed to align it with the fovea.
A striking phenomenon known as ‘change blindness’ (e.g., Grimes, 1996; for reviews and analyses, see Simons & Levin, 1997; Simons & Ambinder, 2005; Simons & Rensink, 2005) also seems inconsistent with the notion of an integrative visual buffer. Research on change blindness may be considered extension of the work discussed above demonstrating saccadic suppression of displacement. In change blindness, we find that rather dramatic changes in visual features, like changes in location, may escape subjects’ notice provided that they occur during a saccade or other visual disruption (such as a blank interval; see Rensink, O'Regan, & Clark 1997).10 In one study, for instance, Grimes (1996) presented subjects with naturalistic scenes and asked them to study the images for an upcoming memory test. He warned subjects in advance to monitor the scene for changes and asked them to indicate when they noticed any changes by pressing a button. Even given this prior warning, subjects’ sensitivity to the change was surprisingly low. For example, when the heads of two prominent figures in the scene were switched during an eye movement, subjects reported seeing something change only about half the time. And when a flock of birds perched on the shoreline decreased in number by about a third during a saccade, subjects reported noticing something odd in only about 10 % of trials. If the changing object is pre-cued or lies at or near the target of the critical saccade (during which the change occurs), however, detection performance improves (Rensink et al., 1997; Irwin & Gordon, 1998), suggesting that attention mitigates change blindness.
As noted above, these findings seem inconsistent with the notion that we form fine-grained, spatiotopic representations of the full scene across saccades. Provided that these representations are accessible to conscious awareness, we would expect that such profound changes in the scene would be detected more reliably. Can we go further, however? Can we conclude that transsaccadic visual memory is extremely sparse for all but the immediate objects of our attention (see, e.g., Dennett, 1991; O’Regan, 1992; Rensink et al., 1997)? Before arriving at this conclusion, a number of factors must be considered.
First, it is possible that changes that fail to reach conscious awareness may nonetheless be detected at unconscious levels within the system. Indeed, evidence for unconscious change detection come from research in which subjects ‘guess’ at levels above chance which element in an array has changed even when they report seeing no change occur (Fernandez-Duque & Thornton, 2000). Inflated fixation durations on changed objects, even in the absence of conscious change detection, may also provide support for this idea (Henderson & Hollingworth, 2003; see also Hayhoe, Bensinger, & Ballard, 1998). Second, having viewed the changing object, at some point, in both its original and its altered state would seem a prerequisite for detecting a change, no matter how robust the transsaccadic visual memory system might be. Because visual acuity declines dramatically with growing eccentricity from the fovea and because most change blindness studies have not controlled for whether the critical object received such fixations, some proportion of ‘misses’ in these studies may derive from a failure not of memory but of exposure to vital visual input (Henderson & Hollingworth, 1999). Third, failures at the stages of retrieval or comparison, rather than retention, appear to account for some detection failures (Hollingworth, 2003). Thus, while change blindness is certainly informative about transsaccadic perception—it belies our intuition that we monitor our full visual environments rather keenly even across disruptions (Levin, Momen, Drivdahl, & Simons, 2000)—it does not imply that only extremely sparse information survives each saccade.
Indeed, other lines of research provide evidence that we do retain and use substantial information from prior fixations. First, we can, it seems, discern external spatial relationships among objects viewed in disparate fixations. In a study by Hayhoe, Lachter, and Feldman (1991), for instance, subjects were successively presented with three points of light that would, if simultaneous, form the vertices of a triangle. Participants then determined whether the central angle of the triangle would have been acute or obtuse when maintaining central fixation or when sequentially fixating each point (either in isolation or in the continual presence of a static visual landmark). Though performance was best during sustained fixation, it was nonetheless significantly above chance in both eye movement conditions. More recently, Prime, Niemeier, and Crawford (2006) presented subjects with two tilted lines in succession and asked them to indicate the point at which the lines would have intersected had they appeared at once upon the screen. Subjects in this study performed equally well when a saccade intervened between stimulus presentations as they did when maintaining steady fixation.
An additional important line of research on transsaccadic integration concerns parafoveal preview (or, more broadly, extrafoveal preview). This work directly examines the ways in which implicit transsaccadic memory may be used to facilitate subsequent processing (for a review of parafoveal processing, see Schotter, Angele, & Rayner, 2012). These studies demonstrate that information does persist across saccades and that transsaccadic memory traces are used in the following fixation. However, the nature of the information that subserves this preview effect appears to differ subtly depending on the task at hand or the characteristics of the stimuli involved.
What sort of information is preserved and integrated in this manner? In reading, it does not appear that facilitation is driven by the integration of visual features. This view is evidenced by changing case experiments (McConkie & Zola, 1979; Rayner, McConkie, & Zola, 1980; Slattery, Angele, & Rayner, 2011; see also Henderson, 1994 for a single letter version of the task). In such studies, the case of each letter in a word is reversed during the saccade. Alternating upper and lower case letters are used in both the preview and the target in order to avoid highly salient changes in the word’s size from one presentation to the next. Indeed, subjects are typically not consciously aware that such a change has occurred. The word ‘tArGeT’, for example, might be preceded either by an identical preview or by the reversed-case preview ‘TaRgEt’. Although the degree of visual overlap clearly differs between the identical and reversed-case previews, the resulting facilitation is found to be equal, suggesting that the preview benefits found in these studies are not subserved by visual integration. Rather, it appears that orthography (i.e., abstract letter identity), a somewhat higher level of representation, contributes to such facilitation effects in reading. Furthermore, because phonological similarity between the preview and the target confers an advantage over and above that obtained from orthographic overlap, phonology also appears to contribute to preview benefits (Pollatsek, Lesch, Morris, & Rayner, 1992; see also Henderson, Dixon, Petersen, Twilley, & Ferreira, 1995). Finally, it is noteworthy that facilitation does not require spatiotopic congruity of the preview with the target; indeed, when the retinal conditions experienced in a saccade are simulated during steady fixation (with the successive presentation of the preview at the parafovea followed by the target at the fovea), preview benefits are nonetheless obtained (Rayner, McConkie, & Ehrlich, 1978). In summary, then, these studies indicate that, at least in reading, preview benefits result not from a spatiotopic combination of low-level visual information across saccades, but rather from the integration of higher-level information (e.g., abstract letter identity).
Preview facilitation has also been studied in pictorial object recognition. The stimulus, in this case, is simply a drawing or photograph depicting objects or a scene. Following the extrafoveal preview (often presented beyond the parafovea in such tasks), the subject fixates the object and either silently identifies it, often before moving on to other objects in a sequence, or names it aloud (as in Schotter, Ferreira, & Rayner, 2013). As in reading research, phonological overlap between the preview and the target appears to speed responses in object naming (Meyer, Ouellet, & Häcker, 2008; Morgan & Meyer, 2005; Pollatsek, Rayner, & Collins, 1984). Furthermore, just as described above, preview facilitation is obtained even when spatial congruity between preview and target is violated in a simulated saccade condition (Pollatsek, Rayner, & Henderson, 1990). Therefore, precise spatial congruity is not strictly necessary for preview benefits to arise. The question of whether spatial congruity enhances preview benefits, however, has yielded somewhat mixed findings, with Pollatsek et al. (1990) finding little or no such effect, and others (e.g., Henderson & Anes, 1994; Kahneman, Treisman, & Gibbs, 1992) finding an increase in preview facilitation due to spatial congruity (see Henderson & Anes, 1994 for a useful discussion of possible explanations of these differences). Finally, Pollatsek, et al. (1984) found that facilitation in this paradigm was unaffected by relatively minor changes in size (on the order of about 10 %) from preview to target.
Interestingly, some differences in the role of visual information in preview effects appear to occur between reading and pictorial object recognition. Pollatsek et al. (1990) found that a preview that was visually similar (but phonologically and semantically quite dissimilar) to the target speeded subsequent naming. For example, a preview of a carrot speeded naming of a baseball bat upon direct fixation. Furthermore, Henderson and Siefert (1999, 2001) found that an identical preview of an object lead to greater facilitation than its (still perfectly recognizable) mirror image (see also Pollatsek et al., 1984). In addition, Demeyer, De Graef, Wagemans, and Verfaillie (2009) presented subjects with artificial shapes during preview that were either identical to the target or different in very minor visual detail. They found that identification performance following the identical preview (in this case measured primarily by correct identifications rather than response latency) was superior to performance following even the highly similar but subtly altered preview (see also De Graef & Verfaillie, 2002). Taken together, these findings suggest that, unlike in reading, visual contour information is used to facilitate subsequent processing when viewing pictorial objects.
Why might preview effects in reading and visual object recognition differ? Irwin et al. (1983) pointed out that, with respect to the importance of visual information, reading may be a special case. The transsaccadic accumulation of visual information per se may be unnecessary during reading. When reading in an alphabetic language, the reader must transform arbitrary visual symbols into higher level representations of letters, words, sounds, and eventually meanings. Once this has been accomplished, visual details are entirely meaningless. ‘ChAiR’, ‘cHaIr’, ‘chair’, and ‘chair’ are, after all, fully equivalent in meaning. In contrast, visual differences in form may be more significant when viewing real objects, even after the object’s category has been identified; chairs of different shapes might, for example, feel quite different to sit on.11 It is possible the transsaccadic integration system is somehow sensitive to these differences in the importance of visual information across domains. It is also worth noting, however, that differences in basic physical properties of the stimuli generally obtain between reading and object recognition tasks. For instance, a previewed word is typically presented much closer to the fovea than is a previewed visual object. Previewed objects are typically also much larger than words. It is possible that such physical variables may modulate the role of visual information in preview effects.
While the theory of the integrative visual buffer—a purported stable, fine-grained, spatiotopic image of the scene that builds up across eye movements—has not generally received support in the literature, some more recent findings have provided evidence of certain, perhaps more local, spatiotopic effects across eye movements. However, a number of these findings have either failed to replicate or been proposed to be explainable by alternative hypotheses. Therefore, additional research is required before this question can be settled with certainty.
Melcher and Morrone (2003; see also Ong, Hooshvar, Zhang, & Bisley, 2009) found evidence of spatiotopic integration of motion signals across a saccade. They presented subjects with two consecutive weak motion signals, elicited by a subset of dots moving in unison amid a field of random motion. Each brief presentation fell below the subjects’ detection threshold. However, when the stimuli occupied the spatial location (but different retinal locations) before and after an eye movement, the signals could seemingly be temporally integrated so as to enable motion detection. This was not the case in a control condition, when the stimuli appeared at different spatial locations during a single fixation.12 Morris et al. (2010), however, offered a different interpretation for this result. They argued that what appeared to be low-level, spatiotopic integration of information across saccades was, instead, the result of a decision process. Specifically, they maintain, the effect arises because of noise in the system regarding the onset time of each motion signal.
Findings of spatiotopic aftereffects have also been reported, though some of these findings have been disputed. Ezzati, Golzar, and Afraz (2008), for instance, found evidence for a transsaccadic motion aftereffect, wherein adaptation to a moving stimulus induces a bias in the opposite direction when perceiving subsequent stimuli, that was anchored in external spatial coordinates (although the effect was stronger at the retinotopic location of the adaptor). Turi and Burr, however, (2012) found no evidence for such spatiotopic transfer of traditional motion aftereffects. They did, though, find spatiotopic positional motion aftereffects, thought to be a distinct phenomenon wherein adaptation to moving stimuli induces opposite biases in subsequent judgments of position. Melcher (2005) also reported finding several kinds of spatiotopic aftereffects across eye movements, including for faces and tilted gratings, though some of these findings were later disputed (see Afraz & Cavanagh, 2008; Knapen, Rolfs, Wexler, & Cavanagh, 2010; Mathôt & Theeuwes, 2013). Mathôt and Theeuwes (2013) found, for instance, that when an adaptor and test grating were presented before and after a saccade respectively, a tilt aftereffect occurred when the two stimuli were retinotopically aligned, but not when they were spatiotopically aligned. Indeed, upon computing the Bayes factor to assess their results, they concluded that there was substantial evidence against a spatiotopic tilt aftereffect.
In addition, a study by Wittenberg, Bremmer, and Wachtler (2008) found evidence of transsaccadic color priming for discs presented at the same external spatial location across eye movements. The perceived color of an orange test stimulus presented after completing a saccade appeared biased in the direction of a presaccadic, spatially congruent red or yellow prime. It is not entirely clear from this study, however, whether this priming effect was due to spatiotopic transsaccadic integration, as Wittenberg et al. suggested, or a more general phenomenon wherein feature information attributed to a single object may be combined.
Research on transsaccadic memory and integration has been complex, yielding some apparently contradictory findings. Nonetheless, several points seem clear. Notably, several lines of classic evidence do not seem compatible with the notion that we have and use a fine-grained, spatiotopic representation of the full scene, integrated across multiple fixations. First, we do not seem able to perceptually fuse complex stimuli across saccades in quite the same way that we can, under certain conditions (e.g., with low ISIs), within a single fixation (recall, e.g., the study by O'Regan & Lévy-Schoen, 1983, involving the integration of line segments to form letters). Second, change blindness indicates that we sometimes fail to notice even rather dramatic changes in the scene when they take place across eye movements (or other visual disruptions). Returning to the scene described in the introductory paragraph above, change blindness reveals that if a contemporary sailboat in the scene were replaced with a seventeenth-century frigate during one of your saccades, this would likely escape your notice, provided that the ship were not the immediate object of your attention. Third, at least in the case of reading, preview benefits are not affected when the visual form—the case—of letters changes from one fixation to the next (though visual information appears to support preview benefits when viewing objects). Nonetheless, there is ample evidence that substantial information is preserved and, indeed, used from prior fixations. First, external spatial relationships among objects can be determined over multiple fixations (recall Hayhoe et al., 1991). Furthermore, as was noted above, previews of objects glimpsed in the parafovea or periphery before they are fixated directly provide us with a ‘head start’ in future processing. Interestingly, the nature of these preview effects appears to differ somewhat across domains. In particular, while visual feature information is unimportant in preview effects during reading, there is some evidence that feature congruity between preview and target confers facilitation during pictorial object recognition. Finally, while the notion of a full-scene integrative visual buffer is not well supported, some recent research suggests that certain, perhaps more local spatiotopic integration across saccades may be possible under some conditions. However, a number of these findings have been subject to debate; thus, more research is required before this matter can be decided with certainty.
It should also be noted, however, that some research suggests that, while receptive fields do change before saccades, these changes may not, in fact, reflect predictive remapping. For instance, Tolias et al. (2001) found that receptive fields in V4 seemed to shrink and compress toward the saccade target rather than shift so as to anticipate the post-saccadic pattern of activity. More recently, Zirnsak, Steinmetz, Noudoost, Xu, & Moore (2014) reported similar effects in the frontal eye fields (FEF) of monkeys. Using multiple electrodes to record output from a number of neurons simultaneously, they found that receptive fields seemed to converge around the target of the saccade prior to execution. They suggested, furthermore, that findings of this kind may help explain enhanced presaccadic visual attention at the saccadic target (e.g., Deubel & Schneider, 1996) as well as mislocalization of stimuli toward the target location (Ross et al., 1997). There is currently some uncertainty, then, about the exact nature of the changes in receptive fields that occur around the time of saccades. Nonetheless, because a significant number of papers have reported findings that appear to reflect remapping and because this transformation is regularly cited as an explanation for aspects of transsaccadic processing, the remainder of this section will focus upon this phenomenon and its potential behavioral implications.
Remapping was discovered by Duhamel, Colby, and Goldberg (1992) in the lateral intraparietal cortex (LIP) of the rhesus macaque (see also Kusunoki & Goldberg, 2003; Heiser & Colby, 2006). Additional findings from non-human primates suggest that the frontal eye fields (FEF) (Umeno & Goldberg, 1997) and superior colliculus (SC) (Dunn, Hall, & Colby, 2010; Walker, Fitzgibbon, & Goldberg, 1995) also exhibit robust remapping. Notably, these areas are primarily involved in allocating visual attention and selecting the targets of upcoming saccades rather than processing visual feature information (see, e.g., Findlay & Gilchrist, 2003 for an overview; but see also Sereno & Maunsell, 1998 for an account of shape selectivity in LIP). Remapping has also been found, though in much lower proportions of cells, in relatively early visual areas. Nakamura and Colby (2002) reported, for instance, that cells in areas V3A, V3, and V2 of the rhesus macaque all showed detectable levels of remapping. Almost no cells in V1, however, exhibited the effect. Broadly consistent findings have also been obtained in humans. Using functional magnetic resonance imaging (fMRI), Merriam, Genovese, and Colby (2003) found evidence of predictive remapping in the posterior parietal cortex (see also Parks & Corballis, 2008), a region involved in spatial cognition and movement planning (Colby & Goldberg, 1999). As in monkeys, remapping in humans appears less common in early visual areas (Merriam, Genovese, & Colby, 2007). In addition, Joiner, Cavanaugh, & Wurtz (2011) found that more salient stimuli were subject to greater remapping than less salient stimuli. Indeed, Gottlieb, Kusunoki, and Goldberg (1998) suggested that remapping may only occur for salient stimuli (see also Melcher & Colby, 2008; Wurtz, 2008 for further discussions of this idea). However, a recent study by Mirpour and Bisley (2012) found that both target and distractor items in a display were remapped across saccades. This may suggest that remapping, in fact, occurs for all stimuli in the visual field. However, as Mirpour and Bisley point out, it is also possible that the distractors used in this study were, in fact, relatively salient (as they were bright and unstable), thus triggering remapping.
We do not yet fully understand how remapping, which entails the transfer of neural activity across cells, occurs. Evidence suggests, however, that corollary discharge drives the effect, selecting the appropriate mappings across cells on the basis of the upcoming eye movement vector (Sommer & Wurtz, 2006, 2008). Projections from the superior colliculus to oculomotor nuclei specify the targets of impending saccades (e.g., Wurtz, 2009). Recent experiments suggest that projections from this structure also run in the opposite direction, conveying a corollary discharge signal through mediodorsal nucleus of the thalamus (MD) to the frontal eye fields in frontal cortex (Sommer & Wurtz, 2002, 2004). Interestingly, unilateral inactivation of the MD nucleus of the thalamus appears to impair remapping in FEF for contraversive saccades in monkeys (Sommer & Wurtz, 2006). Thus, corollary discharge traveling along this pathway may underlie predictive remapping.
After a brief introduction to remapping, we can now turn to the fundamental question of this section: what role, if any, does the phenomenon play in transsaccadic perception? Remapping is often proposed to enable stability (e.g., Hall & Colby, 2011) or to compensate for retinal shifts across eye movements. The precise meaning of such statements is, however, often somewhat ambiguous. At least two alternative explanations should be differentiated. One proposal is that remapping creates a stable, spatiotopic representation of the scene. A second proposal is that remapping contributes to the perceptual elimination of the eye movement by predicting post-saccadic activation given a stable external world (see Bridgeman et al., 1994 for a relevant comparison of what they call ‘translation’ and ‘elimination’ theories).
The first proposal holds that remapping establishes a representation in which mappings from neurons and external locations are invariant across eye movements. Such a representation, which might be called spatiotopic, could in theory be useful in several ways. First, it would position objects viewed in distinct fixations within a single reference frame so that the spatial relations among them could be discerned. Second, it might facilitate the programming of visually guided movements, simplifying the computations required to direct an effector to a visual target.
However, as Burr and Morrone (2011) pointed out, remapping does not, in fact, annul saccade-induced changes in receptive fields to create a single, invariant representation. Rather, it anticipates the very changes the saccade will bring about (see, e.g., Talsma, White, Mathôt, Munoz, & Theeuwes, 2013; Wurtz, 2008). Suppose, for example, that a cue is first glimpsed at the periphery and is then fixated directly following a saccade. A truly compensatory mechanism would enable a fixed neural locus to respond to this stationary cue both before and after the eye movement. Instead, in remapping, as in its absence, cue-related activity in retinotopic areas will shift to a new set of cells because of the saccade. The difference is that, in remapping, this may occur slightly before the eyes actually move. Thus, remapping does not produce true spatiotopy.
A second hypothesis is that remapping supports the phenomenal experience of stability across eye movements. Sommer and Wurtz (2006), for example, claimed that the neural pathway driving remapping provides ‘a circuit-level explanation’ for transsaccadic visual stability. In order to maintain phenomenal stability, remapping may produce a prediction of the expected postsaccadic retinal image given that the external world remains stable (and no new objects appear) during the saccade. If the prediction is upheld upon refixation, according to this argument, we will perceive a stable world (e.g., Crapse & Sommer, 2008a, 2008b; Heiser & Colby, 2006; Wurtz, 2008; Wurtz, Joiner, et al., 2011).
Though this proposal is theoretically tenable, it is important to note that the successive presence of two activation patterns on a map—the remapped, presaccadic ‘prediction’ followed by the postsaccadic reality—does not constitute a comparison between them. It seems analogous to the process of overwriting a set of variables in a computer program with new values. In this example, as in remapping, the overwriting process does not entail a check for equality. Some external ‘comparator’ is required to actually perform the needed comparison (Bays & Husain, 2007; Wurtz, Joiner, et al., 2011). At present, however, the nature and locus of such a comparator is unknown. Therefore, remapping is not sufficient to enable prediction-based verification of external stability, though it may play some role in such a process. Furthermore, it not clear, a priori, that remapping should be necessary for this sort of verification. As was noted above, remapping appears to be driven by a corollary discharge signal specifying the vector of the upcoming saccade. In theory, a ‘comparator’ could alternatively verify external stability using this corollary discharge signal, the typical (i.e., not remapped) presaccadic activation pattern, and the postsaccadic activation pattern as inputs, comparing the first input to the difference in location between the latter two; indeed, this seems to be the sort of mechanism suggested by the classic form of the cancellation theory of stability. Additionally, as Currie and colleagues have pointed out (McConkie & Currie, 1996; Currie et al., 2000), simply retaining a visual short-term memory trace of features of the saccade target object could enable the system to engage in a prediction and verification process without tracking the saccadic vector. In this case, the prediction is simply that an object matching the memory trace of the saccade target will appear near the fovea after the eye movement is complete provided that the world has remained stable across the saccade.
A more fundamental point, however, is that prior research suggests that corollary discharge-based verification, at least in simple form, is likely not critical to phenomenal stability during most real-world saccades. First, we are not terribly sensitive to artificial, transsaccadic shifts of the scene—shifts that should lead to failures of remapping-based predictions and hence to the detection of external instability (at least if they exceed whatever the tolerance of the system may be). Second, during paresis or paralysis, while attempting to make an eye movement does disrupt stability when the stimulus is an isolated point of light, the sensation of stability is maintained in the presence of a full, complex scene. This research does not preclude some role for prediction-based verification mechanisms in phenomenal stability. For example, it might be the case, as Deubel and colleagues suggested, that comparison of the corollary discharge vector and retinal shift does occur, but only after we obtain dramatic evidence—such as the introduction of a blank interval upon refixation—that something is amiss. What these findings do suggest, however, is that the maintenance of perceptual stability under natural conditions does not hinge on a mechanism whereby retinal shift must be annulled by corollary discharge after each saccade. Rather, a more subtle operation seems to be at work, one in which a wider range of evidence is considered.
Finally, Wurtz, Joiner, et al. (2011) proposed that remapping may contribute to phenomenal stability by simply serving as an indicator that a saccade has taken place, causing the system to disregard visual disruptions associated with the movement. Though it is possible that the system uses remapping in this manner, a number of other potential signals (such as the corollary discharge signal that remapping itself relies upon) could alternatively be used for such a purpose.
In the preceding paragraphs, we have approached the potential role of remapping in phenomenal stability mainly from a theoretical perspective. What does the empirical evidence suggest? One relevant study, conducted by Ostendorf, Liebermann, and Ploner (2010), examined transsaccadic processing in a human patient, N.P., who had suffered a localized, unilateral stroke affecting a small area around the medial dorsal region of the right thalamus. Recall that Sommer and Wurtz (2002, 2006) found evidence that a pathway from the superior colliculus, through this region, to the FEF may carry a corollary discharge signal driving remapping. If remapping occurring due to this signal is critical to perceptual stability across eye movements, disrupting this pathway should lead to illusory stimulus motion across saccades (Sommer & Wurtz, 2006). Interestingly, N.P.’s performance in a transsaccadic shift detection task did not differ from controls under naturalistic conditions, when the stimulus was visible immediately upon refixatation following the saccade. However, when a brief blank interval was inserted after refixation, while typical subjects showed a marked improvement in performance, perhaps due to a weakening of the bias toward stability and an increased reliance on the corollary discharge signal, N.P.’s performance declined. In particular, for saccades that were ipsiversive to the lesion, his performance diminished significantly from baseline in this blanking condition. Although his saccade targeting remained accurate, he tended to report inward stimulus shifts (with respect to the direction of the saccade) when the cue had remained still or even shifted slightly outward. This finding seems consistent with the idea that, in the blanking condition, a hypometric extraretinal signal was recruited to determine visual stability during ipsiversive saccades. In addition, N.P.’s judgments about stimulus shifts in the blanking condition were better predicted by visual error upon landing than by actual shifts. This suggests that, unlike in healthy individuals tested in this paradigm (Collins et al., 2009; Deubel et al., 1996), he was not able to track and account for targeting error. More recently, Ostendorf, Liebermann, & Ploner (2013) conducted a follow-up study with a larger number of patients and healthy control subjects. The pattern of results obtained in this study was similar to that reported in Ostendorf et al. (2010).13 Critically, patients in this study reported no deficits in visual stability in day-to-day life. Thus, these findings simultaneously suggest a possible role for remapping in phenomenal stability under certain specific laboratory conditions (i.e., when using the blanking paradigm) and highlight the circumscribed nature of its importance. Under naturalistic conditions, after all, no disruptions of stability occurred. A final point that should be noted is that possible perceptual effects of an MD lesion cannot be attributed, with certainty, to remapping in particular as a corollary discharge signal passing through this region may instead contribute to stability via other mechanisms.
Some recent studies have also used transcranial magnetic stimulation (TMS) in human subjects to examine transsaccadic perceptual effects when activity in specific brain regions is disrupted. Chang and Ro (2007), for instance, studied transsaccadic shift detection performance after TMS was applied unilaterally to the posterior parietal cortex (PPC). A control condition, in which a region in the frontal cortex instead received the stimulation, was also included in the study. Subjects’ sensitivity to shifts was found to be higher for ipsiversive than for contraversive saccades when TMS was applied to the PPC but not the frontal cortex shortly before the saccade. Chang and Ro suggested that spatial processing and, in particular, remapping in the PPC may have been disrupted at the site of stimulation, thus impairing shift detection following contraversive saccades. However, evidence for PPC’s role in visual stability should not be taken to show that remapping, in particular, is involved; alternative processes taking place within region may have driven the effect.14 Ostendorf, Kilias, and Ploner (2012) also employed TMS, applying stimulation to the frontal eye fields. This appeared to impair the detection of certain transsaccadic stimulus shifts, providing some evidence of this structure’s role in transsaccadic processing but, once again, not exclusively implicating remapping.15
After examining evidence concerning the potential role of remapping in stability, we will now address the phenomenon in the context of transsaccadic memory integration (see Prime, Vesia, & Crawford, 2011 for relevant review). Interestingly, Prime, Vesia, and Crawford found that when TMS was applied to the human right posterior parietal cortex (Prime et al., 2008) or the frontal eye fields (Prime et al., 2010), transsaccadic memory for visual features appeared to be somewhat selectively suppressed. This may, they proposed, result from a stimulation-induced disruption of remapping these regions, although, as noted above, the effect cannot be attributed with certainty to remapping in particular. How might remapping facilitate transsaccadic memory?
Recall that, in the discussion above, we characterized true spatiotopy as a neural representation with fixed mappings between cells and the external spatial locations to which they respond. Although remapping does not support sustained spatiotopy of this sort, it may give rise to a narrow, ‘transient spatiotopy’ wherein a given cell responds to the same region of external space both immediately before and immediately after a saccade (Burr & Morrone, 2012). This transient spatiotopy could, in principle, facilitate transsaccadic integration by transferring visual feature information associated with particular objects to the correct postsaccadic retinal locations.
As was discussed in the previous section of this article, the evidence for low-level spatiotopic visual integration has been mixed. Furthermore, even the evidence that does support this notion—e.g., positive findings of spatiotopic motion integration—cannot be attributed to remapping in particular; instead, stable, spatiotopic maps may underlie these effects. However, findings that show certain presaccadic perceptual anomalies may provide more convincing evidence for the involvement of remapping. Examples of such findings include the result that remapping may lead to pre-saccadic visual crowding (Harrison, Retell, Remington, & Mattingley, 2013) as well as masking effects (Hunt & Cavanagh, 2011).
In one influential study of presaccadic effects, a tilt aftereffect was induced among subjects immediately before they performed a saccade (Melcher, 2007). The tilt aftereffect is a negative aftereffect wherein, following adaptation to a tilted stimulus, the perception of subsequently presented stimuli is biased in the opposite direction (Gibson, 1937; Gibson & Radner, 1937). In the first experiment of the study, the adapting stimulus was presented at the fovea. A saccadic cue then appeared at the periphery. While the subject was preparing the eye movement, a test stimulus was, in one condition, briefly flashed at the saccadic landing site. Since the eye movement had not yet been executed, the test stimulus appeared at a distant retinal location from the adapting stimulus. During steady fixation, the tilt aftereffect was retinotopically constrained to the region immediately surrounding the adaptation site (Melcher, 2007). Nonetheless, prior to making the saccade, subjects exhibited a fairly robust tilt aftereffect for the stimulus presented at the saccadic target location, peaking about 150 ms before the saccade was executed. At the same time, interestingly, the aftereffect for stimuli presented at the fovea—the retinal location previously occupied by the adaptor—declined from previous levels. Biber and Ilg (2011) reported similar findings using adaptation to motion rather than tilted gratings. These findings have been proposed as evidence that predictive remapping affects perceptual experience.
Recently, however, this conception of remapping has been questioned (see Cavanagh, Hunt, Afraz, & Rolfs, 2010; Rolfs, Jonikaitis, Deubel, & Cavanagh, 2011).16 Cavanagh, Rolfs and colleagues argued that visual feature information is, in fact, unlikely to be remapped before a saccade. Instead, they proposed, ‘attention pointers’ likely shift to new retinotopic locations prior to the eye movement to reflect the correct, post-saccadic locations of objects of interest in the scene (see also Bays & Husain, 2007; Quaia et al., 1998, Wurtz, 2008). Afferent visual input is thought, on this account, to be processed in the standard manner, with cells in feature-processing regions of the brain responding to stimuli present in their classic receptive fields. In regions that largely represent not features, per se, but the salience of particular retinotopic locations, however, areas of interest are predictively updated before the saccade. This view is consistent, they argued, with findings that remapping occurs largely in structures such as the lateral intraparietal area, frontal eye fields, and superior colliculus that are thought to contain salience maps of the scene, marking important locations and potential targets for upcoming saccades (see, e.g., Munoz, 2002; Schall, 1995; Findlay & Gilchrist, 2003).
Furthermore, Rolfs et al. (2011) recently argued that the findings reported in Melcher (2007; first experiment; see also Mathôt & Theeuwes, 2010, Experiments 2 and 317) do not reflect remapping as it is currently understood.18 The result, they maintained, relies on a transfer of activity that is reversed in direction with respect to standard remapping. In the original study, an aftereffect produced at the fovea appeared to exert a presaccadic effect on the perception of a stimulus at the saccadic target site. One explanation for this effect, the idea that the foveal adaptation state was remapped to the presaccadic retinal locus of the target, would, as Rolfs et al. maintained, reflect a reversal of the known mechanism of remapping. In standard remapping, cells that encompass a given external spatial location in their classic receptive fields—here, cells at the fovea—are thought to transfer activity to neurons that will encompass the same external location after the saccade is complete. These neurons lie at the retinal location opposite to those at saccadic target. If, for instance, a saccade were planned 5 ° to the right, foveal cells would transfer their activity to those lying 5 ° to the left of fixation. These are the neurons that will encompass the region currently aligned with the fovea after the saccade is complete. An alternative explanation for the finding—one that is consistent with standard remapping—could, however, be conceived. On this account, adaptation-related activity at the fovea would remain in place while afferent visual information from the saccadic target was remapped through these gain-adapted, foveal channels. Though this explanation would agree with prior research on the direction of remapping, it would not, interestingly, seem to explain prior results of spatiotopic aftereffects for stimuli that remain stationary in external space.
In order to examine their own theory of remapping, in which salience markers are shifted to the proper postsaccadic locations before the eye movement to support efficient selection upon refixation, Rolfs et al. (2011) employed a ‘double step’ task (Hallett & Lightstone, 1976; Becker & Jürgens, 1979). This requires subjects to make two successive saccades based upon visual cues, both presented in sequence before the eyes began to move. Previous findings suggested that perceptual processing is enhanced at the site of a saccadic target (e.g., Kowler et al., 1995; Deubel & Schneider, 1996). Rolfs et al., however, found that, before the first eye movement, the retinal location that would be occupied by the second target after completion of the first saccade seemed to receive attentional enhancement. Perceptual judgments about stimuli at this retinal location were more accurate than at a control location. This finding is, they maintained, consistent with the view that remapping realigns peaks of attention to the proper, updated retinal locations prior to eye movements, thus enhancing processing at these locations (see also Jonikaitis, Szinte, Rolfs, & Cavanagh, 2013).
However, spatiotopic updating of attended locations with saccades appears to be neither obligatory nor immediate (Golomb, Chun, & Mazer, 2008). Golomb et al. (2008) measured covert attentional facilitation after a saccade at a location that matched a pre-saccadic stimulus either in retinotopic or spatiotopic coordinates. When a behavioral task required tracking the spatiotopic location of the stimulus, they found evidence of facilitation (e.g., faster response times when compared with a control location) at both spatially and retinally congruent locations. Very shortly after the saccade, this facilitation was stronger at the retinally congruent location. At longer delays, facilitation at the spatially congruent location grew dominant. These results show that while predictive remapping may begin to realign peaks of attention to their proper, post-saccadic locations before the eye movement begins (as argued in Rolfs et al., 2011), this remapping process appears to continue for some duration after the saccade is complete, and to leave a ‘retinotopic trace’ of the former retinal location of a cue even after refixation (see also Golomb, Marino, Chun, & Mazer, 2011; Jonikaitis et al., 2013; Mathôt & Theeuwes, 2010, Experiment 1; Talsma et al., 2013; but see Pertzov, Zohary, & Avidan, 2010 for a finding of early spatiotopic effects even for behaviorally irrelevant stimuli).
A final proposal to consider is that remapping contributes to the planning of visually guided actions. Bays and Husain (2007) proposed this view based, in part, on the findings of Henriques, Klier, Smith, Lowy, and Crawford (1998). The study provided evidence that when pointing toward a remembered stimulus, visible only during a previous fixation, subjects may have relied on a signal that was first remapped to the expected, postsaccadic retinal location and only subsequently localized within a supraretinal reference frame when this information was needed to direct the pointing gesture. If this theory is correct, note that remapping does not actually perform the necessary conversion of locations between reference frames; this essential transformation remains to be accomplished by some other mechanism. One important exception, however, is the special case when the action to be performed is itself an eye movement. Because saccade targets are specified in retinotopic coordinates, remapping could, in this case, provide accurate target location information in the required reference frame for motor output. This may be useful when planning sequences of saccades (Wurtz, Joiner, et al., 2011). Suppose that two saccades are to be performed in short succession. The retinotopic location of the second target will, of course, be different after the first saccade is complete than before any movement began. Remapping could provide the system with earlier access to this updated retinotopic location than if reafferent visual information were required after the first saccade (see Quaia et al., 1998 for a computational model). Indeed, Sommer and Wurtz (2002) found that disrupting the corollary discharge signal thought to drive remapping decreased targeting accuracy for the second saccade, but not the first, when two eye movements were performed in sequence, perhaps suggesting a role for remapping in localizing this second target (see also Berman, Heiser, Dunn, Saunders, & Colby, 2007).19
In the preceding paragraphs, we provided a selective review of proposals about how remapping may contribute to transsaccadic perception. In our view, the idea that remapping serves to realign salience maps with the proper retinotopic locations upon each new fixation is currently among the most plausible of these accounts. This is in part because the regions in which remapping effects are most robust are largely understood to encode such maps, selecting particular regions of the scene for enhanced processing and designating a subset of these as the targets of upcoming saccades. However, direct empirical evidence for this claim, while suggestive, is not yet overwhelming. Some other proposals appear to fare less well under careful scrutiny. In particular, the idea that remapping creates a stable, spatiotopic map across eye movements does not appear to be theoretically sound. A distinct proposal, which holds that remapping helps to annul the eye movement by predicting postsaccadic activation given a stable outside world is, however, logically tenable. Nonetheless, two caveats should be noted. First, remapping cannot, by itself, perform this computation because it does not compare predicted and actual activation. Second, and more crucially, the real-world importance of this sort of computation in maintaining visual stability across eye movements appears to be quite limited, given a substantial body of classic research on the topic. From a purely theoretical standpoint, remapping may also be a suitable neural substrate for fusing visual features in a fleetingly spatiotopic manner across eye movements. However, there is currently some debate in the literature as to whether this actually occurs.
Finally, the possibility that the neural phenomenon of predictive remapping plays no functional role in the experience of transsaccadic perception, while rather unsatisfying, should also be given fair consideration, as the evidence linking these neural changes with perceptual experience is, in our view, not yet overwhelming.
In the first sections of this article, we reviewed seminal findings about how we perceive a stable world and integrate information across eye movements. In both domains, conceptually straightforward (but, in some cases, seemingly computationally demanding) solutions were proposed. These solutions were tasked, effectively, with ‘undoing’ the eye movement in one way or another, so as to create the sort of representation that might arise more naturally from a system in which the eyes remained fixed at all times. For example, the retinal shift induced by each saccade was thought to be ‘cancelled’ by an equal and opposite corollary discharge signal. Further, saccadic suppression was proposed to arise from a transient period of ‘central anesthesia’ that would prohibit all visual processing during eye movements. Finally, an integrative visual buffer was proposed to fuse detailed traces of retinal images across eye movements in a spatiotopic manner.
Each of these proposals appears to contain an element of truth. Nonetheless, in light of subsequent research, each also appears incomplete—insufficient to account for the full range of data. For example, in most real-world situations, we do not appear to carry out a cancellation computation following each saccade. In addition, while selective, active neural suppression may contribute to our failure to see motion blur during saccades, other factors such as backward masking are also fundamentally important. Finally, while we do clearly retain information from prior fixations and, indeed, use it to speed subsequent processing (as we see from extrafoveal preview benefits), it does not appear that this process is generally driven by low-level, highly detailed spatiotopic fusion. In the domains of both phenomenal stability and integration, we see evidence that challenges raised by the saccadic nature of the visual system are met not with a single mechanism for ‘undoing’ our eye movements, but rather with a flexible system of multiple ‘strategies’. As we see in many other complex domains of processing, there is evidence that, where possible, simple heuristics are used to find approximate—and often entirely acceptable—solutions. For example, the system appears to take visual stability across saccades as a working assumption, only to be overturned in the face of overwhelming evidence to the contrary. Indeed, given the low probability stimulus shifts during saccades, this seems the rational approach. Furthermore, we see evidence of the flexible use of ‘strategies’ depending upon the nature of the task (and perhaps stimulus) involved; while purely visual preview benefits do arise during pictorial object recognition, this does not appear to be the case during reading.
With the discovery of predictive remapping, this understanding of transsaccadic perception as the product of a flexible and redundant set of multiple strategies may have been overshadowed, to some degree, by these exciting new findings from neuroscience. This may be partly because remapping has some of the properties once sought in a mechanism that could ‘annul’ a saccade. Specifically, remapping entails a shift of neural activity with each saccade that seems to depend on a corollary discharge signal. This correspondence may, at times, have caused certain important complications to be overlooked. Examples of such complications include: (1) the theoretical point that remapping is not an appropriate transformation for creating a sustained, spatiotopic map, and (2) empirical findings suggesting that we may not typically create (or, in any case, use) a detailed, spatiotopic representation of the visual world from one fixation to the next. In several recent articles, however, these and related points appear to be garnering more attention (see, e.g., Burr & Morrone, 2011; Wurtz, Joiner, et al., 2011). These developments will, we believe, support a return to a more complex understanding of the mechanisms of transsaccadic perception, now supplemented by a careful consideration of the potential contributions of predictive remapping (or other neural changes) that accompany saccades.
The problem of displacement, however, will be covered in more detail than the problem of motion. This is because displacement suppression is more closely linked with the remaining topics to be addressed in this article—integration and remapping—than is motion suppression.
This term is used somewhat inconsistently in the literature. It is sometimes used fairly narrowly to describe only active, neural suppression of visual input. Following Matin (1974), however, we use it more broadly to describe our failure to perceive the world during saccades, however this may be achieved.
Another potential extraretinal signal of the eye movement is proprioceptive feedback from the extraocular muscles as they perform the actual eye movement. However, this factor will not be discussed until the next section, which concerns displacement suppression.
The further question of how we might track the shift of the image on the retina across the eye movement, however, also seems non-trivial.
As evidence for the second component of their theory, they point to the finding that shifts of the target alone were found to be more noticeable than shifts of the target and background as a coherent unit.
Why, however, was improved performance in the blanking condition attributed to reliance on corollary discharge? Instead, it seems, blanking might somehow trigger a more careful evaluation of the target location, when it reappears, relative to the fovea; the improved performance may arise from an elevated sensitivity to visual error. Deubel et al. argued against this view on the basis that the target displacement vector was a better predictor of displacement judgments than was visual error. For example, if the saccade overshot the original target location on the same trial in which the target was displaced slightly in the forward direction, visual error might be near zero. However, subjects would be likely, nonetheless, to report the forward shift if a brief blank period was inserted upon refixation (see also Collins, Rolfs, Deubel, & Cavanagh, 2009).
Note that the focus here will mainly be on memory and integration across a single fixation-saccade-fixation cycle rather than the gradual accumulation of information over an extended viewing period.
Note, however, that in one of these studies there was a significant effect of spatial congruity at one of the time windows in the saccade condition. Because it did not replicate in a subsequent experiment, however, Irwin suggested that this finding was likely due to Type 1 error.
Interestingly, however, there is some evidence that transsaccadic image shifts along the axis of the saccade may be particularly difficult to detect. Verfaillie, De Troy, and Van Rensbergen (1994) compared detection rates for different sorts of changes in a point light figure across a saccade or a blank interval within a single fixation. In one condition, the figure was translated laterally while in another its postural position changed. Relative change detection performance across the saccade (compared with during fixation) was lower for the shift condition than the postural change condition. In addition, Crapse & Sommer (2010) found greater sensitivity to shifts that were perpendicular to saccades than to those that were parallel to the eye movements. Both findings are consistent with the notion that some degree of uncertainty about position along saccade axis may be expected and accounted for in the system.
Note, however, that in an object-naming task, differences in form among members of the same category are not relevant for successful performance (since the correct name is the same). Thus, the theory would require that sensitivity to such differences must ‘carry over’ from real-world scene viewing experience where such differences may be more consequential.
Some recent neuroimaging evidence suggests that MT (the medial temporal region) might contain spatiotopic receptive fields, a potential neural substrate for this effect (Crespi et al., 2011; d'Avossa, Tosetti, Crespi, Biagi, Burr, & Morrone, 2006). However, this finding has also been contested (Gardner, Merriam, Movshon, & Heeger, 2008; Golomb and Kanwisher, 2012). Interestingly, Ong and Bisley (2011) also failed to find evidence of predictive remapping responses, a topic to be discussed below, in MT.
Interestingly, there was no clear relationship in this study between the location of the lesion and the saccade direction for which impairments emerged. Furthermore, note that in Ostendorf et al. (2010), saccades that were ipsiversive to the lesion were affected. Given the findings of Sommer and Wurtz (2002, 2006), this effect might be expected to be lateralized, affecting only saccades that were contraversive to the lesion. As Ostendorf et al. pointed out, however, there is also some precedent in the literature for ipsiversive effects in some MD-damaged patients (see, e.g., Bellebaum, Daum, Koch, Schwarz, & Hoffmann, 2005).
Indeed, the PPC has been implicated in transsaccadic spatial cognition independently of any contribution of remapping. Andersen and colleagues (Andersen & Mountcastle, 1983; Zipser & Andersen, 1988), for instance, observed that cells in area 7a of the posterior parietal cortex of monkeys were retinotopic but that the gain of cellular responsiveness in this area was modulated by eye position. The cells were retinotopic in the sense that, for each cell, a Gaussian responsitivity curve was centered at a particular, preferred location on the retina. The gain of these cells was modulated, however, by eye position such that response amplitude changed (in an approximately linear fashion) depending upon the direction of gaze (Salinas & Abbott, 2001). “Thus”, Zipser and Andersen (1988) explained, “the activity of the cell for retinopically identical stimuli varied as a function of the angle of gaze” (p. 491). Such a map could, as Bridgeman et al. (1994) pointed out, in principle, be sampled at the population level to extract spatiotopic location information.
Interestingly, using single cell recording, Crapse and Sommer (2012) found that some cells in FEF appeared to have response properties that would enable them to serve as indicators of a stimulus’ stability or instability across a saccade. They found evidence of this response property, which they call ‘translation tuning’, both among cells that exhibited remapping and those that did not, though the proportion of tuned cells was greater among remapping cells (42 %) than non-remapping cells (29 %). This suggests that, if their speculation about the function of these cells is correct, and they serve to indicate whether stimuli remained at the same spatial location across the saccade, remapping may not be a necessary contributor.
Interestingly, Harrison, Mattingley, and Remington (2012) found consistent effects with those reported in Mathôt and Theeuwes (2010) providing additional evidence that the reported effect – namely, attentional benefits prior to a saccade at a location shifted from a cued location in the same direction as eye movement – may indeed be a reliable effect. However, such effects do not appear to result from predictive remapping as it is currently understood.
This objection would also seem to apply to the subsequent results of Biber and Ilg (2011).
See Van Grootel, Van der Willigen, and Van Opstal (2012), however, for data suggesting that dynamic motor feedback, as opposed to a fixed, pre-saccadic signal of the planned initial saccade, may best account for the movement vector selected for the second saccade.
Preparation of this review was made possible by funds from the Atkinson Endowment at UCSD. We would like to thank Karen Dobkins, Julie Golomb, David Irwin, Don MacLeod, Martin Rolfs, and Jeremy Wolfe for their helpful comments on previous drafts. This article was based on a qualifying paper submitted in partial fulfillment of the requirements for doctoral candidacy at UCSD.
- Anstis, S. (1978). Apparent movement. In R. Held, H. Leibowitz, & H.-L. Teuber (Eds.), Handbook of sensory physiology VIII: Perception (pp. 655–673). Berlin: Springer-Verlag.Google Scholar
- Bridgeman, B. (1979). Adaptation and the two-visual-systems hypothesis. Behavioral and Brain Sciences, 2(01), 64–65.Google Scholar
- Bridgeman, B., & Mayer, M. (1983). Failure to integrate visual information from successive fixations. Bulletin of the Psychonomic Society, 21(4), 285–286.Google Scholar
- Bridgeman, B., Van der Heijden, A., & Velichkovsky, B. M. (1994). A theory of visual stability across saccadic eye movements. Behavioral and Brain Sciences, 17(2), 247–257.Google Scholar
- Brooks, B. A., Impelman, D. M. K., & Lum, J. T. (1981). Backward and forward masking associated with saccadic eye movement. Perception & Psychophysics, 30(1), 62–70.Google Scholar
- Crapse, T., & Sommer, M. (2010). Translation of a visual stimulus during a saccade is more detectable if it moves perpendicular, rather than parallel, to the saccade. Journal of Vision, 10(7), 521.Google Scholar
- Currie, C. B., McConkie, G. W., Carlson-Radvansky, L. A., & Irwin, D. E. (2000). The role of the saccade target object in the perception of a visually stable world. Perception & Psychophysics, 62(4), 673–683.Google Scholar
- Davidson, M. L., Fox, M. J., & Dick, A. (1973). Effect of eye movements on backward masking and perceived location. Perception & Psychophysics, 14(1), 110–116.Google Scholar
- Demeyer, M., De Graef, P., Wagemans, J., & Verfaillie, K. (2009). Transsaccadic identification of highly similar artificial shapes. Journal of Vision, 9(4), 28.1. 14.Google Scholar
- Dennett, D. C. (1991). Consciousness explained. Boston: Little, Brown and Co.Google Scholar
- Descartes, R. (1972). Treatise of man (Thomas Steele Hall, Trans.) Cambridge, MA: Harvard University Press. (Original work published in 1664).Google Scholar
- Dodge, R. (1900). Visual perception during eye movement. Psychological Review, 7(5), 454.Google Scholar
- Erdmann, B., & Dodge, R. (1898). Psychologische Untersuchungen über das Lesen auf experimenteller Grundlage. Halle: Niemeyer.Google Scholar
- Ezzati, A., Golzar, A., & Afraz, A. S. R. (2008). Topography of the motion aftereffect with and without eye movements. Journal of Vision, 8(14).Google Scholar
- Feldman, J. A. (1985). Four frames suffice: A provisional model of vision and space. Behavioral and Brain Sciences, 8(02), 265–289.Google Scholar
- Fernandez-Duque, D., & Thornton, I. M. (2000). Change detection without awareness: Do explicit reports underestimate the representation of change in the visual system? Visual Cognition, 7(1–3), 323–344.Google Scholar
- Findlay, J. M., & Gilchrist, I. D. (2003). Active vision: The psychology of looking and seeing. Oxford, UK: Oxford University Press.Google Scholar
- Fracasso, A., Caramazza, A., & Melcher, D. (2010). Continuous perception of motion and shape across saccadic eye movements. Journal of Vision, 13, 14. 1-17.Google Scholar
- Gibson, J. J. (1937). Adaptation with negative after-effect. Psychological Review, 44(3), 222.Google Scholar
- Gibson, J. J., & Radner, M. (1937). Adaptation, after-effect and contrast in the perception of tilted lines. I. quantitative studies. Journal of Experimental Psychology, 20(5), 453.Google Scholar
- Golomb, J. D., & Kanwisher, N. (2012). Higher level visual cortex represents retinotopic, not spatiotopic, object location. Cerebral Cortex, bhr357.Google Scholar
- Golomb, J. D., Marino, A. C., Chun, M. M., & Mazer, J. A. (2011). Attention doesn’t slide: spatiotopic updating after eye movements instantiates a new, discrete attentional locus. Attention, Perception, & Psychophysics, 73(1), 7–14.Google Scholar
- Grimes, J. (1996). On the failure to detect changes in scenes across saccades. In K. Akins (Ed.), Perception. Vancouver studies in cognitive science, Vol. 5 (pp. 89–110). New York, NY: Oxford University Press.Google Scholar
- Grüsser O. J. (1995). On the history of the ideas of efference copy and reafference. 33:35–55.Google Scholar
- Henderson, J. M. (1994). Two representational systems in dynamic visual identification. Journal of Experimental Psychology: General, 123(4), 410.Google Scholar
- Henderson, J. M., Dixon, P., Petersen, A., Twilley, L. C., & Ferreira, F. (1995). Evidence for the use of phonological representations during transsaccadic word recognition. Journal of Experimental Psychology: Human Perception and Performance, 21(1), 82.Google Scholar
- Henderson, J. M., & Hollingworth, A. (1999). The role of fixation position in detecting scene changes across saccades. Psychological Science, 10(5), 438–443.Google Scholar
- Henderson, J. M., & Hollingworth, A. (2003). Eye movements and visual memory: Detecting changes to saccade targets in scenes. Perception & Psychophysics, 65(1), 58–71.Google Scholar
- Henderson, J. M., & Siefert, A. B. C. (1999). The influence of enantiomorphic transformation on transsaccadic object integration. Journal of Experimental Psychology: Human Perception and Performance, 25(1), 243.Google Scholar
- Henderson, J. M., & Siefert, A. B. C. (2001). Types and tokens in transsaccadic object identification: Effects of spatial position and left-right orientation. Psychonomic Bulletin & Review, 8(4), 753–760.Google Scholar
- Hoffman, J. E., & Subramaniam, B. (1995). The role of visual attention in saccadic eye movements. Perception & Psychophysics, 57(6), 787–795.Google Scholar
- Hollingworth, A., Richard, A. M., & Luck, S. J. (2008). Understanding the function of visual short-term memory: Transsaccadic memory, object correspondence, and gaze correction. Journal of Experimental Psychology: General, 137(1), 163.Google Scholar
- Holt, E. B. (1903). Eye-movement and central anaesthesia. The Psychological Review: Monograph Supplements, 4(1), 1–45.Google Scholar
- Irwin, D. E. (1992). Memory for position and identity across eye movements. Journal of Experimental Psychology: Learning, Memory, and Cognition, 18(2), 307.Google Scholar
- Irwin, D. E. (1996). Integrating information across saccadic eye movements. Current Directions in Psychological Science, 5(3), 94–100.Google Scholar
- Irwin, D. E., Brown, J. S., & Sun, J. (1988). Visual masking and visual integration across saccadic eye movements. Journal of Experimental Psychology: General, 117(3), 276–287.Google Scholar
- Irwin, D. E., & Gordon, R. D. (1998). Eye movements, attention and trans-saccadic memory. Visual Cognition, 5(1–2), 127–155.Google Scholar
- Irwin, D. E., Yantis, S., & Jonides, J. (1983). Evidence against visual integration across saccadic eye movements. Perception & Psychophysics, 34(1), 49–57.Google Scholar
- Irwin, D. E., Zacks, J. L., & Brown, J. S. (1990). Visual memory and the perception of a stable visual environment. Perception & Psychophysics, 47(1), 35–46.Google Scholar
- Jones, L. A., & Higgins, G. C. (1947). Photographic granularity and graininess. JOSA, 37(4), 217–258.Google Scholar
- Latour, P. (1962). Visual threshold during eye movements. Vision Research, 2(7–8), 261–262.Google Scholar
- Levin, D. T., Momen, N., Drivdahl, S. B., & Simons, D. J. (2000). Change blindness blindness: The metacognitive error of overestimating change-detection ability. Visual Cognition, 7(1–3), 397–412.Google Scholar
- Li, W., & Matin, L. (1990). The influence of saccade length on the saccadic suppression of displacement detection. Perception & Psychophysics, 48(5), 453–458.Google Scholar
- Mathôt, S., & Theeuwes, J. (2013). A reinvestigation of the reference frame of the tilt-adaptation aftereffect. Scientific Reports, 3, e1152.Google Scholar
- Matin, L., Matin, E., & Pearce, D. G. (1969). Visual perception of direction when voluntary saccades occur. I. relation of visual direction of a fixation target extinguished before a saccade to a flash presented during the saccade. Perception & Psychophysics, 5(2), 65–80.Google Scholar
- McConkie, G. W., & Rayner, K. (1976). Identifying the span of the effective stimulus in reading: Literature review and theories of reading. Theoretical Models and Processes of Reading. In H. Singer & R. B. Ruddell (Eds.), Theoretical models and processes in reading (pp. 137–162). Newark, DE: International Reading Association.Google Scholar
- McConkie, G. W., & Zola, D. (1979). Is visual information integrated across successive fixations in reading? Perception & Psychophysics, 25(3), 221–224.Google Scholar
- Nakamura, K., & Colby, C. L. (2002). Updating of the visual representation in monkey striate and extrastriate cortex during saccades. Proceedings of the National Academy of Sciences, 99(6), 4026.Google Scholar
- Neisser, U. (1967). Cognitive psychology. Englewood Cliffs, NJ: Prentice-Hall.Google Scholar
- O'Regan, J. K. (1992). Solving the" real" mysteries of visual perception: The world as an outside memory. Canadian Journal of Psychology/Revue Canadienne De Psychologie, 46(3), 461.Google Scholar
- Ostendorf, F., Liebermann, D., & Ploner, C. J. (2010). Human thalamus contributes to perceptual stability across eye movements. Proceedings of the National Academy of Sciences, 107(3), 1229–1234.Google Scholar
- Ostendorf, F., Liebermann, D., & Ploner, C. J. (2013). A role of the human thalamus in predicting the perceptual consequences of eye movements. Frontiers in Systems Neuroscience, 7.Google Scholar
- Pollatsek, A., Rayner, K., & Collins, W. E. (1984). Integrating pictorial information across eye movements. Journal of Experimental Psychology: General, 113(3), 426.Google Scholar
- Rayner, K. (1975). The perceptual span and peripheral cues in reading. Cognitive Psychology, 7(1), 65–81.Google Scholar
- Rayner, K. (1978). Foveal and parafoveal cues in reading. In J. Requin (Ed.), Attention and performance VII. Hillsdale, NJ: Erlbaum.Google Scholar
- Rayner, K., & Pollatsek, A. (1983). Is visual information integrated across saccades? Perception & Psychophysics, 34(1), 39–48.Google Scholar
- Rensink, R. A., O'Regan, J. K., & Clark, J. J. (1997). To see or not to see: The need for attention to perceive changes in scenes. Psychological Science, 8(5), 368–373.Google Scholar
- Richards, W. (1969). Saccadic suppression. JOSA, 59(5), 617–623.Google Scholar
- Rock, I., & Ebenholtz, S. (1962). Stroboscopic movement based on change of phenomenal rather than retinal location. The American Journal of Psychology, 193-207.Google Scholar
- Ross, J., Morrone, M. C., & Burr, D. C. (1997). Compression of visual space before saccades. Nature, 386(6625), 598–601.Google Scholar
- Schiller, P. H. (1998). The neural control of visually guided eye movements. In J. Richards (Ed.), Cognitive neuroscience of attention: A developmental perspective (pp. 3–50). New Jersey: Lawrence Erlbaum Associates.Google Scholar
- Schotter, E. R., Angele, B., & Rayner, K. (2012). Parafoveal processing in reading. Attention, Perception, & Psychophysics, 74, 5–35.Google Scholar
- Schotter, E. R., Ferreira, V. F., & Rayner, K. (2013). Parallel object identification and attentional gating of information: Evidence from eye movements in the multiple object naming paradigm. Journal of Experimental Psychology: Learning, Memory, and Cognition, 39, 649–662.Google Scholar
- Sherrington, C. (1918). Observations on the sensual role of the proprioceptive nerve-supply of the extrinsic ocular muscles. Brain, 41(3–4), 332–343.Google Scholar
- Simons, D. J., & Ambinder, M. S. (2005). Change blindness theory and consequences. Current Directions in Psychological Science, 14(1), 44–48.Google Scholar
- Skavenski, A. A., Haddad, G., & Steinman, R. M. (1972). The extraretinal signal for the visual perception of direction. Perception & Psychophysics, 11(4), 287–290.Google Scholar
- Uttal, W. R., & Smith, P. (1968). Recognition of alphabetic characters during voluntary eye movements. Perception & Psychophysics, 3(4), 257–264.Google Scholar
- Volkmann, F. C. (1962). Vision during voluntary saccadic eye movements. JOSA, 52(5), 571–577.Google Scholar
- Volkmann, F. C., Schick, A., & Riggs, L. A. (1968). Time course of visual inhibition during voluntary saccades. JOSA, 58(4), 562–569.Google Scholar
- von Helmholtz , H. (1962). Treatise on physiological optics. (James Southall, Trans.) New York, NY: Dover. (Original work published in 1867).Google Scholar
- von Holst, E., & Mittelstaedt, H. (1971). The principle of reafference: Interactions between the central nervous system and peripheral organs. In P. C. Dodwell (Ed.), Perceptual processing: Stimulus equivalence and pattern recognition (pp. 41-72). New York: Appleton. (Original work published 1950).Google Scholar
- Woodworth, R. (1906). Vision and localization during eye movements. Psychological Bulletin, 3, 68–70.Google Scholar
- Wurtz, R. H. (1969). Comparison of effects of eye movements and stimulus movements on striate cortex neurons of the monkey. Journal of Neurophysiology, 32(98), 994.Google Scholar
- Wurtz, R. H. (2009). Superior Colliculus. In Encyclopedia of neuroscience (pp. 627–634). Elsevier Ltd.Google Scholar