Experimental Brain Research

, 215:13

Trans-saccadic processing of visual and motor planning during sequential eye movements

  • Supriya Ray
  • Neha Bhutani
  • Vishal Kapoor
  • Aditya Murthy
Research Article

DOI: 10.1007/s00221-011-2866-x

Cite this article as:
Ray, S., Bhutani, N., Kapoor, V. et al. Exp Brain Res (2011) 215: 13. doi:10.1007/s00221-011-2866-x

Abstract

How the brain maintains perceptual continuity across eye movements that yield discontinuous snapshots of the world is still poorly understood. In this study, we adapted a framework from the dual-task paradigm, well suited to reveal bottlenecks in mental processing, to study how information is processed across sequential saccades. The pattern of RTs allowed us to distinguish among three forms of trans-saccadic processing (no trans-saccadic processing, trans-saccadic visual processing and trans-saccadic visual processing and saccade planning models). Using a cued double-step saccade task, we show that even though saccade execution is a processing bottleneck, limiting access to incoming visual information, partial visual and motor processing that occur prior to saccade execution is used to guide the next eye movement. These results provide insights into how the oculomotor system is designed to process information across multiple fixations that occur during natural scanning.

Keywords

Concurrent processing Eye movements Attention Saccadic suppression 

Introduction

Fast ballistic eye movements called saccades have evolved to counter the design of our visual system, where high acuity is largely limited to the central region of the retina. Such architecture necessitates the production of saccades to foveate objects of interest in the visual scene for further examination. Since visual processing largely takes place in retinal coordinates, the coordinates of all objects shift dramatically with each saccadic eye movement (Melcher and Colby 2008). This poses a considerable challenge on the visual system to integrate spatial information across saccades. Alternate views have been put forth to explain the apparent stability of vision despite eye movements. At one end, it is hypothesized that there are mechanisms that enable the integration of spatial information across saccades (Jonides et al. 1982; Irwin 1991; Pollatsek et al. 1990). Evidence for this derives from the finding of receptive fields in the visual system that are either head centered (craniotopic; Galletti et al. 1993) or in external spatial coordinates (spatiotopic; d’Avossa et al. 2007; Duhamel et al. 1997) rather than in retinal coordinates; or cells that remap spatial information into coordinates that take into account the impending eye movement (Berman et al. 2007; Duhamel et al. 1992); or of a visual short-term memory that operates across saccades (Hayhoe et al. 1992; Irwin et al. 1990; Prime et al. 2007). In contrast, another view posits that little or no visual information is maintained across saccades: Vision effectively begins anew after each eye movement (O’Regan and Noe 2001; Rensink 2000). Evidence for this point of view comes from failure to detect changes in complex scenes, when these changes occur during eye movements.

While much of the work on trans-saccadic processing of spatial information has been motivated to explain the apparent stability of vision despite eye movements, and involves measuring perceptual benefits of presaccadic information, matching an object across saccades is also crucial for forming a decision and guiding motor actions. The study of trans-saccadic vision for action provides a novel and complementary approach to study the nature of trans-saccadic information processing since actions typically involve transforming the location of stimuli into a representations suitable for movement planning to occur. Here, we investigate whether the visual information of a stimulus available before the saccade can be used to plan a subsequent saccadic movement after the saccade. This requires that the brain must update the location of objects that are represented in retinal coordinates into a stable frame of reference, which can be integrated across the saccade. This transformation is essential because presaccadic and postsaccadic retinal coordinates are not aligned (see Fig. 1). In this study, we test among alternative hypotheses regarding the time course of this putative transformation. In one case, we hypothesize that this transformation, which marks the beginning of motor planning, can only occur after the identification of the target is complete and after the primary saccade is complete. Here, only visual information concerning the identity of the target is capable of being integrated across the saccade. On the other hand, it is possible that in addition to target identity, target location can be specified and integrated across the saccade. The latter computation would necessarily involve transforming from retinal coordinates to a more stable reference frame (craniocentric, for example) that can be trans-saccadically integrated and mark the beginning of motor planning (for the subsequent saccade) that could occur before and during the primary saccade.
Fig. 1

Schematic showing the visuomotor transformation required for advance preparation of a second saccade vector in a classic double-step task. During fixation, targets T1 and T2 are presented one after another. Subjects are instructed to make a sequence of saccades to these targets in the order of their presentation as soon as possible. While retinal vector V1 is the same as the saccade motor vector M1 required for foveating the first target, retinal vector V2 needs to be eventually transformed to saccade motor vector M2′ for correctly capturing the second target at the end of the first saccade

To understand the nature and the dynamics of trans-saccadic information processing, we used a task that required subjects to produce sequential saccades. We studied the latency of saccades in a framework adopted from the dual-task paradigm, in which two independent tasks are performed in a rapid succession in response to stimuli introduced one after another. When the stimuli for these tasks appear in a close temporal proximity, there is often a delay in performance of one or both of the tasks, called the psychological refractrory period or PRP effect (Welford 1952). The PRP effect has been observed in different manual tasks (e.g., Pashler 1984) and is thought to reflect bottlenecks in information-processing stages. Models that postulate a bottleneck make distinctive predictions. Consider the case when a visual stimulus is flashed during the execution of a saccade. If the visual processing of the stimulus that triggers a subsequent saccade cannot occur because of the blurring of the image caused by the retinal motion during the ongoing saccade (Beeler and George 1967; Mackay 1970; Campbell and Wurtz 1978; Sperling 1990) or by the saccade-induced reduction in the visual sensitivity (Volkmann 1986), the reaction time (RT) of the second saccade should increase (Fig. 2). More specifically, if the execution of the first saccade acts as a bottleneck preventing any visual information processing, then RT should increase by an amount that is proportionate to the interval between the appearance of the stimulus and the first saccade onset, which we refer to as the cue–saccade overlap time or just saccade overlap time (SOT) for simplicity. In other words, the slope of RT plotted against SOT during the execution of a saccade should approach unity.
Fig. 2

Schematic shows three alternate hypotheses concerning how trans-saccadic processing of information may occur in the context of a processing bottleneck during saccadic execution (M1). a If no trans-saccadic processing occurs, then RTs are expected to keep increasing as the cue–saccade overlap time increases. For simplicity, we show two hypothetical data points corresponding to SOTs of duration  M1 ms and M1 + X ms. These points correspond to instances when the cue for the second saccade either begins at the start of M1 or X ms before the start of M1, respectively. b If only visual processing (V2) was to occur trans-saccadically, then RTs are expected to remain constant for +ve cue–saccade overlap times up to a limit set by the duration of the visual stage (V2). c If the trans-saccadic processing of visual information (V2) and saccade planning (P2) were to occur, then RTs are expected to decrease for +ve cue–saccade overlap times up to a limit

Figure 2 describes graphically the logic of using the PRP effect to assess whether the processing of the second saccade reduces in a given interval during the time period in which the first saccade is being prepared and executed. We assume, without loss of generality, that every sensory-motor task consists of at least three processing stages, a visual stage (V), a planning/motor preparation stage (P) and an execution stage (M) (e.g., Sigman and Dehaene 2005). For simplicity, these stages are shown as serial but the same logic can be applied when these stages are cascaded or continuous (McClelland 1979; Meyer et al. 1988). We also assume that the duration of the individual stages is constant. Three alternative hypotheses of the nature of information processing across a saccade are considered. In the first case, where no trans-saccadic processing occurs across the saccade, one expects that RT should increase for targets presented in the period prior to the start of the saccade (Fig. 2a). Figure 2b describes a second hypothetical scenario in which if only visual processing is allowed to occur across the primary saccade, RT of the cued saccade is expected to stay constant as a consequence of parallel processing. In this model, parts of visual information gathered before and after the execution of the primary saccade are integrated trans-saccadically to initiate planning of the subsequent saccade. Therefore, this scenario is expected to generate a plateau in the plot of RT against SOT for the period equivalent to the duration of the visual processing stage of the second saccade. In Fig. 2c, a third hypothetical situation is described in which trans-saccadic processing is permitted for both the visual and planning stages. In this scheme, motor plans of the subsequent saccade based on partial visual information gathered before and after the execution of the primary saccade are integrated to initiate the planning of the next saccade. Chronometrically, this model is different from the previous case by the absence of a plateau. Rather, RTs are expected to gradually decrease with SOT down to a limit as a consequence of processing of the planning stage of the second saccade during the execution of the primary saccade. Such a decrease in RT could, however, be obscured by idiosyncratic variations in the duration of the visual stage. For example, if the visual stage (V1) in the first saccade was to be very long, it stands to reason that the corresponding visual stage (V2) in the second saccade should be at least as long. In the context of serial processing, this implies that for the durations of the cue–saccade overlap time tested, sufficient scope for motor planning to occur may not be available. In such a case, the RT profile will reduce to model 2 (visual trans-saccadic processing). However, if the output of the visual stage is continuously fed into the planning stage (cascade or continuous processing models; see McClelland 1979; Meyer et al. 1988), as is likely to be the case physiologically (Bichot et al. 2001), then some degree of planning is still expected to occur even when the visual stage is somewhat long, and reductions in RT are still expected to occur. Thus, since these three hypotheses provide distinct chronometric signatures, we tested the nature of trans-saccadic processing across saccades by measuring the reaction time of subsequent saccades.

Methods

A total of ten human subjects with normal or corrected vision performed a visually guided saccade task while their eye movements were recorded with their heads stabilized by means of a chin rest with forehead support. Nine subjects participated in Experiment 1, and six subjects participated in Experiment 2; five subjects participated in both experiments. All subjects gave their informed consent in accordance with the institutional human ethics committee of National Brain Research Centre. Subjects received monetary reward for every successful trial, which was marked by an auditory tone that served as a secondary reinforcer during the experiment. TEMPO/VIDEOSYNC software (Reflective Computing, St. Louis, USA) displayed visual stimuli, sampled and stored eye positions and other behavioral parameters. Analog data of eye positions were sampled at 240 Hz with an infrared pupil tracker (ISCAN, Boston, USA) that interfaced with the TEMPO software in real time. All stimuli were presented on a Sony Trinitron 500 GDM monitor (21 inch; 70 Hz refresh rate) placed 57 cm in front of the subject. Stimuli were calibrated for luminance with a Minolta CA-96 colorimeter to measure contrast and ensure that color stimuli were physically isoluminant.

Stimuli

Experiment 1: A pair of identical gray circular disks of 2° diameter was presented to either the left or right and above the fixation spot at an eccentricity of 10° visual angle on a uniform gray background of luminance 10 cd/m2 (see Fig. 3). On half of the trials, either of these stimuli flickered at 18 Hz between two luminance levels around a background luminance of 10 cd/m2 such that the Michelson contrast [(Lmax − Lmin)/(Lmax + Lmin)] was 0.20, Lmax and Lmin being the maximum and minimum luminance of the stimuli, respectively. Once started, modulation of luminance (flicker) of the stimulus continued till the end of the trial.
Fig. 3

Temporal sequence of stimuli used in the cued sequential saccade task. Trials started with the appearance of two identical circular disks displayed horizontally and vertically relative to a central fixation spot. a In non-cued trials, subjects were instructed to make a primary saccade to the horizontal stimulus. b In cued trials, subjects were also instructed to make a primary saccade to the horizontal stimulus. After an unpredictable stimulus onset asynchrony (SOA), one of the two circular disks (horizontal or vertical cue) began to flicker till the end of the trial instructing the subject to make a secondary return saccade to the central fixation spot. Cued and non-cued trials of the same proportion were randomly interleaved. Subjects also performed the cued sequential saccade task in a complementary condition consisting of a vertical primary saccade followed by either a horizontally or vertically cued second saccade

Experiment 2: The size, shape and spatial arrangements of the red stimuli of luminance 6.8 cd/m2 displayed on a 6.6-cd/m2 gray background remained the same as in Experiment 1. On half of the trials, either of these stimuli underwent a single photometric equiluminant change in color from red to green.

Behavioral task

On 50% of all trials, referred to as non-cued trials, after fixation for a random duration that ranged from 300 to 800 ms, subjects made a horizontal saccade to the target that appeared at either the left or right of the central fixation spot. Subjects were encouraged to shift their gazes within 400 ms from the appearance of the target. A postsaccadic fixation time mandated subjects to keep their eyes on the target for 800 ms.

On the remaining trials, called cued trials, subjects made a quick horizontal saccade to the target. The signal that distinguished cued trials from non-cued trials was a rapid continuous modulation in luminance (Experiment 1) or a physically equiluminant modulation in chromaticity (Experiment 2) of either the horizontal target or the accompanying vertical circular disk, after random delay, called the stimulus onset asynchrony (SOA), following the appearance of the initial target. Subjects were instructed to make a second saccade back to the fixation spot as soon as they detected the cue. Trials were aborted when subjects failed to respond to the cue within 1,600 ms. Cued and non-cued trials were randomly interleaved. Stimulus onset asynchrony (SOA) for each trial was randomly selected from a total of ten SOAs, 14 ms apart from each other, spanning the period (180–306 ms) during which the first saccade was likely to occur. In cued trials, no compulsory fixation period after the first saccade was imposed. Cued trials were further divided into two conditions depending on whether the vertical or horizontal stimulus was cued, which varied unpredictably across trials. In addition, four subjects were reexamined when they were asked to make a primary saccade to a target either 10° vertically up or down from the central fixation spot accompanied by an identical stimulus at 10° right to fixation spot; all other stimulus parameters remained the same as in Experiment 1.

Data analysis

Blinks were removed from the filtered analogue data of eye positions. A velocity threshold of 30°/s was used to demarcate the beginning and end of saccades. The saccade detection algorithm was subsequently verified manually for every saccade. All trials producing blink-perturbed saccades were eliminated from the analyses. All offline and statistical analyses were done using Matlab and SPSS.

Results

Figure 3 illustrates the cued double-step task used in the present study (see “Methods”). Subjects were instructed to make a 10° saccade to the target that appeared to the left or right of the central fixation spot on every trial. In some randomly selected trials, a visual cue appeared either at the end point of the primary saccade or orthogonal to the direction of the saccade following the appearance of the target. Subjects were instructed to make a return saccade to the fixation spot in response to the cue. Analogous to the two-alternative-forced choice paradigm, this enabled us to calculate the rate of false alarms made by each subject during the task. False alarm rate in each session was measured by calculating the proportion of erroneous non-cued trials when subjects made return saccades with latency shorter than the usual latency of primary saccades. In practice, the ninety-fifth percentile of the saccade latencies in correct non-cued trials was considered as the limit to ensure that the erroneous return saccades with shorter latencies in incorrect non-cued trials were preplanned. Low false alarm rates (mean ± SD = 3.3 ± 2.5%) suggest that the second saccades were not preplanned, rather made in response to the cue. We also tested whether subjects adopted an overtly conservative strategy by deliberately delaying their primary saccades in anticipation of the cue. We compared the mean RT of saccades made in response to the cue occurring after the primary saccade with the mean RT of saccades in non-cued trials (see horizontal arrows in Figs. 5, 7). The average RTs (min = 230 ms, max = 274 ms, mean = 250 ms) in correct non-cued trials were significantly shorter (P < 0.001) than the average second saccade RTs (min = 307 ms, max = 454 ms, mean = 371 ms) in correct cued trials for all but one subject. We, therefore, excluded the data of the subject that did not match this pattern from further analyses.

Figure 4 shows the behavior of three representative subjects in both conditions where the cue appeared horizontally (left panel) and vertically (right panel). Cued trials were separated into groups such that their corresponding cue–saccade overlap times (SOT) fell within bins of 14 ms. For each group, the mean (±SEM) RT of cued saccades is plotted with its corresponding mean SOT. The shaded region on each plot indicates the SOT interval corresponding to instances when the cue occurred while the first saccade was being executed. A positive SOT indicates that the cue appeared before the end of the primary saccade, resulting in a temporal overlap between two saccades. In contrast, a negative SOT indicates the cue appeared after the primary saccade, resulting in the serial processing of the two saccades.
Fig. 4

Plots of cued saccade RT versus cue–saccade overlap time (SOT) for horizontal (left panel) and vertical cued saccades (right panel) for three representative subjects. The shaded region represents the epoch when the cue appeared during execution of the primary saccade. The epoch to the right and left of the shaded region represents the situation when the cue appeared after the onset and before the execution of the primary saccade, respectively. Plots of RT against SOT in “During” and “Before” epochs are weighted by the reciprocal of the standard error in the measurement of mean RT to fit using a linear regression and an exponential function, respectively. Going from right to left along the SOT axis, RT increased monotonically with SOT when the cue appeared during execution of the primary saccade, indicating a bottleneck of processing. When the cue appeared before the primary saccade, as SOT increased, initially RT decreased to a minimum from which it increased again. The horizontal arrow on the ordinate indicates the mean saccade RT in non-cued trials. The vertical arrow on the abscissa indicates the locus of the minimum of the exponential fit

According to the logic of the dual-task paradigm, a processing bottleneck is revealed when a monotonic increase in RTs of second saccades across adjacent SOT intervals occurs. Across subjects, we observed two such points. The first point occurred at a SOT of zero, which corresponds to the instance when the appearance of the cue instructing the second saccade coincided with the termination of the primary saccade. Increase in RTs to the left of this point in Fig. 4 suggests a bottleneck that prevents parallel processing of two saccades during the execution of primary saccade. In other words, a visual cue provided during execution of a saccade will not be processed until the rapid eye movement ceases. To quantify the degree of intrasaccadic visual processing, data points during the interval of execution of the primary saccades (~0–50 ms) were fitted by a regression line for each subject as shown in Fig. 4. Across the population of subjects, the mean (±SD) R2 of linear regression fits was 0.83 (±0.18) and the average slope of these fits was not significantly different from 1.0 (horizontal cue: mean (±SD) = 0.98 ± 0.25, P = 0.89; vertical cue: mean (±SD) = 1.26 ± 0.52, P = 0.2). A paired t test shows the rate of increase in RT with SOT was indifferent between the horizontal and vertical cue positions across the population of subjects (P = 0.21, t = 1.37).

In addition, all subjects displayed a second point after which RTs progressively increased. Typically, this point occurred around 100 ms before the end of the first saccade (see also Fig. 5a). The increase in RT to the left of this point, corresponding to longer SOTs, implies that greater the overlap between two planning stages of saccades, the slower the processing of the second saccade. Such a capacity limitation has been observed in the previous studies of double-step saccades (Becker and Jürgens 1979; Lünenberger et al. 2003; Ray et al. 2004) and is likely to be a consequence of a competition between two motor programs for the same processing resource. Although of interest, it is not the focus of the current study. Of interest to the current study, however, is the observed pattern of RTs occurring to the right of the putative response preparation bottleneck, in the region between cue–saccade overlap times of ~100 and 65 ms, and corresponds to the epoch just prior to the beginning of the primary saccade.
Fig. 5

Population data describing the dynamics of cued saccade latencies. a Plot of the mean (±SEM) cued saccade RT from exponential fits for horizontal and vertical primary saccades and vertical and horizontal secondary cued saccades as a function of cue–saccade overlap time (SOT) when the cue appeared before the onset of the primary saccade (65 ms). The RT reached its minimum (362 ms) at a SOT of 102 ms. b Distribution of the delay between onsets of the cue for the cued second saccade and the primary saccade for which the cued saccade RT reached its minimum across subjects. Ordinate of the stacked histogram indicates the number of groups of trials with one of the four possible spatial combinations of the target and the cue: horizontal target–horizontal cue (HT–HQ), horizontal target–vertical cue (HT–VQ), vertical target–horizontal cue (VT–HQ) and vertical target–vertical cue (VT–VQ). c Linear regression line displayed with 95% prediction bound shows a correlation between minimum cued saccade latency obtained from exponential fits (Fig. 4) and locus of the minimum relative to the end of the primary saccades. Data points that did not contribute to the regression analysis are encircled

On the basis of distinct chronometric signatures provided by the three schemes described in Fig. 2, we tested the nature of trans-saccadic processing. In contrast to the prediction of model 1 (No trans-saccadic processing; Fig. 2a), the RT of cued saccades did not increase with increasing cue–saccade overlap time (SOT) when the cue appeared just before the onset of the primary saccade as shown in Figs. 4, 5a and 7. This was assessed by measuring the occurrence of the maximum in the fit to the data. Across the population of subjects, the maximum RT occurred on average (±SD) at a SOT of 53 (±11) ms, which was of the order of the execution time of a saccade. This suggests that the maximum RT generally occurred when the cue appeared at the beginning of the saccade. Additionally, the ‘no trans-saccadic processing’ model predicts that at a sufficiently high SOT a sudden drop in RT is expected to occur, when visual processing is completed before the start of the saccade. This was never observed; rather, the decrease in RT with increasing SOT was smooth. Taken together, these results are not consistent with the predictions of model 1. The pattern of RT of cued saccade also appears to be inconsistent with the prediction of model 2 (trans-saccadic visual processing; Fig. 2b) as well. This is primarily because if only visual integration, but no integration of saccade plans, was to occur across the saccade, then RTs at positive SOTs are expected to be constant for the epoch just prior to the beginning of the primary saccade. This was never observed either; instead, RT decreased gradually with increasing SOT between 60 and 100 ms, commonly across three data points. This trend was best captured by fitting a double exponential function that had an initial descending arm followed by an ascending arm, which characterized the response-related bottleneck at the longest SOTs (mean R2 ± SD of fit = 0.75 ± 0.22). Mathematically, the double exponential function is described by \( F(x) = \alpha {\text{e}}^{ - \beta (x - \theta )} + \gamma {\text{e}}^{\delta (x - \theta )} \), where α, β, δ, γ, θ were independent coefficients. The fits were weighted by the reciprocal of the standard errors contributing to the measurement of the mean RT of cued saccades. The use of the double exponential was motivated to simulate the occurrence of parallel programming and saccadic bottlenecks with opposing effects on the RT of the cued saccade.

An additional four subjects were examined in a complementary condition in a separate set of experiments. Here, subjects were asked to make a saccade to the upward vertical target instead of the horizontal target. All other experimental conditions remained the same as the previous experiment. Their data showed the same trend as described before. Figure 5a shows the mean (±SEM) cued saccade RT from exponential fits across subjects for all targets and cue positions, as a function of SOT. Figure 5b plots the distribution of the time of occurrence of the minima in the exponential fit measured relative to the beginning of the first saccade across the population for the four possible combinations of the target and the cue position: horizontal target–horizontal cue (HT–HQ), horizontal target–vertical cue (HT–VQ), vertical target–horizontal cue (VT–HQ) and vertical target–vertical cue (VT–VQ). Excluding just one data point from the bin (0–10 ms), for which we did not observe the characteristic gradual drop in RT of the second saccade, the minimum RT across all four conditions occurred on average (±SD) at 36 (±18) ms prior to the onset of the primary saccade. Furthermore, Fig. 5c shows a significant correlation (coeff = 0.557, P = 0.01) between the minimum RT of cued saccades obtained from the fit in the “before” epoch and the locus of that minimum relative to the end of the primary saccades, i.e., between the horizontal dotted line and vertical dotted line in Fig. 4 (top row), across the population of subjects in all possible combinations of target and cue locations. This suggests that the bimodal distribution of the loci of minima of fits with two peaks, one at 10–20 ms and the other at 40–50 ms, was due to variability in the RT of cued saccades. Such flexibility in the slope of the swivel in the “before” epoch is consistent with the idea of model 3 (trans-saccadic visual processing and motor planning: Fig. 2c).

We also tested whether the pattern of RTs could be explained by a speed/accuracy trade-off. If this was the case, the SOT associated with the minimum RT is expected to yield the maximum errors and vice versa. To test whether the data can be explained by a speed/accuracy trade-off, we measured the proportion of errant trials when the cue appeared around the time of execution of primary saccades to the target. Errors were a consequence of failure to respond to the cue by making a return saccade to the fixation spot in time, presumably because they failed to detect the cue. All cued trials that yielded identical (±10 ms) SOTs were grouped together to calculate the proportion of error. The proportion of error was calculated from the ratio of the number of erroneous cued trials to the number of all cued trials in a group. Proportions of errant cued trials are plotted against corresponding SOTs for a representative subject in Fig. 5a. The proportion of erroneous cued trials peaked during the execution of trials, i.e., at a positive SOT (20 ms) less than the average time (60 ms) of saccade execution. To measure the span of SOT when the subject made frequent errors and the instance when the subject made maximum errors, we optimized the ‘standard deviation (σ)’ and ‘mean (μ)’ of a Gaussian distribution function that fitted the data best (see “Methods”). In Fig. 6b, fits across the population of subjects are overlaid. The average (±SD) R2 of the fits was 0.7 (±0.27). The maximum proportion of failures to detect the cue occurred on average (±SD) at a SOT of 18 (±5) ms, which was much shorter than the SOT (102 ms) associated with the minimum RT as shown in Fig. 5a. An increase in the proportion of errors was observed during SOTs that spanned on average (±SD) across 40 (±12) ms. This interval of SOT was calculated by taking three times the standard deviation (σ) on either side of the mean (μ). These results indicate that a speed/accuracy trade-off cannot explain the observed pattern of RTs and provide corroborative evidence of a saccade-related bottleneck that prevents visual processing during saccade execution.
Fig. 6

Plots of the proportion of erroneous cued trials as a function of the cue–saccade overlap time for a a representative subject and b across the population. Subjects failed to respond to the cue more often when it appeared during the execution of the primary saccade

We repeated the experiment with a cue having a physically equiluminant change in a chromatic stimulus. Here, unlike the previous experiment, the equiluminant change did not flicker and persisted throughout the duration of the trial. Figure 7 shows the behavior of four participants in a session in which the equiluminant cue appeared either at the end point of saccades (horizontal cue) or orthogonal to the direction of saccades (vertical cue). Qualitatively, similar results were obtained as in Fig. 4. The average (±SD) slope of the linear regression fits for the cued saccade latencies plotted against the SOT when the cue appeared during the first saccade across subjects and cue positions was 0.87 ± 0.55 ms, which was not significantly different from 1.0 (P = 0.42). We also observed that the minima of the exponential fits occurred on average (±SD) at 54 (±25) ms prior to the primary saccade in the RT plot, similar to the data from the previous experiment. As in the previous experiment, subjects made frequent mistakes when the chromatic cue appeared during the execution of primary saccades. The average (±SD) R2 of Gaussian fits of the proportion of erroneous cued trials plotted against the corresponding SOT across subjects was 0.6 (±0.35). The maximum proportion of failures to detect the cue happened when the cue appeared on average (±SD) at 16 (±6) ms before the end of the primary saccade. An increase in the proportion of errors occurred during SOTs that spanned on average (±SD) across 34 (±15) ms. Taken together, our results suggest that trans-saccadic integration for visual processing and motor planning occurred independent of the nature of the cued stimulus.
Fig. 7

Plots of cued saccade RT versus cue–saccade overlap time (SOT) when the cue appeared as an equiluminant change in chromatic stimuli in four subjects. Conventions are the same as in Fig 4. A gradual decrease followed by an increase in RT is observed for every subject when the cue appeared before the onset of the primary saccade

Discussion

Mental chronometry provides a complementary approach to study trans-saccadic information processing since actions are not only characterized by spatial decisions that determine where to act but also by temporal decisions that govern when to act. Here, we have studied the accumulation of information across a primary saccade that functions as a natural bottleneck, limiting access to incoming visual information. In this study, we provide evidence of the integration of visual and motor planning across saccades using a novel chronometric technique designed to probe for capacity limitations of information processing.

A critical factor on which the interpretation of these results hinges is the notion that little or no visual processing occurs during a saccade. This assumption appears reasonable based on the plots of the mean RT of cued saccades as a function of the cue–saccade overlap time (SOT), which show that the rate of change in RT is close to unity across subjects. Interpreted in the framework of the dual-task paradigm, this implies a bottleneck of processing. Numerous studies have documented impaired visual processing before, during and after saccadic eye movements based on subject’s perceptual performance (Campbell and Wurtz 1978; Schlag and Schlag-Rey 1995; Thilo et al. 2004; Volkmann 1986; Zuber and Stark 1966). However, there are only few reports about the effects of intrasaccadic visual stimulation on either the ongoing primary saccade or the secondary corrective saccade in a double-step task (Becker 1993; Prablanc et al. 1978). While Becker (1993) observed an increase in saccade amplitude of the primary saccade, Eggert et al. (1999) reported no effect on either the RT or gain of the primary saccade. Our results, therefore, appear to be more compatible with the results of Eggert et al. (1999), and the difference in results might be explained by the stronger visual stimulation (full-field stroboscopic flashes) used in Becker’s experiment. However, Eggert et al. (1999) did provide some evidence of increased saccade RT of the secondary corrective saccades, particularly if the target step occurred during the deceleration stage, suggesting some intrasaccadic visual processing (Brooks and Fuchs 1975). Since their experiments used low illumination levels (0.1 cd/m²), it is likely that the detection threshold were only modestly raised allowing for some intrasaccadic visual processing (Brooks and Fuchs 1975). However, in their study, the degree of such intrasaccadic processing was not quantified. By adopting principles from the dual-task paradigm that highlight capacity limitations in executing two tasks concurrently (Pashler 1984; Welford 1952), we show how chronometry can reveal the dynamics of intrasaccadic information processing.

Another assumption that has bearing on the interpretation of our results is the notion that the duration of individual stages that together comprise the RT does not change across the cue–saccade overlap time. This assumption needs to be examined in the context of evidence from psychophysical studies that show increases in RT either due to the presence of a remote distracter presented at fixation or contralateral to the hemifield of the target (Walker et al. 1997; Ludwig et al. 2005; Honda 2005; White et al. 2005) or decreased contrast sensitivity for stimuli flashed around the time of a saccade (saccadic suppression: Burr and Ross 1982; Burr et al. 1994). In relation to the remote distractor effect, in the cued trials of our task, the first saccade to the horizontal target places the non-target vertical stimulus (distractor) on the hemifield opposite to the target hemifield, which eventually acted as a distractor for the subsequent saccades as well. While the impact of the appearance of the non-target stimulus on the first saccade latency cannot be ruled out, it is unlikely that the increase in the second saccade latency was due to the presence of the non-target stimulus, because the remote distractor has no effect on saccade latency when it appears ~80 ms or earlier than the onset of the saccade target (Bompas and Sumner 2009) and when the fixation spot remains on the screen with the saccade target (Honda 2005). In relation to saccadic suppression, a transient reduction in contrast sensitivity might contribute to transient increases in RTs of the cued saccade if the accumulation of visual information that eventually produces a response is based on the quality of the visual stimulus, in this case the contrast of the flickering cue (Pieron’s Law; for review, see Bonnet and Dresp 2001; Palmer et al. 2005). More specifically, the effect of such suppression should be highest for the cue occurring just prior to the saccade and lowest for cue that occur before the start of the suppression. Since the time course of suppression measured perceptually for stimuli either flashed or displaced during a saccade reveal decreased contrast sensitivity and increased displacement thresholds occur at least 50 ms prior to the saccade (Burr and Ross 1982; Diamond et al. 2000; Ross et al. 2001; Thilo et al. 2004; Riggs et al. 1974; Zuber and Stark 1966), this hypothesis predicts that the minimum cued saccade RT should occur 50 ms prior to the primary saccade onset, similar to what we have observed.

However, the assumption that the duration of visual stage might increase prior to the saccade in and of itself does not invalidate the prediction of Fig. 2c (trans-saccadic visual processing and motor planning). In principle, the predictions of Fig. 2c still hold, although slope of the RT versus SOT curve is expected to be steeper in the “before” epoch. This is because the reaction times of saccades generated in response to the cue appearing closer in time to the onset of the primary saccade are expected to be longer (because of a longer duration of the visual processing stage), compared to that in response to the cues appearing at longer SOTs relative to the primary saccade onset, since visual processing will not be as distorted due to decreased amount of saccadic suppression of vision. Thus, it is important to note that the model of trans-saccadic visual and motor planning in and of itself is not antagonistic to the notion of presaccadic suppression and consequent changes in the duration of the visual stage. However, transient changes in the duration of the visual stage may blur the distinction between the prediction of model 2 (trans-saccadic visual processing) and 3 (trans-saccadic visual processing and motor planning). However, this requires further assumptions, to be built into model 2. Thus, we must first assume presaccadic suppression. Although saccadic suppression has been extensively investigated psychophysically, the source (and hence timing) of suppression is still uncertain. Some investigators have suggested the source to be extraretinal (Riggs et al. 1974; Holt 1903; Zuber and Stark 1966; Duffy and Lombroso 1968; Matin 1974) and inherent to saccadic planning itself. If saccadic motor planning is one of such extraretinal sources, then the presaccadic suppression is expected to be potentially selective (Magnocellular: Burr et al. 1994) can occur before the start of the saccade. In contrast, many investigators have argued for the retinal image motion produced by the saccades, combined with masking effects of successive fixations, as the reasons of suppression (Beeler and George 1967; Mackay 1970; Campbell and Wurtz 1978; Sperling 1990). More recently, some investigators (Castet et al. 2002; García-Pérez and Peli 2001) have provided fairly compelling arguments to suggest that motion perception can occur during a saccade. Second, although the phenomenon of presaccadic suppression has been extensively studied, the studies have been largely limited to perceptual reports. In this context, Burr et al. (2001) have found evidence that saccade affects the representation of the visual space for perception but not for action. Therefore, the time course of saccadic suppression of perceptual contrast sensitivity as measured from the report made verbally by observers may not be enough to justify the pattern of the RTs of second saccades in our task (Diamond et al. 2000).

There are additional reasons to suggest that saccade suppression per se cannot provide an adequate explanation for the pattern of results observed. First, although our models assume that the duration of visual processing of the cue is constant irrespective of the time of its onset, the main finding of this study does not hinge on this assumption. That is, the first two models, namely, the no trans-saccadic processing model and the trans-saccadic visual processing model, still fail to account for the observed decrease in the cued saccade reaction time (RT) with increasing cue–saccade overlap time (SOT). Given that RT increases to RT + M1 due to the intrasaccadic processing bottleneck, when the cue onset coincides with the onset of a primary saccade, RT will further increase when the cue appears just prior to the onset of the saccade, if we consider an increase in the duration of the visual processing of the cue due to saccadic suppression (Fig. 2a, b). Even if the span of this postponement of the cue processing gradually decreases with increasing SOT, RT is expected to reach its maximum when the cue appears sometime before the primary saccade onset, unlike at the time of the primary saccade onset as our data show (Figs. 4, 7). Although this conclusion maybe biased by discontinuities introduced as a consequence of binning the data, given the bin size used (14 ms) and the fact that almost all the data unequivocally indicate that the maximum RT was centered around the beginning of the primary saccade (we would expect a greater variation if the position of the max RT was only sensitive to the binning procedure), we believe that visuomotor processing hypothesis is a more parsimonious explanation of the data. Alignment of the peak RT with the primary saccade onset is only feasible if the elongation of the visual processing stage is counterbalanced by a shortening of the planning stage. With the present set of data, we do not see any means of such facilitation but initiation of motor preparation of the cued saccade during execution of the primary saccade, as shown in Fig. 2c (trans-saccadic visual processing and motor planning). Second, previous work (e.g., Deubel and Schneider 1996; Hoffman and Subramaniam 1995; Kowler et al. 1995) has shown that spatial attention is invariably located at the end point of a saccade. Since spatial attention is known to enhance contrast sensitivity at the cued location (Herrmann et al. 2010), one expects different patterns of response when the primary saccade and the cue location were the same or different. However, we obtained similar patterns with each of the four possible combinations of the target and cue (horizontal target–horizontal cue (H–H), horizontal target–vertical cue (H–V), horizontal target–vertical cue (H–V) and vertical target–vertical cue (V–V), in which the location of the cue and saccades were either spatially congruent or dissociated. Thus, we believe that the modulation of contrast sensitivity, either as a consequence of saccade suppression or spatial attention, is not a viable explanation for the observed effects. Third, studies have shown that predictive saccadic suppression occurs primarily for low contrast, low spatial frequency and high temporal frequency stimuli that stimulate the magnocellular pathway but does not occur for equiluminant stimuli that might selectively stimulate the parvocellular pathway (Burr and Ross 1982). We did not find any qualitative difference in RT profiles between the cue conditions (Figs. 4, 7), which might differentially stimulate the parvocellular (equiluminant color cue) and magnocellular pathway (flicker cue). Furthermore, in both conditions, subjects failed more often to detect the cue that appeared while the eyes were in flight and not before the start of the saccade. Fourth, several studies have shown dissociation between perception and action during voluntary movements (Trevarthen 1968; Goodale and Milner 1992) and that perceptual judgements based on relative locations of visual stimuli are different from those of goal-directed movements that may use information about absolute, egocentric locations (Hallett and Lightstone 1976; Honda 1989; Burr et al. 2001). The suggested dichotomy between visual representations dedicated to perception versus action goal-directed eye movements may also underlie the relative insensitivity of saccade target selection to the effects of saccade suppression. Fifth, it is possible that the opposing effects and similar time course of presaccadic suppression of vision and concurrent facilitation of spatial attention at the end point of the saccade cancel out leaving saccade target selection relatively unaffected. Sixth, we also performed the flicker experiment at 100% contrast in two subjects and obtained a similar pattern of results as with the low-contrast flicker cue (data not shown).

Although the absence of presaccadic suppression might appear to run counter to the notion that suppression contributes to perceptual stability, this might facilitate behaviors like error correction that demand quick responses in dynamic environments (Ray et al. 2004; Vaziri et al. 2006; Murthy et al. 2007). Instead, the minimum time spent to respond to the cue occurring 50 ms prior to the onset of the primary saccade could be due to 50-ms visual RT of neurons in the cortical oculomotor areas that identify the cue (Schmolesky et al. 1998; Bisley et al. 2004). In this scenario, a cue–saccade overlap time of 50 ms represents the “sweet spot” (Fig. 2), when the visual processing of the cue is finished exactly at the time of onset of the first saccade allowing uninterrupted processing of the second saccade. Alternatively, the location of the minimum RT at ~50 ms prior to saccade beginning may also be a consequence of the saccade duration of ~50 ms that limits the maximum amount of planning that can occur during saccade execution. Taken together, we interpret the reduction in RT with cue–saccade overlap time up to a limit as a consequence of presaccadic visual information being used for planning the next saccade while the primary saccade is being executed (see Fig. 2c, visual and plan integration).

Conceptually, our model envisions that the oculomotor system can make decisions and plan movements from the continuous accumulation of information. This critical idea has been formalized in numerous other models of reaction times that posit that the total duration of planning a saccade can be segmented into processing stages such as target identification and motor planning, with each stage involving the integration of transient noisy information. By incorporating a stopping rule in the form of a threshold that elicits a response when crossed, these models have been very successful in predicting RTs distributions in a variety of tasks that require decision making and motor planning (Smith and Ratcliff 2004). While these models have motivated many investigations on how a response is chosen from alternatives by accumulating continuous sensory information, they have not been subject to manipulations in which the accumulation of information is made to halt temporarily. This is accomplished in our cued double-step experiment by the introduction of an intervening saccade that acts as a bottleneck. A number of important implications follow from the current work. First, since the presaccadic visual information is being used for planning the next saccade while the primary saccade is being executed; this implies parallel programming of two saccades, for which a number of studies have provided converging behavioral evidence (McPeek et al. 2000; Ray et al. 2004; Caspi et al. 2004; Sharika et al. 2008). Specifically, this model predicts that movement-related activity in oculomotor areas for a forthcoming saccade can co-occur with the execution of a prior saccade. Evidence for this derives from recent work in frontal eye fields during a search-step task where movement-related activity coding for a corrective saccade could begin during the execution of the erroneous saccade (Murthy et al. 2007, Phillips and Segraves 2010).

Our results also have bearing on a central issue in mental chronometry, which is whether information transfer between processing stages such as stimulus evaluation and response preparation occurs in a continuous or discrete manner (reviewed in Meyer et al. 1988). Discrete information-processing models assume that one process must finish before a subsequent process can begin, so different processes operate in a strictly sequential manner. In contrast, continuous processing allows the partial accumulated output from the visual stage to influence the subsequent response preparation stage. Since we have obtained evidence that the motor planning may proceed even though the visual stage may be interrupted by an intervening saccade, these results support the model of continuous information processing and argue against a strictly discrete model. Such a model of continuous information processing is a consequence of RT models that assume that moment-to-moment changes in the quality of sensory evidence are held in memory by neural networks that perform the essential function of integration and thought to be a necessary part of movement planning (Wang 2002). Recent neurophysiological evidence using a motion discrimination task has provided clear evidence that the neural activity of cells in sensorimotor areas such as LIP and FEF does, in fact, reflect the dynamic changes in the sensory environment that have occurred in the recent past (Hanks et al. 2006). It is such a buffering mechanism that is likely to integrate visual information before, during and after the saccade into motor plans.

Acknowledgments

This work was supported by grants from the Department of Science & Technology and the Department of Biotechnology, Govt. of India. S. Ray was supported by Council of Scientific and Industrial Research, India. We thank Dr. A. Sripati, Dr. S.J. Heinen and A. Ramakrishnan for their critical comments on the manuscript.

Copyright information

© Springer-Verlag 2011

Authors and Affiliations

  • Supriya Ray
    • 1
    • 2
  • Neha Bhutani
    • 1
  • Vishal Kapoor
    • 1
    • 3
  • Aditya Murthy
    • 1
    • 4
  1. 1.National Brain Research CentreManesarIndia
  2. 2.The Smith-Kettlewell Eye Research InstituteSan FransiscoUSA
  3. 3.Max Plank Institute for Biological CyberneticsTübingenGermany
  4. 4.Centre for Neuroscience, Indian Institute of ScienceBangaloreIndia

Personalised recommendations