The latency for correcting a movement depends on the visual attribute that defines the target
Neurons in different cortical visual areas respond to different visual attributes with different latencies. How does this affect the on-line control of our actions? We studied hand movements directed toward targets that could be distinguished from other objects by luminance, size, orientation, color, shape or texture. In some trials, the target changed places with one of the other objects at the onset of the hand’s movement. We determined the latency for correcting the movement of the hand in the direction of the new target location. We show that subjects can correct their movements at short latency for all attributes, but that responses for the attributes color, form and texture (that are relevant for recognizing the object) are 50 ms slower than for the attributes luminance, orientation and size. This dichotomy corresponds to both to the distinction between magno-cellular and parvo-cellular pathways and to a dorsal–ventral distinction. The latency also differed systematically between subjects, independent of their reaction time.
KeywordsDorsal stream Hand movement Color Shape Visual processing Target perturbation Double step
Neurons in different cortical visual areas respond to different visual attributes with different latencies (Livingstone and Hubel 1987, 1988; Schmolesky et al. 1998; Bullier 2001). How do such differences affect the way in which visual information guides our movements? Two main streams of visual processing have been identified within the brain; each specialized for processing a different kind of information (Trevarthen 1968; Mishkin et al. 1983; Goodale and Milner 1992; Ungerleider and Haxby 1994). A ventral stream in which information travels from V1 to the inferotemporal cortex is crucial for the identification of objects, whereas a dorsal stream passing through the posterior parietal region plays a major role in spatial vision. A widely accepted explanation for this distinction is that the dorsal stream is specialized for quickly processing spatial information to control action, whereas the slower, detailed visual analysis of other attributes that are important for recognizing objects takes place in the ventral pathway (DeYoe and Van Essen 1988; Goodale and Milner 1992; Tanne et al. 1995; Desmurget et al. 1999; Rossetti et al. 2000; Lee and van Donkelaar 2002). This view implies that not all information can be used to quickly adjust hand movements. We tested this prediction.
An issue that cannot be ignored is how conspicuous the distinction between target and reference is within each attribute; i.e., how easily the target can be distinguished from the references. When comparing responses to targets defined by the same attribute, the response becomes faster if the relevant contrast is increased (Brenner and Smeets 2003). We therefore have to show that any differences in latency that we find cannot be simply caused by differences in conspicuousness. The question is how to evaluate conspicuousness independently from the response latency.
In previous studies, this issue was resolved by equating a critical stimulus parameter (e.g. cone contrast, White et al. 2006), or by equating the stimuli in terms of an independent perceptual judgment (e.g., how conspicuous the targets are judged to be, Brenner and Smeets 2003). The first approach is not feasible in our current experiment because it is not at all clear how stimulus parameters for different attributes could be matched. The second approach is also problematic, because the visual processing for the perceptual judgment task might differ from that for the real experiment. We will therefore develop a new method to deal with differences in conspicuousness. This method, which will be developed in the results section, is based on examining responses to various targets that are all distinguished on the basis of the same attribute, but that differ in conspicuousness. We will show that beside leading to shorter latencies, more conspicuous targets also lead to shorter reaction times, higher velocities, and a steeper slope of the responses. We will then compare latencies for targets defined by different attributes when they give rise to comparable reaction times, velocities, and response slopes. Differences in latency after such matching cannot be caused by differences in conspicuousness.
Materials and methods
Twelve subjects took part in this study. Three of them were the authors and the others were colleagues and friends who were unaware of the hypothesis that was being tested. All of them had normal or corrected to normal vision. Two of the participants were left-handed. Each participant made 120 pointing movements in each of 13 different sessions. Six of the participants also made 120 pointing movements in each of 7 additional sessions in which various luminance contrasts were used. Each participant performed the sessions in a different random order. The ethics committee of the Faculty of Human Movement Sciences approved the study.
Apparatus and stimuli
Participants sat behind an A2-sized graphic tablet (Digitizer II, Wacom Ltd, Tokyo, Japan) and viewed a projection surface via a semitransparent mirror that was placed above this graphic tablet. The images were back-projected from above the projection surface. The distance between the mirror and the projection surface was identical to that between the mirror and the surface of the tablet, so that the projected image appeared to be at the surface of the tablet. Lamps underneath the mirror ensured that the participants could see their hands. The resolution of the projected image was 1,024 by 768 pixels, with 1 pixel corresponding to about 0.5 mm. The position of the pen was determined every 5 ms (200 Hz).
At the near edge of the screen was a starting point (a black, 1 cm diameter disk with a luminance of 2.3 cd/m2 as measured with a Minolta LS-110 luminance meter). At a distance of 25 cm from this point in the sagittal direction, three objects were projected next to each other on a white background (35 × 45 cm; 28.0 cd/m2). The distance between the centers of the objects was 3 cm (Fig. 1a). One of the three objects served as the target; this object differed from the other two objects (the references).
In the luminance (390%) condition, the target differed from the references in luminance only (for further details about this and subsequent conditions see Fig. 2). There were seven additional conditions with different luminance contrasts between the target and the references. These were used to determine how responses depend on how conspicuous the distinction is between the target and the references.
In the three orientation conditions, the target and the references were rectangles (0.67 × 1.50 cm, 5.1 cd/m2). The target rectangle was orientated at an angle of 45°, 60° or 90° with respect to the parallel references. In the three size conditions the target squares had different sizes. The reference squares had surface areas of 1 cm2, as did all the objects in the other conditions. In the shape session the target was a circle, and the references were squares. In the color condition, the target was a green square (5.1 cd/m2) and the references were two red squares (5.1 cd/m2). In the texture condition, the target square was filled with ten 1-mm wide stripes, alternating between dark (2.3 cd/m2) and bright (7.9 cd/m2) ones. The average luminance of the stripes was equal to the luminance of the references (5.1 cd/m2), to ensure that participants had to analyze the texture to recognize the target.
During pilot-experiments, some of the participants had the impression (especially in the orientation and the shape conditions) that they were not responding to a difference between the target and the references, but to the change at the new target position. To investigate whether this was really an issue we performed two control conditions in which we masked such effects by also changing aspects of the references when the target changed its position. In a masked orientation condition, the references both changed their orientation by 90° when the target jumped to a new location. In a masked shape condition, the reference squares increased in size at the same moment that the target circle changed position.
Participants started each trial by placing the pen on the starting point. After a random time (about 2 s), the target and references appeared (Fig. 1a). The task was to reach the target with the pen and stop there as soon as possible. The sum of the movement time and the reaction time was to be minimized. Participants were shown which object was the target (and which the references) before they started each session.
Within each session of 120 trials, the target appeared 40 times at each of the 3 positions. In half of the trials the target and the references remained at their initial position (unperturbed trials, Fig. 1c). In the other half of the trials the target changed its position to one of the two reference positions about 62 ms after the subject’s finger reached a velocity threshold of 0.02 m/s (perturbed trials, Fig. 1b). At the same moment the reference (that had been at that position) changed its position to where the target had been. Whenever the target changed position participants had to adjust their hand movement to reach the new target location (Fig. 1d). If a movement ended on the target within 1.2 s of the target first appearing, the participant was rewarded with auditory feedback. The 120 trials (three initial positions, change in position or not, two positions to which to change, ten replications) were presented in random order. Since different positions to which to change are irrelevant when there is no change, there were actually 20 trials in which the target remained at each initial position.
Velocities were calculated for the interval between every two measurements by dividing the displacement of the tip of the pen by the 5 ms between the measurements. The beginning of the movement was defined as the first position after the tangential velocity reached 0.02 m/s. The end of the movement was defined as the first position after the tangential velocity fell below 0.02 m/s. To evaluate the corrections, we only used the lateral component of the velocity (parallel to the displacement of the target).
In order to isolate the responses to changes in target position, we first synchronized all the measurements relative to the moment that the target changed—or would have changed—position. We then separately averaged the lateral velocity for each combination of initial and final target position. We defined the lateral velocity in the direction of the position change as being positive; when the position did not change, we defined the direction in which it would have changed as being positive. We characterize the response to a change in target position by the additional lateral velocity: the difference in the lateral component of the velocity between the trials in which the target changed from a certain initial position to a certain final position and the ones in which it remained at the same initial position. To obtain a single response per attribute for each participant we averaged the additional lateral velocity across the six combinations of initial and final target positions. We used these curves to determine the latencies of the responses.
The slope of the line through the 25 and 75% points of the average additional lateral velocity does not only depend on the intensity of the response on individual trials (lower acceleration results in a shallower slope). If the response does not always occur at the same time, averaging will result in an average response that has a lower peak velocity, shallower slope and longer duration than the responses in individual trials. To evaluate the shape of the response curve without the influence of variability in timing, we used a second way of synchronizing trials before determining the additional lateral velocity.
To estimate the intensity of individual responses for each condition and participant, we synchronized the lateral velocity curves of all perturbed trials at the peak lateral velocity in the direction of the new target position (irrespective of when the target position changed), and produced an additional lateral velocity curve for each subject by averaging across replications and perturbation directions. We averaged these synchronized additional lateral velocity curves across positions and across participants to investigate whether there are systematic differences between the intensity of the responses for different attributes. We averaged the curves across conditions rather than participants to investigate whether there are systematic differences between the intensity of different participants’ responses. In order to determine whether the response was proportional to the size of the perturbation, we also averaged the curves separately for all pairs of positions in which the target jumped 3 cm, and all pairs of positions in which the target jumped 6 cm.
Overview of responses
On average, it took participants 365 ms to react to the appearance of the target (and the two references). The average movement time (time until participants stopped their movement on the target) was another 275 ms. Figure 1 shows one participant’s paths in one condition with the target initially positioned on the left. When the target stayed at the initial target position (Fig. 1c), the hand moved to the target along a slightly curved path (note that the lateral deviations are exaggerated due to the unequal scaling of the figure). When the target jumped at the onset of the movement (Fig. 1d), participants obviously initially moved along similar paths to those seen in Fig. 1c, but approximately half way to the target the path curved toward the new target position.
Figure 3 shows the additional lateral velocity for the various conditions (summarized in Fig. 2), averaged over all subjects. Each curve represents the difference in lateral velocity between trials in which the target did and did not change position (see “materials and methods” for details). The dotted lines are drawn through the points at which the mean additional lateral velocity curves reached 25 and 75% of their peak values. We consider the points at which these lines intersected the line at an additional lateral velocity of zero as the latencies of the responses. We used targets that only differed in luminance from the references to evaluate effects of conspicuousness (for half of our subjects). We found that lower luminance contrast (a less conspicuous target) results in both a longer latency and a shallower slope (Fig. 3a). The conclusion that we can draw from the main conditions depicted in Fig. 3b, is that both the timing and the slopes of the curves differ between conditions. Figure 3 shows an analysis over responses averaged over subjects. The rest of the analysis is done on the basis of determining slopes and latencies for each subject (and condition) separately.
Dealing with conspicuousness
Different attributes, different latencies?
Figure 4d and e shows the same relationships as Fig. 4a, b for the main experimental conditions (Fig. 2a). The dashed lines reproduce the relationship that we found for the luminance conditions (Fig. 4a, b). Participants responded to the various size and orientation conditions with a relationship between latency and conspicuousness that was similar to that for the responses to luminance-defined targets. The targets defined by the attributes color, shape and texture were not less conspicuous than the other targets (reaction times and slopes are similar), but subjects take exceptionally long to react to changes in their position (t-test, P < 0.05). These responses were about 50 ms longer than the ones indicated by the dashed lines. The movements toward the targets defined by color, shape, or texture had similar peak velocities, both of the movements themselves and of the corrections, as the movements toward other targets (Fig. 4f).
For two attributes (orientation and shape) we constructed a “masked” condition, in which not only the target changed location, but also the references changed. This prevents subjects from using the change itself as an indicator of the new target location. The latency for these two conditions is 210 ms, which is longer than for the two corresponding original conditions (paired t-test, P < 0.05). The masking decreased the velocity of the movements and of the corrections (Fig. 4f), but did not change the conspicuousness of the targets, either in terms of the reaction time (Fig. 4d) or of the slope of the average response (Fig. 4e). So the increase in latency due to the masking is not caused by masking making the targets less conspicuous, but by requiring additional processing, presumably for a renewed selection of the target. Due to the blocked design, subjects could counteract the effects of their later responses by slowing down their movements (low values for the masked targets in Fig. 4f).
Subjects and amplitudes
Before discussing how to interpret the results, we want to answer two additional questions. The first is, how does the variability between participants contribute to the results? To find this out, we plotted the data averaged over the conditions for each of the participants separately (Figs. 4g–i, 5b). We coded subjects according to the latency of their adjustments: bright curves and symbols correspond to subjects with short latencies. The intensity of the response differed much more between subjects (SD = 91 m/s2, Fig. 5b) than between conditions (SD = 37 m/s2, Fig. 5a), with no apparent correlation between the peak velocity and latency (Fig. 4i: brighter symbols are not concentrated in one region of the figure). All subjects show more or less the same pattern of latencies as we found for the average data (not shown). This was so for the different attributes as well as for the different levels of conspicuity. There is quite some variability between the subjects, but subjects that are slow in one condition are generally slow in the other conditions as well. The result is that subjects differ systematically in their latency, without any correlation with their reaction time (Fig. 4g) or with the slope of their response (Fig. 4h). This performance difference cannot easily be explained by factors such as age, visual acuity or body size.
The second question is whether the similarity of the curves in Fig. 5a means that subjects have a pre-programmed response to a perturbation that is used irrespective of the precise visual information. To answer this question, we compared the responses for the two amplitudes of the perturbation. When the target jumped 6 cm from its initial location (from one extreme position to the other), participants responded about twice as strongly as when the target jumped 3 cm (Fig. 5c). Thus, the response is proportional to the position change, and therefore clearly tailored to the visual information rather than that pre-programmed and triggered by the change.
By considering the relationship between response latency and measures of conspicuousness (reaction time and variability in the latency), we can conclude that participants reacted with a similar latency to differences in orientation and size than to corresponding differences in luminance. The attributes color, shape and texture gave rise to approximately 50 ms longer latencies than did differences in luminance, orientation and size that were equally conspicuous. Note that these latencies are still 150 ms faster than the original reaction times. We found a similar order of fast and more slowly processed attributes across participants, which supports our idea that we can use this method to measure systematic differences between attributes in terms of the time that it takes to process them, even if participants respond differently (see different velocity profiles in Fig. 5b).
An important question in this kind of research is whether the apparent response to a particular attribute could actually be a response to a small difference in another attribute. For instance, in the color condition, participants could respond to small differences in luminance instead of to differences in color between the target and the references, since the luminance of the target and the references was not equated for individual participants. They clearly do not, because if they had done so, the response would be similar to that for one of the small luminance contrast conditions, which is clearly not the case (the slope of the response is much steeper for color).
As mentioned earlier, it takes about as long to respond to changes in position defined by target size as to those defined by target luminance. We cannot reject the possibility that the reactions to the difference in surface area between the target and the references in the size condition were actually responses to the average luminance within an area larger than that of the target itself. To be sure that this was not the case we would have had to vary the luminance of the targets and references, which we did not do in the present study.
Desmurget et al. (1999), showed that applying transcranial magnetic stimulation over the left posterior parietal cortex disrupted corrections of reaching movements of the right hand after a perturbation in target location in four of their five subjects, thereby demonstrating that the dorsal pathway is necessary for fast corrections of the hand. The fact that participants are able to respond very quickly to changes in target location when the target is defined by the attributes orientation, size and luminance (but not color, texture or shape) is therefore an indication for dorsal processing of orientation, size and luminance (but not of color, texture and shape). This division of attributes is consistent with the distinction between pathways for “what” and “where” (Mishkin et al. 1983; Ungerleider and Haxby 1994). The fact that subjects were still able to correct their movements to targets defined by color, texture or shape (albeit at a longer delay) implies that these attributes also reach the parietal cortex, although at a longer latency. This is in line with studies using electrophysiology (Murata et al. 2000; Sakata et al. 2005), positron emission tomography (Faillenot et al. 1997; Vidnyanszky et al. 2000) and functional magnetic resonance imaging (Oliver and Thompson-Schill 2003) that all suggest that the parietal cortex is involved in at least some aspects of discriminating shape. These studies were, however, not concerned with the exact timing of the activity.
Why are responses for targets defined by shape relatively slow? Assuming that all the responses that we measured are indeed mediated by the posterior parietal cortex, the dichotomy in latencies could be explained in two ways. It could be that the parietal cortex mediates the responses to some targets after initial processing of the attributes that are relevant for the “what” of the object (like color, shape and texture (Mishkin et al. 1983) in the ventral pathway. This indirect pathway is likely to result in longer processing times than the direct dorsal pathway for the “where” attributes.
Alternatively, the dichotomy may arise earlier, with the distinction between the magnocellular retinogeniculate pathway that is faster and more sensitive to luminance contrast, but is color-blind and has a low spatial resolution, and the parvocellular pathway that is color selective and has a higher spatial resolution, but is slower and less sensitive to luminance contrast (Livingstone and Hubel 1987, 1988). This distinction corresponds nicely with the dichotomy in the latencies that we found. Assuming that both the magnocellular and the parvocellular pathways provide input to the dorsal pathway could therefore account for the different latencies that we found for different attributes (Brenner and Smeets 2003), without requiring input from the ventral pathway.
Although we cannot decide between these two lines of explanation, the two masked conditions indicate what is essential for the very fast responses. Being able to identify the target on the basis of “where” attributes (size, orientation, luminance) is necessary for fast responses. However this is not sufficient, because the latency of responses to the change in location of a target defined by orientation increases by more than 50 ms if the references change their orientation. Masking changes prevents responses to the location of the change. Presumably, the target has to be found (identified) again, which requires ventral processing (according to the first explanation).
The conclusion that identifying targets by their color takes 50 ms longer than doing so on the basis of luminance seems to be in conflict with earlier results where we reported that subjects could respond within 120 ms to color information (Brenner and Smeets 2004). In that study, we compared two conditions: one with one reference object of a different color and one with no reference object (similar to the control condition in the present experiment). The main difference between the responses in these conditions was the larger variation in latency for the condition with one reference object (a more shallow slope). It seems as if the distribution of responses consisted of a small peak at short latency, and a larger peak 50 ms later (Fig. 3 in Brenner and Smeets 2004). Perhaps, the few very fast responses in the condition with the reference object were related to the very simple design of that experiment: the position to which the target could change was known in advance, so detecting the change in color at the original target location was enough to start an adequate correction. In the present experiment, the location of the new target position had to be determined on the basis of color information. That directly responding to a transient can lead to exceptional fast responses is evident from the fact that masking the jump of the oriented target by orientation changes of the references caused an increase in latency of 50 ms.
A last issue to discuss is the variability between subjects. Some subjects responded consistently later to the perturbations than others (Fig. 4g, h). The difference between the slowest and fastest subject was 70 ms; larger than the difference between the responses to targets defined by color and luminance. Such a large variation is in line with the remarkable finding that stimulating the posterior parietal cortex directly after movement onset disrupts adjustments in some subjects, but not in others (Desmurget et al. 1999). We can draw two conclusions from these variations in latency. The first is that since none of the subjects had particular complaints of clumsiness, the exact value of the latency is apparently not critical for performing every day tasks. The second is that in situations in which short latencies are important, such as when playing table tennis or boxing, some participants may perform better than others could ever hope to, due to differences between their minimal visuomotor latencies. This latter implication (and in particular whether a shorter latency can be achieved by training) is open for experimental verification.
This article is distributed under the terms of the Creative Commons Attribution Noncommercial License which permits any noncommercial use, distribution, and reproduction in any medium, provided the original author(s) and source are credited.