Humans use continuous visual feedback from the hand to control both the direction and distance of pointing movements
- First Online:
- Cite this article as:
- Saunders, J.A. & Knill, D.C. Exp Brain Res (2005) 162: 458. doi:10.1007/s00221-004-2064-1
- 246 Views
Vision of the hand during reaching provides dynamic feedback that can be used to control movement. We investigated the relative contributions of feedback about the direction and distance of the hand relative to a target. Subjects made pointing movements in a 3-D virtual environment, in which a small sphere provided dynamic visual feedback about the position of their unseen fingertip. On a subset of trials, the position of the virtual fingertip was smoothly shifted by 2 cm during movement, either (1) in the direction of movement, which would require adjustments to the distance moved, or (2) orthogonal to the direction of movement, which would require adjustments to the direction moved. Despite not noticing the perturbations, subjects adjusted their movements to compensate for both types of visual shifts. Corrective responses to direction perturbations were observed within 117 ms, and response latencies were invariant to movement speed and perturbation onset time. Initial corrections to distance perturbations were smaller and appeared after longer delays of 130–200 ms, and both the speed and magnitude of responses were reduced for early onset perturbations. Simulations of a feedback control model that optimally integrates visual information over time show that the results can be explained by differences in the sensory noise levels in the visual dimensions relevant for direction and distance control.
KeywordsHand movementsMotor controlSensorimeterOn-line controlVision
The question of whether movement direction and distance are controlled jointly or independently has been the topic of considerable study. Evidence for the independence of these dimensions has been observed for movement planning (Gordon et al. 1994; Chieffi and Allport 1997; Sainburg et al. 2003), and for sensory-motor adaptation (Krakauer et al. 2000). Independent control of direction and distance is also observed in responses to unexpected target perturbations (Turrell et al. 1998). Whether these movement dimensions are processed differently for on-line visual feedback control remains unknown, however. In previous studies addressing control of direction and distance, visual feedback from the hand was either absent (Turrell et al. 1998) or maintained a constant relationship with the physical hand across many trials (Krakauer et al. 2000). In the latter case, it is difficult to distinguish between effects due to movement planning and on-line feedback control.
In the previous study, the fingertip perturbations were applied perpendicular to the path of movement. Sarlegna et al. (2003) recently performed a similar manipulation of visual feedback from the hand, but in the principle direction of movement. They did not observe fast corrective responses. On the perturbed trials in their study, visual feedback about the location of the hand was displaced relative to the hand during subjects’ initial orienting saccades to the target. Responses during the primary movement phase only partially compensated (~20%), and began after relatively long latencies, averaging 470 ms. The difference between the two results suggests that the visuomotor system may rely on visual feedback about the direction of the hand relative to the target, but not its distance from the target, at least early in a movement.
There are a number of reasons to expect different contributions from feedback about movement direction and distance. First, as mentioned earlier, there is considerable evidence that direction and distance are processed separately for other aspects of movement control, such as planning and adaptation. Second, perturbations in different directions applied to visual feedback from the hand produce effects that likely differ in perceptual salience. Since hand movements tend to follow straight paths, uncertainty about hand position would be expected to be greatest along the axis of movement, such that there would effectively be more noise in feedback about target distance than direction. Similarly, the human visual system is much more sensitive to changes in velocity orthogonal to an objects’ motion path than to changes in velocity along the path (Mateef et al. 2000). Finally, the control problem for these dimensions differs. Once movement is initiated, the direction of movement is relatively constant, and even small errors are diagnostic of future errors in the final position of the hand. In contrast, target distance and hand speed both change markedly throughout the movement. If initial kinematics are only weakly predictive of final movement extent, as was observed by Messier and Kalaska (1999), there might be less incentive for early online feedback control of movement extent, and consequently greater need for later feedback control.
On the other hand, there is evidence that the visual motor system is at least capable of fast online adjustments to both direction and distance movement parameters. Perturbations of the visual target of a reach have been tested both in the direction of movement (Pelisson et al. 1986), requiring adjustment to movement distance, and in a nearly orthogonal direction (Prablanc and Martin 1992), requiring adjustment to movement direction. In both experiments, similar fast corrective responses were observed. While this does not imply that analogous perturbations to visual feedback from the hand would have similar effects, it does demonstrate that both movement direction and distance are potentially subject to online control. Furthermore, the results of the Sarlegna et al. study cannot be directly compared to those of the Saunders and Knill study due to methodological differences. Perturbed feedback occurred earlier in movements in the Sarlegna et al. experiment, and pointing movements were directed to points in space rather than to targets on a solid surface. Finally, as we will describe later, the analysis method used by Sarlegna et al. was significantly less sensitive than that used by Saunders and Knill.
We observed online corrections to both direction and distance perturbations, but with differing magnitudes and latencies. To test whether these differences could be attributed to the amounts of visual noise in the direction and distance feedback signals, we simulated a feedback control model that optimally integrates noisy visual information during movements, and compared its performance to human data for the conditions tested in the experiment. Under plausible noise assumptions, mostly determined from human psychophysics on motion and position acuity, the model exhibited qualitatively similar effects to the experimental subjects.
Design and conditions
Subjects performed a simple pointing task in a virtual environment (Fig. 1a), with visual feedback provided by a small sphere that moved in real-time with their unseen fingertip. On half of the trials, the virtual fingertip was perturbed during the reach, as illustrated in Fig. 1b. On these trials, the virtual finger initially coincided with the actual finger position, but at some point along the movement it was shifted in one of four directions. Perturbations were small (2 cm) and smooth, appearing as a blurred step that extended over 4 cm of the movement. Subjects uniformly reported that they were unaware of any unusual changes in the perceived movement of the virtual finger during the experiment, even when told later about the perturbations.
The main experimental manipulation was the axis of perturbation displacements, which was either aligned with the direction between the initial hand location and the target (distance perturbations, Fig. 2a) or orthogonal to that direction (direction perturbations, Fig. 2b). We also varied the time of perturbation onset and the movement speed. In the early onset condition, perturbations occurred when the finger had moved 25% the distance to the target, while in the mid-reach onset condition, perturbations occurred when the position of the fingertip had moved 50% of the distance to the target. We ran subjects in one of two speed conditions: a fast condition, in which subjects were given feedback to keep their movement durations within a range around 450 ms, and a slow condition, in which subjects were given feedback to keep their movement durations in a range around 600 ms. These speeds spanned the range of what pilot subjects subjectively reported as natural movement speeds for the distances used. Each type of perturbation was presented in both possible directions (up or down, forward or backward). Direction and distance perturbations were presented in separate blocks, and the two movement speeds were tested in separate subject sets. Other parameters, including the presence, onset and sign of perturbations, were randomized across trials within blocks.
Apparatus and display
Visual displays were presented in stereo on a computer monitor viewed through a mirror (Fig. 1), using CrystalEyes shutter glasses to present different stereo views to the left and right eyes. The left and right eye views were accurate perspective renderings of the simulated environment. In stereo mode, the monitor had a resolution of 1024×768 pixels and a refresh rate of 150 Hz, or 75 Hz for each eye’s view. The stimuli and feedback were all drawn in red to take advantage of the comparatively faster red phosphor of the monitor and prevent inter-ocular cross-talk.
The horizontal mirror prevented a subject’s hand from being visible. Visual feedback about hand position was instead provided by a 1 cm diameter virtual sphere that moved in real-time along with the subject’s actual finger. The targets of the subjects’ movements were 1 cm diameter circles rendered to be aligned with an unseen tabletop (~55 cm from the eyes).
An Optotrak 3020 system recorded the time-varying position of a subject’s finger at 150 Hz. The data was used to dynamically update the position of the virtual fingertip. Subjects wore a finger splint on their right index finger, which had a triad of active infrared markers. The position of a subject’s fingertip within the splint was computed from the position of the three markers attached to the splint. The optical measurements had very small latencies (<2 ms), but the speed of the graphical rendering loop imposed an additional latency on visual feedback of ~10 ms. When computing the rendered position of the virtual fingertip, we compensated for this delay by linearly extrapolating from the latest marker position, using the position from previous frames to estimate velocity. We checked the effectiveness of this extrapolation subjectively by viewing the actual and virtual finger simultaneously (using a half-silvered mirror). Residual errors were not readily apparent.
Spatial calibration of the virtual environment required computing the coordinate transformation from the reference frame of the Optotrak to the reference frame of the computer monitor, and the location of a subject’s eyes relative to the monitor. These parameters were measured at the start of each experimental session using an optical matching procedure. The backing of the half-silvered mirror was temporarily removed, so that subjects could see their hand and the monitor simultaneously, and subjects aligned an Optotrak marker to a sequence of visually cued locations. Cues were presented monocularly, and matches were performed in separate sequences for left and right eyes. Thirteen positions on the monitor were cued, and each position was matched twice at different depth planes. The combined responses for both eyes were used to determine a globally optimal combination of 3-D reference frame and eye position. After the calibration procedure, a rough test was performed in which subjects moved a marker viewed through the half-silvered mirror and checked that the position of a rendered dot was perceptually coincident with the marker. Calibration was deemed acceptable if deviations appeared to be less than 1–2 mm. Otherwise, the calibration procedure was repeated.
The task for each subject was to move their finger back and forth to touch a series of visually-presented targets. At the start of a trial, a new target would appear on the opposite side of the workspace as the current hand position. The target onset was the cue to move. Subjects were instructed to move to touch the target in a fast and “natural” manner. Upon reaching the target, they were to keep their finger in place until the next target appeared. Data collection started once a subject’s finger moved 0.25 cm, and continued for 3 s. The recording period was followed by a delay of 800 ms with a blank screen, after which a new target appeared initiating the next trial. The time between trials varied depending on how quickly subjects initiated movement, averaging approximately 4 s.
Subjects were instructed to try to move at about the same speed on each trial. Feedback was provided throughout the experiment to train subjects to reach the target within a range of time around a goal time, which was 450 ms for subjects in the fast condition and 600 ms for subjects in the slow condition. Subjects received positive feedback for movement durations within 75% to 125% of the goal time (a larger circle appeared around target) and negative feedback otherwise (an “X” through the target indicating too fast, no change indicating too slow). For purposes of feedback, reaching the target was defined as bringing the virtual finger within 1 cm of the center of the target. Prior to the main experiment blocks, subjects were allowed 20–40 unperturbed practice trials to familiarize themselves with the timing constraint.
The targets varied in location within two ellipses on the table, which were 10 cm wide, 16 cm tall and separated by 24 cm. The target position for a trial was chosen relative to the end location of the previous trial. For experimental trials, the target distance was always 26 cm (~24° of visual angle), while the direction from the starting position to the target randomly varied between −15° and +15° relative to the left/right axis along the tabletop, subject to the constraint that the endpoint lay within the target ellipse. Randomly interspersed with the experimental trials were a smaller number of trials (20%) with targets at a distance of 22 cm. These trials were included to add some variation in the distance moved, but were not analyzed. Occasionally, subjects moved their hands toward the center of the workspace too early (contrary to instructions), such that it was impossible to satisfy the criteria. In these cases, a random position within the opposite ellipse was chosen as target location, and the trial was discarded from analysis.
On perturbed trials, 2 cm offsets between the actual and visual position of the finger were phased-in gradually over a 4 cm transition range. The amount of displacement on a given frame was a function of the position of a subject’s fingertip along the axis from the starting position to the target. On trials with early perturbations, offsets were phased-in when the fingertip was between 4.5 cm and 8.5 cm away from the starting position, as measured along the target axis. On trials with mid-reach perturbations, the transition region was from 11 cm to 15 cm. The centers of these two transition ranges corresponded to where a subject’s fingertip was 25% or 50% of the way to the target. Within the transition ranges, the proportion of the perturbation applied was computed using a scaled logistic function. The base function was f(x)=1/(1+3−x), where x was the difference between the current position of the fingertip along the target axis and the midpoint of the transition region. This function was linearly re-scaled to span from 0 to 1 over the 4 cm transition range, f’(x)=(f(x)−f(−2))/(f(2)−f(−2)), and the result was multiplied by the magnitude of the eventual displacement (2 cm) to determine the partial displacement. The vector direction of the perturbations was either aligned with the axis from starting position to the target, in the case of distance perturbations, or aligned with the perpendicular axis along the tabletop, in the case of direction perturbations.
The perturbation transitions caused transient discrepancies between the velocity of the virtual fingertip and the velocity of the actual fingertip, which was maximal at the midpoint of the transition. In the case of distance perturbations, the maximal differences in speed were approximately 35–45 cm/s, which corresponds to 60–70% of the actual finger speed, and the differences in the directions of motion caused by the perturbations were negligible. In the case of direction perturbations, maximal differences in speed were 10–12 cm/s, corresponding to 15–25% of the actual finger speed, and the maximal differences in the directions of motion averaged 34°.
Prior to analysis, data were filtered to remove various types of irregular trials. These included trials with incomplete data due to occluded markers, trials with mistimed recordings due to false starts, trials in which subjects began moving before the target appeared, and trials in which motion was not complete within ±40% of the goal time. Some trials contained isolated single frames with occluded markers. If there were no more than four such frames on a trial, the trial was not excluded. Instead, the data from missing frames was filled in by linearly interpolating between adjacent frames. Trials with adjacent missing frames or with more than four isolated missing frames were excluded from analysis.
Subjects participated in two experimental sessions on separate days. Each session began with calibration of the virtual environment, followed by practice trials to familiarize a subject with the task and movement speed, and then two blocks of experimental trials separated by a brief break. Subjects performed 300 trials in each block, of which 240 were experimental trials. This yielded a total of 120 trials per perturbed condition (x-axis onset) and 240 unperturbed trials for each subject.
Perturbations were applied on half of the trials. Perturbed and unperturbed trials were randomly intermixed within blocks, as were the onsets and signs of perturbations. Direction and distance perturbations were blocked by experimental session, with order counterbalanced across subjects. Fast and slow movements were tested in separate subjects.
For the analysis, finger positions were represented in a normalized coordinate frame, with the x-axis being the direction from initial position to target, and the y-axis being the direction in the tabletop plane orthogonal to the x-axis. We ignored the vertical (z-axis) components of the data series, treating the finger positions as 2-D points in the xy-plane.
Movements showed large variability in overall trajectories, but were also very smooth, so much of the variability could be due to differences in initial conditions, motor plans, goal paths, and so on. To derive a sensitive measure of the effect of perturbations, we temporally de-correlated the data series using linear filters fit to the data (Saunders and Knill 2003).
The weights w1(t), w2(t), ... wn(t) were allowed to vary as a function of time, but were assumed to be constant across trials. To align the data series temporally, we defined t=0 to be the midpoint of the perturbation transition, so that t represents approximate time after perturbation. Since there was variability in the speed and time-courses of the trials, the assumption that weights are the same across trials corresponds to assuming that the weights are relatively constant over small shifts in t.
For each subject, we applied linear regression to the baseline trials to estimate the autoregressive weights in Eq. 1. We set n=6 for the analysis, but found that the results were insensitive to the exact value for n>6. To compute influence functions for each perturbation type wpert(t), we correlated the sign of the perturbations (+1,−1) on perturbation trials with the error between the hand position at time t and the position predicted by the autoregressive model. For trials with distance perturbations, perturbation influence functions were computed using only the x-component of the kinematic data (approximately the movement extent), while for trials with direction perturbations, influence functions were computed using only the y-component of the data (approximately equivalent to movement direction).
Response times were computed by locating the first point in time at which the mean perturbation influence functions, averaged across subjects, were significantly different from zero. The raw mean influence function was first smoothed with a one-sided (causal) exponential filter, f(t)=exp(t/λ) for t<0, with λ=35 ms, producing a time-weighted cumulative measure of the perturbation influence. The filtered result, f(t)*wpert(t), was compared at each time t to a statistical threshold derived from re-sampling baseline trials, and the earliest super-threshold time was taken to be the response latency. The re-sampling procedure consisted of repeatedly applying the analysis to randomly-chosen baseline trials (sampling with replacement) and using the results to estimate the null-model noise distribution of filtered wpert(t).
To determine error bounds on the response time estimates, we re-sampled across subjects to compute mean influence functions, and used these samples to compute a distribution of response times. The filtering and threshold comparison applied to the samples was the same as applied to the actual data. The error bounds plotted in the figures correspond to 5% and 95% points in the sample histograms. Comparisons across conditions were based on the distribution of differences between response times. For a given pair of conditions, the difference in response times was computed for each sample, and the resulting set was taken as representing the distribution of actual differences for purposes of statistical comparisons.
In addition to response times, we report a measure of the amount of correction in response to perturbations. We wanted this measure to reflect only corrections that occurred during the main portion of movements, and not any additional secondary corrections, which requires a criterion for identifying the end of initial movements. We defined this to be the first minimum in the speed profile after the deceleration phase had begun. To avoid detecting spurious re-accelerations, we smoothed the time series with a 10 ms Gaussian filter prior to computing minima, and further required that the hand speed be below 6 cm/s. These criteria were ad hoc, chosen to produce subjectively accurate and reliable measures of the end of initial movements, and small variations in the parameter values did not significantly affect the results.
Sixteen naïve, paid subjects participated in the experiment—eight in the fast movement condition and eight in the slow movement condition. Subjects provided informed consent in accordance with guidelines from the University of Rochester Research Subjects Review Board. Our apparatus required that subjects use their right hand, so only right-handed subjects were accepted.
Response times for direction perturbations
Our finding of constant response times for direction perturbations is in agreement with the results of our previous study (Saunders and Knill 2003), which tested conditions identical to the direction perturbations used here. The only apparent discrepancy is the overall magnitude of response latencies: 117 ms in the present experiment versus 160 ms in our previous study. Most of this difference can be attributed to the increased sensitivity of our current analysis. The method of marking the point at which the perturbation influence function exceeds a fixed significance level has an inherent positive bias, which will depend on the noise in the measured influence functions. In the present experiment, response times were computed from the mean influence functions averaged across subjects, rather than separately for individual subjects. This, by itself, increases sensitivity and reduces bias. Additionally, because the mean influence functions are less noisy, less smoothing is required to compute stable estimates of when responses occurred. Decreasing the temporal extent of the smoothing kernel further reduces the amount of positive bias in the estimates. To measure the effect of these changes, we applied our previous analysis to the direction perturbation conditions in the current experiment, and found mean response times of 140 ms. The remaining discrepancy between the current results and our previous results can be explained by our recent discovery of an added one frame delay (17 ms) in the rendering of the virtual fingertip using the graphics card used for the previous experiment—a delay that we had not accounted for in the previous estimates. Thus, the present data replicate our previous findings, and provide further evidence that visual feedback from the hand is used for continuous control of movement direction. The shorter reaction time measured here, 117 ms, further supports this conclusion.
There is some ambiguity in the response times, due to the gradual application of the perturbations, which was spread over a period of 50–70 ms. Depending on how sensitive subjects are to the small conflicts present early in the transition period, or to the initial changes in velocity, a perturbed visual signal could have been available as much as 35 ms earlier than the “onset” time we used for analysis. Note, however, that the perturbations were introduced sub-linearly during the transition, so the initial conflicts were small. For example, during the first 15% of the transition, the conflicts remained less than 2 mm. Thus, it is likely that a perturbed signal effectively becomes available only at some time after the start of the transition. On the other hand, there is no reason to expect that the midpoint of the transition, which we treat as the “onset”, would accurately correspond to the initial availability of a perturbed signal. Thus, our RT estimates should be interpreted as having additional uncertainty, beyond measurement noise, of up to 35 ms.
Continuous feedback for distance
A further reason to doubt an end phase feedback explanation is that, in the conditions that produced the slowest responses, corrections began relatively early in the movements. In the case of early onset distance perturbations, responses occurred by the time the hand had moved approximately 70% of the distance to the target. Assuming even a minimal sensorimotor delay of 80 ms, these corrections would have to have been initiated while the hand was less than half-way to the target, prior to the point of peak hand velocity. The observation of such early perturbation responses, combined with the smaller-than-expected differences in response latencies, argues strongly against a model in which distance feedback is only used at the end of movements.
While the responses we observed to distance perturbations were slower overall than those to direction perturbations, the difference was much smaller than would have been expected by comparing the Sarlegna et al. and Saunders and Knill results. This could largely be due to the more sensitive analysis applied here to measure response times. Still, the data show significant differences in response times between direction and distance perturbations, as well as differences across distance perturbation subconditions. One possibility is that these effects result from differences in sensory uncertainty of visual feedback about finger direction and distance relative to the target. These two components correspond to different perceptual dimensions (speed versus motion direction) and have differing amounts of variability over time. To test whether sensory uncertainty could explain the differences in responses we observed, we simulated a feedback controller that optimally integrates sensory feedback with an ongoing estimate of hand state to generate online control signals for the hand, and compared its performance to the human data for the same experimental conditions.
In Saunders and Knill (2004), we applied a feedback control model to the conditions of a motion direction feedback perturbation study to characterize the expected response of a feedback to a controller limited by noise in the visual coding of finger position and velocity. The model combines an optimal sensory estimator for finger position and velocity with an online control law derived from the minimum jerk criterion (Hoff and Arbib 1993) for generating motor commands to move the finger. The sensory component of the model is an adaptive Kalman filter that maintains a running estimate of the state of the hand using noisy sensory information and outgoing motor signals. The filter continuously updates its estimate of hand state by combining its current estimate with the information derived from new sensory (visual) signals and an estimate of the motor command that was most recently sent out to the system. To account for sensorimotor delays, the filter runs in a delayed loop, and the delayed estimates of hand state are input into a forward model that uses estimates of the motor commands sent out during the delay period to “predict” the current state of the hand from the delayed estimate.
We adapted our previous model to the present experiments to test whether responses to direction and distance perturbations could be similarly explained in terms of sensory feedback noise. As before, we incorporated both signal-dependent noise on the control signal and state-dependent noise on the sensory feedback. The latter models speed-dependent changes in motion discrimination thresholds and eccentricity-dependent changes in spatial acuity. We also assumed that the sensory estimates of both hand position and velocity were temporally low-pass filtered in concordance with observed psychophysical behavior. Details are given in Appendix A. Most of the parameters for the sensory filters—the noise levels for visual estimates of the direction and speed of finger movement, and the noise levels for visual position estimates—were derived from published psychophysical data, as described in the Appendix. The one noise parameter that we have added to model the current data is designed to capture the uncertainty in position estimates that would be induced by temporal blurring of the visual input. The early visual coding of position is derived from temporally low-pass filtered images—effectively limiting positional acuity for a moving object like the finger by the spatial extent of the temporally blurred image of the object. We modeled this motion-blur induced uncertainty with a noise term whose standard deviation was proportional to the velocity of the finger. The total noise on position estimates is then given by the sum of a noise term that models static positional acuity (incorporating the effects of eccentricity on spatial sampling), and a term that models the positional noise induced by motion blur. The latter noise is zero in the direction perpendicular to the finger’s motion and is proportional to finger speed in the direction of motion. The constant of proportionality was set to equate the motion blur noise to what would be expected for a visual signal blurred over a 40 ms window, equal to the time constant used to model the low-pass filtering of the visual signal (see Appendix A). The only free parameters in the model were the noise terms for the control signal and the sensorimotor delay. These were set to the same values used to fit the data in the previously reported study (Saunders and Knill 2004). Thus, all model parameters were fixed prior to running the simulations and none were fit to the current data.
In our model, we assume that feedback control is based on the system’s estimate of the current state of the hand relative to the target. Under this assumption, perturbations evoke responses because they change the observed position and velocity of the hand relative to the target. It is also possible that subjects reacted to the conflict between observed and expected hand position on perturbed trials. For example, if subjects attempted to guide their hand along a predetermined kinematic trajectory, the relevant control signal would be the difference between observed and desired current position, which would not directly involve comparisons between hand and target. The present results cannot distinguish these possibilities, since perturbations alter both the visual position of the hand relative to the target and the visual position of the hand relative to its expected, physical position. Thus, there is some ambiguity as to the specific control variable that drives subjects’ responses to the perturbations tested here. In another related study (Saunders and Knill 2004), we tested conditions that did allow greater comparison of control strategies. The observed results were inconsistent with a strategy of steering the hand along a predetermined trajectory, which is one natural strategy that would depend solely on the error between observed and expected state, rather than target-relative state. Note also that our model is not insensitive to conflicts between observed and expected hand state. However, its performance is not directly driven by differences between observed and expected hand state; rather, these variables affect performance via their contributions to the integrated estimate of instantaneous hand state.
The data show evidence for online feedback control of both movement direction and distance. For both dimensions, perturbations produced responses after relatively short latencies: 117 ms for direction perturbations, and 130–200 ms for distance perturbations. In the case of early onset perturbations, responses were observed by the time the hand was 50–70% of the distance to the target, implying that the corrections were initiated prior to the hand reaching peak velocity.
For direction perturbations, response latencies were the same for early and mid-reach perturbations, and for fast and slow movements. This replicates our previously-reported findings (Saunders and Knill 2003), and further supports the hypothesis that visual feedback is used continuously for control of reaching movements. The estimate of feedback delay computed from the direction perturbation data, 117 ms, is close to the reaction times observed for corrections in response to target perturbations (Pelisson et al. 1986; Prablanc and Martin 1992; Brenner and Smeets 1997). It also closely corresponds to the feedback delay we used to model results from another experiment, in which visual feedback was perturbed in ways that selectively altered motion and position feedback signals (Saunders and Knill 2004). These consistencies support our current, shorter estimate of feedback latency. As discussed above, we believe that our previous estimate of feedback latency had larger positive bias, and that the response time computed from the present data more accurately reflects the feedback delay.
The responses to distance perturbations were not invariant to onset time and movement speed. However, the differences in response times were smaller than would be expected under a hypothesis of end-phase feedback control. A control model that relies on an optimal Kalman filter for updating estimates of finger state from sensory feedback replicates all of the main patterns found in the data. The sensory noise parameters used for model simulations were drawn from psychophysical data on position and velocity acuity and were the same as those used to model previous data (Saunders and Knill 2004). The one piece that we added to the model here was a position noise term that models the effects of motion blur on position estimates. The effects of motion-blur on position estimates effectively gives rise to the differences between the model’s responses to direction and distance perturbations. The model simulations demonstrate that the differences across conditions observed in the human data are consistent with reasonable assumptions about the amount of perceptual noise present in the relevant control dimensions. Thus, despite some variations in measured reaction times, the results for distance perturbations remain consistent with the hypothesis of continuous feedback control of movement distance.
A conventional view is that visual feedback contributes to control of reaching movements only during an end-stage, when the hand is close to the target. As described above, our results clearly contradict this model. One reason that feedback might not be expected to contribute during the initial portion of a movement is because visual feedback about the state of the hand relative to the target would be poor. However, under the conditions tested here, the hand could still be visually compared to the target throughout the movement, even if subjects fixate the target. The quality of information about the hand state does strongly depend on its eccentricity relative to the target, which is one reason that we included eccentricity-dependent noise parameters in the model. In fact, the model performance demonstrates that varying amounts of sensory noise are sufficient to explain apparent differences in sensorimotor reaction times. Note that visual feedback from the hand is not strictly necessary to perform reaching tasks, as evidenced by our ability to pick up objects with our eyes closed. The accuracy of our model similarly decreases only modestly with the removal of visual feedback. Thus, it is not surprising that the initial influence of visual feedback would be small and difficult to detect, particularly if accuracy were used as a measure. However, the perturbation responses observed here demonstrate that the human sensorimotor system does take advantage of visual feedback about the hand throughout movements, even for the minimal feedback provided in our experiment (a small dot representing the fingertip).
The current results seem to conflict with those of Sarlegna et al. (2003), who found only weak responses to perturbations of visual feedback from the hand, and after a much longer latency of 470 ms. While our results demonstrate that there is indeed a difference between responses to perturbations affecting movement direction and distance, we still observed relatively fast responses to distance perturbations as well as direction perturbations. Response latencies ranged from 130–200 ms depending on perturbation onset and movement speed. While these latencies are longer than those observed for direction perturbations, in all cases they are much shorter than the latencies reported by Sarlegna et al. The cumulative amount of correction was also greater in the present study.
The analysis method used here largely accounts for the differences. Were we to have marked responses as beginning when the average finger trajectories from perturbation trials deviated significantly from those from unperturbed trials, we would have found similar response times to Sarlegna et al., in that responses would only have appeared at the end of movements. This overestimation results from the high variance in subjects’ raw trajectories. Only by using the autoregressive model to account for correlations within trajectories is one able to obtain a high-enough signal-to-noise ratio to measure response times with any degree of sensitivity. Even with this improvement, our estimates are likely to be high, as demonstrated by the model results: when the same method was applied to the simulated data, response times were significantly higher than the sensorimotor delay built into the model. Furthermore, in the Sarlegna et al. study, the “onset” of perturbations was immediately after initial saccades, while the perturbations used here occurred during the main transport portion of movements, well after initial saccades. There could be some refractory period after saccades during which feedback cannot effectively be used, resulting in somewhat longer apparent feedback delays. Other methodological factors could account for the remaining differences in our findings (for example in proportional correction). In the Sarlegna et al. experiment, movements were directed to positions in space, such that the virtual target provided no impact resistance or haptic feedback. This is less natural than the pointing movements used in our experiment, which were directed toward virtual targets coincident with a physical tabletop, and might have somehow altered subjects’ strategies. Finally, we marked the ends of movements differently than Salegna et al., in part because contact with the tabletop effectively stopped movements in our experiments. Small differences in this variable could easily have led to the differences we found in the proportional amount of correction for perturbations.
While the perturbation responses reported here clearly demonstrate that visual feedback contributes to online control, the data are ambiguous as to the specific perceptual dimensions that underlie this control. The distance perturbation conditions required subjects to adjust primarily the extent of movement, and the direction perturbation conditions required adjustments primarily in the direction of movement, so we interpret responses to these perturbations as reflecting feedback control of distance and direction respectively. However, there are many possible models for controlling these task dimensions, with differing relevant control variables, which these data cannot distinguish. For example, are corrections based on the current state of the hand relative to the target, or relative to some predetermined kinematic plan? Another question would be whether the representation of hand state used for control includes only position and velocity, or higher derivatives as well. Such questions are beyond the scope of the present study, but are addressed to some extent in recent related paper (Saunders and Knill 2004).
We have also neglected to consider the role of depth information. Since movements were largely within a single plane, we have treated the task and data as two-dimensional. However, because the tabletop was slanted relative to the viewer, the visual information about the state of the hand relative to the target included a depth component. In particular, direction perturbations displaced the virtual finger in depth as well as in the frontal (projection) plane. For movements that are nearly planar, as we observed, the depth and frontal components of hand kinematics are highly correlated. When we re-analyzed the kinematic data for the direction perturbations using the vertical position of the hand in viewer-centered coordinates (in the image plane) instead of in tabletop coordinates, we found essentially equivalent results. The main complication is that the effective magnitude of the perturbation signal depends on perceptual sensitivity to both depth and frontal dimensions. If observers were much less sensitive to motion in depth than to frontal motion, as one would expect, direction perturbations would have been attenuated somewhat due to foreshortening. In fact, the model has this foreshortening effect built into its model of the sensory signal, which it assumed to be the movement of the finger in the image plane (see details in Appendix A). This quantitative ambiguity does not affect the distinction between direction and distance perturbation conditions. Control of movement direction might be based to varying extents on either 2-D or 3-D position and velocity, but regardless of the role of depth information, the task-relevant dimensions would be different than those for controlling movement extent (distance and speed).
In conclusion, the results provide evidence that visual feedback from the hand is used for online control of both the direction and distance of the hand. The results for direction perturbations strongly implicate continuous feedback control of movement direction. The results for distance perturbations are also consistent with continuous feedback control, particularly when the perceptual noise in relevant control dimensions is considered. While there is evidence that movement direction and distance are controlled independently during movement planning, we found no evidence for a qualitative distinction between these dimensions with regard to online feedback control. Rather, our results are consistent with a model in which available visual information is continuously integrated in an optimal manner throughout movement to estimate hand state and generate corrective adjustments to movements. Apparent differences in subjects’ sensitivity to direction and distance perturbations are well accounted for by differences in the sensory noise that corrupts position and speed estimates in the principal direction of movement versus in the perpendicular direction.
The second term is the control law given in Eq. 6 above, the third term represents the signal-dependent motor noise, and the fourth term represents the state-dependent noise used to model state-dependent uncertainty in the sensory estimates of position and velocity.
The Kalman filter runs delayed by Δ ms, so the delayed state estimates are propagated forward through the state update equations using the remembered values of the motor commands, ut, to arrive at an estimate for the current hand state.
In order to simulate the model, we had to set parameters for the visual noise in position and velocity. We chose values that fit with observed psychophysical data, as described below. We fit the motor noise parameters and the sensorimotor delay by hand to subjects’ data.
The constants of proportionality in Eq. 17 determine the appropriate elements of the Di matrices that model the state-dependent noise in position. The constant noise term is the standard deviation of the corresponding elements of the constant noise vector, ξ. The noise in visual estimates of target position was modeled as being constant, since the target is stationary and assumed to be fixated during movement. We assumed a standard deviation of 0.5 mm (equivalent to a visual angle of 3′ of arc).
Modeling motion noise
Again, we approximated the principle direction of motion to be in the x-direction. We scaled the parameters by a constant factor of 14.3 so that a sensory version of the Kalman filter run on static stimuli for 500 ms would give velocity estimates with the standard deviations listed above. The constants of proportionality in Eqs. 20 and 21 determine the appropriate diagonal elements of the Di matrices that model the state-dependent noise in position. The constant noise terms are the standard deviations of the corresponding elements of the constant noise vector, ξ.
Other model parameters
The coefficient of proportionality (0.05) for the proportional noise determined the diagonal elements of the C1 and C2 matrices corresponding to the motor commands (the jerk signals in x and y), while the constant noise term determined the standard deviation of the same elements of the constant noise vector, ξ. We simulated the model with a sensorimotor delay of 115 ms, for a duration of 750 ms, iterating the system with a time step of ∂=2 ms. Perturbations were induced at t=214 ms for the early perturbation and t=326 ms for the mid-reach perturbation. The duration of movement was set based on estimates for the average time that subjects’ fingers came to rest for the slow movement conditions, and the time of perturbation was set based on the average time after the start of movement that subjects’ fingers reached a quarter or half of the distance to the target.