Introduction

It is known that saccades are initiated before the visual information is fully processed. This implies that eye movement will not always be appropriate. Indeed when reading, 14% of the saccades are regressions (Starr and Rayner 2001), and during visual search the eyes often (in 5–55% of the trials) move away from the target and then immediately return (Hooge and Erkelens 1996). We recently reported that when hand movements are used in conjunction with eye movements in a search task, not only the eyes, but also the hand makes return movements (Liesker et al. 2008). We concluded that the hand movement is planned before visual information processing is completed and that the visual information has not been fully processed by the last moment at which the hand movement to the next item can still be cancelled.

When the hands are moving to process tactile items, they never pass the target (Overvliet et al. 2007). One might conclude that hand movements are only initiated after tactile information processing is complete. However, the two studies did not only differ with respect to sensory information: the eye and hand had to move together in the study by Liesker et al. (2008), whereas eye movements were irrelevant in the study by Overvliet et al. (2007).

Our eyes and our hands often move together to perform everyday tasks. For instance, when making tea or preparing sandwiches, our eyes often move ahead of our hands (Land and Hayhoe 2001). In a pointing task, Neggers and Bekkering (2000) even found that subjects were unable to make a saccade towards a new target before their hand had reached the initial target location. However, Rotman et al. (2004) found that subjects did not make saccades to the position they tapped, but kept pursuing the disk with their eyes when tapping targets flashed near a moving disk. Smeets et al. (1996) studied subjects’ head movements while executing various natural manual tasks requiring gaze shifts. The movements of hand and gaze were uncoupled to a large extent, allowing them to conclude that the head not only followed the gaze, but also the hand. Also when searching with the eyes and hand, there is evidence that the effectors can move independently (Liesker et al. 2009). Thus, the exact details of a task determine whether eye and hand movements are coupled. Does how fast hand movements are planned depend on whether or not the eyes move to the same positions?

To investigate whether these return movements are linked to return movements made by the eye, we compare the two methods of moving the field of view relative to a scene in visual search. In one method, a window is moved by hand over the scene to make an item visible. In the other, the window is fixed at the centre of the screen and the scene is moved behind it by hand. The scene was a circle of seven items: six Cs and one O. The items were projected onto a graphics tablet (see Fig. 1). Participants could always see the item positions (as dark grey disks that had the same diameter as the items; grey spots) and a window through which the actual shape of the items could be perceived. Participants had to move a pen across the graphics tablet to either move the window over the scene or to move the scene behind the window, to make the items visible.

Fig. 1
figure 1

a The experimental set-up. The stimulus was projected onto a projection surface. Participants saw this stimulus via a mirror, making it appear to coincide with the surface of a graphics tablet. Participants moved a pen over the graphics tablet and indicated that they had found the target by pressing the space bar on the keyboard. b, c The stimulus. The dark grey spots indicate the item positions. The white circle is the window through which an item is visible. Image B is an example of the condition in which moving the pen over the graphics tablet moved the window over the scene, so the pen was ‘linked’ to the window. Image C is an example of the condition in which moving the pen over the graphics tablet moved the scene behind the window, so the pen was ‘linked’ to the scene

Full size image

When moving the window over the scene, we can expect the eyes to either follow the moving window (and the hand) or to make saccades between item locations and wait at each location for the window to arrive. When moving the scene behind the window, the eyes are forced to move in a different way from the hand. The eyes could either remain fixated on the window at the screen centre, or they could saccade away from the target that was processed (which was visible in the window) to the next item (which was visible as a grey spot) and track this spot as it was moved into the window. The two methods of moving the visible part of the scene require the same hand movements (stepwise movements along a circle) and in both cases the eyes may make saccades and pursue the target or window that is being moved by the hand, but the relationship between what is visible and the eye and hand movements is different: if the window is moved, the saccades will be in the same direction as the hand movements, whereas if the scene is moved, the saccades will be in the direction opposite to the hand movements.

If there is an intrinsic benefit in moving eye and hand together, then moving the window over the scene will be easier than moving the scene behind the window, resulting in shorter viewing times. If the hand makes a return movement to the target because it was following the eyes, then moving the window over the scene will also lead to more (or larger) return movements. Because the number of return movements to the target depends on the viewing time (Liesker et al. 2008), we varied the difficulty of the task by varying the window size and the contrast between the items and the background to see how these variables affect viewing time, hand movement velocity and the extent of return movements, and in particular whether the two tasks differ in the relationship between these variables.

Methods

Participants

Twelve participants, five male and seven female, aged between 19 and 32 years, participated in this experiment. All participants were right handed and had normal or corrected to normal vision. None of the participants were aware of the goal of the study.

Apparatus

Participants were seated in a chair in the set-up shown in Fig. 1a. The stimulus (Fig. 1b, c) was generated by an Apple Power Mac G4 (resolution 1,024 × 768 pixels for a 57.5 × 43-cm image; refresh rate 85 Hz) and projected by a video projector (Boxlight) onto a back-projection screen. Participants looked downwards into a mirror where they saw the reflection of the projected image that exactly coincided with the felt surface of the graphics tablet (Wacom Digitizer II, sampling frequency 200 Hz). Participants adjusted the height of the chair, so that they could see the whole image in the mirror and move the pen comfortably over the graphics tablet. The distance from the eyes to the projection of the image was about 50 cm, so that 1 cm corresponds with about 1° of visual angle (depending on the location on the screen). Participants put their non-dominant hand on the keyboard, which was positioned under the graphics tablet. They indicated that they had found the target by pressing the keyboard’s space bar.

Procedure

At the beginning of each trial, a dark grey spot (radius 2 cm) appeared at the centre of the bright grey image on the tablet. Participants were instructed to place the tip of the pen on this spot, using their dominant hand. Once they did so, the 7-item positions appeared as 2-cm radius dark grey spots, positioned at equal distances on a 12-cm radius circle around the centre of the screen. At the same time, the central spot was replaced by a circular window (size varied between conditions), which revealed the white background of the scene behind the bright grey surface. By moving the pen across the graphics tablet, participants either moved the window across the scene (move window condition; Fig. 1b) or they moved the scene behind the window (move scene condition; Fig. 1c). In the latter case, the pattern of grey spots moved with the pen. In both conditions, there was a delay between moving the pen and the change in appearance in the image of about 60 ms. As the window moved over an item, or an item moved behind the window, the part of the item that was within the window was visible as a black drawing on the white background. To keep the relationship between arm movements and what is visible within the window equal for the two movement conditions, the scenes in the two conditions were (vertical) mirror images of each other. This means that the ‘top’ item in the move window condition was the ‘bottom’ item in the move scene condition.

The items were six non-target Cs and one target O. The task for the participants was to find the O as quickly as possible. The outer radius of each item (C or O) and that of the dark grey spots indicating the item positions was 2 cm. The inner radius of all items was 1.3 cm. The gaps in the Cs were 0.4-cm wide. The participants were instructed not to lift the pen from the tablet during the task. They were instructed to find the target and then press the space bar on the keyboard to indicate that they had found the target, while keeping the pen on the tablet and the target within the window. Thus, if they moved the window beyond the target or the target beyond the window and realised that they saw the target, they were to move back before pressing the space bar. Subsequently, the starting position for the next trial was presented. The participants were given six practice trials before starting each of the two movement conditions of the experiment. We varied the size of the window (radius 3.3 or 2 cm) and the contrast between the items and the background: The luminance of the white background was 21.5 cd/m2 that of the bright grey screen was 8.0 cd/m2 and that of the dark grey spots was 5.8 cd/m2. The luminance of the stimuli was 1.7 cd/m2 (high contrast), 11.5 cd/m2 (moderate contrast) or 18.0 cd/m2 (low contrast), respectively. The position of the gap in the Cs was chosen at random from the top, bottom, left and right sides for each item on each presentation.

The different movement conditions, window sizes and contrast levels were presented in separate blocks of trials. Hence, for each participant, the experiment consisted of 2 (movement condition) × 2 (window size) × 3 (contrast) = 12 blocks. Each block consisted of 21 trials, with each of the 7 possible target positions appearing three times, in random order. Participants could take a break between blocks. The order of the two movement conditions was counterbalanced across participants. Within each of the two sets of blocks with the same movement condition, a random order of the window size and contrast blocks was selected for each participant. During the experiment, the x and y positions of the tip of the pen on the graphics tablet were collected at 200 Hz. Trials in which the tip of the pen was not positioned on the target when participants gave their response were considered invalid trials and were not included in the data analysis.

To ascertain that subjects moved their eyes in the way we expect them to in the two types of tasks we measured the eye movements of one participant during one block in the move window condition and one block in the move scene condition (both with a large window and high contrast), using the Eyelink II system (SR Research Ltd.; sampling frequency 500 Hz).

Data analysis

The first part of each trial, from when participants put the tip of the pen on the dark grey spot at the centre of the stimulus circle until the first item became visible within the window, was excluded from further analysis because during that time there was no information to process. The position of the tip of the pen was analysed from the moment the first item became visible within the window until the moment the space bar was pressed. For each trial, the mean duration per item inspected was determined (the total trial duration divided by the number of items inspected), as was the peak velocity, the total distance covered and the distance covered until the first moment the target became visible within the window.

Return movements are movements away from and back to the target that has just been seen. These were quantified by the distance covered after the first moment the target became visible within the window (and before the space bar was pressed). This measure captures all movements when the target is visible through the window as well as the movements away from the target and back. The advantage of using this measure is that it is sensitive to small return movements within the target area, and measures these movements independent of window size. The eye movement data were smoothed using a second-order low-pass Butterworth filter (cut-off frequency 100 Hz).

Repeated measures analyses of variance were conducted using SPSS software (version 11.0.2 for Mac OS X), with movement condition (2; move window vs. move scene) window size (2) and contrast (3) as independent variables and mean duration per item, peak velocity, total distance covered, distance covered until the target became visible within the window and distance covered after the target first became visible within the window as dependent variables. For Figs. 2, 3, 4, 5, values were averaged across trials and then across participants; standard errors (Fig. 2, 3, 4) were calculated across participants.

Fig. 2
figure 2

The total trial duration divided by the number of items inspected for the two different movement conditions, the two window sizes and three contrast levels. Error bars indicate the standard error of the mean across participants

Full size image
Fig. 3
figure 3

Peak velocities of the movements between every two items for the two different movement conditions and the two window sizes. Error bars indicate the standard error of the mean across participants’ average values

Full size image
Fig. 4
figure 4

Total distance covered during a trial (total bars), consisting of the distance covered until the target first became visible (lower, brighter bars) and the distance covered after the target first became visible (upper, darker bars) in the two different movement conditions for the two window sizes. Error bars indicate the standard error of the mean across participants

Full size image
Fig. 5
figure 5

Distance covered after the target first became visible as a function of the viewing duration for a distractor. Separate dots represent items 2–6, both when they were a distractor (x-axis durations) and when they were the target (y-axis distances). Data were averaged across contrasts and subjects. Viewing durations increase and return movement lengths decrease from the second item to the sixth item

Full size image

Results

Trials in which the tip of the pen was not positioned on the target when participants gave their response were considered invalid trials and were not included in the data analysis. This was so for 69 of the 3,024 trials (2%).

Before interpreting any differences between the conditions in the extent to which subjects moved on after reaching the target, we must first examine how performance in general was affected by the conditions. Did the durations that the items were inspected differ for the two movement conditions, for the two window sizes and for the three contrasts? Figure 2 shows that the mean duration per item is shorter when subjects move the window over the scene (circles), than when they move the scene behind the window (squares) [F(1,11) = 46.618, P < 0.001]. The figure also shows that the duration per item is shorter for the large window (filled symbols) [F(1,11) = 141.130, P < 0.001] and when the contrast between items and background is higher [F(2,22) = 53.441 P < 0.001]. Finally, there is a significant interaction between window size and contrast for the duration per item [F(2,22) = 14.638, P < 0.001]: the effect of contrast is larger for the small window.

The longer durations are mainly caused by longer viewing times (not shown), but might also partly be caused by the movements being made at different speeds (Fig. 3). The peak velocity is higher when subjects move the window over the scene than when they move the scene behind the window [F(1,11) = 19.452, P < 0.005]. The peak velocity is also higher for the large window [F(1,11) = 39.504, P < 0.001]. The effect of contrast on peak velocity was not significant.

Figure 4 shows that, on average, the total distance was longer when subjects moved the window over the scene, than when they moved the scene behind the window [F(1,11) = 5.041, P < 0.05]. The total distance covered was clearly longer for the large window than for the small window [F(1,11) = 12.895, P < 0.005]. There was no significant interaction between the two factors. For the distance covered until the target became visible, there were no significant effects. Subjects covered on average a distance of 34.9 cm before the target first became visible, which is only slightly more than the distance of 31.5 cm that we expect them to cover if they move straight between the items’ centres (3 times the inter-item distance of 10.5 cm).

The distance covered after the target first became visible (upper dark bars in Fig. 4) was much more than was necessary for the target to become completely visible within the window. This was so for all four (two movement and two window size) conditions. On average, the distance covered after the target’s edge first entered the window was 12.2 cm. Subjects only needed to move 4 cm (twice the item radius of 2 cm) for the target to be fully visible within the window (in both window size conditions) and only 5.3 cm to place it at the centre of the large window (4 cm for the small window). Hence, in all conditions, they made return movements.

All the effects on total distance mentioned above appeared in the distance covered after the target first became visible (upper, darker bars). This means that larger return movements must have been made when subjects moved the window over the scene, compared to when they moved the scene behind the window. Testing this directly showed that this effect is significant [F(1,11) = 14.256, P < 0.005]. The distance after the target first became visible was 8.5 cm when the scene moved behind the small window and 12.3 cm when the scene moved behind the large window. It was 11.6 cm when the small window moved across the scene and 16.5 cm when the large window moved across the scene. Overall, return movements were significantly longer for the large window than for the small window [F(1,11) = 38.394, P < 0.001]. The differences in return movement lengths are too large to be accounted for by the higher movement speed (and a constant overshoot duration). The effect of contrast on the length of the return movements was not significant.

Does the length of the return movement depend on the time taken to inspect the items? In the analyses above, we showed that the latter depends on the task and window size. It also increased as more and more items were scanned, probably because the likelihood of the next item being the target increased with each distractor seen (as in our previous study Liesker et al. 2008) Figure 5 shows a scatter graph of the mean distance covered after the target first became visible as a function of the mean viewing duration per distractor. Separate points represent the second to sixth item to be inspected, averaged across contrasts and subjects. The horizontal values are the viewing durations for distractors being the second to sixth item inspected during the trial, and the vertical values are the distance covered after the target was first visible when it was the second to sixth item that was inspected. Item 1 is not included, because we had previously seen that this item is treated differently. Item 7 is not included, because if the sixth inspected item was a distractor, subjects knew that the seventh and last item had to be the target, so no return movements are expected and indeed none are found.

The durations per item in Fig. 5 are shorter than the durations per item in Fig. 2 because Fig. 2 shows the total trial duration divided by the number of items inspected, whereas the values in Fig. 5 exclude the target viewing time and the return movement. Figure 5 shows that there is a systematic relationship between the duration per item and the distance covered after the target first became visible. Irrespective of the condition, the shorter the duration the items are inspected, the larger the return movement. Apart from its influence on the duration per item, the task (move window or scene) does not appear to make a fundamental difference.

For the move window condition, the recorded gaze direction data (Fig. 6a) and the velocity profile (Fig. 6c) confirms that the eyes moved with the window (and the hand) over the items, with the eyes making saccades between item locations. For the move scene condition, the gaze direction data (Fig. 6b) and the velocity profile (Fig. 6d) confirm that the do not follow the hand (which moved in exactly the same manner as in the move window condition). Gaze did not remain at the window in the centre of the screen; there were many saccades and intervals with pursuit movements, presumably because the next spot was fixated before reaching the window and tracked as it was moved into the window.

Fig. 6
figure 6

Eye movements of one participant. a, b Gaze directions across a complete block (21 trials) in the large window and high-contrast condition. c, d Velocity profiles for a period of 2 s taken from the gaze direction data shown in a and b. The ellipse gives an indication of the window size in this condition

Full size image

Discussion

When the hand moved the scene behind the window, the eyes and hand did not always move together. This is obvious because otherwise the subjects would have been unable to fixate the items. In particular, when making a saccade to the next item while moving that item into the window, the eyes and hand moved in opposite directions. When the hand moved the window, the saccades were in the same direction as the hand movement. Thus, in terms of the coordination between eyes and hand, the two conditions are quite different.

The duration of 503 ms per item (including the saccade; Fig. 5) that we found in the move large window condition is comparable with the duration per item found in our previous experiment (Liesker et al. 2008; 519 ms in a large window condition). These durations are much longer than the 280 ms fixation duration (excluding the saccade) found by Cornelissen et al. (2005) in a search task (finding an O amongst Cs with a comparable gap size: 0.2°) in which subjects moved a window that provided a comparable field of view (as large as one item and its surrounding background: 5°) by moving their eyes (here too all item locations were indicated). These results suggested that having to move the eyes together with the hand has a big disadvantage over moving just the eyes to see the item. We here show that having to move the hand in a different direction than the eye has some additional costs.

We found performance differences between the move window condition and the move scene condition: in the move window condition subjects searched faster. Most importantly, however, the relationship between the duration the items were inspected and the return movement when comparing the two movement conditions was the same as the relationship between these factors when comparing the different window sizes. Thus, the task apparently becomes more difficult in the move scene condition, as it does when window size is reduced, so subjects take longer to move and spend more time inspecting each item. As a result of this they make smaller return movements. The extent to which they make smaller return movements is similar to what one would find for a different window size that would give rise to the same inspection time.

The relevant difficulty appears to be the difficulty in making the movements: the more difficult the movements are, the more time is needed for planning the movement, and thus less chance that visual processing is not completed in the time used for planning the movement. That planning of the movement is critical, rather than perception, is evident from the influences of contrast. The longer search times for low contrast stimuli (Fig. 2) are not accompanied by smaller return movements. Thus, moving the scene beneath the window appears to be the more complicated motor task, probably because the hand and eyes do not simply move together.

We can conclude that the hand making return movements to the target is not directly related to the eyes doing so, but rather that moving on before the information has been fully processed (and therefore making return movements to the target) is a general principle of visuomotor control.