The way we perceive the visual world is strongly influenced by what we are doing or intending to do. From the vast amount of information available at every moment, the visual system filters out what is relevant for our current behavioral goals, and may thus be seen as a system optimized for gathering action relevant visual information about our environment. This action-related selective processing of visual information is often referred to as “selection for action” (Allport, 1987). The assumption of a close relationship between visual perception and action has received substantial empirical support (e.g., Schütz-Bosbach & Prinz, 2007), but selective processing continues to be essential for the visual system beyond the perceptual stage, namely for maintaining relevant visual information over short periods of time. In these experiments, we investigated whether selection for action also occurs during maintenance in visual working memory (VWM).

Action planning and visual attention

Early evidence for a coupling of action planning and visual selection was provided by studies in which participants were to perform saccadic eye movements in combination with a visual detection, discrimination, or identification task. Hoffman and Subramaniam (1995) had participants saccade to one of four locations and detect a target letter briefly presented at one of the locations before the movement was initiated. Detection accuracy was best when the target letter had been presented at the saccade goal, even when participants were explicitly cued to attend to another location. Similar results were obtained by Kowler, Anderson, Dosher, and Blaser (1995), who combined an eye-movement with a letter-identification task. This obligatory coupling between saccade programming and visual attention appears to be spatially specific to the intended location rather than to the actual landing position of the saccade (Deubel & Schneider, 1996). When sequences of saccadic eye movements were executed, performance in an identification task was better at any of the saccade goal locations than at any of the other locations, suggesting that attention was allocated in parallel to all movement goals (Baldauf & Deubel, 2008a; Godijn & Theeuwes, 2003).

One could assume that eye movements are special in that the link between overt and covert attention is particularly strong (see also Rizzolatti, Riggio, Dascola, & Umiltà, 1987), but remarkably similar conclusions have been drawn for hand movements (for a review, see Baldauf & Deubel, 2010). In several studies, Deubel and colleagues have shown that during the preparation of manual pointing movements, performance in a visual task was best at the location of the pointing goal, indicating that attention was shifted to the goal location prior to movement onset (Baldauf & Deubel, 2008b, 2009; Baldauf, Wolf, & Deubel, 2006; Deubel, Schneider, & Paprotta, 1998). This coupling of attention to the movement goal location was observed in spite of participants’ knowledge of the location of the visual target (Deubel et al., 1998), suggesting that it is obligatory. In addition to these studies on how spatial attention is linked to an action goal, others have demonstrated the impact of manual actions on attentional weighting of features (e.g., Craighero, Fadiga, Giacomo, & Umiltà, 1999; Müsseler & Hommel, 1997) and feature-dimensions (e.g., Fagioli, Hommel, & Schubotz, 2007; Wykowska, Schubö, & Hommel, 2009) in visual search or visual discrimination tasks.

Presumably, the deployment of spatial attention to a movement goal serves to ensure that all the relevant information necessary to specify movement parameters is available and preferentially processed. Given that processing efficiency has been shown to decrease when attention is distributed over a larger region of the visual field (e.g., Castiello & Umiltà, 1990; Müller, Bartelt, Donner, Villringer, & Brandt 2003), attentional deployment should ideally be spatially specific to the actual movement goal. Indeed, a high spatial specificity has been demonstrated for manual pointing movements (Baldauf et al., 2006; Deubel et al., 1998). When there are multiple pointing movement goals, such as for bimanual movements and movement sequences, attention appears not to be uniformly distributed across the visual field. Rather, movement-relevant goal locations are selected in parallel by spatially distinct attentional foci, whereas intermediate locations remain unattended (Baldauf & Deubel, 2008b, 2009; Baldauf et al., 2006).

Interestingly, the intention to perform a movement does not seem to be necessary to affect visual perception, but the mere presence of a hand near visual stimuli alters their processing. Reed, Grubb, and Steele (2006) had participants place one hand next to one of the target locations in a covert attention task and observed facilitated detection of targets near the hand. The authors proposed that this nearby-hand effect might be due to an attentional prioritization of space near the hand. Other studies have corroborated this idea. Using three classic attention paradigms (visual search, inhibition of return, and attentional blink), Abrams, Davoli, Du, Knapp, and Paull (2008) were able to show that a nearby hand disrupted attentional disengagement, indicating a more detailed evaluation of objects in the perihand space. An extended analysis of objects that are near the hand can be assumed to optimize potentially upcoming actions by providing the relevant visual information. Indeed, visual sensitivity in the perihand space has been shown to be improved (Dufour & Touzalin, 2008). Moreover, a recent neurophysiological study provides evidence of a modulation of neuronal responses in an early visual area, namely a sharpening of orientation tuning and reduced response variability of neurons in macaque area V2 in the presence of a nearby hand (Perry, Sergio, Crawford, & Fallah, 2015).

Attentional modulation of maintenance in VWM

VWM, as that part of the visual system that allows us to maintain and manipulate visual information over short periods of time, is important for higher cognitive functions and even simple actions such as saccades. Because the capacity of VWM is highly limited (Cowan, 2001; Fukuda, Awh, & Vogel, 2010; Luck & Vogel, 1997), selective processing is crucial for its optimal use, necessitating attentional mechanisms not only for selecting the most relevant information for encoding, but also for updating and weighting VWM contents. Indeed, attentional mechanisms modulate VWM throughout all processing stages, from encoding to retrieval (Gazzaley & Nobre, 2012). During maintenance, attention can be directed toward specific representations, improving memory for the respective selected information (e.g., Astle, Summerfield, Griffin, & Nobre, 2012; Griffin & Nobre, 2003; Nobre et al., 2004). Irrelevant information can be excluded from memory (Kuo, Stokes, & Nobre, 2012; Williams, Hong, Kang, Carlisle, & Woodman, 2013), or a weighting of information according to its relevance can be introduced by means of maintenance in different representational states, within and outside an internal focus of attention (e.g., Heuer & Schubö, 2016b; LaRocque, Lewis-Peacock, & Postle, 2014b; Rerko & Oberauer, 2013; Zokaei, Ning, Manohar, Feredoes, & Husain, 2014). Moreover, representations can be flexibly selected based on whatever type of stimulus characteristic determines their relevance, be it their spatial location or a feature (Heuer & Schubö, 2016a; Heuer, Schubö, & Crawford, 2016; Li & Saiki, 2014; Pertzov, Bays, Joseph, & Husain, 2013).

Experimentally, the attentional selection of relevant representations is typically induced by presenting a so-called retrocue during the retention interval, indicating certain items as more behaviorally relevant than others. Most studies used spatial retrocues that indicated one or several item(s) by pointing toward the location(s) at which the item(s) had previously been presented. Outside the laboratory, however, the relevance of certain aspects of our visual environment is not specified by an external event, but mostly (if not always) determined by what we are intending to do. Thus, it seems reasonable to assume that our actions do not only influence which visual information we prioritize in perception but also which visual information we maintain. One study reported improved performance with nearby hands in a change detection task, which requires the use of VWM (Tseng & Bridgeman, 2011). However, in these experiments, the hands were continuously placed at the monitor during the trials so that, as the authors themselves suggest, the observed improvement could be the result of perceptual facilitation and not an improvement of working memory per se. To our knowledge, the idea that intended action might influence attention in VWM has not been directly tested.

Rationale of the experiments

In two experiments, we investigated whether the contents of VWM are weighted according to their potential action relevance. As outlined above, spatial attention is automatically drawn to the location of an action goal, and the deployment of spatial attention to items in VWM improves memory for these items. Thus, we hypothesized that performing a movement toward a specific location would result in the allocation of spatial attention to that location, and that memory items that had previously been presented at that location would benefit from this attentional bias, yielding improved memory for these items. In a dual-task paradigm, participants had to memorize a number of objects and perform a pointing movement during the retention interval. The location of the item subsequently tested in the memory task either corresponded to the movement goal location or it was presented at an action-irrelevant location.

In contrast to studies that used retrocues to manipulate task relevance, all memory items were equally relevant for the memory task, but only differed in their potential action relevance. We expected better performance for items that had been presented at movement goal locations than for items that had been presented at action-irrelevant locations. As selective processing is particularly important for the capacity-limited VWM, we additionally investigated whether action-related selective processing would receive priority when VWM reaches its capacity, that is, when memory load is increased to its limit (Experiment 1), and the spatial specificity of the effect relative to the movement goal (Experiment 2). Control conditions without a movement (Experiment 1) and with a movement to a goal that never corresponded to the test item location in the memory task (Experiment 2) served to ensure that any observed effects were not due to perceptual priming resulting from the movement cue or general, spatially unspecific action planning processes.

Experiment 1

Experiment 1 tested whether representations in VWM are weighted with respect to their potential action relevance. In a dual-task paradigm, participants memorized the orientation of various memory items. The memory items were presented among task-irrelevant distractor items, which were included so that the number of memory items could be varied while keeping the total number of items constant. During the retention interval, a cue indicated the movement goal. In movement blocks, participants then executed a pointing movement toward that location. In control blocks without movement, participants were instructed not to respond to the cue. The subsequent test item in the memory task was either presented at a location corresponding to the cued location or at a location that did not correspond to the cued location.

Several potential outcomes may be expected. First, the cue may have no effect on memory performance, given that it is largely irrelevant to the memory task: It pointed to the memory item that would be tested in only one third of all trials (see Method section). If so, memory performance should be similar for all items independent of whether their location corresponded to the cued location. Second, as spatial cues such as arrows are known to guide attention, at least to some extent (Hommel, Pratt, Colzato, & Godijn, 2001; Ranzini, Dehaene, Piazza, & Hubbard, 2009), the cue may cause participants to deploy their attention to the location indicated by the cue. This would result in cueing benefits, namely better memory performance for items presented at locations corresponding to the cued location compared to items at noncorresponding locations. Finally, if action planning causes a specific attention bias, that is, a weighting of representations in VWM due to an attentional bias at movement goal locations, such cueing benefits should be even more pronounced in movement blocks than in control blocks without movement execution (pointing benefits).

Method

Participants

Thirty students of Philipps-University Marburg participated in the experiment. Data from two participants had to be excluded due to poor performance in the memory task (<50 %). The remaining participants (21 female, seven male, mean age = 22 years) were right-handed and had normal or corrected-to-normal visual acuity and normal color vision. Visual acuity and color vision were tested with the OCULUS Binoptometer 3 (OCULUS Optikgeräte GmbH, Wetzlar, Germany). All participants provided informed written consent and were naïve to the purpose of the experiment.

Apparatus

Participants were seated in a comfortable chair in a dimly lit room, facing a monitor approximately 104 cm from their eyes. In front of the monitor, placed at approximately 55 cm from the participants’ eyes, a framed glass plate was mounted on a table. Pointing movements were performed toward this glass plate. For each participant, the glass plate was adjusted in height to ensure that it covered the entire monitor. Participants had a wooden board in front of them with a response box to the left and a movement pad to the right. To respond to the memory task, participants pressed the two buttons on the response box using their left middle and index fingers. The right hand was positioned on the movement pad with a cross to mark the starting position for index finger and thumb. Stimuli were presented on a 22-in. screen (1680 × 1050 pixels). Stimulus presentation and response collection were controlled by a Windows PC using E-Prime 2.0 software (Psychology Software Tools, Inc.).

Pointing movements were recorded using a Polhemus Liberty 240/8 (Polhemus Inc) magnetic motion tracking device measuring x, y and z coordinates at a sampling rate of 240 Hz. Its source was placed 70 cm in front of the participant, under the table upon which the glass plate was mounted. A sensor was fixed on top of the tip of participants’ right index finger. Movement data collection was controlled using MATLAB.

Task and stimuli

The task is illustrated in Fig. 1. A trial started with the presentation of a memory array consisting of differently oriented bars for 200 ms. The number of memory items (set size) was varied between one and five. Color (red/blue) distinguished memory and distractor items. For instance, memory items were presented in blue and distractor items in red (see Fig. 1). The color assignment that defined memory and distractor items was balanced across participants. Participants were instructed to memorize the orientations of the memory items and ignore the other items. After 800 ms, the memory array was followed by the presentation of a spatial cue for 200 ms. This cue pointed to one of the eight locations of the items, which were marked by circle outlines throughout the experiment. The cued location could be the previous location of a distractor item, of the memory item that would subsequently be tested in that trial, or of another memory item. Participants were to point toward this location by touching the glass plate between them and the monitor with the tip of their right index finger. Participants then moved their hand back to the start position in front of them, where it remained during the experimental trials. In control blocks without movement execution, participants were instructed to ignore the cue. After another interval of 800 ms after the end of the movement (or after 1,500 ms in control blocks), a test item was presented at one of the locations of the memory items, and participants were to indicate whether this test item was of the same or of a different orientation as the memory item that had previously been presented at that location. The response assignment was balanced across participants. The test item was present until response, but participants were encouraged to respond quickly. The fixation dot was present throughout the experimental trials, and participants were instructed to maintain fixation, even during pointing movements.

Fig. 1
figure 1

Trial procedure of Experiment 1. A trial started with the presentation of a memory array for 200 ms. Color (here, blue) indicated the memory items. Participants memorized the orientation of the memory items. After an interval of 800 ms, a cue was presented for 200 ms that pointed to one of the eight locations. In movement blocks, participants then performed a pointing movement to the cued location. In no movement (control) blocks, participants were instructed not to perform a movement, but to ignore the cue. After completion of the pointing movement in movement blocks, or after a fixed interval of 1,500 ms in control blocks, and after another 800 ms, a test item was presented at one of the previous memory item locations. Participants then indicated whether this test item was of the same orientation as the memory item that had previously been presented at that location. The box shows examples of trials in which the test item was presented at the cued or at a noncued location. (Color figure online)

Full size image

All stimuli were presented against a gray background. The orientation of each memory item was randomly selected from a set of six orientations (15°, 45°, 75°, 105°, 135°, and 165°), with the restriction that no two memory items could have the same orientation. The orientations of the distractors were randomly chosen from the same set, but without any restrictions so that multiple distractors could have the same orientation. The orientation of the test item was either identical to that of the respective memory item or randomly selected from the remaining five orientations. Memory and distractor items were blue and red, and the test item was always gray. All item colors were isoluminant.

Eight fixed locations served as item locations in the memory task and as goals in the pointing task. These locations were arranged on an imaginary circle with a radius of approximately 5.07° of visual angle. Circle outlines (diameter 1.6° of visual angle) marked these locations and were present throughout the experiment. The memory, distractor, and test items were 0.28° × 1.49° of visual angle in size. The distance between memory items (center to center) was 3.53° of visual angle. The movement cue was a line (0.07° × 0.44° of visual angle) originating from the fixation dot. The fixation dot subtended 0.17° of visual angle.

Design

Testing took place in two sessions on consecutive days. On the first day, participants performed short training versions of the memory task and of the combined memory and movement task. These data were not entered into the analyses. On the second day, participants performed the main experiment. Afterwards, they filled in a questionnaire to assess strategies and other factors that might affect performance.

The main experiment consisted of 560 trials, organized in blocks of 35 trials each. Set size in the memory task (1, 2, 3, 4, or 5 memory items) and cued location (tested memory item, another memory item, and a distractor item; equally likelyFootnote 1) were varied trialwise. Half of all trials were movement trials, in which participants executed a pointing movement toward the cued location. The other half were control trials, in which participants were instructed to ignore the cue and not to execute a movement. This was varied blockwise, with a change every two blocks. The order of movement blocks and control blocks was balanced across participants.

Analysis of movement performance

Positional data were used to determine the endpoints of the pointing movements on the glass plate. Trials in which participants failed to execute a movement, and trials with pointing errors or outliers were excluded from further analysis (on average, 2.5 % of all movement trials). Trials with pointing errors or outliers were determined separately for each participant and pointing goal. They were defined as trials in which the movement endpoint deviated by more than 2.5 standard deviations from the mean of all movement endpoints on the horizontal or vertical axis.

To control that participants sufficiently differentiated between the eight possible goal locations, we calculated an index of location differentiation. This was the ratio of the mean Euclidean distance between the mean movement endpoints for neighboring goal locations (“distance between locations”) and the mean Euclidean distance between each movement endpoint and the mean movement endpoint calculated separately for each goal location and then averaged (“distance within locations”). This index becomes larger the more the endpoints of the pointing movements cluster for each goal and differ for different goal locations.

Analysis of memory performance

Trials with excessively long reaction times (>2.5 SD from mean RT calculated separately for each participant; on average, 2.9 % of all trials), pointing errors or pointing outliers were excluded from further analysis. Accuracy in percent and mean reaction time were calculated separately for each set size, for trials with and without a movement, and for trials in which the test item position corresponded to the cued location (corresponding trials) and for trials in which the test item was presented at a noncued location (noncorresponding trials). Trials in which the cued location was the location of a distractor item were not entered into the analyses, because we were interested in a weighting of items within VWM. For reaction times, only correct responses were included.

A three-way ANOVA with the factors movement (movement vs. no movement), set size, and test item position (cued location vs. noncued location) was computed for both accuracy and reaction time. To calculate cueing benefits, the means for noncorresponding trials were subtracted from the means for corresponding trials, separately for each set size and for movement and control trials. To isolate the benefits resulting from pointing movements, the cueing benefits in control trials were then subtracted from cueing benefits in movement trials, separately for each set size. These pointing benefits indicate the enhanced weighting of maintained items induced by the pointing movement as compared with a potential benefit of the cue itself. One-tailed t tests were computed separately for each set size to compare cueing and pointing benefits against zero. For these tests we report the p values corrected for multiple comparisons using the Bonferroni–Holm method.

Results and discussion

The mean index of location differentiation was 5.63 (SD = 1.63, range: 1.69–9.24). Thus, as illustrated for an example participant in Fig. 2a, the endpoints of the movements were well clustered according to the different target locations: On average, the distance between endpoint clusters was almost 6 times the distance of endpoints within each cluster from their respective mean.

Fig. 2
figure 2

Pointing movement endpoints of a single participant in movement trials in Experiment 1 (a) and in Experiment 2 (b). Circles show the mean endpoints of the different movement goals

Full size image

The ANOVAs revealed significant main effects of movement [accuracy, F(1, 27) = 28.83, p < .001, ηp 2 = 0.52, reaction time, F(1, 27) = 4.74, p = .038, partial ηp 2 = 0.15] set size [accuracy, F(3, 81) = 54.26, p < .001, ηp 2 = 0.67, reaction time, F(3, 81) = 9.08, p < .001, ηp 2 = 0.25] and test item position [accuracy, F(1, 27) = 8.4, p = .007, ηp 2 = 0.24, reaction time, F(1, 27) = 6.13, p = .02, ηp 2 = 0.19]. Performance in the memory task was better in control trials without a movement, at lower set sizes, and at the cued location. Cueing and pointing benefits are shown in Figs. 3a and b. Figure 3a shows that performance was improved for test items presented at cued locations (solid lines) as compared to test items presented at noncued locations (dashed lines). Cueing benefits in movement trials were observed for a set size of four in terms of both accuracy [t(27) = 2.9, p = .014], and reaction time [t(27) = 2.5, p = .03]. In control trials, there was a cueing benefit for a set size of three in terms of reaction time [t(27) = 2.94, p = .014]. This cueing benefit in the control trials indicates that the presentation of the cue itself influenced performance, even when it was largely task irrelevant and only marked the tested item in one third of trials. Figure 3b shows the pointing benefits, that is, the cueing benefits in movement blocks relative to the cueing benefits in control blocks. Note that positive values for accuracy and negative values for reaction time indicate greater cueing benefits in movement trials as compared to control trials. A pointing benefit, indicating a greater cueing benefit in movement trials as compared to control trials, was only observed for a set size of four items in terms of higher accuracy [t(27) = 2.59, p = .03], but not for smaller set sizes of two and three or for a set size of five (see Fig. 3b).

Fig. 3
figure 3

Results of Experiment 1. a Accuracy (top) and reaction time (bottom) shown separately for movement (left) and control trials (right), and for the different memory item set sizes. Solid lines show performance for test items presented at the cued location; dashed lines show performance for test items presented at noncued memory item locations. Cueing benefits are the differences in performance for test items presented at cued and at noncued locations. b Pointing benefits (cueing benefits in movement trials minus cueing benefits in control trials) in accuracy (top) and reaction time (bottom). Asterisks mark significant differences from zero (one-tailed t tests). Error bars show standard errors of the means. *p < .05

Full size image

Given that for the memory task all items were equally task-relevant, it seems plausible that a weighting of information according to potential action relevance would mainly take effect when the demand on the system is high, that is, when it is likely that not all items can be maintained. Indeed, a set size of four items corresponds to the mean capacity limit of VWM of about four representations (Cowan, 2001; Luck & Vogel, 1997; Zhang & Luck, 2008). Thus, our finding that action intention had the greatest effect for four items suggests that a preferential maintenance of information that may be important for action control because of a spatial correspondence with an action goal becomes behaviorally evident under high load conditions. In analogy to selective visual processing at the perceptual level, this would then be a sort of “selection for action” at the mnemonic level in VWM.

At first glance, it might seem surprising that the action-intention advantage disappeared for five items. However, there may be a straightforward reason for the reduction of the pointing benefit, though not necessarily for its (statistical) disappearance. Weighting by potential action relevance can only produce a benefit when the item at the movement goal is maintained at the moment the action is planned or executed. With a set size of four this is likely the case in almost all trials, but with a set size of five there is a nonnegligible proportion of trials (perhaps more than 20 %, assuming that at presentation of the memory set almost four of the five presented items are selected randomly for maintenance), in which a stronger weighting of the relevant representation would not be possible because that representation is not available.Footnote 2 Thus, the statistical significance of the effect would be expected to dissipate for set sizes that exceeded the limits of VWM.

The control condition without movement execution was designed to control for automatic shifts of attention not related to action planning, but triggered by the cue itself. Although we instructed participants to ignore the cue in control trials, it is possible that they continued to use the information it contained, as cues presented during the retention interval of a VWM task appear not to be under full strategic control (Berryhill, Richmond, Shay, & Olson, 2012). Centrally presented cues can be considered endogenous cues, that is, they symbolically indicate a location, and automatic attraction of attention has traditionally mainly been associated with exogenous cues abruptly appearing at the stimulus location. However, it has been shown that endogenous cues can trigger automatic shifts of attention as well if they are sufficiently (over-)learned symbols such as arrows (e.g., Hommel et al., 2001; Ranzini et al., 2009). A line originating from fixation may not be a symbol as overlearned as an arrow, but reflexive shifts of attention have also been observed when associations between a nonpredictive cue and space were arbitrarily chosen and newly learned (Dodd & Wilson, 2009). As control blocks were interleaved with movement blocks in our experiment, it is likely that a strong association between these cues and a spatial location was established. Consistent with this, we observed a slightly improved performance at cued locations in control blocks in the present experiment (see Fig. 3a). This may have dampened the relative advantage of intended action in experimental trials, but we still observed it for four items.

Because we did not have a motor task in the control trials, one might argue that the experimental effect was not specific to the influence of intended pointing toward a cued item, but was due to a general task-related enhancement. This possibility seems unlikely, because overall performance was higher in the control condition (see Fig. 3a), suggesting that action planning interfered with performance for noncued items. However, this concern motivated a second experiment designed to test the spatial specificity of the influence of action planning relative to both a directional cue and the location of the memory item: In the control condition of Experiment 2, participants had to point to a fixed position irrespective of where the cue pointed.

Experiment 2

Experiment 2 had two aims. First, we sought to further corroborate the effect of an action-induced weighting of information in VWM observed in Experiment 1. Second, we investigated how spatially specific this effect is to the movement goal location. A set size of four (for which the effect was maximal in Experiment 1) was chosen, and the design was modified to allow for a systematic analysis of the impact of spatial distance between movement goal and test item. More specifically, we tested whether items that had been presented at locations neighboring to the movement target location would also benefit from the higher degree of attentional engagement at that location, as compared to items that had been presented at nonneighboring locations. To allow for the presentation of items spaced closely enough to be considered as neighboring or nonneighboring with a set size of four, the task was lateralized, as used successfully in previous studies (Heuer & Schubö, 2016a, 2016b). In each trial, participants memorized four colors presented in one hemifield. The feature to be memorized was changed from orientation to color in order to increase overall performance, which was rather low at the set size of four in Experiment 1. A second movement condition was included, in which participants pointed to the same goal location (the fixation dot) in each trial, irrespective of where the cue had pointed to.

Method

Unless stated otherwise, the methods used in Experiment 2 were identical to those in Experiment 1.

Participants

Twenty students participated in the experiment. Data from five participants had to be excluded, one because performance did not exceed chance level and four because of the self-reported use of strategies that violated the instructions and were likely to systematically distort performance, as assessed by a questionnaire after the experiment. These participants reported that they consistently only memorized a self-chosen subset of items (e.g., only the upper three items or the two items in the middle). Analyses were performed on the remaining participants (10 female, five male, mean age = 23 years).

Procedure

The task is illustrated in Fig. 4. Each trial started with the presentation of an arrow (precue) above the fixation dot for 200 ms, pointing to the left or right and thereby indicating the relevant hemifield for that trial. After an interval of 800 ms, the memory array was presented for 200 ms. It consisted of four colored squares in each hemifield. Participants were instructed to memorize the colors of the squares in the hemifield indicated by the precue. After an interval of 600 ms, a spatial cue was presented for 200 ms. This cue was a line originating from the fixation dot, which pointed symmetrically to one of the four locations in each hemifield. To render the two hemifields physically identical, this line was mirrored along the vertical midline. After a short interval with a duration varying randomly between 280 and 320 ms, a go-signal in the form of an enlargement of the fixation dot was presented until the movement was initiated. In one half of the experiment, participants were then to perform a pointing movement to the location indicated by the cue, and in the other half they were to point to the fixation dot irrespective of where the cue had pointed; 600 ms after completion of the movement (i.e., after return to the start position), the test item was presented until response, and participants were to indicate whether or not there was a change in color compared to the memory item that had previously been presented at that location.

Fig. 4
figure 4

Trial procedure of Experiment 2. A trial started with an arrow (precue) presented above the fixation dot for 200 ms. This precue pointed to the left or right, thereby indicating the relevant visual hemifield for that trial. After 800 ms, the memory array was presented for 200 ms. Participants were instructed to memorize the colors of the four items in the hemifield indicated by the precue. After 600 ms, a cue was presented for 200 ms that symmetrically pointed to one of the four locations in each hemifield. Upon presentation of a go-signal (enlargement of the fixation dot), presented after a variable delay of 280–320 ms, participants were to perform a pointing movement. In one half of the experiment, participants pointed to the location indicated by the cue; in the other half of the experiment, participants pointed to the fixation dot; 600 ms after completion of the movement, a test item was presented at one of the locations until response. Participants were to indicate whether this test item was of the same color as the memory item that had previously been presented at that location. In 25 % of all trials, the test item was presented at the cued location; in 75 %, it was presented at one of the other three locations. (Color figure online)

Full size image

Apparatus and stimuli

Memory items were squares of different colors. The color of each memory item was randomly selected from a set of seven isoluminant colors (blue, green, ochre, orange, pink, red, violet). All memory items within one hemifield had different colors. The color of the test item was either identical to that of the memory item previously presented at that location or randomly selected from the remaining six colors.

Eight fixed locations served as memory item locations in the memory task and as targets in the movement task. These locations were arranged on an imaginary circle with a radius of approximately 4.68° of visual angle. Circle outlines (diameter 1.6° of visual angle) marked these locations and were present throughout the entire experiment. The memory items were 0.44° × 0.44° of visual angle in size, and the distance between memory items (center to center) was 3.31° of visual angle. The cue was a line (0.39° × 0.07° of visual angle) originating from the fixation dot. The fixation dot subtended 0.17° of visual angle.

Design

The main experiment consisted of 640 trials, organized in blocks of 32 trials each. Movement goal (peripheral vs. fixation) was changed after the first half of the experiment: In one half of the experiment, the movement goal corresponded to the location of a memory item indicated by the cue (peripheral goal), and in the other half the movement goal was the fixation dot. The order of these two movement goal conditions was balanced across participants. Test item position was varied on a trial-by-trial basis. All four memory item positions were equally likely to be tested, meaning that the test item was presented at the cued location in 25 % and at another location in 75 % of all trials.

Analyses of movement performance

As in Experiment 1, movement endpoints were determined using the recorded positional data, pointing errors and outliers were excluded from further analysis (on average, 3.5 % of all trials), and the index of location differentiation was calculated. In addition, movement onset and movement duration were compared for pointing toward fixation and pointing toward peripheral positions. This was done to ensure that the overall duration of the movements did not differ between movement conditions, as this would systematically affect the duration of the maintenance interval and therefore most likely also performance. Movement onset was defined as the time from the onset of the cue until the onset of the movement (i.e., when the hand left the start position), and movement duration as the time from the start of the movement until return to the start position.

Analyses of memory performance

Trials with excessively long reaction times (>2.5 SD from mean RT calculated separately for each participant; on average, 2.7 % of all trials), pointing errors, or pointing outliers were excluded from further analysis.

ANOVAs with the factors movement goal (peripheral vs. fixation) and test item position (cued location vs. noncued location) were computed for accuracy in percent and mean reaction time. Trials in which test items were presented at noncued locations were further split with respect to the distance to the cued location, that is, into trials in which the test item was presented at a location neighboring or nonneighboring to the cued location. Two-tailed t tests (cued vs. neighboring, cued vs. nonneighboring, and neighboring vs. nonneighboring) were computed separately for each type of movement goal (peripheral vs. fixation).

Results and discussion

Similar to Experiment 1, the endpoints of the movements were well clustered according to the different movement goals (Fig. 2b shows the movement endpoints of a single participant) with a mean index of location differentiation of 5.42 (SD = 1.75, range: 3.46–7.80). Timing parameters for movements toward peripheral memory item positions (movement onset: 443 ms ± 33 ms; movement duration: 1,891 ms ± 138 ms) and for movements toward fixation (movement onset: 470 ms ± 29 ms; movement duration: 1,918 ms ± 170 ms) did not differ significantly [movement onset: t(14) = 0.96, p = .355; movement duration: t(14) = 0.35, p = .731]. Thus, the overall duration of the maintenance interval was the same in both pointing target conditions.

Figure 5a shows memory performance for the test items presented at cued and noncued locations, separately for the two types of movement goal. Accuracy (Fig. 5a, left panel) was highest [F(1, 14) = 6.75, p = .021, ηp 2 = .33] when the test item was presented at the cued location, as revealed by a main effect of test item position. Importantly, an interaction between movement goal and test item position [F(1, 14) = 5.56, p = .033, ηp 2 = .28] showed that the difference in accuracy for test items presented at cued and noncued locations was larger when the movement goal corresponded to the position of a memory item compared to when the movement goal was the fixation dot. In fact, t tests comparing accuracy for test items presented at the cued location against accuracy for test items presented at noncued locations reached significance only when the movement goal was a peripheral memory item location, that is, when the test item position corresponded to the movement goal [t(14) = 3.17, p = .014] but not when the movement goal was at fixation. Reaction times (see Fig. 5a, right panel) were shortest when the test item was presented at the cued location [F(1, 14) = 11.39, p = .005, ηp 2 = .45]. The difference in performance for test items presented at cued and at noncued locations was larger when the movement goal was the position of a memory item, although this pattern was less pronounced for reaction times than for accuracy, and the interaction fell just short of statistical significance [F(1, 14) = 4.27, p = .058, ηp 2 = .23]. Reaction times were significantly shorter for test items presented at cued compared to noncued locations not only when the movement goal was a peripheral memory item position, that is, when the test item position corresponded to the movement goal [t(14) = 2.93, p = .022] but also when the movement goal was the fixation dot [t(14) = 2.26, p = .04].

Fig. 5
figure 5

Results of Experiment 2. a Accuracy (left) and reaction time (right) shown separately for the two movement goal conditions (position of a memory item and fixation) and for test items presented at the cued (light gray) or at a noncued location (dark gray). b Accuracy (left) and reaction time (right) shown separately for the two movement goal conditions (position of a memory item and fixation) and for test items presented at the cued location (light gray), and for noncued locations that were either neighboring (striped medium gray) or nonneighboring (dotted dark gray) to the cued location. Asterisks mark significant differences. Error bars show standard errors of the means. *p < .05. **p < .01

Full size image

To investigate the spatial specificity of the effect, trials in which test items were presented at noncued locations were further split according to the distance to the cued location (see Fig. 5b). When the movement goal corresponded to the position of a memory item, performance for test items presented at the cued location was better than for items presented at nonneighboring locations in terms of both accuracy [t(14) = 4.16, p = .006] and reaction time [t(14) = 3.34, p = .03]. Interestingly, performance at locations neighboring to the cued location was still better than performance at nonneighboring locations [accuracy: t(14) = 3.01, p = .045]. Performance at the cued and at neighboring locations did not differ significantly. When the movement goal was at fixation and did not correspond to the position of a memory item, performance was equivalent for all test item positions in terms of both accuracy and reaction time.

These results corroborate our finding of an action-induced weighting of information in VWM: Performance was better for memory items at locations that corresponded to the action goal than for items that had been presented at noncorresponding, action-irrelevant locations. Notably, this was the case even though the cue and therefore the movement goal had no predictive value for the memory task, as all items were equally likely to be tested and thus equally relevant for the memory task. They only differed in their potential action relevance as indicated by the spatial correspondence between the memory item representation and the action goal. Indeed, when all memory items had been presented at action-irrelevant locations, that is, when the movement goal was the fixation dot, no difference in performance for items presented at cued and noncued locations was observed. Both memory representations directly corresponding to action goals and representations of items presented at neighboring locations benefitted from the stronger attentional engagement at that location. This suggests that there might be an attentional gradient within the spatial layout of VWM, with enhanced maintenance dropping gradually with increasing distance from the action goal location.

General discussion

The present experiments demonstrate that information in VWM is weighted according to its relevance for a current action goal: Performance for memory items that had been presented at movement goal locations was better than that for items at action-irrelevant locations.

Notably, the movement goal was cued during the retention interval and well after the offset of the display containing the memory items. Therefore, this weighting cannot be explained by perceptual enhancement at movement goal locations prior to encoding, but was introduced at the representational level when the items were already being maintained. Neither can this weighting be attributed to a strategic allocation of visual selective attention to cued locations. In Experiment 1, the cue only indicated the memory item that would be tested in one third of all trials. Even though this is a rather low percentage, the cue was not entirely task irrelevant, because all eight item positions of the memory task were not cued with equal probability. This might explain the cueing benefit observed for the set size of three in control trials. In Experiment 2, however, all items were equally likely to be tested in the memory task, and the cue had no predictive value for the upcoming test item location. Here, the control condition confirmed that the mere perception of the cue did not induce comparable differences in performance. Thus, the observed weighting of information can be attributed to differences between items in their potential action relevance as indicated by a spatial correspondence between the retinotopic representation in VWM (Eimer & Kiss, 2010; Gratton, 1998) and the action goal. We propose that this effect was mediated by the automatic deployment of spatial attention to the action goal (e.g., Baldauf & Deubel, 2010) during item maintenance. Representations of items that had previously been presented at that location then benefitted from this stronger attentional engagement in a similar manner as when attention is explicitly deployed toward specific representations (e.g., Griffin & Nobre, 2003).

The pattern of results in Experiment 2 appears to suggest that the weighting of maintained information was due to an impairment of items presented at action-irrelevant locations rather than due to a facilitation of items spatially corresponding to the action goal: Performance at noncued locations was worse for peripheral movement goals compared to when the movement goal was fixation, whereas performance at cued locations was at the same level. However, it should be noted that the two movement goal conditions differed in overall difficulty. Pointing to a peripheral location varying in each trial was certainly more demanding than pointing to the fixation dot throughout blocks of trials. It is thus likely that overall accuracy was lower in the peripheral movement goal condition. Consequently, it is hard to draw conclusions as to whether the observed weighting of VWM contents was due to a facilitation of action-relevant information, an inhibition of action-irrelevant information, or both. But based on the studies of an action-induced modulation of perception (e.g., Baldauf & Deubel, 2008b; Baldauf et al., 2006), in which performance at action-irrelevant locations and in controls without an action was close to chance level, but well above that at action goal locations, it seems reasonable to assume that observed weighting at the mnemonic level can be attributed to a similar facilitation of representations corresponding to action goals.

Experiment 1 also showed that this weighting was sensitive to the number of items to be maintained, in that it was only observed for a set size at around average VWM capacity. It appears that when demand on the system is high and when it is accordingly likely that not all items can be successfully maintained, items that hold potential relevance for an action are prioritized. This sensitivity to memory load, however, may be related to the fact that all items were also and equally important for the memory task. When no other factors besides action intentions determine the relevance of visual information, a weighting may presumably be observed at smaller set sizes, or the action-irrelevant information would simply be excluded from further maintenance (Kuo et al., 2012; Williams, Hong, Kang, Carlisle, & Woodman, 2013; Zokaei et al., 2014).

Attentional enhancement of maintenance in VWM was not restricted to items that had been presented at the action goal location. Instead, the results of Experiment 2 suggest that there was an attentional gradient spreading out from the action goal location: Performance for items presented next to that location was slightly worse, but still better than for items presented at locations even farther away. This finding contrasts with findings obtained for perceptual enhancement in perihand space. Tseng and Bridgeman (2011) tested whether placing one or two hands at the sides of the display would facilitate performance in such a graded fashion, with performance dropping with increasing distance from the hand(s). They found no evidence of a gradient, but equally improved performance across the entire display. This finding was confirmed by another study investigating altered visual sensitivity in perihand space (Le Bigot & Grosjean, 2012). However, there are important differences between these studies and the present experiments that can account for these seemingly divergent findings. First, in these studies, one or both hands were continuously placed at the display throughout the experimental trials. Presumably, this highlighted the display itself as a potentially action-relevant object. When attention is directed to part of an object— in this case to the parts of the display where the hands were placed—it typically spreads over the entire object (Abrams & Law, 2000; Egly, Driver, & Rafal, 1994; Moore, Yantis, & Vaughan, 1998). Thus, the uniform improvement of performance observed in these studies might be the result of object-based instead of spatial attention. Second, when we think of the functional implications in everyday life, it makes perfect sense that the mere presence of effectors near visual stimuli may have other effects on the allocation of attention than performing a pointing movement. Whereas the presence of an effector increases the general action affordance of objects in its vicinity, a pointing movement is usually performed to highlight very specific aspects of the environment. In a way, spatial specificity is the point of pointing. Interestingly, the abovementioned studies on altered visual processing in perihand space observed differential effects of placing the right, the left, or both hands near the display for right- and left-handers, which reflect the way they manually interact with their environment using their dominant and nondominant hands (Le Bigot & Grosjean, 2012; Lloyd, Azañón, & Poliakoff, 2010; Tseng & Bridgeman, 2011). This indicates that typical functional implications do indeed influence the attentional engagement associated with different effectors and actions.

Correspondingly, a high spatial selectivity of attentional focusing has been observed for pointing movements. Baldauf et al. (2006) had participants perform a sequence of pointing movements and found an improvement of perceptual discrimination at the movement goal locations, but not at intermediate or other action-irrelevant locations (see also Baldauf & Deubel, 2009). Given that the locations were only 3.6° of visual angle apart, these results show that attentional selection was highly specific to the movement goals. Even though the distance between locations was about the same in our experiments (3.5° and 3.3°), we did not observe a similar specificity, but a graded improvement of performance. This may be related to the domain in which the stronger attentional engagement at the action goal took effect: Spatial specificity may be reduced when an action-related enhancement is introduced at the representational level in VWM compared to at the perceptual level. Another possibility is that there was a similarly graded enhancement in the experiments of Baldauf et al. (2006), but that the enhancement at neighboring locations was not strong enough to yield a performance benefit in their highly demanding perceptual discrimination task. In comparison, the task of detecting a change in color (Experiment 2) was relatively easy, and therefore even a weak attentional enhancement of spatially corresponding representations might have sufficed to result in improved performance.

A potential concern with the present findings could be that they might have been influenced by eye movements, because eye position was not monitored. However, participants were instructed and trained to maintain central fixation throughout the experimental trials, even during movement execution. Previous studies using similar tasks (Heuer & Schubö, 2016b; LaRocque, Lewis-Peacock, Drysdale, Oberauer, & Postle, 2013; LaRocque, Eichenbaum, Starrett, Rose, Emrich, & Postle, 2014a; Matsukura, Cosman, Roper, Vatterott, & Vecera, 2014) have shown that participants easily maintained fixation, with eye movements affecting only a very small proportion (<2 %) of trials, and otherwise minimal eye deflections. Given that the memory items in the present experiments were presented at locations about 5° of visual angle in the periphery, it is highly unlikely that eye movements with sufficiently large amplitudes to fixate these locations were executed in a sufficiently large number of trials to bring about the observed effects. While allowing participants to freely move their eyes during the retention interval of a change detection task can improve overall accuracy, there is no evidence indicating that spontaneously fixating the previous location of a specific item benefits memory for that item (Williams, Pouget, Boucher, & Woodman, 2013). Thus, even if fixations of memory item locations occurred, these cannot account for the item-specific benefits observed here.

To conclude, we have shown that differences in potential action relevance induce a weighting of representations in VWM: Memory performance for information that may be or may become important for an action due to a spatial correspondence with the action goal is better than for information corresponding to action-irrelevant locations. Presumably, this action-induced weighting is mediated by the deployment of spatial attention to action goals. These findings demonstrate that our actions continue to influence visual processing beyond the perceptual stage and extend our knowledge about how attentional processes optimize the efficient use of VWM by ensuring preferential maintenance of relevant information.