Spatial action-effect binding
The temporal interval between an action and its ensuing effect is perceptually compressed. Specifically, the perceived onset of actions is shifted towards their effects in time and, vice versa, the perceived onset of effects is shifted towards their causing actions. In four experiments, we report evidence showing that action-effect binding also occurs in the spatial domain. Participants controlled the location of a visual stimulus by performing stylus movements before they judged either the position of the stylus or the position of the visual stimulus. The results yielded spatial binding between the perceived stylus position and the perceived stimulus position when the stimulus was under full control of the hand movement compared to control conditions without direct control.
KeywordsEmbodied perception Perception and action Multisensory processing Intentional binding
The perceived time of a voluntary action shifts towards the time of an ensuing external effect of that action whereas the perceived time of the action effect shifts towards the time of the action (Haggard, Clark, & Kalogeras, 2002). This perceptual attraction between actions and their effects in time has been labeled intentional binding (see Moore & Obhi, 2012 for a review). Here we ask whether an analogous phenomenon exists in the spatial domain – i.e., a subjective compression of space between a moving effector and a visual consequence of that movement.
The possibility of such a spatial action-effect binding is suggested by several observations. For example, wearing prism glasses which displace the optical position of the hand causes the hand to feel like it was located where it looks to be (Hay, Plek, & Ikeda, 1965; cf. also Redding, Rossetti, & Wallace, 2005). A similar proprioceptive drift is apparent in the so-called “rubber hand illusion.” Participants often indicate that their hand is closer to a dummy hand after seeing the dummy hand being stroked by a paintbrush at the same time as they felt stroking of their actual hand (e.g., Botvinick & Cohen, 1998). Comparable effects are observed when the position of the hand is represented by a visual marker (such as a dot) moving in the same plane as the hand: after visuomotor adaptation to misaligned visual feedback, the felt position of the hand drifts towards the seen marker position (e.g., Izawa, Criscimagna-Hemminger, & Shadmehr, 2012; Salomonczyk, Henriques, & Cressman 2012). Similar effects were reported recently even without extensive visuomotor adaptation when hand movements were translated into cursor movements in a different plane (Rand, Wang, Müsseler, & Heuer, 2013). Thus, the proprioceptive perception of the hand seems to be attracted by its visual position even if that position is indicated by an extrinsic object.
According to basic principles of multisensory integration, visual and proprioceptive information are optimally combined in the estimation of hand position (van Beers, Sittig, & Denier van der Gon, 1999; van Beers, Baraduc, & Wolpert, 2002; cf. also Ernst & Bülthoff, 2004). In light of these principles, however, not only proprioceptive perception of the hand should be affected by the vision of an object that visually represents hand position, but also the visual perception of that object could simultaneously be biased by proprioception. In other words, it is reasonable to assume that not only the effector’s perception is affected by its visual effect but also the perception of the effect is affected by its actual cause.
Such mutual biases were in fact recently observed in a study on tool use (Rand & Heuer, 2013). Participants performed arm movements on a digitizing tablet (horizontal plane) while visual feedback was displayed on a monitor (vertical plane). The felt hand direction shifted towards the seen cursor direction (see also Rand et al., 2013) and, vice versa, the judged cursor direction shifted (albeit much less) toward the felt hand direction after the visual movement direction was misaligned in respect to the actual movement direction. These findings might be taken as first evidence for spatial action-effect binding. In order to substantiate this preliminary interpretation, however, at least one additional precondition needs to be met: A mutual spatial attraction of a visual object and the hand should crucially depend on a causal link between hand movement and object movement. Specifically, such attraction should be reduced or vanished when the visual object does not represent an effect of the movement (i.e., when it does not serve as a cursor). This critical condition, however, has not to our knowledge been tested so far. Thus, to substantiate evidence for the spatial attraction between an effector producing an action and visual consequences of that action, one has to show that multisensory integration is reduced or disappears when the visual object does not relate to the effector movement.
Four experiments were conducted to meet this criterion. We implemented varying relations of effector movements to a visual target in conditions where effector movements and the visual target both occurred on a horizontal plane (note that this approach is even more conservative than the abovementioned studies of Rand & Heuer, 2013, and Rand et al., 2013). In Experiment 1, participants moved their right hand horizontally while holding a stylus. These stylus movements made a visual cursor move on the screen (called target1 hereafter). The target moved under full control of the hand but, crucially, it could be displaced by a certain distance relative to the hand. After a certain stylus position was reached participants estimated either the perceived position of the stylus or of the target in separate blocks of trials. In Experiment 2, judgments of either the stylus or the target were requested in a trial-by-trial rather than a blocked manner, to ensure that the main results of Experiment 1 generalize to the conditions used in previous studies (Rand & Heuer, 2013; Rand et al., 2013). In Experiment 3, hand and target had the same displacements during estimation as in Experiment 1. However, we used a stationary target, i.e., its position was no longer controlled by the hand. If spatial binding between the real and the virtual hand position exists then the estimates of the stylus position should be biased in the direction of the target position in Experiment 1 as compared to Experiment 3. Simultaneously, the estimates of the target position should be attracted by the current stylus position in Experiment 1 as compared to Experiment 3. Results were in line with these predictions. Finally, Experiment 4 provided additional baseline conditions to further validate the observed effects.
With Experiment 1 we aimed to show mutual attraction effects between the perceived position of the acting hand that was operating a stylus, and the perceived position of a continuously controlled target stimulus. To this end, participants first moved the target stimulus with their stylus while target and stylus positions were spatially displaced in one of five mapping conditions (target far/close to the left of the stylus, target and stylus at the same position, target close/far to the right of the stylus). After each movement, participants were either asked to report the perceived stylus position or the perceived target position in different blocks of trials. We expected the perceived stylus positions to be shifted towards the target position, and, vice versa, we expected the perceived target position to be shifted towards the stylus.
The main apparatus consisted of a graphics tablet (Intuos 4 XL, Wacom, Kazo, Saitama, Japan), a digitizing stylus, a monitor, and a semi-silvered mirror (see Fig. 1). The monitor was mounted above a table at a distance of approximately 47 cm. The tablet was mounted on the table; it was covered by a semi-silvered mirror that was positioned in-between the monitor and the tablet. This apparatus ensured that stimuli presented on the monitor appeared to the participant to be on the plane of the tablet. The mirror prevented the vision of the participant’s hand in the dimmed lab. One pixel (px) of the monitor was approximately 0.382 mm in size.
The monitor and the mirror were supported by a metal frame. The participants were asked to lean their forehead on the upper part of this frame to reduce a possible impact of head movements (see Fig. 1). Stylus movements were performed with the right hand. Perceptual adjustments, in contrast, were made with the left hand using a keyboard placed to the left of the main apparatus.
Procedure and design
Participants were asked to judge the position of a visual stimulus (target) in one type of block and the position of the stylus in another type of block. For these judgments, a gray vertical line (about 15 mm in length and 0.5 mm in width) was presented about 141 mm above either the current target or the current stylus position. The initial horizontal position randomly deviated from the current position of the stylus or of the target by 9.5 mm to the left or to the right. The task was to align the line with the position of the stylus or of the target by pressing the left and right Arrow keys on the keyboard. The estimate had to be confirmed by the Enter key. When the Enter key was pressed without changing the initial line position an error feedback was presented and the trial was repeated. The respective judgment instruction was presented before each block. We also presented a judgment cue in each trial to encourage participants to keep the instruction in mind throughout a block of trials (German words for cursor or stylus presented in white on a gray background).
The target stimulus was fully controlled by the movement of the stylus. That is, the stimulus served as a cursor of the stylus. After participants were informed about which estimate was to be made and pressed the Space bar on the keyboard, a gray horizontal line (about 382 mm in length and 0.5 mm in width) and the target (green dot, about 2 mm in size) appeared (see Fig. 1). The task here was to move the stylus along the line until the target changed its color from green to red and then to press a stylus button. When the target did not touch the line during the button-press an error feedback appeared and the trial was repeated. The color change could occur at five different positions: when the target was either in the middle of the display, or about 17 or 34 mm left or right of the middle (the allowed deviation was about 2 mm). A comparison line appeared after the stylus button was pressed (see above) and the participant judged the position of the target or of the stylus. In response to the first press of one of the Arrow keys the stimulus changed its color back from red to green and it was slightly adjusted at the middle of the horizontal line when necessary to ensure identical visual conditions during the judgment procedure across the different conditions. Participants had to maintain the current hand position during the judgment. Otherwise they received error feedback and the trial was repeated.
The critical experimental manipulation was related to the spatial mapping between the horizontal position of the stylus and the horizontal position of the target (target-stylus mapping hereafter). The target could be at the position of the stylus or could deviate from it by 38 or 76 mm to the left or to the right. This manipulation was implemented in a trial-wise manner.
The experiment was divided into four blocks with 50 trials each. Two blocks included target judgments, two other blocks included stylus judgments. The succession of blocks was counterbalanced across the participants. In each block, each combination of the position of the target (five levels) and of the mapping between the stylus and the target (five levels) was presented twice in a randomized order. At the beginning of the session, participants performed 20 practice trials (ten for each judgment condition), which were not included in the analyses.
A difference score was computed between the actual and the estimated position for each judgment mode and each trial (constant error hereafter). Then, median constant error values were calculated for each participant and each experimental condition. One participant seemed either to have misunderstood the task instructions or to have a not reported visual impairment. Her estimates of target position deviated considerably from those of other participants as well as from the actual target positions (see Fig. S1). This participant was removed prior to analysis.
Results and discussion
These results indicate that the judgment of target position is biased towards the current stylus position, whereas the judgment of stylus position is biased towards the current target position, i.e., spatial action-effect binding. With Experiment 2 we aimed to conceptually replicate this basic finding. In particular, we asked whether a random succession of the different types of judgments would reproduce the results of Experiment 1. This would indicate that possible conclusions can be generalized to the designs used in previous studies (Rand & Heuer, 2013; Rand et al., 2013).
Experiment 2 aimed at replicating the spatial attraction effects between the perceived position of the operating stylus and the perceived position of a controlled target as observed in Experiment 1. Instead of blocked presentations of target judgments and stylus judgments, however, judgment modes now varied from trial to trial. Despite this procedural change, we still expected perceived stylus positions to be attracted towards the controlled target, and vice versa.
A new group of 24 right-handed participants was recruited (Mage = 26 years, SD = 4; 20 female, four male). Participants gave their informed consent for the procedures and received monetary compensation for their participation.
Apparatus, design, and procedure
Apparatus, design, and procedure were as in Experiment 1 with the following modifications. Most importantly, participants were now required to judge both the target position and the stylus position in each block of trials. In each block, each combination of the position of the target, of the mapping between the stylus and the target, and each judgment mode was presented once in a randomized order.
One participant seemed to have misunderstood the task instructions (see Fig. S1). This participant was removed prior to analysis.
Results and discussion
Experiments 3 and 4 aimed to test for boundary conditions of the observed effects in order to draw a full analogy to previous studies on intentional binding (e.g., Haggard et al., 2002). In particular, we asked whether the mutual biases between the stylus position and the target position would vanish when the target does not represent stylus movements. In these experiments stylus and target judgments were made in separate blocks of trials as in Experiment 1.
As a first control condition with a less direct action-effect relation between the stylus movements and the target, the spatial location of the target was no longer coupled to the stylus. Rather, the target remained stationary but changed its color to indicate that the participant had successfully completed the movement. We therefore expected the spatial attraction effect to be reduced and possibly even absent.
We recruited a new group of 24 right-handed participants (Mage = 22 years, SD = 8; 21 female, three male). Participants gave their informed consent for the procedures and received monetary compensation or course credit for their participation.
Apparatus, design, and procedure
Apparatus, design, and procedure were as in Experiment 1 with the following modifications. Most importantly, the visual target stimulus no longer served as a cursor of stylus movements. After the judgment cue disappeared, the target was immediately presented on the horizontal line at one of five horizontal positions (see Exp. 1). As in Experiment 1, participants were asked to move the stylus along the horizontal line until the target changed its color, but the target now remained stationary (it was termed a dot in the instructions). Two small horizontal markers (about 4 mm in length and 1 mm in width) presented at the left and right edges of the display now served as indicators of the y-coordinate of the stylus. That is, any deviations of the stylus position from the position of the horizontal line were indicated by a spatial deviation of the markers from the position of the horizontal line. The target changed its color when a “virtual” cursor (that was not visible) reached the current target position (i.e., when the stylus position corresponded to that of Exp. 1 in the respective condition; see also below). Thus, except for the color change, the movement of the stylus was completely unrelated to the target stimulus in Experiment 3, whereas the spatial relation between the stylus and the target was preserved.
Two participants seemed to have misunderstood the task instructions (see Fig. S1). These participants were removed prior to analysis.
Results and discussion
As in Experiments 1 and 2, participants tended to judge the target to be more right the more right the target was. The effect of target-stylus mapping on stylus-judgments seemed (as in Exp. 1) slightly to increase the more right the target was. This trend was not significant, however. Instead, the stylus position was generally judged to be more right the more left the target was.
A comparison of the results of Experiments 1 and 3 reveals the expected pattern with reduced attraction effects in Experiment 3 compared to Experiment 1. However, the continued effect of target-stylus mapping on the stylus estimates was not predicted beforehand. Two reasons may underlie this outcome. First, participants might still have experienced some sense of control over the target in Experiment 3 (due to the color change of the target after achievement of a certain stylus position). Second, a type of central tendency effect often observed in perceptual tasks may also explain this result. That is, independent of the relation between the stylus position and the target, participants’ estimates may shift in the direction of the mean of all stylus positions (which can be termed as “adaptation level”; see e.g., Helson, 1964). In a similar vein, the observed variation of target estimates depending on the absolute target position might be related to some categorization or optical factors which are not directly related to the movement of the stylus. Experiment 4, in which hand movements were absent before target judgments and the color change was omitted before stylus judgments, aimed to explore these possibilities.
The rationale behind Experiment 4 was twofold. First, we aimed to further reduce a possible sense of control over the target that may still have been present in Experiment 3 due to the fact that the target changed its color in response to certain movement characteristics (i.e., when a certain stylus position was achieved). Therefore, Experiment 4 provided an even more controlled baseline condition to Experiment 1. Second, we also aimed to access possible biases in judgments of the target which are unrelated to stylus movements. Hand movements were therefore absent in blocks of target judgments. For stylus judgments, a stationary visual target was still present in Experiment 4 as in Experiment 3. However, an acoustic signal indicated the end of the movement instead of the change in target color used in Experiment 3. Because these procedural changes eliminated any type of action-effect relation between stylus movements and the target, Experiment 4 provides the critical baseline to assess to what extent the previously observed effects do indeed resemble spatial action-effect binding.
We recruited a sample of 25 participants, four of whom had also participated in Experiment 3 (Mage = 20 years, SD = 2; 21 female, four male). Participants gave their informed consent for the procedures and received course credit for their participation.
Apparatus, design, and procedure
The apparatus was as in Experiments 1 and 3, except that headphones were used for the presentation of acoustic signals. Furthermore, in Experiment 4, participants were asked to put the stylus aside during the blocks of trials in which target judgments were made. That is, hand movements were absent in these blocks. The target (that was termed a dot in the instructions) and a comparison line appeared immediately after the judgment cue disappeared. Stylus judgments, in contrast, were preceded by stylus movements as in Experiments 1 and 3. The visual target was present but did not change its color, however. Instead, an acoustic signal (a sequence of short beep tones) signaled the reaching of the respective stylus position. Each block of target judgments now simply featured ten repetitions of each target position because there was no stylus movement during these blocks. Due to a slight mistake of the experimenter, the succession of the blocks was not fully counterbalanced across the participants.
Results and discussion
Similar to the results of Experiment 1, the effect of target-stylus mapping on stylus-judgments tended to increase the more rightward the target was. This indicates a rather unspecific origin of this slight interaction. As in Experiments 1, 2, and 3, participants also judged the target to be more right the more right the target was. This effect can thus be considered as another side effect of the given task setting rather than be directly related to the critical manipulation of the target-stylus mapping.
To ensure comparability, only experiments in which stylus and target judgments were made in separate block of trials were compared with each other (i.e., Exps. 1, 3, and 4). Mean coefficients associated with the judgments of the stylus position were maximal in Experiment 1 and were lower in Experiment 3, t(43) = 2.21, p = .032, and Experiment 4, t(46) = 3.89, p < .001. Also, the mean regression coefficient in Experiment 4 was lower than that of Experiment 3, t(45) = 2.58, p = .013. In the judgments of the target position, the mean slope was more positive in Experiment 1 than in Experiment 3, t(43) = 1.85, p = .071, and Experiment 4, t(46) = 2.44, p = .018.
These results substantiate the differences across the individual experiments observed in the main analyses of variance. In particular, when a visual stimulus is used as a cursor of stylus movements, the estimates of that stimulus are attracted by the current stylus position and simultaneously, the estimates of the stylus position are attracted by the current stimulus position. These effects are reduced or even disappear when the visual stimulus is unrelated to the stylus movement.
In the present study we report an extension of intentional binding to the spatial domain. Participants either exerted control over the movement of a visual stimulus by their own hand movement or were prevented from doing so before they judged either the position of that stimulus or the position of their hand. We observed a mutual bias between the perceived positions of the hand and the visual stimulus when that stimulus was used as a movement cursor, compared to conditions without control over the stimulus location. The present results thus extend the previous observations (Rand & Heuer, 2013), by revealing that visuomotor distortions are accompanied by changes in the perception of a visual stimulus as well as of the effector as long as the latter can be considered as an effect of the former.
Considered from a multisensory perspective, the present finding might reflect the results of sensory integration processes which combine and weight proprioceptive and visual signals depending on their reliability in order to provide a robust percept of the current hand position (see e.g., Ernst & Bülthoff, 2004; van Beers et al., 2002). This would imply that the visual target position was treated, at least in part, as equivalent to the felt hand position. In other words, the results can be considered as indicative for the incorporation of a visual stimulus into a body representation. This conclusion corresponds well with the assumed constraints of multisensory integration: different sensory signals are combined as long as these signals are likely to be produced by the same object or event (Bedford, 1995; Ernst, 2006; see also Bresciani et al., 2005; Gepshtein, Burge, Ernst, & Banks, 2005). For example, an increase in spatial separation between visual and haptic signals decreases their integration (Gepshtein et al. 2005). Against this theoretical background, one might wonder why integration occurred in the mentioned studies of Rand and colleagues (Rand & Heuer, 2013; Rand et al., 2013) in which proprioception and vision obviously indicated different positions of clearly different objects. This can possibly be understood in terms of the changed body representation when using a tool (cf. e.g., Canzonieri et al., 2013; Cardinali et al., 2009; Sposito, Bolognini, Vallar, & Maravita, 2012). That is, mutual biases between vision and proprioception may inform here that the internal representation of the body was changed by the acquisition of the kinematic transformation of the tool. As a result, the hand and the cursor were treated, at least to some degree, as one and the same object.4
It has been suggested that the temporal binding is due to a cognitive module linking voluntary actions to their effects (e.g., Haggard et al., 2002; Walsh & Haggard, 2013). Specifically, the perceptual attraction (at least of the sensory effect toward the action) is assumed to result from matching between the predicted and observed action consequences (Waszak, Cardoso-Leite, & Hughes, 2012). In the essence, a sensory action effect is assumed to be perceived earlier (than the same sensory event being not a consequence of own action) due to a preactivation of the representation associated with that effect during action selection (see also Pfister, Obhi, Rieger, & Wenke, 2014). A related mechanism may also be involved in spatial binding reported here. As demonstrated by Buehner and Humphreys (2010), however, similar phenomena can also be observed without intentional actions when a causal connection between events exists. Thus, the critical mechanism involved in action-effect binding might be more abstract and primarily be related to causality (see also Eagleman & Holcombe, 2002).
Explanations based on multisensory mechanisms and on processes akin to temporal binding might be not mutually exclusive. For example, following the ideomotor approach actions are initiated by the anticipation of sensory effects associated with those actions (e.g., Hommel, Müsseler, Aschersleben, & Prinz, 2001; Kunde, 2001). In the given task setting, such a preactivation of body-related action consequences and those in the environment could be a basic prerequisite for multisensory integration to occur. If, for example, an external object does not represent a predicted effect (as in the control experiments of the present study) then there would be no reason to integrate its visual position with proprioceptive information derived from the acting effector. In other words, spatial binding could basically result from concurrently anticipated body-related and environment-related effects of an action which then mutually attract each other. The relation between these types for action effects is currently not completely understood and seems to depend on task conditions (e.g., Sutter, Sülzenbrück, Rieger, & Müsseler, 2013). Accordingly, the suggested explanations are tentative and more research is needed before definite conclusions about the involved mechanisms will be possible.
In summary, the results of the present study indicate that the proprioceptive perception of an effector producing an action and the perception of visual consequences of that action are spatially attracted by each other. This appears to be a particular instance of sensory coupling following the rules that also apply to action-unrelated sensory signals. This finding may have important implications for the research of perceptual-motor interactions and may enable deeper insights into the origins of related phenomena. In particular, the present results indicate that the observed sensory coupling is specific to action-effect relations. Thus, in task situations where body-related signals are accompanied by visual stimulation being a likely consequence of those signals sensory coupling (or spatial binding) should be observed. Research on the rubber hand illusion, for example, showed that the perception of the felt hand position is biased by the position of a dummy hand. However, whether the visual perception of the dummy hand is affected by the position of the real hand has not been examined so far to our knowledge. The results of the present study suggest this possibility and point to a potential explanation of such an outcome.
Note, the term “target” is used here to designate a visual object independently of the relation between this object and a hand movement. Thus, it does not imply other meanings generally used in the context of hand movements (such as of objects to be reached by a movement).
Note, this rounded value was used in Experiments 1, 3, and 4 to convert pixels into mm. The more exact size of one pixel was about 0.3853 mm. Accordingly, the reported mm values could slightly underestimate the real measures.
Note that in Exp. 4 there were no stylus movements before the judgment of the target. The used software, however, randomly classified trials according to the target-stylus conditions used previously. Thus, any residual effects of target-stylus mapping on target estimates should be due to random noise.
The fact that participants did not completely fuse visual and proprioceptive signals into a unified percept in those studies does not speak against this conclusion. A rather low strength of coupling might enable maintaining access to the individual signals and thus enable learning new mappings (Ernst, 2006; Hillis, Ernst, Banks, & Landy, 2002). The mere presence of integration (albeit low), however, indicates that different signals are spatially related to each other (otherwise they should be kept separate, cf. Ernst, 2006). Alternatively, and as noted by Rand and Heuer (2013), however, multisensory integration could merely improve the precision of estimates which refer to different objects.
This research was supported by grant KI 1620/1-2 awarded to W. Kirsch by the German Research Council (DFG).