Responding to a stimulus leads to the integration of the stimulus, the response, and any sensory effect triggered by the response in a mental representation that has been called “event-file” or “instance.” Most theoretical models assume that event files are composed of sets of binary bindings between individual stimulus, response, and effect features. Repeating any of the integrated features on a subsequent occasion would then retrieve the entire episode. However, previous studies mainly focused on either stimulus-response (SR) binding or response-effect (RE) binding while not assessing S-R-E episodes in their entirety. Here we analyzed for the first time bindings within entire action episodes including stimulus, response, and effect. We found clear evidence for SR- and RE-binding, but no indication of integration between stimulus and effect. We conclude that representations of actions are structured according to the current task, possibly providing a base for learning mechanisms to draw on.
Action episodes typically consist of a bodily movement and effects of this movement on the environment, and each action episode is further nested in the current situational context. The cognitive representation of one such action has been assumed to include binding of all these elements into what has been called an instance or event-file (e.g., Frings et al., 2018; Hommel & Wiers, 2017; Logan, 1988; Schmidt, De Houwer, & Rothermund, 2016). More specifically, event-files were described as loose networks of binary bindings, with “binary” indicating bindings of always two features that are each independent of all other bindings in the event-file (Hommel, 2004). The recent Binding and Retrieval in Action Control (BRAC) framework (Frings et al., 2018) follows this notion by proposing explicitly that stimulus-(S), response-(R), and effect-(E) features are integrated into an event-file and that repetition of any of those features can retrieve the remaining features of the event file later on. According to the assumption that repeating one feature can retrieve all of the other features (see also Hommel, 2004; Logan, 1988; Schmidt et al., 2016), one would naturally expect bindings between all features in an event-file. In addition, previous observations also seem to support the assumption that each individual feature in an event-file shares binary bindings with each of the other integrated features (Giesen & Rothermund, 2014; Hommel, 2005; Moeller, Frings, & Pfister, 2016). However, past studies that supported this claim mainly focused either on SR-binding (e.g., Hommel, 1998; Moeller, Frings et al., 2016), or on RE-binding (e.g., Dutzi & Hommel, 2009; Janczyk, Heinemann, & Pfister, 2012) while not assessing S-R-E episodes in their entirety. In other words, bindings between features of entire S-R-E-episodes are yet to be analyzed.
Binding of stimulus features and effect features in particular comes with the critical challenge that these events are distributed in time with the response as intervening step between S and E. Because spatiotemporal proximity is typically seen as a critical precondition for integrating feature codes in an event file (Akyürek, Toffanin, & Hommel, 2008; Kahneman, Treisman, & Gibbs, 1992), evidence from SR- and RE-binding cannot necessarily be generalized to entire S-R-E episodes. In fact, selective binding of only some features in an event file might provide flexibility in action control. In particular, multiple different stimuli can imply the same response, and this response in turn can, depending on the particular situation, lead to different effects. Binding between those features, likely to share a functional relationship, namely S and R as well as R and E, would therefore be expected, while binding between features that are only indirectly related via the response (S and E) would be less likely. Here we analyzed all of these possible bindings within one action episode. Participants experienced complete S-R-E episodes, and we measured SR-, RE-, and SE-binding in a subsequent free- or forced-choice response. To anticipate the results, we found evidence for SR- and RE-binding, but no indication of SE-binding.
We adapted the paradigm from Moeller, Pfister, Kunde, and Frings (2016) to study binding of both stimulus and effect features. Participants executed simple key-press responses in the presence of a distractor stimulus and each response produced an effect. Experimental designs to capture SR- or RE-bindings descend from different research traditions, so that their paradigms differ slightly, resulting in different dependent variables. Both paradigms include two responses and assume integration of features at the first (prime) response and retrieval at the second (probe) response, measuring binding effects in the probe response. In SR-binding paradigms all responses are typically forced choice and binding-effects are measured via response time (RT) differences. Here, binding is assessed by a facilitation of response repetitions (compared to response changes) when the stimulus repeats rather than changes. The critical response in RE-binding paradigms is typically free-choice and binding is measured in differences in percentage response repetitions from prime to probe (for an exception, see Herwig & Waszak, 2012).Footnote 1 Here, larger response repetition rates for effect repetition compared to effect changes indicate binding. To account for both traditions and any aspect of binding-effects that might preferentially affect either dependent variable, we included two experimental blocks, one adapting the RE-paradigm to the full S-R-E case and one adapting the SR-paradigm (see Moeller, Pfister, et al., 2016).
RE-block: The design comprised the two within-subject factors Stimulus Relation (repetition vs. change) and Effect Relation (repetition vs. change).
SR-block: The design comprised the three within-subject factors Response Relation (repetition vs. change), Stimulus Relation (repetition vs. change), and Effect Relation (repetition vs. change).
Thirty-two students (26 female) from the University of Trier took part in the experiment. Their median age was 20 years (range 18–52). All participants received course credit. This sample size ensured a power of 1-β > .99 for detecting retrieval effects in the RE-procedure (assuming dz ≥ 0.98 as reported for Exp. 1 in Moeller, Pfister, et al., 2016, and a two-tailed test) and a power of 1-β = 0.80 for the smallest effect size reported for the SR-procedure (dz = 0.51 for Exp. 2 in Moeller, Pfister, et al., 2016). Effect sizes for SE-binding cannot be estimated from the literature, though it seems plausible to assume a similar magnitude as that for SR- and RE-binding if such bindings were to be built up. Using the mean effect size for each experimental procedure in the previous study suggests a power of 1-β > .99 for the RE-procedure and a power of 1-β = .87 for the SR-procedure.
The experiment was conducted using E-prime 2.0. Instructions and target stimuli were shown in black on a white background on a standard TFT screen. Targets were the letters D, F, J, and K. They had a horizontal visual angle of 0.8–1.1° and a vertical visual angle of 0.9–1.0°. Visual (additional) stimuli and effects were colored rectangles (red, yellow, blue, or green) that were centered on the screen and subtended a horizontal visual angle of 2.2° and a vertical visual angle of 1.4°. Auditory stimuli and effects were two 300-ms and 80-dB sine wave sounds of 400 and 800 Hz, respectively. Viewing distance was approximately 60 cm.
Participants were tested individually in soundproof chambers. Instructions were given on screen. Participants placed their left index finger on the key F and their right index finger on the key J of a standard QWERTZ keyboard. The letters D and F were mapped to one key and the letters J and K to the other. Stimuli and effects could either occur in the visual modality or the auditory modality, and modalities (visual/auditory vs. auditory/visual) were balanced across participants. Each participant completed two experimental blocks that differed slightly in the probe task. One block followed a typical procedure, used to study SR-binding in the past. The other block followed a procedure that has been used to study RE-binding. We will refer to them as the SR-procedure (SR-block) and the RE-procedure (RE-block). In the SR-block, participants’ task was always to classify the target letter that was presented in the center of the display by pressing a key with the corresponding finger and ignoring any sound or colored rectangle that might be presented as an additional stimulus simultaneously with the target or as an effect of the response. Depending on the condition, each prime target was accompanied by either a colored rectangle or a sound. Prime response execution was followed by an effect of the other modality, respectively (i.e., by a sound or by a colored rectangle). The probe target was then accompanied by a colored rectangle and a sound. In the RE-block, a target (accompanied by an additional stimulus and followed by an effect) was only presented in the prime. The probe display included three exclamation marks in black, which were accompanied by a sound and a colored rectangle. Participants were instructed to press either key in response to these stimuli. They were asked to select this response key randomly but try to use both keys for about the same number of trials throughout the experiment.
In stimulus repetition trials, the previous sound/rectangle that accompanied the prime target was repeated at probe presentation. In stimulus change trials, both sound and rectangle presented in the probe differed from the additional stimulus that had been presented together with the prime target. In effect repetition trials, the colored rectangle/sound that followed the prime response was presented in the probe and in effect change trials neither sound nor rectangle color in the probe had been presented as an effect of the prime response. The factor Stimulus Relation (repetition vs. change) was varied orthogonally to the factor Effect Relation (repetition vs. change).
Each trial (see Fig. 1) started with 2,000-ms presentation of an asterisk. Then the prime target and additional stimulus appeared for 300 ms. Every prime response triggered a 300-ms effect sound or rectangle. 500 ms after this effect, the probe display appeared for 300 ms and participants responded by pressing one of the response keys. Each block comprised 128 prime-probe sequences and block order was counterbalanced across participants. Before the experimental blocks, participants worked through a practice block of 32 prime-probe sequences in which they received feedback after each response. Before the second experimental block started, participants were familiarized with the slightly different procedure in 16 practice trials.
To enable a maximally pure test of binding and retrieval influences we opted to remove all participants who reported strategy use or whose data suggested the use of decision strategies from the analysis of the RE-block (for recent evidence for the impact of decision strategies in free-choice designs, see Vogel, Scherbaum, & Janczyk, 2018; Weller, Kunde & Pfister, 2017). Four participants reported having used strategies and another nine (as well as three of the participants reporting strategies) repeated the response from the prime in the probe in more than 90% or less than 10% of the trials if either stimulus or effect was repeated, which we interpreted as a strong indication for strategy use. The result pattern was identical if all of these participants were included.Footnote 2
Only trials with a correct answer in the prime were considered. Due to this constraint, 6.3% of all trials were discarded. Participants used the left and right key for the probe response in about half of the trials each (47% left and 53% right key-presses). These rates did not differ from chance, t(18) = 1.01, p = .328, d = 0.24. That is, participants managed to follow the instruction to use both keys about equally often.
Our dependent variable of interest was the rate with which participants repeated their response from prime to probe. For mean percentages of repetitions, see Table 1. In a 2 (Stimulus Relation: repetition vs. change) × 2 (Effect Relation: repetition vs. change) repeated-measures ANOVA on response repetition rates, the main influences of Effect Relation, F(1,18) = 7.93, p = .011, ηp2 = .31, and Stimulus Relation, F(1,18) = 9.08, p = .007, ηp2 = .34, were significant. Participants repeated their response more often if the prime response effect was again presented in the probe (31.0% response repetitions) than if the sound/color that was presented as effect in the prime differed in the probe (26.5% response repetitions) and they repeated their response more often if the additional stimulus in the prime was again presented in the probe (30.7% response repetitions) than if the sound/color that was presented as an additional stimulus in the prime differed in the probe (26.8% response repetitions). Apparently, prime responses were integrated with the stimulus and also with the effect. Importantly, Stimulus and Effect Relation did not interact, F(1,18) < 1, p = .760, ηp2 < .01. That is, responses were independently integrated with the stimulus and the effect.
For the analysis of the SR-block, only trials with correct responses in prime and probe were considered for analyses of the RT-data. RTs that were more than 1.5 interquartile ranges above the third quartile of the RT distribution of the participant (Tukey, 1977) and those that were shorter than 200 ms were excluded from the analysis. Due to these constraints, 11.5% of all trials were discarded (prime error rate was 3.0%, and probe error rate, after correct primes, was 4.8%). Further, trials in which an identical letter was presented as target in prime and probe were excluded from the analyses.Footnote 3 Mean RTs and error rates are shown in Table 2.
In a 2 (Response Relation: repetition vs. change) × 2 (Stimulus Relation: repetition vs. change) × 2 (Effect Relation: repetition vs. change) repeated-measures ANOVA on RTs, the main influence of Stimulus Relation was significant, F(1,31) = 4.16, p = .050, ηp2 = .12. Participants responded faster if the additional sound or color stimulus from the prime was repeated in the probe (M = 588 ms, SD = 95) than if the stimulus changed from prime to probe (M = 597 ms, SD = 109). More importantly, the interaction of Response Relation and Stimulus Relation, F(1,31) = 20.52, p < .001, ηp2 = .40, and also the interaction of Response Relation and Effect Relation, F(1,31) = 8.25, p = .007, ηp2 = .21, were significant, while the interaction of Stimulus Relation and Effect Relation was not, F(1,31) < 1, p = .382, ηp2 = .03 (see Fig. 2). In addition, the three-way interaction of Response Relation, Stimulus Relation, and Effect Relation, F(1,31) = 2.72, p = .109, ηp2 = .08, was not significant either. That is, stimulus and effect were not integrated with each other and their integration with the response was independent of the relation of the other, respectively. None of the other effects was significant, Fs<1.
In the same analysis on error rates the pattern was identical but only the main influence of Stimulus Relation, F(1,31) = 3.77, p = .061, ηp2 = .11 and the interaction of Response Relation and Effect Relation, F(1,31) = 3.37, p = .076, ηp2 = .10, were close to significance. None of the other effects reached significance, all Fs<2.3, ps > .14, ηp2 < .07.
The representation of an action has widely been assumed to include bindings between S, R, and E codes so that reactivating one of them triggers retrieval of the entire episode (e.g., Frings et al., 2018; Hommel & Wiers, 2017; Logan, 1988; Schmidt et al., 2016). This central assumption of theories regarding binding and retrieval has never been addressed from a perspective on complete S-R-E episodes. We therefore studied the binding of stimulus, response, and effect in an orthogonal experimental design and found that only some but not all of the features become integrated with each other. Namely, we found evidence for SR- and RE-binding, but no evidence for bindings between S and E.
The question remains how to explain why specific features or types of features enter bindings when an event file is compiled while others do not. Several, not mutually exclusive, mechanisms are likely to contribute to this pattern. A first factor is the relatively large temporal distance between S and E, which may affect SE-binding, as spatiotemporal proximity is commonly regarded as a prerequisite for binding to occur (Kahneman et al., 1992; though newer findings suggest that time itself can be integrated as a feature in event files; Bogon, Thomaschke, & Dreisbach, 2017). Alternatively, bindings may emerge only between functionally related features, and, hence, the response plays a central role for binding in action control. Being triggered by S and bringing about E, the response is integrated with both, while at the same time S and E are only indirectly related via the response and not bound to each other.
Another potentially important difference between responses on the one hand and distractors or effects on the other hand, is that the former were task-relevant, whereas the latter were not. Participants were always instructed to produce a response (even a specific one in the stimulus procedure), whereas distractors and effects were completely irrelevant aspects of the task. It has been suggested that task-relevant features have a higher chance to become part of event files (see Memelink & Hommel, 2013). Possibly, differences in other features’ weights might be equally important, such that a given feature is more likely bound to other features of high rather than low weight. Any distractor (or effect) feature might have a higher chance to become bound to a heavily weighted (relevant) response feature, rather than to a relatively weaker effect (or distractor) feature. Thus, differences in feature weight might induce selectivity among several potentially possible bindings. A strongly weighted feature is integrated with many of the other features, and specifically also with features with less weight. By contrast, the weaker the weight of one integrated feature, the larger the weight of the other feature needs to be for binding of the two.
The present results may also be of interest with regard to the domain of RE-association learning. Even though binding and learning mechanisms sometimes appeared to be independent in previous work (Colzato, Raffone, & Hommel, 2006; Herwig & Waszak, 2012; Moeller & Frings, 2017), binding has also been discussed as a very early form of learning (e.g., Elsner & Hommel, 2001, 2004; Giesen & Rothermund, 2014), and several characteristics of bindings closely resemble those of longer-lasting associations after a learning phase (e.g., Giesen & Rothermund, 2014; Moeller & Frings, 2014; Singh, Moeller, & Frings, 2016). From this point of view, it is unsurprising that the structures we found in event-files after a single encounter of individual SRE-episodes are also present in learned associations after repeated encounters of the same episode. Specifically, in the (early) phase of learning RE-contingencies, associations between stimulus and effect do not seem to play a major role (e.g., Hoffmann, Berner, Butz, Herbort, Kiesel, Kunde, & Lenhard, 2007; Wolfensteller & Ruge, 2011; Ziessler & Nattkemper, 2001, 2002). This is in line with the assumption that the representational structure of a current event (i.e., event-file) sets the stage for what might be learned in the long run.
In conclusion, in the representation of a complete action, including S, R, and E, each of the features is not necessarily integrated with all of the others. Instead, entire action episodes seem to be represented by a limited number of structured bindings. Functional relations among features and/or an extended concept of intentional weighting may be able to explain which of the integrated features in an event-file share bindings.
The data of the reported experiment are not openly available, and the experiment was not preregistered.
Notably, the latter RE-procedure only allows for probing whether there are binary SR- and RE-bindings, whereas it does not probe for SE-binding directly.
Analyses including all participants: 5.1% of all trials discarded, participants used left (48%) and right (52%) keys in half of the trials each (not different from chance, t(31) = 1.06, p = .297, d = 0.19). 2 (Stimulus Relation: repetition vs. change) × 2 (Effect Relation: repetition vs. change) repeated-measures ANOVA on response repetition rates: Main influences of Effect Relation, F(1,31) = 9.41, p = .004, ηp2 = .23, and Stimulus Relation, F(1,31) = 12.64, p = .001, ηp2 = .29, were significant, while the interaction of Stimulus and Effect Relation was not, F(1,31) < 1, p = .621, ηp2 < .01. Notably, an effective sample size of 19 participants still ensures a power of 1-β = .98 for the smallest retrieval effect reported for the RE-procedure of Moeller, Pfister, et al. (2016).
The results were identical when we included target repetition trials.
Akyürek, E.G., Toffanin, P., & Hommel, B. (2008). Adaptive control of event integration. Journal of Experimental Psychology: Human Perception and Performance, 34, 569-577.
Bogon, J., Thomaschke, R., & Dreisbach, G. (2017). Binding time: Evidence for integration of temporal stimulus features. Attention, Perception, & Psychophysics, 79, 1290-1296.
Colzato, L.S., Raffone, A., & Hommel, B. (2006). What do we learn from binding features? Evidence for multilevel feature integration. Journal of Experimental Psychology: Human Perception and Performance, 32, 705-716.
Dutzi, I., & Hommel, B. (2009). The microgenesis of action-effect binding. Psychological Research, 73, 425-435.
Elsner, B., & Hommel, B. (2001). Effect anticipation and action control. Journal of Experimental Psychology: Human Perception and Performance, 27, 229-240.
Elsner, B., & Hommel, B. (2004). Contiguity and contingency in the acquisition of action effects. Psychological Research, 68, 138-154.
Frings, C., Koch, I., Rothermund, K., Dignath, D., Giesen, C., Hommel, B., ... Philipp, A. (2018). Merkmalsintegration und Abruf als wichtige Prozesse der Handlungs-steuerung – eine Paradigmen-übergreifende Perspektive [Feature integration and retrieval as core processes in action control - a cross-paradigm perspective]. Psychologische Rundschau.
Giesen, C., & Rothermund, K. (2014). Distractor repetitions retrieve previous responses and previous targets: Experimental dissociations of distractor-response and distractor-target bindings. Journal of Experimental Psychology: Learning, Memory, and Cognition, 40, 645-659.
Herwig, A. & Waszak, F. (2012). Action-effect bindings and ideomotor learning in intention and stimulus-based actions. Frontiers in Psychology, 3, 1-18.
Hoffmann, J., Berner, M., Butz, M. V., Herbort, O., Kiesel, A., Kunde, W., & Lenhard, A. (2007). Explorations of anticipatory behavioral control (ABC): A report from the cognitive psychology unit of the University of Würzburg. Cognitive Processing, 8, 133-142.
Hommel, B. (1998). Event files: Evidence for automatic integration of stimulus-response episodes. Visual Cognition, 5, 183-216.
Hommel, B. (2004). Event files: Feature binding in and across perception and action. Trends in Cognitive Sciences, 8, 494-500.
Hommel, B. (2005). How much attention does an event file need? Journal of Experimental Psychology. Human Perception and Performance, 31, 1067–1082.
Hommel, B., & Wiers, R. W. (2017). Towards a unitary approach to human action control. Trends in Cognitive Sciences, 21, 940-949.
Janczyk, M., Heinemann, A., & Pfister, R. (2012). Instant attraction: Immediate action-effect bindings occur for both, stimulus-and goal-driven actions. Frontiers in Psychology, 3, 446.
Kahneman, D., Treisman, A., & Gibbs, B. J. (1992). The reviewing of object files: Object-specific integration of information. Cognitive Psychology, 24, 175-219.
Logan, G. D. (1988). Toward an instance theory of automatization. Psychological Review, 95, 492-527.
Memelink, J., & Hommel, B. (2013). Intentional weighting: A basic principle in cognitive control. Psychological Research, 77, 249-259.
Moeller, B., & Frings, C. (2014). Attention meets binding: Only attended distractors are used for the retrieval of event files. Attention, Perception, & Psychophysics, 76, 959-978.
Moeller, B., & Frings, C. (2017). Dissociation of binding and learning processes. Attention, Perception, & Psychophysics, 79, 2590-2605.
Moeller, B., Frings, C., & Pfister, R. (2016). The structure of distractor-response bindings: Conditions for configural and elemental integration. Journal of Experimental Psychology: Human Perception and Performance, 42, 464-479.
Moeller, B., Pfister, R., Kunde, W., & Frings, C. (2016). A common mechanism behind distractor-response and response-effect binding? Attention, Perception, & Psychophysics, 78, 1074-1086.
Pfister, R., & Janczyk, M. (2013). Confidence intervals for two sample means: Calculation, interpretation, and a few simple rules. Advances in Cognitive Psychology, 9, 74-80.
Schmidt, J. R., De Houwer, J., & Rothermund, K. (2016). The parallel episodic processing (PEP) model 2.0: A single computational model of stimulus-response binding, contingency learning, power curves, and mixing costs. Cognitive Psychology, 91, 82-108.
Singh, T., Moeller, B., & Frings, C. (2016). Five shades of grey: Generalization in distractor-based retrieval of SR episodes. Attention, Perception, & Psychophysics, 78, 2307-2312.
Tukey, J. (1977). Exploratory data analysis. Reading, MA: Addison-Wesley.
Vogel, D., Scherbaum, S., & Janczyk, M. (2018). Dissociating decision strategies in free-choice tasks–A mouse tracking analysis. Acta Psychologica, 190, 65-71.
Waszak, F., Hommel, B., & Allport, A. (2004). Semantic generalization of stimulus–task bindings. Psychonomic Bulletin & Review, 11, 1027-1033.
Weller, L., Kunde, W., & Pfister, R. (2017). Non-action effect binding: A critical re-assessment. Acta Psychologica, 180, 137-146.
Wolfensteller, U., & Ruge, H. (2011). On the timescale of stimulus-based action–effect learning. The Quarterly Journal of Experimental Psychology, 64, 1273-1289.
Ziessler, M., & Nattkemper, D. (2001). Learning of event sequences is based on response-effect learning: Further evidence from a serial reaction task. Journal of Experimental Psychology: Learning, Memory, and Cognition, 27, 595-613.
Ziessler, M., & Nattkemper, D. (2002). Effect anticipation in action planning. In W. Prinz & B. Hommel (Eds.), Common mechanisms in perception and action: Attention & Performance XIX (pp. 645–672). Oxford, UK: University Press.
Birte Moeller and Christian Frings, University of Trier, Universitätsring 15, D-54296 Trier, Germany. Wilfried Kunde and Roland Pfister, University of Würzburg, Röntgenring 11, D-97070 Würzburg, Germany.
The research reported in this article was supported by grants of the Deutsche Forschungsgemeinschaft (FOR 2790 and MO 2839/2-2).
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
About this article
Cite this article
Moeller, B., Pfister, R., Kunde, W. et al. Selective binding of stimulus, response, and effect features. Psychon Bull Rev 26, 1627–1632 (2019). https://doi.org/10.3758/s13423-019-01646-1
- Action control
- Stimulus-response binding
- Response-effect binding
- Event file