Abstract
In natural scenes, audiovisual events deriving from the same source are synchronized at their origin. However, from the perspective of the observer, there are likely to be significant multisensory delays due to physical and neural latencies. Fortunately, our brain appears to compensate for the resulting latency differences by rapidly adapting to asynchronous audiovisual events by shifting the point of subjective synchrony (PSS) in the direction of the leading modality of the most recent event. Here we examined whether it is the perceived modality order of this prior lag or its physical order that determines the direction of the subsequent rapid recalibration. On each experimental trial, a brief tone pip and flash were presented across a range of stimulus onset asynchronies (SOAs). The participants’ task alternated over trials: On adaptor trials, audition either led or lagged vision with fixed SOAs, and participants judged the order of the audiovisual event; on test trials, the SOA as well as the modality order varied randomly, and participants judged whether or not the event was synchronized. For test trials, we showed that the PSS shifted in the direction of the physical rather than the perceived (reported) modality order of the preceding adaptor trial. These results suggest that rapid temporal recalibration is determined by the physical timing of the preceding events, not by one’s prior perceptual decisions.
Similar content being viewed by others
In natural environments the brain receives information via multiple sensory modalities. This multisensory information often interacts when it is presented in close temporal proximity, to facilitate perception (Alais & Burr, 2004; Morein-Zamir, Soto-Faraco, & Kingstone, 2003; Olivers & Van der Burg, 2008; Shipley, 1964; Van der Burg, Olivers, Bronkhorst, & Theeuwes, 2008; Van der Burg, Talsma, Olivers, Hickey, & Theeuwes, 2011; Vroomen & de Gelder, 2000). For instance, in a noisy auditory environment, speech comprehension is better if one can observe the speaker’s lip movements (Sumby & Pollack, 1954). Typically, multisensory interactions are strongest when the information from different sensory modalities is approximately synchronized, and they reduce with increasing asynchrony (Slutsky & Recanzone, 2001; Van der Burg, Cass, & Alais, 2014; Van der Burg, Cass, Olivers, Theeuwes, & Alais, 2010; van Wassenhove, Grant, & Poeppel, 2007). This is an important point, because although multisensory information deriving from a single source will be emitted simultaneously, multisensory asynchronies are almost inevitable from the perspective of the observer, due to differences in physical propagation speeds and neural processing latencies (Alais, Newell, & Mamassian, 2010). For instance, light propagates vastly more quickly through the air than does sound. Auditory sensory information, by contrast, is processed more quickly in the brain than is visual information, although this may vary depending on factors such as the stimulus intensity (Los & Van der Burg, 2013; Simith, 1933) and attention (i.e., prior entry; Shore, Spence, & Klein, 2001; Spence & Driver, 1996, 1997).
Given that multisensory delays are inevitable, one might argue that multisensory interactions are always suboptimal. Fortunately, the brain appears to compensate for asynchronous multisensory signals. In 2004, two studies independently reported that following several minutes of exposure to a fixed audiovisual asynchrony, participants shifted their point of subjective synchrony (PSS) in the direction of the asynchrony to which they had been exposed (Fujisaki, Shimojo, Kashino, & Nishida, 2004; Vroomen, Keetels, de Gelder, & Bertelson, 2004). This phenomenon is known as temporal recalibration and has been studied extensively using prolonged adaptation procedures in which participants are exposed to a fixed multisensory asynchrony for periods extending from several seconds to several minutes (Cass, Oake, & Van der Burg, 2015; Di Luca, Machulla, & Ernst, 2009; Harrar & Harris, 2008; Heron, Whitaker, McGraw, & Horoshenkov, 2007; Keetels & Vroomen, 2008; Machulla, Di Luca, Froehlich, & Ernst, 2012; Navarra, Garcia-Morera, & Spence, 2012; Roseboom & Arnold, 2011; Roseboom, Kawabe, & Nishida, 2013; Van der Burg, Alais, & Cass, 2015; Vroomen, van Linden, de Gelder, & Bertelson, 2007; Yarrow, Jahn, Durant, & Arnold, 2011; Yarrow, Roseboom, & Arnold, 2011; Yuan, Li, Bi, Yin, & Huang, 2012). Recently, we showed that temporal recalibration occurs rapidly without the need for an explicit adaptation procedure (Van der Burg, Alais, & Cass, 2013). In our study, each trial consisted of a brief tone pip and flash presented at one of several possible stimulus onset asynchronies (SOAs). Participants performed a synchrony judgment (SJ) on each audiovisual pair, indicating whether the stimuli were perceived as synchronous or asynchronous, and we fitted Gaussian functions to the distribution of SJs as a function of SOA in order to estimate the PSS. We then binned the responses on each trial on the basis of the modality order of the preceding trial (i.e., whether vision was leading or lagging) and fitted Gaussians separately to each bin. This analysis showed that the PSS was strongly influenced by the modality order of the preceding trial, with the PSS shifting in the direction of the leading modality of the previous trial.
Our results demonstrated that temporal recalibration can occur extremely quickly, requiring exposure to just a single asynchronous audiovisual event, and this finding has now been replicated numerous times by others using both audiovisual stimuli (e.g., De Niear, Noel, & Wallace, 2017; Noel, De Niear, Stevenson, Alais & Wallace (2017); Noel, De Niear, Van der Burg, & Wallace, 2016; Roseboom, 2017; Simon, Noel, & Wallace, 2017; Turi, Karaminis, Pellicano, & Burr, 2016) and visuotactile stimuli (Lange, Kapala, Krause, Baumgarten, & Schnitzler, 2018). In audiovisual spatial tasks, too, rapid recalibration has been observed that is driven by the previous spatial discrepancy (Mendonça, Escher, van de Par, & Colonius, 2015; Wozny & Shams, 2011). Rapid temporal recalibration suggests that our percept of synchrony is far more malleable than was previously thought and that the brain may be engaged in a continual process of temporal realignment in order to adapt rapidly to changes in the audiovisual environment. One aspect of this process that remains unknown is whether rapid temporal recalibration is determined by the physical modality order of prior events or by subjective factors such as their perceived modality order. Although it may seem unlikely at first that the true physical timing would be available as a basis for recalibration, several recent studies have suggested that estimates of physical timing are indeed available. Leone and McCourt (2015) showed that in a multisensory context timing for action is veridical, despite perceived time lags, and work by Harrar, Harris, and Spence (2017) also demonstrated that physical audiovisual synchrony, rather than perceived synchrony, affects multisensory integration. Recalibration can occur without feedback (unsupervised recalibration; Zaidel, Ma, & Angelaki, 2013), and various factors may guide the computation of physical synchrony, such as internal models of spatial layout and distances and known lags for various actions and modalities. It is unknown, however, whether these processes can act automatically, without awareness.
In our original study (Van der Burg et al., 2013), we argued that rapid temporal recalibration occurs automatically, because recalibration occurred even when the prior trial had required no response from observers (so-called passive trials). It is conceivable, however, that during passive observing participants still generated a subjective decision regarding the timing of the audiovisual events, and simply suppressed the keypress response. Furthermore, in our study we showed that the perceived synchrony on a given trial t was contingent upon the modality order not only on the preceding trial (t–1), but also on trial t–2, making it very unlikely that rapid recalibration is due to a decisional process, since then participants would have to recall the modality order over the last two trials when performing the task on trial t. Another relevant study of visuo-motor temporal recalibration investigated whether subjective awareness of the adaptor lag was crucial for observing recalibration (Tsujita & Ichikawa, 2015). In this study, Tsujita and Ichikawa allocated participants to either a single-step or a multistep lag condition. In the single-step condition, the temporal lag between the motor response and the flash was fixed (200 ms) throughout the adaptation procedure, and participants were informed about this delay so as to make them fully aware of it. In the multistep condition, an initially small temporal lag of 40 ms was systematically increased to 200 ms during the adaptation procedure, so that participants were not aware of the extent of the temporal lag at the end. Tsujita and Ichikawa concluded that being aware of the lag was critical for observing visuo-motor temporal recalibration, because there was recalibration in the single-step condition but not in the multistep condition. On the face of it, this finding seems consistent with another recent study showing that temporal recalibration to audiovisual asynchrony is absent when participants were not aware of a subliminal visual stimulus during the prolonged adaptation procedure, even though the researchers found clear evidence that participants had processed the unseen visual stimulus (Gallagher, Yarrow, & Arnold, 2014). Although Gallagher and colleagues showed that awareness for the visual event during adaptation is crucial for observing a temporal recalibration effect, it remains unclear whether awareness of the modality order is also a prerequisite for observing temporal recalibration.
Recently, Simon, Noel, and Wallace (2017) measured event-related potentials (ERPs) in order to examine the neural basis of rapid temporal recalibration. Simon and colleagues reported a behavioral rapid recalibration effect (Van der Burg et al., 2013) and showed that ERPs over parietal electrodes were modulated by the modality order on the preceding trial. Interestingly, the latency of this effect (> 125 ms after the second stimulus onset) suggests that recalibration influences rather late processing stages, possibly decisional processes (see also Stekelenburg, Sugano, & Vroomen, 2011). However, as the authors noted, the lack of early ERP components in their study does not necessarily rule out the possibility that rapid temporal recalibration influences early processing stages. For instance, when measuring ERPs, it is simply impossible to measure systematic changes in the phase of oscillations, and such changes could underlie temporal recalibration (see, e.g., Kösem, Gramfort, & van Wassenhove, 2014, for an interesting example based on a prolonged adaptation procedure).
Overall, it remains unclear whether rapid temporal recalibration occurs automatically or whether subjective experiences modulate rapid temporal recalibration. In this study we used a novel task-interleaving procedure to examine whether rapid temporal recalibration follows the physical modality order of audiovisual events or whether the perceived modality order determines rapid realignment.
Experiment 1
The aim of Experiment 1 was to examine whether awareness of the modality order of immediately preceding audiovisual events determines whether or not rapid temporal recalibration to asynchronous audiovisual events occurs. The paradigm is shown in Fig. 1. On every trial, a brief tone pip and flash were presented across a range of SOAs. As in our previous study (Van der Burg et al., 2013), we instructed participants to perform a temporal task on each trial. In the present experiment, however, participants were presented with two kinds of trials in alternating order: adaptor trials and test trials. On adaptor trials, participants judged the order of the audiovisual events, to ascertain whether or not their perception matched the physical temporal order (respectively, correct or incorrect responses). On each adapter trial, the auditory pip either led or lagged vision by a fixed SOA (i.e., one SOA in which the visual [V] stimulus led, and one SOA in which the auditory [A] stimulus led). It was important to fix the SOA on adapter trials, because it is known that the interval between audiovisual events influences temporal recalibration (see, e.g., Roseboom, 2017; Simon et al., 2017; Van der Burg et al., 2013; Van der Burg & Goodbourn, 2015). Indeed, if one does not fix the SOA, a difference between temporal recalibration for correct responses versus incorrect responses on trial t–1 could be assigned to a difference between the SOAs used on for correct and incorrect trials t–1, since it is easier to make a temporal order judgment for long than for short SOAs. On test trials, the SOA, as well as the modality order, varied randomly, and participants judged whether or not the events were synchronized. If temporal recalibration occurs regardless of the perceived modality order, then we would expect the PSS on test trials to depend on the physical modality order on the preceding adaptor trial, regardless of whether or not participants perceived the order to those adaptor stimuli correctly. In contrast, if awareness of the modality order is crucial for observing temporal recalibration, then we would expect the PSS to be contingent on the perceived modality order on the preceding trial. In other words, we would expect the PSS on a given trial t to be shifted in time toward the perceived leading modality in trial t–1.
The paradigm used in Experiment 1. Participants were presented with two kinds of trials in alternating order—adaptor trials and test trials—each preceded by a word cue indicating the upcoming task. On adaptor trials, participants performed a temporal order judgment (TOJ) to indicate the perceived order of an audiovisual (AV) stimulus presented with a fixed stimulus onset asynchrony (SOA) that could have either stimulus order (vision leading or sound leading). On test trials, participants performed a synchrony judgment (SJ) on an AV stimulus that varied over a range of positive and negative SOAs, so that a psychometric function could be measured and a point of subjective synchrony obtained.
Method
Participants
Twenty-six students (16 females, ten males; mean age = 22.6 years, ranging from 18 to 59 years) participated in the experiment. All were naïve as to the purpose of the experiment and received €8/h or course credits for their participation.
Stimuli and apparatus
Participants were seated in a dimly lit cabin. The E-Prime 2.0 software was used to program and run the experiment. The visual stimulus was a white ring (radius 2.1°, width 1.0°; 112 cd/m2), presented around a white fixation dot (radius 0.5°) on a black background (< 0.5 cd/m2) for a duration of 50 ms. The visual stimuli were presented on an LCD monitor (refresh rate of 120 Hz) viewed from approximately 80 cm distance. The auditory stimulus was a pure tone (500 Hz, duration 50 ms, sampling rate of 44.1 kHz), presented using headphones at a comfortable suprathreshold listening level and worn throughout the experiment (see also Van der Burg et al., 2013).
Procedure and design
A trial started with the presentation of a word cue for a duration of 1,000 ms, to inform the participant whether the task was a temporal order judgment (odd trials, here called adaptor trials) or a synchrony judgment (even trials, called test trials). On adaptor trials, the word cue was “aud or vis,” and participants judged the order of the audiovisual events by pressing the “a” or “v” key to indicate whether the auditory or the visual stimulus led, respectively. The sound either preceded the visual stimulus by 114 ms or followed the visual event by 186 ms. The audiovisual timing was confirmed using a Le Croy oscilloscope (Wavesurfer 24XS; standard deviation = 4.8 ms, estimated over 100 trials). The SOAs for audition-leading and vision-leading adaptor trials were fixed in order to ensure that any difference on the subsequent test trials could not be explained by the SOAs used during adaptation. On test trials, the word cue was “sim or not,” and participants judged whether or not the audiovisual events were synchronized by pressing the “1” or the “0” key, respectively. The sound either preceded or followed the visual stimulus by an SOA randomly drawn from the set – 364, – 114, – 14, 86, 186, or 436 ms. Here, a negative SOA indicates that audition leads vision, whereas a positive SOA indicates that the sound follows vision. The next trial was initiated after participants had made the unspeeded response. Participants performed four experimental blocks of 216 trials each (108 TOJ trials and 108 SJ trials, in alternating order). Prior to the experiment, participants completed a practice block in order to get familiar with both tasks.
Results
The results are shown in Fig. 2. The data from four participants were excluded (leaving a sample size of 22 participants) from further analyses, since one participant was guessing in the TOJ task (50% correct) and had an overall performance of ~ 50% simultaneous in the SJ task for each SOA condition, whereas three other participants performed too well in the TOJ task, leaving an insufficient number of trials to fit a Gaussian to the synchrony distribution following incorrect TOJ responses.
Results of Experiment 1. (A) Proportions of synchrony responses on test trials as a function of stimulus onset asynchrony (SOA), for trials following an incorrect temporal order judgment on the previous (i.e., adaptor) trial. (B) Proportions of synchrony responses on test trials as a function of SOA, for trials following a correctly judged temporal order on the previous trial. Negative SOAs indicate that audition was leading, whereas positive SOAs indicate that vision was leading. (C) Mean points of subjective synchrony (PSSs) as a function of the response and modality order on the preceding trial. In all panels, the error bars represent ± 1 standard error of the mean.
TOJ performance
The modality order was correctly perceived on 64.1% of the trials, and the performance was significantly better than chance level (50%), t(21) = 7.172, p < .001 (two-tailed t test), confirming that participants were able to perform the task. The mean accuracies were 62.2% correct when audition was leading and 66.0% correct when vision was leading. A two-tailed t test yielded no significant difference between the two modality orders, t(21) = 0.82, p = .422.
SJ performance
Figures 2a and b illustrate the proportions of simultaneous responses as a function of the SOA on test trials and the modality order performance on adaptor trials. The synchrony distributions following incorrect TOJs are shown in Fig. 2a, and the synchrony distributions following correct TOJs are in Fig. 2b. For each participant, we fitted the data with Gaussian distributions in order to estimate the PSS (i.e., the mean of the distribution), amplitude, and synchrony bandwidth (the distribution’s standard deviation) for each modality order on the preceding trial for correct and incorrect responses on adaptor trials. The continuous lines in Figs. 2a and b illustrate the best Gaussian fits to the group mean data. Temporal recalibration is defined by the difference between the PSSs when vision leads relative to when audition leads on the preceding adaptor trials. Here and elsewhere in the article, a positive PSS value indicates that audition must follow vision in order to be perceived simultaneously, and vice versa for a negative PSS value.
Figure 2c illustrates the mean PSSs as a function of modality order and the TOJ response on adaptor trials. An analysis of variance (ANOVA) on the mean PSSs was conducted with modality order and TOJ response on adaptor trials as within-subjects variables. The ANOVA yielded a significant modality order effect, F(1, 21) = 9.9, p = .005, η2 = .321, in that the PSS on test trials was significantly smaller when audition led on the preceding trial (11 ms) than when vision led on the preceding trial (35 ms). Importantly, this rapid temporal recalibration effect did not depend upon the response to the preceding trial, since the main effect of TOJ response as well as the two-way interaction failed to reach significance (Fs < .223), suggesting that the perceived order of the audiovisual pair on adaptor trials had no influence on temporal recalibration. A Bayesian analysis evaluating the strength of this null interaction yielded a Bayes factor of 3.029, signifying moderate evidence for the absence of this interaction (Wagenmakers et al., 2018).
The same classical statistical analyses were conducted on the Gaussian bandwidth and amplitude parameters. An ANOVA on the mean bandwidth, with adaptor trial TOJ response and modality order as within-subjects variables, yielded no significant main effect of modality order, F(1, 21) = 4.2, p = .054, η2 = .166, or TOJ response, F(1, 21) = 1.8, p = .189, η2 = .081. Neither was the two-way interaction significant, F(1, 21) = 1.7, p = .20, η2 = .074. An ANOVA on the mean amplitudes yielded a trend toward a significant main effect of TOJ response, F(1, 21) = 4.3, p = .050, η2 = .171, as the amplitude was higher when participants were correct on the preceding adaptor trials (.86) than when they were incorrect (.83). The main effect of modality order on the preceding trial and the two-way interaction failed to reach significance, F(1, 21) = 1.2, p = .282, η2 = .055, and F(1, 21) = 0.4, p = .560, η2 = .016, respectively.
Experiment 2
The aim of Experiment 2 was to examine whether rapid recalibration to asynchronous audiovisual events occurs also when participants are asked to perform a visual localization task, so that participants were not actively engaged in a subjective ordering task (i.e., the relative timing between the auditory and visual event). The paradigm used in Experiment 2 is shown in Fig. 3. Experiment 2 used the same task-interleaving paradigm as in Experiment 1, except that, on adaptor trials, the white ring had a small notch on either its left or right side, and participants reported on which side the notch was located. This task replaced the TOJ task used in the adaptor trials of Experiment 1. Importantly, the audiovisual asynchronies presented during the notch localization task (vision leading or auditory leading by a fixed SOA) were identical to those used in the adaptor trials of Experiment 1. If recalibration occurs unconsciously and automatically, then we would expect to find a recalibration effect on test trials that would correspond to the physical audiovisual asynchrony presented in the adaptor trials, even though participants were actively engaged in a notch localization task on those trials.
The paradigm used in Experiment 2 was similar to that of Experiment 1. The trial sequences again alternated between adaptor trials (with a fixed positive or negative SOA) and test trials (with a range of positive and negative SOAs), exactly as in Experiment 1. The only difference was that the TOJ task previously used on adaptor trials was changed to a visual spatial localization task, while the stimulus timing was retained from Experiment 1. The white ring was modified on adaptor trials to have a small notch on the left or right side, and the task was to report on which side the notch was located. The task on test trials remained an SJ task, exactly as in Experiment 1.
Method
This experiment was identical to Experiment 1 in all respects, except that on adaptor trials, participants performed a visual localization instead of a TOJ task. For this purpose, the white ring was modified on those trials to have a small notch on the left or the right side, and the task was to report on which side the notch was located (see Fig. 3). Participants were asked to press the “s” or the “d” key when the notch was presented on the left or the right side of the ring, respectively. Twenty-two new participants (14 females, eight males; mean age = 22.7 years, ranging from 19 to 32 years) participated in the experiment. All participants were naïve as to the purpose of the experiment and received €8/h or course credit for their participation. On adaptor trials, the word cue “left or right” was used to indicate that a visual spatial task was required. The participants performed four experimental blocks of 216 trials each (108 spatial trials and 108 SJ trials, in alternating order).
Results
Four participants were excluded from further analyses. Three of these participants responded “simultaneous” on > 85% of the synchrony judgment trials (group mean: 68%). The other participant performed at approximately 50% regardless of the SOA, indicating that he or she was simply guessing. The results of the remaining 18 participants for Experiment 2 are shown in Fig. 4.
Results of Experiment 2: Proportions of synchrony responses on test trials as a function of stimulus onset asynchrony (SOA) and of the modality order on adaptor trials. Negative SOAs indicate that audition was leading, whereas positive SOAs indicate that vision was leading. Error bars represent ± 1 standard error of the mean.
Visual localization performance
Overall, performance on the visual localization task was very good (99.5% correct).
SJ performance
Figure 4 illustrates the proportions of simultaneity responses as a function of the SOA on test trials and the modality order on adaptor trials. Three separate ANOVAs were conducted in order to examine whether the PSS, bandwidth, and amplitude depended on the modality order on adapter trials (in which participants performed the notch localization task). The PSS was significantly smaller when audition led (19.9 ms) than when vision led (55.1 ms) on adaptor trials, F(1, 17) = 22.6, p < .001, η2 = .570, indicating that participants adapted to the asynchrony in odd trials while they were performing a purely visual spatial localization task. We conducted the same analysis on the bandwidth and amplitude of the Gaussian fits. The modality order on adaptor trials also had a significant effect on the bandwidth, F(1, 17) = 5.7, p = .029, η2 = .251, since the bandwidth was smaller when audition led (244 ms) than when vision led (254 ms) on the preceding trial. The modality order on the adaptor trials had no significant effect on the amplitude, F(1, 17) = 0.5, p = .484, η2 = .029.
General discussion
In this study, we showed that temporal recalibration occurs after exposure to a single asynchronous audiovisual event (Noel, De Niear, Stevenson, et al., 2016; Noel, De Niear, Van der Burg, & Wallace, 2016; Turi et al., 2016; Van der Burg et al., 2013; Van der Burg & Goodbourn, 2015). That is, the PSS on a given trial was contingent on the modality order on the preceding trial. Importantly, we showed that this shift in PSS was unaffected by the perceptual decision made in the preceding trial about the temporal order of the audiovisual stimuli (Exp. 1). That performance was determined exclusively by the physical modality order and not by one’s decision regarding this order (audition first vs. vision first) might suggest that this rapid form of temporal recalibration is likely to be mediated by early sensory processes, which operate without reference to one’s conscious appraisal of prior temporal events. This interpretation was reinforced by the results of Experiment 2, which showed that even when observers were engaged in a spatial visual task (making the relative timing of the prior events irrelevant), they still demonstrated PSS shifts consistent with the physical temporal order of the prior events.
Whereas Gallagher et al. (2014) showed that temporal recalibration is absent when participants are not aware of the visual event during a prolonged adaptation procedure, here we showed that recalibration is present when participants are aware of the audiovisual events, but that awareness of the temporal order of these events is not a prerequisite for observing rapid temporal recalibration. The results are inconsistent with a recent study showing that temporal subjective awareness of the adaptor lag during prolonged adaptation is crucial for observing visuo-motor temporal recalibration (Tsujita & Ichikawa, 2015). However, it might be important to note that, in contrast to the present study, both of these previous studies (Gallagher et al., 2014; Tsujita & Ichikawa, 2015) applied an explicit adaptation procedure. Recently, we showed that the classical form of temporal recalibration, which involves a prolonged period of asynchronous adaptation, and the rapid form demonstrated here and elsewhere (Noel, De Niear, Stevenson, et al., 2016; Noel, De Niear, Van der Burg, & Wallace, 2016; Turi et al., 2016; Van der Burg et al., 2013; Van der Burg & Goodbourn, 2015), operate with different, noninteracting time courses, suggesting that they might be distinct and independent processes (Van der Burg, Alais, & Cass, 2015).
A recent study by Bruns and Röder (2015) corroborated the notion that two distinct recalibration mechanisms operate at different timescales. Assuming that the prolonged and rapid forms of temporal recalibration are separable, it remains a possibility that the prolonged form may be determined by decisional factors that may or may not be consciously mediated. This may also explain why Tsujita and Ichikawa (2015) failed to observe visuo-motor recalibration when participants were unaware of the temporal lag. Related to this, it is not clear whether rapid recalibration to other sensory combinations, such as auditory–tactile or visual–tactile stimuli, would show evidence for recalibration without awareness. We have previously shown that rapid recalibration fails to occur for these other stimulus combinations (Van der Burg, Orchard-Mills, & Alais, 2015), just as it fails to occur for unimodal combinations (Harvey, Van der Burg, & Alais, 2014). The findings of the present study suggest a unique mechanism for rapid audiovisual recalibration. It is not yet clear whether the present finding of automatic recalibration for audiovisual stimuli would extend to these other sensory combinations.
What are the functional implications of the results reported in the present study? It has been suggested that audiovisual temporal recalibration may serve to realign multisensory neural/perceptual responses to events that, although synchronous at their source, are generally misaligned in the brain due to stimulus factors including distance and stimulus intensity (Heron et al., 2007; Navarra, Fernández-Prieto, & Garcia-Morera, 2013; Yuan et al., 2012). Our results are consistent with this view. Not only does this form of temporal recalibration operate extremely rapidly, signifying a highly adaptable process (Van der Burg et al., 2013), but it is informed by the physical timing of prior events rather than by one’s decision (and hence, possible misinterpretation) regarding these events. Not only does the latter property ensure that one’s perception will align more precisely with ongoing physical variations in proximal event timing, it reduces the possibility that one’s perception of event timing will be subject to cumulative decisional drift, which would be expected in a purely decisionally determined system following consecutive instances of a given decision (Rhodes, Roseboom, & Seth, 2018; Yarrow, Minaei, & Arnold, 2015).
Another outstanding question concerns our interleaving of temporal order and synchrony judgments. Yarrow, Jahn, et al. (2011) proposed that temporal order and synchrony judgments are based on separate decision criteria. This view finds some support from a study by van Eijk, Kohlrausch, Juola, and van de Par (2008), who found that the PSS estimates derived from a set of SJ and TOJ tasks are uncorrelated. It is possible, therefore, that the observed absence of any interaction between the temporal order and synchrony judgments in our Experiment 1 may have been due to different sets of decisional criteria being activated on the adapter and test trials. If this is true, it is indeed remarkable that the sign of physical asynchrony presented during the TOJ task predicted significant shifts in the PSSs measured using SJs. Future research disentangling the effects of physical lag and temporal order decisions on subsequent TOJs will be necessary in order to determine whether temporal recalibration does in fact depend on dissociable decisional processes similar to those observed here. How temporal recalibration might operate upon temporal order and simultaneity judgment tasks is poorly understood. To complicate matters, as our own work has shown, prolonged and short-term (i.e., trial-by-trial) temporal recalibration exhibit distinct time courses (Van der Burg, Alais, & Cass, 2015). The question of which of these various recalibration tasks and effects are mediated by adaptive variation in sensory timing and/or high-level decisional criteria is the topic of current and ongoing investigation.
To summarize, we found that the brain rapidly adapts to its immediate past. That is, the perceived synchrony on a given trial was contingent on the physical modality order of the preceding trial and not the modalities’ perceived temporal order. Whereas others have proposed that this rapid recalibration is due to rather late decisional processes (Simon et al., 2017; Yarrow, Jahn, et al., 2011), here we have illustrated that subjective measures such as perceived order do not influence the sign of rapid temporal recalibration. From an ecological point of view, one might argue that the brain rapidly adapts to multisensory asynchronies in order to create a coherent picture of the world, such that signals belonging to the same object are synchronized so that interference may be minimized and multisensory benefits maximized. That rapid recalibration should be contingent on the physical asynchronies to which one has previously been exposed rather than on one’s subjective evaluation of these asynchronies implies that one need not be consciously aware of the temporal structure of events during everyday tasks to experience the perceptual benefits that recalibration may afford. This notion is consistent with results from recent studies by Leone and McCourt (2015) and Harrar, Harris, and Spence (2017) showing that multisensory integration depends on physical temporal alignment and not on the perceived temporal alignment.
References
Alais, D., & Burr, D. (2004). The ventriloquism effect results from near-optimal bimodal integration. Current Biology, 14, 257–262. https://doi.org/10.1016/j.cub.2004.01.029
Alais, D., Newell, F. N., & Mamassian, P. (2010). Multisensory processing in review: From physiology to behaviour. Seeing and Perceiving, 23, 3–38.
Bruns, P., & Röder, B. (2015). Sensory recalibration integrates information from the immediate and the cumulative past. Scientific Reports, 5, 12739. https://doi.org/10.1038/srep12739
Cass, J., Oake, D., & Van der Burg, E. (2015). Stretching time: Relativistic lag-induced shifts in perceived audiovisual synchrony using cluttered displays. Journal of Vision, 15(11), 9. https://doi.org/10.1167/15.11.9
De Niear, M. A., Noel, J. P., & Wallace, M. T. (2017). The impact of feedback on the different time courses of multisensory temporal recalibration. Neural Plasticity, 2, 3478742. https://doi.org/10.1155/2017/3478742
Di Luca, M., Machulla, T., & Ernst, M. O. (2009). Recalibration of multisensory simultaneity: Cross-modal transfer coincides with a change in perceptual latency. Journal of Vision, 9(12), 7. https://doi.org/10.1167/9.12.7
Fujisaki, W., Shimojo, S., Kashino, M., & Nishida, S. (2004). Recalibration of audiovisual simultaneity. Nature Neuroscience, 7, 773–778.
Gallagher, R., Yarrow, K., & Arnold, D. (2014). In which’s the time? Temporal recalibration is absent without awareness. Journal of Vision, 14(10), 1155. https://doi.org/10.1167/14.10.1155
Harrar, V., & Harris, L. R. (2008). The effect of exposure to asynchronous audio, visual, and tactile stimulus combinations on the perception of simultaneity. Experimental Brain Research, 186, 517–524.
Harrar, V., Harris, L. R., & Spence, C. (2017). Multisensory integration is independent of perceived simultaneity. Experimental Brain Research, 253, 763–775. https://doi.org/10.1007/s00221-016-4822-2
Harvey, C., Van der Burg, E., & Alais, D. (2014). Rapid temporal recalibration occurs crossmodally without stimulus specificity but is absent unimodally. Brain Research, 1585, 120–130.
Heron, J., Whitaker, D., McGraw, P. V., & Horoshenkov, K. V. (2007). Adaptation minimizes distance-related audiovisual delays. Journal of Vision, 7(13), 5. https://doi.org/10.1167/7.13.5
Keetels, M., & Vroomen, J. (2008). Temporal recalibration to tactile-visual asynchronous stimuli. Neuroscience Letters, 430, 130–134.
Kösem, A., Gramfort, A., & van Wassenhove, V. (2014). Encoding of event timing in the phase of neural oscillations. NeuroImage, 92, 274–284. https://doi.org/10.1016/j.neuroimage.2014.02.010
Lange, J., Kapala, K., Krause, H., Baumgarten, T. J., & Schnitzler, A. (2018). Rapid temporal recalibration to visuo-tactile stimuli. Experimental Brain Research, 236, 347–354. https://doi.org/10.1007/s00221-017-5132-z
Leone, L. M., & McCourt, M. E. (2015). Dissociation of perception and action in audiovisual multisensory integration. European Journal of Neuroscience, 42, 2915–2922.
Los, S. A., & Van der Burg, E. (2013). Sound speeds vision through preparation, not integration. Journal of Experimental Psychology: Human Perception and Performance, 36, 1612–1624.
Machulla, T., Di Luca, M., Froehlich, E., & Ernst, M. O. (2012). Multisensory simultaneity recalibration: Storage of the aftereffect in the absence of counterevidence. Experimental Brain Research, 217, 89–97.
Mendonça, C., Escher, A., van de Par, S., & Colonius, H. (2015). Predicting auditory space calibration from recent multisensory experience. Experimental Brain Research, 233, 1983–1991. https://doi.org/10.1007/s00221-015-4259-z
Morein-Zamir, S., Soto-Faraco, S., & Kingstone, A. (2003). Auditory capture of vision: Examining temporal ventriloquism. Cognitive Brain Research, 17, 154–163.
Navarra, J., Fernández-Prieto, J., & Garcia-Morera, J. (2013). Realigning thunder and lightning: Temporal adaptation to spatiotemporally distant events. PLoS ONE, 8, e84278. https://doi.org/10.1371/journal.pone.0084278
Navarra, J., Garcia-Morera, J., & Spence, C. (2012). Temporal adaptation to audiovisual asynchrony generalizes across different sound frequencies. Frontiers in Psychology, 3, 152. https://doi.org/10.3389/fpsyg.2012.00152.
Noel, J.-P., De Niear, M., Van der Burg, E., & Wallace, M. T. (2016). Audiovisual simultaneity judgment and rapid recalibration throughout the lifespan. PLoS ONE, 11,e0161698. https://doi.org/10.1371/journal.pone.0161698
Noel, J.-P., De Niear, M. A., Stevenson, R., Alais, D., & Wallace, M. T. (2017). Atypical rapid audio-visual temporal recalibration in autism spectrum disorders. Autism Research, 10, 121–129. https://doi.org/10.1002/aur.1633
Olivers, C. N. L., & Van der Burg, E. (2008). Bleeping you out of the blink: Sound saves vision from oblivion. Brain Research, 1242, 191–199.
Rhodes, D., Roseboom, W., & Seth, A. K. (2018). Multiple modality dependent priors in human temporal perception. Manuscript submitted for publication.
Roseboom, W. (2017). Serial dependence across multisensory relative timing tasks. PsyArXiv preprint. https://doi.org/10.17605/OSF.IO/6BKDA
Roseboom, W., & Arnold, D. (2011). Twice upon a time: Multiple concurrent temporal recalibrations of audiovisual speech. Psychological Science, 22, 872–877.
Roseboom, W., Kawabe, T., & Nishida, S. (2013). Audio-visual temporal recalibration can be constrained by content cues regardless of spatial overlap. Frontiers in Psychology, 4, 189. https://doi.org/10.3389/fpsyg.2013.00189
Shipley, T. (1964). Auditory flutter-driving of visual flicker. Science, 145, 1328–1330.
Shore, D. I., Spence, C., & Klein, R. M. (2001). Visual prior entry. Psychological Science, 12, 205–212. https://doi.org/10.1111/1467-9280.00337
Simith, W. F. (1933). The relative quickness of visual and auditory perception. Journal of Experimental Psychology, 16, 239–270.
Simon, D. M., Noel, J. P., & Wallace, M. T. (2017). Event related potentials index rapid recalibration to audiovisual temporal asynchrony. Frontiers in Integrative Neuroscience, 11, 5.
Slutsky, D. A., & Recanzone, G. H. (2001). Temporal and spatial dependency of the ventriloquism effect. NeuroReport, 12, 7–10.
Spence, C., & Driver, J. (1996). Audiovisual links in endogenous covert spatial attention. Journal of Experimental Psychology: Human Perception and Performance, 22, 1005–1030. https://doi.org/10.1037/0096-1523.22.4.1005
Spence, C., & Driver, J. (1997). Audiovisual links in exogenous covert spatial orienting. Perception & Psychophysics, 59, 1–22. https://doi.org/10.3758/BF03206843
Stekelenburg, J. J., Sugano, Y., & Vroomen, J. (2011). Neural correlates of motor–sensory temporal recalibration. Brain Research, 1397, 46–54.
Sumby, W. H., & Pollack, I. (1954). Visual contribution to speech intelligibility in noise. Journal of the Acoustical Society of America, 26, 212–215.
Tsujita, M., & Ichikawa, M. (2015). Awareness of temporal lag is necessary for motor–visual temporal recalibration. Frontiers in Integrative Neuroscience, 9, 64. https://doi.org/10.3389/fnint.2015.00064
Turi, M., Karaminis, T., Pellicano, E., & Burr, D. (2016). No rapid audiovisual recalibration in adults on the autism spectrum. Scientific Reports, 6, 21756. https://doi.org/10.1038/srep21756
Van der Burg, E., Alais, D., & Cass, J. (2013). Rapid recalibration to asynchronous audiovisual stimuli. Journal of Neuroscience, 33, 14633–14637.
Van der Burg, E., Alais, D., & Cass, J. (2015). Audiovisual temporal recalibration occurs independently at two different time scales. Scientific Reports, 5, 14526. https://doi.org/10.1038/srep14526
Van der Burg, E., Cass, J., & Alais, D. (2014). Window of audio-visual simultaneity is unaffected by spatio-temporal visual clutter. Scientific Reports, 4, 5098. https://doi.org/10.1038/srep5098
Van der Burg, E., Cass, J., Olivers, C. N. L., Theeuwes, J., & Alais, D. (2010). Efficient visual search from synchronized auditory signals requires transient audiovisual events. PLoS ONE, 5, e10664. https://doi.org/10.1371/journal.pone.0010664
Van der Burg, E., & Goodbourn, P. T. (2015). Rapid, generalized adaptation to asynchronous audiovisual speech. Proceedings of the Royal Society B, 282, e20143083.
Van der Burg, E., Olivers, C. N. L., Bronkhorst, A. W., & Theeuwes, J. (2008). Pip and pop: Non-spatial auditory signals improve spatial visual search. Journal of Experimental Psychology: Human Perception and Performance, 34, 1053–1065. https://doi.org/10.1037/0096-1523.34.5.1053
Van der Burg, E., Orchard-Mills, E., & Alais, D. (2015). Rapid temporal recalibration is unique to audiovisual stimuli. Experimental Brain Research, 233, 53–59.
Van der Burg, E., Talsma, D., Olivers, C. N. L., Hickey, C., & Theeuwes, J. (2011). Early multisensory interactions affect the competition among multiple visual objects. NeuroImage, 55, 1208–1218.
van Eijk, R. L. J., Kohlraush, A., Juola, J. E., & van de Par, S. (2008). Audiovisual synchrony and temporal order judgments: Effects of experimental method and stimulus type. Perception & Psychophysics, 70, 955–968. https://doi.org/10.3758/PP.70.6.955
van Wassenhove, V., Grant, K. W., & Poeppel, D. (2007). Temporal window of integration in auditory–visual speech perception. Neuropsychologia, 45, 598–607.
Vroomen, J., & de Gelder, B. (2000). Sound enhances visual perception: Cross-modal effects of auditory organization on vision. Journal of Experimental Psychology: Human Perception and Performance, 26, 1583–1590. https://doi.org/10.1037/0096-1523.26.5.1583
Vroomen, J., Keetels, M., de Gelder, B., & Bertelson, P. (2004). Recalibration of temporal order perception by exposure to audio-visual asynchrony. Cognitive Brain Research, 22, 32–35.
Vroomen, J., van Linden, S., de Gelder, B., & Bertelson, P. (2007). Visual recalibration and selective adaptation in auditory–visual speech perception: Contrasting build-up courses. Neuropsychologia, 45, 572–577.
Wagenmakers, E.-J., Marsman, M., Jamil, T., Ly, A., Verhagen, J., Love, J.,… Morey, R. D. (2018). Bayesian inference for psychology: Part I. Theoretical advantages and practical ramifications. Psychonomic Bulletin & Review, 25, 35–57. https://doi.org/10.3758/s13423-017-1343-3
Wozny, D. R., & Shams, L. (2011). Recalibration of auditory space following milliseconds of cross-modal discrepancy. Journal of Neuroscience, 31, 4607–4612.
Yarrow, K., Jahn, N., Durant, S., & Arnold, D. H. (2011). Shifts of criteria or neural timing? The assumptions underlying timing perception studies. Consciousness and Cognition, 20, 1518–1531. https://doi.org/10.1016/j.concog.2011.07.003
Yarrow, K., Minaei, S., & Arnold, D. H. (2015). A model-based comparison of three theories of audiovisual temporal recalibration. Cognitive Psychology, 83, 54–76.
Yarrow, K., Roseboom, W., & Arnold, D. (2011). Spatial grouping resolves ambiguity to drive temporal recalibration. Journal of Experimental Psychology: Human Perception and Performance, 37, 1657–1661.
Yuan, X., Li, B., Bi, C., Yin, H., & Huang, X. (2012). Audiovisual temporal recalibration: Space-based versus context-based. Perception, 41, 1218–1233.
Zaidel, A., Ma, W. J., & Angelaki, D. E. (2013). Supervised calibration relies on the multisensory percept. Neuron, 80, 1544–1557. https://doi.org/10.1016/j.neuron.2013.09.026
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Van der Burg, E., Alais, D. & Cass, J. Rapid recalibration to audiovisual asynchrony follows the physical—not the perceived—temporal order. Atten Percept Psychophys 80, 2060–2068 (2018). https://doi.org/10.3758/s13414-018-1540-9
Published:
Issue Date:
DOI: https://doi.org/10.3758/s13414-018-1540-9