Visual perception traditionally was thought to be almost impenetrable by non-visual information and thus immune to influences of information from different modalities such as audition and touch. However, although visual information may dominate information from other modalities (e.g., Rock & Victor, 1964; Warren, Welch, & McCarthy, 1981), there is neuropsychological (e.g., Falchier, Clavagnier, Barone, & Kennedy, 2002) as well as behavioral (e.g., Shams, Kamitani, & Shimojo, 2000) evidence that non-visual information is capable of altering visual processing. For instance, Sekuler, Sekuler, and Lau (1997) showed that a brief “click” that coincides with the moment of overlap of two moving discs switches the dominant impression of the event from two discs streaming past each other streaming to two discs bouncing off each other. To date, the mechanism driving this qualitative shift in the visual impression is not yet understood, although perceptual (Meyerhoff & Scholl, 2018), attentional (Grassi & Casco, 2009), and cognitive (Grassi & Casco, 2010; Grove, Ashton, Kawachi, & Sakurai, 2012a) mechanisms have been discussed recently.

Tactile-visual interactions

Beyond audiovisual interactions, vision could also be cross-modally affected by touch. A central challenge of studying cross-modal interactions, however, is to ensure that observed effects actually stem from an integration of the sensory information and not just from a shift in the response criterion (e.g., Odgaard, Arieh, & Marks, 2003). One promising way to isolate perceptual processes is to study cross-modal interactions with illusory percepts (e.g., Shimojo & Shams, 2001) or changes in perceptual performance. Regarding tactile-visual interactions, such changes in performance have been reported for localization tasks (rubber hand illusion; Pavani, Spence, & Driver, 2000) and temporal order judgments (temporal ventriloquism, Keetels & Vroomen, 2008). Further, a compelling demonstration of perceptual interactions between vision and touch comes from the touch-induced flash illusion (Violentyev, Shimojo, & Shams, 2005). In this study, the observers were more likely to report the perception of two visual flashes when two rather than one tactile transients coincided with the presentation of one visual flash. Because the coinciding tactile information impaired the sensitivity of the participants to distinguish between trials with one or two (physical) flashes, Violentyev et al. attributed the quantitative change in the visual impression to perceptual processing.

The current project

Most studies addressing perceived bouncing have studied audio-visual stimuli. However, as interactions between objects in the real world typically occur with coinciding auditory information, the visual impression of the discs bouncing off each other might be influenced from previous exposure (rather than perception; see Rips, 2011). In order to study the impact of coinciding transients themselves, we therefore investigated if (and how) coinciding tactile transients induce bouncing impressions. Given the large differences between tactile and auditory modality in terms of cortical organization (e.g., Rauschecker & Scott, 2009) as well as perceived simultaneity (Fujisaki & Nishida, 2009), this approach is also promising with regard to tracing down the perceptual root of perceived bouncing versus streaming. Our aims were twofold. First, we aimed to demonstrate that coinciding tactile information is capable of inducing qualitative changes in the visual impression of the bouncing/streaming display (Experiment 1). Second, we aimed to show that coinciding tactile information also alters perceptual processing in a similar task in order to isolate perceptual processing from (potential) cognitive influences such as biases or inferences (Experiment 2).

Experiment 1

The effect of cross-modal information on perceived bouncing/streaming mostly has been studied with coinciding auditory information (Grove & Sakurai, 2009; Watanabe & Shimojo, 2001; Zhou, Wong, & Sekuler, 2007). In this experiment, we investigated a tactile-visuo variant of the bouncing/streaming illusion in order to study the modality-specifity of the corresponding qualitative change in the visual impression. A similar approach has been reported in the dissertation of Watanabe (2001; for reviews of his data see Shimojo & Shams, 2001; Watanabe and Shimojo, 2005). If coinciding transients in general elicit the impression of discs bouncing off each other, we should observe the typical bouncing vs. streaming illusion also with the vibrotactile transients in this experiment (thus providing an independent replication of Watanabe, 2001Footnote 1).

Methods

Participants

The final sample of Experiment 1 consisted of 24 students from the University of Trier (19–29 years old, 22 females). Data from one additional participant was replaced because this participant did not vary her responses.Footnote 2 All participants reported normal or corrected-to-normal vision.

Apparatus, stimuli, and procedure

The experiment took place in a completely darkened room (no daylight, black furniture) all sources of illumination except the stimuli were eliminated. All stimuli were programmed using the PsychoPy libraries (Pierce, 2007). The experiments were presented on a 24-in. TFT screen (60 Hz, 1,920 × 1,200 pixels) at a restricted viewing distance of 73 cm. The vibrotactile stimulation was administered by a tactor (C-2, Engineering Acoustic, Inc.) that was attached to the heel of the non-dominant hand of the participants. The tactor was controlled by a standard amplifier (Power Amplifier Module PM40C, t.amp). In order to prevent any influence of the noise of the vibrating tactors (250 Hz, 0.3 mm peak-to-peak amplitude), our participants wore headphones playing white noise (95 dB) as well as ear plugs (29 dB reduction).

Each trial began with the presentation of a central fixation cross. The participants were instructed to maintain the central fixation for the entire trial. After 300 ms, two red dots (0.5 deg in diameter) appeared 5° above the center and immediately started moving toward each other along a horizontal path at a constant speed of 10 °/s. The initial position of the discs was arranged so that they were fully overlapped above the fixation cross after 700 ms of object motion. In half of all trials, the onset of a 100-ms vibrotactile stimulation coincided with this moment of overlap between the discs. Thereafter, the discs moved apart for another interval of 700 ms (see Fig. 1). Finally, the participants indicated whether or not they perceived the discs as bouncing off each other by pressing the corresponding key on a keyboard. Each participant completed seven blocks consisting of eight trials (four per condition). The first block was considered to be practice and was removed from the analysis.

Fig. 1
figure 1

A: Illustration of the experimental task of Experiment 1. In half of all trials, a brief vibrotactile stimulation coincided with the moment of overlap between the moving discs. B: Vibrotactile stimulation was provided to the heel of the non-dominant hand of the participants

Full size image

Results

For each participant, we calculated the proportion of “bouncing” responses for the conditions with and without coinciding transients (see Fig. 2). A t-test for paired samples confirmed that the proportion of “bouncing” responses was higher when the coinciding tactile transient was present than when the coinciding transient was absent, t(23) = 3.89, p < .001, d = .79. Therefore, our results replicate an experiment of Watanabe (2001) and show that coinciding tactile information is capable of altering the visual impression.

Fig. 2
figure 2

Standard-boxplot of the results of Experiment 1. Proportion of “bouncing” responses for trials with and without coinciding tactile transients

Full size image

Experiment 2

The results of Experiment 1 show that brief vibrotactile stimulations increase the probability of a visual bouncing impression. However, this result was obtained from explicit verbal reports regarding the perceptual impression. In research on auditory-induced bouncing, Grove et al. (2012) have challenged the attribution of changes in reported bouncing/streaming to perceptual processing. When they asked participants to distinguish between trials with objectively bouncing and streaming pairs of objects (implemented by differently dense moving patches), coinciding auditory information altered the response criterion of the participants toward bouncing reports but had no effect on the sensitivity to discriminate between objective bouncing and streaming (see also Grassi & Casco, 2012). From these results, Grove et al. (2012) concluded that the coinciding tones affected decisional processes rather than perception. Recently, however, Meyerhoff and Scholl (2018) showed that coinciding tones reduce the perceived amount of overlap between the moving discs (i.e., auditory-induced illusory crescents). Because less overlap between the moving discs also comes along with a larger proportion of bouncing impressions (e.g., Grassi & Casco, 2012; Scholl & Nakayama, 2004), the shift in response bias observed by Grove et al. (2012) might in fact be deeply perceptual in nature. In order to provide direct evidence for our suggestion that coinciding vibrotactile transients alter perceptual processing of bouncing/streaming impressions, we therefore tested whether vibrotactile transients also decrease the amount of perceived overlap between the moving discs.

Methods

Participants

The final sample of Experiment 2 consisted of 21 students from the University of Trier (18–33 years old, 17 females). Data from three additional participants were removed (one equipment failure, one participant switched off the white noise during the experiment, and one participant failed to comply with the task instructions).Footnote 3

Apparatus, stimuli, and procedure

The apparatus was identical to Experiment 1. The remaining parameters were adopted from previous studies on illusory crescents (Meyerhoff & Scholl, 2018). Each trial began with the appearance of two discs (3° in diameter; red and green with random color assignment). Both discs were located 3° above the center of the screen. Whereas one disc remained stationary in the central position, the other disc started moving immediately 19.2° off-center (randomly assigned to the left and right) toward the central disc at a constant speed of 38.3 °/sFootnote 4 until the discs overlapped in the center of the screen. In half of all trials, the moment(s) of maximal overlap between the moving discs coincided with the onset of a 100-ms vibrotactile stimulation, presented to the heel of the left hand of the participants. At the moment of overlap, the first disc stopped moving whereas the second disc started moving along the same trajectory. After reaching the opposite location on the screen, the event appeared in reversed order so that the objects returned to their original locations. A full cycle of the event lasted 2,000 ms (see Fig. 3). The whole event was repeated until the participants adjusted and confirmed the overlap of two additional static discs (yellow and blue), which appeared in the lower right corner of the screen (10° to the right and 10° below the center of the screen). Initially there was no overlap between the probe discs and the participants were able to adjust the overlap of these discs in steps of 5% or 1% of the width of the discs by pressing the corresponding keys on a keyboard. The participants were instructed to adjust the overlap of the probe discs until they matched the perceived (maximal) overlap of the discs of the ongoing event in the upper half of the screen. We tested objective overlaps of 100% (i.e., full overlaps) or 80% of the width of the discs. To avoid offsets in the initial locations, the objects moved slightly slower (37.8 °/s) in the condition with 80% overlap. The adjusted overlap between the probe discs served as the dependent variable. Each participant completed seven blocks consisting of eight trials (four per condition). The first block was considered to be practice and was removed from the analysis.

Fig. 3
figure 3

A: Illustration of the illusory-crescent task. The participants adjust the overlap of the two discs in the lower right corner of the screen until it matches the overlap of the ongoing event in the center of the screen. In half of all trials, a brief vibrotactile stimulation coincides with the moment of (maximal) overlap between the discs. B: Illustration of touch-induced illusory crescents. The participants generally misperceive the overlap of the discs; however, this underestimation is more pronounced with than without coinciding tactile transients

Full size image

Results

In agreement with the suggestion that coinciding tactile transients have an impact on visual processing, our participants adjusted less overlap between the probe discs when the coinciding vibrotactile stimulation was present. Prior to the analysis, we excluded trials with a response that deviated more than 3 SD from the individual mean of the corresponding experimental condition (0.69% of all trials). Across all trials, our participants adjusted less overlap than presented objectively (see Fig. 4). We conducted a repeated measures ANOVA with the coinciding transient (none vs. vibrotactile stimulation) and the objective overlap of the discs (80% vs. 100%) as the independent variables as well as the adjusted overlap of the probe discs as the dependent variable. We observed a main effect of the coinciding transient, F(1, 20) = 12.08, p = .002, indicating that our participants adjusted less overlap between the probe discs when a brief vibrotactile stimulation coincided with the moment of maximum overlap. Further, we observed a main effect of the objective overlap, F(1, 20) = 20.73, p < .001, indicating that participants adjusted more overlap in trials with an objectively larger overlap between the moving discs. There was no interaction between the two independent variables, indicating that the touch-induced reduction in adjusted overlap is equally pronounced across both objective overlaps, F(1, 20) < 1. The effect sizes (Cohen’s d) of the touch-induced illusory crescent (i.e., the difference between trials without and with coinciding vibrotactile stimulation) were d = 0.74 for trials with full overlaps and d = 0.62 for trials with 80% overlaps.

Fig. 4
figure 4

Standard-boxplot of the results of Experiment 2. Adjusted overlap on the probe discs for objective overlaps of 80% and 100% of the width of the discs

Full size image

General discussion

The results of our experiments provide two straightforward insights into tactile-visual interactions. First, a brief vibrotactile stimulation alters the quality of the visual impression (from streaming to bouncing; Experiment 1; see also Watanabe, 2001). Second, the coinciding tactile stimulation indeed is capable of altering perceptual processing. This conclusion stems from the results of Experiment 2, in which we used an implicit dependent variable to assess perception. Whereas the open responses of Experiment 1 could be explained in terms of shifts in the response criterion or even task demands, such cognitive influences on the adjusted overlap are unlikely due to the less explicit response format as well as the continuous repetition of the display until the participants were satisfied with their response. Thus, the dependent measure in this experiment allowed for a continuous perceptual comparison between the perceived overlap in the dynamic event and the adjusted overlap of the probe discs. We argue that this likely captures perceptual processing. [Of course, we did not mention the concepts under investigation such as bouncing, streaming, or causality in the instructions of Experiment 2.]

The two experiments differ not only in the experimental task at hand, but also in surface features such as the number of simultaneously moving objects or the speed of the moving object. Nevertheless, their spatiotemporal structure is remarkably similar: In both experiments, two objects approach each other until they fully overlap and then move apart again (see also Michotte, 1963). Most importantly, however, they pose the same question for the visual system, namely, establishing object correspondence (i.e., which object went where). Given this structural similarity, we would like to interpret our results as a first insight into how the qualitative change in the visual impression might arise on a perceptual level (for converging evidence that misperceptions of the actual overlap also arise in the bouncing/streaming displays, see Grassi & Casco, 2012; Kawachi, 2016). Because previous research has demonstrated that less (objective) overlap between the discs comes along with a higher proportion of bouncing reports (Scholl & Nakayama, 2004), it seems likely that the quantitative shift in perceived overlap induced the qualitative shift in the visual impression. As such, the bouncing/streaming illusion is a bistable stimulus in which observers could see bouncing or streaming but not any transitions in between (similar to other bistable stimuli such as the Necker cube). The perceived overlap between the moving discs, however, might alter the probability of one or other perceptual impression. Of course, further research is necessary in order to fully establish the proposed connection between illusory crescents and perceived bouncing.

From coincidence to bouncing

A central challenge for the visual system is to maintain the correspondence of objects when they change their location over time. In order to solve this task, the visual system relies on information from surface features such as color and shape (Feldman & Tremoulet, 2006), occlusion cues (Remijn & Ito, 2007), or the spatial alignment (Kawabe & Miura, 2006). Our results show that the visual system also incorporates tactile signals in order to disambiguate the visual percept. Although we did not directly compare auditory and tactile transients in our study (but see Watanabe & Shimojo, 2005), our results from tactile stimulation closely match findings from audio-visual as well as purely visual experiments exploring the impact of coinciding transients on the bouncing/streaming impression (Sekuler et al., 1997; Watanabe & Shimojo, 1998) as well as illusory crescents (Meyerhoff & Scholl, 2018; Scholl & Nakayama, 2004).

One possible interpretation of these similar findings across different modalities is that the visual system might interpret any kind of coinciding information as heuristic in favor of a causal interaction, which in turn increases the likelihood for the reversal of the motion paths that are perceived as two discs bouncing off each other. From an ecological point of view, such a perceptual heuristic seems plausible as most causal interactions between objects come along with coincident sounds (i.e., colliding billiard balls) or tactile vibrations (i.e., putting a mug on the table). Our observation that tactile information disambiguates stimuli when there is no naturalistic link between the stimuli (i.e., normally, we hear but do not feel the collision of external objects) further indicates that such a heuristic would work on an abstract level rather than with concrete situations and experiences.

The influence of attentional processing

An alternative to explain tactile-induced illusory crescents as well as tactile-induced perceived bouncing comes from cross-modal attentional processing. Indeed, the cross-modal consistency of bouncing impressions as well as illusory crescents between our experiments with tactile stimulation and previous work with auditory or visual stimulation (e.g., Watanabe & Shimojo, 1998; 2005) leans toward a general mechanism (rather than modality-specific mechanisms) that alters the visual percept (see Grove, Kawachi, & Sakurai, 2012b; Watanabe, 2001; Zhou et al., 2007). A good candidate for such a cross-modal mechanism is attentional processing because previous research has identified cross-modal interactions between the tactile and the visual modality (e.g., Frings, Amendt, & Spence, 2011; van der Burg, Olivers, Bronkhorst, & Theeuwes, 2009). Within the visual domain, Watanabe and Shimojo (1998) have reported that bouncing impressions become more likely if attention is distracted from the bouncing/streaming display by coinciding visual transients or dual tasks. Therefore, it is possible that any coinciding transient distracts attention from the bouncing/streaming display. Such an attentional disruption might result in missing the central frame which in turn induces illusory crescents as well as bouncing impressions.

Conclusion

Coinciding vibrotactile information induces perceived bouncing as well as illusory crescents between overlapping discs. Our study suggests that the qualitative change in the visual impression is accompanied by a quantitative change in the perceived overlap between the moving discs (i.e., touch-induced illusory crescents). This matches remarkably well with previous findings from studies with coinciding auditory information which in turn supports the idea that perceived bouncing stems from a modality-independent mechanism such as heuristical processing, or the distraction of attention, or a combination of both.