Abstract
Events are often perceived in multiple modalities. The co-occurring proximal visual and auditory stimuli events are mostly also causally linked to the distal event, which makes it difficult to evaluate whether learned correlation or perceived causation guides binding in multisensory perception. Piano tones are an interesting exception: They are associated with the act of the pianist striking keys, an event that is visible to the perceiver, but directly results from hammers hitting strings, an event that typically is not visible to the perceiver. We examined the influence of seeing the hammer or the keystroke on auditory temporal order judgments (TOJs). Participants judged the temporal order of a dog bark and a piano tone, while seeing the piano stroke shifted temporally relative to its audio signal. Visual lead increased “piano-first” responses in auditory TOJ, but more so if the associated keystroke was visible than if the sound-producing hammer was visible, even though both were equally visually salient. This provides evidence for a learning account of audiovisual perception.
Article PDF
References
Diehl, R. L., & Kluender, K. R., (1989). On the objects of speech perception. Ecological Psychology, 1, 121–144.
Dixon, P., (2008). Models of accuracy in repeated-measures design. Journal of Memory & Language, 59, 447–456. doi:10.1016/j.jml.2007.11.004
Fowler, C. A., (1996). Listeners do hear sounds, not tongues. Journal of the Acoustical Society of America, 99,1730–17411.
Fowler, C. A., & Dekle, D. J., (1991). Listening with eye and hand: Cross-modal contributions to speech perception. Journal of Experimental Psychology: Human Perception & Performance, 17, 816–828.
Frost, R., & Ziegler, J. C., (2007). Speech and spelling interaction: The interdependence of visual and auditory word recognition. In M. G. Gaskell (Ed.), The Oxford handbook of psycholinguistics (pp. 107–118). Oxford: Oxford University Press.
Gaver, W. W., (1993). What in the world do we hear? An ecological approach to auditory event perception. Ecological Psychology, 5, 1–29.
Gibson, J. J., (1979). The ecological approach to visual perception. Boston: Houghton Mifflin.
Goswami, U., Thomson, J., Richardson, U., Stainthorp, R., Hughes, D., Rosen, S., & Scott, S. K., (2002). Amplitude envelope onsets and developmental dyslexia: A new hypothesis. Proceedings of the National Academy of Sciences, 99, 10911–10916. doi:10.1073/pnas.122368599
Hasegawa, T., Matsuki, K.-I., Ueno, T., Maeda, Y., Matsue, Y., Konishi, Y., & Sadato, N., (2004). Learned audio-visual crossmodal associations in observed piano playing activate the left planum temporale: An fMRI study. Cognitive Brain Research, 20, 510–518. doi:10.1016/j.cogbrainres.2004.04.005
Massaro, D. W., (1998). Perceiving talking faces: From speech perception to a behavioral principle. Cambridge, MA: MIT Press.
McGurk, H., & MacDonald, J., (1976). Hearing lips and seeing voices. Nature, 264, 746–748.
Rosenblum, L. D., Schmuckler, M. A., & Johnson, J. A., (1997). The McGurk effect in infants. Perception & Psychophysics, 59, 347–357.
Saldaña, H. M., & Rosenblum, L. D., (1993). Visual influences on auditory pluck and bow judgments. Perception & Psychophysics, 54, 406–416.
Schutz, M., & Lipscomb, S., (2007). Hearing gestures, seeing music: Vision influences perceived tone duration. Perception, 36, 888–897.
Scott, S. K., (1998). The point of P-centres. Psychological Research, 61, 4–11. doi:10.1007/PL00008162
Soto-Faraco, S., & Alsius, A., (2009). Deconstructing the McGurk-MacDonald illusion. Journal of Experimental Psychology: Human Perception & Performance, 35, 580–587.
Sumby, W. H., & Pollack, I., (1954). Visual contribution to speech intelligibility in noise. Journal of the Acoustical Society of America, 26, 212–215.
Summerfield, A. Q., & McGrath, M., (1984). Detection and resolution of audio-visual incompatibility in the perception of vowels. Quarterly Journal of Experimental Psychology, 36A, 51–74.
Vatakis, A., & Spence, C., (2007). Crossmodal binding: Evaluating the “unity assumption” using audiovisual speech stimuli. Perception & Psychophysics, 69, 744–756.
Vatakis, A., & Spence, C., (2008). Evaluating the influence of the “unity assumption” on the temporal perception of realistic audiovisual stimuli. Acta Psychologica, 127, 12–23.
Author information
Authors and Affiliations
Corresponding author
Additional information
This research was supported in part by an Innovational Research Incentive Scheme Veni Grant awarded to A.J. by the Netherlands Organization for Scientific Research (NWO).
Rights and permissions
About this article
Cite this article
Mitterer, H., Jesse, A. Correlation versus causation in multisensory perception. Psychonomic Bulletin & Review 17, 329–334 (2010). https://doi.org/10.3758/PBR.17.3.329
Received:
Accepted:
Issue Date:
DOI: https://doi.org/10.3758/PBR.17.3.329