Abstract
In the McGurk effect, visual information specifying a speaker’s articulatory movements can influence auditory judgments of speech. In the present study, we attempted to find an analogue of the McGurk effect by using nonspeech stimuli—the discrepant audiovisual tokens of plucks and bows on a cello. The results of an initial experiment revealed that subjects’ auditory judgments were influenced significantly by the visual pluck and bow stimuli. However, a second experiment in which speech syllables were used demonstrated that the visual influence on consonants was significantly greater than the visual influence observed for pluck-bow stimuli. This result could be interpreted to suggest that the nonspeech visual influence was not a true McGurk effect. In a third experiment, visual stimuli consisting of the wordspluck andbow were found to have no influence over auditory pluck and bow judgments. This result could suggest that the nonspeech effects found in Experiment 1 were based on the audio and visual information’s having an ostensive lawful relation to the specified event. These results are discussed in terms of motor-theory, ecological, and FLMP approaches to speech perception.
Article PDF
Similar content being viewed by others
Avoid common mistakes on your manuscript.
References
Ades, A. E. (1977). Vowels, consonants, speech and non-speech.Psychological Review,84, 524–530.
Bermant, R. I., &Welch, R. B. (1976). The effect of degree of visual-auditory stimulus separation and eye position upon the spatial inter-action of vision and audition.Perception & Motor Skills,43, 487–493.
Choe, C. S., Welch, R. B., Guilford, R. M., &Juola, J. F. (1975). The “ventriloquist effect”: Visual dominance or response bias?Perception & Psychophysics,18, 55–60.
Cohen, M. M., & Massaro, D. W. (1993).Perceiving visual and auditory information in consonant-vowel and vowel syllables. Manuscript submitted for publication.
Cutting, J. E. (1982). Plucks and bows are categorically perceived, sometimes.Perception & Psychophysics,31, 462–476.
Cutting, J. E., &Rosner, B. S. (1974). Categories and boundaries in speech and music.Perception & Psychophysics,16, 564–570.
Cutting, J. E., Rosner, B. S., &Foard, C. F. (1976). Perceptual categories for musiclike sounds: Implications for theories of speech perception.Quarterly Journal of Experimental Psychology,28, 361–378.
Easton, R. D., &Moran, P. W. A. (1978). A quantitative confirmation of visual capture of curvature: Implications for a visual dominance phenomenon.Journal of General Psychology,98, 105–112.
Fowler, C. A. (1986). An event approach to the study of speech perception from a direct realist perspective.Journal of Phonetics,14, 3–28.
Fowler, C. A., &Dekle, D. J. (1991). Listening with eye and hand: Cross-modal contributions to speech perception.Journal of Experimental Psychology: Human Perception & Performance,17, 816–828.
Fowler, C. A., &Rosenblum, L. D. (1991). Perception of the phonetic gesture. In I. G. Mattingly & M. Studdert-Kennedy (Eds.),Modularity and the motor theory (pp. 33–59). Hillsdale, NJ: Erlbaum.
Gibson, J. J. (1979).The senses considered as perceptual systems. Boston: Houghton Mifflin.
Kewley-Port, D., Watson, C. S., &Foyle, D. C. (1988). Auditory temporal acuity in relation to category boundaries: Speech and non-speech stimuli.Journal of the Acoustical Society of America,83, 1133–1145.
Kinney, J. A. S., &Luria, S. M. (1970). Conflicting visual and tactualkinesthetic stimulation.Perception & Psychophysics,8, 189–192.
Ladefoged, P., Harshman, R. Goldstein, L., &Rice, B. (1978). Generating vocal tract shapes from formant frequencies.Journal of the Acoustical Society of America,64, 1027–1035.
Liberman, A. M., Cooper, F. S., Shankweiler, D. P., &Studdert-Kennedy, M. (1967). Perception of the speech code.Psychological Review,74, 431–461.
Liberman, A. M., Harris, K. S., Hoffman, H. S., &Griffith, B. C. (1957). The discrimination of the speech sound within and across phoneme boundaries.Journal of Esperimental Psychology,54, 358–368.
Liberman, A. M., &Mattingly, I. G. (1985). The motor theory of speech perception revised.Cognition,21, 1–36.
MacDonald, J., &McGurk, H. (1978). Visual influences on speech perception processes.Perception & Psychophysics,24, 253–257.
Massaro, D. W. (1987).Speech perception by ear and eye: A paradigm for psychological inquiry. Hillsdale, NJ: Erlbaum.
Massaro, D. W., &Cohen, M. M. (1983). Evaluation and integration of visual and auditory information in speech perception.Journal of Experimental Psychology: Human Perception & Performance,9, 753–771.
Massaro, D. W., Cohen, M. M., &Thompson, L. A. (1988). Visible language in speech perception: Lipreading and reading. In R. Campbell (Ed.),Visible language (Vol. 22, pp. 8–31). Providence, RI: Rhode Island School of Design.
Mattingly, I. G., &Liberman, A. M. (1990). Speech and other auditory modules. In G. M. Edelman, W. E. Gall, & W. M. Cowan (Eds.),Signal and sense: Local and global order in perceptual maps (pp. 501–520). New York: Wiley.
McGrath, M., &Summerfield, A. Q. (1985). Intermodal timing relations and audio-visual speech recognition by normal-hearing adults.Journal of the Acoustical Society of America,77, 678–685.
McGurk, H., &MacDonald, J. W. (1976). Hearing lips and seeing voices.Nature,264, 746–748.
Michaels, C. F., &Carello, C. (1981).Direct perception. Engle-wood Cliffs, NJ: Prentice-Hall.
O’Leary, A., &Rhodes, G. (1984). Cross-modal effects on visual and auditory object perception.Perception & Psychophysics,35, 565–569.
Pisoni, D. B. (1973). Auditory and phonetic memory codes in the discrimination of consonants and vowels.Perception & Psychophysics,13, 253–260.
Repp, B. H. (1984). Categorical perception: Issues, methods, findings. In N. J. Lass (Ed.),Speech and language: Advances in basic research and practice (Vol. 10, pp. 2–36). New York: Academic Press.
Repp, B. H., Manuel, S. Y., Liberman, A. M., & Studdert-Kennedy, M. (1983).Exploring the “McGurk effect.” Paper presented at the meeting of the Psychonomic Society, San Diego.
Rosen, S. M., &Howell, P. (1981). Plucks and bows are not categorically perceived.Perception & Psychophysics,30, 156–168.
Rosenblum, L. D. (1989).Effort perception of speech and nonspeech events: An audio-visual investigation. Unpublished doctoral dissertation, University of Connecticut.
Rosenblum, L. D., &Fowler, C. A. (1991). Audiovisual investigation of the loudness-effort effect for speech and nonspeech events.Journal of Experimental Psychology: Human Perception & Performance,17, 976–985.
Rosenblum, L. D., &Saldara, H. M. (1992). Discrimination tests of visually influenced syllables.Perception & Psychophysics,52, 461–473.
Schouten, M. E. H. (1992). Modeling phoneme perception I: Categorical perception.Journal of the Acoustical Society of America,92, 1841–1855.
Singer, G., &Day, R. H. (1969). Visual capture of haptically judged depth.Perception & Psychophysics,5, 315–316.
Smurzyński, J. (1985). Noncategorical identification of rise time.Perception & Psychophysics,38, 540–542.
Studdert-Kennedy, M. (1976). Speech perception. In N. J. Lass (Ed.),Contemporary issues in experimental phonetics (pp. 243–293). New York: Academic Press.
Summerfield, A. Q. (1979). Use of visual information in phonetic perception.Phonetica,36, 314–331.
Summerfield, A. Q., &McGrath, M. (1984). Detection and resolution of audio-visual incompatibility in the perception of vowels.Quarterly Journal of Experimental Psychology,36A, 51–74.
Walker, J. T., &Scott, K. J. (1981). Auditory-visual conflicts in the perceived duration of lights, tones, and gaps.Journal of Experimental Psychology: Human Perception & Performance,7, 1327–1339.
Watson, C. S., &Kewley-Port, D. (1988). Some remarks on Pastore.Journal of the Acoustical Society of America,84, 2266–2270.
Welch, R. B., DuttonHurt, L. D., &Warren, D. H. (1986). Contributions of audition and vision to temporal rate perception.Perception & Psychophysics,39, 294–300.
Author information
Authors and Affiliations
Additional information
This research was supported by an intramural grant from the University of California.
Rights and permissions
About this article
Cite this article
Saldaña, H.M., Rosenblum, L.D. Visual influences on auditory pluck and bow judgments. Perception & Psychophysics 54, 406–416 (1993). https://doi.org/10.3758/BF03205276
Received:
Accepted:
Issue Date:
DOI: https://doi.org/10.3758/BF03205276