Abstract
The “McGurk effect” demonstrates that visual (lip-read) information is used during speech perception even when it is discrepant with auditory information. While this has been established as a robust effect in subjects from Western cultures, our own earlier results had suggested that Japanese subjects use visual information much less than American subjects do (Sekiyama & Tohkura, 1993). The present study examined whether Chinese subjects would also show a reduced McGurk effect due to their cultural similarities with the Japanese. The subjects were 14 native speakers of Chinese living in Japan. Stimuli consisted of 10 syllables (/ba/, /pa/, /ma/, /wa/, /da/, /ta/, /na/, /ga/, /ka/, /ra/ ) pronounced by two speakers, one Japanese and one American. Each auditory syllable was dubbed onto every visual syllable within one speaker, resulting in 100 audiovisual stimuli in each language. The subjects’ main task was to report what they thought they had heard while looking at and listening to the speaker while the stimuli were being uttered. Compared with previous results obtained with American subjects, the Chinese subjects showed a weaker McGurk effect. The results also showed that the magnitude of the McGurk effect depends on the length of time the Chinese subjects had lived in Japan. Factors that foster and alter the Chinese subjects’ reliance on auditory information are discussed.
Article PDF
Similar content being viewed by others
References
Binie, C. A., Montgomery, A. A., &Jackson, P. L. (1974). Auditory and visual contributions to the perception of consonants.Journal of Speech & Hearing Research,17, 619–630.
Dekle, D. J., Fowler, C. A., &Funnell, M. G. (1992). Audiovisual integration in perception of real words.Perception & Psychophysics,51, 355–362.
Fuster-Duran, A. (1996). Perception of conflicting audio-visual speech: An examination across Spanish and German. In D. G. Stork & M. E. Hennecke (Eds.),Speechreading by humans and machines (pp. 135–143). Berlin: Springer-Verlag.
Green, K. P., Kuhl, P. K., Meltzoff, A. N., &Stevens, E. B. (1991). Integrating speech information across speakers, gender, and sensory modality: Female faces and male voices in the McGurk effect.Perception & Psychophysics,50, 524–536.
MacDonald, J., &McGurk, H. (1978). Visual influences on speech perception processes.Perception & Psychophysics,24, 253–257.
Massaro, D. W., &Cohen, M. M. (1983). Evaluation and integration of visual and auditory information in speech perception.Journal of Experimental Psychology: Human Perception & Performance,9, 753–771.
Massaro, D. W., Cohen, M. M., &Smeele, P. M. (1995). Crosslinguistic comparisons in the integration of visual and auditory speech.Memory & Cognition,23, 113–131.
Massaro, D. W., Tsuzaki, M., Cohen, M. M., Gesi, A., &Heredia, R. (1993). Bimodal speech perception: An examination across languages.Journal of Phonetics,21, 445–478.
McGurk, H., &MacDonald, J. (1976). Hearing lips and seeing voices.Nature,264, 746–748.
Nomura, M. (1984).Body language wo yomu [Understanding body language]. Tokyo: Heibonsha.
Rosenblum, L. D., &Saldaña, H. M. (1992). Discrimination tests of visually influenced syllables.Perception & Psychophysics,52, 461–473.
Sekiyama, K. (1994a). Differences in auditory-visual speech perception between Japanese and Americans: McGurk effect as a function of incompatibility.Journal of the Acoustical Society of Japan (E),15, 143–158.
Sekiyama, K. (1994b). McGurk effect and incompatibility: A crosslanguage study on auditory-visual speech perception.Studies and Essays in Behavioral Sciences & Philosophy (Kanazawa University),14, 29–62.
Sekiyama, K., Braida, L. D., Nishino, K., Hayashi, M., &Tuyo, M. (1995). The McGurk effect in Japanese and American perceivers. In K. Elenius & P. Branderud (Eds.),Proceedings of the 13th International Congress of Phonetic Sciences (Vol. 3, pp. 214–217). Stockholm: Congress organizers at Kungliga Tekniska Högskolan and Stockholm University.
Sekiyama, K., Joe, K., & Umeda, M. (1988). Perceptual components of Japanese syllables in lipreading: A multidimensional study.Technical Report of IEICE (The Institute of Electronics, Information, and Communication Engineers of Japan), IE87-127. [In Japanese with English abstract]
Sekiyama, K., &Tohkura, Y. (1991). McGurk effect in non-English listeners: Few visual effects for Japanese subjects hearing Japanese syllables of high auditory intelligibility.Journal of the Acoustical Society of America,90, 1797–1805.
Sekiyama, K., &Tohkura, Y. (1993). Inter-language differences in the influence of visual cues in speech perception.Journal of Phonetics,21, 427–444.
Sumby, W. H., &Pollack, I. (1954). Visual contribution to speech intelligibility in noise.Journal of the Acoustical Society of America,26, 212–215.
Walden, B. E., Prosek, R. A., Montgomery, A. A., Scherr, C. K., &Jones, C. J. (1977). Effects of training on the visual recognition of consonants.Journal of Speech & Hearing Research,20, 130–145.
Author information
Authors and Affiliations
Corresponding author
Additional information
This study was supported by a long-term fellowship from the International Human Frontier Science Program (LT-277/93), a grant-in-aid for scientific research from the Japanese Ministry of Education, Science, and Culture (04851019), and a Sasakawa scientific research grant from the Japan Science Society (4-053).
Rights and permissions
About this article
Cite this article
Sekiyama, K. Cultural and linguistic factors in audiovisual speech processing: The McGurk effect in Chinese subjects. Perception & Psychophysics 59, 73–80 (1997). https://doi.org/10.3758/BF03206849
Received:
Accepted:
Issue Date:
DOI: https://doi.org/10.3758/BF03206849