Abstract
We investigated the perceptual processing of facial information and vocal information to test the nature of the multisensory integration process. Single-syllable words were presented in background noise at one of eight stimulus durations ranging from 45 to 80% of the total word duration. Performance was evaluated relative to a fuzzy logical model of perception (FLMP) and an additive model (ADD). These two models differ in the nature of the integration process: optimal multiplicative integration in the FLMP and additive in the ADD. The FLMP provided a significantly better description of the word identifications relative to the ADD. Consistent with the outcomes of several other tasks in audiovisual speech perception, the FLMP also accurately describes the continuous uptake of information during word perception.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Anastasio TJ, Patton PE, Belkacem-Boussaid K (2000) Using Bayes’ rule to model multisensory enhancement in the superior colliculus. Neural Comp 12:1165–1187
Bernstein LE, Eberhardt SP (1986) Johns Hopkins lipreading corpus video-disk set. The Johns Hopkins University Press, Baltimore
Chandler JP (1969) Finds local minima of a smooth function of several parameters. Behav Sci 14:81–82
Cotton S, Grosjean F (1984) The gating paradigm: a comparison of successive and individual presentation formats. Percept Psychophys 35:41–48
Cutting JE, Bruno N, Brady NP, Moore C (1992) Selectivity, scope, and simplicity of models: a lesson from fitting judgments of perceived depth. J Exp Psychol 121:362–381
Grant KW, Walden BE (1996) Evaluating the articulation index for auditory–visual consonant recognition. J Acoust Soc Am 100:2415–2424
Green KP, Kuhl PK (1989) The role of visual information in the processing of place and manner features in speech perception. Percept Psychophys 45:34–42
Grosjean F (1980) Spoken word recognition processes and the gating paradigm. Percept Psychophys 28:267–283
Grosjean F (1997) Gating. In: Grosjean F, Frauenfelder UH (eds) A guide to spoken word recognition paradigms. Psychology, Sussex
Luce RD (1959) Individual choice behavior. Wiley, New York
Marslen-Wilson WD, Warren P (1994) Levels of perceptual representation and process in lexical access: words, phonemes, and features. Psychol Rev 101:653–675
Massaro DW (1975) Understanding language: an information processing analysis of speech perception, reading, and psycholinguistics. Academic, New York
Massaro DW (1987) Speech perception by ear and eye: a paradigm for psychological inquiry. Lawrence Erlbaum Associates, Hillsdale
Massaro DW (1993) Broadening the domain of the fuzzy logical model of perception. In: Pick HL Jr, Van Den Broek P, Knill DC (eds) Cognition: conceptual and methodological issues. APA, Washington, D.C., pp 51–84
Massaro DW (1998) Perceiving talking faces: from speech perception to a behavioral principle. MIT Press, Cambridge
Massaro DW (2002) Multimodal speech perception: a paradigm for speech science. In: Granstrom B, House D, Karlsson I (eds) Multilmodality in language and speech systems. Kluwer Academic, Dordrecht, The Netherlands, pp 45–71
Massaro DW, Friedman D (1990) Models of integration given multiple sources of information. Psychol Rev 97:225–252
Massaro DW, Weldon MS, Kitzis SN (1991) Integration of orthographic and semantic information in memory retrieval. J Exp Psychol Learn Mem Cogn 17:277–287
McQueen JM, Norris D, Cutler A (1999) Lexical influence in phonetic decision making: evidence from subcategorical mismatches. J Exp Psychol Hum Percept Perform 25:1363–1389
McGurk H, MacDonald J (1976) Hearing lips and seeing voices. Nature 264:744–748
Miller GA, Nicely P (1955) An analysis of perception confusions among some English consonants. J Acoust Soc Am 27:338–352
Oden GC, Massaro DW (1978) Integration of featural information in speech perception. Psychol Rev 85:172–191
Salasoo A, Pisoni DB (1985) Interaction of information in spoken word identification. J Mem Lang 24:210–231
Smeele PT (1994) Perceiving speech: integrating auditory and visual speech. Unpublished doctoral dissertation, Delft University of Technology
Smits R, Warner N, McQueen JM, Cutler A (2003) Unfolding of phonetic information over time: a database of Dutch diphone perception. J Acoust Soc Am 113:563–574
Sumby WH, Pollack I (1954) Visual contribution to speech intelligibility in noise. J Acoust Soc Am 26:212–215
Summerfield AQ (1979) Use of visual information in phonetic perception. Phonetica 36:314–331
Teigland AD, Wilson WR (1982) Visual backward masking of selected visemes. J Speech Hear Res 25:269–274
Tyler LK (1985) The structure of the initial cohort: evidence from gating. Percept Psychophys 36:417–427
Walley A, Michela V, Wood D (1995) The gating paradigm: effects of presentation format on spoken word recognition by children and adults. Percept Psychophys 57:343–351
Warren P, Marslen-Wilson WD (1987) Continuous uptake of acoustic cues in spoken word recognition. Percept Psychophys 41:262–275
Warren P, Marslen-Wilson WD (1988) Cues to lexical choice: discriminating place and voice. Percept Psychophys 43:21–30
Weldon MS, Massaro DW (1996) Integration of orthographic, conceptual, and episodic information on implicit and explicit tests. Can J Exp Psychol 50:72–85
Acknowledgements
This work was supported in part by grants from the National Science Foundation (NSF CHALLENGE Grant CDA-9726363 and NSF Grant BCS-9905176), a grant from the Public Health Service (Grant PHS R01 DC00236), cooperative grants from the Intel Corporation and the University of California Digital Media Program (D97-04), and grants from the University of California, Santa Cruz. The authors thank Alexandra Jesse for a critical reading of the manuscript.
Author information
Authors and Affiliations
Corresponding author
Additional information
Edited by: Marie-Hélène Giard and Mark Wallace
Rights and permissions
About this article
Cite this article
de la Vaux, S.K., Massaro, D.W. Audiovisual speech gating: examining information and information processing. Cogn Process 5, 106–112 (2004). https://doi.org/10.1007/s10339-004-0014-2
Received:
Revised:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10339-004-0014-2