Skip to main content

Effects of seeing and hearing speech on speech production: a response time study

Experimental Brain Research volume 195, Article number: 175 (2009) Cite this article

Abstract

Research demonstrates that listening to and viewing speech excites tongue and lip motor areas involved in speech production. This perceptual-motor relationship was investigated behaviourally by presenting video clips of a speaker producing vowel-consonant-vowel syllables in three conditions: visual-only, audio-only, and audiovisual. Participants identified target letters that were flashed over the mouth during the video, either manually or verbally as quickly as possible. Verbal responses were fastest when the target matched the speech stimuli in all modality conditions, yet optimal facilitation was observed when participants were presented with visual-only stimuli. Critically, no such facilitation occurred when participants were asked to identify the target manually. Our findings support previous research suggesting a close relationship between speech perception and production by demonstrating that viewing speech can ‘prime’ our motor system for subsequent speech production.

This is a preview of subscription content, access via your institution.

Access options

Buy single article

Instant access to the full article PDF.

43,34 €

Price includes VAT (Finland)
Tax calculation will be finalised during checkout.

Fig. 1
Fig. 2

Notes

  1. Note, this is a very simplified time course for speech processing, and audiovisual speech perception is indeed much more complex (see Munhall and Tohkura 1998)

References

  • Alsius A, Navarra J, Campbell R et al (2005) Audiovisual integration of speech falters under high attentional demands. Curr Biol 15:839–843

    PubMed  Article  CAS  Google Scholar 

  • Anderson TS, Tiipana K, Laarni J, Kojo I, Sams M (2009) The role of visual spatial attention in audiovisual speech perception. Speech Commun 51:184–193

    Article  Google Scholar 

  • Biccino G, Solodkin A, Small SL (2006) Functions of the mirror neuron system: Implications for neurorehabilitation. Cogn Behav Neurol 19:56–63

    Google Scholar 

  • Binkofski F, Buccino G (2004) Motor functions of the Broca’s region. Brain Lang 89:362–369

    PubMed  Article  Google Scholar 

  • Brass M, Bekkering H, Wohlschlager A, Prinz W (2000) Compatibility between observed and executed finger movements: comparing symbolic, spatial, and imitative cues. Brain Cogn 44:124–143

    PubMed  Article  CAS  Google Scholar 

  • Brass M, Bekkering H, Prinz W (2001) Movement observation affects movement execution in a simple response task. Acta Psychol 106:3–22

    Article  CAS  Google Scholar 

  • Calvert GA, Campbell R (2003) Reading speech from still and moving faces: The neural substrates of visible speech. J Cogn Neurosci 15:57–70

    PubMed  Article  Google Scholar 

  • Di Pellegrino G, Fadiga L, Fogassi L, Gallese V, Rizzolatti G (1992) Understanding motor events: a neurophysiological study. Exp Brain Res 91:176–180

    PubMed  CAS  Google Scholar 

  • Fadiga L, Craighero L, Buccino G et al (2002) Speech listening specifically modulates the excitability of tongue muscles: a TMS study. Eur J Neurosci 15:399–402

    PubMed  Article  Google Scholar 

  • Ferrari PF, Gallese V, Rizzolatti G, Fogassi L (2003) Mirror neurons responding to the observation of ingestive and communicative mouth actions in the monkey ventral premotor cortex. Eur J Neurosci 17:1703–1714

    PubMed  Article  Google Scholar 

  • Fowler CA (2004) Speech as a supramodal or amodal phenomenon. In: Calvert GA, Spense C, Stein BE (eds) The handbook of multisensory processes. MIT Press, Cambridge

    Google Scholar 

  • Heyes C, Bird G, Johnson H, Haggard P (2005) Experience modulates automatic imitation. Cogn Brain Res 22:233–240

    Article  Google Scholar 

  • Hickok G, Poeppel D (2004) Dorsal and ventral streams: a framework for understanding aspects of the functional anatomy of language. Cognition 92:67–99

    PubMed  Article  Google Scholar 

  • Jarick M, Jones JA (2008) Observation of static gestures influences speech perception. Exp Brain Res 189:221–228

    PubMed  Article  Google Scholar 

  • Kerzel D (2002) Evidence for effects of phonological correspondence between visible speech and written syllables. Psychol Res 66:195–200

    PubMed  Article  Google Scholar 

  • Kerzel D, Bekkering H (2000) Motor activation from visible speech: evidence from stimulus response compatibility. J Exp Psychol Hum Percept Perform 26:634–647

    PubMed  Article  CAS  Google Scholar 

  • Kohler E, Keysers C, Umilta MA, Fogassi L, Gallese V, Rizzolatti G (2002) Hearing sounds, understanding actions: action representation in mirror neurons. Science 297:846–848

    PubMed  Article  CAS  Google Scholar 

  • Lahav A, Saltzman E, Schlaug G (2007) Action representation of sound: audiomotor recognition network while listening to newly acquired actions. J Neurosci 27:308–314

    PubMed  Article  CAS  Google Scholar 

  • Leslie KR, Johnson-Frey SH, Grafton S (2004) Functional imaging of face and hand imitation: towards a motor theory of empathy. Neuroimage 21:601–607

    PubMed  Article  Google Scholar 

  • Liberman AM, Mattingly I (1985) The motor theory revised. Cognition 21:1–36

    PubMed  Article  CAS  Google Scholar 

  • McGurk H, MacDonald J (1976) Hearing lips and seeing voices. Nature 264:746–748

    PubMed  Article  CAS  Google Scholar 

  • Mottonen R, Krause CM, Tiippana et al (2002) Processing of changes in visual speech in the human auditory cortex. Brain Res Cogn Brain Res 13:417–425

  • Munhall KG, Tohkura Y (1998) Audiovisual gating and the time course of speech perception. J Acoust Soc Am 104:530–539

    PubMed  Article  CAS  Google Scholar 

  • Nishitani N, Hari R (2002) Viewing lips forms: cortical dynamics. Neuron 36:1211–1220

    PubMed  Article  CAS  Google Scholar 

  • Petrides M, Pandya DN (1999) Dorsolateral prefrontal cortex: comparative cytoarchitectonic analysis in the human and the macaque brain and corticocortical connection patterns. Eur J Neurosci 11:1011–1136

    PubMed  Article  CAS  Google Scholar 

  • Pulvermuller F, Huss M, Kherif F et al (2006) Motor cortex maps articulatory features of speech sounds. Proc Natl Acad Sci USA 103:7865–7870

    PubMed  Article  CAS  Google Scholar 

  • Rizzolatti G, Arbib MA (1998) Language within our grasp. Trends Neurosci 21:188–194

    PubMed  Article  CAS  Google Scholar 

  • Rizzolatti G, Craighero L (2004) The mirror-neuron system. Annu Rev Neurosci 27:169–192

    PubMed  Article  CAS  Google Scholar 

  • Rizzolatti G, Fadiga L, Gallese V et al (1996) Premotor cortex and the recognition of motor actions. Brain Res Cogn Brain Res 3:131–141

    PubMed  Article  CAS  Google Scholar 

  • Skipper JI, Nusbaum HC, Small SL (2005) Listening to talking faces: motor cortical activation during speech production. Neuroimage 25:76–89

    PubMed  Article  Google Scholar 

  • Soto-Faraco S, Navarra J, Alsius A (2004) Assessing automaticity in audiovisual speech integration: evidence from the speeded classification task. Cognition 92:B13–B23

    PubMed  Article  Google Scholar 

  • Sturmer B, Aschersleben G, Prinz W (2000) Correspondence effects with manual gestures and postures: a study of imitation. J Exp Psychol Hum Percept Perform 26:1746–1759

    PubMed  Article  CAS  Google Scholar 

  • Summerfield Q (1987) Some preliminaries to a comprehensive account of audio-visual speech perception. In: Dodd B, Campbell R (eds) Hearing by eye: the psychology of lip-reading. Lawrence Erlbaum Associates, London

    Google Scholar 

  • Sundara M, Namasivayam AK, Chen R (2001) Observation-execution matching system for speech: a magnetic stimulation study. Neuroreport 12:1341–1344

    PubMed  Article  CAS  Google Scholar 

  • Tiippana K, Anderson TS, Sams M (2004) Visual attention modulates audiovisual speech perception. Eur J Cogn Psychol 16:457–472

    Article  Google Scholar 

  • Van Strien JW (1988) Handedness and hemispheric laterality. Dissertation, Vrije Universiteit, Amsterdam

  • Watkins K, Paus T (2004) Modulation of motor excitability during speech perception: the role of Broca’s area. J Cogn Neurosci 16:978–987

    PubMed  Article  Google Scholar 

  • Watkins K, Strafella AP, Paus T (2003) Seeing and hearing speech excites the motor system involved in speech production. Neuropsychologia 41:989–994

    PubMed  Article  CAS  Google Scholar 

  • Wilson SM, Saygin AP, Sereno MI, Iacoboni M (2004) Listening to speech activates motor areas involved in speech production. Nat Neurosci 7:701–702

    PubMed  Article  CAS  Google Scholar 

  • Wohlschlager A, Bekkering H (2002) Is human imitation based on a mirror-neurone system? Some behavioural evidence. Exp Brain Res 143:341–355

    Article  Google Scholar 

Download references

Acknowledgements

This research was funded by a Natural Sciences and Engineering Research Council of Canada (NSERC) Discovery grant and Research Fellowship from Wilfrid Laurier University awarded to Jeffery A. Jones.

Author information

Affiliations

  1. Department of Psychology, Wilfrid Laurier University, Waterloo, ON, N2L 3C5, Canada

    Michelle Jarick & Jeffery A. Jones

  2. Centre for Cognitive Neuroscience, Wilfrid Laurier University, Waterloo, ON, N2L 3C5, Canada

    Jeffery A. Jones

Authors
  1. Michelle Jarick
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Jeffery A. Jones
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Jeffery A. Jones.

Rights and permissions

Reprints and Permissions

About this article

Cite this article

Jarick, M., Jones, J.A. Effects of seeing and hearing speech on speech production: a response time study. Exp Brain Res 195, 175 (2009). https://doi.org/10.1007/s00221-009-1765-x

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s00221-009-1765-x

Keywords

  • Mirror neurons
  • Speech perception
  • Speech production
  • Motor theory
  • Perceptual-motor
  • Audiovisual speech