Skip to main content
SpringerLink
Log in

Silent articulation modulates auditory and audiovisual speech perception

  • Research Article
  • Published:
Experimental Brain Research Aims and scope Submit manuscript

Abstract

The concept of an internal forward model that internally simulates the sensory consequences of an action is a central idea in speech motor control. Consistent with this hypothesis, silent articulation has been shown to modulate activity of the auditory cortex and to improve the auditory identification of concordant speech sounds, when embedded in white noise. In the present study, we replicated and extended this behavioral finding by showing that silently articulating a syllable in synchrony with the presentation of a concordant auditory and/or visually ambiguous speech stimulus improves its identification. Our results further demonstrate that, even in the case of perfect perceptual identification, concurrent mouthing of a syllable speeds up the perceptual processing of a concordant speech stimulus. These results reflect multisensory-motor interactions during speech perception and provide new behavioral arguments for internally generated sensory predictions during silent speech production.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3

Similar content being viewed by others

References

  • Abbs JH, Gracco VL (1984) Control of complex motor gestures: orofacial muscle responses to load perturbations of lip during speech. J Neurophysiol 51:705–723

    PubMed  CAS  Google Scholar 

  • Alho J, Sato M, Sams M, Schwartz J-L, Tiitinen H, Jääskeläinen IP (2012) Enhanced early-latency electromagnetic activity in the left premotor cortex is associated with successful phonetic categorization. Neuroimage 60:1937–1946

    Article  PubMed  Google Scholar 

  • Benoît C, Mohamadi T, Kandel SD (1994) Effects on phonetic context on audio-visual intelligibility of French. J Speech Lang Hear Res 37:1195–1203

    Google Scholar 

  • Binder JR, Liebenthal E, Possing ET, Medler DA, DouglasWard B (2004) Neural correlates of sensory and decision processes in auditory object identification. Nat Neurosci 7:295–301

    Article  PubMed  CAS  Google Scholar 

  • Burnett TA, Freedland MB, Larson CR, Hain TC (1998) Voice F0 responses to manipulations in pitch feedback. J Acoust Soc Am 103:3153–3161

    Article  PubMed  CAS  Google Scholar 

  • Callan D, Callan A, Gamez M, Sato MA, Kawato M (2010) Premotor cortex mediates perceptual performance. Neuroimage 51:844–858

    Article  PubMed  Google Scholar 

  • Campbell CS, Massaro DW (1997) Perception of visible speech: influence of spatial quantization. Perception 26:627–644

    Article  PubMed  CAS  Google Scholar 

  • Cathiard M-A, Schwartz J-L, Abry C (2001) Asking a naive question to the McGurk effect: why does audio [b] give more [d] percepts with usual [g] than with visual [d]? In: Proceedings of the auditory visual speech processing, AVSP’2001. Aalborg, Copenhague, pp 138–142

  • Christoffels IK, Formisano E, Schiller NO (2007) Neural correlates of verbal feedback processing: an fMRI study employing overt speech. Hum Brain Mapp 28:868–879

    Article  PubMed  Google Scholar 

  • Christoffels IK, van de Ven V, Waldorp LJ, Formisano E, Schiller NO (2011) The sensory consequences of speaking: parametric neural cancellation during speech in auditory cortex. PLoS One 6:e18307

    Article  PubMed  CAS  Google Scholar 

  • Curio G, Neuloh G, Numminen J, Jousmaki V, Hari R (2000) Speaking modifies voice-evoked activity in the human auditory cortex. Hum Brain Mapp 9:183–191

    Article  PubMed  CAS  Google Scholar 

  • d’Ausilio A, Bufalari I, Salmas P, Fadiga L (2011) The role of the motor system in discriminating degraded speech sounds. Cortex 48:882–887

    Article  PubMed  Google Scholar 

  • Elman JL (1981) Effects of frequency—shifted feedback on the pitch of vocal productions. J Acoust Soc Am 70:45–50

    Article  PubMed  CAS  Google Scholar 

  • Fadiga L, Craighero L, Buccino G, Rizzolatti G (2002) Speech listening specifically modulates the excitability of tongue muscles: a TMS study. Eur J Neurosci 15:399–402

    Article  PubMed  Google Scholar 

  • Feng Y, Gracco VL, Max L (2011) Integration of auditory and somatosensory error signals in the neural control of speech movements. J Neurophysiol 106:667–679

    Article  PubMed  Google Scholar 

  • Folkins JW, Abbs JH (1975) Lip and jaw motor control during speech: responses to resistive loading of the jaw. J Speech Hear Res 18:207–219

    PubMed  CAS  Google Scholar 

  • Ford JM, Mathalon DH (2004) Electrophysiological evidence of corollary discharge dysfunction in schizo-phrenia during talking and thinking. J Psychiatr Res 38:37–46

    Article  PubMed  Google Scholar 

  • Ford JM, Mathalon DH, Heinks T, Kalba S, Faustman WO, Roth WT (2001) Neurophysiological evidence of corollary discharge dysfunction in schizophrenia. Am J Psychiatry 158:2069–2071

    Article  PubMed  CAS  Google Scholar 

  • Fowler CA, Dekle DJ (1991) Listening with eye and hand: cross-modal contributions to speech perception. J Exp Psychol Hum Percept Perform 17:816–828

    Article  PubMed  CAS  Google Scholar 

  • Francis BA, Wonham WM (1976) The internal model principle of control theory. Automatica 12:457–651

    Article  Google Scholar 

  • Gentilucci M, Cattaneo L (2005) Automatic audiovisual integration in speech perception. Exp Brain Res 167:66–75

    Article  PubMed  Google Scholar 

  • Golfinopoulos E, Tourville JA, Bohland JW, Ghosh SS, Nieto-Castanon A, Guenther FH (2011) fMRI investigation of unexpected somatosensory feedback perturbation during speech. Neuroimage 55:1324–1338

    Article  PubMed  Google Scholar 

  • Grabski K, Schwartz JL, Lamalle L, Vilain C, Vallée N, Baciu M, Le Bas JF, Sato M Shared and distinct neural correlates of vowel perception and production. J Neurolinguistics (in press)

  • Gracco VL, Abbs JH (1985) Dynamic control of the perioral system during speech: kinematic analyses of autogenic and nonautogenic sensorimotor processes. J Neurophysiol 54:418–432

    PubMed  CAS  Google Scholar 

  • Guenther FH (1995) Speech sound acquisition, coarticulation, and rate effects in a neural network model of speech production. Psychol Rev 102:594–621

    Article  PubMed  CAS  Google Scholar 

  • Guenther FH (2006) Cortical interactions underlying the production of speech sounds. J Commun Disord 39:350–365

    Article  PubMed  Google Scholar 

  • Guenther FH, Vladusich T (2012) A neural theory of speech acquisition and production. J Neurolinguistics 25:408–422

    Article  PubMed  Google Scholar 

  • Guenther FH, Hampson M, Johnson D (1998) A theoretical investigation of reference frames for the planning of speech movements. Psychol Rev 105:611–633

    Article  PubMed  CAS  Google Scholar 

  • Hashimoto Y, Sakai KL (2003) Brain activations during conscious self-monitoring of speech production with delayed auditory feedback: an fMRI study. Hum Brain Mapp 20:22–28

    Article  PubMed  Google Scholar 

  • Heinks-Maldonado TH, Nagarajan SS, Houde JF (2006) Magnetoencephalographic evidence for a precise forward model in speech production. Neuroreport 17:1375–1379

    Article  PubMed  Google Scholar 

  • Hickok G (2012) Computational neuroanatomy of speech production. Nat Rev Neurosci 13:135–145

    Article  PubMed  CAS  Google Scholar 

  • Hickok G, Buchsbaum B, Humphries C, Muftuler T (2003) Auditory- motor interaction revealed by fMRI: speech, music, and working memory in area Spt. J Cogn Neurosci 15:673–682

    PubMed  Google Scholar 

  • Hickok G, Houde JF, Rong F (2011) Sensorimotor integration in speech processing: computational basis and neural organization. Neuron 69:407–422

    Article  PubMed  CAS  Google Scholar 

  • Hirano S, Kojima H, Naito Y, Honjo I, Kamoto Y, Okazawa H, Konishi J (1996) Cortical speech processing mechanisms while vocalizing visually presented languages. Neuroreport 8:363–367

    Article  PubMed  CAS  Google Scholar 

  • Hirano S, Kojima H, Naito Y, Honjo I, Kamoto Y, Okazawa H, Konishi J (1997) Cortical processing mechanism for vocalization with auditory verbal feedback. Neuroreport 8:2379–2382

    Article  PubMed  CAS  Google Scholar 

  • Houde JF, Jordan MI (1998) Sensorimotor adaptation in speech production. Science 279(5354):1213–1216

    Article  PubMed  CAS  Google Scholar 

  • Houde JF, Nagarajan SS, Sekihara K, Merzenich MM (2002) Modulation of the auditory cortex during speech: an MEG study. J Cogn Neurosci 14:1125–1138

    Article  PubMed  Google Scholar 

  • Jeannerod M (1994) The representing brain. Neural correlates of motor intention and imagery. Behav Brain Sci 17:187–245

    Article  Google Scholar 

  • Jeannerod M (2001) Neural simulation of action: a unifying mechanism for motor cognition. Neuroimage 14:S103–S109

    Article  PubMed  CAS  Google Scholar 

  • Jones JA, Munhall KG (2000) Perceptual calibration of F0 production: evidence from feedback perturbation. J Acoust Soc Am 108:1246–1251

    Article  PubMed  CAS  Google Scholar 

  • Jones JA, Munhall KG (2005) Remapping auditory–motor representations in voice production. Curr Biol 15:1768–1772

    Article  PubMed  CAS  Google Scholar 

  • Kawato M, Furukawa K, Suzuki R (1987) A hierarchical neural network model for the control and learning of voluntary movements. Biol Cybern 56:1–17

    Article  Google Scholar 

  • Kerzel D, Bekkering H (2000) Motor activation from visible speech: evidence from stimulus response compatibility. J Exp Psychol Hum Percept Perform 26:634–647

    Article  PubMed  CAS  Google Scholar 

  • Klucharev V, Möttönen R, Sams M (2003) Electrophysiological indicators of phonetic and non-phonetic multisensory interactions during audiovisual speech perception. Cogn Brain Res 18:65–75

    Article  Google Scholar 

  • Lamettti DR, Nasir SM, Ostry DJ (2012) Sensory preference in speech production revealed by simultaneous alteration of auditory and somatosensory feedback. J Neurosci 32:9351–9358

    Article  Google Scholar 

  • Levelt WJM (1989) Speaking: from intention to articulation. The MIT Press, Cambridge, MA

    Google Scholar 

  • MacDonald J, Andersen S, Bachmann T (2000) Hearing by eye: how much spatial degradation can be tolerated? Perception 29:1155–1168

    Article  PubMed  CAS  Google Scholar 

  • MacLeod A, Summerfield Q (1987) Quantifying the contribution of vision to speech perception in noise. Br J Audiol 21:131–141

    Article  PubMed  CAS  Google Scholar 

  • Massaro DW (1998) Perceiving talking faces: from speech perception to a behavioral principle. The MIT Press, Cambridge, MA

    Google Scholar 

  • McGurk H, MacDonald J (1976) Hearing lips and seeing voices. Nature 264:746–748

    Article  PubMed  CAS  Google Scholar 

  • Nasir SM, Ostry DJ (2006) Somatosensory precision in speech production. Curr Biol 16:1918–1923

    Article  PubMed  CAS  Google Scholar 

  • Numminen J, Curio G (1999) Differential effects of overt, covert and replayed speech on vowel-evoked responses of the human auditory cortex. Neurosci Lett 272:29–32

    Article  PubMed  CAS  Google Scholar 

  • Numminen J, Salmelin R, Hari R (2000) Subject’s own speech reduces reactivity of the human auditory cortex. Neurosci Lett 265:119–122

    Article  Google Scholar 

  • Oldfield RC (1971) The assessment and analysis of handedness: the Edinburgh inventory. Neuropsychologia 9:97–114

    Article  PubMed  CAS  Google Scholar 

  • Perkell JS (2012) Movement goals and feedback and feed forward control mechanisms in speech production. J Neurolinguistics 25:382–407

    Article  PubMed  Google Scholar 

  • Perkell JS, Matthies ML, Lane H, Guenther FH, Wilhelms-Tricarico R, Wozniak J et al (1997) Speech motor control: acoustic goals, saturation effects, auditory feedback & internal models. Speech Commun 22:227–250

    Article  Google Scholar 

  • Perkell JS, Guenther FH, Lane H, Matthies LM, Perrier P, Vick J, Wilhelms-Tricarico R, Zandipour M (2000) A theory of speech motor control and supporting data from speakers with normal hearing and with profound hearing loss. J Phon 28:233–272

    Article  Google Scholar 

  • Perrier P (2005) Control and representations in speech production. ZAS Pap Linguist 40:109–132

    Google Scholar 

  • Perrier P (2012) Gesture planning integrating knowledge of the motor plant’s dynamics: a literature review from motor control and speech motor control. In: Speech planning and dynamics, collection: speech production and perception

  • Price CJ, Crinion JT, MacSweeney M (2011) A generative model of speech production in Broca’s and Wernicke’s areas. Front Psychol 2:237

    Article  PubMed  Google Scholar 

  • Pulvermuller F, Huss M, Kherif F, del Prado Martin FM, Hauk O, Shtyrov Y (2006) Motor cortex maps articulatory features of speech sounds. Proc Natl Acad Sci USA 103:7865–7870

    Article  PubMed  CAS  Google Scholar 

  • Purcell DW, Munhall KG (2006a) Compensation following real-time manipulation of formants in isolated vowels. J Acoust Soc Am 119:2288–2297

    Article  PubMed  Google Scholar 

  • Purcell DW, Munhall KG (2006b) Adaptive control of vowel formant frequency: evidence from real-time formant manipulation. J Acoust Soc Am 120:966–977

    Article  PubMed  Google Scholar 

  • Rizzolatti G, Craighero L (2004) The mirror-neuron system. Annu Rev Neurosci 27:169–192

    Article  PubMed  CAS  Google Scholar 

  • Sams M, Möttönen R, Sihvonen T (2005) Seeing and hearing others and oneself talk. Brain Res Cogn Brain Res 23:429–435

    Article  PubMed  Google Scholar 

  • Sato M, Tremblay P, Gracco V (2009) A mediating role of the premotor cortex in phoneme segmentation. Brain Lang 111:1–7

    Article  PubMed  Google Scholar 

  • Sato M, Buccino G, Gentilucci M, Cattaneo L (2010) On the tip of the tongue: modulation of the primary motor cortex during audiovisual speech perception. Speech Commun 52:533–541

    Article  Google Scholar 

  • Schwartz JL, Abry C, Boë LJ, Cathiard MA (2002) Phonology in a theory of perception-for-action-control. In: Durand J, Lacks B (eds) Phonology: from phonetics to cognition. Oxford University Press, Oxford, pp 240–280

    Google Scholar 

  • Schwartz JL, Ménard L, Basirat A, Sato M (2012) The perception for action control theory (PACT): a perceptuo-motor theory of speech perception. J Neurolinguistics 25:336–354

    Article  Google Scholar 

  • Shergill SS, Brammer MJ, Fukuda R, Bullmore E, Amaro E Jr, Murray RM, McGuire PK (2002) Modulation of activity in temporal cortex during generation of inner speech. Hum Brain Mapp 16:219–227

    Article  PubMed  Google Scholar 

  • Shiller DM, Sato M, Gracco VL, Baum S (2009) Perceptual recalibration of speech sounds following speech motor learning. J Acoust Soc Am 125:1103–1113

    Article  PubMed  Google Scholar 

  • Shiller DM, Gracco VL, Rvachew S (2010) Auditory-motor learning during speech production in 9–11 year-old children. PLoS One 5:e12975

    Article  PubMed  Google Scholar 

  • Shuster LI, Lemieux SK (2005) An fMRI investigation of covertly and overtly produced mono- and multisyllabic words. Brain Lang 93:20–31

    Article  PubMed  Google Scholar 

  • Skipper JI, Nusbaum HC, Small SL (2005) Listening to talking faces: motor cortical activation during speech perception. Neuroimage 25:76–89

    Article  PubMed  Google Scholar 

  • Skipper JI, Van Wassenhove V, Nusbaum HC, Small SL (2007) Hearing lips and seeing voices: how cortical areas supporting speech production mediate audiovisual speech perception. Cereb Cortex 17:2387–2399

    Article  PubMed  Google Scholar 

  • Sumby WH, Pollack I (1954) Visual contribution to speech intelligibility in noise. J Acoust Soc Am 26:212–215

    Article  Google Scholar 

  • Tian X, Poeppel D (2010) Mental imagery of speech and movement implicates the dynamics of internal forward models. Front Psychol 1:166

    PubMed  Google Scholar 

  • Tourville JA, Reilly KJ, Guenther FH (2008) Neural mechanisms underlying auditory feedback control of speech. Neuroimage 39:1429–1443

    Article  PubMed  Google Scholar 

  • Tremblay P, Small SL (2011) On the context-dependent nature of the contribution of the ventral premotor cortex to speech perception. Neuroimage 57:1561–1571

    Article  PubMed  Google Scholar 

  • Tremblay S, Shiller DM, Ostry DJ (2003) Somatosensory basis of speech production. Nature 423:866–869

    Article  PubMed  CAS  Google Scholar 

  • Ventura MI, Nagarajan SS, Houde JF (2009) Speech target modulates speaking induced suppression in auditory cortex. BMC Neurosci 10:58

    Article  PubMed  Google Scholar 

  • von Holst E, Mittelstaedt H (1950) Das Reafferenzprinzip. Wechselwirkungen zwischen Zentralnervensystem und Peripherie. Naturwissenchaften 37:464–476

    Article  Google Scholar 

  • Watkins KE, Strafelle AP, Paus T (2003) Seeing and hearing speech excites the motor system involved in speech production. Neuropsychologia 41:989–994

    Article  PubMed  CAS  Google Scholar 

  • Wilson SM, Saygin AP, Sereno MI, Iacoboni M (2004) Listening to speech activates motor areas involved in speech production. Nat Neurosci 7:701–702

    Article  PubMed  CAS  Google Scholar 

  • Wise RJ, Scott SK, Blank SC, Mummery CJ, Murphy K, Warburton EA (2001) Separate neural subsystems within “Wernicke’s area”. Brain 124:83–95

    Article  PubMed  CAS  Google Scholar 

  • Yetkin FZ, Hammeke TA, Swanson SJ, Morris GL, Mueller WM, McAuliffe TL, Haughton VM (1995) A comparison of functional MR activation patterns during silent and audible language tasks. Am J Neuroradiol 16:1087–1092

    PubMed  CAS  Google Scholar 

  • Zekveld AA, Heslenfeld DJ, Festen JM, Schoonhoven R (2006) Top–down and bottom–up processes in speech comprehension. Neuroimage 32:1826–1836

    Article  PubMed  Google Scholar 

Download references

Acknowledgments

We thank Jérome Aubin for his help in this study.

Author information

Authors and Affiliations

  1. GIPSA-LAB, UMR CNRS 5216, Département Parole and Cognition, Grenoble Université, 1180, Avenue centrale, BP 25, 38040, Grenoble Cedex 9, France

    Marc Sato & Emilie Troille

  2. Centre for Research on Brain, Language and Music, McGill University, Montreal, Canada

    Marc Sato, Lucie Ménard & Vincent Gracco

  3. Centre de Recherche sur l’Imaginaire, Université Stendhal, Grenoble, France

    Emilie Troille & Marie-Agnès Cathiard

  4. Département de Linguistique, Université du Québec à Montréal, Montréal, Canada

    Lucie Ménard

  5. School of Communication Sciences and Disorders, McGill University, Montreal, Canada

    Vincent Gracco

  6. Haskins Laboratories, New Haven, CT, USA

    Vincent Gracco

Authors
  1. Marc Sato
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Emilie Troille
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Lucie Ménard
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Marie-Agnès Cathiard
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Vincent Gracco
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Marc Sato.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Sato, M., Troille, E., Ménard, L. et al. Silent articulation modulates auditory and audiovisual speech perception. Exp Brain Res 227, 275–288 (2013). https://doi.org/10.1007/s00221-013-3510-8

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00221-013-3510-8

Keywords

Search

Navigation