Experimental Brain Research

, Volume 227, Issue 2, pp 275–288 | Cite as

Silent articulation modulates auditory and audiovisual speech perception

  • Marc Sato
  • Emilie Troille
  • Lucie Ménard
  • Marie-Agnès Cathiard
  • Vincent Gracco
Research Article

Abstract

The concept of an internal forward model that internally simulates the sensory consequences of an action is a central idea in speech motor control. Consistent with this hypothesis, silent articulation has been shown to modulate activity of the auditory cortex and to improve the auditory identification of concordant speech sounds, when embedded in white noise. In the present study, we replicated and extended this behavioral finding by showing that silently articulating a syllable in synchrony with the presentation of a concordant auditory and/or visually ambiguous speech stimulus improves its identification. Our results further demonstrate that, even in the case of perfect perceptual identification, concurrent mouthing of a syllable speeds up the perceptual processing of a concordant speech stimulus. These results reflect multisensory-motor interactions during speech perception and provide new behavioral arguments for internally generated sensory predictions during silent speech production.

Keywords

Speech perception Speech production Silent speech Audiovisual speech perception Internal forward models Sensory-motor interactions Efference copy McGurk effect 

References

  1. Abbs JH, Gracco VL (1984) Control of complex motor gestures: orofacial muscle responses to load perturbations of lip during speech. J Neurophysiol 51:705–723PubMedGoogle Scholar
  2. Alho J, Sato M, Sams M, Schwartz J-L, Tiitinen H, Jääskeläinen IP (2012) Enhanced early-latency electromagnetic activity in the left premotor cortex is associated with successful phonetic categorization. Neuroimage 60:1937–1946PubMedCrossRefGoogle Scholar
  3. Benoît C, Mohamadi T, Kandel SD (1994) Effects on phonetic context on audio-visual intelligibility of French. J Speech Lang Hear Res 37:1195–1203Google Scholar
  4. Binder JR, Liebenthal E, Possing ET, Medler DA, DouglasWard B (2004) Neural correlates of sensory and decision processes in auditory object identification. Nat Neurosci 7:295–301PubMedCrossRefGoogle Scholar
  5. Burnett TA, Freedland MB, Larson CR, Hain TC (1998) Voice F0 responses to manipulations in pitch feedback. J Acoust Soc Am 103:3153–3161PubMedCrossRefGoogle Scholar
  6. Callan D, Callan A, Gamez M, Sato MA, Kawato M (2010) Premotor cortex mediates perceptual performance. Neuroimage 51:844–858PubMedCrossRefGoogle Scholar
  7. Campbell CS, Massaro DW (1997) Perception of visible speech: influence of spatial quantization. Perception 26:627–644PubMedCrossRefGoogle Scholar
  8. Cathiard M-A, Schwartz J-L, Abry C (2001) Asking a naive question to the McGurk effect: why does audio [b] give more [d] percepts with usual [g] than with visual [d]? In: Proceedings of the auditory visual speech processing, AVSP’2001. Aalborg, Copenhague, pp 138–142Google Scholar
  9. Christoffels IK, Formisano E, Schiller NO (2007) Neural correlates of verbal feedback processing: an fMRI study employing overt speech. Hum Brain Mapp 28:868–879PubMedCrossRefGoogle Scholar
  10. Christoffels IK, van de Ven V, Waldorp LJ, Formisano E, Schiller NO (2011) The sensory consequences of speaking: parametric neural cancellation during speech in auditory cortex. PLoS One 6:e18307PubMedCrossRefGoogle Scholar
  11. Curio G, Neuloh G, Numminen J, Jousmaki V, Hari R (2000) Speaking modifies voice-evoked activity in the human auditory cortex. Hum Brain Mapp 9:183–191PubMedCrossRefGoogle Scholar
  12. d’Ausilio A, Bufalari I, Salmas P, Fadiga L (2011) The role of the motor system in discriminating degraded speech sounds. Cortex 48:882–887PubMedCrossRefGoogle Scholar
  13. Elman JL (1981) Effects of frequency—shifted feedback on the pitch of vocal productions. J Acoust Soc Am 70:45–50PubMedCrossRefGoogle Scholar
  14. Fadiga L, Craighero L, Buccino G, Rizzolatti G (2002) Speech listening specifically modulates the excitability of tongue muscles: a TMS study. Eur J Neurosci 15:399–402PubMedCrossRefGoogle Scholar
  15. Feng Y, Gracco VL, Max L (2011) Integration of auditory and somatosensory error signals in the neural control of speech movements. J Neurophysiol 106:667–679PubMedCrossRefGoogle Scholar
  16. Folkins JW, Abbs JH (1975) Lip and jaw motor control during speech: responses to resistive loading of the jaw. J Speech Hear Res 18:207–219PubMedGoogle Scholar
  17. Ford JM, Mathalon DH (2004) Electrophysiological evidence of corollary discharge dysfunction in schizo-phrenia during talking and thinking. J Psychiatr Res 38:37–46PubMedCrossRefGoogle Scholar
  18. Ford JM, Mathalon DH, Heinks T, Kalba S, Faustman WO, Roth WT (2001) Neurophysiological evidence of corollary discharge dysfunction in schizophrenia. Am J Psychiatry 158:2069–2071PubMedCrossRefGoogle Scholar
  19. Fowler CA, Dekle DJ (1991) Listening with eye and hand: cross-modal contributions to speech perception. J Exp Psychol Hum Percept Perform 17:816–828PubMedCrossRefGoogle Scholar
  20. Francis BA, Wonham WM (1976) The internal model principle of control theory. Automatica 12:457–651CrossRefGoogle Scholar
  21. Gentilucci M, Cattaneo L (2005) Automatic audiovisual integration in speech perception. Exp Brain Res 167:66–75PubMedCrossRefGoogle Scholar
  22. Golfinopoulos E, Tourville JA, Bohland JW, Ghosh SS, Nieto-Castanon A, Guenther FH (2011) fMRI investigation of unexpected somatosensory feedback perturbation during speech. Neuroimage 55:1324–1338PubMedCrossRefGoogle Scholar
  23. Grabski K, Schwartz JL, Lamalle L, Vilain C, Vallée N, Baciu M, Le Bas JF, Sato M Shared and distinct neural correlates of vowel perception and production. J Neurolinguistics (in press)Google Scholar
  24. Gracco VL, Abbs JH (1985) Dynamic control of the perioral system during speech: kinematic analyses of autogenic and nonautogenic sensorimotor processes. J Neurophysiol 54:418–432PubMedGoogle Scholar
  25. Guenther FH (1995) Speech sound acquisition, coarticulation, and rate effects in a neural network model of speech production. Psychol Rev 102:594–621PubMedCrossRefGoogle Scholar
  26. Guenther FH (2006) Cortical interactions underlying the production of speech sounds. J Commun Disord 39:350–365PubMedCrossRefGoogle Scholar
  27. Guenther FH, Vladusich T (2012) A neural theory of speech acquisition and production. J Neurolinguistics 25:408–422PubMedCrossRefGoogle Scholar
  28. Guenther FH, Hampson M, Johnson D (1998) A theoretical investigation of reference frames for the planning of speech movements. Psychol Rev 105:611–633PubMedCrossRefGoogle Scholar
  29. Hashimoto Y, Sakai KL (2003) Brain activations during conscious self-monitoring of speech production with delayed auditory feedback: an fMRI study. Hum Brain Mapp 20:22–28PubMedCrossRefGoogle Scholar
  30. Heinks-Maldonado TH, Nagarajan SS, Houde JF (2006) Magnetoencephalographic evidence for a precise forward model in speech production. Neuroreport 17:1375–1379PubMedCrossRefGoogle Scholar
  31. Hickok G (2012) Computational neuroanatomy of speech production. Nat Rev Neurosci 13:135–145PubMedCrossRefGoogle Scholar
  32. Hickok G, Buchsbaum B, Humphries C, Muftuler T (2003) Auditory- motor interaction revealed by fMRI: speech, music, and working memory in area Spt. J Cogn Neurosci 15:673–682PubMedGoogle Scholar
  33. Hickok G, Houde JF, Rong F (2011) Sensorimotor integration in speech processing: computational basis and neural organization. Neuron 69:407–422PubMedCrossRefGoogle Scholar
  34. Hirano S, Kojima H, Naito Y, Honjo I, Kamoto Y, Okazawa H, Konishi J (1996) Cortical speech processing mechanisms while vocalizing visually presented languages. Neuroreport 8:363–367PubMedCrossRefGoogle Scholar
  35. Hirano S, Kojima H, Naito Y, Honjo I, Kamoto Y, Okazawa H, Konishi J (1997) Cortical processing mechanism for vocalization with auditory verbal feedback. Neuroreport 8:2379–2382PubMedCrossRefGoogle Scholar
  36. Houde JF, Jordan MI (1998) Sensorimotor adaptation in speech production. Science 279(5354):1213–1216PubMedCrossRefGoogle Scholar
  37. Houde JF, Nagarajan SS, Sekihara K, Merzenich MM (2002) Modulation of the auditory cortex during speech: an MEG study. J Cogn Neurosci 14:1125–1138PubMedCrossRefGoogle Scholar
  38. Jeannerod M (1994) The representing brain. Neural correlates of motor intention and imagery. Behav Brain Sci 17:187–245CrossRefGoogle Scholar
  39. Jeannerod M (2001) Neural simulation of action: a unifying mechanism for motor cognition. Neuroimage 14:S103–S109PubMedCrossRefGoogle Scholar
  40. Jones JA, Munhall KG (2000) Perceptual calibration of F0 production: evidence from feedback perturbation. J Acoust Soc Am 108:1246–1251PubMedCrossRefGoogle Scholar
  41. Jones JA, Munhall KG (2005) Remapping auditory–motor representations in voice production. Curr Biol 15:1768–1772PubMedCrossRefGoogle Scholar
  42. Kawato M, Furukawa K, Suzuki R (1987) A hierarchical neural network model for the control and learning of voluntary movements. Biol Cybern 56:1–17CrossRefGoogle Scholar
  43. Kerzel D, Bekkering H (2000) Motor activation from visible speech: evidence from stimulus response compatibility. J Exp Psychol Hum Percept Perform 26:634–647PubMedCrossRefGoogle Scholar
  44. Klucharev V, Möttönen R, Sams M (2003) Electrophysiological indicators of phonetic and non-phonetic multisensory interactions during audiovisual speech perception. Cogn Brain Res 18:65–75CrossRefGoogle Scholar
  45. Lamettti DR, Nasir SM, Ostry DJ (2012) Sensory preference in speech production revealed by simultaneous alteration of auditory and somatosensory feedback. J Neurosci 32:9351–9358CrossRefGoogle Scholar
  46. Levelt WJM (1989) Speaking: from intention to articulation. The MIT Press, Cambridge, MAGoogle Scholar
  47. MacDonald J, Andersen S, Bachmann T (2000) Hearing by eye: how much spatial degradation can be tolerated? Perception 29:1155–1168PubMedCrossRefGoogle Scholar
  48. MacLeod A, Summerfield Q (1987) Quantifying the contribution of vision to speech perception in noise. Br J Audiol 21:131–141PubMedCrossRefGoogle Scholar
  49. Massaro DW (1998) Perceiving talking faces: from speech perception to a behavioral principle. The MIT Press, Cambridge, MAGoogle Scholar
  50. McGurk H, MacDonald J (1976) Hearing lips and seeing voices. Nature 264:746–748PubMedCrossRefGoogle Scholar
  51. Nasir SM, Ostry DJ (2006) Somatosensory precision in speech production. Curr Biol 16:1918–1923PubMedCrossRefGoogle Scholar
  52. Numminen J, Curio G (1999) Differential effects of overt, covert and replayed speech on vowel-evoked responses of the human auditory cortex. Neurosci Lett 272:29–32PubMedCrossRefGoogle Scholar
  53. Numminen J, Salmelin R, Hari R (2000) Subject’s own speech reduces reactivity of the human auditory cortex. Neurosci Lett 265:119–122CrossRefGoogle Scholar
  54. Oldfield RC (1971) The assessment and analysis of handedness: the Edinburgh inventory. Neuropsychologia 9:97–114PubMedCrossRefGoogle Scholar
  55. Perkell JS (2012) Movement goals and feedback and feed forward control mechanisms in speech production. J Neurolinguistics 25:382–407PubMedCrossRefGoogle Scholar
  56. Perkell JS, Matthies ML, Lane H, Guenther FH, Wilhelms-Tricarico R, Wozniak J et al (1997) Speech motor control: acoustic goals, saturation effects, auditory feedback & internal models. Speech Commun 22:227–250CrossRefGoogle Scholar
  57. Perkell JS, Guenther FH, Lane H, Matthies LM, Perrier P, Vick J, Wilhelms-Tricarico R, Zandipour M (2000) A theory of speech motor control and supporting data from speakers with normal hearing and with profound hearing loss. J Phon 28:233–272CrossRefGoogle Scholar
  58. Perrier P (2005) Control and representations in speech production. ZAS Pap Linguist 40:109–132Google Scholar
  59. Perrier P (2012) Gesture planning integrating knowledge of the motor plant’s dynamics: a literature review from motor control and speech motor control. In: Speech planning and dynamics, collection: speech production and perceptionGoogle Scholar
  60. Price CJ, Crinion JT, MacSweeney M (2011) A generative model of speech production in Broca’s and Wernicke’s areas. Front Psychol 2:237PubMedCrossRefGoogle Scholar
  61. Pulvermuller F, Huss M, Kherif F, del Prado Martin FM, Hauk O, Shtyrov Y (2006) Motor cortex maps articulatory features of speech sounds. Proc Natl Acad Sci USA 103:7865–7870PubMedCrossRefGoogle Scholar
  62. Purcell DW, Munhall KG (2006a) Compensation following real-time manipulation of formants in isolated vowels. J Acoust Soc Am 119:2288–2297PubMedCrossRefGoogle Scholar
  63. Purcell DW, Munhall KG (2006b) Adaptive control of vowel formant frequency: evidence from real-time formant manipulation. J Acoust Soc Am 120:966–977PubMedCrossRefGoogle Scholar
  64. Rizzolatti G, Craighero L (2004) The mirror-neuron system. Annu Rev Neurosci 27:169–192PubMedCrossRefGoogle Scholar
  65. Sams M, Möttönen R, Sihvonen T (2005) Seeing and hearing others and oneself talk. Brain Res Cogn Brain Res 23:429–435PubMedCrossRefGoogle Scholar
  66. Sato M, Tremblay P, Gracco V (2009) A mediating role of the premotor cortex in phoneme segmentation. Brain Lang 111:1–7PubMedCrossRefGoogle Scholar
  67. Sato M, Buccino G, Gentilucci M, Cattaneo L (2010) On the tip of the tongue: modulation of the primary motor cortex during audiovisual speech perception. Speech Commun 52:533–541CrossRefGoogle Scholar
  68. Schwartz JL, Abry C, Boë LJ, Cathiard MA (2002) Phonology in a theory of perception-for-action-control. In: Durand J, Lacks B (eds) Phonology: from phonetics to cognition. Oxford University Press, Oxford, pp 240–280Google Scholar
  69. Schwartz JL, Ménard L, Basirat A, Sato M (2012) The perception for action control theory (PACT): a perceptuo-motor theory of speech perception. J Neurolinguistics 25:336–354CrossRefGoogle Scholar
  70. Shergill SS, Brammer MJ, Fukuda R, Bullmore E, Amaro E Jr, Murray RM, McGuire PK (2002) Modulation of activity in temporal cortex during generation of inner speech. Hum Brain Mapp 16:219–227PubMedCrossRefGoogle Scholar
  71. Shiller DM, Sato M, Gracco VL, Baum S (2009) Perceptual recalibration of speech sounds following speech motor learning. J Acoust Soc Am 125:1103–1113PubMedCrossRefGoogle Scholar
  72. Shiller DM, Gracco VL, Rvachew S (2010) Auditory-motor learning during speech production in 9–11 year-old children. PLoS One 5:e12975PubMedCrossRefGoogle Scholar
  73. Shuster LI, Lemieux SK (2005) An fMRI investigation of covertly and overtly produced mono- and multisyllabic words. Brain Lang 93:20–31PubMedCrossRefGoogle Scholar
  74. Skipper JI, Nusbaum HC, Small SL (2005) Listening to talking faces: motor cortical activation during speech perception. Neuroimage 25:76–89PubMedCrossRefGoogle Scholar
  75. Skipper JI, Van Wassenhove V, Nusbaum HC, Small SL (2007) Hearing lips and seeing voices: how cortical areas supporting speech production mediate audiovisual speech perception. Cereb Cortex 17:2387–2399PubMedCrossRefGoogle Scholar
  76. Sumby WH, Pollack I (1954) Visual contribution to speech intelligibility in noise. J Acoust Soc Am 26:212–215CrossRefGoogle Scholar
  77. Tian X, Poeppel D (2010) Mental imagery of speech and movement implicates the dynamics of internal forward models. Front Psychol 1:166PubMedGoogle Scholar
  78. Tourville JA, Reilly KJ, Guenther FH (2008) Neural mechanisms underlying auditory feedback control of speech. Neuroimage 39:1429–1443PubMedCrossRefGoogle Scholar
  79. Tremblay P, Small SL (2011) On the context-dependent nature of the contribution of the ventral premotor cortex to speech perception. Neuroimage 57:1561–1571PubMedCrossRefGoogle Scholar
  80. Tremblay S, Shiller DM, Ostry DJ (2003) Somatosensory basis of speech production. Nature 423:866–869PubMedCrossRefGoogle Scholar
  81. Ventura MI, Nagarajan SS, Houde JF (2009) Speech target modulates speaking induced suppression in auditory cortex. BMC Neurosci 10:58PubMedCrossRefGoogle Scholar
  82. von Holst E, Mittelstaedt H (1950) Das Reafferenzprinzip. Wechselwirkungen zwischen Zentralnervensystem und Peripherie. Naturwissenchaften 37:464–476CrossRefGoogle Scholar
  83. Watkins KE, Strafelle AP, Paus T (2003) Seeing and hearing speech excites the motor system involved in speech production. Neuropsychologia 41:989–994PubMedCrossRefGoogle Scholar
  84. Wilson SM, Saygin AP, Sereno MI, Iacoboni M (2004) Listening to speech activates motor areas involved in speech production. Nat Neurosci 7:701–702PubMedCrossRefGoogle Scholar
  85. Wise RJ, Scott SK, Blank SC, Mummery CJ, Murphy K, Warburton EA (2001) Separate neural subsystems within “Wernicke’s area”. Brain 124:83–95PubMedCrossRefGoogle Scholar
  86. Yetkin FZ, Hammeke TA, Swanson SJ, Morris GL, Mueller WM, McAuliffe TL, Haughton VM (1995) A comparison of functional MR activation patterns during silent and audible language tasks. Am J Neuroradiol 16:1087–1092PubMedGoogle Scholar
  87. Zekveld AA, Heslenfeld DJ, Festen JM, Schoonhoven R (2006) Top–down and bottom–up processes in speech comprehension. Neuroimage 32:1826–1836PubMedCrossRefGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2013

Authors and Affiliations

  • Marc Sato
    • 1
    • 2
  • Emilie Troille
    • 1
    • 3
  • Lucie Ménard
    • 2
    • 4
  • Marie-Agnès Cathiard
    • 3
  • Vincent Gracco
    • 2
    • 5
    • 6
  1. 1.GIPSA-LAB, UMR CNRS 5216, Département Parole and CognitionGrenoble UniversitéGrenoble Cedex 9France
  2. 2.Centre for Research on Brain, Language and MusicMcGill UniversityMontrealCanada
  3. 3.Centre de Recherche sur l’ImaginaireUniversité StendhalGrenobleFrance
  4. 4.Département de LinguistiqueUniversité du Québec à MontréalMontréalCanada
  5. 5.School of Communication Sciences and DisordersMcGill UniversityMontrealCanada
  6. 6.Haskins LaboratoriesNew HavenUSA

Personalised recommendations