Abstract
Understanding the interactions between audition and sensorimotor processes is of theoretical importance, particularly in relation to speech processing. Although one current focus in this area is on interactions between auditory perception and the motor system, there has been less research on connections between the auditory and somatosensory modalities. The current study takes a novel approach to this omission by examining specific auditory–tactile interactions in the context of speech and non-speech sound production. Electroencephalography was used to examine brain responses when participants were presented with speech syllables (a bilabial sound /pa/ and a non-labial sound /ka/) or finger-snapping sounds that were simultaneously paired with tactile stimulation of either the lower lip or the right middle finger. Analyses focused on the sensory-evoked N1 in the event-related potential and the extent of alpha band desynchronization elicited by the stimuli. N1 amplitude over fronto-central sites was significantly enhanced when the bilabial /pa/ sound was paired with tactile lip stimulation and when the finger-snapping sound was paired with tactile stimulation of the finger. Post-stimulus alpha desynchronization at central sites was also enhanced when the /pa/ sound was accompanied by tactile stimulation of the lip. These novel findings indicate that neural aspects of somatosensory–auditory interactions are influenced by the congruency between the location of the bodily touch and the bodily origin of a perceived sound.
Similar content being viewed by others
References
Alais D, Newell FN, Mamassian P (2010) Multisensory processing in review: from physiology to behaviour. See Perceiv 23(1):3–38
Arsenault JS, Buchsbaum BR (2015) No evidence of somatotopic place of articulation feature mapping in motor cortex during passive speech perception. Psychon Bull Rev 23(4):1231–1240
Avikainen S, Forss N, Hari R (2002) Modulated activation of the human SI and SII cortices during observation of hand actions. NeuroImage 15(3):640–646
Badin P, Tarabalka Y, Elisei F (2010) Can you “read” tongue movements? Evaluation of the contribution of tongue display to speech understanding. Speech Commun 52(6):493–503
Barraclough NE, Xiao DK, Baker CI, Oram MW, Perrett DI (2005) Integration of visual and auditory information by superior temporal sulcus neurons responsive to the sight of actions. J Cogn Neurosci 17(3):377–391
Bauer M, Oostenveld R, Peeters M, Fries P (2006) Tactile spatial attention enhances gamma-band activity in somatosensory cortex and reduces low-frequency activity in parieto-occipital areas. J Neurosci 11 26(2):490–501
Beauchamp MS (2005) See me, hear me, touch me: multisensory integration in lateral occipital-temporal cortex. Curr Opin Neurobiol 15(2):145–153
Behroozmand R, Shebek R, Hansen DR, Oya H, Robin DA, Howard MA, Greenlee JDW (2015) Sensory-motor networks involved in speech production and motor control: an fMRI study. NeuroImage 109:418–428
Bizley JK, Nodal FR, Bajo VM, Nelken I, King AJ (2007) Physiological and anatomical evidence for multisensory interactions in auditory cortex. Cereb Cortex 17(9):2172–2189
Boersma P, Weenink D (2009) Praat: doing phonetics by computer (Version 5.1. 05). Retrieved 1 May 2009
Brett-Green BA, Miller LJ, Gavin WJ, Davies PL (2008) Multisensory integration in children: a preliminary ERP study. Brain Res 1242:283–290
Bruderer AG, Danielson DK, Kandhadai P, Werker JF (2015) Sensorimotor influences on speech perception in infancy. Proc Natl Acad Sci 112(14):13531–13536
Buccino G, Binkofski F, Fink GR, Fadiga L, Fogassi L, Gallese V, Freund H-J (2001) Action observation activates premotor and parietal areas in a somatotopic manner: an fMRI study. Eur J Neurosci 13(2):400–404
Callan DE, Jones JA, Callan AM, Akahane-Yamada R (2004) Phonetic perceptual identification by native- and second-language speakers differentially activates brain regions involved with acoustic phonetic processing and those involved with articulatory-auditory/orosensory internal models. NeuroImage 22(3):1182–1194
Calvert GA, Thesen T (2004) Multisensory integration: methodological approaches and emerging principles in the human brain. J Physiol Paris 98(1–3):191–205
Calvert GA, Brammer MJ, Bullmore ET, Campbell R, Iversen SD, David AS (1999) Response amplification in sensory-specific cortices during crossmodal binding. NeuroReport 10(2):2619–2623
Calvert GA, Campbell R, Brammer MJ (2000) Evidence from functional magnetic resonance imaging of crossmodal binding in the human heteromodal cortex. Curr Biol 10(12):649–657
Campanella S, Belin P (2007) Integrating face and voice in person perception. Trends Cogn Sci 11(12):535–543
Campbell T, Winkler I, Kujala T (2007) N1 and the mismatch negativity are spatiotemporally distinct ERP components: disruption of immediate memory by auditory distraction can be related to N1. Psychophysiology 44:530–540
Champoux F, Shiller DM, Zatorre RJ (2011) Feel what you say: an auditory effect on somatosensory perception. PLoS One 6(8):e22829
Cheyne D, Gaetz W, Garnero L, Lachaux JP, Ducorps A, Schwartz D, Varela FJ (2003) Neuromagnetic imaging of cortical oscillations accompanying tactile stimulation. Cogn Brain Res 17(3):599–611
Clayson PE, Baldwin SA, Larson MJ (2013) How does noise affect amplitude and latency measurement of event-related potentials (ERPs)? A methodological critique and simulation study. Psychophysiology 50(2):174–186
Correia JM, Jansma BMB, Bonte M (2015) Decoding articulatory features from fMRI responses in dorsal speech regions. J Neurosci 35(45):15015–15025
D’Ausilio A, Bartoli E, Maffongelli L, Berry JJ, Fadiga L (2014) Vision of tongue movements bias auditory speech perception. Neuropsychologia 63:85–91
Della Penna S, Torquati K, Pizzella V, Babiloni C, Franciotti R, Rossini PM, Romani GL (2004) Temporal dynamics of alpha and beta rhythms in human SI and SII after galvanic median nerve stimulation. A MEG study. NeuroImage 22(4):1438–1446
Delorme A, Makeig S (2004) EEGLAB: an open source toolbox for analysis of single-trial EEG dynamics including independent component analysis. J Neurosci Methods 134(1):9–21
Deschrijver E, Wiersema JR, Brass M (2016) The interaction between felt touch and tactile consequences of observed actions: an action-based somatosensory congruency paradigm. Soc Cogn Affect Neurosci 11(7):1162–1172
Dockstader C, Gaetz W, Cheyne D, Wang F, Castellanos FX, Tannock R (2008) MEG eventrelated desynchronization and synchronization deficits during basic somatosensory processing in individuals with ADHD. Behav Brain Funct 4(1):8
Doehrmann O, Naumer MJ (2008) Semantics and the multisensory brain: how meaning modulates processes of audio-visual integration. Brain Res 1242:136–150
Driver J, Noesselt T (2008) Multisensory interplay reveals crossmodal influences on ‘sensory specific’ brain regions, neural responses, and judgments. Neuron 57(1):11–23
Eulitz C, Diesch E, Pantev C, Hampson S, Elbert T (1995) Magnetic and electric brain activity evoked by the processing of tone and vowel stimuli. J Neurosci 15(4):2748–2755
Foxe JJ, Simpson GV, Ahlfors SP (1998) Parieto-occipital approximately 10 Hz activity reflects anticipatory state of visual attention mechanisms. NeuroReport 9(17):3929–3933
Foxe JJ, Morocz IA, Murray MM, Higgins BA, Javitt DC, Schroeder CE (2000) Multisensory auditory-somatosensory interactions in early cortical processing revealed by high-density electrical mapping. Brain Res Cogn Brain Res 10(1–2):77–83
Fujioka T, Ross B (2008) Auditory processing indexed by stimulus-induced alpha desynchronization in children. Int J Psychophysiol 68(2):130–140
Gazzola V, Aziz-Zadeh L, Keysers C (2006) Empathy and the somatotopic auditory mirror system in humans. Curr Biol 16(18):1824–1829
Gick B, Derrick D (2009) Aero-tactile integration in speech perception. Nature 462(7272):502–504
Gick B, Ikegami Y, Derrick D (2010) The temporal window of audio-tactile integration in speech perception. J Acoust Soc Am 128(5):342–346
Hartcher-O’Brien J, Soto-faraco S, Adam R (2017) Editorial: a matter of bottom-up or top-down processes: the role of attention in multisensory integration. Front Integr Neurosci 11:5. doi:10.3389/fnint.2017.00005
Hasson U, Nir Y, Levy I, Fuhrmann G, Malach R (2004) Intersubject synchronization of cortical activity during natural vision. Science 303(5664):1634–1640
Henschke JU, Noesselt T, Scheich H, Budinger E (2015) Possible anatomical pathways for short-latency multisensory integration processes in primary sensory cortices. Brain Struct Funct 22(2):955–977
Hickok G (2014) The myth of mirror neurons: the real neuroscience of communication and cognition. WW Norton & Company, New York
Hickok G, Houde J, Rong F (2011) Sensorimotor integration in speech processing: computational basis and neural organization. Neuron 69(3):407–422
Hoffmann S, Falkenstein M (2008) The correction of eye blink artefacts in the EEG: a comparison of two prominent methods. PLoS One 3(8):e3004
Ito T, Ostry DJ (2012) Speech sounds alter facial skin sensation. J Neurophysiol 107(1):442–447
Ito T, Gracco VL, Ostry DJ (2014) Temporal factors affecting somatosensory-auditory interactions in speech processing. Front Psychol 5(10):1–10
Kayser C, Petkov CI, Augath M, Logothetis NK (2005) Integration of touch and sound in auditory cortex. Neuron 48(2):373–384
Kayser C, Petkov CI, Logothetis NK (2008) Visual modulation of neurons in auditory cortex. Cereb Cortex 18(7):1560–1574
Keysers C, Kaas JH, Gazzola V (2010) Somatosensation in social perception. Nat Rev Neurosci 11(6):417–428
Kuhl PK, Meltzoff AN (1982) The bimodal perception of speech in infancy. Science 218(4577):1138–1141
Kuhl PK, Meltzoff AN (1984) The intermodal representation of speech in infants. Infant Behav Dev 7(3):361–381
Kuhl PK, Meltzoff AN (1996) Infant vocalizations in response to speech: vocal imitation and developmental change. J Acoust Soc Am 100(4):2425–2438
Kuhl PK, Ramirez RR, Bosseler A, Lin JF, Imada T (2014) Infants’ brain responses to speech suggest analysis by synthesis. Proc Natl Acad Sci 111(31):11238–11245
Lahav A, Saltzman E, Schlaug G (2007) Action representation of sound: audiomotor recognition network while listening to newly acquired actions. J Neurosci 27(2):308–314
Lakatos P, Chen CM, O’Connell MN, Mills A, Schroeder CE (2007) Neuronal oscillations and multisensory interaction in primary auditory cortex. Neuron 53(2):279–292
Lange J, Oostenveld R, Fries P (2011) Perception of the touch-induced visual double-flash illusion correlates with changes of rhythmic neuronal activity in human visual and somatosensory areas. Neuroimage 15 54(2):1395–1405
Lange J, Christian N, Schnitzler A (2013) Audio-visual congruency alters power and coherence of oscillatory activity within and between cortical areas. NeuroImage 79:111–120
Macaluso E, Driver J (2005) Multisensory spatial interactions: a window onto functional integration in the human brain. Trends Neurosci 28(5):264–271
Macaluso E, Noppeney U, Talsma D, Vercillo T, Hartcher-O’Brien J, Adam R (2016) The curious incident of attention in multisensory integration: bottom-up vs top-down. Multisens Res 29(6–7):557–583
Makeig S (1993) Auditory event-related dynamics of the EEG spectrum and effects of exposure to tones. Electroencephalogr Clin Neurophysiol 86(4):283–293
Marshall PJ, Meltzoff AN (2015) Body maps in the infant brain. Trends Cogn Sci 19(9):499–505
McGurk H, MacDonald J (1976) Hearing lips and seeing voices. Nature 264(5588):746–748
Meltzoff AN, Moore MK (1997) Explaining facial imitation: a theoretical model. Early Dev Parent 6(3–4):179
Molholm S, Ritter W, Javitt DC, Foxe JJ (2004) Multisensory visual-auditory object recognition in humans: a high-density electrical mapping study. Cereb Cortex 14(4):452–465
Möttönen R, Järveläinen J, Sams M, Hari R (2005) Viewing speech modulates activity in the left SI mouth cortex. NeuroImage 24(3):731–737
Müller N, Weisz N (2012) Lateralized auditory cortical alpha band activity and interregional connectivity pattern reflect anticipation of target sounds. Cereb Cortex 22(7):1604–1613
Murray MM, Molholm S, Michel CM, Heslenfeld DJ, Ritter W, Javitt DC, Foxe JJ (2005) Grabbing your ear: rapid auditory-somatosensory multisensory interactions in low-level sensory cortices are not constrained by stimulus alignment. Cereb Cortex 15(7):963–974
Näätänen R, Picton T (1987) The N1 wave of the human electric and magnetic response to sound: a review and an analysis of the component structure. Psychophysiology 24:375–425
Nishimura S, Tomita Y, Tanaka T, Kasagi S, Takashima S, Takeshita K (1986) Developmental change of short latency somatosensory evoked potential waves between P3 and N1 components in children. Brain Dev 8(1):6–9
Paulus M, Hunnius S, Van Elk M, Bekkering H (2012) How learning to shake a rattle affects 8-month-old infants’ perception of the rattle’s sound: electrophysiological evidence for action-effect binding in infancy. Dev Cogn Neurosci 2(1):90–96
Pihko E, Nevalainen P, Stephen J, Okada Y, Lauronen L (2009) Maturation of somatosensory cortical processing from birth to adulthood revealed by magnetoencephalography. Clin Neurophysiol 120(8):1552–1561
Pizzamiglio L, Aprile T, Spitoni G, Pitzalis S, Bates E, D’Amico S, Di Russo F (2005) Separate neural systems for processing action- or non-action-related sounds. NeuroImage 24(3):852–861
Plöchl M, Gaston J, Mermagen T, König P, Hairston WD (2016) Oscillatory activity in auditory cortex reflects the perceptual level of audio-tactile integration. Sci Rep 6:33693
Pulvermüller F, Fadiga L (2010) Active perception: sensorimotor circuits as a cortical basis for language. Nat Rev Neurosci 11(5):351–360
Pulvermüller F, Huss M, Kherif F, del Prado M, Martin F, Hauk O, Shtyrov Y (2006) Motor cortex maps articulatory features of speech sounds. Proc Natl Acad Sci USA 103(2):7865–7870
Ro T, Hsu J, Yasar NE, Elmore LC, Beauchamp MS (2009) Sound enhances touch perception. Exp Brain Res 195(1):135–143
Ro T, Ellmore TM, Beauchamp MS (2012) A neural link between feeling and hearing. Cereb Cortex 23(7):1724–1730
Rossi S, Tecchio F, Pasqualetti P, Ulivelli M, Pizzella V, Romani G, Rossini P (2002) Somatosensory processing during movement observation in humans. Clin Neurophysiol 113(1):16–24
Russo N, Foxe JJ, Brandwein AB, Altschuler T, Gomes H, Molholm S (2010) Multisensory processing in children with autism: high-density electrical mapping of auditory–somatosensory integration. Autism Res 3(5):253–267
Saby JN, Meltzoff AN, Marshall PJ (2013) Infants’ somatotopic neural responses to seeing human actions: i’ve got you under my dkin. PLoS One 8(10):e77905
Sadaghiani S, Kleinschmidt A (2016) Brain networks and α- oscillations: structural and functional foundations of cognitive control. Trends Cogn Sci 20(11):805–817
Schomers MR, Pulvermüller F (2016) Is the sensorimotor cortex relevant for speech perception and understanding? An integrative review. Front Hum Neurosci 10(9):435
Schubert R, Haufe S, Blankenburg F, Villringer A, Curio G (2008) Now you’ll feel It-now You won’t: EEG rhythms predict the effectiveness of perceptual masking. J Cogn Neurosci 21:2407–2419
Senkowski D, Talsma D, Grigutsch M, Herrmann CS, Woldorff MG (2007) Good times for multisensory integration: effects of the precision of temporal synchrony as revealed by gamma-band oscillations. Neuropsychologia 45(3):561–571
Shen G, Saby JN, Drew AR, Marshall PJ (2017) Exploring potential social influences on brain potentials during anticipation of tactile dtimulation. Brain Res 1659:8–18
Stanford TR, Stein BE (2007) Superadditivity in multisensory integration: putting the computation in context. NeuroReport 18(8):787–792
Stein BE, Stanford TR (2008) Multisensory integration: current issues from the perspective of the single neuron. Nature Rev Neurosci 9(4):255
Talsma D, Senkowski D, Soto-Faraco S, Woldorff MG (2010) The multifaceted interplay between attention and multisensory integration. Trends Cogn Sci 14(9):400–410
Thomas R, Sink J, Haggard P (2013) Sensory effects of action observation. Exp Psycol 60(5):335–346
Tiihonen J, Hari R, Kajola M, Karhu J, Ahlfors S, Tissari S (1991) Magnetoencephalographic 10-Hz rhythm from the human auditory cortex. Neurosci Lett 129(2):303–305
Tiitinen H, Sivonen P, Alku P, Virtanen J, Näätänen R (1999) Electromagnetic recordings reveal latency differences in speech and tone processing in humans. Cogn Brain Res 8(3):355–363
Töllner T, Gramann K, Müller HJ, Eimer M (2009) The anterior N1 component as an index of modality shifting. J Cogn Neurosci 21(9):1653–1669
Tsilionis E, Vatakis A (2016) Multisensory binding: is the contribution of synchrony and semantic congruency obligatory? Current Opinion Behav Sci 8:7–13
van Atteveldt N, Formisano E, Goebel R, Blomert L (2004) Integration of letters and speech sounds in the human brain. Neuron 43(2):271–282
Van Atteveldt NM, Formisano E, Blomert L, Goebel R (2007) The effect of temporal asynchrony on the multisensory integration of letters and speech sounds. Cereb Cortex 17(4):962–974
Virtanen J, Ahveninen J, Ilmoniemi RJ, Näätänen R, Pekkonen E (1998) Replicability of MEG and EEG measures of the auditory N1/N1m-response. Electroencephalogr Clin Neurophysiol 108(3):291–298
Welch RB (1999) Meaning, attention, and the “unity assumption” in the intersensory bias of spatial and temporal perceptions. Adv Psychol 129:371–387
Wheaton KJ, Thompson JC, Syngeniotis A, Abbott DF, Puce A (2004) Viewing the motion of human body parts activates different regions of premotor, temporal, and parietal cortex. NeuroImage 22(1):277–288
Wilson SM, Saygin AP, Sereno MI, Iacoboni M (2004) Listening to speech activates motor areas involved in speech production. Nat Neurosci 7(7):701–702
Wilson EC, Braida LD, Reed CM (2010) Perceptual interactions in the loudness of combined auditory and vibrotactile stimuli. J Acoust Soc Am 127(5):3038–3043
Wright TM, Pelphrey KA, Allison T, McKeown MJ, McCarthy G (2003) Polysensory interactions along lateral temporal regions evoked by audiovisual speech. Cereb Cortex 13(10):1034–1043
Wu C, Stefanescu RA, Martel DT, Shore SE (2015) Listening to another sense: somatosensory integration in the auditory system. Cell Tissue Res 361(1):233–250
Acknowledgements
The authors thank Ashley Drew, Staci Weiss, Rebecca Laconi, Jeb Taylor and Rebecca Feldman for their help with data collection. The writing of this article was supported in part by awards from the National Institutes of Health (1R21HD083756) and the National Science Foundation (BCS-1460889 and SMA-1540619).
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Shen, G., Meltzoff, A.N. & Marshall, P.J. Touching lips and hearing fingers: effector-specific congruency between tactile and auditory stimulation modulates N1 amplitude and alpha desynchronization. Exp Brain Res 236, 13–29 (2018). https://doi.org/10.1007/s00221-017-5104-3
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s00221-017-5104-3