Brain Structure and Function

, Volume 223, Issue 7, pp 3073–3089 | Cite as

Spatial–temporal dynamics of gesture–speech integration: a simultaneous EEG-fMRI study

  • Yifei HeEmail author
  • Miriam Steines
  • Jens Sommer
  • Helge Gebhardt
  • Arne Nagels
  • Gebhard Sammer
  • Tilo Kircher
  • Benjamin Straube
Original Article


The semantic integration between gesture and speech (GSI) is mediated by the left posterior temporal sulcus/middle temporal gyrus (pSTS/MTG) and the left inferior frontal gyrus (IFG). Evidence from electroencephalography (EEG) suggests that oscillations in the alpha and beta bands may support processes at different stages of GSI. In the present study, we investigated the relationship between electrophysiological oscillations and blood-oxygen-level-dependent (BOLD) activity during GSI. In a simultaneous EEG-fMRI study, German participants (n = 19) were presented with videos of an actor either performing meaningful gestures in the context of a comprehensible German (GG) or incomprehensible Russian sentence (GR), or just speaking a German sentence (SG). EEG results revealed reduced alpha and beta power for the GG vs. SG conditions, while fMRI analyses showed BOLD increase in the left pSTS/MTG for GG > GR ∩ GG > SG. In time-window-based EEG-informed fMRI analyses, we further found a positive correlation between single-trial alpha power and BOLD signal in the left pSTS/MTG, the left IFG, and several sub-cortical regions. Moreover, the alpha-pSTS/MTG correlation was observed in an earlier time window in comparison to the alpha-IFG correlation, thus supporting a two-stage processing model of GSI. Our study shows that EEG-informed fMRI implies multiple roles of alpha oscillations during GSI, and that the method is a best candidate for multidimensional investigations on complex cognitive functions such as GSI.


Simultaneous EEG-fMRI Gesture Language Alpha oscillations Multisensory integration 



This research project is supported by a grant from the ‘Von Behring-Röntgen-Stiftung’ (Project no. 59-0002; 64-0001) and by the ‘Deutsche Forschungsgemeinschaft’ (Project no. DFG: STR 1146/11-2, STR 1146/9-1 and SFB/TRR135 project A3). M. S. is supported by the DFG (project no. STR 1146/4-1). B. S. is supported by the DFG (Project no. STR 1146/8-1).

Supplementary material

429_2018_1674_MOESM1_ESM.docx (1.6 mb)
Supplementary material 1 (DOCX 1626 KB)


  1. Allen PJ, Polizzi G, Krakow K, Fish DR, Lemieux L (1998) Identification of EEG events in the MR scanner: the problem of pulse artifact and a method for its subtraction. Neuroimage 8(3):229–239CrossRefPubMedGoogle Scholar
  2. Andric M, Small SL (2012) gesture’s neural language. Front Psychol 3:99CrossRefPubMedPubMedCentralGoogle Scholar
  3. Arnstein D, Cui F, Keysers C, Maurits NM, Gazzola V (2011) µ-suppression during action observation and execution correlates with BOLD in dorsal premotor, inferior parietal, and SI cortices. J Neurosci 31(40):14243–14249CrossRefPubMedGoogle Scholar
  4. Beauchamp MS (2005) See me, hear me, touch me: multisensory integration in lateral occipital-temporal cortex. Curr Opin Neurobiol 15(2):145–153CrossRefPubMedGoogle Scholar
  5. Beauchamp MS, Argall BD, Bodurka J, Duyn JH, Martin A (2004a) Unraveling multisensory integration: patchy organization within human STS multisensory cortex. Nat Neurosci 7(11):1190–1192CrossRefPubMedGoogle Scholar
  6. Beauchamp MS, Lee KE, Argall BD, Martin A (2004b) Integration of auditory and visual information about objects in superior temporal sulcus. Neuron 41(5):809–823CrossRefPubMedGoogle Scholar
  7. Biau E, Soto-Faraco S (2013) Beat gestures modulate auditory integration in speech perception. Brain Lang 124(2):143–152CrossRefPubMedGoogle Scholar
  8. Biau E, Soto-Faraco S (2015) Synchronization by the hand: the sight of gestures modulates low-frequency activity in brain responses to continuous speech. Front Hum Neurosci 9:527CrossRefPubMedPubMedCentralGoogle Scholar
  9. Biau E, Torralba M, Fuentemilla L, de Diego Balaguer R, Soto-Faraco S (2015) Speaker’s hand gestures modulate speech perception through phase resetting of ongoing neural oscillations. Cortex 68:76–85CrossRefPubMedGoogle Scholar
  10. Braboszcz C, Delorme A (2011) Lost in thoughts: neural markers of low alertness during mind wandering. Neuroimage 54(4):3040–3047CrossRefPubMedGoogle Scholar
  11. Buccino G, Binkofski F, Fink GR, Fadiga L, Fogassi L, Gallese V, Freund HJ (2001) Action observation activates premotor and parietal areas in a somatotopic manner: an fMRI study. Eur J Neurosci 13(2):400–404PubMedGoogle Scholar
  12. Buzsáki G, Draguhn A (2004) Neuronal oscillations in cortical networks. Science 304(5679):1926–1929CrossRefPubMedGoogle Scholar
  13. Callan DE, Jones JA, Munhall K, Kroos C, Callan AM, Vatikiotis-Bateson E (2004) Multisensory integration sites identified by perception of spatial wavelet filtered visual speech gesture information. J Cogn Neurosci 16(5):805–816CrossRefPubMedGoogle Scholar
  14. Calvert GA, Campbell R, Brammer MJ (2000) Evidence from functional magnetic resonance imaging of crossmodal binding in the human heteromodal cortex. Curr Biol 10(11):649–657CrossRefPubMedGoogle Scholar
  15. Calvert G, Spence C, Stein BE (2004) The handbook of multisensory processes. MIT Press, CambridgeGoogle Scholar
  16. Debener S, Ullsperger M, Siegel M, Fiehler K, Von Cramon DY, Engel AK (2005) Trial-by-trial coupling of concurrent electroencephalogram and functional magnetic resonance imaging identifies the dynamics of performance monitoring. J Neurosci 25(50):11730–11737CrossRefPubMedGoogle Scholar
  17. Debener S, Ullsperger M, Siegel M, Engel AK (2006) Single-trial EEG–fMRI reveals the dynamics of cognitive function. Trends Cogn Sci 10(12):558–563CrossRefPubMedGoogle Scholar
  18. Dick AS, Goldin-Meadow S, Hasson U, Skipper JI, Small SL (2009) Co-speech gestures influence neural activity in brain regions associated with processing semantic information. Hum Brain Mapp 30(11):3509–3526CrossRefPubMedPubMedCentralGoogle Scholar
  19. Dick AS, Goldin-Meadow S, Solodkin A, Small SL (2012a) Gesture in the developing brain. Dev Sci 15(2):165–180CrossRefPubMedGoogle Scholar
  20. Dick AS, Mok EH, Beharelle AR, Goldin-Meadow S, Small SL (2012b) Frontal and temporal contributions to understanding the iconic co-speech gestures that accompany speech. Hum Brain Mapp 35:900–917CrossRefPubMedPubMedCentralGoogle Scholar
  21. Dosenbach NU, Fair DA, Miezin FM, Cohen AL, Wenger KK, Dosenbach RA, Raichle ME (2007) Distinct brain networks for adaptive and stable task control in humans. Proc Natl Acad Sci 104(26):11073–11078CrossRefPubMedGoogle Scholar
  22. Drijvers L, Özyürek A, Jensen O (2018) Hearing and seeing meaning in noise: alpha, beta, and gamma oscillations predict gestural enhancement of degraded speech comprehension. Hum Brain Mapp 1–13 (in press)Google Scholar
  23. Eichele T, Specht K, Moosmann M, Jongsma ML, Quiroga RQ, Nordby H, Hugdahl K (2005) Assessing the spatiotemporal evolution of neuronal activation with single-trial event-related potentials and functional MRI. Proc Natl Acad Sci USA 102(49):17798–17803CrossRefPubMedGoogle Scholar
  24. Foxe JJ, Snyder AC (2011) The role of alpha-band brain oscillations as a sensory suppression mechanism during selective attention. Front Psychol 2:154CrossRefPubMedPubMedCentralGoogle Scholar
  25. Goldin-Meadow S (2005) Hearing gesture: how our hands help us think. Harvard University Press, CambridgeGoogle Scholar
  26. Goldman RI, Stern JM, Engel J Jr, Cohen MS (2002) Simultaneous EEG and fMRI of the alpha rhythm. Neuroreport 13(18):2487CrossRefPubMedPubMedCentralGoogle Scholar
  27. Green A, Straube B, Weis S, Jansen A, Willmes K, Konrad K, Kircher T (2009) Neural integration of iconic and unrelated coverbal gestures: a functional MRI study. Hum Brain Mapp 30(10):3309–3324CrossRefPubMedGoogle Scholar
  28. Groppe DM, Urbach TP, Kutas M (2011) Mass univariate analysis of event-related brain potentials/fields I: a critical tutorial review. Psychophysiology 48(12):1711–1725CrossRefPubMedPubMedCentralGoogle Scholar
  29. He Y, Gebhardt H, Steines M, Sammer G, Kircher T, Nagels A, Straube B (2015) The EEG and fMRI signatures of neural integration: an investigation of meaningful gestures and corresponding speech. Neuropsychologia 72:27–42CrossRefPubMedGoogle Scholar
  30. Holle H, Gunter TC, Rüschemeyer S-A, Hennenlotter A, Iacoboni M (2008) Neural correlates of the processing of co-speech gestures. Neuroimage 39(4):2010–2024CrossRefPubMedGoogle Scholar
  31. Holle H, Obleser J, Rueschemeyer S-A, Gunter TC (2010) Integration of iconic gestures and speech in left superior temporal areas boosts speech comprehension under adverse listening conditions. Neuroimage 49(1):875–884CrossRefPubMedGoogle Scholar
  32. Hubbard AL, Wilson SM, Callan DE, Dapretto M (2009) Giving speech a hand: gesture modulates activity in auditory cortex during speech perception. Hum Brain Mapp 30(3):1028–1037CrossRefPubMedPubMedCentralGoogle Scholar
  33. Järveläinen J, Schuermann M, Hari R (2004) Activation of the human primary motor cortex during observation of tool use. Neuroimage 23(1):187–192CrossRefPubMedGoogle Scholar
  34. Kelly SD, Kravitz C, Hopkins M (2004) Neural correlates of bimodal speech and gesture comprehension. Brain Lang 89(1):253–260CrossRefPubMedGoogle Scholar
  35. Kircher T, Straube B, Leube D, Weis S, Sachs O, Willmes K, Green A (2009) Neural interaction of speech and gesture: differential activations of metaphoric co-verbal gestures. Neuropsychologia 47(1):169–179CrossRefPubMedGoogle Scholar
  36. Kita S, Özyürek A (2003) What does cross-linguistic variation in semantic coordination of speech and gesture reveal?: evidence for an interface representation of spatial thinking and speaking. J Mem Lang 48(1):16–32CrossRefGoogle Scholar
  37. Laufs H, Kleinschmidt A, Beyerle A, Eger E, Salek-Haddadi A, Preibisch C, Krakow K (2003a) EEG-correlated fMRI of human alpha activity. Neuroimage 19(4):1463–1476CrossRefPubMedGoogle Scholar
  38. Laufs H, Krakow K, Sterzer P, Eger E, Beyerle A, Salek-Haddadi A, Kleinschmidt A (2003b) Electroencephalographic signatures of attentional and cognitive default modes in spontaneous brain activity fluctuations at rest. Proc Natl Acad Sci 100(19):11053–11058CrossRefPubMedGoogle Scholar
  39. Laufs H, Holt JL, Elfont R, Krams M, Paul JS, Krakow K, Kleinschmidt A (2006) Where the BOLD signal goes when alpha EEG leaves. Neuroimage 31(4):1408–1418CrossRefPubMedGoogle Scholar
  40. Leske S, Tse A, Oosterhof NN, Hartmann T, Müller N, Keil J, Weisz N (2014) The strength of alpha and beta oscillations parametrically scale with the strength of an illusory auditory percept. Neuroimage 88:69–78CrossRefPubMedGoogle Scholar
  41. Liu Y, Bengson J, Huang H, Mangun GR, Ding M (2014) Top-down modulation of neural activity in anticipatory visual attention: control mechanisms revealed by simultaneous EEG–fMRI. Cereb Cortex 26(2):517–529PubMedPubMedCentralGoogle Scholar
  42. Luo H, Liu Z, Poeppel D (2010) Auditory cortex tracks both auditory and visual stimulus dynamics using low-frequency neuronal phase modulation. PLoS Biol 8(8):e1000445CrossRefPubMedPubMedCentralGoogle Scholar
  43. Makeig S, Inlow M (1993) Lapse in alertness: coherence of fluctuations in performance and EEG spectrum. Electroencephalogr Clin Neurophysiol 86(1):23–35CrossRefPubMedGoogle Scholar
  44. Maris E, Oostenveld R (2007) Nonparametric statistical testing of EEG- and MEG-data. J Neurosci Methods 164(1):177–190CrossRefPubMedGoogle Scholar
  45. McNeill D (2006) Gesture: a psycholinguistic approach. In: Brown E, Anderson A (eds) The Encyclopedia of Language and Linguistics. Elsevier, Amsterdam, Boston, pp 58–66Google Scholar
  46. McNeill D (2008) Gesture and thought. University of Chicago Press, ChicagoGoogle Scholar
  47. Nagels A, Kircher T, Steines M, Straube B (2015) Feeling addressed! The role of body orientation and co-speech gesture in social communication. Hum Brain Mapp 36(5):1925–1936CrossRefPubMedGoogle Scholar
  48. Nichols T, Brett M, Andersson J, Wager T, Poline JB (2005) Valid conjunction inference with the minimum statistic. Neuroimage 25(3):653–660CrossRefPubMedGoogle Scholar
  49. Obleser J, Weisz N (2012) Suppressed alpha oscillations predict intelligibility of speech and its acoustic details. Cereb Cortex 22(11):2466–2477CrossRefPubMedGoogle Scholar
  50. Oldfield RC (1971) The assessment and analysis of handedness: the Edinburgh inventory. Neuropsychologia 9(1):97–113CrossRefPubMedGoogle Scholar
  51. Oostenveld R, Fries P, Maris E, Schoffelen J-M (2011) FieldTrip: open source software for advanced analysis of MEG, EEG, and invasive electrophysiological data. Intell Neurosci 2011:1–9CrossRefGoogle Scholar
  52. Özyürek A, Willems RM, Kita S, Hagoort P (2007) On-line integration of semantic information from speech and gesture: insights from event-related brain potentials. J Cogn Neurosci 19(4):605–616CrossRefPubMedGoogle Scholar
  53. Perry A, Bentin S (2009) Mirror activity in the human brain while observing hand movements: a comparison between EEG desynchronization in the µ-range and previous fMRI results. Brain Res 1282:126–132CrossRefPubMedGoogle Scholar
  54. Perry A, Stein L, Bentin S (2011) Motor and attentional mechanisms involved in social interaction—evidence from mu and alpha EEG suppression. Neuroimage 58(3):895–904CrossRefPubMedGoogle Scholar
  55. Pfurtscheller G, Stancak Jr A, Neuper C (1996) Event-related synchronization (ERS) in the alpha band—an electrophysiological correlate of cortical idling: a review. Int J Psychophysiol 24(1):39–46CrossRefPubMedGoogle Scholar
  56. Posner MI (2008) Measuring alertness. Ann N Y Acad Sci 1129(1):193–199CrossRefPubMedGoogle Scholar
  57. Quandt LC, Marshall PJ, Shipley TF, Beilock SL, Goldin-Meadow S (2012) Sensitivity of alpha and beta oscillations to sensorimotor characteristics of action: an EEG study of action production and gesture observation. Neuropsychologia 50(12):2745–2751CrossRefPubMedPubMedCentralGoogle Scholar
  58. Romero YR, Keil J, Balz J, Niedeggen M, Gallinat J, Senkowski D (2016) Alpha-band oscillations reflect altered multisensory processing of the McGurk illusion in schizophrenia. Front Hum Neurosci 10:41Google Scholar
  59. Sadaghiani S, Kleinschmidt A (2016) Brain networks and α-oscillations: structural and functional foundations of cognitive control. Trends Cogn Sci 20(11):805–817CrossRefPubMedGoogle Scholar
  60. Sadaghiani S, Scheeringa R, Lehongre K, Morillon B, Giraud A-L, Kleinschmidt A (2010) Intrinsic connectivity networks, alpha oscillations, and tonic alertness: a simultaneous electroencephalography/functional magnetic resonance imaging study. J Neurosci 30(30):10243–10250CrossRefPubMedGoogle Scholar
  61. Sammer G, Blecker C, Gebhardt H, Bischoff M, Stark R, Morgen K, Vaitl D (2007) Relationship between regional hemodynamic activity and simultaneously recorded EEG-theta associated with mental arithmetic-induced workload. Hum Brain Mapp 28(8):793–803CrossRefPubMedGoogle Scholar
  62. Scheeringa R, Petersson KM, Oostenveld R, Norris DG, Hagoort P, Bastiaansen M (2009) Trial-by-trial coupling between EEG and BOLD identifies networks related to alpha and theta EEG power increases during working memory maintenance. Neuroimage 44(3):1224–1238CrossRefPubMedGoogle Scholar
  63. Scheeringa R, Fries P, Petersson K-M, Oostenveld R, Grothe I, Norris DG, Bastiaansen M (2011) Neuronal dynamics underlying high-and low-frequency EEG oscillations contribute independently to the human BOLD signal. Neuron 69(3):572–583CrossRefPubMedGoogle Scholar
  64. Schroeder CE, Lakatos P, Kajikawa Y, Partan S, Puce A (2008) Neuronal oscillations and visual amplification of speech. Trends Cogn Sci 12(3):106–113CrossRefPubMedPubMedCentralGoogle Scholar
  65. Seeley WW, Menon V, Schatzberg AF, Keller J, Glover GH, Kenna H, Greicius MD (2007) Dissociable intrinsic connectivity networks for salience processing and executive control. J Neurosci 27(9):2349–2356CrossRefPubMedPubMedCentralGoogle Scholar
  66. Senkowski D, Schneider TR, Foxe JJ, Engel AK (2008) Crossmodal binding through neural coherence: implications for multisensory processing. Trends Neurosci 31(8):401–409CrossRefPubMedGoogle Scholar
  67. Skipper JI, Goldin-Meadow S, Nusbaum HC, Small SL (2009) Gestures orchestrate brain networks for language understanding. Curr Biol 19(8):661–667CrossRefPubMedPubMedCentralGoogle Scholar
  68. Slotnick SD, Schacter DL (2004) A sensory signature that distinguishes true from false memories. Nat Neurosci 7(6):664–672CrossRefPubMedGoogle Scholar
  69. Straube B, Green A, Bromberger B, Kircher T (2011) The differentiation of iconic and metaphoric gestures: common and unique integration processes. Hum Brain Mapp 32(4):520–533CrossRefPubMedGoogle Scholar
  70. Straube B, He Y, Steines M, Gebhardt H, Kircher T, Sammer G, Nagels A (2013) Supramodal neural processing of abstract information conveyed by speech and gesture. Front Behav Neurosci 7:120CrossRefPubMedPubMedCentralGoogle Scholar
  71. Straube B, Green A, Sass K, Kircher T (2014) Superior temporal sulcus disconnectivity during processing of metaphoric gestures in schizophrenia. Schizophr Bull 40(4):936–944CrossRefPubMedGoogle Scholar
  72. Sturm W, Longoni F, Fimm B, Dietrich T, Weis S, Kemna S, Willmes K (2004) Network for auditory intrinsic alertness: a PET study. Neuropsychologia 42(5):563–568CrossRefPubMedGoogle Scholar
  73. Thut G, Nietzel A, Brandt SA, Pascual-Leone A (2006) α-Band electroencephalographic activity over occipital cortex indexes visuospatial attention bias and predicts visual target detection. J Neurosci 26(37):9494–9502CrossRefPubMedGoogle Scholar
  74. Tseng H-H, Bossong MG, Modinos G, Chen K-M, McGuire P, Allen P (2015) A systematic review of multisensory cognitive–affective integration in schizophrenia. Neurosci Biobehav Rev 55:444–452CrossRefPubMedGoogle Scholar
  75. Tzourio-Mazoyer N, Landeau B, Papathanassiou D, Crivello F, Etard O, Delcroix N, Joliot M (2002) Automated anatomical labeling of activations in SPM using a macroscopic anatomical parcellation of the MNI MRI single-subject brain. Neuroimage 15(1):273–289CrossRefPubMedGoogle Scholar
  76. Uhlhaas PJ, Singer W (2010) Abnormal neural oscillations and synchrony in schizophrenia. Nat Rev Neurosci 11(2):100–113CrossRefPubMedGoogle Scholar
  77. van Wassenhove V, Grant KW, Poeppel D (2005) Visual speech speeds up the neural processing of auditory speech. Proc Natl Acad Sci USA 102(4):1181–1186CrossRefPubMedGoogle Scholar
  78. Wildgruber D, Ethofer T, Grandjean D, Kreifelts B (2009) A cerebral network model of speech prosody comprehension. Int J Speech Lang Pathol 11(4):277–281CrossRefGoogle Scholar
  79. Willems RM, Özyürek A, Hagoort P (2007) When language meets action: the neural integration of gesture and speech. Cereb Cortex 17(10):2322–2333CrossRefPubMedGoogle Scholar
  80. Willems RM, Oostenveld R, Hagoort P (2008) Early decreases in alpha and gamma band power distinguish linguistic from visual information during spoken sentence comprehension. Brain Res 1219:78–90CrossRefPubMedGoogle Scholar
  81. Willems RM, Özyürek A, Hagoort P (2009) Differential roles for left inferior frontal and superior temporal cortex in multimodal integration of action and language. Neuroimage 47(4):1992–2004CrossRefPubMedGoogle Scholar
  82. Wu YC, Coulson S (2010) Gestures modulate speech processing early in utterances. Neuroreport 21(7):522CrossRefPubMedPubMedCentralGoogle Scholar
  83. Zhao W, Riggs K, Schindler I, Holle H (2018) Transcranial magnetic stimulation over left inferior frontal and posterior temporal cortex disrupts gesture–speech integration. J Neurosci 38(8):1891–1900PubMedGoogle Scholar

Copyright information

© Springer-Verlag GmbH Germany, part of Springer Nature 2018

Authors and Affiliations

  1. 1.Translational Neuroimaging Lab, Department of Psychiatry and Psychotherapy, Marburg Center for Mind, Brain and Behavior (MCMBB)Philipps-University MarburgMarburgGermany
  2. 2.Department of Psychiatry and Psychotherapy, Marburg Center for Mind, Brain and Behavior (MCMBB)Philipps-University MarburgMarburgGermany
  3. 3.Department of General LinguisticsJohannes-Gutenberg University MainzMainzGermany
  4. 4.Cognitive Neuroscience at Centre for PsychiatryJustus Liebig University GiessenGiessenGermany

Personalised recommendations