Spatial–temporal dynamics of gesture–speech integration: a simultaneous EEG-fMRI study

Abstract

The semantic integration between gesture and speech (GSI) is mediated by the left posterior temporal sulcus/middle temporal gyrus (pSTS/MTG) and the left inferior frontal gyrus (IFG). Evidence from electroencephalography (EEG) suggests that oscillations in the alpha and beta bands may support processes at different stages of GSI. In the present study, we investigated the relationship between electrophysiological oscillations and blood-oxygen-level-dependent (BOLD) activity during GSI. In a simultaneous EEG-fMRI study, German participants (n = 19) were presented with videos of an actor either performing meaningful gestures in the context of a comprehensible German (GG) or incomprehensible Russian sentence (GR), or just speaking a German sentence (SG). EEG results revealed reduced alpha and beta power for the GG vs. SG conditions, while fMRI analyses showed BOLD increase in the left pSTS/MTG for GG > GR ∩ GG > SG. In time-window-based EEG-informed fMRI analyses, we further found a positive correlation between single-trial alpha power and BOLD signal in the left pSTS/MTG, the left IFG, and several sub-cortical regions. Moreover, the alpha-pSTS/MTG correlation was observed in an earlier time window in comparison to the alpha-IFG correlation, thus supporting a two-stage processing model of GSI. Our study shows that EEG-informed fMRI implies multiple roles of alpha oscillations during GSI, and that the method is a best candidate for multidimensional investigations on complex cognitive functions such as GSI.

This is a preview of subscription content, log in to check access.

Fig. 1
Fig. 2
Fig. 3
Fig. 4

References

  1. Allen PJ, Polizzi G, Krakow K, Fish DR, Lemieux L (1998) Identification of EEG events in the MR scanner: the problem of pulse artifact and a method for its subtraction. Neuroimage 8(3):229–239

    CAS  PubMed  Google Scholar 

  2. Andric M, Small SL (2012) gesture’s neural language. Front Psychol 3:99

    PubMed  PubMed Central  Google Scholar 

  3. Arnstein D, Cui F, Keysers C, Maurits NM, Gazzola V (2011) µ-suppression during action observation and execution correlates with BOLD in dorsal premotor, inferior parietal, and SI cortices. J Neurosci 31(40):14243–14249

    CAS  PubMed  PubMed Central  Google Scholar 

  4. Beauchamp MS (2005) See me, hear me, touch me: multisensory integration in lateral occipital-temporal cortex. Curr Opin Neurobiol 15(2):145–153

    CAS  PubMed  Google Scholar 

  5. Beauchamp MS, Argall BD, Bodurka J, Duyn JH, Martin A (2004a) Unraveling multisensory integration: patchy organization within human STS multisensory cortex. Nat Neurosci 7(11):1190–1192

    CAS  PubMed  Google Scholar 

  6. Beauchamp MS, Lee KE, Argall BD, Martin A (2004b) Integration of auditory and visual information about objects in superior temporal sulcus. Neuron 41(5):809–823

    CAS  PubMed  Google Scholar 

  7. Biau E, Soto-Faraco S (2013) Beat gestures modulate auditory integration in speech perception. Brain Lang 124(2):143–152

    PubMed  Google Scholar 

  8. Biau E, Soto-Faraco S (2015) Synchronization by the hand: the sight of gestures modulates low-frequency activity in brain responses to continuous speech. Front Hum Neurosci 9:527

    PubMed  PubMed Central  Google Scholar 

  9. Biau E, Torralba M, Fuentemilla L, de Diego Balaguer R, Soto-Faraco S (2015) Speaker’s hand gestures modulate speech perception through phase resetting of ongoing neural oscillations. Cortex 68:76–85

    PubMed  Google Scholar 

  10. Braboszcz C, Delorme A (2011) Lost in thoughts: neural markers of low alertness during mind wandering. Neuroimage 54(4):3040–3047

    PubMed  Google Scholar 

  11. Buccino G, Binkofski F, Fink GR, Fadiga L, Fogassi L, Gallese V, Freund HJ (2001) Action observation activates premotor and parietal areas in a somatotopic manner: an fMRI study. Eur J Neurosci 13(2):400–404

    CAS  PubMed  Google Scholar 

  12. Buzsáki G, Draguhn A (2004) Neuronal oscillations in cortical networks. Science 304(5679):1926–1929

    Google Scholar 

  13. Callan DE, Jones JA, Munhall K, Kroos C, Callan AM, Vatikiotis-Bateson E (2004) Multisensory integration sites identified by perception of spatial wavelet filtered visual speech gesture information. J Cogn Neurosci 16(5):805–816

    PubMed  Google Scholar 

  14. Calvert GA, Campbell R, Brammer MJ (2000) Evidence from functional magnetic resonance imaging of crossmodal binding in the human heteromodal cortex. Curr Biol 10(11):649–657

    CAS  PubMed  Google Scholar 

  15. Calvert G, Spence C, Stein BE (2004) The handbook of multisensory processes. MIT Press, Cambridge

    Google Scholar 

  16. Debener S, Ullsperger M, Siegel M, Fiehler K, Von Cramon DY, Engel AK (2005) Trial-by-trial coupling of concurrent electroencephalogram and functional magnetic resonance imaging identifies the dynamics of performance monitoring. J Neurosci 25(50):11730–11737

    CAS  PubMed  Google Scholar 

  17. Debener S, Ullsperger M, Siegel M, Engel AK (2006) Single-trial EEG–fMRI reveals the dynamics of cognitive function. Trends Cogn Sci 10(12):558–563

    PubMed  Google Scholar 

  18. Dick AS, Goldin-Meadow S, Hasson U, Skipper JI, Small SL (2009) Co-speech gestures influence neural activity in brain regions associated with processing semantic information. Hum Brain Mapp 30(11):3509–3526

    PubMed  PubMed Central  Google Scholar 

  19. Dick AS, Goldin-Meadow S, Solodkin A, Small SL (2012a) Gesture in the developing brain. Dev Sci 15(2):165–180

    PubMed  Google Scholar 

  20. Dick AS, Mok EH, Beharelle AR, Goldin-Meadow S, Small SL (2012b) Frontal and temporal contributions to understanding the iconic co-speech gestures that accompany speech. Hum Brain Mapp 35:900–917

    PubMed  PubMed Central  Google Scholar 

  21. Dosenbach NU, Fair DA, Miezin FM, Cohen AL, Wenger KK, Dosenbach RA, Raichle ME (2007) Distinct brain networks for adaptive and stable task control in humans. Proc Natl Acad Sci 104(26):11073–11078

    CAS  PubMed  Google Scholar 

  22. Drijvers L, Özyürek A, Jensen O (2018) Hearing and seeing meaning in noise: alpha, beta, and gamma oscillations predict gestural enhancement of degraded speech comprehension. Hum Brain Mapp 1–13 (in press)

  23. Eichele T, Specht K, Moosmann M, Jongsma ML, Quiroga RQ, Nordby H, Hugdahl K (2005) Assessing the spatiotemporal evolution of neuronal activation with single-trial event-related potentials and functional MRI. Proc Natl Acad Sci USA 102(49):17798–17803

    CAS  PubMed  Google Scholar 

  24. Foxe JJ, Snyder AC (2011) The role of alpha-band brain oscillations as a sensory suppression mechanism during selective attention. Front Psychol 2:154

    PubMed  PubMed Central  Google Scholar 

  25. Goldin-Meadow S (2005) Hearing gesture: how our hands help us think. Harvard University Press, Cambridge

    Google Scholar 

  26. Goldman RI, Stern JM, Engel J Jr, Cohen MS (2002) Simultaneous EEG and fMRI of the alpha rhythm. Neuroreport 13(18):2487

    PubMed  PubMed Central  Google Scholar 

  27. Green A, Straube B, Weis S, Jansen A, Willmes K, Konrad K, Kircher T (2009) Neural integration of iconic and unrelated coverbal gestures: a functional MRI study. Hum Brain Mapp 30(10):3309–3324

    PubMed  Google Scholar 

  28. Groppe DM, Urbach TP, Kutas M (2011) Mass univariate analysis of event-related brain potentials/fields I: a critical tutorial review. Psychophysiology 48(12):1711–1725

    PubMed  PubMed Central  Google Scholar 

  29. He Y, Gebhardt H, Steines M, Sammer G, Kircher T, Nagels A, Straube B (2015) The EEG and fMRI signatures of neural integration: an investigation of meaningful gestures and corresponding speech. Neuropsychologia 72:27–42

    PubMed  Google Scholar 

  30. Holle H, Gunter TC, Rüschemeyer S-A, Hennenlotter A, Iacoboni M (2008) Neural correlates of the processing of co-speech gestures. Neuroimage 39(4):2010–2024

    PubMed  Google Scholar 

  31. Holle H, Obleser J, Rueschemeyer S-A, Gunter TC (2010) Integration of iconic gestures and speech in left superior temporal areas boosts speech comprehension under adverse listening conditions. Neuroimage 49(1):875–884

    PubMed  Google Scholar 

  32. Hubbard AL, Wilson SM, Callan DE, Dapretto M (2009) Giving speech a hand: gesture modulates activity in auditory cortex during speech perception. Hum Brain Mapp 30(3):1028–1037

    PubMed  PubMed Central  Google Scholar 

  33. Järveläinen J, Schuermann M, Hari R (2004) Activation of the human primary motor cortex during observation of tool use. Neuroimage 23(1):187–192

    PubMed  Google Scholar 

  34. Kelly SD, Kravitz C, Hopkins M (2004) Neural correlates of bimodal speech and gesture comprehension. Brain Lang 89(1):253–260

    PubMed  Google Scholar 

  35. Kircher T, Straube B, Leube D, Weis S, Sachs O, Willmes K, Green A (2009) Neural interaction of speech and gesture: differential activations of metaphoric co-verbal gestures. Neuropsychologia 47(1):169–179

    PubMed  Google Scholar 

  36. Kita S, Özyürek A (2003) What does cross-linguistic variation in semantic coordination of speech and gesture reveal?: evidence for an interface representation of spatial thinking and speaking. J Mem Lang 48(1):16–32

    Google Scholar 

  37. Laufs H, Kleinschmidt A, Beyerle A, Eger E, Salek-Haddadi A, Preibisch C, Krakow K (2003a) EEG-correlated fMRI of human alpha activity. Neuroimage 19(4):1463–1476

    CAS  PubMed  Google Scholar 

  38. Laufs H, Krakow K, Sterzer P, Eger E, Beyerle A, Salek-Haddadi A, Kleinschmidt A (2003b) Electroencephalographic signatures of attentional and cognitive default modes in spontaneous brain activity fluctuations at rest. Proc Natl Acad Sci 100(19):11053–11058

    CAS  PubMed  Google Scholar 

  39. Laufs H, Holt JL, Elfont R, Krams M, Paul JS, Krakow K, Kleinschmidt A (2006) Where the BOLD signal goes when alpha EEG leaves. Neuroimage 31(4):1408–1418

    CAS  PubMed  Google Scholar 

  40. Leske S, Tse A, Oosterhof NN, Hartmann T, Müller N, Keil J, Weisz N (2014) The strength of alpha and beta oscillations parametrically scale with the strength of an illusory auditory percept. Neuroimage 88:69–78

    PubMed  Google Scholar 

  41. Liu Y, Bengson J, Huang H, Mangun GR, Ding M (2014) Top-down modulation of neural activity in anticipatory visual attention: control mechanisms revealed by simultaneous EEG–fMRI. Cereb Cortex 26(2):517–529

    PubMed  PubMed Central  Google Scholar 

  42. Luo H, Liu Z, Poeppel D (2010) Auditory cortex tracks both auditory and visual stimulus dynamics using low-frequency neuronal phase modulation. PLoS Biol 8(8):e1000445

    PubMed  PubMed Central  Google Scholar 

  43. Makeig S, Inlow M (1993) Lapse in alertness: coherence of fluctuations in performance and EEG spectrum. Electroencephalogr Clin Neurophysiol 86(1):23–35

    CAS  PubMed  Google Scholar 

  44. Maris E, Oostenveld R (2007) Nonparametric statistical testing of EEG- and MEG-data. J Neurosci Methods 164(1):177–190

    PubMed  Google Scholar 

  45. McNeill D (2006) Gesture: a psycholinguistic approach. In: Brown E, Anderson A (eds) The Encyclopedia of Language and Linguistics. Elsevier, Amsterdam, Boston, pp 58–66

  46. McNeill D (2008) Gesture and thought. University of Chicago Press, Chicago

    Google Scholar 

  47. Nagels A, Kircher T, Steines M, Straube B (2015) Feeling addressed! The role of body orientation and co-speech gesture in social communication. Hum Brain Mapp 36(5):1925–1936

    PubMed  Google Scholar 

  48. Nichols T, Brett M, Andersson J, Wager T, Poline JB (2005) Valid conjunction inference with the minimum statistic. Neuroimage 25(3):653–660

    PubMed  Google Scholar 

  49. Obleser J, Weisz N (2012) Suppressed alpha oscillations predict intelligibility of speech and its acoustic details. Cereb Cortex 22(11):2466–2477

    PubMed  Google Scholar 

  50. Oldfield RC (1971) The assessment and analysis of handedness: the Edinburgh inventory. Neuropsychologia 9(1):97–113

    CAS  Google Scholar 

  51. Oostenveld R, Fries P, Maris E, Schoffelen J-M (2011) FieldTrip: open source software for advanced analysis of MEG, EEG, and invasive electrophysiological data. Intell Neurosci 2011:1–9

    Google Scholar 

  52. Özyürek A, Willems RM, Kita S, Hagoort P (2007) On-line integration of semantic information from speech and gesture: insights from event-related brain potentials. J Cogn Neurosci 19(4):605–616

    PubMed  Google Scholar 

  53. Perry A, Bentin S (2009) Mirror activity in the human brain while observing hand movements: a comparison between EEG desynchronization in the µ-range and previous fMRI results. Brain Res 1282:126–132

    CAS  PubMed  Google Scholar 

  54. Perry A, Stein L, Bentin S (2011) Motor and attentional mechanisms involved in social interaction—evidence from mu and alpha EEG suppression. Neuroimage 58(3):895–904

    PubMed  Google Scholar 

  55. Pfurtscheller G, Stancak Jr A, Neuper C (1996) Event-related synchronization (ERS) in the alpha band—an electrophysiological correlate of cortical idling: a review. Int J Psychophysiol 24(1):39–46

    CAS  PubMed  Google Scholar 

  56. Posner MI (2008) Measuring alertness. Ann N Y Acad Sci 1129(1):193–199

    PubMed  Google Scholar 

  57. Quandt LC, Marshall PJ, Shipley TF, Beilock SL, Goldin-Meadow S (2012) Sensitivity of alpha and beta oscillations to sensorimotor characteristics of action: an EEG study of action production and gesture observation. Neuropsychologia 50(12):2745–2751

    PubMed  PubMed Central  Google Scholar 

  58. Romero YR, Keil J, Balz J, Niedeggen M, Gallinat J, Senkowski D (2016) Alpha-band oscillations reflect altered multisensory processing of the McGurk illusion in schizophrenia. Front Hum Neurosci 10:41

    Google Scholar 

  59. Sadaghiani S, Kleinschmidt A (2016) Brain networks and α-oscillations: structural and functional foundations of cognitive control. Trends Cogn Sci 20(11):805–817

    PubMed  Google Scholar 

  60. Sadaghiani S, Scheeringa R, Lehongre K, Morillon B, Giraud A-L, Kleinschmidt A (2010) Intrinsic connectivity networks, alpha oscillations, and tonic alertness: a simultaneous electroencephalography/functional magnetic resonance imaging study. J Neurosci 30(30):10243–10250

    CAS  PubMed  PubMed Central  Google Scholar 

  61. Sammer G, Blecker C, Gebhardt H, Bischoff M, Stark R, Morgen K, Vaitl D (2007) Relationship between regional hemodynamic activity and simultaneously recorded EEG-theta associated with mental arithmetic-induced workload. Hum Brain Mapp 28(8):793–803

    PubMed  Google Scholar 

  62. Scheeringa R, Petersson KM, Oostenveld R, Norris DG, Hagoort P, Bastiaansen M (2009) Trial-by-trial coupling between EEG and BOLD identifies networks related to alpha and theta EEG power increases during working memory maintenance. Neuroimage 44(3):1224–1238

    PubMed  Google Scholar 

  63. Scheeringa R, Fries P, Petersson K-M, Oostenveld R, Grothe I, Norris DG, Bastiaansen M (2011) Neuronal dynamics underlying high-and low-frequency EEG oscillations contribute independently to the human BOLD signal. Neuron 69(3):572–583

    CAS  PubMed  Google Scholar 

  64. Schroeder CE, Lakatos P, Kajikawa Y, Partan S, Puce A (2008) Neuronal oscillations and visual amplification of speech. Trends Cogn Sci 12(3):106–113

    PubMed  PubMed Central  Google Scholar 

  65. Seeley WW, Menon V, Schatzberg AF, Keller J, Glover GH, Kenna H, Greicius MD (2007) Dissociable intrinsic connectivity networks for salience processing and executive control. J Neurosci 27(9):2349–2356

    CAS  PubMed  PubMed Central  Google Scholar 

  66. Senkowski D, Schneider TR, Foxe JJ, Engel AK (2008) Crossmodal binding through neural coherence: implications for multisensory processing. Trends Neurosci 31(8):401–409

    CAS  PubMed  Google Scholar 

  67. Skipper JI, Goldin-Meadow S, Nusbaum HC, Small SL (2009) Gestures orchestrate brain networks for language understanding. Curr Biol 19(8):661–667

    CAS  PubMed  PubMed Central  Google Scholar 

  68. Slotnick SD, Schacter DL (2004) A sensory signature that distinguishes true from false memories. Nat Neurosci 7(6):664–672

    CAS  PubMed  Google Scholar 

  69. Straube B, Green A, Bromberger B, Kircher T (2011) The differentiation of iconic and metaphoric gestures: common and unique integration processes. Hum Brain Mapp 32(4):520–533

    PubMed  Google Scholar 

  70. Straube B, He Y, Steines M, Gebhardt H, Kircher T, Sammer G, Nagels A (2013) Supramodal neural processing of abstract information conveyed by speech and gesture. Front Behav Neurosci 7:120

    PubMed  PubMed Central  Google Scholar 

  71. Straube B, Green A, Sass K, Kircher T (2014) Superior temporal sulcus disconnectivity during processing of metaphoric gestures in schizophrenia. Schizophr Bull 40(4):936–944

    PubMed  Google Scholar 

  72. Sturm W, Longoni F, Fimm B, Dietrich T, Weis S, Kemna S, Willmes K (2004) Network for auditory intrinsic alertness: a PET study. Neuropsychologia 42(5):563–568

    PubMed  Google Scholar 

  73. Thut G, Nietzel A, Brandt SA, Pascual-Leone A (2006) α-Band electroencephalographic activity over occipital cortex indexes visuospatial attention bias and predicts visual target detection. J Neurosci 26(37):9494–9502

    CAS  PubMed  Google Scholar 

  74. Tseng H-H, Bossong MG, Modinos G, Chen K-M, McGuire P, Allen P (2015) A systematic review of multisensory cognitive–affective integration in schizophrenia. Neurosci Biobehav Rev 55:444–452

    PubMed  Google Scholar 

  75. Tzourio-Mazoyer N, Landeau B, Papathanassiou D, Crivello F, Etard O, Delcroix N, Joliot M (2002) Automated anatomical labeling of activations in SPM using a macroscopic anatomical parcellation of the MNI MRI single-subject brain. Neuroimage 15(1):273–289

    CAS  PubMed  Google Scholar 

  76. Uhlhaas PJ, Singer W (2010) Abnormal neural oscillations and synchrony in schizophrenia. Nat Rev Neurosci 11(2):100–113

    CAS  PubMed  Google Scholar 

  77. van Wassenhove V, Grant KW, Poeppel D (2005) Visual speech speeds up the neural processing of auditory speech. Proc Natl Acad Sci USA 102(4):1181–1186

    PubMed  Google Scholar 

  78. Wildgruber D, Ethofer T, Grandjean D, Kreifelts B (2009) A cerebral network model of speech prosody comprehension. Int J Speech Lang Pathol 11(4):277–281

    Google Scholar 

  79. Willems RM, Özyürek A, Hagoort P (2007) When language meets action: the neural integration of gesture and speech. Cereb Cortex 17(10):2322–2333

    PubMed  Google Scholar 

  80. Willems RM, Oostenveld R, Hagoort P (2008) Early decreases in alpha and gamma band power distinguish linguistic from visual information during spoken sentence comprehension. Brain Res 1219:78–90

    CAS  PubMed  Google Scholar 

  81. Willems RM, Özyürek A, Hagoort P (2009) Differential roles for left inferior frontal and superior temporal cortex in multimodal integration of action and language. Neuroimage 47(4):1992–2004

    PubMed  Google Scholar 

  82. Wu YC, Coulson S (2010) Gestures modulate speech processing early in utterances. Neuroreport 21(7):522

    CAS  PubMed  PubMed Central  Google Scholar 

  83. Zhao W, Riggs K, Schindler I, Holle H (2018) Transcranial magnetic stimulation over left inferior frontal and posterior temporal cortex disrupts gesture–speech integration. J Neurosci 38(8):1891–1900

    CAS  PubMed  Google Scholar 

Download references

Acknowledgements

This research project is supported by a grant from the ‘Von Behring-Röntgen-Stiftung’ (Project no. 59-0002; 64-0001) and by the ‘Deutsche Forschungsgemeinschaft’ (Project no. DFG: STR 1146/11-2, STR 1146/9-1 and SFB/TRR135 project A3). M. S. is supported by the DFG (project no. STR 1146/4-1). B. S. is supported by the DFG (Project no. STR 1146/8-1).

Author information

Affiliations

Authors

Corresponding author

Correspondence to Yifei He.

Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary material 1 (DOCX 1626 KB)

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

He, Y., Steines, M., Sommer, J. et al. Spatial–temporal dynamics of gesture–speech integration: a simultaneous EEG-fMRI study. Brain Struct Funct 223, 3073–3089 (2018). https://doi.org/10.1007/s00429-018-1674-5

Download citation

Keywords

  • Simultaneous EEG-fMRI
  • Gesture
  • Language
  • Alpha oscillations
  • Multisensory integration