Spatial–temporal dynamics of gesture–speech integration: a simultaneous EEG-fMRI study
- 332 Downloads
The semantic integration between gesture and speech (GSI) is mediated by the left posterior temporal sulcus/middle temporal gyrus (pSTS/MTG) and the left inferior frontal gyrus (IFG). Evidence from electroencephalography (EEG) suggests that oscillations in the alpha and beta bands may support processes at different stages of GSI. In the present study, we investigated the relationship between electrophysiological oscillations and blood-oxygen-level-dependent (BOLD) activity during GSI. In a simultaneous EEG-fMRI study, German participants (n = 19) were presented with videos of an actor either performing meaningful gestures in the context of a comprehensible German (GG) or incomprehensible Russian sentence (GR), or just speaking a German sentence (SG). EEG results revealed reduced alpha and beta power for the GG vs. SG conditions, while fMRI analyses showed BOLD increase in the left pSTS/MTG for GG > GR ∩ GG > SG. In time-window-based EEG-informed fMRI analyses, we further found a positive correlation between single-trial alpha power and BOLD signal in the left pSTS/MTG, the left IFG, and several sub-cortical regions. Moreover, the alpha-pSTS/MTG correlation was observed in an earlier time window in comparison to the alpha-IFG correlation, thus supporting a two-stage processing model of GSI. Our study shows that EEG-informed fMRI implies multiple roles of alpha oscillations during GSI, and that the method is a best candidate for multidimensional investigations on complex cognitive functions such as GSI.
KeywordsSimultaneous EEG-fMRI Gesture Language Alpha oscillations Multisensory integration
This research project is supported by a grant from the ‘Von Behring-Röntgen-Stiftung’ (Project no. 59-0002; 64-0001) and by the ‘Deutsche Forschungsgemeinschaft’ (Project no. DFG: STR 1146/11-2, STR 1146/9-1 and SFB/TRR135 project A3). M. S. is supported by the DFG (project no. STR 1146/4-1). B. S. is supported by the DFG (Project no. STR 1146/8-1).
- Calvert G, Spence C, Stein BE (2004) The handbook of multisensory processes. MIT Press, CambridgeGoogle Scholar
- Drijvers L, Özyürek A, Jensen O (2018) Hearing and seeing meaning in noise: alpha, beta, and gamma oscillations predict gestural enhancement of degraded speech comprehension. Hum Brain Mapp 1–13 (in press)Google Scholar
- Goldin-Meadow S (2005) Hearing gesture: how our hands help us think. Harvard University Press, CambridgeGoogle Scholar
- McNeill D (2006) Gesture: a psycholinguistic approach. In: Brown E, Anderson A (eds) The Encyclopedia of Language and Linguistics. Elsevier, Amsterdam, Boston, pp 58–66Google Scholar
- McNeill D (2008) Gesture and thought. University of Chicago Press, ChicagoGoogle Scholar
- Romero YR, Keil J, Balz J, Niedeggen M, Gallinat J, Senkowski D (2016) Alpha-band oscillations reflect altered multisensory processing of the McGurk illusion in schizophrenia. Front Hum Neurosci 10:41Google Scholar
- Sadaghiani S, Scheeringa R, Lehongre K, Morillon B, Giraud A-L, Kleinschmidt A (2010) Intrinsic connectivity networks, alpha oscillations, and tonic alertness: a simultaneous electroencephalography/functional magnetic resonance imaging study. J Neurosci 30(30):10243–10250CrossRefPubMedGoogle Scholar