Experimental Brain Research

, Volume 235, Issue 12, pp 3743–3755 | Cite as

Cross-modal integration of polyphonic characters in Chinese audio-visual sentences: a MVPA study based on functional connectivity

  • Zhengyi Zhang
  • Gaoyan Zhang
  • Yuanyuan Zhang
  • Hong Liu
  • Junhai Xu
  • Baolin LiuEmail author
Research Article


This study aimed to investigate the functional connectivity in the brain during the cross-modal integration of polyphonic characters in Chinese audio-visual sentences. The visual sentences were all semantically reasonable and the audible pronunciations of the polyphonic characters in corresponding sentences contexts varied in four conditions. To measure the functional connectivity, correlation, coherence and phase synchronization index (PSI) were used, and then multivariate pattern analysis was performed to detect the consensus functional connectivity patterns. These analyses were confined in the time windows of three event-related potential components of P200, N400 and late positive shift (LPS) to investigate the dynamic changes of the connectivity patterns at different cognitive stages. We found that when differentiating the polyphonic characters with abnormal pronunciations from that with the appreciate ones in audio-visual sentences, significant classification results were obtained based on the coherence in the time window of the P200 component, the correlation in the time window of the N400 component and the coherence and PSI in the time window the LPS component. Moreover, the spatial distributions in these time windows were also different, with the recruitment of frontal sites in the time window of the P200 component, the frontal-central-parietal regions in the time window of the N400 component and the central-parietal sites in the time window of the LPS component. These findings demonstrate that the functional interaction mechanisms are different at different stages of audio-visual integration of polyphonic characters.


Polyphonic characters Audio-visual integration Functional connectivity ERP MVPA 



This work was supported by the National Basic Research Program (973 Program) of China (no. 2013CB329301) and National Natural Science Foundation of China (no. 61571327 and no. 61503278).

Compliance with ethical standards

Conflict of interest

We declare that we have no actual or potential conflict of interest including any financial, commercial, personal or other relationships with other people or organizations that can inappropriately influence our work.

Supplementary material

221_2017_5086_MOESM1_ESM.docx (566 kb)
Supplementary material 1 (DOCX 565 kb)


  1. Barnea A, Breznitz Z (1998) Phonological and orthographic processing of Hebrew words: electrophysiological aspects. J Genet Psychol 159:492–504. doi: 10.1080/00221329809596166 CrossRefPubMedGoogle Scholar
  2. Benz HL, Zhang H, Bezerianos A, Acharya S, Crone NE, Zheng X, Thakor NV (2012) Connectivity analysis as a novel approach to motor decoding for prosthesis control. IEEE Trans Neural Syst Rehabil Eng 20:143–152. doi: 10.1109/TNSRE.2011.2175309 CrossRefPubMedGoogle Scholar
  3. Chang CC, Lin CJ (2011) LIBSVM: a library for support vector machines. Acm Trans Intell Syst Technol 2:389–396. doi: 10.1145/1961189.1961199 CrossRefGoogle Scholar
  4. Chen YW, Lin CJ (2006) Combining SVMs with various feature selection strategies. Stud Fuzziness Soft Comput 207:315–524. doi: 10.1007/978-3-540-35488-8_13 CrossRefGoogle Scholar
  5. Connolly J, Phillips N (1994) Event-related potential components reflect phonological and semantic processing of the terminal word of spoken sentences. J Cogn Neurosci 6:256–266. doi: 10.1162/jocn.1994.6.3.256 CrossRefPubMedGoogle Scholar
  6. Connolly JF, Phillips NA, Forbes KA (1995) The effects of phonological and semantic features of sentence-ending words on visual event-related brain potentials. Electroencephalogr Clin Neurophysiol 94:276–287. doi: 10.1016/0013-4694(95)98479-R CrossRefPubMedGoogle Scholar
  7. Dosenbach NUF, Binyam N, Cohen AL et al (2010) Prediction of individual brain maturity using fMRI. Science 329:1358–1361CrossRefPubMedPubMedCentralGoogle Scholar
  8. Frenck-Mestre C, Osterhout L, McLaughlin J, Foucart A (2008) The effect of phonological realization of inflectional morphology on verbal agreement in French: evidence from ERPs. Acta Physiol (Oxf) 128:528–536. doi: 10.1016/j.actpsy.2007.12.007 Google Scholar
  9. Guevara MA, Corsi-Cabrera M (1996) EEG coherence or EEG correlation? Int J Psychophysiol 23:145–153. doi: 10.1016/S0167-8760(96)00038-4 CrossRefPubMedGoogle Scholar
  10. Hagoort P, Brown C, Groothusen J (1993) The syntactic positive shift (SPS) as an ERP measure of syntactic processing. Lang Cogn Process 8:439–483. doi: 10.1080/01690969308407585 CrossRefGoogle Scholar
  11. Hagoort P, Hald L, Bastiaansen M, Petersson KM (2004) Integration of word meaning and world knowledge in language comprehension. Science 304:438–441. doi: 10.1126/science.1095455 CrossRefPubMedGoogle Scholar
  12. Hausfeld L, De Martino F, Bonte M, Formisano E (2012) Pattern analysis of EEG responses to speech and voice: influence of feature grouping. NeuroImage 59:3641–3651. doi: 10.1016/j.neuroimage.2011.11.056 CrossRefPubMedGoogle Scholar
  13. Kasabov N, Capecci E (2015) Spiking neural network methodology for modelling, classification and understanding of EEG spatio-temporal data measuring cognitive processes. Inf Sci 294:565–575. doi: 10.1016/j.ins.2014.06.028 CrossRefGoogle Scholar
  14. Kim A, Osterhout L (2005) The independence of combinatory semantic processing: evidence from event-related potentials. J Mem Lang 52:205–225. doi: 10.1016/j.jml.2004.10.002 CrossRefGoogle Scholar
  15. Kohavi R (2001) A study of cross-validation and bootstrap for accuracy estimation and model selection. In: International joint conference on artificial intelligence, pp 1137–1143Google Scholar
  16. Kolk H, Chwilla D (2007) Late positivities in unusual situations. Brain Lang 100:257–261. doi: 10.1016/j.bandl.2006.07.006 CrossRefPubMedGoogle Scholar
  17. Kuperberg GR (2007) Neural mechanisms of language comprehension: challenges to syntax. Brain Res 1146:23–49. doi: 10.1016/j.brainres.2006.12.063 CrossRefPubMedGoogle Scholar
  18. Kuperberg GR, Sitnikova T, Caplan D, Holcomb PJ (2003) Electrophysiological distinctions in processing conceptual relationships within simple sentences. Cogn Brain Res 17:117–129. doi: 10.1016/S0926-6410(03)00086-7 CrossRefGoogle Scholar
  19. Kutas M, Hillyard SA (1980) Reading senseless sentences: brain potentials reflect semantic incongruity. Science 207:203–205. doi: 10.1126/science.7350657 CrossRefPubMedGoogle Scholar
  20. Lachaux J-P, Rodriguez E, Martinerie J, Varela FJ (1999) Measuring phase synchrony in brain signals. Hum Brain Mapp 8:194–208. doi: 10.1002/(SICI)1097-0193(1999)8:43.0.CO;2-C CrossRefPubMedGoogle Scholar
  21. Landi N, Perfetti CA (2007) An electrophysiological investigation of semantic and phonological processing in skilled and less-skilled comprehenders. Brain Lang 102:30–45. doi: 10.1016/j.bandl.2006.11.001 CrossRefPubMedGoogle Scholar
  22. Lee Y-Y, Hsieh S (2014) Classifying different emotional states by means of EEG-based functional connectivity patterns. PLoS One 9:e95415. doi: 10.1371/journal.pone.0095415 CrossRefPubMedPubMedCentralGoogle Scholar
  23. Liu Y, Shu H, Wei J (2006) Spoken word recognition in context: evidence from Chinese ERP analyses. Brain Lang 96:37–48. doi: 10.1016/j.bandl.2005.08.007 CrossRefPubMedGoogle Scholar
  24. Liu B, Jin Z, Li W, Li Y, Wang Z (2009) The pragmatic meanings conveyed by function words in Chinese sentences: an ERP study. J Neurolinguist 22:548–562. doi: 10.1016/j.jneuroling.2009.06.003 CrossRefGoogle Scholar
  25. Liu B, Jin Z, Qing Z, Wang Z (2011a) The processing of phonological, orthographical, and lexical information of Chinese characters in sentence contexts: an ERP study. Brain Res 1372:81–91. doi: 10.1016/j.brainres.2010.11.068 CrossRefPubMedGoogle Scholar
  26. Liu B, Jin Z, Wang Z, Xin S (2011b) An ERP study on whether the P600 can reflect the presence of unexpected phonology. Exp Brain Res 212:399–408. doi: 10.1007/s00221-011-2739-3 CrossRefPubMedGoogle Scholar
  27. Liu B, Wu G, Wang Z, Meng X, Wang Q (2011c) Semantic association of ecologically unrelated synchronous audio-visual information in cognitive integration: an event-related potential study. Neuroscience 192:494–499. doi: 10.1016/j.neuroscience.2011.05.072 CrossRefPubMedGoogle Scholar
  28. Liu F, Guo W, Fouche JP et al (2015) Multivariate classification of social anxiety disorder using whole brain functional connectivity. Brain Struct Funct 220:101–115. doi: 10.1007/s00429-013-0641-4 CrossRefPubMedGoogle Scholar
  29. Liu H, Zhang G, Liu B (2017) Semantic integration of audio-visual information of polyphonic characters in a sentence context: an event-related potential study. Exp Brain Res 235(4):1119–1128. doi: 10.1007/s00221-017-4872-0 CrossRefPubMedGoogle Scholar
  30. Lobier M, Siebenhühner F, Palva S, Palva JM (2014) Phase transfer entropy: a novel phase-based measure for directed connectivity in networks coupled by oscillatory interactions. NeuroImage 85(Part 2):853–872. doi: 10.1016/j.neuroimage.2013.08.056 CrossRefPubMedGoogle Scholar
  31. Luck SJ, Hillyard SA (1994) Electrophysiological correlates of feature analysis during visual search. Psychophysiology 31:291–308. doi: 10.1111/j.1469-8986.1994.tb02218.x CrossRefPubMedGoogle Scholar
  32. Luo H, Poeppel D (2007) Phase patterns of neuronal responses reliably discriminate speech in human auditory cortex. Neuron 54:1001–1010. doi: 10.1016/j.neuron.2007.06.004 CrossRefPubMedPubMedCentralGoogle Scholar
  33. Meng X, Jian J, Shu H, Tian X, Zhou X (2008) ERP correlates of the development of orthographical and phonological processing during Chinese sentence reading. Brain Res 1219:91–102. doi: 10.1016/j.brainres.2008.04.052 CrossRefPubMedGoogle Scholar
  34. Osterhout L, Holcomb PJ (1992) Event-related brain potentials elicited by syntactic anomaly. J Mem Lang 31:785–806CrossRefGoogle Scholar
  35. Pereira F, Mitchell T, Botvinick M (2009) Machine learning classifiers and fMRI: a tutorial overview. NeuroImage 45:S199–S209. doi: 10.1016/j.neuroimage.2008.11.007 CrossRefPubMedGoogle Scholar
  36. Sargolzaei S, Cabrerizo M, Goryawala M, Eddin AS, Adjouadi M (2015) Scalp EEG brain functional connectivity networks in pediatric epilepsy. Comput Biol Med 56:158–166. doi: 10.1016/j.compbiomed.2014.10.018 CrossRefPubMedGoogle Scholar
  37. Sereno SC, Rayner K, Posner MI (1998) Establishing a time-line of word recognition: evidence from eye movements and event-related potentials. NeuroReport 9:2195–2200. doi: 10.1097/00001756-199807130-00009 CrossRefPubMedGoogle Scholar
  38. Siuly Li Y, Wen P (2014) Modified CC-LR algorithm with three diverse feature sets for motor imagery tasks classification in EEG based brain–computer interface. Comput Methods Programs Biomed 113:767–780. doi: 10.1016/j.cmpb.2013.12.020 CrossRefPubMedGoogle Scholar
  39. Smith E, Weinberg A, Moran T, Hajcak G (2013) Electrocortical responses to NIMSTIM facial expressions of emotion. Int J Psychophysiol 88:17–25. doi: 10.1016/j.ijpsycho.2012.12.004 CrossRefPubMedGoogle Scholar
  40. Zheng Y, Zhou X (2008) Involvement of cognitive control in sentence comprehension: evidence from ERPs. Brain Res 1203:103–115. doi: 10.1016/j.brainres.2008.01.090 CrossRefGoogle Scholar

Copyright information

© Springer-Verlag GmbH Germany 2017

Authors and Affiliations

  • Zhengyi Zhang
    • 1
  • Gaoyan Zhang
    • 1
  • Yuanyuan Zhang
    • 1
  • Hong Liu
    • 1
  • Junhai Xu
    • 1
  • Baolin Liu
    • 1
    • 2
    Email author
  1. 1.Tianjin Key Laboratory of Cognitive Computing and Application, School of Computer Science and TechnologyTianjin UniversityTianjinPeople’s Republic of China
  2. 2.State Key Laboratory of Intelligent Technology and Systems, National Laboratory for Information Science and TechnologyTsinghua UniversityBeijingPeople’s Republic of China

Personalised recommendations