Advertisement

Hand Gestures Alert Auditory Cortices

Possible Impacts of Learning on Foreign Language Processing
  • Arne NagelsEmail author
  • Spencer D. Kelly
  • Tilo Kircher
  • Benjamin Straube
Chapter

Abstract

When acquiring a foreign language, the first challenge is to break into the speech stream to identify basic linguistic units. The present study tested the hypothesis that hand gestures facilitate this process by alerting auditory cortices to attend to and identify meaningful phonemic information. During fMRI data acquisition, participants watched videos of an actor speaking in Russian under three conditions. Sentences were produced with just speech alone or were accompanied by two types of hand gestures: 1) metaphoric gesture and 2) free gesture. The main finding was that there was increased auditory cortex activation when both types of gestures accompanied speech compared to speech alone, but there were no differences between the two speech + gesture conditions (or gesture alone conditions). These results suggest that hand gestures may play a role in focusing attention to auditory processing to increase capacity when listening to novel speech in a foreign language.

Keywords

Language Foreign Language Learning L2 Speech Hand Gesture Auditory Cortex Metaphoric 

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

Bibliography

  1. Calvert, G. A., Brammer, M. J., Bullmore, E. T., Campbell, R., Iversen, S. D., & David, A. S. (1999). Response amplification in sensory-specific cortices during crossmodal binding. Neuroreport 10(12), 2619–2623.Google Scholar
  2. Clark, H. H. (1996). Using language. Cambridge: University Press Cambridge.Google Scholar
  3. Colin, C., Radeau, M., Soquet, A, Demolin, D., Colin, F., & Deltenre, P. (2002). Mismatch negativity evoked by the McGurk-MacDonald effect: A phonetic representation within short-term memory. Clinical Neurophysiology 113(4), 495–506.Google Scholar
  4. Dick, A. S., Goldin-Meadow, S., Hasson, U., Skipper, J. I., & Small, S. L. (2009). Co-speech gestures infl uence neural activity in brain regions associated with processing semantic information. Hum Brain Mapp 30(11), 3509–3526.Google Scholar
  5. Feyereisen, P., Van de Wiele, M., & Dubois, F. (1988). The meaning of gestures: What can be understood without speech? Curr Psychol Cogn 8, 3–25.Google Scholar
  6. Friston, K. J., Fletcher, P., Josephs, O., Holmes, A., Rugg, M. D., & Turner, R. (1998). Event-related fMRI: Characterizing differential responses. Neuroimage 7(1), 30–40.Google Scholar
  7. Green, A., Straube, B., Weis, S., Jansen, A., Willmes, K., Konrad, K., & Kircher, T. (2009). Neural integration of iconic and unrelated coverbal gestures: A functional MRI study. Hum Brain Mapp 30(10), 3309–3324.Google Scholar
  8. Gullberg, M., Roberts, L., Dimroth, C., Veroude, K., & Indefrey, P. (2010). Adult language learning after minimal exposure to an unknown natural language. Language Learning 60, 5–24.Google Scholar
  9. Habets, B., Kita, S., Shao, Z., Ozyurek, A., & Hagoort, P. (2011). The Role of Synchrony and Ambiguity in Speech-Gesture Integration during Comprehension. J Cogn Neurosci 23(8), 1845–1854.Google Scholar
  10. Holle, H., & Gunter, T. C. (2007): The role of iconic gestures in speech disambiguation: ERP evidence. J Cogn Neurosci 19(7), 1175–1192.Google Scholar
  11. Holle, H., Gunter, T. C., Rueschemeyer, S.-A., Hennenlotter, A, & Iacoboni, M. (2008). Neural correlates of the processing of co-speech gestures. Neuroimage 39(4), 2010–2024.Google Scholar
  12. Holle, H., Obleser, J., Rueschemeyer, S.-A., Gunter, T. C. (2010). Integration of iconic gestures and speech in left superior temporal areas boosts speech comprehension under adverse listening conditions. Neuroimage 49(1), 875–884.Google Scholar
  13. Hubbard, A. L., Wilson, S. M., Callan, D. E., Dapretto, M. (2009). Giving speech a hand: Gesture modulates activity in auditory cortex during speech perception. Hum Brain Mapp 30(3), 1028–1037.Google Scholar
  14. Ibáñez A., Manes, F., Escobar, J., Trujillo, N., Andreucci, P., & Hurtado, E. (2010). Gesture influences the processing of figurative language in non-native speakers: ERP evidence. Neurosci Lett 471(1), 48–52.Google Scholar
  15. Ibáñez, A., Toro, P., Cornejo, C., Urquina, H., Hurquina, H., Manes, F., Weisbrod, M., & Schröder, J. (2011). High contextual sensitivity of metaphorical expressions and gesture blending: A video event-related potential design. Psychiatry Res 191(1), 68–75.Google Scholar
  16. Kelly, S. D., Barr, D. J., Church, R. B., & Lynch, K. (1999). Offering a Hand to Pragmatic Understanding: The Role of Speech and Gesture in Comprehension and Memory. Journal of Memory and Language 40, 577–592.Google Scholar
  17. Kelly, S. D., Creigh, P., & Bartolotti, J. (2010a). Integrating speech and iconic gestures in a Stroop-like task: Evidence for automatic processing. J Cogn Neurosci 22(4), 683–694.Google Scholar
  18. Kelly, S. D., Kravitz, C., & Hopkins, M. (2004). Neural correlates of bimodal speech and gesture comprehension. Brain Lang 89(1), 253–260.Google Scholar
  19. Kelly, S. D., McDevitt, T., & Esch, M. (2009). Brief training with co-speech gesture lends a hand to word learning in a foreign language. Language and cognitive processes 24(2), 313–334.Google Scholar
  20. Kelly, S. D., Ozyürek, A., & Maris, E. (2010b): Two sides of the same coin: Speech and gesture mutually interact to enhance comprehension. Psychol Sci 21(2), 260–267.Google Scholar
  21. Kelly, S. D., Ward, S., Creigh, P., & Bartolotti, J. (2007). An intentional stance modulates the integration of gesture and speech during comprehension. Brain Lang 101(3), 222–233.Google Scholar
  22. Kircher, T., Straube, B., Leube, D., Weis, S., Sachs, O., Willmes, K., Konrad, K., & Green, A. (2009). Neural interaction of speech and gesture: Differential activations of metaphoric co-verbal gestures. Neuropsychologia 47(1), 169-179.Google Scholar
  23. Krahmer, E., & Swerts, M. (2007). The effects of visual beats on prosodic prominence: Acoustic analyses, auditory perception and visual perception. Journal of Memory and Language 57(3), 396–414.Google Scholar
  24. Macedonia, M., Müller, K., & Friederici, A. D. (2011). The impact of iconic gestures on foreign language word learning and its neural substrate. Hum Brain Mapp 32(6), 982–998.Google Scholar
  25. MacSweeney, M., Woll, B., Campbell, R., McGuire, P. K., David, A. S., Williams, S. C., Suckling, J., Calvert, G. A., & Brammer, M. J. (2002). Neural systems underlying British Sign Language and audio-visual English processing in native users. Brain 125(7), 1583–1593.Google Scholar
  26. McNeill, D. (1992). Hand and mind: What gestures reveal about thought. Chicago: Univ. of Chicago Press.Google Scholar
  27. Oldfield, R. C. (1971). The assessment and analysis of handedness: The Edinburgh inventory. Neuropsychologia 9(1), 97–113.Google Scholar
  28. Ozyürek, A., Willems, R. M., Kita, S., & Hagoort, P. (2007). On-line integration of semantic information from speech and gesture: Insights from event-related brain potentials. J Cogn Neurosci 19(4), 605–616.Google Scholar
  29. Sams, M., Aulanko, R., Hämäläinen, M., Hari, R., Lounasmaa, O. V., Lu, S. T., & Simola, J. (1991): Seeing speech: Visual information from lip movements modifies activity in the human auditory cortex. Neurosci Lett 127(1), 141–145.Google Scholar
  30. Skipper, J. I., Goldin-Meadow, S., Nusbaum, H. C., & Small, S. L. (2007). Speech-associated gestures, Broca’s area, and the human mirror system. Brain Lang 101(3), 260-277.Google Scholar
  31. Skipper, J. I., Goldin-Meadow, S., Nusbaum, H. C., & Small, S. L. (2009). Gestures orchestrate brain networks for language understanding. Curr Biol 19(8), 661-667.Google Scholar
  32. Slotnick, S. D., Moo, L. R., Segal, J. B., & Hart, J., Jr. (2003). Distinct prefrontal cortex activity associated with item memory and source memory for visual shapes. Brain Res Cogn Brain Res 17(1), 75–82.Google Scholar
  33. Stein, B. E., & Meredith, M. A. (1993). The merging of the senses. Cambridge: MIT Press.Google Scholar
  34. Straube, B., Green, A., Bromberger, B., & Kircher, T. (2011a). The differentiation of iconic and metaphoric gestures: Common and unique integration processes. Hum Brain Mapp 32(4), 520–533.Google Scholar
  35. Straube, B., Green, A., Chatterjee, A., & Kircher, T. (2011b). Encoding social interactions: The neural correlates of true and false memories. J Cogn Neurosci 23(2), 306–324.Google Scholar
  36. Straube, B., Green, A., Jansen, A., Chatterjee, A., & Kircher, T. (2010). Social cues, mentalizing and the neural processing of speech accompanied by gestures. Neuropsychologia 48(2), 382–393.Google Scholar
  37. Straube, B., Green, A., Weis, S., Chatterjee, A., & Kircher, T. (2009). Memory effects of speech and gesture binding: Cortical and hippocampal activation in relation to subsequent memory performance. J Cogn Neurosci 21(4), 821–836.Google Scholar
  38. Willems, R. M., Ozyurek, A., & Hagoort, P. (2007). When language meets action: the neural integration of gesture and speech. Cereb Cortex 17(10), 2322–2333.Google Scholar
  39. Willems, R. M., Ozyurek, A., & Hagoort, P. (2009). Differential roles for left inferior frontal and superior temporal cortex in multimodal integration of action and language. Neuroimage 47(4), 1992–2004.Google Scholar
  40. Wu, Y. C., Coulson, S. (2005). Meaningful gestures: Electrophysiological indices of iconic gesture comprehension. Psychophysiology 42(6), 654–667.Google Scholar
  41. Wu, Y. C., & Coulson, S. (2007a). How iconic gestures enhance communication: An ERP study. Brain Lang 101(3), 234–245.Google Scholar
  42. Wu, Y. C., & Coulson, S. (2007b). Iconic gestures prime related concepts: An ERP study. Psychon Bull Rev 14(1), 57–63.Google Scholar
  43. Wu, Y. C., & Coulson, S. (2010). Gestures modulate speech processing early in utterances. Neuroreport 21(7), 522–526.Google Scholar

Copyright information

© Springer Fachmedien Wiesbaden GmbH 2018

Authors and Affiliations

  • Arne Nagels
    • 1
    Email author
  • Spencer D. Kelly
    • 2
  • Tilo Kircher
    • 3
  • Benjamin Straube
    • 3
  1. 1.Johannes Gutenberg University MainzMainzGermany
  2. 2.Colgate University HamiltonHamiltonUSA
  3. 3.Philipps-University MarburgMarburgGermany

Personalised recommendations