Skip to main content
SpringerLink
Log in

The Role of Iconic Gestures in Production and Comprehension of Language: Evidence from Brain and Behavior

  • Conference paper
Gesture in Embodied Communication and Human-Computer Interaction (GW 2009)

Part of the book series: Lecture Notes in Computer Science ((LNAI,volume 5934))

Included in the following conference series:

Abstract

Speakers in all cultures and ages use gestures as they speak (i.e., cospeech gestures). There have been different views in the literature with regard to whether and how a specific type of gestures speakers use, i.e., iconic gestures, interacts with language processing. Here I review evidence showing that iconic gestures are not produced merely from the spatial and/or motoric imagery but from an in interface representation of imagistic and linguistic representation during online speaking Similarly, for comprehension, neuroimaging and behavioral studies indicate that speech and gesture influences semantic processing of each other during online comprehension. These findings show overall that processing of information in both modalities interacts during both comprehension and production of language arguing against models that propose independent processing of each modality. They also have implications for AI models that aim to simulate cospeech gesture use in conversational agents.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 39.99
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. Calvert, A.: Crossmodal processing in the human brain: Insights from functional neuroimaging studies. Cerebral Cortex 11, 1110–1123 (2001)

    Article  Google Scholar 

  2. McNeill, D.: Hand and mind. University of Chicago Press, Chicago (1992)

    Google Scholar 

  3. Bavelas, J.B., Gerwing, J., Sutton, C., Prevost, D.: Gesturing on the telephone: Independent effects of dialogue and visibility. Journal of Memory and Language 58, 495–520 (2008)

    Article  Google Scholar 

  4. Özyürek, A.: Gesture in sign and spoken language. In: Pfau, R., Steinbach, M., Woll, B. (eds.) Sign language: An international handbook. Mouton, Berlin (in press)

    Google Scholar 

  5. Kendon, A.: Gesture. Cambridge University Press, Cambridge (2004)

    Google Scholar 

  6. McNeill, D.: So you think gestures are nonverbal? Psychological Review 92, 350–371 (1985)

    Article  Google Scholar 

  7. Clark, H.: Using language. Cambridge University Press, Cambridge (1996)

    Book  Google Scholar 

  8. Kita, S., Özyürek, A.: What does cross-linguistic variation in semantic coordination of speech and gesture reveal?: Evidence for an interface representation of spatial thinking and speaking. Journal of Memory and Language 48, 16–32 (2003)

    Article  Google Scholar 

  9. Hostetter, A.B., Alibali, M.W.: Visible embodiment: Gestures as simulated action. Psychonomic Bulletin and Review (2008)

    Google Scholar 

  10. Krauss, R.M., Chen, Y., Gottesman, R.: Lexical gestures and lexical access: A process model. In: McNeill, D. (ed.) Language and Gesture, pp. 261–284. Cambridge University Press, Cambridge (2000)

    Chapter  Google Scholar 

  11. Feyreisen, P., Lanoy, J.D.: Gestures and speech: Psychological investigations. Cambridge University Press, Cambridge (1991)

    Google Scholar 

  12. De Ruiter, J.P.: The production of gesture and speech. In: McNeill, D. (ed.) Language and Gesture, pp. 284–312. Cambridge University Press, Cambridge (2000)

    Chapter  Google Scholar 

  13. Mayberry, R., Jaques, J.: Gesture productino during stuttered speech: insights into the nature of speech-gesture integration. In: McNeill, D. (ed.) Language and Gesture, pp. 199–215. Cambridge University Press, Cambridge (2000)

    Chapter  Google Scholar 

  14. Özyürek, A.: Do speakers design their co-speech gestures for their addressees? The effects of addressee location on representational gestures. Journal of Memory and Language 46, 688–704 (2002)

    Article  Google Scholar 

  15. Krauss, R.M., Morrel-Samuels, P., Colasante, C.: Do conversational hand gestures communicate? Journal of Personality and Social Psychology 61, 743–754 (1991)

    Article  Google Scholar 

  16. Kelly, S., Özyürek, A., Maris, E.: Two sides of the same coin: Speech and gesture interact to enhance comprehension. Psychological Science (in press)

    Google Scholar 

  17. Özyürek, A., Kita, S.: Expressing manner and path in English and Turkish: Differences in speech, gesture, and conceptualization. In: Hahn, M., Stoness, S.C. (eds.) Proceedings of the twenty first annual conference of the Cognitive Science Society, pp. 507–512. Lawrence Erlbaum, Mahwah (1999)

    Google Scholar 

  18. Kita, S.: How representational gestures help speaking. In: McNeill, D. (ed.) Language and Gesture, pp. 261–284. Cambridge University Press, Cambridge (2000)

    Google Scholar 

  19. Talmy, L.: Semantics and syntax of motion. In: Shopen, T. (ed.) Language typology and syntactic description. Grammatical categories and the lexicon, vol. 3, pp. 57–149. Cambridge University Press, Cambridge (1985)

    Google Scholar 

  20. Bock, K.: Towards a cognitive psychology of syntax: Information processing contributions to sentence formulation. Psychological Review 89, 1–47 (1982)

    Article  Google Scholar 

  21. Levelt, P.: Speaking. MIT Press, Cambridge (1989)

    Google Scholar 

  22. Özyürek, A., Kita, S., Allen, S., Furman, R., Brown, A.: How does linguistic framing influence co-speech gestures? Insights from cross-linguistic differences and similarities. Gesture 5, 216–241 (2005)

    Article  Google Scholar 

  23. Pederson, E., Danziger, E., Wilkins, D., Levinson, S.C., Kita, S., Senft, G.: Semantic typology and spatial conceptualization. Language 74, 557–589 (1998)

    Article  Google Scholar 

  24. Kita, S., Özyürek, A., Allen, S., Brown, A., Furman, R., Ishizuka, T.: Relations between syntactic encoding and co-speech gestures: Implications for a model of speech and gesture production. Language and Cognitive Processes 22(8), 1212–1236 (2007)

    Article  Google Scholar 

  25. Beattie, G., Shovelton, H.: Do iconic hand gestures really contribute anything to the semantic information conveyed by speech? An experimental investigation. Semiotica 123, 1 (1999)

    Article  Google Scholar 

  26. Özyürek, A., Willems, R., Kita, S., Hagoort, P.: On-line integration of information from speech and gesture: Insight from event-related potentials. Journal of Cognitive Neuroscience 19(4), 605–616 (2007)

    Article  Google Scholar 

  27. Willems, R., Özyürek, A., Hagoort, P.: When language meets action. The neural integration of speech and gesture. Cerebral Cortex 17, 2322–2333 (2007)

    Article  Google Scholar 

  28. Hagoort, P.: How the brain solves the binding problem for language: a neurocomputational model of syntactic processing. Neuroimage 20, S18–S29 (2003)

    Article  Google Scholar 

  29. Hagoort, P., Hald, L., Bastiaansen, M., Petersson, K.M.: Integration of word meaning and world knowledge in language comprehension. Science 304, 438–441 (2004)

    Article  Google Scholar 

  30. Kopp, S., Bergmann, K., Wachsmuth, I.: Multimodal communication from multimodal thinking – Towards an integrated model of speech and gesture production. Semantic Computing 2(1), 115–136 (2008)

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Center for Language Studies & Max Planck Institute for Psycholinguistics, Radboud University Nijmegen,  

    Asli Özyürek

Authors
  1. Asli Özyürek
    View author publications

    You can also search for this author in PubMed Google Scholar

Editor information

Editors and Affiliations

  1. CITEC, Bielefeld University, P.O. Box 100131, 33501, Bielefeld, Germany

    Stefan Kopp

  2. Artificial Intelligence Group, Faculty of Technology, Bielefeld University, 33594, Bielefeld, Germany

    Ipke Wachsmuth

Rights and permissions

Reprints and permissions

Copyright information

© 2010 Springer-Verlag Berlin Heidelberg

About this paper

Cite this paper

Özyürek, A. (2010). The Role of Iconic Gestures in Production and Comprehension of Language: Evidence from Brain and Behavior. In: Kopp, S., Wachsmuth, I. (eds) Gesture in Embodied Communication and Human-Computer Interaction. GW 2009. Lecture Notes in Computer Science(), vol 5934. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-12553-9_1

Download citation

  • DOI: https://doi.org/10.1007/978-3-642-12553-9_1

  • Publisher Name: Springer, Berlin, Heidelberg

  • Print ISBN: 978-3-642-12552-2

  • Online ISBN: 978-3-642-12553-9

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation