Communicative Gestures Facilitate Problem Solving for Both Communicators and Recipients

  • Sandra C. Lozano
  • Barbara Tversky
Part of the Studies in Computational Intelligence book series (SCI, volume 64)

Summary. Gestures are a common, integral part of communication. Here, we investigate the roles of gesture and speech in explanations, both for communicators and recipients. Communicators explained how to assemble a simple object, using either speech with gestures or gestures alone. Gestures used for explaining included pointing and exhibiting to indicate parts, action models to demonstrate assembly, and gestures used to convey narrative structure. Communicators using gestures alone learned assembly better, making fewer assembly errors than those communicating via speech with gestures. Recipients understood and learned better from gesture-only instructions than from speech-only instructions. Gestures demonstrating action were particularly crucial, suggesting that superiority of gestures to speech may reside, at least in part, in compatibility between gesture and action.


Hand Gesture Action Information Assembly Error Assembly Performance Assembly Task 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Alibali, M.M., Kita, S., Young, A.: Gesture and the process of speech production: We think, therefore we gesture. Language and Cognitive Processes 15 (2000) 593-613CrossRefGoogle Scholar
  2. 2.
    Alibali, M.W., DiRusso, A.A.: The function of gesture in learning to count: More than keeping track. Cognitive Development 14 (1999) 37-56CrossRefGoogle Scholar
  3. 3.
    Alibali, M.W., Bassok, M., Olseth Solomon, K., Syc, S.E., Goldin-Meadow, S.: Illuminating mental representation through speech and gesture. Psychological Science 10 (1999) 327-333CrossRefGoogle Scholar
  4. 4.
    Alibali, M.W., Heath, D., Myers, H.: Effects of visibility between speaker and listener on gesture production: Some gestures are meant to be seen. Journal of Memory and Language 44 (2001) 169-188CrossRefGoogle Scholar
  5. 5.
    Anderson, R.C., Ortony, A.: (1975). On putting apples into bottles: A problem of polysemy. Cognitive Psychology, 7, 167-180CrossRefGoogle Scholar
  6. 6.
    Barsalou, L.W.: Perceptual symbol systems. Behavioral and Brain Sciences 22 (1999) 577-609Google Scholar
  7. 7.
    Clark, H.H.: Using language. Cambridge: Cambridge University Press. (1996)Google Scholar
  8. 8.
    Clark, H.H., Krych, M.A.: Speaking while monitoring addressees for understand- ing. Journal of Memory and Language 50 (2004) 62-81CrossRefGoogle Scholar
  9. 9.
    Cohen, A.A.: The communicative function of hand gestures. Journal of Com- munication 27 (1977) 54-63CrossRefGoogle Scholar
  10. 10.
    Cohen, A.A., Harrison, R. P.: Intentionality in the use of hand illustrators in face-to-face communication situations. Journal of Personality and Social Psy-chology 28 (1973) 276-279CrossRefGoogle Scholar
  11. 11.
    Cohen, J.D., MacWhinney, B., Flatt, M., Provost, J.: PsyScope: An interactive graphic system for designing and controlling experiments in the psychology lab-oratory using Macintosh computers. Behavior Research, Methods, Instruments & Computers 25 (1993) 257-271Google Scholar
  12. 12.
    Dittmann, A.T., Llewelyn, L.G.: Body movement and speech rhythm in social conversation. Journal of Personality and Social Psychology 23 (1969) 283-292Google Scholar
  13. 13.
    Emmorey, K., Tversky, B., Taylor, H.A.: Using space to describe space: Perspec-tive in speech, sign, and gesture. Journal of Spatial Cognition and Computation 2 (2000) 157-180CrossRefGoogle Scholar
  14. 14.
    Engle, R.A.: Not channels but composite signals: Speech, gesture, diagrams, and object demonstrations are integrated in multimodal explanations. In M.A. Gernsbacher & S.J. Derry, eds., Spatial language: Cognition and computational perspectives. Kluwer Academic, Dordrecht, The Netherlands (1998)Google Scholar
  15. 15.
    Frick-Horbury, D., Guttentag, R.E.: The effects of restricting hand gesture pro-duction on lexical retrieval and free recall. American Journal of Psychology 111 (1998) 43-62CrossRefGoogle Scholar
  16. 16.
    Glenberg, A.M.: What memory is for. Behavioral and Brain Sciences 2 (1997) 1-55Google Scholar
  17. 17.
    Glenberg, A.M., Kaschak, M.P.: Grounding language in action. Psychonomic Bulletin & Review 9 (2002) 558-565Google Scholar
  18. 18.
    Graham, J.A., Argyle, M.: A cross-cultural study of the communication of extra-verbal meaning by gestures. International Journal of Psychology 10 (1975) 57-67CrossRefGoogle Scholar
  19. 19.
    Graham, J.A., Heywood, S.: The effects of elimination of hand gestures and of verbal codability on speech performance. European Journal of Social Psychology 5 (1975) 189-195CrossRefGoogle Scholar
  20. 20.
    Goldin-Meadow, S.: Hearing gesture: How our hands help us think. Belknap Press, Cambridge (2003)Google Scholar
  21. 21.
    Goldin-Meadow, S., Alibali, M.W., Church, R.B.: Transitions in concept acquisi-tion: Using the hand to read the mind. Psychological Review 100 (1993) 279-297CrossRefGoogle Scholar
  22. 22.
    Haviland, J.: Pointing, gesture spaces, and mental maps. In D. McNeill (Ed.), Language and gesture. Cambridge: Cambridge University Press (2000) 13-46Google Scholar
  23. 23.
    Heiser, J., Phan, D., Agrawala, M., Tversky, B., Hanrahan, P.: Identification and validation of cognitive design principles for automated generation of assembly instructions. Proceedings of Advanced Visual Interfaces (2004) 311-319Google Scholar
  24. 24.
    Heiser, J., Tversky, B.: Descriptions and depictions in acquiring complex sys-tems. In Proceedings of the 24th annual meeting of the Cognitive Science Society. Fairfax, VA. (2002)Google Scholar
  25. 25.
    Heiser, J., Tversky, B.: Cognitive design principles for visualizations: Revealing and instantiating. In Proceedings of the 25th Annual Meeting of the Cognitive Science Society. Boston, MA. (2003)Google Scholar
  26. 26.
    Hewes, G.: Primate communication and the gestural origins of language. Current Anthropology 14 (1973) 5-24CrossRefGoogle Scholar
  27. 27.
    Kendon, A.: Gesture: Visible action in utterance. New York: Cambridge University Press. (2004)Google Scholar
  28. 28.
    Kita, S., Ö zyürek, A.: What does cross-linguistic variation in semantic coordi-nation of speech and gesture reveal?: Evidence for an interface representation of spatial thinking and speaking. Journal of Memory and Language 48 (2003) 16-32CrossRefGoogle Scholar
  29. 29.
    Krauss, R.M.: Why do we gesture when we speak? Current Directions in Psy-chological Science 7 (1998) 54-59CrossRefGoogle Scholar
  30. 30.
    Krauss, R.M., Hadar, U.: The role of speech-related arm/hand gestures in word retrieval. In L. Messing & R. Campbell (Eds.), Gesture, speech, and sign. New York, NY: Oxford University Press. (1999) 93-116Google Scholar
  31. 31.
    Krauss, R.M., Chen, Y., Chawla, P.: Nonverbal behavior and nonverbal commu-nication: What do conversational hand gestures tell us? In M.P. Zanna (Ed.). Advances in experimental social psychology 28. San Diego, CA: Academic Press. (1996)Google Scholar
  32. 32.
    Levinson, S.: Pragmatics. Cambridge: Cambridge University Press. (1983)Google Scholar
  33. 33.
    McNeill, D.: Hand and mind: What gestures reveal about thought. University of Chicago Press, Chicago (1992)Google Scholar
  34. 34.
    McNeill, D.: Analogic/analytic representations and cross-linguistic differences in thinking for speaking. Cognitive Linguistics Special Issue: Language Acquisition 11 (2000) 43-60Google Scholar
  35. 35.
    Özyürek, A.: So speakers design their cospeech gestures for their addressees. The effects of addressee location on representational gestures. Journal of Memory and Language 46 (2002) 688-704CrossRefGoogle Scholar
  36. 36.
    Quintilian: In H.E. Butler (Ed. and Trans.), Institutes of oratory. London: William Heinemann Press. (1920)Google Scholar
  37. 37.
    Rauscher, F.H., Krauss, R.M., Chen, Y.: Gesture, speech, and lexical access: The role of lexical movements in speech production. Psychological Science 7 (1996) 226-231CrossRefGoogle Scholar
  38. 38.
    Riseborough, M.G.: Physiographic gestures as decoding facilitators: Three experiments exploring a neglected facet of communication. Journal of Nonverbal Behavior 5 (1981) 172-183CrossRefGoogle Scholar
  39. 39.
    Rogers, W.T.: The contribution of kinesic illustrators toward the comprehension of verbal behavior within utterances. Human Communication Research 5 (1978) 54-62CrossRefGoogle Scholar
  40. 40.
    Rogers, W.T.: The relevance of body motion cues to both functional and dys-functional communicative behavior. Journal of Communication Disorders 12 (1979) 273-282CrossRefGoogle Scholar
  41. 41.
    Schooler, J.W., Engstler-Schooler, T.Y.: Verbal overshadowing of visual memo-ries: Some things are better left unsaid. Cognitive Psychology 22 (1990) 36-71Google Scholar
  42. 42.
    Tucker, M., Ellis, R.: On the relations between seen objects and components of potential actions. Journal of Experimental Psychology: Human Perception and Performance 24 (1998) 830-846CrossRefGoogle Scholar
  43. 43.
    Tversky, B.: Spatial schemas in depictions. In M. Gattis, ed., Spatial schemas and abstract thought. MIT Press, Cambridge (2001) 79-111Google Scholar
  44. 44.
    Vandenburg, S.G., Kuse, A.R.: Mental rotations, a group test of three-dimensional spatial visualization. Perceptual and Motor Skills 47 (1978) 599-604Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2007

Authors and Affiliations

  • Sandra C. Lozano
    • 1
  • Barbara Tversky
    • 2
  1. 1.Department of PsychologyStanford UniversityStanfordUSA
  2. 2.Department of PsychologyStanford UniversityStanfordUSA

Personalised recommendations