Abstract
Previous research suggests that speakers gesture more when they are describing information learned in a visuo-spatial format than when describing information learned in a verbal format. The present study examined whether this difference in gesture is due to decreased verbal accessibility when information has been learned visuo-spatially or to the decreased accessibility of mental images when information has been learned verbally. Participants described the events of stories learned in two conditions. In one condition, the participants heard a verbal description twice. In the other condition, the participants heard the verbal description once and then watched an animated cartoon of the events. We found increased representational gestures when speakers had watched the cartoon in addition to hearing its description. Importantly, this increase in gesture does not appear to be due to differences in speech accessibility or output, suggesting that speakers gesture when they have a clear image in mind and not just when they do not have ready access to verbal labels.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Alibali, M. W. (2005). Gesture in spatial cognition: Expressing, communicating, and thinking about spatial information. Spatial Cognition and Computation, 5, 307–331.
Alibali, M. W., Bassok, M., Solomon, K., Syc, S. E., & Goldin-Meadow, S. (1999). Illuminating mental representations through speech and gesture. Psychological Science, 10, 327–333.
Alibali, M. W., Heath, D. C., & Myers, H. J. (2001). Effects of visibility between speaker and listener on gesture production. Journal of Memory and Language, 44, 169–188.
Alibali, M. W., Kita, S., & Young, A. J. (2000). Gesture and the process of speech production: We think, therefore, we gesture. Language and Cognitive Processes, 15, 593–613.
Barsalou, L. W. (1999). Perceptual symbol systems. Behavioral and Brain Sciences, 22, 577–660.
Bavelas, J., Gerwing, J., Sutton, C., & Prevost, D. (2008). Gesturing on the telephone: Independent effects of dialogue and visibility. Journal of Memory and Language, 58, 495–520.
Bavelas, J., Kenwood, C., Johnson, T., & Phillips, B. (2002). An experimental study of when and how speakers use gestures to communicate. Gesture, 2, 1–18.
Beattie, G., & Shovelton, H. (1999). Do iconic hand gestures really contribute anything to the semantic information conveyed by speech? An experimental investigation. Semiotica, 123, 1–30.
Bortfeld, H., Leon, S. D., Bloom, J. E., Schober, M. F., & Brennan, S. E. (2001). Disfluency rates in conversation: Effects of age, relationship, topic, role, and gender. Langauge and Speech, 44(2), 123–147.
Butterworth, G., & Hadar, U. (1989). Gesture, speech, and computational stages: A reply to McNeill. Psychological Review, 96, 168–174.
Chartrand, T. L., & Bargh, J. A. (1999). The chameleon effect: The perception-behavior link and social interaction. Journal of Personality and Social Psychology, 76, 893–910.
Chawla, P., & Krauss, R. M. (1994). Gesture and speech in spontaneous and rehearsed narratives. Journal of Experimental Social Psychology, 30, 580–601.
Christenfeld, N., Schachter, S., & Bilous, F. (1991). Filled pauses and gestures: It’s not coincidence. Journal of Psycholinguistic Research, 20, 1–10.
de Ruiter, J. P. (2000). The production of gesture and speech. In D. McNeill (Ed.), Language and gesture (pp. 284–311). Cambridge: Cambridge University Press.
Feyereisen, P., & Havard, I. (1999). Mental imagery and production of hand gestures while speaking in younger and older adults. Journal of Nonverbal Behavior, 23, 153–171.
Fridlund, A. J., Kenworthy, K. G., & Jaffey, A. K. (1992). Audience effects in affective imagery: Replication and extension to dysphoric imagery. Journal of Nonverbal Behavior, 16(3), 191–212.
Glenberg, A. M. (1997). What memory is for. Behavioral and Brain Sciences, 20, 1–55.
Glenberg, A. M., & Kaschak, M. P. (2002). Grounding language in action. Psychonomic Bulletin & Review, 9, 558–565.
Goodale, M. A. (2001). Different spaces and different times for perception and action. Progress in Brain Research, 134, 313–331.
Graham, J. A., & Argyle, M. (1975). A cross-cultural study of the communication of extra-verbal meaning by gestures. International Journal of Psychology, 10, 57–67.
Hostetter, A. B. (2011). When do gestures communicate? A meta-analysis. Psychological Bulletin, 137, 297–315.
Hostetter, A. B., & Alibali, M. W. (2007). Raise your hand if you’re spatial: Relations between verbal and spatial skills and representational gesture production. Gesture, 7, 73–95.
Hostetter, A. B., & Alibali, M. W. (2008). Visible embodiment: Gestures as simulated action. Psychonomic Bulletin & Review, 15, 495–514.
Hostetter, A. B., & Alibali, M. W. (2010). Language, gesture, action! A test of the Gesture as Simulated Action framework. Journal of Memory and Language, 63, 245–257.
Hostetter, A. B., Alibali, M. W., & Kita, S. (2007). I see it in my hands’ eye: Representational gestures reflect conceptual demands. Language and Cognitive Processes, 22, 313–336.
Hostetter, A. B., & Hopkins, W. D. (2002). The effect of thought structure on the production of lexical movements. Brain and Language, 82, 22–29.
Kita, S. (2000). How representational gestures help speaking. In D. McNeill (Ed.), Language and gesture (pp. 162–185). Cambridge: Cambridge University Press.
Kita, S., & Davies, T. S. (2009). Competing conceptual representations trigger co-speech representational gestures. Language and Cognitive Processes, 24(5), 761–775.
Kita, S., & Özyürek, A. (2003). What does cross-linguistic variation in semantic coordination of speech and gesture reveal? Evidence for an interface representation of thinking and speaking. Journal of Memory and Language, 48, 16–32.
Kosslyn, S. M. (2005). Mental images and the brain. Cognitive Neuropsychology, 22, 333–347.
Krauss, R. M. (1998). Why do we gesture when we speak? Current Directions in Psychological Science, 7, 54–60.
Krauss, R. M., Chen, Y., & Gottesman, R. F. (2000). Lexical gestures and lexical access: A process model. In D. McNeill (Ed.), Language and gesture (pp. 261–283). Cambridge: Cambridge University Press.
McNeill, D. (1992). Hand and mind: What gestures reveal about thought. Chicago: University of Chicago Press.
Melinger, A., & Kita, S. (2007). Conceptualization load triggers gesture production. Language and Cognitive Processes, 22, 473–500.
Morsella, E., & Krauss, R. M. (2004). The role of gestures in spatial working memory and speech. The American Journal of Psychology, 117, 411–424.
Paivio, A. (1991). Dual coding theory: Retrospect and current status. Canadian Journal of Psychology, 45, 255–287.
Parrill, F., Bullen, J., & Hoburg, H. (2010). Effects of input modality on gesture-speech Integration. Journal of Pragmatics, 42, 3130–3137.
Pine, K. J., Gurney, D. J., & Fletcher, B. (2010). The semantic specificity hypothesis: When gestures do not depend upon the presence of a listener. Journal of Nonverbal Behavior, 34, 169–178.
Rauscher, F., Krauss, R. M., & Chen, Y. (1996). Gesture, speech, and lexical access: The role of lexical movements in speech production. Psychological Science, 7, 226–231.
Reichle, E. D., Carpenter, P. A., & Just, M. A. (2000). The neural bases of strategy and skill in sentence-picture verification. Cognitive Psychology, 40, 261–295.
Roberts, P. M., Meltzer, A., & Wilding, J. (2009). Disfluencies in non-stuttering adults across sample lengths and topics. Journal of Communication Disorders, 42, 414–427.
Schachter, S., Christenfeld, N., Ravina, B., & Bilous, F. (1991). Speech disfluency and the structure of knowledge. Journal of Personality and Social Psychology, 60, 362–367.
Spivey, M. J., Richardson, D. C., & Gonzales-Marquez, M. (2005). On the perceptual-motor and image-schematic infrastructure of language. In D. Pecher & R. A. Zwaan (Eds.), Grounding cognition: The role of perception and action in memory, language, and thinking (pp. 246–281). Cambridge: Cambridge University Press.
Tucker, M., & Ellis, R. (1998). On the relations between seen objects and components of potential actions. Journal of Experimental Psychology: Human Perception and Performance, 24, 830–846.
Wagner, S., Nusbaum, H., & Goldin-Meadow, S. (2004). Probing the mental representation of gesture: Is hand-waving spatial? Journal of Memory and Language, 50, 395–407.
Willems, R. M., Hagoort, P., & Casasanto, D. (2010). Body-specific representations of action verbs: Neural evidence from right- and left-handers. Psychological Science, 21, 67–74.
Zwaan, R. A., Stanfield, R. A., & Yaxley, R. H. (2002). Language comprehenders mentally represent the shapes of objects. Psychological Science, 13, 168–171.
Acknowledgments
This work was initiated as part of Catherine J. Skirving’s senior individualized project at Kalamazoo College. We thank Justine Dolorfino, Emily Patzelt, Andrea Potthoff, and Carmen Wilson for their assistance with data collection and transcription. We also thank Keith Moreno for assistance with coding unfilled pauses, and Chelsea Baumgarten, Andrea Potthoff, and Erin Sullivan for their comments on the manuscript.
Author information
Authors and Affiliations
Corresponding author
Appendix
Appendix
Pancake
The mouse is making a pancake in a pan on the stove. He tries to flip the pancake by tossing it out of the pan and into the air, but it falls on the ground. The mouse picks up the pancake and continues to cook it. The mouse flips the pancake and it falls on the floor again. The mouse then continues cooking the pancake, flips it again, and it lands on the floor for the third time. The mouse picks up the pancake and begins to cook it again. The mouse flips the pancake for a fourth time and it lands on top of the mouse’s face. He slides the pancake off his face and back into the pan. The mouse then whistles for the elephant who enters the room. The mouse takes another pan off of the wall and hands it to the elephant to hold in his trunk. The mouse signals for the elephant to back up a little bit, which the elephant does. The mouse then flips the pancake and it lands in the elephant’s pan. The elephant flips the pancake back to the mouse’s pan, and they continue to happily flip the pancake back and forth.
High Bar
The Mouse stands looking at a gymnastics high bar and walks up to stand underneath it. He stretches his body by squatting up and down and pumping his arms to the side, and then he jumps up and hangs on to the bar with both hands. The Mouse starts to swing back and forth slowly, until he gets more momentum and is spinning in complete circles around the bar. The elephant enters and watches the mouse spinning. Then the mouse dismounts and the elephant applauds by stomping his front and back feet back and forth. The elephant walks up to the bar and jumps up trying to grab it. He finally gets high enough to grab it with his trunk, and the bar bends in the middle. The mouse lets out a noise of frustration and tries to straighten the bar. A little green man with a tall hat enters and skips underneath the bar. His hat touches the bar and it is magically fixed. The elephant laughs at the mouse.
Banana
The mouse pulls a banana out of his back pocket. He tosses it in the air, catches it, and then peels it. He throws the peel into a nearby basket, but as he takes a bite of the banana, the peel bounces out of the basket and comes back to hit him in the face. He tosses the peel towards the basket again, but again as he takes a bite of the banana, the peel bounces back out of the basket and onto his face. He finishes eating his banana and walks over to the basket. He places the peel inside the basket and then turns the basket upside down. As the mouse walks away, the basket lifts up to reveal the elephant’s four blue feet. The elephant makes a honking noise from under the basket and follows the mouse out of the scene.
Pottery
The mouse walks up to a potter’s wheel. He places a bucket full of clay on the bench behind the wheel and then sits down on the bench next to it. The mouse takes some clay out of the bucket and places it on the wheel. He spins the wheel with his foot, and the clay flies off the wheel and hits the wall on his right. The mouse puts more clay on the wheel and spins it again with his foot. The clay again flies off the wheel and this time hits the wall on his left. The mouse then sits on the wheel himself and begins to spin around. He spins so fast that he flies off of the wheel to his left.
Practice: Circus
The big orange mouse is dressed up like the ringmaster at a circus. He has a hoop that he waves back and forth several times. Then the little blue elephant comes out dressed up like a clown. The mouse holds out the hoop, and the elephant jumps through it. He stands up on his back legs as though taking a bow. Then the elephant jumps up on a small pedestal and does a handstand. He bounces back to upright and there are noises in the background as though an audience is cheering.