Skip to main content
SpringerLink
Log in

The Role of Representational Gestures and Speech Synchronicity in Auditory Input by L2 and L1 Speakers

  • Published:
Journal of Psycholinguistic Research Aims and scope Submit manuscript

Abstract

Speech and gesture are two integrated and temporally coordinated systems. Manual gestures can help second language (L2) speakers with vocabulary learning and word retrieval. However, it is still under-investigated whether the synchronisation of speech and gesture has a role in helping listeners compensate for the difficulties in processing L2 aural information. In this paper, we tested, in two behavioural experiments, how L2 speakers process speech and gesture asynchronies in comparison to native speakers (L1). L2 speakers responded significantly faster when gestures and the semantic relevant speech were synchronous than asynchronous. They responded significantly slower than L1 speakers regardless of speech/gesture synchronisation. On the other hand, L1 speakers did not show a significant difference between asynchronous and synchronous integration of gestures and speech. We conclude that gesture-speech asynchrony affects L2 speakers more than L1 speakers.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3

Similar content being viewed by others

Data Availability

Experiment 1 stimuli can be found at doi: https://doi.org/10.6084/m9.figshare.13421336.

Experiment 2 stimuli are available at doi: https://doi.org/10.6084/m9.figshare.13421495.

The data sets of experiment 1 and 2 are available at doi: https://doi.org/10.6084/m9.figshare.13377161.

Notes

  1. Videos for experiment 1 are at DOI: https://doi.org/10.6084/m9.figshare.13421336.

  2. The Common European Framework of Reference for Languages (CEFR) is a guideline used to describe foreign language learners’ language proficiency. It has six reference levels: from A1 for beginners to C2 for proficient learners. B2 and C1 correspond to upper-intermediate to low-advanced, respectively.

  3. The data sets for experiments 1 and 2 can be found at DOI: https://doi.org/10.6084/m9.figshare.13377161.

  4. The videos for experiment 2 are at DOI: https://doi.org/10.6084/m9.figshare.13421495.

References

  • Alibali, M. W., Kita, S., & Young, A. J. (2000). Gesture and the process of speech production: We think, therefore we gesture. Language and Cognitive Processes, 15, 593–613.

    Article  Google Scholar 

  • Allen, L. Q. (1995). The effects of emblematic gestures on the development and access of mental representations of french expressions. Modern Language Journal, 79, 521–529.

    Article  Google Scholar 

  • Anikin, A., Nirme, J., Alomari, S., Bonnevier, J., & Haake, M. (2015). Compensation for a large gesture-speech asynchrony in instructional videos. In G. Ferré & M. Tutton (Eds.), Gesture and Speech in Interaction – 4th edition (GESPIN 4, pp. 19–23).

  • Bäckman, L., & Nilsson, L. G. (1984). Aging effects in free recall: An exception to the rule. Human Learning, 3, 53–69.

    Google Scholar 

  • Baroni, M., Bernardini, S., Comastri, F., Piccioni, L., Volpi, A., Aston, G. (2004). Introducing the la Repubblica corpus: a large, annotated, TEI(XML)-compliant corpus of newspaper Italian. In Proceedings of LREC 2004 (pp. 1771–1774). ELDA.

  • Barr, D. J., Levy, R., Scheepers, C., & Tily, H. J. (2013). Random effects structure for confirmatory hypothesis testing: Keep it maximal. Journal of Memory and Language, 68, 255–278.

    Article  Google Scholar 

  • Bates, D., Maechler, M., Bolker, B., & Walker, S. (2014). lme4: Linear mixed-effects models using Eigen and S4 (R package version 1.1-7) [Statistical software]. Available: CRAN.R-project.org/package = lme4.

  • Beattie, G., & Shovelton, H. (2000). Iconic hand gestures and the predictability of words in context in spontaneous speech. British Journal of Psychology, 91, 473–492.

    Article  PubMed  Google Scholar 

  • Beattie, G., & Coughlan, J. (1999). An experimental investigation of the role of iconic gestures in lexical access using the tip-of-the-tongue phenomenon. British Journal of Psychology, 90, 35–56.

    Article  PubMed  Google Scholar 

  • Butterworth, B., & Beattie, G. (1978). Gesture and silence as indicators of planning in speech. In R. Campbell, & P. T. Smith (Eds.), Recent advances in the psychology of language: Formal and experimental approaches (pp. 347–360). Plenum.

  • Butterworth, B., & Hadar, U. (1989). Gesture, speech, and computational stages: A reply to McNeill. Psychological Review, 96, 168–174.

    Article  PubMed  Google Scholar 

  • Campbell, R., & Dodd, B. (1980). Hearing by eye. Quarterly Journal of Experimental Psychology, 32, 85–99.

    Article  PubMed  Google Scholar 

  • Chui, K. (2005). Temporal patterning of speech and iconic gestures in conversational discourse. Journal of Pragmatics, 37, 871–887.

    Article  Google Scholar 

  • Cohen, R. L. (1981). On the generality of some memory laws. Scandinavian Journal of Psychology, 22, 267–281.

    Article  Google Scholar 

  • Cohen, R. L., & Stewart, M. (1982). How to avoid developmental effects in free recall. Scandinavian Journal of Psychology, 23, 9–15.

    Article  Google Scholar 

  • Cohen, R. L. (1983). The effect of encoding variables on the free recall of words and action events. Memory & Cognition, 11, 575–582.

    Article  Google Scholar 

  • Cohen, R. L., Sandler, S. P., & Schroeder, K. (1987). Aging and memory for words and action events: Effects of item repetition and list length. Psychology and Aging, 2, 280–285.

    Article  PubMed  Google Scholar 

  • Cook, S. W., Yip, T. K., & Goldin-Meadow, S. (2010). Gesturing makes memories that last. Journal of Memory and Language, 63, 465–475.

    Article  PubMed  PubMed Central  Google Scholar 

  • Davies, M. (2013). Corpus of News on the Web (NOW): 3 + billion words from 20 countries, updated every day. Available online at https://corpus.byu.edu/now/.

  • de Ruiter, J. P. (2000). The production of gesture and speech. In D. McNeill (Ed.), Language and gesture (pp. 284–311). Cambridge University Press.

  • Engelkamp, J., & Krumnacker, H. (1980). Imaginale und motorische Prozesse beim Behalten verbalen Materials Zeitschrift für Experimentelle und Angewandte Psychologie, 27, 511–533.

  • Ekman, P., & Friesen, W. V. (1969). The repertoire of Nonverbal Behavior: Categories, Origins, usage, and Coding (1 vol., pp. 49–98). Semiotica.

  • Esteve-Gibert, N., & Prieto, P. (2014). Infants temporally coordinate gesture–speech combinations before they produce their first words. Speech Communication, 57, 301–316.

    Article  Google Scholar 

  • Esteve-Gibert, N., & Prieto, P. (2015). Nine-month-old infants are sensitive to the temporal alignment of prosodic and gesture prominences. Infant Behaviour & Development, 38, 126–129.

    Article  Google Scholar 

  • Feyereisen, P. (2009). Enactment effects and integration processes in younger and older adults’ memory for actions. Memory (Hove, England), 17, 374–385.

    Article  PubMed  Google Scholar 

  • Frick-Horbury, D., & Guttentag, R. E. (1998). The effects of restricting hand gesture production on lexical retrieval and free recall. The American Journal of Psychology, 111, 43–62.

    Article  Google Scholar 

  • Frick-Horbury, D. (2002a). The effects of hand gestures on verbal recall as a function of high- and-low-verbal-skill levels. Journal of General Psychology, 129, 137–147.

    Article  PubMed  Google Scholar 

  • Frick-Horbury, D. (2002b). The use of hand gestures as self-generated cues for recall of verbally associated targets. The American Journal of Psychology, 115, 1–20.

    Article  PubMed  Google Scholar 

  • Giorgolo, G., & Verstraten, F. A. J. (2008). Perception of speech-and-gesture integration. In AVSP-2008, International Conference on Auditory-Visual Speech Processing (pp. 31 – 6).

  • Graziano, M., & Gullberg, M. (2018). When speech stops, gesture stops: Evidence from cross-linguistic and developmental comparisons. Frontiers in Psychology, 9, 879.

    Article  PubMed  PubMed Central  Google Scholar 

  • Graziano, M., Nicoladis, E., & Marentette, P. (2020). How referential gestures align with speech: Evidence from monolingual and bilingual speakers. Language Learning, 70, 266–304.

    Article  Google Scholar 

  • Gullberg, M. (2006). Handling discourse: Gestures, reference Tracking, and communication strategies in early L2. Language Learning, 56, 155–196.

    Article  Google Scholar 

  • Habets, B., Kita, S., Shao, Z., Özyurek, A., & Hagoort, P. (2011). The role of synchrony and ambiguity in speech-gesture integration during comprehension. Journal of Cognitive Neuroscience, 23, 1845–1854.

    Article  PubMed  Google Scholar 

  • Helstrup, T. (1984). Serial position phenomena: Memory for acts, contents and spatial position patterns. Scandinavian Journal of Psychology, 25, 131–146.

    Article  Google Scholar 

  • Hopp, H. (2010). Ultimate attainment in L2 inflection: Performance similarities between non-native and native speakers. Lingua, 120, 901–931.

    Article  Google Scholar 

  • Kelly, S. D., McDevitt, T., & Esch, M. (2009). Brief training with co-speech gesture lends a hand to word learning in a foreign language. Language and Cognitive Processing, 24, 313–334.

    Article  Google Scholar 

  • Kendon, A. (1980). Gesture and speech: Two aspects of the process of utterance. In M. R. Key (Ed.), Nonverbal communication and language (pp. 207 – 27). Mouton.

  • Kendon, A. (2000). Language and gesture. Unity or duality. In D. McNeill (Ed.), Language and gesture (pp. 47–63). Cambridge University Press.

  • Kendon, A. (2004). Gesture: Visible action as utterance. Cambridge University Press.

  • Kirchhof, C. (2014). Desynchronized speech-gesture signals still get the message across. The 7th International Conference on Multimodality (7ICOM), Hongkong. Retrieved from: https://pub.uni-bielefeld.de/download/2522747/2901722/ISGS_Caro_presented.pdf

  • Kita, S. (2000). How representational gestures help speaking. In D. McNeill (Ed.), Language and gesture (pp. 162–185). Cambridge University Press.

  • Kita, S., & Ozyurek, A. (2003). What does cross-linguistic variation in semantic coordination of speech and gesture reveal? Evidence for an interface representation of spatial thinking and speaking. Journal of Memory and Language, 48, 16–32.

    Article  Google Scholar 

  • Krauss, R. M., & Hadar, U. (1999). The role of speech-related arm/hand gestures in word retrieval. In L. S. Messing & R. Campbell (Eds.), Gesture, speech, and sign (pp. 93–116). Oxford University Press. https://doi.org/10.1093/acprof:oso/9780198524519.003.0006

  • Lantolf, J. P. (2010). Minding your hands: The function of gesture in L2 learning. In R. Batstone (Ed.), Sociocognitive perspectives on language use and language learning (pp. 131–150). Oxford University Press.

  • Loehr, D. (2007). Aspects of rhythm in gesture and speech. Gesture, 7, 179–214.

    Article  Google Scholar 

  • Lopez-Ozieblo, R. (2019). Cut-offs and co-occurring gestures: Similarities between speakers’ first and second languages.IRAL -. International Review of Applied Linguistics in Language Teaching, 4, 219–265.

  • Macedonia, M., & Knösche, T. R. (2011). Body in Mind: How Gestures Empower Foreign Language Learning. Mind, Brain, and Education, 5, 196–211.

  • McCafferty, S. G. (1998). Nonverbal expression and L2 private speech. Applied Linguistics, 19, 73–96.

    Article  Google Scholar 

  • McDonald, J. L. (2006). Beyond the critical period: Processing-based explanations for poor grammaticality judgment performance by late second language learners. Journal of Memory and Language, 55, 381–401.

    Article  Google Scholar 

  • McNeill, D. (1985). So you think gestures are nonverbal? Psychological Review, 92, 271–295.

    Article  Google Scholar 

  • McNeill, D. (1992). Hand and mind: What gestures reveal about thought. University of Chicago Press.

  • McNeill, D. (Ed.). (2000).Language and gesture.Cambridge University Press.

  • McNeill, D. (2015). Why we gesture. Cambridge University Press.

  • McNeill, D., & Duncan, S. D. (2000). Growth points in thinking-for-speaking. In D. McNeill (Ed.), Language and gesture (pp. 141–161). Cambridge University Press.

  • Morett, L. M. (2018). Hand and in mind: Effects of gesture production and viewing on second language word learning. Applied Psycholinguistics, 39, 355–381.

    Article  Google Scholar 

  • Mulligan, N. W., & Hornstein, S. L. (2003). Memory for actions: Self-performed tasks and the reenactment effect. Memory & Cognition, 31, 412–421.

    Article  Google Scholar 

  • Navarra, J., Alsius, A., Soto-Faraco, S., & Spence, C. (2010). Assessing the role of attention in the audio-visual integration of speech. Information Fusion, 11, 4–11.

    Article  Google Scholar 

  • Nobe, S. (1993). Cognitive processes of speaking and gesturing: A comparison between first language speakers and foreign language speakers Masters thesis, Dept. of Psychology, University of Chicago.

  • Obermeier, C., Holle, H., & Gunter, T. C. (2011). What iconic gesture fragments reveal about Gesture–Speech Integration: When Synchrony is lost, memory can help. Journal of Cognitive Neuroscience, 23, 1648–1663.

    Article  PubMed  Google Scholar 

  • Obermeier, C., & Gunter, T. C. (2015). Multisensory integration: The case of a time window of gesture-speech integration. Journal of Cognitive Neuroscience, 27, 292–307.

    Article  PubMed  Google Scholar 

  • Peirce, J. W., Gray, J. R., Simpson, S., MacAskill, M. R., Höchenberger, R., Sogo, H., Kastman, E., & Lindeløv, J. (2019). PsychoPy2: Experiments in behavior made easy. Behavioral Reserch Method, 51, 195–203.

    Article  Google Scholar 

  • Saltz, E., & Donnenwerthnolan, S. E. (1981). Does motoric imagery facilitate memory for sentences? A selective interference test. Journal of Verbal Learning and Verbal Behaviour, 20, 322–332.

    Article  Google Scholar 

  • Segalowitz, N. S. (2003). Automaticity and second language learning. In C. Doughty, & M. Long (Eds.), The handbook of second language acquisition (pp. 382–408). Blackwell.

  • Seyfeddinipur, M. (2006). Disfluency: Interrupting speech and gesture, Ph.D. thesis, Radboud University Nijmegen, Nijmegen.

  • Smotrova, T., & Lantolf, J. P. (2013). The function of gesture in Lexically focused L2 instructional conversations. Modern Language Journal, 97, 397–416.

    Article  Google Scholar 

  • Sweller, N., Shinooka-Phelan, A., & Austin, E. (2020). The effects of observing and producing gestures on japanese word learning. Acta Psychologica, 207, 1039.

    Article  Google Scholar 

  • Tellier, M. (2005). How do teacher’s gestures help young children in second language acquisition? Université Paris 7, UFR Linguistique, Laboratoire ARP.

  • Tellier, M. (2008). The effect of gestures on second language memorisation by young children. Gesture, 8, 219–235.

    Article  Google Scholar 

  • Treffner, P., Peter, M., & Kleidon, M. (2008). Gestures and phases: The dynamics of speech-hand communication. Ecological Psychology, 20, 32–64.

    Article  Google Scholar 

  • van Compernolle, R. A., & Williams, L. (2011). Thinking with your hands: Speech–gesture activity during an L2 awareness-raising task. Language Awareness, 20, 203–219.

    Article  Google Scholar 

  • Wesp, R., Hesse, J., Keutmann, D., & Wheaton, K. (2001). Gestures maintain spatial imagery. The American Journal of Psychology, 144, 591–600.

    Article  Google Scholar 

  • Westfall, J., Kenny, D. A., & Judd, C. M. (2014). Statistical power and optimal design in experiments in which samples of participants respond to samples of stimuli. Journal of Experimental Psychology: General, 143, 2020–2045.

    Article  PubMed  Google Scholar 

  • Yasinnik, Y., Shattuck-Hufnagel, S., & Veilleux, N. (2005). Gesture marking of disfluencies in spontaneous speech. Retrieved from http://www.isca-speech.org/archive_open/archive_papers/diss_05/dis5_173.pdf

Download references

Acknowledgements

The authors thank Andrea Liotto for videorecording and editing the stimuli, as well as helping collect the data of Experiment 2. We also thank Debora Lupini for her help with the data collection of Experiment 1.

Author information

Authors and Affiliations

  1. Dipartimento di Studi Linguistici e Letterari, Università degli Studi di Padova, Padova, Italy

    Maria Grazia Busà

  2. Dipartimento di Studi Umanistici, Università del Salento, Lecce, Italy

    Federica Cavicchio

Authors
  1. Federica Cavicchio
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Maria Grazia Busà
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

Author 1 and author 2 have contributed equally to the research and the final manuscript.

Corresponding author

Correspondence to Federica Cavicchio.

Ethics declarations

Ethical Approval

Informed consent to participate in the experiment was obtained from each participant included in the study. The informed consent module subscribed by the participants was approved by the Ethical Committee for Research at the Università degli Studi di Padova (http://ethos.psy.unipd.it/it/composizione/) and can be found at the following link: http://ethos.psy.unipd.it/it/documentazione/. All procedures performed in the studies involving human participants followed the institutional and national research committee’s ethical standards and with the 1964 Helsinki declaration and its later amendments or comparable ethical standards. For further information, see: http://ethos.psy.unipd.it/it/collegamenti/.

Conflict of Interest/Competing interests -Funding

The research has not been funded.

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Cavicchio, F., Busà, M. The Role of Representational Gestures and Speech Synchronicity in Auditory Input by L2 and L1 Speakers. J Psycholinguist Res 52, 1721–1735 (2023). https://doi.org/10.1007/s10936-023-09947-2

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10936-023-09947-2

Keywords

Search

Navigation