Skip to main content
SpringerLink
Log in

A Protocol for Comparing Gesture and Prosodic Boundaries in Multimodal Corpora

  • Conference paper
  • First Online:
Computational Processing of the Portuguese Language (PROPOR 2022)

Part of the book series: Lecture Notes in Computer Science ((LNAI,volume 13208))

  • 723 Accesses

Abstract

This paper aims to summarize the key points and results of an analysis of the alignment of gesture and prosodic boundaries collected through manual an-notation and automated treatment. For this purpose, we used a multimodal corpus of spontaneous speech, the BGEST corpus [1], part of the C-ORAL-BRASIL language resources [10, 23]. Gestures and prosodic boundaries were manually annotated with ELAN. The gestural an-notation followed partly the Linguistic Annotation System for Gestures guidelines [5], whereas the prosodic boundaries followed the perceptual criterion adopted in other C-ORAL resources. The tabular data outputted by ELAN was then analyzed by a script that compares the alignment of gesture phrases and intonation units. The results point to a larger gestural phrase that encompasses the intonation unit, while the stroke is almost always within the intonation unit. Around 85% of all information units align with a gesture. The analysis also points that the time values of alignment are shorter than analyzed by [17] both for initial as well for final boundaries and more complex than proposed by [6]. This indicates that more research is needed before we can set an approximate value for the overlap of larger units.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 69.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 89.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    All examples are taken of [8]. The transcription conventions can be summarized as follows: the speaker is indicated by an acronym, as “*ABC:" (regex: [A-Z]{3}:) and separated by turns in different line breaks. Each utterance is separated by double dash “//" and intonation units by a single dash “/". The utterance number is indicated by numbers in brackets, as “[3]". The information unit tags are between equal signs, as “=COM=" for the Comment unit. Time taking units are always transcribed as “&he" regardless of the vowels. Further information can be found in [8] and [23], the audios can be accessed in https://www.c-oral-brasil.org.

  2. 2.

    As above mentioned, the speech chunk formed by non-terminal boundaries between the beginning and the end of the utterance (marked by a terminal boundary).

References

  1. Barros, C.: A relação entre unidades gestuais e quebras prosódicas: o caso da unidade informacional Parentético. Ph.D. thesis, Universidade Federal de Minas Gerais, Belo Horizonte (2021). Accessed on 11 May 2021

    Google Scholar 

  2. Boersma, P., Weenink, D.: Praat: doing phonetics by computer (2020). Accessed on 07 March 2020

    Google Scholar 

  3. Bosker, H.R., Peeters, D.: Beat gestures influence which speech sounds you hear. Proc. Roy. Soc. B: Biol. Sci. 288(1943), 20202419 (2021). https://doi.org/10.1098/rspb.2020.2419

  4. Bressem, J.: 124. repetitions in gesture. In: Müller, C., Cienki, A., Fricke, E., Ladewig, S., McNeill, D., Bressem, J. (eds.) Handbücher zur Sprach- und Kommunikationswissenschaft/Handbooks of Linguistics and Communication Science (HSK) 38/2. DE GRUYTER, Berlin, München, Boston (2014). https://doi.org/10.1515/9783110302028.1641

  5. Bressem, J., Ladewig, S., Müller, C.: 71. linguistic annotation system for gestures. In: Müller, C., Cienki, A., Fricke, E., Ladewig, S., McNeill, D., Tessendorf, S. (eds.) Handbücher zur Sprach- und Kommunikationswissenschaft/Handbooks of Linguistics and Communication Science (HSK) 38/1. De Gruyter, Berlin, Boston (2013). https://doi.org/10.1515/9783110261318.1098

  6. Cantalini, G.: La gestualitá co-verbale nel parlato spontaneo e nel recitato. Ph.D. thesis, Universitá degli studi Roma Tre, Roma Tre (2018)

    Google Scholar 

  7. Cantalini, G., Moneglia, M.: The annotation of gesture and gesture/prosody synchronization in multimodal speech corpora. J. Speech Sci. 9, 7–30 (2020)

    Google Scholar 

  8. Cavalcante, F.A., Ramos, A.C.: The american english spontaneous speech minicorpus. CHIMERA. Romance Corpora Linguis. Stud. 3(2), 99–124 (2016)

    Google Scholar 

  9. Cresti, E.: Corpus del italiano parlato. Accademia della Crusca, Firenze (2000)

    Google Scholar 

  10. Cresti, E., Moneglia, M. (eds.): C-ORAL-ROM: Integrated Reference Corpora for Spoken Romance Languages. Studies in corpus linguistics, J. Benjamins, Amsterdam; Philadelphia, PA (2005), oCLC: ocm57506724

    Google Scholar 

  11. Esteve-Gibert, N., Guellaï, B.: Prosody in the auditory and visual domains: a developmental perspective. Front. Psychol. 9, 338 (2018). https://doi.org/10.3389/fpsyg.2018.00338

  12. Esteve-Gibert, N., Prieto, P.: Prosodic structure shapes the temporal realization of intonation and manual gesture movements. J. Speech, Lang. Hear. Res. 56(3), 850–864 (2013). https://doi.org/10.1044/1092-4388(2012/12-0049)

  13. Esteve-Gibert, N., Prieto, P.: Infants temporally coordinate gesture-speech combinations before they produce their first words. Speech Commun. 57, 301–316 (2014). https://doi.org/10.1016/j.specom.2013.06.006

  14. Kendon, A.: Some relationships between body motion and speech. In: Seigman, A., Pope, B. (eds.) Studies in Dyadic Communication, pp. 177–216. Pergamon Press, Elsmford, NY (1972)

    Chapter  Google Scholar 

  15. Ladewig, S.: 118. recurrent gestures. In: Müller, C., Cienki, A., Fricke, E., Ladewig, S., McNeill, D., Bressem, J. (eds.) Handbücher zur Sprach- und Kommunikationswissenschaft/Handbooks of Linguistics and Communication Science (HSK) 38/2. DE GRUYTER, Berlin, München, Boston (2014). https://doi.org/10.1515/9783110302028.1558

  16. Ladewig, S.: 126. creating multimodal utterances: the linear integration of gestures into speech. In: Müller, C., Cienki, A., Fricke, E., Ladewig, S., McNeill, D., Bressem, J. (eds.) Handbücher zur Sprach- und Kommunikationswissenschaft/Handbooks of Linguistics and Communication Science (HSK) 38/2. DE GRUYTER, Berlin, München, Boston (2014). https://doi.org/10.1515/9783110302028.1662

  17. Loehr, D.: Intonation and Gesture. Ph.D. thesis, University of Georgetown, Washington, D.C. (2004)

    Google Scholar 

  18. McNeill, D.: Hand and Mind: What Gestures Reveal About Thought. Hand and Mind: What Gestures Reveal About Thought, pp. xi, 416. University of Chicago Press, Chicago, IL, US (1992)

    Google Scholar 

  19. Mittelberg, I.: 130. gestures and iconicity. In: Müller, C., Cienki, A., Fricke, E., Ladewig, S., McNeill, D., Bressem, J. (eds.) Handbücher zur Sprach- und Kommunikationswissenschaft/Handbooks of Linguistics and Communication Science (HSK) 38/2. DE GRUYTER, Berlin, München, Boston (2014). https://doi.org/10.1515/9783110302028.1712

  20. Moneglia, M., Raso, T.: Appendix: notes on the language into act theory. In: Raso, T., Mello, H. (eds.) Studies in Corpus Linguistics, vol. 61, pp. 468–495. John Benjamins Publishing Company, Amsterdam (2014). https://doi.org/10.1075/scl.61.15mon

  21. Pouw, W., Harrison, S.J., Dixon, J.A.: Gesture-speech physics: the biomechanical basis for the emergence of gesture-speech synchrony. J. Exp. Psychol.: General 149(2), 391–404 (2020). https://doi.org/10.1037/xge0000646

  22. Pouw, W., Trujillo, J.P., Dixon, J.A.: The quantification of gesture–speech synchrony: a tutorial and validation of multimodal data acquisition using device-based and video-based motion tracking. Behav. Res. Meth. 52(2), 723–740 (2019). https://doi.org/10.3758/s13428-019-01271-9

    Article  Google Scholar 

  23. Raso, T., Mello, H. (eds.): C-ORAL-BRASIL I: Corpus de referência do português brasileiro falado informal. Editora UFMG, Belo Horizonte (2012)

    Google Scholar 

  24. RCoreTeam: R: a language and environment for statistical computing. R Foundation for Statistical Computing, Vienna, Austria (2020)

    Google Scholar 

  25. Rocha, B., Mello, H., Raso, T.: Para a compilação do c-oral-angola. Filologia e Linguística Portuguesa 20(Especial), 139–157 (2018). https://doi.org/10.11606/issn.2176-9419.v20iEspecialp139-157

  26. Wagner, P., Malisz, Z., Kopp, S.: Gesture and speech in interaction: an overview. Speech Commun. 57, 209–232 (2014). https://doi.org/10.1016/j.specom.2013.09.008

  27. Wittenburg, P., Brugman, H., Russel, A., Klassmann, A., Sloetjes, H.: Elan: a professional framework for multimodality research. In: Proceedings of LREC 2006, pp. 1556–1559. Fifth International Conference on Language Resources and Evaluation, Max Planck Institute for Psycholinguistics, The Language Archive, Nijmegen (2006). Accessed on 07 March 2020

    Google Scholar 

Download references

Acknowledgement

Both authors acknowledge the support of C-ORAL-BRASIL Research Group, S. M. S. acknowledges the support of CAPES and C. A. B. acknowledges the support of Fapemig.

Author information

Authors and Affiliations

  1. Federal University of Minas Gerais, UFMG, Belo Horizonte, Brazil

    Camila Barros

  2. Université Paris-Saclay, UPSaclay, Paris, France

    Saulo Santos

Authors
  1. Camila Barros
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Saulo Santos
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Camila Barros .

Editor information

Editors and Affiliations

  1. Universidade de Fortaleza, Fortaleza, Brazil

    Vládia Pinheiro

  2. CiTIUS - Universidade de Santiago de Compostela, Santiago de Compostela, Spain

    Pablo Gamallo

  3. Universidade Nova de Lisboa, Lisbon, Portugal

    Raquel Amaro

  4. University of Sheffield, Sheffield, UK

    Carolina Scarton

  5. INESC-ID, Lisbon, Portugal

    Fernando Batista

  6. Federal University of São Carlos, São Carlos, Brazil

    Diego Silva

  7. University of Lisbon, Lisbon, Portugal

    Catarina Magro

  8. Sentimonitor, Porto Alegre, Brazil

    Hugo Pinto

Rights and permissions

Reprints and permissions

Copyright information

© 2022 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Barros, C., Santos, S. (2022). A Protocol for Comparing Gesture and Prosodic Boundaries in Multimodal Corpora. In: Pinheiro, V., et al. Computational Processing of the Portuguese Language. PROPOR 2022. Lecture Notes in Computer Science(), vol 13208. Springer, Cham. https://doi.org/10.1007/978-3-030-98305-5_29

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-98305-5_29

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-98304-8

  • Online ISBN: 978-3-030-98305-5

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation