Distinguishing the Communicative Functions of Gestures

An Experiment with Annotated Gesture Data
  • Kristiina Jokinen
  • Costanza Navarretta
  • Patrizia Paggio
Part of the Lecture Notes in Computer Science book series (LNCS, volume 5237)


This paper deals with the results of a machine learning experiment conducted on annotated gesture data from two case studies (Danish and Estonian). The data concern mainly facial displays, that are annotated with attributes relating to shape and dynamics, as well as communicative function. The results of the experiments show that the granularity of the attributes used seems appropriate for the task of distinguishing the desired communicative functions. This is a promising result in view of a future automation of the annotation task.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. Allwood, J.: Dialog Coding – Function and Grammar. Gothenburg Papers. In: Theoretical Linguistics. Department of Linguistics, vol. 85, Gothenburg University (2001a)Google Scholar
  2. Allwood, J.: The Structure of Dialog. In: Taylor, M., Bouwhuis, D., Nel, F. (eds.) The Structure of Multimodal Dialogue II, pp. 3–24. Amsterdam, Benjamins (2001b)Google Scholar
  3. Allwood, J., Cerrato, L.: A study of gestural feedback expressions. In: Paggio, P., et al. (eds.) Proceedings of the First Nordic Symposium on Multimodal Communication, pp. 7–22 (2003)Google Scholar
  4. Allwood, J., Nivre, J., Ahlsén, E.: On the Semantics and Pragmatics of Linguistic Feedback. Journal of Semantics 9, 1–26 (1992)CrossRefGoogle Scholar
  5. Allwood, J., Cerrato, L., Jokinen, K., Navarretta, C., Paggio, P.: The MUMIN coding scheme for the annotation of feedback, turn management and sequencing phenomena. In: Martin, J.C., et al. (eds.) Multimodal Corpora for Modelling Human Multimodal Behaviour. Special issue of the International Journal of Language Resources and Evaluation, vol. 41(3–4), pp. 273–287. Springer, Heidelberg (2007)Google Scholar
  6. Campbell, N.: Tools and Resources for Visualising Conversational-Speech Interaction. In: LREC 2008, Marrakesh, Morocco (2008)Google Scholar
  7. Carletta, J.: Assessing agreement on classification tasks: the kappa statistics. Computational Linguistics 22(2), 249–254 (1996)Google Scholar
  8. Carletta, J.: Unleashing the killer corpus: experiences in creating the multi-everything AMI Meeting Corpus. In: Language Resources and Evaluation, vol. 41(2), pp. 181–190. Springer, Heidelberg (2007)Google Scholar
  9. Cerrato, L.: Investigating Communicative Feedback Phenomena across Languages and Modalities. PhD Thesis in Speech and Music Communication, Stockholm, KTH (2007)Google Scholar
  10. Clark, H.H., Schaefer, E.F.: Contributing to Discourse. Cognitive Science 13, 259–294 (1989)CrossRefGoogle Scholar
  11. Douxchamps, D., Campbell, N.: Robust real-time tracking for the analysis of human behaviour. In: Popescu-Belis, A., Renals, S., Bourlard, H. (eds.) MLMI 2007. LNCS, vol. 4892. Springer, Heidelberg (2008)CrossRefGoogle Scholar
  12. Duncan Jr., S., Fiske, D.W.: Face-to-Face Interaction: Research, Methods and Theory. John Wiley & Sons/Lawrence Erlbaum, Mahwah (1977)Google Scholar
  13. Jokinen, K., Ragni, A.: Clustering experiments on the communicative properties of gaze and gestures. In: Proceeding of the 3rd. Baltic Conference on Human Language Technologies, Kaunas (2007)Google Scholar
  14. Kendon, A.: Gesture, Cambridge (2004)Google Scholar
  15. Kipp, M.: Anvil – A Generic Annotation Tool for Multimodal Dialogue. In: Proceedings of Eurospeech 2001, pp. 1367–1370 (2001)Google Scholar
  16. Martin, J.C., Paggio, P., Kuenlein, P., Stiefelhagen, R., Pianesi, F. (eds.): Multimodal Corpora for Modelling Human Multimodal Behaviour. Special issue of the International Journal of Language Resources and Evaluation, vol. 41(3–4). Springer, Heidelberg (2007)Google Scholar
  17. McNeill, D.: Hand and Mind: What Gestures Reveal About Thought. University of Chicago Press, Chicago (1992)Google Scholar
  18. Mostefa, D., Moreau, N., Choukri, K., Potamianos, G., Chu, S.M., Tyagi, A., Casas, J.R., Turmo, J., Cristoforetti, L., Tobia, F., Pnevmatikakis, A., Mylonakis, V., Talantzis, F., Burger, S., Stiefelhagen, R., Bernardin, K., Rochet, C.: The CHIL audiovisual corpus for lecture and meeting analysis inside smart rooms. In: Language Resources and Evaluation, vol. 41(3-4), pp. 389–407. Springer, Heidelberg (2007)Google Scholar
  19. Peirce, C.S.: Elements of Logic. In: Hartshorne, C., Weiss, P. (eds.) Collected Papers of Charles Sanders Peirce, vol. 2, Harvard University Press, Cambridge (1931)Google Scholar
  20. Witten, I.H., Frank, E.: Data Mining: Practical machine learning tools and techniques, 2nd edn. Morgan Kaufmann, San Francisco (2005)MATHGoogle Scholar
  21. Platt, J.: Sequential minimal optimization: A fast algorithm for training support vector machines. Technical Report MSR-TR-98-14, Microsoft Research (1998)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2008

Authors and Affiliations

  • Kristiina Jokinen
    • 1
  • Costanza Navarretta
    • 2
  • Patrizia Paggio
    • 2
  1. 1.University of Tartu and University of Helsinki 
  2. 2.University of CopenhagenCST

Personalised recommendations