A Story About Gesticulation Expression

  • Celso de Melo
  • Ana Paiva
Part of the Lecture Notes in Computer Science book series (LNCS, volume 4133)


Gesticulation is essential for the storytelling experience thus, virtual storytellers should be endowed with gesticulation expression. This work proposes a gesticulation expression model based on psycholinguistics. The model supports: (a) real-time gesticulation animation described as sequences of constraints on static (Portuguese Sign Language hand shapes, orientations and positions) and dynamic (motion profiles) features; (b) multimodal synchronization between gesticulation and speech; (c) automatic reproduction of annotated gesticulation according to GestuRA, a gesture transcription algorithm. To evaluate the model two studies, involving 147 subjects, were conducted. In both cases, the idea consisted of comparing the narration of the Portuguese traditional story “The White Rabbit” by a human storyteller with a version by a virtual storyteller. Results indicate that synthetic gestures fared well when compared to real gestures however, subjects preferred the human storyteller.


Virtual Human Motion Profile Conversational Agent Body Expression Iconic Gesture 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    McNeill, D.: Hand and Mind: What gestures reveal about thought. University of Chicago Press, Chicago (1992)Google Scholar
  2. 2.
    McNeill, D.: Gesture and Thought. University of Chicago Press, Chicago (2005)Google Scholar
  3. 3.
    Kendon, A.: Sign languages of Aboriginal Australia: Cultural, semiotic and communicative perspectives. Cambridge University Press, Cambridge (1988)Google Scholar
  4. 4.
    Kita, S.: The temporal relationship between gesture and speech: A study of Japanese-English bilingual. MhD, Department of Psychology, University of Chicago (1990)Google Scholar
  5. 5.
    Nobe, S.: Where do most spontaneous representational gestures actually occur with respect to speech? In: McNeill, D. (ed.) Language and Gesture, pp. 186–198. Cambridge University Press, Cambridge (2000)CrossRefGoogle Scholar
  6. 6.
    de Ruiter, J.: The production of gesture and speech. In: McNeill, D. (ed.) Language and gesture, pp. 284–311. Cambridge University Press, Cambridge (2000)CrossRefGoogle Scholar
  7. 7.
    Krauss, M., Chen, Y., Gottesman, R.: Lexical gestures and lexical access: A process model. In: McNeill, D. (ed.) Language and gesture, pp. 261–283. Cambridge University Press, Cambridge (2000)CrossRefGoogle Scholar
  8. 8.
    Kita, S., Özyürek, A.: What does cross-linguistic variation in semantic coordination of speech and gesture reveal? Evidence for an interface representation of spatial thinking and speaking. Journal of Memory and Language 48, 16–32 (2003)Google Scholar
  9. 9.
    Cassell, J., Pelachaud, C., Badler, N., Steedman, M., Achorn, B., Becket, T., Douville, B., Prevost, S., Stone, M.: Animated Conversation: Rule-based Generation of Facial Expression, Gesture & Spoken Intonation for Multiple Conversational Agent. In: Proc. of SIGGRAPH 1994, pp. 413–420 (1994)Google Scholar
  10. 10.
    Cassell, J., Bickmore, T., Billinghurst, M., Campbell, L., Chang, K., Vilhjálmsson, H., Yan, H.: Embodiment in Conversational Interfaces: Rea. In: Proc. of the CHI 1999 Conference, Pittsburgh, PA, pp. 520–527 (1999)Google Scholar
  11. 11.
    Cassell, J., Stone, M.: Living Hand to Mouth: Psychological Theories about Speech and Gesture in Interactive Dialogue Systems. In: Proc. of the AAAI 1999 Fall Symposium on Psychological Models of Communication in Collaborative Systems, North Falmouth, MA, pp. 34–42 (1999)Google Scholar
  12. 12.
    Cassell, J., Vilhjálmsson, H., Bickmore, T.: BEAT: the Behavior Expression Animation Toolkit. In: Proc. of SIGGRAPH 2001, pp. 477–486 (2001)Google Scholar
  13. 13.
    Kopp, S., Wachsmuth, I.: A knowledge-based approach for lifelike gesture animation. In: Proc. of the 14th European Conf. on Artificial Intelligence. IOS Press, Amsterdam (2000)Google Scholar
  14. 14.
    Wachsmuth, I., Kopp, S.: Lifelike Gesture Synthesis and Timing for Conversational Agents. In: Wachsmuth, I., Sowa, T. (eds.) GW 2001. LNCS (LNAI), vol. 2298, pp. 120–133. Springer, Heidelberg (2002)CrossRefGoogle Scholar
  15. 15.
    SABLE: A Synthesis Markup Language (v. 1.0), http://www.bell-labs.com/project/tts/sable.html
  16. 16.
    Kopp, S., Tepper, P., Cassell, J.: Towards Integrated Microplanning of Language and Iconic Gesture for Multimodal Output. In: Proc. of the International Conference on Multi-modal Interfaces (ICMI 2004), pp. 97–104. ACM Press, New York (2004)CrossRefGoogle Scholar
  17. 17.
    Arafa, Y., Kamyab, K., Mamdani, E.: Character Animation Scripting Languages: A Comparison. In: Proc. of the 2nd Intl. Conference of Autonomous Agents and Multiagent Systems, pp. 920–921 (2003)Google Scholar
  18. 18.
    VHML: VHML – Virtual Human Markup Language, http://www.vhml.org/
  19. 19.
    SMIL: SMIL - Synchronized Multimedia, http://www.w3.org/AudioVideo/
  20. 20.
    Kranstedt, A., Kopp, S., Wachsmuth, I.: MURML: A Multimodal Utterance Representation Markup Language for Conversational Agents. In: AAMAS 2002 Workshop Embodied conversational agents- let’s specify and evaluate them!, Bologna, Italy (2002)Google Scholar
  21. 21.
    Ruttkay, Z., Noot, H.: Variations in Gesturing and Speech by GESTYLE. International Journal of Human-Computer Studies, Special Issue on Subtle Expressivity for Characters and Robots 62(2), 211–229 (2005)Google Scholar
  22. 22.
    de Carolis, B., Pelachaud, C., Poggi, I., Steedman, M.: APML, a Mark-up Language for Believable Behavior Generation. In: Prendinger, H. (ed.) Life-like Characters. Tools, Affective Functions and Applications. Springer, Heidelberg (2004)Google Scholar
  23. 23.
    Pavlovic, V., Sharma, R., Huang, T.: Visual Interpretation of hand gestures for human computer interaction: A review. IEEE Trans. Pattern Analysis Machine Intelligence 19, 677–695 (1997)CrossRefGoogle Scholar
  24. 24.
    Gavrila, D.: The visual analysis of human movement: A survey in Computer Vision and Image Understanding  73, 82–98 (1999)Google Scholar
  25. 25.
    de Melo, C., Paiva, A.: Multimodal Expression in Virtual Humans. Accepted for Computer Animation & Social Agents 2006 (CASA 2006) and Journal of Computer Animation and Virtual Worlds (2006)Google Scholar
  26. 26.
    Blumberg, B., Galyean, T.: Multi-Level Direction of Autonomous Creatures for Real-Time Virtual Environments. In: Proc. of SIGGRAPH 1995, vol. 30(3), pp. 47–54 (1995)Google Scholar
  27. 27.
    Perlin, K., Goldberg, A.: Improv: A System for Scripting Interactive Actors in Virtual Worlds. In: Proc. of SIGGRAPH 1996, pp. 205–216 (1996)Google Scholar
  28. 28.
    NASA Man-Systems Integration Manual (NASA-STD-3000)Google Scholar
  29. 29.
    Black, A., Taylor, P., Caley, R., Clark, R.: Festival, http://www.cstr.ed.ac.uk/projects/festival/
  30. 30.
    Gut, U., Looks, K., Thies, A., Trippel, T., Gibbon, D.: CoGesT – Conversational Gesture Transcription System. Technical Report, University of Bielefeld (1993)Google Scholar
  31. 31.
    Kipp, M.: ANVIL – A Generic Annotation Tool for Multimodal Dialogue. In: Proc. of the 7th European Conference on Speech Comm. and Technology, Aalborg, pp. 1367–1370 (2001)Google Scholar
  32. 32.
    Secretariado Nacional para a Reabilitação e Integração das Pessoas com Deficiência. Gestuário – Língua Gestual Portuguesa – 5th editionGoogle Scholar
  33. 33.
    SMIL. SMIL: Synchronized Multimedia, http://www.w3.org/AudioVideo/
  34. 34.
    Thompson, D., Buford, W., Myers, L., Giurintano, D., Brewer III, J.: A Hand Biomechanics Workstation. Computer Graphics 22(4), 335–343 (1988)CrossRefGoogle Scholar
  35. 35.
    Albrecht, I., Haber, J., Siedel, H.: Construction and Animation of Anatomically Based Human Hand Models. In: SIGGRAPH 2003, pp. 98–109 (2003)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2006

Authors and Affiliations

  • Celso de Melo
    • 1
  • Ana Paiva
    • 1
  1. 1.ISTTechnical University of Lisbon and INESC-IDPorto SalvoPortugal

Personalised recommendations