Towards Natural Gesture Synthesis: Evaluating Gesture Units in a Data-Driven Approach to Gesture Synthesis

  • Michael Kipp
  • Michael Neff
  • Kerstin H. Kipp
  • Irene Albrecht
Part of the Lecture Notes in Computer Science book series (LNCS, volume 4722)


Virtual humans still lack naturalness in their nonverbal behaviour. We present a data-driven solution that moves towards a more natural synthesis of hand and arm gestures by recreating gestural behaviour in the style of a human performer. Our algorithm exploits the concept of gesture units to make the produced gestures a continuous flow of movement. We empirically validated the use of gesture units in the generation and show that it causes the virtual human to be perceived as more natural.


Embodied Conversational Agents Nonverbal Behavior Generation Gesture Synthesis 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Vinayagamoorthy, V., Gillies, M., Steed, A., Tanguy, E., Pan, X., Loscos, C., Slater, M.: Building Expression into Virtual Characters. In: Eurographics Conference State of the Art Report, Vienna (2006)Google Scholar
  2. 2.
    Nass, C., Isbister, K., Lee, E.J.: Truth is beauty: Researching embodied conversational agents. In: Cassell, J., Sullivan, J., Prevost, S., Churchill, E. (eds.) Embodied Conversational Agents, pp. 374–402. MIT Press, Cambridge, MA (2000)Google Scholar
  3. 3.
    Thomas, F., Johnston, O.: The Illusion of Life: Disney Animation. Hyperion Press, New York (1981)Google Scholar
  4. 4.
    Cassell, J., Pelachaud, C., Badler, N., Steedman, M., Achorn, B., Becket, T., Douville, B., Prevost, S., Stone, M.: Animated Conversation: Rule-Based Generation of Facial Expression, Gesture & Spoken Intonation for Multiple Conversational Agents. In: Proceedings of SIGGRAPH 1994, pp. 413–420 (1994)Google Scholar
  5. 5.
    Gratch, J., Rickel, J., André, E., Badler, N., Cassell, J., Petajan, E.: Creating interactive virtual humans: Some assembly required. IEEE Intelligent Systems, 54–63 (July/ August, 2002)Google Scholar
  6. 6.
    Kita, S., van Gijn, I., van der Hulst, H.: Movement phases in signs and co-speech gestures, and their transcription by human coders. In: Wachsmuth, I., Fröhlich, M. (eds.) Gesture and Sign Language in Human-Computer Interaction, pp. 23–35. Springer, Berlin (1998)CrossRefGoogle Scholar
  7. 7.
    Ruttkay, Z., Pelachaud, C. (eds.): From Brows to Trust: Evaluating Embodied Conversational Agents. Kluwer Academic Publishers, Dordrecht (2004)zbMATHGoogle Scholar
  8. 8.
    Neff, M., Kipp, M., Albrecht, I., Seidel, H.-P.: Gesture Modeling and Animation Based on a Probabilistic Recreation of Speaker Style. Transactions on Graphics (accepted, 2007)Google Scholar
  9. 9.
    Kipp, M., Neff, M., Albrecht, I.: An Annotation Scheme for Conversational Gestures: How to economically capture timing and form. Journal on Language Resources and Evaluation - Special Issue on Multimodal Corpora (2007)Google Scholar
  10. 10.
    Kipp, M.: Gesture Generation by Imitation: From Human Behavior to Computer Character Animation., Boca Raton, Florida (2004)Google Scholar
  11. 11.
    Kendon, A.: Gesture – Visible Action as Utterance. Cambridge University Press, Cambridge (2004)Google Scholar
  12. 12.
    McNeill, D.: Hand and Mind: What Gestures Reveal about Thought. University of Chicago Press, Chicago (1992)Google Scholar
  13. 13.
    Stone, M., DeCarlo, D., Oh, I., Rodriguez, C., Stere, A., Lees, A., Bregler, C.: Speaking with hands: Creating animated conversational characters from recordings of human performance. In: Proc. SIGGRAPH 2004, pp. 506–513 (2004)Google Scholar
  14. 14.
    Kopp, S., Wachsmuth, I.: Synthesizing multimodal utterances for conversational agents. Computer Animation and Virtual Worlds 15(1), 39–52 (2004)CrossRefGoogle Scholar
  15. 15.
    de Ruiter, J.P.: The production of gesture and speech. In: McNeill, D. (ed.) Language and Gesture: Window into Thought and Action, pp. 284–311. Cambridge University Press, Cambridge (2000)Google Scholar
  16. 16.
    Cassell, J., Vilhjálmsson, H., Bickmore, T.: BEAT: the Behavior Expression Animation Toolkit. In: Proceedings of SIGGRAPH 2001, pp. 477–486 (2001)Google Scholar
  17. 17.
    Lee, J., Marsella, S.: Nonverbal behavior generator for embodied conversational agents. In: Gratch, J., Young, M., Aylett, R., Ballin, D., Olivier, P. (eds.) IVA 2006. LNCS (LNAI), vol. 4133, pp. 243–255. Springer, Heidelberg (2006)CrossRefGoogle Scholar
  18. 18.
    Hartmann, B., Mancini, M., Buisine, S., Pelachaud, C.: Design and evaluation of expressive gesture synthesis for ecas. In: Proc. AAMAS (2005)Google Scholar
  19. 19.
    Chi, D.M., Costa, M., Zhao, L., Badler, N.I.: The EMOTE model for effort and shape. In: Proc. SIGGRAPH 2000, pp. 173–182 (2000)Google Scholar
  20. 20.
    Hartmann, B., Mancini, M., Pelachaud, C.: Implementing Expressive Gesture Synthesis for Embodied Conversational Agents. In: Gibet, S., Courty, N., Kamp, J.-F. (eds.) GW 2005. LNCS (LNAI), vol. 3881, pp. 188–199. Springer, Heidelberg (2006)CrossRefGoogle Scholar
  21. 21.
    Neff, M., Fiume, E.: AER: Aesthetic Exploration and Refinement for expressive character animation. In: Proc. ACM SIGGRAPH / Eurographics Symposium on Computer Animation 2005, pp. 161–170. ACM Press, New York (2005)CrossRefGoogle Scholar
  22. 22.
    Hodgins, J.K., Wooten, W.L., Brogan, D.C., O’Brien, J.F.: Animating human athletics. In: Proc. SIGGRAPH 1995, pp. 71–78 (1995)Google Scholar
  23. 23.
    Faloutsos, P., van de Panne, M., Terzopoulos, D.: The virtual stuntman: Dynamic characters with a repertoire of autonomous motor skills. Computers & Graphics 25(6), 933–953 (2001)CrossRefGoogle Scholar
  24. 24.
    Zordan, V.B., Hodgins, J.K.: Motion capture-driven simulations that hit and react. In: Proc. ACM SIGGRAPH Symposium on Computer Animation, pp. 89–96. ACM Press, New York (2002)CrossRefGoogle Scholar
  25. 25.
    Noot, H., Ruttkay, Z.: Gesture in style. In: Camurri, A., Volpe, G. (eds.) GW 2003. LNCS (LNAI), vol. 2915, pp. 324–337. Springer, Heidelberg (2004)Google Scholar
  26. 26.
    Webb, R.: Linguistic Properties of Metaphoric Gestures. UMI, New York (1997)Google Scholar
  27. 27.
    Boersma, P., Weenink, D.: Praat: doing phonetics by computer (version 4.3.14) [computer program] (2005), Retrieved from
  28. 28.
    Kipp, M.: Anvil – a Generic Annotation Tool for Multimodal Dialogue. In: Proceedings of Eurospeech, pp. 1367–1370 (2001)Google Scholar
  29. 29.
    McNeill, D.: Gesture and Thought. University of Chicago Press, Chicago (2005)Google Scholar
  30. 30.
    Schegloff, E.A.: On some gestures’ relation to talk. In: Atkinson, J.M., Heritage, J. (eds.) Structures of Social Action, pp. 266–296. Cambridge University Press, Cambridge (1984)Google Scholar
  31. 31.
    Steedman, M.: Information structure and the syntax-phonology interface. Linguistic Inquiry 34, 649–689 (2000)CrossRefGoogle Scholar
  32. 32.
    Kendon, A.: Gesticulation and speech: Two aspects of the process of utterance. In: Key, M.R. (ed.) Nonverbal Communication and Language, pp. 207–227. Mouton, The Hague (1980)Google Scholar
  33. 33.
    Neff, M., Fiume, E.: Modeling tension and relaxation for computer animation. In: Proc. ACM SIGGRAPH Symposium on Computer Animation 2002, pp. 81–88. ACM Press, New York (2002)CrossRefGoogle Scholar
  34. 34.
    Press, W.H., Tukolsky, S.A., Vetterling, W.T., Flannery, B.P.: Numerical Recipes in C: The Art of Scientific Computing, 2nd edn. Cambridge University Press, Cambridge (1992)Google Scholar
  35. 35.
    Hollars, M.G., Rosenthal, D.E., Sherman, M.A.: SD/FAST User’s Manual. Symbolic Dynamics Inc. (1994)Google Scholar
  36. 36.
    McCrae, R.R., John, O.P.: An Introduction to the Five-Factor Model and Its Applications. Journal of Personality 60, 175–215 (1992)CrossRefGoogle Scholar
  37. 37.
    Martin, J.C., Niewiadomski, R., Devillers, L., Buisine, S., Pelachaud, C.: Multimodal Complex Emotions: Gesture Expressivity And Blended Facial Expressions. Special issue of the Journal of Humanoid Robotics 3, 269–291 (2006)CrossRefGoogle Scholar
  38. 38.
    Krämer, N.C., Tietz, B., Bente, G.: Effects of embodied interface agents and their gestural activity. In: Rist, T., Aylett, R., Ballin, D., Rickel, J. (eds.) IVA 2003. LNCS (LNAI), vol. 2792, pp. 292–300. Springer, Heidelberg (2003)Google Scholar
  39. 39.
    Frey, S.: Die Macht des Bildes: der Einfluß der nonverbalen Kommunikation auf Kultur und Politik. Verlag Hans Huber, Bern (1999)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2007

Authors and Affiliations

  • Michael Kipp
    • 1
  • Michael Neff
    • 2
  • Kerstin H. Kipp
    • 3
  • Irene Albrecht
    • 4
  1. 1.DFKIGermany
  2. 2.UC DavisUSA
  3. 3.Saarland University, Experimental Neuropsychology UnitGermany
  4. 4.TomTec Imaging Systems GmbHGermany

Personalised recommendations