Challenges for the Animation of Expressive Virtual Characters: The Standpoint of Sign Language and Theatrical Gestures

  • Sylvie Gibet
  • Pamela Carreno-Medrano
  • Pierre-Francois Marteau
Part of the Springer Tracts in Advanced Robotics book series (STAR, volume 111)


Designing and controlling virtual characters endowed with expressive gestures requires the modeling of multiple processes, involving high-level abstract representations to low-level sensorimotor models. An expressive gesture is here defined as a meaningful bodily motion which intrinsically associates sense, style, and expressiveness. The main challenges rely both on the capability to produce a large spectrum of parametrized actions executed with some variability in various situations, and on the biological plausibility of the motion of the virtual characters. The goals of the paper are twofold. First we review the different formalisms used to describe expressive gestures, from notations to computational languages. Secondly we identify and discuss remaining challenges in the generation of expressive virtual characters. The different models and formalisms are illustrated more particularly for theatrical and sign language gestures.


Sign Language Motion Capture Notation System Virtual Character Virtual Agent 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.



This work is part of the Ingredible project, supported by the French National Research Agency (ANR).


  1. 1.
    M. Thiebaux, S. Marsella, A.N. Marshall, M. Kallmann, Smartbody: behavior realization for embodied conversational agents, in Proceedings of AAMAS’08—Volume 1, Richland, SC (2008) (pp. 151–158)Google Scholar
  2. 2.
    J. Bates et al., The role of emotion in believable agents. Commun. ACM 37(7), 122–125 (1994)CrossRefGoogle Scholar
  3. 3.
    Y. Jung, A. Kuijper, M. Kipp, J. Miksatko, J. Gratch, D, Thalmann, Believable Virtual Characters in Human-Computer Dialogs. In EUROGRAPHICS, Llandudno, United Kingdom (2011)Google Scholar
  4. 4.
    M. Neff, Lessons from the arts: what the performing arts literature can teach us about creating expressive character movement, in Nonverbal Communication in Virtual Worlds: Understanding and Designing Expressive Characters (ETC Press, 2014)Google Scholar
  5. 5.
    K. Elam, The Semiotics of Theatre and Drama (Routledge, 2002)Google Scholar
  6. 6.
    S.C Marsella, S.M. Carnicke, J. Gratch, A. Okhmatovskaia, A. Rizzo, An exploration of delsartes structural acting system, in Intelligent Virtual Agents (Springer, 2006), pp. 80–92Google Scholar
  7. 7.
    S. Gibet, T. Lebourque, P.F. Marteau, High level specification and animation of communicative gestures. J. Visual Lang. Comput. 12, 657–687 (2001)CrossRefGoogle Scholar
  8. 8.
    A. Bogart, T. Landau, The Viewpoints Book: A Practical Guide To Viewpoints and Composition. Theatre Communications Group (2005)Google Scholar
  9. 9.
    R.E. Johnson, S.K. Liddell, Toward a phonetic representation of signs: sequentiality and contrast. Sign Lang. Stud. 11(2), 241–274 (2011)CrossRefGoogle Scholar
  10. 10.
    K. Duarte, Motion capture and avatars as portals for analyzing the linguistic structure of signed languages, in PhD thesis, Université de Bretagne Sud (2012)Google Scholar
  11. 11.
    A. Kendon, Gesticulation and speech two aspects of the process of utterance, in The Relation Between Verbal and Nonverbal Communication (1980), pp. 207–227Google Scholar
  12. 12.
    D. McNeill, Hand and Mind—What Gestures Reveal about Thought (The University of Chicago Press, Chicago, IL, 1992)Google Scholar
  13. 13.
    S. Kita, I. van Gijn, H. van der Hulst, Movement phase in signs and co-speech gestures, and their transcriptions by human coders, Proceedings of the International Gesture Workshop on Gesture and Sign Language in Human-Computer Interaction, vol. 1371, Lecture Notes in Computer Science (Springer, London, 1997), pp. 23–35CrossRefGoogle Scholar
  14. 14.
    C. Awad, N. Courty, K. Duarte, T. Le Naour, S. Gibet, A combined semantic and motion capture database for real-time sign language synthesis, in IVA (2009), pp. 432–438Google Scholar
  15. 15.
    R. Laban. The Mastery of Movement (Plays Inc. 1971)Google Scholar
  16. 16.
    V. Maletik, Body, Space, Expression: The Development of Rudolf Laban’s Movement and Dance Concepts (Mouton de Gruyte, New York, 1987)CrossRefGoogle Scholar
  17. 17.
    D.M. Chi, M. Costa, L. Zhao, N.I. Badler, The EMOTE model for effort and shape, in SIGGRAPH (2000), pp. 173–182Google Scholar
  18. 18.
    L. Zhao, N.I. Badler, Acquiring and validating motion qualities from live limb gestures. Graph. Models 67(1), 1–16 (2005)CrossRefGoogle Scholar
  19. 19.
    L. Torresani, P. Hackney, C. Bregler, Learning motion style synthesis from perceptual observations, in Advances in Neural Information Processing Systems (2006), pp. 1393–1400Google Scholar
  20. 20.
    D. Bouchard, N.I. Badler, Semantic segmentation of motion capture using laban movement analysis, in Intelligent Virtual Agents (IVA 2007) (2007), pp. 37–44Google Scholar
  21. 21.
    M. Karg, A. Samadani, R. Gorbet, K. Kühnlenz, J. Hoey, D. Kulic, Body movements for affective expression: a survey of automatic recognition and generation. T. Affect. Comput. 4(4), 341–359 (2013)CrossRefGoogle Scholar
  22. 22.
    A. Kleinsmith, N. Bianchi-Berthouze, Affective body expression perception and recognition: a survey. T. Affect. Comput. 4(1), 15–33 (2013)CrossRefGoogle Scholar
  23. 23.
    N. Eshkol, J. Harries, Ewmn: Eshkol-wachman movement notation (2011). Accessed 10 Mar 2015 [Online]Google Scholar
  24. 24.
    J.G. Harries, Symmetry in the movements of T’ai Chi Chuan. Comput. Math Appl. 17(46), 827–835 (1989)MathSciNetCrossRefGoogle Scholar
  25. 25.
    N. Eshkol, The Hand Book: The Detailed Notation of Hand and Finger Movements and Forms. [Tel Aviv]: Movement Notation Society, 1971Google Scholar
  26. 26.
    O. Teitelbaum, T. Benton, P.K. Shah, A. Prince, J.L. Kelly, P. Teitelbaum, Eshkol-wachman movement notation in diagnosis: the early detection of Asperger’s syndrome. Proc. Natl. Acad. Sci. USA 101(32), 11909–11914 (2004)CrossRefGoogle Scholar
  27. 27.
    J. Tanenbaum, M. Seif El-Nasr, M. Nixon, Basics of nonverbal communication in the physical world, in Nonverbal Communication in Virtual Worlds: understanding and Designing Expressive Characters (ETC Press, 2014)Google Scholar
  28. 28.
    M. Nixon, P. Pasquier, M.S. El-Nasr, Delsartmap: applying delsartes aesthetic system to virtual agents, in Intelligent Virtual Agents (Springer, 2010), pp. 139–145Google Scholar
  29. 29.
    W.C. Stokoe, Semiotics and Human Sign Language. Walter de Gruyter Inc. (1972)Google Scholar
  30. 30.
    D. Brentari, A Prosodic Model of Sign Language Phonology (MIT Press, Cambridge, MA, 1999)Google Scholar
  31. 31.
    S. Prillwitz, R. Leven, H. Zienert, T. Hanke, J. Henning, Hamburg Notation System for Sign Languages—An Introductory Guide (University of Hamburg Press, 1989)Google Scholar
  32. 32.
    T. Johnston, The lexical database of AUSLAN (Australian Sign Language), in Proceedings of the First Intersign Workshop: Lexical Databases, Hamburg (1998)Google Scholar
  33. 33.
    K. Duarte, S. Gibet, Heterogeneous data sources for signed language analysis and synthesis: the signcom project, in Proceedings of LREC’10 (ELRA, 2010)Google Scholar
  34. 34.
    D. Vernon, G. Metta, G. Sandini, A survey of artificial cognitive systems: implications for the autonomous development of mental capabilities in computational agents. Trans. Evol. Comp 11(2), 151–180 (2007)CrossRefGoogle Scholar
  35. 35.
    W. Duch, R.J. Oentaryo, M. Pasquier, Cognitive architectures: where do we go from here? in Proceedings of the 2008 Conference on Artificial General Intelligence 2008: Proceedings of the First AGI Conference, Amsterdam, The Netherlands. (IOS Press, The Netherlands, 2008) pp. 122–136Google Scholar
  36. 36.
    F. Devillers, S. Donikian, F. Lamarche, J.F. Taille, A programming environment for behavioural animation. J. Vis. Comput. Animation 13(5), 263–274 (2002)CrossRefMATHGoogle Scholar
  37. 37.
    J. Cassell, J. Sullivan, S. Prevost, E.F. Churchill, Embodied Conversational Agents (The MIT Press, 2000)Google Scholar
  38. 38.
    A. Kranstedt, S. Ko, I. Wachsmuth, MURML: a multimodal utterance representation markup language for conversational agents, in Proceedings of the AAMAS02 Workshop on ECA (2002)Google Scholar
  39. 39.
    H. Noot, Z. Ruttkay, Variations in gesturing and speech by gestyle. Int. J. Hum.-Comput. Stud. 62(2), 211–229 (2005)CrossRefGoogle Scholar
  40. 40.
    B. Hartmann, M. Mancini, S. Buisine, C. Pelachaud, Implementing expressive gesture synthesis for embodied conversational agents, in Gesture Workshop (Springer, 2005)Google Scholar
  41. 41.
    C. Pelachaud, Studies on Gesture Expressivity for a Virtual Agent, vol. 63(1) (Springer, 2009)Google Scholar
  42. 42.
    H. Vilhjálmsson, N. Cantelmo, J. Cassell, N. Ech Chafai, M. Kipp, S. Kopp, M. Mancini, S. Marsella, A.N. Marshall, C. Pelachaud, Z. Ruttkay, K.R. Thórisson, H. van Welbergen, R.J. van der Werf, The behavior markup language: recent developments and challenges, in Intelligent Virtual Agents, 7th International Conference, IVA 2007, Paris, France, September 17–19, 2007, (2007), pp. 99–111Google Scholar
  43. 43.
    S. Kopp, B. Krenn, S. Marsella, A.N. Marshall, C. Pelachaud, H. Pirker, K.R. Thórisson, H.H. Vilhjálmsson, Towards a common framework for multimodal generation: the behavior markup language, in Intelligent Virtual Agents, 6th International Conference, IVA 2006, Marina Del Rey, CA, USA, August 21–23 (2006), pp. 205–217Google Scholar
  44. 44.
    D. Tolani, A. Goswami, N.I. Badler, Real-time inverse kinematics techniques for anthropomorphic limbs. Graph. Models 62(5), 353–388 (2000)CrossRefMATHGoogle Scholar
  45. 45.
    S. Kopp, I. Wachsmuth, Synthesizing multimodal utterances for conversational agents. J. Vis. Comput. Animation 15(1), 39–52 (2004)Google Scholar
  46. 46.
    A. Heloir, M. Kipp, Real-time animation of interactive agents: specification and realization. Appl. Artif. Intell. 24(6), 510–529 (2010)CrossRefGoogle Scholar
  47. 47.
    M. Gillies, X. Pan, M. Slater, Piavca: a framework for heterogeneous interactions with virtual characters, in Intelligent Virtual Agents, 8th International Conference, IVA 2008, Tokyo, Japan, September 1–3, 2008. Proceedings, (2008) pp. 494–495Google Scholar
  48. 48.
    L. Kovar, M. Gleicher, F. Pighin, Motion graphs. ACM Trans. Graph. 21(3), 473–482 (2002)CrossRefGoogle Scholar
  49. 49.
    M. Brand, A. Hertzmann, Style machines, in ACM SIGGRAPH 2000 (2000) pp. 183–192Google Scholar
  50. 50.
    A. Hertzmann, Machine learning for computer graphics: a manifesto and tutorial, in Computer Graphics and Applications, 2003. (IEEE, 2003), pp. 22–36Google Scholar
  51. 51.
    K. Grochow, S.L. Martin, A. Hertzmann, Z. Popovisć, Style-based inverse kinematics. ACM Trans. Graph. 23(3), 522–531 (2004)CrossRefGoogle Scholar
  52. 52.
    E. Hsu, K. Pulli, J. Popović, Style translation for human motion, in ACM Transactions on Graphics (TOG) (ACM, 2005), pp. 1082–1089. (Volume 24:3)Google Scholar
  53. 53.
    O. Arikan, D.A. Forsyth, J.F. O’Brien, Motion synthesis from annotations. ACM Trans. Graph. 22(3), 402–408 (2003)CrossRefMATHGoogle Scholar
  54. 54.
    M. Müller, B. Andreas, H.P. Seidel, Efficient and robust annotation of motion capture data, in Proceedings of the ACM SIGGRAPHEurographics Symposium on Computer Animation (2009), pp. 17–26Google Scholar
  55. 55.
    C. Rose, B. Bodenheimer, M.F. Cohen, Verbs and adverbs: Multidimensional motion interpolation using radial basis functions. IEEE Comput. Graph. Appl. 18, 32–40 (1998)CrossRefGoogle Scholar
  56. 56.
    M. Stone, D. DeCarlo, O. Insuk, C. Rodriguez, A. Stere, A. Lees, C. Bregler, Speaking with hands: creating animated conversational characters from recordings of human performance. ACM Trans. Graph. 23(3), 506–513 (2004)CrossRefGoogle Scholar
  57. 57.
    S. Jörg, J.K. Hodgins, A. Safonova, Data-driven finger motion synthesis for gesturing characters. ACM Trans. Grap. 31(6), 189:1–189:7 (2012)Google Scholar
  58. 58.
    A. Safonova, J.K. Hodgins, Construction and optimal search of interpolated motion graphs, in ACM SIGGRAPH 2007 Papers, SIGGRAPH’07 (2007)Google Scholar
  59. 59.
    J. Starck, A. Hilton, Surface capture for performance-based animation. IEEE Comput. Graph. Appl. 27(3), 21–31 (2007)CrossRefGoogle Scholar
  60. 60.
    S. Gibet, N. Courty, K. Duarte, T. Le Naour, The signcom system for data-driven animation of interactive virtual signers: methodology and evaluation. Trans. Interact. Intel. Syst. 1(1), 6 (2011)Google Scholar
  61. 61.
    A. Héloir, N. Courty, S. Gibet, F. Multon, Temporal alignment of communicative gesture sequences. Comput. Animation Virtual Worlds 17, 347–357 (2006)CrossRefGoogle Scholar
  62. 62.
    A. Héloir, S. Gibet, A qualitative and quantitative characterisation of style in sign language gestures, in Gesture in Human-Computer Interaction and Simulation, GW 2007, Lisboa, Portugal. Lecture Notes in Artificial Intelligence (Springer, 2009)Google Scholar
  63. 63.
    Ed Hooks, Acting for Animators (Routledge, 2013)Google Scholar
  64. 64.
    L. Bishko, Our emphatic experience of believable characters, in Nonverbal Communication in Virtual Worlds: Understanding and Designing Expressive Characters (ETC Press, 2014)Google Scholar
  65. 65.
    J. Lecoq, J.G. Carasso, J.C. Lallias, D. Bradby, The Moving Body (Le Corps Poétique): Teaching Creative Theatre. Methuen Drama Modern Plays. Bloomsbury Academic (2009)Google Scholar
  66. 66.
    S. Murray, J. Keefe, Physical Theatres: A Critical Introduction (Taylor & Francis, 2007)Google Scholar

Copyright information

© Springer International Publishing Switzerland 2016

Authors and Affiliations

  • Sylvie Gibet
    • 1
  • Pamela Carreno-Medrano
    • 1
  • Pierre-Francois Marteau
    • 1
  1. 1.IRISAUniversity of Bretagne SudVannesFrance

Personalised recommendations