Towards a Common Framework for Multimodal Generation: The Behavior Markup Language

  • Stefan Kopp
  • Brigitte Krenn
  • Stacy Marsella
  • Andrew N. Marshall
  • Catherine Pelachaud
  • Hannes Pirker
  • Kristinn R. Thórisson
  • Hannes Vilhjálmsson
Part of the Lecture Notes in Computer Science book series (LNCS, volume 4133)


This paper describes an international effort to unify a multimodal behavior generation framework for Embodied Conversational Agents (ECAs). We propose a three stage model we call SAIBA where the stages represent intent planning, behavior planning and behavior realization. A Function Markup Language (FML), describing intent without referring to physical behavior, mediates between the first two stages and a Behavior Markup Language (BML) describing desired physical realization, mediates between the last two stages. In this paper we will focus on BML. The hope is that this abstraction and modularization will help ECA researchers pool their resources to build more sophisticated virtual humans.


Facial Expression Behavior Planning Common Framework Computer Animation Communicative Intent 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Cassell, J., Pelachaud, C., Badler, N., Steedman, M., Achorn, B., Becket, T., Douville, B., Prevost, S., Stone, M.: Animated Conversation: Rule-Based Generation of Facial Expression, Gesture and Spoken Intonation for Multiple Conversational Agents. In: Siggraph 1994 Conference Proceedings, ACM SIGGRAPH, pp. 413–420. Addison Wesley, Reading (1994)CrossRefGoogle Scholar
  2. 2.
    Cassell, J., Vilhjálmsson, H., Bickmore, T.: BEAT: the Behavior Expression Animation Toolkit. In: Proc. ACM SIGGRAPH 2001, Los Angeles, August 12-17, pp. 477–486 (2001)Google Scholar
  3. 3.
    Cassell, J., Vilhjálmsson, H., Chang, K., Bickmore, T., Campbell, L., Yan, H.: Requirements for an Architecture for Embodied Conversational Characters. In: Computer Animation and Simulation 1999. Eurographics Series. Springer, Austria (1999)Google Scholar
  4. 4.
    DeCarolis, B., Pelachaud, C., Poggi, I., Steedman, M.: APML, a mark-up language for believable behavior generation. In: Prendinger, H., Ishizuka, M. (eds.) Life-like Characters. Tools, Affective Functions and Applications, pp. 65–85. Springer, Heidelberg (2004)Google Scholar
  5. 5.
    Hartmann, B., Mancini, M., Pelachaud, C.: Formational parameters and adaptive prototype instantiation for MPEG-4 compliant gesture synthesis. In: Computer Animation 2002, Geneva, Switzerland. IEEE Computer Society Press, Los Alamitos (2002)Google Scholar
  6. 6.
    Kopp, S., Jung, B., Lessmann, N., Wachsmuth, I.: Max–A Multimodal Assistant in Virtual Reality Construction. KI 4/03, 11–17 (2003)Google Scholar
  7. 7.
    Kopp, S., Wachsmuth, I.: Synthesizing Multimodal Utterances for Conversational Agents. Computer Animation and Virtual Worlds 15(1), 39–52 (2004)CrossRefGoogle Scholar
  8. 8.
    Krenn, B.: Representational Lego for ECAs. In: Background paper for a presentation held at the FP6 NoE HUMAINE Workshop on Emotion and Interaction, Paris (March 10-11, 2005)Google Scholar
  9. 9.
    Krenn, B., Pirker, H.: Defining the Gesticon: Language and Gesture Coordination for Interacting Embodied Agents. In: Krenn, B., Pirker, H. (eds.) Proc. of the AISB 2004 Symposium on Language, Speech and Gesture for Expressive Characters, University of Leeds, UK, pp. 107–115 (2004)Google Scholar
  10. 10.
    Martell, C.: FORM: An Extensible, Kinematically-based Gesture Annotation Scheme. In: Proceedings of the 2002 International Conference on Language Resources and Evaluation, Las Palmas, Canary Island (2002)Google Scholar
  11. 11.
    Piwek, P., Krenn, B., Schröder, M., Grice, M., Baumann, S., Pirker, H.: RRL: A Rich Representation Language for the Description of Agent Behaviour in NECA. In: Proceedings of the Workshop Embodied conversational agents - let’s specify and evaluate them!, held in conjunction with AAMAS 2002, Bologna, Italy (July 16, 2002)Google Scholar
  12. 12.
    Prendinger, H., Descamps, S., Ishizuka, M.: MPML: A markup language for controlling the behavior of life-like characters. Journal of Visual Languages and Computing 15(2), 183–203 (2004)CrossRefGoogle Scholar
  13. 13.
    Stokoe, W.C., Casterline, D.C., Croneberg, C.G.: A dictionary of American sign language on linguistic principles. Linstok Press (1976)Google Scholar
  14. 14.
    Searle, J.R.: Speech acts: An essay in the philosophy of language. Cambridge Univ. Press, London (1969)Google Scholar
  15. 15.
    Thórisson, K.R.: Dialogue Control in Social Interface Agents. In: InterCHI Adjunct Proceedings 1993, Amsterdam, pp. 139–140 (April 1993)Google Scholar
  16. 16.
    Thórisson, K.R.: Computational Characteristics of Multimodal Dialogue. In: AAAI Fall Symposium on Embodied Language and Action, Massachusetts Institute of Technology, Cambridge, Massachusetts, November 10-12, pp. 102–108 (1995)Google Scholar
  17. 17.
    Thórisson, K.R.: A Mind Model for Multimodal Communicative Creatures and Humanoids. International Journal of Applied Artificial Intelligence 13(4-5), 449–486 (1999)CrossRefGoogle Scholar
  18. 18.
    Thórisson, K.R., Vilhjalmsson, H., Kopp, S., Pelachaud, C.: Report on Representations for Multimodal Generation Workshop. AI Magazine 27(1), 108 (2006)Google Scholar
  19. 19.
    Vilhjalmsson, H.: Animating Conversation in Online Games. In: Rauterberg, M. (ed.) ICEC 2004. LNCS, vol. 3166, pp. 139–150. Springer, Heidelberg (2004)CrossRefGoogle Scholar
  20. 20.
    Vilhjalmsson, H.: Augmenting Online Conversation through Automated Discourse Tagging. In: 6th annual minitrack on Persistent Conversation at the 38th Hawaii International Conference on System Sciences, Hilton Waikoloa Village, Big Island, Hawaii, January 3-6. IEEE, Los Alamitos (2005)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2006

Authors and Affiliations

  • Stefan Kopp
    • 1
  • Brigitte Krenn
    • 2
  • Stacy Marsella
    • 4
  • Andrew N. Marshall
    • 4
  • Catherine Pelachaud
    • 3
  • Hannes Pirker
    • 2
  • Kristinn R. Thórisson
    • 5
  • Hannes Vilhjálmsson
    • 4
  1. 1.Artificial .Intelligence GroupUniversity of BielefeldGermany
  2. 2.Austrian Research Institute for AI (OFAI)ViennaAustria
  3. 3.IUT de MontreuilUniversity de Paris 8France
  4. 4.Information Sciences InstituteUniversity of Southern CaliforniaUSA
  5. 5.CADIA, Dept. Of Computer ScienceReykjavik UniversityIceland

Personalised recommendations