Coloring Multi-character Conversations through the Expression of Emotions

  • Patrick Gebhard
  • Martin Klesen
  • Thomas Rist
Part of the Lecture Notes in Computer Science book series (LNCS, volume 3068)


In this paper we describe how to generate affective dialogs for multiple virtual characters based on a combination of both automatically generated and pre-scripted scenes. This is done by using the same technique for emotion elicitation and computation that takes either input from the human author in the form of appraisal and dialog act tags or from a dialog planner in the form inferred emotion eliciting conditions. In either case, the system computes the resulting emotions and their intensities. Emotions are used to inform the selection of pre-scripted scenes and dialog strategies, and their surface realization. The approach has been integrated in two fully operable systems, the CrossTalk II installation and the NECA eShowroom.


Affective State Speech Synthesis Virtual Character Facial Animation Conversational Agent 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    André E., Rist T., van Mulken S., Klesen M., and Baldes S.: The automated design of believable dialogues for animated presentation teams. In: [32], pp. 220–255Google Scholar
  2. 2.
    Rist, T., André, E., Baldes, S.: In: Proc of IUI 2003, A Flexible Platform for Building Applications with Life-Like Characters, pp. 158–165. ACM Press, New York (2003)Google Scholar
  3. 3.
    Klesen, M., Kipp, M., Gebhard, P., Rist, T.: Staging Exhibitions: Methods and tools for modelling narrative structure to produce interactive performances with virtual actors. Journal of Virtual Reality – Special issue on Storytelling in Virtual Environments (2003)Google Scholar
  4. 4.
    Gebhard, P., Kipp, M., Klesen, M., Rist, T.: Authoring Scenes for Adaptive, Interactive Performances. In: Proceedings of the Second International Joint Conference on Autonomous Agents and Multiagent Systems, AAMAS 2003 (2003)Google Scholar
  5. 5.
    Laurel, B.: Computers as theatre. Addison-Wesley, Reading (1993)Google Scholar
  6. 6.
    Swartout, W., Hill, R., Gratch, J., Johnson, W.L., Kyriakakis, C., LaBore, C., Lindheim, R., Marsella, S., Miraglia, D., Moore, B., Morie, J., Rickel, J., Thiébaux, M., Tuch, L., Whitney, R., Douglas, J.: Towards the Holodeck: Integrating Graphics, Sound, Character and Story. In: Proc. of Autonomous Agents 2001, pp. 409–416 (2001)Google Scholar
  7. 7.
    Rocchi, C., Stock, O., Zancanaro, M., Kruppa, M., Krüger, A.: The Museum Visit: Generating Seamless Personalized Presentations on Multiple Devices. In: Proc. of Intelligent User Interfaces (2004)Google Scholar
  8. 8.
    Murray, J.H.: Hamlet on the Holodeck: The Future of Narrative in Cyberspace. The MIT Press, Cambridge (2000)Google Scholar
  9. 9.
    Ryokai, K., Vaucelle, C., Cassell, J.: Virtual peers as partners in storytelling and literacy learning. Journal of Computer Assisted Learning 19(2), 195–208 (2003)CrossRefGoogle Scholar
  10. 10.
    McBreen, H., Jack, M.: Evaluating Humanoid Synthetic Agents in E-Retail Applications. IEEE Trans. On Systems, Man, and Cybernetics-Part A: Systems and Humans 31(5), 394–405 (2001)CrossRefGoogle Scholar
  11. 11.
    Diederiks, E.: Buddies in a Box Animated Characters in Consumer Electronics. In: Proc. of IUI 2003, pp. 34–38 (2003)Google Scholar
  12. 12.
    Lester, J., Voerman, J.L., Towns, S.G., Callaway, C.B.: Cosmo: A life-like animated pedagogical agent with deictic believability. In: Proc. of the IJCAI 1997 Workshop on Animated Interface Agents: Making them Intelligent, Nagoya (1997)Google Scholar
  13. 13.
    Émile, G.J.: Marshalling Passions in Training and Education. In: Proc. of Autonomous Agents 2000, pp. 325–332 (2000)Google Scholar
  14. 14.
    Ball, G., Breese, J.: Emotion and personality in a conversational agent. In: [32], pp. 189–219Google Scholar
  15. 15.
    de Carolis, B., Pelachaud, C., Poggi, I., Steedman, M.: APML, a Markup Language for Believable Behavior Generation. In: [31], pp. 65–85Google Scholar
  16. 16.
    Ortony, A., Clore, G.L., Collins, A.: The Cognitive Structure of Emotions. Cambridge University Press, Cambridge (1988)Google Scholar
  17. 17.
    Cassell, J., Pelachaud, C., Badler, N., Steedman, M., Achorn, B., Becket, T., Douville, B., Prevost, S., Stone, M.: Animated Conversation: Rule-Based Generation of Facial Expression, Gesture and Spoken Intonation for Multiple Conversational Agents. In: Proc. of SIGGRAPH 1994, pp. 413–420 (1994)Google Scholar
  18. 18.
    Nitta, K., Hasegawa, O., Akiba, T., Kamishima, T., Kurita, T., Hayamizu, S., Itoh, K., Ishizuka, M., Dohi, H., Okamura, M.: An experimental multimodal disputation system. In: Proc. of the IJCAI 1997 Workshop on Intelligent Multimodal Systems (1997)Google Scholar
  19. 19.
    Rickel, J., Johnson, W.L.: Virtual Humans for Team Training in Virtual Reality. In: Proc of the Ninth International Conference on Artificial Intelligence in Education, pp. 578–585. IOS Press, Amsterdam (1999)Google Scholar
  20. 20.
    Traum, D.R., Rickel, J.: Embodied agents for multi-party dialogue in immersive virtual worlds. In: Proc. of the First International Joint conference on Autonomous Agents and Multiagent systems (2002)Google Scholar
  21. 21.
    Prendinger, H., Saeyor, S., Ishizuka, M.: MPML and SCREAM: Scripting the Bodies and Minds of Life-Like Characters. In: [31], pp. 213–242Google Scholar
  22. 22.
    Guye-Vuillieme, A., Thalmann, D.: A High Level Architecture For Believable Social Agents. VR Journal 5, 95–106 (2001)Google Scholar
  23. 23.
    Rist, T., Schmitt, M.: Applying Socio-Psychological Concepts of Cognitive Consistency to Negotiation Dialog Scenarios with Embodied Conversational Characters. In: Proc. of AISB 2002 Symposium on Animated Expressive Characters for Social Interactions, pp. 79–84 (2002)Google Scholar
  24. 24.
    Walker, M., Cahn, J., Whittaker, S.J.: Improving linguistic style: Social and affective bases for agent personality. In: Proc. of the First International Conference on Autonomous Agents, pp. 96–105 (1997)Google Scholar
  25. 25.
    McCrae, R.R., John, O.P.: An introduction to the five-factor model and its implications. Journal of Personality 60, 171–215 (1992)Google Scholar
  26. 26.
    Krenn, B., Pirker, H., Grice, M., Piwek, P., van Deemter, K., Schröder, M., Klesen, M.: Generation of multimodal dialogue for net environments. In: Proc. der 6. Konferenz zur Verarbeitung natürlicher Sprache (2002)Google Scholar
  27. 27.
    Gebhard, P., Kipp, M., Klesen, M., Rist, T.: Adding the Emotional Dimension to Scripting Character Dialogues. In: Rist, T., Aylett, R.S., Ballin, D., Rickel, J. (eds.) IVA 2003. LNCS (LNAI), vol. 2792, pp. 48–56. Springer, Heidelberg (2003)CrossRefGoogle Scholar
  28. 28.
    Kendon, A.: Nonverbal Communication, Interaction, and Gesture.Mouton, The Hague (1980)Google Scholar
  29. 29.
    Russel, J.A.: How shall an emotion be called. In: Plutchik, R., Conte, H.R. (eds.) Circumplex models of personality and emotions, American Psychological Association (1997)Google Scholar
  30. 30.
    Schröder, M.: Speech and Emotion Research: An overview of research frameworks and a dimensional approach to emotional speech synthesis. PhD thesis, Institute of Phonetics, Saarland University (2003)Google Scholar
  31. 31.
    Prendinger, H., Ishizuka, M.: Life-Like Characters – Tools, Affective Functions, and Applications. Springer, Heidelberg (2004)Google Scholar
  32. 32.
    Cassell, J., Sullivan, J., Prevost, S., Churchill, E. (eds.): Embodied Conversational Agents. The MIT Press, Cambridge (2000)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2004

Authors and Affiliations

  • Patrick Gebhard
    • 1
  • Martin Klesen
    • 1
  • Thomas Rist
    • 1
  1. 1.DFKI GmbHSaarbrückenGermany

Personalised recommendations