Affect Detection and an Automated Improvisational AI Actor in E-Drama

  • Li Zhang
  • Marco Gillies
  • John A. Barnden
  • Robert J. Hendley
  • Mark G. Lee
  • Alan M. Wallington
Part of the Lecture Notes in Computer Science book series (LNCS, volume 4451)


Enabling machines to understand emotions and feelings of the human users in their natural language textual input during interaction is a challenging issue in Human Computing. Our work presented here has tried to make our contribution toward such machine automation. We report work on adding affect-detection to an existing e-drama program, a text-based software system for dramatic improvisation in simple virtual scenarios, for use primarily in learning contexts. The system allows a human director to monitor improvisations and make interventions, for instance in reaction to excessive, insufficient or inappropriate emotions in the characters’ speeches. Within an endeavour to partially automate directors’ functions, and to allow for automated affective bit-part characters, we have developed an affect-detection module. It is aimed at detecting affective aspects (concerning emotions, moods, value judgments, etc.) of human-controlled characters’ textual “speeches”. The work also accompanies basic research into how affect is conveyed linguistically. A distinctive feature of the project is a focus on the metaphorical ways in which affect is conveyed. Moreover, we have also introduced how the detected affective states activate the animation engine to produce gestures for human-controlled characters. The description of our approach in this paper is taken in part from our previous publications [1, 2] with new contributions mainly on metaphorical language processing (practically and theoretically), 3D emotional animation generation and user testing evaluation. Finally, Our work on affect detection in open-ended improvisational text contributes to the development of automatic understanding of human language and emotion. The generation of emotional believable animations based on detected affective states and the production of appropriate responses for the automated affective bit-part character based on the detection of affect contribute greatly to the ease and innovative user interface in e-drama, which leads to high-level user engagement and enjoyment.


E-drama affect detection improvisational AI actor emotional behaviour and metaphor 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Zhang, L., et al.: Exploitation in Affect Detection in Open-ended Improvisational Text. In: Proceedings of Workshop on Sentiment and Subjectivity at COLING-ACL 2006, Sydney, July 2006 (2006a)Google Scholar
  2. 2.
    Zhang, L., et al.: Developments in Affect Detection in E-drama. In: Proceedings of EACL 2006, 11th Conference of the European Chapter of the Association for Computational Linguistics, 2006, Trento, Italy, pp. 203–206 (2006b)Google Scholar
  3. 3.
    Picard, R.W.: Affective Computing. MIT Press, Cambridge (2000)Google Scholar
  4. 4.
    Ortony, A., Clore, G.L., Collins, A.: The Cognitive Structure of Emotions. Cambridge Univ. Press, Cambridge (1988)Google Scholar
  5. 5.
    Prendinger, H., Ishizuka, M.: Simulating Affective Communication with Animated Agents. In: Proceedings of Eighth IFIP TC.13 Conference on Human-Computer Interaction, Tokyo, Japan, pp. 182–189 (2001)Google Scholar
  6. 6.
    Wiltschko, W.R.: Emotion Dialogue Simulator. eDrama learning, Inc. eDrama Front DeskGoogle Scholar
  7. 7.
    Mehdi, E.J., et al.: Modeling Character Emotion in an Interactive Virtual Environment. In: Proceedings of AISB 2004 Symposium: Motion, Emotion and Cognition, Leeds, UK (2004)Google Scholar
  8. 8.
    McCrae, R.R., John, O.P.: An Introduction to the Five Factor Model and Its Application. Journal of Personality 60, 175–215 (1992)CrossRefGoogle Scholar
  9. 9.
    Gratch, J., Marsella, S.: A Domain-Independent Framework for Modeling Emotion. Journal of Cognitive Systems Research 5(4), 269–306 (2004)CrossRefGoogle Scholar
  10. 10.
    Egges, A., Kshirsagar, S., Magnenat-Thalmann, N.: A Model for Personality and Emotion Simulation. In: Palade, V., Howlett, R.J., Jain, L. (eds.) KES 2003. LNCS, vol. 2773, Springer, Heidelberg (2003)Google Scholar
  11. 11.
    Elliott, C., Rickel, J., Lester, J.: Integrating Affective Computing into Animated Tutoring Agents. In: Proceedings of IJCAI’97 Workshop on Intelligent Interface Agents (1997)Google Scholar
  12. 12.
    Aylett, R.S., Dias, J., Paiva, A.: An affectively-driven planner for synthetic characters. In: Proceedings of ICAPS (2006)Google Scholar
  13. 13.
    Mateas, M.: Interactive Drama, Art and Artificial Intelligence. Ph.D. Thesis, School of Computer Science, Carnegie Mellon University (2002)Google Scholar
  14. 14.
    Zhe, X., Boucouvalas, A.C.: Text-to-Emotion Engine for Real Time Internet Communication. In: Proceedings of International Symposium on Communication Systems, Networks and DSPs, Staffordshire University, UK, pp. 164–168 (2002)Google Scholar
  15. 15.
    Boucouvalas, A.C.: Real Time Text-to-Emotion Engine for Expressive Internet Communications. In: Riva, G., Davide, F., IJsselsteijn, W. (eds.) Being There: Concepts, Effects and Measurement of User Presence in Synthetic Environments, pp. 305–318 (2002)Google Scholar
  16. 16.
    Craggs, R., Wood, M.: A Two Dimensional Annotation Scheme for Emotion in Dialogue. In: Proceedings of AAAI Spring Symposium: Exploring Attitude and Affect in Text (2004)Google Scholar
  17. 17.
    Fussell, S., Moss, M.: Figurative Language in Descriptions of Emotional States. In: Fussell, S.R., Kreuz, R.J. (eds.) Social and cognitive approaches to interpersonal communication, Lawrence Erlbaum, Mahwah (1998)Google Scholar
  18. 18.
    Kövecses, Z.: Are There Any Emotion-Specific Metaphors? In: Athanasiadou, A., Tabakowska, E. (eds.) Speaking of Emotions: Conceptualization and Expression, pp. 127–151. Mouton de Gruyter, Berlin (1998)Google Scholar
  19. 19.
    Watson, D., Tellegen, A.: Toward a Consensual Structure of Mood. Psychological Bulletin 98, 219–235 (1985)CrossRefGoogle Scholar
  20. 20.
    Ekman, P.: An Argument for Basic Emotions. Cognition and Emotion 6, 169–200 (1992)CrossRefGoogle Scholar
  21. 21.
    Werry, C.: Linguistic and Interactional Features of Internet Relay Chat. In: Computer-Mediated Communication: Linguistic: Social and Cross-Cultural Perspectives. Pragmatics and Beyond New Series, vol. 39, pp. 47–64. John Benjamins, Amsterdam (1996)Google Scholar
  22. 22.
    Briscoe, E., Carroll, J.: Robust Accurate Statistical Annotation of General Text. In: Proceedings of the 3rd International Conference on Language Resources and Evaluation, Las Palmas, Gran Canaria, pp. 1499–1504 (2002)Google Scholar
  23. 23.
    Heise, D.R.: Semantic Differential Profiles for 1,000 Most Frequent English Words. Psychological Monographs 70(8) (Whole 601) (1965)Google Scholar
  24. 24.
    Barnden, J., et al.: Varieties and Directions of Inter-Domain Influence in Metaphor. Metaphor and Symbol 19(1), 1–30 (2004)CrossRefGoogle Scholar
  25. 25.
    Barnden, J.A.: Forthcoming. Metaphor, Semantic Preferences and Context-sensitivity. Invited chapter for a Festschrifft volume. KluwerGoogle Scholar
  26. 26.
    Esuli, A., Sebastiani, F.: Determining Term Subjectivity and Term Orientation for Opinion Mining. In: Proceedings of EACL-06, 11th Conference of the European Chapter of the Association for Computational Linguistics, Trento, IT, pp. 193–200 (2006)Google Scholar
  27. 27.
    Sharoff, S.: How to Handle Lexical Semantics in SFL: a Corpus Study of Purposes for Using Size Adjectives. In: Systemic Linguistics and Corpus, Continuum, London (2005)Google Scholar
  28. 28.
    Garau, M., et al.: The impact of eye gaze on communication using humanoid avatars. In: Proceedings of the SIG-CHI conference on Human factors in computing systems, Seattle, WA, USA, March 31 - April 5, 2001, pp. 309–316 (2001)Google Scholar
  29. 29.
    Vilhjálmsson, H., Cassell, J.: BodyChat: Autonomous Communicative Behaviors in Avatars. In: Proceedings of ACM Second International Conference on Autonomous Agents, Minneapolis, Minnesota, May 9-13 (1998)Google Scholar
  30. 30.
    Gillies, M., Crabtree, I.B., Ballin, D.: Individuality and Contextual Variation of Character Behaviour for Interactive Narrative. In: Proceedings of the AISB Workshop on Narrative AI and Games (2006)Google Scholar
  31. 31.
    Carletta, J.: Assessing Agreement on Classification Tasks: The Kappa statistic. Computational Linguistics 22(2), 249–254 (1996)Google Scholar
  32. 32.
    Pantic, M., et al.: Human Computing and Machine Understanding of Human Behavior: A Survey. In: Proc. Int’l Conf. Multimodal Interfaces, pp. 239–248 (2006)Google Scholar
  33. 33.
    Cohn, J.F.: Foundations of human-centered computing: Facial expression and emotion. In: Proceedings of the International Joint Conference on Artificial Intelligence (IJCAI’07), Hyderabad, India (2007)Google Scholar
  34. 34.
    Nogueiras, A., et al.: Speech emotion recognition using hidden Markov models. In: Proceedings of Eurospeech 2001, Denmark (2001)Google Scholar
  35. 35.
    Weizenbaum, J.: ELIZA - A Computer Program For the Study of Natural Language Communication Between Man and Machine. Communications of the ACM 9(1), 36–45 (1966)CrossRefGoogle Scholar

Copyright information

© Springer Berlin Heidelberg 2007

Authors and Affiliations

  • Li Zhang
    • 1
  • Marco Gillies
    • 2
  • John A. Barnden
    • 3
  • Robert J. Hendley
    • 3
  • Mark G. Lee
    • 3
  • Alan M. Wallington
    • 3
  1. 1.School of Computing and Technology, University of East London, Dockland Campus, 4-6 University Way, London, E16 4LZ 
  2. 2.Department of Computer Science, University College London, London, WC1E 6BT 
  3. 3.School of Computer Science, University of Birmingham, Birmingham, B15 2TT 

Personalised recommendations