True Emotion vs. Social Intentions in Nonverbal Communication: Towards a Synthesis for Embodied Conversational Agents

  • Jonathan Gratch
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 4930)

Abstract

Does a facial expression convey privileged information about a person’s mental state or is it a communicative act, divorced from “true” beliefs, desires and intentions? This question is often cast as a dichotomy between competing theoretical perspectives. Theorists like Ekman argue for the primacy of emotion as a determinant of nonverbal behavior: emotions “leak” and only indirectly serve social ends. In contrast, theorists such as Fridlund argue for the primacy of social ends in determining nonverbal displays. This dichotomy has worked to divide virtual character research. Whereas there have been advances in modeling emotion, this work is often seen as irrelevant to the generation of communicative behavior. In this chapter, I review current findings on the interpersonal function of emotion. I’ll discuss recent developments in Social Appraisal theory as a way to bridge this dichotomy and our attempts to model these functions within the context of embodied conversational agents.

Keywords

emotion nonverbal behavior virtual humans cognitive modeling 
This is a preview of subscription content, log in to check access.

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. 1.
    Smith, C.A., Lazarus, R.: Emotion and Adaptation, in Handbook of Personality: theory & research. In: Pervin, L.A. (ed.), pp. 609–637. Guilford Press, New York (1990)Google Scholar
  2. 2.
    Ekman, P.: Universals and cultural differences in facial expressions of emotions. In: Cole, J. (ed.) Nebraska Symposium on Motivation, pp. 207–283. University of Nebraska Press, Lincoln, Nebraska (1972)Google Scholar
  3. 3.
    Frank, R.H.: Introducing Moral Emotions into Models of Rational Choice. In: Manstead, A., Frijda, N., Fischer, A. (eds.) Feelings and Emotions, pp. 422–440. Cambridge University Press, Cambridge (2004)Google Scholar
  4. 4.
    Fridlund, A.: Human facial expression: An evolutionary view. Academic Press, San Diego (1994)Google Scholar
  5. 5.
    Chovil, N.: Social determinants of facial displays. Journal of Nonverbal Behavior 15, 163–167 (1991)CrossRefGoogle Scholar
  6. 6.
    Krämer, N.C., Iurgel, I., Bente, G.: Emotion and motivation in embodied conversational agents. In: Proceedings of the Symposium “Agents that Want and Like”, Artificial Intelligence and the Simulation of Behavior (AISB). Hatfield, SSAISB (2005)Google Scholar
  7. 7.
    Cassell, J., et al.: Human conversation as a system framework: Designing embodied conversational agents. In: Cassell, J., et al. (eds.) Embodied Conversational Agents, pp. 29–63. MIT Press, Boston (2000)Google Scholar
  8. 8.
    Heylen, D.: Challenges Ahead. Head Movements and other social acts in conversation. In: AISB. 2005, Hertfordshire, UK (2005)Google Scholar
  9. 9.
    Poggi, I., Pelachaud, C.: Emotional Meaning and Expression in Performative Faces. In: Paiva, A. (ed.) Affective Interactions: Towards a New Generation of Computer Interfaces, pp. 182–195. Springer, Berlin (2000)Google Scholar
  10. 10.
    Lester, J.C., Towns, S.G., FitzGerald, P.J.: Achieving Affective Impact: Visual Emotive Communication in Lifelike Pedagogical Agents. International Journal of Artificial Intelligence in Education 10(3-4), 278–291 (1999)Google Scholar
  11. 11.
    Elliott, C., Rickel, J., Lester, J.: Lifelike Pedagogical Agents and Affective Computing: An Exploratory Synthesis. In: Wooldridge, M., Veloso, M. (eds.) Artificial Intelligence Today: Recent Trends and Developments, pp. 195–212. Springer, Berlin/Heidelberg (1999)Google Scholar
  12. 12.
    Lisetti, C., Gmytrasiewicz, P.: Can a rational agent afford to be affectless? A formal approach. Applied Artificial Intelligence 16, 577–609 (2002)CrossRefGoogle Scholar
  13. 13.
    Scheutz, M., Sloman, A.: Affect and agent control: Experiments with simple affective states. In: IAT. 2001, World Scientific Publisher, Singapore (2001)Google Scholar
  14. 14.
    Blanchard, A., Cañamero, L.: Developing Affect-modulated behaviors: stability, exploration, exploitation, or imitation? in 6th International Workshop on Epigenetic Robotics, Paris (2006)Google Scholar
  15. 15.
    Marsella, S., Gratch, J.: Modeling coping behaviors in virtual humans: Don’t worry, be happy. In: Second International Joint Conference on Autonomous Agents and Multi-agent Systems, Melbourne, Australia (2003)Google Scholar
  16. 16.
    Neal Reilly, W.S.: Believable Social and Emotional Agents, Carnegie Mellon University: Pittsburgh, PA (1996)Google Scholar
  17. 17.
    Smith, C.A., David, B., Kirby, L.D.: Emotion-Eliciting Appraisals of Social Situations. In: Forgas, J. (ed.) Affective Influences on Social Behavior, Psychology Press, New York (2006)Google Scholar
  18. 18.
    Parkinson, B.: Putting appraisal in context. In: Scherer, K., Schorr, A., Johnstone, T. (eds.) Appraisal processes in emotion: Theory, methods, research, pp. 173–186. Oxford University Press, London (2001)Google Scholar
  19. 19.
    Keltner, D., Haidt, J.: Social Functions of Emotions at Four Levels of Anysis. Cognition and Emotion 13(5), 505–521 (1999)CrossRefGoogle Scholar
  20. 20.
    Barrett, K.C.: A functionalist approach to shame and guilt. In: Tangney, J.P., Fischer, K.W. (eds.) Self-conscious emotions: The psychology of shame, guilt, embarrassment, and pride, pp. 25–63. Guilford Publications, New York (1995)Google Scholar
  21. 21.
    Spoor, J.R., Kelly, J.R.: The evolutionary significance of affect in groups: communication and group bonding. Group Processes and Intergroup Prelations 7(4), 398–412 (2004)CrossRefGoogle Scholar
  22. 22.
    de Waal, F.: Primates and Philosophers: How Morality Evolved. In: Macedo, S., Ober, J. (eds.), Princeton University Press, Princeton (2006)Google Scholar
  23. 23.
    Frank, R.: Passions with reason: the strategic role of the emotions. W. W. Norton, New York (1988)Google Scholar
  24. 24.
    Lutz, C., White, G.M.: The anthropology of emotions. Annual Review of Anthropology 15, 405–436 (1986)CrossRefGoogle Scholar
  25. 25.
    Clore, G.L., Storbeck, J.: Affect as Information about Liking, Efficacy, and Importance. In: Forgas, J. (ed.) Affective Influences on Social Behavior, Psychology Press, New York (2006)Google Scholar
  26. 26.
    Tiedens, L.Z.: The Effect of Anger on the Hostile Inferences of Aggressive and Nonaggressive People: Specific Emotions. Cognitive Processing, and Chronic Accessibility Motivation and Emotion 25(3) (2001)Google Scholar
  27. 27.
    Zak, P.J.: Neuroeconomics. In: Philosophical Transactions of the Royal Society Biology, vol. 359, pp. 1737–1748 (2004)Google Scholar
  28. 28.
    Izard, C.: Human emotion. Plenum, New York (1977)Google Scholar
  29. 29.
    Klauer, K.C., Musch, J.: Affective priming: Findings and theories. In: Klauer, K.C., Musch, J., Mahwah, N.J. (eds.) The Psychology of Evaluation: Affective Processes in Cognition and Emotion, pp. 7–49. Lawrence Erlbaum, Mahwah (2003)Google Scholar
  30. 30.
    Dimberg, U., Öhman, A.: Beyond the wrath: Psychophysiological responses to facial stimuli. Motivation and Emotion 20, 149–182 (1996)CrossRefGoogle Scholar
  31. 31.
    Emde, R.N., Gaensbauer, T.J., Harmon, R.J.: Emotional expression in infancy: A biobehavioral study. Psychological Issues 10 (1976) (1, Monograph No. 37)Google Scholar
  32. 32.
    Eisenberg, N., et al.: Sympathy and personal distress: Development, gender differences, and interrelations of indexes. In: Eisenberg, N. (ed.) Empathy and related emotional responses, pp. 107–126. Jossey-Bass, San Franciso (1989)Google Scholar
  33. 33.
    Haviland, J.M., Lelwica, M.: The induced affect response: 10-week-old infants’ responses to three emotion expressions. Developmental Psychology 23, 97–104 (1987)CrossRefGoogle Scholar
  34. 34.
    Brosig, J.: Identifying cooperative behavior: some experimental results in a prisoner’s dilemma game. Journal of Economic Behavior and Organization 47, 275–290 (2002)CrossRefGoogle Scholar
  35. 35.
    Frank, R.H., Gilovich, T., Regan, D.T.: The evolution of one-shot cooperation: an experiment. Ethology and Sociobiology 14, 247–256 (1993)CrossRefGoogle Scholar
  36. 36.
    Dunn, J.: Children as psychologists: The later correlates of individual differences in understanding of emotions and other minds. Cognition and Emotion 9, 187–201 (1995)CrossRefMathSciNetGoogle Scholar
  37. 37.
    Harris, P.L.: Children and Emotion. Blackwell, Oxford (1989)Google Scholar
  38. 38.
    Eder, R.A.: Uncovering young children’s psychological selves: Individual and developmenal differences. Child Development 61, 849–863 (1990)CrossRefGoogle Scholar
  39. 39.
    Phillips, M.L.: Facial processing deficits and social dysfunction: how are they related? Brain 127(8), 1691–1692 (2004)CrossRefGoogle Scholar
  40. 40.
    Parkinson, B.: Do facial movements express emotion or communicate motives? Personality and Social Psychology Review 9(4), 278–311 (2005)CrossRefGoogle Scholar
  41. 41.
    Manstead, A., Fischer, A.H., Jakobs, E.B.: The Social and Emotional Functions of Facial Displays. In: Philippot, P., Feldman, R.S., Coats, E.J. (eds.) The Social Context of Nonverbal Behavior (Studies in Emotion and Social Interaction), pp. 287–316. Cambridge Univ Press, Cambridge (1999)Google Scholar
  42. 42.
    Fernández Dols, J.M., Ruiz-Belda, M.-A.: Spontaneous facial behavior during intense emotional episodes: artistic truth and optical truth. In: Russell, J.A., Fernández Dols, J.M. (eds.) The Psychology of Facial Expression (1997)Google Scholar
  43. 43.
    Scherer, K.R., Schorr, A., Johnstone, T.: Appraisal Processes in Emotion. In: Davidson, R.J., Ekman, P., Scherer, K.R. (eds.) Affective Science, Oxford University Press, Oxford (2001)Google Scholar
  44. 44.
    Parkinson, B., Fischer, A.H., Manstead, A.S.R.: Emotion in social relations: Cultural, group, and interpersonal processes. Psychology Press, New York (2005)Google Scholar
  45. 45.
    Tangney, J.P.: Assessing individual differences in proneness to shame and guilt: development of the self-conscious affect and attribution inventory. Journal of Personality and Social Psychology 59, 102–111 (1990)CrossRefGoogle Scholar
  46. 46.
    Leary, M.R., Landel, J.L., Patton, K.M.: The motivated expression of embarrassment following a self-presentational predicament. Journal of Personality and Social Psychology 64, 619–636 (1996)Google Scholar
  47. 47.
    Gratch, J., Marsella, S.: Tears and Fears: Modeling Emotions and Emotional Behaviors in Synthetic Agents. In: Fifth International Conference on Autonomous Agents, ACM Press, Montreal, Canada (2001)Google Scholar
  48. 48.
    Gratch, J., Marsella, S.: A domain independent framework for modeling emotion. Journal of Cognitive Systems Research 5(4), 269–306 (2004)CrossRefGoogle Scholar
  49. 49.
    Marsella, S., Gratch, J., Rickel, J.: Expressive Behaviors for Virtual Worlds. In: Prendinger, H., Ishizuka, M. (eds.) Life-like Characters Tools, Affective Functions and Applications, pp. 317–360. Springer, Berlin (2003)Google Scholar
  50. 50.
    Swartout, W., et al.: Toward Virtual Humans. AI Magazine 27(1) (2006)Google Scholar
  51. 51.
    Rickel, J., et al.: Toward a New Generation of Virtual Humans for Interactive Experiences. In: IEEE Intelligent Systems, pp. 32–38 (2002)Google Scholar
  52. 52.
    Weiner, B.: Responsibility for Social Transgressions: An Attributional Analysis. In: Malle, B.F., Moses, L.J., Baldwin, D.A. (eds.) Intentions and Intentionality: Foundations of Social Cognition, The MIT Press, Cambridge (2001)Google Scholar
  53. 53.
    Shaver, K.G.: The attribution of blame: Causality, responsibility, and blameworthiness. Springer, Heidelberg (1985)Google Scholar
  54. 54.
    Mao, W., Gratch, J.: Evaluating a computational model of social causality and responsibility. In: 5th International Joint Conference on Autonomous Agents and Multiagent Systems, Hakodate, Japan (2006)Google Scholar
  55. 55.
    Mao, W., Gratch, J.: Social Judgment in Multiagent Interactions. In: Third International Joint Conference on Autonomous Agents and Multiagent Systems, New York (2004)Google Scholar
  56. 56.
    Austin, J.: How to Do Things with Words. Harvard University Press (1962)Google Scholar
  57. 57.
    Brown, P., Levenson, S.C.: Politeness: Some Universals in Language Usage. Cambridge University Press, New York (1987)Google Scholar
  58. 58.
    Martinovski, B., et al.: Mitigation Theory: An integrated approach. In: Cognitive Science (2005)Google Scholar
  59. 59.
    Shannon, C.E., Weaver, W.: A mathematical theory of communication. University of Illinois Press, Urbana (1948)Google Scholar
  60. 60.
    Fogel, A.: Developing through relationships: Origins of communication, self and culture. Harvester Wheatsheaf, New York (1993)Google Scholar
  61. 61.
    Agre, P., Chapman, D.: Pengi: An implementation of a theory of activity. In: National Conference on Artificial Intelligence (1987)Google Scholar
  62. 62.
    Suchman, L.A.: Plans and situated actions: The problem of human-machine communication. Cambridge University Press, New York (1987)Google Scholar
  63. 63.
    Simon, H.: The sciences of the Artificial. MIT Press, Cambridge (1969)Google Scholar
  64. 64.
    Jondottir, G.R., et al.: Fluid Semantic Back-Channel Feedback in Dialogue: Challenges and Progress. In: Intelligent Virtual Agents, Paris, France, Springer, Heidelberg (2007)Google Scholar
  65. 65.
    Ward, N., Tsukahara, W.: Prosodic features which cue back-channel responses in English and Japanese. Journal of Pragmatics 23, 1177–1207 (2000)CrossRefGoogle Scholar
  66. 66.
    Bartlett, M.S., et al.: Fully automatic facial action recognition in spontaneous behavior. In: 7th International Conference on Automatic Face and Gesture Recognition, Southampton, UK (2006)Google Scholar
  67. 67.
    Morency, L.-P., et al.: Contextual Recognition of Head Gestures. In: 7th International Conference on Multimodal Interactions, Torento, Italy (2005)Google Scholar
  68. 68.
    Gratch, J., et al.: Virtual Rapport. In: Gratch, J., et al. (eds.) IVA 2006. LNCS (LNAI), vol. 4133, Springer, Heidelberg (2006)Google Scholar
  69. 69.
    Breazeal, C., Aryananda, L.: Recognition of Affective Communicative Intent in Robot-Directed Speech. Autonomous Robots 12, 83–104 (2002)MATHCrossRefGoogle Scholar
  70. 70.
    Gratch, J., et al.: Creating Rapport with Virtual Agents. In: 7th International Conference on Intelligent Virtual Agents, Paris, France (2007)Google Scholar
  71. 71.
    Burleson, W.: Affective Learning Companions: Strategies for Empathetic Agents with Real-Time Multimodal Affective Sensing to Foster Meta-Cognitive and Meta-Affective Approaches to Learning, Motivation, and Perseverance, Unpublished PhD Thesis, MIT Media Lab: Boston (2006)Google Scholar
  72. 72.
    Wang, N., et al.: Experimental evaluation of polite interaction tactics for pedagogical agents. In: Intelligent User Interfaces (2005)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2008

Authors and Affiliations

  • Jonathan Gratch
    • 1
  1. 1.University of Southern California 

Personalised recommendations