User Modeling and User-Adapted Interaction

, Volume 11, Issue 4, pp 279–295 | Cite as

How Convincing is Mr. Data's Smile: Affective Expressions of Machines

  • Christoph Bartneck


Emotions should play an important role in the design of interfaces because people interact with machines as if they were social actors. This paper presents a literature review on affective expressions through speech, music and body language. It summarizes the quality and quantity of their parameters and successful examples of synthesis. Moreover, a model for the convincingness of affective expressions, based on Fogg and Hsiang Tseng (1999), was developed and tested. The empirical data did not support the original model and therefore this paper proposes a new model, which is based on appropriateness and intensity of the expressions. Furthermore, the experiment investigated if the type of emotion (happiness, sadness, anger, surprise, fear and disgust), knowledge about the source (human or machine), the level of abstraction (natural face, computer rendered face and matrix face) and medium of presentation (visual, audio/visual, audio) of an affective expression influences its convincingness and distinctness. Only the type of emotion and multimedia presentations had an effect on convincingness. The distinctness of an expression depends on the abstraction and the media through which it is presented.

abstraction affective expressions convincingness distinctness emotion face modality music speech 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. Bartneck, C.: 2000, Affective Expressions of Machines, Technical University of Eindhoven, The Netherlands. Also available at Scholar
  2. Barnes, A. and Thagard, P.: 1996, Emotional decisions. Proceedings of the Eighteen's Annual Conference of the Cognitive Science Society, University of California, pp. 426-429.Google Scholar
  3. Brennan, S. E.: 1985, The caricature generator. Leonardo 18, 170-178.Google Scholar
  4. Bresin, R. and Friberg, A.: 1999, Synthesis and decoding of emotionally expressive music performance. Proceedings of the IEEE 1999 Systems, Man and Cybernetics Conference (SMC'99), Tokyo, pp. 317-322.Google Scholar
  5. Cahn, J. E.: 1990, The generation of affect in synthesized speech. Journal of American Voice I/O Society, 8, 1-19.Google Scholar
  6. Cassell, J., Pelachaud, C., Badler, N., Steedman, M., Achorn, B., Becket, W., Douville, B., Prevost, S. and Stone, M.: 1998, Animated conversation: rule-based generation of facial expression, gesture and spoken intonation for multiple conversational agents. In: M. Maybury and W. Wahlster (eds.), Readings in Intelligent User Interfaces, Morgan Kaufmann Publishers.Google Scholar
  7. Crowder, R. G.: 1984, Perception of the major/minor distinction: I. Historical and theoretical foundations. Psychomusicology, 4, 3-10.Google Scholar
  8. CSLU, 1999, CSLU Toolkit. Scholar
  9. Cunningham, J. G. and Sterling, R. S.: 1988, Development change in the understanding of affective meaning in music. Motivation and Emotion, 12, 399-413.Google Scholar
  10. Davies, J. B. (ed.): 1978, The Psychology of Music. Stanford, Stanford University Press.Google Scholar
  11. Ekman, P. (ed.): 1973, Darwin and Facial Expression: A Century of Research in Review. New York: Academic Press.Google Scholar
  12. Ekman, P., Friesen, W. V. and Ellsworth, P. (eds.): 1972, Emotion in the Human Face: Guidelines for Research and an Integration of Findings. New York: Pergamon Press.Google Scholar
  13. Ekman, P., Friesen, W.V., O'Sullivan, M. and Scherer, K.: 1980, Relative importance of face, body, and speech in judgments of personality and affect. Journal of Personality and Social Psychology, 38, 270-277.Google Scholar
  14. Ekman, P. and Frieser, W. (eds.): 1976, Pictures of Facial Affects. Palo Alto: Consulting Psychologist Press.Google Scholar
  15. Etcoff, N. L. and Magee, J. J.: 1992, Categorical perception of facial expressions. Cognition, 44, 227-240.Google Scholar
  16. Feist, G.: 1994, The affective cpmsequences of artistic and scientific problem solving. Cognition and Emotion, 8, 489-502.Google Scholar
  17. Fogg, B.J. and Hsiang Tseng: 1999, The elements of computer credibility. Proceedings of CHI 99, Pittsburgh, pp. 80-87.Google Scholar
  18. Frijda, N. H. (ed.): 1986, The Emotions. Cambridge, Cambridge University Press.Google Scholar
  19. Frijda, N. H.: 1969, Recognition of emotion. Advances in Experimental Social Psychology, 4, 167-223.Google Scholar
  20. Gerardi, G. M. and Gerken, L.: 1995, The development of affective responses to modality and melodic contour. Music Perception, 12, 279-290.Google Scholar
  21. Izard, C. E. (ed.): 1977, Human Emotions. New York: Plenum Press.Google Scholar
  22. Juslin, P. N.: 1997a, Emotional communication in music performance: a functionalist perspective and some data. Music Perception, 14, 383-418.Google Scholar
  23. Juslin, P. N.: 1997b, Perceived emotional expression in synthesized performances of a short melody: capturing the listener's judgment policy. Musicae Scientiae, 2, 225-256.Google Scholar
  24. Kastner, M. P. and Crowder, R. G.: 1990, Perception of major/minor: IV. Emotional connotation in young children. Music Perception, 8, 189-202.Google Scholar
  25. Krauss, R. M., Morrel-Samuels, P. and Colasante, C.: 1991, Do conversational hand gestures communicate? Journal of Personality and Social Psychology, 743-754.Google Scholar
  26. Lazarus, R. S. (ed.): 1991, Emotion and Adaptation. Oxford, Oxford University Press.Google Scholar
  27. Massaro, D. W. (ed.): 1998, Perceiving Talking Faces: From Speech Perception to a Behavioral Principle. MIT Press.Google Scholar
  28. McNeill, D. (ed.): 1992, Hand and Mind: What Gestures Reveal about Thought. Chicago, University of Chicago.Google Scholar
  29. Miller, G.A.: 1956, The Magical Number Seven, Plus or Minus Two: Some Limits on Our Capacity for Processing Information. The Psychological Review, 63, 81-97.Google Scholar
  30. Mozziconacci, S. (ed.): 1998, Speech Variability and Emotions: Production and Perception. Ph.D. thesis, IPO, Technical University of Eindhoven.Google Scholar
  31. Murray, I. R. and Arnott, J. L.: 1992, Towards the simulation of emotion in synthetic speech: A review of the literature on human vocal emotion. Journal of the Acoustic Society of America, 93, 1097-1108.Google Scholar
  32. Nass, C. and Reeves, B. (eds.): 1996, The Media Equation. Cambridge, SLI Publications, Cambridge University Press.Google Scholar
  33. Nielsen, J. (ed.): 1994, Usability Engineering. San Diego: AP Professional.Google Scholar
  34. Nobe, S., Hayamizu, S., Hasegawa, O. and Takahashi, H.: 1997, Are listeners paying attention to the hand gestures of an anthropomorphic agent? An evaluation using a gaze tracking method'. In: I. Wachsmuth and M. Fröhlich (eds.), Gesture and Sign Language in Human-Computer Interaction, Springer.Google Scholar
  35. Norman, D. A.: 1981, Twelve issues for cognitive science. In: D. Norman (ed.), Perspectives on Cognitive Science, Norwood: Ablex Publishing, pp. 265-295.Google Scholar
  36. Nowlis, V.: 1966, Research with the mood adjective check list. In: S. S. Tomkin and C. E. Izard (eds.), Affect, Cognition and Personality, New York, Springer, pp. 352-389.Google Scholar
  37. Oatley, K. and Jenkins, J. M. (eds.): 1996, Understanding Emotions. Oxford: Blackwell.Google Scholar
  38. Osgood, C. E., Suci, G. J. and Tannenbaum, P. H. (eds.): 1957, The Measurements of Meaning. Champaign: University of Illinois Press.Google Scholar
  39. Picard, R. (ed.): 1997a, Affective Computing. Cambridge: MIT Press.Google Scholar
  40. Picard, R.: 1997b, Does HAL cry digital tears? Emotion and computers. In: D. Stork (ed.): Hal's Legacy: 2001's Computer as Dream and Reality, Cambridge: MIT Press.Google Scholar
  41. Pixar, 1998, Geri's game. Scholar
  42. Plutchik, R. (ed.): 1980, Emotion: A Psycho Evolutionary Synthesis. New York, Harper and Row.Google Scholar
  43. Russel, J. A.: 1979, Affective space is bipolar. Journal of Personality and Social Psychology, 37, 345-356.Google Scholar
  44. Scherer, K. R.: 1979, Nonlinguistic vocal indicators of emotion and psychopathology. In: C. E. Izard (eds.), Emotions in Personality and Psychopathology, New York, Plenum.Google Scholar
  45. Scherer, K. R. and Oshinsky, J. S.: 1977, Cue utilization in emotion attribution from auditory stimuli. Motivation and Emotion, 4, 331-346.Google Scholar
  46. Schiano, D. J., Ehrlich, S. M., Rahardja, K. and Sheridan, K.: 2000, Face to InterFace: Facial affect in (Hu)Man and Machine. CHI 2000 Conference Proceedings, The Hague, pp. 193-200.Google Scholar
  47. Schlossberg, H.: 1954, Three dimensions of emotion. Psychological Review, 61, 81-88.Google Scholar
  48. Smith, C. A. and Ellsworth, P. C.: 1985, Patterns of cognitive appraisal in emotion. Journal of Personality and Social Psychology, 48, 813-838.Google Scholar
  49. Sony, 1999, Aibo. Scholar

Copyright information

© Kluwer Academic Publishers 2001

Authors and Affiliations

  • Christoph Bartneck
    • 1
  1. 1.Eindhoven University of TechnologyEindhovenThe Netherlands.

Personalised recommendations