Foundations of Human Computing: Facial Expression and Emotion

  • Jeffrey F. Cohn
Part of the Lecture Notes in Computer Science book series (LNCS, volume 4451)


Many people believe that emotions and subjective feelings are one and the same and that a goal of human-centered computing is emotion recognition. The first belief is outdated; the second mistaken. For human-centered computing to succeed, a different way of thinking is needed. Emotions are species-typical patterns that evolved because of their value in addressing fundamental life tasks. Emotions consist of multiple components, of which subjective feelings may be one. They are not directly observable, but inferred from expressive behavior, self-report, physiological indicators, and context. I focus on expressive facial behavior because of its coherence with other indicators and research. Among the topics included are measurement, timing, individual differences, dyadic interaction, and inference. I propose that design and implementation of perceptual user interfaces may be better informed by considering the complexity of emotion, its various indicators, measurement, individual differences, dyadic interaction, and problems of inference.


Emotion measurement facial expression automatic facial image analysis human-computer interaction temporal dynamics 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Lazarus, R.S.: Emotion and adaptation. Oxford University Press, New York (1991)Google Scholar
  2. 2.
    Ekman, P.: Emotions revealed. Times, New York (2003)Google Scholar
  3. 3.
    Keltner, D., Haidt, J.: Social functions of emotions at multiple levels of analysis. Cognition and Emotion 13, 505–522 (1999)CrossRefGoogle Scholar
  4. 4.
    Darwin, C.: The expression of the emotions in man and animals, 3rd edn. Oxford University Press, New York (1872/1998)Google Scholar
  5. 5.
    Ekman, P., Rosenberg, E. (eds.): What the face reveals. Oxford University Press, New York (2005)Google Scholar
  6. 6.
    Pantic, M., Patras, I.: Dynamics of facial expressions: Recognition of facial actions and their temporal segments from profile image sequences. IEEE Transactions on Systems, Man, and Cybernetics, Part B 36, 443–449 (2006)CrossRefGoogle Scholar
  7. 7.
    Tian, Y., Cohn, J.F., Kanade, T.: Facial expression analysis. In: Li, S.Z., Jain, A.K. (eds.) Handbook of face recognition, pp. 247–276. Springer, Heidelberg (2005)CrossRefGoogle Scholar
  8. 8.
    Pantic, M., Rothkrantz, M.: Automatic analysis of facial expressions: The state of the art. IEEE Transactions on Pattern Analysis and Machine Intelligence 22, 1424–1445 (2000)CrossRefGoogle Scholar
  9. 9.
    Davidson, R.J., et al.: Approach-withdrawal and cerebral asymmetry: Emotional expression and brain physiology I. Journal of Personality and Social Psychology 58, 330–341 (1990)CrossRefGoogle Scholar
  10. 10.
    Levenson, R.W., Ekman, P., Friesen, W.V.: Voluntary facial action generates emotion-specific autonomic nervous system activity. Psychophysiology 27, 363–384 (1990)CrossRefGoogle Scholar
  11. 11.
    Watson, D., Tellegen, A.: Toward a consensual structure of mood. Psychological Bulletin 98, 219–235 (1985)CrossRefGoogle Scholar
  12. 12.
    Cohn, J.F., Elmore, M.: Effects of contingent changes in mothers’ affective expression on the organization of behavior in 3-month old infants. Infants Behavior and Development 11, 493–505 (1988)CrossRefGoogle Scholar
  13. 13.
    Schmidt, K.L., Cohn, J.F.: Human facial expressions as adaptations: Evolutionary perspectives in facial expression research. Yearbook of Physical Anthropology 116, 8–24 (2001)CrossRefGoogle Scholar
  14. 14.
    Gottman, J., Levenson, R., Woodin, E.: Facial expressions during marital conflict. Journal of Family Communication 1, 37–57 (2001)CrossRefGoogle Scholar
  15. 15.
    Cohn, J.F., Campbell, S.B.: Influence of maternal depression on infant affect regulation. In: Cicchetti, D., Toth, S.L. (eds.) Developmental perspectives on depression, pp. 103–130. University of Rochester Press, Rochester (1992)Google Scholar
  16. 16.
    Cohn, J.F., et al.: Individual differences in facial expression: Stability over time, relation to self-reported emotion, and ability to inform person identification. In: International Conference on Multimodal User Interfaces, Pittsburgh, PA, pp. 491–496 (2002)Google Scholar
  17. 17.
    Bartlett, M.S., et al.: Fully automatic facial action recognition in spontaneous behavior. In: IEEE International Conference on Automatic Face and Gesture Recognition, vol. FG 2006, Southampton, England, pp. 223–228 (2006)Google Scholar
  18. 18.
    Valstar, M.F., et al.: Spontaneous vs. posed facial behavior: Automatic analysis of brow actions. In: ACM International Conference on Multimodal Interfaces, Banff, Canada, pp. 162–170 (2006)Google Scholar
  19. 19.
    Lucey, S., et al.: AAM derived face representations for robust facial action recognition. In: Seventh IEEE International Conference on Automatic Face and Gesture Recognition, Southampton, UK, pp. 155–160 (2006)Google Scholar
  20. 20.
    Boker, S.M., et al.: Coordinated motion and facial expression in dyadic conversation. In: NSF Human and Social Dynamics Principal Investigators Meeting, Washington, DC (2006)Google Scholar
  21. 21.
    Cohn, J.F., Ambadar, Z., Ekman, P.: Observer-based measurement of facial expression with the Facial Action Coding System. In: Coan, J.A., Allen, J.B. (eds.): The handbook of emotion elicitation and assessment. Oxford University Press Series in Affective Science. Oxford University Press, New York (In press)Google Scholar
  22. 22.
    Ekman, P.: An argument for basic emotions. Cognition and Emotion 6, 169–200 (1992)CrossRefGoogle Scholar
  23. 23.
    Ekman, P., Friesen, W.V.: Unmasking the face: A guide to emotions from facial cues. Prentice-Hall Inc., Englewood Cliffs (1975)Google Scholar
  24. 24.
    Ekman, P., Friesen, W.V.: Facial action coding system. Consulting Psychologists Press, Palo Alto (1978)Google Scholar
  25. 25.
    Kanade, T., Cohn, J.F., Tian, Y.: Comprehensive database for facial expression analysis. In: Fourth IEEE International Conference on Automatic Face and Gesture Recognition, vol. 4, Grenoble, pp. 46–53 (2000)Google Scholar
  26. 26.
    Pantic, M., et al.: Affective multimodal human-computer interaction. In: ACM International Conference on Multimedia, pp. 669–676 (2005)Google Scholar
  27. 27.
    Pantic, M., Rothkrantz, M.: Toward an affect-sensitive multimodal human-computer interaction. Proceedings of the IEEE 91, 1371–1390 (2003)CrossRefGoogle Scholar
  28. 28.
    Keltner, D., Ekman, P.: Facial expression of emotion. In: Lewis, M., Haviland, J.M. (eds.) Handbooks of emotions, pp. 236–249. Guilford, New York (2000)Google Scholar
  29. 29.
    Scherer, K.R.: What does facial expression express? In: Strongman, K.T. (ed.) International Review of Studies on Emotion, vol. 2, pp. 138–165. John Wiley & Sons Ltd, Chichester (1992)Google Scholar
  30. 30.
    Izard, C.E., Dougherty, L.M., Hembree, E.A.: A system for identifying affect expressions by holistic judgments. Instructional Resources Center, University of Delaware, Newark, Delaware (1983)Google Scholar
  31. 31.
    Ekman, P., Friesen, W.V., O’Sullivan, M.: Smiles when lying. Journal of Personality and Social Psychology 54, 414–420 (1988)CrossRefGoogle Scholar
  32. 32.
    Cohn, J.F., Ekman, P.: Measuring facial action by manual coding, facial EMG, and automatic facial image analysis. In: Harrigan, J.A., Rosenthal, R., Scherer, K. (eds.) Handbook of nonverbal behavior research methods in the affective sciences, pp. 9–64. Oxford University Press, New York (2005)Google Scholar
  33. 33.
    Ekman, P., Friesen, W.V., Hager, J.C. (eds.): Facial action coding system. Research Nexus, Network Research Information, Salt Lake City, UT (2002)Google Scholar
  34. 34.
    Rosenthal, R.: Conducting judgment studies. In: Harrigan, J.A., Rosenthal, R., Scherer, K.R. (eds.) Handbook of nonverbal behavior research methods in the affective sciences, pp. 199–236. Oxford University Press, New York (2005)Google Scholar
  35. 35.
    Oster, H., Hegley, D., Nagel, L.: Adult judgments and fine-grained analysis of infant facial expressions: Testing the validity of a priori coding formulas. Developmental Psychology 28, 1115–1131 (1992)CrossRefGoogle Scholar
  36. 36.
    Cohn, J.F., Kanade, T.: Use of automated facial image analysis for measurement of emotion expression. In: Coan, J.A., Allen, J.B. (eds.) The handbook of emotion elicitation and assessment, Oxford Unversity Press, New York (in press)Google Scholar
  37. 37.
    Eibl-Eibesfeldt, I.: Human ethology. Aldine de Gruvier, New York (1989)Google Scholar
  38. 38.
    Sayette, M.A., et al.: A psychometric evaluation of the Facial Action Coding System for assessing spontaneous expression. Journal of Nonverbal Behavior 25, 167–186 (2001)CrossRefGoogle Scholar
  39. 39.
    Fridlund, A.J., et al.: Audience effects on solitary faces during imagery: Displaying to the people in your head. Journal of Nonverbal Behavior 14, 113–137 (1990)CrossRefGoogle Scholar
  40. 40.
    Cohn, J.F., Schmidt, K.L.: The timing of facial motion in posed and spontaneous smiles. International Journal of Wavelets, Multiresolution and Information Processing 2, 1–12 (2004)CrossRefMathSciNetGoogle Scholar
  41. 41.
    Fleiss, J.L.: Statistical methods for rates and proportions. Wiley, New York (1981)MATHGoogle Scholar
  42. 42.
    Messinger, D.S., et al.: Continuously measured smile dynamics in infant-mother interaction. Miami (2006)Google Scholar
  43. 43.
    Ibanez, L., et al.: Automated measurement of infant and mother interactive smiling. American Psychological Society (2006)Google Scholar
  44. 44.
    Russell, J.A., Fernandez-Dols, J.M. (eds.): The psychology of facial expression. Cambridge University Press, Cambridge (1997)Google Scholar
  45. 45.
    Ekman, P.: Facial expression and emotion. American Psychologist 48, 384–392 (1993)CrossRefGoogle Scholar
  46. 46.
    Cohn, J.F.: Automated analysis of the configuration and timing of facial expression. In: Ekman, P., Rosenberg, E. (eds.) What the face reveals, pp. 388–392. Oxford University Press, New York (2005)Google Scholar
  47. 47.
    Theobald, B.J., Cohn, J.F.: Facial image synthesis. In: Sander, D., Scherer, K.R. (eds.): Oxford companion to affective sciences: An encyclopedic dictionary for the affective sciences. Oxford University Press, New York (In press)Google Scholar
  48. 48.
    Edwards, K.: The face of time: Temporal cues in facial expressions of emotion. Psychological Science 9, 270–276 (1998)CrossRefGoogle Scholar
  49. 49.
    Krumhuber, E., Kappas, A.: Moving smiles: The role of dynamic components for the perception of the genuineness of smiles. Journal of Nonverbal Behavior 29, 3–24 (2005)CrossRefGoogle Scholar
  50. 50.
    Cohn, J.F., et al.: Multimodal coordination of facial action, head rotation, and eye motion. In: Sixth IEEE International Conference on Automatic Face and Gesture Recognition, vol. FG’04, Seoul, Korea, pp. 645–650 (2004)Google Scholar
  51. 51.
    Ambadar, Z., Schooler, J., Cohn, J.F.: Deciphering the enigmatic face: The importance of facial dynamics to interpreting subtle facial expressions. Psychological Science 16, 403–410 (2005)CrossRefGoogle Scholar
  52. 52.
    Ambadar, Z., Cohn, J.F., Reed, L.I.: All smiles are not created equal: Timing characteristics and interpretation of spontaneous smiles (2006)Google Scholar
  53. 53.
    Dimberg, U., Thunberg, M., Grunedal, S.: Facial reactions to emotional stimuli: Automatically controlled emotional responses. Cognition and Emotion 16, 449–471 (2002)CrossRefGoogle Scholar
  54. 54.
    Moore, G.A., Cohn, J.F., Campbell, S.B.: Mothers’ affective behavior with infant siblings: Stability and change. Developmental Psychology 33, 856–860 (1997)CrossRefGoogle Scholar
  55. 55.
    Oster, H., et al.: The patterning of facial expressions in Chinese, Japanese, and American infants in fear- and anger- eliciting situations. Poster presented at the International Conference on Infant Studies, Providence, RI (1996)Google Scholar
  56. 56.
    Matsumoto, D., Willingham, B.: The thrill of victory and the agony of defeat: spontaneous expressions of medal winners of the 2004 Athens olympic games. Journal of Personality & Social Psychology 91, 568–581 (2006)CrossRefGoogle Scholar
  57. 57.
    Camras, L.A., Chen, Y.: Culture, ethnicity, and children’s facial expressions: A study of European American, mainland Chinese, Chinese American, and adopted Chinese girls. 6, 103-114 (2006)Google Scholar
  58. 58.
    Markus, H.R., Kitayama, S.: Culture and the self: Implications for cognition, emotion, and motivation. Psychological Review 98, 224–253 (1991)CrossRefGoogle Scholar
  59. 59.
    Tsai, J.L., Levenson, R.W., McCoy, K.: Cultural and temperamental variation in emotional response. Emotion 6, 484–497 (2006)CrossRefGoogle Scholar
  60. 60.
    Zlochower, A.J., Cohn, J.F.: Vocal timing in face-to-face interaction of clinically depressed and nondepressed mothers and their 4-month-old infants. Infant Behavior and Development 19, 373–376 (1996)CrossRefGoogle Scholar
  61. 61.
    Haidt, J., Keltner, D.: Culture and facial expression: Open-ended methods find more expressions and a gradient of recognition. Emotion and Cognition 13, 225–266 (1999)CrossRefGoogle Scholar
  62. 62.
    Jaffe, J., et al.: Rhythms of dialogue in early infancy. Monographs of the Society for Research in Child Development 66 (2001)Google Scholar
  63. 63.
    Cohn, J.F., Tronick, E.Z.: Mother-Infant face-to-face interaction: Influence is bidirectional and unrelated to periodic cycles in either partner’s behavior. Developmental Psychology 34, 386–392 (1988)CrossRefGoogle Scholar
  64. 64.
    Cohn, J.F., Tronick, E.Z.: Mother infant interaction: The sequence of dyadic states at three, six, and nine months. Developmental Psychology 23, 68–77 (1987)CrossRefGoogle Scholar
  65. 65.
    Chow, S.M., et al.: Emotion as a thermostat: representing emotion regulation using a damped oscillator model. Emotion 5, 208–225 (2005)CrossRefGoogle Scholar
  66. 66.
    Boker, S.M., et al.: Windowed cross-correlation and peak picking for the analysis of variability in the association between behavioral time series. Psychological Methods 7, 338–355 (2002)CrossRefGoogle Scholar

Copyright information

© Springer Berlin Heidelberg 2007

Authors and Affiliations

  • Jeffrey F. Cohn
    • 1
  1. 1.Department of Psychology, University of Pittsburgh, 3137 SQ, 210 S. Bouquet Street, Pittsburgh, PA 15260USA

Personalised recommendations