Facial Expression Analysis

  • Ying-Li Tian
  • Takeo Kanade
  • Jeffrey F. Cohn

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. 1.
    J. Ahlberg and R. Forchheimer. Face tracking for model-based coding and face animation. International Journal of Imaging Systems and Technology, pages 8–22, Vol. 13,Issue 1, 2003.CrossRefGoogle Scholar
  2. 2.
    S. Baker and T. Kanade. Limits on super-resolution and how to break them. IEEE Transactions on Pattern Analysis and Machine Intelligence, 24(9):1167–1183, 2002.CrossRefGoogle Scholar
  3. 3.
    M. Bartlett, B. Braathen, G. Littlewort-Ford, J. Hershey, I. Fasel, T. Marks, E. Smith, T. Sejnowski, and J.R. Movellan. Automatic analysis of spontaneous facial behavior: A final project report. Technical Report INC-MPLab-TR-2001.08, Machine Perception Lab, Institute for Neural Computation, University of California, San Diego, 2001.Google Scholar
  4. 4.
    M. Bartlett, J. Hager, P. Ekman, and T. Sejnowski. Measuring facial expressions by computer image analysis. Psychophysiology, 36:253–264, 1999.CrossRefGoogle Scholar
  5. 5.
    M. Black. Robust incremental optical flow. In PhD thesis, Yale University, 1992.Google Scholar
  6. 6.
    M. Black and Y. Yacoob. Tracking and recognizing rigid and non-rigid facial motions using local parametric models of image motion. In Proc. of International conference on Computer Vision, pages 374–381, 1995.Google Scholar
  7. 7.
    M. Black and Y. Yacoob. Recognizing facial expressions in image sequences using local parameterized models of image motion. International Journal of Computer Vision, 25(1):23–48, October 1997.CrossRefGoogle Scholar
  8. 8.
    L. Brown and Y.-L. Tian. Comparative study of coarse head pose estimation. In IEEE Workshop on Motion and Video Computing, Orlando, 2002.Google Scholar
  9. 9.
    L. Camras, L. Lambrecht, and G. Michel. Infant surprise expressions as coordinative motor structures. Journal of Nonverbal Behavior, 20:183–195, 1966.CrossRefGoogle Scholar
  10. 10.
    J. Carroll and J. Russell. Do facial expression signal specific emotions? Journal of Personality and Social Psychology., 70:205–218, 1996.CrossRefGoogle Scholar
  11. 11.
    J. Carroll and J. Russell. Facial expression in hollywood’s portrayal of emotion. Journal of Personality and Social Psychology., 72:164–176, 1997.CrossRefGoogle Scholar
  12. 12.
    L. Chen. Joint processing of ausio-visual information for the recognition of emotional expressions in human-computer interaction. In PhD thesis, University of Illinois at Urbana-Champaign, Department of Electrical Engineering, 2000.Google Scholar
  13. 13.
    I. Cohen, N. Sebe, F. Cozman, M. Cirelo, and T. Huang. Coding, analysis, interpretation, and recognition of facial expressions. Journal of Computer Vision and Image Understanding Special Issue on Face Recognition, 2003.Google Scholar
  14. 14.
    J. Cohn, T. Kanade, T. Moriyama, Z. Ambadar, J. Xiao, J. Gao, and H. Imamura. A comparative study of alternative facs coding algorithms. Technical Report CMU-RI-TR-02-06, Robotics Institute, Carnegie Mellon University, Pittsburgh, November 2001.Google Scholar
  15. 15.
    J. Cohn and G. Katz. Bimodal expression of emotion by face and voice. In ACM and ATR Workshop on Face/Gesture Recognition and Their Applications, pages 41–44, 1998.Google Scholar
  16. 16.
    J. Cohn, K. Schmidt, R. Gross, and P. Ekman. Individual differences in facial expression: stability over time, relation to self-reported emotion, and ability to inform person identification. In Proceedings of the International Conference on Multimodal User Interfaces (ICMI 2002), pages 491–496, 2002.Google Scholar
  17. 17.
    J. Cohn, A. Zlochower, J. Lien, and T. Kanade. Automated face analysis by feature point tracking has high concurrent validity with manual facs coding. Psychophysiology, 36:35–43, 1999.CrossRefGoogle Scholar
  18. 18.
    C. Darwin. The Expression of Emotions in Man and Animals. John Murray, reprinted by University of Chicago Press, 1965, 1872.Google Scholar
  19. 19.
    J. Daugmen. Complete discrete 2d gabor transforms by neutral networks for image analysis and compression. IEEE Transaction on Acoustic, Speech and Signal Processing, 36(7):1169–1179, 1988.CrossRefGoogle Scholar
  20. 20.
    G. Donato, M. Bartlett, J. Hager, P. Ekman, and T. Sejnowski. Classifying facial actions. IEEE Transaction on Pattern Analysis and Machine Intelligence, 21(10):974–989, 1999.CrossRefGoogle Scholar
  21. 21.
    Eihl-Eihesfeldt. Human Ethology. Aldine de Gruvter, New York, 1989.Google Scholar
  22. 22.
    P. Ekman. The Argument and Evidence about Universals in Facial Expressions of Emotion, pages 143–164. Wiley, New York, 1989.Google Scholar
  23. 23.
    P. Ekman. Facial expression and emotion. American Psychologist, 48:384–392, 1993.CrossRefGoogle Scholar
  24. 24.
    P. Ekman and W. Friesen. Pictures of Facial Affect. Palo Alto, CA: Consulting Psychologist., 1976.Google Scholar
  25. 25.
    P. Ekman and W. Friesen. The Facial Action Coding System: A Technique for the Measurement of Facial Movement. Consulting Psychologists Press, San Francisco, 1978.Google Scholar
  26. 26.
    P. Ekman, J. Hager, C. Methvin, and W. Irwin. Ekman-Hager Facial Action Exemplars. Human Interaction Laboratory, University of California, San Francisco.Google Scholar
  27. 27.
    P. Ekman and E. e. Rosenberg. What the Face Reveals. Oxford University, New York, 1997.Google Scholar
  28. 28.
    I. Essa and A. Pentland. Coding, analysis, interpretation, and recognition of facial expressions. IEEE Trans. on Pattern Analysis and Machine Intell., 19(7):757–763, 1997.CrossRefGoogle Scholar
  29. 29.
    L. Farkas and I. Munro. Anthropometric Facial Proportions in Medicine. Charles C Thomas, Springfield, IL, 1987.Google Scholar
  30. 30.
    B. Fasel and J. Luttin. Recognition of asymmetric facial action unit activities and intensities. In Proceedings of International Conference of Pattern Recognition, 2000.Google Scholar
  31. 31.
    B. Fasel and J. Luttin. Automatic facial expression analysis: Survey. Pattern Recognition, 36(1):259–275, 2003.CrossRefMATHGoogle Scholar
  32. 32.
    J. Fleiss. Statistical Methods for Rates and Proportions. Wiley, New York, 1981.MATHGoogle Scholar
  33. 33.
    G. Ford. Fully automatic coding of basic expressions from video. Technical Report INC-MPLab-TR-2002.03, Machine Perception Lab, Institute for Neural Computation, University of California, San Diego, 2002.Google Scholar
  34. 34.
    N. Fox, R. Gross, P. de Chazal, J. Cohn, and R. Reilly. Person identification using multi-modal features: speech, lip, and face. In Proc. of ACM Multimedia Workshop in Biometrics Methods and Applications (WBMA 2003), CA, 2003.Google Scholar
  35. 35.
    N. Fox and R. Reilly. Audio-visual speaker identification. In Proc. of the 4th International Conference on Audio-and Video-Based Biometric Person Authentication, 2003.Google Scholar
  36. 36.
    M. Frank and P. Ekman. The ability to detect deceit generalizes across different types of high-stake lies. Personality and Social Psychology, 72:1429–1439, 1997.CrossRefGoogle Scholar
  37. 37.
    W. Friesen and P. Ekman. Emfacs-7: emotional facial action coding system. Unpublished manuscript, University of California at San Francisco, 1983.Google Scholar
  38. 38.
    H. Fujiyoshi and A. Lipton. Real-time human motion analysis by image skeletonization. In Proc. of the Workshop on Application of Computer Vision, 1998.Google Scholar
  39. 39.
    K. Fukui and O. Yamaguchi. Facial feature point extraction method based on combination of shape extraction and pattern matching. Systems and Computers in Japan, 29(6):49–58, 1998.CrossRefGoogle Scholar
  40. 40.
    A. Hampapur, S. Pankanti, A. Senior, Y. Tian, L. Brown, and R. Bolle. Face cataloger: multi-scale imaging for relating identity to location. In Proceedings of IEEE Conference on Advanced Video and Signal Based Surveillance, 2003.Google Scholar
  41. 41.
    B. Heisele, T. Serre, M. Pontil, and T. Poggio. Component-based face detection. In Proc. IEEE Conf. on Computer Vision and Pattern Recogn. (CVPR), 2001.Google Scholar
  42. 42.
    C. Izard, L. Dougherty, and E. A. Hembree. A system for identifying affect expressions by holistic judgments. In Unpublished Manuscript, University of Delaware, 1983.Google Scholar
  43. 43.
    T. Kanade, J. Cohn, and Y.-L. Tian. Comprehensive database for facial expression analysis. In Proceedings of International Conference on Face and Gesture Recognition, pages 46–53, 2000.Google Scholar
  44. 44.
    S. Kimura and M. Yachida. Facial expression recognition and its degree estimation. In Proc. Of the International Conference on Computer Vision and Pattern Recognition, pages 295–300, 1997.Google Scholar
  45. 45.
    R. Kleck and M. Mendolia. Decoding of profile versus full-face expressions of affect. Journal of Nonverbal Behavior, 14(1):35–49, 1990.CrossRefGoogle Scholar
  46. 46.
    H. Kobayashi, K. Tange, and F. Hara. Dynamic recognition of six basic facial expressions by discretetime recurrent neural network. In Proc. of the International Joint Conference on Neural Networks, pages 155–158, 1993.Google Scholar
  47. 47.
    H. Kobayashi, K. Tange, and F. Hara. Real-time recognition of six basic facial expressions. In Proc. IEEE Workshop on Robot and Human Communication, pages 179–186, 1995.Google Scholar
  48. 48.
    R. Kraut and R. Johnson. Social and emotional messages of smiling: an ethological approach. Journal of Personality and Social Psychology, 37:1539–1523, 1979.CrossRefGoogle Scholar
  49. 49.
    S. Li and L. Gu. Real-time multi-view face detection, tracking, pose estimation, alignment, and recognition. In IEEE Conf. on Computer Visioin and Pattern Recognition Demo Summary, 2001.Google Scholar
  50. 50.
    J.-J. Lien, T. Kanade, J. Cohn, and C. Li. Subtly different facial expression recognition and expression intensity estimation. In Proceedings of the IEEE International Conference on Computer Vision and Pattern Recognition, pages 853–859, 1998.Google Scholar
  51. 51.
    J.-J. Lien, T. Kanade, J. Cohn, and C. Li. Detection, tracking, and classification of action units in facial expression. Journal of Robotics and Autonomous System, 31:131–146, 2000.CrossRefGoogle Scholar
  52. 52.
    M. Lyons, S. Akamasku, M. Kamachi, and J. Gyoba. Coding facial expressions with gabor wavelets. In Proceedings of International Conference on Face and Gesture Recognition, 1998.Google Scholar
  53. 53.
    A. Manstead. Expressiveness as an individual difference, pages 285–328. Cambridge University Press, Cambridge, UK, 1991.Google Scholar
  54. 54.
    P. Martin and P. Bateson. Measuring behavior: An Introductory Guide. Cambridge University Press, Cambridge, UK, 1986.Google Scholar
  55. 55.
    K. Mase. Recognition of facial expression from optical flow. IEICE Transactions, E. 74(10):3474–3483, 1991.Google Scholar
  56. 56.
    R. Matias, J. Cohn, and S. Ross. A comparison of two systems to code infants’ affective expression. Developmental Psychology, 25:483–489, 1989.CrossRefGoogle Scholar
  57. 57.
    T. Moriyama, T. Kanade, J. Cohn, J. Xiao, Z. Ambadar, J. Gao, and M. Imanura. Automatic recognition of eye blinking in spontaneously occurring behavior. In Proceedings of the 16th International Conference on Pattern Recognition (ICPR’ 2002), volume 4, pages 78–81, 2002.Google Scholar
  58. 58.
    Y. Moses, D. Reynard, and A. Blake. Determining facial expressions in real time. In Proc. of Int. Conf. On Automatic Face and Gesture Recognition, pages 332–337, 1995.Google Scholar
  59. 59.
    M. Pantic and L. Rothkrantz. Automatic analysis of facial expressions: the state of the art. IEEE Transaction on Pattern Analysis and Machine Intelligence, 22(12):1424–1445, 2000.CrossRefGoogle Scholar
  60. 60.
    M. Pantic and L. Rothkrantz. Expert system for automatic analysis of facial expression. Image and Vision Computing, 18(11):881–905, 2000.CrossRefGoogle Scholar
  61. 61.
    A. Pentland, B. Moghaddam, and T. Starner. View-based and modular eigenspaces for face recognition. In Proc. IEEE Conf. Computer Vision and Pattern Recognition, pages 84–91, 1994.Google Scholar
  62. 62.
    F. Pighin, H. Szeliski, and D. Salesin. Synthesizing realistic facial expressions from photographs. In Proc of SIGGRAPH, 1998.Google Scholar
  63. 63.
    W. Rinn. The neuropsychology of facial expression: a review of the neurological and psychological mechanisms for producing facial expressions. Psychological Bulletin, (95):52–77, 1984.Google Scholar
  64. 64.
    S. Rizvi, P. Phillips, and H. Moon. The feret verification testing protocol for face recognition algorithms. In Proceedings of the Third International Conference on Automatic Face and Gesture Recognition, pages 48–55, 1998.Google Scholar
  65. 65.
    M. Rosenblum, Y. Yacoob, and L. Davis. Human expression recognition from motion using a radial basis function network architecture. IEEE Transactions On Neural Network, 7(5):1121–1138, 1996.CrossRefGoogle Scholar
  66. 66.
    H. Rowley, S. Baluja, and T. Kanade. Neural network-based face detection. IEEE Transactions On Pattern Analysis and Machine intelligence, 20(1):23–38, 1998.CrossRefGoogle Scholar
  67. 67.
    J. Russell. Culture and the categorization. Psychological Bulletin, 110:426–450, 1991.CrossRefGoogle Scholar
  68. 68.
    J. Russell. Is there universal recognition of emotion from facial expression? a review of the crosscultural studies. Psychological Bulletin, 115:102–141, 1991.CrossRefGoogle Scholar
  69. 69.
    A. Samal and P. Iyengar. Automatic recognition and analysis of human faces and facil expressions: A survey. Pattern Recognition, 25(1):65–77, 1992.CrossRefGoogle Scholar
  70. 70.
    M. Sayette, J. Cohn, J. Wertz, M. Perrott, and D. Parrott. A psychometric evaluation of the facial action coding system for assessing spontaneous expression. Journal of Nonverbal Behavior, 25:167–186, 2001.CrossRefGoogle Scholar
  71. 71.
    K. Scherer and P. Ekman. Handbook of Methods in Nonverbal Behavior Research. Cambridge University Press, Cambridge, UK, 1982.Google Scholar
  72. 72.
    K. Schmidt and J. Cohn. Human facial expressions as adaptations: Evolutionary questions in facial expression. Yearbook of Physical Anthropology, 44:3–24, 2002.Google Scholar
  73. 73.
    K. Schmidt, J. F. Cohn, and Y.-L. Tian. Signal characteristics of spontaneous facial expressions: Automatic movement in solitary and social smiles. Biological Psychology, 2003.Google Scholar
  74. 74.
    H. Schneiderman and T. Kanade. A statistical model for 3d object detection applied to faces and cars. In IEEE Conference on Computer Vision and Pattern Recognition. IEEE, 2000.Google Scholar
  75. 75.
    K. Sung and T. Poggio. Example-based learning for view-based human face detection. IEEE Transactions on Pattern Analysis and Machine intelligence, 20(1):39–51, 1998.CrossRefGoogle Scholar
  76. 76.
    M. Suwa, N. Sugie, and K. Fujimora. A preliminary note on pattern recognition of human emotional expression. In International Joint Conference on Pattern Recognition, pages 408–410, 1978.Google Scholar
  77. 77.
    H. Tao and T. Huang. Explanation-based facial motion tracking using a piecewise bezier volume deformation model. In Proc. IEEE Conf. Computer Vision and Pattern Recognition, 1999.Google Scholar
  78. 78.
    D. Terzopoulos and K. Waters. Analysis of facial images using physical and anatomical models. In IEEE International Conference on Computer Vision, pages 727–732, 1990Google Scholar
  79. 79.
    Y.-L. Tian and R. Bolle. Automatic detecting neutral face for face authentication. In Proceedings of AAAI-03 Spring Symposium on Intelligent Multimedia Knowledge Management, CA, 2003.Google Scholar
  80. 80.
    Y.-L. Tian, L. Brown, A. Hampapur, S. Pankanti, A. Senior, and R. Bolle. Real world real-time automatic recognition of facial expressions. In Proceedings of IEEE Workshop on Performance Evaluation of Tracking and Surveillance, Graz, Austria, 2003.Google Scholar
  81. 81.
    Y.-L. Tian, T. Kanade, and J. Cohn. Recognizing action units for facial expression analysis. IEEE Trans. on Pattern Analysis and Machine Intell., 23(2):1–19, 2001.MATHGoogle Scholar
  82. 82.
    Y.-L. Tian, T. Kanade, and J. Cohn. Evaluation of gabor-wavelet-based facial action unit recognition in image sequences of increasing complexity. In Proceedings of the 5th IEEE International Conference on Automatic Face and Gesture Recognition (FG’02), Washington, DC, 2002.Google Scholar
  83. 83.
    Y.-L. Tian, T. Kanade, and J. Cohn. Eye-state action unit detection by gabor wavelets. In Proceedings of International Conference on Multi-modal Interfaces (ICMI 2000), pages 143–150, Sept, 2000.Google Scholar
  84. 84.
    K. Toyama. look, ma-no hands! hands-free cursor control with real-time 3d face tracking. In Proc. Workshop on Perceptual User Interfaces(PUI’98), 1998.Google Scholar
  85. 85.
    J. VanSwearingen, J. Cohn, and A. Bajaj-Luthra. Specific impairment of smiling increases severity of depressive symptoms in patients with facial neuromuscular disorders. Journal of Aesthetic Plastic Surgery, 23:416–423, 1999CrossRefGoogle Scholar
  86. 86.
    T. Vetter. Learning novels views to a single face image. In Proceedings of the IEEE International Conference on Automatic Face and Gesture Recognition, pages 22–29, 1995.Google Scholar
  87. 87.
    P. Viola and M. Jones. Robust real-time object detection. In International Workshop on Statistical and Computational Theories of Vision — Modeling, Learning, Computing, and Sampling, 2001.Google Scholar
  88. 88.
    Z. Wen and T. Huang. Capturing subtle facial motions in 3d face tracking. In Proc. of Int. Conf. On Computer Vision, 2003.Google Scholar
  89. 89.
    Y. Wu and K. Toyama. Wide-range person and illumination-insensitve head orientation estimation. In Proceedings of International Conference on Automatic Face and Gesture Recognition, pages 183–188, 2000.Google Scholar
  90. 90.
    J. Xiao, T. Kanade, and J. Cohn. Robust full motion recovery of head by dynamic templates and reregistration techniques. In Proceedings of International Conference on Automatic Face and Gesture Recognition, pages 163–169, 2002.Google Scholar
  91. 91.
    J. Xiao, T. Moriyama, T. Kanade, and J. Cohn. Robust full-motion recovery of head by dynamic templates and re-registration techniques. International Journal of Imaging Systems and Technology, September, 2003.Google Scholar
  92. 92.
    Y. Yacoob and M. Black. Parameterized modeling and recognition of activities. In Proc. 6th IEEE Int. Conf. on Computer Vision, pages 120–127, Bombay, 1998.Google Scholar
  93. 93.
    Y. Yacoob and L. Davis. Recognizing human facial expression from long image sequences using optical flow. IEEE Trans. on Pattern Analysis and Machine Intell., 18(6):636–642, June 1996.CrossRefGoogle Scholar
  94. 94.
    Y. Yacoob, H.-M. Lam, and L. Davis. Recognizing faces showing expressions. In Proc. Int.Workshop on Automatic Face-and Gesture-Recognition, pages 278–283, Zurich, Switserland, 1995.Google Scholar
  95. 95.
    M. Yang, D. Kriegman, and N. Ahuja. Detecting faces in images: a survey. IEEE Trans. on Pattern Analysis and Machine Intell., 24(1), 2002.Google Scholar
  96. 96.
    Z. Zhang. Feature-based facial expression recognition: sensitivity analysis and experiments with a multi-layer perceptron. International Journal of Pattern Recognition and Artificial Intelligence, 13(6):893–911, 1999.CrossRefGoogle Scholar
  97. 97.
    Z. Zhang, M. Lyons, M. Schuster, and S. Akamatsu. Comparison between geometry-based and gaborwavelets-based facial expression recognition using multi-layer perceptron. In InternationalWorkshop on Automatic Face and Gesture Recognition, pages 454–459, 1998.Google Scholar
  98. 98.
    L. Zhao, G. Pingali, and I. Carlbom. Real-time head orientation estimation using neural networks. In Proc of the 6th International Conference on Image Processing, 2002.Google Scholar
  99. 99.
    A. Zlochower, J. Cohn, J. Lien, and T. Kanade. A computer vision based method of facial expression analysis in parent-infant interaction. In International Conference on Infant Studies, Atlanta, 1998.Google Scholar

Copyright information

© Springer Science+Business Media, Inc. 2005

Authors and Affiliations

  • Ying-Li Tian
    • 1
  • Takeo Kanade
    • 2
  • Jeffrey F. Cohn
    • 2
    • 3
  1. 1.IBM T. J. Watson Research CenterHawthorneUSA
  2. 2.Carnegie Mellon UniversityRobotics InstitutePittsburghUSA
  3. 3.University of PittsburghDepartment of PsychologyPittsburghUSA

Personalised recommendations