Abstract
This paper presents an approach to automatic visual emotion recognition from two modalities: expressive face and body gesture. Face and body movements are captured simultaneously using two separate cameras. For each face and body image sequence single “expressive” frames are selected manually for analysis and recognition of emotions. Firstly, individual classifiers are trained from individual modalities for mono-modal emotion recognition. Secondly, we fuse facial expression and affective body gesture information at the feature and at the decision-level. In the experiments performed, the emotion classification using the two modalities achieved a better recognition accuracy outperforming the classification using the individual facial modality. We further extend the affect analysis into a whole image sequence by a multi-frame post integration approach over the single frame recognition results. In our experiments, the post integration based on the fusion of face and body has shown to be more accurate than the post integration based on the facial modality only.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
Balomenos, T., et al.: Emotion Analysis in Man-Machine Interaction Systems. In: Bengio, S., Bourlard, H. (eds.) MLMI 2004. LNCS, vol. 3361, pp. 318–328. Springer, Heidelberg (2005)
Bartlett, M.S., et al.: Machine learning methods for fully automatic recognition of face expressions and face actions. In: Proc. of IEEE SMC, pp. 592–597 (2004)
Boone, R.T., Cunningham, J.G.: Children’s decoding of emotion in expressive body movement: The development of cue attunement. Developmental Psyc. 34, 1007–1016 (1998)
Burgoon, J.K., et al.: Augmenting Human Identification of Emotional States in Video. In: Proc. of Int. Conference on Intelligent Data Analysis (2005) (in press)
Bradski, G.R.: Computer vision face tracking for use in a perceptual user interface. Intel Techn. J. 2nd Quarter (1998)
Brave, S., Nass, C.: Emotion in HCI. In: Jacko, J., Sears, A. (eds.) The HCI Hand-book. Lawrence Erlbaum Associates, Hillsdale (2002)
Chen, L.S., Huang, T.S.: Emotional expressions in audiovisual human computer interaction. In: Proc. of IEEE ICME, vol. 1, pp. 423–426 (2000)
Coulson, M.: Attributing Emotion to Static Body Postures: Recognition Accuracy, Confusions, and Viewpoint Dependence. J. of Nonverbal Behavior 28(2) (2004)
De Silva, L.C., Ng, P.C.: Bi-modal emotion recognition. In: Proc. FG, pp. 332–335 (2000)
Ekman, P., Friesen, W.V.: Unmasking the face: a guide to recognizing emotions from facial clues, Imprint. Prentice-Hall, Englewood Cliffs (1975)
Givens, D.B.: The Nonverbal Dictionary of Gestures, Signs & Body Language Cues. Center for Nonverbal Studies Press, Washington (2005)
Gunes, H., Piccardi, M.: Fusing Face and Body Gesture for Machine Recognition of Emotions. In: Proc. IEEE RO-MAN 2005, Nashville, USA (2005) (in press)
Hudlicka, E.: To feel or not to feel: The role of affect in human-computer interaction. Int. J. Hum.-Comput. Stud. 59(1-2), 1–32 (2003)
Kapoor, A., et al.: Probabilistic Combination of Multiple Modalities to Detect Interest. In: Proc. IEEE ICPR (2004)
Kuncheva, L.I.: A Theoretical Study on Six Classifier Fusion Strategies. IEEE Trans. On PAMI 24(2) (2002)
Lucas, B.D., Kanade, T.: An iterative image registration technique with an application to stereo vision. In: Proc. of 7th Int. Jnt. Conf. on Artificial Intelligence, pp. 674–680 (1981)
Pantic, M., Rothkrantz, L.J.M.: Towards an affect-sensitive multimodal humancom-puter interaction. Proc. of the IEEE 91(9), 1370–1390 (2003)
Shapiro, L.G., Rosenfeld, A.: Computer Vision and Image Processing. Academic Press, Boston (1992)
Witten, H., Frank, E.: Data Mining: Practical machine learning tools with Java implementations. Morgan Kaufmann, San Francisco (2000)
Wu, L., Oviatt, S.L., Cohen, P.R.: Multimodal Integration-A Statistical View. IEEE Trans. on Multimedia 1(4), 334–341 (1999)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2005 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Gunes, H., Piccardi, M. (2005). Fusing Face and Body Display for Bi-modal Emotion Recognition: Single Frame Analysis and Multi-frame Post Integration. In: Tao, J., Tan, T., Picard, R.W. (eds) Affective Computing and Intelligent Interaction. ACII 2005. Lecture Notes in Computer Science, vol 3784. Springer, Berlin, Heidelberg. https://doi.org/10.1007/11573548_14
Download citation
DOI: https://doi.org/10.1007/11573548_14
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-540-29621-8
Online ISBN: 978-3-540-32273-3
eBook Packages: Computer ScienceComputer Science (R0)