Abstract
Changes in a speaker’s emotion are a fundamental component in human communication. Some emotions motivate human actions while others add deeper meaning and richness to human interactions. In this paper, we explore the development of a computing algorithm that uses audio and visual sensors to recognize a speaker’s affective state. Within the framework of Multi-stream Hidden Markov Model (MHMM), we analyze audio and visual observations to detect 11 cognitive/emotive states. We investigate the use of individual modality confidence measures as a means of estimating weights when combining likelihoods in the audio-visual decision fusion. Person-independent experimental results from 20 subjects in 660 sequences suggest that the use of stream exponents estimated on training data results in classification accuracy improvement of audio-visual affect recognition.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
Similar content being viewed by others
References
Pantic, M., Rothkrantz, L.J.M.: Toward an affect-sensitive multimodal human-computer interaction. Proceedings of the IEEE 91(9), 1370–1390 (2003)
Chen, L., Huang, T.S.: Emotional expressions in audiovisual human computer interaction. In: Int. Conf. on Multimedia & Expo 2000, pp. 423–426 (2000)
Chen, L., Huang, T.S., Miyasato, T., Nakatsu, R.: Multimodal human emotion/expression recognition. In: Int. Conf. on Automatic Face & Gesture Recognition 1998, pp. 396–401 (1998)
De Silva, L.C., Ng, P.C.: Bimodal emotion recognition. In: Int. Conf. on Automatic Face & Gesture Recognition 2000, pp. 332–335 (2000)
Picard, R.W.: Affective Computing. MIT Press, Cambridge (1997)
Tao, H., Huang, T.S.: Explanation-based facial motion tracking using a piecewise Bezier volume deformation mode. In: CVPR 1999, vol. 1, pp. 611–617 (1999)
Talkin, D.: A Robust Algorithm for Pitch Tracking. In: Kkeijn, W.B., Paliwal, K.K. (eds.) Speech Coding and Synthesis. Elsevier Science, Amsterdam (1995)
Kwon, O.W., Chan, K., Hao, J., Lee, T.W.: Emotion Recognition by Speech Signals. In: EUROSPEECH 2003 (2003)
Mehrabian, A.: Communication without words. Psychol. Today 2(4), 53–56 (1968)
Jain, A.J., Duin, R.P.W., Mao, J.: Statistical Pattern Recognition: A Review. IEEE PAMI 22(1), 4–37 (2000)
Bourlard, H., Dupont, S.: A New ASR Approach Based on Independent Processing And Recombination of Partial Frequency Bands (1996)
Potamianos, G., Neti, C., Gravier, G., Garg, A.: Automatic Recognition of audio-visual speech: Recent progress and challenges. Proceedings of the IEEE 91(9) (September 2003)
Chen, L.S.: Joint Processing of Audio-Visual Information for the Recognition of Emotional Expressions in Human-Computer Interaction, PhD thesis, UIUC (2000)
Zeng, Z., Tu, J., Pianfetti, B., Liu, M., Zhang, T., Zhang, Z., Huang, T.S., Levinson, S.: Audio-visual Affect Recognition through Multi-stream Fused HMM for HCI. In: CVPR 2005 (2005)
Tu, J., Zhang, Z., Zeng, Z., Huang, T.S.: Face Localization via Hierarchical Condensation with Fisher Boosting Feature Selection. In: Proc. Computer Vision and Pattern Recognition (2004)
Zeng, Z., Tu, J., Liu, M., Zhang, T., Rizzolo, N., Zhang, Z., Huang, T.S., Roth, D., Levinson, S.: Bimodal HCI-related Affect Recognition. In: ICMI 2004 (2004)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2005 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Zeng, Z., Tu, J., Liu, M., Huang, T.S. (2005). Multi-stream Confidence Analysis for Audio-Visual Affect Recognition. In: Tao, J., Tan, T., Picard, R.W. (eds) Affective Computing and Intelligent Interaction. ACII 2005. Lecture Notes in Computer Science, vol 3784. Springer, Berlin, Heidelberg. https://doi.org/10.1007/11573548_123
Download citation
DOI: https://doi.org/10.1007/11573548_123
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-540-29621-8
Online ISBN: 978-3-540-32273-3
eBook Packages: Computer ScienceComputer Science (R0)