Multimodal Emotion Classification in Naturalistic User Behavior

  • Steffen Walter
  • Stefan Scherer
  • Martin Schels
  • Michael Glodek
  • David Hrabal
  • Miriam Schmidt
  • Ronald Böck
  • Kerstin Limbrecht
  • Harald C. Traue
  • Friedhelm Schwenker
Part of the Lecture Notes in Computer Science book series (LNCS, volume 6763)

Abstract

The design of intelligent personalized interactive systems, having knowledge about the user’s state, his desires, needs and wishes, currently poses a great challenge to computer scientists. In this study we propose an information fusion approach combining acoustic, and bio-physiological data, comprising multiple sensors, to classify emotional states. For this purpose a multimodal corpus has been created, where subjects undergo a controlled emotion eliciting experiment, passing several octants of the valence arousal dominance space. The temporal and decision level fusion of the multiple modalities outperforms the single modality classifiers and shows promising results.

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. 1.
    Bishop, C.M.: Pattern Recognition and Machine Learning (Information Science and Statistics), 1st edn. Springer, Heidelberg (2006)MATHGoogle Scholar
  2. 2.
    Brown, G., Kuncheva, L.I.: “Good” and “Bad” diversity in majority vote ensembles. In: El Gayar, N., Kittler, J., Roli, F. (eds.) MCS 2010. LNCS, vol. 5997, pp. 124–133. Springer, Heidelberg (2010)CrossRefGoogle Scholar
  3. 3.
    Davis, S., Mermelstein, P.: Comparison of parametric representations for monosyllabic word recognition in continuously spoken sentences. IEEE Transactions on Acoustics, Speech and Signal Processing 28(4), 357–366 (1980)CrossRefGoogle Scholar
  4. 4.
    Dietrich, C., Schwenker, F., Palm, G.: Classification of time series utilizing temporal and decision fusion. In: Kittler, J., Roli, F. (eds.) MCS 2001. LNCS, vol. 2096, pp. 378–387. Springer, Heidelberg (2001)CrossRefGoogle Scholar
  5. 5.
    Ekman, P., Friesen, W.V.: Facial Action Coding System: A Technique for the Measurement of Facial Movement. Consulting Psychologists Press, Palo Alto (1978)Google Scholar
  6. 6.
    Hermansky, H.: The modulation spectrum in automatic recognition of speech. In: Proceedings of IEEE Workshop on Automatic Speech Recognition and Understanding, pp. 140–147. IEEE, Los Alamitos (1997)CrossRefGoogle Scholar
  7. 7.
    Hermansky, H., Hanson, B., Wakita, H.: Perceptually based linear predictive analysis of speech. In: Proceedings of IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP 1985), vol. 10, pp. 509–512 (1985)Google Scholar
  8. 8.
    Kim, J., André, E.: Emotion recognition based on physiological changes in music listening. IEEE Trans. Pattern Anal. Mach. Intell. 30, 2067–2083 (2008), http://portal.acm.org/citation.cfm?id=1477073.1477535 CrossRefGoogle Scholar
  9. 9.
    Kuncheva, L.I.: Combining Pattern Classifiers: Methods and Algorithms. Wiley Interscience, Hoboken (2004)CrossRefMATHGoogle Scholar
  10. 10.
    Maganti, H.K., Scherer, S., Palm, G.: A novel feature for emotion recognition in voice based applications. In: Paiva, A.C.R., Prada, R., Picard, R.W. (eds.) ACII 2007. LNCS, vol. 4738, pp. 710–711. Springer, Heidelberg (2007)CrossRefGoogle Scholar
  11. 11.
    Peter, J., Lang, M.M.B., Cuthbert, B.N.: International affective picture system (iaps): Affective ratings of pictures and instruction manual. Tech. rep., NIMH Center for the Study of Emotion & Attention, University of Florida (2008)Google Scholar
  12. 12.
    Picard, R.W.: Affective Computing. MIT Press, Cambridge (2000)Google Scholar
  13. 13.
    Platt, J.: Probabilistic Outputs for Support Vector Machines and Comparisons to Regularized Likelihood Methods. Advances in Large Margin Classifiers, 61–74 (1999)Google Scholar
  14. 14.
    Rabiner, L.R., Schafer, R.W.: Digital processing of speech signals. Prentice-Hall Signal Processing Series. Prentice-Hall, Englewood Cliffs (1978)Google Scholar
  15. 15.
    Russell, J.: A circumplex model of affect. Journal of Personality and Social Psychology 39, 1161–1178 (1980)CrossRefGoogle Scholar
  16. 16.
    Schwenker, F., Dietrich, C., Thiel, C., Palm, G.: Learning of decision fusion mappings for pattern recognition. International Journal on Artificial Intelligence and Machine Learning (AIML) 6, 17–21 (2006)Google Scholar
  17. 17.
    Smyth, P.: Clustering sequences with hidden markov models. Advances in Neural Information Processing Systems 9, 648–654 (1997)Google Scholar
  18. 18.
    Tax, D.M.J., van Breukelen, M., Duin, R.P.W., Kittler, J.: Combining multiple classifiers by averaging or by multiplying. Pattern Recognition 33(9), 1475–1485 (2000)CrossRefGoogle Scholar
  19. 19.
    Vapnik, V.N.: The nature of statistical learning theory. Springer-Verlag New York, Inc., New York (1995)CrossRefMATHGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2011

Authors and Affiliations

  • Steffen Walter
    • 1
  • Stefan Scherer
    • 2
  • Martin Schels
    • 2
  • Michael Glodek
    • 2
  • David Hrabal
    • 1
  • Miriam Schmidt
    • 2
  • Ronald Böck
    • 3
  • Kerstin Limbrecht
    • 1
  • Harald C. Traue
    • 1
  • Friedhelm Schwenker
    • 2
  1. 1.Medical PsychologyUniversity of UlmGermany
  2. 2.Institute of Neural Information ProcessingUniversity of UlmGermany
  3. 3.Chair of Cognitive SystemsOtto von Guericke University MagdeburgGermany

Personalised recommendations