Fusion of Multichannel Biosignals Towards Automatic Emotion Recognition

  • Jonghwa Kim
  • Elisabeth André
Part of the Lecture Notes in Electrical Engineering book series (LNEE, volume 35)


Endowing the computer with the ability to recognize human emotional states is the most important prerequisites for realizing an affect-sensitive human-computer interaction. In this paper, we deal with all the essential stages of an automatic emotion recognition system using multichannel physiological measures, from data collection to the classification process. Particularly we develop two different classification methods, feature-level fusion and emotion-specific classification scheme. Four-channel biosensors were used to measure electromyogram, electrocardiogram, skin conductivity, and respiration changes while subjects were listening to music. A wide range of physiological features from various analysis domains is proposed to correlate them with emotional states. Classification of four musical emotions is performed by using feature-level fusion combined with an extended linear discriminant analysis (pLDA). Furthermore, by exploiting a dichotomic property of the 2D emotion model, we developed a novel scheme of emotion-specific multilevel dichotomous classification (EMDC) and compare its performance with direct multiclass classification using the pLDA feature-level fusion. Improved recognition accuracy of 95% and 70% for subject-dependent and subject-independent classification, respectively, is achieved by using the EMDC scheme.


Biosignal Multisensor data fusion ECG EMG SC RSP Emotion recognition Multichannel biosignals Pattern recognition Human-computer interaction Music and emotion 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    R. Cowie, E. Douglas-Cowie, N. Tsapatsoulis, G. Votsis, S. Kollias, W. Fellenz, and J. G. Taylor, Emotion recognition in human-computer interaction, IEEE Signal Process. Mag., 18, 32–80, 2001.CrossRefGoogle Scholar
  2. 2.
    J. Healey and R. W. Picard, Digital processing of affective signals, in Proc. IEEE Int. Conf. Acoust., Speech, and Signal Proc., Seattle, WA, 1998, pp. 3749–3752.Google Scholar
  3. 3.
    R. Picard, E. Vyzas, and J. Healy, Toward machine emotional intelligence: Analysis of affective physiological state, IEEE Trans. Pattern Anal. Mach Intell., 23(10), 1175–1191, 2001.CrossRefGoogle Scholar
  4. 4.
    F. Nasoz, K. Alvarez, C. Lisetti, and N. Finkelstein, Emotion recognition from physiological signals for presence technologies, Int. J. Cogn., Technol., and Work, 6(1), 4–14, 2003.Google Scholar
  5. 5.
    J. J. Gross and R. W. Levenson, Emotion elicitation using films, Cogn. Emot., 9, pp. 87–108, 1995.CrossRefGoogle Scholar
  6. 6.
    K. H. Kim, S. W. Bang, and S. R. Kim, Emotion recognition system using short-term monitoring of physiological signals, Med. Biol. Eng. Comput., 42, pp. 419–427, 2004.CrossRefGoogle Scholar
  7. 7.
    J. Pan and W. Tompkins, A real-time qrs detection algorithm, IEEE Trans. Biomed. Eng., 32(3), 230–323, 1985.CrossRefGoogle Scholar
  8. 8.
    P. W. Kamen, H. Krum, and A. M. Tonkin, Poincare plot of heart rate variability allows quantitative display of parasympathetic nervous activity, Clin. Sci., 91, 201–208, 1996.Google Scholar
  9. 9.
    J. Richmann and J. Moorman, Physiological time series analysis using approximate entropy and sample entropy, Am. J. Physiol. Heart Circ. Physiol. 278, H2039, 2000.Google Scholar
  10. 10.
    M. Costa, A. L. Goldberger, and C.-K. Peng, Multiscale entropy analysis of biological signals,” Phys. Rev., E 71, 021906, 2005.MathSciNetGoogle Scholar
  11. 11.
    R. Thuraisingham and G. Gottwald, On multiscale entropy analysis for physiological data, Physica A, 336, 323–333, 2006.Google Scholar
  12. 12.
    J. Ye and Q. Li, A two-stage linear discriminant analysis via Qr-decomposition, IEEE Trans. on PAMI, 27(6), 929–941, 2005.Google Scholar
  13. 13.
    J. Kittler, Feature Selection and Extraction. San Diego, CA:Academic Press, Inc., 59–83, 1986.Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2009

Authors and Affiliations

  1. 1.Institute of Computer ScienceUniversity of AugsburgGermany

Personalised recommendations