Journal on Multimodal User Interfaces

, Volume 8, Issue 1, pp 5–16 | Cite as

Using unlabeled data to improve classification of emotional states in human computer interaction

  • Martin Schels
  • Markus Kächele
  • Michael Glodek
  • David Hrabal
  • Steffen Walter
  • Friedhelm Schwenker
Original Paper


The individual nature of physiological measurements of human affective states makes it very difficult to transfer statistical classifiers from one subject to another. In this work, we propose an approach to incorporate unlabeled data into a supervised classifier training in order to conduct an emotion classification. The key idea of the method is to conduct a density estimation of all available data (labeled and unlabeled) to create a new encoding of the problem. Based on this a supervised classifier is constructed. Further, numerical evaluations on the EmoRec II corpus are given, examining to what extent additional data can improve classification and which parameters of the density estimation are optimal.


Partially supervised learning Clustering Affective computing 



This paper is based on work done within the Transregional Collaborative Research Center SFB/TRR 62 “Companion-Technology for Cognitive Technical Systems” funded by the German Research Foundation (DFG).


  1. 1.
    Baluja S (1999) Probabilistic modeling for face orientation discrimination: learning from labeled and unlabeled data. In: Proceedings of the 1998 conference on advances in neural information processing systems II (NIPS), pp 854–860Google Scholar
  2. 2.
    Baumert JH, Frey AW, Adt M (1995) Analysis of heart rate variability. Background, method, and possible use in anesthesia. Der Anaesth 44(10):677–686CrossRefGoogle Scholar
  3. 3.
    Blum A, Mitchell T (1998) Combining labeled and unlabeled data with co-training. In: Proceedings conference on computational learning theory, pp 92–100Google Scholar
  4. 4.
    Boiten FA, Frijda NH, Wientjes CJ (1994) Emotions and respiratory patterns: review and critical analysis. Int J Psychophysiol 17(2):103–128CrossRefGoogle Scholar
  5. 5.
    Breiman L (1996) Bagging predictors. Mach Learn 24:123–140MATHMathSciNetGoogle Scholar
  6. 6.
    Bruns T, Praun N (2002) Biofeedback: Ein Handbuch für die therapeutische Praxis. Vandenhoeck & Ruprecht, GermanyGoogle Scholar
  7. 7.
    Burkhardt F, Paeschke A, Rolfes M, Sendlmeier W, Weiss B (2005) A database of German emotional speech. In: Proceeding of interspeech, pp 1517–1520Google Scholar
  8. 8.
    Christensen H, Fuglsang-Frederiksen A (1986) Power spectrum and turns analysis of EMG at different voluntary efforts in normal subjects. Electroencephalogr Clin Neurophysiol 64(6):528–535CrossRefGoogle Scholar
  9. 9.
    Cohn DA, Ghahramani Z, Jordan MI (1996) Active learning with statistical models. J Artif Intell Res 4:129–145MATHGoogle Scholar
  10. 10.
    Cooper DB, Freeman JH (1970) On the asymptotic improvement in the out-come of supervised learning provided by additional nonsupervised learning. IEEE Trans Comput 19(11):1055–1063CrossRefMATHGoogle Scholar
  11. 11.
    Cover TM (1965) Geometrical and statistical properties of systems of linear inequalities with applications in pattern recognition. IEEE Trans Electron 14(3):326–334CrossRefMATHGoogle Scholar
  12. 12.
    Deng J, Schuller B (2012) Confidence measures in speech emotion recognition based on semi-supervised learning. In: proceeding of the INTERSPEECH 2012, pp 2226–2229Google Scholar
  13. 13.
    Eckmann JP, Kamphorst SO, Ruelle D (1987) Recurrence plots of dynamical systems. Europhys Lett 4(9):973CrossRefGoogle Scholar
  14. 14.
    Ekman P, Friesen WV (1977) Facial action coding system. Consulting Psychologists Press, USAGoogle Scholar
  15. 15.
    Esparza J, Scherer S, Schwenker F (2012) Studying self- and active-training methods for multi-feature set emotion recognition. In: Proceeding of the partially supervised learning (PSL 11), LNCS. Springer, New York, pp 19–31Google Scholar
  16. 16.
    Fahrenberg J, Foerster F (1982) Covariation and consistency of activation parameters. Biol Psychol 15(3–4):151–169CrossRefGoogle Scholar
  17. 17.
    Haynes JDD, Rees G (2006) Decoding mental states from brain activity in humans. Nature reviews. Neuroscience 7(7):523–534Google Scholar
  18. 18.
    Joachims T (2006) Transductive support vector machines. In: Semi-supervised learning. The MIT Press, London, pp 105–117Google Scholar
  19. 19.
    Kanade T, Cohn J, Tian Y (2000) Comprehensive database for facial expression analysis. In: IEEE international conference on automatic face and gesture recognition, pp 46–53Google Scholar
  20. 20.
    Karmakar C, Khandoker A, Voss A, Palaniswami M (2011) Sensitivity of temporal heart rate variability in poincar plot to changes in parasympathetic nervous system activity. BioMed Eng Online 10:1–14CrossRefGoogle Scholar
  21. 21.
    Kelley JF (1983) An empirical methodology for writing user-friendly natural language computer applications. In: SIGCHI conference on human factors in computing systems, pp 193–196Google Scholar
  22. 22.
    Kestler HA, Schwenker F, Hoher M, Palm G (1995) Adaptive class-specific partitioning as a means of initializing RBF-networks. IEEE Int Conf Syst Man Cybern 1:46–49Google Scholar
  23. 23.
    Kierkels JJM, Soleymani M, Pun T (2009) Queries and tags in affect-based multimedia retrieval. In: IEEE international conference on multimedia and expo, ICME ’09, pp 1436–1439Google Scholar
  24. 24.
    Kim J, André E (2008) Emotion recognition based on physiological changes in music listening. IEEE Trans Pattern Anal 12(12):2067–2083Google Scholar
  25. 25.
    Krikler DM (1990) The QRS complex. Ann N Y Acad Sci 601(1):24–30CrossRefGoogle Scholar
  26. 26.
    Kuncheva LI, Whitaker CJ (2003) Measures of diversity in classifier ensembles and their relationship with the ensemble accuracy. Mach Learn 51(2):181–207CrossRefMATHGoogle Scholar
  27. 27.
    Kuppens P, Van Mechelen I, Nezlek JB, Dossche D, Timmermans T (2007) Individual differences in core affect variability and their relationship to personality and psychological adjustment. Emotion 7(2):262–274CrossRefGoogle Scholar
  28. 28.
    Lang PJ, Greenwald MK, Bradley MM, Hamm AO (1993) Looking at pictures: affective, facial, visceral, and behavioral reactions. Psychophysiology 30(3):261–273CrossRefGoogle Scholar
  29. 29.
    Lisetti CL, Nasoz F (2004) Using noninvasive wearable computers to recognize human emotions from physiological signals. EURASIP J Appl Signal Process 2004(4):1672–1687CrossRefGoogle Scholar
  30. 30.
    Malik M (1996) Heart rate variability: standards of measurement, physiological interpretation, and clinical use. Circulation 93(5):1043–1065CrossRefGoogle Scholar
  31. 31.
    Mehrabian A (1996) Pleasure–arousal–dominance: a general framework for describing and measuring individual differences in temperament. Curr Psychol 14(4):261–292CrossRefMathSciNetGoogle Scholar
  32. 32.
    Mewett D, Reynolds K, Nazeran H (1999) Recurrence plot features of ECG signals. In: Conference on engineering in medicine and biology and 1999 annual fall meeting of the Biomedical Engineering Society, vol 2, p 193Google Scholar
  33. 33.
    Picard RW (2000) Affective computing. MIT Press, CambridgeGoogle Scholar
  34. 34.
    Picard RW, Vyzas E, Healey J (2001) Toward machine emotional intelligence: analysis of affective physiological state. IEEE Trans Pattern Anal Mach Intell 23(10):1175–1191CrossRefGoogle Scholar
  35. 35.
    Pincus SM (1991) Approximate entropy as a measure of system complexity. Natl Acad Sci USA 88(6):2297–2301CrossRefMATHMathSciNetGoogle Scholar
  36. 36.
    Qiu G (2002) Indexing chromatic and achromatic patterns for content-based colour image retrieval. Pattern Recogn 35(8):1675–1686CrossRefMATHGoogle Scholar
  37. 37.
    Richman JS, Moorman JR (2000) Physiological time-series analysis using approximate entropy and sample entropy. Am J Physiol Heart C 278(6):H2039–H2049Google Scholar
  38. 38.
    Rosenberg C, Hebert M, Schneiderman H (2005) Semi-supervised self-training of object detection models. In: Workshop on application of computer vision, pp 29–36Google Scholar
  39. 39.
    Rösner D, Frommer J, Friesen R, Haase M, Lange J, Otto M (2012) Last minute: a multimodal corpus of speech-based user-companion interactions. In: Conference on language resources and evaluation (LREC ’12), pp 2559–2566Google Scholar
  40. 40.
    Rudnicki M, Strumiłło P (2007) A real-time adaptive wavelet transform-based QRS complex detector. In: International conference on adaptive and natural computing algorithms, pp 281–289Google Scholar
  41. 41.
    Russell J (1980) A circumplex model of affect. J Pers Soc Psychol 39:1161–1178CrossRefGoogle Scholar
  42. 42.
    Schels M, Kächele M, Schwenker F (2012) Classification of emotional states in a Woz scenario exploiting labeled and unlabeled bio-physiological data. In: Proceedings of partially supervised learning 2011, pp 138–147Google Scholar
  43. 43.
    Scherer KR (2005) What are emotions? and how can they be measured? Soc Sci Inf 44(4):693–727CrossRefGoogle Scholar
  44. 44.
    Schuller B, Valstar M, Eyben F, McKeown G, Cowie R, Pantic M (2011) AVEC 2011—the first international audio visual emotion challenge. In: Proceeding of the affective computing and intelligent, interaction, pp 415–424Google Scholar
  45. 45.
    Schwenker F, Kestler HA, Palm G (2001) Three learning phases for radial-basis-function networks. Neural Netw 14:439–458CrossRefGoogle Scholar
  46. 46.
    Simson M (1981) Use of signals in the terminal QRS complex to identify patients with ventricular tachycardia after myocardial infarction. Circulation 64(2):235–242CrossRefGoogle Scholar
  47. 47.
    Soleymani M, Chanel G, Kierkels J, Pun T (2008) Affective characterization of movie scenes based on multimedia content analysis and user’s physiological emotional responses. In: IEEE international symposium on multimedia, pp 228–235Google Scholar
  48. 48.
    Stemmler G (1989) The autonomic differentiation of emotions revisited: convergent and discriminant validation. Psychophysiology 26(6):617–632CrossRefGoogle Scholar
  49. 49.
    Stemmler G, Heldmann M, Pauls CA, Scherer T (2001) Constraints for emotion specificity in fear and anger: the context counts. Psychophysiology 38(2):275–291CrossRefGoogle Scholar
  50. 50.
    Strauss PM, Hoffmann H, Minker W, Neumann H, Palm G, Scherer S, Traue H, Weidenbacher U (2008) The pit corpus of German multi-party dialogues. In: International language resources and evaluation (LREC ’08), pp 2442–2445 Google Scholar
  51. 51.
    Van Boxtel A (2010) Facial EMG as a tool for inferring affective states. In: Proceedings of measuring behavior, pp 104–108Google Scholar
  52. 52.
    Vapnik A (1998) Statistical learning theory. Wiley, New YorkMATHGoogle Scholar
  53. 53.
    Vogt T, André E (2009) Exploring the benefits of discretization of acoustic features for speech emotion recognition. In: Proceeding of the interspeech, pp 328–331Google Scholar
  54. 54.
    Walter S, Kim J, Hrabal D, Crawcour S, Kessler H, Traue H (2013) Transsituational individual-specific biopsychological classification of emotions. IEEE Trans Syst Man Cybern 43(4):988–995CrossRefGoogle Scholar
  55. 55.
    Wright RA, Dill JC (1993) Blood pressure responses and incentive appraisals as a function of perceived ability and objective task demand. Psychophysiology 30(2):152–160CrossRefGoogle Scholar
  56. 56.
    Zhang Z, Deng J, Schuller B (2013) Co-training succeeds in computational paralinguistics. In: IEEE international conference on acoustics, speech, and signal processing (ICASSP), pp 8505–8509Google Scholar
  57. 57.
    Zhang Z, Schuller B (2012) Active learning by sparse instance tracking and classifier confidence in acoustic emotion recognition. In: Proceeding of the INTERSPEECH 2012, pp 362–365Google Scholar

Copyright information

© OpenInterface Association 2013

Authors and Affiliations

  • Martin Schels
    • 1
  • Markus Kächele
    • 1
  • Michael Glodek
    • 1
  • David Hrabal
    • 2
  • Steffen Walter
    • 2
  • Friedhelm Schwenker
    • 1
  1. 1.Institute of Neural Information ProcessingUlm UniversityUlmGermany
  2. 2.Medical PsychologyUlm UniversityUlmGermany

Personalised recommendations