Emotion Recognition Based on Physiological Sensor Data Using Codebook Approach

  • Kimiaki ShirahamaEmail author
  • Marcin Grzegorzek
Conference paper
Part of the Advances in Intelligent Systems and Computing book series (AISC, volume 472)


This paper addresses emotion recognition based on physiological signal sequences (e.g., blood pressure, galvanic skin response and respiration) that can be obtained using state-of-the-art wearable sensors. We formulate this as a machine learning problem to distinguish sequences labelled with a certain emotion from the other sequences. In particular, we explore how to extract a feature that effectively characterises a sequence and yields accurate emotion recognition. With respect to this, existing methods rely on hand-crafted features that are manually defined based on prior knowledge about physiological signals. However, in addition to intensive labour, it is difficult to manually design features which can represent the details of a sequence. To overcome this, we propose a codebook approach where a sequence is represented with a feature describing the distribution of characteristic subsequences, called codewords. These are statistically justified because they are obtained by clustering a large number of subsequences. In addition, the details of the sequence can be maintained by considering the distribution of hundreds of codewords. Experimental results validate the effectiveness of our codebook-based emotion recognition method.


Ambient assisted living Emotion recognition Physiological sensor data Codebook approach Sequence classification 



Research and development activities leading to this article have been supported by the German Federal Ministry of Education and Research within the project Cognitive Village: Adaptively Learning Technical Support System for Elderly (Grant Number: 16SV7223K).


  1. 1.
    Bengio, Y., Courville, A., Vincent, P.: Representation learning: a review and new perspectives. IEEE Trans. Pattern Anal. Mach. Intell. 35(8), 1798–1828 (2013)CrossRefGoogle Scholar
  2. 2.
    Chang, C.C., Lin, C.J.: Libsvm: A library for support vector machines. ACM Trans. Intell. Syst. Technol. 2(3), 27:1–27:27 (2011)Google Scholar
  3. 3.
    Chatfield, K., Lempitsky, V., Vedaldi, A., Zisserman, A.: The devil is in the details: an evaluation of recent feature encoding methods. In: Proceedings of the 22th British Machine Vision Conference (BMVC 2011), pp. 76.1–76.12 (2011)Google Scholar
  4. 4.
    Chen, Y., Keogh, E., Hu, B., Begum, N., Bagnall, A., Mueen, A., Batista, G.: The UCR time series classification archive (2015). Accessed 24 Feb 2016
  5. 5.
    Garbarino, M. et al.: Empatica E3 - A wearable wireless multi-sensor device for real-time computerized biofeedback and data acquisition. In: Proceedings of the 2014 EAI 4th International Conference on Wireless Mobile Communication and Healthcare (Mobihealth 2014), pp. 39–42 (2014)Google Scholar
  6. 6.
    Han, J., Kamber, M., Pei, J.: Data Mining: Concepts and Techniques, 3rd edn. Morgan Kaufmann (2011)Google Scholar
  7. 7.
    Healey, J.A.: Wearable and Automotive Systems for Affect Recognition from Physiology. Ph.D. thesis, Massachusetts Institute of Technology (2000)Google Scholar
  8. 8.
    Hong, J.H., Ramos, J., Dey, A.K.: Understanding physiological responses to stressors during physical activity. In: Proceedings of the 14th ACM International Conference on Ubiquitous Computing (UbiComp 2012), pp. 270–279 (2012)Google Scholar
  9. 9.
    Jiang, Y.G., Yang, J., Ngo, C.W., Hauptmann, A.G.: Representations of keypoint-based semantic concept detection: a comprehensive study. IEEE Trans. Multimed. 12(1), 42–53 (2010)CrossRefGoogle Scholar
  10. 10.
    Kim, J., Andre, E.: Emotion recognition based on physiological changes in music listening. IEEE Tran. Pattern Anal. Mach. Intell. 30(12), 2067–2083 (2008)CrossRefGoogle Scholar
  11. 11.
    Koelstra, S., Mühl, C., Soleymani, M., Lee, J.S., Yazdani, A., Ebrahimi, T., Pun, T., Nijholt, A., Patras, I.: DEAP: a database for emotion analysis using physiological signals. IEEE Trans. Affect. Comput. 3(1), 18–31 (2012)CrossRefGoogle Scholar
  12. 12.
    Metz, R.: This fitness wristband wants to play doctor. MIT Technol. Rev. (2014). Accessed 24 Feb 2016
  13. 13.
    Microsoft Corporation: Microsoft Band Official Site. Accessed 24 Feb 2016
  14. 14.
    Pentland, A., Lazer, D., Brewer, D., Heibeck, T.: Using reality mining to improve public health and medicine. In: Bushko, R.G. (ed.) Strategy for the Future of Health, pp. 93–102. IOS press (2009)Google Scholar
  15. 15.
    Picard, R.W., Vyzas, E., Healey, J.: Toward machine emotional intelligence: analysis of affective physiological state. IEEE Trans. Pattern Anal. Mach. Intell. 23(10), 1175–1191 (2001)CrossRefGoogle Scholar
  16. 16.
    Plarre, K. et al.: Continuous inference of psychological stress from sensory measurements collected in the natural environment. In: Proceedings of the 10th ACM/IEEE International Conference on Information Processing in Sensor Networks (IPSN 2011), pp. 97–108 (2011)Google Scholar
  17. 17.
    Rashidi, P., Mihailidis, A.: A survey on ambient-assisted living tools for older adults. IEEE J. Biomed. Health Inf. 17(3), 579–590 (2013)CrossRefGoogle Scholar
  18. 18.
    Shirahama, K., Grzegorzek, M.: Towards large-scale multimedia retrieval enriched by knowledge about human interpretation: retrospective survey. Multimed. Tools Appl. 75(1), 297–331 (2016)CrossRefGoogle Scholar
  19. 19.
    Shirahama, K., Uehara, K.: Kobe university and Muroran institute of technology at TRECVID 2012 semantic indexing task. In: Proceedings of the TREC Video Retrieval Evaluation Workshop (TRECVID 2012), pp. 239–247 (2012)Google Scholar
  20. 20.
    Snoek, C.G.M., Worring, M., Smeulders, A.W.M.: Early versus late fusion in semantic video analysis. In: Proceedings of the 13th Annual ACM International Conference on Multimedia (MM 2005), pp. 399–402 (2005)Google Scholar
  21. 21.
    Soleymani, M., Lichtenauer, J., Pun, T., Pantic, M.: A multimodal database for affect recognition and implicit tagging. IEEE Trans. Affect. Comput. 3(1), 42–55 (2012)CrossRefGoogle Scholar
  22. 22.
    van de Sande, K.E., Gevers, T., Snoek, C.G.: Evaluating color descriptors for object and scene recognition. IEEE Trans. Pattern Anal. Mach. Intell. 32(9), 1582–1596 (2010)CrossRefGoogle Scholar
  23. 23.
    Van Gemert, J.C., Veenman, C.J., Smeulders, A.W.M., Geusebroek, J.M.: Visual word ambiguity. IEEE Trans. Pattern Anal. Mach. Intell. 32(7), 1271–1283 (2010)CrossRefGoogle Scholar
  24. 24.
    Vapnik, V.N.: Statistical Learning Theory. Wiley-Interscience (1998)Google Scholar

Copyright information

© Springer International Publishing Switzerland 2016

Authors and Affiliations

  1. 1.Pattern Recognition GroupUniversity of SiegenSiegenGermany

Personalised recommendations