Abstract
As is known for centuries, humans exhibit an electrical profile. This profile is altered through various psychological and physiological proce-sses, which can be measured through biosignals; e.g., electromyography (EMG) and electrodermal activity (EDA). These biosignals can reveal our emotions and, as such, can serve as an advanced man-machine interface (MMI) for empathic consumer products. However, such a MMI requires the correct classification of biosignals to emotion classes. This chapter starts with an introduction on biosignals for emotion detection. Next, a state-of-the-art review is presented on automatic emotion classification. Moreover, guidelines are presented for affective MMI. Subsequently, a research is presented that explores the use of EDA and three facial EMG signals to determine neutral, positive, negative, and mixed emotions, using recordings of 21 people. A range of techniques is tested, which resulted in a generic framework for automated emotion classification with up to 61.31% correct classification of the four emotion classes, without the need of personal profiles. Among various other directives for future research, the results emphasize the need for parallel processing of multiple biosignals.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
Picard, R.W.: Affective Computing. MIT Press, Boston (1997)
van Tulder, M., Malmivaara, A., Koes, B.: Repetitive strain injury. The Lancet 369(9575), 1815–1822 (2007)
Schuler, J.L.H., O’Brien, W.H.: Cardiovascular recovery from stress and hypertension factors: A meta-analytic view. Psychophysiology 34(6), 649–659 (1997)
Frederickson, B.L., Manusco, R.A., Branigan, C., Tugade, M.M.: The undoing effect of positive emotions. Motivation and Emotion 24(4), 237–257 (2000)
Ader, R., Cohen, N., Felten, D.: Psychoneuroimmunology: Interactions between the nervous system and the immune system. The Lancet 345(8942), 99–103 (1995)
Solomon, G.F., Amkraut, A.A., Kasper, P.: Immunity, emotions, and stress with special reference to the mechanisms of stress effects on the immune system. Psychotherapy and Psychosomatics 23(1-6), 209–217 (1974)
Fairclough, S.H.: Fundamentals of physiological computing. Interacting with Computers 21(1-2), 133–145 (2009)
Mauss, I.B., Robinson, M.D.: Measures of emotion: A review. Cognition and Emotion 23(2), 209–237 (2009)
Picard, R.W., Vyzas, E., Healey, J.: Toward machine emotional intelligence: Analysis of affective physiological state. IEEE Transactions on Pattern Analysis and Machine Intelligence 23(10), 1175–1191 (2001)
van den Broek, E.L., Janssen, J.H., Westerink, J.H.D.M., Healey, J.A.: Prerequisits for Affective Signal Processing (ASP). In: Encarnação, P., Veloso, A. (eds.) Biosignals 2009: Proceedings of the International Conference on Bio-Inspired Systems and Signal Processing, Porto – Portugal, pp. 426–433 (2009)
Critchley, H.D., Elliott, R., Mathias, C.J., Dolan, R.J.: Neural activity relating to generation and representation of galvanic skin conductance responses: A functional magnetic resonance imaging study. The Journal of Neuroscience 20(8), 3033–3040 (2000)
Zeng, Z., Pantic, M., Roisman, G.I., Huang, T.S.: A survey of affect recognition methods: Audio, visual, and spontaneous expressions. IEEE Transactions on Pattern Analysis and Machine Intelligence 31(1), 39–58 (2009)
James, W.: Review: La pathologie des emotions by Ch. Féré. The Philosophical Review 2(3), 333–336 (1893)
Marwitz, M., Stemmler, G.: On the status of individual response specificity. Psychophysiology 35(1), 1–15 (1998)
Gunes, H., Piccardi, M.: Automatic temporal segment detection and affect recognition from face and body display. IEEE Transactions on Systems, Man, and Cybernetics – Part B: Cybernetics 39(1), 64–84 (2009)
Whitehill, J., Littlewort, G., Fasel, I., Bartlett, M., Movellan, J.: Towards practical smile detection. IEEE Transactions on Pattern Analysis and Machine Intelligence 31(11), 2106–2111 (2009)
Daly, A.: Movement analysis: Piecing together the puzzle. TDR – The Drama Review: A Journal of Performance Studies 32(4), 40–52 (1988)
Ververidis, D., Kotropoulos, C.: Emotional speech recognition: Resources, features, and methods. Speech Communication 48(9), 1162–1181 (2006)
Van den Broek, E.L.: Emotional Prosody Measurement (EPM): A voice-based evaluation method for psychological therapy effectiveness. Studies in Health Technology and Informatics (Medical and Care Compunetics) 103, 118–125 (2004)
van den Broek, E.L., Schut, M.H., Westerink, J.H.D.M., Tuinenbreijer, K.: Unobtrusive Sensing of Emotions (USE). Journal of Ambient Intelligence and Smart Environments 1(3), 287–299 (2009)
Gamboa, H., Silva, F., Silva, H., Falcão, R.: PLUX – Biosignals Acquisition and Processing (2010), http://www.plux.info (Last accessed January 30, 2010)
van den Broek, E.L., Westerink, J.H.D.M.: Considerations for emotion-aware consumer products. Applied Ergonomics 40(6), 1055–1064 (2009)
Berntson, G.G., Bigger, J.T., Eckberg, D.L., Grossman, P., Kaufmann, P.G., Malik, M., Nagaraja, H.N., Porges, S.W., Saul, J.P., Stone, P.H., van der Molen, M.W.: Heart rate variability: Origins, methods, and interpretive caveats. Psychophysiology 34(6), 623–648 (1997)
Boucsein, W.: Electrodermal activity. Plenum Press, New York (1992)
Grossman, P., Taylor, E.W.: Toward understanding respiratory sinus arrhythmia: Relations to cardiac vagal tone, evolution and biobehavioral functions. Biological Psychology 74(2), 263–285 (2007)
Fridlund, A.J., Cacioppo, J.T.: Guidelines for human electromyographic research. Psychophysiology 23(5), 567–589 (1986)
Reaz, M.B.I., Hussain, M.S., Mohd-Yasin, F.: Techniques of EMG signal analysis: detection, processing, classification and applications. Biological Procedures Online 8(1), 11–35 (2006)
Grandjean, D., Scherer, K.R.: Unpacking the cognitive architecture of emotion processes. Emotion 8(3), 341–351 (2008)
Lotte, F., Congedo, M., Lécuyer, A., Lamarche, F., Arnaldi, B.: A review of classification algorithms for EEG-based brain-computer interfaces. Journal of Neural Engineering 4(2), R1–R13 (2007)
Bimber, O.: Brain-Computer Interfaces. IEEE Computer 41(10) (2008); [special issue]
Minsky, M.: The Emotion Machine: Commonsense Thinking, Artificial Intelligence, and the Future of the Human Mind. Simon & Schuster, New York (2006)
Aarts, E.: Ambient intelligence: Vision of our future. IEEE Multimedia 11(1), 12–19 (2004)
Kim, J., André, E.: Emotion recognition based on physiological changes in music listening. IEEE Transactions on Pattern Analysis and Machine Intelligence 30(12), 2067–2083 (2008)
Liu, C., Rani, P., Sarkar, N.: Human-robot interaction using affective cues. In: Proceedings of the 15th IEEE International Symposium on Robot and Human Interactive Communication (RO-MAN 2006), Hatfield, UK, pp. 285–290. IEEE Computer Society, Los Alamitos (2006)
Rani, P., Sims, J., Brackin, R., Sarkar, N.: Online stress detection using psychophysiological signals for implicit human-robot cooperation. Robotica 20(6), 673–685 (2002)
Cacioppo, J.T., Tassinary, L.G., Berntson, G.: Handbook of Psychophysiology, 3rd edn. Cambridge University Press, New York (2007)
Sinha, R., Parsons, O.A.: Multivariate response patterning of fear. Cognition and Emotion 10(2), 173–198 (1996)
Scheirer, J., Fernandez, R., Klein, J., Picard, R.W.: Frustrating the user on purpose: A step toward building an affective computer. Interacting with Computers 14(2), 93–118 (2002)
Nasoz, F., Alvarez, K., Lisetti, C.L., Finkelstein, N.: Emotion recognition from physiological signals for presence technologies. International Journal of Cognition, Technology and Work 6(1), 4–14 (2003)
Takahashi, K.: Remarks on emotion recognition from bio-potential signals. In: Proceedings of the IEEE International Conference on Systems, Man and Cybernetics, Palmerston North, New Zealand, October 5-8, vol. 2, pp. 1655–1659 (2003)
Haag, A., Goronzy, S., Schaich, P., Williams, J.: Emotion recognition using bio-sensors: First steps towards an automatic system. In: André, E., Dybkjær, L., Minker, W., Heisterkamp, P. (eds.) ADS 2004. LNCS (LNAI), vol. 3068, pp. 36–48. Springer, Heidelberg (2004)
Kim, K.H., Bang, S.W., Kim, S.R.: Emotion recognition system using short-term monitoring of physiological signals. Medical & Biological Engineering & Computing 42(3), 419–427 (2004)
Lisetti, C.L., Nasoz, F.: Using noninvasive wearable computers to recognize human emotions from physiological signals. EURASIP Journal on Applied Signal Processing 2004(11), 1672–1687 (2004)
Wagner, J., Kim, J., André, E.: From physiological signals to emotions: Implementing and comparing selected methods for feature extraction and classification. In: Proceedings of the IEEE International Conference on Multimedia and Expo. (ICME), Amsterdam, The Netherlands, July 6-8, pp. 940–943 (2005)
Yoo, S.K., Lee, C.K., Park, J.Y., Kim, N.H., Lee, B.C., Jeong, K.S.: Neural network based emotion estimation using heart rate variability and skin resistance. In: Wang, L., Chen, K., S. Ong, Y. (eds.) ICNC 2005. LNCS, vol. 3610, pp. 818–824. Springer, Heidelberg (2005)
Choi, A., Woo, W.: Physiological sensing and feature extraction for emotion recognition by exploiting acupuncture spots. In: Tao, J., Tan, T., Picard, R.W. (eds.) ACII 2005. LNCS, vol. 3784, pp. 590–597. Springer, Heidelberg (2005)
Healey, J.A., Picard, R.W.: Detecting stress during real-world driving tasks using physiological sensors. IEEE Transactions on Intelligent Transportation Systems 6(2), 156–166 (2005)
Rani, P., Liu, C., Sarkar, N., Vanman, E.: An empirical study of machine learning techniques for affect recognition in human-robot interaction. Pattern Analysis & Applications 9(1), 58–69 (2006)
Zhai, J., Barreto, A.: Stress detection in computer users through noninvasive monitoring of physiological signals. Biomedical Science Instrumentation 42, 495–500 (2006)
Jones, C.M., Troen, T.: Biometric valence and arousal recognition. In: Thomas, B.H. (ed.) Proceedings of the Australasian Computer-Human Interaction Conference (OzCHI), Adelaide, Australia, pp. 191–194 (2007)
Leon, E., Clarke, G., Callaghan, V., Sepulveda, F.: A user-independent real-time emotion recognition system for software agents in domestic environments. Engineering Applications of Artificial Intelligence 20(3), 337–345 (2007)
Liu, C., Conn, K., Sarkar, N., Stone, W.: Physiology-based affect recognition for computer-assisted intervention of children with Autism Spectrum Disorder. International Journal of Human-Computer Studies 66(9), 662–677 (2008)
Katsis, C.D., Katertsidis, N., Ganiatsas, G., Fotiadis, D.I.: Toward emotion recognition in car-racing drivers: A biosignal processing approach. IEEE Transactions on Systems, Man, and Cybernetics–Part A: Systems and Humans 38(3), 502–512 (2008)
Yannakakis, G.N., Hallam, J.: Entertainment modeling through physiology in physical play. International Journal of Human-Computer Studies 66(10), 741–755 (2008)
Task Force: Heart rate variability: Standards of measurement, physiological interpretation, and clinical use. European Heart Journal 17(3), 354–381 (1996)
Ravenswaaij-Arts, C.M.A.V., Kollee, L.A.A., Hopman, J.C.W., Stoelinga, G.B.A., Geijn, H.P.: Heart rate variability. Annals of Internal Medicine 118(6), 436–447 (1993)
Butler, E.A., Wilhelm, F.H., Gross, J.J.: Respiratory sinus arrhythmia, emotion, and emotion regulation during social interaction. Psychophysiology 43(6), 612–622 (2006)
van den Broek, E.L., Schut, M.H., Westerink, J.H.D.M., van Herk, J., Tuinenbreijer, K.: Computing emotion awareness through facial electromyography. In: Huang, T.S., Sebe, N., Lew, M., Pavlović, V., Kölsch, M., Galata, A., Kisačanin, B. (eds.) ECCV 2006 Workshop on HCI. LNCS, vol. 3979, pp. 52–63. Springer, Heidelberg (2006)
Westerink, J.H.D.M., van den Broek, E.L., Schut, M.H., van Herk, J., Tuinenbreijer, K.: 14. In: Computing emotion awareness through galvanic skin response and facial electromyography. Philips Research Book Series, vol. 8, pp. 137–150. Springer, Dordrecht (2008)
Cacioppo, J., Tassinary, L.: Inferring psychological significance from physiological signals. American Psychologist 45(1), 16–28 (1990)
Mitchell, T.M.: Machine Learning. The McGraw-Hill Companies, Inc., Columbus (1997)
Bishop, C.M.: Pattern Recognition and Machine Learning. Information Science and Statistics. Springer Science+Business Media, LLC, New York (2006)
Schölkopf, B., Smola, A.J.: Learning with kernels: Support Vector Machines, Regularization, Optimization, and Beyond. In: Adaptive Computation and Machine Learning. The MIT Press, Cambridge (2002)
Rencher, A.C.: Methods of Multivariate Analysis, 2nd edn. Wiley Series in Probability and Statistics. John Wiley & Sons, Inc., New York (2002)
Rottenberg, J., Ray, R.R., Gross, J.J.: 1. In: Emotion elicitation using films, pp. 9–28. Oxford University Press, New York (2007)
Kreibig, S.D., Wilhelm, F.H., Roth, W.T., Gross, J.J.: Cardiovascular, electrodermal, and respiratory response patterns to fear- and sadness-inducing films. Psychophysiology 44(5), 787–806 (2007)
Kring, A.M., Gordon, A.H.: Sex differences in emotion: Expression, experience, and physiology. Journal of Personality and Social Psychology 74(3), 686–703 (1998)
Carrera, P., Oceja, L.: Drawing mixed emotions: Sequential or simultaneous experiences? Cognition & Emotion 21(2), 422–441 (2007)
Russell, J.A.: A circumplex model of affect. Journal of Personality and Social Psychology 39(6), 1161–1178 (1980)
Cover, T.M., van Campenhout, J.M.: On the possible orderings in the measurement selection problem. IEEE Transactions on Systems, Man, and Cybernetics SMC-7(9), 657–661 (1977)
Lawrence, S., Giles, C.L., Tsoi, A.: What size neural network gives optimal generalization? Convergence properties of backpropagation. Technical Report UMIACS-TR-96-22 and CS-TR-3617 (April 1996)
Barrett, L.F.: Valence as a basic building block of emotional life. Journal of Research in Personality 40, 35–55 (2006)
Russel, J.A., Barrett, L.F.: Core affect, prototypical emotional episodes, and other things called emotion: Dissecting the elephant. Journal of Personality and Social Psychology 26(5), 805–819 (1999)
Gendolla, G.H.E.: On the impact of mood on behavior: An integrative theory and a review. Review of General Psychology 4(4), 378–408 (2000)
Cooper, C.L., Pervin, L.A.: Personality: Critical concepts in psychology, 1st edn. Critical concepts in psychology. Routledge, New York (1998)
Lukowicz, P.: Wearable computing and artificial intelligence for healthcare applications. Artificial Intelligence in Medicine 42(2), 95–98 (2008)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2010 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
van den Broek, E.L., Lisý, V., Janssen, J.H., Westerink, J.H.D.M., Schut, M.H., Tuinenbreijer, K. (2010). Affective Man-Machine Interface: Unveiling Human Emotions through Biosignals. In: Fred, A., Filipe, J., Gamboa, H. (eds) Biomedical Engineering Systems and Technologies. BIOSTEC 2009. Communications in Computer and Information Science, vol 52. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-11721-3_2
Download citation
DOI: https://doi.org/10.1007/978-3-642-11721-3_2
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-642-11720-6
Online ISBN: 978-3-642-11721-3
eBook Packages: Computer ScienceComputer Science (R0)