Affective Man-Machine Interface: Unveiling Human Emotions through Biosignals

  • Egon L. van den Broek
  • Viliam Lisý
  • Joris H. Janssen
  • Joyce H. D. M. Westerink
  • Marleen H. Schut
  • Kees Tuinenbreijer
Part of the Communications in Computer and Information Science book series (CCIS, volume 52)

Abstract

As is known for centuries, humans exhibit an electrical profile. This profile  is  altered  through various  psychological  and  physiological proce-sses, which can be measured through biosignals; e.g., electromyography (EMG) and electrodermal activity (EDA). These biosignals can reveal our emotions and, as such, can serve as an advanced man-machine interface (MMI) for empathic consumer products. However, such a MMI requires the correct classification of biosignals to emotion classes. This chapter starts with an introduction on biosignals for emotion detection. Next, a state-of-the-art review is presented on automatic emotion classification. Moreover, guidelines are presented for affective MMI. Subsequently, a research is presented that explores the use of EDA and three facial EMG signals to determine neutral, positive, negative, and mixed emotions, using recordings of 21 people. A range of techniques is tested, which resulted in a generic framework for automated emotion classification with up to 61.31% correct classification of the four emotion classes, without the need of personal profiles. Among various other directives for future research, the results emphasize the need for parallel processing of multiple biosignals.

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. 1.
    Picard, R.W.: Affective Computing. MIT Press, Boston (1997)Google Scholar
  2. 2.
    van Tulder, M., Malmivaara, A., Koes, B.: Repetitive strain injury. The Lancet 369(9575), 1815–1822 (2007)CrossRefGoogle Scholar
  3. 3.
    Schuler, J.L.H., O’Brien, W.H.: Cardiovascular recovery from stress and hypertension factors: A meta-analytic view. Psychophysiology 34(6), 649–659 (1997)CrossRefGoogle Scholar
  4. 4.
    Frederickson, B.L., Manusco, R.A., Branigan, C., Tugade, M.M.: The undoing effect of positive emotions. Motivation and Emotion 24(4), 237–257 (2000)CrossRefGoogle Scholar
  5. 5.
    Ader, R., Cohen, N., Felten, D.: Psychoneuroimmunology: Interactions between the nervous system and the immune system. The Lancet 345(8942), 99–103 (1995)CrossRefGoogle Scholar
  6. 6.
    Solomon, G.F., Amkraut, A.A., Kasper, P.: Immunity, emotions, and stress with special reference to the mechanisms of stress effects on the immune system. Psychotherapy and Psychosomatics 23(1-6), 209–217 (1974)CrossRefGoogle Scholar
  7. 7.
    Fairclough, S.H.: Fundamentals of physiological computing. Interacting with Computers 21(1-2), 133–145 (2009)CrossRefGoogle Scholar
  8. 8.
    Mauss, I.B., Robinson, M.D.: Measures of emotion: A review. Cognition and Emotion 23(2), 209–237 (2009)CrossRefGoogle Scholar
  9. 9.
    Picard, R.W., Vyzas, E., Healey, J.: Toward machine emotional intelligence: Analysis of affective physiological state. IEEE Transactions on Pattern Analysis and Machine Intelligence 23(10), 1175–1191 (2001)CrossRefGoogle Scholar
  10. 10.
    van den Broek, E.L., Janssen, J.H., Westerink, J.H.D.M., Healey, J.A.: Prerequisits for Affective Signal Processing (ASP). In: Encarnação, P., Veloso, A. (eds.) Biosignals 2009: Proceedings of the International Conference on Bio-Inspired Systems and Signal Processing, Porto – Portugal, pp. 426–433 (2009)Google Scholar
  11. 11.
    Critchley, H.D., Elliott, R., Mathias, C.J., Dolan, R.J.: Neural activity relating to generation and representation of galvanic skin conductance responses: A functional magnetic resonance imaging study. The Journal of Neuroscience 20(8), 3033–3040 (2000)Google Scholar
  12. 12.
    Zeng, Z., Pantic, M., Roisman, G.I., Huang, T.S.: A survey of affect recognition methods: Audio, visual, and spontaneous expressions. IEEE Transactions on Pattern Analysis and Machine Intelligence 31(1), 39–58 (2009)CrossRefGoogle Scholar
  13. 13.
    James, W.: Review: La pathologie des emotions by Ch. Féré. The Philosophical Review 2(3), 333–336 (1893)CrossRefGoogle Scholar
  14. 14.
    Marwitz, M., Stemmler, G.: On the status of individual response specificity. Psychophysiology 35(1), 1–15 (1998)CrossRefGoogle Scholar
  15. 15.
    Gunes, H., Piccardi, M.: Automatic temporal segment detection and affect recognition from face and body display. IEEE Transactions on Systems, Man, and Cybernetics – Part B: Cybernetics 39(1), 64–84 (2009)CrossRefGoogle Scholar
  16. 16.
    Whitehill, J., Littlewort, G., Fasel, I., Bartlett, M., Movellan, J.: Towards practical smile detection. IEEE Transactions on Pattern Analysis and Machine Intelligence 31(11), 2106–2111 (2009)CrossRefGoogle Scholar
  17. 17.
    Daly, A.: Movement analysis: Piecing together the puzzle. TDR – The Drama Review: A Journal of Performance Studies 32(4), 40–52 (1988)MathSciNetGoogle Scholar
  18. 18.
    Ververidis, D., Kotropoulos, C.: Emotional speech recognition: Resources, features, and methods. Speech Communication 48(9), 1162–1181 (2006)CrossRefGoogle Scholar
  19. 19.
    Van den Broek, E.L.: Emotional Prosody Measurement (EPM): A voice-based evaluation method for psychological therapy effectiveness. Studies in Health Technology and Informatics (Medical and Care Compunetics) 103, 118–125 (2004)Google Scholar
  20. 20.
    van den Broek, E.L., Schut, M.H., Westerink, J.H.D.M., Tuinenbreijer, K.: Unobtrusive Sensing of Emotions (USE). Journal of Ambient Intelligence and Smart Environments 1(3), 287–299 (2009)Google Scholar
  21. 21.
    Gamboa, H., Silva, F., Silva, H., Falcão, R.: PLUX – Biosignals Acquisition and Processing (2010), http://www.plux.info (Last accessed January 30, 2010)
  22. 22.
    van den Broek, E.L., Westerink, J.H.D.M.: Considerations for emotion-aware consumer products. Applied Ergonomics 40(6), 1055–1064 (2009)CrossRefGoogle Scholar
  23. 23.
    Berntson, G.G., Bigger, J.T., Eckberg, D.L., Grossman, P., Kaufmann, P.G., Malik, M., Nagaraja, H.N., Porges, S.W., Saul, J.P., Stone, P.H., van der Molen, M.W.: Heart rate variability: Origins, methods, and interpretive caveats. Psychophysiology 34(6), 623–648 (1997)CrossRefGoogle Scholar
  24. 24.
    Boucsein, W.: Electrodermal activity. Plenum Press, New York (1992)Google Scholar
  25. 25.
    Grossman, P., Taylor, E.W.: Toward understanding respiratory sinus arrhythmia: Relations to cardiac vagal tone, evolution and biobehavioral functions. Biological Psychology 74(2), 263–285 (2007)CrossRefGoogle Scholar
  26. 26.
    Fridlund, A.J., Cacioppo, J.T.: Guidelines for human electromyographic research. Psychophysiology 23(5), 567–589 (1986)CrossRefGoogle Scholar
  27. 27.
    Reaz, M.B.I., Hussain, M.S., Mohd-Yasin, F.: Techniques of EMG signal analysis: detection, processing, classification and applications. Biological Procedures Online 8(1), 11–35 (2006)CrossRefGoogle Scholar
  28. 28.
    Grandjean, D., Scherer, K.R.: Unpacking the cognitive architecture of emotion processes. Emotion 8(3), 341–351 (2008)CrossRefGoogle Scholar
  29. 29.
    Lotte, F., Congedo, M., Lécuyer, A., Lamarche, F., Arnaldi, B.: A review of classification algorithms for EEG-based brain-computer interfaces. Journal of Neural Engineering 4(2), R1–R13 (2007)Google Scholar
  30. 30.
    Bimber, O.: Brain-Computer Interfaces. IEEE Computer 41(10) (2008); [special issue]Google Scholar
  31. 31.
    Minsky, M.: The Emotion Machine: Commonsense Thinking, Artificial Intelligence, and the Future of the Human Mind. Simon & Schuster, New York (2006)Google Scholar
  32. 32.
    Aarts, E.: Ambient intelligence: Vision of our future. IEEE Multimedia 11(1), 12–19 (2004)CrossRefGoogle Scholar
  33. 33.
    Kim, J., André, E.: Emotion recognition based on physiological changes in music listening. IEEE Transactions on Pattern Analysis and Machine Intelligence 30(12), 2067–2083 (2008)CrossRefGoogle Scholar
  34. 34.
    Liu, C., Rani, P., Sarkar, N.: Human-robot interaction using affective cues. In: Proceedings of the 15th IEEE International Symposium on Robot and Human Interactive Communication (RO-MAN 2006), Hatfield, UK, pp. 285–290. IEEE Computer Society, Los Alamitos (2006)CrossRefGoogle Scholar
  35. 35.
    Rani, P., Sims, J., Brackin, R., Sarkar, N.: Online stress detection using psychophysiological signals for implicit human-robot cooperation. Robotica 20(6), 673–685 (2002)CrossRefGoogle Scholar
  36. 36.
    Cacioppo, J.T., Tassinary, L.G., Berntson, G.: Handbook of Psychophysiology, 3rd edn. Cambridge University Press, New York (2007)CrossRefGoogle Scholar
  37. 37.
    Sinha, R., Parsons, O.A.: Multivariate response patterning of fear. Cognition and Emotion 10(2), 173–198 (1996)CrossRefGoogle Scholar
  38. 38.
    Scheirer, J., Fernandez, R., Klein, J., Picard, R.W.: Frustrating the user on purpose: A step toward building an affective computer. Interacting with Computers 14(2), 93–118 (2002)Google Scholar
  39. 39.
    Nasoz, F., Alvarez, K., Lisetti, C.L., Finkelstein, N.: Emotion recognition from physiological signals for presence technologies. International Journal of Cognition, Technology and Work 6(1), 4–14 (2003)CrossRefGoogle Scholar
  40. 40.
    Takahashi, K.: Remarks on emotion recognition from bio-potential signals. In: Proceedings of the IEEE International Conference on Systems, Man and Cybernetics, Palmerston North, New Zealand, October 5-8, vol. 2, pp. 1655–1659 (2003)Google Scholar
  41. 41.
    Haag, A., Goronzy, S., Schaich, P., Williams, J.: Emotion recognition using bio-sensors: First steps towards an automatic system. In: André, E., Dybkjær, L., Minker, W., Heisterkamp, P. (eds.) ADS 2004. LNCS (LNAI), vol. 3068, pp. 36–48. Springer, Heidelberg (2004)Google Scholar
  42. 42.
    Kim, K.H., Bang, S.W., Kim, S.R.: Emotion recognition system using short-term monitoring of physiological signals. Medical & Biological Engineering & Computing 42(3), 419–427 (2004)CrossRefGoogle Scholar
  43. 43.
    Lisetti, C.L., Nasoz, F.: Using noninvasive wearable computers to recognize human emotions from physiological signals. EURASIP Journal on Applied Signal Processing 2004(11), 1672–1687 (2004)CrossRefGoogle Scholar
  44. 44.
    Wagner, J., Kim, J., André, E.: From physiological signals to emotions: Implementing and comparing selected methods for feature extraction and classification. In: Proceedings of the IEEE International Conference on Multimedia and Expo. (ICME), Amsterdam, The Netherlands, July 6-8, pp. 940–943 (2005)Google Scholar
  45. 45.
    Yoo, S.K., Lee, C.K., Park, J.Y., Kim, N.H., Lee, B.C., Jeong, K.S.: Neural network based emotion estimation using heart rate variability and skin resistance. In: Wang, L., Chen, K., S. Ong, Y. (eds.) ICNC 2005. LNCS, vol. 3610, pp. 818–824. Springer, Heidelberg (2005)Google Scholar
  46. 46.
    Choi, A., Woo, W.: Physiological sensing and feature extraction for emotion recognition by exploiting acupuncture spots. In: Tao, J., Tan, T., Picard, R.W. (eds.) ACII 2005. LNCS, vol. 3784, pp. 590–597. Springer, Heidelberg (2005)CrossRefGoogle Scholar
  47. 47.
    Healey, J.A., Picard, R.W.: Detecting stress during real-world driving tasks using physiological sensors. IEEE Transactions on Intelligent Transportation Systems 6(2), 156–166 (2005)CrossRefGoogle Scholar
  48. 48.
    Rani, P., Liu, C., Sarkar, N., Vanman, E.: An empirical study of machine learning techniques for affect recognition in human-robot interaction. Pattern Analysis & Applications 9(1), 58–69 (2006)CrossRefGoogle Scholar
  49. 49.
    Zhai, J., Barreto, A.: Stress detection in computer users through noninvasive monitoring of physiological signals. Biomedical Science Instrumentation 42, 495–500 (2006)Google Scholar
  50. 50.
    Jones, C.M., Troen, T.: Biometric valence and arousal recognition. In: Thomas, B.H. (ed.) Proceedings of the Australasian Computer-Human Interaction Conference (OzCHI), Adelaide, Australia, pp. 191–194 (2007)Google Scholar
  51. 51.
    Leon, E., Clarke, G., Callaghan, V., Sepulveda, F.: A user-independent real-time emotion recognition system for software agents in domestic environments. Engineering Applications of Artificial Intelligence 20(3), 337–345 (2007)CrossRefGoogle Scholar
  52. 52.
    Liu, C., Conn, K., Sarkar, N., Stone, W.: Physiology-based affect recognition for computer-assisted intervention of children with Autism Spectrum Disorder. International Journal of Human-Computer Studies 66(9), 662–677 (2008)CrossRefGoogle Scholar
  53. 53.
    Katsis, C.D., Katertsidis, N., Ganiatsas, G., Fotiadis, D.I.: Toward emotion recognition in car-racing drivers: A biosignal processing approach. IEEE Transactions on Systems, Man, and Cybernetics–Part A: Systems and Humans 38(3), 502–512 (2008)CrossRefGoogle Scholar
  54. 54.
    Yannakakis, G.N., Hallam, J.: Entertainment modeling through physiology in physical play. International Journal of Human-Computer Studies 66(10), 741–755 (2008)CrossRefGoogle Scholar
  55. 55.
    Task Force: Heart rate variability: Standards of measurement, physiological interpretation, and clinical use. European Heart Journal 17(3), 354–381 (1996)Google Scholar
  56. 56.
    Ravenswaaij-Arts, C.M.A.V., Kollee, L.A.A., Hopman, J.C.W., Stoelinga, G.B.A., Geijn, H.P.: Heart rate variability. Annals of Internal Medicine 118(6), 436–447 (1993)Google Scholar
  57. 57.
    Butler, E.A., Wilhelm, F.H., Gross, J.J.: Respiratory sinus arrhythmia, emotion, and emotion regulation during social interaction. Psychophysiology 43(6), 612–622 (2006)CrossRefGoogle Scholar
  58. 58.
    van den Broek, E.L., Schut, M.H., Westerink, J.H.D.M., van Herk, J., Tuinenbreijer, K.: Computing emotion awareness through facial electromyography. In: Huang, T.S., Sebe, N., Lew, M., Pavlović, V., Kölsch, M., Galata, A., Kisačanin, B. (eds.) ECCV 2006 Workshop on HCI. LNCS, vol. 3979, pp. 52–63. Springer, Heidelberg (2006)CrossRefGoogle Scholar
  59. 59.
    Westerink, J.H.D.M., van den Broek, E.L., Schut, M.H., van Herk, J., Tuinenbreijer, K.: 14. In: Computing emotion awareness through galvanic skin response and facial electromyography. Philips Research Book Series, vol. 8, pp. 137–150. Springer, Dordrecht (2008)Google Scholar
  60. 60.
    Cacioppo, J., Tassinary, L.: Inferring psychological significance from physiological signals. American Psychologist 45(1), 16–28 (1990)CrossRefGoogle Scholar
  61. 61.
    Mitchell, T.M.: Machine Learning. The McGraw-Hill Companies, Inc., Columbus (1997)MATHGoogle Scholar
  62. 62.
    Bishop, C.M.: Pattern Recognition and Machine Learning. Information Science and Statistics. Springer Science+Business Media, LLC, New York (2006)MATHCrossRefGoogle Scholar
  63. 63.
    Schölkopf, B., Smola, A.J.: Learning with kernels: Support Vector Machines, Regularization, Optimization, and Beyond. In: Adaptive Computation and Machine Learning. The MIT Press, Cambridge (2002)Google Scholar
  64. 64.
    Rencher, A.C.: Methods of Multivariate Analysis, 2nd edn. Wiley Series in Probability and Statistics. John Wiley & Sons, Inc., New York (2002)MATHGoogle Scholar
  65. 65.
    Rottenberg, J., Ray, R.R., Gross, J.J.: 1. In: Emotion elicitation using films, pp. 9–28. Oxford University Press, New York (2007)Google Scholar
  66. 66.
    Kreibig, S.D., Wilhelm, F.H., Roth, W.T., Gross, J.J.: Cardiovascular, electrodermal, and respiratory response patterns to fear- and sadness-inducing films. Psychophysiology 44(5), 787–806 (2007)CrossRefGoogle Scholar
  67. 67.
    Kring, A.M., Gordon, A.H.: Sex differences in emotion: Expression, experience, and physiology. Journal of Personality and Social Psychology 74(3), 686–703 (1998)CrossRefGoogle Scholar
  68. 68.
    Carrera, P., Oceja, L.: Drawing mixed emotions: Sequential or simultaneous experiences? Cognition & Emotion 21(2), 422–441 (2007)CrossRefGoogle Scholar
  69. 69.
    Russell, J.A.: A circumplex model of affect. Journal of Personality and Social Psychology 39(6), 1161–1178 (1980)CrossRefGoogle Scholar
  70. 70.
    Cover, T.M., van Campenhout, J.M.: On the possible orderings in the measurement selection problem. IEEE Transactions on Systems, Man, and Cybernetics SMC-7(9), 657–661 (1977)Google Scholar
  71. 71.
    Lawrence, S., Giles, C.L., Tsoi, A.: What size neural network gives optimal generalization? Convergence properties of backpropagation. Technical Report UMIACS-TR-96-22 and CS-TR-3617 (April 1996)Google Scholar
  72. 72.
    Barrett, L.F.: Valence as a basic building block of emotional life. Journal of Research in Personality 40, 35–55 (2006)CrossRefMathSciNetGoogle Scholar
  73. 73.
    Russel, J.A., Barrett, L.F.: Core affect, prototypical emotional episodes, and other things called emotion: Dissecting the elephant. Journal of Personality and Social Psychology 26(5), 805–819 (1999)CrossRefGoogle Scholar
  74. 74.
    Gendolla, G.H.E.: On the impact of mood on behavior: An integrative theory and a review. Review of General Psychology 4(4), 378–408 (2000)CrossRefGoogle Scholar
  75. 75.
    Cooper, C.L., Pervin, L.A.: Personality: Critical concepts in psychology, 1st edn. Critical concepts in psychology. Routledge, New York (1998)Google Scholar
  76. 76.
    Lukowicz, P.: Wearable computing and artificial intelligence for healthcare applications. Artificial Intelligence in Medicine 42(2), 95–98 (2008)CrossRefGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2010

Authors and Affiliations

  • Egon L. van den Broek
    • 1
  • Viliam Lisý
    • 2
  • Joris H. Janssen
    • 3
    • 4
  • Joyce H. D. M. Westerink
    • 3
  • Marleen H. Schut
    • 5
  • Kees Tuinenbreijer
    • 5
  1. 1.Center for Telematics and Information Technology (CTIT)University of TwenteAE EnschedeThe Netherlands
  2. 2.Agent Technology Center, Dept. of Cybernetics, FEECzech Technical UniversityPraha 6Czech Republic
  3. 3.User Experience GroupPhilips ResearchAE EindhovenThe Netherlands
  4. 4.Dept. of Human Technology InteractionEindhoven University of TechnologyEindhovenThe Netherlands
  5. 5.Philips Consumer Lifestyle Advanced TechnologyAE EindhovenThe Netherlands

Personalised recommendations