Cognition, Technology & Work

, Volume 6, Issue 1, pp 4–14 | Cite as

Emotion recognition from physiological signals using wireless sensors for presence technologies

  • Fatma Nasoz
  • Kaye Alvarez
  • Christine L. Lisetti
  • Neal Finkelstein
Original Article


In this article we describe a new approach to enhance presence technologies. First, we discuss the strong relationship between cognitive processes and emotions and how human physiology is uniquely affected when experiencing each emotion. Secondly, we introduce our prototype multimodal affective user interface. In the remainder of the paper we describe the emotion elicitation experiment we designed and conducted and the algorithms we implemented to analyse the physiological signals associated with emotions. These algorithms can then be used to recognise the affective states of users from physiological data collected via non-invasive technologies. The affective intelligent user interfaces we plan to create will adapt to user affect dynamically in the current context, thus providing enhanced social presence.


Emotion recognition Social presence User interfaces 


  1. Allen, A. Roman, L. Cox, R. and Cardwell, B. (1996) Home health visits using a cable television network: user satisfaction. J Telemed Telecare 2:92–94CrossRefPubMedGoogle Scholar
  2. Baylor AL (2000) Beyond butlers: Intelligent agents as mentors. J Educ Comput Res 22:373–382Google Scholar
  3. Bianchi N, Lisetti CL (2002) Modeling multimodal expression of user’s affective subjective experience. Int J User Model User-adapt Interact 12(1):49–84Google Scholar
  4. Birdwhistle R (1970) Kinesics and context: essays on body motion and communication. University of Pennsylvania PressGoogle Scholar
  5. Bower G (1981) Mood and memory. Am Psychol 36(2)Google Scholar
  6. Briggs P, Burford B, Dracup C (1998) Modeling self-confidence in users of a computer system showing unrepresentative design. Int J Hum Comput Stud 49:717–742CrossRefGoogle Scholar
  7. Casanueva JS, Blake EH (2001) The effects of avatars on co-presence in a collaborative virtual environment. Technical Report CS01–02–00, Department of Computer Science, University of Cape Town, South AfricaGoogle Scholar
  8. Chovil N (1991) Discourse-oriented facial displays in conversation. Res Lang Soc Interact 25:163–194Google Scholar
  9. Collet C, Vernet-Maury E, Delhomme G, Dittmar A (1997) Autonomic nervous system response patterns specificity to basic emotions. J Auton Nerv Syst 62(1–2):45–57Google Scholar
  10. Colquitt JA, LePine JA, Noe RA (2000) Toward an integrative theory of training motivation: A meta-analytic path analysis of 20 years of research. J Appl Psychol 85:678–707Google Scholar
  11. Crist TM, Kaufman SB, Crampton KR (1996) Home telemedicine: a home health care agency strategy for maximizing resources. Home Health Care Manage Pract 8:1-9Google Scholar
  12. Damasio A (1994) Descartes’ error. Avon, New YorkGoogle Scholar
  13. Darkins AW, Carey MA (2000) Telemedicine and telehealth: principles, policies performance and pitfalls. Springer, Berlin Heidelberg New YorkGoogle Scholar
  14. Derryberry D, Tucker D (1992) Neural mechanisms of emotion. J Consult Clin Psychol 60(3):329–337PubMedGoogle Scholar
  15. Dillon C, Keogh E, Freeman J, Davidoff J (2000) Aroused and immersed: the psychophysiology of presence. In: Proceedings of 3rd International Workshop on Presence, Delft University of Technology, Delft, The Netherlands, March 2000, pp 27–28Google Scholar
  16. Ekman P (1989) Handbook of social psychophysiology, pp 143–146. Wiley, ChichesterGoogle Scholar
  17. Ekman P, Levenson RW, Friesen WV (1983) Autonomic nervous system activity distinguishes between emotions. Science 221:1208–1210PubMedGoogle Scholar
  18. Ekman P, Friesen WV (1975) Unmasking the face: a guide to recognizing emotions from facial expressions. Prentice Hall, Englewood Cliffs, NJGoogle Scholar
  19. Feldman Barrett L, Gross JJ, Conner Christensen T, Benvenuto M (2001) Knowing what you’re feeling and knowing what to do about it: mapping the relation between emotion differentiation and emotion regulation. Cognit Emotion 15:713–724CrossRefGoogle Scholar
  20. Frijda N (1986) The emotions. New York: Cambridge University Press. MIT PressGoogle Scholar
  21. Goleman D (1995) Emotional intelligence. Bantam, New YorkGoogle Scholar
  22. Gross JJ, Levenson RW (1997) Hiding feelings: the acute effects of inhibiting negative and positive emotions. J Abnorm Psychol 10(1):95–103Google Scholar
  23. Gross JJ, Levenson RW (1995) Emotion elicitation using films. Cognit Emotion 9:87–108Google Scholar
  24. Guinn C, Hubal H (2003) Extracting emotional information from the text of spoken dialog. In: Proceedings of user modeling (UM) 03 Workshop “assessing and adapting to user attitudes and affect: why, when and how?”, Pittsburgh, PAGoogle Scholar
  25. Hagan MT, Menhaj MB (1994) Training feedforward networks with the marquardt algorithm. IEEE Trans Neural Netw 5(6): 989–993CrossRefGoogle Scholar
  26. IJsselsteijn WA (2002) Elements of a multi-level theory of presence: phenomenology, mental processing and neural correlates. In: Proceedings of PRESENCE 2002, pp 245–259 Universidade Fernando Pessoa, Porto, Portugal, 9–11 October 2002Google Scholar
  27. IJsselsteijn WA, de Ridder H, Freeman J, Avons SE (2000) Presence: concept, determinants and measurement. In: Proceedings of the SPIE, Human Vision and Electronic Imaging V, 3959–76Google Scholar
  28. James L (2000) Road rage and aggressive driving. Prometheus, Amherst, NYGoogle Scholar
  29. Kalawsky RS (2000) The validity of presence as a reliable human performance metric in immersive environments. In: Proceedings of 3rd international workshop on presence, Delft University of Technology, Delft, The NetherlandsGoogle Scholar
  30. Larson J, Rodriguez C (1999) Road rage to road-wise. Tom Doherty Associates, New YorkGoogle Scholar
  31. Lewis VE, Williams RN (1989) Mood-congruent vs mood-state-dependent learning: implications for a view of emotion. J Soc Behav Pers 4:157–171Google Scholar
  32. Ledoux J (1992) Brain mechanisms of emotion and emotional learning. Curr Opin Neurobiol 2:191–197PubMedGoogle Scholar
  33. Lisetti CL (1999) A user model of emotion-cognition. In Proceedings of the UM’99 Workshop on Attitude, Personality, and Emotions in User-Adapted Interaction (Banff, Canada, June 1999)Google Scholar
  34. Lisetti CL, Nasoz F (2002) MAUI: a multimodal affective user interface. In: Proceedings of ACM Multimedia International Conference, Juan les Pins, France, December 2002Google Scholar
  35. Lisetti CL, Nasoz F, Lerouge C, Ozyer O, Alvarez K (2003) Developing multimodal intelligent affective interfaces for tele-home health care. Int J Hum Comput Stud 59(1–2):245–255Google Scholar
  36. Lombard M, Ditton T (1997) At the heart of it all: the concept of presence. J Comput Mediated Commun 3(2)Google Scholar
  37. Lorenz R, Gregory RP, Davis DL (2000) Utility of a brief self-efficacy scale in clinical training program evaluation. Eval Health Prof 23:182–193PubMedGoogle Scholar
  38. Martocchio JJ, Dulebohn J (1994) Performance feedback effects in training: the role of perceived controllability. Pers Psychol 47:357–373Google Scholar
  39. Martocchio JJ, Judge TA (1997) Relationship between conscientiousness and learning in employee training: mediating influences of self-deception and self-efficacy. J Appl Psychol 82:764–773CrossRefPubMedGoogle Scholar
  40. Martocchio JJ (1994) Effects of conceptions of ability on anxiety, self-efficacy, and learning in training. J Appl Psychol 79:819–825CrossRefPubMedGoogle Scholar
  41. Minsky M (1980) Telepresence. Omni, June 1980:45–51Google Scholar
  42. Mitchell TM (1997) Machine learning. McGraw-HillGoogle Scholar
  43. Nicol AA (1999) Presenting your findings: a practical guide for creating tables. American Physiological Association, Washington, DCGoogle Scholar
  44. Picard RW, Healey J, Vyzas E (2001) Toward machine emotional intelligence analysis of affective physiological state. IEEE Trans Pattern Anal 23(10):1175–1191CrossRefGoogle Scholar
  45. Rozell EJ, Gardner WL (2000) Cognitive, motivation, and affective processes associated with computer-related performance: a path analysis. Comput Hum Behav 16:199–222CrossRefGoogle Scholar
  46. Takeuchi A, Nagao K (1993) Communicative facial displays as a new conversational modality. In: Proceedings of the INTERCHI’93 conference on human factors in computing systems, Amsterdam pp 187–193Google Scholar
  47. Warner I (1997) Telemedicine applications for home health care. J Telemed Telecare 3:65–66CrossRefPubMedGoogle Scholar
  48. Warr P, Bunce D (1995) Trainee characteristics and the outcomes of open learning. Pers Psychol 48:347–375Google Scholar
  49. Zajonc R (1984) On the primacy of affect. Am Psychol 39:117–124Google Scholar

Copyright information

© Springer-Verlag London Limited 2004

Authors and Affiliations

  • Fatma Nasoz
    • 1
  • Kaye Alvarez
    • 2
  • Christine L. Lisetti
    • 1
  • Neal Finkelstein
    • 3
  1. 1.Department of Computer ScienceUniversity of Central FloridaOrlandoUSA
  2. 2.Personnel Board of Jefferson CountyBirminghamUSA
  3. 3.Simulation Technology CenterOrlandoUSA

Personalised recommendations