Human Computer Intelligent Interaction Using Augmented Cognition and Emotional Intelligence

  • Jim X. Chen
  • Harry Wechsler
Part of the Lecture Notes in Computer Science book series (LNCS, volume 4563)


Human-Computer Interaction (HCI) has mostly developed along two competing methodologies: direct manipulation and intelligent agents. Other possible but complementary methodologies are those of augmented cognition and affective computing and their adaptive combination. Augmented cognition harnesses computation to exploit explicit or implicit knowledge about context, mental state, and motivation for the user, while affective computing provides the means to recognize emotional intelligence and affects human-computer interfaces and interactions people are engaged with. Most HCI studies elicit emotions in relatively simple settings, whereas augmented cognition and affective computing include bodily (physical) embedded within mental (cognitive) and emotional events. Recognition of affective states currently focuses on their physical form (e.g., blinking or face distortions underlying human emotions) rather than implicit behavior and function (their impact on how the user employs the interface or communicates with others). Augmented cognition and affective computing are examined throughout this paper regarding design, implementation, and benefits. Towards that end we have designed an HCII interface that diagnoses and predicts whether the user was fatigued, confused, frustrated, momentarily distracted, or even alive through non-verbal information, namely paralanguage, in a virtual reality (VR) learning environment.


Virtual Reality Independent Component Analysis Emotional Intelligence Independent Component Analysis Facial Action 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Anderson, J.R., Matessa, M., Lebiere, C.: ACT-R: A Theory of Higher Level Cognition and Its Relation to Visual Attention. Human-Computer Interaction 12, 439–462 (1997)CrossRefGoogle Scholar
  2. 2.
    Anderson, J.R., Lebiere, C. (eds.): Atomic Components of Thought. Erlbaum, Hillsdale, NJ (1998)Google Scholar
  3. 3.
    Baron-Cohen, S., Cross, P.: Reading the Eyes: Evidence for the Role of Perception in the Development of a Theory of Mind. Mind and Language 7, 182–186 (1992)CrossRefGoogle Scholar
  4. 4.
    Byrne, M.D., Anderson, J.R.: Perception and Action. In: Anderson, J.R., Lebiere, C. (eds.) Atomic Components of Thought, pp. 167–200. Erlbaum, Mahwah (1998)Google Scholar
  5. 5.
    Chen, J.X.: Learning Abstract Concepts. Computers & Graphics 30(1), 10–19 (2006)CrossRefGoogle Scholar
  6. 6.
    Chen, J.X., Yang, Y., Loftin, R.B.: MUVEES: a PC-based Multi-User Virtual Environment for Learning. In: IEEE Virtual Reality (VR 2003), pp. 163–170. IEEE Computer Society Press, New York (2003)CrossRefGoogle Scholar
  7. 7.
    Cipolla, R., Pentland, A. (eds.): Computer Vision for Human-Machine Interaction. Camb. Univ. Press, Cambridge (1998)Google Scholar
  8. 8.
    Cohn, J., Schmidt, K., Gross, R., Ekman, P.: Individual Differences in Facial Expression. In: Int. Conf. on Multimodal User Interfaces, Pittsburgh, PA, pp. 491–496 (2002)Google Scholar
  9. 9.
    Cole, J.: About Face. MIT Press, Cambridge, MA (1998)Google Scholar
  10. 10.
    Donato, G., Bartlett, M.S., Hager, J.C., Ekman, P., Sejnowski, T.J.: Classifying Facial Actions. IEEE Trans. on Pattern Analysis and Machine Intelligence 21(10), 974–989 (1999)CrossRefGoogle Scholar
  11. 11.
    Duric, Z., et al.: Integrating Perceptual and Cognitive Modeling for Adaptive and Intelligent Human-Computer Interaction. Proc. IEEE 90(7), 1272–1289 (2002)CrossRefGoogle Scholar
  12. 12.
    Ekman, P., Friesen, W.V.: Facial Action Coding System (FACS): A Technique for the Measurement of Facial Actions. Consulting Psychologists Press, Palo Alto, CA (1978)Google Scholar
  13. 13.
    Ekman, P.: An Argument About Basic Emotions. Cognition and Emotion 6, 162–200 (1992)Google Scholar
  14. 14.
    Etkoff, N.L., et al.: Lie Detection and Language Comprehension. Nature 405, 139 (2000)CrossRefGoogle Scholar
  15. 15.
    Fasel, B., Luettin, J.: Automatic Facial Expression Analysis. Pattern Recognition 36, 259–275 (2003)zbMATHCrossRefGoogle Scholar
  16. 16.
    Fernandez-Dols, J., et al.: The Meaning of Expressions: Views from Art and Other Sources. In: Anolli, L., et al. (eds.) Say not to Say: New Perspectives on Miscommunication, IOS Press, Amsterdam, Trento, Italy (2001)Google Scholar
  17. 17.
    Fidaleo, D., Neumann, U.: Coart: Coarticulation Region Analysis for Control of 2D Faces. Computer Animation, 17–22 (2002)Google Scholar
  18. 18.
    Fidaleo, D.: G-Folds: An Appearance Based Model of Facial Gestures for Performance Driven Facial Animation, PhD Thesis, University of Southern California (2003)Google Scholar
  19. 19.
    Fogassi, L., Ferrari, P.F., Gesierich, B., Rozzi, S., Chersi, F., Rizzolatti, G.: Parietal Lobe: From Action Organization to Intention Understanding. Science 308, 662–667 (2005)CrossRefGoogle Scholar
  20. 20.
    Friedlund, A.J.: Evolution and Facial Action in Reflex, Social Motive and Paralanguage. Biological Psychology 32, 3–100 (1991)CrossRefGoogle Scholar
  21. 21.
    Fu, G.: A New Strategy for K-12 Students to Learn Math, Dissertation Proposal. George Mason Univ. (2004)Google Scholar
  22. 22.
    Ginsburg, G.P., Harrington, M.E.: Bodily States and Context in Situated Line of Actions. In: Harre, R., Parrot, W.G. (eds.) The Emotions. Social, Cultural and Biological Dimensions, Sage, London (1996)Google Scholar
  23. 23.
    Gray, J.A.: The Neuropsychology of Anxiety. Oxford University Press, Oxford (1982)Google Scholar
  24. 24.
    Izard, C.E.: The Face of Emotion, Appleton-Century-Crofts (1971)Google Scholar
  25. 25.
    Kieras, D.E., Meyer, D.E.: An Overview of the EPIC Architecture for Cognition and Performance with Application to Human Computer Interaction. Human-Computer Interaction 12, 391–438 (1997)CrossRefGoogle Scholar
  26. 26.
    Knapp, M.I., Hall, J.A.: Nonverbal Communication in Human Interaction, 6th edn. WadsworthGoogle Scholar
  27. 27.
    Koenen, R.: MPEG-4 Project Overview, ISO/IEC JTC1/SC29/WG11 (2000)Google Scholar
  28. 28.
    Maes, P., Shneiderman, B.: Direct Manipulation vs. Interface Agents: a Debate. Interactions 4(6) (1997)Google Scholar
  29. 29.
    McNeill, D.: The Face - A Natural History, Little, Brown and Company (1998)Google Scholar
  30. 30.
    Nakahara, K., Miyashita, Y.: Understanding Intentions. Science 308, 644–645 (2005)CrossRefGoogle Scholar
  31. 31.
    Ortony, A., Taylor, T.J.: What’s Basic About Emotions. Psychological Review 97, 315–331Google Scholar
  32. 32.
    Panksepp, J.: Toward a General Psychobiological Theory of Emotions. Behavioral and Brain Sciences 5, 405–467Google Scholar
  33. 33.
    Pearl, J.: Probabilistic Reasoning in Intelligent Systems: Networks of Plausible Inference. Morgan Kaufmann, Seattle, Washington, USA (1988)Google Scholar
  34. 34.
    Pentland, A.: Perceptual Intelligence. Comm. of ACM 43(3), 35–44 (2000)CrossRefGoogle Scholar
  35. 35.
    Pentland, A.: Socially Aware Computing and Communication. Computer 38(3), 33–40 (2005)CrossRefGoogle Scholar
  36. 36.
    Picard, R.: Affective Computing. MIT Press, Cambridge, MA (1998)Google Scholar
  37. 37.
    Salzman, M.C., Dede, C.J., Loftin, R.B., Chen, J.X.: Understanding How Immersive VR Learning Environments Support Learning through Modeling. Presence 8(3), 293–316 (1999)CrossRefGoogle Scholar
  38. 38.
    Wechsler, H., Duric, Z., Li, F., Cherkassky, V.: Motion Estimation Using Statistical Learning Theory. IEEE Trans. on Pattern Analysis and Machine Intelligence 26(4), 466–478 (2004)CrossRefGoogle Scholar
  39. 39.
    Wechsler, H.: Reliable Face Recognition Systems: System Design, Implementation, and Evaluation. Springer, Heidelberg (2006)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2007

Authors and Affiliations

  • Jim X. Chen
    • 1
  • Harry Wechsler
    • 1
  1. 1.Computer Science Department, George Mason University, Fairfax, VA 22030USA

Personalised recommendations