Nonverbally Smart User Interfaces: Postural and Facial Expression Data in Human Computer Interaction

  • G. Susanne Bahr
  • Carey Balaban
  • Mariofanna Milanova
  • Howard Choe
Part of the Lecture Notes in Computer Science book series (LNCS, volume 4555)


We suggest that User Interfaces (UIs) can be designed to serve as cognitive tools based on a model of nonverbal human interaction. Smart User Interfaces (SUIs) have the potential to support the human user when and where appropriate and thus indirectly facilitate higher mental processes without the need for end-user programming or external actuation. Moreover, graphic nonverbally sensitive SUIs are expected to be less likely to interfere with ongoing activity and disrupt the user. We present two non-invasive methods to assess postural and facial expression components and propose a contextual analysis to guide SUI actuation and supportive action. The approach is illustrated in a possible redesign of the Microsoft helper agent “Clippit” ®.


Facial Expression Nonverbal Communication Cognitive Tool Facial Animation Engagement Index 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Ahlberg, J.: Model-based Coding-Extraction, Coding, and Evaluation of Face Model Parameter. Diss. Linköping University, Sweden (2002)Google Scholar
  2. 2.
    Balaban, C.D., Cohn, J.V., Redfern, M.S., Prinkey, J., Stripling, R., Hoffer, M.: Postural control as a probe for cognitive state: exploiting human information processing to enhance performance. International Journal of Human-Computer Interaction 17, 275–286 (2004)CrossRefGoogle Scholar
  3. 3.
    Balaban, C.D., Prinkey, J., Frank, G., Redfern, M.: Postural measurements seated subjects as gauges of cognitive engagement. In: Schmorrow, D. (ed.) Foundations of Augmented Cognition, pp. 321–328. Laurence Erlbaum Associates Publishers, Mahwah (2005)Google Scholar
  4. 4.
    Bull, P.E.: Posture and Gesture. Pergamon, Oxford (1987)Google Scholar
  5. 5.
    Condensation Algorithm. Last retrieved 16 (December 2006), from
  6. 6.
    Cornelius, R.R.: The Science of Emotion: Research and Tradition in the psychology of Emotion. Prentice Hall, Upper Saddle River (1996)Google Scholar
  7. 7.
    Darwin, C.R.: The Expression of Emotion in Man and Animals. John Murray, London (1872)Google Scholar
  8. 8.
    Ekman, P.: Strong evidence for universals in facial expressions: A reply to Russell’s mistaken critique. Psychological Bulletin 115, 268–287 (1994)CrossRefGoogle Scholar
  9. 9.
    Fridlund, A.J.: Ekman, Paul; Oster, Harriet, Facial expressions of emotion. In: Siegman, Aron, W., Feldstein, Stanley (ed.) Nonverbal behavior and communication 2nd ed. pp. 143–223. Lawrence Erlbaum (1987)Google Scholar
  10. 10.
    Fraenkel, G.S., Gunn, D.L.: The orientation of animals: Kineses, taxes and compass reactions. Dover, New York (1961)Google Scholar
  11. 11.
    Johnson, L., Shaw, E., Ganeshan, R.: Pedagogical Agents on the Web. In: Proceedings of the Third annual conference on Autonomous Agents, pp. 283–290, Seattle, Washington, United States ( 1999)Google Scholar
  12. 12.
    Jonassen, D.H.: What are cognitive Tools. In: Kommers, P., Jonassen, D.H., Mayes, T. (eds.) Cognitive tools for learning, Springer, Heidelberg (1992)Google Scholar
  13. 13.
    Klein, J., Moon, Y., Picard, R.W.: This computer responds to user frustration. Interacting with Computers 14, 119–140 (2002)Google Scholar
  14. 14.
    Knapp, M.L., Hall, J.A.: Nonverbal communication in human interaction, 6th edn. Thomson Wadsworth (2006)Google Scholar
  15. 15.
    Luening, E.: Microsoft tool Clippy gets pink slip. Retrieved October 15, 2005, (2001, April 11), from
  16. 16.
    Lever, N.: Real-time 3D Characteranimation. Focal press, Woburn (2003)Google Scholar
  17. 17.
    Pandzic, I.S., Forchheimer, R.: MPEG4 Facial Animation. John Wiley & Sons Publishers, Hoboken, NJ (2002)Google Scholar
  18. 18.
    Parke, F.I., Waters, K.: Computer Facial animation. A.K. Peters Ltd, Wellesley, MA (1996)Google Scholar
  19. 19.
    Redican, W.K.: An Evolutionary on Human Facial Displays. In: Ekman, P. (ed.) Emotion in the Human Face, 2nd edn. Cambridge University Press, Cambridge (1982)Google Scholar
  20. 20.
    Thurrott, P.: Microsoft Office XP reviewed (January 22, 2002), Retrieved December 8, 2006, from
  21. 21.
    office10_whatsnew.aspGoogle Scholar
  22. 22.
    Vygotsky, L.S. (ed.): Educational Psychology, 1st edn.(reprint) Pentagon Press, New Delhi, India (1926–2006)Google Scholar
  23. 23.
    Zacks, J.M., Tversky, B.: Event structure in perception and conception. Psychological Bulletin 127, 3–21 (2001)CrossRefGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2007

Authors and Affiliations

  • G. Susanne Bahr
    • 1
  • Carey Balaban
    • 2
  • Mariofanna Milanova
    • 3
  • Howard Choe
    • 4
  1. 1.Florida Institute of Technology, Melbourne, FloridaU.S.A.
  2. 2.University of Pittsburgh, Pittsburgh, PennsylvaniaU.S.A.
  3. 3.University of Arkansas at Little Rock, Little Rock, ArkansasU.S.A.
  4. 4.Raytheon - Network Centric Systems (NCS), Plano, TexasU.S.A.

Personalised recommendations