Face for Ambient Interface

  • Maja Pantic
Part of the Lecture Notes in Computer Science book series (LNCS, volume 3864)


The human face is used to identify other people, to regulate the conversation by gazing or nodding, to interpret what has been said by lip reading, and to communicate and understand social signals, including affective states and intentions, on the basis of the shown facial expression. Machine understanding of human facial signals could revolutionize user-adaptive social interfaces, the integral part of ambient intelligence technologies. Nonetheless, development of a face-based ambient interface that detects and interprets human facial signals is rather difficult. This article summarizes our efforts in achieving this goal, enumerates the scientific and engineering issues that arise in meeting this challenge and outlines recommendations for accomplishing this objective.


Facial Expression Face Image Facial Feature Emotion Category Ambient Intelligence 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Aarts, E.: Ambient Intelligence – Visualizing the Future. In: Proc. Conf. Smart Objects & Ambient Intelligence (2005), http://www.soc-eusai2005.org/
  2. 2.
    Ambady, N., Rosenthal, R.: Thin Slices of Expressive Behavior as Predictors of Interpersonal Consequences: A Meta-Analysis. Psychological Bulletin 111(2), 256–274 (1992)CrossRefGoogle Scholar
  3. 3.
    Anderson, K., McOwan, P.W.: A Real-Time Automated System for Recognition of Human Facial Expressions. IEEE Trans. Systems, Man, and Cybernetics, Part B 36(1), 96–105 (2006)CrossRefGoogle Scholar
  4. 4.
    Aristotle: Physiognomonica. In: Ross, W.D. (ed.) The works of Aristotle, pp. 805–813. Clarendon, Oxford (1913)Google Scholar
  5. 5.
    Baker, S., Matthews, I., Xiao, J., Gross, R., Kanade, T.: Real-time non-rigid driver head tracking for driver mental state estimation. In: Proc. World Congress on Intelligent Transportation Systems (2004), http://www.ri.cmu.edu/projects/project_448.html
  6. 6.
    Barron, J., Fleet, D., Beauchemin, S.: Performance of optical flow techniques. J. Computer Vision 12(1), 43–78 (1994)CrossRefGoogle Scholar
  7. 7.
    Bartlett, M.S., Hager, J.C., Ekman, P., Sejnowski, T.J.: Measuring facial expressions by computer image analysis. Psychophysiology 36, 253–263 (1999)CrossRefGoogle Scholar
  8. 8.
    Bartlett, M.S., Littlewort, G., Lainscsek, C., Fasel, I., Movellan, J.R.: Machine Learning Methods for Fully Automatic Recognition of Facial Expressions and Facial Actions. In: Proc. Conf. Systems, Man, and Cybernetics, vol. 1, pp. 592–597 (2004)Google Scholar
  9. 9.
    Bassili, J.N.: Facial Motion in the Perception of Faces and of Emotional Expression. J. Experimental Psychology 4, 373–379 (1978)Google Scholar
  10. 10.
    Black, M., Yacoob, Y.: Recognizing facial expressions in image sequences using local parameterized models of image motion. Computer Vision 25(1), 23–48 (1997)CrossRefGoogle Scholar
  11. 11.
    Bobick, A.F., Davis, J.W.: The Recognition of Human Movement Using Temporal Templates. IEEE Trans. Pattern Analysis and Machine Intelligence 23(3), 257–267 (2001)CrossRefGoogle Scholar
  12. 12.
    Bowyer, K.W.: Face Recognition Technology – Security vs. Privacy. IEEE Technology and Society Magazine 23(1), 9–19 (2004)CrossRefGoogle Scholar
  13. 13.
    Bruce, V.: Recognizing Faces. Lawrence Erlbaum Assoc., Hove (1986)Google Scholar
  14. 14.
    Cohen, I., Sebe, N., Garg, A., Chen, L.S., Huang, T.S.: Facial expression recognition from video sequences – temporal and static modeling. Computer Vision and Image Understanding 91, 160–187 (2003)CrossRefGoogle Scholar
  15. 15.
    Cohn, J.F., Reed, L.I., Ambadar, Z., Xiao, J., Moriyama, T.: Automatic analysis and recognition of brow actions and head motion in spontaneous facial behavior. In: Proc. Conf. Systems, Man and Cybernetics, vol. 1, pp. 610–616 (2004)Google Scholar
  16. 16.
    Cohn, J.F., Zlochower, A.J., Lien, J., Kanade, T.: Automated face analysis by feature point tracking has high concurrent validity with manual faces coding. Psychophysiology 36, 35–43 (1999)CrossRefGoogle Scholar
  17. 17.
    Cristinacce, D., Cootes, T.F.: A Comparison of Shape Constrained Facial Feature Detectors. In: Proc. Conf. Automatic Face and Gesture Recognition, pp. 375–380 (2004)Google Scholar
  18. 18.
    Darwin, C.: The expression of the emotions in man and animals. University of Chicago Press, Chicago (1965 / 1872)CrossRefGoogle Scholar
  19. 19.
    DeCarlo, D., Metaxas, D.: The integration of optical flow and deformable models with applications to human face shape and motion estimation. In: Proc. Conf. Computer Vision and Pattern Recognition, pp. 231–238 (1996)Google Scholar
  20. 20.
    Dishman, E.: Inventing wellness systems for aging in place. IEEE Computer Magazine, Spec. Issue on Computing and the Aging 37(5), 34–41 (2004)CrossRefGoogle Scholar
  21. 21.
    Donato, G., Bartlett, M.S., Hager, J.C., Ekman, P., Sejnowski, T.J.: Classifying Facial Actions. IEEE Trans. Pattern Analysis and Machine Intelligence 21(10), 974–989 (1999)CrossRefGoogle Scholar
  22. 22.
    Ekman, P.: Emotions Revealed. Times Books, New York (2003)Google Scholar
  23. 23.
    Ekman, P., Friesen, W.V.: The repertoire of nonverbal behavior. Semiotica 1, 49–98 (1969)CrossRefGoogle Scholar
  24. 24.
    Ekman, P., Friesen, W.V.: Unmasking the face. Prentice-Hall, New Jersey (1975)Google Scholar
  25. 25.
    Ekman, P., Friesen, W.V.: Facial Action Coding System. Consulting Psychologist Press, Palo Alto (1978)Google Scholar
  26. 26.
    Ekman, P., Friesen, W.V., Hager, J.C.: Facial Action Coding System. A Human Face, Salt Lake City (2002)Google Scholar
  27. 27.
    Fasel, I., Fortenberry, B., Movellan, J.R.: GBoost: A generative framework for boosting with applications to real-time eye coding. Computer Vision and Image Understanding, under review, http://mplab.ucsd.edu/publications/
  28. 28.
    Friedman, J., Hastie, T., Tibshirani, R.: Additive logistic regression: A statistical view of boosting. Annals of Statistics 28(2), 337–374 (2000)MathSciNetCrossRefMATHGoogle Scholar
  29. 29.
    Gokturk, S.B., Bouguet, J.Y., Tomasi, C., Girod, B.: Model-based face tracking for view-independent facial expression recognition. In: Proc. Conf. Automatic Face and Gesture Recognition, pp. 272–278 (2002)Google Scholar
  30. 30.
    Gross, T.: Ambient Interfaces – Design Challenges and Recommendations. In: Proc. Conf. Human-Computer Interaction, pp. 68–72 (2003)Google Scholar
  31. 31.
    Gu, H., Ji, Q.: Information extraction from image sequences of real-world facial expressions. Machine Vision and Applications 16(2), 105–115 (2005)CrossRefGoogle Scholar
  32. 32.
    Guo, G., Dyer, C.R.: Learning From Examples in the Small Sample Case – Face Expression Recognition. IEEE Trans. Systems, Man, and Cybernetics, Part B 35(3), 477–488 (2005)CrossRefGoogle Scholar
  33. 33.
    Haykin, S., de Freitas, N., (eds): Special Issue on Sequential State Estimation. Proceedings of the IEEE 92(3), 399–574 (2004)Google Scholar
  34. 34.
    Isard, M., Blake, A.: Condensation – conditional density propagation for visual tracking. J. Computer Vision 29(1), 5–28 (1998)CrossRefGoogle Scholar
  35. 35.
    Jacobs, D.W., Osadchy, M., Lindenbaum, M.: What Makes Gabor Jets Illumination Insensitive, http://rita.osadchy.net/papers/gabor-3.pdf
  36. 36.
    Kalman, R.E.: A new approach to linear filtering and prediction problems. Trans. ASME J. Basic Eng. 82, 35–45 (1960)CrossRefGoogle Scholar
  37. 37.
    Kanade, T., Cohn, J., Tian, Y.: Comprehensive database for facial expression analysis. In: Proc. Conf. Automatic Face and Gesture Recognition, pp. 46–53 (2000)Google Scholar
  38. 38.
    Keltner, D., Ekman, P.: Facial Expression of Emotion. In: Lewis, M., Haviland-Jones, J.M. (eds.) Handbook of Emotions, 2nd edn., pp. 236–249. The Guilford Press, New York (2004)Google Scholar
  39. 39.
    Li, S.Z., Jain, A.K. (eds.): Handbook of Face Recognition. Springer, New York (2005)MATHGoogle Scholar
  40. 40.
    van Loenen, E.J.: On the role of Graspable Objects in the Ambient Intelligence Paradigm. In: Proc. Conf. Smart Objects (2003), http://www.grenoble-soc.com/
  41. 41.
    Lucas, B.D., Kanade, T.: An iterative image registration technique with an application to stereo vision. In: Proc. Conf. Artificial Intelligence, pp. 674–679 (1981)Google Scholar
  42. 42.
    Martinez, A.M.: Matching expression variant faces. Vision Research 43, 1047–1060 (2003)CrossRefGoogle Scholar
  43. 43.
    Mase, K.: Recognition of facial expression from optical flow. IEICE Transactions E74(10), 3474–3483 (1991)Google Scholar
  44. 44.
    Moghaddam, B., Pentland, A.: Probabilistic Visual Learning for Object Recognition. IEEE Trans. Pattern Analysis and Machine Intelligence 19(7), 696–710 (1997)CrossRefGoogle Scholar
  45. 45.
    Norman, D.A.: The Invisible Computer. MIT Press, Cambridge (1999)Google Scholar
  46. 46.
    Ortony, A., Turner, T.J.: What is basic about basic emotions? Psychological Review 74, 315–341 (1990)CrossRefGoogle Scholar
  47. 47.
    Pantic, M.: Face for Interface. In: Pagani, M. (ed.) The Encyclopedia of Multimedia Technology and Networking 1, pp. 308–314. Idea Group Reference, Hershy (2005)Google Scholar
  48. 48.
    Pantic, M., Patras, I.: Detecting facial actions and their temporal segments in nearly frontal-view face image sequences. In: Proc. Conf. Systems, Man, and Cybernetics (2005)Google Scholar
  49. 49.
    Pantic, M., Patras, I.: Dynamics of Facial Expressions – Recognition of Facial Actions and their Temporal Segments from Face Profile Image Sequences. IEEE Trans. Systems, Man, and Cybernetics, Part B 36 (2006)Google Scholar
  50. 50.
    Pantic, M., Rothkrantz, L.J.M.: Expert system for automatic analysis of facial expression. Image and Vision Computing 18(11), 881–905 (2000)CrossRefGoogle Scholar
  51. 51.
    Pantic, M., Rothkrantz, L.J.M.: Automatic Analysis of Facial Expressions – The State of the Art. IEEE Trans. Pattern Analysis and Machine Intelligence 22(12), 1424–1445 (2000)CrossRefGoogle Scholar
  52. 52.
    Pantic, M., Rothkrantz, L.J.M.: Toward an Affect-Sensitive Multimodal Human-Computer Interaction. Proceedings of the IEEE, Spec. Issue on Human-Computer Multimodal Interface 91(9), 1370–1390 (2003)Google Scholar
  53. 53.
    Pantic, M., Rothkrantz, L.J.M.: Facial Action Recognition for Facial Expression Analysis from Static Face Images. IEEE Trans. Systems, Man, and Cybernetics, Part B 34(3), 1449–1461 (2004)CrossRefGoogle Scholar
  54. 54.
    Pantic, M., Rothkrantz, L.J.M.: Case-based reasoning for user-profiled recognition of emotions from face images. In: Proc. Conf. Multimedia and Expo, vol. 1, pp. 391–394 (2005)Google Scholar
  55. 55.
    Pantic, M., Sebe, N., Cohn, J.F., Huang, T.: Affective Multimodal Human-Computer Interaction. In: Proc. ACM Conf. Multimedia (2005)Google Scholar
  56. 56.
    Pantic, M., Valstar, M.F., Rademaker, R., Maat, L.: Web-based database for facial expression analysis. In: Proc. Conf. Multimedia and Expo (2005), http://www.mmifacedb.com/
  57. 57.
    Patras, I., Pantic, M.: Particle Filtering with Factorized Likelihoods for Tracking Facial Features. In: Proc. Conf. Automatic Face and Gesture Recognition, pp. 97–102 (2004)Google Scholar
  58. 58.
    Patras, I., Pantic, M.: Tracking Deformable Motion. In: Proc. Conf. Systems, Man, and Cybernetics (2005)Google Scholar
  59. 59.
    Pentland, A.: Looking at people – Sensing for ubiquitous and wearable computing. IEEE Trans. Pattern Analysis and Machine Intelligence 22(1), 107–119 (2000)CrossRefGoogle Scholar
  60. 60.
    Pentland, A., Moghaddam, B., Starner, T.: View-Based and Modular Eigenspaces for Face Recognition. In: Proc. Conf. Computer Vision and Pattern Recognition, pp. 84–91 (1994)Google Scholar
  61. 61.
    Picard, R.W.: Affective Computing. MIT Press, Cambridge (1997)CrossRefGoogle Scholar
  62. 62.
    Pitt, M.K., Shephard, N.: Filtering via simulation: auxiliary particle filtering. J. Amer. Stat. Assoc. 94, 590–599 (1999)CrossRefMATHGoogle Scholar
  63. 63.
    Preece, J., Rogers, Y., Sharp, H.: Interaction Design – Beyond Human-Computer Interaction. John Wiley & Sons, New York (2002)Google Scholar
  64. 64.
    Raisinghani, M.S., Benoit, A., Ding, J., Gomez, M., Gupta, K., Gusila, V., Power, D., Schmedding, O.: Ambient Intelligence – Changing Forms of Human-Computer Interaction and their Social Implications. J. Digital Information 5(4), 1–8 (2004)Google Scholar
  65. 65.
    Remagnino, P., Foresti, G.L.: Ambient Intelligence – A New Multidisciplinary Paradigm. IEEE Trans. Systems, Man, and Cybernetics, Part A, Spec. Issue on Ambient Intelligence 35(1), 1–6 (2005)CrossRefGoogle Scholar
  66. 66.
    Rowley, H., Baluja, S., Kanade, T.: Neural Network-Based Face Detection. IEEE Trans. Pattern Analysis and Machine Intelligence 20(1), 23–38 (1998)CrossRefGoogle Scholar
  67. 67.
    Russell, J.A., Fernandez-Dols, J.M. (eds.): The Psychology of Facial Expression. Cambridge University Press, Cambridge (1997)Google Scholar
  68. 68.
    Samal, A., Iyengar, P.A.: Automatic recognition and analysis of human faces and facial expressions: A survey. Pattern Recognition 25(1), 65–77 (1992)CrossRefGoogle Scholar
  69. 69.
    Scherer, K.R., Ekman, P. (eds.): Handbook of methods in non-verbal behavior research. Cambridge University Press, Cambridge (1982)Google Scholar
  70. 70.
    Schmidt, K.L., Cohn, J.F.: Dynamics of facial expression: Normative characteristics and individual differences. In: Proc. Conf. Multimedia and Expo, pp. 547–550 (2001)Google Scholar
  71. 71.
    Shadbolt, N.: Ambient Intelligence. IEEE Intelligent Systems 18(4), 2–3 (2003)CrossRefGoogle Scholar
  72. 72.
    Shi, J., Tomasi, C.: Good features to track. In: Proc. Conf. Computer Vision and Pattern Recognition, pp. 593–600 (1994)Google Scholar
  73. 73.
    Stephanidis, C., Akoumianakis, D., Sfyrakis, M., Paramythis, A.: Universal accessibility in HCI. In: Proc. ERCIM Workshop. User Interfaces For All (1998), http://ui4all.ics.forth.gr/UI4ALL-98/proceedings.html
  74. 74.
    Streitz, N., Nixon, P.: The Disappearing Computer. ACM Communications, Spec. Issue on The Disappearing Computer 48(3), 33–35 (2005)Google Scholar
  75. 75.
    Sung, K.K., Poggio, T.: Example-Based Learning for View-Based Human Face Detection. IEEE Trans. Pattern Analysis and Machine Intelligence 20(1), 39–51 (1998)CrossRefGoogle Scholar
  76. 76.
    Tao, H., Huang, T.S.: Connected vibrations – a model analysis approach to non-rigid motion tracking. In: Proc. Conf. Computer Vision and Pattern Recognition, pp. 735–740 (1998)Google Scholar
  77. 77.
    Tian, Y., Kanade, T., Cohn, J.F.: Recognizing action units for facial expression analysis. IEEE Trans. Pattern Analysis & Machine Intelligence 23(2), 97–115 (2001)CrossRefGoogle Scholar
  78. 78.
    Tian, Y.L., Kanade, T., Cohn, J.F.: Facial Expression Analysis. In: Li, S.Z., Jain, A.K. (eds.) Handbook of Face Recognition. Springer, New York (2005)Google Scholar
  79. 79.
    Tscheligi, M.: Ambient Intelligence – The Next Generation of User Centeredness. ACM Interactions, Spec. Issue on Ambient Intelligence 12(4), 20–21 (2005)CrossRefGoogle Scholar
  80. 80.
    Valstar, M., Pantic, M., Patras, I.: Motion History for Facial Action Detection from Face Video. In: Proc. Conf. Systems, Man and Cybernetics, vol. 1, pp. 635–640 (2004)Google Scholar
  81. 81.
    Valstar, M., Patras, I., Pantic, M.: Facial Action Unit Detection using Probabilistic Actively Learned Support Vector Machines on Tracked Facial Point Data. In: Proc. Conf. Computer Vision and Pattern Recognition (2005)Google Scholar
  82. 82.
    Viola, P., Jones, M.: Robust real-time object detection. In: Proc. Int’l Conf. Computer Vision, Workshop on Statistical and Computation Theories of Vision (2001)Google Scholar
  83. 83.
    Vukadinovic, D., Pantic, M.: Fully automatic facial feature point detection using Gabor feature based boosted classifiers. In: Proc. Conf. Systems, Man and Cybernetics (2005)Google Scholar
  84. 84.
    Weiser, M.: The world is not a desktop. ACM Interactions 1(1), 7–8 (1994)CrossRefGoogle Scholar
  85. 85.
    Xiao, J., Baker, S., Matthews, I., Kanade, T.: Real-time Combined 2D+3D Active Appearance Models. In: Proc. Conf. Computer Vision and Pattern Recognition, vol. 2, pp. 535–542 (2004)Google Scholar
  86. 86.
    Yacoob, Y., Davis, L., Black, M., Gavrila, D., Horprasert, T., Morimoto, C.: Looking at People in Action. In: Cipolla, R., Pentland, A. (eds.) Computer Vision for Human-Machine Interaction, pp. 171–187. Cambridge University Press, Cambridge (1998)CrossRefGoogle Scholar
  87. 87.
    Yang, M.H., Kriegman, D.J., Ahuja, N.: Detecting faces in images: A survey. IEEE Trans. Pattern Analysis and Machine Intelligence 24(1), 34–58 (2002)CrossRefGoogle Scholar
  88. 88.
    Zhai, S., Bellotti, V.: Introduction to Sensing-Based Interaction. ACM Trans. Computer-Human Interaction, Spec. Issue on Sensing-Based Interaction 12(1), 1–2 (2005)CrossRefGoogle Scholar
  89. 89.
    Zhang, Y., Ji, Q.: Active and Dynamic Information Fusion for Facial Expression Understanding from Image Sequence. IEEE Trans. Pattern Analysis & Machine Intelligence 27(5), 699–714 (2005)CrossRefGoogle Scholar
  90. 90.
    Zhao, W., Chellappa, R., Rosenfeld, A., Phillips, P.J.: Face Recognition – A literature survey. ACM Computing Surveys 35(4), 399–458 (2003)CrossRefGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2006

Authors and Affiliations

  • Maja Pantic
    • 1
  1. 1.Computing DepartmentImperial CollegeLondonU.K.

Personalised recommendations