Visual Interaction in Natural Human-Machine Dialogue

  • Joseph Machrouh
  • Franck Panaget
Part of the Lecture Notes in Computer Science book series (LNCS, volume 4021)


In this article, we describe a visual component able to detect and track a human face in video streaming. This component is integrated into an embodied conversational agent. Depending on the presence or absence of a user in front of the camera and the orientation of his head, the system begins, continues, resumes or closes the interaction. Several constraints have been taken into account: a simple webcam, a low error rate and a minimum computing time that permits the whole system to run on a simple pc.


Skin Colour Video Streaming Face Detection Face Tracking Conversational Agent 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. [Ahmad, 1995]
    Ahmad, S.: A usable real-time 3d hand tracker. In: Proceeding of the 28th Asilomar Conference on Signals, Systems and Computers, Pacific Grove, CA, USA, pp. 1257–1261 (1995)Google Scholar
  2. [Bradski, 1998]
    Bradski, G.R.: Computer vision face tracking for use in a perceptual user interface. In: Proceeding of IEEE Workshop on Applications of Computer Vision, Princeton, NJ, USA, pp. 214–219 (1998)Google Scholar
  3. [Cai and Goshtasby, 1999]
    Cai, J., Goshtasby, A.: Detecting human faces in color images. Image Vision Computing 18, 63–75 (1999)CrossRefGoogle Scholar
  4. [Cassell et al., 1999]
    Cassell, J., Bickmore, T., Billinghurst, M., Campbell, L., Chang, K., Vilhjalmsson, H., Yan, H.: Embodiment in conversational interfaces: Rea. In: CHI 1999: Proceedings of the SIGGHI Conference on Human Factors in Computing Systems, Pittsburgh, Pennsylvania, USA, pp. 520–527 (1999)Google Scholar
  5. [Chai and Ngan, 1999]
    Chai, D., Ngan, K.: Face segmentation using skin-color map in videophone applications. IEEE Transactions on Circuits and Systems for Video Technology 9(4), 551–564 (1999)CrossRefGoogle Scholar
  6. [Chen et al., 2003]
    Chen, M., Chi, M., Hsu, C., Chen, J.: Roi video coding based on h.263+ with robust skin-color detection technique. IEEE Transactions on Consumer Electronics 49(3), 724–730 (2003)CrossRefGoogle Scholar
  7. [Crowley and Bedrune, 1994]
    Crowley, J.L., Bedrune, J.M.: Integration and control of reactive visual process. In: Eklundh, J.-O. (ed.) ECCV 1994. LNCS, vol. 800. Springer, Heidelberg (1994)Google Scholar
  8. [Feng and Yuen, 2001]
    Feng, G.C., Yuen, P.: Multi-cues eye detection on gray intensity image. Pattern Recogntion 34(5), 1033–1046 (2001)MATHCrossRefGoogle Scholar
  9. [Foresti et al., 2003]
    Foresti, G., Micheloni, C., Snidaro, L., Marchiol, C.: Face detection for visual surveillance. In: Proceedings of the 12th International Conference on Image Analysis and Processing (ICIAP 2003), Mantova, Italy (2003)Google Scholar
  10. [Garcia and Delakis, 2004]
    Garcia, C., Delakis, M.: Convolution face finder: A neural architecture for fast and robust face detection. IEEE Transaction on Pattern Analysis and Machine Intelligence 26(11), 1408–1423 (2004)CrossRefGoogle Scholar
  11. [Gourier et al., 2004]
    Gourier, N., Hall, D., Crowley, J.L.: Estimating face orientation from robust detection of salient facial features. In: Proceeding of Pointing 2004, ICPR, International Workshop on Visual Observation of Deictic Gestures, Cambridge, UK (2004)Google Scholar
  12. [He et al., 2003]
    He, X., Liu, Z., Zhou, J.: Real-time human face detection in color image. In: Proceedings of the Second lnternational Conference on Machine Learning and Cybernetics, Xi’an, China (2003)Google Scholar
  13. [Hsu et al., 2002]
    Hsu, R.L., Abdel-Mottaleb, M., Jain, A.K.: Face detection in color images. IEEE Transaction on Pattern Analysis and Machine Intelligence 24(5), 696–706 (2002)CrossRefGoogle Scholar
  14. [Kovac et al., 2003]
    Kovac, J., Peer, P., Solina, F.: Human skin colour clustering for face detection. In: Zajc, B. (ed.) EUROCON 2003 - International Conference on Computer as a Tool, Ljubljana, Slovenia (2003)Google Scholar
  15. [Kumar et al., 2002]
    Kumar, R.T., Raja, S.K., Ramakrishnan, A.G.: Eye detection using color cues and projection functions. In: Proceeding of International Conference on Image Processing, Rochester, NY, USA, vol. 3, pp. 337–340 (2002)Google Scholar
  16. [Machrouh et al., 2006]
    Machrouh, J., Panaget, F., Bretier, P., Garcia, C.: Face and eyes detection to improve natural human-computer dialogue. In: Proceeding of the second IEEE-EURASIP International Symposium on Control, Communications, and Signal Processing, Marrakech, Morocco (2006)Google Scholar
  17. [Marcel and Bernier, 1999]
    Marcel, S., Bernier, O.: Hand posture recognition in a body-face centred space. In: Braffort, A., Gibet, S., Teil, D., Gherbi, R., Richardson, J. (eds.) GW 1999. LNCS (LNAI), vol. 1739, pp. 97–100. Springer, Heidelberg (2000)CrossRefGoogle Scholar
  18. [Menezes et al., 2003]
    Menezes, P., Brethes, L., Lerasle, F., Dans, P., Dias, J.: Visual tracking of silhouettes for human-robot interaction. In: Proceeding of International Conference on Advanced Robotics (ICAR 2001), Coimbra, Portugal, vol. 2, pp. 971–976 (2003)Google Scholar
  19. [Pelé et al., 2003]
    Pelé, D., Breton, G., Panaget, F., Loyson, S.: Let’s find a restaurant with nestor: A 3d embodied conversational agent on the web. In: Proceeding of AAMAS Workshop on embodied conversational characters as individual, Australia (2003)Google Scholar
  20. [Sadek et al., 1997]
    Sadek, D., Bretier, P., Panaget, F.: Artimis: Natural dialogue meets rational agency. In: Proceeding of the 15th International Joint Conference on Artificial Intelligence (IJCAI 1997), Nagoya, Japan, pp. 1030–1035 (1997)Google Scholar
  21. [Tomaz et al., 2003]
    Tomaz, F., Candeias, T., Shahbazkia, H.: Improved automatic skin detection in color images. In: Sun, C., Talbot, H., Ourselin, S., Adriaansen, T. (eds.) Proceeding of VIIth Digital Computing: Techniques and Applications, Sydney, Australia, pp. 419–427 (2003)Google Scholar
  22. [Turk, 2004]
    Turk, M.: Computer vision in the interface. Communications of the ACM 47(1), 60–67 (2004)CrossRefGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2006

Authors and Affiliations

  • Joseph Machrouh
    • 1
  • Franck Panaget
    • 1
  1. 1.France Telecom R&D, TECH/EASY labsLannionFrance

Personalised recommendations