Skip to main content
SpringerLink
Log in

Visual tracking of hands, faces and facial features of multiple persons

  • Original Paper
  • Published:
Machine Vision and Applications Aims and scope Submit manuscript

Abstract

This paper presents an integrated approach for tracking hands, faces and specific facial features (eyes, nose, and mouth) in image sequences. For hand and face tracking, we employ a state-of-the-art blob tracker which is specifically trained to track skin-colored regions. In this paper we extend the skin color tracker by proposing an incremental probabilistic classifier, which can be used to maintain and continuously update the belief about the class of each tracked blob, which can be left-hand, right hand or face as well as to associate hand blobs with their corresponding faces. An additional contribution of this paper is related to the employment of a novel method for the detection and tracking of specific facial features within each detected facial blob which consists of an appearance-based detector and a feature-based tracker. The proposed approach is intended to provide input for the analysis of hand gestures and facial expressions that humans utilize while engaged in various conversational states with robots that operate autonomously in public places. It has been integrated into a system which runs in real time on a conventional personal computer which is located on a mobile robot. Experimental results confirm its effectiveness for the specific task at hand.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. Yang M.H., Kriegman D., Ahuja D.: Detecting faces in images: a survey. IEEE Trans. Pattern Anal. Mach. Intell. 24(1), 34–58 (2002)

    Article  Google Scholar 

  2. Zabulis X., Baltzakis H., Argyros A.: Vision-based hand gesture recognition for human-computer interaction. In: Stefanides, C. (eds) The Universal Access Handbook, Human Factors and Ergonomics., pp. 34.1–34.30. Lawrence Erlbaum Associates, Inc. (LEA), New Jersey (2009)

    Google Scholar 

  3. Jones M.J., Rehg J.M.: Statistical color models with application to skin detection. Int. J. Comput. Vis. 46(1), 81–96 (2002)

    Article  MATH  Google Scholar 

  4. Argyros, A.A., Lourakis, M.I.A.: Real-time tracking of multiple skin-colored objects with a possibly moving camera. In: Proceedings of the European Conference on Computer Vision, Prague, Czech Republic, May 2004, pp. 368–379

  5. Nickel, K., Seemann, E., Stiefelhagen, R.: 3D-tracking of head and hands for pointing gesture recognition in a human-robot interaction scenario. In: Proceedings of the IEEE International Conference on Automatic Face and Gesture Recognition, Seoul, Korea, May 2004, pp. 565–570

  6. Baltzakis, H., Argyros, A., Lourakis, M., Trahanias, P.: Tracking of human hands and faces through probabilistic fusion of multiple visual cues. In: Proceedings of the International Conference on Computer Vision Systems (ICVS), Santorini, Greece, May 2008, pp. 33–42

  7. Sigalas, M., Baltzakis, H., Trahanias, P.: Visual tracking of independently moving body and arms. In: Proceedings of the IEEE International Conference on Intelligent Robots and Systems (IROS ’09), St. Louis, MO, USA, October 2009

  8. Deutscher, J., Blake, A., Reid, I.: Articulated body motion capture by annealed particle filtering. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, vol. 2, pp. 126–133 (2000)

  9. Stenger B., Thayananthan A., Torr P.H.S., Cipolla R.: Model-based hand tracking using a hierarchical bayesian filter. IEEE Trans. Pattern Anal. Mach. Intell. 28(9), 1372–1384 (2006)

    Article  Google Scholar 

  10. Baltzakis, H., Argyros, A.: Propagation of pixel hypotheses for multiple objects tracking. In: Proceedings of the International Symposium on Visual Computating (ISVC), Las Vegas, Nevada, USA, November 2009

  11. Zhang, X., Xu, Y., Du, L.: Locating facial features with color information. In: Proceedings of the Fourth International Conference on Signal Processing (ICSP), vol. 2, pp. 889–892 (1998)

  12. Hsu R.-L., Abdel-Mottaleb M., Jain A.K.: Face detection in color images. IEEE Trans. Pattern Anal. Mach. Intell. 24(5), 696–706 (2002)

    Article  Google Scholar 

  13. Sobottka K., Pitas I.: A novel method for automatic face segmentation, facial feature extraction and tracking. Signal Process. Image Commun. 12(3), 263–281 (1998)

    Article  Google Scholar 

  14. Nguyen, T., Nguyen, V., Kim, H.: Robust feature extraction for facial image quality assessment. In: Chung, Y., Yung, M. (eds.) Information Security Applications. Lecture Notes in Computer Science, vol. 6513, pp. 292–306. Springer, Berlin (2011)

  15. Yuille A.L., Hallinan P.W., Cohen D.S.: Feature extraction from faces using deformable templates. Int. J. Comput. Vis. 8(2), 99–111 (1992)

    Article  Google Scholar 

  16. Herpers, R., Sommer, G.: An attentive processing strategy for the analysis of facial features. In: Face recognition: From Theory to Applications, pp. 457–468 (1998)

  17. Pardas, M., Losada, M.: Facial parameter extraction system based on active contours. In Proceedings of the International Conference on Image Processing (ICIP’01), Thessaloniki, Greece, October 2001, vol. 1, pp. 1058–1061

  18. Kawaguchi T., Rizon M., Hidaka D.: Detection of eyes from human faces by hough transform and separability filter. Electron. Commun. Japan 88(5), 29–39 (2005)

    Article  Google Scholar 

  19. Cootes T., Edwards G., Taylor C.: Active appearance models. IEEE Trans. Pattern Anal. Mach. Intell. 23(6), 681–685 (2001)

    Article  Google Scholar 

  20. Cristinacce, D., Cootes, T.F., Scott, I.: A multi-stage approach to facial feature detection. In: 15th British Machine Vision Conference, pp. 231–240 (2004)

  21. Zuo F., de With P.H.: Facial feature extraction by a cascade of model-based algorithms. Signal Process Image Commun. 23(3), 194–211 (2008)

    Article  Google Scholar 

  22. Zhou Y., Li Y., Wu Z., Ge M.: Robust facial feature point extraction in color images. Eng. Appl. Artif. Intell. 24, 195–200 (2011)

    Article  Google Scholar 

  23. Kim, H.-C., Kim, D., Bang, S.-Y.: A pca mixture model with an efficient model selection method. In: Proceedings of the International Joint Conference on Neural Networks, 2001 (IJCNN ’01), vol. 1, pp. 430–435 (2001)

  24. Phimoltares, S., Lursinsap, C., Chamnongthai, K.: Locating essential facial features using neural visual model. In: Proceedings of the International Conference on Machine Learning and Cybernetics, 2002, vol. 4, pp. 1914–1919 (2002)

  25. Wilson P., Fernandez J.: Facial feature detection using haar classifiers. J. Comput. Sci. Coll. 21(4), 127–133 (2006)

    Google Scholar 

  26. Viola P., Jones M.: Robust real-time face detection. Int. J. Comput. Vis. 57(2), 137–154 (2004)

    Article  Google Scholar 

  27. Pateraki, M., Baltzakis, H., Kondaxakis, P., Trahanias, P.: Tracking of facial features to support human-robot interaction. In: Proceedings of the IEEE International Conference on Robotics and Automation (ICRA ’09), pp. 3755–3760 (2009)

  28. Stauffer, C., Grimson, W.E.L.: Adaptive background mixture models for real-time tracking. In: Proceedings of the IEEE Computer Vision and Pattern Recognition (CVPR),Ft. Collins, USA, June 1999, pp. 2246–2252

  29. Tian, Y.: Evaluation of face resolution for expression analysis. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshop (CVPRW’04), pp. 82–89. IEEE Computer Society, New York (2004)

  30. Sohail A.S.Md, Bhattacharya P.: Detection of facial feature points using anthropometric face model. In: Damiani, E., Ytongnon, K., Schelkens, P., Dipanda, A., Legrand, L., Chbeir, R. (eds) Signal Processing for Image Enhancement and Multimedia Processing. Multimedia Systems and Applications, vol. 31, pp. 189–200. Springer US, USA (2008)

    Chapter  Google Scholar 

  31. Kanade, T., Cohn, J., Tian, Y.: Comprehensive database for facial expression analysis. In Proceedings of the IEEE 4th International Conference on Automatic Face and Gesture Recognition (FG ’00), Grenoble, France, pp. 46–53 (2000)

  32. Gunes, H., Piccardi, M.: A bimodal face and body gesture database for automatic analysis of human nonverbal affective behavior. In: Proceedings of ICPR 2006 the 18th International Conference on Pattern Recognition, Hong Kong, China, August 2006

  33. Jesorsky, O., Kirchberg, K.J., Frischholz, R.W.: Robust face detection using the hausdorff distance. In: Proceedings of the 3rd International Conference on Audio- and Video-based Biometric Person Authentication, Halmstad, Sweden, June 2001. Lecture Notes in Computer Science, vol. 2091, pp. 90–95. Springer, Berlin

Download references

Author information

Authors and Affiliations

  1. Institute of Computer Science, Foundation for Research and Technology-Hellas (FORTH), P.O. Box 1385, 711 10, Heraklion, Crete, Greece

    Haris Baltzakis, Maria Pateraki & Panos Trahanias

  2. Department of Computer Science, University of Crete, 714 09, Heraklion, Crete, Greece

    Panos Trahanias

Authors
  1. Haris Baltzakis
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Maria Pateraki
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Panos Trahanias
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Haris Baltzakis.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Baltzakis, H., Pateraki, M. & Trahanias, P. Visual tracking of hands, faces and facial features of multiple persons. Machine Vision and Applications 23, 1141–1157 (2012). https://doi.org/10.1007/s00138-012-0409-5

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00138-012-0409-5

Keywords

Search

Navigation