Machine Vision and Applications

, Volume 23, Issue 6, pp 1141–1157 | Cite as

Visual tracking of hands, faces and facial features of multiple persons

  • Haris BaltzakisEmail author
  • Maria Pateraki
  • Panos Trahanias
Original Paper


This paper presents an integrated approach for tracking hands, faces and specific facial features (eyes, nose, and mouth) in image sequences. For hand and face tracking, we employ a state-of-the-art blob tracker which is specifically trained to track skin-colored regions. In this paper we extend the skin color tracker by proposing an incremental probabilistic classifier, which can be used to maintain and continuously update the belief about the class of each tracked blob, which can be left-hand, right hand or face as well as to associate hand blobs with their corresponding faces. An additional contribution of this paper is related to the employment of a novel method for the detection and tracking of specific facial features within each detected facial blob which consists of an appearance-based detector and a feature-based tracker. The proposed approach is intended to provide input for the analysis of hand gestures and facial expressions that humans utilize while engaged in various conversational states with robots that operate autonomously in public places. It has been integrated into a system which runs in real time on a conventional personal computer which is located on a mobile robot. Experimental results confirm its effectiveness for the specific task at hand.


Blob tracking Hand and face tracking Facial features Incremental classification Template matching 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Yang M.H., Kriegman D., Ahuja D.: Detecting faces in images: a survey. IEEE Trans. Pattern Anal. Mach. Intell. 24(1), 34–58 (2002)CrossRefGoogle Scholar
  2. 2.
    Zabulis X., Baltzakis H., Argyros A.: Vision-based hand gesture recognition for human-computer interaction. In: Stefanides, C. (eds) The Universal Access Handbook, Human Factors and Ergonomics., pp. 34.1–34.30. Lawrence Erlbaum Associates, Inc. (LEA), New Jersey (2009)Google Scholar
  3. 3.
    Jones M.J., Rehg J.M.: Statistical color models with application to skin detection. Int. J. Comput. Vis. 46(1), 81–96 (2002)zbMATHCrossRefGoogle Scholar
  4. 4.
    Argyros, A.A., Lourakis, M.I.A.: Real-time tracking of multiple skin-colored objects with a possibly moving camera. In: Proceedings of the European Conference on Computer Vision, Prague, Czech Republic, May 2004, pp. 368–379Google Scholar
  5. 5.
    Nickel, K., Seemann, E., Stiefelhagen, R.: 3D-tracking of head and hands for pointing gesture recognition in a human-robot interaction scenario. In: Proceedings of the IEEE International Conference on Automatic Face and Gesture Recognition, Seoul, Korea, May 2004, pp. 565–570Google Scholar
  6. 6.
    Baltzakis, H., Argyros, A., Lourakis, M., Trahanias, P.: Tracking of human hands and faces through probabilistic fusion of multiple visual cues. In: Proceedings of the International Conference on Computer Vision Systems (ICVS), Santorini, Greece, May 2008, pp. 33–42Google Scholar
  7. 7.
    Sigalas, M., Baltzakis, H., Trahanias, P.: Visual tracking of independently moving body and arms. In: Proceedings of the IEEE International Conference on Intelligent Robots and Systems (IROS ’09), St. Louis, MO, USA, October 2009Google Scholar
  8. 8.
    Deutscher, J., Blake, A., Reid, I.: Articulated body motion capture by annealed particle filtering. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, vol. 2, pp. 126–133 (2000)Google Scholar
  9. 9.
    Stenger B., Thayananthan A., Torr P.H.S., Cipolla R.: Model-based hand tracking using a hierarchical bayesian filter. IEEE Trans. Pattern Anal. Mach. Intell. 28(9), 1372–1384 (2006)CrossRefGoogle Scholar
  10. 10.
    Baltzakis, H., Argyros, A.: Propagation of pixel hypotheses for multiple objects tracking. In: Proceedings of the International Symposium on Visual Computating (ISVC), Las Vegas, Nevada, USA, November 2009Google Scholar
  11. 11.
    Zhang, X., Xu, Y., Du, L.: Locating facial features with color information. In: Proceedings of the Fourth International Conference on Signal Processing (ICSP), vol. 2, pp. 889–892 (1998)Google Scholar
  12. 12.
    Hsu R.-L., Abdel-Mottaleb M., Jain A.K.: Face detection in color images. IEEE Trans. Pattern Anal. Mach. Intell. 24(5), 696–706 (2002)CrossRefGoogle Scholar
  13. 13.
    Sobottka K., Pitas I.: A novel method for automatic face segmentation, facial feature extraction and tracking. Signal Process. Image Commun. 12(3), 263–281 (1998)CrossRefGoogle Scholar
  14. 14.
    Nguyen, T., Nguyen, V., Kim, H.: Robust feature extraction for facial image quality assessment. In: Chung, Y., Yung, M. (eds.) Information Security Applications. Lecture Notes in Computer Science, vol. 6513, pp. 292–306. Springer, Berlin (2011)Google Scholar
  15. 15.
    Yuille A.L., Hallinan P.W., Cohen D.S.: Feature extraction from faces using deformable templates. Int. J. Comput. Vis. 8(2), 99–111 (1992)CrossRefGoogle Scholar
  16. 16.
    Herpers, R., Sommer, G.: An attentive processing strategy for the analysis of facial features. In: Face recognition: From Theory to Applications, pp. 457–468 (1998)Google Scholar
  17. 17.
    Pardas, M., Losada, M.: Facial parameter extraction system based on active contours. In Proceedings of the International Conference on Image Processing (ICIP’01), Thessaloniki, Greece, October 2001, vol. 1, pp. 1058–1061Google Scholar
  18. 18.
    Kawaguchi T., Rizon M., Hidaka D.: Detection of eyes from human faces by hough transform and separability filter. Electron. Commun. Japan 88(5), 29–39 (2005)CrossRefGoogle Scholar
  19. 19.
    Cootes T., Edwards G., Taylor C.: Active appearance models. IEEE Trans. Pattern Anal. Mach. Intell. 23(6), 681–685 (2001)CrossRefGoogle Scholar
  20. 20.
    Cristinacce, D., Cootes, T.F., Scott, I.: A multi-stage approach to facial feature detection. In: 15th British Machine Vision Conference, pp. 231–240 (2004)Google Scholar
  21. 21.
    Zuo F., de With P.H.: Facial feature extraction by a cascade of model-based algorithms. Signal Process Image Commun. 23(3), 194–211 (2008)CrossRefGoogle Scholar
  22. 22.
    Zhou Y., Li Y., Wu Z., Ge M.: Robust facial feature point extraction in color images. Eng. Appl. Artif. Intell. 24, 195–200 (2011)CrossRefGoogle Scholar
  23. 23.
    Kim, H.-C., Kim, D., Bang, S.-Y.: A pca mixture model with an efficient model selection method. In: Proceedings of the International Joint Conference on Neural Networks, 2001 (IJCNN ’01), vol. 1, pp. 430–435 (2001)Google Scholar
  24. 24.
    Phimoltares, S., Lursinsap, C., Chamnongthai, K.: Locating essential facial features using neural visual model. In: Proceedings of the International Conference on Machine Learning and Cybernetics, 2002, vol. 4, pp. 1914–1919 (2002)Google Scholar
  25. 25.
    Wilson P., Fernandez J.: Facial feature detection using haar classifiers. J. Comput. Sci. Coll. 21(4), 127–133 (2006)Google Scholar
  26. 26.
    Viola P., Jones M.: Robust real-time face detection. Int. J. Comput. Vis. 57(2), 137–154 (2004)CrossRefGoogle Scholar
  27. 27.
    Pateraki, M., Baltzakis, H., Kondaxakis, P., Trahanias, P.: Tracking of facial features to support human-robot interaction. In: Proceedings of the IEEE International Conference on Robotics and Automation (ICRA ’09), pp. 3755–3760 (2009)Google Scholar
  28. 28.
    Stauffer, C., Grimson, W.E.L.: Adaptive background mixture models for real-time tracking. In: Proceedings of the IEEE Computer Vision and Pattern Recognition (CVPR),Ft. Collins, USA, June 1999, pp. 2246–2252Google Scholar
  29. 29.
    Tian, Y.: Evaluation of face resolution for expression analysis. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshop (CVPRW’04), pp. 82–89. IEEE Computer Society, New York (2004)Google Scholar
  30. 30.
    Sohail A.S.Md, Bhattacharya P.: Detection of facial feature points using anthropometric face model. In: Damiani, E., Ytongnon, K., Schelkens, P., Dipanda, A., Legrand, L., Chbeir, R. (eds) Signal Processing for Image Enhancement and Multimedia Processing. Multimedia Systems and Applications, vol. 31, pp. 189–200. Springer US, USA (2008)CrossRefGoogle Scholar
  31. 31.
    Kanade, T., Cohn, J., Tian, Y.: Comprehensive database for facial expression analysis. In Proceedings of the IEEE 4th International Conference on Automatic Face and Gesture Recognition (FG ’00), Grenoble, France, pp. 46–53 (2000)Google Scholar
  32. 32.
    Gunes, H., Piccardi, M.: A bimodal face and body gesture database for automatic analysis of human nonverbal affective behavior. In: Proceedings of ICPR 2006 the 18th International Conference on Pattern Recognition, Hong Kong, China, August 2006Google Scholar
  33. 33.
    Jesorsky, O., Kirchberg, K.J., Frischholz, R.W.: Robust face detection using the hausdorff distance. In: Proceedings of the 3rd International Conference on Audio- and Video-based Biometric Person Authentication, Halmstad, Sweden, June 2001. Lecture Notes in Computer Science, vol. 2091, pp. 90–95. Springer, BerlinGoogle Scholar

Copyright information

© Springer-Verlag 2012

Authors and Affiliations

  • Haris Baltzakis
    • 1
    Email author
  • Maria Pateraki
    • 1
  • Panos Trahanias
    • 1
    • 2
  1. 1.Institute of Computer ScienceFoundation for Research and Technology-Hellas (FORTH)HeraklionGreece
  2. 2.Department of Computer ScienceUniversity of CreteHeraklionGreece

Personalised recommendations