Dynamically Adaptive Tracking of Gestures and Facial Expressions

  • D. Metaxas
  • G. Tsechpenakis
  • Z. Li
  • Y. Huang
  • A. Kanaujia
Part of the Lecture Notes in Computer Science book series (LNCS, volume 3993)


We present a dynamic data-driven framework for tracking gestures and facial expressions from monocular sequences. Our system uses two cameras, one for the face and one for the body view for processing in different scales. Specifically, and for the gesture tracking module, we track the hands and the head, obtaining as output the blobs (ellipses) of the ROIs, and we detect the shoulder positions with straight lines. For the facial expressions, we first extract the 2D facial features, using a fusion between KLT tracker and a modified Active Shape Model, and then we obtain the 3D face mask with fitting a generic model to the extracted 2D features. The main advantages of our system are (i) the adaptivity, i.e., it is robust to external conditions, e.g., lighting, and independent from the examined individual, and (ii) its computational efficiency, providing us results off- and online with a rates higher than 20fps.


Facial Expression Facial Feature Skin Region Active Appearance Model Face Tracking 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


  1. 1.
    Birchfield, S.: Derivation of Kanade-Lucas-Tomasi Tracking Equation (May 1996), web-published at
  2. 2.
    Cootes, T.F., Taylor, C.J., Cooper, D.H., Graham, J.: Active Shape Models - their training and application. Computer Vision and Image Understanding 61(1), 389 (1995)CrossRefGoogle Scholar
  3. 3.
    Cootes, T.F., Edwards, G.J., Taylor, C.J.: Active Appearance Model. In: 5th Europen Conference on Computer Vision, Freiburg, Germany (1998)Google Scholar
  4. 4.
    Cootes, T.F., Kittipanya, P.: Comparing Variations on the Active Appearance Model Algorithm. In: Brittish Machine Vision Conderence, University of Cardiff (September 2002)Google Scholar
  5. 5.
    de Carlo, D., Metaxas, D.: Optical Flow Constraints on Deformable Models with Applications to Face Tracking. International Journal of Computer Vision 38(2), 99–127 (2000)CrossRefGoogle Scholar
  6. 6.
    Devore, J.L.: Probability and Statistics for Engineering and the Sciences, Pacific Grove, Calif. Brooks/Cole Pub. Co (2004)Google Scholar
  7. 7.
    Goldenstein, S., Vogler, C., Metaxas, D.: Statistical Cue Integration in Deformable Models. Pattern Analysis and Machine Intelligence 25(7), 801–813 (2003)CrossRefGoogle Scholar
  8. 8.
    Gavrila, D.M.: The Visual Analysis of Human Movement: A Survey. Computer Vision and Image Understanding 73(1), 82–98 (1999)MATHCrossRefGoogle Scholar
  9. 9.
    Kalman, R.E.: A New Approach to Linear Filtering and Prediction Problems. ASME Journal of Basic Engineering, 35–45 (March 1960)Google Scholar
  10. 10.
    Li, S.Z., Zou, X.L., Hu, Y.X., Zhang, Z.Q., Yan, S.C., Peng, X.H., Huang, L., Zhang, H.J.: Real-Time Multi-View Face Detection, Tracking, Pose Estimation, Alignment, and Recognition. In: IEEE Conference on Computer Vision and Pattern Recognition, Demo Summary, Hawaii (December 2001)Google Scholar
  11. 11.
    Lu, S., Tsechpenakis, G., Metaxas, D., Jensen, M.L., Kruse, J.: Blob Analysis of the Head and Hands: A Method for Deception Detection and Emotional State Identification. In: Hawaii International Conference on System Sciences, Big Island, Hawaii (January 2005)Google Scholar
  12. 12.
    Lucas, B.D., Kanade, T.: An Iterative Image Registration Technique with an Application to Stereo Vision. In: International Joint Conference on Artificial Intelligence, pp. 674–679 (1981)Google Scholar
  13. 13.
    Shi, J., Tomasi, C.: Good Features to Track. In: IEEE Conference on Computer Vision and Pattern Recognition, Seattle, WA (June 1994)Google Scholar
  14. 14.
    Tao, H., Huang, T.: Visual Estimation and Compression of Facial Motion Parameters: Elements of a 3D Model-based Video Coding System. International Journal of Computer Vision 50(2), 111–125 (2002)MATHCrossRefGoogle Scholar
  15. 15.
    Tomasi, C., Kanade, T.: Detection and Tracking of Point Features. Carnegie Mellon University Technical Report CMU-CS-91-132 (April 1991)Google Scholar
  16. 16.
    Viola, P.A., Jones, M.J.: Robust real-time face detection. International Journal of Computer Vision 57(2), 137–154 (2004)CrossRefGoogle Scholar
  17. 17.
    Zhang, H., Fritts, J.E., Goldman, S.A.: A Fast Texture Feature Extraction Method for Region-based Image Segmentation. In: 16th Annual Symposium on Image and Video Communication and Processing, January 2005. SPIE, vol. 5685 (2005)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2006

Authors and Affiliations

  • D. Metaxas
    • 1
  • G. Tsechpenakis
    • 1
  • Z. Li
    • 1
  • Y. Huang
    • 1
  • A. Kanaujia
    • 1
  1. 1.Center for Computational Biomedicine, Imaging and Modeling (CBIM), Computer Science DepartmentRutgers UniversityPiscatawayUSA

Personalised recommendations