Occluded Facial Expression Tracking

  • Hugo Mercier
  • Julien Peyras
  • Patrice Dalle
Part of the Lecture Notes in Computer Science book series (LNCS, volume 4522)


The work presented here takes place in the field of computer aided analysis of facial expressions displayed in sign language videos. We use Active Appearance Models to model a face and its variations of shape and texture caused by expressions. The inverse compositional algorithm is used to accurately fit an AAM to the face seen on each video frame. In the context of sign language communication, the signer’s face is frequently occluded, mainly by hands. A facial expression tracker has then to be robust to occlusions. We propose to rely on a robust variant of the AAM fitting algorithm to explicitly model the noise introduced by occlusions. Our main contribution is the automatic detection of hand occlusions. The idea is to model the behavior of the fitting algorithm on unoccluded faces, by means of residual image statistics, and to detect occlusions as being what is not explained by this model. We use residual parameters with respect to the fitting iteration i.e., the AAM distance to the solution, which greatly improves occlusion detection compared to the use of fixed parameters. We also propose a robust tracking strategy used when occlusions are too important on a video frame, to ensure a good initialization for the next frame.


Active Appearance Model occlusion facial expression tracking inverse compositional 


  1. 1.
    Cootes, T.F., Edwards, G.J., Taylor, C.J.: Active appearance models. IEEE Transactions on Pattern Analysis and Machine Intelligence 23, 681–685 (2001), CrossRefGoogle Scholar
  2. 2.
    Matthews, I., Baker, S.: Active appearance models revisited. International Journal of Computer Vision 60(2), 135 (2004)CrossRefGoogle Scholar
  3. 3.
    Baker, S., Gross, R., Matthews, I., Ishikawa, T.: Lucas-Kanade 20 years on: A unifying framework: Part 2. Technical Report CMU-RI-TR-03-01, Robotics Institute, Carnegie Mellon University, Pittsburgh, PA (February 2003)Google Scholar
  4. 4.
    Gross, R., Matthews, I., Baker, S.: Active appearance models with occlusion. Image and Vision Computing 24(6), 593–604 (2006)CrossRefGoogle Scholar
  5. 5.
    Theobald, B.J., Matthews, I., Baker, S.: Evaluating error functions for robust active appearance models. In: Proceedings of the International Conference on Automatic Face and Gesture Recognition, April 2006, pp. 149–154 (2006)Google Scholar
  6. 6.
    Baker, S., Gross, R., Matthews, I.: Lucas-Kanade 20 years on: A unifying framework: Part 3. Technical Report CMU-RI-TR-03-35, Robotics Institute, Carnegie Mellon University, Pittsburgh, PA (November 2003)Google Scholar

Copyright information

© Springer Berlin Heidelberg 2007

Authors and Affiliations

  • Hugo Mercier
    • 1
  • Julien Peyras
    • 2
  • Patrice Dalle
    • 1
  1. 1.Institut de Recherche en Informatique de Toulouse, 118, route de Narbonne, F-31062 Toulouse Cedex 9 
  2. 2.Dipartimento di Scienze dell’Informazione, via Comelico 39/41, I-20135 Milano 

Personalised recommendations