Using 3D Models for Real-Time Facial Feature Tracking, Pose Estimation, and Expression Monitoring

  • Angela Caunce
  • Tim Cootes
Part of the Lecture Notes in Computer Science book series (LNCS, volume 7585)


We present an application which uses 3D statistical shape models to track a subject in real time using a single fixed camera. The system can handle large pose variation; variable illumination; occlusion; glasses. Since the models are 3D, the application can report pose information which may be vital in a safety context such as driving attentiveness. Two models are used in tandem, one for identity and one for facial actions, enabling the system to also estimate the user’s behavioural state at a basic level. The system works directly on the captured images, with no pre-processing, and tracks the facial features using simple template matching and boundary detection. The parameters of the identity model adapt over time to the model subspace occupied by the subject, and this allows the second model to describe simple actions such as eye, brow, and mouth movement. The parameters of the actions model are then used to identify smiling, frowning, talking, and blinking using simple linear discriminants.


Face Gesture Recognition Facial Action Variable Illumination Active Shape Model Model Subspace 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


  1. 1.
    Belhumeur, P.N., Jacobs, D.W., Kriegman, D.J., Kumar, N.: Localizing Parts of Faces Using a Consensus of Exemplars. In: Computer Vision and Pattern Recognition, pp. 545–552 (2011)Google Scholar
  2. 2.
    Cristinacce, D., Cootes, T.F.: Automatic feature localisation with constrained local models. Pattern Recognition 41, 3054–3067 (2007)CrossRefGoogle Scholar
  3. 3.
    Valstar, M., Martinez, B., Binefa, X., Pantic, M.: Facial Point Detection Using Boosted Regression and Graph Models. In: Computer Vision and Pattern Recognition, pp. 2729–2736 (2010)Google Scholar
  4. 4.
    Vogler, C., Li, Z., Kanaujia, A., Goldenstein, S., Metaxas, D.: The Best of Both Worlds: Combining 3D Deformable Models with Active Shape Models. In: International Conference on Computer Vision, pp. 1–7 (2007)Google Scholar
  5. 5.
    Xiao, J., Baker, S., Matthews, I., Kanade, T.: Real-Time Combined 2D+3D Active Appearance Models. In: Conference on Computer Vision and Pattern Recognition, vol. 2, pp. 535–542 (2004)Google Scholar
  6. 6.
    Romdhani, S., Ho, J., Vetter, T., Kriegman, D.J.: Face Recognition Using 3-D Models: Pose and Illumination. Proceedings of the IEEE 94, 1977–1999 (2006)CrossRefGoogle Scholar
  7. 7.
    Caunce, A., Cristinacce, D., Taylor, C., Cootes, T.: Locating Facial Features and Pose Estimation Using a 3D Shape Model. In: Bebis, G., Boyle, R., Parvin, B., Koracin, D., Kuno, Y., Wang, J., Wang, J.-X., Wang, J., Pajarola, R., Lindstrom, P., Hinkenjann, A., Encarnação, M.L., Silva, C.T., Coming, D. (eds.) ISVC 2009, Part I. LNCS, vol. 5875, pp. 750–761. Springer, Heidelberg (2009)CrossRefGoogle Scholar
  8. 8.
    Caunce, A., Taylor, C., Cootes, T.: Adding Facial Actions into 3D Model Search to Analyse Behaviour in an Unconstrained Environment. In: Bebis, G., Boyle, R., Parvin, B., Koracin, D., Chung, R., Hammoud, R., Hussain, M., Kar-Han, T., Crawfis, R., Thalmann, D., Kao, D., Avila, L. (eds.) ISVC 2010, Part I. LNCS, vol. 6453, pp. 132–142. Springer, Heidelberg (2010)CrossRefGoogle Scholar
  9. 9.
    Amberg, B., Knothe, R., Vetter, T.: Expression Invariant 3D Face Recognition with a Morphable Model. In: International Conference on Automatic Face Gesture Recognition, Amsterdam, pp. 1–6 (2008)Google Scholar
  10. 10.
    Basso, C., Vetter, T.: Registration of Expressions Data Using a 3D Morphable Model. Journal of Multimedia 1, 37–45 (2006)CrossRefGoogle Scholar
  11. 11.
    Cootes, T.F., Cooper, D.H., Taylor, C.J., Graham, J.: Active Shape Models - Their Training and Application. Computer Vision and Image Understanding 61, 38–59 (1995)CrossRefGoogle Scholar
  12. 12.
    Viola, P., Jones, M.J.: Robust Real-Time Face Detection. International Journal of Computer Vision 57, 137–154 (2004)CrossRefGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2012

Authors and Affiliations

  • Angela Caunce
    • 1
  • Tim Cootes
    • 1
  1. 1.Imaging Science and Biomedical EngineeringThe University of ManchesterUK

Personalised recommendations