3D Model-Based Face Recognition in Video

  • Unsang Park
  • Anil K. Jain
Part of the Lecture Notes in Computer Science book series (LNCS, volume 4642)


Face recognition in video has gained wide attention due to its role in designing surveillance systems. One of the main advantages of video over still frames is that evidence accumulation over multiple frames can provide better face recognition performance. However, surveillance videos are generally of low resolution containing faces mostly in non-frontal poses. Consequently, face recognition in video poses serious challenges to state-of-the-art face recognition systems. Use of 3D face models has been suggested as a way to compensate for low resolution, poor contrast and non-frontal pose. We propose to overcome the pose problem by automatically (i) reconstructing a 3D face model from multiple non-frontal frames in a video, (ii) generating a frontal view from the derived 3D model, and (iii) using a commercial 2D face recognition engine to recognize the synthesized frontal view. A factorization-based structure from motion algorithm is used for 3D face reconstruction. The proposed scheme has been tested on CMU’s Face In Action (FIA) video database with 221 subjects. Experimental results show a 40% improvement in matching performance as a result of using the 3D models.


Face recognition video surveillance 3D face modeling view synthesis structure from motion factorization active appearance model 


  1. 1.
    Pentland, A., Moghaddam, B., Starner, T.: View-based and Modular Eigenspace for Face Recognition. In: Proc. CVPR, pp. 84–91 (1994)Google Scholar
  2. 2.
    Chai, X., Shan, S., Chen, X., Gao, W.: Local Linear Regression (LLR) for Pose Invariant Face Recognition. In: Proc. AFGR, pp. 631–636 (2006)Google Scholar
  3. 3.
    Beymer, D., Poggio, T.: Face Recognition from One Example View. In: Proc. ICCV, pp. 500–507 (1995)Google Scholar
  4. 4.
    Blanz, V., Vetter, T.: Face Recognition based on Fitting a 3D Morphable Model. IEEE Trans. PAMI 25, 1063–1074 (2003)Google Scholar
  5. 5.
    FaceVACS Software Developer Kit, Cognitec,
  6. 6.
    Tomasi, C., Kanade, T.: Shape and motion from image streams under orthography: A factorization method. Int. Journal of Computer Vision 9(2), 137–154 (1992)CrossRefGoogle Scholar
  7. 7.
    Xiao, J., Chai, J., Kanade, T.: A Closed-Form Solution to Non-Rigid Shape and Motion Recovery. In: Pajdla, T., Matas, J(G.) (eds.) ECCV 2004. LNCS, vol. 3024, pp. 668–675. Springer, Heidelberg (2004)Google Scholar
  8. 8.
    Brand, M.: A Direct Method for 3D Factorization of Nonrigid Motion Observation in 2D. In: Proc. CVPR, vol. 2, pp. 122–128 (2005)Google Scholar
  9. 9.
    Stegmann, M.B.: The AAM-API: An Open Source Active Appearance Model Implementation. In: Ellis, R.E., Peters, T.M. (eds.) MICCAI 2003. LNCS, vol. 2879, pp. 951–952. Springer, Heidelberg (2003)Google Scholar
  10. 10.
    Goh, R., Liu, L., Liu, X., Chen, T.: The CMU Face In Action (FIA) Database. In: Zhao, W., Gong, S., Tang, X. (eds.) AMFG 2005. LNCS, vol. 3723, pp. 255–263. Springer, Heidelberg (2005)CrossRefGoogle Scholar
  11. 11.
    Zhao, W., Chellappa, R.: SFS Based View Synthesis for Robust Face Recognition. In: Proc. FGR, pp. 285–292 (2000)Google Scholar
  12. 12.
    Phillips, P.J., Grother, P., Micheals, R.J., Blackburn, D.M., Tabassi, E., Bone, J.M.: FRVT: 2002: Evaluation Report, Tech. Report NISTIR 6965, NIST (2003)Google Scholar
  13. 13.
    Phillips, P.J., Flynn, P.J., Scruggs, T., Bowyer, K.W., Worek, W.: Preliminary Face Recognition Grand Challenge Results. In: Proc. AFGR, pp. 15–24 (2006)Google Scholar
  14. 14.
    Lee, K., Ho, J., Yang, M., Kriegman, D.: Video-based face recognition using probabilistic appearance manifolds. CVPR I, 313–320 (2003)Google Scholar
  15. 15.
    Zhou, S., Krueger, V., Chellappa, R.: Probabilistic recognition of human faces from video. Computer Vision and Image Understanding 91, 214–245 (2003)CrossRefGoogle Scholar
  16. 16.
    Barber, C.B., Dobkin, D.P., Huhdanpaa, H.: The Quickhull Algorithm for Convex Hulls. ACM Trans. Mathematical Software 22(4), 469–483 (1996)zbMATHCrossRefMathSciNetGoogle Scholar
  17. 17.
    Ullman, S.: The Interpretation of Visual Motion. MIT Press, Cambridge, MA (1979)Google Scholar
  18. 18.
    Matthews, I., Baker, S.: Active Appearance Models Revisited. International Journal of Computer Vision 60(2), 135–164 (2004)CrossRefGoogle Scholar
  19. 19.
    Maurer, T., Guigonis, D., Maslov, I., Pesenti, B., Tsaregorodtsev, A., West, D., Medioni, G.: Performance of Geometrix ActiveIDTM 3D Face Recognition Engine on the FRGC Data. In: Proc. CVPR, pp. 154–160 (2005)Google Scholar
  20. 20.
    Tu, J., Huang, T., Tao, H.: Accurate Head Pose Tracking in Low Resolution Video. In: Proc. FGR, pp. 573–578 (2006)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2007

Authors and Affiliations

  • Unsang Park
    • 1
  • Anil K. Jain
    • 1
  1. 1.Department of Computer Science and Engineering, Michigan State University, 3115 Engineering Building, East Lasing, MI 48824USA

Personalised recommendations