Face Recognition from Video Using the Generic Shape-Illumination Manifold

  • Ognjen Arandjelović
  • Roberto Cipolla
Part of the Lecture Notes in Computer Science book series (LNCS, volume 3954)


In spite of over two decades of intense research, illumination and pose invariance remain prohibitively challenging aspects of face recognition for most practical applications. The objective of this work is to recognize faces using video sequences both for training and recognition input, in a realistic, unconstrained setup in which lighting, pose and user motion pattern have a wide variability and face images are of low resolution. In particular there are three areas of novelty: (i) we show how a photometric model of image formation can be combined with a statistical model of generic face appearance variation, learnt offline, to generalize in the presence of extreme illumination changes; (ii) we use the smoothness of geodesically local appearance manifold structure and a robust same-identity likelihood to achieve invariance to unseen head poses; and (iii) we introduce an accurate video sequence “reillumination” algorithm to achieve robustness to face motion patterns in video. We describe a fully automatic recognition system based on the proposed method and an extensive evaluation on 171 individuals and over 1300 video sequences with extreme illumination, pose and head motion variation. On this challenging data set our system consistently demonstrated a nearly perfect recognition rate (over 99.7%), significantly outperforming state-of-the-art commercial software and methods from the literature.


Face Recognition Video Sequence Recognition Rate Face Image Gaussian Mixture Model 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Adini, Y., Moses, Y., Ullman, S.: Face recognition: The problem of compensating for changes in illumination direction. PAMI 19(7), 721–732 (1997)CrossRefGoogle Scholar
  2. 2.
    Arandjelović, O., Cipolla, R.: A new look at filtering techniques for illumination invariance in automatic face recogition. In: FG (2006)Google Scholar
  3. 3.
    Arandjelović, O., Shakhnarovich, G., Fisher, J., Cipolla, R., Darrell, T.: Face recognition with image sets using manifold density divergence. In: CVPR (2005)Google Scholar
  4. 4.
    Arandjelović, O., Zisserman, A.: Automatic face recognition for film character retrieval in feature-length films. In: CVPR, vol. 1, pp. 860–867 (2005)Google Scholar
  5. 5.
    Barrett, W.A.: A survey of face recognition algorithms and testing results. Systems and Computers 1, 301–305 (1998)Google Scholar
  6. 6.
    BBC. Doubts over passport face scans. BBC Online, UK Edition (October 2004)Google Scholar
  7. 7.
    Belhumeur, P.N., Kriegman, D.J.: What is the set of images of an object under all possible illumination conditions? IJCV 28(3), 245–260 (1998)CrossRefGoogle Scholar
  8. 8.
    Berg, T.L., Berg, A.C., Edwards, J., Maire, M., White, R., Teh, Y.W., Learned-Miller, E., Forsyth, D.A.: Names and faces in the news. In: CVPR, vol. 2, pp. 848–854 (2004)Google Scholar
  9. 9.
    Bichsel, M., Pentland, A.P.: Human face recognition and the face image set’s topology. Computer Vision, Graphics and Image Processing 59(2), 254–261 (1994)CrossRefGoogle Scholar
  10. 10.
    Blanz, V., Vetter, T.: Face recognition based on fitting a 3D morphable model. PAMI 25(9), 1063–1074 (2003)CrossRefGoogle Scholar
  11. 11.
    Craw, I., Costen, N.P., Kato, T., Akamatsu, S.: How should we represent faces for automatic recognition? PAMI 21, 725–736 (1999)CrossRefGoogle Scholar
  12. 12.
    Duda, R.O., Hart, P.E., Stork, D.G.: Pattern Classification, 2nd edn. John Wily & Sons, Inc., New York (2000)MATHGoogle Scholar
  13. 13.
    Everingham, M., Zisserman, A.: Automated person identification in video. In: Enser, P.G.B., Kompatsiaris, Y., O’Connor, N.E., Smeaton, A.F., Smeulders, A.W.M. (eds.) CIVR 2004. LNCS, vol. 3115, pp. 289–298. Springer, Heidelberg (2004)CrossRefGoogle Scholar
  14. 14.
    Fitzgibbon, A.W., Zisserman, A.: On affine invariant clustering and automatic cast listing in movies. In: Heyden, A., Sparr, G., Nielsen, M., Johansen, P. (eds.) ECCV 2002. LNCS, vol. 2352, pp. 304–320. Springer, Heidelberg (2002)CrossRefGoogle Scholar
  15. 15.
    Fukui, K., Yamaguchi, O.: Facial feature point extraction method based on combination of shape extraction and pattern matching. Systems and Computers in Japan 29(6), 2170–2177 (1998)CrossRefGoogle Scholar
  16. 16.
    Fukui, K., Yamaguchi, O.: Face recognition using multi-viewpoint patterns for robot vision. In: Int’l Symp. of Robotics Research (2003)Google Scholar
  17. 17.
    Gavrila, D.M.: Pedestrian detection from a moving vehicle. In: Vernon, D. (ed.) ECCV 2000. LNCS, vol. 1843, pp. 37–49. Springer, Heidelberg (2000)CrossRefGoogle Scholar
  18. 18.
    Georghiades, A.S., Kriegman, D.J., Belhumeur, P.N.: Illumination cones for recognition under variable lighting: Faces. In: CVPR, pp. 52–59 (1998)Google Scholar
  19. 19.
    Gorodnichy, D.O.: Associative neural networks as means for low-resolution video-based recognition. In: International Joint Conference on Neural Networks (2005)Google Scholar
  20. 20.
    Gross, R., Matthews, I., Baker, S.: Generic vs. person specific active appearance models. In: BMVC (2004)Google Scholar
  21. 21.
    Hall, P., Marshall, D., Martin, R.: Merging and splitting eigenspace models. PAMI 22(9), 1042–1049 (2000)CrossRefGoogle Scholar
  22. 22.
    Identix. Faceit,
  23. 23.
    Kepenekci, B.: Face Recognition Using Gabor Wavelet Transform. PhD thesis, The Middle East Technical University (2001)Google Scholar
  24. 24.
    Kim, T., Arandjelović, O., Cipolla, R.: Learning over sets using boosted manifold principal angles (BoMPA). In: BMVC (to appear, 2005)Google Scholar
  25. 25.
    Kimmel, R., Elad, M., Shaked, D., Keshet, R., Sobel, I.: A variational framework for retinex. IJCV 52(1), 7–23 (2003)CrossRefMATHGoogle Scholar
  26. 26.
    Lee, K., Kriegman, D.: Online learning of probabilistic appearance manifolds for video-based recognition and tracking. In: CVPR, vol. 1, pp. 852–859 (2005)Google Scholar
  27. 27.
    Lee, K., Yang, M., Kriegman, D.: Video-based face recognition using probabilistic appearance manifolds. In: CVPR, vol. 1, pp. 313–320 (2003)Google Scholar
  28. 28.
    Li, Y., Gong, S., Liddell, H.: Modelling faces dynamically across views and over time. ICCV 1, 554–559 (2001)Google Scholar
  29. 29.
    Liu, X., Chen, T.: Video-based face recognition using adaptive hidden Markov models. In: CVPR, vol. 1, pp. 340–345 (2003)Google Scholar
  30. 30.
    Murase, H., Nayar, S.: Visual learning and recognition of 3-D objects from appearance. IJCV 14, 5–24 (1995)CrossRefGoogle Scholar
  31. 31.
    Palanivel, S., Venkatesh, B.S., Yegnanarayana, B.: Real time face recognition system using autoassociative neural network models. ASSP 2, 833–836 (2003)Google Scholar
  32. 32.
    Pentland, A., Moghaddam, B., Starner, T.: View-based and modular eigenspaces for face recognition. In: CVPR, pp. 84–91 (1994)Google Scholar
  33. 33.
    Phillips, P.J., Grother, P., Micheals, R.J., Blackburn, D.M., Tabassi, E., Bone, J.M.: FRVT 2002: Overview and summary. Technical report, National Institute of Justice (March 2003)Google Scholar
  34. 34.
    Riklin-Raviv, T., Shashua, A.: The quotient image: Class based re-rendering and recognition with varying illuminations. PAMI 23(2), 219–239 (2001)Google Scholar
  35. 35.
    Shakhnarovich, G., Fisher III, J.W., Darrell, T.: Face recognition from long-term observations. In: Heyden, A., Sparr, G., Nielsen, M., Johansen, P. (eds.) ECCV 2002. LNCS, vol. 2352, pp. 851–865. Springer, Heidelberg (2002)CrossRefGoogle Scholar
  36. 36.
    Sim, T., Zhang, S.: Exploring face space. Face Processing in Video (2004)Google Scholar
  37. 37.
    Sivic, J., Everingham, M., Zisserman, A.: Person spotting: Video shot retrieval for face sets. In: Leow, W.-K., Lew, M., Chua, T.-S., Ma, W.-Y., Chaisorn, L., Bakker, E.M. (eds.) CIVR 2005. LNCS, vol. 3568, pp. 226–236. Springer, Heidelberg (2005)CrossRefGoogle Scholar
  38. 38.
    Stenger, B., Thayananthan, A., Torr, P., Cipolla, R.: Filtering using a tree-based estimator. ICCV 2, 1063–1070 (2003)MATHGoogle Scholar
  39. 39.
    Tenenbaum, J.B., de Silva, V., Langford, J.C.: A global geometric framework for nonlinear dimensionality reduction. Science 290(5500), 2319–2323 (2000)CrossRefGoogle Scholar
  40. 40.
    Tipping, M.E., Bishop, C.M.: Mixtures of probabilistic principal component analyzers. Neural Computation 11(2), 443–482 (1999)CrossRefGoogle Scholar
  41. 41.
  42. 42.
    Viola, P., Jones, M.: Robust real-time face detection. IJCV 57(2) (2004)Google Scholar
  43. 43.
    Wang, H., Li, S.Z., Wang, Y.: Face recognition under varying lighting conditions using self quotient image. In: FG, pp. 819–824 (2004)Google Scholar
  44. 44.
    Wang, X., Tang, X.: Unified subspace analysis for face recognition. In: ICCV (2003)Google Scholar
  45. 45.
    Wolf, L., Shashua, A.: Learning over sets using kernel principal angles. JMLR 4(10), 913–931 (2003)MathSciNetMATHGoogle Scholar
  46. 46.
    Zhao, W., Chellappa, R., Phillips, P.J., Rosenfeld, A.: Face recognition: A literature survey. ACM Computing Surveys 35(4), 399–458 (2004)CrossRefGoogle Scholar
  47. 47.
    Zhou, S., Krueger, V., Chellappa, R.: Probabilistic recognition of human faces from video. Computer Vision and Image Understanding 91(1), 214–245 (2003)CrossRefGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2006

Authors and Affiliations

  • Ognjen Arandjelović
    • 1
  • Roberto Cipolla
    • 1
  1. 1.Department of EngineeringUniversity of CambridgeUK

Personalised recommendations