Learning Discriminative Canonical Correlations for Object Recognition with Image Sets

  • Tae-Kyun Kim
  • Josef Kittler
  • Roberto Cipolla
Part of the Lecture Notes in Computer Science book series (LNCS, volume 3953)


We address the problem of comparing sets of images for object recognition, where the sets may represent arbitrary variations in an object’s appearance due to changing camera pose and lighting conditions. The concept of Canonical Correlations (also known as principal angles) can be viewed as the angles between two subspaces. As a way of comparing sets of vectors or images, canonical correlations offer many benefits in accuracy, efficiency, and robustness compared to the classical parametric distribution-based and non-parametric sample-based methods. Here, this is demonstrated experimentally for reasonably sized data sets using existing methods exploiting canonical correlations. Motivated by their proven effectiveness, a novel discriminative learning over sets is proposed for object recognition. Specifically, inspired by classical Linear Discriminant Analysis (LDA), we develop a linear discriminant function that maximizes the canonical correlations of within-class sets and minimizes the canonical correlations of between-class sets. The proposed method significantly outperforms the state-of-the-art methods on two different object recognition problems using face image sets with arbitrary motion captured under different illuminations and image sets of five hundred general object categories taken at different views.


Face Recognition Singular Value Decomposition Object Recognition Linear Discriminant Analysis Canonical Correlation 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


  1. 1.
    Lee, K., Yang, M., Kriegman, D.: Video-based face recognition using probabilistic appearance manifolds. In: CVPR, pp. 313–320 (2003)Google Scholar
  2. 2.
    Zhou, S., Krueger, V., Chellappa, R.: Probabilistic recognition of human faces from video. Computer Vision and Image Understanding 91(1), 214–245 (2003)CrossRefGoogle Scholar
  3. 3.
    Shakhnarovich, G., Fisher, J.W., Darrel, T.: Face recognition from long-term observations. In: Heyden, A., Sparr, G., Nielsen, M., Johansen, P. (eds.) ECCV 2002. LNCS, vol. 2352, pp. 851–865. Springer, Heidelberg (2002)CrossRefGoogle Scholar
  4. 4.
    Arandjelović, O., Shakhnarovich, G., Fisher, J., Cipolla, R., Darrell, T.: Face recognition with image sets using manifold density divergence. In: CVPR (2005)Google Scholar
  5. 5.
    Satoh, S.: Comparative Evaluation of Face Sequence Matching for Content-based Video Access. In: Int’l Conf. on Automatic Face and Gesture Recognition (2000)Google Scholar
  6. 6.
    Belhumeur, P.N., Hespanha, J.P., Kriegman, D.J.: Eigenfaces vs. Fisherfaces: Recognition Using Class Specific Linear Projection. PAMI 19(7), 711–720 (1997)Google Scholar
  7. 7.
    Bressan, M., Vitria, J.: Nonparametric discriminant analysis and nearest neighbor classification. Pattern Recognition Letters 24, 2743–2749 (2003)CrossRefGoogle Scholar
  8. 8.
    Yamaguchi, O., Fukui, K., Maeda, K.: Face recognition using temporal image sequence. AFG 10, 318–323 (1998)Google Scholar
  9. 9.
    Wolf, L., Shashua, A.: Learning over sets using kernel principal angles. JMLR 4(10), 913–931 (2003)CrossRefMathSciNetGoogle Scholar
  10. 10.
    Kim, T.K., Arandjelović, O., Cipolla, R.: Learning over Sets using Boosted Manifold Principal Angles (BoMPA). In: BMVC (2005)Google Scholar
  11. 11.
    Fukui, K., Yamaguchi, O.: Face recognition using multi-viewpoint patterns for robot vision. In: Int’l Symp. of Robotics Research (2003)Google Scholar
  12. 12.
    Hotelling, H.: Relations between two sets of variates. Biometrika 28, 321–372 (1936)zbMATHGoogle Scholar
  13. 13.
    Björck, Å., Golub, G.H.: Numerical methods for computing angles between linear subspaces. Mathematics of Computation 27(123), 579–594 (1973)zbMATHCrossRefMathSciNetGoogle Scholar
  14. 14.
    Viola, P., Jones, M.: Robust real-time face detection. IJCV 57(2), 137–154 (2004)CrossRefGoogle Scholar
  15. 15.
    Geusebroek, J.M., Burghouts, G.J., Smeulders, A.W.M.: The Amsterdam library of object images. IJCV 61(1), 103–112 (2005)CrossRefGoogle Scholar
  16. 16.
  17. 17.
    Oja, E.: Subspace Methods of Pattern Recognition. Research Studies Press (1983)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2006

Authors and Affiliations

  • Tae-Kyun Kim
    • 1
  • Josef Kittler
    • 2
  • Roberto Cipolla
    • 1
  1. 1.Department of EngineeringUniversity of CambridgeCambridgeUK
  2. 2.Centre for Vision, Speech and Signal ProcessingUniversity of SurreyGuildfordUK

Personalised recommendations