Local Regularization for Multiclass Classification Facing Significant Intraclass Variations

  • Lior Wolf
  • Yoni Donner
Part of the Lecture Notes in Computer Science book series (LNCS, volume 5305)


We propose a new local learning scheme that is based on the principle of decisiveness: the learned classifier is expected to exhibit large variability in the direction of the test example. We show how this principle leads to optimization functions in which the regularization term is modified, rather than the empirical loss term as in most local learning schemes. We combine this local learning method with a Canonical Correlation Analysis based classification method, which is shown to be similar to multiclass LDA. Finally, we show that the classification function can be computed efficiently by reusing the results of previous computations. In a variety of experiments on new and existing data sets, we demonstrate the effectiveness of the CCA based classification method compared to SVM and Nearest Neighbor classifiers, and show that the newly proposed local learning method improves it even further, and outperforms conventional local learning schemes.


Training Image Canonical Correlation Analysis Local Regularization Local Learning Class Vector 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


  1. 1.
    Fei-Fei, L., Fergus, R., Perona, P.: A bayesian approach to unsupervised one-shot learning of object categories. In: ICCV, Nice, France, pp. 1134–1141 (2003)Google Scholar
  2. 2.
    Belkin, M., Niyogi, P.: Semi-supervised learning on riemannian manifolds. Machine Learning 56, 209–239 (2004)zbMATHCrossRefGoogle Scholar
  3. 3.
    Bart, E., Ullman, S.: Cross-generalization: learning novel classes from a single example by feature replacement. In: CVPR (2005)Google Scholar
  4. 4.
    Yamada, M., Pezeshki, A., Azimi-Sadjadi, M.: Relation between kernel cca and kernel fda. In: IEEE International Joint Conference on Neural Networks (2005)Google Scholar
  5. 5.
    Bottou, L., Vapnik, V.: Local learning algorithms. Neural Computation 4 (1992)Google Scholar
  6. 6.
    Zhang, H., Berg, A.C., Maire, M., Malik, J.: Svm-knn: Discriminative nearest neighbor classification for visual category recognition. In: CVPR (2006)Google Scholar
  7. 7.
    Hotelling, H.: Relations between two sets of variates. Biometrika 28, 321–377 (1936)zbMATHGoogle Scholar
  8. 8.
    Akaho, S.: A kernel method for canonical correlation analysis. In: International Meeting of Psychometric Society (2001)Google Scholar
  9. 9.
    Wolf, L., Shashua, A.: Learning over sets using kernel principal angles. J. Mach. Learn. Res. 4, 913–931 (2003)CrossRefMathSciNetGoogle Scholar
  10. 10.
    Neumaier, A.: Solving ill-conditioned and singular linear systems: A tutorial on regularization (1998)Google Scholar
  11. 11.
    Hastie, T., Tibshirani, R., Friedman, J.: The elements of statistical learning: data mining, inference and prediction. Springer, Heidelberg (2001)zbMATHGoogle Scholar
  12. 12.
    Sherman, J., Morrison, W.J.: Adjustment of an inverse matrix corresponding to changes in the elements of a given column or a given row of the original matrix. Annals of Mathematical Statistics 20, 621 (1949)Google Scholar
  13. 13.
    Golub, G.: Some modified eigenvalue problems. Technical report, Stanford (1971)Google Scholar
  14. 14.
    Bunch, J.R., Nielsen, C.P., Sorensen, D.C.: Rank-one modification of the symmetric eigenproblem. Numerische Mathematik 31, 31–48 (1978)zbMATHCrossRefMathSciNetGoogle Scholar
  15. 15.
    CalPhotos: A database of photos of plants, animals, habitats and other natural history subjects [web application], animal–mammals collection. bscit, University of California, Berkeley,–Mammal
  16. 16.
    Huang, G.B., Ramesh, M., Berg, T., Learned-Miller, E.: Labeled faces in the wild: A database for studying face recognition in unconstrained environments. University of Massachusetts, Amherst, Technical Report 07-49 (2007)Google Scholar
  17. 17.
    Rifkin, R., Klautau, A.: In defense of one-vs-all classification. Journal of Machine Learning Research 5 (2004)Google Scholar
  18. 18.
    Vedaldi, A.: Bag of features: A simple bag of features classifier (2007),
  19. 19.
    Nister, D., Stewenius, H.: Scalable recognition with a vocabulary tree. In: CVPR (2006)Google Scholar
  20. 20.
    Nowak, E., Jurie, F., Triggs, B.: Sampling strategies for bag-of-features image classification. In: European Conference on Computer Vision. Springer, Heidelberg (2006)Google Scholar
  21. 21.
    Ojala, T., Pietikainen, M., Harwood, D.: A comparative-study of texture measures with classification based on feature distributions. Pattern Recognition 29 (1996)Google Scholar
  22. 22.
    Ahonen, T., Hadid, A., Pietikainen, M.: Face recognition with local binary patterns. In: Pajdla, T., Matas, J(G.) (eds.) ECCV 2004. LNCS, vol. 3024. Springer, Heidelberg (2004)Google Scholar
  23. 23.
    Huang, G.B., Jain, V., Learned-Miller, E.: Unsupervised joint alignment of complex images. ICCV (2007)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2008

Authors and Affiliations

  • Lior Wolf
    • 1
  • Yoni Donner
    • 1
  1. 1.The School of Computer ScienceTel Aviv UniverisyTel AvivIsrael

Personalised recommendations