Generalization Bounds for Subspace Selection and Hyperbolic PCA

  • Andreas Maurer
Part of the Lecture Notes in Computer Science book series (LNCS, volume 3940)


We present a method which uses example pairs of equal or unequal class labels to select a subspace with near optimal metric properties in a kernel-induced Hilbert space. A representation of finite dimensional projections as bounded linear functionals on a space of Hilbert-Schmidt operators leads to PAC-type performance guarantees for the resulting feature maps. The proposed algorithm returns the projection onto the span of the principal eigenvectors of an empirical operator constructed in terms of the example pairs. It can be applied to meta-learning environments and experiments demonstrate an effective transfer of knowledge between different but related learning tasks.


Face Recognition Equivalence Constraint Empirical Operator Machine Learn Research Handwritten Character 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Bartlett, P.L., Mendelson, S.: Rademacher and Gaussian Complexities: Risk Bounds and Structural Results. Journal of Machine Learning Research (2002)Google Scholar
  2. 2.
    Bartlett, P., Bousquet, O., Mendelson, S.: Local Rademacher complexities, Available online:
  3. 3.
    Bar-Hillel, A., Hertz, T., Shental, N., Weinshall, D.: Learning a Mahalanobis Metric from Equivalence Constraints. Journal of Machine Learning Research 6, 937–965 (2005)MathSciNetMATHGoogle Scholar
  4. 4.
    Baxter, J.: A Model of Inductive Bias Learning. Journal of Artificial Intelligence Research 12, 149–198 (2000)MathSciNetMATHGoogle Scholar
  5. 5.
    Cristianini, N., Shawe-Taylor, J.: Support Vector Machines. Cambridge University Press, Cambridge (2000)CrossRefMATHGoogle Scholar
  6. 6.
    Hammer, R., Hertz, T., Hochstein, S., Weinshall, D.: Category learning from equivalence constraints. In: XXVII Conference of Cognitive Science Society (CogSci 2005) (available online)Google Scholar
  7. 7.
    Koltchinskii, V., Panchenko, D.: Empirical margin distributions and bounding the generalization error of combined classifiers. The Annals of Statistics 30(1), 1–50Google Scholar
  8. 8.
    Ledoux, M., Talagrand, M.: Probability in Banach Spaces: isoperimetry and processes. Springer, Heidelberg (1991)CrossRefMATHGoogle Scholar
  9. 9.
    McDiarmid, C.: Concentration. In: Probabilistic Methods of Algorithmic Discrete Mathematics, pp. 195–248. Springer, Berlin (1998)CrossRefGoogle Scholar
  10. 10.
    Mika, S., Schölkopf, B., Smola, A., Müller, K.-R., Scholz, M., Rätsch, G.: Kernel PCA and De-noising in Feature Spaces. Advances in Neural Information Processing Systems 11 (1998)Google Scholar
  11. 11.
    Shawe-Taylor, J., Christianini, N.: Estimating the moments of a random vector. In: Proceedings of GRETSI 2003 Conference, vol. I, pp. 47–52 (2003)Google Scholar
  12. 12.
    Reed, M., Simon, B.: Functional Analysis, part I of Methods of Mathematical Physics. Academic Press, London (1980)MATHGoogle Scholar
  13. 13.
    Robins, A.: Transfer in Cognition. In: Thrun, S., Pratt, L. (eds.) Learning to Learn. Springer, Heidelberg (1998)Google Scholar
  14. 14.
    Shawe-Taylor, J., Williams, C.K.I., Cristianini, N., Kandola, J.S.: On the eigenspectrum of the gram matrix and the generalization error of kernel-PCA. IEEE Transactions on Information Theory 51(7), 2510–2522 (2005)MathSciNetCrossRefMATHGoogle Scholar
  15. 15.
    Thrun, S.: Lifelong Learning Algorithms. In: Thrun, S., Pratt, L. (eds.) Learning to Learn. Springer, Heidelberg (1998)CrossRefGoogle Scholar
  16. 16.
    Xing, E.P., Ng, A.Y., Jordan, M.I., Russel, S.: Distance metric learning, with application to clustering with side information. In: Becker, S., Thrun, S., Obermayer, K. (eds.) Advances in Neural Information Processing Systems, vol. 14, MIT Press, Cambridge (2002)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2006

Authors and Affiliations

  • Andreas Maurer
    • 1
  1. 1.MünchenGermany

Personalised recommendations