Advertisement

Accurate Probabilistic Error Bound for Eigenvalues of Kernel Matrix

  • Lei Jia
  • Shizhong Liao
Part of the Lecture Notes in Computer Science book series (LNCS, volume 5828)

Abstract

The eigenvalues of the kernel matrix play an important role in a number of kernel methods. It is well known that these eigenvalues converge as the number of samples tends to infinity. We derive a probabilistic finite sample size bound on the approximation error of an individual eigenvalue, which has the important property that the bound scales with the dominate eigenvalue under consideration, reflecting the accurate behavior of the approximation error as predicted by asymptotic results and observed in numerical simulations. Under practical conditions, the bound presented here forms a significant improvement over existing non-scaling bound. Applications of this theoretical finding in kernel matrix selection and kernel target alignment are also presented.

Keywords

Kernel Method Kernel Matrix Kernel Principal Component Analysis True Alignment Machine Learn Research 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. 1.
    Shawe-Taylor, J., Cristianini, N.: Kernel Methods for Pattern Analysis. Cambridge University Press, Cambridge (2005)Google Scholar
  2. 2.
    Shawe-Taylor, J., Cristianini, N., Kandola, J.: On the concentration of spectral properties. In: Dietterich, T.G., Becker, S., Ghahramani, Z. (eds.) Advances in Neural Information Processing Systems 14, pp. 511–517. MIT Press, Cambridge (2002)Google Scholar
  3. 3.
    Schölkopf, B., Smola, A., Muller, K.: Kernel principal component analysis. In: Schölkopf, B., Burges, C., Smola, A. (eds.) Advances in Kernel Methods: Support Vector Machines, pp. 327–352. MIT Press, Cambridge (1998)Google Scholar
  4. 4.
    Saunders, C., Gammerman, A., Vovk, V.: Ridge regression learning algorithm in dual variables. In: Proceedings of the 15th International Conference on Machine Learning, pp. 24–27 (1998)Google Scholar
  5. 5.
    Trefethen, L., Bau, D.: Numerical Linear Algebra. SIAM, Philadelphia (1997)zbMATHGoogle Scholar
  6. 6.
    Ng, A.Y., Zheng, A.X., Jordan, M.I.: Link analysis, eigenvectors and stability. In: Proceedings of the 7th International Joint Conference on Artificial Intelligence, pp. 903–910 (2001)Google Scholar
  7. 7.
    Martin, S.: The numerical stability of kernel methods. Technical report, Sandia National Laboratory, New Mexico (2005)Google Scholar
  8. 8.
    Jia, L., Liao, S.: Perturbation stability of computing spectral data on kernel matrix. In: The 26th International Conference on Machine Learning, Workshop on Numerical Mathematics on Machine Learning (2009)Google Scholar
  9. 9.
    Williams, C., Seeger, M.: The effect of the input density distribution on kernel-based classifiers. In: Langley, P. (ed.) Proceedings of the 17th International Conference on Machine Learning, pp. 1159–1166. Morgan Kaufmann, San Francisco (2000)Google Scholar
  10. 10.
    Bengio, Y., Vincent, P., Paiement, J., Delalleau, O., Ouimet, M., Le Roux, N.: Spectral clustering and kernel PCA are learning eigenfunctions. Technical Report TR 1239, University of Montreal (2003)Google Scholar
  11. 11.
    Shawe-Taylor, J., Williams, C.K., Cristianini, N., Kandola, J.: On the eigenspectrum of the gram matrix and the generalization error of kernel-PCA. IEEE Transactions on Information Theory 51(7), 2510–2522 (2005)CrossRefMathSciNetGoogle Scholar
  12. 12.
    Zwald, L., Blanchard, G.: On the convergence of eigenspaces in kernel principal component analysis. In: Weiss, Y., Schölkopf, B., Platt, J. (eds.) Advances in Neural Information Processing Systems 18, pp. 1649–1656. MIT Press, Cambridge (2006)Google Scholar
  13. 13.
    Koltchinskii, V.: Asymptotics of spectral projections of some random matrices approximating integral operators. Progress in Probability 43, 191–227 (1998)MathSciNetGoogle Scholar
  14. 14.
    Koltchinskii, V., Giné, E.: Random matrix approximation of spectra of integral operators. Bernoulli 6(1), 113–167 (2000)zbMATHCrossRefMathSciNetGoogle Scholar
  15. 15.
    Horn, R.A., Johnson, C.R.: Matrix Analysis. Cambridge University Press, Cambridge (1985)zbMATHGoogle Scholar
  16. 16.
    Braun, M.L.: Accurate error bounds for the eigenvalues of the kernel matrix. Journal of Machine Learning Research 7, 2303–2328 (2006)MathSciNetGoogle Scholar
  17. 17.
    Braun, M.L.: Spectral Properties of the Kernel Matrix and their Relation to Kernel Methods in Machine Learning. PhD thesis, Bonn University (2005)Google Scholar
  18. 18.
    Cristianini, N., Kandola, J., Elisseeff, A., Shawe-Taylor, J.: On kernel-target alignment. Journal of Machine Learning Research 1, 1–31 (2002)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2009

Authors and Affiliations

  • Lei Jia
    • 1
  • Shizhong Liao
    • 1
  1. 1.School of Computer Science and Technology, Institute of Knowledge Science and EngineeringTianjin UniversityTianjinP. R. China

Personalised recommendations