Learning to Predict the Leave-One-Out Error of Kernel Based Classifiers

  • Koji Tuda
  • Gunnar Rätsch
  • Sebastian Mika
  • Klaus-Robert Müller
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 2130)

Abstract

We propose an algorithm to predict the leave-one-out (LOO) error for kernel based classifiers. To achieve this goal with computational efficiency, we cast the LOO error approximation task into a classification problem. This means that we need to learn a classification of whether or not a given training sample - if left out of the data set - would be misclassified. For this learning task, simple data dependent features are proposed, inspired by geometrical intuition. Our approach allows to reliably select a good model as demonstrated in simulations on Support Vector and Linear Programming Machines. Comparisons to existing learning theoretical bounds, e.g. the span bound, are given for various model selection scenarios.

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. 1.
    K.P. Bennett and O.L. Mangasarian. Robust linear programming discrimination of two linearly inseparable sets. Optimization Methods and Software, 1:23–34, 1992.CrossRefGoogle Scholar
  2. 2.
    O. Chapelle and V.N. Vapnik. Choosing kernel parameters for support vector machines. Personal communication, mar 2000.Google Scholar
  3. 3.
    T.S. Jaakkola and D. Haussler. Probabilistic kernel regression models. In Proceedings of the 1999 Conference on AI and Statistics, 1999.Google Scholar
  4. 4.
    K.-R. Müller, S. Mika, G. Rätsch, K. Tsuda, and B. Schölkopf. An introduction to kernel-based learning algorithms. IEEE Transactions on Neural Networks, 2001. in Press.Google Scholar
  5. 5.
    M. Opper and O. Winther. Gaussian processes and SVM: Mean field and leave-one-out. In A.J. Smola, P.L. Bartlett, B. Schölkopf, and D. Schuurmans, editors, Advances in Large Margin Classifiers, pages 311–326, Cambridge, MA, 2000. MIT Press.Google Scholar
  6. 6.
    V.N. Vapnik. Estimation of Dependences Based on Empirical Data. Springer-Verlag, Berlin, 1982.MATHGoogle Scholar
  7. 7.
    V.N. Vapnik. The nature of statistical learning theory. Springer Verlag, New York, 1995.MATHGoogle Scholar
  8. 8.
    V.N. Vapnik and O. Chapelle. Bounds on error expectation for support vector machines. Neural Computation, 12(9):2013–2036, September 2000.Google Scholar
  9. 9.
    G. Whaba, Y. Lin, and H. Zhang. Generalized approximate cross-validation for support-vector-machines: another way to look at margin-like quantities. Technical Report TR-1006, Dept. of Statistics, University of Wisconsin, April 1999.Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2001

Authors and Affiliations

  • Koji Tuda
    • 1
    • 3
  • Gunnar Rätsch
    • 1
    • 2
  • Sebastian Mika
    • 1
  • Klaus-Robert Müller
    • 1
    • 2
  1. 1.GMD FIRSTBerlinGermany
  2. 2.University of PotsdamPotsdam
  3. 3.AIST Computational Biology Research CenterTokyoJapan

Personalised recommendations