Advertisement

Selection of Basis Functions Guided by the L2 Soft Margin

  • Ignacio Barrio
  • Enrique Romero
  • Lluís Belanche
Conference paper
  • 1.8k Downloads
Part of the Lecture Notes in Computer Science book series (LNCS, volume 4668)

Abstract

Support Vector Machines (SVMs) for classification tasks produce sparse models by maximizing the margin. Two limitations of this technique are considered in this work: firstly, the number of support vectors can be large and, secondly, the model requires the use of (Mercer) kernel functions. Recently, some works have proposed to maximize the margin while controlling the sparsity. These works also require the use of kernels. We propose a search process to select a subset of basis functions that maximize the margin without the requirement of being kernel functions. The sparsity of the model can be explicitly controlled. Experimental results show that accuracy close to SVMs can be achieved with much higher sparsity. Further, given the same level of sparsity, more powerful search strategies tend to obtain better generalization rates than simpler ones.

Keywords

Basis Function Forward Selection Relevance Vector Machine Radial Basis Function Sparse Model 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. 1.
    Vapnik, V.: The Nature of Statistical Learning Theory. Springer, Heidelberg (1995)zbMATHGoogle Scholar
  2. 2.
    Schölkopf, B., Tsuda, J.P.V.K.: Kernel methods in computational biology. MIT Press, Cambridge (2004)Google Scholar
  3. 3.
    Balcan, M.F., Blum, A.: On a theory of learning with similarity functions. In: Proceedings of the 23rd International Conference on Machine Learning, pp. 73–80. ACM Press, New York (2006)CrossRefGoogle Scholar
  4. 4.
    Steinwart, I.: Sparseness of support vector machines. Journal of Machine Learning Research 4, 1071–1105 (2003)CrossRefGoogle Scholar
  5. 5.
    Tipping, M.: Sparse Bayesian learning and the relevance vector machine. Journal of Machine Learning Research 1, 211–244 (2001)zbMATHCrossRefGoogle Scholar
  6. 6.
    Vincent, P., Bengio, Y.: Kernel matching pursuit. Machine Learning 48(1-3), 165–187 (2002)zbMATHCrossRefGoogle Scholar
  7. 7.
    Bradley, P.S., Mangasarian, O.L.: Feature selection via concave minimization and support vector machines. In: 15th International Conf. on Machine Learning, pp. 82–90. Morgan Kaufmann, San Francisco (1998)Google Scholar
  8. 8.
    Wu, M., Schölkopf, B., Bakir, G.: Building sparse large margin classifiers. In: 22nd International Conf. on Machine learning, pp. 996–1003. ACM Press, New York (2005)CrossRefGoogle Scholar
  9. 9.
    Keerthi, S., Chapelle, O., DeCoste, D.: Building Support Vector Machines with Reduced Classifier Complexity. Journal of Machine Learning Research 8, 1–22 (2006)Google Scholar
  10. 10.
    Keerthi, S., DeCoste, D.: A modified finite Newton method for fast solution of large scale linear SVMs. Journal of Machine Learning Research 6, 341–361 (2005)Google Scholar
  11. 11.
    Kittler, J.: Feature selection and extraction. In: Young, F. (ed.) Handbook of Pattern Recognition and Image Processing, Academic Press, London (1986)Google Scholar
  12. 12.
    Pudil, P., Novovičová, J., Kittler, J.: Floating Search Methods in Feature Selection. Pattern Recognition Letters 15(11), 1119–1125 (1994)CrossRefGoogle Scholar
  13. 13.
    Lee, Y.J., Mangasarian, O.L.: Rsvm: Reduced support vector machines. In: SIAM International Conference on Data Mining (2001)Google Scholar
  14. 14.
    Lin, K.M., Lin, C.J.: A study on reduced support vector machines. IEEE Transactions on Neural Networks 14(6), 1449–1559 (2003)CrossRefGoogle Scholar
  15. 15.
    Schölkopf, B., Sung, K., Burges, C., Girosi, F., Niyogi, P., Poggio, T., Vapnik, V.: Comparing support vector machines with Gaussian kernels to radial basis function classifiers. IEEE Transactions on Signal Processing 45(11), 2758–2765 (1997)CrossRefGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2007

Authors and Affiliations

  • Ignacio Barrio
    • 1
  • Enrique Romero
    • 1
  • Lluís Belanche
    • 1
  1. 1.Departament de Llenguatges i Sistemes Informàtics, Universitat Politècnica de Catalunya, BarcelonaSpain

Personalised recommendations