The Huller: A Simple and Efficient Online SVM

  • Antoine Bordes
  • Léon Bottou
Part of the Lecture Notes in Computer Science book series (LNCS, volume 3720)


We propose a novel online kernel classifier algorithm that converges to the Hard Margin SVM solution. The same update rule is used to both add and remove support vectors from the current classifier. Experiments suggest that this algorithm matches the SVM accuracies after a single pass over the training examples. This algorithm is attractive when one seeks a competitive classifier with large datasets and limited computing resources.


Support Vector Machine Support Vector Online Algorithm Cache Size Machine Learn Research 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


  1. 1.
    Vapnik, V.N.: The Nature of Statistical Learning Theory. Springer, New York (1995)zbMATHGoogle Scholar
  2. 2.
    Aizerman, M.A., Braverman, É.M., Rozonoér, L.I.: Theoretical foundations of the potential function method in pattern recognition learning. Automation and Remote Control 25, 821–837 (1964)Google Scholar
  3. 3.
    Vapnik, V., Lerner, A.: Pattern recognition using generalized portrait method. Automation and Remote Control 24, 774–780 (1963)Google Scholar
  4. 4.
    Rosenblatt, F.: The perceptron: A probabilistic model for information storage and organization in the brain. Psychological Review 65, 386–408 (1958)CrossRefMathSciNetGoogle Scholar
  5. 5.
    Freund, Y., Schapire, R.E.: Large margin classification using the perceptron algorithm. In: Shavlik, J. (ed.) Machine Learning: Proceedings of the Fifteenth International Conference. Morgan Kaufmann, San Francisco (1998)Google Scholar
  6. 6.
    Frieß, T.T., Cristianini, N., Campbell, C.: The kernel Adatron algorithm: a fast and simple learning procedure for support vector machines. In: Shavlik, J. (ed.) 15th International Conf. Machine Learning, pp. 188–196. Morgan Kaufmann Publishers, San Francisco (1998)Google Scholar
  7. 7.
    Gentile, C.: A new approximate maximal margin classification algorithm. Journal of Machine Learning Research 2, 213–242 (2001)CrossRefMathSciNetGoogle Scholar
  8. 8.
    Li, Y., Long, P.: The relaxed online maximum margin algorithm. Machine Learning 46, 361–387 (2002)zbMATHCrossRefGoogle Scholar
  9. 9.
    Crammer, K., Singer, Y.: Ultraconservative online algorithms for multiclass problems. Journal of Machine Learning Research 3, 951–991 (2003)zbMATHCrossRefMathSciNetGoogle Scholar
  10. 10.
    Crammer, K., Kandola, J., Singer, Y.: Online classification on a budget. In: Thrun, S., Saul, L., Schölkopf, B. (eds.) Advances in Neural Information Processing Systems 16, MIT Press, Cambridge (2004)Google Scholar
  11. 11.
    Keerthi, S.S., Shevade, S.K., Bhattacharyya, C., Murthy, K.R.K.: A fast iterative nearest point algorithm for support vector machine classifier design. Technical Report Technical Report TR-ISL-99-03, Indian Institute of Science, Bangalore (1999),
  12. 12.
    Bennett, K.P., Bredensteiner, E.J.: Duality and geometry in SVM classifiers. In: Langley, P. (ed.) Proceedings of the 17th International Conference on Machine Learning, pp. 57–64. Morgan Kaufmann, San Francisco (2000)Google Scholar
  13. 13.
    Crisp, D.J., Burges, C.J.C.: A geometric interpretation of ν-SVM classifiers. In: Solla, S.A., Leen, T.K., Müller, K.R. (eds.) Advances in Neural Information Processing Systems 12. MIT Press, Cambridge (2000)Google Scholar
  14. 14.
    Gilbert, E.G.: Minimizing the quadratic form on a convex set. SIAM J. Control 4, 61–79 (1966)zbMATHCrossRefMathSciNetGoogle Scholar
  15. 15.
    Haffner, P.: Escaping the convex hull with extrapolated vector machines. In: Dietterich, T., Becker, S., Ghahramani, Z. (eds.) Advances in Neural Information Processing Systems 14, pp. 753–760. MIT Press, Cambridge (2002)Google Scholar
  16. 16.
    Cortes, C., Vapnik, V.: Support vector networks. Machine Learning 20, 273–297 (1995)zbMATHGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2005

Authors and Affiliations

  • Antoine Bordes
    • 1
    • 2
  • Léon Bottou
    • 2
  1. 1.Ecole Supérieure de Physique et de Chimie IndustriellesParisFrance
  2. 2.NEC Labs AmericaPrincetonUSA

Personalised recommendations