A Note on Extending Generalization Bounds for Binary Large-Margin Classifiers to Multiple Classes

  • Ürün Dogan
  • Tobias Glasmachers
  • Christian Igel
Part of the Lecture Notes in Computer Science book series (LNCS, volume 7523)


A generic way to extend generalization bounds for binary large-margin classifiers to large-margin multi-category classifiers is presented. The simple proceeding leads to surprisingly tight bounds showing the same \(\tilde{O}(d^2)\) scaling in the number d of classes as state-of-the-art results. The approach is exemplified by extending a textbook bound based on Rademacher complexity, which leads to a multi-class bound depending on the sum of the margin violations of the classifier.


Support Vector Machine Multiple Classis Empirical Risk Canonical Extension Machine Learn Research 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


  1. 1.
    Vapnik, V.: Statistical Learning Theory. John Wiley and Sons (1998)Google Scholar
  2. 2.
    Schölkopf, B., Smola, A.J.: Learning with Kernels: Support Vector Machines, Regularization, Optimization, and Beyond. MIT Press (2002)Google Scholar
  3. 3.
    Shawe-Taylor, J., Cristianini, N.: Kernel Methods for Pattern Analysis. Cambridge University Press (2004)Google Scholar
  4. 4.
    Boucheron, S., Bousquet, O., Lugosi, G.: Theory of classification: A survey of some recent advances. ESAIM: Probability and Statistics 9, 323–375 (2005)MathSciNetzbMATHCrossRefGoogle Scholar
  5. 5.
    Bartlett, P.L., Jordan, M.I., McAuliffe, J.D.: Convexity, classification, and risk bounds. Journal of the American Statistical Association 101, 138–156 (2006)MathSciNetzbMATHCrossRefGoogle Scholar
  6. 6.
    Wu, Q., Ying, Y., Zhou, D.X.: Multi-kernel regularized classifiers. Journal of Complexity 23, 108–134 (2007)MathSciNetzbMATHCrossRefGoogle Scholar
  7. 7.
    Steinwart, I., Scovel, C.: Fast rates for support vector machines using Gaussian kernels. The Annals of Statistics 35, 575–607 (2007)MathSciNetzbMATHCrossRefGoogle Scholar
  8. 8.
    Steinwart, I.: Oracle inequalities for svms that are based on random entropy numbers. Journal of Complexity 25, 437–454 (2009)MathSciNetzbMATHCrossRefGoogle Scholar
  9. 9.
    Guermeur, Y.: VC theory for large margin multi-category classifiers. Journal of Machine Learning Research 8, 2551–2594 (2007)MathSciNetzbMATHGoogle Scholar
  10. 10.
    Guermeur, Y.: Sample complexity of classifiers taking values in ℝQ, Application to multi-class SVMs. Communications in Statistics: Theory and Methods 39, 543–557 (2010)MathSciNetzbMATHCrossRefGoogle Scholar
  11. 11.
    Bordes, A., Usunier, N., Bottou, L.: Sequence Labelling SVMs Trained in One Pass. In: Daelemans, W., Goethals, B., Morik, K. (eds.) ECML PKDD 2008, Part I. LNCS (LNAI), vol. 5211, pp. 146–161. Springer, Heidelberg (2008)CrossRefGoogle Scholar
  12. 12.
    Aronszajn, N.: Theory of reproducing kernels. Transactions of the American Mathematical Society 68, 337–404 (1950)MathSciNetzbMATHCrossRefGoogle Scholar
  13. 13.
    Boser, B.E., Guyon, I.M., Vapnik, V.N.: A training algorithm for optimal margin classifiers. In: Proceedings of the Fifth Annual Workshop on Computational Learning Theory (COLT 1992), pp. 144–152. ACM (1992)Google Scholar
  14. 14.
    Cortes, C., Vapnik, V.: Support-vector networks. Machine Learning 20, 273–297 (1995)zbMATHGoogle Scholar
  15. 15.
    Weston, J., Watkins, C.: Support vector machines for multi-class pattern recognition. In: Verleysen, M. (ed.) Proceedings of the Seventh European Symposium On Artificial Neural Networks (ESANN), pp. 219–224. d-side Publications, Belgium (1999)Google Scholar
  16. 16.
    Bredensteiner, E.J., Bennett, K.P.: Multicategory classification by support vector machines. Computational Optimization and Applications 12, 53–79 (1999)MathSciNetzbMATHCrossRefGoogle Scholar
  17. 17.
    Crammer, K., Singer, Y.: On the algorithmic implementation of multiclass kernel-based vector machines. Journal of Machine Learning Research 2, 265–292 (2002)zbMATHGoogle Scholar
  18. 18.
    Tsochantaridis, I., Joachims, T., Hofmann, T., Altun, Y.: Large margin methods for structured and interdependent output variables. Journal of Machine Learning Research 6, 1453–1484 (2005)MathSciNetzbMATHGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2012

Authors and Affiliations

  • Ürün Dogan
    • 1
  • Tobias Glasmachers
    • 2
  • Christian Igel
    • 3
  1. 1.Institut für MathematikUniversität PotsdamGermany
  2. 2.Institut für NeuroinformatikRuhr-Universität BochumGermany
  3. 3.Department of Computer ScienceUniversity of CopenhagenDenmark

Personalised recommendations