Multi-Class Support Vector Machine

  • Zhe Wang
  • Xiangyang Xue


Support vector machine (SVM) was initially designed for binary classification. To extend SVM to the multi-class scenario, a number of classification models were proposed such as the one by Crammer and Singer (J Mach Learn Res 2:265–292, 2001). However, the number of variables in Crammer and Singer’s dual problem is the product of the number of samples (l) by the number of classes (k), which produces a large computational complexity. This chapter sorts the existing classical techniques for multi-class SVM into the indirect and direct ones and further gives the comparison for them in terms of theory and experiments. Especially, this chapter exhibits a new Simplified Multi-class SVM (SimMSVM) that reduces the size of the resulting dual problem from l × k to l by introducing a relaxed classification error bound. The experimental discussion demonstrates that the SimMSVM approach can greatly speed up the training process, while maintaining a competitive classification accuracy.


  1. 1.
    Asuncion, A., Newman, D.: UCI machine learning repository. (2007)
  2. 2.
    Baldi, P., Pollastri, G.: A machine-learning strategy for protein analysis. IEEE Intell. Syst. 17(2), 28–35 (2002)Google Scholar
  3. 3.
    Bartlett, P., Jordan, M., McAuliffe, J.: Convexity, classification, and risk bounds. J. Am. Stat. Assoc. 101, 138–156 (2006)CrossRefMATHMathSciNetGoogle Scholar
  4. 4.
    Bredensteiner, E., Bennett, K.: Multicategory classification by support vector machines. Comput. Optim. Appl. 12, 53–79 (1999)CrossRefMATHMathSciNetGoogle Scholar
  5. 5.
    Chawla, N.V., Japkowicz, N., Kolcz, A.: Editorial: special issue on learning from imbalanced data sets. ACM SIGKDD Explor. 6(1), 1–6 (2004)CrossRefGoogle Scholar
  6. 6.
    Cortes, C., Vapnik, V.: Support vector networks. Mach. Learn. 20(3), 273–297 (1995)MATHGoogle Scholar
  7. 7.
    Crammer, K., Singer, Y.: On the algorithmic implementation of multiclass kernel-based vector machines. J. Mach. Learn. Res. 2, 265–292 (2001)Google Scholar
  8. 8.
    Dietterich, T., Bakiri, G.: Solving multiclass learning problems via error-correcting output codes. J. Artif. Intell. Res. 2, 263–286 (1995)MATHGoogle Scholar
  9. 9.
    Fung, G., Mangasarian, O.: Proximal support vector machine classifiers. In: Provost, F., Srikant, R. (eds.) Proceedings KDD-2001: Knowledge Discovery and Data Mining, August 26–29, 2001, San Francisco, CA, pp. 77–86. Asscociation for Computing Machinery, New York (2001)Google Scholar
  10. 10.
    Fung, G., Mangasarian, O.: Multicategory proximal support vector machine classifiers. Mach. Learn. 59(1–2), 77–97 (2005)CrossRefMATHGoogle Scholar
  11. 11.
    Ganapathiraju, A., Hamaker, J., Picone, J.: Applications of support vector machines to speech recognition. IEEE Trans. Signal Process. 52(8), 2348–2355 (2004)CrossRefGoogle Scholar
  12. 12.
    Guermeur, Y.: Combining discriminant models with new multi-class svms. Pattern Anal. Appl. 5(2), 168–179 (2002)CrossRefMATHMathSciNetGoogle Scholar
  13. 13.
    He, X., Wang, Z., Jin, C., Zheng, Y., Xue, X.Y.: A simplified multi-class support vector machine with reduced dual optimization. Pattern Recognit. Lett. 33, 71–82 (2012)CrossRefGoogle Scholar
  14. 14.
    Hsu, C., Lin, C.: A comparison of methods for multi-class support vector machines. IEEE Trans. Neural Netw. 13, 415–425 (2002)CrossRefGoogle Scholar
  15. 15.
    Hsu, C., Lin, C.: A simple decomposition method for support vector machines. Mach. Learn. 46, 291–314 (2002)CrossRefMATHGoogle Scholar
  16. 16.
    Hsu, C., Lin, C.: Bsvm. (2008)
  17. 17.
    Hull, J.: A database for handwritten text recognition research. IEEE Trans. Pattern Anal. Mach. Intell. 16(5), 550–554 (1994)CrossRefGoogle Scholar
  18. 18.
    Khan, L., Awad, M., Thuraisingham, B.: A new intrusion detection system using support vector machines and hierarchical clustering. VLDB J. 16(4), 507–521 (2007)CrossRefGoogle Scholar
  19. 19.
    King, R., Feng, C., Sutherland, A.: Statlog: comparison of classification algorithms on large real-world problems. Appl. Artif. Intell. 9(3), 289–333 (1995)CrossRefGoogle Scholar
  20. 20.
    Knerr, S., Personnaz., L., Dreyfus, G.: Single-layer learning revisited: a stepwise procedure for building and training neural network. In: Fogelman, J. (ed.) Neurocomputing: Algorithms, Architectures and Applications. Springer, Berlin (1990)Google Scholar
  21. 21.
    Kreßel, U.: Pairwise classification and support vector machines. In: Schölkopf, B., Burges, C., Smola, A. (eds.) Advances in Kernel Methods: Support Vector Learning, pp. 255–268. MIT Press, Cambridge (1999)Google Scholar
  22. 22.
    Lee, Y., Lin, Y., Wahba, G.: Multicategory support vector machines. In: Wegman, E., Braverman, A., Goodman, A., Smyth, P. (eds.) Computing Science and Statistics, vol. 33, pp. 498–512. Interface Foundation of North America, Inc., Fairfax Station, VA, USA (2002)Google Scholar
  23. 23.
    Lee, Y., Lin, Y., Wahba, G.: Multicategory support vector machines: theory and application to the classification of microarray data and satellite radiance data. J. Am. Stat. Assoc. 99, 67–81 (2004)CrossRefMATHMathSciNetGoogle Scholar
  24. 24.
    Liu, Y.: Fisher consistency of multicategory support vector machines. In: Proceedings of the Eleventh International Conference on Artificial Intelligence and Statistics (AISTATS-07) (2007)Google Scholar
  25. 25.
    Mangasarian, O., Musicant, D.: Successive overrelaxation for support vector machines. IEEE Trans. Neural Netw. 10(5), 1032–1037 (1999)CrossRefGoogle Scholar
  26. 26.
    Mangasarian, O., Musicant, D.: Lagrangian support vector machines. J. Mach. Learn. Res. 1, 161–177 (2001)MATHMathSciNetGoogle Scholar
  27. 27.
    Mori, S., Suen, C., Yamamoto, K.: Historical review of OCR research and development, pp. 244–273. IEEE Computer Society Press, Los Alamitos (1995)Google Scholar
  28. 28.
    Platt, J., Cristianini, N., Shawe-Taylor, J.: Large margin dags for multiclass classification. In: Advances in Neural Information Processing Systems, vol. 12, pp. 547–553. MIT Press, Cambridge (2000)Google Scholar
  29. 29.
    Rifkin, R., Klautau, A.: In defense of one-vs-all classification. J. Mach. Learn. Res. 5, 101–141 (2004)MATHMathSciNetGoogle Scholar
  30. 30.
    Schölkopf, B., Platt, J., Shawe-Taylor, J., Smola, A., Williamson, R.: Estimating the support of a high-dimensional distribution. Neural Comput. 13(7), 1443–1471 (2001)CrossRefMATHGoogle Scholar
  31. 31.
    Suykens, J., Vandewalle, J.: Least squares support vector machine classifiers. Neural Process. Lett. 9(3), 293–300 (1999)CrossRefMathSciNetGoogle Scholar
  32. 32.
    Suykens, J., Vandewalle, J.: Multiclass least squares support vector machines. In: Proceedings of the International Joint Conference on Neural Networks (IJCNN99). World Scientific, Washington, DC (1999)Google Scholar
  33. 33.
    Szedmak, S., Shawe-Taylor, J., Saunders, C., Hardoon, D.: Multiclass classification by l1 norm support vector machine. In: Pattern Recognition and Machine Learning in Computer Vision Workshop (2004)Google Scholar
  34. 34.
    Tang, Y., Zhang, Y., Chawla, N., Krasser, S.: Svms modeling for highly imbalanced classification. IEEE Trans. Syst. Man Cybern. Part B 39(1), 281–288 (2009)CrossRefGoogle Scholar
  35. 35.
    Tax, D., Duin, R.: Data domain description using support vectors. In: Proceedings of the European Symposium on Artificial Neural Networks, pp. 251–256 (1999)Google Scholar
  36. 36.
    Vapnik, V.: Statistical Learning Theory. Wiley, New York (1998)MATHGoogle Scholar
  37. 37.
    Wang, L., Shen, X.: On l1-norm multiclass support vector machines: methodology and theory. J. Am. Stat. Assoc. 102, 583–594 (2007)CrossRefMATHMathSciNetGoogle Scholar
  38. 38.
    Weston, J., Watkins, C.: Multi-class support vector machines. In: Proceedings of ESANN99 (1999)Google Scholar
  39. 39.
    Xia, X., Li, K.: A sparse multi-class least-squares support vector machine. In: IEEE International Symposium on Industrial Electronics, 2008 (ISIE 2008), pp. 1230–1235 (2008)Google Scholar

Copyright information

© Springer International Publishing Switzerland 2014

Authors and Affiliations

  1. 1.Department of Computer Science and EngineeringEast China University of Science and TechnologyShanghaiChina
  2. 2.School of Computer ScienceFudan UniversityShanghaiChina

Personalised recommendations