Optimal arrangements of hyperplanes for SVM-based multiclass classification

  • Víctor BlancoEmail author
  • Alberto Japón
  • Justo Puerto
Regular Article


In this paper, we present a novel SVM-based approach to construct multiclass classifiers by means of arrangements of hyperplanes. We propose different mixed integer (linear and non linear) programming formulations for the problem using extensions of widely used measures for misclassifying observations where the kernel trick can be adapted to be applicable. Some dimensionality reductions and variable fixing strategies are also developed for these models. An extensive battery of experiments has been run which reveal the powerfulness of our proposal as compared with other previously proposed methodologies.


Multiclass support vector machines Mixed integer non linear programming Classification, hyperplanes 

Mathematics Subject Classification

62H30 90C11 68T05 32S22 



The authors were partially supported by the research Project MTM2016-74983-C2-1-R (MINECO, Spain). The first author has been also supported by Project PP2016-PIP06 (Universidad de Granada) and the research group SEJ-534 (Junta de Andalucía). We would also like to thank the three anonymous referees for their helpful and constructive comments that greatly contributed to improving the final version of the paper.


  1. Agarwal N, Balasubramanian VN, Jawahar C (2018) Improving multiclass classification by deep networks using DAGSVM and triplet loss. Pattern Recognit Lett 112:184–190Google Scholar
  2. Allwein EL, Schapire RE, Singer Y (2001) Reducing multiclass to binary. Reducing multiclass to binary: a unifying approach for margin classifiers. J Mach Learn Res 1:113–141MathSciNetzbMATHGoogle Scholar
  3. Bagirov AM, Ugon J, Webb D, Ozturk G, Kasimbeyli R (2013) A novel piecewise linear classifier based on polyhedral conic and max-min separabilities. TOP 21(1):3–24MathSciNetzbMATHGoogle Scholar
  4. Bahlmann C, Haasdonk B, Burkhardt H (2002) On-line handwriting recognition with support vector machines-a kernel approach. In: Eighth international workshop on frontiers in handwriting recognition, pp 49–54Google Scholar
  5. Benders JF (1962) Partitioning procedures for solving mixed-variables programming problems. Numerische Mathematik 4(1):238–252MathSciNetzbMATHGoogle Scholar
  6. Benítez-Peña S, Blanquero R, Carrizosa E, Ramírez-Cobo P (2018) On support vector machines under a multiple-cost scenario. Adv Data Anal Classif. zbMATHGoogle Scholar
  7. Bennett KP, Demiriz A (1999) Semi-supervised support vector machines. Adv Neural Inf Process Syst 11:368–374Google Scholar
  8. Blanco V, Ben Ali S, Puerto J (2014) Revisiting several problems and algorithms in continuous location with \(\ell _p\) norms. Comput Optim Appl 58(3):563–595MathSciNetzbMATHGoogle Scholar
  9. Blanco V, Puerto J, Salmerón R (2018) Locating hyperplanes to fitting set of points: a general framework. Comput Oper Res 95:172–193MathSciNetzbMATHGoogle Scholar
  10. Blanco V, Japón A, Puerto J (2019) Optimal arrangements of hyperplanes for multiclass classification. arXiv preprint: arXiv:1810.09167
  11. Blanco V, Puerto J, Rodríguez-Chía A M (2019) On \(\ell _p \)-Support Vector Machines and Multidimensional Kernels. arXiv preprint: arXiv:1711.10332
  12. Cortes C, Vapnik V (1995) Support-vector networks. Mach Learn 20(3):273–297zbMATHGoogle Scholar
  13. Cover T, Hart P (1967) Nearest neighbor pattern classification. IEEE Trans Inf Theory 13:21–27zbMATHGoogle Scholar
  14. Crammer K, Singer Y (2001) On the algorithmic implementation of multiclass kernel-based vector machines. J Mach Learn Res 2:265–292zbMATHGoogle Scholar
  15. Dietterich TG, Bakiri G (1995) Solving multiclass learning problems via error-correcting output codes. J Artif Intell Res 2:263–286zbMATHGoogle Scholar
  16. Geoffrion AM (1972) Generalized Benders decomposition. J Optim Theory Appl 10(4):237–260MathSciNetzbMATHGoogle Scholar
  17. Ghaddar B, Naoum-Sawaya J (2018) High dimensional data classification and feature selection using support vector machines. Eur J Oper Res 265(3):993–1004MathSciNetzbMATHGoogle Scholar
  18. Guermeur Y, Monfrini E (2011) A quadratic loss multi-class SVM for which a radius-margin bound applies. Informatica 22(1):73–96MathSciNetzbMATHGoogle Scholar
  19. Harris T (2013) Quantitative credit risk assessment using support vector machines: broad versus narrow default definitions. Expert Syst Appl 40(11):4404–4413Google Scholar
  20. Horn D, Demircioglu A, Bischl B, Glasmachers T, Weihs C (2016) A comparative study on large scale kernelized support vector machines. Adv Data Anal Classif 12(4):867–883MathSciNetzbMATHGoogle Scholar
  21. Ikeda K, Murata N (2005) Geometrical properties of Nu support vector machines with different norms. Neural Comput 17(11):2508–2529MathSciNetzbMATHGoogle Scholar
  22. Ikeda K, Murata N (2005) Effects of norms on learning properties of support vector machines. Proc IEEE Int Conf Acoust Speech Signal Process 5:241–244Google Scholar
  23. Kašćelan V, Kašćelan L, Novović Burić M (2016) A nonparametric data mining approach for risk prediction in car insurance: a case study from the montenegrin market. Econ Res Ekonomska istraživanja 29(1):545–558Google Scholar
  24. Labbé M, Martínez-Merino LI, Rodríguez-Chía AM (2018) Mixed integer linear programming for feature selection in support vector machine. Discrete Appl Math 261:276–304MathSciNetzbMATHGoogle Scholar
  25. Lauer F, Guermeur Y (2011) MSVMpack: a multi-class support vector machine package. J Mach Learn Res 12:2269–2272MathSciNetzbMATHGoogle Scholar
  26. Lee Y, Lin Y, Wahba G (2004) Multicategory support vector machines: theory and application to the classification of microarray data and satellite radiance data. J Am Stat Assoc 99(465):67–81MathSciNetzbMATHGoogle Scholar
  27. Lewis D.D (1998) Naive (bayes) at forty: the independence assumption in information retrieval. In: European conference on machine learning, pp 4–15Google Scholar
  28. Lichman M (2013) UCI machine learning repository. Accessed Oct 2018
  29. López J, Maldonado S, Carrasco M (2018) Double regularization methods for robust feature selection and SVM classification via DC programming. Inf Sci 429:377–389MathSciNetGoogle Scholar
  30. Majid A, Ali S, Iqbal M, Kausar N (2014) Prediction of human breast and colon cancers from imbalanced data using nearest neighbor and support vector machines. Comput Methods Progr Biomed 113(3):792–808Google Scholar
  31. Maldonado S, Pérez J, Weber R, Labbé M (2014) Feature selection for support vector machines via mixed integer linear programming. Inf Sci 279:163–175MathSciNetzbMATHGoogle Scholar
  32. Mangasarian OL (1999) Arbitrary-norm separating plane. Oper Res Lett 24(1–2):15–23MathSciNetzbMATHGoogle Scholar
  33. Meyer D, Dimitriadou E, Hornik K, Weingessel A, Leisch F (2017) e1071: misc functions of the department of statistics, probability theory group (Formerly: E1071), TU Wien. R package version 1.6-8. Accessed Oct 2018
  34. Ortigosa-Hernández J, Inza I, Lozano JA (2016) Semisupervised multiclass classification problems with scarcity of labeled data: a theoretical study. IEEE Trans Neural Netw Learn Syst 27(12):2602–2614MathSciNetGoogle Scholar
  35. Pedregosa F, Varoquaux G, Gramfort A, Michel V, Thirion B, Grisel O, Blondel M, Prettenhofer P, Weiss R, Dubourg V, Vanderplas J, Passos A, Cournapeau D, Brucher M, Perrot M, Duchesnay E (2011) Scikit-learn: machine learning in Python. J Mach Learn Res 12:2825–2830MathSciNetzbMATHGoogle Scholar
  36. Platt JC, Cristianini N, Shawe-Taylor J (2000) Large margin DAGs for multiclass classification. In: Solla SA, Leen TK, Mülle K (eds) Advances in neural information processing systems, vol 12. The MIT Press, Cambridge, pp 547–553Google Scholar
  37. Radhimeenakshi S (2016) Classification and prediction of heart disease risk using data mining techniques of support vector machine and artificial neural network. In: International conference on computing for sustainable global development, INDIACom, pp 3107–3111Google Scholar
  38. Tang X, Xu A (2016) Multi-class classification using kernel density estimation on k-nearest neighbours. Electron Lett 52(8):600–602Google Scholar
  39. Üney F, Türkay M (2006) A mixed-integer programming approach to multi-class data classification problem. Eur J Oper Res 173(3):910–920MathSciNetzbMATHGoogle Scholar
  40. van den Burg GJJ, Groenen PJF (2016) GenSVM: a generalized multiclass support vector machine. J Mach Learn Res 17(225):1–42MathSciNetzbMATHGoogle Scholar
  41. Weston J, Watkins C (1999) Support vector machines for multi-class pattern recognition. In: European symposium on artificial neural networks, pp 219–224Google Scholar

Copyright information

© Springer-Verlag GmbH Germany, part of Springer Nature 2019

Authors and Affiliations

  1. 1.IEMath-GRUniversidad de GranadaGranadaSpain
  2. 2.IMUSUniversidad de SevillaSevillaSpain

Personalised recommendations