Soft Computing

, Volume 18, Issue 10, pp 1985–1998 | Cite as

Electromagnetism-like algorithm for support vector machine parameter tuning

  • Aleksandar Kartelj
  • Nenad Mitić
  • Vladimir Filipović
  • Dušan Tošić
Methodologies and Application

Abstract

This paper introduces an electromagnetism-like (EM) approach for solving the problem of parameter tuning in the support vector machine (SVM). The proposed method is used to tune binary SVM classifiers in single and multiple kernel mode. The internal kernel structure is based on linear and radial basis functions (RBF). An appropriate encoding scheme of EM enables easy transformation of real-valued EM points directly to real-valued parameter combinations. Estimations of the generalization error based on the cross-validation and validation set error are used as objective functions. The efficient local search procedure uses variable size interval movement in order to improve the convergence of the method. The quality of the proposed method is tested on four collections of testing benchmarks through five separate experiments. The first three collections consist of small-size to medium-size classification data sets with up to 60 features and 1,300 training vectors, while the fourth collection is formed of large heterogeneous data sets with up to 1,554 features and 2,186 training vectors. The obtained results indicate that EM outperforms the comparison algorithms in 10 out of 13 instances from the first collection, 5 out of 5 instances from the second, and 13 out of 15 instances from the third collection. The last two experiments, conducted on the fourth collection, show that the proposed method outperforms 14 successful methods in 3 out of 5 data sets where RBF multiple kernel learning is used, and behaves competitively in cases when linear kernels are used.

Keywords

SVM parameter tuning  Electromagnetism-like metaheuristic Classification 

References

  1. Ali M, Golalikhani M (2010) An electromagnetism-like method for nonlinearly constrained global optimization. Comput Math Appl 60(8):2279–2285CrossRefMATHMathSciNetGoogle Scholar
  2. Allwein EL, Schapire RE, Singer Y (2001) Reducing multiclass to binary: a unifying approach for margin classifiers. J Mach Learn Res 1:113–141MATHMathSciNetGoogle Scholar
  3. Aydin I, Karakose M, Akin E (2011) A multi-objective artificial immune algorithm for parameter optimization in support vector machine. Appl Soft Comput 11(1):120–129CrossRefGoogle Scholar
  4. Bach FR, Lanckriet GRG, Jordan MI (2004) Multiple kernel learning, conic duality, and the SMO algorithm. In: Proceeding of 21st International Conference on Machine Learning, ACM, New York, NY, USA, ICML ’04, pp 6–6Google Scholar
  5. Barbero Jiménez A, López Lázaro J, Dorronsoro JR (2009) Finding optimal model parameters by deterministic and annealed focused grid search. Neurocomput 72(13–15):2824–2832CrossRefGoogle Scholar
  6. Birbil SI, Fang SC (2003) An electromagnetism-like mechanism for global optimization. J Global Optim 25:263–282CrossRefMATHMathSciNetGoogle Scholar
  7. Birbil SI, Fang SC, Sheu RL (2004) On the convergence of a population-based global optimization algorithm. J Global Optim 30:301–318CrossRefMATHMathSciNetGoogle Scholar
  8. Boser BE, Guyon IM, Vapnik VN (1992) A training algorithm for optimal margin classifiers. In: Proceedings of 5th Annual ACM Workshop Computational Learning Theory, ACM Press, pp 144–152Google Scholar
  9. Campbell C, Ying Y (2011) Learning with support vector machines. Synth Lect Artif Intell Mach Learn 5(1):1–95CrossRefGoogle Scholar
  10. Carrizosa E, Martín-Barragán B, Romero Morales D (2012) Variable neighborhood search for parameter tuning in support vector machines. Tech. rep.Google Scholar
  11. Chapelle O, Vapnik V, Bousquet O, Mukherjee S (2002) Choosing multiple parameters for support vector machines. Mach Learn 46:131–159CrossRefMATHGoogle Scholar
  12. Conforti D, Guido R (2010) Kernel based support vector machine via semidefinite programming: application to medical diagnosis. Comput Oper Res 37(8):1389–1394CrossRefMATHGoogle Scholar
  13. Cortes C, Vapnik V (1995) Support-vector networks. Mach Learn 20:273–297MATHGoogle Scholar
  14. Črepinšek M, Liu SH, Mernik L (2012) A note on teaching-learning-based optimization algorithm. Inform Sci 212:79–93CrossRefGoogle Scholar
  15. Cuevas E, Oliva D, Zaldivar D, Prez-Cisneros M, Sossa H, (2012) Circle detection using electro-magnetism optimization. Inf Sci 182(1):40–55Google Scholar
  16. Diebold FX, Mariano RS (2002) Comparing predictive accuracy. J Bus Econ Stat 20(1)Google Scholar
  17. Duan K, Keerthi S, Poo AN (2003) Evaluation of simple performance measures for tuning svm hyperparameters. Neurocomput 51:41– 59Google Scholar
  18. Filipović V (2011) An electromagnetism metaheuristic for the uncapacitated multiple allocation hub location problem. Serdica J Comput 5(3):261–272Google Scholar
  19. Filipović V, Kartelj A, Matić D (2013) An electromagnetism metaheuristic for solving the maximum betweenness problem. Appl Soft Comput 13(2):1303–1313CrossRefGoogle Scholar
  20. Franc V, Hlaváč V (2003) Greedy algorithm for a training set reduction in the kernel methods. In: Computer Analysis of Image and Pattern. Springer, Berlin, pp 426–433Google Scholar
  21. Frank A, Asuncion A (2010) UCI machine learning repository. http://archive.ics.uci.edu/ml
  22. Friedrichs F, Igel C (2005) Evolutionary tuning of multiple SVM parameters. Neurocomput 64:107–117CrossRefGoogle Scholar
  23. Gascón-Moreno J, Ortiz-García E, Salcedo-Sanz S, Paniagua-Tineo A, Saavedra-Moreno B, Portilla-Figueras J (2011) Multi-parametric gaussian kernel function optimization for \(\varepsilon \)-SVMr using a genetic algorithm. Adv Comput Intell 113–120Google Scholar
  24. Gascón-Moreno J, Ortiz-García E, Salcedo-Sanz S, Carro-Calvo L, Saavedra-Moreno B, Portilla-Figueras A (2013) Evolutionary optimization of multi-parametric kernel \(\varepsilon \)-SVMr for forecasting problems. Soft Comput 17(2):213–221CrossRefGoogle Scholar
  25. Gascón-Moreno J, Ortiz-García E, Salcedo-Sanz S, Paniagua-Tineo A, Saavedra-Moreno B, Portilla-Figueras J (2013) Multi-parametric gaussian kernel function optimization for \(\varepsilon \)-\(\text{ SVM }_{{\rm r}}\) using a genetic algorithm. In: Cabestany J, Rojas I, Joya G (eds) Advances in Computational Intelligence. Lecture notes in computer science, 6692, Springer, Heidelberg, pp 113–120Google Scholar
  26. Hung WM, Hong WC (2009) Application of svr with improved ant colony optimization algorithms in exchange rate forecasting. Control Cybern 38(3):863–891Google Scholar
  27. Imbault F, Lebart K (2004) A stochastic optimization approach for parameter tuning of support vector machines. In: Proceedings of 17th International Conference on Pattern Recognition, vol 4, pp 597– 600Google Scholar
  28. Joachims T (2000) Estimating the generalization performance of an SVM efficiently. In: Proc of the 17th International Conference on Machine LearningGoogle Scholar
  29. Kartelj A (2012) Electromagnetism metaheuristic algorithm for solving the strong minimum energy topology problem. Yug J Oper Res 22(2)Google Scholar
  30. Kecman V (2001) Learning and soft computing: support vector machines, neural networks, and fuzzy logic models. MIT press, CambridgeGoogle Scholar
  31. Keerthi S (2002) Efficient tuning of svm hyperparameters using radius/margin bound and iterative algorithms. IEEE Trans Neural Netw 13(5):1225–1229CrossRefGoogle Scholar
  32. Keerthi SS, Lin CJ (2003) Asymptotic behaviors of support vector machines with gaussian kernel. Neural Comput 15(7):1667–1689CrossRefMATHGoogle Scholar
  33. Lavesson N, Davidsson P (2006) Quantifying the impact of learning algorithm parameter tuning. In: Proceedings of the 21st National Conference on Artificial Intelligence, pp 395–400Google Scholar
  34. Luntz A, Brailovsky V (1969) On estimation of characters obtained in statistical procedure of recognition. Technicheskaya Kibernetica 3Google Scholar
  35. Muller KR, Mika S, Ratsch G, Tsuda K, Scholkopf B (2001) An introduction to kernel-based learning algorithms. IEEE Trans Neural Netw 12(2):181–201CrossRefGoogle Scholar
  36. Naji-Azimi Z, Toth P, Galli L (2010) An electromagnetism metaheuristic for the unicost set covering problem. Eur J Oper Res 205(2):290–300CrossRefMATHMathSciNetGoogle Scholar
  37. Phienthrakul T, Kijsirikul B (2010) Evolutionary strategies for hyperparameters of support vector machines based on multi-scale radial basis function kernels. Soft Comput 14(7):681–699CrossRefGoogle Scholar
  38. Rätsch G, Onoda T, Müller KR (2001) Soft margins for adaboost. Mach Learn 42:287–320CrossRefMATHGoogle Scholar
  39. Samadzadegan F, Soleymani A, Abbaspour R (2010) Evaluation of genetic algorithms for tuning svm parameters in multi-class problems. In: Proceedings of the 11th International Symposium on Computational Intellingence and Information, pp 323–328Google Scholar
  40. Shawe-Taylor J, Cristianini N (2004) Kernel methods for pattern analysis. Cambridge university press, CambridgeGoogle Scholar
  41. Sonnenburg S, Rätsch G, Schäfer C, Schölkopf B (2006) Large scale multiple kernel learning. J Mach Learn Res 7:1531–1565MATHMathSciNetGoogle Scholar
  42. Su CT, Lin HC (2011) Applying electromagnetism-like mechanism for feature selection. Inf Sci 181(5):972–986CrossRefGoogle Scholar
  43. Tavakkoli-Moghaddam R, Khalili M, Naderi B (2009) A hybridization of simulated annealing and electromagnetic-like mechanism for job shop problems with machine availability and sequence-dependent setup times to minimize total weighted tardiness. Soft Comput 13(10):995–1006 Google Scholar
  44. Vapnik V (1995) The nature of statistical learning theory. Springer, BerlinGoogle Scholar
  45. Vapnik VN (1999) An overview of statistical learning theory. IEEE Trans Neural Netw 10(5):988–999CrossRefGoogle Scholar
  46. Yurtkuran A, Emel E (2010) A new hybrid electromagnetism-like algorithm for capacitated vehicle routing problems. Expert Syst Appl 37(4):3427–3433CrossRefGoogle Scholar
  47. Zhang X, Chen X, He Z (2010) An ACO-based algorithm for parameter optimization of support vector machines. Expert Syst Appl 37(9):6618–6628CrossRefGoogle Scholar
  48. Zhiqiang G, Huaiqing W, Quan L (2013) Financial time series forecasting using LPP and SVM optimized by PSO. Soft Comput 17(5):805–818CrossRefGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2013

Authors and Affiliations

  • Aleksandar Kartelj
    • 1
  • Nenad Mitić
    • 1
  • Vladimir Filipović
    • 1
  • Dušan Tošić
    • 1
  1. 1.Faculty of MathematicsUniversity of BelgradeBelgradeSerbia

Personalised recommendations