Abstract
This paper introduces an electromagnetism-like (EM) approach for solving the problem of parameter tuning in the support vector machine (SVM). The proposed method is used to tune binary SVM classifiers in single and multiple kernel mode. The internal kernel structure is based on linear and radial basis functions (RBF). An appropriate encoding scheme of EM enables easy transformation of real-valued EM points directly to real-valued parameter combinations. Estimations of the generalization error based on the cross-validation and validation set error are used as objective functions. The efficient local search procedure uses variable size interval movement in order to improve the convergence of the method. The quality of the proposed method is tested on four collections of testing benchmarks through five separate experiments. The first three collections consist of small-size to medium-size classification data sets with up to 60 features and 1,300 training vectors, while the fourth collection is formed of large heterogeneous data sets with up to 1,554 features and 2,186 training vectors. The obtained results indicate that EM outperforms the comparison algorithms in 10 out of 13 instances from the first collection, 5 out of 5 instances from the second, and 13 out of 15 instances from the third collection. The last two experiments, conducted on the fourth collection, show that the proposed method outperforms 14 successful methods in 3 out of 5 data sets where RBF multiple kernel learning is used, and behaves competitively in cases when linear kernels are used.
Similar content being viewed by others
Notes
Data sets available at: http://mldata.org/repository/tags/data/IDA_Benchmark_Repository/.
References
Ali M, Golalikhani M (2010) An electromagnetism-like method for nonlinearly constrained global optimization. Comput Math Appl 60(8):2279–2285
Allwein EL, Schapire RE, Singer Y (2001) Reducing multiclass to binary: a unifying approach for margin classifiers. J Mach Learn Res 1:113–141
Aydin I, Karakose M, Akin E (2011) A multi-objective artificial immune algorithm for parameter optimization in support vector machine. Appl Soft Comput 11(1):120–129
Bach FR, Lanckriet GRG, Jordan MI (2004) Multiple kernel learning, conic duality, and the SMO algorithm. In: Proceeding of 21st International Conference on Machine Learning, ACM, New York, NY, USA, ICML ’04, pp 6–6
Barbero Jiménez A, López Lázaro J, Dorronsoro JR (2009) Finding optimal model parameters by deterministic and annealed focused grid search. Neurocomput 72(13–15):2824–2832
Birbil SI, Fang SC (2003) An electromagnetism-like mechanism for global optimization. J Global Optim 25:263–282
Birbil SI, Fang SC, Sheu RL (2004) On the convergence of a population-based global optimization algorithm. J Global Optim 30:301–318
Boser BE, Guyon IM, Vapnik VN (1992) A training algorithm for optimal margin classifiers. In: Proceedings of 5th Annual ACM Workshop Computational Learning Theory, ACM Press, pp 144–152
Campbell C, Ying Y (2011) Learning with support vector machines. Synth Lect Artif Intell Mach Learn 5(1):1–95
Carrizosa E, Martín-Barragán B, Romero Morales D (2012) Variable neighborhood search for parameter tuning in support vector machines. Tech. rep.
Chapelle O, Vapnik V, Bousquet O, Mukherjee S (2002) Choosing multiple parameters for support vector machines. Mach Learn 46:131–159
Conforti D, Guido R (2010) Kernel based support vector machine via semidefinite programming: application to medical diagnosis. Comput Oper Res 37(8):1389–1394
Cortes C, Vapnik V (1995) Support-vector networks. Mach Learn 20:273–297
Črepinšek M, Liu SH, Mernik L (2012) A note on teaching-learning-based optimization algorithm. Inform Sci 212:79–93
Cuevas E, Oliva D, Zaldivar D, Prez-Cisneros M, Sossa H, (2012) Circle detection using electro-magnetism optimization. Inf Sci 182(1):40–55
Diebold FX, Mariano RS (2002) Comparing predictive accuracy. J Bus Econ Stat 20(1)
Duan K, Keerthi S, Poo AN (2003) Evaluation of simple performance measures for tuning svm hyperparameters. Neurocomput 51:41– 59
Filipović V (2011) An electromagnetism metaheuristic for the uncapacitated multiple allocation hub location problem. Serdica J Comput 5(3):261–272
Filipović V, Kartelj A, Matić D (2013) An electromagnetism metaheuristic for solving the maximum betweenness problem. Appl Soft Comput 13(2):1303–1313
Franc V, Hlaváč V (2003) Greedy algorithm for a training set reduction in the kernel methods. In: Computer Analysis of Image and Pattern. Springer, Berlin, pp 426–433
Frank A, Asuncion A (2010) UCI machine learning repository. http://archive.ics.uci.edu/ml
Friedrichs F, Igel C (2005) Evolutionary tuning of multiple SVM parameters. Neurocomput 64:107–117
Gascón-Moreno J, Ortiz-García E, Salcedo-Sanz S, Paniagua-Tineo A, Saavedra-Moreno B, Portilla-Figueras J (2011) Multi-parametric gaussian kernel function optimization for \(\varepsilon \)-SVMr using a genetic algorithm. Adv Comput Intell 113–120
Gascón-Moreno J, Ortiz-García E, Salcedo-Sanz S, Carro-Calvo L, Saavedra-Moreno B, Portilla-Figueras A (2013) Evolutionary optimization of multi-parametric kernel \(\varepsilon \)-SVMr for forecasting problems. Soft Comput 17(2):213–221
Gascón-Moreno J, Ortiz-García E, Salcedo-Sanz S, Paniagua-Tineo A, Saavedra-Moreno B, Portilla-Figueras J (2013) Multi-parametric gaussian kernel function optimization for \(\varepsilon \)-\(\text{ SVM }_{{\rm r}}\) using a genetic algorithm. In: Cabestany J, Rojas I, Joya G (eds) Advances in Computational Intelligence. Lecture notes in computer science, 6692, Springer, Heidelberg, pp 113–120
Hung WM, Hong WC (2009) Application of svr with improved ant colony optimization algorithms in exchange rate forecasting. Control Cybern 38(3):863–891
Imbault F, Lebart K (2004) A stochastic optimization approach for parameter tuning of support vector machines. In: Proceedings of 17th International Conference on Pattern Recognition, vol 4, pp 597– 600
Joachims T (2000) Estimating the generalization performance of an SVM efficiently. In: Proc of the 17th International Conference on Machine Learning
Kartelj A (2012) Electromagnetism metaheuristic algorithm for solving the strong minimum energy topology problem. Yug J Oper Res 22(2)
Kecman V (2001) Learning and soft computing: support vector machines, neural networks, and fuzzy logic models. MIT press, Cambridge
Keerthi S (2002) Efficient tuning of svm hyperparameters using radius/margin bound and iterative algorithms. IEEE Trans Neural Netw 13(5):1225–1229
Keerthi SS, Lin CJ (2003) Asymptotic behaviors of support vector machines with gaussian kernel. Neural Comput 15(7):1667–1689
Lavesson N, Davidsson P (2006) Quantifying the impact of learning algorithm parameter tuning. In: Proceedings of the 21st National Conference on Artificial Intelligence, pp 395–400
Luntz A, Brailovsky V (1969) On estimation of characters obtained in statistical procedure of recognition. Technicheskaya Kibernetica 3
Muller KR, Mika S, Ratsch G, Tsuda K, Scholkopf B (2001) An introduction to kernel-based learning algorithms. IEEE Trans Neural Netw 12(2):181–201
Naji-Azimi Z, Toth P, Galli L (2010) An electromagnetism metaheuristic for the unicost set covering problem. Eur J Oper Res 205(2):290–300
Phienthrakul T, Kijsirikul B (2010) Evolutionary strategies for hyperparameters of support vector machines based on multi-scale radial basis function kernels. Soft Comput 14(7):681–699
Rätsch G, Onoda T, Müller KR (2001) Soft margins for adaboost. Mach Learn 42:287–320
Samadzadegan F, Soleymani A, Abbaspour R (2010) Evaluation of genetic algorithms for tuning svm parameters in multi-class problems. In: Proceedings of the 11th International Symposium on Computational Intellingence and Information, pp 323–328
Shawe-Taylor J, Cristianini N (2004) Kernel methods for pattern analysis. Cambridge university press, Cambridge
Sonnenburg S, Rätsch G, Schäfer C, Schölkopf B (2006) Large scale multiple kernel learning. J Mach Learn Res 7:1531–1565
Su CT, Lin HC (2011) Applying electromagnetism-like mechanism for feature selection. Inf Sci 181(5):972–986
Tavakkoli-Moghaddam R, Khalili M, Naderi B (2009) A hybridization of simulated annealing and electromagnetic-like mechanism for job shop problems with machine availability and sequence-dependent setup times to minimize total weighted tardiness. Soft Comput 13(10):995–1006
Vapnik V (1995) The nature of statistical learning theory. Springer, Berlin
Vapnik VN (1999) An overview of statistical learning theory. IEEE Trans Neural Netw 10(5):988–999
Yurtkuran A, Emel E (2010) A new hybrid electromagnetism-like algorithm for capacitated vehicle routing problems. Expert Syst Appl 37(4):3427–3433
Zhang X, Chen X, He Z (2010) An ACO-based algorithm for parameter optimization of support vector machines. Expert Syst Appl 37(9):6618–6628
Zhiqiang G, Huaiqing W, Quan L (2013) Financial time series forecasting using LPP and SVM optimized by PSO. Soft Comput 17(5):805–818
Acknowledgments
This work is supported by the Ministry of Education, Science and Technological Development, Republic of Serbia under Grant Numbers: 174010, 174021 and 44006. The authors would like to thank Tanasanee Phienthrakul and Boonserm Kijsirikul for making their benchmark data sets available.
Author information
Authors and Affiliations
Corresponding author
Additional information
Communicated by V. Piuri.
Rights and permissions
About this article
Cite this article
Kartelj, A., Mitić, N., Filipović, V. et al. Electromagnetism-like algorithm for support vector machine parameter tuning. Soft Comput 18, 1985–1998 (2014). https://doi.org/10.1007/s00500-013-1180-x
Published:
Issue Date:
DOI: https://doi.org/10.1007/s00500-013-1180-x