Bias and Variance Multi-objective Optimization for Support Vector Machines Model Selection

  • Alejandro Rosales-Pérez
  • Hugo Jair Escalante
  • Jesus A. Gonzalez
  • Carlos A. Reyes-Garcia
  • Carlos A. Coello Coello
Part of the Lecture Notes in Computer Science book series (LNCS, volume 7887)


In this paper, we describe a novel model selection approach for a SVM. Each model can be composed by a feature selection method and a pre-processing method besides the classifier. Our approach is based on a multi-objective evolutionary algorithm and on the bias-variance definition. This strategy allows us to explore the hyperparameters space and to select the solutions with the best bias-variance trade-off. The proposed method is evaluated using a number of benchmark data sets for classification tasks. Experimental results show that it is possible to obtain models with an acceptable generalization performance using the proposed approach.


Support vector machines Model selection Bias-variance trade-off Multi-objective optimization 


  1. 1.
    Aydin, I., Karakose, M., Akin, E.: A multi-objective artificial immune algorithm for parameter optimization in support vector machine. Appl. Soft Comput. 11(1), 120–129 (2011)CrossRefGoogle Scholar
  2. 2.
    Chatelain, C., Adam, S., Lecourtier, Y., Heutte, L., Paquet, T.: Multi-objective optimization for svm model selection. In: Ninth ICDAR, vol. 1, pp. 427–431 (2007)Google Scholar
  3. 3.
    Chatelain, C., Adam, S., Lecourtier, Y., Heutte, L., Paquet, T.: A multi-model selection framework for unknown and/or evolutive misclassification cost problems. Pattern Recogn. 43(3), 815–823 (2010)CrossRefMATHGoogle Scholar
  4. 4.
    Coello Coello, C.A., Lamont, G.B., Veldhuizen, D.A.V.: Evolutionary Algorithms for Solving Multi-Objective Problems. In: Genetic and Evolutionary Computation. Springer, US (2007)Google Scholar
  5. 5.
    Deb, K., Agrawal, S., Pratap, A., Meyarivan, T.: A fast elitist non-dominated sorting genetic algorithm for multi-objective optimization: Nsga-ii. In: Deb, K., Rudolph, G., Lutton, E., Merelo, J.J., Schoenauer, M., Schwefel, H.-P., Yao, X. (eds.) PPSN VI. LNCS, vol. 1917, pp. 849–858. Springer, Heidelberg (2000)Google Scholar
  6. 6.
    Demšar, J.: Statistical comparisons of classifiers over multiple data sets. J. Mach. Learn. Res. 7, 1–30 (2006)MathSciNetMATHGoogle Scholar
  7. 7.
    Escalante, H.J., Montes, M., Sucar, L.E.: Particle swarm model selection. J. Mach. Learn. Res. 10, 405–440 (2009)Google Scholar
  8. 8.
    Friedman, J.H.: On bias, variance, 0/1loss, and the curse-of-dimensionality. Data Min. Knowl. Discov. 1, 55–77 (1997)CrossRefGoogle Scholar
  9. 9.
    Ji, Y., Sun, S.: Feature selection for ensembles using non-dominated sorting in genetic algorithms. In: 2010 Sixth International Conference on Natural Computation (ICNC), vol. 2, pp. 888–891 (August 2010)Google Scholar
  10. 10.
    Kohavi, R., Wolpert, D.: Bias plus variance decomposition for zero-one loss functions. In: Proceedings of the 13th ICML, pp. 275–283. Morgan-Kaufmann Publishers (1996)Google Scholar
  11. 11.
    Kong, E., Dietterich, T.: Error-correcting output coding corrects bias and variance. In: Proceedings of the 12th ICML, pp. 313–321. Morgan-Kaufmann Publishers (1995)Google Scholar
  12. 12.
    Li, W., Liu, L., Gong, W.: Multi-objective uniform design as a svm model selection tool for face recognition. Expert Syst. Appl. 38(6), 6689–6695 (2011)CrossRefGoogle Scholar
  13. 13.
    Rätsch, G., Onoda, T., Müller, K.R.: Soft margins for adaboost. Mach. Learn. 42, 287–320 (2001)CrossRefMATHGoogle Scholar
  14. 14.
    Saffari, A., Guyon, I.: Quick start guide for clop. Tech. rep. (May 2006),
  15. 15.
    Suttorp, T., Igel, C.: Multi-objective optimization of support vector machines. In: Jin, Y. (ed.) Multi-Objective Machine Learning. SCI, vol. 16, pp. 199–220. Springer, Heidelberg (2006)CrossRefGoogle Scholar
  16. 16.
    Webb, G.I.: Multiboosting: A technique for combining boosting and wagging. Mach. Learn. 40, 159–196 (2000)CrossRefGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2013

Authors and Affiliations

  • Alejandro Rosales-Pérez
    • 1
  • Hugo Jair Escalante
    • 1
  • Jesus A. Gonzalez
    • 1
  • Carlos A. Reyes-Garcia
    • 1
  • Carlos A. Coello Coello
    • 2
  1. 1.Computer Science DepartmentInstituto Nacional de Astrofísica, Óptica y Electrónica (INAOE)TonantzintlaMexico
  2. 2.Computer Science DepartmentCINVESTAV-IPNMexico CityMexico

Personalised recommendations