Hyper-Parameter Tuning for Support Vector Machines by Estimation of Distribution Algorithms

  • Luis Carlos Padierna
  • Martín Carpio
  • Alfonso Rojas
  • Héctor Puga
  • Rosario Baltazar
  • Héctor Fraire
Chapter
Part of the Studies in Computational Intelligence book series (SCI, volume 667)

Abstract

Hyper-parameter tuning for support vector machines has been widely studied in the past decade. A variety of metaheuristics, such as Genetic Algorithms and Particle Swarm Optimization have been considered to accomplish this task. Notably, exhaustive strategies such as Grid Search or Random Search continue to be implemented for hyper-parameter tuning and have recently shown results comparable to sophisticated metaheuristics. The main reason for the success of exhaustive techniques is due to the fact that only two or three parameters need to be adjusted when working with support vector machines. In this chapter, we analyze two Estimation Distribution Algorithms, the Univariate Marginal Distribution Algorithm and the Boltzmann Univariate Marginal Distribution Algorithm, to verify if these algorithms preserve the effectiveness of Random Search and at the same time make more efficient the process of finding the optimal hyper-parameters without increasing the complexity of Random Search.

Keywords

Parameter tuning Support vector machines Hyper-parameters Estimation of distribution algorithms Pattern classification  

References

  1. 1.
    N. Christianini and J. Shawe-Taylor. An Introduction to SVM and other Kernel Based Methods. Cambridge, U.K.: Cambridge University Press, 2000.Google Scholar
  2. 2.
    V. Vapnik, Statistical Learning Theory, New York: John Wiley and Sons, 1998.Google Scholar
  3. 3.
    M. Kanevski, V. Timonin and A. Pozdnukhov, Machine Learning for Spatial Environmental Data: theory, applications and software, CRC Press, 2009.Google Scholar
  4. 4.
    O. Chapelle, V. Vapnik, O. Bousquet and S. Mukherjee, Choosing Multiple Parameters for Support Vector Machines Machine Learning, vol. 46 (1-3), pp. 131-159, 2002.Google Scholar
  5. 5.
    F. Friedrichs and C. Igel. Evolutionary tuning of multiple SVM parameters. Neurocomputing, vol 64 pp. 107-117, 2005.Google Scholar
  6. 6.
    C. Gold, A. Holub y P. Sollich. Bayesian approach to feature selection and parameter tuning for support vector machine classifiers. Neural Networks, vol 18 (5) pp. 693-701, 2005.Google Scholar
  7. 7.
    C.-L. Huang and W. Chieh-Jen. A GA-based feature selection and parameters optimization for support vector machines. Expert System with Applications, vol 31(2) pp. 231-240, 2006.Google Scholar
  8. 8.
    S.-W. Lin, K.-C. Ying y S.-C. L. Z.-J. Chen. Particle Swarm Optimization for Parameter Determination and Feature Selection of Support Vector Machines. Expert System with Applications, vol 35(4) pp. 1817-1824, 2008.Google Scholar
  9. 9.
    J. Bergstra and Y. Bengio. Random Search for Hyper-Parameter Optimization. Journal of Machine Learning Research, 13(1) pp. 281-305, 2012.Google Scholar
  10. 10.
    R. Mantovani, A. Rossi, J. Vanschoren and B. d.-C. A. Bischl. Effectiveness of Random Search in SVM hyper-parameter tuning. The International Joint Conference on Neural Networks (IJCNN), 2015.Google Scholar
  11. 11.
    A. Shigeo, Support Vector Machines for Pattern Classification, New York: Springer, 2010.Google Scholar
  12. 12.
    N. Deng, Y. Tian and C. Zhang, Support Vector Machines, Boca Raton: CRC Press, 2013.Google Scholar
  13. 13.
    M. Hauschild and M. Pelikan. An introduction and survey of estimation of distribution algorithms. Swarm and Evolutionary Computation, vol 1(3) pp. 111-128, 2011.Google Scholar
  14. 14.
    D. Simon, Evolutionary Optimization Algorithms: Biologically Inspired and Population-Based Approaches to Computer Intelligence, Hoboken: John Wiley and Sons, 2013.Google Scholar
  15. 15.
    S. I. Valdez, A. Hernández and S. Botello. A Boltzmann based estimation of distribution algorithm. Information Sciences, vol 236 pp. 126-137, 2013.Google Scholar

Copyright information

© Springer International Publishing AG 2017

Authors and Affiliations

  • Luis Carlos Padierna
    • 1
  • Martín Carpio
    • 1
  • Alfonso Rojas
    • 1
  • Héctor Puga
    • 1
  • Rosario Baltazar
    • 1
  • Héctor Fraire
    • 2
  1. 1.Tecnológico Nacional de MéxicoInstituto Tecnológico de LeónLeónMexico
  2. 2.Tecnológico Nacional de MéxicoInstituto Tecnológico de Cd. MaderoCiudad MaderoMexico

Personalised recommendations