Tuning L1-SVM Hyperparameters with Modified Radius Margin Bounds and Simulated Annealing
In the design of support vector machines an important step is to select the optimal hyperparameters. One of the most used estimators of the performance is the Radius-Margin bound. Some modifications of this bound have been made to adapt it to soft margin problems, giving a convex optimization problem for the L2 soft margin formulation. However, it is still interesting to consider the L1 case due to the reduction in the support vector number. There have been some proposals to adapt the Radius-Margin bound to the L1 case, but the use of gradient descent to test them is not possible in some of them because these bounds are not differentiable. In this work we propose to use simulated annealing as a method to find the optimal hyperparameters when the bounds are not differentiable, have multiple local minima or the kernel is not differentiable with respect to its hyperparameters.
KeywordsSupport Vector Machine Simulated Annealing Gradient Descent Soft Margin Multiple Local Minimum
Unable to display preview. Download preview PDF.
- 1.Vapnik, N.V.: The Natureof Statistical Learning Theory (1 edn. 1998). Springer, Heidelberg (2000)Google Scholar
- 2.Joachims, T.: Estimating the generalization performance of a SVM efficiently. In: Langley, P. (ed.) Proc. of ICML-00, pp. 431–438. Morgan Kaufmann, San Francisco (2000), citeseer.ist.psu.edu/article/joachims99estimating.html Google Scholar
- 7.Scholkopf, B., Smola, A.J.: Learning with Kernels: Support Vector Machines, Regularization, Optimization, and Beyond. MIT Press, Cambridge (2001)Google Scholar
- 9.Shawe-Taylor, J., Cristianini, N.: Kernel Methods for Pattern Analysis. Cambridge University Press, New York (2004)Google Scholar