Stochastic Subgradient Estimation Training for Support Vector Machines
Subgradient algorithms for training support vector machines have been successful in solving many large-scale and online learning problems. However, for the most part, their applicability has been restricted to linear kernels and strongly convex formulations. This paper describes efficient subgradient approaches without such limitations. Our approaches make use of randomized low-dimensional approximations to nonlinear kernels, and minimization of a reduced primal formulation using an algorithm based on robust stochastic approximation, which does not require strong convexity. Experiments illustrate that our approaches produce solutions of comparable prediction accuracy with the solutions acquired from existing SVM solvers, but often in much shorter time.
The authors acknowledge the support of NSF Grants DMS-0914524 and DMS-0906818, and in part by Deutsche Forschungsgemeinschaft (DFG) within the Collaborative Research Center SFB 876 “Providing Information by Resource-Constrained Analysis,” project C1.
- 2.Boser, B.E., Guyon, I.M., Vapnik, V.N.: A training algorithm for optimal margin classifiers. In: Proceedings of the Fifth Annual Workshop on Computational Learning Theory, pp. 144–152. ACM, New York (1992)Google Scholar
- 3.Bottou, L.: SGD: Stochastic gradient descent, http://leon.bottou.org/projects/sgd (2005). Accessed 30 Mar 2012
- 7.Franc, V., Sonnenburg, S.: Optimized cutting plane algorithm for support vector machines. In: Proceedings of the 25th International Conference on Machine Learning, pp. 320–327. ACM, New York (2008)Google Scholar
- 8.Joachims, T.: Making large-scale support vector machine learning practical. In: Advances in Kernel Methods – Support Vector Learning, pp. 169–184. MIT, Cambridge (1999)Google Scholar
- 9.Joachims, T.: Training linear SVMs in linear time. In: International Conference on Knowledge Discovery and Data Mining, pp. 217–226. ACM, New York (2006)Google Scholar
- 13.Lewis, D.D., Yang, Y., Rose, T.G., Dietterich, G., Li, F., Li, F.: RCV1: A new benchmark collection for text categorization research. J. Mach. Learn. Res. 5, 361–397 (2004)Google Scholar
- 14.Nemirovski, A., Yudin, D.B.: Problem Complexity and Method Efficiency in Optimization. Wiley, New York (1983)Google Scholar
- 16.Rahimi, A., Recht, B.: Random features for large-scale kernel machines. In: Advances in Neural Information Processing Systems, vol. 20, pp. 1177–1184. MIT, Cambridge (2008)Google Scholar
- 17.Scholkopf, B., Smola, A.J.: Learning with Kernels: Support Vector Machines, Regularization, Optimization, and Beyond. MIT, Cambridge (2001)Google Scholar
- 18.Shalev-Shwartz, S., Singer, Y., Srebro, N.: Pegasos: Primal estimated sub-GrAdient SOlver for SVM. In: Proceedings of the 24th International Conference on Machine Learning, pp. 807–814. ACM, New York (2007)Google Scholar
- 20.Zinkevich, M.: Online convex programming and generalized infinitesimal gradient ascent. In: Proceedings of the 20th International Conference on Machine Learning, pp. 928–936. ACM, New York (2003)Google Scholar