Abstract
Subgradient algorithms for training support vector machines have been successful in solving many large-scale and online learning problems. However, for the most part, their applicability has been restricted to linear kernels and strongly convex formulations. This paper describes efficient subgradient approaches without such limitations. Our approaches make use of randomized low-dimensional approximations to nonlinear kernels, and minimization of a reduced primal formulation using an algorithm based on robust stochastic approximation, which does not require strong convexity. Experiments illustrate that our approaches produce solutions of comparable prediction accuracy with the solutions acquired from existing SVM solvers, but often in much shorter time.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Bordes, A., Ertekin, S., Weston, J., Bottou, L.: Fast kernel classifiers with online and active learning. J. Mach. Learn. Res. 6, 1579–1619 (2005)
Boser, B.E., Guyon, I.M., Vapnik, V.N.: A training algorithm for optimal margin classifiers. In: Proceedings of the Fifth Annual Workshop on Computational Learning Theory, pp. 144–152. ACM, New York (1992)
Bottou, L.: SGD: Stochastic gradient descent, http://leon.bottou.org/projects/sgd (2005). Accessed 30 Mar 2012
Chapelle, O.: Training a support vector machine in the primal. Neural Comput. 19, 1155–1178 (2007)
Drineas, P., Mahoney, M.W.: On the Nystrom method for approximating a gram matrix for improved kernel-based learning. J. Mach. Learn. Res. 6, 2153–2175 (2005)
Fan, R.-E., Chen, P.-H., Lin, C.-J.: Working set selection using second order information for training SVM. J. Mach. Learn. Res. 6, 1889–1918 (2005)
Franc, V., Sonnenburg, S.: Optimized cutting plane algorithm for support vector machines. In: Proceedings of the 25th International Conference on Machine Learning, pp. 320–327. ACM, New York (2008)
Joachims, T.: Making large-scale support vector machine learning practical. In: Advances in Kernel Methods – Support Vector Learning, pp. 169–184. MIT, Cambridge (1999)
Joachims, T.: Training linear SVMs in linear time. In: International Conference on Knowledge Discovery and Data Mining, pp. 217–226. ACM, New York (2006)
Joachims, T., Yu, C.-N.J.: Sparse kernel SVMs via cutting-plane training. Mach. Learn. 76(2–3), 179–193 (2009)
Joachims, T., Finley, T., Yu, C.-N.: Cutting-plane training of structural SVMs. Mach. Learn. 77(1), 27–59 (2009)
Kimeldorf, G., Wahba, G.: A correspondence between bayesian estimation on stochastic processes and smoothing by splines. Ann. Math. Statist. 41, 495–502 (1970)
Lewis, D.D., Yang, Y., Rose, T.G., Dietterich, G., Li, F., Li, F.: RCV1: A new benchmark collection for text categorization research. J. Mach. Learn. Res. 5, 361–397 (2004)
Nemirovski, A., Yudin, D.B.: Problem Complexity and Method Efficiency in Optimization. Wiley, New York (1983)
Nemirovski, A., Juditsky, A., Lan, G., Shapiro, A.: Robust stochastic approximation approach to stochastic programming. SIAM J. Optim. 19(4), 1574–1609 (2009)
Rahimi, A., Recht, B.: Random features for large-scale kernel machines. In: Advances in Neural Information Processing Systems, vol. 20, pp. 1177–1184. MIT, Cambridge (2008)
Scholkopf, B., Smola, A.J.: Learning with Kernels: Support Vector Machines, Regularization, Optimization, and Beyond. MIT, Cambridge (2001)
Shalev-Shwartz, S., Singer, Y., Srebro, N.: Pegasos: Primal estimated sub-GrAdient SOlver for SVM. In: Proceedings of the 24th International Conference on Machine Learning, pp. 807–814. ACM, New York (2007)
Shalev-Shwartz, S., Singer, Y., Srebro, N., Cotter, A.: Pegasos: Primal estimated sub-GrAdient SOlver for SVM. Math. Program. Ser. B 127(1), 3–30 (2011)
Zinkevich, M.: Online convex programming and generalized infinitesimal gradient ascent. In: Proceedings of the 20th International Conference on Machine Learning, pp. 928–936. ACM, New York (2003)
Acknowledgements
The authors acknowledge the support of NSF Grants DMS-0914524 and DMS-0906818, and in part by Deutsche Forschungsgemeinschaft (DFG) within the Collaborative Research Center SFB 876 “Providing Information by Resource-Constrained Analysis,” project C1.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2013 Springer Science+Business Media New York
About this paper
Cite this paper
Lee, S., Wright, S.J. (2013). Stochastic Subgradient Estimation Training for Support Vector Machines. In: Latorre Carmona, P., Sánchez, J., Fred, A. (eds) Mathematical Methodologies in Pattern Recognition and Machine Learning. Springer Proceedings in Mathematics & Statistics, vol 30. Springer, New York, NY. https://doi.org/10.1007/978-1-4614-5076-4_5
Download citation
DOI: https://doi.org/10.1007/978-1-4614-5076-4_5
Published:
Publisher Name: Springer, New York, NY
Print ISBN: 978-1-4614-5075-7
Online ISBN: 978-1-4614-5076-4
eBook Packages: Mathematics and StatisticsMathematics and Statistics (R0)