In this paper, we are interested in the analysis of regularized online algorithms associated with reproducing kernel Hilbert spaces. General conditions on the loss function and step sizes are given to ensure convergence. Explicit learning rates are also given for particular step sizes.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
N. Aronszajn, Theory of reproducing kernels, Trans. Amer. Math. Soc. 68 (1950) 337–404.
P.L. Bartlett, M.I. Jordan and J.D. McAuliffe, Convexity, classification, and risk bounds, Preprint, Department of Statistics, University of California Berkeley, 2003.
S. Boyd and L. Vandenberghe, Convex Optimization (Cambridge: Cambridge University Press, 2004).
D.R. Chen, Q. Wu, Y.M. Ying and D.X. Zhou, Support vector machine soft margin classifiers: error analysis, J. Mach. Learn. Res. 5 (2004) 1143–1175.
N. Cesa-Bianchi, P. Long and M. Warmuth, Worst-case quadratic loss bounds for prediction using linear functions and gradient descent, IEEE Trans. Neural Netw. 7 (1996) 604–619.
N. Cristianini and J. Shawe-Taylor, An Introduction to Support Vector Machines and Other Kernel-based Learning Methods (Cambridge: Cambridge University Press, 2000).
F. Cucker and S. Smale, On the mathematical foundations of learning, Bull. Amer. Math. Soc. 39 (2001) 1–49.
L. Devroye, L. Györfi and G. Lugosi, A Probabilistic Theory of Pattern Recognition (Berlin Heidelberg New York: Springer, 1997).
T. Evgeniou, M. Pontil and T. Poggio, Regularization networks and support vector machines, Adv. Comput. Math. 13 (2000) 1–50.
J. Kivinen, A.J. Smola and R.C. Williamson, Online learning with kernels, IEEE Trans. Signal Process. 52 (2004) 2165–2176.
B. Blanchard, G. Lugosi and N. Vayatis, On the rate of convergence of regularized boosting classifiers, J. Mach. Learn. Res. 4 (2003) 861–894.
P. Niyogi and F. Girosi, On the relationships between generalization error, hypothesis complexity and sample complexity for radial basis functions, Neural Comput. 8 (1996) 819–842.
C. Scovel and I. Steinwart, Fast rates for support vector machines, Los Alamos National Laboratory Technical Report, 2005.
S. Smale and Y. Yao, Online learning algorithms, Preprint, Department of Mathematics, University of California Berkeley, 2004.
S. Smale and D.X. Zhou, Estimating the approximation error in learning theory, Anal. Appl. 1 (2003) 17–41.
S. Smale and D.X. Zhou, Shannon sampling and function reconstruction from point values, Bull. Amer. Math. Soc. 41 (2004) 279–305.
S. Smale and D.X. Zhou, Shannon sampling II: Connection to learning theory, Preprint, 2004.
V. Vapnik, Statistical Learning Theory (New York: John Wiley & Sons, 1998).
Q. Wu, Y. Ying and D.X. Zhou, Multi-kernel Regularized Classifiers, Submitted to J. Complexity, Department of Mathematics, City University of Hong Kong, 2004.
Y. Ying and D.X. Zhou, Learnability of Gaussians with flexible variances, Preprint, Department of Mathematics, City University of Hong Kong, 2004.
Y. Ying and D.X. Zhou, Online regularized classification algorithms, Preprint, 2005.
T. Zhang, Statistical behavior and consistency of classification methods based on convex risk minimization, Ann. Statis. 32 (2004) 56–85.
D.X. Zhou, The covering number in learning theory, J. Complex. 18 (2002) 739–767.
D.X. Zhou, Capacity of reproducing kernel spaces in learning theory, IEEE Trans. Inf. Theory 49 (2003) 1743–1752.
Author information
Authors and Affiliations
Corresponding author
Additional information
★The author’s current address: Department of Computer Sciences, University College London, Gower Street, London WC1E, England, UK.
Rights and permissions
About this article
Cite this article
Ying, Y. Convergence analysis of online algorithms. Adv Comput Math 27, 273–291 (2007). https://doi.org/10.1007/s10444-005-9002-z
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10444-005-9002-z
Keywords
- online learning algorithm
- reproducing kernel Hilbert space
- regularized sample error
- general loss function