Skip to main content
SpringerLink
Log in

Learning with coefficient-based regularization and ℓ1 −penalty

  • Published:
Advances in Computational Mathematics Aims and scope Submit manuscript

Abstract

The least-square regression problem is considered by coefficient-based regularization schemes with ℓ1 −penalty. The learning algorithm is analyzed with samples drawn from unbounded sampling processes. The purpose of this paper is to present an elaborate concentration estimate for the algorithms by means of a novel stepping stone technique. The learning rates derived from our analysis can be achieved in a more general setting. Our refined analysis will lead to satisfactory learning rates even for non-smooth kernels.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. Aronszajn, N.: Theory of reproducing kernels. Trans. Am. Math. Soc. 68, 337–404 (1950)

    Article  MathSciNet  MATH  Google Scholar 

  2. Bennett, G.: Probability inequalities for the sum of independent random variables. J. Am. Stat. Assoc. 57, 33–45 (1962)

    Article  MATH  Google Scholar 

  3. Candès, E., Romberg, J.: Sparsity and incoherence in compressive sampling. Inverse Probl. 23, 969–985 (2007)

    Article  MATH  Google Scholar 

  4. Cucker, F., Zhou, D.X.: Learing Theory: An Approxiamtion Theory Viewpoint. Cambridge University Press (2007)

  5. Conway, J.B.: A Course in Operator Theory. AMS (2000)

  6. Guo, Z.C., Zhou, D.X.: Concentration estimates for learning with unbounded sampling. Adv. Comput. Math. doi:10.1007/s10444-011-9238-8

  7. Liu, C.: Gabor-based kernel pca with fractional power polynomial models for face recognition. IEEE Trans. Pattern Anal. Mach. Intell. 26, 572–581 (2004)

    Article  Google Scholar 

  8. Lin, H., Lin, C.: A study on sigmoid kernels for SVM and the training of non-PSD kernles by SMO-type methods. In: Technical report, Department of Computer Science and Information Engineering, National Taiwan University (2003)

  9. Mendelson, S., Neeman, J.: Regularization in kernel learning. Ann. Stat. 38, 526–565 (2010)

    Article  MathSciNet  MATH  Google Scholar 

  10. Opfer, R.: Multiscale kernels. Adv. Comput. Math. 25, 357–380 (2006)

    Article  MathSciNet  MATH  Google Scholar 

  11. Song, G., Zhang, H.: Reproducing kernel Banach spaces with the l1 norm II: error analysis for regularized least square regression. Neural Comput. 23, 2713–2729 (2011)

    Article  MathSciNet  MATH  Google Scholar 

  12. Song, G., Zhang, H., Hickernell, F.J.: Reproducing kernel Banach spaces with the l1 norm. Appl. Comput. Harmon. Anal. 34, 96–116 (2013)

    Article  MathSciNet  MATH  Google Scholar 

  13. Steinwart, I., Scovel, C.: Fast rates for support vector machines. Lect. Notes Comput. Sci. 3559, 279–294 (2005)

    Article  MathSciNet  Google Scholar 

  14. Shi, L., Feng, Y.L., Zhou, D.X.: Concentration estimates for learning with ℓ1-regularizer and data dependent hypothesis spaces. Appl. Comput. Harmon. Anal. 31, 286-302 (2011)

    Article  MathSciNet  MATH  Google Scholar 

  15. Smale, S., Zhou, D.X.: Learning theory estimates via integral operators and their approximations. Constr. Approx. 26, 153–172 (2007)

    Article  MathSciNet  MATH  Google Scholar 

  16. Tibshirani, R.: Regression shrinkage and selection via the lasso. J. R. Stat. Soc. B 58, 267–288 (1996)

    MathSciNet  MATH  Google Scholar 

  17. van der Vaart, A.W., Wellner, J.A.: Weak Convergence and Emprical Processes, Springer-Verlag, New York (1996)

    Book  Google Scholar 

  18. De Vito, E., Caponnetto, A., Rosasco, L.: Model selection for regularized least-squares algorithm in learning theory. Found. Comput. Math. 5, 59–85 (2005)

    Article  MathSciNet  MATH  Google Scholar 

  19. Vapnik, V.: Statistical Learning Theory. Wiley, New York (1998)

    MATH  Google Scholar 

  20. Wahba, G.: Spline Models for Observational Data. SIAM (1990)

  21. Wu, Q., Zhou, D.X.: SVM soft margin classifier: linear programming versus quadratic programming. Neural Comput. 15, 1397–1437 (2003)

    Article  Google Scholar 

  22. Wu, Q., Ying, Y., Zhou, D.X.: Learning rates of least-square regularized regression. Found. Comput. Math. 6, 171–192 (2006)

    Article  MathSciNet  MATH  Google Scholar 

  23. Wu, Q., Zhou, D.X.: Learning with sample dependent hypothesis spaces. Comput. Math. Appl. 56, 2896–2907 (2008)

    Article  MathSciNet  MATH  Google Scholar 

  24. Xiao, Q.W., Zhou, D.X.: Learning by nonsymmetric kernel with data dependent spaces and ℓ1-regularizer. Taiwan. J. Math. 14, 1821–1836 (2010)

    MathSciNet  MATH  Google Scholar 

  25. Xu, Y.S., Zhang, H.Z.: Refinable kernels. J. Mach. Learn. Res. 8, 2083–2120 (2007)

    MathSciNet  MATH  Google Scholar 

  26. Zhang, T.: Leave-one-out bounds for kernel methods. Neural Comput. 15, 1397–1437 (2003)

    Article  MATH  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. College of Engineering, Mathematics and Physical Sciences, University of Exeter, Harrison Building, EX4 4QF, Exeter, UK

    Zheng-Chu Guo

  2. Shanghai Key Laboratory for Contemporary Applied Mathematics, School of Mathematical Sciences, Fudan University, Shanghai, 200433, People’s Republic of China

    Lei Shi

Authors
  1. Zheng-Chu Guo
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Lei Shi
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Lei Shi.

Additional information

Communicated by: Lixin Shen.

The work described in this paper is supported by the National Science Foundation of China under Grand 11201079.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Guo, ZC., Shi, L. Learning with coefficient-based regularization and ℓ1 −penalty. Adv Comput Math 39, 493–510 (2013). https://doi.org/10.1007/s10444-012-9288-6

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10444-012-9288-6

Keywords

Mathematics Subject Classifications (2010)

Search

Navigation