Skip to main content

Sample Complexity of Linear Learning Machines with Different Restrictions over Weights

  • Conference paper
  • 1663 Accesses

Part of the Lecture Notes in Computer Science book series (LNAI,volume 7268)

Abstract

Known are many different capacity measures for learning machines like: Vapnik-Chervonenkis dimension, covering numbers or fat dimension. In this paper we present experimental results of sample complexity estimation, taking into account rather simple learning machines linear in parameters. We show that, sample complexity can be quite different even for learning machines having the same VC-dimension. Moreover, independently from the capacity of a learning machine, the distribution of data is also significant. Experimental results are compared with known theoretical results for sample complexity and generalization bounds.

Keywords

  • Mean Square Error
  • Sample Complexity
  • True Error
  • Bayesian Regularisation
  • Capacity Concept

These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

This work has been financed by the Polish Government, Ministry of Science and Higher Education from the sources for science within years 2010–2012. Research project no.: N N516 424938.

This is a preview of subscription content, access via your institution.

Buying options

Chapter
USD   29.95
Price excludes VAT (Canada)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD   39.99
Price excludes VAT (Canada)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD   54.99
Price excludes VAT (Canada)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Learn about institutional subscriptions

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. Anthony, M., Bartlett, P.L.: Neural Network Learning: Theoretical Foundations. Cambridge University Press (1999)

    Google Scholar 

  2. Bartlett, P.L., Mendelson, S.: Rademacher and gaussian complexities: risk bounds and structural results. J. Mach. Learn. Res. 3, 463–482 (2003)

    MathSciNet  MATH  Google Scholar 

  3. Burges, C.J.C.: A tutorial on support vector machines for pattern recognition. Data Min. Knowl. Discov. 2(2), 121–167 (1998)

    CrossRef  Google Scholar 

  4. Cawley, G.C., Talbot, N.L.C.: Gene selection in cancer classification using sparse logistic regression with bayesian regularisation. Bioinformatics 22(19), 2348–2355 (2006)

    CrossRef  Google Scholar 

  5. Chang, C.C., Lin, C.J.: LIBSVM: a library for support vector machines (2001), Software available at, http://www.csie.ntu.edu.tw/cjlin/libsvm

  6. Domingos, P.: The role of occam’s razor in knowledge discovery. Data Mining and Knowledge Discovery 3, 409–425 (1999)

    CrossRef  Google Scholar 

  7. Efron, B., Hastie, T., Johnstone, I., Tibshirani, R.: Least angle regression. Annals of Statistics 32(2), 407–451 (1996)

    MathSciNet  Google Scholar 

  8. Hastie, T., Tibshirani, R., Friedman, J.: The Elements of Statistical Learning: Data Mining, Inference, and Prediction. Springer (2009)

    Google Scholar 

  9. Hesterberg, T., Choi, N.H., Meier, L., Fraley, C.: Least angle and l 1 penalized regression: A review. Statistics Surveys 2, 61–93 (2008)

    CrossRef  MathSciNet  MATH  Google Scholar 

  10. Klęsk, P., Korzeń, M.: Sets of approximating functions with finite vapnik-chervonenkis dimension for nearest-neighbors algorithms. Pattern Recognition Letters 32(14), 1882–1893 (2011)

    CrossRef  Google Scholar 

  11. MacKay, D.J.C.: Information theory, inference, and learning algorithms. Cambridge University Press (2003)

    Google Scholar 

  12. Minka, T.P.: A comparison of numerical optimizers for logistic regression. Technical report, Dept. of Statistics, Carnegie Mellon Univ. (2003)

    Google Scholar 

  13. Ng, A.Y.: Feature selection, l1 vs. l2 regularization, and rotational invariance. In: ICML 2004: Proceedings of the Twenty-First International Conference on Machine Learning, p. 78. ACM, New York (2004)

    CrossRef  Google Scholar 

  14. Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society, Series B 58(1), 267–288 (1996)

    MathSciNet  MATH  Google Scholar 

  15. Vapnik, V.: Statistical learning theory. Wiley (1998)

    Google Scholar 

  16. Vincent, P., Bengio, Y.: K-local hyperplane and convex distance nearest neighbors algorithms. In: Advances in Neural Information Processing Systems, pp. 985–992 (2001)

    Google Scholar 

  17. Williams, P.M.: Bayesian regularisation and pruning using a laplace prior. Neural Computation 7, 117–143 (1994)

    CrossRef  Google Scholar 

  18. Zahálka, J., Železný, F.: An experimental test of occam’s razor in classification. Machine Learning 82, 475–481 (2011)

    CrossRef  Google Scholar 

  19. Zhang, T.: Covering number bounds of certain regularized linear function classes. Journal of Machine Learning Research 2, 527–550 (2002)

    MATH  Google Scholar 

  20. Zou, H., Hastie, T.: Regularization and variable selection via the elastic net. J. R. Statist. Soc. B 67(2), 301–320 (2005)

    CrossRef  MathSciNet  MATH  Google Scholar 

Download references

Author information

Authors and Affiliations

Authors

Editor information

Editors and Affiliations

Rights and permissions

Reprints and Permissions

Copyright information

© 2012 Springer-Verlag Berlin Heidelberg

About this paper

Cite this paper

Korzeń, M., Klęsk, P. (2012). Sample Complexity of Linear Learning Machines with Different Restrictions over Weights. In: Rutkowski, L., Korytkowski, M., Scherer, R., Tadeusiewicz, R., Zadeh, L.A., Zurada, J.M. (eds) Artificial Intelligence and Soft Computing. ICAISC 2012. Lecture Notes in Computer Science(), vol 7268. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-29350-4_13

Download citation

  • DOI: https://doi.org/10.1007/978-3-642-29350-4_13

  • Publisher Name: Springer, Berlin, Heidelberg

  • Print ISBN: 978-3-642-29349-8

  • Online ISBN: 978-3-642-29350-4

  • eBook Packages: Computer ScienceComputer Science (R0)