Encyclopedia of Machine Learning

2010 Edition
| Editors: Claude Sammut, Geoffrey I. Webb

Generalization Bounds

  • Mark Reid
Reference work entry
DOI: https://doi.org/10.1007/978-0-387-30164-8_328

Synonyms

Definition

In the theory of statistical machine learning, a generalization bound – or, more precisely, a generalization error bound – is a statement about the predictive performance of a learning algorithm or class of algorithms. Here, a learning algorithm is viewed as a procedure that takes some finite training sample of labeled instances as input and returns a hypothesis regarding the labels of all instances, including those which may not have appeared in the training sample. Assuming labeled instances are drawn from some fixed distribution, the quality of a hypothesis can be measured in terms of its risk – its incompatibility with the distribution. The performance of a learning algorithm can then be expressed in terms of the expected risk of its hypotheses given randomly generated training samples.

Under these assumptions, a generalization bound is a theorem, which holds for any distribution and states that, with high probability, applying...

This is a preview of subscription content, log in to check access.

References

  1. Anthony, M., & Bartlett, P. L. (1999). Neural network learning: Theoretical foundations. Cambridge: Cambridge University Press.MATHCrossRefGoogle Scholar
  2. Banerjee, A. (2006). On Bayesian bounds. ICML ’06: Proceedings of the 23rd International Conference on Machine learning, Pittsburgh, pp. 81–88.Google Scholar
  3. Bartlett, P. L., & Mendelson, S. (2003). Rademacher and Gaussian complexities: risk bounds and structural results. Journal of Machine Learning Research, 3, 463–482.MathSciNetMATHCrossRefGoogle Scholar
  4. Beygelzimer, A., Langford, J., & Zadrozny, B. (2008). Machine learning techniques – reductions between prediction quality metrics. In Liu, Zhen; Xia, Cathy H. (Eds.) Performance modeling and engineering (pp. 3–28). Springer.Google Scholar
  5. Blumer, A., Ehrenfeucht, A., Haussler, D., & Warmuth, M. K. (1989). Learnability and the Vapnik-Chervonenkis dimension. Journal of the ACM (JACM), 36(4), 929–965.MathSciNetMATHCrossRefGoogle Scholar
  6. Boucheron, S., Bousquet, O., & Lugosi, G. (2005). Theory of classification: A survey of some recent advances. ESAIM Probability and statistics, 9, 323–375.MathSciNetMATHCrossRefGoogle Scholar
  7. Bousquet, O., Boucheron, S., & Lugosi, G. (2004). Introduction to statistical learning theory, volume 3176 of lecture notes in artificial intelligence (pp. 169–207). Berlin: Springer.Google Scholar
  8. Herbrich, R., & Williamson, R. C. (2002). Learning and generalization: Theory and bounds. In M. Arbib (Ed.), Handbook of brain theory and neural networks (2nd ed.). Cambridge: MIT Press.Google Scholar
  9. Kearns, M. J., & Vazirani, U. V. (1994). An introduction to computational learning theory. Cambridge: MIT Press.Google Scholar
  10. Koltchinskii, V. (2001). Rademacher penalties and structural risk minimization. IEEE Transactions on Information Theory, 47(5), 1902–1914.MathSciNetMATHCrossRefGoogle Scholar
  11. Langford, J. (2005). Tutorial on practical prediction theory for classification. Journal of Machine Learning Research, 6(1), 273–306.MathSciNetGoogle Scholar
  12. McAllester, D. A. (1999). Some PAC-Bayesian theorems. Machine Learning, 37(3), 355–363.MATHCrossRefGoogle Scholar
  13. Shawe-Taylor, J., & Williamson, R. C. (1997). A PAC analysis of a Bayesian estimator. Proceedings of the Tenth Annual Conference on Computational Learning Theory, ACM, p. 7.Google Scholar
  14. Valiant, L. G. (1984). A theory of the learnable. Communications of the ACM, 27(11), 1142.CrossRefGoogle Scholar
  15. Vapnik, V. N. (1998). Statistical learning theory. New York: Wiley.MATHGoogle Scholar
  16. Vapnik, V. N., & Chervonenkis, A. Y., (1971). On the uniform convergence of relative frequencies of events to their probabilities. Theory of Probability and Its Applications, 16(2), 264–280.MathSciNetMATHCrossRefGoogle Scholar

Copyright information

© Springer Science+Business Media, LLC 2011

Authors and Affiliations

  • Mark Reid
    • 1
  1. 1.The Australian National UniversityCanberraAustralia