In the theory of statistical machine learning, a generalization bound – or, more precisely, a generalization error bound – is a statement about the predictive performance of a learning algorithm or class of algorithms. Here, a learning algorithm is viewed as a procedure that takes some finite training sample of labeled instances as input and returns a hypothesis regarding the labels of all instances, including those which may not have appeared in the training sample. Assuming labeled instances are drawn from some fixed distribution, the quality of a hypothesis can be measured in terms of its risk – its incompatibility with the distribution. The performance of a learning algorithm can then be expressed in terms of the expected risk of its hypotheses given randomly generated training samples.
Under these assumptions, a generalization bound is a theorem, which holds for any distribution and states that, with high probability, applying...
- Banerjee, A. (2006). On Bayesian bounds. ICML ’06: Proceedings of the 23rd International Conference on Machine learning, Pittsburgh, pp. 81–88.Google Scholar
- Beygelzimer, A., Langford, J., & Zadrozny, B. (2008). Machine learning techniques – reductions between prediction quality metrics. In Liu, Zhen; Xia, Cathy H. (Eds.) Performance modeling and engineering (pp. 3–28). Springer.Google Scholar
- Bousquet, O., Boucheron, S., & Lugosi, G. (2004). Introduction to statistical learning theory, volume 3176 of lecture notes in artificial intelligence (pp. 169–207). Berlin: Springer.Google Scholar
- Herbrich, R., & Williamson, R. C. (2002). Learning and generalization: Theory and bounds. In M. Arbib (Ed.), Handbook of brain theory and neural networks (2nd ed.). Cambridge: MIT Press.Google Scholar
- Kearns, M. J., & Vazirani, U. V. (1994). An introduction to computational learning theory. Cambridge: MIT Press.Google Scholar
- Shawe-Taylor, J., & Williamson, R. C. (1997). A PAC analysis of a Bayesian estimator. Proceedings of the Tenth Annual Conference on Computational Learning Theory, ACM, p. 7.Google Scholar