Abstract
Various authors have proposed probabilistic extensions of Valiant's PAC learning model in which the target to be learned is a conditional (or unconditional) probability distribution. In this paper, we improve upon the best known upper bounds on the sample complexity of learning an important class of stochastic rules called 'stochastic rules with finite partitioning' with respect to the classic notion of distance between distributions, the Kullback-Leibler divergence (KL-divergence). In particular, we improve the upper bound of order O(1/e2) due to Abe, Takeuchi, and Warmuth [2] to a bound of order O(1/e). Our proof technique is interesting for at least two reasons: First, previously known upper bounds with respect to the KL-divergence were obtained using the uniform convergence technique, while our improved upper bound is obtained by taking advantage of the properties of the maximum likelihood estimator. Second, our proof relies on the fact that only a linear number of examples are required in order to distinguish a true parametric model from a bad parametric model. The latter notion is apparently related to the notion of discrimination proposed and studied by Yamanishi, but the exact relationship is yet to be determined.
Preview
Unable to display preview. Download preview PDF.
References
Abe, N. & Warmuth, M.(1992). On the computational complexity of approximating distributions by probabilistic automata. Machine Learning, a special issue for COLT '90, 9(2/3).
Abe, N., Takeuchi, J., & Warmuth, M.(1991). Polynomial Learnability of Probabilistic Concepts with respect to the Kullback-Leibler Divergence. Proceedings of the Forth Annual Workshop on Computational Learning Theory (pp. 277–289), Rochester, NY: Morgan Kaufmann.
Kearns, M. & Schapire, R.(1990). Efficient distribution-free learning of probabilistic concepts. Proceedings of the 31st Symposium on Foundations of Computer Science, (PP. 382–391), St. Louis, Missouri.
Pollard, D.(1984).Convergence of Stochastic Processes. (pp. 191–193), Springer Verlag.
Takeuchi, J.(1992).Some Improved Sample Complexity Bounds in the Probabilistic PAC Learning Model. To appear.
Valiant, L.G.(1984). A theory of the learnable. Communications of the ACM, 27, 1134–1142.
Yamanishi, K.(1992). A learning criterion for stochastic rules. Machine Learning, a special issue for COLT '90, 9(2/3).
Yamanishi, K.(1992). Probably Almost Discriminative Learning. Proceedings of the Fifth Annual Workshop on Computational Learning Theory, Rochester, NY: Morgan Kaufmann.
Author information
Authors and Affiliations
Editor information
Rights and permissions
Copyright information
© 1993 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Takeuchi, Ji. (1993). Some improved sample complexity bounds in the probabilistic PAC learning model. In: Doshita, S., Furukawa, K., Jantke, K.P., Nishida, T. (eds) Algorithmic Learning Theory. ALT 1992. Lecture Notes in Computer Science, vol 743. Springer, Berlin, Heidelberg. https://doi.org/10.1007/3-540-57369-0_40
Download citation
DOI: https://doi.org/10.1007/3-540-57369-0_40
Published:
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-540-57369-2
Online ISBN: 978-3-540-48093-8
eBook Packages: Springer Book Archive