Abstract
We consider the complexity of concept learning in various common models for on-line learning, focusing on methods for proving lower bounds to the learning complexity of a concept class. Among others, we consider the model for learning with equivalence and membership queries. For this model we give lower bounds on the number of queries that are needed to learn a concept classC in terms of the Vapnik-Chervonenkis dimension ofC, and in terms of the complexity of learningC with arbitrary equivalence queries. Furthermore, we survey other known lower bound methods and we exhibit all known relationships between learning complexities in the models considered and some relevant combinatorial parameters. As it turns out, the picture is almost complete. This paper has been written so that it can be read without previous knowledge of Computational Learning Theory.
Article PDF
Similar content being viewed by others
Avoid common mistakes on your manuscript.
References
Angluin, D. (1981). A note on the number of queries needed to identify regular languages.Information and Control, 51 76–87.
Angluin, D. (1987a). Learning regular sets from queries and counterexamples.Information and Computation, 75 87–106.
Angluin, D. (1987b).Learning k-term DNF formulas using queries and counterexamples. (Technical Report YALEU/DCS/RR-557). New Haven, CT: Yale University, Department of Computer Science.
Angluin, D. (1988). Queries and concept learning.Machine Learning, 2 319–342.
Angluin, D. (1990). Negative results for equivalence queries.Machine Learning, 5 121–150.
Angluin, D., Frazier, M. & Pitt, L. (1990). Learning conjunctions of Horn clauses.Proceedings of the Thirty-First Annual Symposium on Foundatins of Computer Science (pp. 186–192). Washington, DC: IEEE Computer Society.
Angluin, D., Hellerstein, L. & Karpinski, M. (1989).Learning read-once formulas with queries. (Technical Report UCB/CSD 89/527). University of California at Berkeley, Computer Science Division. (Also, Technical Report TR-89-050, International Computer Science Institute, Berkeley, California.)
Barzdin, T.M. & Freiwalds, R.V. (1972). On the prediction of general recursive functions,Sov. Math. Dokl., 13 1224–1228.
Baum, E.B. (1990). Polynomial time algorithms for learning neural nets.Proceedings of the Third Annual Workshop on Computational Learning Theory (pp. 258–272). San Mateo, CA: Morgan Kaufmann.
Berman, P. & Roos, R. (1987). Learning one-counter languages in polynomial time.Proceedings of the Twenty-Eighth Annual Symposium on Foundations of Computer Science (pp. 61–67). Washington, DC: IEEE Computer Society.
Blumer, A., Ehrenfeucht, A., Haussler, D., & Warmuth, M.K. (1989). Learnability and the Vapnik-Chervonenkis dimension.Journal of the ACM, 36 929–965.
Bultman, W. & Maass, W. (1991). On-line learning of geometrical concepts with membership queries.Proceedings of the Fourth Annual Workshop on Computational Learning Theory (pp. 337–353). San Mateo, CA: Morgan Kaufmann.
Csiszár, I. & Körner, J. (1981).Information theory. New York: Academic Press.
Erdös, P. & Spencer, J. (1974).Probabilistic methods in combinatorics. New York-Budapest: Academic Press-Akadémiai Kiadó.
Faigle, U. & Turan, Gy. (1988). Sorting and recognition problems for ordered sets.SIAM Journal on Computing, 17 100–113.
Gaizer, T. (1990). The Vapnik-Chervonenkis dimension of finite automata. Unpublished manuscript.
Goldman, S.A., Rivest, R.L. & Schapire, R.E. (1989). Learning binary relations and total orders.Proceedings of the Thirtieth Annual Symposium on Foundations of Computer Science (pp. 46–51), Washington, DC: IEEE Computer Society.
Hetyei, G. (1964),Pécsi Tanárképzö Föiskola Közleményei, 151–168.
Ishizaka, H. (1990). Polynomial time learnability of simple deterministic languages.Machine Learning, 5 151–164.
Kahn, J. & Saks, M. (1984). Balancing poset extensions.Order, 1 113–126.
Kleitman, D.J., Shearer, J.B., & Sturtevant, D. (1981). Intersections ofk-element sets.Combinatorica, 1 381–384.
Littlestone, N. (1988). Learning quickly when irrelevant attributes abound: A new linear-threshold algorithm.Machine Learning, 2 285–318.
Lovász, L. (1979).Combinatorial problems and exercises. Budapest: Akadémiai Kiadó.
Maass, W. (1991). On-line learning with an oblivious environment and the power of randomization.Proceedings of the Fourth Annual Workshop on Computational Learning Theory (pp. 167–175). San Mateo, CA: Morgan Kaufmann.
Maass, W. & Turán, Gy. (1989). On the complexity of learning from counterexamples.Proceedings of the Thirtieth Annual Symposium on Foundations of Computer Science (pp. 262–267). Washington, DC: IEEE Computer Society Press.
Maass, W. & Turán, Gy. (1990a). On the complexity of learning from counterexamples and membership queries.Proceedings of the Thirty-First Annual Symposium on Foundations of Computer Science (pp. 203–210). Washington, DC: IEEE Computer Society Press.
Maass, W. & Turán, Gy. (1990b). Algorithms and lower bounds for on-line learning of geometrical concepts. To appear inMachine Learning.
Maass, W. & Turán, Gy. (1990c). How fast can a threshold gate learn? In S. Hanson, G. Drastal, R. Rivest, (Eds.),Computational learning theory and natural learning systems: Constraints and prospects, Cambridge, MA: MIT Press, to appear.
Minsky, M. & Papert, S. (1988).Perceptrons: an introduction to computational geometry, Expanded edition. Cambridge, MA: MIT Press.
Nilsson, N.J. (1965).Learning machines. New York: McGraw-Hill.
Pitt, L. & Valiant, L.G. (1988). Computational limitations on learning from examples.Journal of the ACM, 35 965–984.
Rosenblatt, F. (1962).Principles of neurodynamics. New York: Spartan Books.
Rumelhart, D.E. & McClelland, J.L. (1986).Parallel distributed processing: Explorations in the microstructure of cognition. Cambridge, MA: MIT Press.
Sauer, N. (1972). On the density of families of sets.Journal of Combinatorial Theory (A),13 154–147.
Shelah, S. (1972). A combinatorial problem; stability and order for models and theories in infinitary languages.Pacific Journal of Mathematics, 41 247–261.
Valiant, L.G. (1984). A theory of the learnable.Communications of the ACM, 27 1134–1142.
Vapnik, V.N. & Chervonenkis, A. Ya. (1971). On the uniform convergence of relative frequencies of events to their probabilities.Theory of Probability and Its Applications, 16 264–280.
Vitter, J.S. & Lin, J.H. (1988). Learning in parallel.Proceedings of the 1988 Workshop on Computational Learning Theory (pp. 106–124). San Mateo, CA: Morgan Kaufmann.
Author information
Authors and Affiliations
Rights and permissions
About this article
Cite this article
Maass, W., Turán, G. Lower bound methods and separation results for on-line learning models. Mach Learn 9, 107–145 (1992). https://doi.org/10.1007/BF00992674
Issue Date:
DOI: https://doi.org/10.1007/BF00992674