Machine Learning

, Volume 28, Issue 2–3, pp 133–168

Selective Sampling Using the Query by Committee Algorithm

  • Yoav Freund
  • H. Sebastian Seung
  • Eli Shamir
  • Naftali Tishby
Article

Abstract

We analyze the “query by committee” algorithm, a method for filtering informative queries from a random stream of inputs. We show that if the two-member committee algorithm achieves information gain with positive lower bound, then the prediction error decreases exponentially with the number of queries. We show that, in particular, this exponential decrease holds for query learning of perceptrons.

selective sampling query learning Bayesian Learning experimental design 

References

  1. 1.
    Dana Angluin. Queries and concept learning. Machine Learning, 2(4): 319–342, April 1988.Google Scholar
  2. 2.
    A. C. Atkinson and A. N. Donev. Optimum Experimental Designs. Oxford science publications, 1992.Google Scholar
  3. 3.
    Ian Barland. Some ideas on learning with directional feedback. Master's thesis, University of California at Santa Cruz, June 1992.Google Scholar
  4. 4.
    E. Baum. Neural net algorithms that learn in polynomial time from examples and queries. IEEE Transactions on Neural Networks, 2: 5–19, 1991.Google Scholar
  5. 5.
    E. B. Baum and K. Lang. Query learning can work poorly when a human oracle is used. In International Joint Conference in Neural Networks, Beijing, China, 1992.Google Scholar
  6. 6.
    A. Blumer, A. Ehrenfeucht, D. Haussler, and M. K. Warmuth. Learnability and the Vapnik-Chervonenkis dimension. J. ACM, 36(4): 929–965, 1989.Google Scholar
  7. 7.
    T. Bonnesen and W. Fenchel. Theory of Convex Bodies. BCS Associates, Moscow, Idaho, USA, 1987.Google Scholar
  8. 8.
    David Cohn, Les Atlas, and Richard Ladner. Training connectionist networks with queries and selective sampling. In D. Touretzky, editor, Advances in Neural Information Processing Systems 2, San Mateo, CA, 1990. Morgan Kaufmann.Google Scholar
  9. 9.
    Ido Dagan and Sean P. Engelson. Committee-based sampling for training probabilistic classfiers. In Priedits and Russel, editors, The XII International Conference on Machine Learning, pages 150–157. Morgan Kaufmann, 1995.Google Scholar
  10. 10.
    Bonnie Eisenberg and Ronald L. Rivest. On the sample complexity of pac-learning using random and chosen examples. In Proceedings of the 1990Workshop on Computational Learning Theory, pages 154–162, 1990.Google Scholar
  11. 11.
    V. V. Fedorov. Theory of Optimal Experiments. Academic Press, New York, 1972.Google Scholar
  12. 12.
    David Haussler, Michael Kearns, and Robert E. Schapire. Bounds on the sample complexity of Bayesian learning using information theory and the VC dimension. Machine Learning, 14: 83–113, 1994.Google Scholar
  13. 13.
    W. Kinzel and P. Ruján. Improving a network generalization ability by selecting examples. Europhys. Lett., 13: 473–477, 1990.Google Scholar
  14. 14.
    D.V. Lindley. On a measure of the information provided by an experiment. Ann. Math. Statist., 27: 986–1005, 1956.Google Scholar
  15. 15.
    C. McDiarmid. On the method of bounded differences. In Survey of Combinatorics, 10th British Combinatorial Conference, 1989.Google Scholar
  16. 16.
    T.M. Mitchell. Generalization as search. Artificial Intelligence, 18(2), 1982.Google Scholar
  17. 17.
    N. Sauer. On the density of families of sets. J. Combinatorial Theory (A), 13: 145–147, 1972.Google Scholar
  18. 18.
    H.S Seung, M. Opper, and H. Sompolinsky. Query by committee. In Proceedings of the Fifth Workshop on Computational Learning Theory, pages 287–294, San Mateo, CA, 1992. Morgan Kaufmann.Google Scholar
  19. 19.
    Peter Smith. Convexity Methods in Variational Calculus. Research studies press, John Wiley & Sons, 1985.Google Scholar
  20. 20.
    L. G. Valiant. A theory of the learnable. Communications of the ACM, 27(11): 1134–1142, November 1984.Google Scholar

Copyright information

© Kluwer Academic Publishers 1997

Authors and Affiliations

  • Yoav Freund
    • 1
  • H. Sebastian Seung
    • 2
  • Eli Shamir
    • 3
  • Naftali Tishby
    • 3
  1. 1.AT&T LabsFlorham Park
  2. 2.Bell LaboratoriesLucent Technologies
  3. 3.Institute of Computer ScienceHebrew UniversityJerusalemISRAEL

Personalised recommendations