Machine Learning

, Volume 4, Issue 2, pp 161–186 | Cite as

Incremental Induction of Decision Trees

  • Paul E. Utgoff


This article presents an incremental algorithm for inducing decision trees equivalent to those formed by Quinlan's nonincremental ID3 algorithm, given the same training instances. The new algorithm, named ID5R, lets one apply the ID3 induction process to learning tasks in which training instances are presented serially. Although the basic tree-building algorithms differ only in how the decision trees are constructed, experiments show that incremental training makes it possible to select training instances more carefully, which can result in smaller decision trees. The ID3 algorithm and its variants are compared in terms of theoretical complexity and empirical behavior.

Decision tree concept learning incremental learning learning from examples 


  1. Barto, A. G. (1985). Learning by statistical cooperation of self-interested neuron-like computing elements. Human Neurobiology, 4, 229–256.Google Scholar
  2. Breiman, L., Friedman, J. H., Olshen, R. A., and Stone, C. J. (1984). Classification and regression trees. Belmont, CA: Wadsworth International Group.Google Scholar
  3. Buchanan, B. G., and Mitchell, T. M. (1978). Model-directed learning of production rules. In D. A. Waterman and F. Hayes-Roth (Eds.), Pattern-directed inference systems. New York: Academic Press.Google Scholar
  4. Cockett, J. R. B. (1987). Discrete decision theory: Manipulations. Theoretical Computer Science, 54, 215–236.Google Scholar
  5. Fisher, D. H. (1987). Knowledge acquisition via incremental conceptual clustering. Machine Learning, 2, 139–172.Google Scholar
  6. Gallant, S. I. (1988). Connectionist expert systems. Communications of the ACM, 31, 152–169.Google Scholar
  7. Lewis, P. M. (1962). The characteristic selection problem in recognition systems. IRE Transactions on Information Theory, IT-8, 171–178.Google Scholar
  8. Michalski, R. S., and Chilausky, R. L. (1980). Learning by being told and learning from examples: An experimental comparison of the two methods of knowledge acquisition in the context of developing an expert system for soybean disease diagnosis. Policy Analysis and Information Systems, 4, 125–160.Google Scholar
  9. Michalski, R. S. (1983). A theory and methodology of inductive learning. In R. S. Michalski, J. G. Carbonell, and T. M. Mitchell (Eds.), Machine learning: An artificial intelligence approach. San Mateo, CA: Morgan Kaufmann.Google Scholar
  10. Mitchell, T. M. (1978). Version spaces: An approach to concept learning. Doctoral dissertation, Department of Electrical Engineering, Stanford University, Palo Alto, CA.Google Scholar
  11. Moret, B. M. E. (1982). Decision trees and diagrams. Computing Surveys, 14, 593–623.Google Scholar
  12. Quinlan, J. R. (1983). Learning efficient classification procedures and their application to chess end games. In R. S. Michalski, J. G. Carbonell, and T. M. Mitchell (Eds.), Machine learning: An artificial intelligence approach. San Mateo, CA: Morgan Kaufmann.Google Scholar
  13. Quinlan, J. R. (1986). Induction of decision trees. Machine Learning, 1, 81–106.Google Scholar
  14. Quinlan, J. R. (1988). An empirical comparison of genetic and decision-tree classifiers. Proceedings of the Fifth International Conference on Machine Learning (pp. 135–141). Ann Arbor, MI: Morgan Kaufmann.Google Scholar
  15. Schlimmer, J. C., and Fisher, D. (1986). A case study of incremental concept induction. Proceedings of the Fifth National Conference on Artificial Intelligence (pp. 496–501). Philadelphia, PA: Morgan Kaufmann.Google Scholar
  16. Schlimmer, J. C., and Granger, R. H., Jr. (1986). Incremental learning from noisy data. Machine Learning, 1, 317–354.Google Scholar
  17. Shannon, C. E. (1948). A mathematical theory of communication. Bell System Technical Journal, 27, 379–423.Google Scholar
  18. Utgoff, P. E. (1988). ID5: An incremental ID3. Proceedings of the Fifth International Conference on Machine Learning (pp. 107–120). Ann Arbor, MI: Morgan Kaufmann.Google Scholar
  19. Utgoff, P. E. (1989). Improved training via incremental learning. Proceedings of the Sixth International Workshop on Machine Learning. Ithaca, NY: Morgan Kaufmann.Google Scholar
  20. Vere, S. A. (1980). Multilevel counterfactuals for generalizations of relational concepts and productions. Artificial Intelligence, 14, 138–164.Google Scholar
  21. Watanabe, S. (1985). Pattern recognition: Human and mechanical. New York: Wiley and Sons.Google Scholar
  22. Wilson, S. W. (1987). Classifier systems and the animal problem. Machine Learning, 2, 199–228.Google Scholar

Copyright information

© Kluwer Academic Publishers 1989

Authors and Affiliations

  • Paul E. Utgoff
    • 1
  1. 1.Department of Computer and Information ScienceUniversity of MassachusettsAmherst

Personalised recommendations