Decision tree pruning as a search in the state space

  • Floriana Esposito
  • Donate Malerba
  • Giovanni Semeraro
Research Papers Inductive Learning
Part of the Lecture Notes in Computer Science book series (LNCS, volume 667)


This paper presents a study of one particular problem of decision tree induction, namely (post-)pruning, with the aim of finding a common framework for the plethora of pruning methods appeared in literature. Given a tree Tmax to prune, a state space is defined as the set of all subtrees of Tmax to which only one operator, called any-depth branch pruning operator, can be applied in several ways in order to move from one state to another. By introducing an evaluation function f defined on the set of subtrees, the problem of tree pruning can be cast as an optimization problem, and it is also possible to classify each post-pruning method according to both its search strategy and the kind of information exploited by f. Indeed, while some methods use only the training set in order to evaluate the accuracy of a decision tree, other methods exploit an additional pruning set that allows them to get less biased estimates of the predictive accuracy of apruned tree. The introduction of the state space shows that very simple search strategies are used by the postpruning methods considered. Finally, some empirical results allow theoretical observations on strengths and weaknesses of pruning methods to be better understood.


  1. 1.
    L. Breiman, J. Friedman, R. Olshen, C. Stone: Classification and regression trees. Belmont, CA: Wadsworth International 1984Google Scholar
  2. 2.
    J. R. Quinlan: Induction of decision trees. Machine Learning 1, 81–106 (1986)Google Scholar
  3. 3.
    J. R. Quinlan: Simplifying decision trees. International Journal of Man-Machine Studies 27, 221–234 (1987) (also appeared in: B. R. Gaines, J. H. Boose (eds.): Knowledge Acquisition for Knowledge-Based Systems. Academic Press 1988)Google Scholar
  4. 4.
    M. Gams, N. Lavrac: Review of five empirical learning systems within a proposed schemata. In: I. Bratko, N. Lavrac (eds.): Progress in Machine Learning. Wilmslow: Sigma Press 1987Google Scholar
  5. 5.
    J. Mingers: An empirical comparison of selection measures for decision-tree induction. Machine Learning 3, 319–342 (1989)Google Scholar
  6. 6.
    J. Mingers: An empirical comparison of pruning methods for decision tree induction. Machine Learning 4, 227–243 (1989)Google Scholar
  7. 7.
    B. Cestnik, I. Kononenko, I. Bratko: ASSISTANT 86: Aknowledge-elicitation tool for sophisticated users. In: I. Bratko, N. Lavrac (eds.): Progress in Machine Learning. Wilmslow: Sigma Press 1987Google Scholar
  8. 8.
    J. R. Quinlan: Determinate literals in inductive logic programming. Proceedings of the IJCAI 91. San Mateo, CA: Morgan Kaufmann 1991, pp. 746–750Google Scholar
  9. 9.
    T. Niblett: Constructing decision trees in noisy domains. In: I. Bratko, N. Lavrac (eds.): Progress in Machine Learning. Wilmslow: Sigma Press 1987Google Scholar
  10. 10.
    A. V. Aho, J. E. Hopcroft, J. D. Ullman: The design and analysis of computer algorithms. Reading, MA: Addison Wesley 1974Google Scholar
  11. 11.
    F. Esposito, D. Malerba, G. Semeraro: Pruning methods in decision tree induction: a unifying view. Technical report (1992)Google Scholar
  12. 12.
    T. Niblett, I. Bratko: Learning decision rules in noisy domains. Proceedings of Expert Systems 86. Cambridge: University Press 1986Google Scholar
  13. 13.
    B. Cestnik, I. Bratko: On estimating probabilities in tree pruning. Proceedings of the EWSL-91. Berlin: Springer-Verlag 1991, pp. 138–150Google Scholar
  14. 14.
    A. Barr, E. Feigenbaum: The handbook of artificial intelligence, (Vol. 1). Reading, MA: Addison Wesley 1981Google Scholar
  15. 15.
    S. B. Gelfand, C. S. Ravishankar, E. J. Delp: An iterative growing and pruning algorithm for classification tree design. IEEE Transactions on Pattern Analysis and Machine Intelligence PAMI-13, 2, 163–174 (1991)Google Scholar
  16. 16.
    W. Buntine, T. Niblett: A further comparison of splitting rules for decision-tree induction. Machine Learning 8, 75–85 (1992)Google Scholar
  17. 17.
    C. Schaffer: Deconstructing the digit recognition problem. In: Machine Learning: Proceedings of the Ninth International Workshop (ML92). San Mateo, CA: Morgan Kaufmann (1992), pp. 394–399Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 1993

Authors and Affiliations

  • Floriana Esposito
    • 1
  • Donate Malerba
    • 1
  • Giovanni Semeraro
    • 1
  1. 1.Dipartimento di InformaticaUniversità degli Studi di BariBariItaly

Personalised recommendations