Advertisement

On estimating probabilities in tree pruning

Part 3: Numeric And Statistical Approaches
Part of the Lecture Notes in Computer Science book series (LNCS, volume 482)

Abstract

In this paper we introduce a new method for decision tree pruning, based on the minimisation of the expected classification error method by Niblett and Bratko. The original Niblett-Bratko pruning algorithm uses Laplace probability estimates. Here we introduce a new, more general Bayesian approach to estimating probabilities which we call m-probability-estimation. By varying a parameter m in this method, tree pruning can be adjusted to particular properties of the learning domain, such as level of noise. The resulting pruning method improves on the original Niblett-Bratko pruning in the following respects: apriori probabilities can be incorporated into error estimation, several trees pruned to various degrees can be generated, and the degree of pruning is not affected by the number of classes. These improvements are supported by experimental findings. m-probability-estimation also enables the combination of learning data obtained from various sources.

Keywords

Classification Accuracy Class Distribution Tree Pruning Pruning Method Complete Binary Tree 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

References

  1. Berger, J.O. (1985), Statistical Decision Theory and Bayesian Analysis, Springer-Verlag, New York.Google Scholar
  2. Bratko, I., Kononenko, I. (1986), Learning diagnostic rules from incomplete and noisy data, AI Methods in Statistics, UNICOM Seminar, London, December 1986. Also in Interactions in AI and Statistics (ed. B. Phelps) London; Gower Technical Press, 1987.Google Scholar
  3. Breiman, L., Friedman, J.H., Olshen, R.A., Stone, C.J. (1984), Classification and Regression Trees, Belmont, California: Wadsworth Int. Group.Google Scholar
  4. Cestnik, B., Kononenko, I., Bratko, I. (1987), ASSISTANT 86: A Knowledge-Elicitation Tool for Sophisticated Users, Progress in Machine Learning, Eds. I. Bratko & N. Lavrac, Sigma Press, Wilmslow.Google Scholar
  5. Cestnik, B. (1990), Estimating Probabilities: A Crucial Task in Machine Learning. In Proceedings of ECAI 90, Stockholm, August 1990.Google Scholar
  6. Clark, P., Niblett, T. (1987), Induction in Noisy Domains, Progress in Machine Learning, Eds. I. Bratko & N. Lavrac, Sigma Press, Wilmslow.Google Scholar
  7. Good, I.J. (1965), The Estimation of Probabilities, M.I.T. Press, Cambridge, Massachusetts.Google Scholar
  8. Mingers, J. (1989), An Empirical Comparison of Pruning Methods for Decision Tree Induction, Machine Learning vol. 4, no. 2, Kluwer Academic Publishers.Google Scholar
  9. Niblett, T., Bratko, I. (1986), Learning decision rules in noisy domains, Expert Systems 86, Cambridge University Press (Proceddings of Expert Systems 86 Conf., Brighton 1986).Google Scholar
  10. Pearl, J. (1988), Probabilistic Reasoning in Intelligent Systems: Networks of Plausible Inference, Morgan Kaufmann: San Mateo, CA.Google Scholar
  11. Quinlan, J.R. (1986), Learning from noisy data, Machine Learning vol. 2, Eds. R. Michalski, J. Carbonell and T. Mitchel, Palo Alto, CA: Tioga.Google Scholar
  12. Quinlan, J.R. (1987), Simplifying decision trees, International Journal of Man-Machine Studies, 27, pp. 221–234.Google Scholar
  13. Smyth, P., Goodman, R.M., Higgins, C. (1990), A Hybrid Rule-based/Bayesian Classifier, In Proceedings of ECAI 90, Stockholm, August 1990.Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 1991

Authors and Affiliations

  1. 1.Jožef Stefan InstituteLjubljanaYugoslavia
  2. 2.Faculty of Electrical Eng. and Computer ScienceLjubljanaYugoslavia

Personalised recommendations