Abstract
Machine learning algorithms for inferring decision trees typically choose a single “best” tree to describe the training data. Recent research has shown that classification performance can be significantly improved by voting predictions of multiple, independently produced decision trees. This paper describes an algorithm, OB1, that produces a weighted sum over many possible models. Model weights are determined by the prior probability of the model, as well as the performance of the model during training. We describe an implementation of OBI that includes all possible decision trees as well as naive Bayesian models within a single option tree. Constructing all possible decision trees is very expensive, growing exponentially in the number of attributes. However it is possible to use the internal structure of the option tree to avoid recomputing values. In addition, the current implementation allows the option tree to be depth bounded.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
D. W. Aha, D. Kibler, and M. K. Albert. Instance-based learning algorithms. Machine Learning, 6:37–66, 1991.
Peter Auer, Robert C. Holte, and Wolfgang Maass. Theory and applications of agnostic PAC-learning with small decision trees. In Prieditis A. and Russell S., editors, Proc. of the 12th International Conference on Machine Learning (ICML95), 1995.
R.A. Baxter and J.J. Oliver. MDL and MML: Similarities and differences (introduction to minimum encoding inference—part III). Technical Report Technical Report 207, Department of Computer Science, Monash University, Australia, 1994.
Wray Buntine. Classifiers: A theoretical and empirical study. In Proc. of the 1991 International Joint Conference on Artificial Intelligence, 1991.
J. Cleary, S. Legg, and I. H. Witten. An MDL estimate of the significance of rules. In Proc. of the Information, Statistics and Induction in Science Conference, pages 43–53, Melbourne, Australia, 1996.
Usama M. Fayyad and Keki B. Irani. On the handling of continuous-valued attributes in decision tree generation. Machine Learning, 8:87–102, 1992.
G. Holmes, A. Donkin, and I. H. Witten. WEKA: A machine learning workbench. In Proc. of the Second Australia and New Zealand Conference on Intelligent Information Systems, Brisbane, Australia, 1994. [webpage at http://www.cs.waikato.ac.nz/~ml/].
P. Langley, W. Iba, and K. Thompson. An analysis of Bayesian classifiers. In Proc. of the Tenth National Conference on Artificial Intelligence, pages 223–228, 1992.
C.J. Merz and P.M. Murphy. UCI Repository of Machine Learning Data-Bases. University of California, Dept. of Information and Computer Science, Irvine, CA, 1996.
W.J. Oates. The Stoic and Epicurean Philosophers: The Complete Extant Writings of Epicurus, Epictetus, Lucretius, Marcus Aurelius. Random House, New York, 1957.
J. Ross Quinlan. C4.5: Programs for Machine Learning. Morgan Kaufmann, San Mateo, CA, 1994.
R.E. Schapire, Y. Freund, P. Bartlett, and Wee Sun Lee. Boosting the margin: A new explanation for the effectiveness of voting methods. In Proc. of the 14th International Conference on Machine Learning (ICML97), pages 322–330, 1997.
P.A.J Volf. Deriving MDL-decision trees using the context maximizing algorithm. Master’s thesis, Department of Electrical Engineering, Eindhoven University of Technology, 1994.
Frans M. J. Willems, Yuri M. Shtarkov, and Tjalling J. Tjalkens. The context-tree weighting method: Basic properties. IEEE Transactions on Information Theory, 41(3):653–664, May 1995.
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2001 Springer-Verlag London
About this paper
Cite this paper
Cleary, J.G., Trigg, L.E., Holmes, G., Hall, M. (2001). Experiences with a weighted decision tree learner. In: Bramer, M., Preece, A., Coenen, F. (eds) Research and Development in Intelligent Systems XVII. Springer, London. https://doi.org/10.1007/978-1-4471-0269-4_3
Download citation
DOI: https://doi.org/10.1007/978-1-4471-0269-4_3
Publisher Name: Springer, London
Print ISBN: 978-1-85233-403-1
Online ISBN: 978-1-4471-0269-4
eBook Packages: Springer Book Archive