Abstract
We propose a multivariate decision tree inference scheme by using the minimum message length (MML) principle (Wallace and Boulton, 1968; Wallace and Dowe, 1999). The scheme uses MML coding as an objective (goodness-of-fit) function on model selection and searches with a simple evolution strategy. We test our multivariate tree inference scheme on UCI machine learning repository data sets and compare with the decision tree programs C4.5 and C5. The preliminary results show that on average and on most data-sets, MML oblique trees clearly perform better than both C4.5 and C5 on both “right”/“wrong” accuracy and probabilistic prediction – and with smaller trees, i.e., less leaf nodes.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
Blake, C.L., Merz, C.J.: UCI repository of machine learning databases (1998), http://www.ics.uci.edu/~mlearn/MLRepository.html
Breiman, L., Friedman, J.H., Olshen, R.A., Stone, C.J.: Classification And Regression Trees. Wadsworth/Brooks (1984)
Cantu-Paz, E., Kamath, C.: Using evolutionary algorithms to induce oblique decision trees. In: Proc.Genetic and Evolutionary Computation Conference, Las Vegas, Nevada, USA, pp. 1053–1060. Morgan Kaufmann, San Francisco (2000)
Comley, J.W., Dowe, D.L.: Generalised Bayesian networks and asymmetric languages. In: Proc. Hawaii International Conference on Statistics and Related Fields, June 5-8 (2003)
Comley, J.W., Dowe, D.L.: Minimum message length, MDL and generalised Bayesian networks with asymmetric languages. In: Grünwald, P., Pitt, M.A., Myung, I.J. (eds.) Advances in Minimum Description Length: Theory and Applications (MDL Handbook), MIT Press, Cambridge (to appear)
Dowe, D.L., Farr, G.E., Hurst, A.J., Lentin, K.L.: Information-theoretic football tipping. In: de Mestre, N. (ed.) Third Australian Conference on Mathematics and Computers in Sport, Bond University, Qld, Australia, pp. 233–241 (1996) http://www.csse.monash.edu.au/~footy
Dowe, D.L., Krusel, N.: A decision tree model of bushfire activity. In (Technical report 93/190) Dept. Comp. Sci., Monash Uni., Clayton, Australia (1993)
Dowe, D.L., Wallace, C.S.: Kolmogorov complexity, minimum message length and inverse learning. In: 14th Australian Statistical Conference (ASC-14), Gold Coast, Qld, Australia, 6-10 July, p. 144 (1998)
Heath, D.G., Kasif, S., Salzberg, S.: Induction of oblique decision trees. In: International Joint Conference on AI (IJCAI), pp. 1002–1007 (1993)
Murthy, S.K.: On Growing Better Decision Trees from Data. PhD thesis, The John Hopkins University (1997)
Needham, S.L., Dowe, D.L.: Message length as an effective Ockham’s razor in decision tree induction. In: Proc. 8th International Workshop on Artificial Intelligence and Statistics, Key West, Florida, U.S.A, pp. 253–260 (January 2001)
Oliver, J.J., Wallace, C.S.: Inferring Decision Graphs. In: Workshop 8 International Joint Conference on AI (IJCAI), Sydney, Australia (August 1991)
Quinlan, J.R.: C4.5: Programs for Machine Learning. Morgan Kaufmann,San Mateo,CA (1992), The latest version of C5 is, available from http://www.rulequest.com
Quinlan, J.R., Rivest, R.: Inferring Decision Trees Using the Minimum Description Length Principle. Information and Computation 80, 227–248 (1989)
Schack, R., Ariano, G.M.D., Caves, C.M.: Hypersensitivity to perturbation in the quantum kicked top. Physical Review E 50, 972–987 (1994)
Tan, P.J., Dowe, D.L.: MML inference of decision graphs with multi-way joins. In: McKay, B., Slaney, J.K. (eds.) Canadian AI 2002. LNCS (LNAI), vol. 2557, pp. 131–142. Springer, Heidelberg (2002)
Tan, P.J., Dowe, D.L.: MML inference of decision graphs with multi-way joins and dynamic attributes. In: Gedeon, T(T.) D., Fung, L.C.C. (eds.) AI 2003. LNCS (LNAI), vol. 2903, pp. 269–281. Springer, Heidelberg (2003), http://www.csse.monash.edu.au/~dld/Publications/2003/Tan+Dowe2003.ref
Wallace, C.: Statistical and Inductive Inference by Minimum Message Length. Springer, Heidelberg (to appear)
Wallace, C.S., Boulton, D.M.: An Information Measure for Classification. Computer Journal 11, 185–194 (1968)
Wallace, C.S., Dowe, D.L.: Minimum Message Length and Kolmogorov Complexity. Computer Journal 42(4), 270–283 (1999)
Wallace, C.S., Freeman, P.R.: Estimation and Inference by Compact Coding. Journal of the Royal Statistical Society. Series B 49(3), 240–265 (1987)
Wallace, C.S., Patrick, J.D.: Coding Decision Trees. Machine Learning 11, 7–22 (1993)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2004 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Tan, P.J., Dowe, D.L. (2004). MML Inference of Oblique Decision Trees. In: Webb, G.I., Yu, X. (eds) AI 2004: Advances in Artificial Intelligence. AI 2004. Lecture Notes in Computer Science(), vol 3339. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-540-30549-1_105
Download citation
DOI: https://doi.org/10.1007/978-3-540-30549-1_105
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-540-24059-4
Online ISBN: 978-3-540-30549-1
eBook Packages: Computer ScienceComputer Science (R0)