Advertisement

MML Inference of Oblique Decision Trees

  • Peter J. Tan
  • David L. Dowe
Part of the Lecture Notes in Computer Science book series (LNCS, volume 3339)

Abstract

We propose a multivariate decision tree inference scheme by using the minimum message length (MML) principle (Wallace and Boulton, 1968; Wallace and Dowe, 1999). The scheme uses MML coding as an objective (goodness-of-fit) function on model selection and searches with a simple evolution strategy. We test our multivariate tree inference scheme on UCI machine learning repository data sets and compare with the decision tree programs C4.5 and C5. The preliminary results show that on average and on most data-sets, MML oblique trees clearly perform better than both C4.5 and C5 on both “right”/“wrong” accuracy and probabilistic prediction – and with smaller trees, i.e., less leaf nodes.

Keywords

Decision Tree Bayesian Network Leaf Node Internal Node Probabilistic Prediction 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. 1.
    Blake, C.L., Merz, C.J.: UCI repository of machine learning databases (1998), http://www.ics.uci.edu/~mlearn/MLRepository.html
  2. 2.
    Breiman, L., Friedman, J.H., Olshen, R.A., Stone, C.J.: Classification And Regression Trees. Wadsworth/Brooks (1984)Google Scholar
  3. 3.
    Cantu-Paz, E., Kamath, C.: Using evolutionary algorithms to induce oblique decision trees. In: Proc.Genetic and Evolutionary Computation Conference, Las Vegas, Nevada, USA, pp. 1053–1060. Morgan Kaufmann, San Francisco (2000)Google Scholar
  4. 4.
    Comley, J.W., Dowe, D.L.: Generalised Bayesian networks and asymmetric languages. In: Proc. Hawaii International Conference on Statistics and Related Fields, June 5-8 (2003)Google Scholar
  5. 5.
    Comley, J.W., Dowe, D.L.: Minimum message length, MDL and generalised Bayesian networks with asymmetric languages. In: Grünwald, P., Pitt, M.A., Myung, I.J. (eds.) Advances in Minimum Description Length: Theory and Applications (MDL Handbook), MIT Press, Cambridge (to appear)Google Scholar
  6. 6.
    Dowe, D.L., Farr, G.E., Hurst, A.J., Lentin, K.L.: Information-theoretic football tipping. In: de Mestre, N. (ed.) Third Australian Conference on Mathematics and Computers in Sport, Bond University, Qld, Australia, pp. 233–241 (1996) http://www.csse.monash.edu.au/~footy
  7. 7.
    Dowe, D.L., Krusel, N.: A decision tree model of bushfire activity. In (Technical report 93/190) Dept. Comp. Sci., Monash Uni., Clayton, Australia (1993)Google Scholar
  8. 8.
    Dowe, D.L., Wallace, C.S.: Kolmogorov complexity, minimum message length and inverse learning. In: 14th Australian Statistical Conference (ASC-14), Gold Coast, Qld, Australia, 6-10 July, p. 144 (1998)Google Scholar
  9. 9.
    Heath, D.G., Kasif, S., Salzberg, S.: Induction of oblique decision trees. In: International Joint Conference on AI (IJCAI), pp. 1002–1007 (1993)Google Scholar
  10. 10.
    Murthy, S.K.: On Growing Better Decision Trees from Data. PhD thesis, The John Hopkins University (1997)Google Scholar
  11. 11.
    Needham, S.L., Dowe, D.L.: Message length as an effective Ockham’s razor in decision tree induction. In: Proc. 8th International Workshop on Artificial Intelligence and Statistics, Key West, Florida, U.S.A, pp. 253–260 (January 2001)Google Scholar
  12. 12.
    Oliver, J.J., Wallace, C.S.: Inferring Decision Graphs. In: Workshop 8 International Joint Conference on AI (IJCAI), Sydney, Australia (August 1991)Google Scholar
  13. 13.
    Quinlan, J.R.: C4.5: Programs for Machine Learning. Morgan Kaufmann,San Mateo,CA (1992), The latest version of C5 is, available from http://www.rulequest.com
  14. 14.
    Quinlan, J.R., Rivest, R.: Inferring Decision Trees Using the Minimum Description Length Principle. Information and Computation 80, 227–248 (1989)zbMATHCrossRefMathSciNetGoogle Scholar
  15. 15.
    Schack, R., Ariano, G.M.D., Caves, C.M.: Hypersensitivity to perturbation in the quantum kicked top. Physical Review E 50, 972–987 (1994)CrossRefGoogle Scholar
  16. 16.
    Tan, P.J., Dowe, D.L.: MML inference of decision graphs with multi-way joins. In: McKay, B., Slaney, J.K. (eds.) Canadian AI 2002. LNCS (LNAI), vol. 2557, pp. 131–142. Springer, Heidelberg (2002)CrossRefGoogle Scholar
  17. 17.
    Tan, P.J., Dowe, D.L.: MML inference of decision graphs with multi-way joins and dynamic attributes. In: Gedeon, T(T.) D., Fung, L.C.C. (eds.) AI 2003. LNCS (LNAI), vol. 2903, pp. 269–281. Springer, Heidelberg (2003), http://www.csse.monash.edu.au/~dld/Publications/2003/Tan+Dowe2003.ref CrossRefGoogle Scholar
  18. 18.
    Wallace, C.: Statistical and Inductive Inference by Minimum Message Length. Springer, Heidelberg (to appear)Google Scholar
  19. 19.
    Wallace, C.S., Boulton, D.M.: An Information Measure for Classification. Computer Journal 11, 185–194 (1968)zbMATHGoogle Scholar
  20. 20.
    Wallace, C.S., Dowe, D.L.: Minimum Message Length and Kolmogorov Complexity. Computer Journal 42(4), 270–283 (1999)zbMATHCrossRefGoogle Scholar
  21. 21.
    Wallace, C.S., Freeman, P.R.: Estimation and Inference by Compact Coding. Journal of the Royal Statistical Society. Series B 49(3), 240–265 (1987)zbMATHMathSciNetGoogle Scholar
  22. 22.
    Wallace, C.S., Patrick, J.D.: Coding Decision Trees. Machine Learning 11, 7–22 (1993)zbMATHCrossRefGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2004

Authors and Affiliations

  • Peter J. Tan
    • 1
  • David L. Dowe
    • 1
  1. 1.School of Computer Science and Software EngineeringMonash UniversityClaytonAustralia

Personalised recommendations