Abstract
Among the numerous learning tasks that fall within the field of knowledge discovery in databases, classification may be the most common. Furthermore, top-down induction of decision trees is one of the most popular techniques for inducing such classification models. Most of the research in decision tree induction has focused on single attribute trees, but in this chapter we review multi-attribute decision trees induction and discuss how such methods can improve both the accuracy and simplicity of the decision trees. As an example of this approach we consider the recently proposed second order decision tree induction (SODI) algorithm, which uses conjunctive and disjunctive combinations of two attributes for improved decision tree induction in nominal databases. We show via numerical examples that in many cases this generates more accurate classification models and easier to interpret decision trees and rules.
Triantaphyllou, E. and G. Felici (Eds.), Data Mining and Knowledge Discovery Approaches Based on Rule Induction Techniques, Massive Computing Series, Springer, Heidelberg, Germany, pp. 327–358, 2006.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
M. Ali and M. Pazzani, “HYDRA: A nose-tolerant relational concept learning algorithm,” International Joint Conference on Artificial Intelligence, Chambery, France, 1995.
M. Ali and M. Pazzani, “Reducing the small disjuncts problems by learning probabilistic concept descriptions,” Computational Learning Theory and Natural Learning Systems, Vol. 3, pp. 183–199, 1995.
E. Baker and A. K. Jain, “On feature ordering in practice and some finite sample effects,” Proceedings of the 3 rd International Joint Conference on Pattern Recognition, pp. 45–49, San Diego, CA, USA, 1976.
M. Ben-Bassat, “Use of distance measures, information measures and error bounds on feature evaluation,” Classification, Pattern Recognition and Reduction of Dimensionality, In K. P. R. Krishnaiah and L. N. Kanal, editors, Vol. 2 of Handbook of Statistics, North-Holland Publishing Company, Amsterdam, The Netherlands, pp. 773–791, 1987.
M. Ben-Bassat, “Myopic policies in sequential classification,” IEEE Transactions on Computing, Vol. 27, No, 2, pp. 170–174, 1978.
K. P. Bennett and O. L. Mangasarian, “Multicategory Discrimination via Linear Programming,” Optimization Methods and Software, Vol. 3, pp. 29–39, 1994.
J. C. Bioch, O. Van der Meer, and R. Potharst, “Bivariate Decision Trees”, in J. Komorowski, J. Zytkow, eds. Principles of Data Mining and Knowledge Discovery, Lecture Notes in Artificial Intelligence 1263, Springer Verlag, 1997, New York, NY, USA, pp. 232–243.
H. Blocked and L. De Raedt, “Top-down induction of first order logical decision trees,” Artificial Intelligence, Vol. 101, pp.285–297, 1998.
L. Breiman, J. H. Friedman, R. A. Olshen, C. J. Stone, Classification and regression trees, Wadsworh International Group, Belmont, CA, USA, 1984.
C. E. Brodley and P. E. Utgoff, “Multivariate Decision Trees,” Machine Learning, Vol. 19, pp. 45–77, 1995.
W. Buntine, “Learning classification trees,” Statistics and Computing, Vol. 2, pp. 63–73, 1992.
R. G. Casey and G. Nagy, “Decision tree design using a probabilistic model,” IEEE Transactions on Information Theory, IT-30, No. 1, pp. 93–99, 1984.
T. G. Dietterich and E. B. Kong, “Machine learning bias, statistical bias and statistical variance of decision tree algorithms,” Machine Learning: Proceedings of the 12 th International Conference, Tahoe City, CA, USA, 1995.
L. De Màntaras, “Technical note: A distance-based attribute selection measure for decision tree induction,” Machine Learning, Vol. 6, No. 1, pp.81–92, 1991.
U. M. Fayyad, G. Piatetsky-Shapiro, P. Smyth, and R. Uthurusamy. Advances in Knowledge Discovery and Data Mining, MIT Press, Cambridge, MA, USA, 1996.
U. M. Fayyad, J. E. Laird, K. B. Irani, The Fifth International Conference on Machine Learning, AI Magazine Vol. 10, No. 2, pp. 79–84, 1989.
E. Frank and I. H. Witten, “Generating accurate rule sets without global optimization,” Machine Learning: Proceedings of the 15 th International Conference edited by J. Shavlik, Morgan Kaufmann, San Francisco, CA, USA, 1998.
S. B. Gelfand, C. S. Ravishankar, and E. J. Delp, “An iterative growing and pruning algorithm for classification tree design,” IEEE Transaction on Pattern Analysis and Machine Intelligence, Vol. 13, No. 2, pp. 163–174, 1991.
M. Golea and M. Marchand, “A growth algorithm for neural network decision trees,” EuroPhysics Letters, Vol. 12, No. 3, pp.205–210, 1990.
D. Heath, S. Kasif, and S. Salzberg, “Learning oblique decision trees,” IJCAI-93: Proceedings of the 13 th International Joint Conference On Artificial Intelligence, Vol. II, Chambery, France, 1993, Morgan Kaufmann, Vol. 160, pp. 1002–1007.
D. Heath, S. Kasif, and S. Salzberg, “k-DT: A multi-tree learning method,” Proceedings of the 2 nd International Workshop on Multistrategy Learning, Harpers Ferry, WV, 1993. George Mason University, pp. 138–149.
G. Kalkanis, “The application of confidence interval error analysis to the design of decision tree classifiers,” Pattern Recognition Letters, Vol. 14, No. 5, pp. 355–361, 1993.
I. Kononenko, “On biases in estimating multi-valued attributes,” in C. Mellishpages, ed, IJCAI-95: Proceedings of the 14 th International Joint Conference on Artificial Intelligence, Montreal, Canada, August 1995, Morgan Kaufmann, San Francisco, CA, USA, pp. 1034–1040.
S. W. Kwok, and C. Carter, “Multiple decision trees,” Uncertainty in Artificial Intelligence, Elsevier Science, Amsterdam, Vol. 4, pp. 327–335, 1990.
J. Lee, S. Olafsson, “SODI: A new approach of second order decision tree induction,” Working Paper, Department of Industrial and Manufacturing Systems Engineering, Iowa State University, Ames, IA, USA, 2002.
J. K. Martin, “An exact probability metric for decision tree splitting and stopping,” AI&Stats-95: the 5 th International Workshop on Artificial Intelligence and Statistics, Ft. Lauderdale, FL, 1995. Society for AI and Statistics, pp. 379–385.
J. Mingers, “Expert systems: rule induction with statistical data,” Journal of the Operational Research Society, Vol. 38, No. 1, pp. 39–47, 1987.
M. Miyakawa, “Criteria for selecting a variable in the construction of efficient decision trees,” IEEE Transactions on Computers, Vol. 38, No. 1, pp. 130–141, 1989.
P. M. Murphy and M. Pazzani, “ID2-of-3: Constructive induction of M-of-N concepts for discriminators in decision trees,” Proceedings of the 8 th International Workshop of Machine Learning, 1991.
P. M. Murphy and M. Pazzani, “Exploring the decision forest: An empirical investigation of Occam’s Razor in decision tree induction,” Journal of Artificial Intelligence Research, Vol. 1., pp.257–275, 1994.
S. K. Murthy, S. Kasif, and S. Salzberg, “A system for induction of oblique decision trees’” Journal of Artificial Intelligence Research, Vol. 2, pp. 1–33, 1994.
A. L. Oliveria and A. S. Vincentelli, “Learning complex Boolean functions: Algorithms and applications,” Advances in Neural Information Processing Systems 6, Morgan Kaufmann, San Francisco, CA, USA, 1993
K. R. Pattipati and M. G. Alexandridis, “Application of heuristic search and information theory to sequential fault diagnosis,” IEEE Transactions on Systems, Man and Cybernetics, Vol. 20, No. 4, pp. 872–887, 1990.
M. Pazzani, P. Murphy, K. Ali, and D. Schulenburg, “Trading off coverage for accuracy in forecasts: Applications to clinical data analysis,” AAAI Symposium on AI in Medicine, Stanford, CA, USA, pp. 106–110, 1994.
J. R. Quinlan, “Induction of decision trees,” Machine Learning, Vol. 1, pp. 81–106, 1986.
J. R. Quinlan, C4.5: Programs for machine learning, Vol. 29, pp. 5–44, Morgan Kaufmann, 1993.
J. R. Quinlan and R. L. Rivest, “Inferring decision trees using the minimum description length principle. Information and Computation,” Vol. 80, No. 3, pp. 227–248, 1989.
J. Risannen, Stochastic Complexity in Statistica Enquiry, World Scientific, Teaneck, N.J, USA, 1989.
I. K. Sethi and G. P. R. Sarvarayudu, “Hierarchical classifier design using mutual information,” IEEE Transactions on Pattern Analysis and Machine Intelligence, PAMI-4, No. 4, pp. 441–445, 1982.
C.E. Shannon, “A Mathematical Theory of Communication”, Bell System Technical Journal, Vol. 27, pp. 379–423, pp. 623–656, 1948.
S. Shlien, “Multiple binary decision tree classifiers,” Pattern Recognition, Vol. 23, No. 7, pp.757–763, 1990.
S. Shlien, “Nonparametric classification using matched binary decision trees,” Pattern Recognition Letters, Vol. 13, No. 2, pp. 83–88, 1992.
J. L. Talmon, “A multiclass nonparametric partitioning algorithm,” Pattern Recognition Letters, Vol. 4, pp. 31–38, 1986.
P. C. Taylor and B. W. Silverman, “Block diagrams and splitting criteria for classification trees,” Statistics and Computing, Vol. 3, No. 4, pp. 147–161, 1993.
T. Van De Merckt, “Decision trees in numerical attribute spaces,” In IJCAI-93: Proceedings of the 13th International Joint Conference on Artificial Intelligence, Vol. 2, Chambery, France, September 1993, Morgan Kaufmann, pp. 1016–1021.
P. K. Varshney, C. R. P. Hartmann, and J. M. De Faria Jr., “Applications of information theory to sequential fault diagnosis,” IEEE Transactions on Computers, Vol. 31, No. 2, pp. 164–170, 1982.
P. White and W. Z. Liu, “Technical note: Bias in information-based measures in decision tree induction,” Machine Learning, Vol. 15, No. 3, pp. 321–329, 1994.
H. Witten and E. Frank, Data mining: practical machine learning tools and techniques with Java implementations. Morgan Kaufmann, San Francisco, CA, USA, 1999.
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2006 Springer Science+Business Media, LLC
About this chapter
Cite this chapter
Lee, JY., Olafsson, S. (2006). Multi-Attribute Decision Trees and Decision Rules. In: Triantaphyllou, E., Felici, G. (eds) Data Mining and Knowledge Discovery Approaches Based on Rule Induction Techniques. Massive Computing, vol 6. Springer, Boston, MA . https://doi.org/10.1007/0-387-34296-6_10
Download citation
DOI: https://doi.org/10.1007/0-387-34296-6_10
Publisher Name: Springer, Boston, MA
Print ISBN: 978-0-387-34294-8
Online ISBN: 978-0-387-34296-2
eBook Packages: Computer ScienceComputer Science (R0)