Skip to main content

Decision Trees

  • Chapter

Abstract

Decision Trees are considered to be one of the most popular approaches for representing classifiers. Researchers from various disciplines such as statistics, machine learning, pattern recognition, and Data Mining have dealt with the issue of growing a decision tree from available data. This paper presents an updated survey of current methods for constructing decision tree classifiers in a top-down manner. The chapter suggests a unified algorithmic framework for presenting these algorithms and describes various splitting criteria and pruning methodologies.

Keywords

  • Decision tree
  • Information Gain
  • Gini Index
  • Gain Ratio
  • Pruning
  • Minimum Description
  • Length
  • C4.5
  • CART
  • Oblivious Decision Trees

This is a preview of subscription content, access via your institution.

Buying options

Chapter
USD   29.95
Price excludes VAT (USA)
  • DOI: 10.1007/0-387-25465-X_9
  • Chapter length: 28 pages
  • Instant PDF download
  • Readable on all devices
  • Own it forever
  • Exclusive offer for individuals only
  • Tax calculation will be finalised during checkout
eBook
USD   229.00
Price excludes VAT (USA)
  • ISBN: 978-0-387-25465-4
  • Instant PDF download
  • Readable on all devices
  • Own it forever
  • Exclusive offer for individuals only
  • Tax calculation will be finalised during checkout

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  • Almuallim H., An Efficient Algorithm for Optimal Pruning of Decision Trees. Artificial Intelligence 83(2): 347–362, 1996.

    CrossRef  Google Scholar 

  • Almuallim Hv and Dietterich T.G., Learning Boolean concepts in the presence of many irrelevant features. Artificial Intelligence, 69:1–2, 279–306, 1994.

    MathSciNet  CrossRef  Google Scholar 

  • Alsabti K., Ranka S. and Singh V., CLOUDS: A Decision Tree Classifier for Large Datasets, Conference on Knowledge Discovery and Data Mining (KDD-98), August 1998.

    Google Scholar 

  • Attneave F, Applications of Information Theory to Psychology. Holt, Rinehart and Winston, 1959.

    Google Scholar 

  • Baker E., and Jain A. K., On feature ordering in practice and some finite sample effects. In Proceedings of the Third International Joint Conference on Pattern Recognition, pages 45–49, San Diego, CA, 1976.

    Google Scholar 

  • BenBassat M., Myopic policies in sequential classification. IEEE Trans. on Computing, 27(2): 170–174, February 1978.

    MathSciNet  Google Scholar 

  • Bennett X. and Mangasarian O.L., Multicategory discrimination via linear programming. Optimization Methods and Software, 3:29–39, 1994.

    Google Scholar 

  • Bratko I., and Bohanec M., Trading accuracy for simplicity in decision trees, Machine Learning 15: 223–250, 1994.

    MATH  Google Scholar 

  • Breiman L., Friedman J., Olshen R., and Stone C.. Classification and Regression Trees. Wadsworth Int. Group, 1984.

    Google Scholar 

  • Brodley C. E. and Utgoff. P. E., Multivariate decision trees. Machine Learning, 19:45–77, 1995.

    MATH  Google Scholar 

  • Buntine W., Niblett T, A Further Comparison of Splitting Rules for Decision-Tree Induction. Machine Learning, 8: 75–85, 1992.

    Google Scholar 

  • Catlett J., Mega induction: Machine Learning on Vary Large Databases, PhD, University of Sydney, 1991.

    Google Scholar 

  • Chan P.K. and Stolfo S.J, On the Accuracy of Meta-learning for Scalable Data Mining, J. Intelligent Information Systems, 8:5–28, 1997.

    CrossRef  Google Scholar 

  • Crawford S. L., Extensions to the CART algorithm. Int. J. of ManMachine Studies, 31(2): 197–217, August 1989.

    MathSciNet  CrossRef  Google Scholar 

  • Dietterich, T. G., Kearns, M., and Mansour, Y., Applying the weak learning framework to understand and improve C4.5. Proceedings of the Thirteenth International Conference on Machine Learning, pp. 96–104, San Francisco: Morgan Kaufmann, 1996.

    Google Scholar 

  • Duda, R., and Hart, P., Pattern Classification and Scene Analysis, New-York, Wiley, 1973.

    MATH  Google Scholar 

  • Esposito E, Malerba D. and Semeraro G., A Comparative Analysis of Methods for Pruning Decision Trees. EEE Transactions on Pattern Analysis and Machine Intelligence, 19(5):476–492, 1997.

    CrossRef  Google Scholar 

  • Fayyad U., and Irani K. B., The attribute selection problem in decision tree generation. In proceedings of Tenth National Conference on Artificial Intelligence, pp. 104–110, Cambridge, MA: AAAI Press/MIT Press, 1992.

    Google Scholar 

  • Ferri C, Flach P., and Hernandez-Orallo J., Learning Decision Trees Using the Area Under the ROC Curve. In Claude Sammut and Achim Hoffmann, editors, Proceedings of the 19th International Conference on Machine Learning, pp. 139–146. Morgan Kaufmann, July 2002

    Google Scholar 

  • Fifield D. J., Distributed Tree Construction From Large Datasets, Bachelor’s Honor Thesis, Australian National University, 1992.

    Google Scholar 

  • Freitas X., and Lavington S. H., Mining Very Large Databases With Parallel Processing, Kluwer Academic Publishers, 1998.

    Google Scholar 

  • Friedman J. H., A recursive partitioning decision rule for nonparametric classifiers. IEEE Trans, on Comp., C26:404–408, 1977.

    MATH  Google Scholar 

  • Friedman, J. H., “Multivariate Adaptive Regression Splines”, The Annual Of Statistics, 19, 1–141, 1991.

    MATH  Google Scholar 

  • Gehrke J., Ganti V., Ramakrishnan R., Loh W., BOAT-Optimistic Decision Tree Construction. SIGMOD Conference 1999: pp. 169–180, 1999.

    Google Scholar 

  • Gehrke J., Ramakrishnan R., Ganti V., RainForest-A Framework for Fast Decision Tree Construction of Large Datasets, Data Mining and Knowledge Discovery, 4,2/3) 127–162, 2000.

    CrossRef  Google Scholar 

  • Gelfand S. B., Ravishankar C. S., and Delp E. J., An iterative growing and pruning algorithm for classification tree design. IEEE Transaction on Pattern Analysis and Machine Intelligence, 13(2): 163–174, 1991.

    CrossRef  Google Scholar 

  • Gillo M. W., MAID: A Honeywell 600 program for an automatised survey analysis. Behavioral Science 17: 251–252, 1972.

    Google Scholar 

  • Hancock T. R., Jiang T, Li M., Tromp J., Lower Bounds on Learning Decision Lists and Trees. Information and Computation 126(2): 114–122, 1996.

    MathSciNet  CrossRef  MATH  Google Scholar 

  • Holte R. C, Very simple classification rules perform well on most commonly used datasets. Machine Learning, 11:63–90, 1993.

    MATH  CrossRef  Google Scholar 

  • Hyafil L. and Rivest R.L., Constructing optimal binary decision trees is NP-complete. Information Processing Letters, 5(1):15–17, 1976

    MathSciNet  CrossRef  MATH  Google Scholar 

  • Janikow, C.Z., Fuzzy Decision Trees: Issues and Methods, IEEE Transactions on Systems, Man, and Cybernetics, Vol. 28, Issue 1, pp. 1–14. 1998.

    Google Scholar 

  • John G. H., Robust linear discriminant trees. In D. Fisher and H. Lenz, editors, Learning From Data: Artificial Intelligence and Statistics V, Lecture Notes in Statistics, Chapter 36, pp. 375–385. Springer-Verlag, New York, 1996.

    Google Scholar 

  • Kass G. V, An exploratory technique for investigating large quantities of categorical data. Applied Statistics, 29(2):119–127, 1980.

    Google Scholar 

  • Kearns M. and Mansour Y., A fast, bottom-up decision tree pruning algorithm with near-optimal generalization, in J. Shavlik, ed., ‘Machine Learning: Proceedings of the Fifteenth International Conference’, Morgan Kaufmann Publishers, Inc., pp. 269–277, 1998.

    Google Scholar 

  • Kearns M. and Mansour Y, On the boosting ability of top-down decision tree learning algorithms. Journal of Computer and Systems Sciences, 58(1): 109–128, 1999.

    MathSciNet  CrossRef  MATH  Google Scholar 

  • Kohavi R. and Sommerfield D., Targeting business users with decision table classifiers, in R. Agrawal, P. Stolorz & G. Piatetsky-Shapiro, eds, ‘Proceedings of the Fourth International Conference on Knowledge Discovery and Data Mining’, AAAI Press, pp. 249–253, 1998.

    Google Scholar 

  • Langley, P. and Sage, S., Oblivious decision trees and abstract cases. in Working Notes of the AAAI-94 Workshop on Case-Based Reasoning, pp. 113–117, Seattle, WA: AAAI Press, 1994.

    Google Scholar 

  • Last, M., Maimon, O. and Minkov, E., Improving Stability of Decision Trees, International Journal of Pattern Recognition and Artificial Intelligence, 16: 2, 145–159, 2002.

    CrossRef  Google Scholar 

  • Li X. and Dubes R. C, Tree classifier design with a Permutation statistic, Pattern Recognition 19:229–235, 1986.

    CrossRef  Google Scholar 

  • Lim X., Loh W.Y., and Shih X., A comparison of prediction accuracy, complexity, and training time of thirty-three old and new classification algorithms. Machine Learning 40:203–228, 2000.

    CrossRef  MATH  Google Scholar 

  • Lin Y. K. and Fu K., Automatic classification of cervical cells using a binary tree classifier. Pattern Recognition, 16(1):69–80, 1983.

    CrossRef  Google Scholar 

  • Loh W.Y.,and Shih X., Split selection methods for classification trees. Statistica Sinica, 7: 815–840, 1997.

    MathSciNet  MATH  Google Scholar 

  • Loh W.Y. and Shih X., Families of splitting criteria for classification trees. Statistics and Computing 9:309–315, 1999.

    CrossRef  Google Scholar 

  • Loh W.Y. and Vanichsetakul N., Tree-structured classification via generalized discriminant Analysis. Journal of the American Statistical Association, 83: 715–728, 1988.

    MathSciNet  CrossRef  MATH  Google Scholar 

  • Lopez de Mantras R., A distance-based attribute selection measure for decision tree induction, Machine Learning 6:81–92, 1991.

    Google Scholar 

  • Lubinsky D., Algorithmic speedups in growing classification trees by using an additive split criterion. Proc. AI&Statistics93, pp. 435–444, 1993.

    Google Scholar 

  • Martin J. K., An exact probability metric for decision tree splitting and stopping. An Exact Probability Metric for Decision Tree Splitting and Stopping, Machine Learning, 28,2–3):257–291, 1997.

    CrossRef  Google Scholar 

  • Mehta M., Rissanen J., Agrawal R., MDL-Based Decision Tree Pruning. KDD 1995: pp. 216–221, 1995.

    Google Scholar 

  • Mehta M., Agrawal R. and Rissanen J., SLIQ: A fast scalable classifier for Data Mining: In Proc. If the fifth Int’l Conference on Extending Database Technology (EDBT), Avignon, France, March 1996.

    Google Scholar 

  • Mingers J., An empirical comparison of pruning methods for decision tree induction. Machine Learning, 4(2):227–243, 1989.

    CrossRef  Google Scholar 

  • Morgan J. N. and Messenger R. C, THAID: a sequential search program for the analysis of nominal scale dependent variables. Technical report, Institute for Social Research, Univ. of Michigan, Ann Arbor, MI, 1973.

    Google Scholar 

  • Muller W., and Wysotzki F, Automatic construction of decision trees for classification. Annals of Operations Research, 52:231–247, 1994.

    CrossRef  Google Scholar 

  • Murthy S. K., Automatic Construction of Decision Trees from Data: A Multi-Disciplinary Survey. Data Mining and Knowledge Discovery, 2(4):345–389, 1998.

    CrossRef  Google Scholar 

  • Naumov G.E., NP-completeness of problems of construction of optimal decision trees. Soviet Physics: Doklady, 36(4):270–271, 1991.

    MATH  MathSciNet  Google Scholar 

  • Niblett T. and Bratko I., Learning Decision Rules in Noisy Domains, Proc. Expert Systems 86, Cambridge: Cambridge University Press, 1986.

    Google Scholar 

  • Olaru C, Wehenkel L., A complete fuzzy decision tree technique, Fuzzy Sets and Systems, 138(2):221–254, 2003.

    MathSciNet  CrossRef  Google Scholar 

  • Pagallo, G. and Huassler, D., Boolean feature discovery in empirical learning, Machine Learning, 5(1): 71–99, 1990.

    CrossRef  Google Scholar 

  • Peng Y., Intelligent condition monitoring using fuzzy inductive learning, Journal of Intelligent Manufacturing, 15(3): 373–380, June 2004.

    CrossRef  Google Scholar 

  • Quinlan, J.R., Induction of decision trees, Machine Learning 1, 81–106, 1986.

    Google Scholar 

  • Quinlan, J.R., Simplifying decision trees, International Journal of Man-Machine Studies, 27, 221–234, 1987.

    CrossRef  Google Scholar 

  • Quinlan, J.R., Decision Trees and Multivalued Attributes, J. Richards, ed., Machine Intelligence, V. 11, Oxford, England, Oxford Univ. Press, pp. 305–318, 1988.

    Google Scholar 

  • Quinlan, J. R., Unknown attribute values in induction. In Segre, A. (Ed.), Proceedings of the Sixth International Machine Learning Workshop Cornell, New York. Morgan Kaufmann, 1989.

    Google Scholar 

  • Quinlan, J. R., C4.5: Programs for Machine Learning, Morgan Kaufmann, Los Altos, 1993.

    Google Scholar 

  • Quinlan, J. R. and Rivest, R. L., Inferring Decision Trees Using The Minimum Description Length Principle. Information and Computation, 80:227–248, 1989.

    MathSciNet  CrossRef  MATH  Google Scholar 

  • Rastogi, R., and Shim, K., PUBLIC: A Decision Tree Classifier that Integrates Building and Pruning, Data Mining and Knowledge Discovery, 4(4):315–344, 2000.

    CrossRef  MATH  Google Scholar 

  • Rissanen, J., Stochastic complexity and statistical inquiry. World Scientific, 1989.

    Google Scholar 

  • Rounds, E., A combined non-parametric approach to feature selection and binary decision tree design, Pattern Recognition 12, 313–317, 1980.

    CrossRef  Google Scholar 

  • Schlimmer, J. C., Efficiently inducing determinations: A complete and systematic search algorithm that uses optimal pruning. In Proceedings of the 1993 International Conference on Machine Learning: pp 284–290, San Mateo, CA, Morgan Kaufmann, 1993.

    Google Scholar 

  • Sethi, K., and Yoo, J. H., Design of multicategory, multifeature split decision trees using perceptron learning. Pattern Recognition, 27(7):939–947, 1994.

    CrossRef  Google Scholar 

  • Shafer, J. C, Agrawal, R. and Mehta, M., SPRINT: A Scalable Parallel Classifier for Data Mining, Proc. 22nd Int. Conf. Very Large Databases, T. M. Vijayaraman and Alejandro P. Buchmann and C. Mohan and Nandlal L. Sarda (eds), 544–555, Morgan Kaufmann, 1996.

    Google Scholar 

  • Sklansky, J. and Wassel, G. N., Pattern classifiers and trainable machines. SpringerVerlag, New York, 1981.

    MATH  Google Scholar 

  • Sonquist, J. A., Baker E. L., and Morgan, J. N., Searching for Structure. Institute for Social Research, Univ. of Michigan, Ann Arbor, MI, 1971.

    MATH  Google Scholar 

  • Taylor P. C, and Silverman, B. W., Block diagrams and splitting criteria for classification trees. Statistics and Computing, 3(4):147–161, 1993.

    CrossRef  Google Scholar 

  • Utgoff, P. E., Perceptron trees: A case study in hybrid concept representations. Connection Science, 1(4):377–391, 1989.

    Google Scholar 

  • Utgoff, P. E., Incremental induction of decision trees. Machine Learning, 4: 161–186, 1989.

    CrossRef  Google Scholar 

  • Utgoff, P. E., Decision tree induction based on efficient tree restructuring, Machine Learning 29,1):5–44, 1997.

    MATH  CrossRef  Google Scholar 

  • Utgoff, P. E., and Clouse, J. A., A Kolmogorov-Smirnoff Metric for Decision Tree Induction, Technical Report 96-3, University of Massachusetts, Department of Computer Science, Amherst, MA, 1996.

    Google Scholar 

  • Wallace, C. S., and Patrick J., Coding decision trees, Machine Learning 11: 7–22, 1993.

    CrossRef  MATH  Google Scholar 

  • Zantema, H., and Bodlaender H. L., Finding Small Equivalent Decision Trees is Hard, International Journal of Foundations of Computer Science, 11(2): 343–354, 2000.

    MathSciNet  CrossRef  Google Scholar 

Download references

Author information

Authors and Affiliations

Authors

Editor information

Editors and Affiliations

Rights and permissions

Reprints and Permissions

Copyright information

© 2005 Springer Science+Business Media, Inc.

About this chapter

Cite this chapter

Rokach, L., Maimon, O. (2005). Decision Trees. In: Maimon, O., Rokach, L. (eds) Data Mining and Knowledge Discovery Handbook. Springer, Boston, MA. https://doi.org/10.1007/0-387-25465-X_9

Download citation

  • DOI: https://doi.org/10.1007/0-387-25465-X_9

  • Publisher Name: Springer, Boston, MA

  • Print ISBN: 978-0-387-24435-8

  • Online ISBN: 978-0-387-25465-4

  • eBook Packages: Computer ScienceComputer Science (R0)