Evolutionary Induction of Decision Trees for Misclassification Cost Minimization

  • Marek Krȩtowski
  • Marek Grześ
Part of the Lecture Notes in Computer Science book series (LNCS, volume 4431)

Abstract

In the paper, a new method of decision tree learning for cost-sensitive classification is presented. In contrast to the traditional greedy top-down inducer in the proposed approach optimal trees are searched in a global manner by using an evolutionary algorithm (EA). Specialized genetic operators are applied to modify both the tree structure and tests in non-terminal nodes. A suitably defined fitness function enables the algorithm to minimize the misclassification cost instead of the number of classification errors. The performance of the EA-based method is compared to three well-recognized algorithms on real-life problems with known and randomly generated cost-matrices. Obtained results show that the proposed approach is competitive both in terms of misclassification cost and compactness of the classifier at least for some datasets.

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. 1.
    Abe, N., Zadrozny, B., Langford, J.: An iterative method for multi-class cost-sensitive learning. In: KDD’04, pp. 3–11. ACM Press, New York (2004)Google Scholar
  2. 2.
    Blake, C., Keogh, E., Merz, C.: UCI repository of machine learning databases, (1998), http://www.ics.uci.edu/~mlearn/MLRepository.html
  3. 3.
    Bradford, J.P., Kunz, C., Kohavi, R., Brunk, C., Brodley, C.E.: Pruning decision trees with misclassification costs. In: Nédellec, C., Rouveirol, C. (eds.) ECML 1998. LNCS, vol. 1398, pp. 131–136. Springer, Heidelberg (1998)CrossRefGoogle Scholar
  4. 4.
    Breiman, L., Friedman, J., Olshen, R., Stone, C.: Classification and Regression Trees. Wadsworth Int. Group, Belmont (1984)MATHGoogle Scholar
  5. 5.
    Cantu-Paz, E., Kamath, C.: Inducing oblique decision trees with evolutionary algorithms. IEEE Transactions on Evolutionary Computation 7(1), 54–68 (2003)CrossRefGoogle Scholar
  6. 6.
    Chai, B., et al.: Piecewise-linear classifiers using binary tree structure and genetic algorithm. Pattern Recognition 29(11), 1905–1917 (1996)CrossRefMathSciNetGoogle Scholar
  7. 7.
    Domingos, P.: MetaCost: A general method for making classifiers cost-sensitive. In: Proc. of KDD’99, pp. 155–164. ACM Press, New York (1999)Google Scholar
  8. 8.
    Fayyad, U.M., Irani, K.B.: Multi-interval discretization of continuous-valued attributes for classification learning. In: Proc. of IJCAI’93, pp. 1022–1027 (1993)Google Scholar
  9. 9.
    Frank, E., et al.: Weka 3 - Data Mining with Open Source Machine Learning Software in Java. University of Waikato (2000), http://www.cs.waikato.ac.nz/~ml/weka
  10. 10.
    Knoll, U., Nakhaeizadeh, G., Tausend, B.: Cost-sensitive pruning of decision trees. In: Bergadano, F., De Raedt, L. (eds.) ECML 1994. LNCS, vol. 784, pp. 383–386. Springer, Heidelberg (1994)Google Scholar
  11. 11.
    Koza, J.: Concept formation and decision tree induction using genetic programming paradigm. In: Schwefel, H.-P., Männer, R. (eds.) PPSN 1990. LNCS, vol. 496, pp. 124–128. Springer, Heidelberg (1991)CrossRefGoogle Scholar
  12. 12.
    Krȩtowski, M.: An evolutionary algorithm for oblique decision tree induction. In: Rutkowski, L., Siekmann, J.H., Tadeusiewicz, R., Zadeh, L.A. (eds.) ICAISC 2004. LNCS (LNAI), vol. 3070, pp. 432–437. Springer, Heidelberg (2004)Google Scholar
  13. 13.
    Krȩtowski, M., Grześ, M.: Global learning of decision trees by an evolutionary algorithm. In: Information Processing and Security Systems, pp. 401–410. Springer, Heidelberg (2005)CrossRefGoogle Scholar
  14. 14.
    Krȩtowski, M., Grześ, M.: Evolutionary learning of linear trees with embedded feature selection. In: Rutkowski, L., Tadeusiewicz, R., Zadeh, L.A., Żurada, J.M. (eds.) ICAISC 2006. LNCS (LNAI), vol. 4029, pp. 400–409. Springer, Heidelberg (2006)CrossRefGoogle Scholar
  15. 15.
    Krȩtowski, M., Grześ, M.: Mixed decision trees: An evolutionary approach. In: Tjoa, A.M., Trujillo, J. (eds.) DaWaK 2006. LNCS, vol. 4081, pp. 260–269. Springer, Heidelberg (2006)CrossRefGoogle Scholar
  16. 16.
    Kwedlo, W., Krȩtowski, M.: An evolutionary algorithm for cost-sensitive decision rule learning. In: Flach, P.A., De Raedt, L. (eds.) ECML 2001. LNCS (LNAI), vol. 2167, pp. 288–299. Springer, Heidelberg (2001)CrossRefGoogle Scholar
  17. 17.
    Ling, C., Yang, Q., Wang, J., Zhang, S.: Decision trees with minimal costs. In: Proc. of ICML’04, Article No. 69, ACM Press, New York (2004)Google Scholar
  18. 18.
    Michalewicz, Z.: Genetic Algorithms + Data Structures = Evolution Programs, 3rd edn. Springer, Heidelberg (1996)MATHGoogle Scholar
  19. 19.
    Margineantu, D.D., Dietterich, T.G.: Bootstrap methods for the cost-sensitive evaluation of classifiers. In: Proc. of ICML’2000, pp. 583–590. Morgan Kaufmann, San Francisco (2000)Google Scholar
  20. 20.
    Papagelis, A., Kalles, D.: Breeding decision trees using evolutionary techniques. In: Proc. of ICML’01, pp. 393–400. Morgan Kaufmann, San Francisco (2001)Google Scholar
  21. 21.
    Quinlan, J.: C4.5: Programs for Machine Learning. Morgan Kaufmann, San Francisco (1993)Google Scholar
  22. 22.
    Ting, K.M.: An instance-weighting method to induce cost-sensitive trees. IEEE Transactions on Knowledge and Data Engineering 14(3), 659–665 (2002)CrossRefGoogle Scholar
  23. 23.
    Turney, P.: Cost-sensitive classification: Empirical evaluation of a hybrid genetic decision tree induction algorithm. Journal of Artificial Intelligence Research 2, 369–409 (1995)Google Scholar
  24. 24.
    Turney, P.: Types of cost in inductive concept learning. In: Proc. of ICML’2000 Workshop on Cost-Sensitive Learning, Stanford, CA (2000)Google Scholar
  25. 25.
    Zadrozny, B., Langford, J., Abe, N.: Cost-sensitive learning by cost-proportionate example weighting concept learning. In: Proc. of ICDM’03, IEEE Computer Society Press, Los Alamitos (2003)Google Scholar
  26. 26.
    Zhang, S., Qin, Z., Ling, C., Sheng, S.: Missing is usefull: Missing values in cost-sensitive decision trees. IEEE Transactions on Knowledge and Data Engineering 17(12), 1689–1693 (2005)CrossRefGoogle Scholar

Copyright information

© Springer Berlin Heidelberg 2007

Authors and Affiliations

  • Marek Krȩtowski
    • 1
  • Marek Grześ
    • 1
  1. 1.Faculty of Computer Science, Białystok Technical University, Wiejska 45a, 15-351 BiałystokPoland

Personalised recommendations