Advertisement

Soft Computing

, Volume 14, Issue 9, pp 973–993 | Cite as

Lossless fitness inheritance in genetic algorithms for decision trees

  • Dimitris Kalles
  • Athanasios Papagelis
Original Paper

Abstract

When genetic algorithms are used to evolve decision trees, key tree quality parameters can be recursively computed and re-used across generations of partially similar decision trees. Simply storing instance indices at leaves is sufficient for fitness to be piecewise computed in a lossless fashion. We show the derivation of the (substantial) expected speedup on two bounding case problems and trace the attractive property of lossless fitness inheritance to the divide-and-conquer nature of decision trees. The theoretical results are supported by experimental evidence.

Keywords

Decision trees Genetic algorithms Fitness inheritance Fitness approximation Learning speedup 

Notes

Acknowledgments

The study reported in this paper is research that has not previously been undertaken or published by the author. The opening of Sect. 2 borrows some text from Papagelis and Kalles (2001). Zygounas (2004) first ventured into experimenting with GATree and speedup techniques and has influenced the experimental understanding of what reasonable speedup during evolution might be. An anonymous reviewer pointed out that root crossover is very fast regardless of data structures, and several reviewers have made numerous comments that substantially improved the presentation of this work and the proper acknowledgement of work by other researchers.

References

  1. Altenberg L (1997) NK fitness landscapes. In: Back T, Fogel D, Michalewicz Z (eds) The handbook of evolutionary computation, sect. B2.7.2. Oxford University Press, OxfordGoogle Scholar
  2. Asuncion A, Newman DJ (2007) UCI machine learning repository. http://www.ics.uci.edu/~mlearn/MLRepository.html. University of California, School of Information and Computer Science, Irvine
  3. Bot M, Langdon W (2000) Application of genetic programming to induction of linear classification trees. In: Proceedings of the genetic and evolutionary computation conference. Las Vegas, NVGoogle Scholar
  4. Breiman L (2001) Random forests. Machine Learn 45(1):5–32zbMATHCrossRefGoogle Scholar
  5. Breiman L, Friedman JH, Olshen RA, Stone CJ (1984) Classification and regression trees. Wadsworth, BelmontzbMATHGoogle Scholar
  6. Cantú-Paz E, Kamath C (2003) Inducing oblique decision trees with evolutionary algorithms. IEEE Trans Evol Comput 7(1):54–68CrossRefGoogle Scholar
  7. Catlett J (1992). Peepholing: choosing attributes effectively for megainduction. In: Proceedings of the 9th international workshop on machine learning, Aberdeen, Scotland, pp 49–54Google Scholar
  8. Chai B, Huang T, Zhuang X, Zhao Y, Sklansky J (1996) Piecewise-linear classifiers using binary tree structure and genetic algorithm. Pattern Recognit 29(11):1905–1917CrossRefGoogle Scholar
  9. Ehrenburg H (1996) Improved directed acyclic graph evaluation and the combine operator in genetic programming. In: Proceedings of the 1st genetic programming conference. Stanford University, CAGoogle Scholar
  10. Esmeir S, Markovitch S (2007) Anytime learning of decision trees. J Machine Learn Res 8:891–933Google Scholar
  11. Handley S (1994a) On the use of a directed acyclic graph to represent a population of computer programs. Proceedings of the IEEE World Congress on Computational Intelligence, Orlando, FL, pp 154–159Google Scholar
  12. Handley S (1994) On the use of a directed acyclic graph to represent a population of computer programs. In: Proceedings of the IEEE world congress on computational intelligence, Orlando, FL, pp 154–159Google Scholar
  13. Jansen T (1999) On classifications of fitness functions. Technical report CI-76/99. University of Dortmund, GermanyGoogle Scholar
  14. Jin Y (2005) A comprehensive survey of fitness approximation in evolutionary computation. Soft Comput 9(1):3–12CrossRefGoogle Scholar
  15. Kalles D (1994) Decision trees and domain knowledge in pattern recognition. Dissertation, University of Manchester Institute of Science and Technology, UKGoogle Scholar
  16. Kalles D, Papagelis A (2000) Stable decision trees: Using local anarchy for efficient incremental learning. Int J Artif Intell Tools 9(1):79–95CrossRefGoogle Scholar
  17. Kalles D, Pierrakeas C (2006) Analyzing student performance in distance learning with genetic algorithms and decision trees. Appl Artif Intell 20(8):655–674CrossRefGoogle Scholar
  18. Knessl C, Szpankowski W (2006) On the joint path length distribution in random binary trees. Stud Appl Math 117:109–147zbMATHCrossRefMathSciNetGoogle Scholar
  19. Koza JR (1991) Concept formation and decision tree induction using the genetic programming paradigm. In: Proceedings of the 1st workshop on parallel problem solving from nature. Springer, Berlin, pp 124–128Google Scholar
  20. Krętowski M (2004) An evolutionary algorithm for oblique decision tree induction. In: Proceedings of ICAISC 2004. LNCS, vol 3070. Springer, Berlin, pp 432–437Google Scholar
  21. Krętowski M, Grześ M (2005) Global learning of decision trees by an evolutionary algorithm. In: Saeed K, Peja J (eds) Information processing and security systems, Springer, Berlin, pp 401–410Google Scholar
  22. Linton RC (2004) Adapting binary fitness functions in genetic algorithms. In: Proceedings of the 42nd ACM southeast regional conference, Huntsville, AL, pp 391–395Google Scholar
  23. Mansour Y, McAllester D (2000) Generalization bounds for decision trees. In: Proceedings of the 13th annual conference on computer learning theory, San Francisco, CA, pp 69–80Google Scholar
  24. Meisel WS, Michalopoulos DA (1973) A partitioning algorithm with application in pattern classification and the optimization of decision trees. IEEE Trans Comput 22(1):93–103zbMATHCrossRefGoogle Scholar
  25. Mitchell M (1996) An introduction to genetic algorithms. MIT Press, CambridgeGoogle Scholar
  26. Mitchell T (1997) Machine Learning. McGraw Hill, NYzbMATHGoogle Scholar
  27. Musick R, Catlett J, Russel S (1993) Decision theoretic subsampling for induction on large databases. In: Proceedings of the 10th international conference on machine learning, Amherst, MA, pp 212–219Google Scholar
  28. Naumov GE (1991) NP-completeness of problems of construction of optimal decision trees. Sov Phys Doklady 36(4):270–271zbMATHMathSciNetGoogle Scholar
  29. Nikolaev N, Slavov V (1998) Inductive genetic programming with decision trees. Intell Data Anal 2(1):31–40CrossRefGoogle Scholar
  30. Pagallo G (1989) Learning DNF by decision trees. In: Proceedings of the 11th international joint conference on artificial intelligence, pp 639–644Google Scholar
  31. Papagelis A, Kalles D (2001) Breeding decision trees using evolutionary techniques. In: Proceedings of the 18th international conference on machine learning, Williamstown, MA, pp 393–400Google Scholar
  32. Pelikan M, Sastry K (2004) Fitness inheritance in the Bayesian optimization algorithm. In: Proceedings of the genetic and evolutionary computation conference, Seattle, WA, pp 48–59Google Scholar
  33. Poli R, Langdon WB (2006) Backward-chaining evolutionary algorithms. Artif Intell 170:953–982zbMATHCrossRefMathSciNetGoogle Scholar
  34. Quinlan JR (1986) Induction of decision trees. Machine Learn 1(1):81–106Google Scholar
  35. Quinlan JR (1993) C4.5: programs for machine learning. Morgan Kaufmann, San MateoGoogle Scholar
  36. Quinlan JR, Rivest RL (1989) Inferring decision trees using the Minimum Description Length principle. Inf Comput 80(3):227–248zbMATHCrossRefMathSciNetGoogle Scholar
  37. Roberts MA (2003) The effectiveness of cost based sub-tree caching mechanisms in typed genetic programming for image segmentation. In: Proceedings of EvoIASP, applications of evolutionary computation. LNCS, vol 2611, Essex, UK, pp 444–454Google Scholar
  38. Rokach L (2008) Genetic algorithm-based feature set partitioning for classification problems. Pattern Recognit 41(5):1693–1717CrossRefGoogle Scholar
  39. Rokach L, Maimon O (2008) Data mining with decision trees: theory and applications. World Scientific, SingaporezbMATHGoogle Scholar
  40. Sastry K, Goldberg DE, Pelikan M (2001) Don’t evaluate, inherit. In: Proceedings of the genetic and evolutionary computation conference, San Francisco, CA, pp 551–558Google Scholar
  41. Smith RE, Dike BA, Stegmann SA (1995) Fitness inheritance in genetic algorithms. In: Proceedings of the symposium on applied computing, Nashville, TN, pp 345–350Google Scholar
  42. Teller A, Andre D (1997) Automatically choosing the number of fitness cases: The rational allocation of trials. In: Proceedings of the 2nd annual conference on genetic programming, Stanford University, CA, pp 321–328Google Scholar
  43. Turney PD (1995) Cost-sensitive classification: empirical evaluation of a hybrid genetic decision tree induction algorithm. J Artif Intell Res 2:369–409Google Scholar
  44. Witten IH, Frank E (2005) Data mining: practical machine learning tools and techniques, 2nd edn. Morgan Kaufmann, San FranciscoGoogle Scholar
  45. Woodward JR (2006) Complexity and Cartesian genetic programming. In: Proceedings of the 9th European conference on genetic programming. Lecture notes in computer science, vol 3905, Budapest, Hungary, pp 260–269Google Scholar
  46. Zhang B-T, Joung J-G (1999) Genetic programming with incremental data inheritance. In: Proceedings of the Genetic and Evolutionary Computation Conference, Orlando, FL, pp 1217–1224Google Scholar
  47. Zygounas S (2004) Using genetic algorithms in the construction of decision trees. Dissertation (in Greek), Hellenic Open University, GreeceGoogle Scholar

Copyright information

© Springer-Verlag 2009

Authors and Affiliations

  1. 1.Hellenic Open UniversityPatrasGreece
  2. 2.Open University of CyprusNicosiaCyprus
  3. 3.Department of Computer Engineering and InformaticsUniversity of PatrasPatrasGreece

Personalised recommendations