Genetic Programming and Evolvable Machines

, Volume 3, Issue 4, pp 329–343 | Cite as

Genetic Programming-based Construction of Features for Machine Learning and Knowledge Discovery Tasks

  • Krzysztof Krawiec


In this paper we use genetic programming for changing the representation of the input data for machine learners. In particular, the topic of interest here is feature construction in the learning-from-examples paradigm, where new features are built based on the original set of attributes. The paper first introduces the general framework for GP-based feature construction. Then, an extended approach is proposed where the useful components of representation (features) are preserved during an evolutionary run, as opposed to the standard approach where valuable features are often lost during search. Finally, we present and discuss the results of an extensive computational experiment carried out on several reference data sets. The outcomes show that classifiers induced using the representation enriched by the GP-constructed features provide better accuracy of classification on the test set. In particular, the extended approach proposed in the paper proved to be able to outperform the standard approach on some benchmark problems on a statistically significant level.

genetic programming machine learning change of representation feature construction feature selection 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    H. N. Bensusan, Automatic Bias Learning: An Inquiry into the Inductive Basis of Induction. Ph.D. Dissertation, School of Computing and Cognitive Sciences, University of Sussex, Sussex, 1998.Google Scholar
  2. 2.
    H. N. Bensusan and I. Kuscu, “Constructive induction using genetic programming,” in Proc. Int. Conf. Machine Learning, Evolutionary Computing and Machine Learning Workshop, T. Fogarty and G. Venturini (eds.), Bari, Italy, 1996.Google Scholar
  3. 3.
    B. Bhanu and K. Krawiec, “Coevolutionary construction of features for transformation of representation in machine learning,” in Proceedings of the Bird of a Feather Workshops, Genetic and Evolutionary Computation Conference (GECCO 2002), A. M. Barry (ed.), AAAI Press: New York, 2002, pp. 249–254.Google Scholar
  4. 4.
    C. L. Blake and C. J. Merz, “UCI Repository of machine learning databases,” [], University of California: Irvine, CA, 1998.Google Scholar
  5. 5.
    M. Brameier and W. Banzhaf, “Evolving teams of predictors with linear genetic programming,” Genetic Programming and Evolvable Machines vol. 2, pp. 381–407, 2001.Google Scholar
  6. 6.
    C. E. Brodley and A. Pohoreckyj Danyluk (eds.), Proc. Int. Conf. Machine Learning, Morgan Kaufmann: San Francisco 2001.Google Scholar
  7. 7.
    M. Dash and H. Liu, “Feature selection for classification,” Intelligent Data Analysis vol. 1, no. 3, pp. 131–156, 1997.Google Scholar
  8. 8.
    K. A. De Jong, An Analysis of the Behavior of a Class of Genetic Adaptive Systems. Doctoral dissertation, University of Michigan, Ann Arbor, 1975.Google Scholar
  9. 9.
    K. A. De Jong, W. M. Spears, and D. F. Gordon, “Using genetic algorithms for concept learning,” Machine Learning vol. 13, pp. 161–188, 1993.Google Scholar
  10. 10.
    D. Goldberg, Genetic Algorithms in Search, Optimization and Machine Learning, Addison-Wesley: Reading, 1989.Google Scholar
  11. 11.
    J. J. Grefenstette, “Lamarckian learning in multi-agent environments,” in Proc. Fourth Intl. Conf. of Genetic Algorithms, Morgan Kaufmann, San Mateo, CA, 1991, pp. 303–310.Google Scholar
  12. 12.
    J. H. Holland, Adaptation in Natural and Artificial Systems, University of Michigan Press: Ann Arbor, 1975.Google Scholar
  13. 13.
    I. Imam and H. Vafaie, “An empirical comparison between global and greedy-like search for feature selection,” in Proceedings of the Florida AI Research Symposium (FLAIRS-94), Pensacola Beach, FL, 1994, pp. 66–70.Google Scholar
  14. 14.
    J. K. Kishore, L. M. Patnaik, V. Mani, and V. K. Agrawal, “Application of genetic programming for multicategory pattern classification,” IEEE Trans. Evolutionary Comp. vol. 4, no. 3, pp. 242–258, 2000.Google Scholar
  15. 15.
    M. Komosinski and K. Krawiec, “Evolutionary weighting of image features for diagnosing of CNS tumors,” Artif. Intell. in Medicine vol. 19, no. 1, pp. 25–38, 2000.Google Scholar
  16. 16.
    R. Kohavi and G. H. John, “Wrappers for feature subset selection,” Artif. Intell. Journal vol. 1–2, pp. 273–324, 1997.Google Scholar
  17. 17.
    R. Kohavi, D. Sommerfeld, and J. Dougherty, “Data mining using MLC++: Amachine learning library in C++,” in Proc. of the Eight International Conference on Tools with Artificial Intelligence (ICTAI'96), IEEE Computer Society, 1996, pp. 234–245.Google Scholar
  18. 18.
    J. R. Koza, Genetic Programming—2, MIT Press: Cambridge, 1994.Google Scholar
  19. 19.
    K. Krawiec, “Pairwise comparison of hypotheses in evolutionary learning,” in Proc. Int. Conf. Machine Learning, C. E. Brodley and A. Pohoreckyj Danyluk (eds.), Morgan Kaufmann: San Francisco, 2001, pp. 266–273.Google Scholar
  20. 20.
    K. Krawiec, “Genetic programming with local improvement for visual learning from examples,” in Computer Analysis of Images and Patterns (LNCS 2124), W. Skarbek, (ed.), Springer: Berlin, 2001, pp. 209–216.Google Scholar
  21. 21.
    K. Krawiec, “On the use of pairwise comparison of hypotheses in evolutionary learning applied to learning from visual examples,” in Machine Learning and Data Mining in Pattern Recognition (LNAI 2123), P. Perner (ed.), Springer: Berlin, 2001, pp. 307–321.Google Scholar
  22. 22.
    P. Langley, Elements of Machine Learning, Morgan Kaufmann: San Francisco, 1996.Google Scholar
  23. 23.
    T.-S. Lim, W.-Y. Loh, and Y.-S. Shih, “Acomparison of prediction accuracy, complexity, and training time of thirty-three old and new classification algorithms,” Machine Learning, vol. 40, pp. 203–228, 2000.Google Scholar
  24. 24.
    S. Luke, “ECJ 7: An EC and GP system in Java,”, 2001.Google Scholar
  25. 25.
    C. J. Matheus and L. A. Rendell, “Constructive induction on decision trees,” in Proc. of the Eleventh International Joint Conference on Artificial Intelligence, N. S. Sridharan (ed.), Detroit, MI, USA, August 1989, Morgan Kaufmann 1989, pp. 645–650.Google Scholar
  26. 26.
    C. J. Matheus, “Aconstructive induction framework,” in Proc. of the Sixth International Workshop on Machine Learning, Ithaca: New York, 1989, pp. 474–475.Google Scholar
  27. 27.
    G. Mayraz and G. Hinton, “Recognizing hand-written digits using hierarchical products of experts,” Adv. NIPS 13, MIT Press: Cambridge, MA, 2001, pp. 953–959.Google Scholar
  28. 28.
    R. S. Michalski, “Atheory and methodology of inductive learning,” Artificial Intelligence 20, pp. 111–161, 1983.Google Scholar
  29. 29.
    T. M. Mitchell, An Introduction to Genetic Algorithms, MIT Press: Cambridge, MA, 1996.Google Scholar
  30. 30.
    T. M. Mitchell, Machine Learning, McGraw-Hill: New York, 1997.Google Scholar
  31. 31.
    J. R. Quinlan, C4.5: Programs for Machine Learning, Morgan Kaufmann: San Mateo, CA, 1992.Google Scholar
  32. 32.
    M. L. Raymer, W. F. Punch, E. D. Goodman, L. A. Kuhn, and A. K. Jain, “Dimensionality reduction using genetic algorithm,” IEEE Trans. on Evolutionary Computation, vol. 4, no. 2, pp. 164–171, 2000.Google Scholar
  33. 33.
    M. L. Raymer, W. F. Punch, E. D. Goodman, and L. A. Kuhn, “Genetic programming for improved data mining—application to the biochemistry of protein interactions,” in Genetic Programming 1996: Proceedings, J. R. Koza, D. E. Goldberg, D. B. Fogel, and R. L. Riolo, (eds.), MIT Press: Cambridge, MA, 1996, pp. 275–381.Google Scholar
  34. 34.
    S. Thrun, J. Bala, E. Bloedorn, I. Bratko, B. Cestnik, J. Cheng, K. De Jong, S. Dzeroski, R. Hamann, K. Kaufman, S. Keller, I. Kononenko, J. Kreuziger, R. S. Michalski, T. Mitchell, P. Pachowicz, B. Roger, H. Vafaie, W. Van de Velde, W. Wenzel, J. Wnek, and J. Zhang, “The MONK's problems: Aperformance comparison of different learning algorithms,” Technical Report CMU-CS-91-197, Carnegie Mellon University: Pittsburgh, PA, 1991.Google Scholar
  35. 35.
    I. H. Witten and E. Frank, Data Mining: Practical Machine Learning Tools and Techniques with Java Implementations, Morgan Kaufmann: San Francisco, CA, 1999.Google Scholar
  36. 36.
    J. Yang and V. Honavar, “Feature subset selection using a genetic algorithm,” IEEE Intelligent Systems (Special Issue on Feature Transformation and Subset Selection), vol. 13, pp. 44–49, 1998.Google Scholar

Copyright information

© Kluwer Academic Publishers 2002

Authors and Affiliations

  • Krzysztof Krawiec
    • 1
  1. 1.Institute of Computing SciencePoznań University of TechnologyPoznańPoland

Personalised recommendations