Genetic Programming for Feature Ranking in Classification Problems

  • Kourosh Neshatian
  • Mengjie Zhang
  • Peter Andreae
Part of the Lecture Notes in Computer Science book series (LNCS, volume 5361)


Feature ranking (FR) provides a measure of usefulness for the attributes of a classification task. Most existing FR methods focus on the relevance of a single feature to the class labels. Here, we use GP to see how a set of features can contribute towards discriminating different classes and then we score the participating features accordingly. The scoring mechanism is based on the frequency of appearance of each feature in a collection of GP programs and the fitness of those programs. Our results show that the proposed FR method can detect important features of a problem. A variety of different classifiers restricted to just a few of these high-ranked features work well. The ranking mechanism can also shrink the search space of size O(2 n ) of subsets of features to a search space of size O(n) in which there are points that may improve the classification performance.


Feature Selection Genetic Programming Class Label Feature Ranking Decision Stump 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Jong, K., Mary, J., Cornujols, A., Marchiori, E., Sebag, M.: Ensemble feature ranking. In: Boulicaut, J.-F., Esposito, F., Giannotti, F., Pedreschi, D. (eds.) PKDD 2004. LNCS, vol. 3202, pp. 267–278. Springer, Heidelberg (2004)CrossRefGoogle Scholar
  2. 2.
    Ruiz, R., Riquelme, J.C., Aguilar-Ruiz, J.S.: Fast feature ranking algorithm. Knowledge-Based Intelligent Information and Engineering Systems, 325–331 (2003)Google Scholar
  3. 3.
    Biesiada, J., Duch, W., Kachel, A., Maczka, K., Palucha, S.: Feature ranking methods based on information entropy with parzen windows. In: International Conference on Research in Electrotechnology and Applied Informatics (REI 2005), pp. 109–119 (2005)Google Scholar
  4. 4.
    Lin, T.H., Chiu, S.H., Tsai, K.C.: Supervised feature ranking using a genetic algorithm optimized artificial neural network. Journal of Chemical Information and Modeling 46, 1604–1614 (2006)CrossRefGoogle Scholar
  5. 5.
    Oh, I.S., Lee, J.S., Moon, B.R.: Hybrid genetic algorithms for feature selection. IEEE Transactions on Pattern Analysis and Machine Intellignece, 1424–1437 (2004)Google Scholar
  6. 6.
    Smith, M.G., Bull, L.: Genetic programming with a genetic algorithm for feature construction and selection. Genetic Programming and Evolvable Machines 6, 265–281 (2005); Published online: 17 August 2005CrossRefGoogle Scholar
  7. 7.
    Langdon, W.B., Poli, R.: Foundations of Genetic Programming. Springer, Heidelberg (2002)CrossRefzbMATHGoogle Scholar
  8. 8.
    Lin, J.Y., Ke, H.R., Chien, B.C., Yang, W.P.: Classifier design with feature selection and feature extraction using layered genetic programming. Expert Syst. Appl. 34, 1384–1393 (2008)CrossRefGoogle Scholar
  9. 9.
    Davis, R.A., Charlton, A.J., Oehlschlager, S., Wilson, J.C.: Novel feature selection method for genetic programming using metabolomic 1h nmr data. Chemometrics and Intelligent Laboratory Systems 81, 50–59 (2006)CrossRefGoogle Scholar
  10. 10.
    Liu, H., Yu, L.: Toward integrating feature selection algorithms for classification and clustering. IEEE Transactions on Knowledge and Data Engineering 17, 491–502 (2005)CrossRefGoogle Scholar
  11. 11.
    Landry, J.A., Costa, L.D., Bernier, T.: Discriminant feature selection by genetic programming: Towards a domain independent multi-class object detection system. Systemics, Cybernetics and Informatics 3, 76–81 (2006)Google Scholar
  12. 12.
    Parrott, D., Li, X., Ciesielski, V.: Multi-objective techniques in genetic programming for evolving classifiers. In: The 2005 IEEE Congress on Evolutionary Computation, 2005, vol. 2, pp. 1141–1148 (2005)Google Scholar
  13. 13.
    Muni, D.P., Pal, N.R., Das, J.: Genetic programming for simultaneous feature selection and classifier design. IEEE Transactions on Systems, Man and Cybernetics, Part B 36, 106–117 (2006)CrossRefGoogle Scholar
  14. 14.
    Asuncion, A., Newman, D.: Uci machine learning repository (2007),
  15. 15.
    Koza, J.R.: Genetic Programming: On the Programming of Computers by Means of Natural Selection. MIT Press, Cambridge (1992)zbMATHGoogle Scholar
  16. 16.
    Quinlan, J.R.: C4.5: Programs for Machine Learning. Morgan Kaufmann, San Francisco (1993)Google Scholar
  17. 17.
    Witten, I.H., Frank, E.: Data Mining: Practical Machine Learning Tools and Techniques, 2nd edn. Morgan Kaufmann, San Francisco (2005)zbMATHGoogle Scholar
  18. 18.
    John, G., Langley, P.: Estimating continuous distributions in bayesian classifiers, pp. 338–345 (1995)Google Scholar
  19. 19.
    Keerthi, S.S., Shevade, S.K., Bhattacharyya, C., Murthy, K.R.K.: Improvements to platt’s smo algorithm for svm classifier design. Neural Comp. 13, 637–649 (2001)CrossRefzbMATHGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2008

Authors and Affiliations

  • Kourosh Neshatian
    • 1
  • Mengjie Zhang
    • 1
  • Peter Andreae
    • 1
  1. 1.School of Mathematics, Statistics and Computer ScienceVictoria University of WellingtonWellingtonNew Zealand

Personalised recommendations