On the Scalability of Genetic Algorithms to Very Large-Scale Feature Selection

  • Andreas Moser
  • M. Narasimha Murty
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 1803)

Abstract

Feature Selection is a very promising optimisation strategy for Pattern Recognition systems. But, as an NP-complete task, it is extremely difficult to carry out. Past studies therefore were rather limited in either the cardinality of the feature space or the number of patterns utilised to assess the feature subset performance.

This study examines the scalability of Distributed Genetic Algorithms to very large-scale Feature Selection. As domain of application, a classification system for Optical Characters is chosen. The system is tailored to classify hand-written digits, involving 768 binary features. Due to the vastness of the investigated problem, this study forms a step into new realms in Feature Selection for classification.

We present a set of customisations of GAs that provide for an application of known concepts to Feature Selection problems of practical interest. Some limitations of GAs in the domain of Feature Selection are unrevealed and improvements are suggested. A widely used strategy to accelerate the optimisation process, Training Set Sampling, was observed to fail in this domain of application.

Experiments on unseen validation data suggest that Distributed GAs are capable of reducing the problem complexity significantly. The results show that the classification accuracy can be maintained while reducing the feature space cardinality by about 50%. Genetic Algorithms are demonstrated to scale well to very large-scale problems in Feature Selection.

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. 1.
    M. Dash and H. Liu. Feature selection for classification. Intelligent Data Analysis, 1997.Google Scholar
  2. 2.
    G. John, R. Kohavi, and K. Pfleger. Irrelevant features and the subset selection problem. Proceedings of the International Conference on Machine Learning, 11, 1994.Google Scholar
  3. 3.
    W. Siedlecki and J. Sklansky. A note on genetic algorithms for large-scale feature selection. Pattern Recognition Letters, 10:335–347, 1988.CrossRefGoogle Scholar
  4. 4.
    Punch, Goodman, Pei, Lai Chia-Shun, P. Hovland, and R. Enbody. Further research on feature selection and classification using genetic algorithms. Proceedings of the 5th International Conference of Genetic Algorithms, 1993.Google Scholar
  5. 5.
    A. Moser. A distributed vertical genetic algorithm for feature selection. Fifth International Conference on Document Analysis and Recognition, Open Research Forum, 1999.Google Scholar
  6. 6.
    D. Flotzinger. Feature selection by genetic algorithms. IIG Report Series, 369, 1993.Google Scholar
  7. 7.
    M. Prakash and M. N. Murty. Feature selection to improce classification accuracy using a genetic algorithm. Journal of the Indian Institute of Science, 1997.Google Scholar
  8. 8.
    A. K. Jain and D. Zongker. Feature selection: Evaluation, application and small sample performance. IEEE Transactions on Pattern Analysis and Machine Intelligence, 19(2), 1997.Google Scholar
  9. 9.
    J. E. Smith, T. C. Fogarty, and I.R. Johnson. Genetic feature selection for clustering and classification. Proceedings of the IEEE Colloquium on Genetic Algorithms in Image Processing & Vision; IEEE Digest 1994/193, 1994.Google Scholar
  10. 10.
    C. Guerra-Salcedo and D. Whitley. Genetic search for feature selection: A comparison between CHC and GENESIS. Proceedings of the Symposium on Genetic Algorithms, 1998.Google Scholar
  11. 11.
    J. Yang and V. Honavar. Feature subset selection using a genetic algorithm. Feature Extraction, Construction and Selection-A Data Mining Perspective, 1998.Google Scholar
  12. 12.
    F. J. Ferri, P. Pudil, M. Hatef, and J. Kittler. Comparative study of techniques for large-scale feature reduction. Pattern Recognition in Practice IV, 1994.Google Scholar
  13. 13.
    I. F. Imam and H. Vafaie. An emprical comparison between global and greedy-like search for feature selection. Proceedings of the Florida AI Research Symposium, 1994.Google Scholar
  14. 14.
    E. I. Chang and R. P. Lippmann. Using genetic algorithms to improve pattern classification performance. Advances in Neural Information Processing, 3, 1990.Google Scholar
  15. 15.
    D. Beasley, D. R. Bull, and R. R. Martin. An overview of genetic algorithms; part 2: Research topics. University Computing, 15(4):58–69, 1993.Google Scholar
  16. 16.
    F. Z. Brill, D. E. Brown, and W. N. Martin. Fast genetic selection of features for neural network classifiers. IEEE Transactions of Neural Networks, 3(2), 1992.Google Scholar
  17. 17.
    M. Prakash and M. N. Murty. Growing subspace pattern recognition methods and their neural-network models. IEEE Transactions on Neural Networks, 8(1), 1997.Google Scholar
  18. 18.
    V. V. Saradhi. Pattern Representation and Prototype Selection in Classification. Master Thesis, Department of Computer Science and Automation, Indian Institute of Science, Bangalore, 1999.Google Scholar
  19. 19.
    L. Holmstroem, P. Koistinen, and E. Oja. Neural and statistical classifiers-taxonomy and two case studies. IEEE Transactions on Neural Networks, 8(1), 1997.Google Scholar
  20. 20.
    A. Moser. Distributed genetic algorithms for feature selection. Diploma Thesis, University of Kaiserslautern, Germany, 1999.Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2000

Authors and Affiliations

  • Andreas Moser
    • 1
  • M. Narasimha Murty
    • 2
  1. 1.German Research Center for Artificial Intelligence GmbHKaiserslauternGermany
  2. 2.Department of Computer Science and AutomationIndian Institute of ScienceBangaloreIndia

Personalised recommendations