Feature Selection by Combining Multiple Methods

  • Lior Rokach
  • Barak Chizi
  • Oded Maimon
Part of the Studies in Computational Intelligence book series (SCI, volume 23)


Feature selection is the process of identifying relevant features in the dataset and discarding everything else as irrelevant and redundant. Since feature selection reduces the dimensionality of the data, it enables the learning algorithms to operate more effectively and rapidly. In some cases, classification performance can be improved; in other instances, the obtained classifier is more compact and can be easily interpreted. There is much work done on feature selection methods for creating ensemble of classifiers. Thus, these works examine how feature selection can help ensemble of classifiers to gain diversity. This paper examines a different direction, i.e. whether ensemble methodology can be used for improving feature selection performance. In this paper we present a general framework for creating several feature subsets and then combine them into a single subset. Theoretical and empirical results presented in this paper validate the hypothesis that this approach can help finding a better feature subset.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Bartlett P. and Shawe-Taylor J., Generalization Performance of Support Vector Machines and Other Pattern Classifiers, In “Advances in Kernel Methods, Support Vector Learning”, Bernhard Scholkopf, Christopher J. C. Burges, and Alexander J. Smola (eds.), MIT Press, Cambridge, USA, 1998.Google Scholar
  2. 2.
    Cunningham P., and Carney J., Diversity Versus Quality in Classification Ensembles Based on Feature Selection, In: R. L. de Mántaras and E. Plaza (eds.), Proc. ECML 2000, 11th European Conf. On Machine Learning, Barcelona, Spain, LNCS 1810, Springer, 2000Google Scholar
  3. 3.
    Hall, M. Correlation-based Feature Selection for Machine Learning. University of Waikato, 1999.Google Scholar
  4. 4.
    Ho T. K. The random subspace method for constructing decision forests. IEEE Transactions on Pattern Analysis and Machine Intelligence, 20(8):832–844, 1998.CrossRefGoogle Scholar
  5. 5.
    Hu, X., Using Rough Sets Theory and Database Operations to Construct a Good Ensemble of Classifiers for Data Mining Applications. ICDM01. pp. 233–240, 2001.Google Scholar
  6. 6.
    Kohavi R. and John, G. Wrappers for feature subset selection. Artificial Intelligence, special issue on relevance, 97(1–2): 273–324, 1996Google Scholar
  7. 7.
    Masulli, F. and Rovetta, S. Random Voronoi ensembles for gene selection in DNA microarray data, in Udo Seiffert and Lakhmi C. Jain, editors, Bioinformatics using Computational Intelligence Paradigms, World Scientific Publishing, Singapore, 2003Google Scholar
  8. 8.
    Miller, A. Subset Selection in Regression. Chapman and Hall, New York, 1990.zbMATHGoogle Scholar
  9. 9.
    Oliveira L.S., Sabourin R., Bortolozzi F., and Suen C.Y. A Methodology for Feature Selection using Multi-Objective Genetic Algorithms for Handwritten Digit String Recognition, International Journal of Pattern Recognition and Artificial Intelligence, 17(6):903–930, 2003.CrossRefGoogle Scholar
  10. 10.
    Opitz, D., Feature Selection for Ensembles, In: Proc. 16th National Conf. on Artificial Intelligence, AAAI, pages 379–384, 1999.Google Scholar
  11. 11.
    Quinlan, J. C4.5: Programs for machine learning. Morgan Kaufmann, Los Altos, California, 1993.Google Scholar
  12. 12.
    Quinlan, J. Induction of decision trees. Machine Learning, 1: 81–106, 1996Google Scholar
  13. 13.
    Torkkola, K. and Tuv, E. Variable selection using ensemble methods. IEEE Intelligent Systems, 2005, (Vol. 20, No. 6): 68–70.CrossRefGoogle Scholar
  14. 14.
    Tsymbal A., and Puuronen S., Ensemble Feature Selection with the Simple Bayesian Classification in Medical Diagnostics, In: Proc. 15thIEEE Symp. on Computer-Based Medical Systems CBMS’2002, IEEE CS Press, 2002.Google Scholar
  15. 15.
    Turner, K. and Ghosh J., Error Correlation and Error Reduction in Ensemble Classifiers, Connection Science, Special issue on combining artificial neural networks: ensemble approaches, 8(3–4): 385–404, 1999.Google Scholar
  16. 16.
    Tuv, E. and Torkkola, K. Feature filtering with ensembles using artificial contrasts. In Proceedings of the SIAM 2005 Int. Workshop on Feature Selection for Data Mining, Newport Beach, CA, April 23 2005, pp. 69–71.Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2006

Authors and Affiliations

  • Lior Rokach
    • 1
  • Barak Chizi
    • 2
  • Oded Maimon
    • 2
  1. 1.Department of Information Systems EngineeringBen-Gurion University of the NegevIsrael
  2. 2.Department of Industrial EngineeringTel-Aviv UniversityIsrael

Personalised recommendations