Feature Selection by Combining Multiple Methods
Feature selection is the process of identifying relevant features in the dataset and discarding everything else as irrelevant and redundant. Since feature selection reduces the dimensionality of the data, it enables the learning algorithms to operate more effectively and rapidly. In some cases, classification performance can be improved; in other instances, the obtained classifier is more compact and can be easily interpreted. There is much work done on feature selection methods for creating ensemble of classifiers. Thus, these works examine how feature selection can help ensemble of classifiers to gain diversity. This paper examines a different direction, i.e. whether ensemble methodology can be used for improving feature selection performance. In this paper we present a general framework for creating several feature subsets and then combine them into a single subset. Theoretical and empirical results presented in this paper validate the hypothesis that this approach can help finding a better feature subset.
Unable to display preview. Download preview PDF.
- 1.Bartlett P. and Shawe-Taylor J., Generalization Performance of Support Vector Machines and Other Pattern Classifiers, In “Advances in Kernel Methods, Support Vector Learning”, Bernhard Scholkopf, Christopher J. C. Burges, and Alexander J. Smola (eds.), MIT Press, Cambridge, USA, 1998.Google Scholar
- 2.Cunningham P., and Carney J., Diversity Versus Quality in Classification Ensembles Based on Feature Selection, In: R. L. de Mántaras and E. Plaza (eds.), Proc. ECML 2000, 11th European Conf. On Machine Learning, Barcelona, Spain, LNCS 1810, Springer, 2000Google Scholar
- 3.Hall, M. Correlation-based Feature Selection for Machine Learning. University of Waikato, 1999.Google Scholar
- 5.Hu, X., Using Rough Sets Theory and Database Operations to Construct a Good Ensemble of Classifiers for Data Mining Applications. ICDM01. pp. 233–240, 2001.Google Scholar
- 6.Kohavi R. and John, G. Wrappers for feature subset selection. Artificial Intelligence, special issue on relevance, 97(1–2): 273–324, 1996Google Scholar
- 7.Masulli, F. and Rovetta, S. Random Voronoi ensembles for gene selection in DNA microarray data, in Udo Seiffert and Lakhmi C. Jain, editors, Bioinformatics using Computational Intelligence Paradigms, World Scientific Publishing, Singapore, 2003Google Scholar
- 10.Opitz, D., Feature Selection for Ensembles, In: Proc. 16th National Conf. on Artificial Intelligence, AAAI, pages 379–384, 1999.Google Scholar
- 11.Quinlan, J. C4.5: Programs for machine learning. Morgan Kaufmann, Los Altos, California, 1993.Google Scholar
- 12.Quinlan, J. Induction of decision trees. Machine Learning, 1: 81–106, 1996Google Scholar
- 14.Tsymbal A., and Puuronen S., Ensemble Feature Selection with the Simple Bayesian Classification in Medical Diagnostics, In: Proc. 15thIEEE Symp. on Computer-Based Medical Systems CBMS’2002, IEEE CS Press, 2002.Google Scholar
- 15.Turner, K. and Ghosh J., Error Correlation and Error Reduction in Ensemble Classifiers, Connection Science, Special issue on combining artificial neural networks: ensemble approaches, 8(3–4): 385–404, 1999.Google Scholar
- 16.Tuv, E. and Torkkola, K. Feature filtering with ensembles using artificial contrasts. In Proceedings of the SIAM 2005 Int. Workshop on Feature Selection for Data Mining, Newport Beach, CA, April 23 2005, pp. 69–71.Google Scholar