Abstract
Feature selection is an important step for data mining and machine learning. It can be used to reduce the requirement of data measurement and storage, and defy the curse of dimensionality to improve the prediction performance. In this paper, we propose a feature selection method via mutual information estimation. It avoids the calculation of high-dimensional mutual information by transforming the high-dimensional feature space into one dimension through a novel supervised clustering method. Experimental results on ten benchmark data sets show that: (1) the performances of kNN, naive Bayes classifier, and C4.5 using much less features selected by the proposed method are similar or even better than those on the original data sets with the whole feature set; (2) different from most of state-of-the-art methods which require to setting the number of features to select in prior, the proposed method can automatically determine the proper size of selected feature subsets.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Guyon, I., Elisseeff, A.: An introduction to variable and feature selection. J. Mach. Learn. Res. 3, 1157–1182 (2003)
Guyon, I., Gunn, S., Nikravesh, M., Zadeh, L.A. (eds.): Feature Extraction: Foundations and Applications. Studies in Fuzziness and Soft Computing. Springer, Heidelberg (2006)
Liu, H., Motoda, H.: Feature Selection for Knowledge Discovery and Data Mining. Kluwer Academic Publishers, Norwell (1998)
Vergara, J.R., Estevez, P.A.: A review of feature selection methods based on mutual information. Neural Comput. Appl. 24(1), 175–186 (2014)
Kwak, N., Choi, C.-H.: Input feature selection by mutual information based on parzen window. IEEE Trans. Pattern Anal. Mach. Intell. 24(12), 1667–1671 (2002)
Balagani, K.S., Phoha, V.V.: On the feature selection criterion based on an approximation of multidimensional mutual information. IEEE Trans. Pattern Anal. Mach. Intell. 32(7), 1342–1343 (2010)
Battiti, R.: Using mutual information for selecting features in supervised neural net learning. IEEE Trans. Neural Netw. 5(4), 537–550 (1994)
Peng, H., Long, F., Ding, C.: Feature selection based on mutual information criteria of max-dependency, max-relevance, and min-redundancy. IEEE Trans. Pattern Anal. Mach. Intell. 27(8), 1226–1238 (2005)
Brown, G.: A new perspective for information theoretic feature selection. In: AISTATS 2009, pp. 49–56 (2009)
Fano, R.M.: Transmission of Information: A Statistical Theory of Communications. MIT Press, Cambridge (1961)
Hellman, M., Raviv, J.: Probability of error, equivocation, and the chernoff bound. IEEE Trans. Inf. Theory 16(4), 368–372 (1970)
Rodriguez, A., Laio, A.: Clustering by fast search and find of density peaks. Science 344(6191), 1492–1496 (2014)
Asuncion, A., Newman, D.: UCI machine learning repository (2007)
Quinlan, J.R.: C4.5: Programs for Machine Learning. Morgan Kaufmann Series in Machine Learning. Morgan Kaufmann, San Francisco (1992)
Yang, H.H., Moody, J.E.: Data visualization and feature selection: new algorithms for nongaussian data. In: NIPS 1999, pp. 687–702 (1999)
Meyer, P., Schretter, C., Bontempi, G.: Information-theoretic feature selection in microarray data using variable complementarity. IEEE J. Sel. Top. Sig. Process. 2(3), 261–274 (2008)
Brown, G., Pocock, A., Zhao, M.-J., Lujan, M.: Conditional likelihood maximisation: a unifying framework for information theoretic feature selection. J. Mach. Learn. Res. 13, 27–66 (2012)
Kira, K., Rendell, L.A.: The feature selection problem: traditional methods and a new algorithm. In: AAAI 1992, pp. 129–134 (1992)
Hinton, G.E., Salakhutdinov, R.R.: Reducing the dimensionality of data with neural networks. Science 313(5786), 504–507 (2006)
Acknowledgment
This work was supported in part by National Natural Science Fund of China (61232005) and National Key Technology R&D Program of China (2012BAH06B01). Liu is partially sponsored by CCF-Tencent Open Research Fund.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2015 Springer International Publishing Switzerland
About this paper
Cite this paper
Liu, H., Wu, Z., Zhang, X. (2015). Feature Selection Based on Data Clustering. In: Huang, DS., Bevilacqua, V., Premaratne, P. (eds) Intelligent Computing Theories and Methodologies. ICIC 2015. Lecture Notes in Computer Science(), vol 9225. Springer, Cham. https://doi.org/10.1007/978-3-319-22180-9_23
Download citation
DOI: https://doi.org/10.1007/978-3-319-22180-9_23
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-22179-3
Online ISBN: 978-3-319-22180-9
eBook Packages: Computer ScienceComputer Science (R0)