Advances in Feature Selection with Mutual Information

  • Michel Verleysen
  • Fabrice Rossi
  • Damien François
Part of the Lecture Notes in Computer Science book series (LNCS, volume 5400)


The selection of features that are relevant for a prediction or classification problem is an important problem in many domains involving high-dimensional data. Selecting features helps fighting the curse of dimensionality, improving the performances of prediction or classification methods, and interpreting the application. In a nonlinear context, the mutual information is widely used as relevance criterion for features and sets of features. Nevertheless, it suffers from at least three major limitations: mutual information estimators depend on smoothing parameters, there is no theoretically justified stopping criterion in the feature selection greedy procedure, and the estimation itself suffers from the curse of dimensionality. This chapter shows how to deal with these problems. The two first ones are addressed by using resampling techniques that provide a statistical basis to select the estimator parameters and to stop the search procedure. The third one is addressed by modifying the mutual information criterion into a measure of how features are complementary (and not only informative) for the problem at hand.


Feature Selection Mutual Information Linear Discriminant Analysis Smoothing Parameter Relevance Criterion 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Borggaard, C., Thodberg, H.: Optimal minimal neural interpretation of spectra. Analytical Chemistry 64, 545–551 (1992)CrossRefGoogle Scholar
  2. 2.
    Cover, T., Thomas, J.: Elements of Information Theory. Wiley, New York (1991)CrossRefGoogle Scholar
  3. 3.
    Dijck, G.V., Hulle, M.M.V.: Speeding up the wrapper feature subset selection in regression by mutual information relevance and redundancy analysis. In: ICANN 2006: International Conference in Aritificial Neural Networks, Athens, Greece, September 2006, pp. 31–40 (2006) (submitted) Google Scholar
  4. 4.
    François, D., Krier, C., Rossi, F., Verleysen, M.: Estimation de redondance pour le clustering de variables spectrales. In: Agrostat 2008, 10èmes journées Européennes Agro-industrie et Méthodes statistiques, pp. 55–61. Louvain-la-Neuve, Belgium (2008)Google Scholar
  5. 5.
    François, D., Rossi, F., Wertz, V., Verleysen, M.: Resampling methods for parameter-free and robust feature selection with mutual information. Neurocomputing 70(7-9), 1265–1275 (2007)CrossRefGoogle Scholar
  6. 6.
    Goria, M.N., Leonenko, N.N., Mergel, V.V., Inverardi, P.L.N.: A new class of random vector entropy estimators and its application in testing statistical hypotheses. Journal of Nonparametric Statistics 17(3), 277–297 (2005)CrossRefGoogle Scholar
  7. 7.
    Kozachenko, L.F., Leonenko, N.N.: Sample estimate of entropy of a random vector. Probl. Inf. Transm. 23, 95–101 (1987)Google Scholar
  8. 8.
    Kraskov, A., Stögbauer, H., Grassberger, P.: Estimating mutual information. Physical Review E 69, 066138 (2004)CrossRefGoogle Scholar
  9. 9.
    Krier, C., François, D., Rossi, F., Verleysen, M.: Feature clustering and mutual information for the selection of variables in spectral data. In: ESANN 2007, European Symposium on Artificial Neural Networks Advances in Computational Intelligence and Learning, pp. 157–162. Bruges, Belgium (2007)Google Scholar
  10. 10.
    Rossi, F., Lendasse, A., François, D., Wertz, V., Verleysen, M.: Mutual information for the selection of relevant variables in spectrometric nonlinear modelling. Chemometrics and Intelligent Laboratory Systems 2(80), 215–226 (2006)CrossRefGoogle Scholar
  11. 11.
    Scott, D.: Multivariable Density Estimation: Theory, Practice, and Visualization. Wiley, New-York (1992)CrossRefGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2009

Authors and Affiliations

  • Michel Verleysen
    • 1
  • Fabrice Rossi
    • 2
  • Damien François
    • 1
  1. 1.Machine Learning GroupUniversité catholique de LouvainBelgium
  2. 2.INRIA Rocquencourt, Domaine de Voluceau, RocquencourtLe Chesnay CedexFrance

Personalised recommendations