Unsupervised Extraction and Supervised Selection of Features Based on Information Gain

  • Soo-Young Lee
  • Chandra Shahard Dhir
  • Paresh Chandra Barman
  • Sangkyun Lee
Conference paper


For robust recognition we first extract features from sensory data without considering the class labels, and then select important features for the classification. The unsupervised feature extraction may incorporate Principle Component Analysis, Independent Component Analysis, and Non-negative Matrix factorization. For the supervised selection of features we adopt Fisher Score and Information Gain (IG). To avoid the calculation of multivariate joint probability density functions, instead of the IG, we use Mutual Information (MI) between a feature and the class variable. However, in this case the MI among selected features reduces the effectiveness of the feature selection, and the statistically-independent ICA-based features result in the best performance.


Feature extraction feature selection Fisher score information gain mutual information independent component analysis 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Turk, M., and Pentland, A.: Eigenfaces for recognition. Journal of Cognitive Neruoscience 3 (1991) 71–86.CrossRefGoogle Scholar
  2. 2.
    Bartlett, M., Lades, H., and Sejnowski, T.: Independant component representations for face recognition. In T. Rogowitz, B. and Pappas (Ed.): Proceedings of the SPIE Symposium on Electronic Imaging: Science and Technology; Human Vision and Electronic Imaging III, 3299, January 1998. SPIE Press, San Jose, CA, pp. 528–539.Google Scholar
  3. 3.
    Lee, D.D., and Seung, H.S.: Learning the parts of objects by non-negative matrix factorization. Nature 401 (1999) 788–791.PubMedCrossRefGoogle Scholar
  4. 4.
    Bishop, C.: Neural Networks for Pattern Recognition, 2nd ed. New York: Oxford University Press, (1995).Google Scholar
  5. 5.
    Wang, G., and Lochovsky, F.H.: Feature selection with conditional mutual information maxim in text categorization. Proceedings of the thirteenth ACM international conference on Information and Knowledge Management (2004) 342–349.Google Scholar
  6. 6.
    Lee, K.D., Lee, M.J., and Lee, S.Y.: Extraction of frame-difference features based on PCA and ICA for lip-reading. IEEE International Joint Conference on Neural Networks, IEEE Computer Society Press, Los Alamitos (2005) pp. 232–237.Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2008

Authors and Affiliations

  • Soo-Young Lee
    • 1
  • Chandra Shahard Dhir
  • Paresh Chandra Barman
  • Sangkyun Lee
  1. 1.Brain Science Research CenterKorea Advanced Institute of Science and TechnologyDaejeon 305-701Korea

Personalised recommendations