Feature Selection via Maximizing Neighborhood Soft Margin

  • Qinghua Hu
  • Xunjian Che
  • Jinfu Liu
Part of the Lecture Notes in Computer Science book series (LNCS, volume 5828)


Feature selection is considered to be a key preprocessing step in machine learning and pattern recognition. Feature evaluation is one of the key issues for constructing a feature selection algorithm. In this work, we propose a new concept of neighborhood margin and neighborhood soft margin to measure the minimal distance between different classes. We use the criterion of neighborhood soft margin to evaluate the quality of candidate features and construct a forward greedy algorithm for feature selection. We conduct this technique on eight classification learning tasks. Compared with the raw data and other three feature selection algorithms, the proposed technique is effective in most of the cases.


Feature Selection Feature Selection Algorithm Candidate Feature Feature Subset Selection Machine Learn Research 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Wang, H., Bell, D., Murtagh, F.: Axiomatic Approach to Feature Subset Selection Based on Relevance. IEEE Transactions on Pattern Analysis and Machine Intelligence 21, 271–277 (1999)CrossRefGoogle Scholar
  2. 2.
    Yu, L., Liu, H.: Efficient Feature Selection via Analysis of Relevance and Redundancy. Journal of Machine Learning Research 5, 1205–1224 (2004)MathSciNetGoogle Scholar
  3. 3.
    Guyon, I., Elisseeff, A.: An introduction to variable and feature selection. Journal of Machine Learning Research 3, 1157–1182 (2003)zbMATHCrossRefGoogle Scholar
  4. 4.
    Liu, H., Yu, L.: Toward integrating feature selection algorithms for classification and clustering. IEEE Transactions on Knowledge and Data Engineering 17, 491–502 (2005)CrossRefGoogle Scholar
  5. 5.
    Kohavi, R., John, G.H.: Wrappers for feature subset selection. Artificial Intelligence 97, 273–324 (1997)zbMATHCrossRefGoogle Scholar
  6. 6.
    Abe, S., Thawonmas, R., Kobayashi, Y.: Feature selection by analyzing class regions approximated by ellipsoids. IEEE Transactions on Systems, Man, and Cybernetics-Part C: Applications and Reviews 28, 282–287 (1998)CrossRefGoogle Scholar
  7. 7.
    Neumann, J., Schnorr, C., Steidl, G.: Combined SVM-based feature selection and classification. Machine learning 61, 129–150 (2005)zbMATHCrossRefGoogle Scholar
  8. 8.
    Ho, T.K., Basu, M.: Complexity measures of supervised classification problems. IEEE Trans. Pattern Analysis and Machine Intelligence 24, 289–300 (2002)CrossRefGoogle Scholar
  9. 9.
    Hu, Q.H., Yu, D., Liu, J.F., Wu, C.: Neighborhood rough set based heterogeneous feature subset selection. Information sciences 178, 3577–3594 (2008)zbMATHCrossRefMathSciNetGoogle Scholar
  10. 10.
    Hall, M.A.: Correlation-based feature selection for discrete and numeric class machine learning. In: Proc. 17th Int’l. Conf. Machine Learning, pp. 359–366 (2000)Google Scholar
  11. 11.
    Hu, Q.H., Yu, D.R., Xie, Z.X.: Neighborhood classifier. Expert Systems with Applications 34, 866–876 (2008)CrossRefGoogle Scholar
  12. 12.
    Dash, M., Liu, H.: Consistency-based search in feature selection. Artificial Intelligence 151, 155–176 (2003)zbMATHCrossRefMathSciNetGoogle Scholar
  13. 13.
    Crammer, K., Gilad-Bachrach, R., Navot, A., Tishby, N.: Margin analysis of the LVQ algorithm. In: Proc. 17th Conference on Neural Information Processing Systems (2002)Google Scholar
  14. 14.
    Sun, Y., Li, J.: Iterative RELIEF for Feature Weighting. In: Proc. 23rd Int’l. Conf. Machine Learning, pp. 913–920 (2006)Google Scholar
  15. 15.
    Kononenko, I.: Estimating Attributes: Analysis and Extensions of RELIEF. In: Bergadano, F., De Raedt, L. (eds.) ECML 1994. LNCS, vol. 784, pp. 171–182. Springer, Heidelberg (1994)Google Scholar
  16. 16.
    Li, Y., Lu, B.-L.: Feature selection based on loss-margin of nearest neighbor classification. Pattern Recognition (2008), doi:10.1016/j.patcog. 2008.10.011Google Scholar
  17. 17.
    Cortes, C., Vapnik, V.: Support-Vector Networks. Machine Learning 20, 273–297 (1995)zbMATHGoogle Scholar
  18. 18.
    Vapnik, V.: Statistical learning theory. Wiley, NY (1998)zbMATHGoogle Scholar
  19. 19.
    Blake, C.L., Merz, C.J.: UCI repository of machine learning databases (1998),
  20. 20.
    Gilad-Bachrach, R., Navot, A., Tishby, N.: Margin based feature selection- theory and algorithm. In: Proceddings of 21st international conference on machine learning, Banff, Canada (2004)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2009

Authors and Affiliations

  • Qinghua Hu
    • 1
  • Xunjian Che
    • 1
  • Jinfu Liu
    • 1
  1. 1.Harbin Institute of TechnlologyHarbinChina

Personalised recommendations