Orthogonal Relief Algorithm for Feature Selection

  • Jun Yang
  • Yue-Peng Li
Part of the Lecture Notes in Computer Science book series (LNCS, volume 4113)


The Relief is a popular feature selection algorithm. However, it is ineffective in removing redundant features due to its feature evaluation mechanism that all discriminative features are assigned with high relevance scores, regardless of the correlations in between. In the present study, we develop an orthogonal Relief algorithm (O-Relief) to tackle the redundant feature problem. The basic idea of the O-Relief algorithm is to introduce an orthogonal transform to decompose the correlation between features so that the relevance of a feature could be evaluated individually as it is done in the original Relief algorithm. Experiment results on four world problems show that the orthogonal Relief algorithm provides features leading to better classification results than the original Relief algorithm.


Feature Selection Discriminative Feature Redundant Feature Orthogonal Space Wrap Method 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Jain, A.K., Duin, R.P.W., Mao, J.: Statistical Pattern Recognition: A Review. IEEE Transactions on Pattern Analysis and Machine Intelligence 22, 4–37 (2000)CrossRefGoogle Scholar
  2. 2.
    Kohavi, R., John, G.H.: Wrappers for Feature Subset Selection. Artificial Intelligent Journal 1, 273–324 (1997)CrossRefGoogle Scholar
  3. 3.
    Kira, K., Rendell, L.A.: The Feature Selection Problem: Traditional Methods and A New Algorithm. In: Proceedings of AAAI 1992, San Jose, USA, pp. 129–134 (1992)Google Scholar
  4. 4.
    Bins, J., Draper, B.A.: Feature Selection from Huge Feature Sets. In: Proceedings of ICCV 2001, Vancouver, Canada, pp. 159–165 (2001)Google Scholar
  5. 5.
    Florez-lopez, R.: Reviewing RELIEF and its Extensions: A New Approach for Estimating Attributes Considering High-Correlated Features. In: Proceedings of IEEE International Conference on Data Mining, Maebashi, Japan, pp. 605–608 (2002)Google Scholar
  6. 6.
    Kononenko, I.: Estimating Attributes: Analysis and Extensions of RELIEF. In: Proceedings of European Conference on Machine Learning, Catania, Italy, vol. 182, pp. 171–182 (1994)Google Scholar
  7. 7.
    Chen, S., Billings, S.A., Luo, W.: Orthogonal Least Square Methods and their Applications to Nonlinear System Identification. International Journal of Control 50, 1873–1896 (1989)MATHCrossRefMathSciNetGoogle Scholar
  8. 8.
    Mao, K.Z., Billings, S.A.: Algorithms for Minimal Model Structure Detection in Nonlinear Dynamic System Identification. International Journal of Control 68, 311–330 (1997)MATHCrossRefMathSciNetGoogle Scholar
  9. 9.
    Mao, K.Z., Tan, K.C., Ser, W.: Probabilistic Neural Network Structure Determination for Pattern Classification. IEEE Transactions on Neural Networks 11, 1009–1016 (2000)CrossRefGoogle Scholar
  10. 10.
    Mao, K.Z.: RBF Neural Network Center Selection Based on Fisher Ratio Class Separability Measure. IEEE Transactions on Neural Networks 13, 1211–1217 (2002)CrossRefGoogle Scholar
  11. 11.
    Mao, K.Z.: Orthogonal Forward Selection and Backward Elimination Algorithms for Feature Subset Selection. IEEE Transactions on Systems, Man and Cybernetics, Part B 34, 629–634 (2004)CrossRefGoogle Scholar
  12. 12.
    Blake, C.L., Merz, C.J.: UCI Repository of Machine Learning Databases, Department of Information and Computer Sciences, University of California, Irvene, CA, USA (1998), http://www.ics.uci.edu/~mlearn/MLRepository.html
  13. 13.
    Golub, T.R., et al.: Molecular Classification of Cancer: Class Discovery and Class Prediction by Gene Expression Monitoring. Science 286, 531–537 (1999)CrossRefGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2006

Authors and Affiliations

  • Jun Yang
    • 1
  • Yue-Peng Li
    • 1
  1. 1.Institute of AcousticsChinese Academy of SciencesBeijingChina

Personalised recommendations