A Rough Set Approach to Feature Selection Based on Relative Decision Entropy

  • Lin Zhou
  • Feng Jiang
Part of the Lecture Notes in Computer Science book series (LNCS, volume 6954)


Rough set theory has been recognized to be one of the powerful tools for feature selection. The essence of rough set approach to feature selection is to find a minimal subset (or called reduct) of original features, which can predict the decision concepts as well as the original features. Since finding a minimal feature subset is a NP-hard problem, many heuristic algorithms have been proposed, which usually employ the feature significance in rough sets as heuristic information. Shannon’s information theory provides us a feasible way to measure the information of data sets with entropy. Hence, many researchers have used information entropy to measure the feature significance in rough sets, and proposed different information entropy models in rough sets. In this paper, we present a new information entropy model, in which relative decision entropy is defined. Based on the relative decision entropy, we propose a new rough set algorithm to feature selection. To verify the efficiency of our algorithm, experiments are carried out on some UCI data sets. The results demonstrate that our algorithm can provide competitive solutions efficiently.


Rough sets feature selection information entropy relative decision entropy 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Liu, H., Motoda, H.: Feature Selection for Knowledge Discovery and Data Mining. Kluwer Academic Publishers, Dordrecht (1998)CrossRefMATHGoogle Scholar
  2. 2.
    Guyon, I., Elisseeff, A.: An Introduction to Variable and Feature Selection. Journal of Machine Learning Research 3, 1157–1182 (2003)MATHGoogle Scholar
  3. 3.
    Hu, X.H.: Knowledge Discovery in Databases: An Attribute-oriented Rough Set Approach. Ph.D. thesis, Regina University (1995)Google Scholar
  4. 4.
    Hu, X.H., Cereone, N.: Learning in Relational Databases: A Rough Set Approach. Computational Intelligence 11(2), 323–337 (1995)CrossRefGoogle Scholar
  5. 5.
    Hu, K.Y., Lu, Y.C., Shi, C.Y.: Feature Ranking in Rough Sets. AI Commun. 16(1), 41–50 (2003)MATHGoogle Scholar
  6. 6.
    Wang, G.Y., Yu, H., Yang, D.C.: Decision Table Reduction Based on Conditional Information Entropy. Chinese Journal of Computers 25(7), 759–766 (2002)MathSciNetGoogle Scholar
  7. 7.
    Wang, G.Y.: Rough Set Theory and Knowledge Acquisition. Xian Jiaotong University Press, Xian (2001)Google Scholar
  8. 8.
    Liu, S.H., Sheng, Q.J., Wu, B., Shi, Z.Z.: Research on Efficient Algorithms for Rough Set Methods. Chinese Journal of Computers 26(5), 525–529 (2003)Google Scholar
  9. 9.
    Miao, D.Q., Hu, G.R.: An Heuristic Algorithm of Knowledge Reduction. Computer Research and Development 36(6), 681–684 (1999)Google Scholar
  10. 10.
    Düntsch, I., Gediga, G.: Uncertainty Measures of Rough Set Prediction. Artificial Intelligence 106, 109–137 (1998)MathSciNetCrossRefMATHGoogle Scholar
  11. 11.
    Liang, J.Y., Shi, Z.Z.: The Information Entropy, Rough Entropy and Knowledge Granulation in Rough Set Theory. International Journal of Uncertainty, Fuzziness and Knowledge Based Systems 12(1), 37–46 (2004)MathSciNetCrossRefMATHGoogle Scholar
  12. 12.
    Shannon, C.E.: The Mathematical Theory of Communication. Bell System Technical Journal 27(3-4), 373–423 (1948)MathSciNetGoogle Scholar
  13. 13.
    Sui, Y.F., Xia, Y.M., Wang, J.: The Information Entropy of Rough Relational Databases. In: Wang, G., Liu, Q., Yao, Y., Skowron, A. (eds.) RSFDGrC 2003. LNCS (LNAI), vol. 2639, pp. 320–324. Springer, Heidelberg (2003)CrossRefGoogle Scholar
  14. 14.
    Xu, Z.Y., Liu, Z.P., Yang, B.R., Song, W.: A Quick Attribute Reduction Algorithm with Complexity of max(O(|C| |U|),O(|C| 2 |U/C|)). Chinese Journal of Computers 29(3), 391–399 (2006)Google Scholar
  15. 15.
    Jensen, R., Shen, Q.: Finding Rough Set Reducts with Ant Colony Optimization. In: Proc. of the 2003 UK Workshop on Computational Intelligence, pp. 15–22 (2003)Google Scholar
  16. 16.
    Wroblewski, J.: Finding Minimal Reducts Using Genetic Algorithms. In: Proc. of the Second Annual Join Conf. on Information Sciences, NC, pp. 186–189 (1995)Google Scholar
  17. 17.
    Bazan, J.: A Comparison of Dynamic and Non-dynamic Rough Set Methods for Extracting Laws from Decision Table. In: Rough Sets in Knowledge Discovery, pp. 321–365. Physica-Verlag, Heidelberg (1998)Google Scholar
  18. 18.
    Wang, X.Y., Yang, J., Teng, X.L., Xia, W.J., Jensen, R.: Feature Selection Based on Rough Sets and Particle Swarm Optimization. Pattern Recognition Letters 28(4), 459–471 (2007)CrossRefGoogle Scholar
  19. 19.
    Nguyen, S.H., Nguyen, H.S.: Some Efficient Algorithms for Rough Set Methods. In: Proceedings of IPMU 1996, Granada, Spain, pp. 1451–1456 (1996)Google Scholar
  20. 20.
    Skowron, A., Bazan, J., Son, N.H., Wroblewski, J., et al.: RSES 2.2 User’s Guide. Institute of Mathematics, Warsaw University, Warsaw, Poland (2005)Google Scholar
  21. 21.
    Stefanowski, J.: On Rough Set Based Approaches to Induction of Decision Rules. In: Rough Sets in Knowledge Discovery, pp. 500–529. Physica-Verlag, Heidelberg (1998)Google Scholar
  22. 22.
    Bay, S.D.: The UCI KDD Repository (1999), http://kdd.ics.uci.edu
  23. 23.
    Pawlak, Z.: Rough Sets. Int. J. Comput. Informat. Sci. 11(5), 341–356 (1982)MathSciNetCrossRefMATHGoogle Scholar
  24. 24.
    Pawlak, Z.: Rough Sets: Theoretical Aspects of Reasoning about Data. Kluwer Academic Publishing, Dordrecht (1991)CrossRefMATHGoogle Scholar
  25. 25.
    Zhong, N., Dong, J., Ohsuga, S.: Using Rough Sets with Heuristics for Feature Selection. Journal of Intelligent Information Systems 16(3), 199–214 (2001)CrossRefMATHGoogle Scholar
  26. 26.
    Yao, J.T., Zhang, M.: Feature Selection with Adjustable Criteria. In: Ślęzak, D., Wang, G., Szczuka, M.S., Düntsch, I., Yao, Y. (eds.) RSFDGrC 2005. LNCS (LNAI), vol. 3641, pp. 204–213. Springer, Heidelberg (2005)CrossRefGoogle Scholar
  27. 27.
    Pal, S.K., Shankar, B.U., Mitra, P.: Granular Computing, Rough Entropy and Object Extraction. Pattern Recognition Letters 26(16), 2509–2517 (2005)CrossRefGoogle Scholar
  28. 28.
    Sen, D., Pal, S.K.: Generalized Rough Sets, Entropy and Image Ambiguity Measures. IEEE Trans. Syst., Man and Cyberns. Part B 39(1), 117–128 (2009)CrossRefGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2011

Authors and Affiliations

  • Lin Zhou
    • 1
  • Feng Jiang
    • 1
  1. 1.College of Information Science and TechnologyQingdao University of Science and TechnologyQingdaoP.R. China

Personalised recommendations