A New Criterion of Mutual Information Using R-value

Part of the Lecture Notes in Electrical Engineering book series (LNEE, volume 235)


Mutual information has wide area of application including feature selection and classification. To calculate mutual information, statistical equation of information theory has been used. In this paper, we propose a new criterion for mutual information. It is based on R-value which captures overlapping areas among classes in variables (features). Overlapping area of classes reflects uncertainty of the variables; it corresponds to the meaning of entropy. We compare traditional mutual information and R-value on the context of feature selection. From the experiment we confirm that proposed method shows better performance than traditional mutual information.


Entropy Mutual information Attribute interaction R-value Information theory Data mining 



This work was supported by the National Research Foundation of Korea Grant funded by the Korean Government (NRF-2012S1A2A1A01028576).


  1. 1.
    Guiasu S (1977) Information theory with applications. McGraw-Hill, New YorkMATHGoogle Scholar
  2. 2.
    Anastassiou D (2007) Computational analysis of the synergy among multiple interacting genes. Mol Sys Biol 3(83):1–8Google Scholar
  3. 3.
    Definition of feature selection, in 〈
  4. 4.
    Liu H, Yu L (2005) Toward integrating feature selection algorithms for classification and clustering. IEEE Trans Knowl Data Eng 17:491–502Google Scholar
  5. 5.
    Saeys Y, Inza I, Larranaga P (2007) A review of feature selection techniques in bioinformatics. Bioinfo 23(19):2507–2517CrossRefGoogle Scholar
  6. 6.
    Berrar DP, Dubitzky W, Granzow M (2009) A practical approach to microarray data analysis. Springer Publishing Company, IncorporatedGoogle Scholar
  7. 7.
    Cover TM, Thomas JA (1991) Elements of information theory. Wiley, New YorkMATHCrossRefGoogle Scholar
  8. 8.
    Tourassia GD, Frederick ED, Markey MK, Floyd CE (2001) Application of the mutual information criterion for feature selection in computer-aided diagnosis. Med Phys 28(12):2394–2402CrossRefGoogle Scholar
  9. 9.
    Oh S (2011) A new dataset evaluation method based on category overlap. Comput Biol Med 41:115–122CrossRefGoogle Scholar
  10. 10.
    Battiti R (1994) Using mutual information for selecting features in supervised neural net learning. IEEE Trans Neural Netw 5:537–550CrossRefGoogle Scholar
  11. 11.
    Zheng Y, Kwoh CK (2011) A feature subset selection method based on high-dimensional mutual information. Entropy 13:860–901 doi: 10.3390/e13040860 Google Scholar
  12. 12.
    Largeron C, Moulin C, Géry M (2011) Entropy based feature selection for text categorization. ACM Symp Appl Comput doi: 10.1145/1982185.1982389
  13. 13.
    Can-Tao L (2009) Mutual information based on Renyi’s entropy feature selection. IEEE international conference on intelligent computing and intelligent systems, 2009. ICIS 2009, vol 1, pp 816–820Google Scholar
  14. 14.
    Jakulin A, Bratko I, Smrke D, Demsar J, Zupan B (2003) Attribute interactions in medical data analysis. AI in Medicine in Europe (AIME), pp 229–238Google Scholar
  15. 15.
    Lee J, Batnyam N, Oh S RFS: efficient feature selection method based on R-value. Comput Biol Med (In press)Google Scholar
  16. 16.
  17. 17.
    UCI machine learning repository

Copyright information

© Springer Science+Business Media Dordrecht 2013

Authors and Affiliations

  1. 1.School of Computer, Electronic, and Communication EngineeringYanbian University of Science and TechnologyYanji CityChina
  2. 2.Department of Nanobiomedical ScienceDankook UniversityCheonanRepublic of Korea

Personalised recommendations