Advertisement

A Feature Selection Algorithm Based on Equal Interval Division and Minimal-Redundancy–Maximal-Relevance

  • Xiangyuan Gu
  • Jichang GuoEmail author
  • Lijun Xiao
  • Tao Ming
  • Chongyi Li
Article
  • 13 Downloads

Abstract

Minimal-redundancy–maximal-relevance (mRMR) algorithm is a typical feature selection algorithm. To select the feature which has minimal redundancy with the selected features and maximal relevance with the class label, the objective function of mRMR subtracts the average value of mutual information between features from mutual information between features and the class label, and selects the feature with the maximum difference. However, the problem is that the feature with the maximum difference is not always the feature with minimal redundancy maximal relevance. To solve the problem, the objective function of mRMR is first analyzed and a constraint condition that determines whether the objective function can guarantee the effectiveness of the selected features is achieved. Then, for the case where the objective function is not accurate, an idea of equal interval division is proposed and combined with ranking to process the interval of mutual information between features and the class label, and that of the average value of mutual information between features. Finally, a feature selection algorithm based on equal interval division and minimal-redundancy–maximal-relevance (EID–mRMR) is proposed. To validate the performance of EID–mRMR, we compare it with several incremental feature selection algorithms based on mutual information and other feature selection algorithms. Experimental results demonstrate that the EID–mRMR algorithm can achieve better feature selection performance.

Keywords

Minimal-redundancy–maximal-relevance Equal interval division Mutual information Feature selection 

Notes

Acknowledgements

This work was supported by the National Natural Science Foundation of China (61771334).

References

  1. 1.
    Tan MK, Tsang IW, Wang L (2014) Towards ultrahigh dimensional feature selection for big data. J Mach Learn Res 15:1371–1429MathSciNetzbMATHGoogle Scholar
  2. 2.
    Cataron A, Andonie R (2010) Energy supervised relevance neural gas for feature ranking. Neural Process Lett 32(1):59–73CrossRefGoogle Scholar
  3. 3.
    Borja SP, Veronica BC, Amparo AB (2017) Testing different ensemble configurations for feature selection. Neural Process Lett 46:1–24CrossRefGoogle Scholar
  4. 4.
    Tiwari S, Singh B, Kaur M (2017) An approach for feature selection using local searching and global optimization techniques. Neural Comput Appl 28(10):2915–2930CrossRefGoogle Scholar
  5. 5.
    Wang JZ, Wu LS, Kong J, Li YX, Zhang BX (2013) Maximum weight and minimum redundancy: a novel framework for feature subset selection. Pattern Recognit 46(6):1616–1627CrossRefGoogle Scholar
  6. 6.
    Shang CX, Li M, Feng SZ, Jiang QS, Fan JP (2013) Feature selection via maximizing global information gain for text classification. Knowl-Based Syst 54:298–309CrossRefGoogle Scholar
  7. 7.
    Tang B, Kay S, He HB (2016) Toward optimal feature selection in naive Bayes for text categorization. IEEE Trans Knowl Data Eng 28(9):2508–2521CrossRefGoogle Scholar
  8. 8.
    Fei T, Kraus D, Zoubir AM (2012) A hybrid relevance measure for feature selection and its application to underwater objects recognition. In: International conference on image processing, pp 97–100Google Scholar
  9. 9.
    Fei T, Kraus D, Zoubir AM (2015) Contributions to automatic target recognition systems for underwater mine classification. IEEE Trans Geosci Remote Sens 53(1):505–518CrossRefGoogle Scholar
  10. 10.
    Zhang F, Chan PPK, Biggio B, Yeung DS, Roli F (2016) Adversarial feature selection against evasion attacks. IEEE Trans Cybern 46(3):766–777CrossRefGoogle Scholar
  11. 11.
    Veronica BC, Noelia SM, Amparo AB (2013) A review of feature selection methods on synthetic data. Knowl Inf Syst 34(3):483–519CrossRefGoogle Scholar
  12. 12.
    Jia XP, Kuo BC, Crawford MM (2013) Feature mining for hyperspectral image classification. Proc IEEE 101(3):676–697CrossRefGoogle Scholar
  13. 13.
    Lin CH, Chen HY, Wu YS (2014) Study of image retrieval and classification based on adaptive features using genetic algorithm feature selection. Expert Syst Appl 41(15):6611–6621CrossRefGoogle Scholar
  14. 14.
    Zhao YH, Wang GR, Yin Y, Li Y, Wang ZH (2016) Improving ELM-based microarray data classification by diversified sequence features selection. Neural Comput Appl 27(1):155–166CrossRefGoogle Scholar
  15. 15.
    Estevez PA, Tesmer M, Perez CA, Zurada JM (2009) Normalized mutual information feature selection. IEEE Trans Neural Netw 20(2):189–201CrossRefGoogle Scholar
  16. 16.
    Brown G, Pocock A, Zhao MJ, Lujan M (2012) Conditional likelihood maximisation: a unifying framework for information theoretic feature selection. J Mach Learn Res 13:27–66MathSciNetzbMATHGoogle Scholar
  17. 17.
    Vergara JR, Estevez PA (2014) A review of feature selection methods based on mutual information. Neural Comput Appl 24(1):175–186CrossRefGoogle Scholar
  18. 18.
    Lewis DD (1992) Feature selection and feature extraction for text categorization. In: Proceedings of the workshop on speech and natural language, pp 212–217Google Scholar
  19. 19.
    Peng HC, Long FH, Ding C (2005) Feature selection based on mutual information: criteria of max-dependency, max-relevance and min-redundancy. IEEE Trans Pattern Anal Mach Intell 27(8):1226–1238CrossRefGoogle Scholar
  20. 20.
    Hoque N, Bhattacharyya DK, Kalita JK (2014) MIFS-ND: a mutual information-based feature selection method. Expert Syst Appl 41(14):6371–6385CrossRefGoogle Scholar
  21. 21.
    Han M, Ren WJ (2015) Global mutual information-based feature selection approach using single-objective and multi-objective optimization. Neurocomputing 168:47–54CrossRefGoogle Scholar
  22. 22.
    Zhang Y, Ding C, Li T (2008) Gene selection algorithm by combining reliefF and mRMR. BMC Genom 9(2):1–10Google Scholar
  23. 23.
    Unler A, Murat A, Chinnam RB (2011) m\({r^2}\)PSO: a maximum relevance minimum redundancy feature selection method based on swarm intelligence for support vector machine classification. Inf Sci 181(20):4625–4641CrossRefGoogle Scholar
  24. 24.
    Battiti R (1994) Using mutual information for selecting features in supervised neural net learning. IEEE Trans Neural Netw 5(4):537–550CrossRefGoogle Scholar
  25. 25.
    UCI machine learning repository. http://archive.ics.uci.edu/ml. Accessed 6 Mar 2018
  26. 26.
    ASU feature selection datasets. http://featureselection.asu.edu/datasets/. Accessed 6 Mar 2018
  27. 27.
    Fayyad U, Irani K (1993) Multi-interval discretization of continuous-valued attributes for classification learning. In: Proceedings of joint conference on artificial intelligence, pp 1022–1027Google Scholar
  28. 28.
    Hall M, Frank E, Holmes G, Pfahringer B, Reutemann P, Witten IH (2009) The weka data mining software: an update. ACM SIGKDD Explor Newsl 11(1):10–18CrossRefGoogle Scholar
  29. 29.
    Zhao Z, Morstatter F, Sharma S, Alelyani S, Anand A, Liu H (2010) Advancing feature selection research. In: ASU feature selection repository, pp 1–28Google Scholar
  30. 30.
    Kwak N, Choi CH (2002) Input feature selection for classification problems. IEEE Trans Neural Netw 13(1):143–159CrossRefGoogle Scholar
  31. 31.
    Wang ZC, Li MQ, Li JZ (2015) A multi-objective evolutionary algorithm for feature selection based on mutual information with a new redundancy measure. Inf Sci 307:73–88MathSciNetCrossRefGoogle Scholar
  32. 32.
    Foithong S, Pinngern O, Attachoo B (2012) Feature subset selection wrapper based on mutual information and rough sets. Expert Syst Appl 39(1):574–584CrossRefGoogle Scholar
  33. 33.
    Kononenko I (1994) Estimating attributes: analysis and extension of RELIEF. In: Proceedings of European conference on machine learning, pp 171–182CrossRefGoogle Scholar
  34. 34.
    Duda RO, Hart PE, Stork DG (2001) Pattern classification. Wiley-Interscience Publication, New YorkzbMATHGoogle Scholar
  35. 35.
    Rodriguez-Lujan I, Huerta R, Elkan C, Cruz CS (2010) Quadratic programming feature selection. J Mach Learn Res 11:1491–1516MathSciNetzbMATHGoogle Scholar
  36. 36.
    Nguyen XV, Chan J, Romano S, Bailey J (2014) Effective global approaches for mutual information based feature selection. In: Proceedings of ACM SIGKDD international conference on knowledge discovery and data mining, pp 512–521Google Scholar
  37. 37.
    Herman G, Zhang B, Wang Y, Ye GT, Chen F (2013) Mutual information-based method for selecting informative feature sets. Pattern Recognit 46(12):3315–3327CrossRefGoogle Scholar
  38. 38.
    Fleuret F (2004) Fast binary feature selection with conditional mutual information. J Mach Learn Res 5:1531–1555MathSciNetzbMATHGoogle Scholar
  39. 39.
    Sun X, Liu YH, Xu MT, Chen HL, Han JW, Wang KH (2013) Feature selection using dynamic weights for classification. Knowl-Based Syst 37:541–549CrossRefGoogle Scholar
  40. 40.
    Bennasar M, Hicks Y, Setchi R (2015) Feature selection using joint mutual information maximisation. Expert Syst Appl 42(22):8520–8532CrossRefGoogle Scholar
  41. 41.
    Zeng ZL, Zhang HJ, Zhang R, Yin CX (2015) A novel feature selection method considering feature interaction. Pattern Recognit 48(8):2656–2666CrossRefGoogle Scholar
  42. 42.
    Shishkin A, Bezzubtseva AA, Drutsa A, Shishkov I, Gladkikh E, Gusev G, Serdyukov P (2016) Efficient high order interaction aware feature selection based on conditional mutual information. In: Proceedings of the conference on advances in neural information processing systems, pp 4637–4645Google Scholar
  43. 43.
    Vinh NX, Zhou S, Chan J, Bailey J (2016) Can high-order dependencies improve mutual information based feature selection. Pattern Recognit 53:46–58CrossRefGoogle Scholar
  44. 44.
    Wang J, Wei JM, Yang ZL, Wang SQ (2017) Feature selection by maximizing independent classification information. IEEE Trans Knowl Data Eng 29(4):828–841CrossRefGoogle Scholar
  45. 45.
    Gao WF, Hu L, Zhang P (2018) Class-specific mutual information variation for feature selection. Pattern Recognit 79:328–339CrossRefGoogle Scholar
  46. 46.
    Gao WF, Hu L, Zhang P, He JL (2018) Feature selection considering the composition of feature relevancy. Pattern Recognit Lett 112:70–74CrossRefGoogle Scholar

Copyright information

© Springer Science+Business Media, LLC, part of Springer Nature 2019

Authors and Affiliations

  • Xiangyuan Gu
    • 1
  • Jichang Guo
    • 1
    Email author
  • Lijun Xiao
    • 1
  • Tao Ming
    • 1
  • Chongyi Li
    • 2
  1. 1.School of Electrical and Information EngineeringTianjin UniversityTianjinChina
  2. 2.The Department of Computer ScienceCity University of Hong KongHong KongChina

Personalised recommendations