Relevance and Redundancy Analysis for Ensemble Classifiers

  • Rakkrit Duangsoithong
  • Terry Windeatt
Part of the Lecture Notes in Computer Science book series (LNCS, volume 5632)


In machine learning systems, especially in medical applications, clinical datasets usually contain high dimensional feature spaces with relatively few samples that lead to poor classifier performance. To overcome this problem, feature selection and ensemble classification are applied in order to improve accuracy and stability. This research presents an analysis of the effect of removing irrelevant and redundant features with ensemble classifiers using five datasets and compared with floating search method. Eliminating redundant features provides better accuracy and computational time than removing irrelevant features of the ensemble.


Feature selection Ensemble classification Redundant feature Irrelevant feature 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Bellman, R.E.: Adaptive Control Processes: A Guided Tour. Princeton University Press, Princeton (1961)CrossRefMATHGoogle Scholar
  2. 2.
    Liu, H., Dougherty, E., Dy, J., Torkkola, K., Tuv, E., Peng, H., Ding, C., Long, F., Berens, M., Parsons, L., Zhao, Z., Yu, L., Forman, G.: Evolving feature selection. IEEE Intelligent Systems 20(6), 64–76 (2005)CrossRefGoogle Scholar
  3. 3.
    Liu, H., Yu, L.: Toward integrating feature selection algorithms for classification and clustering. IEEE Transactions on Knowledge and Data Engineering 17(4), 491–502 (2005)CrossRefGoogle Scholar
  4. 4.
    Saeys, Y., Inza, I., Larranaga, P.: A review of feature selection techniques in bioinformatics. Bioinformatics 23(19), 2507–2517 (2007)CrossRefGoogle Scholar
  5. 5.
    Windeatt, T.: Ensemble MLP Classifier Design. LNCS, vol. 137, pp. 133–147. Springer, Heidelberg (2008)Google Scholar
  6. 6.
    Windeatt, T.: Accuracy/diversity and ensemble MLP classifier design. IEEE Transactions on Neural Networks 17(5), 1194–1211 (2006)CrossRefGoogle Scholar
  7. 7.
    Breiman, L.: Bagging predictors. Machine Learning 24(2), 123–140 (1996)MATHGoogle Scholar
  8. 8.
    Freund, Y., Schapire, R.E.: Experiments with a new boosting algorithm. In: Proceedings of the Thirteenth International Conference on Machine Learning, pp. 148–156. Morgan Kaufmann, San Francisco (1996)Google Scholar
  9. 9.
    Witten, I.H., Frank, E.: Data Mining: Practical Machine Learning Tools and Techniques, 2nd edn. Morgan Kaufmann, San Francisco (2005)MATHGoogle Scholar
  10. 10.
    Almuallim, H., Dietterich, T.G.: Learning with many irrelevant features. In: Proceedings of the Ninth National Conference on Artificial Intelligence, pp. 547–552. AAAI Press, Menlo Park (1991)Google Scholar
  11. 11.
    Hall, M.A.: Correlation-based feature selection for discrete and numeric class machine learning. In: Proceeding of the 17th International Conference on Machine Learning, pp. 359–366. Morgan Kaufmann, San Francisco (2000)Google Scholar
  12. 12.
    Yu, L., Liu, H.: Efficient feature selection via analysis of relevance and redundancy. J. Mach. Learn. Res. 5, 1205–1224 (2004)MathSciNetMATHGoogle Scholar
  13. 13.
    Malarvili, M., Mesbah, M.: Hrv feature selection based on discriminant and redundancy analysis for neonatal seizure detection. In: 6th International Conference on Information, Communications and Signal Processing, p. 15 (2007)Google Scholar
  14. 14.
    Deisy, C., Subbulakshmi, B., Baskar, S., Ramaraj, N.: Efficient dimensionality reduction approaches for feature selection. In: International Conference on Computational Intelligence and Multimedia Applications, vol. 2, pp. 121–127 (2007)Google Scholar
  15. 15.
    Chou, T., Yen, K., Luo, J., Pissinou, N., Makki, K.: Correlation-based feature selection for intrusion detection design. In: IEEE on Military Communications Conference, MILCOM 2007, pp. 1–7 (2007)Google Scholar
  16. 16.
    Biesiada, J., Duch, W.: Feature Selection for High- Dimensional Data - A Pearson Redundancy Based Filter, vol. 45, pp. 242–249. Springer, Heidelberg (2008)Google Scholar
  17. 17.
    Biesiada, J., Duch, W.: A Kolmogorov-Smirnov Correlation-Based Filter for Microarray Data. In: Ishikawa, M., Doya, K., Miyamoto, H., Yamakawa, T. (eds.) ICONIP 2007, Part II. LNCS, vol. 4985, pp. 285–294. Springer, Heidelberg (2008)CrossRefGoogle Scholar
  18. 18.
    Kudo, M., Sklansky, J.: Comparison of algorithms that select features for pattern classifiers. Pattern Recognition 33, 25–41 (2000)CrossRefGoogle Scholar
  19. 19.
    Zhang, H., Sun, G.: Feature Selection using Tabu search. Pattern Recognition 35, 701–711 (2002)CrossRefMATHGoogle Scholar
  20. 20.
    Pudil, P., Novovicova, J., Kitler, J.: Floating Search Methods in Feature Selection. Pattern Recognition Letters 15, 1119–1125 (1994)CrossRefGoogle Scholar
  21. 21.
    Asuncion, A., Newman, D.: UCI machine learning repository (2007),
  22. 22.
    John, G., Kohavi, R., Pfleger, K.: Irrelevant features and the subset selection problem, pp. 121–129. Morgan Kaufmann, San Francisco (1994)Google Scholar
  23. 23.
    Polat, K., Gunes, S.: A hybrid approach to medical decision support systems: Combining feature selection, fuzzy weighted pre-processing and airs. Computer Methods and Program in Biomedicine 88(2), 164–174 (2007)CrossRefGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2009

Authors and Affiliations

  • Rakkrit Duangsoithong
    • 1
  • Terry Windeatt
    • 1
  1. 1.Center for Vision, Speech and Signal ProcessingUniversity of SurreyGuildfordUnited Kingdom

Personalised recommendations