An Integrated Pruning Criterion for Ensemble Learning Based on Classification Accuracy and Diversity

  • Bin Fu
  • Zhihai Wang
  • Rong Pan
  • Guandong Xu
  • Peter Dolog
Part of the Advances in Intelligent Systems and Computing book series (AISC, volume 172)

Abstract

Ensemble pruning is an important issue in the field of ensemble learning. Diversity is a key criterion to determine how the pruning process has been done and measure what result has been derived. However, there is few formal definitions of diversity yet. Hence, three important factors that should be further considered while designing a pruning criterion is presented, and then an effective definition of diversity is proposed. The experimental results have validated that the given pruning criterion could single out the subset of classifiers that show better performance in the process of hill-climbing search, compared with other definitions of diversity and other criteria.

Keywords

Ensemble Learning Classification Ensemble Pruning Diversity of Classifiers 

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. 1.
    Dietterich, T.G.: Ensemble Methods in Machine Learning. In: Kittler, J., Roli, F. (eds.) MCS 2000. LNCS, vol. 1857, pp. 1–15. Springer, Heidelberg (2000)CrossRefGoogle Scholar
  2. 2.
    Shipp, C., Kuncheva, L.: Relationships between combination methods and measures of diversity in combing classifiers. Information Fusion 3(2), 135–148 (2002)CrossRefGoogle Scholar
  3. 3.
    Kuncheva, L., Whitaker, C.: Measures of diversity in classifier ensembles and their relationship with the ensemble accuracy. Machine Learning 51(2), 181–207 (2003)MATHCrossRefGoogle Scholar
  4. 4.
    Breiman, L.: Bagging predictors. Machine Learning 24(2), 123–140 (1996)MathSciNetMATHGoogle Scholar
  5. 5.
    Freund, Y., Schapire, R.: Experiments with a new boosting algorithm. In: Proceedings of the Thirteenth International Conference on Machine Learning (ICML 1996), pp. 148–156 (1996)Google Scholar
  6. 6.
    Ho, T.: The random subspace method for constructing decision forests. IEEE Transactions on Pattern Analysis and Machine Intelligence 20(8), 832–844 (1998)CrossRefGoogle Scholar
  7. 7.
    Caruana, R., Niculescu-Miil, A., Crew, G., et al.: Ensemble Selection from Libraries of Models. In: Proceedings of the Twenty-First International Conference (ICML 2004), pp. 18–25 (2004)Google Scholar
  8. 8.
    Martinez-Munoz, G., Suarez, A.: Using boosting to prune bagging ensembles. Pattern Recognition Letters 28, 156–165 (2007)CrossRefGoogle Scholar
  9. 9.
    Tsoumakas, G., Partalas, I., Vlahavas, I.: An Ensemble Pruning Primer. In: Okun, O., Valentini, G. (eds.) Applications of Supervised and Unsupervised Ensemble Methods. SCI, vol. 245, pp. 1–13. Springer, Heidelberg (2009)CrossRefGoogle Scholar
  10. 10.
    Martinez-Munoz, G., Suarez, A.: Aggregation ordering in bagging. In: Proceedings of the 2004 International Conference on Artificial Intelligence and Applications, pp. 258–263 (2004)Google Scholar
  11. 11.
    Yang, Y., Korb, K.B., Ting, K.M., Webb, G.I.: Ensemble Selection for SuperParent-One-Dependence Estimators. In: Zhang, S., Jarvis, R.A. (eds.) AI 2005. LNCS (LNAI), vol. 3809, pp. 102–112. Springer, Heidelberg (2005)CrossRefGoogle Scholar
  12. 12.
    Banfield, R., Hall, L., Bowyer, K., et al.: Ensemble diversity measures and their application to thinning. Information Fusion 6(1), 49–62 (2005)CrossRefGoogle Scholar
  13. 13.
    Partalas, I., Tsoumakas, G., Vlahavas, I.: Focused ensemble selection: a diversity-Based method for greedy ensemble selection. In: Proceedings of the 18th European Conference on Artificial Intelligence (ECAI 2008), pp. 117–121 (2008)Google Scholar
  14. 14.
    Caruana, R., Niculescu-Mizil, A.: Data mining in metric space: an empirical analysis of supervised learning performance criteria. In: Proceedings of the 10th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD 2004), pp. 69–78 (2004)Google Scholar
  15. 15.
    Kapp, M., Sabourin, R., Maupin, P.: An empirical study on diversity measures and margin theory for ensembles of classifiers. Information Fusion (2007)Google Scholar
  16. 16.
    Skalak, D.: The sources of increased accuracy for two proposed boosting algorithms. In: Proceedings of the 11th AAAI Conference on Artificial Intelligence, pp. 120–125 (1996)Google Scholar
  17. 17.
    Giacinto, G., Roli, F.: Design of effective neural network ensembles for image classification processes. Image Vision and Computing Journal 19(9), 699–707 (2001)CrossRefGoogle Scholar
  18. 18.
    Kohavi, R., Wolpert, D.: Bias plus variance decomposition for zero-one loss functions. In: Proceedings of the Thirteenth International Conference on Machine Learning, pp. 275–283 (1996)Google Scholar
  19. 19.
    Hashem, L., Salamon, P.: Neural network ensembles. IEEE Transaction on Pattern Analysis and Machine Intelligence 12(10), 993–1001 (1990)CrossRefGoogle Scholar
  20. 20.
    Tang, E., Suganthan, P., Yao, X.: An analysis of diversity measures. Machine Learning 65(1), 241–271 (2006)CrossRefGoogle Scholar
  21. 21.
    Schapire, R., Freund, Y., Bartlett, P., et al.: Boosting the margin: a new explanation for the effectiveness of voting methods. In: Proceedings of the Fourteenth International Conference on Machine Learning (ICML 1997), pp. 322–330 (1997)Google Scholar
  22. 22.
    Witten, I., Frank, E.: Data Mining: Practical Machine Learning Tools and Techniques with Java Implementations, pp. 365–368. Morgan Kaufmann Publishers, San Francisco (2000)Google Scholar
  23. 23.
    Asuncion A, Newman, D.: UCI Machine Learning Repository. University of California, School of Information and Computer Science, Irvine, CA (2007), http://www.ics.uci.edu/~mlearn/MLRepository.html

Copyright information

© Springer-Verlag Berlin Heidelberg 2013

Authors and Affiliations

  • Bin Fu
    • 1
  • Zhihai Wang
    • 1
  • Rong Pan
    • 2
  • Guandong Xu
    • 3
  • Peter Dolog
    • 2
  1. 1.School of Computer and Information TechnologyBeijing Jiaotong UniversityBeijingChina
  2. 2.Department of Computer ScienceAalborg UniversityAalborgDenmark
  3. 3.School of Engineering & ScienceVictoria UniversityVictoriaAustralia

Personalised recommendations