Machine Learning

, Volume 81, Issue 3, pp 257–282 | Cite as

An ensemble uncertainty aware measure for directed hill climbing ensemble pruning

  • Ioannis PartalasEmail author
  • Grigorios Tsoumakas
  • Ioannis Vlahavas


This paper proposes a new measure for ensemble pruning via directed hill climbing, dubbed Uncertainty Weighted Accuracy (UWA), which takes into account the uncertainty of the decision of the current ensemble. Empirical results on 30 data sets show that using the proposed measure to prune a heterogeneous ensemble leads to significantly better accuracy results compared to state-of-the-art measures and other baseline methods, while keeping only a small fraction of the original models. Besides the evaluation measure, the paper also studies two other parameters of directed hill climbing ensemble pruning methods, the search direction and the evaluation dataset, with interesting conclusions on appropriate values.


Ensemble pruning Ensemble selection Ensemble methods 


  1. Asuncion, A., & Newman, D. (2007). UCI machine learning repository.
  2. Banfield, R. E., Hall, L. O., Bowyer, K. W., & Kegelmeyer, W. P. (2005). Ensemble diversity measures and their application to thinning. Information Fusion, 6(1), 49–62. CrossRefGoogle Scholar
  3. Breiman, L. (1996). Bagging predictors. Machine Learning, 24(2), 123–140. zbMATHMathSciNetGoogle Scholar
  4. Caruana, R., Niculescu-Mizil, A., Crew, G., & Ksikes, A. (2004). Ensemble selection from libraries of models. In Proceedings of the 21st international conference on machine learning, p. 18. Google Scholar
  5. Caruana, R., Munson, A., & Niculescu-Mizil, A. (2006). Getting the most out of ensemble selection. In Proceedings of the international conference on data mining (ICDM), pp. 828–833. Google Scholar
  6. Demsar, J. (2006). Statistical comparisons of classifiers over multiple data sets. Journal of Machine Learning Research, 7, 1–30. MathSciNetGoogle Scholar
  7. Dietterich, T. G. (2000). Ensemble methods in machine learning. In Proceedings of the 1st international workshop in multiple classifier systems (pp. 1–15). Google Scholar
  8. Fan, W., Chu, F., Wang, H., & Yu, P. S. (2002). Pruning and dynamic scheduling of cost-sensitive ensembles. In Eighteenth national conference on artificial intelligence, American association for artificial intelligence (pp. 146–151). Google Scholar
  9. Giacinto, G., Roli, F., & Fumera, G. (2000). Design of effective multiple classifier systems by clustering of classifiers. In 15th international conference on pattern recognition, ICPR, 2000 (pp. 160–163). Google Scholar
  10. Kuncheva, L. I., & Whitaker, C. J. (2003). Measures of diversity in classifier ensembles and their relationship with the ensemble accuracy. Machine Learning, 51(2), 181–207. zbMATHCrossRefGoogle Scholar
  11. Margineantu, D., & Dietterich, T. (1997). Pruning adaptive boosting. In Proceedings of the 14th international conference on machine learning (pp. 211–218). Google Scholar
  12. Martinez-Munoz, G., & Suarez, A. (2004). Aggregation ordering in bagging. In International conference on artificial intelligence and applications (IASTED) (pp. 258–263). Calgary: Acta Press. Google Scholar
  13. Martinez-Munoz, G., & Suarez, A. (2006). Pruning in ordered bagging ensembles. In 23rd international conference in machine learning (ICML-2006) (pp. 609–616). New York: ACM Press. CrossRefGoogle Scholar
  14. Partalas, I., Tsoumakas, G., & Vlahavas, I. (2008). Focused ensemble selection: a diversity-based method for greedy ensemble selection. In M. Ghallab, C. D. Spyropoulos, N. Fakotakis, & N. M. Avouris (Eds.), Frontiers in artificial intelligence and applications : Vol. 178. ECAI 2008—18th European conference on artificial intelligence, Patras, Greece, July 21–25, 2008 (pp. 117–121). Amsterdam: IOS Press. Google Scholar
  15. Schapire, R. E. (1990). The strength of weak learnability. Machine Learning, 5(2), 197–227. Google Scholar
  16. Tang, E. K., Suganthan, P. N., & Yao, X. (2006). An analysis of diversity measures. Machine Learning, 65(1), 247–271. CrossRefGoogle Scholar
  17. Tsoumakas, G., Angelis, L., & Vlahavas, I. (2005). Selective fusion of heterogeneous classifiers. Intelligent Data Analysis, 9(6), 511–525. Google Scholar
  18. Tsoumakas, G., Partalas, I., & Vlahavas, I. (2009). An ensemble pruning primer. In O. Okun, & G. Valentino (Eds.), Applications of supervised and unsupervised ensemble methods (pp. 1–13). Berlin: Springer. CrossRefGoogle Scholar
  19. Wilcoxon, F. (1945). Individual comparisons by ranking methods. Biometrics, 1, 80–83. CrossRefGoogle Scholar
  20. Witten, I. H., & Frank, E. (2005). Data mining: practical machine learning tools and techniques. San Mateo: Morgan Kaufmann. zbMATHGoogle Scholar

Copyright information

© The Author(s) 2010

Authors and Affiliations

  • Ioannis Partalas
    • 1
    Email author
  • Grigorios Tsoumakas
    • 1
  • Ioannis Vlahavas
    • 1
  1. 1.Department of InformaticsAristotle University of ThessalonikiThessalonikiGreece

Personalised recommendations