An Empirical Investigation on the Use of Diversity for Creation of Classifier Ensembles

  • Muhammad A. O. Ahmed
  • Luca Didaci
  • Giorgio FumeraEmail author
  • Fabio Roli
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 9132)


We address one of the main open issues about the use of diversity in multiple classifier systems: the effectiveness of the explicit use of diversity measures for creation of classifier ensembles. So far, diversity measures have been mostly used for ensemble pruning, namely, for selecting a subset of classifiers out of an original, larger ensemble. Here we focus on pruning techniques based on forward/backward selection, since they allow a direct comparison with the simple estimation of accuracy of classifier ensemble. We empirically carry out this comparison for several diversity measures and benchmark data sets, using bagging as the ensemble construction technique, and majority voting as the fusion rule. Our results provide further and more direct evidence to previous observations against the effectiveness of the use of diversity measures for ensemble pruning, but also show that, combined with ensemble accuracy estimated on a validation set, diversity can have a regularization effect when the validation set size is small.


Diversity Ensemble pruning Forward/backward selection Ensemble construction 



This work has been partly supported by the project CRP-59872 funded by Regione Autonoma della Sardegna, L.R. 7/2007, Bando 2012.


  1. 1.
    Banfield, R.E., Hall, L.O., Bowyer, K.W., Kegelmeyer, W.P.: EnsembleUWA diversity measures and their application to thinning. Inf. Fusion 6(1), 49–62 (2005)CrossRefGoogle Scholar
  2. 2.
    Brown, G., Wyatt, J.L., Harris, R., Yao, X.: Diversity creation methods: a survey and categorisation. Inf. Fusion 6(1), 5–20 (2005)CrossRefGoogle Scholar
  3. 3.
    Brown, G., Kuncheva, L.I.: “Good” and “Bad” diversity in majority vote ensembles. In: El Gayar, N., Kittler, J., Roli, F. (eds.) MCS 2010. LNCS, vol. 5997, pp. 124–133. Springer, Heidelberg (2010) CrossRefGoogle Scholar
  4. 4.
    Caruana, R., Niculescu-Mizil, A., Crew, G., Ksikes, A.: Ensemble selection from libraries of models. In: 21st International Conference on Machine Learning, p. 18. ACM (2004)Google Scholar
  5. 5.
    Demsar, J.: Statistical comparisons of classifiers over multiple data sets. J. Mach. Learn. Res. 7, 1–30 (2005)MathSciNetGoogle Scholar
  6. 6.
    Didaci, L., Fumera, G., Roli, F.: Diversity in classifier ensembles: fertile concept or dead end? In: Zhou, Z.-H., Roli, F., Kittler, J. (eds.) MCS 2013. LNCS, vol. 7872, pp. 37–48. Springer, Heidelberg (2013) CrossRefGoogle Scholar
  7. 7.
    Ko, A.H.-R., Sabourin, R., de Souza Britto Jr., A.: Compound diversity functions for ensemble selection. Int. J. Patt. Rec. Artif. Int. 23(4), 659–686 (2009)CrossRefGoogle Scholar
  8. 8.
    Krogh, A., Vedelsby, J.: Neural network ensembles, cross validation, and active learning. In: Tesauro, G., Touretzky, D.S., Leen, T.K. (eds.) Advances in Neural Information Processing Systems 7, pp. 231–238. MIT Press, Cambridge (1995) Google Scholar
  9. 9.
    Kuncheva, L.I., Whitaker, C.J.: Measures of diversity in classifier ensembles and their relationship with the ensemble accuracy. Mach. Learn. 51(2), 181–207 (2003)CrossRefzbMATHGoogle Scholar
  10. 10.
    Kuncheva, L.I.: A bound on kappa-error diagrams for analysis of classifier ensembles. IEEE Trans. Knowl. Data Eng. 25(3), 494–501 (2013)CrossRefGoogle Scholar
  11. 11.
    Kuncheva, L.I.: Combining Pattern Classifiers: Methods and Algorithms, 2nd edn. Wiley, Hoboken (2014) CrossRefGoogle Scholar
  12. 12.
    Li, N., Yu, Y., Zhou, Z.-H.: Diversity regularized ensemble pruning. In: Flach, P.A., De Bie, T., Cristianini, N. (eds.) ECML PKDD 2012, Part I. LNCS, vol. 7523, pp. 330–345. Springer, Heidelberg (2012) CrossRefGoogle Scholar
  13. 13.
    Margineantu, D.D., Dietterich, T.G.: Pruning adaptive boosting. In: 14th International Conference Machine Learning, pp. 378–387. Morgan Kaufmann (1997)Google Scholar
  14. 14.
    Martinez-Munoz, G., Suarez, A.: Aggregation ordering in bagging. In: International Conference on Artificial Intelligence and Applications, pp. 258–263 (2004)Google Scholar
  15. 15.
    Partalas, I., Tsoumakas, G., Vlahavas, I.P.: An ensemble uncertainty aware measure for directed hill climbing ensemble pruning. Mach. Learn. 81, 257–282 (2010)CrossRefMathSciNetGoogle Scholar
  16. 16.
    Partridge, D., Yates, W.B.: Engineering multiversion neural-net systems. Neural Comput. 8(4), 869–893 (1996)CrossRefGoogle Scholar
  17. 17.
    Prodromidis, A., Stolfo, S.J.: Pruning meta-classifiers in a distributed data mining system. In: Proceedings of the 1st National Conference on New Information Technologies, pp. 151–160 (1998)Google Scholar
  18. 18.
    Rokach, L.: Collective-agreement-based pruning of ensembles. Comp. Stat. Data Anal. 53(4), 1015–1026 (2009)CrossRefzbMATHMathSciNetGoogle Scholar
  19. 19.
    Tang, E.K., Suganthan, P.N., Yao, X.: An analysis of diversity measures. Mach. Learn. 65, 247–271 (2006)CrossRefGoogle Scholar
  20. 20.
    Tsoumakas, G., Partalas, I., Vlahavas, I.: An ensemble pruning primer. In: Okun, Oleg, Valentini, Giorgio (eds.) Applications of Supervised and Unsupervised Ensemble Methods. SCI, vol. 245, pp. 1–13. Springer, Heidelberg (2009) CrossRefGoogle Scholar
  21. 21.
    Ueda, N., Nakano, R.: Generalization error of ensemble estimators. In: International Conference on Neural Networks, pp. 90–95 (1996)Google Scholar
  22. 22.
    Yu, Y., Li, Y.-F., Zhou, Z.-H.: Diversity regularized machine. In: 22nd International Joint Conference on Artificial Intelligence, pp. 1603–1608 (2011)Google Scholar
  23. 23.
    Zhou, Z.-H., Wu, J., Tang, W.: Ensembling neural networks: many could be better than all. Artif. Intell. 137(1–2), 239–263 (2002)CrossRefzbMATHMathSciNetGoogle Scholar
  24. 24.
    Yu, Y., Li, Y.-F., Zhou, Z.-H.: Diversity regularized machine. In: Proceedings of the 22nd International Joint Conference on Artificial Intelligence, pp. 1603–1608 (2011)Google Scholar
  25. 25.
    Zhou, Z.-H.: Ensemble Methods: Foundations and Algorithms. CRC Press, USA (2012)Google Scholar

Copyright information

© Springer International Publishing Switzerland 2015

Authors and Affiliations

  • Muhammad A. O. Ahmed
    • 1
  • Luca Didaci
    • 1
  • Giorgio Fumera
    • 1
    Email author
  • Fabio Roli
    • 1
  1. 1.Department of Electrical and Electronic EngineeringUniversity of CagliariCagliariItaly

Personalised recommendations