Abstract
Ensemble learning trains and combines multiple base learners for a single learning task, and has been among the state-of-the-art learning techniques. Ensemble pruning tries to select a subset of base learners instead of combining them all, with the aim of achieving a better generalization performance as well as a smaller ensemble size. Previous methods often use the validation error to estimate the generalization performance during optimization, while recent theoretical studies have disclosed that margin distribution is also crucial for better generalization. Inspired by this finding, we propose to formulate ensemble pruning as a three-objective optimization problem that optimizes the validation error, margin distribution, and ensemble size simultaneously, and then employ multi-objective evolutionary algorithms to solve it. Experimental results on 20 binary classification data sets show that our proposed method outperforms the state-of-the-art ensemble pruning methods significantly in both generalization performance and ensemble size.
This work was supported by the National Science Foundation of China (62022039, 61921006).
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Back, T.: Evolutionary Algorithms in Theory and Practice: Evolution Strategies, Evolutionary Programming. Genetic Algorithms. Oxford University Press, Oxford, UK (1996)
Banfield, R.E., Hall, L.O., Bowyer, K.W., Kegelmeyer, W.P.: Ensemble diversity measures and their application to thinning. Inf. Fusion 6(1), 49–62 (2005)
Bartlett, P., Freund, Y., Lee, W.S., Schapire, R.E.: Boosting the margin: a new explanation for the effectiveness of voting methods. Ann. Stat. 26(5), 1651–1686 (1998)
Breiman, L.: Bagging predictors. Mach. Learn. 24(2), 123–140 (1996)
Breiman, L.: Prediction games and arcing algorithms. Neural Comput. 11(7), 1493–1517 (1999)
Breiman, L., Friedman, J., Olshen, R.A., Stone, C.J.: Classification and Regression Trees. Wadsworth and Brooks, Monterey (1984)
Caruana, R., Niculescu-Mizil, A., Crew, G., Ksikes, A.: Ensemble selection from libraries of models. In: Proceedings of the 21st International Conference on Machine Learning (ICML 2004), pp. 18–25. Banff, Canada (2004)
Deb, K., Jain, H.: An evolutionary many-objective optimization algorithm using reference-point-based nondominated sorting approach, Part I: Solving problems with box constraints. IEEE Trans. Evol. Comput. 18(4), 577–601 (2013)
Deb, K., Pratap, A., Agarwal, S., Meyarivan, T.: A fast and elitist multiobjective genetic algorithm: NSGA-II. IEEE Trans. Evol. Comput. 6(2), 182–197 (2002)
Dietterich, T., Margineantu, D.: Pruning adaptive boosting. In: Proceedings of the 14th International Conference on Machine Learning (ICML 1997), pp. 211–218. Nashville, TN (1997)
Drucker, H., Burges, C.J.C., Kaufman, L., Smola, A.J., Vapnik, V.: Support vector regression machines. In: Advances in Neural Information Processing Systems 9 (NIPS 1996), pp. 155–161. Denver, CO (1996)
Dua, D., Graff, C.: UCI machine learning repository (2017). http://archive.ics.uci.edu/ml
Eiben, A., Smith, J.: Introduction to Evolutionary Computing. Springer, Bering (2015)
Gao, W., Zhou, Z.H.: On the doubt about margin explanation of boosting. Artif. Intell. 203, 1–18 (2013)
Grønlund, A., Kamma, L., Larsen, K.G., Mathiasen, A., Nelson, J.: Margin-based generalization lower bounds for boosted classifiers. In: Advances in Neural Information Processing Systems 32 (NeurIPS 2019), Vancouver, Canada, pp. 11940–11949 (2019)
Laumanns, M., Thiele, L., Zitzler, E.: Running time analysis of multiobjective evolutionary algorithms on pseudo-Boolean functions. IEEE Trans. Evol. Comput. 8(2), 170–182 (2004)
Li, N., Yu, Y., Zhou, Z.H.: Diversity regularized ensemble pruning. In: Proceedings of the 23rd European Conference on Machine Learning (ECML 2012), Bristol, UK, pp. 330–345 (2012)
Li, N., Zhou, Z.H.: Selective ensemble under regularization framework. In: Proceedings of the 8th International Workshop on Multiple Classifier Systems (MCS 2009), Reykjavik, Iceland, pp. 293–303 (2009)
Lyu, S.H., Yang, L., Zhou, Z.H.: A refined margin distribution analysis for forest representation learning. In: Advances in Neural Information Processing Systems 32 (NeurIPS 2019), Vancouver, Canada, pp. 5531–5541 (2019)
Martínez-Muñoz, G., Hernández-Lobato, D., Suárez, A.: An analysis of ensemble pruning techniques based on ordered aggregation. IEEE Trans. Pattern Anal. Mach. Intell. 31(2), 245–259 (2008)
Martínez-Muñoz, G., Suárez, A.: Pruning in ordered bagging ensembles. In: Proceedings of the 23rd International Conference on Machine Learning (ICML 2006), Pittsburgh, PA, pp. 609–616 (2006)
Mohri, M., Rostamizadeh, A., Talwalkar, A.: Foundations of Machine Learning. MIT Press, Cambridge (2018)
Qian, C., Yu, Y., Zhou, Z.H.: An analysis on recombination in multi-objective evolutionary optimization. Artif. Intell. 204, 99–119 (2013)
Qian, C., Yu, Y., Zhou, Z.H.: Pareto ensemble pruning. In: Proceedings of the 29th AAAI Conference on Artificial Intelligence (AAAI 2015), Austin, TX, pp. 2935–2941 (2015)
Reyzin, L., Schapire, R.E.: How boosting the margin can also boost classifier complexity. In: Proceedings of the 23rd International Conference on Machine Learning (ICML 2006), Pittsburgh, PA , pp. 753–760 (2006)
Schapire, R.E.: The strength of weak learnability. Mach. Learn. 5(2), 197–227 (1990)
Wilcoxon, F.: Individual comparisons by ranking methods. Biometrics Bull. 1(6), 80–83 (1945)
Zhang, Q., Li, H.: MOEA/D: a multiobjective evolutionary algorithm based on decomposition. IEEE Trans. Evol. Comput. 11(6), 712–731 (2007)
Zhang, T., Zhou, Z.H.: Optimal margin distribution clustering. In: Proceedings of the 32nd AAAI Conference on Artificial Intelligence (AAAI 2018), New Orleans, LA, pp. 4474–4481 (2018)
Zhang, T., Zhou, Z.H.: Optimal margin distribution machine. IEEE Trans. Knowl. Data Eng. 32(6), 1143–1156 (2019)
Zhou, Z.H.: Ensemble Methods: Foundations and Algorithms. Chapman & Hall/CRC Press, Boca Raton, FL (2012)
Zhou, Z.H.: Large margin distribution learning. In: Proceedings of the 6th International Workshop on Artificial Neural Networks in Pattern Recognition (ANNPR 2014), Montreal, Canada, pp. 1–11 (2014)
Zhou, Z.H., Tang, W.: Selective ensemble of decision trees. In: Proceddings of the 9th International Conference on Rough Sets, Fuzzy Sets, Data Mining, and Granular Computing (RSFDGrC 2003), Chongqing, China, pp. 476–483 (2003)
Zhou, Z.H., Wu, J., Tang, W.: Ensembling neural networks: many could be better than all. Artif. Intell. 137(1–2), 239–263 (2002)
Zhou, Z.H., Yu, Y., Qian, C.: Evolutionary Learning: Advances in Theories and Algorithms. Springer, Singapore (2019)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Wu, YC., He, YX., Qian, C., Zhou, ZH. (2022). Multi-objective Evolutionary Ensemble Pruning Guided by Margin Distribution. In: Rudolph, G., Kononova, A.V., Aguirre, H., Kerschke, P., Ochoa, G., Tušar, T. (eds) Parallel Problem Solving from Nature – PPSN XVII. PPSN 2022. Lecture Notes in Computer Science, vol 13398. Springer, Cham. https://doi.org/10.1007/978-3-031-14714-2_30
Download citation
DOI: https://doi.org/10.1007/978-3-031-14714-2_30
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-14713-5
Online ISBN: 978-3-031-14714-2
eBook Packages: Computer ScienceComputer Science (R0)