Abstract
Ensemble learning provides a theoretically well-founded approach to address the bias-variance trade-off by combining many learners to obtain an aggregated model with reduced bias or variance. This same idea of extracting knowledge from the predictions or choices of individuals has been also studied under different perspectives in the domains of social choice theory and collective intelligence. Despite this similarity, there has been little research comparing and relating the aggregation strategies proposed in these different domains. In this article, we aim to bridge the gap between these disciplines by means of an experimental evaluation, done on a set of standard datasets, of different aggregation criteria in the context of the training of ensembles of decision trees. We show that a social-science method known as surprisingly popular decision and the three-way reduction, achieve the best performance, while both bagging and boosting outperform social choice-based Borda and Copeland methods.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Arrow, K.J.: Social Choice and Individual Values. Wiley, New York (1951)
Awasthi, P., Blum, A., Haghtalab, N., Mansour, Y.: Efficient PAC learning from the crowd. arXiv preprint arXiv:1703.07432 (2017)
Brams, S., Fishburn, P.C.: Approval Voting. Springer, New York (2007). https://doi.org/10.1007/978-0-387-49896-6
Brandt, F., Conitzer, V., Endriss, U., Lang, J., Procaccia, A.D.: Handbook of Computational Social Choice. Cambridge University Press, Cambridge (2016)
Breiman, L.: Bagging predictors. Mach. Learn. 24(2), 123–140 (1996)
Breiman, L.: Random forests. Mach. Learn. 45(1), 5–32 (2001)
Bubeck, S., Cesa-Bianchi, N., et al.: Regret analysis of stochastic and nonstochastic multi-armed bandit problems. Found. Trends® Mach. Learn. 5(1), 1–122 (2012)
Cabitza, F., Campagner, A., Ciucci, D.: New frontiers in explainable AI: understanding the GI to interpret the GO. In: Holzinger, A., Kieseberg, P., Tjoa, A.M., Weippl, E. (eds.) CD-MAKE 2019. LNCS, vol. 11713, pp. 27–47. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-29726-8_3
Campagner, A., Ciucci, D., Svensson, C.M., Figge, T., Cabitza, F.: Ground truthing from multi-rater labelling with three-way decisions and possibility theory. Inf. Sci. (2020, submitted)
Chourasia, R., Singla, A.: Unifying ensemble methods for q-learning via social choice theory. CoRR abs/1902.10646 (2019)
Colomer, J.M.: Ramon Llull: from ‘Ars electionis’ to social choice theory. Soc. Choice Welf. 40(2), 317–328 (2013). https://doi.org/10.1007/s00355-011-0598-2
Cornelio, C., Donini, M., Loreggia, A., Pini, M.S., Rossi, F.: Voting with random classifiers (VORACE). arXiv preprint arXiv:1909.08996 (2019)
de Condorcet, J.: Essai sur l’application de l’analyse à la probabilité des decisions rendues à la pluralité des voix. Imprimerie Royale, Paris (1785)
Dua, D., Graff, C.: UCI machine learning repository (2017). http://archive.ics.uci.edu/ml
Dubois, D., Prade, H.: Possibility Theory: An Approach to Computerized Processing of Uncertainty. Springer, Boston (2012). https://doi.org/10.1007/978-1-4684-5287-7
Emerson, P.: The original Borda count and partial voting. Soc. Choice Welf. 40(2), 353–358 (2013). https://doi.org/10.1007/s00355-011-0603-9
Endriss, U.: Trends in Computational Social Choice. Lulu.com, Morrisville (2017)
Fraenkel, J., Grofman, B.: The Borda count and its real-world alternatives: comparing scoring rules in Nauru and Slovenia. Aust. J. Polit. Sci. 49(2), 186–205 (2014)
Freund, Y., Schapire, R.E.: A desicion-theoretic generalization of on-line learning and an application to boosting. In: Vitányi, P. (ed.) EuroCOLT 1995. LNCS, vol. 904, pp. 23–37. Springer, Heidelberg (1995). https://doi.org/10.1007/3-540-59119-2_166
Friedman, M.: A comparison of alternative tests of significance for the problem of m rankings. Ann. Math. Stat. 11(1), 86–92 (1940)
Görzen, T., Laux, F.: Extracting the wisdom from the crowd: a comparison of approaches to aggregating collective intelligence. Paderborn University, Faculty of Business Administration and Economics, Technical report (2019)
Hastie, T., Tibshirani, R., Friedman, J.: The Elements of Statistical Learning: Data Mining, Inference, and Prediction. Springer, New York (2009). https://doi.org/10.1007/978-0-387-84858-7
Heinecke, S., Reyzin, L.: Crowdsourced PAC learning under classification noise. In: Proceedings of the AAAI Conference on Human Computation and Crowdsourcing, vol. 7, pp. 41–49 (2019)
Hertwig, R.: Tapping into the wisdom of the crowd-with confidence. Science 336(6079), 303–304 (2012)
Hollander, M., Wolfe, D.A., Chicken, E.: Nonparametric Statistical Methods, vol. 751. Wiley, Hoboken (2013)
Kearns, M.: Thoughts on hypothesis boosting. Unpublished manuscript 45, 105 (1988)
Kearns, M., Valiant, L.: Cryptographic limitations on learning Boolean formulae and finite automata. J. ACM (JACM) 41(1), 67–95 (1994)
Lee, M.D., Steyvers, M., De Young, M., Miller, B.: Inferring expertise in knowledge and prediction ranking tasks. Top. Cogn. Sci. 4(1), 151–163 (2012)
Leon, F., Floria, S.A., Bădică, C.: Evaluating the effect of voting methods on ensemble-based classification. In: IEEE INISTA, vol. 2017, pp. 1–6 (2017)
Leung, K.T., Parker, D.S.: Empirical comparisons of various voting methods in bagging. In: Proceedings of the Ninth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 595–600 (2003)
Luo, T., Liu, Y.: Machine truth serum. arXiv preprint arXiv:1909.13004 (2019)
Mason, L., Baxter, J., Bartlett, P.L., Frean, M.R.: Boosting algorithms as gradient descent. In: Advances in Neural Information Processing Systems, pp. 512–518 (2000)
Pennock, D.M., Maynard-Reid II, P., Giles, C.L., Horvitz, E.: A normative examination of ensemble learning algorithms. In: ICML, pp. 735–742 (2000)
Prelec, D.: A Bayesian truth serum for subjective data. Science 306(5695), 462–466 (2004)
Prelec, D., Seung, H.S., McCoy, J.: A solution to the single-question crowd wisdom problem. Nature 541(7638), 532–535 (2017)
Rangi, A., Franceschetti, M.: Multi-armed bandit algorithms for crowdsourcing systems with online estimation of workers’ ability. In: Proceedings of ICAAMAS, vol. 2018, pp. 1345–1352 (2018)
Shalev-Shwartz, S., Ben-David, S.: Understanding Machine Learning: From Theory to Algorithms. Cambridge University Press, Cambridge (2014)
Simmons, J.P., Nelson, L.D., Galak, J., Frederick, S.: Intuitive biases in choice versus estimation: implications for the wisdom of crowds. J. Consum. Res. 38(1), 1–15 (2011)
Surowiecki, J.: The Wisdom of Crowds. Anchor (2005)
Yao, Y.: An outline of a theory of three-way decisions. In: Yao, J.T., et al. (eds.) RSCTC 2012. LNCS (LNAI), vol. 7413, pp. 1–17. Springer, Heidelberg (2012). https://doi.org/10.1007/978-3-642-32115-3_1
Zadeh, L.: Fuzzy sets as the basis for a theory of possibility. Fuzzy Sets Syst. 1, 3–28 (1978)
Zhang, H., Conitzer, V.: A PAC framework for aggregating agents’ judgments. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 33, pp. 2237–2244 (2019)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2020 Springer Nature Switzerland AG
About this paper
Cite this paper
Campagner, A., Ciucci, D., Cabitza, F. (2020). Ensemble Learning, Social Choice and Collective Intelligence. In: Torra, V., Narukawa, Y., Nin, J., Agell, N. (eds) Modeling Decisions for Artificial Intelligence. MDAI 2020. Lecture Notes in Computer Science(), vol 12256. Springer, Cham. https://doi.org/10.1007/978-3-030-57524-3_5
Download citation
DOI: https://doi.org/10.1007/978-3-030-57524-3_5
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-57523-6
Online ISBN: 978-3-030-57524-3
eBook Packages: Computer ScienceComputer Science (R0)