Abstract
In recent years crowd-voting and crowd-sourcing systems are attracting increased attention in research and industry. As a part of computational social choice (COMSOC) crowd-voting and crowd-sourcing address important societal problems (e.g. participatory budgeting), but also many industry problems (e.g. sentiment analyses, data labeling, ranking and selection, etc.). Consequently, decisions that are based on aggregation of crowd votes do not guarantee high-quality results. Even more, in many cases majority of crowd voters may not be satisfied with final decisions if votes have high heterogeneity. On the other side in many crowd voting problems and settings it is possible to acquire and formalize knowledge and/or opinions from domain experts. Integration of expert knowledge and “Wisdom of crowd” should lead to high-quality decisions that satisfy crowd opinion. In this research, we address the problem of integration of experts domain knowledge with “Wisdom of crowds” by proposing machine learning based framework that enables ranking and selection of alternatives as well as quantification of quality of crowd votes. This framework enables weighting of crowd votes with respect to expert knowledge and procedures for modeling trade-off between crowd and experts satisfaction with final decisions (ranking or selection).
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Traunmueller, M., Fatah, G., Schieck, A.: Introducing the space recommender system: how crowd-sourced voting data can enrich urban exploration in the digital era. In: Proceedings of the 6th International Conference on Communities and Technologies, pp. 149–156 (2013)
Jung, H.J., Lease, M.: Inferring missing relevance judgments from crowd workers via probabilistic matrix factorization. In: Proceedings of the 35th International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 1095–1096 (2012)
Hong, H., Du, Q., Wang, G., Fan, W., Xu, D.: Crowd wisdom: the impact of opinion diversity and participant independence on crowd performance. In: Twenty-second Americas Conference on Information Systems (2016)
Hill, S., Ready-Campbell, N.: Expert stock picker: the wisdom of (experts in) crowds. Int. J. Electr. Commer. 15(3), 73–102 (2011)
Garcia, A.C., Klein, M.: pBOL: an idea filtering method based on negative multi-voting and Pareto aggregation (2017). http://dx.doi.org/10.2139/ssrn.3175329
Lees, A., Welty, C.: Discovering user bias in ordinal voting systems. In: Companion Proceedings of the 2019 World Wide Web Conference, pp. 1106–1110 (2019)
Bhatt, S., Chen, K., Shalin, V.L., Sheth, A.P., Minnery, B.: Who should be the captain this week? Leveraging inferred diversity-enhanced crowd wisdom for a fantasy premier league captain prediction. In: Proceedings of the International AAAI Conference on Web and Social Media, vol. 13, no. 01, pp. 103–113 (2019)
Suran, S., Pattanaik, V., Draheim, D.: Frameworks for collective intelligence: a systematic literature review. ACM Comput. Surv. (CSUR) 53(1), 1–36 (2020)
Dodevska, Z.A.: Computational social choice and challenges of voting in multi-agent systems. Tehnika 74(5), 724–730 (2019)
Chen, L., Xu, P., Liu, D.: The effect of crowd voting on participation in crowdsourcing contests. In: Working paper (2019). 39 pages
Bennet, A., Bennet, D.: The decision-making process for complex situations in a complex environment. In: Burstein, F., Holsapple, C.W. (eds.) Handbook on Decision Support Systems 1, pp. 3–20. Springer, Heidelberg (2008). https://doi.org/10.1007/978-3-540-48713-5_1
Yu, C., Chai, Y., Liu, Y.: Literature review on collective intelligence: a crowd science perspective. Int. J. Crowd Sci. 2(1), 64–73 (2018)
Mandic, K., Bobar, V., Delibašić, B.: Modeling interactions among criteria in MCDM methods: a review. In: Delibašić, B., Hernández, J.E., Papathanasiou, J., Dargam, F., Zaraté, P., Ribeiro, R., Liu, S., Linden, I. (eds.) ICDSST 2015. LNBIP, vol. 216, pp. 98–109. Springer, Cham (2015). https://doi.org/10.1007/978-3-319-18533-0_9
Rossi, F.: Preferences, Constraints, Uncertainty, and Multi-Agent Scenarios. ISAIM (2008)
Jackson, S.E.: The consequences of diversity in multidisciplinary work teams. In: West, M.A. (ed.) Handbook of Work Group Psychology, pp. 53–75. Wiley, Chichester (1996)
Miller, N.R.: Reflections on Arrow’s theorem and voting rules. Publ. Choice 179(1–2), 113–124 (2019). https://doi.org/10.1007/s11127-018-0524-6
Herings, P.J.-J., Houba, H.: The Condorcet paradox revisited. Soc. Choice Welfare 47(1), 141–186 (2016). https://doi.org/10.1007/s00355-016-0950-7
Aitamurto, T., Landemore, H., Galli, J.S.: Unmasking the crowd: participants’ motivation factors, expectations, and profile in a crowdsourced law reform. Inf. Commun. Soc. 20(8), 1239–1260 (2017)
Goel, A., Krishnaswamy, A.K., Sakshuwong, S., Aitamurto, T.: Knapsack voting for participatory budgeting. ACM Trans. Econ. Comput. (TEAC) 7(2), 1–27 (2019)
Ghezzi, A., Gabelloni, D., Martini, A., Natalicchio, A.: Crowdsourcing: a review and suggestions for future research. Int. J. Manage. Rev. 20(2), 343–363 (2017)
Isinkaye, F.O., Folajimi, Y.O., Ojokoh, B.A.: Recommendation systems: P, methods and evaluation. Egyptian Inform. J. 16(3), 261–273 (2015)
Dodevska, Z., Kovacevic, A., Vukicevic, M., Delibasic, B.: Two sides of collective decision making - votes from crowd and knowledge from experts. In: ICDSST 2020, EWG-DSS 6th International Conference on Decision Support System Technology (2020, in press)
Takács, G., Tikk, D.: Alternating least squares for personalized ranking. In: Proceedings of the Sixth ACM Conference on Recommender Systems (2012)
Mikolov, T., Chen, K., Corrado, G., Dean, J.: Efficient estimation of word representations in vector space. arXiv preprint arXiv:1301.3781 (2013)
Pennington, J., Socher, R., Manning, C.D.: Glove: global vectors for word representation. In: Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), October, pp. 1532–1543 (2014)
Lloyd, S.P.: Least squares quantization in PCM. IEEE Trans. Inf. Theory 28(2), 129–137 (1982)
Liu, F.T, Ting, K.M, Zhou, Z.H.: Isolation forest. In: 2008 Eighth IEEE International Conference on Data Mining, 15 December 2008, pp. 413–422. IEEE (2008)
Acknowledgments
This paper is a result of the project ONR - N62909-19-1-2008 supported by the Office for Naval Research, the United States: Aggregating computational algorithms and human decision-making preferences in multi-agent settings.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2020 Springer Nature Switzerland AG
About this paper
Cite this paper
Kovacevic, A., Vukicevic, M., Radovanovic, S., Delibasic, B. (2020). CrEx-Wisdom Framework for Fusion of Crowd and Experts in Crowd Voting Environment – Machine Learning Approach. In: Bellatreche, L., et al. ADBIS, TPDL and EDA 2020 Common Workshops and Doctoral Consortium. TPDL ADBIS 2020 2020. Communications in Computer and Information Science, vol 1260. Springer, Cham. https://doi.org/10.1007/978-3-030-55814-7_11
Download citation
DOI: https://doi.org/10.1007/978-3-030-55814-7_11
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-55813-0
Online ISBN: 978-3-030-55814-7
eBook Packages: Computer ScienceComputer Science (R0)